Data Engineering Lead - Python, Pyspark, Azure Databricks
GurugramFull-time6–11 yrsPosted 5 days ago
Senior data science role at UnitedHealth Group (Optum) in Gurugram. Needs 6+ years and an MCA, B.Tech or B.E..
About the role
Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale.
What you'll do
- Design, build and maintain scalable batch and streaming data pipelines on Azure and Databricks using the Medallion Architecture (Bronze, Silver, Gold) on Delta Lake
- Ingest and process structured, semi-structured and unstructured data in formats such as CSV, JSON, XML, Parquet, Avro, ORC, Excel and Delta, using Azure Data Factory
- Develop complex transformations in PySpark and Spark SQL, including CDC, SCD Type 1/2, deduplication and handling of nested or schema-evolving data
- Design data models and build the right consumption objects for each use case: managed/external tables, views, temporary views, materialized views and streaming tables
- Implement data governance, lineage and security using Unity Catalog, including row/column-level security and PHI/PII protection
What you need
- Python
- Spark
- Databricks
- Azure
- SQL
- ETL
Good to have
- Kafka
- Terraform
- Data modelling
- CI/CD
- Power BI
- Git