Data Engineering Lead - Python, Pyspark, Azure Databricks

About the role

Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale.

What you'll do

• Design, build and maintain scalable batch and streaming data pipelines on Azure and Databricks using the Medallion Architecture (Bronze, Silver, Gold) on Delta Lake
• Ingest and process structured, semi-structured and unstructured data in formats such as CSV, JSON, XML, Parquet, Avro, ORC, Excel and Delta, using Azure Data Factory
• Develop complex transformations in PySpark and Spark SQL, including CDC, SCD Type 1/2, deduplication and handling of nested or schema-evolving data
• Design data models and build the right consumption objects for each use case: managed/external tables, views, temporary views, materialized views and streaming tables
• Implement data governance, lineage and security using Unity Catalog, including row/column-level security and PHI/PII protection

What you need

• 6–11 yrs of experience
• Python
• Spark
• Databricks
• Azure
• SQL
• ETL
• MCA or B.Tech or B.E. or B.Sc or BCA

Good to have

• Kafka
• Terraform
• Data modelling
• CI/CD
• Power BI
• Git