Lead II - Data Engineering_AWS Databricks
PuneFull-time7–9 yrsPosted 1 week ago
Senior data science role at UST in Pune. Needs 7–9 years.
About the role
We are looking for strong Lead Databricks Developer with streaming expertise.
What you'll do
- Design, develop, and lead enterprise-scale data platforms and pipelines using Databricks, PySpark, Delta Lake, and Spark SQL
- Build and optimize scalable batch and real-time data ingestion solutions using Kafka, Event Hubs, Auto Loader, Structured Streaming, and Delta Live Tables (DLT)
- Develop and manage Lakehouse architectures leveraging Medallion (Bronze, Silver, Gold) design patterns and data engineering best practices
- Implement and govern enterprise data assets using Unity Catalog, including data lineage, metadata management, security policies, and access controls
- Collaborate with architects and cloud teams to build and deploy data solutions on AWS and Azure, leveraging services such as S3, ADLS, Event Hubs, IAM
What you need
- AWS
- Databricks
- Spark
- LLMs
- MLOps
- Kafka
Good to have
- Terraform
- IAM
- CI/CD
- Data privacy
- Azure