Lead Data Engineer
HyderabadFull-time7–10 yrsPosted 4 months ago
Senior data science role at NationsBenefits in Hyderabad. Needs 7–10 years.
About the role
We are seeking a self-driven Lead Data Engineer with 7–10 years of experience to build and optimize scalable ETL/ELT pipelines using Azure Databricks, PySpark, and Delta Lake. The role involves working across scrum teams to develop data solutions, ensure data governance with Unity Catalog, and support real-time and batch processing.
What you'll do
- Work with different scrum teams to develop all the quality database programming requirements of the sprint
- Create and deploy scalable ETL/ELT pipelines with Azure Databricks by utilizing PySpark and SQL
- Create Delta Lake tables with ACID transactions and schema evolution to support real-time and batch processing
- Develop unit tests to be able to test them automatically
- Use SOLID development principles to maintain data integrity and cohesiveness
What you need
- SQL
- Databricks
- Spark
- Azure
- Unity
- Scala
Good to have
- ETL
- Python
- SSIS