Lead Data Engineer

About the role

We are seeking a self-driven Lead Data Engineer with 7–10 years of experience to build and optimize scalable ETL/ELT pipelines using Azure Databricks, PySpark, and Delta Lake. The role involves working across scrum teams to develop data solutions, ensure data governance with Unity Catalog, and support real-time and batch processing.

What you'll do

• Work with different scrum teams to develop all the quality database programming requirements of the sprint
• Create and deploy scalable ETL/ELT pipelines with Azure Databricks by utilizing PySpark and SQL
• Create Delta Lake tables with ACID transactions and schema evolution to support real-time and batch processing
• Develop unit tests to be able to test them automatically
• Use SOLID development principles to maintain data integrity and cohesiveness

What you need

• 7–10 yrs of experience
• SQL
• Databricks
• Spark
• Azure
• Unity
• Scala
• Any graduate

Good to have

• ETL
• Python
• SSIS