Senior Data Engineer

About the role

We are looking for a highly skilled Senior Data Engineer to contribute to the engineering, modernization, and continuous evolution of data processing platform on Databricks on AWS. While supporting the transition from the legacy Cloudera Hadoop platform to Databricks on AWS, this role will continue to play a key part in enhancing performance, simplifying pipelines, and delivering new capabilities on the Databricks platform over the long term.

What you'll do

• Platform Engineering & Modernization
• Refactor and modernize existing Spark pipelines to Databricks native architectures
• Eliminate legacy Hadoop dependencies and adopt cloud native AWS patterns
• Enhance and extend existing processing logic using optimized Spark (ScalaSpark) on Databricks
• Build and optimize solutions using Databricks features, including Delta Lake, Databricks Workflows for orchestration and Auto scaling and job clusters

What you need

• 8–14 yrs of experience
• SQL
• AWS
• Databricks
• Spark
• Data modelling
• Hadoop
• Any graduate