Senior Data Engineer
Chennai, PuneFull-time8–14 yrsPosted 4 days ago
Leadership data science role at Match Me Resume Match | Citi in Chennai and Pune. Needs 8+ years.
About the role
We are looking for a highly skilled Senior Data Engineer to contribute to the engineering, modernization, and continuous evolution of data processing platform on Databricks on AWS. While supporting the transition from the legacy Cloudera Hadoop platform to Databricks on AWS, this role will continue to play a key part in enhancing performance, simplifying pipelines, and delivering new capabilities on the Databricks platform over the long term.
What you'll do
- Platform Engineering & Modernization
- Refactor and modernize existing Spark pipelines to Databricks native architectures
- Eliminate legacy Hadoop dependencies and adopt cloud native AWS patterns
- Enhance and extend existing processing logic using optimized Spark (ScalaSpark) on Databricks
- Build and optimize solutions using Databricks features, including Delta Lake, Databricks Workflows for orchestration and Auto scaling and job clusters
What you need
- SQL
- AWS
- Databricks
- Spark
- Data modelling
- Hadoop