Data Engineer

About the role

We are looking for a hands-on Senior Data Engineer with strong expertise in Python, PySpark and Apache Spark to develop, troubleshoot and optimise production-grade distributed data-processing workloads. Spark engineering role, not a general data engineering or orchestration role.

What you'll do

• PySpark Development and Distributed Processing
• Design, develop, test and maintain production-grade PySpark applications using DataFrame APIs and Spark SQL
• Implement complex transformations, joins, aggregations, window functions, deduplication and incremental processing
• Translate data requirements into efficient, scalable solutions, considering data characteristics, edge cases and execution costs
• Spark Internals, Optimisation and Troubleshooting

What you need

• 3–5 yrs of experience
• Python
• SQL
• Spark
• Data structures
• Reconciliation
• Any graduate

Good to have

• Kafka
• Databricks