Data Engineer

About the role

Experienced Data Engineer to analyze, design and develop enterprise-scale data pipelines across legacy and strategic cloud data platforms. A key component of the position is analyzing existing data pipelines, particularly Omnia-based Spark pipelines, understanding the underlying business and transformation logic, and translating that logic into scalable data solutions on modern cloud platforms.

What you'll do

• Conduct detailed analysis of existing end-to-end data pipelines across multiple platforms, with a particular focus on Omnia - PySpark-based processing
• Review existing source code, SQL, transformation rules, mappings and data flows to understand the current business and technical logic
• Derive, document, and implement new transformation logic based on the analysis of existing pipelines and business requirements
• Design, develop and maintain reliable, scalable, and reusable data pipelines using AWS, Snowflake and dbt
• Develop batch and, where required, event-driven data processing solutions using cloud-native and distributed data technologies

What you need

• 1–3 years in data science & ai
• SQL
• Python
• ETL
• AWS
• Spark
• Snowflake
• Any graduate

Good to have

• dbt
• Airflow
• Data modelling
• Reconciliation
• IAM
• VBA