07.Data Engineer

About the role

This platform is vital to support our increasing business demands. The Data Pipeline team is tasked with creating, deploying, and managing this platform, utilizing leading technologies like Databricks, Delta Lake, Spark, PySpark, Scala, and the AWS suite.

What you'll do

• Create and manage robust data ingestion pipelines leveraging Databricks, Airflow, Kafka, and Terraform
• Ensure high performance, reliability, and efficiency by optimizing large-scale data pipelines
• Develop data processing workflows using Databricks, Delta Lake, and Spark technologies
• Maintain and improve the Data Lakehouse, utilizing Unity Catalog for efficient data management and discovery
• Construct automation, frameworks, and enhanced tools to streamline data engineering workflows

What you need

• SQL
• Python
• Databricks
• Spark
• Airflow
• LLMs

Good to have

• Terraform
• Scala
• ETL
• Kafka
• AWS