Senior Data Engineer

About the role

We are hiring a Senior Data Engineer to own and scale a petabyte-scale data platform on Google Cloud Platform (GCP), processing TB scale of data every day with Apache Spark. You will lead a small team of data engineers, drive cloud cost optimisation across our data workloads, and build AI-driven workflows that change how we support and operate the platform.

What you'll do

• Build and scale the platform
• Design, build and operate highly available, fault-tolerant batch and streaming pipelines on GCP that ingest hundreds of terabytes a day into a petabyte-scale data lake…
• Write and tune large Spark workloads (PySpark, Spark SQL, Scala/Java) on Dataproc or Spark on Kubernetes (GKE): partitioning, shuffle and join optimization
• Build real-time and near-real-time pipelines with Pub/Sub or Kafka and Spark Structured Streaming or Dataflow (Apache Beam), with clear guarantees on latency
• Build data cleaning, validation and reconciliation pipelines that stay correct when source data is late, duplicated, malformed or missing

What you need

• 6–9 yrs of experience
• SQL
• Python
• Spark
• BigQuery
• Kafka
• Airflow
• Any graduate

Good to have

• dbt
• Docker
• ETL
• GCP
• Java
• System design