Lead Data Engineer

About the role

Secure Talent Lake connectivity : secure pipelines that let customers ETL their Eightfold data into their own data warehouses. Today we run on Redshift and Databricks, and we’re migrating to StarRocks across 8+ regions distributed globally on AWS, Azure, and Oracle Cloud. As Lead Platform / Data Engineer, you’ll play a central role in shaping this migration and the multi-cloud platform that comes out of it.

What you'll do

• Architect and manage the lakehouse. Design and maintain Apache Iceberg tables, including schema evolution, partitioning strategies, snapshot management, compaction
• Own the analytical query layer. Deploy, operate, and performance-tune open-source datawarehouse clusters, covering FE/BE sizing, materialized views, query profiling
• Optimize storage formats. Make deliberate choices about Parquet file layout, including row group sizing, compression codecs, encoding, and predicate pushdown
• Build and maintain ETL/ELT pipelines. Develop robust batch and streaming pipelines with strong guarantees around data quality, idempotency, backfills, and observability
• Run event-driven autoscaling on Kubernetes. Design and implement KEDA-based scaling for data workloads, triggered by queue depth, Kafka lag, or custom metrics

What you need

• ETL
• AWS
• Kafka
• Spark
• Azure
• Kubernetes

Good to have

• Terraform
• Redshift
• Databricks
• Helm
• Prometheus
• Grafana