Lead Assistant Manager

About the role

We are hiring a Data Engineer to build and run scalable batch data pipelines on our big data platform. You will write production-grade Python, PySpark and SQL, orchestrate workflows in Airflow, and work with data stored in Hadoop and Hive. You will partner with analysts, data scientists and product teams to deliver reliable, well-modelled data that the business can trust.

What you'll do

• Design, build and maintain scalable ETL/ELT pipelines using Python, PySpark and SQL
• Develop and schedule workflows in Apache Airflow (DAG design, dependencies, retries, SLAs, alerting)
• Ingest, transform and store large datasets on Hadoop (HDFS) and expose them through Hive tables
• Design Hive schemas with sound partitioning, bucketing and file-format choices (Parquet/ORC)
• Tune PySpark jobs and SQL queries for performance and cost (partitioning, joins, caching, skew handling)

What you need

• SQL
• Python
• ETL
• Spark
• Hadoop
• Data modelling
• B.Sc or B.Tech or B.E. or BCA

Good to have

• Airflow
• AWS
• Data warehousing
• Kafka
• Hive
• Root cause analysis