Data Engineer with Kafka - Pune

About the role

We are looking for an experienced Senior Data Engineer with strong expertise in Apache Kafka, Apache Spark, Scala/PySpark, SQL, and modern data engineering technologies . The ideal candidate should have hands-on experience designing, developing, optimizing, and maintaining scalable data pipelines, including batch and real-time streaming solutions.

What you'll do

• Design, develop, and maintain scalable, reliable, and high-performance data pipelines for batch and real-time data processing
• Implement and optimize Kafka-based event streaming solutions , ensuring reliable data ingestion, processing, and integration
• Develop production-grade data processing applications using Apache Spark and Scala/PySpark
• Optimize Spark workloads through effective partitioning, caching, skew handling, join optimization, and performance tuning
• Write advanced SQL queries involving complex joins, window functions, data transformations, and query optimization

What you need

• 5–9 years in data science & ai
• Kafka
• ETL
• Spark
• SQL
• Python
• AWS
• M.Sc or B.Tech or B.E. or B.Sc or BCA or M.Tech or MCA

Good to have

• Airflow
• Azure Data Factory
• Data modelling
• Scala
• Root cause analysis
• Test automation