Lead Software Engineer - Python + Spark

About the role

Design, develop, and maintain scalable data processing applications using Python, PySpark, Spark SQL, and distributed computing frameworks . Build, optimize, and support large-scale batch and real-time data pipelines using Apache Spark and cloud-native technologies. Develop high-performance ETL/ELT solutions for processing structured, semi-structured, and unstructured data from multiple sources.

What you'll do

• Design, develop, and maintain scalable data processing applications using Python, PySpark, Spark SQL, and distributed computing frameworks
• Build, optimize, and support large-scale batch and real-time data pipelines using Apache Spark and cloud-native technologies
• Develop high-performance ETL/ELT solutions for processing structured, semi-structured, and unstructured data from multiple sources
• Write clean, reusable, and efficient Python code following software engineering best practices, design patterns, and coding standards
• Optimize Spark applications through partitioning, caching, memory management, join optimization, and performance tuning techniques

What you need

• 8–12 yrs of experience
• Python
• Spark
• SQL
• ETL
• AWS
• Azure
• Any graduate

Good to have

• CI/CD
• GCP
• Data structures
• Git
• NoSQL
• REST APIs