Lead Software Engineer - Python + Spark
BengaluruFull-time8–12 yrsPosted 1 week ago
Leadership data science role at Societe Generale in Bengaluru. Needs 8–12 years.
About the role
Design, develop, and maintain scalable data processing applications using Python, PySpark, Spark SQL, and distributed computing frameworks . Build, optimize, and support large-scale batch and real-time data pipelines using Apache Spark and cloud-native technologies. Develop high-performance ETL/ELT solutions for processing structured, semi-structured, and unstructured data from multiple sources.
What you'll do
- Design, develop, and maintain scalable data processing applications using Python, PySpark, Spark SQL, and distributed computing frameworks
- Build, optimize, and support large-scale batch and real-time data pipelines using Apache Spark and cloud-native technologies
- Develop high-performance ETL/ELT solutions for processing structured, semi-structured, and unstructured data from multiple sources
- Write clean, reusable, and efficient Python code following software engineering best practices, design patterns, and coding standards
- Optimize Spark applications through partitioning, caching, memory management, join optimization, and performance tuning techniques
What you need
- Python
- Spark
- SQL
- ETL
- AWS
- Azure
Good to have
- CI/CD
- GCP
- Data structures
- Git
- NoSQL
- REST APIs