Pyspark developer
PuneFull-time18–20 yrsPosted 3 months ago
Leadership data science role at Aligned Automation Services in Pune. Needs 18–20 years.
About the role
We are seeking a highly skilled Senior PySpark Data Engineer with 7–8 years of experience in designing, developing, and optimizing large-scale data engineering solutions. The ideal candidate should have extensive experience with PySpark, Python, SQL, Databricks, cloud platforms , and modern data architectures.
What you'll do
- Design, develop, and maintain scalable ETL/ELT pipelines using PySpark
- Build high-performance data pipelines to process large volumes of structured and semi-structured data
- Develop reusable frameworks for data ingestion, transformation, validation, and monitoring
- Optimize Spark jobs by tuning partitions, joins, caching, and memory configurations
- Design and implement Data Lake/Lakehouse architectures using Delta Lake
What you need
- Spark
- SQL
- Python
- Databricks
- AWS
- Airflow
Good to have
- Azure
- ETL
- CI/CD
- Git
- SQL Server
- PostgreSQL