Pyspark Engineer
ChennaiFull-time4–7 yrsPosted 1 week ago
Mid-level data science role at Match Me Resume Match | Citi in Chennai. Needs 4+ years and a B.Tech, B.E. or B.Sc.
About the role
We are seeking an experienced and results-driven PySpark / Big Data Engineer to design, develop, and maintain large-scale batch and real-time data pipelines. In this role, you will be responsible for transforming complex raw data into actionable analytical data models, optimizing distributed computing workloads, and ensuring data integrity, scalability, and security across enterprise data platforms.
What you'll do
- Data Pipeline Design & Development
- Architect, build, and maintain robust, scalable ETL/ELT pipelines using PySpark and Python to ingest data from diverse batch and streaming sources (Kafka
- Design and implement Lakehouse architectures using formats such as Delta Lake, Apache Iceberg, or Apache Hudi
- Build reusable frameworks and data ingestion libraries to standardize enterprise data processing
- Spark Performance Tuning & Optimization
What you need
- Spark
- Python
- SQL
- ETL
- Databricks
- AWS
Good to have
- Linux
- Kafka
- Snowflake
- Airflow
- Redshift
- BigQuery