SPARK Data Onboarding Engineer - Chennai
ChennaiFull-time5–9 yrsPosted open since jan 2025
Senior data science role at Fa Ertb Saasfaprod1 in Chennai. Needs 5+ years and a B.Sc, B.Tech or B.E..
About the role
We are seeking a skilled PySpark Data Engineer to join our team and drive the development of robust data processing and transformation solutions within our data platform. You will be responsible for designing, implementing, and maintaining PySpark-based applications to handle complex data processing tasks, ensure data quality, and integrate with diverse data sources.
What you'll do
- Design, develop, and test PySpark-based applications to process, transform, and analyze large-scale datasets from various sources, including relational databases
- Implement efficient data transformation and aggregation using PySpark and relevant big data frameworks
- Develop robust error handling and exception management mechanisms to ensure data integrity and system resilience within Spark jobs
- Optimize PySpark jobs for performance, including partitioning, caching, and tuning of Spark configurations
- Collaborate with data analysts, data scientists, and data architects to understand data processing requirements and deliver high-quality data solutions
What you need
- Spark
- Onboarding
- ETL
- Airflow
- SQL
- NoSQL
Good to have
- Data structures