SPARK Data Onboarding Engineer - Chennai

About the role

We are seeking a skilled PySpark Data Engineer to join our team and drive the development of robust data processing and transformation solutions within our data platform. You will be responsible for designing, implementing, and maintaining PySpark-based applications to handle complex data processing tasks, ensure data quality, and integrate with diverse data sources.

What you'll do

• Design, develop, and test PySpark-based applications to process, transform, and analyze large-scale datasets from various sources, including relational databases
• Implement efficient data transformation and aggregation using PySpark and relevant big data frameworks
• Develop robust error handling and exception management mechanisms to ensure data integrity and system resilience within Spark jobs
• Optimize PySpark jobs for performance, including partitioning, caching, and tuning of Spark configurations
• Collaborate with data analysts, data scientists, and data architects to understand data processing requirements and deliver high-quality data solutions

What you need

• 5–9 yrs of experience
• Spark
• Onboarding
• ETL
• Airflow
• SQL
• NoSQL
• B.Sc or B.Tech or B.E. or BCA

Good to have

• Data structures