Principal Data Processing Engineer-OSS

About the role

As a Principal Data Processing Engineer (OSS), you will be a key individual contributor in adopting and advancing the capabilities of open-source software (OSS) platforms such as Apache Gluten, Velox, Apache Spark, and Apache Flink in the context of DataPelago’s data processing engine. You will enhance the functional breadth, performance, scale, and reliability of the DataPelago engine through downstream and upstream contributions.

What you'll do

• Influence the architecture of how our data processing engine interfaces with open-source platforms and engines
• Lead the design of functional and performance enhancements to open source platforms such as Apache Gluten and Velox, and their integration with our data processing engine
• Individually design, implement, test, optimize, and maintain components of the data processing engine
• Analyze the technology roadmap of Apache Gluten, Velox, and equivalent platforms and identify opportunities for our engine to enhance technology and product leadership
• Foster best practices in design and code reviews, testing, CI/CD, and issue resolution to maintain the highest product quality, security, efficiency, and productivity

What you need

• 6–11 yrs of experience
• Java
• C++
• Spark
• C
• Linux
• Customer success
• B.Sc or M.Sc or B.Tech or B.E.

Good to have

• CI/CD