Principal Data Processing Engineer-OSS
HyderabadFull-time6–11 yrsPosted 1 week ago
Senior engineering role at Datapelago in Hyderabad. Needs 6+ years and a B.Sc, M.Sc or B.Tech.
About the role
As a Principal Data Processing Engineer (OSS), you will be a key individual contributor in adopting and advancing the capabilities of open-source software (OSS) platforms such as Apache Gluten, Velox, Apache Spark, and Apache Flink in the context of DataPelago’s data processing engine. You will enhance the functional breadth, performance, scale, and reliability of the DataPelago engine through downstream and upstream contributions.
What you'll do
- Influence the architecture of how our data processing engine interfaces with open-source platforms and engines
- Lead the design of functional and performance enhancements to open source platforms such as Apache Gluten and Velox, and their integration with our data processing engine
- Individually design, implement, test, optimize, and maintain components of the data processing engine
- Analyze the technology roadmap of Apache Gluten, Velox, and equivalent platforms and identify opportunities for our engine to enhance technology and product leadership
- Foster best practices in design and code reviews, testing, CI/CD, and issue resolution to maintain the highest product quality, security, efficiency, and productivity
What you need
- Java
- C++
- Spark
- C
- Linux
- Customer success
Good to have
- CI/CD