Data Engineer - Chennai / Bengaluru
Chennai, BengaluruFull-timeJunior levelPosted 2 months ago
Data science role at Fa Ertb Saasfaprod1 in Chennai and Bengaluru.
About the role
Develop and maintain data pipelines, ELT processes, and workflow orchestration using Apache Airflow, Python, PySpark , Hive(Trino) and Snowflake to ensure the efficient and reliable delivery of data. Design and implement custom connectors to facilitate the ingestion of diverse data sources into our platform, including structured and unstructured data from various document formats .
What you'll do
- Develop and maintain data pipelines, ELT processes, and workflow orchestration using Apache Airflow, Python, PySpark
- Design and implement custom connectors to facilitate the ingestion of diverse data sources into our platform
- Collaborate closely with cross-functional teams to gather requirements, understand data needs, and translate them into technical solutions
- Design and implement data CI/CD pipelines to enable automated and efficient data integration, transformation, and deployment processes
- Monitor and troubleshoot data pipelines, proactively identifying and resolving issues related to data ingestion, transformation, and loading
What you need
- Python
- SQL
- Airflow
- Spark
- Snowflake
- Hive
Good to have
- ETL
- CI/CD