Databricks Engineer II (Pyspark+
ChennaiFull-time5–8 yrsPosted 1 week ago
Senior data science role at NCR in Chennai. Needs 5–8 years.
About the role
Databricks - Unity catalog, Auto loader, Spark, pySpark , Workflows and Jobs Advanced SQL - Mastery on Window function, CTE, Store procedure, joins, CTEs, aggregations, query optimization. Delta Lake — ACID transactions, time travel, schema evolution. Pyspark , Incremental pipelines, Batch data pipeline and streaming.
What you'll do
- Design and implement ELT pipelines using PySpark and Databricks Workflows
- Write efficient, reusable, and scalable Python code for data processing and automation tasks
- Orchestrate complex workflows using Azure Data Factory and integrate with other Azure services
- Collaborate with analysts and data scientists to model and deliver trusted datasets
- Monitor and troubleshoot data pipelines, ensuring high availability and reliability
What you need
- Databricks
- ETL
- Python
- Spark
- Azure Data Factory
- SQL
Good to have
- Machine learning
- Azure
- Git
- AWS
- CI/CD
- GCP