Senior Data Engineer
ChennaiFull-timeSenior levelPosted over a month ago
Senior data science role at Toppan Merrill in Chennai.
About the role
Senior Data Engineer at Toppan Merrill. Develop & Optimize Data Pipelines Build, test, and maintain ETL/ELT data pipelines using Azure Databricks & Apache Spark (PySpark).
What you'll do
- Develop & Optimize Data Pipelines Build, test, and maintain ETL/ELT data pipelines using Azure Databricks & Apache Spark (PySpark)
- Optimize performance and cost-efficiency of Spark jobs
- Ensure data quality through validation, monitoring, and alerting mechanisms
- Understand cluster types, configuration, and use-case for serverless
- Implement Unity Catalog for Data Governance Design and enforce access control policies using Unity Catalog
What you need
- SQL
- Python
- ETL
- Databricks
- Spark
- Azure Data Factory
Good to have
- MLOps
- Kafka
- dbt
- SQL Server
- Salesforce
- Azure