Senior Data Engineer

About the role

Senior Data Engineer at Toppan Merrill. Develop & Optimize Data Pipelines Build, test, and maintain ETL/ELT data pipelines using Azure Databricks & Apache Spark (PySpark).

What you'll do

• Develop & Optimize Data Pipelines Build, test, and maintain ETL/ELT data pipelines using Azure Databricks & Apache Spark (PySpark)
• Optimize performance and cost-efficiency of Spark jobs
• Ensure data quality through validation, monitoring, and alerting mechanisms
• Understand cluster types, configuration, and use-case for serverless
• Implement Unity Catalog for Data Governance Design and enforce access control policies using Unity Catalog

What you need

• SQL
• Python
• ETL
• Databricks
• Spark
• Azure Data Factory
• Any graduate

Good to have

• MLOps
• Kafka
• dbt
• SQL Server
• Salesforce
• Azure