Lead Consultant(Product/Domain)

About the role

Key ResponsibilitiesDatabricks & AWS Data Pipeline Engineering Architect end-to-end data products using the Medallion Architecture (Bronze, Silver, Gold layers) natively built on Amazon S3 and Delta Lake. Ingest diverse data streams by integrating Databricks with AWS messaging systems like Amazon Kinesis or Managed Streaming for Apache Kafka (MSK).

What you'll do

• Key ResponsibilitiesDatabricks & AWS Data Pipeline Engineering Architect end-to-end data products using the Medallion Architecture (Bronze, Silver
• Design and scale production pipelines using PySpark, SQL, and Delta Live Tables (DLT) for both high-throughput batch and real-time streaming data
• Ingest diverse data streams by integrating Databricks with AWS messaging systems like Amazon Kinesis or Managed Streaming for Apache Kafka (MSK)
• Optimize Lakehouse storage layouts by leveraging Delta features such as Liquid Clustering, Z-Ordering, and data compaction
• Manage infrastructure security by configuring security roles and personas Optimize cloud spend by managing Databricks clusters

What you need

• 6–11 yrs of experience
• SQL
• Git
• Python
• AWS
• Unit testing
• Kafka
• Any graduate

Good to have

• ETL
• CI/CD
• IAM
• Spark
• Databricks
• Unity