Senior Data Engineer

About the role

You will build and operate the ingestion pipelines, the dbt transformation layer, the FHIR serialization pipelines, the FHIR-Repository integration components, and the data products that depend on them. You will work within a spec-driven development framework — every component begins with an approved specification, and your work is generated, refined, and verified against that specification.

What you'll do

• Build and maintain the ingestion pipelines — Apache Flink streaming jobs on Dataproc for HL7v2 and FHIR feeds, PySpark batch jobs for CCDA and CSV bulk loads
• Implement against the CDM_ingest_mapping shared Python library that defines SourceSpec instances per source × format combination
• Implement source-format parsers (HL7v2, CCDA, CSV, FHIR) as Python classes per the parser component spec. Write fixture-driven tests covering both well-formed inputs and…
• Implement and maintain the synchronous Informatica MDM call pattern in the ingestion path — batched calls, timeout handling, circuit breaker behavior
• Implement the asynchronous MDM event consumer that applies ECI changes to CDM

What you need

• 4–8 yrs of experience
• Python
• SQL
• dbt
• Spark
• Informatica
• Kafka
• M.Sc or B.Tech or B.E. or B.Sc or BCA or M.Tech or MCA

Good to have

• GCP
• IAM
• Snowflake
• Java
• VBA
• CI/CD