Senior Data Engineer
MumbaiFull-time4–8 yrsPosted over a month ago
Senior data science role at Quantiphi in Mumbai. Needs 4+ years and an M.Sc, B.Tech or B.E..
About the role
You will build and operate the ingestion pipelines, the dbt transformation layer, the FHIR serialization pipelines, the FHIR-Repository integration components, and the data products that depend on them. You will work within a spec-driven development framework — every component begins with an approved specification, and your work is generated, refined, and verified against that specification.
What you'll do
- Build and maintain the ingestion pipelines — Apache Flink streaming jobs on Dataproc for HL7v2 and FHIR feeds, PySpark batch jobs for CCDA and CSV bulk loads
- Implement against the CDM_ingest_mapping shared Python library that defines SourceSpec instances per source × format combination
- Implement source-format parsers (HL7v2, CCDA, CSV, FHIR) as Python classes per the parser component spec. Write fixture-driven tests covering both well-formed inputs and…
- Implement and maintain the synchronous Informatica MDM call pattern in the ingestion path — batched calls, timeout handling, circuit breaker behavior
- Implement the asynchronous MDM event consumer that applies ECI changes to CDM
What you need
- Python
- SQL
- dbt
- Spark
- Informatica
- Kafka
Good to have
- GCP
- IAM
- Snowflake
- Java
- VBA
- CI/CD