Senior Data Engineer (Healthcare Data & Pipelines)

In this role, you will:

• Design and implement robust data pipelines (batch and streaming) to ingest and process healthcare data from diverse EHR systems and data sources, ensuring data reliability and correctness.
• Excel in applying medallion architecture principles, developing scalable bronze, silver, and gold layers that support advanced analytics and machine learning.
• Thrive in building and managing event-driven architectures with tools like Kafka or Kinesis, enabling real-time data processing for clinical and operational applications.
• Collaborate closely with healthcare IT teams and third-party vendors (e.g., EPIC, Cerner, Meditech) to integrate, normalize, and secure data from EHRs, HL7, and FHIR standards.
• Are committed to optimizing data pipeline performance for throughput, latency, and cost-effectiveness using best practices for scaling, partitioning, and caching.
• Prioritize compliance with HIPAA, HITECH, and other regulations, implementing strong encryption, access controls, and patient privacy measures.
• Actively partner with data scientists, analysts, and developers to design data models and interfaces that meet diverse project requirements.
• Guide and mentor junior engineers, sharing your expertise in data engineering, cloud architecture, containerization, and orchestration tools.

Requirements:

• 5+ years of experience in data engineering or software engineering, with a focus on Big Data or real-time pipelines.
• Understanding of EHR workflows, HL7, FHIR standards, and integration with major vendors like EPIC, Cerner, or Meditech.
• Proficiency in SQL and Python, with experience in frameworks such as Apache Spark, Databricks, or Hadoop.
• Hands-on experience with event-driven systems like Kafka or Kinesis, and expertise in medallion architecture for data transformations.
• Strong knowledge of ACID transaction principles and their application in data engineering workflows.
• Familiarity with cloud platforms (AWS, Azure, or GCP), and orchestration tools such as Airflow or Step Functions.
• Strong problem-solving skills, attention to detail, and excellent communication abilities.
• Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, or a related technical field.

Nice to haves:

• Experience implementing real-time streaming analytics or IoT solutions in healthcare settings.
• Familiarity with infrastructure-as-code tools like Terraform or CloudFormation.
• Knowledge of data governance frameworks, metadata management, and lineage tracking.
• Previous leadership experience in defining and executing data engineering roadmaps or managing teams.