Sr. Data Engineer

Location: UNAVAILABLE, UNAVAILABLE, US

Employment Type: FULL_TIME

Overview

Manages and evolves the enterprise data lake and data warehouse while ensuring the reliable, secure, and efficient flow of high-quality data. Implements data processes, managing data architecture, designing ETL processes, and analyzing data for business insights.

 

The base salary range for this position is $99,937-$157,044.

 

Actual pay will be determined based upon a candidate’s job-related knowledge, skills, education, experience, geographic location, and may include other job-related factors such as certification(s), professional licensure, or internal equity considerations.

Responsibilities

 

• Design, implement and maintain scalable data pipnes on WS using S3, DMS, Glue, lambda, step function/MWAA & Redshift.
• Develop robust batch and near-real-time ETL/ELT workflow to ingest, cleanse, transform and load data from databases, legacy applications and event streams using Python & Pyspark.
• Design incremental/CDC mechanism, including restart ability, idempotency, duplicate handling and recovery.
• Implement automated controls for completeness, accuracy, reconciliation, schema changes & lineage.
• Optimize Glue/Spark, Athena, Redshift & S3 workload through partitioning, columnar formats, query tuning and appropriate storage/compute design.
• Design near real time/event-driven pipelines using Kinesis/Kafka where required, covering ordering, retry, idempotency and failure recovery.
• Implement AWS data security, least privilege access, data classification and governance controls.
• Monitor pipelines such as CloudWatch, troubleshoot failure and resolving production data incidents.
• Enforce Git/version control, code review, automated testing and CI/CD practices.
• Work with product owners, architect, reporting and business stakeholders to translate requirements into scalable data solutions.
• Document pipelines and operational procedures.

 

Qualifications

Qualifications Required

• Bachelor's degree in a computer-related field from an accredited college or university and five (5) or more years of experience in data engineering, building scalable and distributed ETL data pipelines in enterprise environments.
• Experience building and operating scalable AWS-based data platforms and pipelines using services including Lambda, Glue, Athena, S3, Redshift, DMS, MWAA (Airflow), and Step Functions, supporting batch, CDC, and near real-time data processing.
• Advanced proficiency in Python, SQL, and PySpark with hands-on experience developing reusable ETL/ELT frameworks, data warehouses, data marts, and integrations across databases, APIs, event streams, and analytics environments.
• Experience implementing data quality, governance, and optimization best practices, including automated validation frameworks, Lake Formation and Glue Data Catalog, performance tuning, and cost optimization across AWS data services.
• Strong communication skills with the ability to translate complex data concepts for business stakeholders; experience in healthcare, life sciences, and other highly regulated environments with HIPAA, GDPR, FDA, or similar compliance requirements preferred.
• Experience with metadata management, data lineage, data observability, master data management, or enterprise data catalog solutions.
• Knowledge with data modeling & analytical data model, schema design, schema evolution, and data structure optimized for reporting and analytics.
• Knowledge of data lake and data warehouse architecture include data partitioning and columnar storage format such as Parquet.
• Relevant AWS certification, such as AWS Certified Data Engineer – Associate, or an equivalent cloud or data engineering certification.

 

WORKING CONDITIONS

• Flexible work hours in fun collaborative environment
• Working remote requires a reliable internet connection
• Must have the ability to travel, as needed for company meetings