Senior Data Engineer

We are seeking a highly skilled Senior Data Engineer to join our data team. The ideal candidate will have extensive experience with data engineering tools and technologies, including a strong background in Data Orchestration, Data Pipelines, Medallion architecture, and Data Governance. This is a unique opportunity to work on a core initiative, building and architecting a unique data asset store. 

Responsibilities

• Design, develop, and maintain robust and scalable data pipelines using a combination of batch and streaming technologies. 
• Implement/Implement Apache Airflow to orchestrate and automate complex data workflows and ensure timely delivery of data. 
• Optimize and tune data pipelines to improve performance and reliability.
• Design and implement data solutions following the Medallion (Bronze, Silver, Gold) architecture to provide organized, enriched, and high-quality data layers. 
• Ensure the smooth transition of data from raw to processed and curated stages while maintaining data quality standards.
• Establish and enforce data governance practices, ensuring data integrity, compliance, and best practices.
• Implementing Data Lineage, Metadata controls in collaboration with compliance requirements. 
• Collaborate with business stakeholders to define data quality metrics and implement monitoring mechanisms. Maintain documentation of data lineage, data models, and transformation processes.
• Work closely with data analysts, data scientists, and business stakeholders to understand data requirements and ensure data solutions meet business needs. 
• Guide and mentor junior data engineers and other team members, promoting a culture of best practices and collaboration within and without the organization. 

Experience and Qualifications:

• Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or related field. 
• 5+ years of experience in Data Engineering. 
• Proven experience with Apache Airflow for orchestrating and automating data workflows. 
• Hands-on experience with Medallion architecture and implementing data pipelines across multiple layers. 
• Experience working with cloud data platforms (e.g., AWS, Azure, GCP). 
• Solid experience with SQL, NoSQL and database technologies. 
• Strong programming skills in Python or general purpose programming languages.
• Experience with distributed data processing tools (e.g., Spark, Databricks, DBT). 
• Knowledge of data modeling, ETL/ELT processes, and data warehousing. Understanding of data governance principles, data cataloging, and data quality management. 
• Excellent problem-solving and analytical skills. 
• Strong communication and collaboration skills. 
• Ability to work in a fast-paced, agile environment.

Nice to Have:

• Experience with data visualization tools (e.g., Tableau, Power BI, Looker). 
• Familiarity with CI/CD practices for Data Engineering.
• Familiarity with Snowflake. 
• Experience with Terraform and AWS Manage Data Services, DynamoDB and MWAA. 

Privacy & Data Protection

When you are applying for a job at Prove, we collect and use your personal information in the job application process. To understand more about how Prove uses your personal information, please see our Recruitment Privacy Policy on our website.