Data Engineer II

About the Role:

Are you a curious, critical-thinking Data Engineer? Join our Product Data team where you’ll play a pivotal role in enhancing and architecting the strategy behind our data monitoring and quality assurance systems, as well as having a hand in the construction of ETL processes and engineering decision-making within our big data pipelines. In this dynamic position, you'll serve as both a developer and technical lead for various data pipeline and platform projects. You'll collaborate with a dedicated team to design and implement data pipelines that not only measure metrics but also monitor the quality of our systems, ensuring an unparalleled experience for our customers and internal stakeholders.

Technological Stack:

Our technology stack includes but is not limited to Airflow, Python, PySpark, BigQuery, Snowflake, Google DataProc, Starburst/Apache Trino, Apache Iceberg, AWS EMR and more. This role offers a tremendous opportunity to gain experience with a wide array of cloud-native technologies, develop sophisticated data pipelines using modern big data technologies, and lead significant initiatives on an enterprise scale with exposure to two of the top major cloud providers, AWS and GCP.

What you’ll do:

• Design and implement ETL processes to handle the ingestion of data, thinking considerately of downstream impact to our data ecosystem
• Design and implement data monitoring pipelines to proactively identify and resolve data quality issues, potentially impacting downstream products.
• Collaborate with stakeholders to define project requirements, analyze and think critically to develop metrics representing data pipeline business value, and create corresponding monitoring solutions using Python, Spark, and Airflow.
• Innovate and develop new methodologies to enhance access to trustworthy data, accelerating the value provided by the product data team.
• Own monitoring efforts for your projects from dashboard wireframe to finished pipeline

What you’ll bring:

• Bachelor’s Degree in Computer Science, Engineering, or a related STEM field, with a focus on Data Processing or Data Analysis
• A Master’s Degree is equivalent to 2 years of experience.
• A Ph.D. counts as 5 years of experience
• At least 3 years of experience in Data Engineering or a similar role, with a proven track record of working with big data pipelines and analytics
• Minimum of 2 years of hands-on experience with SQL in scalable data warehouses (e.g., Bigquery, Snowflake)
• Experience in implementing best practices for data engineering and optimizing data pipeline performance/scalability
• Proficiency in cloud technologies, preferably GCP and/or AWS
• Expertise with Apache Airflow
• 2+ years of experience with Apache Spark
• 3+ years of coding experience in Python
• Understanding of Distributed Systems and Effective Data Management
• Knowledge of commonly utilized data structures in databases and data warehouses
• Familiarity with the Decoupled Compute and Storage Paradigm
• Familiarity with containerized applications
• Familiarity with CI/CD tools (e.g., Jenkins)

Desirable Experience:

• Familiarity with Agile methodologies and CI/CD practices
• Familiarity with Infrastructure as Code (e.g., Terraform)
• Proficiency in data visualization tools (e.g., Tableau)

#LI-MH

#LI-Hybrid