Senior Data Engineer, Services

As a Senior Data Engineer, you will play an instrumental role in advancing our core machine learning solution. Your technical expertise and leadership skills will be leveraged to drive the development and deployment of high-quality, scalable data pipelines and products.  You will work alongside scientists and engineers in a collaborative environment, contributing features and data pipelines to power our core recommender systems and our DaVinci Personalization product.  This is an opportunity to work end-to-end on a large-scale machine-learning system that touches millions of customers, and a chance to continuously learn and help improve our solution as the field evolves.

Responsibilities:

• Release features that deliver measurable and tangible business value.
• Implement production data products and pipelines that are scalable, reliable, and of high quality.
• Keep data pipelines up to date with a continuously updated series of services.    
• Maintain data sources and performance of an enterprise scale Data Lake.
• Continuously improve data infrastructure for greater scalability.

Qualifications:

• 5+ years of data engineering experience
• Experience with large-scale data processing frameworks (we use PySpark, SQL)
• Expertise in Spark DataFrame API
• Experience with event stream data (we use Kafka)
• Strong software development skills in Python (unit testing, git, code review, CI/CD)
• Experience with cloud computing platforms and cluster configuration, optimization, and scaling (GCP)
• Experience in data storage formats (we use Parquet, Delta Lake)
• Ability to collaborate with technical partners – you’ll be working closely with other teams to determine requirements for your work and to make design decisions that affect our stack 
• Enjoys working in a fast-paced, goal-driven environment