Senior Data Engineer

The data engineering team develops and maintains robust data pipelines, builds trusted data sources, and creates analytics/data products that inject automation into data science processes, with a goal to empower users via self-serve analytics.

What we do:

1. Trusted data sources: Build and maintain a foundational data layer (data marts), that serve as the single source of truth across Coinbase. 
2. Reliable data pipelines: Design and implement robust data pipelines, guaranteeing data quality and timely data delivery across our organization.
3. Data science developer tools: Build developer tools that inject automation into data science processes, improving efficiency and productivity. For example, data transformation, data modeling, and data quality tooling.
4. Self-serve analytics products: Deliver tailored data products, designed to empower users with self-serve capabilities and ensure accurate answers to their data inquiries.

What you’ll be doing:

Your primary responsibilities will include building tools and products that enhance data science productivity, enable self-serve analytics, and ensure data reliability and quality. Additional responsibilities may include contributing to the maintenance of critical data marts and data pipelines as needed.. 

More specifically: 

• Enforce best practices: Introduce automation mechanisms to enforce data best practices and optimal design principles.
• Data transformation tooling: Enhance our data transformation layer by adding new features and migrating data pipelines from legacy tools.
• Data quality tooling: Build systems to automate monitoring, alerting and fixing data anomalies.
• Productivity tooling: Build internal tools aimed at streamlining workflows, minimizing manual interventions and driving operational efficiencies. 
• Cross-Functional collaboration: Work alongside fellow data engineers and cross-functional partners from Data Science, Data Platform, Machine Learning, and various analytics teams to ensure alignment on priorities and deliverables. 
• Reliable data pipelines: Develop new data pipelines and ETL processes, while enhancing the efficiency of existing ones to reduce costs, improve landing times, and ensure reliability. This may involve indexing, partitioning, and tuning the underlying data models.
• Semantic layer integration: Build the semantic layer to ensure it effectively translates complex data models from the data marts into business terms. This includes defining dimensions, measures, hierarchies, and business logic.
• Self-serve analytics: Contribute to the self-serve analytics chatbot and its infrastructure by adding features, fixing bugs and promoting its use across the company. 
• Knowledge base: Maintain clear and comprehensive documentation of processes to facilitate smooth team collaboration and hand-offs.

What we look for in you: 

• Python: Must be adept at scripting in Python, particularly for data manipulation and integration tasks, with strong experience in Object-Oriented Programming (OOP).
• SQL: Must have a solid grasp of advanced SQL techniques for querying, transformation, and performance optimization.
• ETL/ELT Processes: Experience in designing, building, and optimizing ETL/ELT data pipelines to process large datasets. Experience with both batch and streaming is a plus.
• Apache Airflow: Experience in building, deploying, and optimizing DAGs in Airflow or a similar pipeline orchestration tool.
• GitHub: Experience with version control, branching, and collaboration on GitHub is a must-have.
• Data Visualization: Experience with tools like Superset, Looker, or Python visualization libraries (Matplotlib, Seaborn, Plotly…etc)
• Data Modeling: Understanding of best practices for data modeling, including star schemas, snowflake schemas, and data normalization techniques
• Collaboration and Communication: Ability to work closely with data scientists, analysts, and other stakeholders to translate business requirements into technical solutions. Strong documentation skills for pipeline design and data flow diagrams.
• Fundamental DevOps Practices: Knowledge of unit testing, CI/CD, and repository management.
• Technologies: Familiarity with Docker, Kubernetes, and cloud platforms such as AWS or GCP.
• Prompt Engineering for LLMs: Expertise in crafting, testing, and refining prompts for effective communication with LLMs, like GPT, to generate accurate and contextually relevant responses is a plus. 

Job #: P60857

#LI-Remote