Data Engineer

Key Responsibilities:

• Design, build, and maintain large-scale data pipelines to ingest, process, and transform complex financial data
• Develop data ingestion frameworks to handle high-volume and high-variety data feeds
• Implement data quality control measures to detect and handle data anomalies, duplicates, and inconsistencies
• Optimize data pipeline performance for scalability, reliability, and latency
• Collaborate with data scientists to integrate machine learning models into the data pipeline and ensure seamless data flow
• Develop and maintain model serving infrastructure to support real-time predictions and batch scoring
• Implement model monitoring and logging to track performance, data quality, and model drift
• Manage cloud-based infrastructure to support data pipeline and model serving workloads
• Troubleshoot data pipeline issues, data quality problems, and model performance degradation
• Collaborate with cross-functional teams, including data science, product, and engineering teams to ensure alignment and effective communication

Required Skills, Experience, and Qualifications:

• 5+ years of experience in a data engineering or similar role, with a focus on building and deploying data pipelines and machine learning models
• 3+ years of experience with Databricks, PySpark, and Apache Spark
• 2+ year of experience with Apache Airflow, Apache Kafka, and Kubernetes
• Strong Python skills, with 3+ years of experience
• Experience with production machine learning model deployment and management
• Knowledge of cloud-based infrastructure and services AWS and Azure
• Experience with data warehousing, data lakes, and data marts
• Familiarity with containerization using Docker and orchestration using Kubernetes
• Understanding of messaging and streaming using Apache Kafka
• Experience with workflow management using Apache Airflow
• Strong problem-solving skills, with the ability to troubleshoot complex data pipeline and machine learning model issues
• Excellent communication and collaboration skills, with the ability to work with data scientists, engineers, and senior stakeholders
• Degree educated in Computer Science or relevant subject
• Excellent spoken and written English
• Experience of driving innovation within teams
• Positive, can-do approach to work, delivering on commitments
• Able to earn respect and build trust through high-quality delivery and by demonstrating value added
• Act with integrity, with the ability to make decisions, taking accountability for the consequences of these decisions
• Creative approach to problem solving, can look at existing situations and problems in novel ways and propose innovative solutions