Sr. Lead Engineer - Data Analytics

Duties and Responsibilities

• 
  
Platform Design and Architecture:
• Lead the design and implementation of a robust, scalable, and high-performance data analytics platform capable of handling large-scale data workloads.
• Architect and optimize data pipelines for both real-time and batch processing, ensuring data is delivered accurately and efficiently across the organization.
• Evaluate and select appropriate technologies and frameworks (e.g., Spark, Flink, Kafka, Presto) to build a modern data analytics platform that meets business needs.
Scalability and Performance Optimization:
• Identify and address performance bottlenecks in the data platform, ensuring the system can scale to meet the growing data demands of the business.
• Implement data partitioning, sharding, and indexing strategies to optimize query performance and reduce latency.
• Ensure the platform is designed for high availability, reliability, and fault tolerance, minimizing downtime and ensuring business continuity.
Data Management and Integration:
• Collaborate with software engineering, data engineering, and analytics teams to integrate data sources into the platform and ensure data quality and consistency.
• Develop and enforce best practices for data governance, data security, and data management within the platform.
• Implement data cataloging and metadata management tools to improve data discoverability and usability across the organization.
Collaboration and Leadership:
• Lead cross-functional teams to deliver data-driven solutions that support business objectives and drive strategic initiatives.
• Provide technical leadership and mentorship to junior engineers, fostering a culture of continuous learning and improvement.
• Collaborate with product managers, data scientists, and other stakeholders to understand data requirements and translate them into scalable solutions.
Innovation and Continuous Improvement:
• Stay up to date with the latest advancements in data engineering, analytics, and cloud technologies, and apply them to enhance the platform.
• Drive continuous improvement initiatives to streamline data processing workflows, reduce costs, and improve the overall efficiency of the data platform.
• Contribute to the development of internal tools and frameworks that enhance the capabilities and performance of the data analytics platform.

Qualifications and Experience

• Bachelor’s or Master’s degree in Computer Science, Information Systems, or a related field.
• 10+ years of experience in data engineering, data platform engineering, or a similar role, with a strong focus on building and scaling data analytics platforms.
• Proven experience with big data technologies (e.g., Apache Spark, Apache Flink, Apache Kafka) and cloud-based data platforms (e.g., AWS, Google Cloud, Azure).
• Expertise in designing and managing large-scale data pipelines and distributed data processing systems.
• Strong knowledge of SQL and NoSQL databases, data lake solutions (e.g., Apache Iceberg, Big Query), and ETL processes.
• Proficiency in programming languages such as Go, Java, or Scala for building data applications and automating workflows.
• Experience with containerization and orchestration tools (e.g., Docker, Kubernetes) for deploying and managing data services.
• Excellent problem-solving and analytical skills, with a keen attention to detail.
• Strong communication skills, with the ability to articulate complex technical concepts to both technical and non-technical stakeholders.
• Leadership abilities, with experience mentoring and guiding engineering teams.

Desired Skills and Experience

• Experience with real-time data streaming technologies and frameworks.
• Knowledge of data governance, data quality, and master data management best practices.
• Familiarity with machine learning pipelines and integration with data analytics platforms.