Manager, Data Engineering

We are seeking an experienced and dynamic Manager of Software Engineering to lead a team of engineers responsible for designing and managing data mediation systems that support billing, usage tracking, and real-time event processing. In this role, you will oversee the development of data pipelines, ensure high performance and data integrity, and drive innovation in the automation and efficiency of mediation processes. You will collaborate with cross-functional teams, including product, billing, and engineering, to align technical solutions with business goals. 

Responsibilities: 

• Team Leadership: Lead, mentor, and develop a team of Mediation Data Engineers, fostering a culture of innovation, accountability, and technical excellence. 

• Strategic Oversight: Define and execute the roadmap for mediation systems, ensuring that the team builds scalable, reliable, and efficient data pipelines for real-time and batch data processing. 

• Technical Direction: Provide architectural guidance for the design and implementation of mediation systems, including data collection, transformation, and delivery to billing platforms. Ensure that solutions meet the performance and scalability requirements of high-throughput environments. 

• Operational Excellence: Ensure that mediation systems are running smoothly and efficiently, meeting service-level agreements (SLAs) for data processing and uptime. Implement monitoring, logging, and alerting systems to maintain data flow and catch potential issues early. 

• Cross-Team Collaboration: Collaborate closely with billing, product management, and other engineering teams to understand requirements, align mediation systems with business objectives, and drive initiatives to improve the billing process. 

• Process Automation: Drive automation of manual processes related to mediation and billing data pipelines, reducing errors and improving operational efficiency using tools like Airflow or NiFi. 

• Data Governance and Compliance: Oversee the implementation of data governance best practices, ensuring that mediation systems comply with industry standards, including data privacy regulations (e.g., GDPR, CCPA). 

• Project and Resource Management: Manage multiple projects related to mediation and billing, ensuring timely delivery within scope and budget. Allocate resources effectively and prioritize team efforts to meet business needs. 

• Continuous Improvement: Encourage continuous learning and improvement within the team by staying current with emerging technologies and promoting the adoption of best practices in data engineering and event processing. 

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field. 

• 7+ years of experience in data engineering, with at least 3 years in a managerial or team leadership role. 

• Proven experience managing teams that develop mediation systems or data pipelines for billing, usage tracking, or high-transaction environments. 

• Strong technical background in event-driven architectures and ETL processes, with hands-on experience in tools such as Apache Kafka, Flink, Spark Streaming, Airflow, or NiFi. 

• Expertise in managing large-scale data processing pipelines for real-time and batch data. 

• Strong programming skills in Python, Java, or Scala, with the ability to guide and review the technical work of team members. 

• Experience with cloud platforms like AWS, GCP, or Azure, and hands-on familiarity with their data processing services (e.g., Lambda, Glue, Dataflow). 

• Solid understanding of data governance, including data security and privacy regulations impacting mediation and billing systems. 

Preferred: 

• Experience in industries with complex mediation needs such as telecommunications, messaging, utilities, or SaaS platforms, particularly with billing and usage-based models. 

• Knowledge of mediation engines and practices used in high-transaction industries. 

• Experience with CI/CD pipelines, version control, and infrastructure automation. 

• Knowledge of data privacy regulations (e.g., GDPR, CCPA) and how they apply to large-scale data mediation and billing processes. 

Soft Skills: 

• Strong leadership and communication skills, with the ability to build and inspire high-performing teams. 

• Strategic thinker with a focus on innovation and long-term planning, balancing business needs with technical capabilities. 

• Strong problem-solving skills and ability to troubleshoot complex systems and resolve technical issues. 

• Ability to manage competing priorities and effectively allocate resources to meet business needs.