Machine Learning Engineer

About the team

Our NLP team at Dialpad is a diverse and vibrant group of scientists who come from varied fields such as linguistics, computational linguistics, machine learning, computer science, material science, biology, and quantum physics. We prioritize innovation and creativity in our pursuit to push the boundaries of Natural Language Processing and Artificial Intelligence.

We deliver natural language understanding (NLU) features for transcribed speech and typed inputs. The major focus of the team currently is enhancing DialpadGPT, our in-house LLM specifically designed for the domain of business communication, which powers several features. Beyond the technical skills, we're a team that values collaboration, continuous learning, and the application of diverse perspectives to solve complex problems.

Together, our goal is to revolutionize business communications, making it more efficient, accurate, and accessible by harnessing the power of ASR, NLP and machine learning. Using a variety of tools and technologies we're driven to continuously improve and refine our products, delivering top-tier solutions to our customers. 

Your role

As a Machine Learning Engineer at Dialpad, you'll be an integral part of our NLP team, working on our in-house LLM, DialpadGPT and other LLMs and addressing the unique challenges of training large models on big data across third party and on-prem distributed infrastructure. You will work on maintaining a robust multi-node training infrastructure and clusters, optimizing distributed training workflows including data loading and logging, debugging complex system errors or hardware failures, and ensuring efficient utilization of resources.

If you're passionate about large scale machine learning, generative AI, and contributing to a team that's changing the face of business communications, you'll find yourself right at home with us.

This position reports to the Manager of the NLP team and has the opportunity to be based in our office in Vancouver, BC or Kitchener, ON. 

What you’ll do [i,e., Responsibilities]

• Enhance LLM training efficiency by optimizing scripts and architectures, leveraging CUDA and advanced GPU acceleration techniques to improve performance and reduce training time.
• Optimize preprocessing pipelines by resolving timeout issues and implementing caching strategies.
• Ensure reliable distributed networking by addressing connection failures and implementing monitoring systems.
• Minimize downtime on rental machines through efficient recovery workflows and off-hour error mitigation.
• Streamline debugging of distributed systems, addressing node failures and network partitions.
• Develop scalable logging frameworks for efficient error detection and resolution.
• Optimize system scalability by balancing workloads and reducing communication overhead.
• Design fault-tolerant systems to handle node crashes and ensure seamless job restarts.
• Monitor system health and implement recovery strategies for unresponsive nodes.
• Manage massive data sets for large-scale model training workflows.
• Develop real-time monitoring frameworks for training processes to detect and alert issues promptly.

Skills you’ll bring [i,e., Qualifications]

• Master’s or PhD degree in Computer Science, Machine Learning, or a related field.
• 3+ years of experience in distributed systems or machine learning infrastructure.
• Strong proficiency in Python and experience with distributed ML frameworks (e.g., PyTorch, TensorFlow, Horovod, or Ray).
• Understanding of networking protocols and distributed communication libraries (e.g., NCCL, gRPC).
• Hands-on experience with cloud platforms (AWS, GCP, Azure) and cluster orchestration tools (Kubernetes, Slurm).
• Proven ability to debug and resolve issues in large-scale distributed systems.
• Familiarity with fault tolerance, caching strategies, and scalable logging systems.
• Excellent problem-solving and communication skills.

Dialpad benefits and perks

Equity, balance, and belonging

At Dialpad, we strive to ensure that Dialers are free to bring their full selves to work each day, and champion the intersectionality that exists between gender identity, ethnicity, age, disability status, and the many other aspects of our greater humanity.

Culture

We’ve been named a Top Workplace seven times, and a big part of this is because of our collaborative culture that elevates our teammates, celebrates wins, and brings together passion and talent. 

Compensation and equity

Teamwork makes the dream work, and Dialpad offers competitive salaries in addition to stock options because each and every Dialer participates in our success.

Benefits, time-off, and wellness

An apple a day keeps the doctor away—and it doesn’t hurt that we offer flexible time off and great options for medical, dental, and vision plans for all employees. Along with that, employees also receive a stipend to help cover your cell phone bill, home internet bill, home office equipment, gym membership costs, a variety of wellness events, and more!

Continuing education

Dialpad offers a yearly stipend for continued learning and education expenses.