Senior Quality Engineer (AI)

Team: IT

Location: Canada

Commitment: Full-time

Workplace Type: remote

Accountabilities:

As a Senior Quality Engineer (AI), you will own the development and execution of AI quality strategies, creating scalable validation approaches for complex and non-deterministic systems. You will collaborate with engineering, AI/ML, and data teams to ensure AI-powered solutions meet high standards of accuracy, reliability, and safety.

• Define and lead AI validation strategies for Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) systems, and AI agent workflows.
• Build and maintain automated validation pipelines integrated into CI/CD processes.
• Design evaluation frameworks measuring correctness, relevance, groundedness, consistency, hallucination rates, and overall AI output quality.
• Develop prompt regression testing approaches to identify quality degradation after model or prompt changes.
• Implement semantic validation techniques that evaluate meaning, context, and relevance beyond traditional software testing methods.
• Establish production monitoring strategies and define acceptable quality thresholds for AI-generated outputs.
• Partner with engineering teams to validate AI-generated code, automated test cases, and AI-assisted workflows.
• Introduce AI guardrails, confidence scoring mechanisms, and validation checkpoints to improve safety and reliability.
• Build quality metrics and KPIs to measure defect reduction, output consistency, stability, and user confidence.
• Contribute to the continuous improvement of AI development and validation practices across teams.

Requirements:

The ideal candidate is a technically skilled quality engineering professional with hands-on experience testing and validating AI-driven systems. You have strong expertise in automation, AI evaluation methods, and software quality practices, with the ability to collaborate effectively across technical teams.

• 7+ years of experience in Quality Engineering, Software Development, or a related technical discipline.
• Recent hands-on experience working with AI/ML systems, Large Language Models, or intelligent applications.
• Experience designing validation or evaluation frameworks for probabilistic and non-deterministic systems.
• Strong understanding of RAG architectures, agent-based workflows, and LLM evaluation concepts, including hallucination detection, relevance, groundedness, and consistency.
• Hands-on experience with LLMs and prompt engineering, including prompt design, optimization, and regression validation.
• Experience using AI-assisted development platforms or agentic coding tools such as Claude Code, Claude SDK, or similar technologies.
• Proficiency with Node.js for building automation tools, validation frameworks, and engineering utilities.
• Experience working with MongoDB for managing evaluation data, validation results, or AI output logs.
• Experience integrating automated testing and validation processes into CI/CD pipelines.
• Knowledge of AI quality metrics, confidence scoring, production monitoring, and output consistency measurement.
• Strong understanding of data validation principles, including schema validation, business rules, and data integrity.
• Excellent collaboration and communication skills with experience working across quality, engineering, AI/ML, and governance teams.
• Familiarity with prompt regression testing tools, AI governance, fairness practices, UI/API automation tools, or multi-agent systems is considered an asset.

Benefits:

• Fully remote work opportunity available in Canada.
• Flexible working hours designed to support work-life balance.
• Competitive compensation package with stock option opportunities.
• Benefits package exceeding standard requirements.
• Opportunity to work on innovative AI solutions shaping the future of technology.
• Collaborative environment with experienced professionals across engineering and AI disciplines.
• Access to mentorship, learning opportunities, and professional development support.
• Opportunity to contribute to meaningful projects involving advanced AI systems and emerging technologies.