Principal AI Quality Developer

Location: Toronto, ON, CAN

Time Type: Full time

Job Description

26WD95110

Position Overview

We are hiring a principal quality engineering leader to design, build, and operationalize testing and evaluation for applied AI and ML systems across Autodesk products, with a focus on the Fusion Industry Cloud. This role is responsible for validating the quality, safety, reliability, and performance of AI features and agentic workflows shipped by the applied AI engineering team. You will create repeatable evaluation approaches for predictive and generative AI systems, build automated test harnesses integrated into CI/CD, and partner closely with Applied AI engineers, Product, Security, and Platform teams to ensure AI capabilities meet defined acceptance thresholds and customer expectations.

Autodesk is a global leader in 3D design, engineering, and entertainment software, empowering innovators everywhere to imagine, design, and make a better world. Our software is used by professionals and hobbyists alike, across industries from architecture and manufacturing to media and entertainment. We are committed to diversity, flexibility, and continuous learning, offering a collaborative environment where you can grow your skills and make a real impact.

You will lead the quality strategy for applied AI experiences that directly impact product experience and customer outcomes. This role offers the opportunity to define how we test and trust AI features at scale, partner with senior engineering leaders, and raise the bar for reliability and responsible AI in a high-impact product environment.
 

Responsibilities

• Define and drive the quality strategy for applied AI and agentic features within Autodesk products and services with a focus on Fusion Industry Cloud

• Test planning, acceptance criteria, and release readiness for AI-powered capabilities

• Design and build automated evaluation frameworks for LLM and ML outputs, including curated gold sets, synthetic test sets, adversarial test cases, and regression suites that run in CI/CD

• Create repeatable evaluation metrics and scoring approaches for AI behavior, including relevance, faithfulness, hallucination rate, toxicity, bias, robustness, latency, and cost, and partner with Engineering and Product to define pass or fail thresholds

• Validate retrieval-augmented generation workflows, tool use, and agent behavior across edge cases, including prompt sensitivity, context recall, and retrieval accuracy

• Lead AI safety and abuse testing in partnership with Security, including prompt injection and jailbreak scenarios, data leakage risks, and misuse cases relevant to product integrations

• Establish production monitoring and post-deployment validation for AI features, including drift detection, failure taxonomy, and alerting tied to customer impact and SLAs

• Perform deep issue triage and root-cause analysis for AI-related defects across data, prompts, retrieval, model versions, and integration logic, and drive fixes with Applied AI engineers

• Mentor engineers and QA practitioners on AI testing approaches, improve team testing standards, and contribute to hiring and interview loops for quality roles

• Collaborate with cross-functional partners to ensure responsible AI practices are embedded into development workflows, including documentation of test methodology, evaluation results, and release sign-off criteria

Minimum Qualifications

• 7+ years of experience in quality engineering, test automation, or software engineering, with demonstrated ownership of test strategy and automation for complex systems, including services, platforms, or product integrations

• Strong programming skills in Python and experience with test frameworks such as pytest or unittest and building automated test harnesses integrated with CI/CD

• Hands-on experience testing AI or ML systems, including model output evaluation, error analysis, regression testing, and monitoring of quality over time

• Practical experience validating LLM-based systems, including prompt testing, retrieval-augmented generation workflows, hallucination detection, and robustness testing across edge cases

• Experience defining measurable acceptance criteria and working with product and engineering teams to translate requirements into evaluation metrics and release gates

• Proven ability to debug complex cross-stack issues spanning application logic, APIs, data flows, model behavior, and evaluation pipelines.

• Strong communication skills and ability to influence without direct authority in a collaborative, matrixed environment

• Bachelor’s degree in Computer Science, Engineering, or related field, or equivalent experience

Preferred Qualifications

• Experience with adversarial testing or AI security testing, including prompt injection, jailbreak testing, data leakage assessments, and abuse case modeling

• Familiarity with observability tooling and telemetry pipelines, including tracing and structured logging for multi-step AI workflows

• Experience with cloud platforms, AWS preferred, and containerized test environments using Docker and Kubernetes

• Experience building or maintaining evaluation datasets, including gold, synthetic, and adversarial sets, and implementing automated scoring for quality and safety

• Background in responsible AI testing, including bias evaluation, explainability approaches, and privacy considerations

• Domain knowledge in AEC, design, or manufacturing

The Ideal Candidate

• A technical quality leader who sets high standards, raises the bar through operational rigor, and delivers repeatable testing practices that scale with the organization

• Comfortable operating in a fast-paced, collaborative, and matrixed environment, partnering closely with engineers, product partners, and security stakeholders

• Demonstrates One ORBIT values by being optimistic and solutions-focused, relentless about outcomes and quality, brave in surfacing risks early, ingenious in building new evaluation methods, and trusted in follow-through and transparent communication

Learn More

About Autodesk

Welcome to Autodesk! Amazing things are created every day with our software – from the greenest buildings and cleanest cars to the smartest factories and biggest hit movies. We help innovators turn their ideas into reality, transforming not only how things are made, but what can be made.

We take great pride in our culture here at Autodesk – it’s at the core of everything we do. Our culture guides the way we work and treat each other, informs how we connect with customers and partners, and defines how we show up in the world.

When you’re an Autodesker, you can do meaningful work that helps build a better world designed and made for all. Ready to shape the world and your future? Join us!

Salary transparency

Salary is one part of Autodesk’s competitive compensation package. For Canada-BC based roles, we expect a starting base salary between $99,000 and $145,200. Offers are based on the candidate’s experience and geographic location, and may exceed this range. In addition to base salaries, our compensation package may include annual cash bonuses, commissions for sales roles, stock grants, and a comprehensive benefits package.

Diversity & Belonging
We take pride in cultivating a culture of belonging where everyone can thrive. Learn more here: https://www.autodesk.com/company/diversity-and-belonging

Are you an existing contractor or consultant with Autodesk?

Please search for open jobs and apply internally (not on this external site).