Real job — pulled straight from Jobgether’s careers page · Verified September 27, 2026 · No reposts.
Job description
Jobgether is hiring a Software Engineering Director, Agentic Evaluations — a full-time, remote role ($260k–$320k). Apply directly on Jobgether's careers page below.
Software Engineering Director, Agentic Evaluations
Team: IT
Location: US
Commitment: Full-time
Workplace Type: remote
This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Software Engineering Director, Agentic Evaluations based in United States.
This role will lead the engineering strategy and execution behind credible, scalable evaluations of AI agents offered by software vendors. You will own the core evaluation platform, from APIs and administration tools through workflow orchestration and production delivery. The position combines deep backend engineering with hands-on experience evaluating customer-facing AI agents using modern LLMs and agentic techniques. You will design reusable evaluation infrastructure that can scale across software categories and accelerate the launch of new benchmarks. Alongside building the platform, you will lead and mentor a team of engineers and serve as a technical authority on agent evaluation. This is an opportunity to shape how agent performance, accuracy, reliability, and policy adherence are measured in an AI-driven software ecosystem.
Accountabilities:
- Lead the development and continuous improvement of evaluations that run against live software agents, identifying practical approaches for producing useful, credible, and repeatable results.
- Own the full technology stack supporting the evaluation platform, including administrative tooling, APIs, evaluation workflows, orchestration, and production systems.
- Design reusable evaluation primitives and architecture that can be applied across multiple software categories and reduce the effort required to onboard new agent types.
- Turn recurring integration, onboarding, and maintenance processes into reusable AI skills, agents, or automated workflows that increase evaluation velocity.
- Monitor emerging frameworks, methodologies, and industry practices for agent evaluation, identifying opportunities to improve measurement quality and distribution of results.
- Partner closely with data science teams to develop proprietary benchmarks and richer approaches to evaluating agent performance.
- Lead, mentor, and develop a core engineering team, providing technical direction and subject-matter expertise in AI agent evaluations.
- Promote the effective use of evaluations across agent-focused products and initiatives throughout the wider organization.
- 10+ years of professional software development experience, primarily in backend or full-stack environments.
- 2+ years of direct engineering management experience, including team leadership, mentoring, and technical direction.
- Expert-level backend development skills in languages such as Python, Java/Kotlin, TypeScript/JavaScript, or Go, with strong experience in frameworks such as FastAPI or Node.js.
- Hands-on experience building evaluations for customer-facing AI agents, including the use of agent trajectory traces and evaluation rubrics to measure task completion, accuracy, correctness, and/or policy adherence.
- Direct experience using frontier models from providers such as OpenAI, Anthropic, or Google in LLM-as-a-judge applications.
- Regular experience using coding-agent tools such as Claude Code, Codex, OpenCode, or Pi as part of a modern software development workflow.
- Bachelor’s degree in computer science, engineering, or a related field.
- Experience integrating agent tool use directly or through MCP servers is a plus.
- Familiarity with benchmark frameworks such as STATE-Bench, tau2-bench, or similar evaluation approaches is desirable.
- Experience using browser automation and agent tooling such as Playwright, browser-use, or Chrome DevTools MCP is advantageous.
- Experience with durable execution and workflow platforms such as Temporal, DBOS, Cloudflare Workflows, or Vercel Workflows is helpful.
- Familiarity with agent sandboxing technologies such as AWS E2B, Daytona, Cloudflare Containers, or Vercel Containers is also valuable.
- Strong communication, technical leadership, mentoring, and cross-functional collaboration skills are essential.
- Total earnings of approximately $260,000–$320,000, combining base salary and bonus.
- Equity participation.
- Performance-based bonus opportunities.
- Fully remote position within the United States.
- Flexible working environment designed to support distributed teams.
- Unlimited paid time off.
- Generous parental leave.
- Comprehensive benefits designed to support employee well-being and flexibility.
- Inclusive and diverse workplace with employee-led community initiatives and professional growth opportunities.
Requirements
Benefits
Get Software Engineering Director, Agentic Evaluations jobs like this→
New roles from thousands of companies land hourly, straight from their careers pages. Get the freshest matches by email so you never miss one.
Email me new jobsSimilar jobs
Frequently asked questions
What is the salary for Software Engineering Director, Agentic Evaluations at Jobgether?
The estimated salary range for Software Engineering Director, Agentic Evaluations at Jobgether is $260,000 - $320,000 USD per year.
Is Software Engineering Director, Agentic Evaluations at Jobgether a remote job?
Yes, Software Engineering Director, Agentic Evaluations at Jobgether is a remote position. This role is open to remote candidates.
What skills are required for Software Engineering Director, Agentic Evaluations at Jobgether?
The required skills for Software Engineering Director, Agentic Evaluations at Jobgether include: Python, Java, Kotlin, TypeScript, JavaScript, Go, FastAPI, Node.js, OpenAI, Playwright, LLM, AI.
What is the seniority level for Software Engineering Director, Agentic Evaluations at Jobgether?
Software Engineering Director, Agentic Evaluations at Jobgether is a Director / Manager level position.
How do I apply for Software Engineering Director, Agentic Evaluations at Jobgether?
You can view the full description and apply for Software Engineering Director, Agentic Evaluations at Jobgether on EchoJobs: https://echojobs.io/job/jobgether-software-engineering-director-agentic-evaluations-ego5u.