Test Analyst

Location: Medellín, Colombia

Department: Product

About the position

We’re looking for a full-time Test Analyst to join our team in Medellín, Colombia. As a Test Analyst, your primary responsibility will be to conduct thorough testing of voice-based applications, assessing usability, functionality, and performance of AI-driven patient engagement tools. You'll be at the forefront of testing innovative AI technology that directly impacts patient care, working with cutting-edge voice AI systems in a supportive, learning-focused environment.

Key Responsibilities

• Collaborate with product managers, engineers, and analysts to establish targets and processes for continuously improving system reliability, with a focus on voice interactions and AI-generated responses.
• Design, document, and execute comprehensive test plans for voice-based patient interactions, including system monitoring and data validation to ensure AI responses meet functional and non-functional requirements.
• Partner with engineers and business owners to identify, triage, and address issues with AI-driven interactions, data quality, and software performance, and then implement improvements to preempt similar defects.
• Ensure testing processes comply with healthcare data privacy requirements and patient safety standards.
• Contribute as an integral member of Agile teams, including daily stand-ups, sprint planning, and retrospectives.
• Learn and apply emerging voice testing methodologies and AI evaluation techniques, with mentorship from our engineering team.

Required Skills and Qualifications

• Bachelor's degree in technical field (e.g., Computer Science, Information Systems).
• 3-4 years of software testing experience, including automated testing and API testing.
• Experience with test management tools (Jira, TestRail) and basic scripting/programming skills.
• Strong analytical skills, with the ability to evaluate voice-based interactions, identify sources of errors, and implement improvements.
• Experience testing user interfaces and evaluating user experience quality.
• Excellent English communication skills for collaborating with distributed teams.

Desired Skills and Qualifications

• Experience with automated testing for AI-driven applications and voice UI systems.
• Experience working with voice UI testing tools and platforms, with a focus on evaluating speech generation and response quality.
• Knowledge of accessibility testing and inclusive design principles
• Interest in AI/ML testing methodologies and prompt engineering
• Interest in AI security testing, including methods for jailbreaking guardrails and adversarial testing of AI-generated responses.
• Understanding of healthcare compliance requirements (HIPAA) and patient data sensitivity.
• Proficiency in Spanish in addition to English; additional languages are a plus for testing diverse patient interactions.
• Experience working with distributed teams across time zones.

About the Company

Who we are

Third Way Health helps medical practices across the United States improve the patient experience while reducing the administrative burden on practice owners. We enable practices to enhance the experience of their patients by providing them with a leading service automation platform and a world class team of service representatives. What unites us is our passion to support physicians and provide better access for patients from all backgrounds.