Senior AI Software Developer in Test

Team: Data, AI, & Interoperability Platform - QA

Location: Bogotá, Colombia, Medellin

Commitment: Full Time - Permanent

Workplace Type: remote

What You’ll Be Doing

• Quality & AI-First Mindset 
• Evolve a modern, AI-first quality strategy for our fast-scaling SaaS architecture, including foundational infrastructure and emerging agentic/intelligent systems. 
• Integrate AI enhancements into CI/CD pipelines (e.g., predictive flakiness detection, automated test generation, self-healing scripts) to improve isolation, data setup, & execution reliability using existing/suggesting tools. 
• Establish scalable testing practices that support hyper-growth and petabyte-scale AI data pipelines. 


• AI-Focused Test Strategy, Automation & Evaluation 
• Design deterministic and statistical testing approaches for non-deterministic LLM-based and agentic systems, addressing hallucinations, prompt injection, bias, drift, and safety risks. 
• Build automated evaluation pipelines and harnesses for correctness, faithfulness, retrieval quality, generation accuracy, tool-calling, planning sequences, and multi-agent flows. 
• Execute/Develop test frameworks for the full AI lifecycle: prompts, datasets, embeddings, model versions, RAG pipelines (end-to-end validation), and guardrails. 
• Implement red-teaming, bias/fairness checks, and compliance mechanisms; leverage in trend frameworks for metrics and observability. 
• Integrate AI-specific quality signals into CI/CD for automated gating and continuous monitoring. 


• Cross-Functional & End-to-End Testing 
• Partner closely with product, data science, AI engineering, and dev teams to test AI features, conduct multi-agent simulations, and ensure high-quality roadmap delivery. 
• Facilitate knowledge sharing and upskilling on AI testing best practices across the Quality Function. 


• Metrics, Observability & Continuous Improvement 
• Drive core metrics (DORA, test coverage/effectiveness) plus AI-specific indicators (e.g., hallucination rate, context precision, drift detection). 
• Build real-time dashboards and support A/B testing of models with post-deployment monitoring. 


• Culture, Mentorship & Innovation 
• Champion a quality-first, ethical AI mindset organization-wide. 
• Mentor SDET’s, lead workshops on AI risks/validation, and influence design/deploy/incident processes. 
• As a foundational hire, define roadmaps and best practices for sustainable AI quality assurance. 

Challenges You'll Tackle

• Ensuring reliability in agentic systems amid data drift and non-deterministic behavior. 
• Scaling tests for global SaaS while maintaining low hallucination rates and strong safety guardrails. 
• Building evaluation from scratch in a rapidly evolving landscape (e.g., multi-modal, agentic flows). 

Success in the First 6 Months

• Launch foundational AI test frameworks and pipelines, achieving 80-90% coverage for key AI components. 
• Reduce AI-related defect escapes by 30-40% and integrate automated safety/compliance checks into all releases. 
• Establish metrics dashboards and evaluation loops that enable data-driven iteration on intelligent features. 

What You Will Bring

• 7+ years in Quality Engineering/SDET roles within cloud-native SaaS environments, including 2+ years hands-on with AI/ML/LLM systems. 
• Expertise in automated testing infrastructure, CI/CD (Jenkins/GitHub Actions), and test pyramid strategies (unit → E2E). 
• Strong full-stack testing experience (frontend/backend/API) and collaboration with dev teams. 
• Proven experience testing LLMs, AI agents, RAG pipelines, and related risks (hallucinations, prompt injection, bias, drift). 
• Proficiency in JS/TS, working knowledge of Python or Java; experiance with AI evaluation frameworks (e.g., Ragas, DeepEval, LangChain/LangSmith/LangFuse) and other tools you may have proficiency in. 
• Knowledge of performance, Stress and Load testing tools like K6, JMeter, Blazemeter will be nice to have. 
• Knowledge of observability (NewRelic), statistical testing methods, red-teaming, and ethical AI practices. 
• Excellent communication, and coaching skills; ability to thrive in ambiguity and drive innovation. 
• Bachelor's/Master's in Computer Science, AI, or related; certifications (e.g., ISTQB AI Testing) a plus. 
• Strong English language communication and collaboration skills


We value adaptability in this fast-moving field—equivalent experience and a strong portfolio (e.g., open-source contributions, case studies) are highly regarded. 

Perks & Benefits

• Contrato a termino Indefinido with all the legal benefits
• Prepaid Medicine
• Life insurance and funeral assistance
• Internet allowance
• Home office stipend
• Competitive compensation — above the market average
• 100% remote work environment and an excellent work-life balance
• Opportunity to work for a growing global SaaS leader company
• A culture that promotes independence, innovation, trust, and accountability
• Open space to be creative, innovative, and strategize for the future
• Mentorship by a highly experienced professional 
• Budget for training, we want you to grow
• 5 Personal Time Off days per year
• Sick Leave Top up to total 100% of salary paid by the employer from Day 3 to 90. 
• Recognition Award, additional paid time off in recognition of the corresponding year of service
• Upgrade vacation starting at 5 years of service