Intermediate/Senior Test Engineer Automation - OP01926

Location: São Paulo, State of São Paulo, Brazil, Buenos Aires, Buenos Aires, Argentina

Department: OP

Workplace: remote

Employment Type: full

Description

🟢 Are you in Brazil or Argentina? Join us as we actively recruit in these locations, offering a comfortable remote environment. Submit your CV in English, and we'll get back to you!

We invite an experienced and proactive Test Automation Engineer with strong Python and manual testing skills to join our team. In this role, you’ll design and implement robust testing solutions for cutting-edge AI products, focusing on improving RAG flows and testing LLM integrations to ensure reliability, safety, and performance.

🟩 What's in it for you:

• Work hands-on on test automation for AI-powered solutions that have real-world impact
• Grow your expertise in AI safety and RAG/LLM testing
• Collaborate with a team of experienced engineers and QA professionals in a modern, innovation-focused environment

✅ Is that you?

• 3+ years of experience in automation testing
• Strong programming skills in Python
• Practical experience with pytest or similar Python testing frameworks
• Experience developing and maintaining test automation frameworks
• Solid understanding of test design, strategy, and QA best practices
• Hands-on experience with CI/CD pipelines and tools (Jenkins, GitLab CI)
• Understanding of RAG pipelines and testing AI/ML-based systems
• Familiarity with AI safety testing (prompt validation, hallucination checks, response consistency)
• Familiarity with API (REST, HTTP) testing and data validation
• Good knowledge of Git, test reporting, and bug tracking tools
• Strong analytical and problem-solving skills
• Upper-Intermediate English level

Desirable:

• Experience with major cloud platforms (GCP, AWS, Azure)
• Knowledge of containerization tools (Docker, Kubernetes)

🧩Key responsibilities and your contribution

In this role, you'll develop and execute automated tests to ensure the quality and reliability of AI-driven products.

• Design, develop, and maintain automation frameworks and test strategies for AI products
• Evaluate and improve RAG flows, ensuring accuracy and performance
• Test LLM integrations with prompt validation, hallucination checks, and response consistency checks
• Prepare test data and clarify testing requirements as needed
• Create and run automated test suites for APIs, data pipelines, and model integrations
• Integrate automated tests into CI/CD pipelines to maintain quality and stability
• Analyze test results, report defects, and work with teams to resolve issues
• Participate in code reviews, sprint ceremonies, and QA planning sessions
• Maintain clear test documentation and provide regular updates to stakeholders
• Continuously improve testing tools, processes, and automation efficiency

🎾 What's working at Dev.Pro like?

Dev.Pro is a global company that's been building great software since 2011. Our team values fairness, high standards, openness, and inclusivity for everyone — no matter your background

🌐 We are 99.9% remote — you can work from anywhere in the world
🌴 Get 30 paid days off per year to use however you like — vacations, holidays, or personal time
✔️ 5 paid sick days, up to 60 days of medical leave, and up to 6 paid days off per year for major family events like weddings, funerals, or the birth of a child
⚡️ Partially covered health insurance after the probation, plus a wellness bonus for gym memberships, sports nutrition, and similar needs after 6 months
💵 We pay in U.S. dollars and cover all approved overtime
📓 Join English lessons and Dev.Pro University programs, and take part in fun online activities and team-building events

Our next steps:

✅ Submit a CV in English — ✅ Intro call with a Recruiter — ✅ Internal interview — ✅ Client interview — ✅ Offer

Interested? Find out more:

📋How we work

💻 LinkedIn Page

📈 Our website

💻IG Page