QA Engineer

Location: Ahmedabad, India

Department: AI Division

Experience: 3

Skills: Python, Generative AI, QA

• Design test strategies for LLM-powered features, agents, RAG systems, and workflow automations (n8n, custom pipelines, MCP-based systems)
• Build and maintain golden datasets, evaluation suites, and prompt regression tests so model or prompt changes don't silently break production
• Validate output quality across factual accuracy, format adherence, tone, hallucination rate, instruction-following, and refusal behavior
• Test agentic and multi-step workflows end-to-end tool calls, handoffs, error recovery, loop detection, cost and latency under load
• Stress-test RAG pipelines: retrieval accuracy, chunking quality, context relevance, citation correctness
• Run adversarial testing prompt injection, jailbreak attempts, malformed inputs, edge-case client data

• Own end-to-end QA for custom-built applications: dashboards, AI CRMs, client portals, and internal tools
• Functional, regression, cross-browser, and responsive testing across web UIs (Next.js, React, similar stacks)
• API testing for REST endpoints, webhooks, and third-party integrations (CRMs, email, calendars, payment, file storage)
• Database and data-integrity testing especially for multi-tenant architectures where one client's data must never leak to another
• Auth and access-control testing: SSO, OAuth, role-based permissions, session handling
• Performance, load, and stability testing on production-bound features
• File upload, processing, and storage flows a recurring failure point in AI products

• Monitor production systems for drift, degradation, errors, and cost/token anomalies; set up observability and alerting
• Collaborate with AI engineers, full-stack developers, prompt designers, and delivery leads to catch issues before they hit clients
• Automate testing in Python and/or JavaScript; integrate into CI/CD pipelines
• Document findings, build QA dashboards, and contribute to E2M's internal AI QA playbook

• Bachelor's degree in Computer Science, IT, Engineering, or related field
• 3+ years QA experience covering both web applications and AI-enabled products, with at least 1 year hands-on testing LLM-powered features, AI workflows, or agent systems
• Strong Python skills for evaluation harnesses and automation; working JavaScript/TypeScript familiarity for web app testing
• Hands-on experience with at least one LLM evaluation framework: Promptfoo, DeepEval, RAGAs, LangSmith, Braintrust, or similar
• Web app testing experience with Playwright, Cypress, or Selenium
• API testing with Postman, REST Assured, or equivalent
• Solid understanding of prompt engineering, prompt regression testing, and golden-dataset design
• Database fundamentals (SQL, basic schema reasoning) enough to verify data flows and spot multi-tenancy bugs
• Familiarity with LLM observability tools (Langfuse, Helicone, LangSmith) and token/cost/latency monitoring
• Git, CI/CD principles, and at least one cloud platform (AWS, GCP, or Azure)
• Working knowledge of vector databases, embeddings, and how RAG systems fail

• Experience testing multi-tenant SaaS platforms or client-facing custom apps
• Workflow automation platforms (n8n, Make, Zapier) or agent frameworks
• Exposure to MCP (Model Context Protocol) servers or tool-calling architectures
• Familiarity with Supabase, Postgres, or similar backends commonly used in our stack
• Experience with Claude, OpenAI, or open-source LLM APIs in production
• Agency or client-services background you understand what "client-ready" means
• Prompt injection / red-teaming experience
• SOC 2 or compliance-testing exposure

• Sharp critical eye you assume AI output (and the app around it) is broken until you've verified it
• Comfortable working in ambiguity; AI QA doesn't have settled playbooks yet
• Clear written communication bug reports for AI failures need precise reproduction steps
• Collaborative across AI engineers, full-stack devs, delivery managers, and account leads
• Proactive about catching issues before they reach the client demo