LRL Test Engineering Hub - Automation Test Lead

At Lilly, we unite caring with discovery to make life better for people around the world. We are a global healthcare leader headquartered in Indianapolis, Indiana. Our employees around the world work to discover and bring life-changing medicines to those who need them, improve the understanding and management of disease, and give back to our communities through philanthropy and volunteerism. We give our best effort to our work, and we put people first. We’re looking for people who are determined to make life better for people around the world.

Job Summary

At Lilly, we unite caring with discovery to make life better for people around the world. We are a global healthcare leader headquartered in Indianapolis, Indiana. Our 39,000 employees work to discover and bring life-changing medicines to those who need them, improve the understanding and management of disease, and give back to our communities through philanthropy and volunteerism. We give our best effort to our work, and we put people first. We’re looking for people who are determined to make life better for people around the globe.

Medicines Development serves a strategic function within Lilly Research Laboratories (LRL). Development of most medicines requires about a billion dollars and a decade or more to bring to market. At Lilly, we’re investing to make it faster, better and cheaper, to the benefit of patients... and the Information and Digital Solutions organization is a critical partner in reaching success. Broad trends in science and healthcare are driving integration across Medicines Development, Research, Manufacturing, and global Business Units. New digital capabilities in the broader business model, both internally and externally, will require agile and creative evolution of the processes and digital solutions that support clinical and product development.  

As a Intelligent Automation Test Lead, you will have the opportunity to accelerate the delivery and to improve the quality of Lilly’s Applications and Data. You will be responsible for designing and implementing test frameworks and infrastructure.  To be successful, you must be a strong programmer, proficient at thoroughly testing applications and services, and helping improve team velocity through automation. We are looking for highly motivated individuals, who can continually research and apply the newest technologies to test engineering, independently tackle issues and influence the teams to adopt the best test practices. Deeply engage with the development/engineering teams through all phases of development!

Desired Technologies

• Bachelor’s degree in computer science, information technology, management information systems or equivalent work experience.

• Design, develop, and maintain automated test frameworks and scripts for applications utilizing LLMs, ensuring comprehensive coverage of all functionalities and user scenarios.

• Understand Retrieval Augmented Generation(RAG) strategy to improve testing for knowledge-intensive tasks by retrieving relevant documents or data that can be used as references or benchmarks for evaluating the LLM's responses

• Exposure to various Gen AI Testing Frameworks like RAGAS, Deepeval, BenchLLM 

• Perform thorough regression testing when bugs are resolved and when there are updates to LLMs or application features.

• Collaborate with development teams to understand application features and integration points for Large Language Models, identifying potential areas for automation.

• Execute automated and manual test cases, analyze test results, identify and document defects, and work closely with developers to ensure they are addressed promptly.

• Monitor the performance of LLMs in the chat interface, identifying any issues with latency, accuracy, or relevance of responses.

• Stay up-to-date with the latest developments in AI, machine learning, and chatbot technologies to continually improve testing strategies and tools.

• Collaborate with product managers and UX designers to ensure the chat application meets the highest standards of user satisfaction and engagement.

• Ensure compliance with data privacy and security policies during the testing process, especially when handling sensitive user data.

• Provide detailed reports and documentation on test activities, results, insights, and recommendations for improving the application's quality and performance.

Preferred Qualifications

• Should be a Passionate tester with development exposure for SDET role.

• Good understanding of automation tools, technology landscape, and OOPS concepts.

• Experience in testing and automation in agile and continuous delivery models                      

• Experience in Test Driven and Behaviour Driven Automation Framework Development.

• Experience working in regulated environments and with internal systems quality policies and procedures 

• Experience in working in Scrum - agile methodologies, coding standards, code reviews, source management, build processes (CI/CD Pipeline).

• Knowledge of the data architectures associated with information integration, data visualization, & data warehousing 

• Pharmaceutical or healthcare industry experience

• Defining best practices and developing technical standards, design principals, best practices, and frameworks

• Technical curiosity and desire to innovate

Technical Leadership 

• Partner with and influence vendor resources on testing process, examine the existing testing processes (manual and automation both) and suggest improvements in terms of the adoption of new approaches which can improve quality and shorten test cycles.

• Check the feasibility of requirements whether they could be automated or if there are some cases that must stay as manual.

• Leverage existing frameworks at an enterprise level and build new frameworks by integrating with the existing tools and infrastructure.

• Communicating with internal and external stakeholders regarding testing progress and timelines by providing testing reports and metrics.

• Responsible to communicate across the testing community and development teams; Sharing knowledge via training session.

• Work collaboratively with delivery teams to ensure timely project delivery.

• Responsible for understanding, designing, and writing test scenarios and test cases; updating test cases based on the requirement changes.

• Responsible for bug classification & analysis using defect tracking tools such as JIRA, Bugzilla, Mantis.

• Ability to adapt quickly in a constantly changing environment

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form (https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.

Lilly does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.

#WeAreLilly