Test Lead - GenAI and LLM Testing

Senior QA Automation Engineer / SDET - GenAI & LLM Testing 6 Month Contract Banking Client Remote / Hybrid (UK) Competitive Day Rate We're supporting a leading banking client on a large-scale AI transformation programme and are looking for an experienced Senior QA Automation Engineer / SDET with a strong background in GenAI and LLM testing. This role will focus on defining and implementing automated testing frameworks for AI-driven products, ensuring the quality, reliability and safety of LLM-powered solutions deployed within a highly regulated banking environment. Key Responsibilities

• Design and develop scalable automated test frameworks using Python.
• Create test strategies from architecture and solution design documentation.
• Test RESTful APIs and microservice-based applications.
• Define and execute testing approaches for event-driven architectures and Kafka-based systems.
• Validate GenAI and LLM solutions through functional, non-functional and behavioural testing.
• Develop automated prompt testing and evaluation frameworks.
• Verify AI guardrails, safety controls and compliance requirements.
• Implement validation approaches for non-deterministic LLM outputs.
• Build and maintain contract tests using Pact Flow.
• Collaborate closely with Engineering, Architecture, Data Science and Product teams.
• Support CI/CD pipelines and quality engineering best practices. Required Skills & Experience Core Automation
• Strong Python 3.x development skills with experience building testing frameworks from the ground up.
• Extensive experience with Pytest.
• Strong API testing experience.
• Experience creating test strategies and quality approaches from architectural diagrams and solution designs. Cloud & Platform Working knowledge of AWS services including:
• Lambda
• Bedrock
• S3
• EKS Event-Driven Systems
• Experience testing Kafka-based architectures.
• Strong understanding of asynchronous messaging and event-driven systems. GenAI / LLM Testing (Essential)
• Proven experience testing LLM and GenAI applications.
• Prompt testing and prompt evaluation.
• Non-deterministic output validation.
• AI guardrail verification and safety testing.
• Hallucination and response quality assessment.
• Experience establishing quality standards for AI solutions. LLM Evaluation Tooling Experience with one or more of:
• DeepEval
• Promptfoo
• RAGAS
• LangSmith
• Similar LLM evaluation frameworks Contract Testing
• Hands-on experience with Pact Flow.
• Consumer-driven contract testing within distributed systems. Desirable
• Banking or Financial Services experience.
• Experience working in highly regulated environments.
• Exposure to RAG architectures.
• Knowledge of LLM observability and monitoring.
• Kubernetes experience.
• CI/CD pipeline integration.