Senior Product Manager, AI Evals job opportunity at London Stock Exchange Group (LSEG).

Home Jobs In United Kingdom Tag: AI Evals Senior Product Manager, AI Evals position at London Stock Exchange Group (LSEG)


DateMore Than 30 Days Ago bot
London Stock Exchange Group (LSEG) Senior Product Manager, AI Evals
Experience: General
Pattern: full-time
Country:
apply Apply Now
Salary:
Status:

AI Evals

Copy Link Report
degreeOND
loacation GBR-London-10 Paternoster Square, United Kingdom
loacation GBR-London-10 ..........United Kingdom

As part of our AI@LSEG Product Management team, we are seeking a  Senior  Product Manager  with deep  expertise  in testing, automation, and evaluation systems  to own our  AI Evaluation Framework  for the entire AI workflow from natural language requests to the execution to the response.   This role will define how LSEG measures the accuracy,  reliability  and consistency of AI-powered interactions across our data,  analytics  and market solutions. You will  establish  the frameworks, processes and standards that evaluate LLM  behaviour , MCP tool usage, and natural-language output quality across multiple AI stacks—ensuring our customers  receive  precise, trustworthy, and explainable results.   Key Responsibilities & Accountabilities   Framework Ownership   Design and own the  end-to-end evaluation framework  for AI systems: natural-language prompting, LLM orchestration, MCP execution, and natural-language output quality.   Codify a  rubric-based scoring system  covering task fulfilment, factual accuracy, reasoning quality, safety,  coverage  and instruction adherence.   Develop the taxonomy for  query types, complexity levels, and evaluation profiles  across all LSEG MCPs and AI stacks.   Test Design & Automation   Partner with SMEs to create  seed evaluation  scenarios , and  define the processes for generating structured prompt permutations using LLMs.   Build the strategy for scalable  automated evaluation pipelines , including LLM-based graders, confidence scoring, sampling  strategies  and regression detection.   Define requirements for internal tooling: scenario catalogues, result dashboards, model-versus-model comparisons, performance  tracking  and release gates.   Cross-Functional Leadership   Collaborate with product teams across Data & Analytics, Markets, FTSE  Russell  and Risk Intelligence to integrate the evaluation framework into their AI initiatives.   Work closely with Engineering, AI Governance, Model Risk,  Compliance  and GSSR to ensure alignment with responsible AI standards and regulatory expectations.   Support product teams in interpreting evaluation results,  identifying  failure  modes  and improving model + MCP reliability.   AI Strategy & Governance   Contribute to group-wide AI governance by defining evaluation requirements that support responsible AI, explainability and bias mitigation.   Provide insights on emerging testing and benchmarking methodologies across the AI ecosystem.   Ensure evaluation frameworks adapt as  LLMs,  MCPs and customer workflows evolve.   External & Internal Advocacy   Serve as the internal subject matter expert for AI evaluation and testing.   Present evaluation insights to senior stakeholders,  ExCo , product  leadership  and external partners.     Skills & Competencies   Required   Background in  product management ,  QA/test automation , or  ML evaluation frameworks  with experience designing structured test systems at scale.   Strong knowledge of  LLMs , prompt engineering, and tool-augmented LLM workflows.   Experience with test automation, scenario design,  evaluation   metrics  and quality frameworks in complex systems.   Familiarity with  financial data, analytics workflows, or MCP-style interoperability .   Ability to translate ambiguous real-world user tasks into formal evaluation structures and rubrics.   Preferred   Experience with LLM-based evaluators, rubric scoring systems, structured data validation, and regression frameworks.   Familiarity with financial data pipelines, structured/unstructured data handling, analytics APIs, and Python/JS-based tooling ecosystems.   Strong analytical skills: ability to  identify  quality gaps, failure  patterns  and reliability issues in AI systems.   Comfortable partnering with senior internal stakeholders, engineering  teams  and policy groups.     Measures of Success   Quality & Accuracy   Improvements in task  fulfilment  accuracy, reasoning  clarity  and factual correctness across MCP-powered workflows.   Reduction in hallucinations, constraint violations, and output variability across permutations and AI stacks.   Scalability & Adoption   Uptake of the evaluation framework across LSEG product divisions.   Level of automation achieved in test execution and  evaluation  pipelines.   Stability of evaluation results and speed of  identifying  regressions.   Organisational  Impact   Contribution to Responsible AI, model risk  processes  and governance frameworks.   Improvements in transparency,  auditability  and explainability of AI  behaviour .   Ability to support product teams in shipping AI features with confidence.   Career Stage: Manager London Stock Exchange Group (LSEG) Information: Join us and be part of a team that values innovation, quality, and continuous improvement. If you're ready to take your career to the next level and make a significant impact, we'd love to hear from you. LSEG is a leading global financial markets infrastructure and data provider. Our purpose is driving financial stability, empowering economies and enabling customers to create sustainable growth. Our purpose is the foundation on which our culture is built. Our values of Integrity, Partnership , Excellence and Change underpin our purpose and set the standard for everything we do, every day. They go to the heart of who we are and guide our decision making and everyday actions. Working with us means that you will be part of a dynamic organisation of 25,000 people across 65 countries. However, we will value your individuality and enable you to bring your true self to work so you can help enrich our diverse workforce. We are proud to be an equal opportunities employer. This means that we do not discriminate on the basis of anyone’s race, religion, colour, national origin, gender, sexual orientation, gender identity, gender expression, age, marital status, veteran status, pregnancy or disability, or any other basis protected under applicable law. Conforming with applicable law, we can reasonably accommodate applicants' and employees' religious practices and beliefs, as well as mental health or physical disability needs. You will be part of a collaborative and creative culture where we encourage new ideas. We are committed to sustainability across our global business and we are proud to partner with our customers to help them meet their sustainability objectives. Our charity, the LSEG Foundation provides charitable grants to community groups that help people access economic opportunities and build a secure future with financial independence. Colleagues can get involved through fundraising and volunteering. LSEG offers a range of tailored benefits and support, including healthcare, retirement planning, paid volunteering days and wellbeing initiatives. Please take a moment to read this privacy notice carefully, as it describes what personal information London Stock Exchange Group (LSEG) (we) may hold about you, what it’s used for, and how it’s obtained, your rights and how to contact us as a data subject . If you are submitting as a Recruitment Agency Partner, it is essential and your responsibility to ensure that candidates applying to LSEG are aware of this privacy notice.

Other Ai Matches

Senior Account Manager Applicants are expected to have a solid experience in handling Job related tasks
Junior Research Analyst Arabic/French Applicants are expected to have a solid experience in handling Job related tasks
Trade Support Executive - FXT Applicants are expected to have a solid experience in handling Job related tasks
Senior Engineer, Windows Applicants are expected to have a solid experience in handling Windows related tasks
Senior Sales Specialist - PRS Applicants are expected to have a solid experience in handling Job related tasks
Principal/Senior Lead DevOps Engineer – Cloud Platform Applicants are expected to have a solid experience in handling Job related tasks
Account Manager - Customer Connect Sales team Applicants are expected to have a solid experience in handling Job related tasks
Engineering Graduate Programme (Backend Data Engineer) Applicants are expected to have a solid experience in handling Job related tasks
Senior ServiceNow Architect Applicants are expected to have a solid experience in handling Job related tasks
Lead Network Engineer Applicants are expected to have a solid experience in handling Job related tasks
CDSClear Quantitative Strategist Applicants are expected to have a solid experience in handling Job related tasks
Senior Manager, Business Management Applicants are expected to have a solid experience in handling Business Management related tasks
Senior Principal Quality Engineer (WC Curation Platform) Applicants are expected to have a solid experience in handling Job related tasks
Research specialist, Hydropower markets Applicants are expected to have a solid experience in handling Hydropower markets related tasks
Observability Platform Engineer Applicants are expected to have a solid experience in handling Job related tasks
Assistant Manager - Reporting & Analytics Applicants are expected to have a solid experience in handling Job related tasks
Senior Executive, RNS Applicants are expected to have a solid experience in handling RNS related tasks
Quality Automation Engineer Applicants are expected to have a solid experience in handling Job related tasks
Senior Principal Software Engineer (WC Curation Platform) Applicants are expected to have a solid experience in handling Job related tasks
Sales Specialist, Workflows Applicants are expected to have a solid experience in handling Workflows related tasks
Technical Lead Applicants are expected to have a solid experience in handling Job related tasks
Associate Solution Consultant, Real-Time and Historical Data Applicants are expected to have a solid experience in handling Real-Time and Historical Data related tasks
Senior Implementation Specialist Applicants are expected to have a solid experience in handling Job related tasks