AI Evals/Benchmarking Engineer - A26419

Activate Interactive Pte Ltd (“Activate”) is a leading technology consultancy headquartered in Singapore with a presence in Malaysia and Indonesia. Our clients are empowered with quality, cost-effective, and impactful end-to-end application development, like mobile and web applications, and cloud technology that remove technology roadblocks and increase their business efficiency.

We believe in positively impacting the lives of people around us and the environment we live in through the use of technology. Hence, we are committed to providing a conducive environment for all employees to realise their full potential, who in turn have the opportunity to continuously drive innovation.

We are searching for our next team members to join our growing team.

If you love the idea of being part of a growing company with exciting prospects in mobile and web technologies that create positive impact on people’s lives, then we would love to hear from you.

Co-Development Business Unit is hiring for AI Evals/Benchmarking Engineer

You will work in Singapore Government Agencies

This is a fixed term contract role. The engagement is 1 year.

Internal Code: A26419

What will you do?

• Design and operate the evaluation and benchmarking framework for the AIDD target evaluation workflow and its AI agents. The role establishes retrospective baselines, integrates Therapeutics Data Commons (TDC) datasets where appropriate, runs forward regression testing, and converts structured human-in-the-loop feedback into governed training and improvement signals while maintaining quality, provenance and reproducibility.
• Define the end-to-end evaluation strategy, benchmark suites, test datasets, rubrics, acceptance thresholds and reporting standards for retrieval, generation, reasoning, scoring, gating and meta-agent components.
• Build retrospective benchmarks using historical target evaluations and expert decisions to measure evidence quality, citation and provenance coverage, factual consistency, therapeutic-direction accuracy, score reliability and report quality.
• Assess and integrate relevant TDC datasets and tasks, including data mapping, licensing, preprocessing, versioning and alignment with AIDD evaluation criteria.
• Develop automated forward regression pipelines that compare models, prompts, agents, data sources and workflow versions, detect performance drift and block unsafe or substandard releases.
• Design human-in-the-loop review capture, including structured annotations, corrections, confidence, rationale and outcomes, and transform validated feedback into governed evaluation and training signals.
• Perform error analysis and failure-mode discovery, maintain traceable experiment records, and work with scientists and engineers to prioritize and verify improvements.
• Create dashboards and concise reports on benchmark performance, QA pass rate, provenance, latency, cost and human-review effort. Review results with scientific stakeholders, the Architect and the Delivery & Tech Lead to support model selection, release gates and prioritisation of workflow and engineering improvements.

What are we looking for?

• Strong experience in AI or machine-learning engineering, model evaluation or applied research, including generative AI, LLMs, RAG or agentic systems.
• Proficiency in Python and common ML and data libraries, with the ability to build automated data, evaluation and reporting pipelines.
• Practical knowledge of evaluation methods for retrieval, generation, classification, ranking and scoring, including metric selection, dataset design, statistical comparison and error analysis.
• Experience with experiment tracking, data and model versioning, reproducibility and MLOps practices using tools such as MLflow, DVC, Airflow, Git and containers.
• Ability to analyse false positives, false negatives, edge cases, drift and failure modes, then translate findings into actionable engineering recommendations.
• Strong documentation and stakeholder communication skills, with the ability to work closely with domain experts, AI engineers and governance teams.
• Relevant degree in artificial intelligence, computer science, data science, engineering or a related discipline.
• Experience with TDC or comparable biomedical datasets and evaluation of RAG, semantic search, document intelligence or LLM agents.
• Development of multi-agent evaluation systems, human-feedback datasets and traceable scoring or decision logic.
• Production AI delivery on cloud and data platforms, including containerised infrastructure, MLOps and evaluation or diagnostic tooling.

What do we offer in return? 

Competitive Compensation: Market competitive salary and variable performance bonus aligned to your skills, impact, and contribution. 

Benefits: Outpatient medical, specialist medical coverage and generous customisable flexi benefits, or flexi allowance; life and health insurance, thoughtful perks like special occasions “red packet” and CNY goodies, etc. 

Employee Wellness: Support for your physical, mental, and overall well-being through year-round initiatives. 

Growth & Development: Learning programmes, certification support, and a dedicated staff development budget. (We are a “SHRI 2025 Gold winner” in “Learning & Development; Coaching & Mentoring”) 

Career Progression: Structured career pathways that enable you to grow along a technical/domain expert track or a leadership track. 

Competency Framework: A structured and practical framework to support you to develop, perform, and succeed. 

Flexible Work Arrangement: Staff may choose to work flexi-place, flexi-time, and flexi-load based on existing framework 

Why you'll love working with us? 

If you are looking for opportunities to collaborate with leading industry experts and be surrounded by highly motivated and talented peers, we welcome you to join us. We provide all employees with equal opportunities to grow and develop with us. We believe your success is our success. 

Does it sound like something you are interested in exploring further? Please be in touch with our team for an initial chat at [email protected]

Activate Interactive Singapore is an equal opportunity employer. Employment decisions will be based on merit, qualifications and abilities. Activate Interactive Pte Ltd does not discriminate in employment opportunities or practices on the basis of race, colour, religion, gender, sexuality, national origin, age, disability, marital status or any other characteristics protected by law. 

Protecting your privacy and the security of your data are longstanding top priorities for Activate Interactive Pte Ltd. 

Your personal data will be processed for the purposes of managing Activate Interactive Pte Ltd’s recruitment related activities, which include setting up and conducting interviews and tests for applicants, evaluating and assessing the results, and as is otherwise needed in the recruitment and hiring processes. 

Please consult our Privacy Notice (https://www.activate.sg/privacy-policy ) to know more about how we collect, use, and transfer the personal data of our candidates. Here you can find how you can request for access, correction and/or withdrawal of your Personal Data.