Research
NAIRL’s Professor Jinyoung Yeo of Yonsei University to Present at NeurIPS 2026
A research team led by Professor Jinyoung Yeo of the Department of Artificial Intelligence at Yonsei University, affiliated with the National AI Research Lab (NAIRL), has developed a way to evaluate AI that designs and improves other AI agents. The study, “Towards Direct Evaluation of Harness Optimizers via Priority Ranking,” will be presented at NeurIPS 2026.
AI agents now write software, analyze data, and carry out multi-step tasks, but each one needs a harness to work: the instructions it follows, the tools it can call, what it remembers, and the order in which it proceeds.
Building that harness has been a human job. Increasingly, it is being handed to another AI, an optimizer agent that inspects a target agent’s harness and revises it. For a company running thousands of agents, that automation is the only practical path. The problem is that optimizers have only ever been judged by whether the target agent got better in the end, which says nothing about the individual edits along the way.
The team found those edits matter. In their experiments, current optimizers chose the wrong thing to fix in roughly 45 to 48 percent of their updates, which means a target agent can end up worse off. Rather than run full optimization loops to catch this, the team asks an optimizer a simpler question: given a harness, rank its components by how much changing each one would help or hurt the agent.
This priority ranking, as the team calls it, measures the optimizer’s judgment at each step without expensive rollouts or manual review, and the team showed it correlates with how well an optimizer actually improves agents in practice. It runs about 8 times cheaper and 17 times faster than existing evaluation.
To support the evaluation, the team built SHOR, a set of 182 human-verified optimization scenarios spanning different domains, harness designs, and stages of optimization. The study was co-first-authored by Kai Tzu-iunn Ong, Minseok Kang, and Dongwook Choi of the LangAGI Lab and conducted in collaboration with Samsung Electronics.
As companies move from tuning a single model to building and maintaining agents at scale, checking whether the AI doing that work can tell what to fix becomes a quality-control question in its own right. NAIRL will continue to introduce research that advances the reliability and automation of AI to the broader domestic and international AI community.
Paper: https://arxiv.org/abs/2605.22505
Code and data: https://github.com/k59118/Harness_Optimizer_Evaluation
