Talk recap: How we took an AI agent from 41% to 97% accurate

Milind Shah, a member of technical staff at Brainbase, spoke at AI Engineer about hill-climbing back-office agents: taking an agent that barely works and making it one a team can rely on. His running example was the agent we built with Domino's CMO office to answer a question they kept asking. What does a pizza cost at the competitor down the street, in their app and on their website, with and without promotions?

Before the agent, an analyst did this by hand: open a competitor's site, pick the right store, compare delivery and takeaway, find the promotions, take the cheapest price, write it into a spreadsheet. Each step looks easy. Get one wrong and you have a cell that looks finished and is simply wrong. Version 1 of the agent was 41% accurate. Version 11 was 97%, at about half the cost per source, and almost every change in between was to instructions, skills and playbooks rather than the model. The talk laid out the six steps behind that, which we now use with other customers too. The full story is in the Domino's case study.

#Kickoff

Start by agreeing with the people who do the work on what a win looks like. What is the job today, and what does a person sign off on at the end? How long does it take, what does it cost, and how accurate does the agent need to be? Who judges the output, and what must the agent never do or assume on its own?

For Domino's, an analyst spent about two hours a day building the spreadsheet, the pricing team checked every cell, and the agent would hand off to the analyst for sign-off, flagging anything it was unsure of. The goal was a faster, repeatable way to collect local competitor pricing without losing judgment on the edge cases.

#Ontology

Next, write down every term the agent needs so it never has to guess: what to collect, where each answer comes from, where it goes, what each term means to the analysts, and what to do when it cannot work something out. If you do not write something down, the agent will guess, and probably get it wrong.

For Domino's, the agent worked from a table of target items in Brainbase Memory, each with a brand, size, channel and app. It saved what it found to a price observations table, the price seen, the full price and the source, and filled a spreadsheet of item IDs from there. Most rules came straight from failures in real traces:

• Record the price a customer pays at checkout, not the struck-out one.
• A large pizza is the item itself, or a bundle containing it.
• Delivery is not takeaway.
• Apply promotion maths before entering a price.
• If a price cannot be verified, record it as a gap. Do not guess.

When a cell is wrong, it points to the exact page and the rule behind the price chosen.

#Benchmarks and evals

Work out what you can score offline and what you can score online. Offline, start from past cases that already have answers; for Domino's, that was the analysts' old spreadsheets. Archived snapshots of competitor websites worked for some sources but not for delivery apps, so the team kept browser screenshots from every run, starting with version 1, and built the benchmark environment from those.

For each check, ask whether it can be done in code, by a model as judge, or by an agent as judge: an agent that takes several turns and tool calls to decide whether an output is correct. Code checked that every cell was filled, that the discount maths was right and that a price more than 20% away from the last run was flagged. A model judge read the trace to confirm the right item and promotion. An agent judge revisited every flagged price to confirm the flag was valid.

#A first agent

Ship a narrow first agent you can score from day one. Its job is to be measured, not to be good: how long it runs, what it costs and what it gets wrong. Widen the scope once you have a baseline. Domino's started with one app, one item and one city, then added burgers and takeaway, then another app, then competitor websites.

#Iterate

Score, fix and ship. Find the failures that are most expensive and most frequent, fix them, check what the fix does to run time and cost, and turn each failure into a benchmark case or eval criterion. Change one thing at a time, and fix the instructions before you change the model.

Two fixes from the talk show the range. In one run the agent followed a link to the right section of a menu, found the large pizza and stopped, missing a cheaper deal further down the page. One new rule fixed it: read the whole page, not just the section you were sent to. In another, a competitor's promotions lived inside 26 images. The agent zoomed in again and again, found the right card at minute 13 and kept checking until it was stopped at minute 15. The page's HTML had image titles and alt text all along, so the rules now say to read the DOM first and to stop past a set zoom level. That cut image reads to a third and cost by more than half. Automated checks catch a lot, but some fixes only come from an engineer reading the traces.

#Hand off the loop

At the end you hand over the loop, not just the agent. Write down who runs it, who reviews the output, who turns corrections into rules and benchmark cases, and who decides a new version is ready to ship. At Domino's the agent runs on a schedule or from an email trigger, analysts review only the exceptions, and each correction they make can become a benchmark case. The work is now expanding to more markets.

The framework works best wherever sources are messy and the right answer takes judgment. Thank you to AI Engineer for having us. The full talk is on YouTube.