How to Buy the Wrong AI with the Right Benchmark score

How to compare enterprise RAG systems using your own benchmark, and why a leaderboard score is not enough

Share

Linkedin iconFacebook iconTwitter icon

More deployment options

How do you evaluate an enterprise RAG system? We built two systems for the same public Enterprise RAG leaderboard to do just that. One scored about 83 for seven cents a question. The other topped the board at 93.45 for about $3.

Here is what that gap does and does not tell a buyer.

Image depicting a RAG score.

How did we get here?

It started as an itch. We were weighing architectures for a system that pulls context out of a company’s messy, unstructured documents, and one public leaderboard, EnterpriseRAG-Bench from Onyx, kept coming up. It ranks 40 RAG systems. We traded notes, and a few barbs, about the ones at the top. How much of a top score is architecture, and how much is learning the test and over-engineering the system to beat it?

Finding the best enterprise RAG system

So we made it a bet. Each of us would build a system on our own, with AI coding assistants writing most of the code, and see how close we could get.

Jitender‘s stalled at about 83. He tried a dozen strategies, and none moved the score by more than the judge agent’s own noise. It cost about seven cents a question. Puneet kept going. Between Saturday night and Tuesday evening his system climbed from 81.63 to 93.45, enough to top the board, at about $3 a question. That is roughly 43 times the cost for about 10 points.

We decided to write about it instead of chasing the chart. We could report every number in this post accurately and still leave you with the wrong impression, because the score alone will not tell an enterprise buyer which of those points carry over to their own work.

We did not submit the 93.45 run. The benchmark’s written rules cover submissions only, and nothing in them restricts using its guide files or tuning on its public questions. So this is not a story about cheating. It is a story about what a score can and cannot tell you.

Where the 11.82 points came from

The benchmark asks a system to answer questions from a company’s documents. The company is made up, and the questions and reference answers are public. There are 500 scored questions, each worth up to 0.2 points: a correct answer earns the share of the reference facts it contains, and a wrong one earns zero. So 93.45 is a fact-coverage score, not a count of right answers. 475 of the 500 answers were judged right (95.0%). We built for what this particular test rewards: independent searches, a referee that wants evidence from the documents before it will replace an answer, and a rewrite step. Then we let the system read a lot more. Here is where the points went.

Step Points What it cost or hid What a buyer must check
Independent searches, referee, rewrite and benchmark guides (81.63 to 91.30) 9.67 Not split out on the full 500 questions. On our 400-question development set, the guides alone added about 1.1. Does the evidence rule help on your documents? Do your documents come with equivalent guides?
Deeper search (about 600 look-alike documents per run) 1.33 7 answers fixed, none lost. Run on 114 questions at about $6.50 each: about $750, or $107 per correction. Is $107 per correction worth it for this job?
Grader variation on unchanged answers (91.30 to 93.45) 0.82 490 answers did not change. 4 flipped verdict, all from wrong to right. How noisy is the grader you will rely on?
Leftover generator field no measured gain Original 93.45 run cannot be certified clean. Is anything in your inputs leaking the answer?
Total, 81.63 to 93.45 11.82

What the score hid

The test explained itself. The benchmark ships guides to its own folders and fields, like a map handed to a student before the exam. One of them says the company uses Jira only for support, not project management, and that customer tickets and internal helpdesk tickets live in separate folders with different ticket prefixes. That tells a search agent not to hunt for project plans in Jira and which folder to open.

It helped Puneet’s build by about 1.1 points on our development set. In Jitender’s build, the same guides scored 78.44 against an 80.15 baseline, inside the noise. And according to the benchmark’s own methodology, these files are the instructions used to generate the corpus. A real company has nothing like them. A buyer has to ask whether their documents come with an equivalent map. Yours may not.

More reading cost more. The deeper search read about 600 look-alike documents every time it ran. It corrected 7 answers and added 1.33 points. It ran on 114 questions at about $6.50 each, so roughly $750 in total, or $107 per corrected answer. Whether that is worth it depends on the job. Across the whole build, cost per question went from about $0.50 to about $3 at list prices. The leaderboard showed neither cost nor response time.

The grader contributed too. Across our last two versions, 490 answers did not change, and each version was graded once, so those answers got graded twice. Grader variation on them added 0.82 points. Four answers flipped verdict, every one from wrong to right. Four flips in one direction is too small a sample to call the grader biased: if flips were equally likely either way, all four landing the same way would happen about 1 time in 8. We have not yet re-graded one answer file several times to count flips in each direction. Our stalled attempts ran into the same noise.

We tuned on the questions we were scored on. Of the 500 scored questions, 400 were also Puneet’s development set. The only held-out result we have is 89.2, for the earlier 91.30 configuration. The 93.45 configuration has no held-out number.

A leftover field. We also found a stray field from the test’s generator in our first system’s inputs, the kind of fingerprint a data generator leaves behind. Components that never saw it scored the same or better on the affected questions, so we measured no advantage. After the 93.45 run we built a copy of the search data without that field. We have not rerun on it yet, so we cannot certify the original run as clean.

Build your own benchmark

If we were the ones buying, we would skip the leaderboard and build our own test first. It is less work than it sounds, and nobody can game it, because nobody else has seen it.

Start with a question that made someone’s week worse. You know the kind: it got forwarded to a colleague, then to another, until it landed on the one person who actually knew. Write down the question, the answer that person gave, and the documents they had to open to get there. Then ask two things. What would have made that answer useless? And what decision hung on getting it right?

Here is the case we would want most. Someone asks which cancellation terms apply to a contract. The company has three versions of the policy, and one customer has a side letter that overrides a clause. The accepted answer names the right version, cites the exception, and points to both documents. A fluent, confident summary of the standard terms sounds great and is wrong. That is the failure that costs real money, and a leaderboard average will bury it.

This is the heart of it. Real work is full of conflicts. Two policies disagree. A newer one quietly supersedes an older one. The exception lives in a document nobody remembers. Picking the right document, and noticing when documents disagree, is most of the job, so build cases that test it head on. If missing that exception would rule the system out, mark it as a required case before you test anything, and report it on its own line instead of folding it into an average.

Put in a few cases where the right answer is “the documents don’t say.” A system that always has an answer is a system that sometimes invents one.

What you end up with is a golden dataset: real questions, accepted answers and the evidence behind them. A small team can start it next week, before picking any system. The same idea works for systems that answer from tables. There the accepted answer is the number an analyst who knows the data would sign off on.

A few ground rules save you pain later:

  • Write down who is allowed to see the supporting documents.
  • When your own experts disagree on the right answer, settle it or park it. Unsettled cases stay out of the score.
  • Give the set an owner. Policies change, and a stale golden set quietly turns into a bad test.

How large does it need to be? We have no evidence for a magic number, so this is judgment. About 100 cases is enough to tell a system that misses one in ten from one that misses three in ten. Separating systems that sit a few points apart takes several hundred cases, or fewer if you run every candidate on the same questions and compare them case by case.

Then run it like a fair fight. Let builders (vendors) learn from part of the set and keep the rest sealed. Give every candidate the same permitted information, record the version and documents used, and agree on what counts as a pass before anyone sees an answer.

Finally, put your buying conditions to the test. How much can an answer cost? How long can the user wait? Do answers go straight to people, or through a reviewer? Measure that next to quality, and have humans look at the disputed grades. Then compare against how the team does the work today, counting the hours spent checking and fixing answers either way. The question is not whether the system scores well. It is whether it makes the work better by more than it costs.

Take the failures to the buying meeting

You would want to walk into the buying meeting with your own benchmarks and open the failures before the wins. You should ask questions before deciding the purchase:

  1. Which required detail went missing from failures?
  2. Which answer needed more information than the system could find?
  3. Which correction was too expensive for the task?

Before signing, we would want to know which of those failures we could live with, and who owns the mess they leave behind. We would ask the same of our own system.

Start for Free with Starburst Galaxy

Try our free trial today and see how you can improve your data performance.
Start Free