Skip to content
K-d Tree Explorer, home
All decision records

DR-006Accepteddecided 2026-10-10

Evaluate the AI explanations on a fixed, seeded set of searches, run with the visitor's own key

docs/decisions/DR-006-evaluation-harness-for-explanations.md

Context

DR-005 added an optional AI walkthrough of one search, with five automated checks against the search trace. A check result describes one answer. Nothing showed how often a model gets the numbers right, whether a cheaper model does as well as a dearer one, or how much the answers of one model vary from run to run. DR-005 recorded that no systematic evaluation had been run, because there is no budget for API calls. The rest of the site gives every quantitative result an interval and compares methods on the same inputs; the AI feature was the one place that did neither.

Decision

Add an evaluation harness at /ai-eval that runs the explorer's prompt over a fixed set of twenty searches with the visitor's own key, and reports what comes back with intervals.

  • The set (web/src/lib/explain/eval-set.ts, version explain-eval-v2, seed 2026): cases C01 to C20, four kinds of search (nearest near a business, nearest anywhere in the city, radius 0.0005° and radius 0.003°), each on all five insertion orders. Each kind draws its own five points, so changing one kind never moves the others. Five of the twenty traces are longer than the 150 steps that are sent.
  • The headline is the share of graded answers that are fully grounded: valid JSON that fits the schema and passes all five checks. Invalid JSON, truncated answers and refusals count as failures. Key, quota, network and rate-limit failures are not the model's, so they are listed and kept out of every denominator.
  • Uncertainty: a Wilson 95% interval for every rate and a percentile bootstrap (2,000 resamples, seed 20003) for the median latency, timed on the request that answered (retries and waits excluded). For two runs on the same cases: the paired difference in the grounded rate, with Newcombe's (1998) hybrid score interval for paired proportions (his method 10), and an exact McNemar test.
  • One record of every call: calls run one at a time through the same audited client as the explorer (DR-005), with the run id, the case id and the prompt version in each audit entry. Runs are not stored separately. The page rebuilds them from the audit log and grades every logged answer again against facts recomputed from the data. A call is graded only if it carried exactly the prompt the site sends for that case today; otherwise (an older prompt version, or data that has changed) it is marked stale-input, left out of every rate and asked again if the run is continued. Runs can be exported as CSV or JSON, with the prompt version and request settings logged with each call.
  • No published scores: the site shows only runs made in the visitor's own browser.

Options considered

  1. Keep judging one answer at a time. Rejected. One answer cannot give a rate, an interval or a difference between models.
  2. Run the evaluation myself and publish the numbers. Rejected for now. It needs a paid key, and a score that I produce and display about my own feature is the kind of self-reported number the site avoids.
  3. A second model to grade the prose (a model as judge). Rejected for now. It would double the cost to the visitor and bring in a grader whose own errors would need evaluating. The prose is left to the human decision recorded with each call.
  4. A fixed, seeded set graded mechanically, run by the visitor with their own key and rebuilt from the audit log. Chosen.

Why

The checks from DR-005 are exact and deterministic, so they can grade any number of answers for nothing beyond the calls themselves. A fixed set makes two runs answer the same questions, which allows a paired comparison; with a pass or fail on shared cases, McNemar's test uses exactly the cases where the two runs disagree. The interval for the difference is a closed-form score interval rather than a bootstrap: with pass or fail results on twenty pairs, a bootstrap over the pairs has zero width when the runs agree on every case and cannot reach past zero after one or two disagreements in the same direction. Newcombe's interval is built from the Wilson limits of each run's rate, the same family as every other rate on the site, corrected for the pairing. Rebuilding runs from the audit log means a run cannot show a score that its logged answers do not support, and it keeps one record of every call instead of two that could drift apart.

What happened

  • The harness is tested end to end with fetch mocked (web/src/lib/explain/eval-runner.test.ts): twenty requests, each carrying exactly the explorer's prompt for its case; twenty audit entries, none containing the key; a run rebuilt from the log and graded; a stop after the first rejected key; and a cancelled case that is asked again when the run continues. The intervals and p-values are checked against R in eval.test.ts.
  • I have not run it with a paid key, so neither the site nor this record has any results to report.
  • The first version gave the paired difference a percentile bootstrap interval. Review before release showed that it collapses exactly where real runs are likely to sit: two runs that agreed on every case got "0 points (95% CI 0 to 0)", and one disagreement gave 0 to +15 points beside a McNemar p of 1.000. I replaced it with Newcombe's interval before the harness went live. Two runs that both ground the same 18 of 20 cases now show 0 points (95% CI −12.9 to +12.9 points), and one disagreement on top of that shows +5 points (−8.7 to +20.9 points). The limits are checked against Newcombe's own worked example and against R.
  • Twenty cases give coarse answers. Even 20 fully grounded answers out of 20 leave a Wilson lower bound of 83.9%. Two runs can be told apart (exact McNemar p below 0.05) only with at least six disagreements in one direction, for example 6 against 0 (p = 0.031) or 8 against 1 (p = 0.039); five all one way give only 0.0625. Per-kind results are shown as counts without intervals, because five cases per kind are too few.
  • Review also found that an answer which failed as the model's (cut off at the token limit, refused, or not valid JSON) lost the tokens it had cost and most of its text from the audit log, and that the latency of such a call included the waits between retries. Both are fixed: those calls keep their token counts and full text, and every call records the answering request's latency and, separately, the whole call with its retries.
  • Before release the case ids were renamed from E01–E20 to C01–C20, because the 2020 experiment already numbers its queries E6 to E13, and the set version became explain-eval-v2, so that test calls logged under the first version are never mixed in. The searches themselves did not change.
  • A run sends about 309 kB of prompts in total, roughly 77,000 to 103,000 input tokens at three to four characters per token, plus the answers, all billed to the visitor's key.

What I'd change

With a budget, I would run the set on both Claude models and an OpenAI model, twice each, and publish the runs with their logs, so that the variation between runs of one model is measured rather than assumed. I would also grow the set so the intervals narrow (about 80 cases bring the Wilson half-width below ten points at an 80% rate), add cases drawn from real explorer searches, and rate the prose against a written rubric by two people independently, reporting how often they agree.