AI evaluation · bring your own key
Does the AI explanation stick to the trace?
“Explain this search” in the explorer asks a language model to walk through one search, and checks its numbers against the trace. One answer shows little. Here the same prompt runs over twenty fixed searches, so a model gets a grounding rate with an interval, and two models can be compared on exactly the same cases.
Design (fixed before any run)
- Cases
- 20 searches (explain-eval-v2, seed 2026): 4 kinds of search on each of the five insertion orders. 5 traces are longer than the 150 steps that are sent.
- Prompt
- The explorer's prompt (explain-v1) and answer schema, unchanged
- Headline
- Fully grounded: the answer fits the schema and passes all five checks (four numbers from the trace, and every cited step exists). Invalid JSON, cut-off answers and refusals count as not grounded.
- Not graded
- Key, quota, network and rate-limit failures: listed, but left out of every rate
- Intervals
- Wilson 95% for rates; for two runs, Newcombe's score interval for the paired difference and an exact McNemar test; percentile bootstrap (2,000 resamples, seed 20003) for the median latency of the answering request (retries excluded)
- Provenance
- Each call logs its prompt version and settings. Only calls that carried exactly today's prompt for their case are graded; older ones show as stale input and are asked again if the run is continued
- Not measured
- Whether the prose is clear or its reasoning sound. Record your view of each answer in the AI log.
What a run costs you. 20 calls, one at a time, billed to your own key: about 309 kB of prompts in total, roughly 77,000 to 103,000 input tokens, plus the answers.
No published scores. The site has no key and no budget for API calls, so it shows no results of its own. Every number below comes from a run in your browser, and every call is in your AI log.
Why it is built this way: DR-006. The explanation feature itself: DR-005 and the model card.
Run the evaluation
Loading the data to rebuild the twenty searches…
The twenty searches
Drawn once from seed 2026: each kind of search takes its own five points, one per insertion order, so cheap and expensive trees, short and truncated traces are all represented. Open any case in the explorer to see the search the model is asked about.
- Nearest, near a business. A random business location moved up to 0.0005° (about 50 m): a typical question.
- Nearest, anywhere. Uniform over the data's bounding box, so some queries sit in parkland or on the river.
- Radius 0.0005°. Every business within about 50 m of a point near a business: mostly short traces.
- Radius 0.003°. Every business within about 300 m: long traces, some cut to the first 150 steps.
| Case | Kind · order | Insertion order | Query (x y) | Cmp.Comparisons | Steps sent |
|---|---|---|---|---|---|
| C01 | Nearest, near a businessOriginal | Original | 144.9572513 -37.80983131 | 28 | 41 |
| C02 | Nearest, near a businessRandom | Random | 144.9497478 -37.80251307 | 43 | 69 |
| C03 | Nearest, near a businessRandom, sorted | Random, sorted | 144.9456773 -37.80237994 | 25 | 44 |
| C04 | Nearest, near a businessSorted by x | Sorted by x | 144.9663419 -37.81401189 | 83 | 100 |
| C05 | Nearest, near a businessMedian first | Median first | 144.9664923 -37.79705901 | 13 | 23 |
| C06 | Nearest, anywhereOriginal | Original | 144.9894692 -37.81172995 | 23 | 41 |
| C07 | Nearest, anywhereRandom | Random | 144.9180950 -37.81942138 | 26 | 41 |
| C08 | Nearest, anywhereRandom, sorted | Random, sorted | 144.9319459 -37.81766472 | 29 | 47 |
| C09 | Nearest, anywhereSorted by x | Sorted by x | 144.9738960 -37.80231556 | 723 | 150 of 746 |
| C10 | Nearest, anywhereMedian first | Median first | 144.9812216 -37.80099783 | 60 | 83 |
| C11 | Radius 0.0005°Original | Original | 144.9604668 -37.81625546 | 51 | 71 |
| C12 | Radius 0.0005°Random | Random | 144.9668561 -37.80575068 | 32 | 51 |
| C13 | Radius 0.0005°Random, sorted | Random, sorted | 144.9592224 -37.81639721 | 20 | 32 |
| C14 | Radius 0.0005°Sorted by x | Sorted by x | 144.9598333 -37.81071970 | 134 | 150 of 169 |
| C15 | Radius 0.0005°Median first | Median first | 144.9637753 -37.81180871 | 37 | 58 |
| C16 | Radius 0.003°Original | Original | 144.9675203 -37.79883204 | 403 | 150 of 455 |
| C17 | Radius 0.003°Random | Random | 144.9438687 -37.82061514 | 92 | 121 |
| C18 | Radius 0.003°Random, sorted | Random, sorted | 144.9710206 -37.81788242 | 117 | 137 |
| C19 | Radius 0.003°Sorted by x | Sorted by x | 144.9669869 -37.80374563 | 1,041 | 150 of 1,058 |
| C20 | Radius 0.003°Median first | Median first | 144.9571667 -37.80429103 | 294 | 150 of 332 |
Read with care
What this does not show
- Grounding is agreement with the trace, not quality. An answer can repeat every number correctly and still explain the search badly; that judgement is yours, call by call, in the AI log.
- Twenty cases give wide intervals: even 20 of 20 has a Wilson lower bound near 84%. An exact McNemar p below 0.05 needs at least six disagreements in one direction: 6 against 0 (p = 0.031) or 8 against 1 (p = 0.039), for example; five all one way give only 0.0625.
- Answers vary between runs of the same model. A second run of the same model shows how much; compare it like any other run.
- The cases are synthetic searches from one seed. They cover the kinds of search the explorer offers, not the searches visitors actually make.