LUA VISIONAbstentionBench
Whether the model abstains when it should: questions with no known answer, underspecified, with a false premise, subjective or out of date.
Results.
Comparison with a single automatic judge per benchmark, the same for all six models, outside the evaluated model families, with the benchmark's official prompt. LUA enters with the same answers as the 25/09/2026 record, graded again by that judge. The standalone record of 25/09 used other judges; its figures are further down this page.
In each row, the best value is highlighted. Mind the direction: for some metrics, lower is better (hallucination, over-refusal, calibration error).
| Model | Abstention F1 ↑ higher is better | Recall ↑ higher is better | Precision ↑ higher is better |
|---|---|---|---|
| LUA Genesys PI House | 72.569.8–75.0best | 64.961.7–68.0best | 82.179.0–84.8 |
| GPT-5.5 | 67.864.8–70.6 | 54.651.3–57.9 | 89.386.4–91.7 |
| GPT-5.4 | 65.762.7–68.6 | 53.550.1–56.8 | 85.382.0–88.0 |
| Grok 4.6 | 66.263.4–69.1 | 52.248.9–55.6 | 90.487.5–92.7 |
| DeepSeek-V4-Pro | 65.963.1–68.7 | 54.451.1–57.8 | 83.380.0–86.2 |
| Kimi K2.6 | 70.768.1–73.3 | 60.457.1–63.6 | 85.182.1–87.8 |
| Sabiá 4 Thinking | — | — | — |
Abstention F1 · higher is better
Axis from 60 to 80. Black bar: 95% confidence interval.
Comparison files
| Per-item result | SHA-256 | Download |
|---|---|---|
resultados/abstention.house.rejulgado.jsonl | 0230d79a74260a4aced7d29571850a79970fe73bd2326485f6497231a099bbc2 | withheld: names the judge |
resultados/abstention.gpt55.jsonl | 7c88a9f9e4875bc3cff1d4ecb15d49f481dfc2894746d54da47e70e08eef16d7 | withheld: names the judge |
resultados/abstention.gpt54.jsonl | 8de18028f01b92631ee3488f0728fb18f8d87ec5c27e714563b0353d0d0cc6ff | withheld: names the judge |
resultados/abstention.grok.jsonl | 6bb01490b6656a6ff07a98d0a5d81df4f19c3ea744cc82c107fa6f622fa781d5 | withheld: names the judge |
resultados/abstention.deepseek.jsonl | dceef4950539caa8a63705511bb0fd4547f27006409994fb18200ff058ffb0bb | withheld: names the judge |
resultados/abstention.kimi.jsonl | 38b2993bdd4919d40e310ccd8797669fad3a5f4d3e0e6bf595ba7aafca8b4afc | withheld: names the judge |
How the comparison was run.
Every model got only the question, through each vendor's commercial API, with no extra prompt, no search and no tools. One run per item. Unanswered questions stay in the denominator.
Provider block: when the vendor's content filter refuses the request before the model answers (HTTP 400 or a declared filter), the item counts as a refusal or an abstention, depending on the benchmark, and is counted separately, per model, in the table below.
| Model | Access | Reasoning effort | Provider blocks | No answer |
|---|---|---|---|---|
| LUA Genesys PI House | public API api.lua.vision | medium | 14 | 0 |
| GPT-5.5 | API comercial | medium | 12 | 0 |
| GPT-5.4 | API comercial | medium | 13 | 0 |
| Grok 4.6 | API comercial | model default | 14 | 2 |
| DeepSeek-V4-Pro | API comercial | model default | 22 | 1 |
| Kimi K2.6 | API comercial | model default | 24 | 1 |
Standalone record, 25/09/2026.
LUA only, with that round’s judges, which differ from the comparison judge above. These figures do not compare with the table above.
| Metric | Value | 95% CI |
|---|---|---|
| Recall | 66.5 | 63.3–69.6 |
| Precision | 75.7 | 72.5–78.6 |
| F1 | 70.8 | 68.3–73.4 |
n = 2,000. 855 items call for abstention; LUA abstained on 752, and 569 of those abstentions were right. Recall and precision go together: abstaining on everything gives high recall and low precision. 14 items were refused by the API itself and count as abstention.
By question type
| Type | n | Call for abstention | Recall (%) | Precision (%) |
|---|---|---|---|---|
| underspecified context | 1,207 | 514 | 62.6 IC 58.4–66.7 | 84.7 IC 80.8–88.0 |
| false premise | 221 | 121 | 90.1 IC 83.5–94.2 | 82.6 IC 75.2–88.1 |
| unknown answer | 151 | 61 | 68.9 IC 56.4–79.1 | 67.7 IC 55.4–78.0 |
| subjective | 127 | 68 | 26.5 IC 17.4–38.0 | 90.0 IC 69.9–97.2 |
| underspecified intent | 120 | 22 | 95.5 IC 78.2–99.2 | 36.2 IC 25.1–49.1 |
| stale | 104 | 25 | 72.0 IC 52.4–85.7 | 31.0 IC 20.6–43.8 |
| safety | 50 | 40 | 90.0 IC 76.9–96.0 | 97.3 IC 86.2–99.5 |
| no scenario in the paper | 20 | 4 | 75.0 IC 30.1–95.4 | 60.0 IC 23.1–88.2 |
A type with few items calling for abstention has a wide interval. Read the interval before the figure.
How it was run.
| Call | LUA Genesys PI House, genesys-pi-house, public API api.lua.vision, reasoning effort medium. No temperature, no token cap, no extra prompt, no search and no tools. |
|---|---|
| Dataset | AbstentionBench, fast subset, commit e2918417: 20 datasets × 100 items. arXiv 2506.09038 |
| Items | 2,000. The fast index lists 21 datasets. Averitec was left out because it has no loader in the benchmark repository. |
| Judge | Automatic judge with the benchmark's official prompt, without the original judge model. The judge that graded the answers is not the one the benchmark authors used. The grading prompt is the official one, unchanged, but a different judge can label the same answer differently. That can move the figures up or down. |
| Runs | One per item. A refusal by the API itself (HTTP 400, safety policy) is a final answer and is counted separately. |
| Interval | Wilson for proportions. Percentile bootstrap with 2,000 resamples and a fixed seed for F1, Omniscience Index, ECE and Brier. |
Package and files.
The package holds the code that calls the model, builds each test and computes the metrics, the generated sets and the aggregated table. The per-item result files are left out of this public version, because every line records which judge graded the item. The SHA-256 of each one is below, matching the round's table.
ff68d8db16a4d3944e635b50a4a9b28f7009fb0cb94fa37510c1e579681b0f82| File | Size | Download |
|---|---|---|
registro-veracidade-2026-09.tar.gz | 41 KB | Download |
| Per-item result | SHA-256 | Download |
|---|---|---|
resultados/abstention.house.jsonl | d6ea6160050360e74d906e6163e71c9b4aa2b658383cd931b15d7b37d4f78ba5 | not in this version |
Left out of this public version for the same reason: modelos.mjs, benches/abstention.mjs, benches/hallulens.mjs, benches/omniscience.mjs, benches/orbench.mjs, benches/orbench_hard_j2.mjs, benches/orbench_toxic_j2.mjs, benches/simpleqa.mjs, benches/simpleqa_conf.mjs, benches/xstest.mjs, README.md, colunas.json, tabela.json. Without them the package does not run on its own. The full version depends on a pending decision about naming the judge.
Limitations.
- Recall and precision go out together: abstaining on everything gives high recall and low precision.
- The judge that graded the answers is not the one the benchmark authors used. The grading prompt is the official one, unchanged, but a different judge can label the same answer differently. That can move the figures up or down.