LUA VISIONENAMED: the 2025 medical exam.
Multiple-choice questions from the 2025 Brazilian national exam for graduating medical students (ENAMED). It measures clinical knowledge in Portuguese, in the form the country tests it.
Results.
In each row, the best value is highlighted. Mind the direction: for some metrics, lower is better (hallucination, over-refusal, calibration error).
| Model | Accuracy (%) ↑ higher is better | 95% CI | Answered | Refused or blocked | No answer |
|---|---|---|---|---|---|
| LUA Genesys PI House | 96.6best | 92.0–100.0 | 85 | 2 | 0 |
| GPT-5.5 | 96.6 | 92.0–100.0 | 85 | 2 | 0 |
| GPT-5.4 | 94.3 | 88.5–98.9 | 85 | 2 | 0 |
| Grok 4.6 | 95.4 | 90.8–98.9 | 87 | 0 | 0 |
| DeepSeek-V4-Pro | 94.3 | 88.5–98.9 | 87 | 0 | 0 |
| Kimi K2.6 | 93.1 | 87.4–97.7 | 86 | 1 | 0 |
| Sabiá 4 Thinking† | 94.4 | — | — | — | — |
Accuracy (%) · higher is better
Axis from 85 to 100. Black bar: 95% confidence interval.
How it was run.
| Date | 24/09/2026 |
|---|---|
| Dataset | recogna-nlp/enamed-2025, commit 0f821fc60a2d, file dataset.json |
| Items | 87 of 90. 3 are out: questions with an associated image or with no A-to-E key, for every model. |
| Scoring | Exact letter, extracted by regular expression from the final line "Resposta: X". No judge. |
| Runs | One per item, pass@1. |
| Failures | An unanswered question scores zero and stays in the denominator. That includes refusals by the provider's content filter and refusals by the model itself. |
| Interval | 95% interval by percentile bootstrap, 4,000 resamples over items, fixed seed. |
| SHA-256 of the data | dados/enamed.jsonl2fe5851a37f11f891c1ac4d524d2eb3ce61ee1369310cc244bcd2569f2e386ea |
Setting per model.
Every model got the same messages, through the commercial API of each vendor or of the cloud where the model is published, on the same date. No extra system prompt, no temperature, no tools beyond those the benchmark defines.
| Model | Access | Reasoning effort |
|---|---|---|
| LUA Genesys PI House | public API api.lua.vision | medium |
| GPT-5.5 | API comercial | medium |
| GPT-5.4 | API comercial | medium |
| Grok 4.6 | API comercial | model default |
| DeepSeek-V4-Pro | API comercial | model default |
| Kimi K2.6 | API comercial | model default |
| Sabiá 4 Thinking † | figure published by the vendor | — |
Protocol decisions.
- A single zero-shot multiple-choice prompt, the same for every model: the model reasons and ends with "Resposta: X". The prompt is in the package.
- Questions with images are dropped for every model, because not every compared model reads images.
Defects found in the data.
- ENAMED_64 and ENAMED_67 deal with suicidal ideation and a gunshot wound. The LUA column refused both, and the cloud content filter blocked both for GPT-5.4 before the model answered. They are legitimate medical exam questions and score zero.
Contamination.
The items are in public datasets. Any column, LUA included, may have seen these questions in training. This round did not measure contamination, so read the figure as a ceiling on what the model knows about the subject, not as proof of reasoning on unseen items.
Limitations.
- One run per item. Run-to-run variation of the same model was not measured.
- A single zero-shot multiple-choice prompt. Another prompt may change the order of the columns.
- The figure holds for the model served by the API on 24/09/2026. A new version needs a new round.
- Questions with images were left out for every column. The result says nothing about reading figures.
Reproduce.
The package holds the code that calls the models, builds each test, scores it and builds the table, plus the aggregated table for this round. The data is downloaded by preparar_dados.py at the exact versions and checked by SHA-256. If one byte changes, the script stops.
With Python 3 and Node 20 or newer:
tar -xzf bancada-brasil-2026-09.tar.gz && cd bancada-brasil-2026-09
shasum -a 256 -c <(sed -n '/^Arquivos deste pacote/,/^$/p' VERIFICAR.txt | grep -E '^[0-9a-f]{64}')
pip install huggingface_hub pyarrow
python3 preparar_dados.py # baixa, normaliza e confere SHA-256
export LUA_API_KEY=... # chave de avaliação da API LUA
node run.mjs --bench enamed --modelo house
node tabela.mjs1144df20af11b5c0141614c256fa1733c698bf829be3303842839ec149a1f81111 KB · Raw per-item answers are not in this package.History.
24/09/2026 · first round, published on this page.