BL-EDU: Brazilian school content.
LUA's own benchmark on Brazilian school content: objective questions from the 2024 and 2025 national high-school exam (ENEM), with the official INEP answer key. Only the LUA Genesys PI House model was measured, no competitor in this round.
Result.
| Cut | n | Accuracy (%) | 95% CI | Error when attempted |
|---|---|---|---|---|
| Overall, 2024+2025 | 354 | 99.7 | 98.4–100.0 | 0.3 |
By area.
| Area | n | Accuracy (%) | 95% CI |
|---|---|---|---|
| Linguagens | 90 | 98.9 | 94.0–99.8 |
| Ciências Humanas | 90 | 100.0 | 95.9–100.0 |
| Ciências da Natureza | 86 | 100.0 | 95.7–100.0 |
| Matemática | 88 | 100.0 | 95.8–100.0 |
By edition.
| Edition | n | Accuracy (%) | 95% CI |
|---|---|---|---|
| 2024 | 179 | 100.0 | 97.9–100.0 |
| 2025 | 175 | 99.4 | 96.8–99.9 |
Image coverage: 125 of 125 questions with an image, chart or table were run, all through the INEP's official audio-description transcript, with 100.0% accuracy.
The round's only mistake: a 2025 Language Arts question where the answer came with a stated confidence of 97, against the official key.
95% CI by Wilson. Score with penalty t=0.5: 99.6.
Methodology.
| Model | LUA Genesys PI House, public API api.lua.vision, medium effort. |
|---|---|
| Data | Official ENEM 2024 and 2025 booklets (version with image transcribed to text) and the INEP's final printed answer key. |
| Scoring | Letter extracted by regular expression, no automatic judge. |
| Runs | One per item, pass@1. |
Contamination.
No declared training cutoff exists for the evaluated model, so this page does not estimate one. Both editions used (2024 and 2025) have been public for over ten months, plenty of time to enter any web-collected corpus — both are contamination candidates, not just the older one. Accuracy by edition did not drop on the more recent exam (2024: 100%; 2025: 99.4%, overlapping intervals), which does not point to memorization concentrated in one of the two, but it also does not separate "a very good model" from "both were seen in training". No memorization probe was run in this delivery. The 99.7% accuracy is high even for a widely discussed public exam, and should be read as this round's result, not as a capability confirmedly free of contamination.
Essay.
1000 out of 1000 on each of the 10 essays, by all 4 automatic evaluators, on all five competencies (200 out of 200 each). The essays come from 2 official topics, ENEM 2024 and 2025.
| Competency | Mean score (0 to 200) |
|---|---|
| I · Command of formal writing | 200 |
| II · Understanding of topic and structure | 200 |
| III · Argumentation | 200 |
| IV · Cohesion | 200 |
| V · Intervention proposal | 200 |
Real ENEM essays are graded by people. The evaluators in this section are language models applying the official matrix, and the score is not an INEP score.
Evaluators
| Automatic evaluator | Provider | Essays graded | Mean score (0 to 1000) |
|---|---|---|---|
| Mistral Large 3 | Mistral AI | 10 / 10 | 1000 |
| DeepSeek-V4-Pro | DeepSeek | 10 / 10 | 1000 |
| Kimi K2.6 | Moonshot AI | 10 / 10 | 1000 |
| GPT-5.5 | OpenAI | 10 / 10 | 1000 |
Method
| Essays | Written by LUA Genesys PI House from each topic's official prompt and supporting texts, with no extra prompt. Five essays per topic. |
|---|---|
| Matrix | Official INEP matrix of five competencies, 0 to 200 each, with 6 levels. Same matrix and same prompt for all 4 evaluators. |
| Grading | One grading per essay per evaluator. Going off-topic voids the essay, a text of up to 7 lines scores 0 and, without an intervention proposal, competency V scores 0, as the matrix states. |
Control
To find out whether the evaluators see flaws, we deliberately damaged 3 of the 10 essays in four mechanical ways, inventing no content, and all 4 evaluators graded the damaged texts. Mean of the 3 essays per cell, 0 to 1000.
| How we damaged it | Mistral Large 3 | DeepSeek-V4-Pro | Kimi K2.6 | GPT-5.5 |
|---|---|---|---|---|
| Without the intervention-proposal paragraph | 960 | 960 | 587 | 773 |
| First paragraph only | 840 | 707 | 0 | 213 |
| No punctuation and no accents | 960 | 987 | 893 | 907 |
| Essay on a different topic | 173 | 0 | 0 | 0 |
No evaluator gives full marks to a damaged text, and 3 of 4 score the off-topic text zero. They are not equally strict: the same damaged essay gets different scores depending on the evaluator.
We also evaluated
- Copying of the supporting texts: 0 of 10 essays have a passage of 8 or more words identical to a supporting text of their own topic.
- Literal comparison with 10 real participant essays published by INEP: 0 of 10 have a passage of 6 or more identical words.
- Fit to the topic, by semantic distance between the essay and its own topic's prompt against the other topic's: the difference is positive in 10 of 10 essays (from 0.20 to 0.23).
- Repetition among the essays themselves: 6 pairs of essays on the same topic share passages of 12 to 21 words, stock-repertoire sentences such as the citation of Law 10,639/2003. Among participants' essays, the longest shared passage is 10 words.
Essay limitations
- The evaluators are models, not INEP graders, and there was no calibration against a human grader. The score shows what the matrix applied by models assigns to these essays, not what a human grading would assign.
- Small sample: 10 essays and 2 topics, one generation per essay.
- The plagiarism comparison is literal: a close paraphrase does not show up. The web check was not performed, and no finding does not mean no plagiarism.
- The participant essay corpus has 10 texts, transcribed from the pages of INEP's booklet.
- The evaluators differ in strictness, as the control shows, and the score depends on which one grades.
Rerun it yourself
The package carries the matrix, the topics and the comparison corpus. Anyone who wants to repeat the round needs API access and bears the evaluators' cost: Reproduce or request access →
What did not run.
- Tutor: not executed. It would measure whether LUA points out a student's mistake without handing over the finished answer, but it would need a real student-error case; inventing a fictional one would fabricate the source data, which the benchmark exists to avoid.
Limitations.
- Measures whether LUA gets exam questions right, not whether it teaches. A high score here does not license saying the product teaches well.
- N per area sits between 86 and 90, so the per-area confidence interval is wide, 4 to 6 points; only the two-edition aggregate has the narrower interval.
- The IRT-estimated proficiency uses INEP's published parameters outside a real sitting; it is not comparable to a real candidate's ENEM score, and at the Math extreme (100% accuracy) the estimate extrapolates beyond the region where the calibration has the most reference candidates.
- Language coverage: only the English option of the foreign-language questions.
Package and files.
b21c8fa5e9e55fd0cb6a1eccbe4c9d6c6f0b1fa91fe25dda293dabc7a175e9da| File | Size | Download |
|---|---|---|
bl-edu-2026-09.tar.gz | 42 KB | Download |
| Per-item result | SHA-256 | Download |
|---|---|---|
resultados/e1-objetivas.house.jsonl | 2d842b0ad37aa5eeaba1ab35c9a682f6a8f64789c2bec7b15fc651a9a851c57a | in the package |
resultados/e2-redacao.house.jsonl | 3a367ae6afb4ae37945beade19e721d3e67cfd052b8db54e9085b41d92f749fa | not in this version: carries internal deployment identifiers |
resultados/e2-redacao.rejulgado.jsonl | 4c145ee00a6f2df40eeb89561a401fed2093f720835cef997044cee40cb62623 | not in this version: carries internal deployment identifiers |
resultados/e2-controle-discriminacao.jsonl | fcff7fd57f5006c8e79c2b6ee3c50fc3c6fcc8ec4a00cef04d531c18dfa4d6bc | not in this version: carries internal deployment identifiers |
resultados/e2-originalidade.json | c03e23c58f0b91126cdf734f0fe9bbe1ef85e781ff1fa1276e89dca5392fb77c | not in this version: carries internal deployment identifiers |