BL-OAB: Brazilian law as it stands.

LUA's own benchmark on current Brazilian law: final booklets and answer keys from the unified Bar exam, first and second stage. Only the LUA Genesys PI House model was measured, no competitor in this round.

RunLUA Record · run by LUA Vision · not an independent evaluation

Result.

Would pass the first stage in all seven official editions and would score a passing grade (≥ 6.0) on all 21 complete second-stage exams.

Benchmark	Accuracy (%)	95% CI	Graded	Error when attempted	Score t=0.75
First stage	99.3	98.2–99.7	552/556	0.0	99.3

Benchmark	Score /10	95% CI	Pass rate ≥6.0	Complete exams
Second stage	9.41	9.19–9.59	100.0%	21

95% intervals: Wilson on the first stage, percentile bootstrap (2,000 resamples) on the second. Items annulled by the exam board leave the denominator.

Methodology.

Model	LUA Genesys PI House, public API `api.lua.vision`, medium effort.
Data	Final booklets and answer keys (post-appeal) from the seven most recent unified Bar exam editions, and the final answer standard from the three most recent editions with a published grading guide.
First stage	Letter extracted by a fixed regular expression. No letter counts as an error.
Second stage	Three runs per task. Each answer goes to a panel of two automatic judges from different families, who apply the official grading standard item by item; the task score is the average of the two.
Judge	Automatic judge with the benchmark's official prompt, sample reviewed by a human. No judge is the measured column.

By edition.

Edition	n	Accuracy (%)	Points, benchmark	Points, board	Would pass?
41	80	98.8	79/80	79/80	Yes
42	79	98.7	78/80	79/80	Yes
43	78	98.7	77/80	79/80	Yes
44	80	100.0	80/80	80/80	Yes
45	80	98.8	79/80	79/80	Yes
46	80	100.0	80/80	80/80	Yes
47	79	100.0	79/80	80/80	Yes

Contamination.

The benchmark uses only editions after the oldest one included in a public reference dataset for the Bar exam: earlier editions enter as a separate row, marked as such, and the score for both slices sits side by side in the package. No memorization probe was run in this delivery.

Limitations.

Only the official layer. No newly written items reviewed by lawyers, which would give the open and private layers of the full design.
No memorization probe and no human review of the second stage: the open-ended score comes only from the automatic judge panel.
The grading standard belongs to the exam board, not absolute truth.
A technical failure confirmed by retrying, not instability. Five questions, out of 556 objective items and 105 open-ended tasks, always come back empty from the API (status 200, no content, no refusal or content-filter signal). The call was repeated across three separate rounds: the four objective items failed on every attempt, and the open-ended one failed on all nine attempts (three runs in each of three rounds). These are the only five questions, out of 661 total, that narrate a violent crime in detail — attempted femicide, revenge pornography, armed robbery, rape of a vulnerable person and homicide committed by a minor. They are standard legal-exam prompts, the kind of case any criminal-law student studies; the benchmark did not choose these topics, they came from the exam board's official test. Each one scores zero in the denominator, and the result captures that silence for what it is: a silent, topic-based block on a legitimate professional use, not noise to discard. Investigating the cause of the block is outside this benchmark's scope.

Package and files.

bl-oab-2026-09.tar.gz5edd2382a74e061b67154a73e3de6acde4fff5d0ac9d42cdb0a53aa9def43df4

File	Size	Download
`bl-oab-2026-09.tar.gz`	35 KB	Download

Per-item result	SHA-256	Download
`resultados/oab_1f.house.jsonl`	`d063f77759756fa797d1302153327c7068302a8af66cb7005dd337a39db9339c`	Download
`resultados/oab_2f.house.jsonl`	`eb33e1261be06445d1f05c247fde332a67ecdd0b3753cfb017f38e330f435dc5`	withheld: names the judge