Can a vision model read a nutrition panel? 391 hand-labelled images across 27 languages, and a trade-off nobody had measured.
Nutrition data feeds carb counting for insulin dosing and sodium limits for hypertension, so a misread number travels. I screened 4.73M Open Food Facts products to build a benchmark, then ran four prompt variants over the same images — 1,173 calls, zero failures — to ask not how often the model is right, but when you can tell.
See what it costs to constrain outputHalf of Open Food Facts carries a nutrition photo, but only a quarter of those record values per 100g — the rest are per serving, where the stored figure and the printed one disagree by construction. Requiring all seven nutrients leaves 5.1% of the corpus.
The constraint is not photography. It is the basis products are recorded on.
A benchmark score is an average, and you cannot act on an average. So the question was not how accurate the model is, but whether its own consistency tells you which answers to trust.
An average tells you how good it is.
A confidence signal tells you what to do.
Instructing the model to return JSON and nothing else raised the parse rate from 75% to 97% and cut output tokens by 77% — and made it worse at the task.
European panels state values per 100g; American panels state them per serving and need a conversion. The format instruction costs nothing on the first and 29 points on the second — it removed the working space the arithmetic needed.
European labels print energy twice — 180kJ / 42kcal. The model returned 180 as kilocalories: not a misread digit but the wrong column, a fourfold overstatement recurring on every dual-unit panel.
Detected by ratio, not by eye — any value within 3% of 4.184× the truth. One line of prompt guidance cut it from 38% of energy fields to 10.4%.
Averaging the four variants made things worse — six points below the best single arm. They share a bias, so the median cannot rescue them. But their disagreement still carries information.
Agreement is not correcting errors. A panel that is blurry, oddly laid out, or needs conversion pushes at least one variant off on its own — consensus detects difficulty.
Run four cheap variants. Auto-accept the 49% where they agree at 82%; route the other 51% to a reviewer. That is an operating procedure with a known review rate, not a leaderboard position.
It is also the ceiling: 82% is not good enough for data feeding insulin dosing. Consensus gives you a usable confidence signal — it does not make the extractor ready.
4.7M products
12 GB, never decompressed
242,597 candidates
stratified by language
391 panels by hand
panel / partial / not-panel
4 arms · 1,173 calls
field-level accuracy
The nutrition-image slot is crowdsourced and does not always hold a panel — one product had a photo of the jar. I checked all 391 by eye before spending an API call. The 3% that were not panels were excluded, not counted as model failures.
Equal numbers per language, so English (192,661 candidates) and Czech (2,411) carry the same weight. That is deliberate — it gives equal power to test language effects, and makes any corpus-level figure something you would have to reweight.
Open Food Facts is self-reported. Several products record salt at 0.01g per 100g against label readings of 8–45g — implausible for packaged food, so the database is probably wrong there, not the model. Until each is checked by eye, every figure here is a lower bound.
The 90B variant returned no successful responses for this account and two alternative models were not provisioned. Every finding describes Llama 3.2 11B Vision specifically and may not generalise.
English scores worst of the major languages — but English-language products here are disproportionately American, and American panels are per serving. The language effect is largely a format effect, and this sample cannot separate them.
This measures extraction against a reference database. It does not show the system saves anyone time, and no reviewer has used the triage rule in practice.
A triage rule is only worth stating if someone can work the queue it produces. So I built the queue: the 342 of 391 panels where the consensus got at least one field wrong, ordered by severity, each with the label image, both answers, and a written reason for the flag.
Cards marked kilojoule substitution are the named error. Cards marked ground truth suspect are the opposite case — the stored value is implausible and the model is probably right. Which one a disagreement turns out to be decides whether the figures above are a floor or a ceiling.
No inference happens here. All 1,173 calls ran in one 35-minute batch; the app reads a 435 KB summary of what came back. No API key, nothing to spend, no way to submit your own photograph.
It is also the part I have not finished. Working this queue by hand is what would separate model error from database error, and turn every number on this page into a measurement rather than a lower bound.
Best single-arm accuracy was 58.4% — short of production. The useful output was learning which half you can trust.
Three results worth keeping: format constraints cost accuracy exactly where reasoning is required; a named failure can be found by ratio and reduced by one line of prompt; and inter-prompt agreement predicts correctness well enough to route work.
Review the disagreements by hand to separate model error from database error. Then test whether cropping to the panel beats any prompt change.
Data from Open Food Facts, September 2026 export: product data under the Open Database License, images under CC-BY-SA. Inference via NVIDIA NIM using meta/llama-3.2-11b-vision-instruct. Nutrition values are self-reported and are not a validated reference standard, so findings describe extraction against that database rather than against the physical labels. An evaluation exercise; not intended to support dietary or clinical decisions.