I Measured Four Models on 200 Documents, and What I Did Not Measure Overturned the Result
On numeric fidelity the challenger won. Then I found that Korean number spacing was broken in all 200 outputs
I was trying to replace the model behind an economic-news pipeline, so I measured four candidates against 200 earnings releases. One challenger led on numeric fidelity, and then I looked at the actual outputs and found Korean number spacing broken in all 200 of them. Before that, it turned out my grader had been wrong by 22 percentage points.
What I set out to do
I wanted to build a pipeline that collects and refines economic news around the clock. I suspected there was something better than the server’s gemma-4-26B-A4B, so I looked at replacing it. The target was Qwen3.8-Flash-Next. A 180B total with 6B active MoE seemed right for a machine that is bandwidth-bound.
I could not produce ground-truth labels. So I picked only metrics that compare the output against the source mechanically. I built a test set of 200 earnings releases.
Where I got stuck
The candidate did not fit on the machine.
Six billion active parameters only relieves the bandwidth bottleneck. What has to be resident is decided by the total parameter count. The smallest quantization vLLM would load is 132.7GB of weights alone, and total unified memory is 119GB. It does not fit even with the KV cache at zero.
I picked three alternates and ran them. The first number looked wrong. Presence rate 74.1%, meaning one in four figures was invented, except the outputs looked fine.
source: $37,536 (table in thousands)
output: 3,753만 6,000달러
grader: no 3753, no 6000 → 2 hallucinationsThe model had transcribed it exactly. The grader did not understand Korean myriad grouping and was comparing strings, so it flagged correct values as hallucinations. Twenty-two percentage points were the measuring instrument’s fault.
How I worked it out
I matched on value instead of string. Fold the Korean units together, then compare ignoring digit grouping. It rose to 96.7%.
More defects kept surfacing while I fixed it. A missing unit folded 285만 3천 into 2,850,003. Floating-point tails from the multiplication printed to nineteen digits. And Qwen3-Next-80B scored 0 of 60 on format compliance because I had attached a parser for a reasoning format that model does not emit, so its output was swallowed whole. The Instruct build has no <think>. Five seconds with the chat template would have told me.
Half the metric was missing too. Presence rate only asks whether the numbers you printed are real. Print one correct number and you score 100%. What you left out is invisible. I added recall.
I also compared unrelated documents against each other to get a chance-agreement floor. Everything has to be read above 7.5%.
Putting it together
Four models, 200 documents, thinking off.
| Model | Korean presence | Korean recall | English presence |
|---|---|---|---|
| gemma-4-26B-A4B | 97.0% | 51.8% | 99.9% |
| Qwen3.5-35B-A3B | 93.0% | 52.2% | 99.1% |
| GLM-4.7-Flash | 91.8% | 35.5% | 99.4% |
| Qwen3-Next-80B | 84.6% | 34.5% | 98.7% |
The largest one, at 80B, was the worst. It sits 12.4 points below the incumbent 26B on Korean. Size decided only whether it could be loaded; it predicted nothing about quality.
Only Qwen3.5 led on recall. I pulled the outputs side by side to read them. That is where this showed up.
2025 년 4 분기 서비스 수익은 3 억 810 만 달러로 전년 동기 대비 5.3% 증가했으며There are gaps between the numbers and their units. All 200 documents. gemma had zero. The reason its recall was higher came out at the same time. It writes more numbers, and it writes them in a form you cannot put on a page.
The metric could not see this. It only asks whether a number exists in the source, so the score stays high while the spacing is broken.
What I learned
When a measurement looks wrong, suspect the grader first. This evaluation turned up six defects in the grader and harness, and five of them would have read as model quality problems if left alone. A presence rate of 74.1% was actually 96.7%, and format compliance of 0 out of 60 came from me attaching the wrong parser.
A good metric does not mean it is usable. On numeric fidelity the challenger won. In practice it is unusable. With no ground-truth labels I picked “only what I could measure,” and something outside that selection overturned the conclusion. Narrowing candidates with automatic metrics is right, but you cannot skip the step of reading the outputs after you narrow.
So I dropped the replacement. But the incumbent’s quality did not satisfy me either. That looked like a contradiction with the metrics and was not. The incumbent model does not get things wrong. It is thin. In one case it missed all four headline figures and wrote only quarterly revenue. The prose was clean.
Changing the model does not fix it, and the metric was already pointing at that weakness. What is left is the prompt and the processing structure. The next post goes after that.
Source material: docs/ECON_MODEL_EVAL.md, docs/ECON_SAMPLES_KO.md (per-model outputs side by side), tests/test_numgrade.py (grader regression tests — every case is an input it actually got wrong).