I Cut the Input to a Third, and What I Thought I Had Fixed Split in Two
Two-stage processing fixed both numeric accuracy and omission. Measuring again on short inputs showed one was a length problem and the other was a structure problem
Two-stage processing fixed numeric accuracy and omission together. But the test set had a median of 12,000 characters and real economic articles run 3,000 to 4,000. Stripping the tables and appendices from the same documents and measuring again split what I thought I had fixed into one length problem and one structure problem.
What I set out to do
Last post I split summarization into two stages. Stage one pulls figures from the source, and stage two writes prose from that list without reading the source again. In Korean, numeric presence rose from 97.0% to 99.6% and recall from 51.8% to 69.8%, together.
I could have ended there feeling good. Then I looked at the test set again: a median of 12,000 characters. That is an SEC earnings release with the financial tables attached. Real economic articles run 3,000 to 4,000.
Was what I fixed really a structure problem, or just a problem created by long input?
Where I got stuck
Separating those required a set that differed only in length. I went through English economic news feeds and every one was blocked. No feed gives full text, and descriptions cap at 279 characters. Scraping the article pages runs into paywalls and JavaScript.
Fed and SEC press releases are static pages, so those came through. They are genuine economic news with no redistribution restrictions. Then I measured them and they were unusable.
| Document | Body | Multiplier and percent expressions |
|---|---|---|
| Bank supervision guidance | 2,388 chars | 0 |
| Community banking | 1,771 chars | 2 |
| Monetary policy | 693 chars | 0 |
The length is news-like and short, but there are no numbers. A metric that measures numeric fidelity does not hold.
How I worked it out
I cut the same documents instead. I stripped the financial tables and appendices from the earnings releases and kept only the narrative. That is the actual article copy companies distribute to the wires, and more importantly, because it is the same document, length is the only variable. Median 3,598 characters, 29% of the original.
I cleared out debris three times while building it. Parenthesized area codes survived and got picked up as figures, documents that were slide attachments and opened with a disclaimer got mixed in, and the regex finding the lede matched mid-word, cutting FEBRUARY to EBRUARY. A hundred and twenty-two documents remained.
Putting it together
| Configuration | Long presence | News presence | Long recall | News recall |
|---|---|---|---|---|
| Single stage, Korean | 97.0% | 99.7% | 51.8% | 60.1% |
| Two stage, Korean | 99.6% | 99.8% | 69.8% | 82.0% |
Getting numbers wrong was a length problem. With shorter input, a single stage alone goes from 97.0% to 99.7%. Short inputs do not need two stages.
Leaving numbers out was a structure problem. Even with the length reduced, a single stage stalls at 60.1% while two stages reach 82.0%. The two-stage gain is actually larger at news length: 18.0 points on the long documents, 21.9 points on the news-length ones.
Ask one request to be accurate, complete, and short at once, and the model abandons complete even when the input is short. It is not a needle-in-a-haystack problem. It is having been handed all the tasks at the same time.
That also explains my earlier note that English gains nothing from two stages. On long documents, English recall fell from 56.9% to 48.6%. At news length it rises from 66.6% to 74.3%. The extraction stage was simply recovering fewer figures from long English text. Extraction recall splits 66.9% against 83.9%.
What I learned
When you see an effect, separate out what it is an effect of. Watching two stages lift both metrics together, I nearly stopped there. Left mixed, all you have is the blunt sentence “two-stage processing is good.” Separated, it becomes: one stage is enough for short input, and two stages are needed when completeness matters. The second one is a sentence you can actually act on.
And the test set decides the conclusion. I chose SEC earnings releases because I could get full text and they are dense with numbers. That choice had me measuring with 12,000-character inputs, three times what I will actually feed it. Without changing the set I would have left the wrong conclusion that two stages hurt in English.
As a side effect I fixed this repository’s documentation too. The 262,144 context was described only as “a sliding window, so no concurrency penalty.” That is a throughput statement. Quality collapses at 13,000 tokens, which is 5% of the limit. I wrote one line recording that the length it accepts and the length where accuracy holds differ by roughly twentyfold.
Source material: docs/ECON_MODEL_EVAL.md. Test set generation is tests/build_news_testset.py.