The Prompt Did Not Do It, Splitting the Stages Did
I tried to fix thin summaries with the prompt and made them worse. Separating number extraction from sentence writing raised recall 18 points and presence 2.6 points together
Dropping the model replacement left the incumbent’s weakness in place. It does not get things wrong, but it omits half the figures in the source. Naming the required items in the prompt pushed both metrics down together, and splitting number extraction from sentence writing pushed them both up.
What I set out to do
Last time I dropped the model replacement. All three candidates fell short. But the incumbent’s quality did not satisfy me either. The metrics made it obvious where the problem was. It is not wrong. It is thin.
In earnings-release summaries, 97.0% of the numbers it prints exist in the source. But it only carries 51.8% of the source’s headline figures. It drops half.
Reading the prompt suggested why. It asks for “3 to 5 key points” and says nothing about which figures to include. Naming the items to include seemed like it would do it.
Where I got stuck
It got worse. Both metrics, both languages.
| Condition | Presence | Recall | Item count met |
|---|---|---|---|
| Baseline | 97.0% | 51.8% | 199/200 |
| Required figures named | 95.2% | 46.3% | 140/200 |
| Named + more slots | 96.6% | 48.4% | 131/200 |
baseline: - Q4 revenue was $30.3M, down from $33.4M a year earlier, with a net
loss of $3.9M (a loss of $0.17 per share).
revised: - Q4 revenue was $30.3 million, and full-year revenue was $119.9 million.More figures went in, and the year-over-year comparison disappeared. Mandating specific items made it cram the numbers into one line and throw away the context. Fifty-nine documents produced only two items.
And the first drop of 11.6 points was so large that I went back to the grader. Counting which figures were being missed gave cash, total assets, and equity. I had set the recall target to “the four largest values in the document,” and values an earnings summary has no duty to carry were sitting in there. The revised prompt steered toward revenue, profit, and EPS, so it missed those even more. The editorially better choice was being penalized.
I narrowed the target to income-statement and guidance figures and rescored everything. The gap shrank to 5.5 points and the ranking held. The conclusion is the same, but had I reported before fixing it I would have left an inflated number on the record.
How I worked it out
Asking one request to be accurate, complete, and short at the same time was too much. So I split it.
Stage one pulls only the figures out of the full source. No sentences, one per line, in the source’s own notation. I told it not to pull balance-sheet items. Stage two never reads the source again. It receives the extracted list and the first 1,500 characters and writes the prose.
I split the measurement by stage too. Looking only at the stage-two result tells you nothing about which stage spoiled it. Stage-one extraction presence, stage-one extraction recall, and stage-two leak-in, meaning whether the final output contains numbers absent from the stage-one list. That last one is the safety check.
The first attempt failed. I had stage one pull only six fixed items, and extraction recall stalled at 23.7% while leak-in jumped to 42.9%. Stage one gave so little that stage two pulled numbers from the lede. The bottleneck was not stage two; it was the item count in stage one.
I opened stage one up: pull every income-statement and outlook figure.
Putting it together
| Condition | Lang | Presence | Recall | S1 recall | S2 leak-in | Per min |
|---|---|---|---|---|---|---|
| Single stage baseline | Korean | 97.0% | 51.8% | — | — | 18.8 |
| Single stage baseline | English | 99.9% | 56.9% | — | — | 24.3 |
| Two stage, open extraction | Korean | 99.6% | 69.8% | 78.2% | 6.6% | 4.7 |
| Two stage, open extraction | English | 100.0% | 48.6% | 66.9% | 23.3% | 7.4 |
| Two stage, six fixed items | Korean | 99.7% | 26.3% | 23.7% | 42.9% | 18.5 |
In Korean, recall rose 18.0 points and presence 2.6 points together. Prompt wording had pushed both down.
And this overturned an earlier conclusion. Last post, all four models scored lower on Korean numeric fidelity than English, so I wrote that the pipeline should refine in English and translate only when needed. With two stages, Korean at 69.8% beats the English baseline of 56.9%. The Korean penalty was something that appears when the model tries to do everything at once.
English gained nothing. Presence completes at 100% while recall goes down. Stage-one extraction recall is 66.9%, lower than Korean, and leak-in is 23.3%, higher. I did not measure why it splits by language.
The cost is throughput. It drops fourfold, from 18.8 documents per minute to 4.7. There are two calls per document and stage-one output is long. At 6,700 documents a day that is still surplus for this pipeline, so I took it.
What I learned
It was not a problem to solve by changing models. I spent two days measuring three candidates over 200 documents in two languages, and the conclusion was to keep what I had. The quality actually came from changing the processing structure on the same model. The metric was pointing at “thin” the whole time and I went digging through model lists first.
When you split into stages, you have to measure by stage. The first two-stage attempt had a final recall of 26.3%. Stopping there, I would have written “two-stage processing does not help.” Looking at stage-one recall of 23.7% and leak-in of 42.9% separately identified stage one as the culprit.
And you have to check what your metric rewards. My recall was measuring “did it mention the largest numbers,” which is not the same as “did it carry what matters.” I was judging editorial quality with a metric that pays you for writing down total assets. That is why a prompt improvement scored as a regression.
But stopping here would have been wrong. The test set has a median of 12,000 characters while real economic articles run 3,000 to 4,000. What the two-stage split fixed might actually be a length problem. The next post separates them.
Source material: docs/ECON_MODEL_EVAL.md. The reproduction commands are in that document’s usage section.