#evaluation
7 posts tagged evaluation. · All tags
- The Prompt Did Not Do It, Splitting the Stages Did I tried to fix thin summaries with the prompt and made them worse. Separating number extraction from sentence writing raised recall 18 points and presence 2.6 points together
- I Cut the Input to a Third, and What I Thought I Had Fixed Split in Two Two-stage processing fixed both numeric accuracy and omission. Measuring again on short inputs showed one was a length problem and the other was a structure problem
- I Measured Four Models on 200 Documents, and What I Did Not Measure Overturned the Result On numeric fidelity the challenger won. Then I found that Korean number spacing was broken in all 200 outputs
- Measured Separately Is Not Measured Together, and 3% Utilization Measured thinking mode on an offline set, concluded no effect, then it reversed on a multi-step set. Meanwhile a never-measured combination went into production. The max_tokens cap was a breaker, not a budget, and after four months of tuning, the hardware's utilization was 3%
- The Grader Was Wrong More Often Than the Model Over 800 validation runs while I slept; the morning failure list had 8+ items that were measurement-tool defects. Fixed the tools, then 1,000 config experiments on the real defects that remained. Three cleared, four walls
- I Decided to Re-measure the Hardware from Scratch An orphan number called sm_121, a 6:0 SGLang win that flipped once speculative decoding was on, and a public-benchmark leader losing on my own labeled data. A day re-measuring the DGX Spark at three layers: chip, engine, model
- Shrinking the Prompt Made It 55% Faster, and Lost What Actually Mattered The compression that produced −55% was throwing away the core of the domain. A story about a metric I wasn't measuring collapsing.