Measured Separately Is Not Measured Together, and 3% Utilization
Measured thinking mode on an offline set, concluded no effect, then it reversed on a multi-step set. Meanwhile a never-measured combination went into production. The max_tokens cap was a breaker, not a budget, and after four months of tuning, the hardware's utilization was 3%
I serve a local model myself on the GPU box at home, and this was the day for locking in its final configuration. The one variable left was thinking mode: always on, with reasoning text long enough to be a large share of the latency, and I had never measured whether it paid for itself. I measured it on an offline set, concluded no effect, and shipped it to production, which left one question open, whether that combination had actually been measured.
What I set out to do
With the experiments done, it was time to lock in the final configuration. One question remained: thinking mode. It’s always on and the reasoning text has a median of 2,490 characters. It’s a large share of the latency, and I’d never measured whether the quality was worth it.
How I asked
I compared thinking mode ON and OFF on the offline scenario set. 35/40 vs 35/40, a tie. I concluded “no effect,” turned thinking mode off, layered on the new system prompt that had won in the experiments, and put it in production.
Did you test the unverified combination? And so what are you going to do about thinking mode?
Where it broke
Claude Code’s answer was a confession. The system prompt experiment had run with thinking mode ON, and the thinking mode experiment had run without the new prompt. New prompt with thinking mode OFF had never been measured once. That unverified combination had just gone into production. Measured, flagging of common-sense departures had dropped from 4/5 to 0/5. I reverted immediately.
The thinking mode conclusion itself was also wrong. The offline set was problems where thinking mode had no room to intervene. I’d measured with a set that couldn’t measure it and called it no effect. I built a new multi-step reasoning set and measured again. A problem where the cash flow sign is minus 40 billion, a problem requiring controlling-shareholder net income multiplied by ownership stake, the PER trap in cyclical industries. OFF 56/60 vs ON 60/60. All three misses were trap problems. Confirmed: keep thinking mode. Latency rises from 12 seconds to 28, but it was worth it.
The next day I made the same mistake once more with max_tokens. While building four purpose-specific presets, I tightened the cap to twice the measured generation length. In the process I discovered the general preset had been getting cut off at 4,000. Headroom was 0%. A truncated answer doesn’t look wrong, it looks short.
Isn’t that still too low? Why restrict it like that?
How I fixed it
I measured whether the cap was really a cost. Throwing the same request at caps of 200 and 32,000, median latency was 1.04 seconds and 1.00 seconds, the same. vLLM v1 allocates KV cache as it generates, so the cap is not a reservation. Generation length is determined by the task, not the input; increasing the input 14x left generation flat. The cap was a breaker, not a budget. With no reason to tighten it, I loosened it to four times measured. The real danger was the analysis preset, which had no cap at all. If it runs away, one request holds the GPU for up to 260k tokens.
That was the configuration. Finally, I looked at how much work this hardware actually does.
| Value | |
|---|---|
| GPU average utilization | 30.4% |
| Average power | 21W (idle 12.5W) |
| Effective generation | 9.3t/s |
| Hardware capability | 339t/s |
| Utilization | about 3% |
The answer had already been there in April. About the batch client I had written, “traffic was matched to this, so I just set it to 3 for now.” The disclosure-summary service sends requests sequentially, so average latency is 53.1 seconds. Send at 16 concurrent and the same work finishes far faster. Four months of tuning performance, and the bottleneck wasn’t performance, it was the workload.
Lessons
- Measuring A and B separately is not measuring A+B. The combination that goes into production gets measured as that exact combination.
- Record “what wasn’t measured” alongside measurement results. Twice I believed I’d measured and was wrong. The measurement isn’t the defense; the list of what wasn’t measured is.
- “No effect” means suspect the set. Measure with problems that leave no room to intervene and anything comes out as no effect.
- The max_tokens cap is a breaker. It doesn’t affect latency, and a truncated answer doesn’t look like a wrong one. Leave it generous at four times measured, and add one only where there is none.
- Check utilization before tuning. Rather than raising the t/s of a 3%-utilized machine, switching the client to concurrent sending should have come first.