Settings That Are Bad Alone and Good Together
Shrinking chunked prefill alone made tail latency worse; combined with priority scheduling, worst-case TTFT dropped from 51.8s to 2.0s. A night of rejecting four community recommendations by measurement
The vLLM server on the DGX Spark takes long disclosure documents and short chat requests through the same engine. Throughput under concurrency was fine, but a short request arriving while a long document was being processed waited more than 50 seconds for its first token. I went through the settings the community recommends one at a time, trying to bring that tail latency down.
What I set out to do
On the night of Aug 16 I set a baseline with a concurrency benchmark. At 16 concurrent requests, 283t/s, scaling nicely to 12x single. The problem was mixed load. When a long disclosure document was being processed and I threw in a short chat request, the first token came out 51.8 seconds later. Throughput was good, tail latency was bad.
I pushed the tuning past midnight. Two instructions: verify the settings the community recommends by measurement, and leave a guide covering what the client has to comply with too.
How I asked
I turned on each recommendation one at a time and ran the same benchmark. Every result differed from expectations.
| Setting | Source | Measured | Verdict |
|---|---|---|---|
--max-num-seqs 4 | Official blog recommendation | 12x scaling at 16 concurrent, no cliff | Rejected |
--max-num-batched-tokens 8192 alone | Community | Worst TTFT worsened 51.8s → 65.4s | Rejected alone |
--kv-cache-dtype fp8 | Community | Same-sentence repetition loop after 500 tokens | Rejected |
| ngram speculative decoding | Community | Summaries +12%, free-form prose −36% | Rejected |
max-num-seqs 4 was a recommendation for single-user low-latency environments. We’re a batch server. The premise was different, and that premise wasn’t written in the recommendation.
Where it broke
I thought shrinking the prefill chunk to 8192 would split long documents finely enough to open gaps for short requests to slip in. In practice it got worse. The scheduler is first-come-first-served, so even with gaps the short request still stood at the back, and the chunk-splitting overhead just got added on top.
Turning on priority scheduling was the same. If the client didn’t send a priority field, worst TTFT was 69.6 seconds, worse than baseline. Claude Code nailed this point. “Server settings alone have zero effect.”
How I fixed it
Turn both on and have the client send priorities, and worst TTFT came down to 2.0 seconds. Chunking creates the gap; priority decides who goes into it. With only one, either there’s a gap but no reordering, or there’s an order but no gap. Throughput loss was 0. 294.8t/s, same as baseline.
In the morning I finished looking at speculative decoding. ngram was rejected, and the model’s built-in MTP head gave 2.2x on the real workload. Summaries went from 17.9t/s to 39.8t/s.
One more thing caught here. The benchmark script was counting speculative decoding’s chunk count as the token count. An actual 39.5t/s was recorded as 14.3t/s. I nearly concluded MTP wasn’t worth much; counting the raw responses caught it. The answer to my question “so is MTP worth using?” hinged on one bug in the measurement tool.
Lessons
- Recommendations come with premises, and they’re usually unwritten. Apply a single-user recommendation to a batch server and you get the opposite result. Re-measure with my own load.
- Measure settings in combination. Chunking alone worsened, priority alone worsened, together they were a 25x improvement. Toggling one at a time misses combination effects.
- Some server settings are client contracts. priority only works if the client sends the field. Write “what must be sent” into the guide alongside.
- Speculative decoding differs by kind. ngram lost on free-form prose; the model’s built-in MTP gained across the board.
- When results look strange, suspect the measurement tool first. 14.3t/s was a number produced by the benchmark script, not the model.