I Deleted the Gateway and Fixed the Wiring
The gateway I set up two days earlier was attaching stock-market open days even to cooking questions. I repeated the exact pattern I was stopped from in April, and once it was gone, most of the chat quality problems were silently ignored settings, not the model
The GPU box at home serves a local model with vLLM, and the trading bot and a chat UI for friends sit on top of it. Two days earlier I had put a gateway in front of the server, and the first assessment after wiring up the chat was that the quality was unusable. The first thing to settle was whether the model was bad or whether the layers in between were.
What I set out to do
On Aug 17 I put a gateway in front of the server. On port 8002 it handles the search round trip on the server side and injects the date, market holidays, and grounding-enforcement rules into the prompt. I built it so the trading bot could call in conveniently.
The next day I attached the chat UI for friends on top of it. The first assessment was “quality is definitely unusable. And these aren’t hard questions. Speed is fast.”
How I asked
Before blaming the model, I suspected the wiring.
Doesn’t main.py have a huge number of dependencies? Isn’t that why it’s behaving strangely? Is it even right for a gateway to be mediating like this in the first place?
Digging in, domain rules were being attached to every request. Cooking questions, travel questions, all got “the Korean stock market is on a regular open day today.” The next morning I summed it up in one line. “Get rid of the gateway too. You said we don’t normally use one.” Created Aug 17, 2 pm; removed Aug 19, 1 pm. Lifespan: two days.
I’d been stopped in the exact same spot in April. In episode 10 I cut off an attempt to build a general-purpose endpoint in front of the server and wrote down “don’t put your own schema on top of a standard interface.” Four months later I built the same thing again, except this time the agent proposed it rather than me, and I approved.
What the gateway left behind was three technical lessons. In streaming, if you accumulate chunks before emitting, first token is 6.2 seconds; stream immediately and swallow only the tool-call deltas, 0.22 seconds. During the search round trip no bytes go out, which trips the Cloudflare tunnel’s 100-second idle timeout, so SSE keepalive is required. Search options don’t go in tools; the OpenAI spec is web_search_options.
Where it broke
Removing the gateway didn’t bring chat quality back. This time it was the Open WebUI side. The traps I confirmed by measurement had one thing in common: they don’t tell you you’re wrong, they’re silently ignored.
| Trap | Symptom |
|---|---|
| Container default TZ is UTC | Time off by 9 hours |
chat_template_kwargs outside the allowlist | Thinking mode setting silently dropped |
webSearch placed in settings.ui | Ignored. Must be at settings root to work |
| Settings persisted in DB | Change the env and the first-boot value still wins |
| Model parameters applied twice | Overwrites per-request overrides |
:main image tag | Frontend changes on every restart |
There was one diagnostic failure too. It concluded “this model has no thinking mode,” but vLLM was sending deltas under the reasoning key, not reasoning_content. It looked at the wrong key and said it wasn’t there.
How I fixed it
I had it classify the failed conversations one by one by cause. The conclusion was “mostly wiring, not the model.” The case that refused a factual question had the search toggle off, the wrong dates were TZ, and the missing thinking mode was the key name.
Pinned Open WebUI to v0.11.0, injected the date into the system prompt, kept one canonical copy of the system prompt in one place, and turned search on permanently. The server went back to pure model serving. One port, 8000, one OpenAI-compatible API.
Lessons
- A silently ignored setting is worse than a wrong setting. A wrong setting gives an error. An ignored setting shows up days later as “quality is bad.”
- The no-wrapper rule was confirmed a second time. I wrote it in April and still repeated it. Written down, it still repeats. Because it was written down, this time it ended in two days.
- Quality complaints: look at the wiring first. Before changing the model, check TZ, setting key names, the search toggle. This time it was all wiring.
- After putting in a setting, confirm from the response that it took. Whether the thinking mode setting went in is known from whether the reasoning text length is 0.
- When the agent builds a middle layer “for convenience,” ask “so what does this improve?” before approving. I should have used episode 10’s conclusion this time too.