I Stopped Judging Runaways Per Job
I built a per-call cost cap, raised it, turned it into an alert, and finally removed it. After fixing the same safeguard four times in four days, all that's left is the aggregate gate and a timeout. "Didn't I say just alert me?"
The shared queue lines up LLM calls from several services and runs them one at a time, and the subscription quota it spends is a fixed aggregate. One job reloaded its context several times and burned four times the tokens of a normal one. I tried to stop that runaway with a per-call cost cap.
What I set out to do
A runaway job showed up in the shared queue. A normal job is 78,291 characters of input, 44,519 tokens, 11 seconds. The runaway job was 61,112 characters of input, but 193,395 input tokens, 59,113 output, 477 seconds. Thinking was 0, so it had reloaded the context several times. My guess was a retry loop in structured output.
Length was a trigger, but not a ceiling. Of 143 jobs under 10,000 characters, 0 runaways; of 21 jobs at 40,000–80,000 characters, 8 runaways; and a 150,000-character job finished normally. A timeout can’t stop it either. When I raised it from 180 seconds to 600, jobs that used to die “succeeded” and burned more.
So on September 2 I set --max-budget-usd at 0.15 dollars as a per-call cap.
How I asked
I’m using Claude on a subscription right now. Is the cost really calculated like that?
It’s a subscription, so the marginal cost is 0. The value the flag looks at is a converted figure, the CLI multiplying the token count by API list price, and I measured that the calculation and the blocking work under subscription auth too. It’s a monotonic function of tokens, so in the end it’s a token cap written in dollars. I put it in the docs that way.
That same night I had already said which way to go. “There are only a handful in practice, so let them run, and instead send a Telegram alert when one needs a lot.” But the next morning’s commit kept the block and just raised the cap to 0.20 dollars.
Where it broke
The grounds for raising the cap were 72 measured jobs. Normal jobs were 0.007–0.191 dollars, runaway jobs 0.100–0.489 dollars. The distributions overlap. At 0.15, 2 normal jobs got cut, 10 runaways were blocked, and 5 got through. At 0.20, 0 normal, 8 runaways blocked, 7 through. I picked it as the smallest value with 0 false positives. I wrote that it wasn’t a complete block but a cap on the loss. And I attached a Telegram alert to the block.
At 7:17 in the morning I asked. “The budget is still there, so why is it over budget?” Four minutes later, again. “Didn’t I tell you to just alert me and let it process?”
0.20 dollars was turning 27 jobs a day from success into failure. All 15 jobs classified as runaways were jobs that had completed. They were just expensive. The weekly quota was at 31%. It went into the commit body as is. “The requirement was ‘let it run and just alert,’ but I attached the alert to a block.”
In the meantime, one cache story got corrected too. Put the same document in front and change only the instruction, and cache reads were 0. Only completely identical input gets read. The CLI’s cache is per message, so the prefix caching assumption I had used with local vLLM breaks here. I also pulled the line from the previous day’s docs, “cache writes are 1.25x, so sending the full text is more expensive.” 1.25 is an API list-price multiplier, and on a subscription dollars aren’t the constraint.
How I fixed it
On September 3 I changed the block into an alert. If output tokens exceed 5,000, it only sends an alert. That’s the empty range between the normal maximum of 4,433 and the runaway 41,000. I raised the cost cap to 1.0 dollars and left it only as a last line of defense. The observed maximum is 0.489 dollars, so normally it doesn’t trip.
Two days later I removed even that. “A safeguard that doesn’t trip normally picks the most expensive job and throws it away when it does trip.” The quota is protected in aggregate by the 5-hour window and the weekly gate. There was one side effect. The failure-handling path didn’t record tokens, so the consumption of blocked jobs wasn’t being caught by the gate. Removing the block made the metering more accurate.
The same day I turned the alert off too. 5,000 was a value chosen from the Haiku and effort low distribution, but once I opened it up so each job can choose its model and effort, normal jobs could exceed it too. An alert that normal jobs trip all the time is noise, and the moment it becomes noise, real runaways go unseen along with it. Leaving the detector on after it has lost its premise is worse than turning it off.
Three safeguards remain. The aggregate gate, the 10-minute timeout that is the only safeguard cutting off a single call, and a dashboard for after-the-fact observation. The fundamental response was on the producer side, not the queue. Once the disclosure-summary service clamped its input to 24,000 characters, runaways went to 0.
Lessons
- Don’t judge runaways per job. When the normal and runaway distributions overlap, any cap cuts normal jobs. Protect in aggregate and remove the cause at the producer.
- Don’t attach a block to a request that says “alert only.” 27 failures a day was the cost of mistranslating the requirement.
- A safeguard that doesn’t trip normally makes the worst choice when it trips. It picks the most expensive job, which is the longest document, and throws it away.
- When the premise behind a detection criterion changes, turn the detector off. The moment model selection opened up, the number 5,000 lost its grounds.
- The subscription CLI’s cache is per message. With a document-first, instruction-last structure, the cache doesn’t hit. Don’t carry over the assumptions of local serving as they are.