I Turned Thinking Off

85.8% of the summary job's output tokens were thinking. One environment variable turned it off: output −85%, 4.5x faster, schema 10/10, and the summaries actually had more numbers. And why the one line that turns it off could have killed auth for three projects at once

The shared queue lines up the LLM calls from three projects so they share one subscription quota. After standing it up I ran the first real measurement, 50 disclosures, and output tokens came out at more than four times the prediction. Effort was already set low, so the first job was working out where the excess was coming from.

What I set out to do

It was the first real measurement after standing up the shared queue. I put 50 real disclosures from the disclosure-summary service into the queue and ran them through the subscription CLI. Model Haiku 4.5, effort low. All 50 succeeded, but the numbers were off. Average output was 3,705 tokens. The agent’s prediction had been 800. 4.6x.

I had built the queue to record thinking tokens separately per item. The CLI’s JSON output has that field. Aggregated: thinking averaged 3,178, actual output 527. 85.8% of the output was thinking. That was with effort already set to low.

This measurement was on August 28, and I actually touched it three days later.

How I asked

I didn’t ask for it. On the 28th I pasted a Claude Code usage report into the session, and the agent corrected its own prediction, saying it had “drawn a conclusion without evidence,” and proposed an A/B with thinking turned off. The reasoning: it’s an extraction task, so quality risk is low, and it would save 25% weekly. On the 31st, in the session where I told it to pick up where it left off, that experiment ran.

First it had to find how to turn it off. --help only has --effort; there’s no thinking-specific flag. It dug through the CLI binary with strings and found four candidate environment variables. Comparing on a single item: MAX_THINKING_TOKENS=0 was 577 tokens, CLAUDE_CODE_DISABLE_THINKING=1 was 1,015, and current was 6,697. All three passed the schema. It picked the shorter one.

Where it broke

The first A/B failed. The scratchpad got cleaned up between sessions and the results vanished. It redid it. Sorted the 50 items by body length, picked 10 ranging from 1,220 to 24,016 characters, and re-ran them locally with the same system prompt, same schema, same model and effort.

Processing time was an unexpected bonus. 4.5x. Putting the same 10 items back into production, all of them finished in 48 seconds.

For quality I put the summary sentences of the first 5 pairs side by side. On a cement company’s item, current said “corrected to a scheduled date of August 7 upon completion of balance payment,” and the off side said “balance payment completed on August 7, 2026, following the contract in April 2024.” On another item, only the off side listed both the share count and the ownership stake. Less thinking, and the summary was more specific. To be honest, this isn’t a quantitative metric; it’s an eyeball comparison of 5 pairs, and the one doing the judging was the agent.

How I fixed it

Applying it is one line in the runner. It unconditionally injects MAX_THINKING_TOKENS=0 as a subprocess environment variable. But that one line was the trap.

Go’s exec.Cmd inherits the parent environment when Env is nil, but the moment you put a value into Env, only that slice goes to the child. Put in just the one environment variable, and the OAuth token that came in from the deployment platform never reaches the child claude process. This queue is shared by three projects, and that token is the only means of auth. In that case the AI of all three projects stops at once, silently.

This time I didn’t get hit. Three days earlier, at design time, I had written “if the token expires, every project stops silently” as a risk item, and that awareness carried through to the code. I wrote it as append(os.Environ(), ...) from the start and left “overwrite it wholesale and it dies” as a warning in the comment and the docs. The 10 production items right after deploy had 0 failures.

The weekly projection went from 35.8M to 27.3M tokens, and daily processing time from 77 minutes to 18. I tightened the quota gate to match. Later I opened it up so each job can choose its model and effort, but thinking stays at 0. It was a decision backed by an 85% saving, so I didn’t revert it.

Lessons

• Record output tokens split into thinking and actual output. Until I split them, I couldn’t know why “output is 4.6x the prediction.”
• effort low does not turn thinking off. For jobs where thinking has no room to intervene, like extraction and summarization, give it an explicit 0.
• No flag? Dig through the binary. Four environment variables not in the docs came out of a single strings.
• The moment you put a value into cmd.Env, inheritance is cut. Always append to os.Environ(). On shared infrastructure, that one line is an outage for every project.
• Write the risk in the design doc and your hands remember it when you write the code. One line written three days earlier prevented the incident.

#llm#cost#go#measurement