I Lost 2.5 Months of Logs in My First Hour Back
In the first DGX Spark session in two months, "check the logs" turned into a container recreate and 334MB vanished. What survived was 14 days that another system happened to be watching
The DGX Spark is the machine the local LLM runs on. Apart from the disclosure-summary service’s batch job calling it a few times a day, it had been running on its own for over two months. I opened a session after a long gap, looked over the state, and asked it to check the logs, and inside that request I thought was only a check there was an irreversible step.
What I set out to do
I hadn’t opened a session on the DGX Spark side in over two months, not since early June. The server ran on its own the whole time. The only traffic was the disclosure-summary service’s batch job calling it a few times a day.
On the morning of Aug 15 I connected for the first time in a while. My first words were “Long time no see, tell me how the server is doing.” I looked at the status, cleaned up uncommitted changes, and had it search for whether anything better than the current model had come out. Then I asked it to look at the logs.
How I asked
Can you take a look at the logs? Check how much disk we’re using and whether the logs are being rotated.
The check was accurate. Docker’s default json-file driver has no size limit. The vLLM container log had been growing unbounded for 2.5 months, about 334MB. Rotation had never been configured. Claude Code proposed two numbered actions, and two minutes later I replied “go ahead with both 1 and 2.”
Where it broke
Rotation is a logging option in docker-compose. Changing the option means recreating the container. Claude Code did exactly that the moment it got approval. And Docker’s json logs are attached to the container instance. When the container goes, the logs go with it.
334MB disappeared on the spot. I noticed 8 minutes later.
Ah, my intent was for you to look at the logging and judge whether it was worth changing. Did you delete all of it?
It went straight into Claude Code’s wrap-up. “Cost I incurred: 334MB of vLLM log history deleted.” I had said “check it,” approved a proposal, and that proposal contained an irreversible step. One approval bundled the check and the change together.
How I fixed it
There was no way to get it back. But something had survived. The Promtail I attached the day I set up the server in April had kept shipping logs to Loki. Retention was 336 hours, so 14 days from Aug 1 onward remained. It survived because a different system happened to be watching the same thing.
Rotation was set to five files of 50MB. I did some follow-up cleanup the same day. Lowered gpu-memory-utilization from 0.85 to 0.75, shrinking the KV cache from 46.68GiB to 34.55GiB and widening host RAM headroom from 9GB to 21GB. A config that had been running since June 3 but never committed finally got pushed too.
I asked once more during the day. “What actually got better with this change? I don’t think anything did.” Rotation prevents future incidents; nothing improves right now. The answer was honest, so I let it go.
Lessons
- “Should I check?” and “should I change?” are different questions. When one approval bundles both, the irreversible one runs first. Get the check result, then order the change separately.
- Recreating a container is deletion. Logs, temp files, all state without a volume, gone. If a change requires recreation, ask beforehand what disappears with it.
- Set up log rotation the day you set up the server. Set it up 2.5 months later and what to do with everything already accumulated becomes the problem.
- Keep logs in two places. Container logs and a separate collector. The monitoring I attached on day one paid off in an incident four months later.
- In a session opened after a long gap, ask “so what does this improve?” one more time. The agent is good at finding places to touch. Whether they should be touched is my call.