Five Days of Fine-Tuning Lost to a 108-Character Prompt
I spent five days on LoRA for Gemma 4 26B-A4B and it barely differed from no tuning at all. What happened when I attacked the same task with prompt compression instead.
To push disclosure-summary quality higher, I put LoRA on gemma-4-26B-A4B-it. The reference labels were generated by calling the Claude Sonnet API, so they cost money, and this is the run that killed the whole machine. After five days I looked at what it was worth.
What I set out to do
I wanted better disclosure summaries. So I put LoRA on google/gemma-4-26B-A4B-it. This is the job that killed the whole machine yesterday.
Today the verdict came in. I’m not using it.
How I asked
First I needed labels. Each disclosure needed a reference summary attached, and I generated those by calling the Claude Sonnet API. That costs money, so the whole time I was nagged by the thought that if training didn’t work, that spend would simply evaporate.
Final count: 13,206 labels. The training set was 3,000 of them, averaging 3,476 characters, max 13,519. 6% were over ten thousand characters, and that 6% is the set of samples that killed the machine yesterday.
This is where “I should have looked at the length distribution first” comes from. Short samples were 37%, medium 40%, and I had set max_seq_length to fit the small minority of long ones, so everything was fine most of the time and memory only spiked when a long one came through. The tail decides the config, not the average.
Where it broke
Just getting training started was work. Bringing up a container from Unsloth’s DGX Spark image blocked me twice.
One: uid mismatch. The image’s default user is uid 1001, but the mounted working directory is owned by the host (uid 1000). Training ran all the way through and then died with a permission error at the save step. The most painful place to fail. Fixed with --user "$(id -u):$(id -g)".
Two: cache path. The image bakes in HF_HOME=/workspace/.cache/huggingface, so if you mount the host’s ~/.cache/huggingface at the conventional /root/.cache/huggingface, it can’t find the cache and re-downloads 48GB of weights.
None of this is written down anywhere. You only learn it by stepping on it.
And then the actual results fell short of what I’d hoped. I asked about the knobs one at a time. What if we ran one more epoch, was there a way to raise rank without OOM, how much would quality drop if we shortened the sequence.
The answer was mostly “that knob won’t change much.” Each time it came with reasoning, and the reasoning was mostly plausible. The problem was I had no way to verify it. Actually running each combination would be another several days. When the cost of checking an answer exceeds the cost of getting it, the answer’s usefulness gets murky.
“It was obvious, honestly. I wanted to push quality higher, but right now it’s not much different from just using the model untuned.”
How I fixed it
I dropped fine-tuning and tried shrinking the prompt. The system prompt I’d been using was 277 characters; I compressed it to 108. Re-measured at n=200, with Claude Opus labels as ground truth.
| Metric | Value |
|---|---|
| Processing time | 12.84s → 5.73s (−55%) |
| Opus label agreement | 73.0% (146/200, 95% CI ±6.1%p) |
| JSON parse success | 100% |
Fewer words made it more than twice as fast and quality didn’t drop. I put this version into production today.
A few rounds of prompt editing did what five days of training couldn’t.
Why didn’t it work
I’m not certain, but here’s my current thinking. Summarization and sentiment classification are things the model already knows how to do. Fine-tuning pays off big when you teach the model something it doesn’t know (internal jargon, special formats, domain-specific judgment criteria). “Summarize this disclosure” is a task it saw countless times during pretraining, so showing it 3,000 more doesn’t leave much to learn.
Which means I should have measured what this model already knew in the first place. I skipped the measuring and went straight to teaching.
Lessons
- Squeeze everything out of the prompt before fine-tuning. Get the order backwards and five days disappear.
- Before even that, measure what the model knows. Testing the “it doesn’t know” assumption probably wouldn’t have taken half a day.
- Fine-tuning is worth it when you teach the model something it doesn’t know. Drilling it harder on what it already does has little effect.
- Training containers break on uid and cache paths. And that failure usually blows up at save time, after the whole run finishes.
What hurts most is that I learned “not much different” five days later. Measuring 200 samples took under an hour. All I had to do was swap the order.