I Turned On Auto-Publishing and Four Things Surfaced in Three Days
None of them were visible while a human sent each issue by hand. A rerun pinned an old response to a new list, a slug got rebuilt and broke the links, a shell that only filled the schema reached the page, and one source took nine of twenty-five slots
I turned on auto-publishing for the news pipeline. With the step where a human took one last look removed, four things surfaced over three days. All four were pre-existing defects. They had simply been invisible because a person was filtering them out by eye.
What I set out to do
The news pipeline gathers articles, clusters them, scores importance, writes the copy, and puts it on the page. Everything up to the last step was automatic and I pressed publish myself.
I turned that on. The gate is not a score but “is the fact present in the transcript.” It goes out automatically, and only what gets caught comes to me as an alert.
Where I got stuck
One. A rerun pinned an old response to a new list.
The clustering job’s idempotency key was only date#attempt. The summarization job’s key carries a prompt version; this one did not.
Here is why that matters. The clustering job’s response points at articles by number, and those numbers are the order of the candidate list. So changing the rules that select candidates changes what every number refers to. Rerun a date you already ran in that state, and the queue says “same key” and hands back the old response. Old numbers get applied to the new list.
Importance, classification, and grouping all attach to the wrong articles. And nothing errors. The numbers are within a valid range.
Reruns really happen. September 12 went to a third attempt.
I put a clustering version in the key, and decided to bump it not only when the prompt changes but when the selection rules change.
Two. The slug was built twice.
The slug that becomes the URL is fixed when the editor queues the piece. But the side that writes the copy was not using it. It was rebuilding the slug from the date it picked the job up.
Within a single day the two match. Cross midnight, or fail and retry, and they diverge. Then the read and edit links in the alert 404, and the video step prints the wrong public address.
Three. A shell that only filled the schema reached the page.
The September 14 feature went out like this. Title: “test title.” Lede: “test lede.” Outline: two items. A hundred and thirty-eight bytes total.
It was the response that comes back on the final attempt after structured output has broken several times. The format is valid. So the parser passes it and the quality gate passes it. The gate measures form, not the presence of content. It had happened once before on September 11, at 152 bytes.
I decided that fewer than three outline items, a body under 300 characters, or visible placeholder wording fails the piece before the gate is ever measured. It gets checked right after the copy arrives. That has to be the spot, or quotes, the English edition, and photo tokens get thrown away with it.
Four. One source took nine of twenty-five slots.
The September 13 morning issue did not carry the story about Anthropic’s CEO talking about slowing down. Not because its importance was low. It never made it into the job that scores importance. It ranked 28th on rule score and the cut was at 25.
Nine of the twenty-five slots were a single personal blog. Six of those were point-release version bumps for a tool that person maintains, and the model scored their importance at 1.
The score is 1.5 × number of sources covering it + sum of tier weights. So one curation outlet covering something alone (3.0) always beats one general outlet covering something alone (2.7), regardless of content. On top of that, the cut landed in the middle of a tie block: ten clusters at 3.0 and sixteen at 2.7. Which of those 2.7s survived was being decided by array order.
How I worked it out
The fourth took the most work. I set a per-source cap of four clusters, and capping without distinction made things worse.
The top item on September 13 got cut by the cap. The model had scored its importance at 4. On September 1, three official announcements got pushed out and preprints filled their slots. Announcement originals and index sites are a different kind of thing. Several items from the same place is normal there, and it is not bias.
So I put a category outside the cap. And clusters caught by the cap are held rather than discarded, filling in if slots remain. Ties get broken by number of covering sources, then item count, then recency.
Something unexpected moved into the slots the cap freed up: self-promotional event notices, like booking a booth at a conference. I filtered those in the general-outlet tier only.
Rerunning September 1 through 13 changes zero or one item on most days, and four on September 13. Four version bumps drop out, and the Anthropic CEO story, Altman’s IPO remark, and a data-center regulation story come in.
On the third item, the question was where to put the auto-publish threshold. My first idea was to cut on the number of failed criteria, but 15 of 19 measured pieces had at least one failure. Eleven of those were the same single criterion. Cutting on count effectively turns auto-publishing off. I went with holding anything under a score of 85 for approval instead. Run against 19 pieces, that catches the one shell, holds two, and lets seventeen go out as they do now.
What I learned
- Remove the step where a human looks last, and everything that person was filtering becomes visible. All four predate auto-publishing. It did not create them; it made them show.
- An idempotency key needs a version for everything that changes the result. Include the prompt and omit the selection rules, and a rerun on the day you change the rules lies quietly.
- A response that points by number fails without erroring when the list changes. A number inside the valid range passes validation.
- Do not rebuild a value that was already decided. Build the slug in two places and it matches within a day and diverges after midnight. Tests do not catch this.
- A format check does not guarantee the presence of content. A hundred and thirty-eight bytes passed the schema and reached the page.
- Selecting on score alone breaks diversity, but capping everything equally blocks legitimate concentration too. Announcement originals and index sites are supposed to send several items from one place.