I Built the Non-LLM Preprocessing First: 150 Items Down to 39

The collector for the newly built AI-news digest dumped 248 items from three days. Rule-based preprocessing that started from "I can't use the LLM without limit," rejecting embeddings, and the one regex character that was throwing away every Korean title as a duplicate

The AI-news digest is a project I had started the night before. Twenty-one sources dropped 248 items from three days into the DB at once, and handing all of that to the LLM was more tokens than I was willing to spend. So the stage that had to be built first was the one that folds duplicates and cuts the candidate list down by rule alone, before the LLM is called.

What I set out to do

At 10:27 the night before, I started a new project. “I’m thinking of building a service for reading AI-related news, is that okay? I want to do a mailing too. I’m also building it for myself to read.” I checked 47 candidate sources with curl, 41 were alive, and I picked 21. Thirty minutes later 248 items from three days were in the DB.

The first thing I said on seeing the first screen was “it looks too much like AI made it,” and the second was this.

This is a lot, though. Could we do something about importance, or about overlap?

How I asked

The next afternoon I pinned the constraint.

What will you do when it comes to editing? I can’t use the LLM without limit. I need a stage that filters duplicates, or does preprocessing anyway, without the LLM.

Instead of explaining, the agent ran a Python prototype on the actual articles in the DB and showed me. 150 items became 141 by rule, and of those 18 folded into 4 clusters. The largest cluster was 12. Twelve outlets had reported the same announcement.

The design document draft still had embeddings as an option. If the rules missed often, add a multilingual embedding model, and check whether the serving on the GPU box at home could be used. I cut it mid-turn. “Don’t use embeddings either. I’m thinking of selling that box.” The mention of embeddings and the box disappeared from the document and “no embeddings” went into the rules. What the rules miss, one LLM job merges within the top 25 clusters. If a missed cluster keeps recurring, add a word to the alias dictionary.

Where it broke

The pipeline is ten stages. Per-source cap, noise rules (prereleases, newsletter-type, duplicate releases from the same repository), exact-duplicate titles, tokenization and transliteration aliases (아스트라→astra, 오픈에이아이→openai, particle removal), union-find clustering by IDF-weighted title similarity, rule score (source count, tier weight, log of HN score, −3 for domestic PR-style patterns), and two LLM jobs. Job 1 is cluster confirmation and importance for the top 25 clusters; job 2 is Korean summaries for the top 7.

In the dry run, Korean titles dropped out one after another as “exact-duplicate title.” All nine articles from domestic outlets. The cause was the regex \W+. Without the u flag, all Hangul is non-word, so the key for every Korean title became an empty string, and empty strings were exact duplicates of each other. The Python prototype was fine because \W is Unicode by default, and it blew up when ported to TypeScript. I fixed it to [^\p{L}\p{N}]+ with the u flag and attached the source to the key so duplicates are handled only within the same source. The same title from a different source has to go to a cluster so the link survives.

Right after job 1 succeeded, it died on save. bun’s SQL template didn’t turn a JS array into a PostgreSQL array; it serialized it as a comma-separated string. I built the array literal by hand and cast it. The rerun reused the LLM result thanks to the idempotency key, at 0 tokens.

How I fixed it

The first real run. 149 candidates, 8 excluded by rule, 141 into 126 clusters, and 39 items from the top 25 clusters handed to the LLM. Job 1 took 27 seconds, 5,546 input tokens, 1,935 output tokens, 23 stories. Job 2 succeeded on all of the top 7. By design it’s around 30,000 tokens a day.

One validation came out. The Verge’s “ChatGPT, Grok, and Claude went down at the same time” and Hacker News’s “Why OpenAI, Claude, and Grok went down at the same time” were not clustered by the rules. The titles shared few tokens. Job 1 merged the two into one story and put it at 3rd. The design where the LLM merges within the 25 clusters, without embeddings, held up on the first run.

Looking over the output by eye I caught one more. At the end of the 3rd-place summary the tag </summary_ko></invoke> was attached as is. The model had leaked tags inside the JSON string. I added tag-stripping postprocessing. The next morning, 3 of 7 summaries failed after 5 structured-output retries and then survived on the queue’s second attempt. Because of additionalProperties: false in the schema, the model was failing validation whenever it attached even one extra key. I removed it from the schema and wrote “only three fields” in the prompt.

Lessons

• Put a rule layer in front of the LLM. The LLM sees it only after 149 items become 39. At 30,000 tokens a day it runs daily within the subscription.
• Let the LLM pick up what the rules missed. Instead of embeddings, job 1 merges within the 25 clusters. It actually merged on the first run.
• Regex \W differs by language. Python is Unicode by default; JS needs the u flag. Every Korean title was deduplicated as an empty key.
• Check model output down to the inside of strings. The JSON passed, but there were tags inside a value.
• Don’t put additionalProperties: false in a structured-output schema. It turns one extra key into 5 failed retries. Constrain with the prompt.

#llm#cost#side-project