Three Throttles That Keep a 640K-Row Backfill Behind Live Traffic
Filling the 2020–2025 disclosure archive means running for months inside the external API's daily quota and the LLM's throughput. I split collection from analysis, put the archive behind new disclosures, and built a throttle that stops every worker on a rate-limit error. The completion estimate changed four times in five days
The disclosure-summary service only summarizes disclosures as they come in, so everything back to 2020 sat empty. 2025 alone is 120K items and back to 2020 it is 640K, and getting them in means months of steady running under two ceilings: how many calls the disclosure API allows in a day, and how much the LLM can chew through. The condition was that today’s incoming disclosures must not get pushed behind all of that.
What I set out to do
I want to backfill all the 2025 disclosures, get it ready. And what happened to the stock prices? I can’t see them.
This is about filling in the disclosure-summary service’s past. The stock prices weren’t a bug. By design, the closing price for the day after a disclosure came in the afternoon of the day after that. I filled prices separately the same day, back to 2020. 46 minutes, 3.93 million rows.
Disclosures were a different scale. Measuring the list of listed-company disclosures gave a total of 648,594 for 2020–2025. 2025 alone was 120,381. There were two ceilings: the disclosure API (OpenDART)‘s daily quota and LLM throughput. The agent’s diagnosis was “a months-long standing job; slow and continuous is the only way.” And the real constraint was disk. A 5GB volume shared by four projects was 67% full. I pushed 2020–2024 back until after a volume expansion and went ahead with 2025 only.
How I asked
“Can I keep the LLM running slowly, continuously?” Three throttles came out of that one line.
Split collection from analysis. Putting 640K rows into the queue at once blows through the API rate limit on the spot. Load only the list first, and a separate command puts one day’s worth into the analysis queue.
Put the archive behind live traffic. The queue picks in created_at ascending order, and the backfill rows come before new disclosures. Left as is, the 640K rows get processed first and today’s disclosures slip by months. I sent anything with a filing date more than 30 days old to the back as the first sort key.
Stop every worker on a rate-limit error. A quota overrun was flowing through as an ordinary failure, using up the 3 retries and becoming a permanent failure. The rate limit is per account, so re-queuing row by row is just a busy loop. I split it out as a dedicated error that puts the row back with its retry count preserved and halts all workers for 30 minutes.
Where it broke
Six review rounds attached 33 items. Three throttles became five. I changed the daily pacing from row count to API call count. A periodic report is 4 financial-statement API calls plus 1 raw-text fallback, so 5 calls max. An estimate at queue time can’t count retries, so I counted actual calls per day and kept them in the DB; over budget, it just stops picking the archive and live traffic keeps going.
One of the bot’s rebuttals was exactly right. A rate-limit error gets covered by a successful viewer fallback (a different host), so the halt never triggers. The older the disclosure, the more likely the viewer loads, a combination that blows up precisely in the backfill. And five minutes after the merge, I learned for the first time that production runs 2 replicas. A scenario I had handled only in theory through the whole review.
The completion estimate kept being wrong. The doc said 17 days. When the first daily batch of 7,730 was fully digested in one day, 14 days. The next day I uploaded a screenshot of the API dashboard, and the daily quota was 40,000, twice what we had assumed. Our own counter, 11,196, and the dashboard’s 11,704 agreed within 5%. The bottleneck wasn’t the LLM but the budget. Raising the budget to 25,000 made it 4–5 days.
How I fixed it
Two days later, right after a daily batch started, failures jumped from 37 to 736. In 10 minutes: 0 completed, 466 failed. I stopped the drip (the daily drip-feed job) immediately. The LLM server was dead, and the last normal completion was the afternoon two days earlier. Periodic reports spent their 5 financial-statement API calls first and then died at LLM submission, so a day’s worth of API calls, 10,000 of them, went to waste. The scheduler had no LLM health gate.
Check whether the DGX is alive. The home internet went out.
The cause was the home internet. Collection is on the cloud side, so nothing was missed, and I re-queued the 1,812 failures and recovered. Effective rate 18,000 a day, completion estimate September 5. The fourth number in five days.
Lessons
- A backfill splits collection from analysis. Loading the list takes 37 minutes; analysis takes days. Bundle them and you spend the whole quota on day one.
- The archive goes behind live traffic by sort key. If you push it back only within per-status priority, failed retries get promoted and stand in front. It has to be the first sort key.
- A per-account rate limit can’t be blocked per row. Stopping every worker is the answer. Propagate the error first so a fallback success doesn’t cover a rate-limit error.
- Pacing counts actual calls, not rows. Comparing estimates against measurements daily, they agreed within 5%, and that comparison was what told me the quota assumption was wrong.
- Don’t run the drip without a downstream health gate. In the one day the LLM was dead, 10,000 calls of API budget vanished.