The Replication Job Was Recording "No Upstream" as Success
I asked for a cleanup of two months of data and found the price history had stopped on August 24. An upstream password rejection started it, and then one env var missing from the cutover turned a loud failure into a quiet success. Also, the safety guards passed 518 times in two months
The trading bot is the system I have been building since episode 14. It runs the rule approach and the Claude approach side by side to compare them, and it still buys in a broker paper account rather than with real money. Two months in I asked to organize the results, and while organizing them I saw that the layer that judges the trades had been stopped for eighteen days.
What I set out to do
The trading bot had been running for two months. There is no judging the returns yet. The criterion is thirty periods of twenty trading days, and 43 trading days is two thirtieths. Still, I wanted to see what had accumulated.
I think we need to organize what we have done with the auto-trading so far.. what do you think, we have some data now right
How I asked
I steered it toward an operations review rather than a report card. Not how much it made, but whether this system is in a state where it can be judged at all.
I also pinned down where to read from. Not the dashboard, not a log summary, but the database itself. Job runs, system events, guardrail evaluations, cash flows, page views. I had it check each of the six live-trading gates against those tables one at a time.
Where I got stuck
The returns were what I expected. All three books negative, only the index positive.
| Book | Return |
|---|---|
| Claude-style shadow book | -4.56% |
| Rule-only shadow book | -7.59% |
| Actual paper account | -2.72% |
| Index | +2.58% |
That is just numbers. Two months cannot tell you luck from skill.
The problem showed up in the next section. The signal scorecard was empty. Scores accumulate, 3,855 rows of them, but the table recording whether those scores were right has zero rows. Following why led to price history, and price history had stopped on August 24. The review was running on September 11. Eighteen days.
There were two layers to it.
The first layer is August 25. The upstream database that supplies prices started rejecting the password that day. This was not quiet. The replication job failed twenty-one times and was recorded as failed. Of 27 job failures across two months, 21 are this.
The second layer is September 5. That was the cutover from Railway to the Mac mini, done for cost, moved with a seven-step script covering stop, dump, restore, a full comparison of 37 tables, launchd registration, DNS, deletion, and cleanup. And the upstream address was missing from the list of env vars exported to the desk process.
So the replication code did this.
try:
up = krx_replicate.Upstream()
except RuntimeError as e:
_log_agent(app, "MARKET", "POSTMARKET", "SKIPPED", f"replication skipped — {e}")
return "no upstream"With no address, constructing the upstream object raises. That exception is caught, logged as skipped, and the job exits normally. Not a failure.
So what had been failing loudly since August 25 started succeeding quietly on September 5. The records say all 104 jobs since the cutover succeeded, and this was mixed in among them. I read the alarm stopping as the problem being fixed.
The alarm had not fully stopped, either. The evening market job was ending every day with “the replica is stale,” and a freshness warning arrived every morning at ten. I had been receiving it for over two weeks. The wording is about data being a bit old, so I did not read it as urgent. Same event, but the loud channel went dark and only the quiet one was left, and the whole thing dropped in priority.
How I worked it out
I added the upstream address to the export list. One line. Then I added the same key to the test that verifies that list. Drop it again and the test catches it.
I did not fix the upstream itself. That went down as a decision to be made. If it is alive, new credentials are all it needs. If it has been shut down, the system has to pull from the exchange directly. The backtest code for that already exists so the work is not large, but the replication design has to be revised. I wrote both options into the doc with a recommendation and raised it as a decision request. Fix what can be fixed on the spot, leave what needs deciding to be decided.
The review caught more. These mattered more, honestly.
The safety guards were evaluated 518 times in two months and passed 518 times. They have never once stopped anything. One of the live-trading gates requires that a guard actually fire at least once, and zero firings does not mean well built, it means never verified. I wrote it in exactly that way. A guard that has never fired is the same as no guard. The next steps include a rehearsal that deliberately trips the halt and the loss limit in the paper account.
Of 24 escalations, 23 have no decision. The system asked for a judgment 24 times and a decision was recorded once. The rest came up and drifted away. I wrote that if we are not going to decide, we should at least record that we did not.
The page view table had zero rows. One gate requires viewing the evening screen on 15 of 20 days, and the table that records views is empty. On top of that, that screen’s domain has had no DNS since the cutover, so it does not open. Without records this gate is unmeasurable forever. That is also what I belatedly asked about that night.
where can I see the situation room, is there a web page?
Without the review I would not have seen one of these four. Trading has not stopped in two months and has not failed since August 1. The surface was fine. The layer that evaluates had simply stopped.
What I learned
- Catch an exception and log it as “skipped” and a failure becomes a success. Unable to work because the config is missing is a different state from nothing to do. They must not share a line.
- The alarm stopping may not mean it got fixed. When something that was failing loudly goes quiet, check whether that is good news.
- Bind the env var export list with a test. A list a human copies by hand drops a line every time you move.
- Put a data-flow check in the migration script. All 37 tables were compared, and the flow arriving from upstream was not among the things compared.
- A guard that has never fired has never been verified. A 100% pass rate is not a boast, it is an unknown.
- A fine surface and a working evaluation are different things. Trading went two months without incident while the evaluation layer sat dead for eighteen days.
- Periodic review catches what alerting cannot. None of the four arrived as an alert, or if it did, it arrived in wording that did not sound urgent.