Two AIs Blamed Each Other: What Happens When Every Project Gets Its Own Agent
Each project's agent looked only at its own logs and pleaded innocent, so I laid both logs side by side to reach a conclusion
I ran the disclosure-summary batch and the application API fell over. The model server and the summarization application are separate projects with their own Claude Code session each, and when I asked both, each read only its own logs and pointed at the other. Carrying messages between them became my job.
What I set out to do
I fired 50 concurrent requests from the disclosure-summary batch and the application API fell over. I had to find the cause.
Up to here, it’s an ordinary incident. What was unusual was the debugging structure.
How I asked
I keep a Claude Code session running in each of two projects. One looks at the model server, the other at the summarization application. When the incident hit, I asked each of them.
And each looked only at its own logs and pleaded its own innocence.
When I asked the disclosure side, it said the model server couldn’t keep up. So I took that to the server side, which said it had processed every request. Back to the disclosure side.
“Can you check the server-side logs? It says it’s your fault. Says there was no problem at all over there.”
I was playing messenger between two AIs. Each has only its own container logs, its own codebase, and its own conversation history, and knows the other system only through hearsay: “the other side says so.”
And hearsay gets rewritten. When I relay A’s words to B, B interprets them in its own context and builds a rebuttal; when I relay that rebuttal back to A, A interprets it in its own context again. After two round trips, even I was confused about what had originally been said. It’s exactly what happens between departments in a human organization, just much faster.
Where it broke
Finding the cause was stuck from the start. Grafana was full of 500s, but the application log was empty for that time window. No matter how many times I changed the time range and asked, the only answer was “there are no logs for that time.”
Logs can be lost or filtered out by level, but if a counter counted 500s, then 500s happened. In that case you have to trust the metric. But the agent kept trying to base its reasoning on the side it could verify (no logs).
And it kept crossing the line. The disclosure-side agent, trying to resolve the incident, tried to touch another project’s server config (max_model_len). Without knowing that side’s circumstances. That value was chosen when the server was stood up, weighing context length against memory. None of that context is visible from here.
Inside its own project, an agent is remarkably capable, but it doesn’t know that it doesn’t know what’s outside its boundary.
How I fixed it
The conclusion came only after I laid both logs on top of each other.
“I want to increase total throughput, not raise the timeout. There are no errors on the server. It processed everything that was requested. You’re the one who failed to receive it.”
No errors in the model server log. Every request was processed. The receiving side failed to receive.
So what dried up on the receiving side? It was the database connections. At the time, one analysis flowed as read the target from the DB, fetch the disclosure text, call the AI, write the result, and all of it sat inside one transaction. The AI call is a blocking call with a 120-second read timeout. Which means a DB connection was held for the entire duration of an external call that takes tens of seconds. The pool holds ten. So ten in flight is enough for the next request to get no connection at all. That is why the application API fell over. The model server really was not at fault.
Finding the fix took a long detour. The first proposal was to raise the timeout, but that just postpones the symptom. Next was an async callback. Then you need to handle lost callbacks, which needs somewhere to keep state. gRPC even came up. The structures kept piling up.
What actually happened that evening is right there in the commits.
| Time | What I did |
|---|---|
| 19:37 | Split collection from analysis. Saving a disclosure also inserts a pending row, and a scheduler running every minute handles the analysis |
| 20:00 | Capped the pending fetch at 50 rows per cycle |
| 20:51 | Added a separate in-progress state and ten threads of parallelism. Locked rows so multiple instances cannot grab the same one, and reset rows stuck in progress for over five minutes back to pending |
| 22:08 | Split the flow into four stages so that the external call section releases the DB connection |
| 23:40 | Added an admin API to pause and resume the scheduler, plus a count of in-flight threads |
One line in the 22:08 commit says it: ten connections can now drive twenty threads. That headroom did not come from growing the pool. It came from shortening how long each connection is held.
Nothing new got introduced from that hour of design. But the “somewhere to keep state” that the async-callback route said it would need turned out to be needed anyway. Instead of bringing in a new queue, I put pending and in-progress states on the analysis cache table that already existed and used it as the queue. It was not that no queue was needed. It was that the one already there did the job.
Lessons
- When an incident spans two systems, give one agent access to both sides’ logs. Don’t relay by hearsay. Hearsay gets rewritten.
- An agent’s “that’s the other side’s problem” is not a conclusion; it’s the edge of what it can see.
- When metrics and logs disagree, trust the metrics. Logs might not get written, but counters don’t lie.
- Don’t put a long external call inside a transaction. Ten connections came to drive twenty threads not by growing the pool but by holding each one for less time.
- When an agent tries to touch another project’s config, stop it. That value usually carries a history that isn’t visible from here.
- Connections drying up and throughput being insufficient are separate problems. The former got solved that day; I didn’t touch the latter.