Failing to Reproduce It Locally Did Not Mean There Was No Bug

After the move to Railway, the web frontend climbed from 226MB to 833MB in a straight line over 26 hours. It was not render load, and the sibling service on the same stack was fine. I hammered it with 3,000 requests on my Mac, could not reproduce it, and wrote down "not reproducible by request volume." Then I measured again inside a container and got 388KB per request. The platform had been hiding the signal

The bill tracker’s web frontend is a Next.js app that had just moved onto Railway. On the dashboard it was using more memory than the read API server behind it, and pulling 26 hours of it gave a straight line rather than a sawtooth. Lowering concurrency or shrinking a cache was not going to touch that.

What I set out to do

I was looking at the Railway dashboard at one in the morning when the bill-tracker’s web frontend caught my eye. It was using more memory than the read API server behind it.

Can we get uijeongbogi web to use less memory?? check railway

An hour later I asked one more thing.

why is it higher than the api server??

I thought this was a tuning question. Lower the concurrency, shrink a cache, done. That is why I asked “can we make it less.” That was the wrong question.

How I asked

I had the agent pull a time series instead of the dashboard’s current value. A single number only tells you high or low. Twenty-six hours of it gave me a shape.

Not a sawtooth. A straight line. About 24MB an hour, and it never came back down once in 26 hours. That is not using a lot, that is not giving it back. The tuning question became a leak question.

Where I got stuck

I set up three hypotheses and knocked them down one at a time.

Is it render load? No. The steepest stretch was 09:00 to 11:00 on the 8th, and traffic in that window was 1.0MB per hour, the lowest of the entire span. Meanwhile at 08:00 on the 7th, when traffic was ten times higher, memory actually went down. When load and growth move in opposite directions, load is not the culprit.

Is the CPU busy with something? No. It sat at 0.001 to 0.004 vCPU for the full 26 hours. It was doing nothing and eating memory anyway.

Is this just Next? To check that I pulled in the sibling service. The disclosure-summary frontend runs the same Next 16.2.6, the same standalone build, the same ISR, on the same Railway host. Put 34 hours of it next to this one and it was a sawtooth between 155MB and 725MB. No trend. When the same stack only climbs on one side, the culprit is not the stack, it is the difference between the two.

While we were going through the differences I brought one up myself.

but dartbrief-web is piling up logs in sentry too right now… so why is only uijeongbogi climbing..

Sentry looked enabled on both. It was not. The disclosure side had ripped out server-side Sentry six days earlier, after hitting Error 1102 on Cloudflare Workers, the error that says the worker exceeded its resource limits. The client side was still reporting, so the dashboard kept filling up, which is why both looked enabled to me.

That flipped something. The same configuration produces different symptoms depending on the platform. Workers recycle isolates, so per-request residue shows up as cold-start CPU. Railway runs one long-lived Node process. The same residue accumulates as memory here. If what we diagnosed as a CPU problem six days ago looks like a memory problem now, it could be the same thing wearing a different face.

How I worked it out

To confirm it I tried to reproduce it locally. I ran a production build on my Mac and pushed 3,000 identical requests through it.

Nothing reproduced. heapUsed went from 87MB to 88MB between request 1,500 and 3,000. One megabyte. rss bounced wildly between 47MB and 233MB. There was nothing you could call a signal. So I wrote down “not reproducible by request volume.”

That sentence was wrong. More precisely, its subject was wrong. It was not that nothing reproduced, it was that something different was doing the measuring. macOS compresses memory and reclaims it on its own. The bouncing rss was not noise, it was the result of that management. Production is a Linux container. There the kernel counts what you used, per cgroup, and hands you the number.

So I moved where I was measuring. I brought up a node:22-alpine container and pushed real bill-detail URLs through it.

Roughly 388KB per request. The same order of magnitude as the slope I was seeing in production. Same code, same requests, same count: 1MB on the Mac, 1.1GB in the container. The difference was not the code, it was where I stood to measure.

I left the probe in as frontend/memprobe.cjs. Every five minutes it prints one line combining process.memoryUsage() and /sys/fs/cgroup. I hooked it into the Dockerfile’s run line with --require so production measures the same thing. I marked it as temporary.

I nearly slipped once more here. The file bucket in the container numbers climbed to 625MB, so my first thought was .next/cache. I measured the disk and it was 62MB. Ten times off. The same label does not mean the same thing. So I revised the probe again to split shmem out, and I wrote into the docs that cgroup buckets alone are not enough to name a cause.

And as I write this, there is still no verdict. A branch with the server-side Sentry init commented out is running, and I am watching whether the curve flattens. Early on the two curves overlapped. Later the Sentry-off side did drop. But it gave memory back without a restart, which means it may be page cache rather than the JS heap. It is not time to name anyone. That branch is not merged.

What I learned

• Failing to reproduce locally is not evidence. If you are measuring somewhere else, it is not absent, it is invisible. One megabyte on the Mac was 1.1GB in the container.
• Look at a time series, not a current value. The number 833MB tells you nothing. The shape of a 26-hour straight line tells you it is a leak.
• If load and growth move in opposite directions, load is not the culprit. The slope was steepest when traffic was lowest.
• The sibling service on the same stack is your best control group. Erase what they share until one difference is left.
• The same configuration produces different symptoms on different platforms. What was a CPU problem on serverless is a memory problem in a long-lived process.
• Do not name a thing from the label on its bucket. 625MB of file was not 62MB of disk cache.
• If there is no verdict, write that there is no verdict. Name a culprit off one curve dropping and you will chase the same thing again next month.

#next.js#sentry#railway#memory#measurement