The situation
Friday, 16:02. !301 bulk-export v2 is merged and the production deploy is green. Green means it deployed. It doesn't mean it works for the customers with the biggest exports, and they're the ones who'll find out over the weekend.
The usual options are to stare at three dashboards for twenty minutes, or to go home and hope.
What Triagic looked at
We asked the Console: how is the 16:02 bulk-export deploy doing? The agent wrote a short plan, then ran it read-only:
- Prometheus: error rate and memory for the export workers, before and after 16:02.
- Sentry: new issues first seen since the deploy.
- PostHog: sessions of people using bulk export right now, and where they drop off.
Each call shows up as it runs, with its query.
What it found
Healthy for 97% of exports, but exports over 500k rows OOM the worker. The before and after table had the detail: export 5xx went from 0.1% to 2.4%, and worker-exports was OOMKilled six times. All three affected orgs are on the enterprise plan.
This is where forty minutes usually go: copying graphs and a stack trace into a blank issue form that nobody can reproduce from. We clicked Create issue on the answer instead. The title and description were already drafted from the evidence the agent had just gathered, with Reported, Root cause and Evidence sections, the numbers, the stack trace and the exact orgs. One click filed it in the tracker.
What changed
Monday's fix started from a diagnosis: #427 bulk-export: worker OOM on >500k-row exports, assigned, with everything needed to reproduce it. The three enterprise customers got a heads-up before they wrote in.
Shipping on a Friday is still a choice. Knowing what happened next takes one question. Connect Prometheus, Sentry and your issue tracker to run the same check on your deploys.