Example: Platform Reliability
A playbook for "everything is slow" and "the site is down" — scoped to infrastructure, and the one worth pairing with a scheduled report.
Broad, non-specific complaints: slowness, timeouts, errors across features, "is something going on?". The distinguishing feature of this category is that the ticket gives you almost nothing to filter on, so the playbook has to supply the method.
Fields
Name
Platform ReliabilityDescription
Tickets reporting broad degradation rather than a specific broken feature: the site
is slow, pages time out, errors appear across unrelated areas, or a merchant asks
whether there is an ongoing incident.Routing hints
slow, slowness, timeout, "timing out", down, outage, "is there an incident", 500,
502, 503, "error page", unresponsive, "everything is broken", degraded, latencyTriage instructions
These tickets rarely name a subsystem, so establish scope and timeline first and only
then look for a cause. Resist starting from the merchant.
1. Establish the window. Use the ticket's timestamps and any times the reporter gives.
Everything after this is filtered to that window.
2. Prometheus: error rate and latency across services for that window, compared with
the preceding period. This is the fastest way to tell a real degradation from one
user's bad wifi.
3. Kubernetes: pod restarts, CrashLoopBackOff, pending pods, and recent events in the
window. A restart storm is usually visible here before anywhere else.
4. Sentry: exception volume by service for the window, looking for a new issue or a
sharp rise in an existing one.
5. app-* OpenSearch logs for 5xx clusters, narrowed to whichever service the previous
steps implicated.
Known causes, most to least common:
- a deployment rollout that made pods unhealthy — correlate with pod age
- a dependency (database, cache, external provider) slow or unavailable, which shows
up as latency across unrelated services at once
- resource exhaustion: OOMKilled pods, pending pods with no node capacity
- nothing at all — the reporter's own connectivity, which is a legitimate finding
Conclusions:
- State plainly whether platform-wide degradation is observable in the window. "No
degradation observed between 14:00 and 15:00 UTC; error rate and p95 latency were
within normal range" is a valuable answer and should be given confidently.
- If degradation is real, name the affected services and the time the change starts.
The start time is what makes a deployment correlation possible.
- Do not name a single merchant as the cause of platform-wide symptoms.Data sources
org-prometheus, org-kubernetes, org-opensearch, org-sentryVisibility
Consider restricting to a platform or on-call team, if you have one — see Teams.
Why it is scoped this way
Deliberately no application databases. A platform-wide symptom is not diagnosed by reading one merchant's rows, and excluding them stops the agent starting from the reporter — which is the wrong end of this particular investigation.
Pair it with a scheduled report
This is the playbook that most benefits from running on a cadence. A daily 09:00 report scoped to it gives the team a health summary before the tickets arrive, and because a report body is the first message in a Console thread, anyone who can see the playbook can ask follow-up questions in it.
Schedules are configured on the desktop — see Playbooks in the app.
Suggested brief:
Summarize the last 24 hours of platform health: error rate and p95 latency by service
against the previous 24 hours, any pods that restarted or entered CrashLoopBackOff,
and the top new or sharply rising Sentry issues. State clearly if nothing notable
happened.