Incidents
When a database fails, the API that calls it fails next, and then the web front end that calls the API. That is three alerts and one problem. Seylo puts alerts like these together as one incident, finds the service where the failure started, and has Nightshift investigate it once, instead of three times with three different answers.
Incidents are on Nightshift → Incidents (/incidents). Show filters
them: Open, Closed or All. An alert that belongs to one says
part of incident ... on Observability → Alerts.
Incidents use the service map: what calls what. The better the map knows your services - from traces, a mesh, your configuration or an architecture file - the better incidents group.
What makes alerts one incident
Section titled “What makes alerts one incident”Two alerts are related when both hold:
- They started within 15 minutes of each other. A cascade’s later symptoms arrive late by design: a readiness alert waits 5 minutes, a crash loop backs off.
- The service map connects them within 3 steps, in one of two ways:
- a cascade - one reaches the other along what it calls;
- a shared dependency - both call the same thing, which has at most 10 callers. A dependency half your organisation calls would join every coincidence, so it does not.
Two services that are both called by the same caller are not joined: two dependencies failing at once is no evidence of one cause. A NetworkPolicy that merely allows traffic, and an edge someone hid on the map, never join alerts.
A call that succeeds does not join. A step is followed only when the caller’s calls into the other service are failing. When a trace or mesh sample read since the earlier of the two alerts started shows at least 10 calls with fewer than 5% failing, the two alerts are left apart: the failure did not travel along that call. With no sample, fewer calls, or a sample read before the alerts started, there is no evidence either way, and the pair joins as above. Without this, a busy estate’s unrelated alerts, a few steps apart on the map, were pulled into one incident.
Two or more related alerts, at least one still open, make an incident. A lone alert is not one: it already has its own page, and an incident lets Nightshift read more widely, which should not happen for every alert.
An open incident grows when an alert related to any of its members arrives, and two incidents merge when an alert relates to both. It closes when every member alert has resolved, and reopens if one fires again within 15 minutes. Members are never removed. An incident holds at most 100 alerts. An alert that would have joined a full incident is still part of that incident, beyond its 100-alert limit: it is not one of the 100 that Nightshift reads, it does not open a second incident, and it starts no investigation of its own. Its notification, its alert page and the incident’s page say so. If the incident closes while the alert is still open, the alert is free to open the next one. An incident never spans two clusters. Closed incidents are kept for 90 days.
An alert about something that is not on the map - an uptime check, an agent going offline, a rule whose series names no workload, or a workload the map has not read yet - joins no incident.
Every join is kept and shown: whether it was a cascade or a shared dependency, the path between the two, each edge with how it is known and how sure, and the minutes apart. A configured edge is not proof of a call, and the page says which kind joined each pair.
The incident page
Section titled “The incident page”| Section | What it shows |
|---|---|
| Where it started | The root-cause walk’s answer: a lead for Nightshift to confirm, not a conclusion |
| Who else is hit | The services that depend on the failure |
| On the service map | The part of the map the incident touches: the root outlined strongly, members outlined, affected services tinted |
| Member alerts | The alerts that make up the incident, and why each joined |
| Nightshift | The incident’s one investigation, and Investigate the incident |
| The fix on the root | One proposed fix, through the right door |
| Seen before | Past incidents with a similar shape, from your organisation’s memory |
| What we learned | Written by Seylo when the incident closes: the root, the path, the fix and whether it held |
| Timeline | When it opened, grew, merged, was walked, investigated, fixed and closed |
Where it started: the root-cause walk
Section titled “Where it started: the root-cause walk”From each failing service, Seylo follows what it calls, step by step, and stops at the deepest unhealthy service whose own dependencies are healthy. That service is the root.
A service is:
- unhealthy when it has an alert, when its callers’ calls into it fail (5% or more, over at least 10 calls observed in traces or a mesh), or when a health signal says it is failing - so a database with no alert of its own can still be the root;
- healthy when Seylo watches it and sees no alert and no failing call into it, or a store’s exporter says it answers and nothing is failing;
- unknown when there is no evidence either way: a database or external host nobody’s traces reach, a declared component, something the map names but could not read, a namespace the incident may not read. Unknown is never treated as healthy.
Latency is shown, and decides nothing: without a baseline, a fixed threshold would call every slow-by-design API unhealthy.
The walk also notes what changed just before: changes Seylo made in the hour before the incident started, and, for a Deployment, a new revision (a new ReplicaSet) whether Seylo made it or not. A StatefulSet’s rollouts are not read, and history that was not read is said as not read, never as “no change”.
Asynchronous hops are not followed. A consumer’s failure does not fail the producer that put the message on the queue, so the walk does not descend from a producer through a queue or topic; each such hop is listed as a caveat.
The answer is root, with the chain of evidence, only when one candidate is confirmed and nothing ties it. Otherwise it is inconclusive, with the reason: an unknown dependency, a dependency in a namespace the incident may not read, two failures at once that explain the same symptoms, a cycle, or no member on the map.
Who else is hit
Section titled “Who else is hit”Seylo climbs from the failing service to its callers, up to 4 steps and 50 services:
- affected - they alert, or their calls toward the failure fail;
- at risk - they depend on it and are quiet so far (they may retry, cache, or not have been called yet).
Climbing stops at an asynchronous hop; those producers are listed apart as not affected.
Across clusters
Section titled “Across clusters”When one of your clusters sends traffic to another - a front proxy kept in front of a migrated app, for example - the walk follows that link into the other cluster. The incident page then has an Across clusters section: the links the walk followed, and alerts on the other cluster that the map relates to this incident, listed as symptoms there rather than made members. An incident stays one cluster’s. When Nightshift has found where the 5xx is born, the page says so on a line of its own. See across clusters.
Nightshift investigates the incident once
Section titled “Nightshift investigates the incident once”When an incident’s alert fires and automatic investigation is on, Nightshift
runs once for the incident. The run starts on the incident’s anchor -
the root’s alert when the walk named a root, else the earliest open member -
and every other member’s own run is skipped and says
Part of incident .... The organisation’s
limits apply, and an
incident runs at most once every 6 hours. The run, its findings and its
proposals are kept on the anchor alert; the incident page shows them too.
A new kind of failure gets one follow-up run. When an alert joins the incident after its run started, with a kind of failure none of the incident’s alerts had when the run started, Nightshift runs once more for the incident, on its anchor. The incident is walked again first, and Nightshift is told about the new alert and the earlier run and asked whether it is the same root or a separate failure. The same kind of failure on another workload does not start another, and the organisation’s limits still apply.
The run is Nightshift’s usual investigation, with three differences:
- It reads the incident’s namespaces, not one. While
Read across namespaces during an incident is on (the default, on
Nightshift → Settings → During an incident), that is the namespaces of
its alerts plus the ones the service map connects to them. Namespaces an
adminlists under Never read - one name for every cluster (payments-pci), orcluster/namespacefor one - are never read. With the setting off, the run reads only its alerts’ namespaces. - The walk is evidence. Nightshift gets the walk’s answer as numbered evidence, to confirm or refute, not to repeat.
- It is asked for the root, the failure and where it propagated. Matching skills, including ones scoped to any service in the incident, and Seylo’s incident runbook are listed first.
A releaser can press Investigate the incident at any time. A run a person
starts reads, and shows, only the namespaces that person may see, and is
refused if they cannot see the anchor’s.
The fix on the root
Section titled “The fix on the root”Under the investigation, The fix on the root proposes one fix, on the service the walk named, through one of these doors:
| Door | When |
|---|---|
| in the cluster | The root’s workload is changed directly, through the usual approval |
| pull request | Argo CD or Flux deploys the root from Git: the change goes to the owner’s repository as a pull request. See pull requests for workloads Git owns |
| coding agent | The fault is in the code: a task for the hosted coding agent |
| no safe automatic fix | Nothing can be safely changed for you; the card gives the evidence and the next step |
What it proposes depends on what the evidence shows:
| The root | The fix |
|---|---|
| Is OOM-killed | A memory limit sized from evidence |
| Was released by Seylo in the hour before | A rollback to the previous release |
| Was rolled out outside Seylo, and Git owns it | The previous ReplicaSet’s image, back in Git, as a pull request |
| Is a consumer that runs but whose lag, or its broker’s backlog, is still rising | A restart, verified by the lag falling |
| Has an exception where requests through it drop, and nothing changed on it | A task for the coding agent: the incident’s evidence and “fix it and add a regression test”, in the repository your architecture file names |
| Had a ConfigMap or environment change in the hour before | Explained: the change and where its previous version is. Nightshift never writes a ConfigMap |
| Has its connection pool, or the database’s connections, failing | Explained, with the exact change for a person |
| Anything else, or an inconclusive walk | No fix: the evidence and the next step |
The card shows the evidence, the checks, Verified when: and Undo:, and Next:. When a door is not ready, it lists What is missing for this door and the alternative, for example Propose it in the cluster instead (owner acknowledged on approval).
A person proposes it, and a person approves it. releaser presses
Propose this fix (or Propose as a pull request, or
Start the coding-agent task). Nightshift then plans it from fresh evidence
and the fix appears on its card below, to be approved like any
Nightshift fix. After an automatic incident
run, a fix in the cluster is proposed on its own, and runs unattended only
where a standing grant covers its action. Earned autonomy never covers an
incident’s fix. A pull request and a coding-agent task never start without a
person, and Seylo never merges either.
Who sees what
Section titled “Who sees what”Each person sees an incident cut to the namespaces they may see:
- An incident exists for you only if you can see at least one of its alerts. Otherwise its address answers “not found”, because its existence would name a failure you cannot see.
- A member alert in a namespace you cannot see reads “an alert in a namespace you cannot see”, with no name, workload or reason.
- A service in a namespace you cannot see is “a service in a namespace you cannot see”, and any line that names it is left out whole. The page says how many namespaces you cannot see, never their names.
- The title names the root only when you may see it.
Notifications
Section titled “Notifications”Incidents add a line to alert notifications; who is told does not change.
- The alert whose arrival opened the incident says
Opens incident ...: <n> related alerts on the service map, with the root when the walk has one. - Later members say
Part of incident ... (<title>), <n> alerts since <time>.
Slack, Microsoft Teams, Google Chat and email carry the line. The webhook adds
an incident object; PagerDuty and Opsgenie carry the incident’s ID. Every
member alert still notifies through your
routes as before, because routes may send
each service’s alerts to its own team. Silences and snoozes work exactly as on
alerts.