Alerts
An alert on Observability → Alerts is a pod or workload failure Seylo observed in one of your clusters, or one of your uptime checks going down. It is not the result of a change you made — those are in Activity. This page covers what raises one, how it ends, and what you can do to it. Sending alerts to Slack, Teams, PagerDuty and the rest is on notification channels and routes.
What raises an alert
Section titled “What raises an alert”Seylo reads pod status, Deployment status and Kubernetes Warning events. Each alert carries one of these reasons and its severity:
| Reason | Read from | Severity |
|---|---|---|
CrashLoopBackOff |
pod status | critical |
ImagePullBackOff, ErrImagePull, ErrImageNeverPull, InvalidImageName |
pod status | critical |
CreateContainerConfigError, CreateContainerError, RunContainerError |
pod status | critical |
OOMKilled |
the container’s last termination | critical |
FailedScheduling |
pod condition, or event | warning |
Evicted |
pod status, or event | warning |
ProgressDeadlineExceeded |
Deployment status | critical |
FailedCreate |
Deployment status (replica failure) | critical |
FailedMount, FailedAttachVolume |
event | critical |
ProbeFailure |
event (Unhealthy) |
warning |
BackOff |
event | warning |
Events catch what container state never shows: a volume that will not mount or a probe that keeps failing never reaches a container’s status. Events and pod status are both namespaced, so alerts work in full on a namespace-scoped agent.
Alerts come from three places. The cluster state described below; alert rules you write over your own metrics and logs on Alerts → Rules — a threshold plus how long it must hold, or a one-click starter rule; and your uptime checks. Each rule runs with its author’s own access, re-checked every time it runs. Rules query the observability chart, so they need it installed. On the Rules page, How a rule works sits above the table and folds once you have read it; whether it is open is remembered in your browser.
Uptime checks raise an alert with reason UptimeDown when a check fails
its Down after number of times in a row (2 by default), at the check’s own
Alert severity (critical by default). Its first passing check resolves
the alert. Uptime alerts belong to no cluster or namespace. See
uptime checks.
See and switch off what raises alerts: Detectors
Section titled “See and switch off what raises alerts: Detectors”Observability → Alerts → Detectors lists everything that can raise an alert:
- Pod and workload failures, the table above;
- Nightshift’s checks: problems no pod reports, such as autoscalers, Services, nodes, configuration that was never picked up, volumes, throttling, and a namespace over its ResourceQuota, which blocks every new pod (critical; see a namespace at its quota). Each must persist before it alerts;
- Agent connection: a cluster’s agent offline past its grace period, or two agents claiming one cluster;
- Uptime checks: a check down for its failure threshold. On by default, and switched for the whole organisation only, because uptime checks do not run in a cluster;
- your rules.
An admin can turn any built-in detector off for the whole organisation or
for one cluster (Per cluster). Off means the detector raises nothing
until it is turned on again. Its open alerts are resolved, with a note saying
who turned it off. Every change is recorded in the audit log. A releaser
switches rules, as on the Rules tab.
One alert per failure, not per pod
Section titled “One alert per failure, not per pod”An alert is identified by namespace, workload, container and reason — not by
pod. A crash-looping pod that Kubernetes replaces every few minutes is one alert
with a rising count, not a new alert per pod. The latest pod name is kept on
the alert, and replicas failing the same way at the same time are collapsed
into one alert ((3 pods affected)).
One crash is one alert even when Kubernetes reports it twice: the kubelet’s
BackOff event, which names no container, and the pod’s own
CrashLoopBackOff status are the same failure. A StatefulSet whose pods crash
the same way is one alert on the StatefulSet, not one per pod.
Several alerts that the service map connects,
within minutes of each other, are grouped into an
incident. Each keeps its own row, marked
part of incident ....
Startup events wait before they alert
Section titled “Startup events wait before they alert”Some Kubernetes events are normal while a pod starts: a service-account or
ConfigMap volume that has not mounted yet (FailedMount ... failed to sync configmap cache), a readiness probe before the app listens, or a pod waiting
for the autoscaler’s new node. These raise an alert only if they are still
happening after 5 minutes (10 for a pod waiting on capacity). They
resolve on their own when the workload is Ready again, and they are a
warning, not critical, while the workload is healthy.
How an alert ends on its own
Section titled “How an alert ends on its own”An open alert resolves automatically only when both are true:
- the failure was not seen in the latest read, and
- its workload was seen, and is healthy.
The second condition matters. If the agent goes offline, a namespace leaves the policy, or permissions change, Seylo can no longer see the workload — and absence of evidence is not evidence of health, so the alert stays open.
An old event does not keep an alert open. Kubernetes keeps events for
about an hour, so after a fix the BackOff events about the pod that was
replaced are still there. An event last seen more than 10 minutes ago
neither opens, reopens nor holds an alert. A crash loop that is still going on
keeps its alert through the pod’s own status and through events that keep
coming. Once the workload is seen healthy, the alert resolves.
A deleted namespace’s alerts resolve. When a namespace is deleted, its workloads are never seen again, so the rule above could never close their alerts. Every 2 minutes Seylo reads the Namespace of each open pod alert, and when the cluster answers that it does not exist, resolves the alert with the reason:
Namespace "<namespace>" no longer exists: a read of the cluster at <time> found no such namespace, so <namespace>/<workload> is gone and its <reason> cannot still be happening. Resolved, not deleted; if the namespace is created again and fails the same way, this alert reopens.Only that answer resolves anything. An agent that is offline, a read that is refused, a namespace that is still terminating, or two agents claiming the same cluster resolve nothing.
When an alert notifies
Section titled “When an alert notifies”An alert notifies once per episode:
- when it fires;
- when it comes back after resolving;
- when its severity rises, for example from warning to critical.
While it keeps firing, its count and last seen keep updating on the board without notifying again.
Reminders
Section titled “Reminders”An admin can turn on a reminder under Observability → Alerts → Detectors: Remind while unacknowledged, every 1, 2, 4, 8, 12, 24 or 48 hours, or every 7 days. It is off by default (Never - notify once per episode). A reminder goes out while an alert fires and nobody has acknowledged it. Acknowledged and snoozed alerts never remind.
Statuses
Section titled “Statuses”| Status | Meaning | Notifies? |
|---|---|---|
firing |
Being observed | Once per episode, and on a reminder if your organisation turned one on |
acknowledged |
Someone has said they have it | No, while it stays acknowledged |
resolved |
Ended — by the cluster, or by a person with a reason | Sends “resolved” to channels that were told it fired |
A snooze or a silence keeps the alert visible on the board; it only stops notifications.
Acting on an alert
Section titled “Acting on an alert”Each row has one button, Investigate & fix, and a menu with the rest. On a resolved alert the button reads Investigate: there is nothing left to fix.
| Action | Role | What it does |
|---|---|---|
| Acknowledge | viewer |
“I have it.” Stops notifications and reminders while acknowledged. If it resolves and recurs, it notifies again |
| Un-acknowledge | viewer |
Back to firing. It notifies again when its severity rises, when it recurs after resolving, or when your organisation’s reminder is due |
| Snooze… | viewer |
Quiet until a time you choose — 1 hour, 4 hours, 24 hours, or a time of your own — at most 7 days. Notifications resume by themselves when it ends. End snooze ends it early |
| Assign… | viewer |
Says who is on it: a person your organisation names by email address. Does not change who is notified |
| Timeline and comments | viewer |
Who did what and when, what was sent to which channel, and a comment box |
| Check now | viewer |
Re-reads the cluster and says whether the failure is still there. On an uptime alert, it runs the check |
| What changed | viewer |
Changes Seylo made in the same cluster and namespace in the two hours before the alert first fired |
| Resolve… | releaser |
Closes the alert for everyone, with a reason of at least 12 characters, and tells every channel that was told it fired |
| Investigate & fix | viewer to read the fix; releaser to investigate |
One panel under the row. For a releaser it starts Nightshift, Seylo’s AI SRE: evidence from the diagnose rules, the change history and the crashed container’s last log lines, numbered and redacted, then, with a model configured, reads of the cluster from a fixed menu, within fixed budgets, ending with findings that cite the evidence and up to three proposed changes, each a diff you approve. Under the run is the playbook: what Seylo would do about this alert, whether it may do it unattended, and the button that does it. Below releaser the panel shows the playbook only. An alert that is part of an incident is investigated once, with the incident. See investigate and fix |
Mark all read clears the bell badge. It is not acknowledgement: an unacknowledged alert still reminds, if your organisation turned reminders on.
Comments are redacted before they are stored, because a comment is where somebody pastes the token they just rotated. A comment is at most 2000 characters and is visible to everyone in your organisation.
What changed says “started N minutes after”, never “caused by”. An empty
answer means Seylo recorded no change — not that nothing changed: kubectl, a
reconciler sync or a node upgrade are not in Seylo’s record.
Resolving by hand is a claim — the cluster has the last word
Section titled “Resolving by hand is a claim — the cluster has the last word”If a failure you resolved is observed again, the same alert reopens — same row, same history — and notifies like any recurrence. Its timeline says it was observed failing again after you resolved it. That line is often the most useful thing on the timeline: somebody’s fix did not hold.
Check now is the evidence-based way to close an alert. Resolve is for “we know, it is handled, stop telling us”.
A reason that is too short is refused with a message beginning:
Resolving by hand needs a reason of at least 12 characters, such as "rolled back to 1.4.2".Snoozing for longer than 7 days
Section titled “Snoozing for longer than 7 days”For a time of your own, the picker offers nothing past 7 days from now. A later time typed into the field shows the limit and the latest time allowed under it, and Snooze stays off until you change it. For anything longer, create a silence instead. Snoozes are for “not now”; anything longer is a decision, and a silence records it with a reason and an end that everyone can see.
Alerts are not deleted
Section titled “Alerts are not deleted”An alert is a record of something your cluster did, so there is no delete. It is resolved — by the cluster, by a person with a reason, or with a reason when its namespace is deleted — and resolved alerts are removed by retention (30 days by default).
Silences
Section titled “Silences”A silence quiets many alerts for planned work such as a node pool upgrade or a migration. Observability → Alerts → Silences.
| Role to create or end one | approver (anyone can read the list) |
| Matches on | cluster, namespace, workload, reason, severity — at least one |
| Reason | Required, at least 12 characters. Shown on every alert it quiets |
| End | Required, at most 90 days ahead. There is no permanent silence. The picker offers nothing later, and a later time typed in shows the latest one allowed under the field and keeps Create silence off |
| Active silences per organisation | At most 200 |
What a silenced alert still does: it is recorded, counted, shown on the board, and resolved when the workload is healthy. Only notification is held back, and its timeline names the silence that held it.
Before you save, the form shows how many open alerts the silence would cover.
How matchers work
Section titled “How matchers work”Every field you fill must match; an empty field matches anything.
*matches any run of characters:prod-*matchesprod-euandprod-us.|separates alternatives:critical|warning.- Severity takes
critical,warningorinfo. - A field that is only
*is refused — leave the field empty instead.
There are no regular expressions and no negation, deliberately. A regular
expression typed into a silence is how “silence shop” becomes “silence every
namespace containing shop”, and “everything except X” quietly grows as new
things appear.
Ending a silence
Section titled “Ending a silence”End now ends it at once. Ended silences are not deleted: they stay listed for 7 days, because “who muted this, when, and why” is exactly what gets asked after an incident nobody was told about.
Who gets told
Section titled “Who gets told”With no notification routes, alerts reach people by email according to their role, as a daily digest by default. Once an administrator enables a route, routes alone decide who is told. See notification channels and routes.