Skip to content

Alerts

An alert on Observability → Alerts is a pod or workload failure Seylo observed in one of your clusters, or one of your uptime checks going down. It is not the result of a change you made — those are in Activity. This page covers what raises one, how it ends, and what you can do to it. Sending alerts to Slack, Teams, PagerDuty and the rest is on notification channels and routes.

Seylo reads pod status, Deployment status and Kubernetes Warning events. Each alert carries one of these reasons and its severity:

Reason Read from Severity
CrashLoopBackOff pod status critical
ImagePullBackOff, ErrImagePull, ErrImageNeverPull, InvalidImageName pod status critical
CreateContainerConfigError, CreateContainerError, RunContainerError pod status critical
OOMKilled the container’s last termination critical
FailedScheduling pod condition, or event warning
Evicted pod status, or event warning
ProgressDeadlineExceeded Deployment status critical
FailedCreate Deployment status (replica failure) critical
FailedMount, FailedAttachVolume event critical
ProbeFailure event (Unhealthy) warning
BackOff event warning

Events catch what container state never shows: a volume that will not mount or a probe that keeps failing never reaches a container’s status. Events and pod status are both namespaced, so alerts work in full on a namespace-scoped agent.

Alerts come from three places. The cluster state described below; alert rules you write over your own metrics and logs on Alerts → Rules — a threshold plus how long it must hold, or a one-click starter rule; and your uptime checks. Each rule runs with its author’s own access, re-checked every time it runs. Rules query the observability chart, so they need it installed. On the Rules page, How a rule works sits above the table and folds once you have read it; whether it is open is remembered in your browser.

Uptime checks raise an alert with reason UptimeDown when a check fails its Down after number of times in a row (2 by default), at the check’s own Alert severity (critical by default). Its first passing check resolves the alert. Uptime alerts belong to no cluster or namespace. See uptime checks.

See and switch off what raises alerts: Detectors

Section titled “See and switch off what raises alerts: Detectors”

Observability → Alerts → Detectors lists everything that can raise an alert:

  • Pod and workload failures, the table above;
  • Nightshift’s checks: problems no pod reports, such as autoscalers, Services, nodes, configuration that was never picked up, volumes, throttling, and a namespace over its ResourceQuota, which blocks every new pod (critical; see a namespace at its quota). Each must persist before it alerts;
  • Agent connection: a cluster’s agent offline past its grace period, or two agents claiming one cluster;
  • Uptime checks: a check down for its failure threshold. On by default, and switched for the whole organisation only, because uptime checks do not run in a cluster;
  • your rules.

An admin can turn any built-in detector off for the whole organisation or for one cluster (Per cluster). Off means the detector raises nothing until it is turned on again. Its open alerts are resolved, with a note saying who turned it off. Every change is recorded in the audit log. A releaser switches rules, as on the Rules tab.

An alert is identified by namespace, workload, container and reason — not by pod. A crash-looping pod that Kubernetes replaces every few minutes is one alert with a rising count, not a new alert per pod. The latest pod name is kept on the alert, and replicas failing the same way at the same time are collapsed into one alert ((3 pods affected)).

One crash is one alert even when Kubernetes reports it twice: the kubelet’s BackOff event, which names no container, and the pod’s own CrashLoopBackOff status are the same failure. A StatefulSet whose pods crash the same way is one alert on the StatefulSet, not one per pod.

Several alerts that the service map connects, within minutes of each other, are grouped into an incident. Each keeps its own row, marked part of incident ....

Some Kubernetes events are normal while a pod starts: a service-account or ConfigMap volume that has not mounted yet (FailedMount ... failed to sync configmap cache), a readiness probe before the app listens, or a pod waiting for the autoscaler’s new node. These raise an alert only if they are still happening after 5 minutes (10 for a pod waiting on capacity). They resolve on their own when the workload is Ready again, and they are a warning, not critical, while the workload is healthy.

An open alert resolves automatically only when both are true:

  1. the failure was not seen in the latest read, and
  2. its workload was seen, and is healthy.

The second condition matters. If the agent goes offline, a namespace leaves the policy, or permissions change, Seylo can no longer see the workload — and absence of evidence is not evidence of health, so the alert stays open.

An old event does not keep an alert open. Kubernetes keeps events for about an hour, so after a fix the BackOff events about the pod that was replaced are still there. An event last seen more than 10 minutes ago neither opens, reopens nor holds an alert. A crash loop that is still going on keeps its alert through the pod’s own status and through events that keep coming. Once the workload is seen healthy, the alert resolves.

A deleted namespace’s alerts resolve. When a namespace is deleted, its workloads are never seen again, so the rule above could never close their alerts. Every 2 minutes Seylo reads the Namespace of each open pod alert, and when the cluster answers that it does not exist, resolves the alert with the reason:

Namespace "<namespace>" no longer exists: a read of the cluster at <time> found no such namespace, so <namespace>/<workload> is gone and its <reason> cannot still be happening. Resolved, not deleted; if the namespace is created again and fails the same way, this alert reopens.

Only that answer resolves anything. An agent that is offline, a read that is refused, a namespace that is still terminating, or two agents claiming the same cluster resolve nothing.

An alert notifies once per episode:

  • when it fires;
  • when it comes back after resolving;
  • when its severity rises, for example from warning to critical.

While it keeps firing, its count and last seen keep updating on the board without notifying again.

An admin can turn on a reminder under Observability → Alerts → Detectors: Remind while unacknowledged, every 1, 2, 4, 8, 12, 24 or 48 hours, or every 7 days. It is off by default (Never - notify once per episode). A reminder goes out while an alert fires and nobody has acknowledged it. Acknowledged and snoozed alerts never remind.

Status Meaning Notifies?
firing Being observed Once per episode, and on a reminder if your organisation turned one on
acknowledged Someone has said they have it No, while it stays acknowledged
resolved Ended — by the cluster, or by a person with a reason Sends “resolved” to channels that were told it fired

A snooze or a silence keeps the alert visible on the board; it only stops notifications.

Each row has one button, Investigate & fix, and a menu with the rest. On a resolved alert the button reads Investigate: there is nothing left to fix.

Action Role What it does
Acknowledge viewer “I have it.” Stops notifications and reminders while acknowledged. If it resolves and recurs, it notifies again
Un-acknowledge viewer Back to firing. It notifies again when its severity rises, when it recurs after resolving, or when your organisation’s reminder is due
Snooze… viewer Quiet until a time you choose — 1 hour, 4 hours, 24 hours, or a time of your own — at most 7 days. Notifications resume by themselves when it ends. End snooze ends it early
Assign… viewer Says who is on it: a person your organisation names by email address. Does not change who is notified
Timeline and comments viewer Who did what and when, what was sent to which channel, and a comment box
Check now viewer Re-reads the cluster and says whether the failure is still there. On an uptime alert, it runs the check
What changed viewer Changes Seylo made in the same cluster and namespace in the two hours before the alert first fired
Resolve… releaser Closes the alert for everyone, with a reason of at least 12 characters, and tells every channel that was told it fired
Investigate & fix viewer to read the fix; releaser to investigate One panel under the row. For a releaser it starts Nightshift, Seylo’s AI SRE: evidence from the diagnose rules, the change history and the crashed container’s last log lines, numbered and redacted, then, with a model configured, reads of the cluster from a fixed menu, within fixed budgets, ending with findings that cite the evidence and up to three proposed changes, each a diff you approve. Under the run is the playbook: what Seylo would do about this alert, whether it may do it unattended, and the button that does it. Below releaser the panel shows the playbook only. An alert that is part of an incident is investigated once, with the incident. See investigate and fix

Mark all read clears the bell badge. It is not acknowledgement: an unacknowledged alert still reminds, if your organisation turned reminders on.

Comments are redacted before they are stored, because a comment is where somebody pastes the token they just rotated. A comment is at most 2000 characters and is visible to everyone in your organisation.

What changed says “started N minutes after”, never “caused by”. An empty answer means Seylo recorded no change — not that nothing changed: kubectl, a reconciler sync or a node upgrade are not in Seylo’s record.

Resolving by hand is a claim — the cluster has the last word

Section titled “Resolving by hand is a claim — the cluster has the last word”

If a failure you resolved is observed again, the same alert reopens — same row, same history — and notifies like any recurrence. Its timeline says it was observed failing again after you resolved it. That line is often the most useful thing on the timeline: somebody’s fix did not hold.

Check now is the evidence-based way to close an alert. Resolve is for “we know, it is handled, stop telling us”.

A reason that is too short is refused with a message beginning:

Resolving by hand needs a reason of at least 12 characters, such as "rolled back to 1.4.2".

For a time of your own, the picker offers nothing past 7 days from now. A later time typed into the field shows the limit and the latest time allowed under it, and Snooze stays off until you change it. For anything longer, create a silence instead. Snoozes are for “not now”; anything longer is a decision, and a silence records it with a reason and an end that everyone can see.

An alert is a record of something your cluster did, so there is no delete. It is resolved — by the cluster, by a person with a reason, or with a reason when its namespace is deleted — and resolved alerts are removed by retention (30 days by default).

A silence quiets many alerts for planned work such as a node pool upgrade or a migration. Observability → Alerts → Silences.

Role to create or end one approver (anyone can read the list)
Matches on cluster, namespace, workload, reason, severity — at least one
Reason Required, at least 12 characters. Shown on every alert it quiets
End Required, at most 90 days ahead. There is no permanent silence. The picker offers nothing later, and a later time typed in shows the latest one allowed under the field and keeps Create silence off
Active silences per organisation At most 200

What a silenced alert still does: it is recorded, counted, shown on the board, and resolved when the workload is healthy. Only notification is held back, and its timeline names the silence that held it.

Before you save, the form shows how many open alerts the silence would cover.

Every field you fill must match; an empty field matches anything.

  • * matches any run of characters: prod-* matches prod-eu and prod-us.
  • | separates alternatives: critical|warning.
  • Severity takes critical, warning or info.
  • A field that is only * is refused — leave the field empty instead.

There are no regular expressions and no negation, deliberately. A regular expression typed into a silence is how “silence shop” becomes “silence every namespace containing shop”, and “everything except X” quietly grows as new things appear.

End now ends it at once. Ended silences are not deleted: they stay listed for 7 days, because “who muted this, when, and why” is exactly what gets asked after an incident nobody was told about.

With no notification routes, alerts reach people by email according to their role, as a daily digest by default. Once an administrator enables a route, routes alone decide who is told. See notification channels and routes.