Skip to content

Nightshift's fixes

When Nightshift finds a cause it can act on, it files a fix. A fix is never a bare suggestion: before anyone sees it, Seylo has worked out the exact change, checked it against what was read, and decided how it will check that the fix worked and how it would undo it. Nothing changes in your cluster until a person approves it, unless one of the envelopes on Operate → Automation covers it.

When the cause is outside what Seylo changes - the application’s code, a database, a cloud setting, another team’s system - there is no fix card, but the investigation still gives the remedy: where the change goes, the exact change, who applies it and how it is verified. See the fix is a remedy.

Nightshift → Fixes lists every fix Nightshift proposed, newest first:

Column What it shows
Proposed When
Fix What it changes, and the kind of fix
Where namespace/workload, and the cluster
Door Cluster, or Pull request (with its number once opened) for a workload whose owner reads it from Git
Status Where it got to, and approved by the person, or on its own
From Where it came from - Alert, Incident, Readiness campaign or APM coverage - and a link to the card where it is approved, verified or undone

Filter by Status, Cluster and Namespace. The filters are part of the page’s address, so a filtered view can be shared. Nothing is approved from this list: each row opens the card the fix lives on. You see only fixes in namespaces you may see, and they are the only ones counted.

Status Meaning
waiting for a person Proposed; nobody has approved it
sent for approval Approved on the card and filed as an approval request
pull request open Opened in the owner’s repository; waiting for it to be merged and synced
applied - verifying In the cluster; its check is running
verified Its check passed for the whole window
did not verify - undone Its check failed, and Seylo applied the exact inverse and read it back
undo failed Its check failed and the undo did not take: a person needs to look
did not verify Its check failed and nothing was undone: the fix has no automatic undo, or the object it changed no longer exists (There is nothing to undo: the object is gone). It says which
did not take The write was refused or failed; nothing changed
will not change it Refused before it was filed, with the reason
rejected, dismissed A person said no
No longer needed Its problem went away before it ran; nothing was changed

Every fix, wherever it came from, shows the same parts:

  • the evidence, numbered, that the fix rests on;
  • the change, as a diff: each field, its current value and the new one;
  • Verified when: what Seylo checks afterwards, how long it must hold, and by when - a fix not verified in time is undone;
  • Undo: the exact inverse, or none - and the reason there is none;
  • who approved it, or that it ran on its own under a grant.

Approve files it as an approval request (approvals on) or applies it at once (approvals off), through the same engine as every other change: the ownership check, a server-side dry run, the apply and a read-back. Seylo re-checks the fix just before it runs and refuses one whose problem is gone, or whose target changed since it was planned.

Fix history is kept per workload. A workload that is deleted and created again under the same name starts with no history: the earlier one’s fixes, refusals and memory-limit steps are not counted against it. Its measured usage starts clean too: CPU and memory sizing, and the autoscaler or bigger request judgement, read usage only since the workload was created, so the old workload’s load is not counted. When the workload’s creation time cannot be read, all history is kept, the cautious side. Agents 0.2.16 and 0.2.17 send creation times empty; Seylo then reads them from the observability chart’s metrics where it has them. Agent 0.2.18 sends them again.

Nightshift recognises specific failures - for example an OOM kill, a crash right after a release, or a missing image tag. For each one the panel under the alert shows the evidence, the exact change as a diff, how Nightshift will check that the fix worked (and for how long), and how it would undo it.

Every fix waits for a person until that class of fix has earned autonomy in your organisation: 3 approved fixes that each verified for low-risk classes, 5 for medium-risk ones, and never for high-risk ones - and only inside an envelope an admin turns on with an end date at most 90 days away.

Operate → Automation → Fixes Nightshift may apply holds that envelope: Let proven fixes run on their own, with an Until date. The date picker offers only days within the limit; a date typed past it shows the limit and the latest date allowed under the field, and Save stays off until it is changed. On a cluster where an admin has switched Nightshift off, no fix runs on its own, whatever the envelope says. The page lists every class with four columns: the fix, its Risk, Proven (how many times it was applied here and checked to have worked, out of the number it needs), and Mode: Ask first keeps a class waiting for a person however often it is proven; Auto once proven lets it run on its own once proven, while the envelope is on. High-risk classes show Always asks. Growing a nearly full volume and recreating a claim that never bound show Allowed below or Asks first: they never earn autonomy from approvals, because a person approving five disk growths has not approved the cost of the sixth. Only their own grant, on the cards below the table, lets them run on their own; see what Nightshift may do on its own. Hover the (i) next to a fix to see how Nightshift checks that it worked. Unattended fixes stay inside built-in limits, are limited to one per workload per day, and are still verified. If a fix does not hold, Nightshift applies the exact inverse and reads it back. A workload owned by Argo CD, Flux or Helm is never changed unattended.

When a release causes the failure - pods that crash-loop soon after a change, or an image that cannot be pulled or parsed - the fix is to go back to the revision that worked. Nightshift picks which revision in this order:

  1. Seylo’s own release history, when the live image is the one Seylo recorded as current. The rollback restores the whole recorded revision, as rollback from History does.
  2. The Deployment’s own previous revision, when Seylo did not record the release (it was set with kubectl or a pipeline), or the workload was changed outside Seylo so that Seylo’s history is not what ran before. This is the revision the cluster kept - the previous ReplicaSet, which is what kubectl rollout undo goes back to. The fix’s title ends (the Deployment's own previous revision, ReplicaSet <name>).

The second kind is written as a patch of only the fields that differ between the live pod template and the previous one, container by container, by name. Each field is checked again on the live Deployment just before it is applied, so a change someone made in the meantime refuses the write instead of being overwritten. Its undo is the exact inverse: forward again to the revision it replaced.

It asks a person, whatever a grant or earned autonomy says. The card says why: It goes back to a revision Seylo did not record (the change it undoes was made outside Seylo), so a person decides; it never runs unattended.

One exception: an image that never started. When the new image’s tag does not exist or its name cannot be parsed, none of the release’s pods ever ran. Then, under a grant that names the image-can’t-start rollback, or once that fix has earned autonomy, the Deployment’s own previous revision goes back with nobody approving - only when no pod of the new ReplicaSet is Ready and a pod of the previous one is Ready now. A rollback to the Deployment’s own revision after a crash, or of a stuck rollout, always asks a person.

A crash or stuck rollout right after a Seylo release. The first kind, back to Seylo’s own recorded release, can run on its own under a grant that names the crash or stuck-rollout rollback: only when the failure began within 30 minutes of a release Seylo made, and the release before it is one Seylo recorded as succeeded, with no other change to the workload in between. A grant naming Roll back to the previous revision (any reason), or earned autonomy, covers it as before.

It is refused, with the reason, when:

  • the workload is not a Deployment (the cluster keeps revisions for Deployments);
  • the Deployment’s ReplicaSets could not be read, or none of them carries the live pod template, or there is no earlier one;
  • the previous revision has other containers than the live one: adding or removing a container is kubectl rollout undo for a person;
  • the previous revision is the same as the live one.

When neither kind is possible, the refusal gives both reasons, for example ... the workload was changed outside Seylo, ... Nor can it go back to the Deployment's own previous revision: it has no earlier revision than ReplicaSet <name>.

Reading ReplicaSets is part of the agent’s default write tier (tier 1). A Deployment’s first ReplicaSet is never taken as “a rollout outside Seylo”. Choosing the previous ReplicaSet needs each one’s creation time; see a workload created again for agents 0.2.16 and 0.2.17.

A pod waits in Pending because the PersistentVolumeClaim it mounts names a StorageClass that does not exist, so the claim can never bind. A claim’s storageClassName cannot be changed, so the only fix is to recreate the claim: Nightshift deletes it and creates it again under the same name, with your cluster’s default StorageClass. The workload needs no change, because the claim it names exists again.

Nightshift proposes this only for a claim that provably holds no data. It diagnoses it only when the API server answered “not found” for the class; a read that was refused or failed is never taken as “missing”. It refuses, with the step a person takes instead, when the claim:

  • names a volume (spec.volumeName), or is Bound or Lost - a volume, and data, may exist;
  • is already being deleted;
  • is restored or cloned (dataSource, dataSourceRef);
  • comes from a StatefulSet’s volumeClaimTemplates, or has any other owner, which would recreate it as it was;
  • has a selector, which asks for a volume someone made by hand;
  • requests no size;
  • is used by a pod that is already scheduled: Kubernetes keeps such a claim, so the delete would hang. Scale the workload to 0 (or delete that pod) first, then ask again;

and when the cluster has no single default StorageClass (none, several, or they could not be read). Nightshift never picks a class itself.

The new claim keeps the old one’s labels, annotations (less the binding and provisioning ones), access modes, size and volume mode. Seylo builds it from the live claim; the model never writes a claim.

  1. Open the fix from Nightshift → Fixes or from the alert. It waits for a person: an automatic investigation proposes it but never files it for approval on its own.
  2. Read the card: it names the claim, the class it will use, and that this needs the agent’s destructive tier.
  3. Type the claim’s exact name, tick I understand the claim is deleted and created again., and press Approve. This needs the approver role.

It then runs as one request, in order: re-read the claim (refused if it bound, changed or gained an owner since); a server-side dry run of the new claim; delete the old claim; wait up to 45 seconds until it is gone; create the new one; read it back. Pods are never deleted.

It needs the agent’s destructive tier. The agent’s default role creates and patches claims but never deletes them. Turn on rbac.write.destructive.enabled=true (or rbac.posture=admin) on that cluster’s agent first; see the destructive tier. Without it the delete is refused, the old claim is left exactly as it was, nothing is created, and the fix says so:

Not recreated: the agent's role may not delete PersistentVolumeClaims, so claim <name> was left untouched and nothing was created. Deleting a claim is the agent's destructive tier: set rbac.write.destructive.enabled=true on that cluster's agent (...), then approve the fix again.

It runs unattended only for a claim proven never to have bound, and only when an admin has allowed it under Operate → Automation → Claims that never bound: no volume, no PersistentVolume naming it, and no pod ever running with it, so there is no data to lose. Anything Nightshift could not read keeps it a person’s, and a claim that is or ever was bound is never recreated unattended. Earned autonomy never covers it. See recreate a claim that never bound.

The undo, if it does not verify: delete the new claim, only while it is still unbound, and create the original again, its StorageClass and all. A claim that has bound or names a volume is never deleted by the undo; that is reported to a person instead. The fix is verified when the new claim is Bound on the default class and every pod of the workload is Ready, held for 2 minutes, within 10. If it does not verify, Seylo reports it and changes nothing more.

If the old claim is still terminating after the wait, the new one is not created; the message names what holds it (its finalizers). Approve the fix again once it is gone, and Nightshift re-plans from fresh evidence.

When a container is OOMKilled, Nightshift sizes the new memory limit from evidence, not from a fixed step:

Evidence Read from
What the container declares Its command and arguments, and a short list of environment variables: a JVM -Xmx, -XX:MaxRAMPercentage and similar, Node.js --max-old-space-size, Go GOMEMLIMIT, stress --vm-bytes, JAVA_TOOL_OPTIONS, NODE_OPTIONS, HEAP_SIZE and a few more. Only the matched setting is kept (-Xmx512m), never the rest of the value. A value that comes from a Secret is never read, and one from a ConfigMap is not read either: both are reported as present and unread
Its crash log The kernel’s kill, and log lines such as a failed allocation with its size
The same image elsewhere A healthy workload in the same namespace running the same image tag: all replicas Ready and no OOM kill in 24 hours
Its memory over 24 hours The shape of its working set, cut at each restart

The new limit adds the runtime’s own overhead and some headroom, and stays under the room there is: the largest node the pod may run on, your ResourceQuota and LimitRange, and the maximum on Operate → Automation. A number above the room is refused, naming the bound. The request is left as it is.

It is refused when:

  • the memory rises steadily across restarts - a leak. More memory only delays the next kill: Rising across restarts: more memory only delays the crash. The finding points at the code or its owner instead;
  • the runtime ran out of its own heap setting (-Xmx, --max-old-space-size, PHP’s memory_limit) and the kernel killed nothing: more container memory changes nothing. Raise the setting in a release;
  • the container has no memory limit at all: the node ran out, not the container.

With no sizing evidence, the class takes a step instead: ×1.25, then ×1.5, then ×2 of the same limit, each a separate fix that is verified and undone if it does not hold, and each larger than every step that did not hold that week. After ×2 it stops and asks for evidence. After a memory change is undone, Nightshift looks again at once and files the next change once - and that change always asks a person.

The card says how the limit was sized: How the memory limit was sized:, with one of sized from what the container declares, sized from its crash log, sized from the same image elsewhere, a ladder step - no sizing evidence, or sized by the model, held to the guard’s range.

Operate → Automation → Memory limits sized from evidence (admin):

Control Default What it does
Maximum (Mi) 2048 when empty The largest limit Nightshift sizes, asked or not. Required before Apply them on their own can be saved
Largest increase (%) 200 An unattended change may set at most this share of today’s limit: 200 is twice it, and the most it accepts is 400, four times. A step with no sizing evidence never goes past twice; only a limit sized from evidence uses the room above. A larger change asks a person
Per day 10 Unattended memory changes per organisation per day, undone ones counted
Apply them on their own Off Lets a limit sized from evidence - by Nightshift’s own sizing, or by the model citing evidence - apply without a person, inside the bounds above

Allow a second step the same day once the first held (at most 2 a day), beside Apply them on their own, lets a second evidence-sized step run on its own on the same workload the same day, only after the first verified, and only while the day’s growth stays within Largest increase (%) of the limit before the first change and under Maximum (Mi). See a second memory step.

Off, every model-sized limit waits for a person. On, it covers memory and nothing else: a change is applied without a person only when everything it sets is one container’s memory request or limit. A change that also sets anything else - a command, arguments, an environment variable - or several containers’ memory waits for a person, with the reason: Not applied unattended: The evidence-sized memory grant covers memory values only; .... It still runs only from an automatic investigation, inside the organisation’s autonomy switch and end date, one change per workload per day, verified (no OOM kill and every pod Ready, held 15 minutes) and undone if it does not hold - and never on a workload Argo CD, Flux or Helm owns.

A pod waits in Pending because no node has the CPU or memory it requests, and the container uses less than it asks for. Nightshift proposes a smaller request only if it will schedule: it reads the room each node the pod may run on actually has - its allocatable capacity minus the requests of the pods already placed there, on nodes that are Ready, schedulable and whose taints the pod tolerates.

  • When half of today’s request fits on such a node, the request is halved, the most one step lowers it.
  • When half does not fit anywhere, it is set to the most room a node has: the smallest decrease that schedules.
  • It is never set below the container’s measured use plus 20%.

It is refused, with the reason, when even measured use plus 20% fits on no node (the cluster needs capacity, not a smaller request), when no usage was measured, or when a node’s room could not be read - a number that may not schedule is not proposed. The fix is verified by every pod being scheduled and Ready, with none restarting, for 2 minutes, and fails if a pod is OOM-killed with the smaller request.

When the scheduler says both CPU and memory are short, Nightshift works on the one the pod asks the most of, measured against the largest node. When no node can take the pod at any request of that resource, the refusal names the resource that is really full and why each node was passed over: not Ready, cordoned, not admitted by the pod’s node selector or affinity, a taint it does not tolerate, or the other resource already fully requested. For example: node-2 has its cpu fully requested (1900m of 1900m allocatable; the pod also asks 10m).

A setting or feature flag changed just before the errors

Section titled “A setting or feature flag changed just before the errors”

A feature flag turned on, or a setting in a ConfigMap changed, and errors started soon after. Nightshift puts it back, but only to the value Seylo itself recorded before the change: never a default, and never a flag’s “off” by itself.

How Seylo knows the previous value: while it builds the service map, it reads the ConfigMaps your workloads use and records, when they change, one-line settings and each flagd flag’s default variant. It never records a Secret, a file, or a value that looks like a credential. A ConfigMap holding flags is read every 5 minutes.

The fix is proposed only when:

  • exactly one setting changed, at most 30 minutes before the errors began, and after its old value was last seen;
  • the change was not made through Seylo;
  • at most 3 Deployments or StatefulSets read the ConfigMap, and with the services that hold a flag, at most 3 restart.

On approval, Seylo changes that one key (inside a flagd file, only that flag’s default variant; every other byte stays) and restarts every workload that reads the ConfigMap, the flag service included, so it loads the value.

An app that asks the flag service and keeps the answer in memory holds the old value until it restarts. For a feature flag, Seylo therefore restarts those services too, after the flag service: only the ones its evidence names, from the traces and the service map, never a guess. The workloads that read the ConfigMap and the services that hold the flag are at most 3 together; if the flag reaches more, nothing is proposed and the card names them. When Seylo cannot read which services hold the flag, only the workloads that read the ConfigMap restart, and the card says so.

It counts as working only when the alert stays quiet for 5 minutes in a row, within 20 minutes. A dip under the threshold is not enough. If it does not hold, Seylo puts the setting back as it was, restarts the same workloads again, and tells you so. That does not rule the setting out: a service that caches the value keeps it until it restarts, so the alert stays open for a person to start from its investigation. When you approved the revert with the owner acknowledged (a ConfigMap Helm, Argo CD or Flux manages), the undo runs under your same acknowledgement, so it is not refused.

A person approves the first 3 reverts of the same key in your organisation. After that it can earn autonomy like other low-risk fixes.

A Chaos Mesh NetworkChaos experiment is still cutting or slowing traffic to the failing workload, or to a workload it calls. Nightshift proposes to pause it with Chaos Mesh’s own annotation, experiment.chaos-mesh.org/pause: "true". The experiment is kept; the undo removes the annotation and it resumes.

  • Always a person’s approval. An experiment is somebody’s test. It never runs on its own and no grant covers it.
  • Two running experiments that match: a person chooses which.
  • Verified when Chaos Mesh reports the experiment paused and the workload’s pods are all Ready for 2 minutes, within 10.

The agent needs permission. The Seylo agent’s chart grants nothing on Chaos Mesh. With Chaos Mesh’s security mode on (its default), a pause is checked as an update, so patch alone is refused. To let Nightshift read and pause experiments, give the agent’s ServiceAccount (seylo-agent in the seylo-agent namespace, with the default install) these verbs:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: seylo-agent-chaos-pause
rules:
- apiGroups: ["chaos-mesh.org"]
resources: ["networkchaos"]
verbs: ["get", "list", "watch", "patch", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: seylo-agent-chaos-pause
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: seylo-agent-chaos-pause
subjects:
- kind: ServiceAccount
name: seylo-agent
namespace: seylo-agent

Narrower: the same rules as a Role, with a RoleBinding to the same ServiceAccount, in each namespace whose experiments may be paused. Chaos Mesh checks every namespace the experiment selects, so include those too. These verbs let the agent change an experiment, not create or delete one; Seylo itself only sets and removes the pause annotation.

Without them, the approved pause fails, and the card says why and gives the kubectl annotate command to pause it by hand.

A Deployment or StatefulSet, often a broker or a database, was scaled to 0 and never scaled back, and everything that calls it fails. Nightshift proposes the replica count it last ran, read from your metrics store over the last 7 days. It never picks a number: with no such count, it says what a person must decide.

Always a person’s approval, because someone may have scaled it down on purpose. Verified when every replica is Ready, none restarting, for 2 minutes, within 15. The undo scales it back to 0.

Most fixes change the workload the alert names. Some change something else: CoreDNS and metrics-server run in kube-system and serve every namespace, so restarting one is the fix for an alert in shop.

A fix that changes a namespace other than the alert’s, or any namespace whose name starts with kube-, is approved only by an organisation admin who also has release access to the namespace it changes. An application’s releaser alone cannot approve it. The same rule applies on the fix, on the Approvals page, and for a request Nightshift filed. Such a fix never runs on its own, whatever grants or earned autonomy say. If you may not approve it, the message says who may, or gives the change to make by hand.

Nightshift reads a CronJob’s schedule and its recent runs, and checks for these on its own, without waiting for a pod alert. Each fix below waits for a person. Nightshift never deletes a Job and never runs a failed Job again: it may have done part of its work before it failed, so that is the owner’s call.

Problem What Nightshift proposes Verified when
A CronJob misses its schedules (Kubernetes records MissSchedule or TooManyMissedTimes, or the schedule says runs were due and never started) Sets startingDeadlineSeconds to the schedule’s shortest interval, read from the schedule itself A scheduled run starts after the change and no missed schedule is recorded since. It waits up to the schedule’s longest interval plus 10 minutes
CronJob runs pile up (three or more runs active at once under concurrencyPolicy: Allow, the oldest running longer than the schedule’s shortest interval) Sets concurrencyPolicy to Forbid. The runs already active keep running; stopping one is a person’s call For one whole schedule interval after the change, no run starts while another is active
A Job or CronJob run keeps failing until its backoffLimit The cause, from the last failed pod. An OOM kill or a liveness kill is fixed in the CronJob’s job template, so its next run starts with it The next run, started after the change, completes

Where it refuses, it says why:

  • A suspended CronJob has nothing owed, so it is left alone.
  • Under Forbid, a long-running active run is the reason schedules are skipped, not the deadline, so no deadline changes.
  • If no deadline is set and Kubernetes has not given up counting, the misses point at the cluster’s controller or clocks, and a deadline would only skip more.
  • A one-off Job’s pod template cannot be changed after it is created, so for a Job that is not a CronJob’s run, Nightshift gives the values to use when you create it again.
  • A run that failed in your own code is named, with its exit code, and goes to the owner or the coding agent.

A quota counts what pods declare, not what they use. It adds up every pod’s requests and limits. A namespace can use 7% of its CPU and still be over its quota, because its pods declare far more than they use. And a quota charges every new pod in full, so a namespace over it starts no pod, not even a smaller one: no rollout, no restart, no CronJob run, and no fix that changes a pod.

Nightshift’s checks raise one critical alert for each namespace and quota that refuses new pods: the quota is over its hard limits, or open FailedCreate alerts name it. Its message says which quota, what is in use against its limit, which workloads are refused now, and the remedy, for example:

ResourceQuota team-budget in shop-dev is over its hard limits (limits.cpu 10500m in use of 8): every new pod that needs more of it is refused - a rollout, a restart, a CronJob run, any fix.

A namespace set below what it already runs refuses nothing until something rolls, so before this alert nothing said so.

Nightshift’s first remedy is to lower what other workloads in the namespace declare but do not use, as one fix, approved by a person every time. It is called a rebalance. It follows the same rules as right-sizing:

  • a new request is the busiest 5% of use plus headroom, never below a measured peak, and never lowered on a resource that is already at risk;
  • at most 50% off in one change;
  • the most over-declared workloads first, and as few as are needed to make the room;
  • it never changes anything in kube-system, a workload an HPA scales, a workload with no measured use, or a workload Argo CD, Flux or Helm owns (unless that owner is acknowledged).

The fix is applied in order and read back, step by step. It counts as working when the workload that was refused becomes Ready, and it is undone value by value if it does not.

A namespace that is already over its quota cannot roll even a smaller pod, so the fix’s first step raises the quota just enough for the rollouts to start, and says so. Seylo does not delete pods to make room: in a namespace over its quota a deleted pod may not come back.

Beside it the fix states the alternative: the quota’s owner raises it (each refused workload’s alert carries the exact raise), or, for Seylo’s own namespace budget, an admin raises it on Admin → Clusters → Coding agents → Raise the budget. When no room can be made, Nightshift proposes the exact raise that admits the pods still missing, which is what is used now plus those pods’ requests, and nothing more.

Always a person. A quota is the namespace owner’s budget. No grant and no number of approvals makes this run on its own: a quota an agent can set is not a quota. A rebalance changes workloads the alert is not about, so it is approved each time.

It counts as working when no new refusal names the quota and the workload has the pods it wants, for 2 minutes.

Every fix is checked against the quota first

Section titled “Every fix is checked against the quota first”

The same check runs before any fix that raises requests, limits, replicas or a claim’s storage, including one Nightshift writes with a model:

  • The change must fit what is left of every quota in the namespace, both when it settles and during the rollout, when new pods run beside the old ones.
  • A change that does not fit is refused with the numbers, and says where an admin raises Seylo’s budget or what to ask the quota’s owner.
  • If the quotas cannot be read, the fix stays for a person: no grant runs it.
  • A change that was written and then refused by a quota fails at once and says so: The change could not roll out: ResourceQuota X refused the new pods. It is not reported as a fix that “did not work”.

Nightshift needs agent 0.2.22 or later to read quotas in every posture.

Pods wait in CreateContainerConfigError because a ConfigMap, a Secret or a key they name does not exist. Nightshift names the missing object or key, and where the template refers to it.

  • A release brought the reference. When the Deployment’s own previous revision did not name it, Nightshift proposes rolling back to that revision. It counts as working when every pod is Running and Ready, and none is new in CreateContainerConfigError, for 2 minutes.
  • Otherwise, nothing is changed. You get the remedy: create the object, add the key, correct the name, or mark the reference optional: true if the app runs without it. Seylo never writes a Secret’s value; it reads only that a Secret exists and the names of its keys.

Pods wait in ImagePullBackOff and the registry answered 401 or 403. Nightshift reads which pull secret the pod template and the ServiceAccount name, and whether each exists and is a registry credential, never its value. It tells you which of these is wrong: nothing names a pull secret, a named secret is missing, a named secret is not a registry credential, or the credential exists and the registry still refuses it.

Diagnosis only. Seylo never writes registry credentials: you renew the credential and update the Secret. Docker Hub gives the same answer for a repository that does not exist, and the card says so when that is possible.

A budget allows no disruption while every pod is healthy, so a node drain, an upgrade or the cluster autoscaler removing a node is refused for good. Nightshift checks for this on its own and proposes one more replica on the Deployment, when that is enough for the budget to allow one disruption. It counts as working when the budget allows at least one and every replica is Ready for 2 minutes.

It proposes nothing, and says why, when:

  • the budget sets maxUnavailable: 0, or still allows none with one more replica: relaxing the budget is its owner’s decision;
  • an HPA sets the workload’s replicas, or it is a StatefulSet, or its pods mount a volume claim;
  • one more replica is above a replica ceiling an admin set.

These add-ons run in kube-system and serve every namespace, so a fix here follows the rule for changes outside the app’s namespace. Seylo restarts them and never edits their configuration, never the Corefile. A managed cluster owns these add-ons, so where the platform would put a change back, Nightshift refuses and says so.

CoreDNS is unhealthy. Raised when lookups fail in two or more workloads while CoreDNS is unhealthy, or by Nightshift’s own check of CoreDNS. Nightshift proposes:

  • a rolling restart, only after reading CoreDNS’s own log and finding no configuration error: a restart would load the same bad configuration;
  • or one more replica, when every pod is Ready but CPU-throttled and no autoscaler owns the count, never more than one per node.

It refuses when CoreDNS fails on its configuration, is killed for memory, is scaled to zero, or is not a Deployment named coredns or kube-dns. It counts as working when every replica is Ready and none restarted for 5 minutes.

metrics-server is down. An HPA cannot scale because the metrics API did not answer. Nightshift proposes restarting metrics-server when it is unhealthy. It refuses when metrics-server is scaled to zero, is killed for memory, or runs Ready yet the API server cannot reach it (check the v1beta1.metrics.k8s.io APIService). It counts as working when the HPA reports ScalingActive=True and a current metric for 2 minutes.

Nightshift checks cert-manager’s Certificates on its own. A Certificate that is not Ready, has expired, or is more than an hour past its renewal time gets a diagnosis: its request was denied or failed, the order or the challenge is stuck, the issuer is not ready, or an earlier issuance failed and cert-manager is waiting before trying again. You get the remedy for that cause.

Diagnosis only. Seylo does not trigger the renewal: cert-manager’s trigger writes the Certificate’s status, which the agent’s role does not grant. It needs agent 0.2.22 to read the Certificates; on an older agent the card says the read was refused.

An init container crashes or exits non-zero, so the app containers never run.

  • OOM-killed: Nightshift raises its memory limit one step, as it does for a container, within the memory ceiling. If it does not hold it is undone and the next step is proposed. It counts as working when no init container is OOM-killed again and every pod is Ready for 10 minutes.
  • Any other exit: diagnosed from the log signatures, with nothing changed. Seylo stores no log text.
  • An image that cannot be pulled is handled by the image fixes.

A node’s kubelet reports disk pressure and evicts pods. Nightshift reads the kubelet’s eviction messages and names what fills the node: the largest writer of ephemeral storage, or images. For a writer it says to give that container an ephemeral-storage limit or move what it writes to a volume.

It proposes a cordon only when the kubelet’s image clean-up is failing and at least 2 pods were evicted for disk, because the kubelet already keeps new pods off a node under disk pressure and frees space itself until then. The cordon evicts nothing and is approved by an admin; it is not proposed while the platform is repairing the node. It counts as working when no pod of the workload is evicted from that node again for 10 minutes.

When no class above matches, Nightshift’s model may propose one change of its own: a patch to one object in the namespace, or one new object. Each field must cite the evidence it came from. A guard on Seylo’s side accepts it only when all five checks hold:

Check What it requires
1. Allowed fields Only the fields in the table below
2. Evidence for every field Each value rests on a fact Seylo built from what it read: the same image’s probes elsewhere (only where those pods are Ready), a path seen answering 2xx in logs or traces, the replica range over 7 days, measured usage, the ConfigMap’s keys, the API server’s own 403 in a log line, a value your organisation’s skill names
3. No worse posture The change adds no new finding from the policies on Admin → Posture to the object
4. Dry run The cluster accepts exactly what would be applied, in a server-side dry run
5. Verification and undo A check from a fixed list (rollout healthy, pods Ready, endpoints ready, autoscaler active, budget active, symptom gone), and the exact undo: the inverse patch, or deleting what was created by its UID

What a generated fix may change:

Kind Fields May create
Deployment, StatefulSet A container’s readinessProbe, livenessProbe (HTTP only), startupProbe, resources (CPU and memory requests and limits), env (adding only entries read from an existing ConfigMap key), command and args (only when the evidence shows they are why it crashes); the pod’s nodeSelector and tolerations (only what a Ready node carries) No
Service spec.selector (labels its running pods carry), a port’s targetPort (a port its pods declare) No
HorizontalPodAutoscaler What it scales (new ones only), minReplicas, maxReplicas (from measured load, never above the admin’s ceiling), CPU or memory utilisation between 50% and 90% Yes
PodDisruptionBudget The workload’s own selector (new ones only), maxUnavailable: 1, for two or more replicas Yes
Role One rule, exactly the verb, resource and API group a 403 in the workload’s logs names, on the one Role bound to the workload’s own ServiceAccount. Every existing rule stays as it is No
ConfigMap One existing key of a ConfigMap the workload reads, set to the value your organisation’s skill names. Only a one-line setting, never one that looks like a credential. Every workload that reads it (at most 3) restarts, and again on undo No

A Role rule is never a ClusterRole, a wildcard, or a permission on Roles, bindings, Secrets or ServiceAccounts; never exec, attach, portforward or proxy; never escalate, bind or impersonate. It is approved by an admin, and the agent needs its RBAC write tier on that cluster (rbac.write.rbac.enabled=true).

Never, whatever the evidence: Secrets, RBAC beyond the one Role rule above, ServiceAccounts, Namespaces, CustomResourceDefinitions, Nodes, PersistentVolumes and claims, StorageClasses, webhooks, NetworkPolicies, ConfigMap keys no skill names, quotas and LimitRanges; and on an allowed kind, the image, securityContext, envFrom, volumes, host networking, the ServiceAccount, init containers, replicas and metadata. A probe is never removed and never an exec probe. DaemonSets are not on the list. Nightshift → Settings → Generated fixes shows the same list, read-only.

How a change is written, so it fits the object it changes:

  • A probe keeps its handler. A probe pointed at the wrong port is fixed by correcting the port of the handler it has - an HTTP probe stays an HTTP probe, on its own path - rather than by swapping it for another kind.
  • A container is named, never counted. A change to the second container is made to that container by its name, so it cannot land on another one if the order changes.
  • Only what changes is sent. A field restated at its current value is left out, and said so, instead of refusing the whole change.

Narrow it per workload with an annotation that names the rules it allows:

metadata:
annotations:
seylo.in/nightshift-allowed-paths: "readinessProbe,resources"

The rule names are readinessProbe, livenessProbe, startupProbe, resources, env, command, args, nodeSelector, tolerations, selector, targetPort, hpa and pdb. For a new autoscaler or budget, put it on the workload. An empty annotation allows nothing; a name Seylo does not know is ignored, and said so. The annotation can only narrow the list.

A refused change goes back to the model once with every reason, so it can correct it; then it is filed, or the investigation records why not. The panel says which, for example A generated change was proposed and refused by the guard after 2 attempt(s): ....

A generated fix waits for a person, with four exceptions, each only when an admin turned it on: a memory limit under the memory grant; a probe path copied from a passing probe of the same image, after 3 approved (probe paths); a raised CPU request (a grant); and a new HPA (a grant). There is at most one open generated fix per workload. A workload Argo CD, Flux or Helm owns needs its owner acknowledged; for a new autoscaler or budget, the owner of the workload it covers. Before it runs, Seylo re-checks that the fields still hold the values the change was computed from, and that a new object’s name is still free.

The undo of a created object deletes it by its UID, so an object someone else has since made under the same name is never deleted. With agent 0.2.18 the API server itself enforces the UID; with older agents Seylo compares the UID on a fresh read just before the delete. No UID read back after the create means no undo, and the card says so.

Nightshift can also create what a workload lacks, through the same guard:

What From
A readiness probe The same image’s probe elsewhere; the load balancer’s health path; a route seen answering 2xx; else TCP on the port the Service targets. A Dockerfile HEALTHCHECK is not visible to Seylo, and the fix says so
A liveness probe Only on an HTTP path with evidence, and only for a workload that starts in 15 seconds or less - otherwise a startup probe first
A startup probe The slowest observed start to Ready, × 1.5, plus 10 seconds
An autoscaler 70% utilisation; the minimum from the replicas it runs; the maximum twice what the measured load needs, never above the admin’s ceiling. Refused without requests
A PodDisruptionBudget maxUnavailable: 1 with the workload’s own selector
Requests The 95th percentile of measured usage, rounded up

Admin → Posture → Production-readiness campaign (releaser) does this for a whole namespace. Enter the namespace and press Read the namespace: Seylo reads its workloads’ Posture gaps (no readiness probe, no PodDisruptionBudget, no requests, no liveness probe) and lists one change per workload, each already through the five checks and a dry run, with its evidence, how it is verified and its undo. A campaign reads 15 workloads at a time and names the ones it did not read.

Nothing is applied from the list. You file each change on its own; each becomes its own approval, is verified on its own, and is undone exactly if it does not hold. There is deliberately no “file all”: twenty workloads restarting at once to gain probes is an outage of its own.

Some failures need the damage limited before the cause is fixed. When Nightshift’s answer names a mitigation Seylo can compose, Seylo builds it - the model never writes it - checks it with its own guard and a server-side dry run, and files it as a fix that always waits for a person. No grant covers a mitigation.

Mitigation Only when The change Undone
Capacity buffer Pods are Pending for lack of CPU or memory, a cluster autoscaler or Karpenter is present and not saying it cannot grow, and the pod fits a node Pause pods in the seylo-capacity namespace, at most 3, sized like the largest pending pod, at a priority below every workload, so the autoscaler adds a node By deleting what was created
Rate limit An Ingress served by ingress-nginx receives more than the highest rate it served in the 24 hours before the alert nginx.ingress.kubernetes.io/limit-rps at that rate, rounded up, per client IP By removing the annotation
Maintenance page Every backend of the Ingress has no ready endpoint, and a Service in the namespace serves your maintenance page with a ready endpoint nginx.ingress.kubernetes.io/default-backend pointing at that Service By removing the annotation, automatically when the backend is ready again
Scale down A non-critical Deployment or StatefulSet - not the alerting one - takes capacity the failing path needs, such as a shared database’s connections or a node’s CPU Fewer replicas, in one step: never fewer than half, never 0, never below its PodDisruptionBudget. Refused for a workload with an autoscaler, which would scale it straight back Always: the replicas are restored after an hour. Scale it yourself to keep it lower

With another ingress controller, or Cloudflare or Azure Front Door in front, the answer gives the setting as a step for a person.

The capacity buffer needs two objects a person creates once, because Seylo never creates a cluster-wide object:

apiVersion: v1
kind: Namespace
metadata:
name: seylo-capacity
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: seylo-capacity-buffer
value: -1
preemptionPolicy: Never
globalDefault: false

Seylo reads PriorityClasses to check them before it proposes a buffer. That read is in agent 0.2.18 on a cluster-wide install; without it, the buffer is refused as “could not be read”.

A workload that Argo CD or Flux deploys from Git cannot be fixed durably in the cluster: the reconciler puts the old value back at its next sync. For such a workload, Nightshift can send the fix to the owner’s repository as a pull request instead. Today this is how an incident’s fix on its root and an APM coverage annotation reach a workload Git owns. A fix on a single alert for such a workload asks you to acknowledge the owner, as before.

It is off by default. An admin turns on Let Seylo open pull requests on Admin → Integrations → GitHub; Nightshift → Settings → Pull requests shows whether it is on. It is separate from the skills repository, which stays read-only. Seylo’s GitHub App must be installed on the repository with contents: write and pull requests: write. Each pull request uses a token for that one repository, made when it is needed and never stored.

How it works:

  1. Seylo finds the source: an Argo CD Application, or a Flux Kustomization and its GitRepository, that deploys plain manifests from a GitHub repository. It finds the one file under that path that holds the workload.
  2. It writes a one-line edit and checks it: the file must hold the value the cluster has now, and the edited file must differ in exactly that field.
  3. Approve and open the pull request opens it, with a body that carries no secret. The fix shows pull request open.
  4. Seylo never merges it. Your review and merge is the decision. The fix is verified once the change is merged and seen live in the cluster, in the same way as a change made in the cluster.
  5. If it does not verify, the undo is a revert pull request, proposed for a person to open from the fix card.

It is refused, with the reason, when the source is not on GitHub, renders a Helm chart (an Argo CD chart or a Flux HelmRelease: the value lives in a values file Seylo will not map back), mixes a chart with manifests, has more than one Git source with a path, or is pinned to a revision so that a pull request to a branch would never be deployed. When the door is not ready, the card lists What is missing for this door - each item addressed to whoever can do it - and offers the alternative: the change in the cluster, with the owner acknowledged on approval.

If the problem behind a fix goes away before anyone approves it - for example, your cloud repairs a node - the fix shows No longer needed with the reason, its approval request is withdrawn, and nothing is changed.

Seylo always re-checks a fix just before it runs, including when someone approves it, and refuses one that is no longer needed. A fix approved after its problem went away therefore changes nothing, and says so.

The same happens when the object a fix would change has changed since the fix was proposed - someone edited or scaled it, or it was deleted and created again. The fix was worked out from the old object, so applying it could put old values back; it is refused with the fields that moved, and nothing is changed. If the problem is still there, Nightshift proposes a fresh fix from the object as it is now.

A fix waiting for approval also expires after 72 hours, like any other request.

A node that has been NotReady for one minute, by its own Ready condition, raises an alert. A node that flaps briefly, for example while its container runtime restarts, does not: the minute starts again each time it comes back. Reading nodes needs an agent installed cluster-wide; a namespace-scoped install cannot read them.

Most clouds repair a lost node themselves, so Nightshift does not race your platform. The alert says what is wrong, what Kubernetes did with the node’s pods, and what the platform is doing about it, from what can be seen inside the cluster:

Platform What the alert says
AKS That AKS node auto-repair is repairing the node, which step started when, and whether it finished or failed. Until the first repair event appears, that AKS has not started yet and Seylo is waiting for it
AKS, with no repair after 15 minutes, or a shut-down node That AKS is not repairing it
Karpenter, EKS Auto Mode That the node is marked for replacement, or that its repair is blocked, with the reason
Cluster autoscaler That the autoscaler is removing the node
EKS managed node groups, GKE That the repair, if your platform does one, leaves nothing inside the cluster, so Seylo cannot see it. For GKE, gcloud container operations list shows it

Each change in the repair’s state is added to the alert’s timeline.

While the platform repairs the node, Nightshift does not cordon it or force-delete pods stuck terminating on it. The fix shows the platform’s repair as its reason. It is proposed again when the platform reports that the repair failed, when AKS has shown no repair after 15 minutes, or when the repair has run for more than an hour. A cordon proposed before the repair started shows No longer needed. On a node that no platform repairs, a cordon is proposed once the node has been NotReady for 5 minutes.

The alert closes only after Nightshift has verified the recovery: the node has been Ready for 2 minutes (or is gone, or is cordoned with its pods moved elsewhere), and every workload that had pods on it has all its replicas Ready and none Pending. The resolution lists what was checked, for example Verified: shop/orders-api 2/2 Ready, no Pending pod. If a workload is still unhealthy 15 minutes after the node is back, the timeline says so and the alert stays open.

Seylo reads a node’s events from the default namespace, where Kubernetes records them. On AKS, if your agent’s namespace list leaves out default, the alert says that Seylo could not read the node’s events and so cannot tell whether AKS is repairing it.