Skip to content

What Nightshift may do on its own

By default, every change Nightshift proposes waits for a person. That does not change unless an admin decides it should, one kind of fix at a time. This page lists every fix Nightshift can apply on its own, the bound each one is held to, and how to turn it on.

Every control on this page is on Operate → Automation, under Nightshift fixes it may apply without asking. Everything Nightshift changed on its own, and every refusal, is in that page’s activity log.

Way Where on Automation Who
Earned autonomy Fixes Nightshift may apply. A kind of fix that has been approved and verified enough times in your organisation runs on its own while Let proven fixes run on their own is on, until its end date. See fixes, class by class An admin turns the switch on
A box under that table Memory limits sized from evidence, nearly-full volumes, claims that never bound, and probe paths. Each runs on its own only when its own box is ticked An admin ticks the box and saves
A grant Grants on cluster → New grant, whose Nightshift actions section ticks the action, for a cluster and usually one namespace An admin requests it, an approver signs it

Whichever way, the same envelope holds:

  • never on a cluster where Nightshift is switched off;
  • never on a workload Argo CD, Flux or Helm owns, and never when who owns it could not be read;
  • at most one change per workload per day, except a second memory step;
  • every change is read back and verified, and undone when it has an exact inverse and does not hold;
  • Pause all automation on Automation stops all of it at once;
  • every decision, refusals included, is in Automation’s activity log.

A fix that is not covered says why under it, for example Not applied unattended: No standing grant covers shop/checkout-api, so this needs a person.

Fix Turned on by Its bound
Roll back a crash or stuck rollout after a release Grant Only to the release Seylo saw run healthy, and only within 30 minutes of the bad release
Roll back an image that never started Grant Only when the new image cannot be pulled or parsed
Raise a CPU request Grant Measured use + 20%, at most 4 times today’s, never above your CPU ceiling
Add an HPA Grant Only where load splits across pods; minimum = today’s pods, maximum within your replica ceiling
A probe path Box: Probe paths After 3 approved here, and only a path copied from a passing probe of the same image
A memory limit sized from evidence Box: memory Your maximum, and at most four times today’s limit
A second memory step the same day Box: memory Only after the first held; at most 2 a day per workload
Grow a nearly full volume Box: volumes Your largest step and largest size, and only with a priced cost
Recreate a claim that never bound Box: claims Only a claim proven never to have bound
Restart after a Key Vault secret rotated Grant Only when the pods hold an older version than the vault’s current one
One-fix grants Grant Each covers one fix, inside the bound its label states

Roll back a crash or stuck rollout after a release

Section titled “Roll back a crash or stuck rollout after a release”

Pods crash-loop, or a rollout stalls, soon after a release. The new image did run, so going back undoes what the release changed. Nightshift may do that on its own under a grant that ticks Roll back a crash or stuck rollout to the last release Seylo saw run healthy (within 30 min of the release), only when every one of these is read and true:

  • the change just before the failure is a release Seylo made to this workload;
  • the failure began within 30 minutes of that release. Later than that, something else may have broken it;
  • the operation just before the bad release, on this workload, is also a release, and Seylo’s record says it succeeded: it was applied and its rollout reached healthy. Any other change in between, such as an environment edit or a resource patch, means the revision it would go back to is not that release;
  • Seylo’s history of the workload holds that release.

It goes back to Seylo’s recorded release only, never to a revision Seylo did not record. If any of it is not proven, the fix says which part, and a person approves.

A grant that ticks Roll back to the previous revision (any reason) covers this too, as before, without these conditions: it is the wider of the two. Earned autonomy for rollbacks is unchanged.

When a release’s new image cannot start - its tag does not exist (ImagePullBackOff, ErrImagePull with “not found”), or its reference cannot be parsed (InvalidImageName) - none of the release’s pods ever ran, so going back undoes nothing the release did.

A grant that ticks Roll back when the new image cannot start (pull fails or name invalid) lets Nightshift roll back on its own, in that case only:

  • to Seylo’s recorded previous release, when Seylo’s history is the workload’s history;
  • or to the Deployment’s own previous revision (what kubectl rollout undo does), only when the cluster proves it: no pod of the new ReplicaSet is Ready, and a pod of the previous one is Ready now. Anything unread, or either half false, and a person approves.

Roll back to the previous revision (any reason) also covers this case. Both can also run on their own once they have earned autonomy, under the same proof.

A container uses more CPU than it requests, and Nightshift’s scaling judgement says a bigger request, not more pods, is the fix. A grant that ticks Raise a CPU request to measured use + 20% (at most 4x, never above the CPU ceiling you set) lets Nightshift raise it on its own:

  • one container’s CPU request, raised, nothing else;
  • to at most its measured 95th percentile plus 20%;
  • to at most 4 times today’s request;
  • never above the grant’s CPU request ceiling, in millicores. The form requires it when you tick the action: a grant without one raises nothing;
  • only where the scaling judgement says Request or Both.

A container with no CPU request at all is a different fix: Set a missing CPU or memory request in the one-fix grants.

When an HPA sits at its maxReplicas and is still above target, Nightshift raises the maximum one 50% step. Operate → Automation → Fixes Nightshift may apply → HPAs at their maximum has one field, Highest maxReplicas (a whole number from 1 to 100, admin only, saved with the card’s Save):

  • every maxReplicas raise Nightshift proposes stays at or under it;
  • a grant for raising an HPA’s maximum acts only inside it. Leave it empty and such a grant raises nothing, and any other raise stops at 100;
  • an HPA Nightshift creates takes the tighter of it and the grant’s Most replicas.

Nightshift’s scaling judgement says the load splits across pods, so an autoscaler is the fix. A grant that ticks Add an HPA where load splits across pods (min = today’s pods, max within your replica ceiling) lets Nightshift create one HorizontalPodAutoscaler on its own:

  • only where the scaling judgement says Autoscaler and the change cites it;
  • for the alert’s own Deployment or StatefulSet;
  • minReplicas is the number of pods running now, so it never scales below today;
  • maxReplicas never above the grant’s Most replicas. The form requires it when you tick the action, and Seylo refuses a grant without it, so the number that bounds the HPA is on what the approver signs. If the hpa-at-max replica ceiling (Highest maxReplicas on Automation, see HPAs at their maximum) is also set, the tighter of the two wins;
  • one CPU utilisation target, between 50% and 90%.

It is verified as any new HPA is (see what it does with the answer), and its undo deletes the HPA it created.

A readiness, liveness or startup probe checks a path that does not answer. Nightshift’s fix copies the path from a probe of the same image that passes elsewhere. It is a generated fix, so by default a person approves it.

Probe paths, with Fix them on their own after 3 approved ticked, lets it run on its own once 3 probe fixes here were approved by a person and each verified. A fix that ran on its own never counts towards the 3. Even then:

  • the change may set only a container’s probes on a Deployment or StatefulSet;
  • every path it writes must be copied from a probe of the same image whose pods are Ready. A path seen only in a log line, or a load balancer’s health path, is good enough for a person to approve, not to run on its own;
  • it needs Let proven fixes run on their own on, and stops at its Until date.

It is undone if it does not hold.

When a container is OOMKilled, Nightshift sizes the new limit from evidence: what the container declares (a JVM -Xmx, GOMEMLIMIT and similar), an allocation in its crash log, or the same image running healthy elsewhere. How it sizes is on memory limits sized from evidence.

Memory limits sized from evidence, with Apply them on their own ticked, lets such a limit apply without a person:

Control Default Bound
Maximum (Mi) 2048 when empty The largest limit Nightshift sets, asked or not. Required before the box can be saved
Largest increase (%) 200 At most 400: four times today’s limit
Per day 10 Unattended memory changes per organisation per day
  • Only evidence goes past twice. With no sizing evidence, Nightshift takes a step instead, and a step never more than doubles the limit. A limit taken from evidence may reach four times today’s, when Largest increase allows it.
  • Never on a leak. Memory that rises across restarts is refused: more memory only delays the next kill.
  • Verified by no OOM kill and every pod Ready for 15 minutes, and undone exactly if it does not hold.

Sometimes the first limit holds, and the workload is killed again later the same day at the new limit. Ticking Allow a second step the same day once the first held (at most 2 a day), beside Apply them on their own, lets one more step run on its own, only when:

  • every unattended change on that workload in the last day was a memory change, and each verified. One still being checked, or undone, keeps the second step for a person;
  • it is the second that day for that workload, never a third;
  • the second limit is sized from evidence;
  • the day’s growth stays within Largest increase (%) of the limit before the first change, so two steps never go further than one step may;
  • it stays under Maximum (Mi).

Only this box allows a second change a day. Every other fix, and memory under earned autonomy or a grant, stays at one change per workload per day.

When a volume is more than 85% full and its StorageClass allows expansion, Nightshift grows the claim by one step. A larger disk costs money from then on and can never shrink back, so this has its own box: Nearly-full volumes.

Control Required What it bounds
Grow them on their own - Off by default
Largest step (%) Yes, 1 to 100 How much one growth may add
Largest size (Gi) Yes The largest the volume may ever reach on its own

Both bounds are required, because growth alone compounds: one 50% step a day is more than eleven times the size in a week, and only a size cap bounds what the disk can ever cost. A growth that would land past either bound, even by rounding up, is refused, not trimmed.

Only when its cost is priced from a source. Every proposed growth states what the added capacity costs a month: the difference between the two disk sizes as your cloud bills them, from your own bill or the cloud’s public list price, naming where the figure came from. When it cannot be priced, the fix says unpriced, because ... and waits for a person, whatever the box says. Seylo never estimates a disk price from a similar one.

A volume alert names a claim, not a workload, so Nightshift reads who owns the claim itself. If that cannot be read, the fix says so and a person approves: an owner Seylo cannot see could put the claim back.

It is verified by reading the new size back. It is never undone: a volume cannot shrink. Reading whether a StorageClass allows expansion needs a cluster-wide agent; see RBAC posture.

A pod waits in Pending because its PersistentVolumeClaim names a StorageClass that does not exist. The fix is to recreate the claim under the same name with the cluster’s default StorageClass; how that works is on a claim whose StorageClass does not exist.

Claims that never bound, with Recreate them on their own ticked, lets it run without a person only for a claim proven never to have bound. Every one of these must be read and true:

  • the claim is Pending and names no volume;
  • no PersistentVolume names the claim (more than 60 PersistentVolumes in the cluster is “not proven”);
  • no pod using it is scheduled or running now;
  • over the claim’s whole life as your metrics store saw it, from within 5 minutes of its creation and at most 7 days back, it was never Bound, no pod with it ever ran, and no node ever reported its volume.

A claim that never bound has no volume, so there is no data to lose. Anything Nightshift could not read is “not proven”, and the fix then waits for a person who types the claim’s name, naming the condition. A claim that is or ever was bound is never recreated on its own.

The claim is read again just before the delete, and the delete is refused if it has bound since. The agent’s destructive tier is still needed on that cluster (rbac.write.destructive.enabled=true).

Its undo is exact: delete the new claim and create the original again. The undo refuses to delete a claim that has bound or names a volume, so it never removes a claim that may hold data written since; that case is reported to a person.

Pods that mount a secret through the Secrets Store CSI driver keep the version they mounted until they restart. When the vault’s current version is newer, the fix is a rolling restart. See Nightshift and your cloud.

A grant that ticks Restart a workload whose pods mount an older version of a Key Vault secret than the vault’s current one lets it run on its own. Just before the restart, Nightshift checks again that:

  • the pods mount an older version than the vault’s current one, read from the CSI driver and the vault’s metadata, never the value;
  • the vault’s current version is enabled and not expired, so new pods would mount it;
  • the workload is the alert’s own Deployment or StatefulSet, and no reconciler owns it.

It needs the Azure subscription connected as a cloud account. A restart has no inverse: a new pod still holding an old version, or pods that are not Ready, is reported to a person.

Each of these names covers one Nightshift fix and nothing else, held to the bound its label states. Before applying, Nightshift recomputes the change from what it read, so a grant signed for one kind of change never carries another. Tick them in the grant’s Nightshift actions section.

Action, as the form names it What it changes
Let pods pull their image: imagePullPolicy Never to IfNotPresent Each container that could not start, that field only
Give a slow-starting app longer before its liveness probe (one step, at most 600s) livenessProbe.initialDelaySeconds, raised by one step, never past 600 seconds
Point a readiness probe at the container’s only declared port The probe’s port, only when the container declares exactly one
Fix one wrong selector value of the Service named after a workload (or of one it calls), to its pods’ own label One existing selector value, set to the pods’ own label
Point a Service a failing workload calls at the one port its pods declare (its targetPort, one port) One port’s targetPort, on the Service the failing workload calls
Set a missing CPU or memory request from measured usage (p95 + 20%) so an HPA can scale One missing request
Raise an HPA’s maxReplicas one 50% step, within the ceiling an admin sets (Fixes Nightshift may apply: HPAs at their maximum) maxReplicas, one step, never above 100, and never above the Highest maxReplicas an admin sets on the Automation page (HPAs at their maximum). Without that number, the grant does not cover it
Lower revisionHistoryLimit to the Kubernetes default (10) revisionHistoryLimit, only downwards to 10
Set a refused request or limit to the namespace LimitRange’s own bound The refused value, to the LimitRange’s bound
Lower a request no node has room for, to one a Ready node fits, never below measured use (p95) + 20% One request. See a request no node has room for
Drop a nodeSelector label that no node in the cluster has at all (only when a Ready node can then run the pods) One nodeSelector entry

A grant on one workload covers changes to that workload. A Service one hop away belongs to the workload it selects, so it needs a grant on its namespace or on that workload.

Whatever is allowed, these never run on their own:

  • a workload Argo CD, Flux or Helm owns, or one whose owner could not be read. For Git-owned workloads, Nightshift can open a pull request instead, which a person merges;
  • a rollback to a revision Seylo did not record, except an image that never started, and a rollback proposed for an incident;
  • growing a volume past your bounds, or when its cost is unpriced;
  • recreating a claim that is, or ever was, bound, or that is not proven never to have bound;
  • an HPA where the scaling judgement does not say Autoscaler, or above the grant’s Most replicas;
  • raising a quota or a namespace’s budget;
  • a mitigation;
  • a generated fix other than a memory limit, a CPU request, a new HPA or a probe path under the rules above. That includes a rule added to a Role, a ConfigMap key set, a toleration and a disruption budget;
  • changing a container’s command;
  • pausing a chaos experiment;
  • scaling a workload back up from zero: someone may have scaled it down on purpose;
  • restarting the same workload a second time;
  • a pull request, and a coding-agent task;
  • any change in your cloud, and any change to a front proxy on another cluster: Nightshift gives the exact command or patch, and a person applies it.

A box under the table (memory, volumes, claims, probe paths):

  1. Open Operate → Automation as an admin.
  2. On Fixes Nightshift may apply, turn on Let proven fixes run on their own and choose an Until date, at most 90 days away. Nothing on this card runs on its own while this is off.
  3. Tick the fix’s box (Apply them on their own, Allow a second step the same day once the first held, Grow them on their own, Recreate them on their own or Fix them on their own after 3 approved) and fill in its bounds.
  4. Press Save. Seylo refuses a setting without the bounds it needs, and says which.

A grant (rollbacks, the CPU request, the HPA, the Key Vault restart, one-fix grants):

  1. Open Operate → Automation, pick the cluster, and choose New grant (admin). Name a namespace unless the action needs the whole cluster.
  2. In Nightshift actions, tick each action to grant. There is no wildcard, and a grant signed before an action existed does not cover it. Ticking Raise a CPU request … requires a CPU request ceiling; ticking Add an HPA … requires Most replicas.
  3. Set the hours, rate limit and expiry, and choose Request approval. The form lists every problem at once, for example Raise a CPU request ...: set the CPU request ceiling (millicores); without one it does nothing.
  4. An approver signs it under Approvals, reading each action’s full label and its ceiling.

The whole form is described on grants. Nightshift → Autonomy lists the grants that name a Nightshift action, read-only, across every cluster, with Manage on Automation.