What Nightshift may do on its own
By default, every change Nightshift proposes waits for
a person. That does not change unless an admin decides it should, one kind
of fix at a time. This page lists every fix Nightshift can apply on its own,
the bound each one is held to, and how to turn it on.
Every control on this page is on Operate → Automation, under Nightshift fixes it may apply without asking. Everything Nightshift changed on its own, and every refusal, is in that page’s activity log.
Three ways a fix runs on its own
Section titled “Three ways a fix runs on its own”| Way | Where on Automation | Who |
|---|---|---|
| Earned autonomy | Fixes Nightshift may apply. A kind of fix that has been approved and verified enough times in your organisation runs on its own while Let proven fixes run on their own is on, until its end date. See fixes, class by class | An admin turns the switch on |
| A box under that table | Memory limits sized from evidence, nearly-full volumes, claims that never bound, and probe paths. Each runs on its own only when its own box is ticked | An admin ticks the box and saves |
| A grant | Grants on cluster → New grant, whose Nightshift actions section ticks the action, for a cluster and usually one namespace | An admin requests it, an approver signs it |
Whichever way, the same envelope holds:
- never on a cluster where Nightshift is switched off;
- never on a workload Argo CD, Flux or Helm owns, and never when who owns it could not be read;
- at most one change per workload per day, except a second memory step;
- every change is read back and verified, and undone when it has an exact inverse and does not hold;
- Pause all automation on Automation stops all of it at once;
- every decision, refusals included, is in Automation’s activity log.
A fix that is not covered says why under it, for example
Not applied unattended: No standing grant covers shop/checkout-api, so this needs a person.
The fixes at a glance
Section titled “The fixes at a glance”| Fix | Turned on by | Its bound |
|---|---|---|
| Roll back a crash or stuck rollout after a release | Grant | Only to the release Seylo saw run healthy, and only within 30 minutes of the bad release |
| Roll back an image that never started | Grant | Only when the new image cannot be pulled or parsed |
| Raise a CPU request | Grant | Measured use + 20%, at most 4 times today’s, never above your CPU ceiling |
| Add an HPA | Grant | Only where load splits across pods; minimum = today’s pods, maximum within your replica ceiling |
| A probe path | Box: Probe paths | After 3 approved here, and only a path copied from a passing probe of the same image |
| A memory limit sized from evidence | Box: memory | Your maximum, and at most four times today’s limit |
| A second memory step the same day | Box: memory | Only after the first held; at most 2 a day per workload |
| Grow a nearly full volume | Box: volumes | Your largest step and largest size, and only with a priced cost |
| Recreate a claim that never bound | Box: claims | Only a claim proven never to have bound |
| Restart after a Key Vault secret rotated | Grant | Only when the pods hold an older version than the vault’s current one |
| One-fix grants | Grant | Each covers one fix, inside the bound its label states |
Roll back a crash or stuck rollout after a release
Section titled “Roll back a crash or stuck rollout after a release”Pods crash-loop, or a rollout stalls, soon after a release. The new image did run, so going back undoes what the release changed. Nightshift may do that on its own under a grant that ticks Roll back a crash or stuck rollout to the last release Seylo saw run healthy (within 30 min of the release), only when every one of these is read and true:
- the change just before the failure is a release Seylo made to this workload;
- the failure began within 30 minutes of that release. Later than that, something else may have broken it;
- the operation just before the bad release, on this workload, is also a release, and Seylo’s record says it succeeded: it was applied and its rollout reached healthy. Any other change in between, such as an environment edit or a resource patch, means the revision it would go back to is not that release;
- Seylo’s history of the workload holds that release.
It goes back to Seylo’s recorded release only, never to a revision Seylo did not record. If any of it is not proven, the fix says which part, and a person approves.
A grant that ticks Roll back to the previous revision (any reason) covers this too, as before, without these conditions: it is the wider of the two. Earned autonomy for rollbacks is unchanged.
Roll back an image that never started
Section titled “Roll back an image that never started”When a release’s new image cannot start - its tag does not exist
(ImagePullBackOff, ErrImagePull with “not found”), or its reference cannot
be parsed (InvalidImageName) - none of the release’s pods ever ran, so going
back undoes nothing the release did.
A grant that ticks Roll back when the new image cannot start (pull fails or name invalid) lets Nightshift roll back on its own, in that case only:
- to Seylo’s recorded previous release, when Seylo’s history is the workload’s history;
- or to the Deployment’s own previous revision (what
kubectl rollout undodoes), only when the cluster proves it: no pod of the new ReplicaSet is Ready, and a pod of the previous one is Ready now. Anything unread, or either half false, and a person approves.
Roll back to the previous revision (any reason) also covers this case. Both can also run on their own once they have earned autonomy, under the same proof.
Raise a CPU request
Section titled “Raise a CPU request”A container uses more CPU than it requests, and Nightshift’s scaling judgement says a bigger request, not more pods, is the fix. A grant that ticks Raise a CPU request to measured use + 20% (at most 4x, never above the CPU ceiling you set) lets Nightshift raise it on its own:
- one container’s CPU request, raised, nothing else;
- to at most its measured 95th percentile plus 20%;
- to at most 4 times today’s request;
- never above the grant’s CPU request ceiling, in millicores. The form requires it when you tick the action: a grant without one raises nothing;
- only where the scaling judgement says Request or Both.
A container with no CPU request at all is a different fix: Set a missing CPU or memory request in the one-fix grants.
HPAs at their maximum
Section titled “HPAs at their maximum”When an HPA sits at its maxReplicas and is still above target, Nightshift
raises the maximum one 50% step. Operate → Automation → Fixes Nightshift
may apply → HPAs at their maximum has one field, Highest maxReplicas
(a whole number from 1 to 100, admin only, saved with the card’s Save):
- every
maxReplicasraise Nightshift proposes stays at or under it; - a grant for raising an HPA’s maximum acts only inside it. Leave it empty and such a grant raises nothing, and any other raise stops at 100;
- an HPA Nightshift creates takes the tighter of it and the grant’s Most replicas.
Add an HPA
Section titled “Add an HPA”Nightshift’s scaling judgement says the load splits across pods, so an autoscaler is the fix. A grant that ticks Add an HPA where load splits across pods (min = today’s pods, max within your replica ceiling) lets Nightshift create one HorizontalPodAutoscaler on its own:
- only where the scaling judgement says Autoscaler and the change cites it;
- for the alert’s own Deployment or StatefulSet;
- minReplicas is the number of pods running now, so it never scales below today;
- maxReplicas never above the grant’s Most replicas. The form requires it when you tick the action, and Seylo refuses a grant without it, so the number that bounds the HPA is on what the approver signs. If the hpa-at-max replica ceiling (Highest maxReplicas on Automation, see HPAs at their maximum) is also set, the tighter of the two wins;
- one CPU utilisation target, between 50% and 90%.
It is verified as any new HPA is (see what it does with the answer), and its undo deletes the HPA it created.
A probe path
Section titled “A probe path”A readiness, liveness or startup probe checks a path that does not answer. Nightshift’s fix copies the path from a probe of the same image that passes elsewhere. It is a generated fix, so by default a person approves it.
Probe paths, with Fix them on their own after 3 approved ticked, lets it run on its own once 3 probe fixes here were approved by a person and each verified. A fix that ran on its own never counts towards the 3. Even then:
- the change may set only a container’s probes on a Deployment or StatefulSet;
- every path it writes must be copied from a probe of the same image whose pods are Ready. A path seen only in a log line, or a load balancer’s health path, is good enough for a person to approve, not to run on its own;
- it needs Let proven fixes run on their own on, and stops at its Until date.
It is undone if it does not hold.
Memory up to four times
Section titled “Memory up to four times”When a container is OOMKilled, Nightshift sizes the new limit from
evidence: what the container declares (a JVM -Xmx, GOMEMLIMIT and
similar), an allocation in its crash log, or the same image running healthy
elsewhere. How it sizes is on
memory limits sized from evidence.
Memory limits sized from evidence, with Apply them on their own ticked, lets such a limit apply without a person:
| Control | Default | Bound |
|---|---|---|
| Maximum (Mi) | 2048 when empty | The largest limit Nightshift sets, asked or not. Required before the box can be saved |
| Largest increase (%) | 200 | At most 400: four times today’s limit |
| Per day | 10 | Unattended memory changes per organisation per day |
- Only evidence goes past twice. With no sizing evidence, Nightshift takes a step instead, and a step never more than doubles the limit. A limit taken from evidence may reach four times today’s, when Largest increase allows it.
- Never on a leak. Memory that rises across restarts is refused: more memory only delays the next kill.
- Verified by no OOM kill and every pod Ready for 15 minutes, and undone exactly if it does not hold.
A second memory step the same day
Section titled “A second memory step the same day”Sometimes the first limit holds, and the workload is killed again later the same day at the new limit. Ticking Allow a second step the same day once the first held (at most 2 a day), beside Apply them on their own, lets one more step run on its own, only when:
- every unattended change on that workload in the last day was a memory change, and each verified. One still being checked, or undone, keeps the second step for a person;
- it is the second that day for that workload, never a third;
- the second limit is sized from evidence;
- the day’s growth stays within Largest increase (%) of the limit before the first change, so two steps never go further than one step may;
- it stays under Maximum (Mi).
Only this box allows a second change a day. Every other fix, and memory under earned autonomy or a grant, stays at one change per workload per day.
Grow a nearly full volume
Section titled “Grow a nearly full volume”When a volume is more than 85% full and its StorageClass allows expansion, Nightshift grows the claim by one step. A larger disk costs money from then on and can never shrink back, so this has its own box: Nearly-full volumes.
| Control | Required | What it bounds |
|---|---|---|
| Grow them on their own | - | Off by default |
| Largest step (%) | Yes, 1 to 100 | How much one growth may add |
| Largest size (Gi) | Yes | The largest the volume may ever reach on its own |
Both bounds are required, because growth alone compounds: one 50% step a day is more than eleven times the size in a week, and only a size cap bounds what the disk can ever cost. A growth that would land past either bound, even by rounding up, is refused, not trimmed.
Only when its cost is priced from a source. Every proposed growth states
what the added capacity costs a month: the difference between the two disk
sizes as your cloud bills them, from your
own bill or the cloud’s public list
price, naming where the figure came from. When it cannot be priced, the fix
says unpriced, because ... and waits for a person, whatever the box
says. Seylo never estimates a disk price from a similar one.
A volume alert names a claim, not a workload, so Nightshift reads who owns the claim itself. If that cannot be read, the fix says so and a person approves: an owner Seylo cannot see could put the claim back.
It is verified by reading the new size back. It is never undone: a volume cannot shrink. Reading whether a StorageClass allows expansion needs a cluster-wide agent; see RBAC posture.
Recreate a claim that never bound
Section titled “Recreate a claim that never bound”A pod waits in Pending because its PersistentVolumeClaim names a
StorageClass that does not exist. The fix is to recreate the claim under the
same name with the cluster’s default StorageClass; how that works is on
a claim whose StorageClass does not exist.
Claims that never bound, with Recreate them on their own ticked, lets it run without a person only for a claim proven never to have bound. Every one of these must be read and true:
- the claim is
Pendingand names no volume; - no PersistentVolume names the claim (more than 60 PersistentVolumes in the cluster is “not proven”);
- no pod using it is scheduled or running now;
- over the claim’s whole life as your metrics store saw it, from within 5
minutes of its creation and at most 7 days back, it was never
Bound, no pod with it ever ran, and no node ever reported its volume.
A claim that never bound has no volume, so there is no data to lose. Anything Nightshift could not read is “not proven”, and the fix then waits for a person who types the claim’s name, naming the condition. A claim that is or ever was bound is never recreated on its own.
The claim is read again just before the delete, and the delete is refused if
it has bound since. The agent’s destructive tier is still needed on that
cluster (rbac.write.destructive.enabled=true).
Its undo is exact: delete the new claim and create the original again. The undo refuses to delete a claim that has bound or names a volume, so it never removes a claim that may hold data written since; that case is reported to a person.
Restart after a Key Vault secret rotated
Section titled “Restart after a Key Vault secret rotated”Pods that mount a secret through the Secrets Store CSI driver keep the version they mounted until they restart. When the vault’s current version is newer, the fix is a rolling restart. See Nightshift and your cloud.
A grant that ticks Restart a workload whose pods mount an older version of a Key Vault secret than the vault’s current one lets it run on its own. Just before the restart, Nightshift checks again that:
- the pods mount an older version than the vault’s current one, read from the CSI driver and the vault’s metadata, never the value;
- the vault’s current version is enabled and not expired, so new pods would mount it;
- the workload is the alert’s own Deployment or StatefulSet, and no reconciler owns it.
It needs the Azure subscription connected as a cloud account. A restart has no inverse: a new pod still holding an old version, or pods that are not Ready, is reported to a person.
One-fix grants
Section titled “One-fix grants”Each of these names covers one Nightshift fix and nothing else, held to the bound its label states. Before applying, Nightshift recomputes the change from what it read, so a grant signed for one kind of change never carries another. Tick them in the grant’s Nightshift actions section.
| Action, as the form names it | What it changes |
|---|---|
| Let pods pull their image: imagePullPolicy Never to IfNotPresent | Each container that could not start, that field only |
| Give a slow-starting app longer before its liveness probe (one step, at most 600s) | livenessProbe.initialDelaySeconds, raised by one step, never past 600 seconds |
| Point a readiness probe at the container’s only declared port | The probe’s port, only when the container declares exactly one |
| Fix one wrong selector value of the Service named after a workload (or of one it calls), to its pods’ own label | One existing selector value, set to the pods’ own label |
| Point a Service a failing workload calls at the one port its pods declare (its targetPort, one port) | One port’s targetPort, on the Service the failing workload calls |
| Set a missing CPU or memory request from measured usage (p95 + 20%) so an HPA can scale | One missing request |
| Raise an HPA’s maxReplicas one 50% step, within the ceiling an admin sets (Fixes Nightshift may apply: HPAs at their maximum) | maxReplicas, one step, never above 100, and never above the Highest maxReplicas an admin sets on the Automation page (HPAs at their maximum). Without that number, the grant does not cover it |
| Lower revisionHistoryLimit to the Kubernetes default (10) | revisionHistoryLimit, only downwards to 10 |
| Set a refused request or limit to the namespace LimitRange’s own bound | The refused value, to the LimitRange’s bound |
| Lower a request no node has room for, to one a Ready node fits, never below measured use (p95) + 20% | One request. See a request no node has room for |
| Drop a nodeSelector label that no node in the cluster has at all (only when a Ready node can then run the pods) | One nodeSelector entry |
A grant on one workload covers changes to that workload. A Service one hop away belongs to the workload it selects, so it needs a grant on its namespace or on that workload.
Always a person
Section titled “Always a person”Whatever is allowed, these never run on their own:
- a workload Argo CD, Flux or Helm owns, or one whose owner could not be read. For Git-owned workloads, Nightshift can open a pull request instead, which a person merges;
- a rollback to a revision Seylo did not record, except an image that never started, and a rollback proposed for an incident;
- growing a volume past your bounds, or when its cost is unpriced;
- recreating a claim that is, or ever was, bound, or that is not proven never to have bound;
- an HPA where the scaling judgement does not say Autoscaler, or above the grant’s Most replicas;
- raising a quota or a namespace’s budget;
- a mitigation;
- a generated fix other than a memory limit, a CPU request, a new HPA or a probe path under the rules above. That includes a rule added to a Role, a ConfigMap key set, a toleration and a disruption budget;
- changing a container’s command;
- pausing a chaos experiment;
- scaling a workload back up from zero: someone may have scaled it down on purpose;
- restarting the same workload a second time;
- a pull request, and a coding-agent task;
- any change in your cloud, and any change to a front proxy on another cluster: Nightshift gives the exact command or patch, and a person applies it.
Turn one on
Section titled “Turn one on”A box under the table (memory, volumes, claims, probe paths):
- Open Operate → Automation as an
admin. - On Fixes Nightshift may apply, turn on Let proven fixes run on their own and choose an Until date, at most 90 days away. Nothing on this card runs on its own while this is off.
- Tick the fix’s box (Apply them on their own, Allow a second step the same day once the first held, Grow them on their own, Recreate them on their own or Fix them on their own after 3 approved) and fill in its bounds.
- Press Save. Seylo refuses a setting without the bounds it needs, and says which.
A grant (rollbacks, the CPU request, the HPA, the Key Vault restart, one-fix grants):
- Open Operate → Automation, pick the cluster, and choose New grant
(
admin). Name a namespace unless the action needs the whole cluster. - In Nightshift actions, tick each action to grant. There is no wildcard, and a grant signed before an action existed does not cover it. Ticking Raise a CPU request … requires a CPU request ceiling; ticking Add an HPA … requires Most replicas.
- Set the hours, rate limit and expiry, and choose Request approval. The
form lists every problem at once, for example
Raise a CPU request ...: set the CPU request ceiling (millicores); without one it does nothing. - An
approversigns it under Approvals, reading each action’s full label and its ceiling.
The whole form is described on grants. Nightshift → Autonomy lists the grants that name a Nightshift action, read-only, across every cluster, with Manage on Automation.