Skip to content

What Nightshift handles

Nightshift investigates every alert. What happens next depends on the problem. Some fixes can run on their own, but only after an admin allows them. Most wait for a person. Some changes Seylo never makes itself: you get the cause and the exact fix to apply.

Each of these waits for a person until the setting in the last column is on. Every setting is on Operate → Automation: a grant is requested there by an admin and signed by an approver; a box is ticked on its Fixes Nightshift may apply card. Each fix is checked after it runs. A fix runs on its own at most once per workload per day (a second memory step is the one exception), and never on a workload Argo CD, Flux or Helm owns. How to turn each on, and its exact bound, is on what Nightshift may do on its own.

Problem What Nightshift does Turned on by
Pods crash, or a rollout stalls, within 30 minutes of a release Rolls back to the release before it, only when Seylo recorded that release running healthy Grant: Roll back a crash or stuck rollout to the last release Seylo saw run healthy
A new image whose tag does not exist, or whose name is invalid Rolls back to the release that worked Grant: Roll back when the new image cannot start
A workload that needs a bigger CPU request, not more pods Raises one container’s CPU request to measured use plus 20%, at most 4 times today’s, never above your ceiling Grant: Raise a CPU request, with a CPU request ceiling
A workload whose load splits across pods, with no autoscaler Adds an HPA: minimum today’s pods, maximum within your ceiling Grant: Add an HPA, with Most replicas
A probe that checks a path the app does not answer Copies the path from a passing probe of the same image Box: Probe paths, after 3 approved here
Pods that cannot start because imagePullPolicy is Never Sets it to IfNotPresent One-fix grant
A liveness probe that kills a slow-starting app Gives it longer to start, one step, at most 600 seconds One-fix grant
A readiness probe on a port the container does not serve Points it at the container’s only declared port One-fix grant
A Service whose selector matches no pods Sets the one wrong value to the pods’ own label One-fix grant
A Service that sends to a port its pods do not declare Sets its targetPort to the declared port One-fix grant
An HPA that cannot scale because the pods have no request Sets the missing request from measured use One-fix grant
An HPA at its maximum and still above target Raises maxReplicas one step, within your ceiling One-fix grant, with Highest maxReplicas set on Automation
CPU throttling Raises the CPU limit by 50% A grant whose CPU limit bounds allow it
Evicted pods left behind Deletes them Grant: Delete one controlled pod
A StatefulSet stuck on a pod from an older revision Deletes that pod so it is recreated Grant: Delete one controlled pod
Pods stuck terminating on a lost node Deletes them so they are replaced Grant: Delete one controlled pod
A node not ready for 5 minutes Cordons it, so nothing new lands there Grant for the whole cluster: Cordon a node
A ConfigMap or Secret changed through Seylo that running pods have not picked up Restarts the pods Grant: Restart a Deployment
A queue consumer that stopped consuming while its backlog grows Restarts it, and checks the backlog falls Grant: Restart a Deployment
A Key Vault secret rotated, and the pods still hold the old version Restarts the pods Grant: Restart after a Key Vault secret rotated
Old ReplicaSets piling up Lowers revisionHistoryLimit to the Kubernetes default, 10 One-fix grant
A LimitRange rejecting the pods Sets the request or limit to the range’s own bound One-fix grant
A pending pod whose request no node has room for Lowers the request to one a node fits, never below measured use plus 20% One-fix grant
A pending pod whose nodeSelector names a label no node has Drops that label, only when a ready node can then run it One-fix grant
A container killed for running out of memory Raises the memory limit, sized from evidence, up to four times Box: Memory limits sized from evidence
The same container killed again the same day, after that raise held A second step, sized from evidence, within the same bounds Box: Allow a second step the same day
A nearly full volume Grows it one step, inside your bounds, only when the cost is priced Box: Nearly-full volumes
A claim that never bound, because its StorageClass does not exist Recreates it with the default StorageClass Box: Claims that never bound

Most of these can also run on their own once they have earned autonomy: approved and proven a few times in your organisation, while an admin has that switch on.

Nightshift proposes these with the exact change, how it will check it worked and how it would undo it. A person approves each one.

The CronJob, quota, configuration, budget, init-container, disk, metrics-server and CoreDNS fixes in this table ask a person every time: no grant and no number of earlier approvals lets them run on their own. A change outside the application’s own namespace, or in kube-system, needs an admin who also has release access there: see changes outside the app’s namespace.

Problem What Nightshift proposes
Pods crash, or a rollout stalls, soon after a release, when no grant covers it or Seylo did not record the release before Roll back to the release that worked. See rollback
A feature flag or setting changed just before the errors Put it back to the value Seylo recorded, and restart what reads it. After 3 approved reverts of the same setting, it can earn autonomy. See setting or feature flag
A chaos experiment still cutting or slowing traffic Pause it. See chaos experiment
A workload scaled to zero and never scaled back Back to the replica count it last ran. Someone may have done it on purpose. See scaled to zero
A pod waiting on a taint it does not tolerate A toleration for exactly that taint
A wrong or missing probe, command, request, autoscaler or disruption budget A model-written fix, through five checks. See generated fixes. A probe path, a CPU request and a new HPA can run on its own under the settings above
The app’s logs show the Kubernetes API refusing it (403) The one permission the 403 names, added to the workload’s own Role. An admin approves
A ConfigMap value your own runbook says is wrong That one key set to your runbook’s value, and the readers restarted
Pods waiting for capacity, an overload, or a backend with nothing ready A mitigation: a capacity buffer, a rate limit, a maintenance page, or scaling down another workload
Anything on a workload Argo CD, Flux or Helm owns A pull request to the owner’s repository, or the change with the owner acknowledged
A CronJob that misses its scheduled runs Sets its startingDeadlineSeconds to the schedule’s shortest interval. See Jobs and CronJobs
A CronJob whose runs pile up under concurrencyPolicy: Allow Sets it to Forbid. Runs already active keep running
A Job or CronJob run that keeps failing until its backoffLimit For an OOM kill or a liveness kill, that fix in the CronJob’s job template, so its next run starts with it. Any other failure is named, and a person decides
A namespace over its ResourceQuota, so new pods are blocked Room made inside the quota by lowering workloads that declare far more than they use, or the exact raise. Always a person: no setting lets it run on its own. See a namespace at its quota
Pods that cannot start because a ConfigMap, a Secret or a key they name does not exist Names it. Rolls back only when the release that introduced the reference is proven. See a missing ConfigMap or Secret
A PodDisruptionBudget that allows no disruption while every pod is healthy One more replica, when that lets a drain proceed. See a budget that blocks a drain
An init container that keeps failing After an OOM kill, its memory limit one step up. Any other exit is diagnosed from its log signatures. See an init container that keeps failing
A node under disk pressure Names what fills it. A cordon only when the kubelet’s image clean-up fails and evictions recur. See a node out of disk
The HPA cannot read metrics because metrics-server is down Restarts metrics-server in kube-system. An admin approves. See CoreDNS and metrics-server
Lookups fail across workloads while CoreDNS is unhealthy Restarts CoreDNS, or adds one replica when it is throttled. An admin approves
A second change to the same workload on the same day Whatever the fix is: only one change per workload a day runs on its own, except a second memory step
Missing readiness basics across a namespace The changes you pick from the production-readiness campaign
A fault in the application’s code A coding-agent task that writes the fix and a test. You open the pull request

Here Nightshift reads enough to name the cause, and gives the exact command, patch or step. A person applies it.

Problem What you get
A stopped database, VM or Function App What stopped, and the az command to start it. Check why it stopped first. See Nightshift and your cloud
A database firewall or App Service rule that does not let the cluster in The address the pods leave from, and the az command and Terraform or Bicep that admit it
A private DNS zone, Key Vault firewall, network rule or route that blocks the path The exact change in your cloud
A Front Door origin, probe or WAF rule that fails the requests The exact change in your cloud
A password rotated on one side only The new Job or Secret that appeared just before, and which Secret to update. You set the new value; Nightshift can then restart the workload
A database out of connections The limit and the peak it reached, and the choices: smaller pools in the callers, a connection pooler, or a higher limit. You pick the number
A memory leak Why more memory would only delay the next crash, and a code fix for the coding agent
A NetworkPolicy blocking traffic The exact policy change
A registry that refuses the login (ImagePullBackOff, 401 or 403) Which pull secret the pods and their ServiceAccount name, and whether each exists. You renew the credential: Seylo never writes registry credentials. See a registry that refuses the login
A cert-manager Certificate that is not ready or not renewing The cause, read from its request, order and challenge, and the remedy. Seylo does not trigger the renewal. See a certificate that is not ready
A front proxy on another cluster failing the requests The exact annotation to change. See across clusters

Why Seylo does not make these changes itself:

  • Its cloud access is read-only. Seylo reads your cloud’s settings, never changes them.
  • It never reads or writes a Secret’s value. A new password only you hold stays with you.
  • It never changes your code without a pull request that a person reviews.
  • A quota, a network policy or a database limit is a decision your team made. Seylo does not pick a value it has not measured.

Nothing runs on its own until an admin turns it on. Every fix is checked after it runs, and undone if it does not hold, where it can be.