Skip to content

Grants

By default nothing in Seylo changes your cluster without an approval. A grant relaxes that, narrowly: it says Seylo may make this kind of change to these workloads, by at most this much, during these hours, at most this often, until this date. Inside those limits Seylo acts on its own. Outside them, the change waits for a person.

What an approver signs is not a change. It is a bounded permission, and every decision taken under it, including every refusal, is recorded.

What What a grant lets it do
Nightshift The actions ticked in the grant’s Nightshift actions section, such as a rollback after a release, raising a CPU request or adding an HPA, each inside its own bound. See what Nightshift may do on its own
Nightshift’s and Seylo’s remediation changes to scale and resources Changes to the dimensions the grant names - CPU or memory request or limit, replicas - inside its limits. For example, raising a throttled container’s CPU limit by 50%
APM coverage Instrument a workload for tracing, inside the grant’s namespace and languages

Managed savings has its own grant, signed when an admin switches it on for a cluster. It is shown separately on Automation, never used by anything else, and a grant you sign here never covers a managed saving.

Sizing for savings is not done under a grant any more. Until 0.4.9 a separate engine lowered requests and replicas on its own under grants that named those dimensions. It is retired: sizing for savings is Rightsizing on the Savings tab, sent for approval, or applied by managed savings. A grant naming CPU request, memory request or replicas now covers only the Nightshift and remediation changes that use them.

If Operate → Automation says Unattended changes are off on this deployment: grants are stored, and nothing uses them yet., nothing acts on its own under any grant on this Seylo deployment; every change goes through approval.

  1. Open Operate → Automation and pick the cluster.
  2. Choose New grant (admin). Or, from a resource or replica change waiting on Approvals, choose Allow this kind to open the form pre-filled for that workload. See allow this kind of change.
  3. Fill in the sections below. The form shows every problem at once, using the same rules the server applies. A problem that is the same for several dimensions is listed once. While any problem is left, Request approval stays off, and the line beside it says why, for example Fix the 2 problems listed above to request approval.
  4. Read What the approver will read, then choose Request approval. The grant always becomes an approval request, even on a deployment where approvals are switched off for other changes.
  5. An approver or admin approves it on Approvals. One approval is enough, and an admin may approve their own grant.

Approving a grant changes nothing on the cluster. It creates the grant; each later action is recorded against the grant’s id. The grant is checked again at approval time, so one that sat in the queue past its expiry cannot be approved into existence:

This autoscale policy is no longer valid at approval time, so no grant was created: ...

There is no way to edit a grant’s limits in place. Widening one is a new grant with a new approval.

A cluster, and optionally a Namespace and a Workload. Empty means every namespace, or every workload in scope. Names match exactly — there are no patterns, because prod-* matching prod-experimental is precisely the accident a scope exists to prevent. A workload with no namespace is refused: it would match that name in every namespace.

When several grants cover the same workload, the narrowest decides: workload, then namespace, then cluster. A broader grant is never searched for a more permissive answer, and a suspended narrow grant is not bypassed by a live broad one.

A grant may name five dimensions: CPU request, Memory request, CPU limit, Memory limit and Replicas. Tick the dimensions the grant permits. Anything not ticked is queued for a person. The limits are asked once, and apply to every ticked dimension:

Field Rule
Max increase per change % Required, 1 to 50. The form starts at 20
Max decrease per change % Required, 1 to 50. The form starts at 10

To treat one dimension differently, tick Set a different limit, a floor or a ceiling for one dimension. A row then opens for each ticked dimension:

Field Rule
Max increase %, Max decrease % Optional, 1 to 50. Empty means the shared value above
Floor, Ceiling Optional. Millicores for CPU, bytes for memory, a count for replicas

50% is a hard cap on any single unattended change, whatever a grant asks for. It is fixed in Seylo, not a setting. A bigger move happens as several changes, spaced by the rate limit and cooldown. A dimension listed without limits authorises nothing; its card says no limit: never acts. Increases and decreases are limited separately because they are different decisions — scaling up costs money, scaling down risks availability.

A grant may name Nightshift actions and no dimension.

Each action covers one kind of change Nightshift may make on its own when its investigation of an alert proposes it, inside this grant’s scope, hours and rate limit. Anything not ticked waits for a person. There is no wildcard, and a grant signed before an action existed does not cover it.

Action Notes
Restart a Deployment A rolling restart
Delete one controlled pod Only a pod a controller will recreate
Roll back to the previous revision (any reason) Never implied by anything else: tick it to grant it
Roll back when the new image cannot start (pull fails or name invalid) Only when none of the release’s pods ever ran. See roll back an image that never started
Roll back a crash or stuck rollout to the last release Seylo saw run healthy (within 30 min of the release) Only to a release Seylo recorded as succeeded. See roll back a crash or stuck rollout
Raise a CPU request to measured use + 20% (at most 4x, never above the CPU ceiling you set) Ticking it shows CPU request ceiling (millicores), which is required: without it the action raises nothing. See raise a CPU request
Add an HPA where load splits across pods (min = today’s pods, max within your replica ceiling) Ticking it shows Most replicas, which is required: the highest maxReplicas an HPA Nightshift creates may have. See add an HPA
Cordon a node, Uncordon a node Covered only by a cluster-wide grant (no namespace)
Instrument a workload for tracing (one annotation; restarts its pods) Adds one OpenTelemetry Operator annotation to a workload that serves traffic and sends no spans, and removes it if spans do not arrive or the pods fail. The grant must name its namespace. Tick the languages it covers - Java, Node.js, Python, .NET; never Go - and Workloads instrumented per day, 1 to 20. Never a workload Argo CD, Flux or Helm owns. See APM coverage
Restart a workload whose pods mount an older version of a Key Vault secret than the vault’s current one See restart after a Key Vault secret rotated
The one-fix actions, from Let pods pull their image on Each covers one Nightshift fix and nothing else, inside the bound its label states. See one-fix grants

The two ceilings are kept on the grant, and shown beside the action wherever the grant is: on its card on Automation, on the request an approver signs, and on Nightshift → Autonomy. A grant that names Add an HPA and was signed before Most replicas was required says so on its card: only the Highest maxReplicas on Automation bounds it, and with neither it creates nothing.

The grant still holds the limits per dimension: What the approver will read lists each ticked dimension with its own increase and decrease.

Days of the week and hour ranges, in a timezone — required, never defaulted to UTC, because “not during business hours” is about someone’s local clock. Choose it from the Timezone list: your browser’s zone comes first, then a short list of common zones, then every zone, each shown with its current offset from UTC, such as Asia/Kolkata (GMT+05:30). Hours run 0 to 24 with the start before the end. A window that crosses midnight is two ranges, such as 22–24 and 0–6. An empty window means never.

Field Rule
Max actions per window At least 1, per workload and dimension
Window (hours) Positive
Cooldown (minutes) Zero or more. Time between two unattended changes to the same target, so the effect of one is seen before the next
Minimum baseline confidence medium or high. Never lower: acting on evidence Seylo itself calls weak is what this design exists to prevent
Duration (days) At most 90. A grant always expires; renewing it is a deliberate decision
Notes Carried into the record of every action the grant authorises

For a change to a dimension, the checks run in this order. The first one that fails decides.

  1. Pause all automation, for your organisation or for this cluster → refuse.
  2. No grant covers the workload → a person decides.
  3. The covering grant is suspended or expired → a person decides.
  4. Helm, Argo CD or Flux owns the workload, or ownership cannot be read → a person decides. A reconciler would revert the change within minutes and it would be proposed again — a churn loop. A Helm release would silently discard it at the next helm upgrade.
  5. Baseline confidence below the grant’s minimum → a person decides.
  6. The dimension is not permitted, or has no limits → a person decides.
  7. The size of the change: more than the grant’s percentage or the 50% cap, below the floor, above the ceiling, or from a current value of 0 (no percentage exists) → a person decides.
  8. Outside the window → a person decides. A window that cannot be evaluated is treated as closed.
  9. 3 consecutive failed or reverted unattended changes on this target → refuse, and the grant suspends itself.
  10. Rate limit reached, or inside the cooldown → a person decides.
  11. A reduction whose usage pattern is not actionable — too flat, or too little hourly history → a person decides. Being wrong downward costs an incident, not a bill.
  12. Otherwise → act.

A Nightshift action is decided the same way for scope, window, rate limit, expiry and the pause, and then held to the action’s own bound.

An act is applied through the same engine as a human-approved change: the ownership check, a dry run, server-side apply and a read-back. A replica change applies to Deployments only.

A change no grant covers is not lost. Nightshift files it for approval, once per workload and dimension, and the fix says so, for example Sent for approval as ... Not applied unattended: .... The exact reasons are in troubleshooting.

  • Suspend a grant (releaser): it authorises nothing until resumed. Changes it would have covered wait for a person.
  • Resume it (releaser): it is active again with its original limits and expiry. Resuming re-approves nothing.
  • Auto-suspension: after 3 consecutive unattended changes on one target that failed or were reverted, the grant suspends itself so a person looks. Something is fighting the change, and repeating it unattended will not win. The grant’s card shows the count against 3, for example 2/3 failed.

A change counts as reverted when the write was accepted but reading it back showed something else, and as failed when the write failed. Neither counts as a success, and both use up a slot in the rate limit.

Pause all automation at the top of Operate → Automation (releaser) stops every grant on every cluster in your organisation at once, managed savings included. See pause and limits.

Every change a grant allowed, and every refusal, is a line in What Seylo did on its own on Automation: what changed, why, and which grant allowed it. See what Seylo did on its own. Each grant’s card also lists its own last decisions under Details, with how many changes it made.

Decisions that only waited for a person - not covered, outside the window, inside a cooldown - are not log lines. They are on the fix that was filed, or on Approvals.