Skip to content

Upgrade the agent

The agent is a single Deployment with one replica and a Recreate strategy, and it only makes outbound connections. An upgrade is therefore a short reconnect, not an outage of your cluster. Seylo shows the cluster as briefly offline, with the reason.

The latest release is 0.2.22. What each release added is in the agent release notes. The most recent:

Release What you get
0.2.22 Nightshift reads PodDisruptionBudgets, Jobs, CronJobs, ResourceQuotas and cert-manager’s certificates on every install, including the read-only posture. Adds read-only RBAC and no new chart value: see what 0.2.22 adds
0.2.21 Reads which node autoscaler your cluster runs and how it is set, for Savings. Adds an optional permission, off by default, for managed savings to create and change NodePools, never delete them: see what 0.2.22 adds
0.2.20 Reads which Key Vault secret version each pod mounted, and ServiceAccounts, so Nightshift can tell a pod holding an old secret from a wrong value. Refuses a manifest that leaves out its namespace. A namespace-scoped agent says what it refused and why. Adds read-only RBAC: see what 0.2.20 adds
0.2.19 Reads every served version of an API group, so an operator whose kinds are served only at a version the group does not prefer is seen. An old out-of-memory kill no longer reopens its alert on reconnect. An agent whose credential Seylo refuses waits up to a minute between attempts instead of retrying within seconds. See the release notes
0.2.18 Times in what the agent reads (a container’s start and finish, an object’s creation), which 0.2.16 and 0.2.17 sent empty, are sent again, so Nightshift can diagnose a pod killed during start-up or stuck on an old revision. One crash is one alert, and old events no longer hold an alert open. Adds read-only access to PriorityClasses: see what 0.2.18 adds
0.2.17 Notices a dropped connection in about 25 seconds and reconnects; alerts, metrics and results produced while disconnected are sent after it reconnects. Adds two optional chart values: see what 0.2.17 adds
0.2.16 Nightshift reads a Service’s endpoints and a StorageClass, and each object’s own status; searching archived log days on the Logs page. Adds read-only RBAC: see what 0.2.16 adds
0.2.15 Connect an in-cluster Grafana, Prometheus or Alertmanager by its Service name; the key stays in a seylo-connections Secret in your cluster that only the agent reads
0.2.14 Resources reads CustomResourceDefinitions (no more 403); seylo-observability 0.2.14 stops the log store being OOMKilled (2 GiB, bounded queries) and reads more message fields from structured logs. Replaces 0.2.13
0.2.12 seylo-observability 0.2.12 fixes doubled “of allocatable” dashboard figures with OpenCost on
0.2.11 Chargeback by team, label or namespace from OpenCost in your cluster, filtered to each person’s namespaces in the agent. Needs seylo-observability 0.2.11 as well

Verify the new version’s signature before upgrading — once per version.

Terminal window
helm upgrade seylo-agent \
oci://ghcr.io/seylo-inc/charts/seylo-agent --version 0.2.22 \
--namespace seylo-agent --reset-then-reuse-values \
--set image.tag=0.2.22 \
--set image.digest=sha256:ed9222f8b6e0beb0fb12873388439ce270304bc5b54dc41a80a96707c552661f

--reset-then-reuse-values (Helm 3.14 or later) keeps the values you set - your server URL, cluster ID, RBAC scope, watched namespaces, proxy and CA settings - and takes the new release’s defaults for everything else. The digest above is the signed 0.2.22 image index.

On Helm older than 3.14, save the values you set and pass them back as a file:

Terminal window
helm get values seylo-agent -n seylo-agent -o yaml > mine.yaml
helm upgrade seylo-agent \
oci://ghcr.io/seylo-inc/charts/seylo-agent --version 0.2.22 \
--namespace seylo-agent -f mine.yaml \
--set image.tag=0.2.22 \
--set image.digest=sha256:ed9222f8b6e0beb0fb12873388439ce270304bc5b54dc41a80a96707c552661f

Coming from a version before 0.2.6? Read upgrading from before 0.2.6 first.

Install commands from Seylo’s hosted service connect the agent to wss://agent.seylo.in/agent/connect, a host for agent connections only. Agents installed earlier connect to portal.seylo.in and keep working there: you do not have to move them, and nothing about the connection or what the agent may do is different on either host.

Moving one is worth it for a steadier connection: the agent host is not behind the content delivery network in front of the portal, which closes long-lived connections on its own schedule. Each close costs a reconnect, and a retry for a command that was in flight.

  1. Allow agent.seylo.in:443 first, in your egress rules and in your forward proxy’s allow-list if you use one. Otherwise the agent reconnects into a blocked path. See restricted networks.
  2. Change the agent’s server URL. The agent keeps its credential, so no new enrollment token is needed.
Terminal window
helm upgrade seylo-agent oci://ghcr.io/seylo-inc/charts/seylo-agent \
--version <the chart version you run now> \
--namespace seylo-agent --reset-then-reuse-values \
--set serverUrl=wss://agent.seylo.in/agent/connect

helm list -n seylo-agent shows the version you run. Naming it keeps this a move and nothing else; without --version, Helm takes the newest chart, which upgrades the agent at the same time.

The cluster shows as briefly offline while the agent restarts, then Agent: Connected again. Once no agent dials portal.seylo.in, your egress rules no longer need it for the agent.

Terminal window
helm upgrade seylo-agent oci://ghcr.io/seylo-inc/charts/seylo-agent --version 0.2.22 -n seylo-agent --reset-then-reuse-values \
--set image.tag=0.2.22 \
--set image.digest=sha256:ed9222f8b6e0beb0fb12873388439ce270304bc5b54dc41a80a96707c552661f
  • Read-only access for twelve more things Nightshift fixes. The agent can now read PodDisruptionBudgets, Jobs, CronJobs, ResourceQuotas and cert-manager’s certificates, so Nightshift can find a budget that blocks a drain, a CronJob that missed its schedule, a namespace at its quota and a certificate that is not ready. The agent writes none of them. These reads are part of every install, including the read-only posture, and carry no Secret.
  • On an older agent, Nightshift cannot read PodDisruptionBudgets or cert-manager’s certificates: it says the read was refused and names the upgrade. Jobs, CronJobs and quotas were already readable where tier 1 is on, which is the default.

New read-only RBAC, for platform teams, in the ClusterRole and in the Role of each watched namespace. clusterissuers is cluster-scoped, so it is in the ClusterRole only.

Resource API group Verbs
poddisruptionbudgets policy get, list
jobs, cronjobs batch get, list
resourcequotas core ("") get, list
certificates, certificaterequests, issuers, clusterissuers cert-manager.io get, list
orders, challenges acme.cert-manager.io get, list

With plain manifests, kubectl set image does not add these: download the manifest again from the cluster’s Agent details page and kubectl apply it. seylo-observability 0.2.22 is the same as 0.2.21, so upgrading the chart is optional. Choose an RBAC posture lists every rule.

The command in what 0.2.22 adds installs these too.

  • Your node autoscaler, read. The agent reads Karpenter and AKS node auto-provisioning NodePools and NodeClaims, the AKS, EKS and EKS Auto Mode node classes, and GKE ComputeClasses, so the Savings tab can say which autoscaler manages your nodes and what a cheaper or spot pool would change. An EKS node class carries a node bootstrap script; Seylo reads named fields only and never returns it.
  • An optional write permission, off by default: rbac.write.autoscaler.enabled. It lets Seylo create and update those NodePools and node classes when you switch on managed savings with spot or better sizes. It never deletes one: deleting a NodePool drains every node it created at once, so that stays your step, and the agent refuses such a delete itself, whatever else allows it. No posture turns it on, admin included. A write it is not allowed names the setting to turn on.

New read-only RBAC, for platform teams, in the ClusterRole only (these kinds are cluster-scoped, so a namespace-scoped install does not get them):

Resource API group Verbs
nodepools, nodeclaims karpenter.sh get, list, watch
aksnodeclasses karpenter.azure.com get, list, watch
ec2nodeclasses karpenter.k8s.aws get, list, watch
nodeclasses eks.amazonaws.com get, list, watch
computeclasses cloud.google.com get, list, watch

With rbac.write.autoscaler.enabled=true, a separate ClusterRole adds create, update and patch, never delete, on nodepools (karpenter.sh), aksnodeclasses, ec2nodeclasses and computeclasses. Add --set rbac.write.autoscaler.enabled=true to the command in what 0.2.22 adds to turn it on.

With plain manifests, kubectl set image does not add these: download the manifest again from the cluster’s Agent details page and kubectl apply it. seylo-observability 0.2.21 is the same as 0.2.20, so upgrading the chart is optional.

The command in what 0.2.22 adds installs these too.

  • Key Vault secret versions, never values. The agent reads which version of a Key Vault secret each pod mounted through the Secrets Store CSI driver, and a workload’s ServiceAccount. Nightshift can then tell a pod still holding an old version from a wrong value, and check a workload identity. Until you upgrade, it says it cannot read them, and does not propose the restart.
  • A manifest with no namespace is refused. Before, it landed in the agent’s own namespace. A cluster-wide object cannot be written by naming a namespace on it. Seylo always names the namespace, so nothing you deploy through it changes.
  • Clearer refusals on a namespace-scoped agent. It says what was refused (a list across all namespaces, or a cluster-wide kind) and which setting limits it.

New read-only RBAC, for platform teams. Neither carries a Secret value:

Resource API group Verbs Granted in
secretproviderclasspodstatuses secrets-store.csi.x-k8s.io get, list The ClusterRole, and the Role in each watched namespace
serviceaccounts core ("") get The ClusterRole, and the Role in each watched namespace

With plain manifests, kubectl set image does not add these: download the manifest again from the cluster’s Agent details page and kubectl apply it. No new chart values.

seylo-observability 0.2.20 removes OpenCost’s network-costs collector: opencost.networkCosts.enabled=true now fails with the reason. See network traffic.

The command in what 0.2.22 adds installs these too.

  • Times are sent again. Agents 0.2.16 and 0.2.17 sent the times in what the agent reads - when a container started and finished, when a pod or workload was created - as empty values. Without them, Nightshift could not tell how long a container ran before a liveness probe killed it, or how long a StatefulSet pod has been stuck on an old revision, and could not tell a workload that was deleted and created again from the one before it. Until you upgrade, Seylo fills these times from the observability chart’s metrics where they exist, and never guesses one.
  • One crash is one alert, and old events no longer hold an alert open. The agent reports when an event was last seen, so an event about a pod that has since been replaced stops keeping its alert open. See how an alert ends on its own.
  • A created object is undone by its UID. When Nightshift undoes a fix that created an object, the API server itself checks that the object it deletes is the one Seylo created. With older agents, Seylo compares the UID on a fresh read just before the delete.

New read-only RBAC, for platform teams. get and list only:

Resource API group Granted in
priorityclasses scheduling.k8s.io The ClusterRole only. A namespace-scoped install cannot grant it

Nightshift reads PriorityClasses before it proposes a capacity buffer. See RBAC posture. No new chart values. seylo-observability 0.2.18 is released at the same version with no change to its templates or defaults; you do not need to upgrade the chart for this release.

The command in what 0.2.22 adds installs these too.

  • A dropped connection is noticed in about 25 seconds. The agent pings its connection to Seylo every 15 seconds. When nothing comes back within 10 seconds, it closes the connection and dials again. Before 0.2.17, a connection that a proxy or load balancer dropped without closing it went unnoticed until the agent’s next heartbeat.
  • Nothing is lost while it reconnects. Alerts, metrics and command results produced while the agent was disconnected are kept and sent once it has reconnected. The agent keeps a bounded amount and drops the oldest first.

No RBAC changes. Two new chart values, both optional:

Value Default What it does
connection.pingSeconds 15 How often the agent pings its connection. 0 turns the ping off, for a network path that drops WebSocket control frames
connection.pongTimeoutSeconds 10 How long the agent waits for an answer before it reconnects. Keep it below pingSeconds

Leave both at their defaults unless a proxy between your cluster and Seylo drops pings. An upgrade from an older chart that reuses your values gets the defaults.

The command in what 0.2.22 adds installs these too.

  • Nightshift reads more, and still never a Secret. The agent can read a Service’s endpoints and a StorageClass. Nightshift then finds a Service with no endpoints without the observability chart, and can propose growing a nearly full volume whose StorageClass allows expansion.
  • Each object’s own status. Seylo sees an object’s status beside its definition, for example a pod’s last termination reason. Secrets are never read.
  • Archived log days on the Logs page. With seylo-observability 0.2.16, you can search days older than your retention.

New read-only RBAC, for platform teams. get and list only:

Resource API group Granted in
endpoints core ("") The ClusterRole, and the Role in each watched namespace
endpointslices discovery.k8s.io The ClusterRole, and the Role in each watched namespace
storageclasses storage.k8s.io The ClusterRole only. A namespace-scoped install cannot grant it, and the volume fix there says so

See RBAC posture.

Older agents keep working. Until you upgrade, the new features say needs agent 0.2.16.

0.2.6 changed the agent’s default RBAC. Read this if your agent is older than 0.2.6.

rbac.write.tier1.customResources is a list of {apiGroup, resources} entries, and empty by default: the agent writes no custom resources until you name them. See custom resources: an explicit list.

  • With 0.2.7 and later, the upgrade succeeds whatever your earlier install carried. Earlier releases defaulted customResources to true, and an upgrade that reuses values carries that forward; 0.2.7 reads a carried-over true or false as no custom-resource writes. Add the custom resources your teams use as a list (below).

  • 0.2.6 refused it and stopped with an error that includes the line below. Nothing was changed in the cluster. Upgrade to 0.2.7 instead:

    rbac.write.tier1.customResources is now a LIST of {apiGroup, resources[]} entries, not true.

To grant no custom-resource writes explicitly:

Terminal window
helm upgrade seylo-agent \
oci://ghcr.io/seylo-inc/charts/seylo-agent --version 0.2.22 \
--namespace seylo-agent --reset-then-reuse-values \
--set image.tag=0.2.22 \
--set image.digest=sha256:ed9222f8b6e0beb0fb12873388439ce270304bc5b54dc41a80a96707c552661f \
--set rbac.write.tier1.customResources=null

To grant the ones your teams use, put the list in a values file and pass it instead of the null:

Terminal window
helm upgrade seylo-agent \
oci://ghcr.io/seylo-inc/charts/seylo-agent --version 0.2.22 \
--namespace seylo-agent --reset-then-reuse-values \
--set image.tag=0.2.22 \
--set image.digest=sha256:ed9222f8b6e0beb0fb12873388439ce270304bc5b54dc41a80a96707c552661f \
-f seylo-agent-values.yaml

If a 0.2.6 upgrade stopped with the message above and you never set the value yourself, the upgrade carried it from your earlier install. Upgrading to 0.2.7 fixes it with no extra setting.

  • The agent reads SecretProviderClasses and PersistentVolumeClaims, and tier 1 creates and patches them. It never deletes either. A namespace copy of a Key Vault workload no longer needs extra Helm values.
  • The agent reports only the namespaces in its watch list that actually exist.

Render the result before you apply it, with the same flags plus your values:

Terminal window
helm template seylo-agent \
oci://ghcr.io/seylo-inc/charts/seylo-agent --version 0.2.22 \
--set serverUrl=wss://x/agent/connect --set clusterId=x --set enrollmentToken=x \
-f seylo-agent-values.yaml

After the upgrade, check what the agent may actually do.

You are not required to upgrade in step with the server

Section titled “You are not required to upgrade in step with the server”

The wire protocol is range-negotiated. Seylo accepts any version it still supports and negotiates down. An older agent keeps doing everything it could do before, and any capability it lacks is reported as unsupported — with its version and the version required, never as zeroes or an empty chart.

The one direction that does not work: an agent newer than the server cannot be negotiated down. If you see protocol-version-unsupported after an upgrade, that is the cause.

It is published for registry-UI reasons only. With imagePullPolicy: IfNotPresent, two nodes in the same cluster can end up running different builds, and re-pointing the tag does not fix a node that already pulled.

Pin a version, and preferably a digest.

A tag says which release you meant. A digest says which bytes you got, and it cannot be moved.

Terminal window
crane digest ghcr.io/seylo-inc/seylo-agent:0.2.22
# or
docker buildx imagetools inspect ghcr.io/seylo-inc/seylo-agent:0.2.22

Pass it alongside the tag. The chart renders repository:tag@sha256:… — Kubernetes resolves the digest and ignores the tag, so the image cannot be swapped underneath you while kubectl get pod -o wide still shows a readable version. A malformed digest is rejected by the chart at render time rather than thirty seconds later as an ImagePullBackOff.

The chart published to the registry already defaults to the digest of the image it was released with, so a Helm install from the OCI registry is digest-pinned even if you pass nothing.