Upgrade the agent
The agent is a single Deployment with one replica and a Recreate strategy, and
it only makes outbound connections. An upgrade is therefore a short
reconnect, not an outage of your cluster. Seylo shows the cluster as briefly
offline, with the reason.
The latest release is 0.2.22. What each release added is in the agent release notes. The most recent:
| Release | What you get |
|---|---|
| 0.2.22 | Nightshift reads PodDisruptionBudgets, Jobs, CronJobs, ResourceQuotas and cert-manager’s certificates on every install, including the read-only posture. Adds read-only RBAC and no new chart value: see what 0.2.22 adds |
| 0.2.21 | Reads which node autoscaler your cluster runs and how it is set, for Savings. Adds an optional permission, off by default, for managed savings to create and change NodePools, never delete them: see what 0.2.22 adds |
| 0.2.20 | Reads which Key Vault secret version each pod mounted, and ServiceAccounts, so Nightshift can tell a pod holding an old secret from a wrong value. Refuses a manifest that leaves out its namespace. A namespace-scoped agent says what it refused and why. Adds read-only RBAC: see what 0.2.20 adds |
| 0.2.19 | Reads every served version of an API group, so an operator whose kinds are served only at a version the group does not prefer is seen. An old out-of-memory kill no longer reopens its alert on reconnect. An agent whose credential Seylo refuses waits up to a minute between attempts instead of retrying within seconds. See the release notes |
| 0.2.18 | Times in what the agent reads (a container’s start and finish, an object’s creation), which 0.2.16 and 0.2.17 sent empty, are sent again, so Nightshift can diagnose a pod killed during start-up or stuck on an old revision. One crash is one alert, and old events no longer hold an alert open. Adds read-only access to PriorityClasses: see what 0.2.18 adds |
| 0.2.17 | Notices a dropped connection in about 25 seconds and reconnects; alerts, metrics and results produced while disconnected are sent after it reconnects. Adds two optional chart values: see what 0.2.17 adds |
| 0.2.16 | Nightshift reads a Service’s endpoints and a StorageClass, and each object’s own status; searching archived log days on the Logs page. Adds read-only RBAC: see what 0.2.16 adds |
| 0.2.15 | Connect an in-cluster Grafana, Prometheus or Alertmanager by its Service name; the key stays in a seylo-connections Secret in your cluster that only the agent reads |
| 0.2.14 | Resources reads CustomResourceDefinitions (no more 403); seylo-observability 0.2.14 stops the log store being OOMKilled (2 GiB, bounded queries) and reads more message fields from structured logs. Replaces 0.2.13 |
| 0.2.12 | seylo-observability 0.2.12 fixes doubled “of allocatable” dashboard figures with OpenCost on |
| 0.2.11 | Chargeback by team, label or namespace from OpenCost in your cluster, filtered to each person’s namespaces in the agent. Needs seylo-observability 0.2.11 as well |
Verify the new version’s signature before upgrading — once per version.
helm upgrade seylo-agent \ oci://ghcr.io/seylo-inc/charts/seylo-agent --version 0.2.22 \ --namespace seylo-agent --reset-then-reuse-values \ --set image.tag=0.2.22 \ --set image.digest=sha256:ed9222f8b6e0beb0fb12873388439ce270304bc5b54dc41a80a96707c552661f--reset-then-reuse-values (Helm 3.14 or later) keeps the values you set -
your server URL, cluster ID, RBAC scope, watched namespaces, proxy and CA
settings - and takes the new release’s defaults for everything else. The
digest above is the signed 0.2.22 image index.
On Helm older than 3.14, save the values you set and pass them back as a file:
helm get values seylo-agent -n seylo-agent -o yaml > mine.yamlhelm upgrade seylo-agent \ oci://ghcr.io/seylo-inc/charts/seylo-agent --version 0.2.22 \ --namespace seylo-agent -f mine.yaml \ --set image.tag=0.2.22 \ --set image.digest=sha256:ed9222f8b6e0beb0fb12873388439ce270304bc5b54dc41a80a96707c552661fComing from a version before 0.2.6? Read upgrading from before 0.2.6 first.
kubectl -n seylo-agent set image deployment/seylo-agent \ agent=ghcr.io/seylo-inc/seylo-agent:0.2.22@sha256:ed9222f8b6e0beb0fb12873388439ce270304bc5b54dc41a80a96707c552661f
kubectl -n seylo-agent rollout status deployment/seylo-agentset image changes the image and nothing else. 0.2.6 changed the RBAC
objects, so coming from before 0.2.6, download the manifest again from the
cluster’s Agent details page (Admin → Clusters, the cluster’s menu) and
kubectl apply it
instead.
Move an agent to agent.seylo.in
Section titled “Move an agent to agent.seylo.in”Install commands from Seylo’s hosted service connect the agent to
wss://agent.seylo.in/agent/connect, a host for agent connections only.
Agents installed earlier connect to portal.seylo.in and keep working
there: you do not have to move them, and nothing about the connection or
what the agent may do is different on either host.
Moving one is worth it for a steadier connection: the agent host is not behind the content delivery network in front of the portal, which closes long-lived connections on its own schedule. Each close costs a reconnect, and a retry for a command that was in flight.
- Allow
agent.seylo.in:443first, in your egress rules and in your forward proxy’s allow-list if you use one. Otherwise the agent reconnects into a blocked path. See restricted networks. - Change the agent’s server URL. The agent keeps its credential, so no new enrollment token is needed.
helm upgrade seylo-agent oci://ghcr.io/seylo-inc/charts/seylo-agent \ --version <the chart version you run now> \ --namespace seylo-agent --reset-then-reuse-values \ --set serverUrl=wss://agent.seylo.in/agent/connecthelm list -n seylo-agent shows the version you run. Naming it keeps this a
move and nothing else; without --version, Helm takes the newest chart, which
upgrades the agent at the same time.
kubectl -n seylo-agent set env deployment/seylo-agent SERVER_URL=wss://agent.seylo.in/agent/connect
kubectl -n seylo-agent rollout status deployment/seylo-agentThe cluster shows as briefly offline while the agent restarts, then
Agent: Connected again. Once no agent dials portal.seylo.in, your egress
rules no longer need it for the agent.
What 0.2.22 adds
Section titled “What 0.2.22 adds”helm upgrade seylo-agent oci://ghcr.io/seylo-inc/charts/seylo-agent --version 0.2.22 -n seylo-agent --reset-then-reuse-values \ --set image.tag=0.2.22 \ --set image.digest=sha256:ed9222f8b6e0beb0fb12873388439ce270304bc5b54dc41a80a96707c552661f- Read-only access for twelve more things Nightshift fixes. The agent can
now read PodDisruptionBudgets, Jobs, CronJobs, ResourceQuotas and
cert-manager’s certificates, so
Nightshift can find a budget that
blocks a drain, a CronJob that missed its schedule, a namespace at its quota
and a certificate that is not ready. The agent writes none of them. These
reads are part of every install, including the
read-onlyposture, and carry no Secret. - On an older agent, Nightshift cannot read PodDisruptionBudgets or cert-manager’s certificates: it says the read was refused and names the upgrade. Jobs, CronJobs and quotas were already readable where tier 1 is on, which is the default.
New read-only RBAC, for platform teams, in the ClusterRole and in the Role
of each watched namespace. clusterissuers is cluster-scoped, so it is in the
ClusterRole only.
| Resource | API group | Verbs |
|---|---|---|
poddisruptionbudgets |
policy |
get, list |
jobs, cronjobs |
batch |
get, list |
resourcequotas |
core ("") |
get, list |
certificates, certificaterequests, issuers, clusterissuers |
cert-manager.io |
get, list |
orders, challenges |
acme.cert-manager.io |
get, list |
With plain manifests, kubectl set image does not add these: download the
manifest again from the cluster’s Agent details page and kubectl apply
it. seylo-observability 0.2.22 is the same as 0.2.21, so upgrading the chart
is optional. Choose an RBAC posture lists every
rule.
What 0.2.21 adds
Section titled “What 0.2.21 adds”The command in what 0.2.22 adds installs these too.
- Your node autoscaler, read. The agent reads Karpenter and AKS node auto-provisioning NodePools and NodeClaims, the AKS, EKS and EKS Auto Mode node classes, and GKE ComputeClasses, so the Savings tab can say which autoscaler manages your nodes and what a cheaper or spot pool would change. An EKS node class carries a node bootstrap script; Seylo reads named fields only and never returns it.
- An optional write permission, off by default:
rbac.write.autoscaler.enabled. It lets Seylo create and update those NodePools and node classes when you switch on managed savings with spot or better sizes. It never deletes one: deleting a NodePool drains every node it created at once, so that stays your step, and the agent refuses such a delete itself, whatever else allows it. No posture turns it on,adminincluded. A write it is not allowed names the setting to turn on.
New read-only RBAC, for platform teams, in the ClusterRole only (these kinds are cluster-scoped, so a namespace-scoped install does not get them):
| Resource | API group | Verbs |
|---|---|---|
nodepools, nodeclaims |
karpenter.sh |
get, list, watch |
aksnodeclasses |
karpenter.azure.com |
get, list, watch |
ec2nodeclasses |
karpenter.k8s.aws |
get, list, watch |
nodeclasses |
eks.amazonaws.com |
get, list, watch |
computeclasses |
cloud.google.com |
get, list, watch |
With rbac.write.autoscaler.enabled=true, a separate ClusterRole adds
create, update and patch, never delete, on nodepools
(karpenter.sh), aksnodeclasses, ec2nodeclasses and computeclasses.
Add --set rbac.write.autoscaler.enabled=true to the command in
what 0.2.22 adds to turn it on.
With plain manifests, kubectl set image does not add these: download the
manifest again from the cluster’s Agent details page and kubectl apply
it. seylo-observability 0.2.21 is the same as 0.2.20, so upgrading the chart
is optional.
What 0.2.20 adds
Section titled “What 0.2.20 adds”The command in what 0.2.22 adds installs these too.
- Key Vault secret versions, never values. The agent reads which version of a Key Vault secret each pod mounted through the Secrets Store CSI driver, and a workload’s ServiceAccount. Nightshift can then tell a pod still holding an old version from a wrong value, and check a workload identity. Until you upgrade, it says it cannot read them, and does not propose the restart.
- A manifest with no namespace is refused. Before, it landed in the agent’s own namespace. A cluster-wide object cannot be written by naming a namespace on it. Seylo always names the namespace, so nothing you deploy through it changes.
- Clearer refusals on a namespace-scoped agent. It says what was refused (a list across all namespaces, or a cluster-wide kind) and which setting limits it.
New read-only RBAC, for platform teams. Neither carries a Secret value:
| Resource | API group | Verbs | Granted in |
|---|---|---|---|
secretproviderclasspodstatuses |
secrets-store.csi.x-k8s.io |
get, list |
The ClusterRole, and the Role in each watched namespace |
serviceaccounts |
core ("") |
get |
The ClusterRole, and the Role in each watched namespace |
With plain manifests, kubectl set image does not add these: download the
manifest again from the cluster’s Agent details page and kubectl apply
it. No new chart values.
seylo-observability 0.2.20 removes OpenCost’s network-costs collector:
opencost.networkCosts.enabled=true now fails with the reason. See
network traffic.
What 0.2.18 adds
Section titled “What 0.2.18 adds”The command in what 0.2.22 adds installs these too.
- Times are sent again. Agents 0.2.16 and 0.2.17 sent the times in what the agent reads - when a container started and finished, when a pod or workload was created - as empty values. Without them, Nightshift could not tell how long a container ran before a liveness probe killed it, or how long a StatefulSet pod has been stuck on an old revision, and could not tell a workload that was deleted and created again from the one before it. Until you upgrade, Seylo fills these times from the observability chart’s metrics where they exist, and never guesses one.
- One crash is one alert, and old events no longer hold an alert open. The agent reports when an event was last seen, so an event about a pod that has since been replaced stops keeping its alert open. See how an alert ends on its own.
- A created object is undone by its UID. When Nightshift undoes a fix that created an object, the API server itself checks that the object it deletes is the one Seylo created. With older agents, Seylo compares the UID on a fresh read just before the delete.
New read-only RBAC, for platform teams. get and list only:
| Resource | API group | Granted in |
|---|---|---|
priorityclasses |
scheduling.k8s.io |
The ClusterRole only. A namespace-scoped install cannot grant it |
Nightshift reads PriorityClasses before it proposes a
capacity buffer. See
RBAC posture.
No new chart values. seylo-observability 0.2.18 is released at the same
version with no change to its templates or defaults; you do not need to
upgrade the chart for this release.
What 0.2.17 adds
Section titled “What 0.2.17 adds”The command in what 0.2.22 adds installs these too.
- A dropped connection is noticed in about 25 seconds. The agent pings its connection to Seylo every 15 seconds. When nothing comes back within 10 seconds, it closes the connection and dials again. Before 0.2.17, a connection that a proxy or load balancer dropped without closing it went unnoticed until the agent’s next heartbeat.
- Nothing is lost while it reconnects. Alerts, metrics and command results produced while the agent was disconnected are kept and sent once it has reconnected. The agent keeps a bounded amount and drops the oldest first.
No RBAC changes. Two new chart values, both optional:
| Value | Default | What it does |
|---|---|---|
connection.pingSeconds |
15 |
How often the agent pings its connection. 0 turns the ping off, for a network path that drops WebSocket control frames |
connection.pongTimeoutSeconds |
10 |
How long the agent waits for an answer before it reconnects. Keep it below pingSeconds |
Leave both at their defaults unless a proxy between your cluster and Seylo drops pings. An upgrade from an older chart that reuses your values gets the defaults.
What 0.2.16 adds
Section titled “What 0.2.16 adds”The command in what 0.2.22 adds installs these too.
- Nightshift reads more, and still never a Secret. The agent can read a Service’s endpoints and a StorageClass. Nightshift then finds a Service with no endpoints without the observability chart, and can propose growing a nearly full volume whose StorageClass allows expansion.
- Each object’s own status. Seylo sees an object’s status beside its definition, for example a pod’s last termination reason. Secrets are never read.
- Archived log days on the Logs page. With
seylo-observability0.2.16, you can search days older than your retention.
New read-only RBAC, for platform teams. get and list only:
| Resource | API group | Granted in |
|---|---|---|
endpoints |
core ("") |
The ClusterRole, and the Role in each watched namespace |
endpointslices |
discovery.k8s.io |
The ClusterRole, and the Role in each watched namespace |
storageclasses |
storage.k8s.io |
The ClusterRole only. A namespace-scoped install cannot grant it, and the volume fix there says so |
See RBAC posture.
Older agents keep working. Until you upgrade, the new features say needs agent 0.2.16.
Upgrading from before 0.2.6
Section titled “Upgrading from before 0.2.6”0.2.6 changed the agent’s default RBAC. Read this if your agent is older than 0.2.6.
customResources is now a list
Section titled “customResources is now a list”rbac.write.tier1.customResources is a list of {apiGroup, resources}
entries, and empty by default: the agent writes no custom resources until
you name them. See
custom resources: an explicit list.
-
With 0.2.7 and later, the upgrade succeeds whatever your earlier install carried. Earlier releases defaulted
customResourcestotrue, and an upgrade that reuses values carries that forward; 0.2.7 reads a carried-overtrueorfalseas no custom-resource writes. Add the custom resources your teams use as a list (below). -
0.2.6 refused it and stopped with an error that includes the line below. Nothing was changed in the cluster. Upgrade to 0.2.7 instead:
rbac.write.tier1.customResources is now a LIST of {apiGroup, resources[]} entries, not true.
To grant no custom-resource writes explicitly:
helm upgrade seylo-agent \ oci://ghcr.io/seylo-inc/charts/seylo-agent --version 0.2.22 \ --namespace seylo-agent --reset-then-reuse-values \ --set image.tag=0.2.22 \ --set image.digest=sha256:ed9222f8b6e0beb0fb12873388439ce270304bc5b54dc41a80a96707c552661f \ --set rbac.write.tier1.customResources=nullTo grant the ones your teams use, put the list in a values file and pass it
instead of the null:
helm upgrade seylo-agent \ oci://ghcr.io/seylo-inc/charts/seylo-agent --version 0.2.22 \ --namespace seylo-agent --reset-then-reuse-values \ --set image.tag=0.2.22 \ --set image.digest=sha256:ed9222f8b6e0beb0fb12873388439ce270304bc5b54dc41a80a96707c552661f \ -f seylo-agent-values.yamlIf a 0.2.6 upgrade stopped with the message above and you never set the value yourself, the upgrade carried it from your earlier install. Upgrading to 0.2.7 fixes it with no extra setting.
New in the default grant
Section titled “New in the default grant”- The agent reads SecretProviderClasses and PersistentVolumeClaims, and tier 1 creates and patches them. It never deletes either. A namespace copy of a Key Vault workload no longer needs extra Helm values.
- The agent reports only the namespaces in its watch list that actually exist.
Check what you granted
Section titled “Check what you granted”Render the result before you apply it, with the same flags plus your values:
helm template seylo-agent \ oci://ghcr.io/seylo-inc/charts/seylo-agent --version 0.2.22 \ --set serverUrl=wss://x/agent/connect --set clusterId=x --set enrollmentToken=x \ -f seylo-agent-values.yamlAfter the upgrade, check what the agent may actually do.
You are not required to upgrade in step with the server
Section titled “You are not required to upgrade in step with the server”The wire protocol is range-negotiated. Seylo accepts any version it still supports and negotiates down. An older agent keeps doing everything it could do before, and any capability it lacks is reported as unsupported — with its version and the version required, never as zeroes or an empty chart.
The one direction that does not work: an agent newer than the server cannot
be negotiated down. If you see protocol-version-unsupported after an upgrade,
that is the cause.
Never track :latest
Section titled “Never track :latest”It is published for registry-UI reasons only. With imagePullPolicy: IfNotPresent, two nodes in the same cluster can end up running different
builds, and re-pointing the tag does not fix a node that already pulled.
Pin a version, and preferably a digest.
Pinning a digest
Section titled “Pinning a digest”A tag says which release you meant. A digest says which bytes you got, and it cannot be moved.
crane digest ghcr.io/seylo-inc/seylo-agent:0.2.22# ordocker buildx imagetools inspect ghcr.io/seylo-inc/seylo-agent:0.2.22Pass it alongside the tag. The chart renders repository:tag@sha256:… —
Kubernetes resolves the digest and ignores the tag, so the image cannot be
swapped underneath you while kubectl get pod -o wide still shows a readable
version. A malformed digest is rejected by the chart at render time rather than
thirty seconds later as an ImagePullBackOff.
The chart published to the registry already defaults to the digest of the image it was released with, so a Helm install from the OCI registry is digest-pinned even if you pass nothing.