Skip to content

Choose an RBAC posture

Two independent decisions control what the agent can do in your cluster:

  1. Scope — cluster-wide, or a Role in each of a named set of namespaces.
  2. Posture — how much the agent may change within that scope.

Both are enforced by Kubernetes itself, not by Seylo. That is the point: the tier model in the portal can be argued with in a code review; an RBAC grant cannot be argued with at all.

Pick a named posture rather than assembling individual toggles. The most common RBAC mistake here is a half-set combination nobody meant — browse enabled with writes left on, say.

Posture The agent can Use when
read-only See the cluster and change nothing in it You want visibility first, or a cluster Seylo should never mutate
deploy Read everything it needs, and create/update/patch the workload kinds a release touches. No delete anywhere Most clusters. This is the product’s intended posture
admin deploy, plus the destructive tier: delete and evict pods, cordon and drain nodes, delete workloads Someone is expected to remediate from the portal
(unset) Whatever the individual toggles say An existing install keeps exactly the Role it has

read-only grants exactly one write, and it is not a cluster permission

Section titled “read-only grants exactly one write, and it is not a cluster permission”

get/patch/update on the agent’s own credential Secret, narrowed by name to seylo-agent-credential.

The agent has to persist its own rotated credential or it cannot stay connected. This is stated here rather than left to be discovered in a rendered Role.

Every posture can read one Secret of connection keys

Section titled “Every posture can read one Secret of connection keys”

From agent 0.2.15, every posture may get one more Secret, narrowed by name to seylo-connections in the agent’s own namespace (chart value connections.secretName). It holds the keys you put there for Grafana, Prometheus and Alertmanager connections made through the agent. The agent reads a key by name for one request to that tool’s Service and never sends it to Seylo. It cannot list Secrets or read any other. If you never create seylo-connections, the rule matches nothing.

Every posture can read CustomResourceDefinitions

Section titled “Every posture can read CustomResourceDefinitions”

From agent 0.2.14, the agent may get, list and watch CustomResourceDefinitions in every posture, read-only included. A CRD is a schema, never data: reading it tells Resources which kinds in your cluster are custom. Reading the custom resources themselves still needs rbac.genericRead or a write tier. Before 0.2.14, Resources showed customresourcedefinitions.apiextensions.k8s.io is forbidden and guessed which kinds were custom from their API group.

Every posture can read endpoints and StorageClasses

Section titled “Every posture can read endpoints and StorageClasses”

From agent 0.2.16, every posture adds three read-only grants, get and list only:

Resource API group Where it is granted
endpoints core ("") The ClusterRole, and the Role in each watched namespace
endpointslices discovery.k8s.io The ClusterRole, and the Role in each watched namespace
storageclasses storage.k8s.io The ClusterRole only

They let Nightshift find a Service with no endpoints without the observability chart, and tell whether a nearly full volume’s StorageClass allows expansion before it proposes growing the volume. None of them carries a Secret value.

A StorageClass is cluster-scoped, and a Role cannot grant a cluster-scoped kind, so a namespace-scoped install cannot read it. There, the volume fix says the agent’s role may not read StorageClasses and that a cluster admin expands the claim by hand, rather than guessing whether the class allows it.

Every cluster-wide posture can read PriorityClasses

Section titled “Every cluster-wide posture can read PriorityClasses”

From agent 0.2.18, the ClusterRole adds get and list on scheduling.k8s.io priorityclasses. Nightshift reads them before it proposes a capacity buffer: whether the buffer’s own PriorityClass exists below every workload, and which class is the cluster’s default. Seylo never creates or changes a PriorityClass. A PriorityClass is cluster-scoped, so a namespace-scoped install cannot read it, and the buffer is refused there.

Every cluster-wide posture can read your node autoscaler

Section titled “Every cluster-wide posture can read your node autoscaler”

From agent 0.2.21, the ClusterRole adds get, list and watch on Karpenter NodePools and NodeClaims (karpenter.sh), AKS node auto-provisioning AKSNodeClasses (karpenter.azure.com), EKS EC2NodeClasses (karpenter.k8s.aws), EKS Auto Mode node classes (eks.amazonaws.com) and GKE ComputeClasses (cloud.google.com). The Savings tab reads them to say which autoscaler manages your nodes. None holds a Secret value. An EKS node class carries a node bootstrap script; Seylo reads named fields only and never returns it. They are cluster-scoped, so a namespace-scoped install cannot read them.

Every posture can read budgets, Jobs, quotas and certificates

Section titled “Every posture can read budgets, Jobs, quotas and certificates”

From agent 0.2.22, every posture adds read-only grants, get and list only, including read-only:

Resource API group Where it is granted
poddisruptionbudgets policy The ClusterRole, and the Role in each watched namespace
jobs, cronjobs batch The ClusterRole, and the Role in each watched namespace
resourcequotas core ("") The ClusterRole, and the Role in each watched namespace
certificates, certificaterequests, issuers cert-manager.io The ClusterRole, and the Role in each watched namespace
clusterissuers cert-manager.io The ClusterRole only
orders, challenges acme.cert-manager.io The ClusterRole, and the Role in each watched namespace

Nightshift reads them to find a PodDisruptionBudget that blocks a drain, a CronJob that missed its schedule or keeps failing, a namespace at its ResourceQuota, and a certificate that is not ready. None of them carries a Secret value, and the agent writes none of them with these grants. cert-manager’s objects are read only where cert-manager is installed.

Before 0.2.22, Jobs, CronJobs and quotas were readable only where tier 1 is on (the default). On an older agent, the PodDisruptionBudget and certificate reads are refused: Nightshift says so, and names the upgrade. See upgrade the agent.

An agent that can edit Roles can widen its own permissions. That is a decision no preset should make on your behalf, so it is never included in a posture — admin included. If you need it, you turn it on deliberately, and you should read the warning below first.

  • pods/attach appears in no grant this chart can produce, and there is no interactive shell.
  • pods/exec appears in no posture. The one way to add it is the separate chart value sre.exec.enabled=true, off by default, which lets Nightshift, Seylo’s AI SRE, run a fixed list of read-only commands — and only once a Seylo admin also opts the cluster in. See commands in containers.

Secret values are never returned by Seylo, at any setting. That guarantee is enforced in code, not by RBAC, because the default grant does include Secrets. See the next section.

Tier 1 (rbac.write.tier1.enabled, on by default) grants get, list, create, update, patch and delete on core secrets.

In the default cluster scope that is every Secret in every namespace, kube-system included, not only the namespaces Seylo manages. The rule is part of a ClusterRole bound cluster-wide. With rbac.scope=namespaced it is a Role in each watched namespace and nowhere else.

Seylo uses it to:

  • set and rotate Secret values;
  • apply Secrets in releases, provisioning and a namespace copy;
  • carry a Secret value over inside your cluster during a copy;
  • show the key names of Secrets in the environment view.

Values are never shown. The agent strips them before anything is serialised onto the wire, and the server strips them again. See how secrets are handled. But the agent’s token itself can read a Secret, so anyone able to replace the agent’s image could read every Secret that token reaches.

For a regulated cluster, install namespace-scoped, so the token reaches Secrets only in the namespaces you list:

Terminal window
--set rbac.scope=namespaced --set 'watchNamespaces={prod,staging}'

Namespace scoping has one real consequence for metrics. See cluster-wide vs namespace-scoped.

To give the token no Secret rule from tier 1 at all, set rbac.write.tier1.enabled=false, or choose the read-only or deploy posture, which leave tier 1 off. Setting, rotating, applying and carrying over Secrets then stop working, and you manage Secrets outside Seylo.

Both are off by default.

Destructive (tier 3). Permits deleting PersistentVolumeClaims — which deletes the real disk when the bound volume’s reclaim policy is Delete — PersistentVolumes, StorageClasses, Namespaces (cascading to everything inside), CustomResourceDefinitions (deleting every custom resource of that type) and VolumeSnapshots.

The portal additionally requires the object’s exact name to be typed and the consequences acknowledged, and shows the reclaim policy, the cascade contents and the custom-resource count first. Those are guardrails on someone who already holds the permission. Leaving it off is the control that does not depend on anybody reading a dialog.

Turn it on with rbac.write.destructive.enabled=true, or with the admin posture, which includes it. One Nightshift fix needs it: recreating a volume claim whose StorageClass does not exist, so that a Pending pod can start. That fix touches only a claim that never bound and holds no data, always waits for an approver who types the claim’s name, and never runs unattended. Without the destructive tier, the delete is refused, the old claim is left exactly as it was, and the fix names this value. The default role creates and patches claims but never deletes them, and that default does not change. See a claim whose StorageClass does not exist.

RBAC (tier 4). With it the agent can write Roles, RoleBindings, ClusterRoles and ClusterRoleBindings — including ones that widen its own permissions.

Node autoscaler (rbac.write.autoscaler.enabled). From agent 0.2.21, off by default, and in no posture, admin included: whether Seylo may change which machines your cluster buys is a choice you make for managed savings, lever by lever, not one a preset makes for you. It grants create, update and patch on Karpenter NodePools, AKSNodeClasses, EC2NodeClasses and ComputeClasses. These objects decide spot or on-demand, machine families, CPU and memory ceilings and how eagerly nodes are consolidated, so a wrong value can leave pods Pending or put workloads on spot capacity. It never grants delete: deleting a NodePool drains every node it created at once, and the agent refuses such a delete itself, whatever else allows it. No cloud credential is involved; your cluster’s own autoscaler creates and removes the machines.

The agent writes a custom resource only if you name it. rbac.write.tier1.customResources is a list of {apiGroup, resources} entries, empty by default. Empty means the agent writes no custom resources.

Each entry grants get, list, watch, create, update, patch and delete on exactly the listed resources of exactly that group. The chart’s own values file carries this recommended starting list, for teams that edit these kinds through Seylo:

rbac:
write:
tier1:
customResources:
- apiGroup: cert-manager.io
resources: [certificates]
- apiGroup: external-secrets.io
resources: [externalsecrets]
- apiGroup: monitoring.coreos.com
resources: [servicemonitors, podmonitors]
- apiGroup: keda.sh
resources: [scaledobjects]
- apiGroup: networking.istio.io
resources: [virtualservices]
- apiGroup: argoproj.io
resources: [rollouts]

Keep only the entries you use.

SecretProviderClasses are not on this list. Their create and patch is a rule of its own in tier 1. See what a namespace copy needs.

helm install, helm upgrade and helm template refuse to render, so nothing reaches the cluster, when the list contains:

Entry Helm’s error includes
The old value true rbac.write.tier1.customResources is now a LIST of {apiGroup, resources[]} entries, not true.
apiGroup: "" (the core group) rbac.write.tier1.customResources[0].apiGroup is empty. The empty group is the CORE group (pods, secrets, namespaces...), whose kinds the tiers grant narrowly.
apiGroup: "*" rbac.write.tier1.customResources[0].apiGroup is "*": "*" matches every API group, the core group and RBAC included, which is the wildcard this list replaced.
A built-in group, such as apps rbac.write.tier1.customResources[0].apiGroup "apps" is a built-in Kubernetes group.
"*" in resources rbac.write.tier1.customResources[0].resources contains "*" for cert-manager.io: "*" (or an empty name) would cover every resource of the group, including ones installed later.
An entry with no resources rbac.write.tier1.customResources[0].resources must be a non-empty list of resource plurals of cert-manager.io, such as [certificates].

The [0] is the position of the offending entry, and the group is yours. Each message goes on to say what to write instead. For true it ends:

Name what the agent may write, for example in a values file: rbac.write.tier1.customResources: [{apiGroup: cert-manager.io, resources: [certificates]}]; or --set rbac.write.tier1.customResources=null to grant none.

A "*" anywhere in the group name is refused, not only on its own.

The built-in groups refused are apps, batch, rbac.authorization.k8s.io, apiextensions.k8s.io, admissionregistration.k8s.io, storage.k8s.io, certificates.k8s.io, coordination.k8s.io, node.k8s.io, scheduling.k8s.io, policy, networking.k8s.io, autoscaling, discovery.k8s.io, flowcontrol.apiserver.k8s.io, apiregistration.k8s.io, events.k8s.io, metrics.k8s.io, authentication.k8s.io, authorization.k8s.io, resource.k8s.io, internal.apiserver.k8s.io, storagemigration.k8s.io, core and v1. Their kinds are granted, narrowly, by the tiers.

The list is checked even while tier 1 is off, so a refused entry cannot wait in a values file for someone to switch tier 1 back on.

When Seylo refuses a custom-resource write

Section titled “When Seylo refuses a custom-resource write”

A write to a custom resource that is not on the list is refused. The message begins PERMISSION MISSING, not an empty or failed result:, names the verb and the resource, and names the exact entry to add. For a KEDA ScaledObject it includes:

scaledobjects.keda.sh is a custom resource, which the agent may write only when it is listed: add {apiGroup: keda.sh, resources: [scaledobjects]} to rbac.write.tier1.customResources in the agent's Helm values (helm upgrade --reuse-values -f <values file>). A "*" entry is refused; name the resource.

Add that entry to your values file and run the upgrade. Use helm upgrade seylo-agent oci://ghcr.io/seylo-inc/charts/seylo-agent --namespace seylo-agent --reset-then-reuse-values -f <values file> rather than the --reuse-values the message names: --reuse-values also keeps the old chart’s defaults, which can make an upgrade fail. On Helm older than 3.14, see upgrade the agent. Nothing was changed by the refused write.

Cluster-wide Namespace-scoped
Binding A ClusterRole + ClusterRoleBinding for the base grant and for each tier that is on A Role + RoleBinding for the base grant and each tier that is on, in each managed namespace
Blast radius Every namespace in the cluster Exactly the listed namespaces, enforced by Kubernetes
Namespace discovery The agent lists namespaces itself Impossible — watchNamespaces is required
Create namespace / namespace wizard Works Does not work
Auditing it yourself Read the agent’s ClusterRoles, one per tier: kubectl get clusterrole -l app.kubernetes.io/name=seylo-agent kubectl get rolebinding -A | grep agent lists the complete reachable set
Choose it when You own the cluster and want Seylo to onboard namespaces Someone else owns the cluster and wants a hard, provable boundary

Seylo additionally enforces the scope itself: a command naming a namespace outside the watched list is refused before any API call. So even a compromised Seylo server cannot widen the blast radius beyond what you configured — defence in depth on top of Kubernetes RBAC.

A namespace list on a cluster-wide install

Section titled “A namespace list on a cluster-wide install”

A cluster-wide install has no namespace list by default. Seylo’s namespace policy, which you change in the portal, then decides where Seylo acts. A new or copied namespace needs no helm upgrade.

You can still fix a list into a cluster-wide install. On the cluster’s Agent details page, tick Also limit the agent to a fixed list of namespaces, inside the cluster, or set watchNamespaces with Helm. The agent intersects what Seylo asks for with that list. Seylo cannot widen the list remotely, by design. A mistaken or compromised Seylo server can narrow what the agent does, but never extend it past what you installed.

The trade-off is a helm upgrade for every namespace outside the list. See a fixed namespace list on a cluster-wide install.

From agent 0.2.6, the agent reads each namespace in its list and reports only the ones that actually exist. A listed namespace that does not exist yet is not shown as present. A namespace copy creates it, or you can remove it from the list. Where the agent’s grant cannot read a Namespace object, as with a namespace-scoped install, it still reports the name, and Seylo treats it as unknown rather than present.

The one real consequence of namespace scoping

Section titled “The one real consequence of namespace scoping”

A namespace-scoped install cannot read Nodes, so the agent cannot scrape the kubelet itself. Without a metrics store in the cluster, the default kubelet metrics source therefore collects nothing for that cluster.

Node is a cluster-scoped object. A Role cannot grant nodes, nodes/proxy or nodes/metrics at all — not as a configuration choice, but because Kubernetes RBAC does not permit a namespaced Role to reference a cluster-scoped resource. No amount of editing the manifest changes this.

Still works, in full: inventory, the dashboard matrix, drift and environment comparison, history, releases, image patches, rollout restarts, manifest edits, ConfigMap and env edits, rollout status and follow-through, dry-run validation, approvals, audit, scheduled deployments, and pod-failure alerts (those come from events and pod status, both namespaced).

Does not work without a metrics store: node-proxy metrics — so no CPU, memory, network I/O or CPU throttling figures, and therefore no history and no right-sizing recommendations. These are reported as unknown, never as zero; the Insights page names the reason rather than drawing an empty chart.

Also unavailable, whatever you install: creating and listing namespaces, and the namespace wizard.

What to do instead, in order of preference

Section titled “What to do instead, in order of preference”
  1. Install the seylo-observability chart and run an agent later than 0.2.4. On the default kubelet source, such an agent reads usage from the chart’s metrics store instead of scraping the kubelet, so a namespace-scoped agent gets CPU, memory, network and throttling, with history, and its own Role stays exactly as narrow as it was. The store still answers only for the agent’s namespaces. Restart counts and OOM kills come from the Kubernetes API as before. Agents 0.2.4 and earlier keep scraping the kubelet, so they gain nothing from the store here. Admin → Integrations → Observability stores shows the install command.

    The trade-off: the chart’s metrics store scrapes the kubelet with its own read-only ClusterRole on nodes. The agent does not gain that permission, but the cluster owner has to install the chart, and it holds cluster-wide read on nodes. The chart is also what managed vCPU is measured from.

  2. Set the cluster’s metrics source to prometheus and give the agent an in-cluster Prometheus URL. Best coverage — CPU, memory, network and throttling with history — and it needs no cluster-scoped RBAC. The agent scrapes it from inside your cluster; Seylo never connects to it.

  3. Set the source to metrics-server. Works with namespaced RBAC for pod metrics, so CPU and memory work. There is no network I/O or throttling from this source for anyone, namespaced or not.

  4. Grant the cluster-scoped variant, if the cluster owner is willing. This is the only way for the agent to scrape the kubelet itself, and it is also what the namespace wizard needs.

  5. Accept metrics-free management. Everything in the “still works” list is unaffected. This is a perfectly reasonable posture for a cluster whose owner wants a provable, namespace-shaped boundary.

The chart prints all of this in its helm install output when you choose namespace scoping.

The base grant covers only what releases, environment edits, restarts and metrics need. Browsing arbitrary kinds — StatefulSets, DaemonSets, CronJobs, Jobs and any CRD — needs broad read, so it is a deliberate opt-in, off by default. With it off, the Resources page says exactly which permission is missing rather than showing an empty list.

Kubernetes RBAC has no deny rule and cannot express “everything except secrets”, so there are only two honest choices:

Mode What it grants Trade-off
wildcard (default) get/list/watch on everything, including CRDs installed later Includes secrets, in every namespace the binding reaches. Seylo’s refusal to expose values is enforced in code, not by RBAC
enumerated The core group without secrets, plus the API groups you list This block adds no Secret read. Cost: a CRD group must be listed before it can be browsed

Both modes are read-only. No write verb is ever granted by this block.

A namespace copy from an agent-managed cluster reads and writes through the agent. From agent 0.2.6, the default grants cover everything a copy writes:

  • tier 1 writes Deployments, StatefulSets, DaemonSets, CronJobs, Jobs, Services, Ingresses, NetworkPolicies, PodDisruptionBudgets, ConfigMaps, Secrets, ServiceAccounts and HPAs;
  • the base grant reads SecretProviderClasses (secrets-store.csi.x-k8s.io) and PersistentVolumeClaims, and tier 1 creates and patches them, through a rule of its own. It never deletes either. Deleting a claim can delete the real disk, so that stays in the destructive tier.

These are two named kinds, not a broad grant. rbac.genericRead stays the separate opt-in it was, and a copy does not need it. Nor does it need rbac.write.destructive or anything in customResources.

Two cases still stop a copy that mounts a SecretProviderClass or a claim:

Case What happens Fix
An agent older than 0.2.6 Its default grant cannot read either kind, so the plan blocks the workload and names the upgrade Upgrade the agent to 0.2.6
Posture read-only or deploy, or rbac.write.tier1.enabled=false The agent reads them but cannot create them, because tier 1 is off Turn tier 1 on, create the object in the target yourself, or skip the workload

On a copy between two agent-managed clusters, the target’s agent does the writing, so the target needs 0.2.6 too.

A write the agent cannot perform is not refused opaquely. The error names the missing verb, the resource, and the exact setting that turns it on. Seylo also reads the agent’s real grant back and shows it in the portal, so you can see what the agent may do before something is refused rather than after.