Skip to content

Commands in containers (opt-in)

By default the Seylo agent never execs into or attaches to a container, and its RBAC holds no pods/exec or pods/attach. That stays true for every cluster you do not opt in.

Some failures cannot be explained from the Kubernetes API alone: is the config file there, does the Service name resolve, is the port actually listening. For those, a cluster admin can let Nightshift, Seylo’s AI SRE, run a fixed list of read-only commands inside a container while it investigates an alert. It is opt-in per cluster, and it takes two switches.

Switch Who sets it What it does
The agent’s chart value sre.exec.enabled=true Whoever runs Helm on your cluster The only thing that adds pods/exec (verbs create and get) to the agent’s RBAC — cluster-wide, or in each watched namespace for a namespace-scoped install — and tells the agent it may run commands. Default false
Allow commands in containers in Seylo An admin in Seylo Lets Nightshift offer commands for this cluster. Confirmed in a dialog and recorded in the audit log with your name

Seylo cannot set the chart value: it holds no credential for your cluster. It reads it back from the agent and shows both halves side by side, so “on in Seylo, off in the cluster” is a visible state, not a silent one.

Nothing else is widened. The agent still has no pods/attach and no port-forward, and a namespace-scoped agent still cannot reach a namespace outside its install.

You need agent 0.2.10 or later. An older agent is reported as needs 0.2.10, never as a switch that silently does nothing.

  1. On your cluster, turn on the chart value (Helm 3.14 or later):

    Terminal window
    helm upgrade seylo-agent oci://ghcr.io/seylo-inc/charts/seylo-agent \
    --namespace seylo-agent --reset-then-reuse-values \
    --set sre.exec.enabled=true

    On Helm older than 3.14, save your values with helm get values seylo-agent -n seylo-agent -o yaml > mine.yaml and pass -f mine.yaml instead of --reset-then-reuse-values.

  2. In Seylo, open Admin → Clusters, the cluster’s Agent install / status, and the Commands in containers card. Choose Allow commands in containers, then Allow in the dialog.

  3. Check the card. It shows:

    Field Example
    in Seylo on, by admin@acme.example at ...
    in the cluster (sre.exec.enabled) true
    agent version 0.2.22 (needs 0.2.10)

    and the badge reads On.

You can do the two steps in either order. If they disagree, the card says so and shows the exact helm upgrade command that brings the cluster’s half into line:

Badge Meaning
Off Seylo’s switch is off. Off. Nightshift troubleshoots through Kubernetes API reads only (no exec, no attach).
On in Seylo only On in Seylo, but OFF in the cluster: the agent's install has sre.exec.enabled=false, so it refuses every command.
On On. Nightshift may run its fixed list of read-only commands in this cluster's containers.

A cluster installed from the plain manifest (kubectl apply) has no chart value to set. To opt it in you would add the pods/exec rule and the agent’s environment variable by hand; moving the install to Helm is simpler.

A cluster Seylo reaches directly (without the agent) cannot be opted in: commands run only through the agent.

Either switch alone turns it off.

  • In Seylo: Stop allowing commands in containers on the same card. The agent’s RBAC still holds pods/exec until you also change the chart value; the card then shows the command that takes it away:

    Terminal window
    helm upgrade seylo-agent oci://ghcr.io/seylo-inc/charts/seylo-agent \
    --namespace seylo-agent --reset-then-reuse-values \
    --set sre.exec.enabled=false
  • On the cluster: the command above removes pods/exec from the RBAC, and the agent refuses every command from then on, whatever Seylo’s switch says.

This is the whole list. The model sends a command’s name and its typed arguments; the agent builds the command line from this table. Nothing runs through a shell, so a pipe, a redirect or a second command cannot be smuggled in an argument.

Name Runs Argument
cat-file cat <path> An absolute path
ls ls -la <path> An absolute path
env-names env, reduced to variable names None
df df -h None
ps ps -ef None
ss ss -tln None
netstat netstat -tln, for images without ss None
nslookup nslookup <host> An in-cluster name
getent-hosts getent hosts <host> An in-cluster name
http-status curl -sS -m 5 -o /dev/null -w %{http_code} <url>: the HTTP status only, body discarded An in-cluster URL

Each command has 10 seconds and 16 KiB of output (4 KiB of error output), and a run may make at most 6 of them. The command must also exist in the container’s image; a minimal image without ss or curl answers with the error it gets.

Paths must be absolute, at most 256 characters, made of letters, digits and . _ / @ + - only (no wildcards, spaces or shell characters), and may not climb with ... Never read or listed:

  • anything under /var/run/secrets, /run/secrets, /root, /home or /dev, and /etc/shadow, /etc/gshadow, /etc/sudoers;
  • anything under /proc except /proc/meminfo, /proc/loadavg, /proc/cpuinfo, /proc/uptime, /proc/mounts and /proc/pressure — a process’s environment and command line can hold secrets;
  • anything under /sys except /sys/fs/cgroup (limits and pressure);
  • any path with a part whose name suggests a credential: secret, passw, token, credential, private, apikey, key and certificate files (.pem, .key, .p12, .pfx, .jks, .keystore), SSH keys, .ssh, .kube, kubeconfig, .aws, .azure, .gcloud, .docker, .netrc, .pgpass, htpasswd, .env files.

Names and URLs must be inside the cluster: a bare Service name (redis), a name ending .svc, .svc.cluster.local or .cluster.local, kubernetes.default, or localhost. A name outside the cluster is refused, because a looked-up name is a way out of it. A URL takes the form http(s)://<in-cluster host>[:port][/path], with no credentials, query string or fragment, for example http://api.shop.svc:8080/healthz.

A refused command is recorded as a refused step, with the reason, like any other refused read.

The agent, not only the server. Before a byte reaches pods/exec, the agent:

  1. refuses everything if its own chart value is off, with a message naming the Helm value;
  2. refuses a namespace outside its install’s scope;
  3. checks the command and its arguments again against the same list, with the same rules.

A buggy or compromised Seylo server could send any request; the agent refuses it on its own.

  • Output is redacted in the agent before it is sent: private keys, passwords in URLs, JWTs, Bearer/Basic tokens, cloud access keys, key=value pairs whose key names a credential, and long opaque strings are replaced with [redacted]. The rule is deliberately blunt: a false positive costs the model one fact, a false negative would cost you a credential.
  • Seylo’s server redacts it again before the model or the run’s record sees it.
  • env-names returns names only. The values never leave the pod.
  • The agent never logs output. Its log line names the command, the pod and the exit code.
  • The audit log records each change to the Seylo switch as cluster-sre-exec: who, when, on or off, and what the agent reported about its own chart value at that moment.
  • Each investigation lists every command as a step, marked with a terminal icon, with its arguments and its result (ok, refused or failed). The header counts them: (… reads, … model calls, 2 command(s) in containers).
  • The panel says, every run, whether commands were available, for example No commands in containers: A cluster admin has not turned on "Allow commands in containers" for this cluster (Clusters page), so only Kubernetes API reads are used.

There is still no interactive shell, for anyone, and no attach.