Troubleshooting the agent

The agent runs a preflight on startup and after a failed reconnect, isolating the failing layer — config → DNS → TCP/proxy → TLS — and printing one actionable line per failure.

Failures above the transport (the HTTP upgrade, enrollment, authentication, protocol version) are classified from the real connection attempt, so nothing is probed twice and a single-use enrollment token is never consumed just to produce a diagnostic.

Start here, always

kubectl -n seylo-agent logs deploy/seylo-agent | grep preflight

A failing agent prints its diagnosis before the first reconnect line, and every reconnect line then carries the layer:

ERROR [agent] preflight FAIL [tls-unknown-ca] The server's TLS certificate for
      portal.seylo.in is not signed by a CA this agent trusts: mount your
      internal CA bundle and set AGENT_CA_FILE to it ...
INFO  [agent] reconnect attempt 3 in 4211ms - last failure [tls-unknown-ca]
      at the tls layer

The code in brackets is stable and is the index for the tables below. The layer tells you which half of the stack to stop looking at.

config — the agent never reached the network

Code	Symptom	Fix
`config-server-url-missing`	Pod restarts; `SERVER_URL is required`	`--set serverUrl=wss://<host>/agent/connect`
`config-server-url-malformed`	Scheme is not `ws`/`wss`/`http`/`https`, or no host — often a leftover shell quote	Use `wss://<host>/agent/connect`. `https://` is accepted and upgraded automatically
`config-server-url-insecure-scheme`	Warning. Resolves to `ws://` for a remote host	The enrollment token and every command would cross the network in the clear. Use `wss://`
`config-cluster-id-missing`	No cluster ID	Copy the id, not the display name, from the Clusters page
`config-no-credential`	No stored credential and no enrollment token	Re-issue a token and `helm upgrade --set enrollmentToken=…`. If it enrolled before, also find out why persistence failed
`config-proxy-url-malformed`	A proxy variable is set but unusable, so the agent went direct	Use `http://[user:pass@]host:port`. Only `CONNECT` proxies are supported. The value is never echoed — it may hold a password
`config-ca-file-unreadable`	The CA path is not in the container	Check the CA Secret is mounted and the key name matches
`config-tls-insecure`	Warning. TLS verification is off	Confirm your diagnosis, then set a CA file and remove the insecure flag

dns / tcp / proxy — the network path

Code	Symptom	Fix
`dns-server-unresolved`	`getaddrinfo ENOTFOUND`	Check the name, and that CoreDNS can resolve it. An internal-only name needs a record or a stub zone
`dns-proxy-unresolved`	The proxy’s hostname does not resolve in-cluster	Fix the proxy URL, or use the proxy’s IP address
`tcp-refused`	`ECONNREFUSED` — something answered and said no	Confirm the port (443 for `wss://`) and that the reverse proxy is listening
`tcp-timeout`	`ETIMEDOUT`, no answer. The classic blocked-egress signature	Allow egress to the host on that port, or configure a forward proxy. See restricted networks
`tcp-unreachable`	`EHOSTUNREACH` / `ENETUNREACH`	Check egress NetworkPolicies and the cluster’s NAT and routes
`proxy-connect-refused`	The proxy itself cannot be reached	Check host and port, and that the pod may reach the proxy
`proxy-auth-required`	The proxy answered 407	Put credentials in the proxy URL, preferably via an existing Secret
`proxy-upstream-refused`	Proxy reached, but it could not open the tunnel (502/503/504)	The proxy’s allow-list must permit `CONNECT` to that host on that port
`proxy-connect-rejected`	The proxy refused `CONNECT`, commonly 403	Many proxies allow `CONNECT` only to 443. Confirm the port and that the host is allow-listed

tls

Code	Symptom	Fix
`tls-unknown-ca`	`self-signed certificate`, `unable to verify the first certificate`. Also what a TLS-inspecting proxy looks like	Mount your internal CA bundle and point the agent at it. See restricted networks
`tls-hostname-mismatch`	`ERR_TLS_CERT_ALTNAME_INVALID`	Use the exact hostname the certificate names. A cert for the ingress host will not match an IP or an internal alias
`tls-cert-expired`	`CERT_HAS_EXPIRED`, or not yet valid	Renew it on the reverse proxy — the agent reconnects by itself. “Not yet valid” almost always means the node’s clock is wrong
`tls-handshake-failed`	Anything else TLS-shaped	Check the cipher and TLS-version policy on the reverse proxy

http — TLS worked, the WebSocket upgrade did not

Code	Symptom	Fix
`http-upgrade-not-forwarded`	`Unexpected server response: 200`. Almost always a reverse proxy not forwarding `Upgrade`	Set `proxy_http_version 1.1`, `proxy_set_header Upgrade $http_upgrade`, `proxy_set_header Connection "upgrade"`
`http-endpoint-not-found`	`404`	The path must be exactly `/agent/connect`, and the ingress must route it to the backend
`http-forbidden`	`401`/`403` before the WebSocket exists	A WAF, ingress auth, or an mTLS requirement in front of Seylo. The gateway never authenticates at the HTTP layer — it authenticates inside the WebSocket
`http-bad-gateway`	`502`/`503`/`504`	The reverse proxy is up; the backend behind it is not reachable
`http-unexpected-status`	Any status other than `101`	Compare with a manual upgrade request from the same pod

auth / protocol — the transport is fine

Stop looking at the network.

Code	Symptom	Fix
`auth-enrollment-token-invalid`	The gateway rejected the token	Re-issue one for this cluster ID
`auth-enrollment-token-used`	Tokens are single-use	Re-issue. If it enrolled before and lost the credential, fix persistence — the credential Secret’s RBAC, or a writable volume rather than an `emptyDir`
`auth-enrollment-token-expired`	Older than the enrollment TTL	Re-issue and apply promptly
`auth-invalid-credential`	The stored credential is not valid for this agent	Delete the credential Secret and re-enroll with a fresh token
`auth-agent-revoked`	An administrator revoked it	Delete the credential Secret, re-issue a token, re-install
`auth-unknown-cluster`	Seylo knows no such cluster ID	Copy the ID exactly as shown on the Clusters page
`auth-cluster-not-agent-managed`	The cluster exists but its connection method is not the agent	Switch it on the Clusters page
`auth-handshake-timeout`	Socket opened, handshake never finished	Usually a proxy buffering the connection. Turn buffering off and raise the read timeout for `/agent/connect`
`auth-not-authenticated`	A frame arrived before authentication	Version skew — upgrade the agent image
`protocol-version-unsupported`	Agent newer than the server, or below the minimum	Install an agent in the server’s supported range, or upgrade the server first. An agent newer than the server cannot be negotiated down
`server-internal-error`	Rejected for another reason	The message is verbatim from the server
`unknown-failure`	Not attributable to a layer	Reproduce from a debug pod in the same namespace to establish whether it is the agent or the cluster’s egress

Symptoms that are not preflight failures

Symptom	Cause	Fix
`Cannot list namespaces cluster-wide` warning, empty matrix	Namespace-scoped RBAC with no namespace list	Set the watched namespaces to the ones whose RoleBindings exist
`FAILED to persist the agent credential`	The credential Secret’s RBAC is missing, or the credential path is read-only	Keep the shipped credential `Role`/`RoleBinding`, or point it at a writable volume
Healthy agent, but no CPU/memory/network figures	A namespace-scoped install cannot read Nodes, so the default `kubelet` metrics source collects nothing	The namespaced-install limits
An action is refused naming a verb and a resource	The posture does not grant it	That is the design — the message names the setting that turns it on. Postures

Still stuck

Collect these three before asking for help — they are what anyone will ask for:

kubectl -n seylo-agent logs deploy/seylo-agent --tail=100
kubectl -n seylo-agent get pods -o wide
kubectl -n seylo-agent get deploy seylo-agent -o yaml

The agent log never contains a token or a Secret value, so it is safe to share.