Skip to content

Troubleshooting the agent

The agent runs a preflight on startup and after a failed reconnect, isolating the failing layer — config → DNS → TCP/proxy → TLS — and printing one actionable line per failure.

Failures above the transport (the HTTP upgrade, enrollment, authentication, protocol version) are classified from the real connection attempt, so nothing is probed twice and a single-use enrollment token is never consumed just to produce a diagnostic.

Terminal window
kubectl -n seylo-agent logs deploy/seylo-agent | grep preflight

A failing agent prints its diagnosis before the first reconnect line, and every reconnect line then carries the layer:

ERROR [agent] preflight FAIL [tls-unknown-ca] The server's TLS certificate for
agent.seylo.in is not signed by a CA this agent trusts: mount your
internal CA bundle and set AGENT_CA_FILE to it ...
INFO [agent] reconnect attempt 3 in 4211ms - last failure [tls-unknown-ca]
at the tls layer

The code in brackets is stable and is the index for the tables below. The layer tells you which half of the stack to stop looking at.

config — the agent never reached the network

Section titled “config — the agent never reached the network”
Code Symptom Fix
config-server-url-missing Pod restarts; SERVER_URL is required --set serverUrl=wss://<host>/agent/connect
config-server-url-malformed Scheme is not ws/wss/http/https, or no host — often a leftover shell quote Use wss://<host>/agent/connect. https:// is accepted and upgraded automatically
config-server-url-insecure-scheme Warning. Resolves to ws:// for a remote host The enrollment token and every command would cross the network in the clear. Use wss://
config-cluster-id-missing No cluster ID Copy the id, not the display name, from the Clusters page
config-no-credential No stored credential and no enrollment token Re-issue a token and helm upgrade --set enrollmentToken=…. If it enrolled before, also find out why persistence failed
config-proxy-url-malformed A proxy variable is set but unusable, so the agent went direct Use http://[user:pass@]host:port. Only CONNECT proxies are supported. The value is never echoed — it may hold a password
config-ca-file-unreadable The CA path is not in the container Check the CA Secret is mounted and the key name matches
config-tls-insecure Warning. TLS verification is off Confirm your diagnosis, then set a CA file and remove the insecure flag
Code Symptom Fix
dns-server-unresolved getaddrinfo ENOTFOUND Check the name, and that CoreDNS can resolve it. An internal-only name needs a record or a stub zone
dns-proxy-unresolved The proxy’s hostname does not resolve in-cluster Fix the proxy URL, or use the proxy’s IP address
tcp-refused ECONNREFUSED — something answered and said no Confirm the port (443 for wss://) and that the reverse proxy is listening
tcp-timeout ETIMEDOUT, no answer. The classic blocked-egress signature Allow egress to the host on that port (agent.seylo.in:443 for a new install on Seylo’s hosted service), or configure a forward proxy. See restricted networks
tcp-unreachable EHOSTUNREACH / ENETUNREACH Check egress NetworkPolicies and the cluster’s NAT and routes
proxy-connect-refused The proxy itself cannot be reached Check host and port, and that the pod may reach the proxy
proxy-auth-required The proxy answered 407 Put credentials in the proxy URL, preferably via an existing Secret
proxy-upstream-refused Proxy reached, but it could not open the tunnel (502/503/504) The proxy’s allow-list must permit CONNECT to that host on that port
proxy-connect-rejected The proxy refused CONNECT, commonly 403 Many proxies allow CONNECT only to 443. Confirm the port and that the host is allow-listed
Code Symptom Fix
tls-unknown-ca self-signed certificate, unable to verify the first certificate. Also what a TLS-inspecting proxy looks like Mount your internal CA bundle and point the agent at it. See restricted networks
tls-hostname-mismatch ERR_TLS_CERT_ALTNAME_INVALID Use the exact hostname the certificate names. A cert for the ingress host will not match an IP or an internal alias
tls-cert-expired CERT_HAS_EXPIRED, or not yet valid Renew it on the reverse proxy — the agent reconnects by itself. “Not yet valid” almost always means the node’s clock is wrong
tls-handshake-failed Anything else TLS-shaped Check the cipher and TLS-version policy on the reverse proxy

http — TLS worked, the WebSocket upgrade did not

Section titled “http — TLS worked, the WebSocket upgrade did not”
Code Symptom Fix
http-upgrade-not-forwarded Unexpected server response: 200. Almost always a reverse proxy not forwarding Upgrade Set proxy_http_version 1.1, proxy_set_header Upgrade $http_upgrade, proxy_set_header Connection "upgrade"
http-endpoint-not-found 404 The path must be exactly /agent/connect, and the ingress must route it to the backend
http-forbidden 401/403 before the WebSocket exists A WAF, ingress auth, or an mTLS requirement in front of Seylo. The gateway never authenticates at the HTTP layer — it authenticates inside the WebSocket
http-bad-gateway 502/503/504 The reverse proxy is up; the backend behind it is not reachable
http-unexpected-status Any status other than 101 Compare with a manual upgrade request from the same pod

Stop looking at the network.

Code Symptom Fix
auth-enrollment-token-invalid The gateway rejected the token Re-issue one for this cluster ID
auth-enrollment-token-used Tokens are single-use Re-issue. If it enrolled before and lost the credential, fix persistence — the credential Secret’s RBAC, or a writable volume rather than an emptyDir
auth-enrollment-token-expired Older than the enrollment TTL Re-issue and apply promptly
auth-invalid-credential The stored credential is not valid for this agent Delete the credential Secret and re-enroll with a fresh token
auth-agent-revoked An administrator revoked it Delete the credential Secret, re-issue a token, re-install
auth-unknown-cluster Seylo knows no such cluster ID Copy the ID exactly as shown on the Clusters page
auth-cluster-not-agent-managed The cluster exists but its connection method is not the agent Switch it on the Clusters page
auth-handshake-timeout Socket opened, handshake never finished Usually a proxy buffering the connection. Turn buffering off and raise the read timeout for /agent/connect
auth-not-authenticated A frame arrived before authentication Version skew — upgrade the agent image
protocol-version-unsupported Agent newer than the server, or below the minimum Install an agent in the server’s supported range, or upgrade the server first. An agent newer than the server cannot be negotiated down
server-internal-error Rejected for another reason The message is verbatim from the server
unknown-failure Not attributable to a layer Reproduce from a debug pod in the same namespace to establish whether it is the agent or the cluster’s egress

An agent whose credential is refused keeps trying until it gets a new one, because only you can fix the credential. From agent 0.2.19 it waits up to a minute between those attempts. Seylo’s side slows an agent that keeps presenting a refused credential, whatever its version: each repeated refusal is answered later than the last, up to 5 minutes, so an agent that retries within seconds does little harm to anyone. The wait starts again after 30 quiet minutes, or once the agent connects.

A note on the cluster’s page. While an agent pod keeps presenting a revoked credential, or one Seylo does not accept, the cluster’s agent status says so, even when another agent for that cluster is connected. It names the pod and its agent version, how many times it was refused and when last, and the two ways out: re-enroll it (issue an enrollment token and run the install command from the cluster’s Agent page) or remove it (helm uninstall of that release in its namespace). The note goes away 30 minutes after the last refusal.

A cluster that has been offline for a while. Seylo waits a short time for an agent that has just dropped to reconnect before it answers a page or a command. For a cluster whose agent has been gone longer, it does not wait: the page answers at once that the agent has been offline since the time it was last heard from, instead of hanging.

What you see What it means and what to do
Connection method "<method>" is not available on this server: it connects only through the Seylo agent ... Seylo connects to clusters only through the in-cluster agent. Choose In-cluster agent and install the agent. A cluster saved earlier with another method shows the same message wherever Seylo would reach it: edit it on Clusters and switch it to the agent. See connection methods
Registry type "acr" needs an Azure Container Registry host (*.azurecr.io) (or "gar" and a Google host) The registry type does not match the host you entered
Symptom Cause Fix
The agent stopped connecting right after you moved it to agent.seylo.in; the log shows tcp-timeout, proxy-connect-rejected or proxy-upstream-refused Your egress rules or proxy allow the portal’s host but not the agent host Allow agent.seylo.in:443, or move the agent back with --set serverUrl=wss://portal.seylo.in/agent/connect. See move an agent
Clusters shows the agent reconnecting often (the Agent: Connected tooltip counts reconnects in the last 24 hours), and it dials portal.seylo.in Long-lived connections through the portal’s host are closed from time to time on the way, and each close is a reconnect Move it to agent.seylo.in, the host for agent connections. See move an agent
Opening https://agent.seylo.in/agent/connect in a browser or with curl answers HTTP 426, /agent/connect accepts only a WebSocket upgrade from a Seylo agent. Expected. Only the agent connects there, with a WebSocket upgrade Nothing to fix. To test the path from inside the cluster, read the agent’s own log
Cannot list namespaces cluster-wide warning, empty matrix Namespace-scoped RBAC with no namespace list Set the watched namespaces to the ones whose RoleBindings exist
FAILED to persist the agent credential The credential Secret’s RBAC is missing, or the credential path is read-only Keep the shipped credential Role/RoleBinding, or point it at a writable volume
Healthy agent, but no CPU/memory/network figures A namespace-scoped install cannot read Nodes, so without a metrics store the default kubelet metrics source collects nothing Install the seylo-observability chart with an agent later than 0.2.4, or another option in the namespaced-install limits
An action is refused naming a verb and a resource The posture does not grant it That is the design — the message names the setting that turns it on. Postures
… is a custom resource, which the agent may write only when it is listed: add {apiGroup: …, resources: […]} to rbac.write.tier1.customResources The agent writes only the custom resources you list, and this one is not listed Add the entry the message names to your values file and helm upgrade. See custom resources
helm upgrade fails with rbac.write.tier1.customResources is now a LIST of {apiGroup, resources[]} entries, not true. The install carries customResources: true, which 0.2.6 refuses Upgrade to 0.2.7, which accepts it and grants no custom-resource writes; or add --set rbac.write.tier1.customResources=null. See upgrading the agent
helm fails on rbac.write.tier1.customResources[…], with is a built-in Kubernetes group, "*" matches every API group, is empty or "*" (or an empty name) An entry in the custom-resource list is refused Name one custom group and each resource plural. See what the chart refuses

A message about a feature rather than the connection — the agent being too old for it, the Datadog connector being off, no OpenCost figures, or Helm’s nil pointer evaluating on upgrade — is in messages you may see, with the command that fixes each.

Collect these three before asking for help — they are what anyone will ask for:

Terminal window
kubectl -n seylo-agent logs deploy/seylo-agent --tail=100
kubectl -n seylo-agent get pods -o wide
kubectl -n seylo-agent get deploy seylo-agent -o yaml

The agent log never contains a token or a Secret value, so it is safe to share.