Troubleshooting the agent
The agent runs a preflight on startup and after a failed reconnect, isolating the failing layer — config → DNS → TCP/proxy → TLS — and printing one actionable line per failure.
Failures above the transport (the HTTP upgrade, enrollment, authentication, protocol version) are classified from the real connection attempt, so nothing is probed twice and a single-use enrollment token is never consumed just to produce a diagnostic.
Start here, always
Section titled “Start here, always”kubectl -n seylo-agent logs deploy/seylo-agent | grep preflightA failing agent prints its diagnosis before the first reconnect line, and every reconnect line then carries the layer:
ERROR [agent] preflight FAIL [tls-unknown-ca] The server's TLS certificate for agent.seylo.in is not signed by a CA this agent trusts: mount your internal CA bundle and set AGENT_CA_FILE to it ...INFO [agent] reconnect attempt 3 in 4211ms - last failure [tls-unknown-ca] at the tls layerThe code in brackets is stable and is the index for the tables below. The layer tells you which half of the stack to stop looking at.
config — the agent never reached the network
Section titled “config — the agent never reached the network”| Code | Symptom | Fix |
|---|---|---|
config-server-url-missing |
Pod restarts; SERVER_URL is required |
--set serverUrl=wss://<host>/agent/connect |
config-server-url-malformed |
Scheme is not ws/wss/http/https, or no host — often a leftover shell quote |
Use wss://<host>/agent/connect. https:// is accepted and upgraded automatically |
config-server-url-insecure-scheme |
Warning. Resolves to ws:// for a remote host |
The enrollment token and every command would cross the network in the clear. Use wss:// |
config-cluster-id-missing |
No cluster ID | Copy the id, not the display name, from the Clusters page |
config-no-credential |
No stored credential and no enrollment token | Re-issue a token and helm upgrade --set enrollmentToken=…. If it enrolled before, also find out why persistence failed |
config-proxy-url-malformed |
A proxy variable is set but unusable, so the agent went direct | Use http://[user:pass@]host:port. Only CONNECT proxies are supported. The value is never echoed — it may hold a password |
config-ca-file-unreadable |
The CA path is not in the container | Check the CA Secret is mounted and the key name matches |
config-tls-insecure |
Warning. TLS verification is off | Confirm your diagnosis, then set a CA file and remove the insecure flag |
dns / tcp / proxy — the network path
Section titled “dns / tcp / proxy — the network path”| Code | Symptom | Fix |
|---|---|---|
dns-server-unresolved |
getaddrinfo ENOTFOUND |
Check the name, and that CoreDNS can resolve it. An internal-only name needs a record or a stub zone |
dns-proxy-unresolved |
The proxy’s hostname does not resolve in-cluster | Fix the proxy URL, or use the proxy’s IP address |
tcp-refused |
ECONNREFUSED — something answered and said no |
Confirm the port (443 for wss://) and that the reverse proxy is listening |
tcp-timeout |
ETIMEDOUT, no answer. The classic blocked-egress signature |
Allow egress to the host on that port (agent.seylo.in:443 for a new install on Seylo’s hosted service), or configure a forward proxy. See restricted networks |
tcp-unreachable |
EHOSTUNREACH / ENETUNREACH |
Check egress NetworkPolicies and the cluster’s NAT and routes |
proxy-connect-refused |
The proxy itself cannot be reached | Check host and port, and that the pod may reach the proxy |
proxy-auth-required |
The proxy answered 407 | Put credentials in the proxy URL, preferably via an existing Secret |
proxy-upstream-refused |
Proxy reached, but it could not open the tunnel (502/503/504) | The proxy’s allow-list must permit CONNECT to that host on that port |
proxy-connect-rejected |
The proxy refused CONNECT, commonly 403 |
Many proxies allow CONNECT only to 443. Confirm the port and that the host is allow-listed |
| Code | Symptom | Fix |
|---|---|---|
tls-unknown-ca |
self-signed certificate, unable to verify the first certificate. Also what a TLS-inspecting proxy looks like |
Mount your internal CA bundle and point the agent at it. See restricted networks |
tls-hostname-mismatch |
ERR_TLS_CERT_ALTNAME_INVALID |
Use the exact hostname the certificate names. A cert for the ingress host will not match an IP or an internal alias |
tls-cert-expired |
CERT_HAS_EXPIRED, or not yet valid |
Renew it on the reverse proxy — the agent reconnects by itself. “Not yet valid” almost always means the node’s clock is wrong |
tls-handshake-failed |
Anything else TLS-shaped | Check the cipher and TLS-version policy on the reverse proxy |
http — TLS worked, the WebSocket upgrade did not
Section titled “http — TLS worked, the WebSocket upgrade did not”| Code | Symptom | Fix |
|---|---|---|
http-upgrade-not-forwarded |
Unexpected server response: 200. Almost always a reverse proxy not forwarding Upgrade |
Set proxy_http_version 1.1, proxy_set_header Upgrade $http_upgrade, proxy_set_header Connection "upgrade" |
http-endpoint-not-found |
404 |
The path must be exactly /agent/connect, and the ingress must route it to the backend |
http-forbidden |
401/403 before the WebSocket exists |
A WAF, ingress auth, or an mTLS requirement in front of Seylo. The gateway never authenticates at the HTTP layer — it authenticates inside the WebSocket |
http-bad-gateway |
502/503/504 |
The reverse proxy is up; the backend behind it is not reachable |
http-unexpected-status |
Any status other than 101 |
Compare with a manual upgrade request from the same pod |
auth / protocol — the transport is fine
Section titled “auth / protocol — the transport is fine”Stop looking at the network.
| Code | Symptom | Fix |
|---|---|---|
auth-enrollment-token-invalid |
The gateway rejected the token | Re-issue one for this cluster ID |
auth-enrollment-token-used |
Tokens are single-use | Re-issue. If it enrolled before and lost the credential, fix persistence — the credential Secret’s RBAC, or a writable volume rather than an emptyDir |
auth-enrollment-token-expired |
Older than the enrollment TTL | Re-issue and apply promptly |
auth-invalid-credential |
The stored credential is not valid for this agent | Delete the credential Secret and re-enroll with a fresh token |
auth-agent-revoked |
An administrator revoked it | Delete the credential Secret, re-issue a token, re-install |
auth-unknown-cluster |
Seylo knows no such cluster ID | Copy the ID exactly as shown on the Clusters page |
auth-cluster-not-agent-managed |
The cluster exists but its connection method is not the agent | Switch it on the Clusters page |
auth-handshake-timeout |
Socket opened, handshake never finished | Usually a proxy buffering the connection. Turn buffering off and raise the read timeout for /agent/connect |
auth-not-authenticated |
A frame arrived before authentication | Version skew — upgrade the agent image |
protocol-version-unsupported |
Agent newer than the server, or below the minimum | Install an agent in the server’s supported range, or upgrade the server first. An agent newer than the server cannot be negotiated down |
server-internal-error |
Rejected for another reason | The message is verbatim from the server |
unknown-failure |
Not attributable to a layer | Reproduce from a debug pod in the same namespace to establish whether it is the agent or the cluster’s egress |
An agent whose credential is refused keeps trying until it gets a new one, because only you can fix the credential. From agent 0.2.19 it waits up to a minute between those attempts. Seylo’s side slows an agent that keeps presenting a refused credential, whatever its version: each repeated refusal is answered later than the last, up to 5 minutes, so an agent that retries within seconds does little harm to anyone. The wait starts again after 30 quiet minutes, or once the agent connects.
A note on the cluster’s page. While an agent pod keeps presenting a
revoked credential, or one Seylo does not accept, the cluster’s agent
status says so, even when another agent for that cluster is connected. It
names the pod and its agent version, how many times it was refused and when
last, and the two ways out: re-enroll it (issue an enrollment token and run
the install command from the cluster’s Agent page) or remove it
(helm uninstall of that release in its namespace). The note goes away 30
minutes after the last refusal.
A cluster that has been offline for a while. Seylo waits a short time for an agent that has just dropped to reconnect before it answers a page or a command. For a cluster whose agent has been gone longer, it does not wait: the page answers at once that the agent has been offline since the time it was last heard from, instead of hanging.
When you add or change a cluster
Section titled “When you add or change a cluster”| What you see | What it means and what to do |
|---|---|
Connection method "<method>" is not available on this server: it connects only through the Seylo agent ... |
Seylo connects to clusters only through the in-cluster agent. Choose In-cluster agent and install the agent. A cluster saved earlier with another method shows the same message wherever Seylo would reach it: edit it on Clusters and switch it to the agent. See connection methods |
Registry type "acr" needs an Azure Container Registry host (*.azurecr.io) (or "gar" and a Google host) |
The registry type does not match the host you entered |
Symptoms that are not preflight failures
Section titled “Symptoms that are not preflight failures”| Symptom | Cause | Fix |
|---|---|---|
The agent stopped connecting right after you moved it to agent.seylo.in; the log shows tcp-timeout, proxy-connect-rejected or proxy-upstream-refused |
Your egress rules or proxy allow the portal’s host but not the agent host | Allow agent.seylo.in:443, or move the agent back with --set serverUrl=wss://portal.seylo.in/agent/connect. See move an agent |
Clusters shows the agent reconnecting often (the Agent: Connected tooltip counts reconnects in the last 24 hours), and it dials portal.seylo.in |
Long-lived connections through the portal’s host are closed from time to time on the way, and each close is a reconnect | Move it to agent.seylo.in, the host for agent connections. See move an agent |
Opening https://agent.seylo.in/agent/connect in a browser or with curl answers HTTP 426, /agent/connect accepts only a WebSocket upgrade from a Seylo agent. |
Expected. Only the agent connects there, with a WebSocket upgrade | Nothing to fix. To test the path from inside the cluster, read the agent’s own log |
Cannot list namespaces cluster-wide warning, empty matrix |
Namespace-scoped RBAC with no namespace list | Set the watched namespaces to the ones whose RoleBindings exist |
FAILED to persist the agent credential |
The credential Secret’s RBAC is missing, or the credential path is read-only | Keep the shipped credential Role/RoleBinding, or point it at a writable volume |
| Healthy agent, but no CPU/memory/network figures | A namespace-scoped install cannot read Nodes, so without a metrics store the default kubelet metrics source collects nothing |
Install the seylo-observability chart with an agent later than 0.2.4, or another option in the namespaced-install limits |
| An action is refused naming a verb and a resource | The posture does not grant it | That is the design — the message names the setting that turns it on. Postures |
… is a custom resource, which the agent may write only when it is listed: add {apiGroup: …, resources: […]} to rbac.write.tier1.customResources |
The agent writes only the custom resources you list, and this one is not listed | Add the entry the message names to your values file and helm upgrade. See custom resources |
helm upgrade fails with rbac.write.tier1.customResources is now a LIST of {apiGroup, resources[]} entries, not true. |
The install carries customResources: true, which 0.2.6 refuses |
Upgrade to 0.2.7, which accepts it and grants no custom-resource writes; or add --set rbac.write.tier1.customResources=null. See upgrading the agent |
helm fails on rbac.write.tier1.customResources[…], with is a built-in Kubernetes group, "*" matches every API group, is empty or "*" (or an empty name) |
An entry in the custom-resource list is refused | Name one custom group and each resource plural. See what the chart refuses |
Messages on other pages
Section titled “Messages on other pages”A message about a feature rather than the connection — the agent being too old
for it, the Datadog connector being off, no OpenCost figures, or Helm’s
nil pointer evaluating on upgrade — is in
messages you may see, with the command
that fixes each.
Still stuck
Section titled “Still stuck”Collect these three before asking for help — they are what anyone will ask for:
kubectl -n seylo-agent logs deploy/seylo-agent --tail=100kubectl -n seylo-agent get pods -o widekubectl -n seylo-agent get deploy seylo-agent -o yamlThe agent log never contains a token or a Secret value, so it is safe to share.