Skip to content

Troubleshooting observability

Start at Admin → Integrations → Observability stores. Pick the cluster and press Check again. Most problems show up there first, as a store’s State or Namespace filter. In the messages below, <cluster>, <url> and similar placeholders stand for what you will see in their place.

The metrics store’s memory refusal on long ranges, the Datadog connector being off, an agent too old for a feature, no OpenCost figures, and Helm’s nil pointer evaluating on upgrade are in messages you may see.

You see What it means What to do
not installed, with Not installed: http://seylo-vm.seylo-observability.svc:8428 does not resolve in this cluster. (or seylo-vl / seylo-vt) The agent looked for the chart’s Service and it does not exist Install the chart. If you installed it in another namespace, set the agent’s observability.namespace to that namespace
not installed, with no address The agent was installed with observability.enabled=false Upgrade the agent with --set observability.enabled=true
not installed, with The API server answered 404: no reachable seylo-vm Service. (or 503) A cluster Seylo reaches directly, without an agent. Seylo looks only in the namespace seylo-observability Install the chart into seylo-observability
unreachable, with The store could not be reached at <url>: fetch failed (ECONNREFUSED) or another code The Service exists, but nothing answers on it kubectl -n seylo-observability get pods. Look for a store pod that is Pending or restarting (see in the cluster)
unreachable, with The store did not answer within 5s (<url>/health). The store is reachable but slow, or the traffic is being dropped Check the store pod’s resources and logs. If the agent is not in the namespace seylo-agent, or has different labels, the NetworkPolicy blocks it. Set agent.namespace / agent.podLabels on the chart
unreachable on a cluster reached directly The chart’s NetworkPolicy admits only the agent. The API server’s service proxy is not a pod the policy can name Set networkPolicy.enabled=false, or list the API server’s addresses in networkPolicy.extraIngressFrom
unreachable, with HTTP <status>: … Something answered the health check with an error The address may point at the wrong Service. Check the agent’s observability.*Url values
Keeps data for: not reported The store answered, but its flags did not include a retention period Usually a store you run yourself. Seylo shows nothing rather than a guess
Disk cap: none set The store has no disk cap On the chart, the caps are always set. On your own store, set one

These matter for anyone whose view is limited to some namespaces, and for everyone when the agent is namespace-scoped. See namespace-scoped installs.

You see What it means What to do
namespace filter enforced The store keeps to the namespaces it is given Nothing
no data yet to test the filter The store has nothing to compare yet. Common straight after install, and for traces until a service sends some Wait for data, then Check again
namespace filter NOT enforced The store returned data for a namespace that does not exist when asked to filter. Limited readers are refused You are probably pointing the agent at your own store, and it is older or behind a proxy that drops query arguments. Use the chart’s store, or fix the path to yours
filter check failed The test itself failed. Hover for the reason Fix the reason (usually the store is unreachable), then Check again
Message begins Cause What to do
Your access is limited to some namespaces of this cluster, and <signal> cannot be shown to you yet: Your Seylo access is namespace-limited, and the filter for that signal is not proved See the Namespace filter verdict above. Someone who can see every namespace can still query
This cluster's agent is installed for some namespaces only, and Seylo shows <signal> through it only once the store is proved to keep to those namespaces. The agent is namespace-scoped, and the filter is not proved As above. This applies to everyone on that cluster
You can see no namespaces in this cluster, so there is nothing to query. Your grants and the cluster’s read policy leave no namespace Ask an administrator for access to a namespace
The agent on "<cluster>" is version <v>, which cannot query observability stores. Agent 0.2.0 adds it The agent predates observability Upgrade the agent. Use 0.2.6
The agent on "<cluster>" is version <v>, which reads traces only through VictoriaTraces' Jaeger API Agent 0.2.0 cannot filter traces by namespace Upgrade the agent. 0.2.1 or later reads traces in a way that can be filtered
This agent was installed with observability turned off (observability.enabled=false), or from an older agent This agent has no metrics store configured (or logs / traces) As it says. The message is followed by an Open Admin > Integrations link, whichever agent version sent it Upgrade the agent with --set observability.enabled=true, then check the stores on Admin → Integrations → Observability stores
No <signal> store is installed: <url> does not resolve in this cluster. The chart is not installed where the agent looks Install the chart
"<cluster>" is a demo cluster with no in-cluster stores Demo clusters have no stores Use a real cluster with the chart
The store did not answer within 30s A query took longer than the agent waits Narrow the time range or the label filters
The metrics response was larger than Seylo relays (4 MiB). Too many series or points Aggregate with sum by (…), add label filters, or use a larger step
That range at step <n>s is <m> points per series; at most 11000. The step is too small for the range Use the step the message suggests
VictoriaMetrics refused the query (HTTP 4xx): (or VictoriaLogs / VictoriaTraces) The store rejected the query, usually a syntax error Fix the query. The rest of the message is the store’s own reason. Start from the query catalogue
VictoriaMetrics answered with something that is not JSON; the store URL may point at the wrong service. The agent’s observability.metricsUrl points at something other than VictoriaMetrics Correct the URL, or remove it to use the chart’s store
No span metrics found (looked for traces_span_metrics_calls and traces_span_metrics_duration_*). APM has nothing to read. No service sends traces yet Send traces to the chart’s collector
… has no unit suffix and its bucket boundaries do not say whether they are milliseconds or seconds Span metrics from a collector other than the chart’s, with an unknown unit APM shows no latency rather than one that may be wrong by a factor of 1000. Use the chart’s collector
No trace <id> between … and now Not in the time range, past retention, or not in your namespaces Widen the range. Check retention.period on Admin → Integrations → Observability stores
You see Meaning
Data found in <cluster>. Every metric the template reads was seen in the last hour
Not seen in <cluster> in the last hour: <metrics>. Some of its metrics are missing. Install or annotate what the template names
None of its metrics were seen in <cluster> in the last hour (or they are in namespaces you cannot see). None of its metrics were seen. “Not seen” can mean “not visible to you”, never “absent”
This template needs data the seylo-observability chart does not collect. The template needs an exporter. See templates that need an exporter
Network panels empty Chart 0.2.4 or earlier. Upgrade to 0.2.6
Node panels empty on Kubernetes cluster overview or Cluster nodes Node series carry no namespace. They show only for people who see every namespace, on a cluster-wide agent
Requests panels empty No service sends OTLP traces, or spans could not be matched to a pod in your namespaces
Annotated pods missing The annotations are on the workload’s metadata, not the pod template, or a NetworkPolicy in your namespace blocks the scrape. See the scrape annotations
You see Cause What to do
Agent warning the metrics store at <url> could not supply usage (<reason>); reading the kubelet directly, and trying the store again in 10 minutes. The agent fell back from the store. The reason says why, for example no container CPU series for running pods Fix the store (see Admin → Integrations → Observability stores). On a namespace-scoped agent, the fallback cannot work, so usage is unknown until the store answers
Warning beginning Cannot list nodes, so the kubelet metrics source cannot be scraped: A namespace-scoped agent with no usable store Install the chart and run agent 0.2.5 or later. See how usage is collected
Usage shows n/a for network and throttling The source cannot report them (metrics-server), or chart 0.2.4 or earlier Use the default kubelet source with chart and agent 0.2.5 or later
Managed vCPU: Unmeasured, because the metrics store could not be queried: … No chart, the store is unreachable, or the agent was offline that day Fix what the message names. Unmeasured days are left out, never estimated
Managed vCPU: Unmeasured, because the metrics store has no kube_pod_container_resource_requests series for this cluster. kubeStateMetrics.enabled is false, or it has not been scraped yet Turn kube-state-metrics on (the default)

These come from Kubernetes, not Seylo. Search for the quoted part.

You see Cause What to do
vlagent pods not created, with violates PodSecurity The namespace enforces the baseline or restricted Pod Security level, which refuses vlagent’s hostPath volumes Allow privileged for the chart’s namespace, or accept no logs. See prerequisites
A store pod Pending, with unbound immediate PersistentVolumeClaims No default StorageClass, or it cannot provision ReadWriteOnce Set storageClassName, or fix the cluster’s default
helm upgrade fails with updates to statefulset spec for fields other than You changed a volume size or storageClassName See growing a volume
ImagePullBackOff on a chart pod Nodes cannot reach Docker Hub or registry.k8s.io Mirror the images and set *.image.repository. See images
VictoriaMetrics stops taking new data while queries still answer Free space on its volume fell below metrics.minFreeDiskSpace Grow the volume or shorten retention.period

Agent connection problems, which stop every page above, are covered in agent troubleshooting.