Troubleshooting observability
Start at Admin → Integrations → Observability stores. Pick the cluster and
press Check again.
Most problems show up there first, as a store’s State or Namespace
filter. In the messages below, <cluster>, <url> and similar placeholders
stand for what you will see in their place.
The metrics store’s memory refusal on long ranges, the Datadog connector being
off, an agent too old for a feature, no OpenCost figures, and Helm’s
nil pointer evaluating on upgrade are in
messages you may see.
Observability stores: State
Section titled “Observability stores: State”| You see | What it means | What to do |
|---|---|---|
not installed, with Not installed: http://seylo-vm.seylo-observability.svc:8428 does not resolve in this cluster. (or seylo-vl / seylo-vt) |
The agent looked for the chart’s Service and it does not exist | Install the chart. If you installed it in another namespace, set the agent’s observability.namespace to that namespace |
not installed, with no address |
The agent was installed with observability.enabled=false |
Upgrade the agent with --set observability.enabled=true |
not installed, with The API server answered 404: no reachable seylo-vm Service. (or 503) |
A cluster Seylo reaches directly, without an agent. Seylo looks only in the namespace seylo-observability |
Install the chart into seylo-observability |
unreachable, with The store could not be reached at <url>: fetch failed (ECONNREFUSED) or another code |
The Service exists, but nothing answers on it | kubectl -n seylo-observability get pods. Look for a store pod that is Pending or restarting (see in the cluster) |
unreachable, with The store did not answer within 5s (<url>/health). |
The store is reachable but slow, or the traffic is being dropped | Check the store pod’s resources and logs. If the agent is not in the namespace seylo-agent, or has different labels, the NetworkPolicy blocks it. Set agent.namespace / agent.podLabels on the chart |
unreachable on a cluster reached directly |
The chart’s NetworkPolicy admits only the agent. The API server’s service proxy is not a pod the policy can name | Set networkPolicy.enabled=false, or list the API server’s addresses in networkPolicy.extraIngressFrom |
unreachable, with HTTP <status>: … |
Something answered the health check with an error | The address may point at the wrong Service. Check the agent’s observability.*Url values |
Keeps data for: not reported |
The store answered, but its flags did not include a retention period | Usually a store you run yourself. Seylo shows nothing rather than a guess |
Disk cap: none set |
The store has no disk cap | On the chart, the caps are always set. On your own store, set one |
Observability stores: Namespace filter
Section titled “Observability stores: Namespace filter”These matter for anyone whose view is limited to some namespaces, and for everyone when the agent is namespace-scoped. See namespace-scoped installs.
| You see | What it means | What to do |
|---|---|---|
namespace filter enforced |
The store keeps to the namespaces it is given | Nothing |
no data yet to test the filter |
The store has nothing to compare yet. Common straight after install, and for traces until a service sends some | Wait for data, then Check again |
namespace filter NOT enforced |
The store returned data for a namespace that does not exist when asked to filter. Limited readers are refused | You are probably pointing the agent at your own store, and it is older or behind a proxy that drops query arguments. Use the chart’s store, or fix the path to yours |
filter check failed |
The test itself failed. Hover for the reason | Fix the reason (usually the store is unreachable), then Check again |
Metrics, Logs, Traces and APM pages
Section titled “Metrics, Logs, Traces and APM pages”| Message begins | Cause | What to do |
|---|---|---|
Your access is limited to some namespaces of this cluster, and <signal> cannot be shown to you yet: |
Your Seylo access is namespace-limited, and the filter for that signal is not proved | See the Namespace filter verdict above. Someone who can see every namespace can still query |
This cluster's agent is installed for some namespaces only, and Seylo shows <signal> through it only once the store is proved to keep to those namespaces. |
The agent is namespace-scoped, and the filter is not proved | As above. This applies to everyone on that cluster |
You can see no namespaces in this cluster, so there is nothing to query. |
Your grants and the cluster’s read policy leave no namespace | Ask an administrator for access to a namespace |
The agent on "<cluster>" is version <v>, which cannot query observability stores. Agent 0.2.0 adds it |
The agent predates observability | Upgrade the agent. Use 0.2.6 |
The agent on "<cluster>" is version <v>, which reads traces only through VictoriaTraces' Jaeger API |
Agent 0.2.0 cannot filter traces by namespace | Upgrade the agent. 0.2.1 or later reads traces in a way that can be filtered |
This agent was installed with observability turned off (observability.enabled=false), or from an older agent This agent has no metrics store configured (or logs / traces) |
As it says. The message is followed by an Open Admin > Integrations link, whichever agent version sent it | Upgrade the agent with --set observability.enabled=true, then check the stores on Admin → Integrations → Observability stores |
No <signal> store is installed: <url> does not resolve in this cluster. |
The chart is not installed where the agent looks | Install the chart |
"<cluster>" is a demo cluster with no in-cluster stores |
Demo clusters have no stores | Use a real cluster with the chart |
The store did not answer within 30s |
A query took longer than the agent waits | Narrow the time range or the label filters |
The metrics response was larger than Seylo relays (4 MiB). |
Too many series or points | Aggregate with sum by (…), add label filters, or use a larger step |
That range at step <n>s is <m> points per series; at most 11000. |
The step is too small for the range | Use the step the message suggests |
VictoriaMetrics refused the query (HTTP 4xx): (or VictoriaLogs / VictoriaTraces) |
The store rejected the query, usually a syntax error | Fix the query. The rest of the message is the store’s own reason. Start from the query catalogue |
VictoriaMetrics answered with something that is not JSON; the store URL may point at the wrong service. |
The agent’s observability.metricsUrl points at something other than VictoriaMetrics |
Correct the URL, or remove it to use the chart’s store |
No span metrics found (looked for traces_span_metrics_calls and traces_span_metrics_duration_*). |
APM has nothing to read. No service sends traces yet | Send traces to the chart’s collector |
… has no unit suffix and its bucket boundaries do not say whether they are milliseconds or seconds |
Span metrics from a collector other than the chart’s, with an unknown unit | APM shows no latency rather than one that may be wrong by a factor of 1000. Use the chart’s collector |
No trace <id> between … and now |
Not in the time range, past retention, or not in your namespaces | Widen the range. Check retention.period on Admin → Integrations → Observability stores |
Dashboards
Section titled “Dashboards”| You see | Meaning |
|---|---|
Data found in <cluster>. |
Every metric the template reads was seen in the last hour |
Not seen in <cluster> in the last hour: <metrics>. |
Some of its metrics are missing. Install or annotate what the template names |
None of its metrics were seen in <cluster> in the last hour (or they are in namespaces you cannot see). |
None of its metrics were seen. “Not seen” can mean “not visible to you”, never “absent” |
This template needs data the seylo-observability chart does not collect. |
The template needs an exporter. See templates that need an exporter |
| Network panels empty | Chart 0.2.4 or earlier. Upgrade to 0.2.6 |
| Node panels empty on Kubernetes cluster overview or Cluster nodes | Node series carry no namespace. They show only for people who see every namespace, on a cluster-wide agent |
| Requests panels empty | No service sends OTLP traces, or spans could not be matched to a pod in your namespaces |
| Annotated pods missing | The annotations are on the workload’s metadata, not the pod template, or a NetworkPolicy in your namespace blocks the scrape. See the scrape annotations |
Usage and managed vCPU
Section titled “Usage and managed vCPU”| You see | Cause | What to do |
|---|---|---|
Agent warning the metrics store at <url> could not supply usage (<reason>); reading the kubelet directly, and trying the store again in 10 minutes. |
The agent fell back from the store. The reason says why, for example no container CPU series for running pods |
Fix the store (see Admin → Integrations → Observability stores). On a namespace-scoped agent, the fallback cannot work, so usage is unknown until the store answers |
Warning beginning Cannot list nodes, so the kubelet metrics source cannot be scraped: |
A namespace-scoped agent with no usable store | Install the chart and run agent 0.2.5 or later. See how usage is collected |
Usage shows n/a for network and throttling |
The source cannot report them (metrics-server), or chart 0.2.4 or earlier | Use the default kubelet source with chart and agent 0.2.5 or later |
Managed vCPU: Unmeasured, because the metrics store could not be queried: … |
No chart, the store is unreachable, or the agent was offline that day | Fix what the message names. Unmeasured days are left out, never estimated |
Managed vCPU: Unmeasured, because the metrics store has no kube_pod_container_resource_requests series for this cluster. |
kubeStateMetrics.enabled is false, or it has not been scraped yet |
Turn kube-state-metrics on (the default) |
In the cluster
Section titled “In the cluster”These come from Kubernetes, not Seylo. Search for the quoted part.
| You see | Cause | What to do |
|---|---|---|
vlagent pods not created, with violates PodSecurity |
The namespace enforces the baseline or restricted Pod Security level, which refuses vlagent’s hostPath volumes |
Allow privileged for the chart’s namespace, or accept no logs. See prerequisites |
A store pod Pending, with unbound immediate PersistentVolumeClaims |
No default StorageClass, or it cannot provision ReadWriteOnce |
Set storageClassName, or fix the cluster’s default |
helm upgrade fails with updates to statefulset spec for fields other than |
You changed a volume size or storageClassName |
See growing a volume |
ImagePullBackOff on a chart pod |
Nodes cannot reach Docker Hub or registry.k8s.io |
Mirror the images and set *.image.repository. See images |
| VictoriaMetrics stops taking new data while queries still answer | Free space on its volume fell below metrics.minFreeDiskSpace |
Grow the volume or shorten retention.period |
Agent connection problems, which stop every page above, are covered in agent troubleshooting.