Skip to content

Messages you may see

Each entry starts with the message as Seylo (or Helm) shows it, so you can find it by pasting it into search. <cluster>, <n> and <version> stand for what you see in their place.

The commands assume the install’s default namespaces, seylo-agent and seylo-observability. If you installed into others, change --namespace to match.

The metrics store could not hold this query in memory

Section titled “The metrics store could not hold this query in memory”
The metrics store could not hold this query in memory: it covers <n> series over the chosen range.
Long ranges see every pod that existed in them, and pods that come and go (CronJobs, restarts,
rollouts) each add series. Pick a shorter range, or give the store more memory (it is capped at
1 GiB by default): helm upgrade seylo-observability ...

Where: a Metrics query or a dashboard panel over a long time range.

Why: a query over a range has to hold, at once, every series that existed in that range. Every pod that ever ran counts — each CronJob run and each restart gets a new pod name — so the number grows with the range, not with how many pods run now. VictoriaMetrics refuses a query that does not fit in its memory limit, which the chart sets to 1 GiB.

Seylo already splits any range longer than two days into three-day windows and joins the answers, so each window holds only the pods alive in it. You see this message only when a single window is still too big for the store, which is rare.

What to do — either:

  • Pick a shorter range, or narrow the query with label filters (a namespace, a workload); or

  • Give the store more memory:

    Terminal window
    helm upgrade seylo-observability oci://ghcr.io/seylo-inc/charts/seylo-observability \
    --version 0.2.22 --namespace seylo-observability --reset-then-reuse-values \
    --set metrics.resources.limits.memory=2Gi

    The node the store runs on needs that much memory free.

The Datadog connector is off on this agent

Section titled “The Datadog connector is off on this agent”
The Datadog connector is off on this agent (datadog.enabled=false).

Where: Admin → Integrations → Datadog, or anything that reads from Datadog.

Why: the agent reads Datadog only when its chart turns the connector on, and it is off by default. The keys stay in a Secret in your cluster; Seylo never receives them.

What to do: create the Secret, then turn the connector on. Run both yourself (the full walk-through is on compare with Datadog):

Terminal window
kubectl -n seylo-agent create secret generic seylo-datadog \
--from-literal=DD_API_KEY=<api key> \
--from-literal=DD_APP_KEY=<application key>
Terminal window
helm upgrade seylo-agent oci://ghcr.io/seylo-inc/charts/seylo-agent \
--namespace seylo-agent --reset-then-reuse-values \
--set datadog.enabled=true \
--set datadog.site=datadoghq.com \
--set datadog.clusterName=<kube_cluster_name> \
--set datadog.existingSecret=seylo-datadog

Use your own Datadog site (datadoghq.eu, us3.datadoghq.com, …) and the cluster name your Datadog Agent reports. A scoped application key needs only the timeseries_query permission. The connector needs agent 0.2.10 or later; if the agent is older, upgrade it first (next entry).

The agent on "<cluster>" is version <version>, which has no Datadog connector.
Upgrade to agent 0.2.10 or later (helm upgrade seylo-agent ... --reset-then-reuse-values), then set datadog.enabled=true.

or, for the observability pages:

The agent on "<cluster>" is version <version>, which cannot query observability stores.

On Admin → Clusters the same thing shows as a chip on the cluster’s row, for example Agent 0.2.10 required; hover it for what does not work until you upgrade.

Why: Seylo keeps working with older agents and says which feature needs a newer one, rather than showing empty charts. Everything the older agent could do keeps working.

What to do: upgrade the agent. No new enrollment token is needed:

Terminal window
helm upgrade seylo-agent oci://ghcr.io/seylo-inc/charts/seylo-agent \
--version 0.2.22 --namespace seylo-agent --reset-then-reuse-values \
--set image.tag=0.2.22 \
--set image.digest=sha256:ed9222f8b6e0beb0fb12873388439ce270304bc5b54dc41a80a96707c552661f

The digest is the signed 0.2.22 image. The tag and digest are passed explicitly because an install that set them keeps the old ones otherwise. See upgrade the agent for the details and verify what you install to check the signature first.

Where: the Cost tab of FinOps → Cost & savings: Where the money went (titled No cost overview for this cluster) and the cost table (No cost by namespace for this cluster).

There are two causes, and the card says which one applies.

1. OpenCost is not on. The card says:

OpenCost is not running in this cluster. Turn it on with:

followed by the command:

Terminal window
helm upgrade seylo-observability oci://ghcr.io/seylo-inc/charts/seylo-observability --version 0.2.22 --namespace seylo-observability --reset-then-reuse-values --set opencost.enabled=true

The first figures appear a few minutes after its pod is Ready (kubectl -n seylo-observability get pods). The agent must also have observability.enabled=true.

2. Seylo reads this cluster for some namespaces only. The card says:

Seylo reads this cluster's store for some namespaces only (<namespaces>), and OpenCost's node prices
belong to no namespace, so the namespace filter hides them even when OpenCost is running.

Cost allocation needs the whole cluster. Edit the cluster in Admin → Clusters and set What this tool can see to The whole cluster. If the agent itself was installed for some namespaces only, reinstall it for the whole cluster from the cluster’s Agent details page (Admin → Clusters, the cluster’s menu).

The card is titled Cost by namespace needs agent 0.2.11 and the 0.2.11 observability chart (or Cost overview needs … on Where the money went), and says one of:

The agent on "<cluster>" is version <version>; Seylo's cost figures come from OpenCost's API, which needs agent 0.2.11 and seylo-observability chart 0.2.11 or later. Run both upgrades on that cluster, in this order:
The agent could not reach OpenCost: the seylo-observability chart on this cluster predates 0.2.11, whose NetworkPolicy closes OpenCost's port to everything but VictoriaMetrics. Run both upgrades on that cluster, in this order:

Where: the Cost tab of FinOps → Cost & savings.

Why: Seylo’s cost figures ask OpenCost’s own API through the agent. That needs agent 0.2.11, and seylo-observability 0.2.11, whose NetworkPolicy lets the agent reach OpenCost.

What to do: run both commands the card shows, in that order. They are also on chargeback.

A browser set to refuse site data (strict privacy settings, cleared site data, some managed browser profiles) used to stop Seylo before the sign-in screen appeared, with an error such as The operation is insecure.

Nothing to do now: Seylo signs in and works in such a browser. It only cannot remember small preferences between visits, such as which panes you folded or which organisation you signed in to last. If you use more than one organisation, open Seylo with your organisation’s own link (/t/<your organisation>) so you land in the right one.

<n> automatic investigations in the last 24 hours, at the organisation's daily cap of <n>. Press Investigate to run one by hand, or ask for a higher limit under Help > Contact support.

or ... in the last hour, at the organisation's limit of <n> per hour. ..., or ... on cluster <cluster> in the last hour, at the limit of <n> per cluster per hour. ..., or, for the same alert:

Nightshift already investigated this alert automatically at <time>; it starts again for the same alert after 6 hours. Press Investigate to run it now.

Where: on an alert, where Nightshift says why it did not start.

Why: automatic investigations are bounded per organisation. The defaults are 40 a day, 10 an hour, 4 per cluster an hour, and the same alert at most once every 6 hours. See how often it starts on its own.

What to do: press Investigate on the alert to run one now; the limits apply only to automatic starts. If your organisation needs more, ask through Help → Contact support: see ask for higher Nightshift limits. The 6-hour wait for the same alert does not change.

Not recreated: the agent’s role may not delete PersistentVolumeClaims

Section titled “Not recreated: the agent’s role may not delete PersistentVolumeClaims”
Not recreated: the agent's role may not delete PersistentVolumeClaims, so claim <name> was left untouched and nothing was created. Deleting a claim is the agent's destructive tier: set rbac.write.destructive.enabled=true on that cluster's agent (helm upgrade --reuse-values --set rbac.write.destructive.enabled=true), then approve the fix again. The cluster said: ...

Where: on a Nightshift fix that recreates a claim whose StorageClass does not exist, after it was approved.

Why: recreating the claim deletes it first, and the agent’s default role never deletes claims. Nothing changed: the old claim is exactly as it was.

What to do: if you want Nightshift to make this fix, turn on the agent’s destructive tier on that cluster, then approve the fix again. Use --reset-then-reuse-values (Helm 3.14 or later) rather than the --reuse-values the message shows; see the last entry:

Terminal window
helm upgrade seylo-agent oci://ghcr.io/seylo-inc/charts/seylo-agent \
--version 0.2.22 --namespace seylo-agent --reset-then-reuse-values \
--set image.tag=0.2.22 \
--set image.digest=sha256:ed9222f8b6e0beb0fb12873388439ce270304bc5b54dc41a80a96707c552661f \
--set rbac.write.destructive.enabled=true

Read what the destructive tier allows first: see the two grants to think hardest about. Or leave it off and recreate the claim yourself, with the StorageClass the fix names.

Claim … was deleted but is still Terminating

Section titled “Claim … was deleted but is still Terminating”
Claim <name> was deleted but is still Terminating after <n> s, held by <finalizers>. ... The new claim was NOT created. Once <name> is gone, approve the fix again (Nightshift re-plans from fresh evidence) or create it by hand with the class shown in the fix.

Where: on a Nightshift fix that recreates a claim.

Why: a finalizer still holds the old claim. When it is kubernetes.io/pvc-protection, a pod was scheduled with the claim after the fix was planned, and the message says so. Seylo does not create the new claim while the old one exists, and never removes a finalizer.

What to do: find what holds it (kubectl get pvc <name> -n <namespace> -o yaml, under metadata.finalizers). Once the claim is gone, approve the fix again.

The live image tag is <tag>, but Seylo's latest recorded release is tag <tag>: the workload was changed outside Seylo, so Seylo's previous revision is not what ran before. Roll back with the tool that made the change. Nor can it go back to the Deployment's own previous revision: <reason>

Where: a Nightshift rollback that was refused.

Why: Seylo’s history does not describe what ran before, and the Deployment’s own previous revision could not be used either. The reason after Nor can it go back says why: its ReplicaSets could not be read, it has no earlier revision, or the previous revision adds or removes a container.

What to do: roll back with the tool that made the change, or with kubectl rollout undo deployment/<name> -n <namespace> after checking what it restores. See rollback.

Rising across restarts: more memory only delays the crash

Section titled “Rising across restarts: more memory only delays the crash”
Rising across restarts: more memory only delays the crash. ... Send instead: a finding that container <name> leaks (no change), pointing at the code or its owner - a heap dump or profile near the limit shows what grows.

Where: a refused memory change, on a Nightshift fix or investigation.

Why: the container’s memory climbs steadily until it is killed, and does so again after a restart. A bigger limit only moves the next kill later, so Nightshift will not raise it.

What to do: treat it as a leak in the application: a heap dump or a profile taken near the limit shows what grows. See memory limits sized from evidence.

The pull request cannot be proposed yet: An admin turns on "Let Seylo open pull requests" (Admin > Integrations > GitHub). It is off by default and separate from the skills repository, which stays read-only.

The list after yet: names each missing step, for example installing Seylo’s GitHub App on the repository, adding the repository to the installation, or accepting the contents: write and pull requests: write permissions.

Where: a fix or an APM coverage proposal for a workload Argo CD or Flux deploys from GitHub.

What to do: each item says who can do it. Or make the change in the cluster instead, acknowledging the owner. See pull requests for workloads Git owns.

Your organisation already has 4 assistant requests waiting for the model

Section titled “Your organisation already has 4 assistant requests waiting for the model”
Your organisation already has 4 assistant requests waiting for the model (at most 4 at once, so one organisation cannot use up the model for everyone). Wait for one to finish - most take under two minutes - or cancel one you no longer need, then ask again.

Where: Ask AI on Metrics and Logs, or Build with AI on Dashboards.

What to do: wait, or press Cancel on a question you no longer need, then ask again.

This assistant only writes queries over this cluster’s metrics, logs and traces

Section titled “This assistant only writes queries over this cluster’s metrics, logs and traces”
This assistant only writes queries over this cluster's metrics, logs and traces, and this reads as a request for creative writing, so it was not sent to the model. Ask about your workloads instead, e.g. "which pods restarted most today?" or "5xx rate of checkout in the last hour".

Where: Ask AI on Metrics and Logs, or Build with AI on Dashboards (which says it builds dashboard panels).

What to do: ask about your workloads, a signal or a metric. See what the assistants answer.

You have used Ask AI … times in the last 24 hours

Section titled “You have used Ask AI … times in the last 24 hours”
You have used Ask on the Logs page 100 times in the last 24 hours, the most one person may (100 a day, so no one person spends the organisation's model budget alone). The oldest stops counting at 15:20 UTC. Your query editor works without it; an admin can ask Seylo for a higher allowance through Help > Contact support.

Where: Ask AI on Metrics and Logs, or Build with AI on Dashboards (40 a day).

What to do: write the query in the editor, or wait until the time the message names. An admin can ask for a higher allowance through Help → Contact support.

“nil pointer evaluating” on helm upgrade

Section titled ““nil pointer evaluating” on helm upgrade”
Error: UPGRADE FAILED: template: ... nil pointer evaluating ...

for example about opencost.image.repository, when you upgrade a chart with --reuse-values.

Why: --reuse-values re-applies all of the previous release’s values, including the old chart’s defaults. A setting block that the new chart added then has nothing in it, and the template stops on it.

What to do: upgrade with --reset-then-reuse-values instead. It takes the new chart’s defaults and keeps only the values you set yourself:

Terminal window
helm upgrade seylo-observability oci://ghcr.io/seylo-inc/charts/seylo-observability \
--version 0.2.22 --namespace seylo-observability --reset-then-reuse-values

The same applies to seylo-agent.

On Helm older than 3.14 (no --reset-then-reuse-values), save the values you set and pass them back as a file:

Terminal window
helm get values seylo-observability -n seylo-observability -o yaml > mine.yaml
helm upgrade seylo-observability oci://ghcr.io/seylo-inc/charts/seylo-observability \
--version 0.2.22 --namespace seylo-observability -f mine.yaml

helm get values without --all prints only what you set, which is exactly what you want to keep.