Messages you may see
Each entry starts with the message as Seylo (or Helm) shows it, so you can find
it by pasting it into search. <cluster>, <n> and <version> stand for what
you see in their place.
The commands assume the install’s default namespaces, seylo-agent and
seylo-observability. If you installed into others, change --namespace to
match.
The metrics store could not hold this query in memory
Section titled “The metrics store could not hold this query in memory”The metrics store could not hold this query in memory: it covers <n> series over the chosen range.Long ranges see every pod that existed in them, and pods that come and go (CronJobs, restarts,rollouts) each add series. Pick a shorter range, or give the store more memory (it is capped at1 GiB by default): helm upgrade seylo-observability ...Where: a Metrics query or a dashboard panel over a long time range.
Why: a query over a range has to hold, at once, every series that existed in that range. Every pod that ever ran counts — each CronJob run and each restart gets a new pod name — so the number grows with the range, not with how many pods run now. VictoriaMetrics refuses a query that does not fit in its memory limit, which the chart sets to 1 GiB.
Seylo already splits any range longer than two days into three-day windows and joins the answers, so each window holds only the pods alive in it. You see this message only when a single window is still too big for the store, which is rare.
What to do — either:
-
Pick a shorter range, or narrow the query with label filters (a namespace, a workload); or
-
Give the store more memory:
Terminal window helm upgrade seylo-observability oci://ghcr.io/seylo-inc/charts/seylo-observability \--version 0.2.22 --namespace seylo-observability --reset-then-reuse-values \--set metrics.resources.limits.memory=2GiThe node the store runs on needs that much memory free.
The Datadog connector is off on this agent
Section titled “The Datadog connector is off on this agent”The Datadog connector is off on this agent (datadog.enabled=false).Where: Admin → Integrations → Datadog, or anything that reads from Datadog.
Why: the agent reads Datadog only when its chart turns the connector on, and it is off by default. The keys stay in a Secret in your cluster; Seylo never receives them.
What to do: create the Secret, then turn the connector on. Run both yourself (the full walk-through is on compare with Datadog):
kubectl -n seylo-agent create secret generic seylo-datadog \ --from-literal=DD_API_KEY=<api key> \ --from-literal=DD_APP_KEY=<application key>helm upgrade seylo-agent oci://ghcr.io/seylo-inc/charts/seylo-agent \ --namespace seylo-agent --reset-then-reuse-values \ --set datadog.enabled=true \ --set datadog.site=datadoghq.com \ --set datadog.clusterName=<kube_cluster_name> \ --set datadog.existingSecret=seylo-datadogUse your own Datadog site (datadoghq.eu, us3.datadoghq.com, …) and the
cluster name your Datadog Agent reports. A scoped application key needs only
the timeseries_query permission. The connector needs agent 0.2.10 or later;
if the agent is older, upgrade it first (next entry).
The agent is too old for a feature
Section titled “The agent is too old for a feature”The agent on "<cluster>" is version <version>, which has no Datadog connector.Upgrade to agent 0.2.10 or later (helm upgrade seylo-agent ... --reset-then-reuse-values), then set datadog.enabled=true.or, for the observability pages:
The agent on "<cluster>" is version <version>, which cannot query observability stores.On Admin → Clusters the same thing shows as a chip on the cluster’s row,
for example Agent 0.2.10 required; hover it for what does not work until you
upgrade.
Why: Seylo keeps working with older agents and says which feature needs a newer one, rather than showing empty charts. Everything the older agent could do keeps working.
What to do: upgrade the agent. No new enrollment token is needed:
helm upgrade seylo-agent oci://ghcr.io/seylo-inc/charts/seylo-agent \ --version 0.2.22 --namespace seylo-agent --reset-then-reuse-values \ --set image.tag=0.2.22 \ --set image.digest=sha256:ed9222f8b6e0beb0fb12873388439ce270304bc5b54dc41a80a96707c552661fThe digest is the signed 0.2.22 image. The tag and digest are passed explicitly because an install that set them keeps the old ones otherwise. See upgrade the agent for the details and verify what you install to check the signature first.
No OpenCost figures for this cluster
Section titled “No OpenCost figures for this cluster”Where: the Cost tab of FinOps → Cost & savings: Where the money went (titled No cost overview for this cluster) and the cost table (No cost by namespace for this cluster).
There are two causes, and the card says which one applies.
1. OpenCost is not on. The card says:
OpenCost is not running in this cluster. Turn it on with:followed by the command:
helm upgrade seylo-observability oci://ghcr.io/seylo-inc/charts/seylo-observability --version 0.2.22 --namespace seylo-observability --reset-then-reuse-values --set opencost.enabled=trueThe first figures appear a few minutes after its pod is Ready
(kubectl -n seylo-observability get pods). The agent must also have
observability.enabled=true.
2. Seylo reads this cluster for some namespaces only. The card says:
Seylo reads this cluster's store for some namespaces only (<namespaces>), and OpenCost's node pricesbelong to no namespace, so the namespace filter hides them even when OpenCost is running.Cost allocation needs the whole cluster. Edit the cluster in Admin → Clusters and set What this tool can see to The whole cluster. If the agent itself was installed for some namespaces only, reinstall it for the whole cluster from the cluster’s Agent details page (Admin → Clusters, the cluster’s menu).
Cost by namespace needs agent 0.2.11
Section titled “Cost by namespace needs agent 0.2.11”The card is titled Cost by namespace needs agent 0.2.11 and the 0.2.11 observability chart (or Cost overview needs … on Where the money went), and says one of:
The agent on "<cluster>" is version <version>; Seylo's cost figures come from OpenCost's API, which needs agent 0.2.11 and seylo-observability chart 0.2.11 or later. Run both upgrades on that cluster, in this order:The agent could not reach OpenCost: the seylo-observability chart on this cluster predates 0.2.11, whose NetworkPolicy closes OpenCost's port to everything but VictoriaMetrics. Run both upgrades on that cluster, in this order:Where: the Cost tab of FinOps → Cost & savings.
Why: Seylo’s cost figures ask OpenCost’s own API through the agent. That
needs agent 0.2.11, and seylo-observability 0.2.11, whose NetworkPolicy lets
the agent reach OpenCost.
What to do: run both commands the card shows, in that order. They are also on chargeback.
A browser that blocks site data
Section titled “A browser that blocks site data”A browser set to refuse site data (strict privacy settings, cleared site data,
some managed browser profiles) used to stop Seylo before the sign-in screen
appeared, with an error such as The operation is insecure.
Nothing to do now: Seylo signs in and works in such a browser. It only
cannot remember small preferences between visits, such as which panes you
folded or which organisation you signed in to last. If you use more than one
organisation, open Seylo with your organisation’s own link (/t/<your organisation>) so you land in the right one.
Nightshift did not start on its own
Section titled “Nightshift did not start on its own”<n> automatic investigations in the last 24 hours, at the organisation's daily cap of <n>. Press Investigate to run one by hand, or ask for a higher limit under Help > Contact support.or ... in the last hour, at the organisation's limit of <n> per hour. ...,
or ... on cluster <cluster> in the last hour, at the limit of <n> per cluster per hour. ...,
or, for the same alert:
Nightshift already investigated this alert automatically at <time>; it starts again for the same alert after 6 hours. Press Investigate to run it now.Where: on an alert, where Nightshift says why it did not start.
Why: automatic investigations are bounded per organisation. The defaults are 40 a day, 10 an hour, 4 per cluster an hour, and the same alert at most once every 6 hours. See how often it starts on its own.
What to do: press Investigate on the alert to run one now; the limits apply only to automatic starts. If your organisation needs more, ask through Help → Contact support: see ask for higher Nightshift limits. The 6-hour wait for the same alert does not change.
Not recreated: the agent’s role may not delete PersistentVolumeClaims
Section titled “Not recreated: the agent’s role may not delete PersistentVolumeClaims”Not recreated: the agent's role may not delete PersistentVolumeClaims, so claim <name> was left untouched and nothing was created. Deleting a claim is the agent's destructive tier: set rbac.write.destructive.enabled=true on that cluster's agent (helm upgrade --reuse-values --set rbac.write.destructive.enabled=true), then approve the fix again. The cluster said: ...Where: on a Nightshift fix that recreates a claim whose StorageClass does not exist, after it was approved.
Why: recreating the claim deletes it first, and the agent’s default role never deletes claims. Nothing changed: the old claim is exactly as it was.
What to do: if you want Nightshift to make this fix, turn on the agent’s
destructive tier on that cluster, then approve the fix again. Use
--reset-then-reuse-values (Helm 3.14 or later) rather than the
--reuse-values the message shows; see the last entry:
helm upgrade seylo-agent oci://ghcr.io/seylo-inc/charts/seylo-agent \ --version 0.2.22 --namespace seylo-agent --reset-then-reuse-values \ --set image.tag=0.2.22 \ --set image.digest=sha256:ed9222f8b6e0beb0fb12873388439ce270304bc5b54dc41a80a96707c552661f \ --set rbac.write.destructive.enabled=trueRead what the destructive tier allows first: see the two grants to think hardest about. Or leave it off and recreate the claim yourself, with the StorageClass the fix names.
Claim … was deleted but is still Terminating
Section titled “Claim … was deleted but is still Terminating”Claim <name> was deleted but is still Terminating after <n> s, held by <finalizers>. ... The new claim was NOT created. Once <name> is gone, approve the fix again (Nightshift re-plans from fresh evidence) or create it by hand with the class shown in the fix.Where: on a Nightshift fix that recreates a claim.
Why: a finalizer still holds the old claim. When it is
kubernetes.io/pvc-protection, a pod was scheduled with the claim after the
fix was planned, and the message says so. Seylo does not create the new claim
while the old one exists, and never removes a finalizer.
What to do: find what holds it (kubectl get pvc <name> -n <namespace> -o yaml,
under metadata.finalizers). Once the claim is gone, approve the fix again.
The workload was changed outside Seylo
Section titled “The workload was changed outside Seylo”The live image tag is <tag>, but Seylo's latest recorded release is tag <tag>: the workload was changed outside Seylo, so Seylo's previous revision is not what ran before. Roll back with the tool that made the change. Nor can it go back to the Deployment's own previous revision: <reason>Where: a Nightshift rollback that was refused.
Why: Seylo’s history does not describe what ran before, and the
Deployment’s own previous revision could not be used either. The reason after
Nor can it go back says why: its ReplicaSets could not be read, it has no
earlier revision, or the previous revision adds or removes a container.
What to do: roll back with the tool that made the change, or with
kubectl rollout undo deployment/<name> -n <namespace> after checking what it
restores. See rollback.
Rising across restarts: more memory only delays the crash
Section titled “Rising across restarts: more memory only delays the crash”Rising across restarts: more memory only delays the crash. ... Send instead: a finding that container <name> leaks (no change), pointing at the code or its owner - a heap dump or profile near the limit shows what grows.Where: a refused memory change, on a Nightshift fix or investigation.
Why: the container’s memory climbs steadily until it is killed, and does so again after a restart. A bigger limit only moves the next kill later, so Nightshift will not raise it.
What to do: treat it as a leak in the application: a heap dump or a profile taken near the limit shows what grows. See memory limits sized from evidence.
The pull request cannot be proposed yet
Section titled “The pull request cannot be proposed yet”The pull request cannot be proposed yet: An admin turns on "Let Seylo open pull requests" (Admin > Integrations > GitHub). It is off by default and separate from the skills repository, which stays read-only.The list after yet: names each missing step, for example installing Seylo’s
GitHub App on the repository, adding the repository to the installation, or
accepting the contents: write and pull requests: write permissions.
Where: a fix or an APM coverage proposal for a workload Argo CD or Flux deploys from GitHub.
What to do: each item says who can do it. Or make the change in the cluster instead, acknowledging the owner. See pull requests for workloads Git owns.
Your organisation already has 4 assistant requests waiting for the model
Section titled “Your organisation already has 4 assistant requests waiting for the model”Your organisation already has 4 assistant requests waiting for the model (at most 4 at once, so one organisation cannot use up the model for everyone). Wait for one to finish - most take under two minutes - or cancel one you no longer need, then ask again.Where: Ask AI on Metrics and Logs, or Build with AI on Dashboards.
What to do: wait, or press Cancel on a question you no longer need, then ask again.
This assistant only writes queries over this cluster’s metrics, logs and traces
Section titled “This assistant only writes queries over this cluster’s metrics, logs and traces”This assistant only writes queries over this cluster's metrics, logs and traces, and this reads as a request for creative writing, so it was not sent to the model. Ask about your workloads instead, e.g. "which pods restarted most today?" or "5xx rate of checkout in the last hour".Where: Ask AI on Metrics and Logs, or Build with AI on Dashboards (which says it builds dashboard panels).
What to do: ask about your workloads, a signal or a metric. See what the assistants answer.
You have used Ask AI … times in the last 24 hours
Section titled “You have used Ask AI … times in the last 24 hours”You have used Ask on the Logs page 100 times in the last 24 hours, the most one person may (100 a day, so no one person spends the organisation's model budget alone). The oldest stops counting at 15:20 UTC. Your query editor works without it; an admin can ask Seylo for a higher allowance through Help > Contact support.Where: Ask AI on Metrics and Logs, or Build with AI on Dashboards (40 a day).
What to do: write the query in the editor, or wait until the time the message names. An admin can ask for a higher allowance through Help → Contact support.
“nil pointer evaluating” on helm upgrade
Section titled ““nil pointer evaluating” on helm upgrade”Error: UPGRADE FAILED: template: ... nil pointer evaluating ...for example about opencost.image.repository, when you upgrade a chart with
--reuse-values.
Why: --reuse-values re-applies all of the previous release’s values,
including the old chart’s defaults. A setting block that the new chart added
then has nothing in it, and the template stops on it.
What to do: upgrade with --reset-then-reuse-values instead. It takes the
new chart’s defaults and keeps only the values you set yourself:
helm upgrade seylo-observability oci://ghcr.io/seylo-inc/charts/seylo-observability \ --version 0.2.22 --namespace seylo-observability --reset-then-reuse-valuesThe same applies to seylo-agent.
On Helm older than 3.14 (no --reset-then-reuse-values), save the values you
set and pass them back as a file:
helm get values seylo-observability -n seylo-observability -o yaml > mine.yamlhelm upgrade seylo-observability oci://ghcr.io/seylo-inc/charts/seylo-observability \ --version 0.2.22 --namespace seylo-observability -f mine.yamlhelm get values without --all prints only what you set, which is exactly
what you want to keep.