Skip to content

Keeping data longer (retention)

Seylo’s observability stores live in your cluster, so keeping data longer costs you disk and bucket storage, billed by your cloud. Seylo’s price does not change with retention.

Admin → Integrations → Observability stores → Retention opens with How long to keep data. Each store has its own row, Keep logs for, Keep metrics for and Keep traces for, with one button per choice:

Store Choices
Logs 7, 15, 30 or 90 days
Traces 7, 15, 30 or 90 days
Metrics 7, 15, 30, 90, 180 or 400 days

The button marked (now) is what the store reports. Two clicks change it:

  1. Click a number of days. Seylo shows what the choice does to your disk and your data, measured on this cluster, and what happens while the store restarts.

  2. Click Keep N days. The agent saves your choice, restarts the store, and the page reads the store’s own setting back. restarting shows until the store reports the new period, then read back.

What the page tells you before you click:

  • The disk. For logs and traces, if your disk cannot hold the days you asked for, the page says how many are kept and why: the store drops the oldest whole days for space first, and the disk does not fill. For metrics, which have no such cap, a choice whose measured need does not fit the volume is refused, with the numbers; grow the volume first (see growing a volume). If Seylo could not measure whether a longer period fits, you must tick a box to go ahead.
  • Your data. A shorter period deletes data, and the page says how much is held now. Logs and traces are deleted at the store’s next retention check after it restarts; metrics are deleted a whole month at a time.
  • The restart. Logs: lines written meanwhile wait in the collector’s on-disk buffer on each node and arrive when the store is back. Metrics: nothing is scraped while the store is down, so dashboards show a short gap. Traces: spans sent while it is down are kept only if the sender retries them.
  • Someone else managing it. If Argo CD, Flux, an operator or another tool owns the store or the setting, you must tick a box saying this page’s choice wins over it, and the page names the owner and where to set it instead. Helm alone does not ask: it is how you installed the chart.

Your choice is kept outside the Helm release, so a later helm upgrade does not put the old period back. To return a store to the chart’s value, delete its key from the seylo-retention ConfigMap in the stores’ namespace.

Only an organisation admin can change retention or the archive, because a shorter period deletes data and every change restarts a store. Everyone who can see the page can read every figure, with the buttons switched off.

One-click retention needs seylo-observability 0.2.23 and an agent that is allowed to write ConfigMaps and evict pods in the stores’ namespace (rbac.write.pods.enabled, which the deploy posture leaves off and admin includes). On an older chart, or without those permissions, the row says which, and the commands under Advanced: commands do the same job.

Below the buttons, the table for each store (metrics, logs and traces) shows:

  • how long it keeps data, read back from the running store;
  • how much it adds per day, measured on your cluster;
  • how many days your disk holds at that rate;
  • the exact commands to keep more, under Advanced: commands, which also take a number of days the buttons do not offer.

A busy cluster can fill a disk before the retention period is reached. The logs store then drops the oldest whole days, so “how many days your disk holds” is the number that tells you how far back you can actually look. See retention: capped by time and by disk.

Each store can have its own period, from seylo-observability 0.2.16:

Store Value Range the commands under Advanced offer
Logs logs.retentionPeriod 7 to 90 days
Traces traces.retentionPeriod 7 to 90 days
Metrics metrics.retentionPeriod 7 to 400 days

Empty (the default) means retention.period, so an install that never set them behaves as before. Beyond 90 days, logs belong in the archive, where they cost bucket storage instead of disk.

The metrics store keeps data in monthly partitions, so its disk holds up to the period you set plus a month.

Keeping more days usually needs a bigger volume. Growing one is three commands, and the Retention page gives them with your names filled in:

  1. grow the claim;
  2. remove the StatefulSet object with --cascade=orphan, which leaves its pod and volume running;
  3. helm upgrade with the new size.

This needs a StorageClass that allows expansion: check the ALLOWVOLUMEEXPANSION column of kubectl get storageclass. The full procedure is in growing a volume.

The archive is a copy, not a move. Below the retention buttons, Keep a copy in my cloud account is one switch.

  • Every signal. Logs, metrics and traces are copied, as many of the three as are installed on the cluster. A status line per signal reads copying, behind, failing, not copied, not installed or not known yet, with one sentence saying what is copied and up to when, taken from what is in your bucket.
  • Into your own bucket. Azure Blob, S3 or GCS, written by a job in your cluster that signs in with workload identity. Seylo never holds a key to your bucket, and the chart refuses a value that looks like a key.
  • Hourly by default. Each finished hour of each signal becomes one compressed file. An hour in which a store held nothing is marked empty, so a reader can tell “nothing happened” from “not archived”.
  • Your data stays on disk for your whole retention, so dashboards and the Logs page show everything still on disk, exactly as before.

Files are named:

<prefix>/logs/cluster=<name>/year=YYYY/month=MM/day=DD/…ndjson.gz

so several clusters can share one bucket. The job never overwrites a file, so immutable (write-once) buckets work: S3 Object Lock, Azure immutable storage, or a GCS retention policy.

For an AKS cluster whose subscription you have connected under Integrations → Cloud accounts (the Seylo Reader role), the switch needs one step from you, because Seylo’s connection is read-only and cannot create your storage:

  1. On Retention, choose Download seylo-archive.json under the switch. Every value is already filled in for this cluster.

  2. Run the one command shown, az deployment group create --template-file seylo-archive.json, with your own sign-in. It creates a storage account dedicated to this cluster (shared key access off, no public blobs), the container, a managed identity trusted by this cluster only, Storage Blob Data Contributor on the container only for that identity, and Storage Blob Data Reader on the container for Seylo, so the copy can be read when the cluster is gone.

  3. Press Check again. Seylo reads every part back and marks each one read back, missing or unknown. Turn on is available only when all are read back. Click it.

To use a storage account you already have, open Advanced: commands and name it under Keep the copy in a storage account you already have. Seylo finds it in your connected subscriptions and reads back the same things, and checks two more: shared key access must be off on the account, and the contributor role must be scoped to the container itself. The command does not change your account’s settings.

The switch applies its own job, so it works on any published chart that has the logs store. Turn off suspends the job; nothing in the bucket is deleted.

On AWS and Google Cloud the switch says it does not cover that cloud. The archive still writes to S3 or GCS, every signal, hourly, with the chart values under Advanced: commands; once it runs, Seylo records where it is and can read it when the cluster is gone, after you create the read identity those commands include. Where Seylo cannot read the cluster’s cloud, or it runs on none of the three, Advanced turns the archive on with a bucket you name.

If the archive is already on through the chart’s own Helm values, the switch says so and shows its status and cost; change it with the commands under Advanced, which the switch does not override.

A switch turned on by an earlier version shows Update. One click re-applies it so it copies every signal and lets Seylo read the copy.

Once the switch is on, Archive now runs the job once, straight away. It first writes one small check file and lists it back, so the identity is proven the same day instead of at the next scheduled run. The result is read from the job: the objects it wrote and listed in your container, or the error in one sentence (a 403 names the container-scoped role; a sign-in failure names the federated credential).

It can run at most once every 30 minutes, and not while a run is going: each run reads up to four whole days back out of the logs store, competing with the Logs page and Nightshift, and lists and writes your container, which your cloud bills per operation. The scheduled runs do the rest.

How often to copy appears on the switch, one choice per signal:

Signal Choices
Logs every hour (the default), every 15 minutes, every 5 minutes (near-real-time)
Metrics every hour (the default), every 15 minutes
Traces every hour (the default), every 15 minutes

Each choice shows the most that would be lost if the cluster were deleted, worked out exactly from the copy interval and the job’s schedule. On their own the figures are 77 minutes for hourly, 22 for every 15 minutes and 9 for every 5 minutes; with different signals on different cadences the page shows the figures for your combination. To that, add the time the run in progress takes, which depends on your volume, and any line that reached the store late.

Each choice also shows what it costs in storage operations a month, as a lower bound: At least $N a month in operations, at the per-operation rates on your own cloud bill. Without those rates it says unpriced, with the reason. Renames, size checks and split uploads cost more than the bound and are not estimated. Pick a cadence, then Turn on or Update to apply it.

On a chart-written archive, archive.cadence.logs, archive.cadence.metrics and archive.cadence.traces set the same choices (1h, 15m, and 5m for logs only). A faster cadence than the chart’s nightly archive.schedule fails the render and names the schedule to set. archive.metrics.enabled is off by default, so an upgrade never starts copying metrics unasked, and archive.lagMinutes (15) is how long after an hour ends the job waits.

How long to keep the copy, and in which storage tier offers one choice, applied as a rule in your storage account. Days count from when an hour’s file was written:

Choice Result
Keep everything No rule. Files stay readable until you delete them.
Keep 90 days; cooler tier after 30 Cooler online tier after 30 days, deleted after 90
Keep 180 days; cooler tier after 30 Cooler online tier after 30 days, deleted after 180
Keep 1 year; cooler tier after 30 Cooler online tier after 30 days, deleted after 1 year
Keep 1 year; readable for 90 days, then the offline archive tier Offline after 90 days, deleted after 1 year
Keep 2 years; readable for 180 days, then the offline archive tier Offline after 180 days, deleted after 2 years

Every choice keeps at least the last 90 days in a tier the pages can read, and none moves a file before the cloud’s minimum storage time for its tier, so you are not charged an early-deletion fee. A file in an offline tier (Azure Archive, S3 Glacier Deep Archive) cannot be read until your cloud restores it, which can take hours; Seylo refuses a read that reaches those hours, with the date. Google Cloud Storage has no offline class, so its choices use Nearline and Coldline.

Each choice shows a monthly cost once full, priced from your own bill and the archive’s measured growth, or unpriced with the reason. It covers storage only: moves between tiers, reads and restores are billed too.

Seylo’s connection is read-only, so you apply the rule with your own sign-in: the setup command carries it for the dedicated account, and the page gives the command for an account you named. Each command replaces the account’s existing rule, and says so. Press Check again and the choice is marked applied once Seylo reads the rule back. Prefer your bucket’s own lifecycle rules? Seylo sets none unless you apply one here.

The copy can be searched from the Logs, Metrics and Traces pages, including after the cluster is removed, and a replacement cluster can continue its history. See read the archive when a cluster is gone.

To search days older than your retention with full query power, load them back into a running cluster:

  1. On the Retention page, under Advanced: commands, plan the window: up to 31 days per load. The page gives the command that loads them into a temporary store in your cluster.

  2. Run the command.

  3. On Logs, choose Source: Rehydrated archive. Each person still sees only their own namespaces. The results and the facet counts both come from the archive.

  4. When you are done, remove the temporary store with the command the Retention page gives. It is not removed for you.

Rehydrate needs seylo-observability 0.2.16 and agent 0.2.16 or later. With an older agent, the Logs page says needs agent 0.2.16. See upgrade the agent.

Retention does not change what Seylo costs: the stores and the archive run in your cluster and your bucket, billed by your cloud. Searching the archive from Seylo is a read your cloud bills.

  • With your Azure cost export connected (Cost → Make the figures match your invoice; see match your invoice), the Retention page shows what the archive’s storage account and the stores’ disks were actually billed.
  • Without it, those costs are shown as unpriced, with the reason. Seylo does not estimate them.

The cadence and storage-tier choices are priced the same way, from your bill or unpriced.