Skip to content

Read the archive when a cluster is gone

If you keep a copy in your cloud account, that copy outlives the cluster’s disks. Seylo can read it back, keylessly, through the cloud account you connected, and show it on the same Logs, Metrics and Traces pages, labelled with how far it goes.

On a cluster whose archive Seylo can read:

  • Healthy cluster. A line above the results reads A copy is kept in your cloud account (up to …) with Search the archive instead. On Logs, the Source menu also has Archive in your cloud account. Back to the live store returns.
  • Agent offline. The page opens on the archive and says why: “The cluster’s agent is offline: these results come from the copy in your Azure account (<region>).” (On AWS or Google Cloud it names that account instead.)
  • Cluster removed. The cluster picker lists it under Removed - archive only, as <name> (removed - archive only). Pick it and the page opens on the archive.

Every archived answer is labelled “from the archive, up to <time>”, the end of the newest archived hour, and names the hours missing from your range. When your range runs past it, the answer adds: “The archive ends at <time>; anything later has not been copied to it yet (it is copied on the archive’s schedule, and only while the cluster answers), so it is not in this answer.” If a signal cannot be read, the page says why in one sentence: no archive is recorded for the cluster, the archive does not copy that signal, the cloud account is not connected, or the bucket is in another geography.

What Seylo needs:

  • An archive that was on while the cluster could be reached. Seylo records where it is (names only, never a key) so it survives the cluster. A removed cluster that never had one has nothing to read.
  • The cloud account the archive is in, connected under Integrations → Cloud accounts. On Azure, the setup you ran grants Seylo Storage Blob Data Reader on the archive container only. On AWS and Google Cloud the archive is read through a separate read identity, seylo-archive-read, limited to that bucket; the commands under Advanced on the Retention page create it.
  • A bucket in the same geography as the Seylo region you use. The raw files pass through Seylo’s memory on their way to an answer, are not stored, and never cross that line. A bucket in another geography, or in a region Seylo has not placed, is refused with its name, and you can rehydrate instead.

Each search reads your bucket, and your cloud bills the read. Seylo keeps nothing it reads; only the list of file names is held, in memory, for minutes.

Archived data is filtered the way live data is: by each person’s namespace access, by the cluster’s read policy as last recorded, and by the namespaces the agent was installed for, whichever is narrowest. A row with no namespace is shown only to someone with no limit.

Seylo reads the files directly, so the archive supports part of each query language. A refusal names the construct and the rehydrate path.

  • Logs: words and phrases, prefixes, any field, field:value and the common field filters, numeric comparisons, _time comparisons with a time, stream filters, and boolean logic, plus the pipes sort by (_time) and limit. No other pipes, and no relative time filter such as _time:5m: the page’s range sets the time.
  • Metrics: selectors; rate, irate, increase and delta without extrapolation; *_over_time; aggregations with by and without; topk and bottomk; histogram_quantile; and one-to-one arithmetic and comparisons. offset, @, subqueries and many-to-one matching are refused.
  • Traces: services and operations, search by service, operation and duration over a range, and a trace by ID. The service map, APM and Follow one request read live data and are not available from the archive.

Limits, per query: 31 days of range, 256 MiB of compressed files for a logs or traces search, 512 MiB for a metrics query, and 50,000 series and 20 million samples held at once. Over a limit, the query is refused with the size it would have read; narrow the range. A line that reached the store more than the copy’s lag after its hour ended is not in that hour’s file.

When you rebuild a cluster, its history is in the old cluster’s archive. Linking tells Seylo that the new cluster replaces the removed one, so a range that starts before the new cluster existed reads the old archive.

  1. Open Logs, Metrics, Traces or Retention on the new cluster. If Seylo finds a removed cluster with an archive that may be the one it replaced, a banner asks Does this cluster replace <name> (removed …)? and says what matched: the same name apart from a trailing counter, date or “new”, and the same cloud, region or resource group. The name or the resource group must match.

  2. As an admin, click This replaces <name>, and confirm It replaces it. If the suggestion is wrong, Choose another lists the other removed clusters with an archive.

A link is always a person’s choice, an organisation admin’s, and is recorded in the audit log. Unlink undoes it; nothing in any bucket changes either way. Only a removed cluster with a recorded archive can be linked. A cluster that is still running is refused with the reason, because two running clusters are two histories. A cluster with no archive, one already replaced, a second predecessor and a loop are refused too.

The archive continues. On Azure, when the old archive was the switch’s, the new cluster’s setup command reuses its storage account, container and managed identity and adds credentials of its own. The new cluster writes under its own cluster= prefix, so the old files are never written again. Where the old archive was written through chart values, or on S3 or GCS, the page says to point the new cluster’s archive.* values at the same bucket under Advanced.

With a link, a range that begins before the new cluster joined Seylo is split into labelled parts, shown in time order and merged into one answer:

Part Where it comes from
archive the old cluster’s archive, up to its newest archived file
gap, “not archived before deletion” the old cluster’s hours that had not reached the archive when it was removed
gap, “no cluster running” the time between the removal and the new cluster joining
live the new cluster’s own store (or its archive, then “not archived yet”, when its agent is offline)

A gap is always a named part with its times, never an empty stretch that could be read as “nothing happened”. Log lines, metric series and traces carry the cluster they came from.

What does not cross a handover, said under What does not cross a handover on every split answer: a rate, increase or *_over_time window, topk and histogram_quantile are computed within one part; series of two clusters are not joined; a trace that crossed clusters shows in each part; split-by-field volume and live follow stop at the handover; and archived parts keep the query subset and limits above. Nightshift reads the same split, so an investigation sees the history from before the deletion.

An admin can open a removed cluster’s archive and choose Forget this archive. Seylo stops reading it. Nothing in your bucket is touched or deleted, and turning an archive on again from a cluster records it anew. The action is audited.