Skip to content

Leaving Seylo - remove it from a cluster

Everything Seylo runs in your cluster came from two Helm charts: seylo-agent and, if you installed it, seylo-observability. Removing them is a short list of commands, but helm uninstall alone leaves four things behind, and this page removes those too:

  • The stored metrics, logs and traces. The three stores keep their data on volumes that Kubernetes creates for them, and Helm never deletes those.
  • Secrets Helm did not create. The agent writes its own credential into a Secret, and you may have created Secrets for it (for example the Datadog keys).
  • A log buffer on each node’s disk, kept by the log collector.
  • The namespaces the install created.
  • Point helm and kubectl at the cluster, as someone who may delete namespaces, cluster roles and persistent volumes.
  • The commands are for bash or zsh. On Windows, run them from WSL or Git Bash.
  • They assume the install’s default namespaces, seylo-agent and seylo-observability. Step 1 shows the ones you actually used; if they differ, change the commands to match.
  • Every delete names exactly what it deletes, and running a step twice is harmless.
  1. See what is installed. The second command shows the names of any Secrets you pointed the agent at (a proxy, a CA certificate, the Datadog keys). Note them: the uninstall leaves them behind.

    Terminal window
    helm list --all-namespaces --filter '^(seylo-agent|seylo-observability)$'
    helm get values seylo-agent --namespace seylo-agent | grep -i secret
  2. Optional: keep the stored metrics, logs and traces. Skip this step to delete them. Run it to keep them: it sets each store’s volume to Retain, so the disks stay in your cloud account after everything below. You can then attach them elsewhere, or delete them later.

    Terminal window
    for pv in $(kubectl --namespace seylo-observability get pvc data-seylo-vm-0 data-seylo-vl-0 data-seylo-vt-0 --ignore-not-found -o jsonpath='{.items[*].spec.volumeName}'); do
    kubectl patch pv "$pv" -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
    done
  3. Uninstall the agent. Seylo stops acting on the cluster at once. This removes the agent’s Deployment, ServiceAccount and enrollment Secret, and every Role, RoleBinding, ClusterRole and ClusterRoleBinding the chart created — including the Roles in other namespaces if you installed it for some namespaces only.

    Terminal window
    helm uninstall seylo-agent --namespace seylo-agent
  4. Uninstall the observability stack. This removes VictoriaMetrics, VictoriaLogs, VictoriaTraces, the collectors (kube-state-metrics, the log collector, the OpenTelemetry collector), OpenCost if you turned it on, and their Services, ConfigMaps, NetworkPolicies and cluster roles. It does not remove the stores’ volumes.

    Terminal window
    helm uninstall seylo-observability --namespace seylo-observability
  5. Delete the Secrets the agent and you created. seylo-agent-credential holds the agent’s credential; the agent wrote it, so Helm does not know it exists. seylo-datadog exists only if you connected Datadog, and the seylo-agent volume claim only if you kept the credential on a volume. Add any other Secret step 1 listed.

    Terminal window
    kubectl --namespace seylo-agent delete secret seylo-agent-credential seylo-datadog --ignore-not-found
    kubectl --namespace seylo-agent delete pvc seylo-agent --ignore-not-found
  6. Delete the stored metrics, logs and traces.

    Terminal window
    kubectl --namespace seylo-observability delete pvc data-seylo-vm-0 data-seylo-vl-0 data-seylo-vt-0 --ignore-not-found
  7. Delete volumes that outlived their claims. A volume whose reclaim policy is Retain stays after its claim is deleted. The first command lists them, with the cloud disk behind each; the loop deletes the volume objects.

    Terminal window
    kubectl get pv --no-headers -o custom-columns=NAME:.metadata.name,NS:.spec.claimRef.namespace,CLAIM:.spec.claimRef.name,POLICY:.spec.persistentVolumeReclaimPolicy,DISK:.spec.csi.volumeHandle | awk '$2=="seylo-observability" && $3 ~ /^data-seylo-(vm|vl|vt)-0$/ {print}'
    for pv in $(kubectl get pv --no-headers -o custom-columns=NAME:.metadata.name,NS:.spec.claimRef.namespace,CLAIM:.spec.claimRef.name | awk '$2=="seylo-observability" && $3 ~ /^data-seylo-(vm|vl|vt)-0$/ {print $1}'); do
    kubectl delete pv "$pv"
    done

    Deleting a Retain volume never deletes its disk. If you did not mean to keep the data, delete the disks the first command listed in your cloud provider’s console.

  8. Optional: remove the log collector’s buffer from each node. The log collector kept a buffer of log lines in /var/lib/seylo-vlagent on every node. Each command starts a short-lived pod on one node that deletes that directory. Wait until kubectl --namespace seylo-observability get pods shows them Completed before the next step. Skip this if logs were off, or if you are about to replace the nodes anyway.

    Terminal window
    for node in $(kubectl get nodes -o name); do
    kubectl debug "$node" --namespace seylo-observability --image=busybox:1.36 -- rm -rf /host/var/lib/seylo-vlagent
    done

    It needs a namespace that allows privileged pods; seylo-observability does, because the collector ran there.

  9. Optional: delete anything the uninstall missed. Normally this deletes nothing. If an uninstall was interrupted, or a release was removed some other way, it removes the remaining cluster roles, Roles and RoleBindings by Seylo’s labels.

    Terminal window
    kubectl delete clusterrole,clusterrolebinding -l 'app.kubernetes.io/name in (seylo-agent,seylo-observability)' --ignore-not-found
    kubectl delete role,rolebinding --all-namespaces -l 'app.kubernetes.io/name in (seylo-agent,seylo-observability)' --ignore-not-found
  10. Delete Seylo’s namespaces. The install created them with --create-namespace, and Helm never deletes a namespace. Deleting a namespace deletes everything still in it, so look first: you should see only Seylo’s leftovers, plus the kube-root-ca.crt ConfigMap Kubernetes puts in every namespace.

    Terminal window
    kubectl get all,secret,configmap,pvc --namespace seylo-observability
    kubectl get all,secret,configmap,pvc --namespace seylo-agent
    kubectl delete namespace seylo-observability seylo-agent --ignore-not-found
  11. Verify nothing of Seylo’s is left. This lists Seylo’s releases, every object carrying Seylo’s labels in any namespace, the cluster-wide roles, the stores’ volumes, the Secrets by name and the namespaces. It prints nothing when the cluster is clean; whatever it prints is what is left.

    Terminal window
    {
    helm list --all-namespaces --short --filter '^(seylo-agent|seylo-observability)$'
    kubectl get clusterrole,clusterrolebinding -l 'app.kubernetes.io/name in (seylo-agent,seylo-observability)' -o name
    kubectl get deployment,statefulset,daemonset,pod,service,configmap,secret,serviceaccount,role,rolebinding,networkpolicy,persistentvolumeclaim --all-namespaces -l 'app.kubernetes.io/name in (seylo-agent,seylo-observability)' -o name
    kubectl --namespace seylo-observability get pvc data-seylo-vm-0 data-seylo-vl-0 data-seylo-vt-0 --ignore-not-found -o name
    kubectl get pv --no-headers -o custom-columns=NAME:.metadata.name,NS:.spec.claimRef.namespace,CLAIM:.spec.claimRef.name | awk '$2=="seylo-observability" && $3 ~ /^data-seylo-(vm|vl|vt)-0$/ {print $1}'
    kubectl --namespace seylo-agent get secret seylo-agent-credential seylo-datadog --ignore-not-found -o name
    kubectl get namespace seylo-observability seylo-agent --ignore-not-found -o name
    } 2>&1 | grep -v '^No resources found'

    kubectl says No resources found for an empty list; the last line hides only that sentence, so an error such as Forbidden still shows.

  12. Remove the cluster from Seylo. Admin → Clusters, the cluster’s ⋯ menu, Remove cluster, and type the cluster’s ID to confirm. Do this last: once the cluster is removed, its page in Seylo is gone.

Applications you deployed or approved through Seylo keep running; they are yours. Two marks of Seylo stay on them, and both are harmless once Seylo is gone:

  • Objects Seylo generated carry the label app.kubernetes.io/managed-by=seylo.
  • A Deployment Seylo restarted carries the pod-template annotation kubestrator.io/restartedAt. Removing it restarts the pods, so leave it unless you have a reason.

To list them (these commands change nothing):

Terminal window
kubectl get deployment,statefulset,service,ingress,configmap,resourcequota,limitrange --all-namespaces -l app.kubernetes.io/managed-by=seylo
kubectl get deployment --all-namespaces --no-headers -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,RESTARTED:.spec.template.metadata.annotations.kubestrator\.io/restartedAt' | awk '$3 != "<none>"'

Deleted at once:

  • The cluster’s entry, and any connection credential Seylo stored for it.
  • The agent’s identity and unused enrollment tokens. The agent’s credential stops working, and a connected agent is disconnected. For an agent-managed cluster Seylo holds no other credential.
  • The inventory the agent last reported (namespaces, workloads).
  • The usage summaries (metric aggregates) for this cluster.
  • Its alerts. Their timelines go at the next hourly alerting cleanup.
  • Policy scan results, findings and suppressions for this cluster.

If you connected the cluster directly rather than with the agent, Seylo deletes its encrypted copy of the credential you gave it. The credential itself still works until you revoke it where it was issued (the service account or token in the kubeconfig, or the cloud identity).

Kept afterwards:

What What happens to it
The audit log, including the removal itself Kept permanently. It is your organisation’s record
Operations on this cluster (deploys, rollbacks, restarts) Deleted with all finished operations after 30 days, or beyond the newest 2,000 per organisation (Seylo’s default settings)
Approval requests that named this cluster Kept. No automatic deletion
Manifest history (the versions Seylo applied) Kept. No automatic deletion
Investigations of this cluster’s alerts, including the evidence they quote (log lines, events) Kept. No automatic deletion
Scheduled deployments, grants and their history, and alert rules that name this cluster Kept, and unable to act: there is no cluster to reach. Delete them on their own pages
Per-cluster access grants, service owners, promotions, and deploy tokens for this cluster Kept. A deploy token is refused when used, because each use re-checks its cluster
A record of the agent’s permissions when it enrolled Kept. A cluster added again under the same ID is compared against it

Raw metrics, logs and traces were never copied to Seylo: they were in the stores in your cluster, which the steps above delete.

Removing every cluster does not close your organisation, and Seylo has no self-service export of your organisation’s records and no self-service “close this organisation” yet. An administrator who wants the organisation closed writes to support@seylo.in from the address on their Seylo account.