Skip to content

Copy a namespace

A namespace copy clones the workloads, Services, Ingresses, ConfigMaps, Secrets and the rest of a namespace into a target namespace, on the same cluster or on another. Use it to stand up a new environment from one that works: shop-staging on acme-staging into a new shop-uat, say.

It goes through the normal approval queue as a Namespace copy request. Nothing is written to the target until an approver approves it.

What you need:

  • releaser on the target to plan and submit the copy. The Copy namespace button, on Overview and on each namespace’s page, is shown to releaser and above.
  • Someone with approver or admin to approve it. That can be you. See approvals.
  • admin, only if the target namespace is outside the cluster’s namespace policy and you want the copy to add it.

Copy when the target should start as a working clone of the source and then diverge: a new environment, a per-customer namespace, a namespace on a new cluster.

To move one value or a few keys between two environments that already exist, use Copy selected keys on the environments page instead. To move a build along a path, promote it.

What is never copied: volume data. A PersistentVolumeClaim is recreated as a claim only. The wizard says so in its own words:

PVC DATA IS NOT COPIED. Only the CLAIM (size, storageClassName, accessModes) is recreated, so the target's volumes start EMPTY.

Restore the data separately, with a volume snapshot, a database dump or your backup tool. ServiceAccount tokens, Helm release records and cert-manager certificates are not copied either: the target cluster issues its own.

The wizard has six steps. Planning is read-only: nothing is written anywhere until the request is approved.

  1. Source & target.

    Choose Copy namespace on Overview, or on the source namespace’s page. Choose the source cluster and namespace, then the target cluster and type the target namespace (shop-uat). It may be a new name on the same cluster, or the same name on another cluster.

    As soon as you type the target, Seylo checks whether it can write there at all, and says so on this step rather than after the review. Two things can stop it: the cluster’s namespace policy, and a namespace list fixed into the agent’s own install. A blocker here starts with This copy cannot run yet. See the agent’s own namespace list.

    If the target is outside the cluster’s namespace policy, an admin can tick Add the namespace to the cluster’s scope if it is outside it. The namespace is then added to the allowed namespaces when the copy is applied. A namespace the policy denies stays denied.

  2. Kinds to copy.

    Every copyable kind is selected by default except Job, because copying a completed Job runs it again in the target.

  3. Transforms.

    The namespace is renamed. You can rewrite image registries, override replicas, and point SecretProviderClasses at the target’s vault. Press Plan the copy (read-only).

  4. Config review.

    Hostnames in the target, values that name the source environment, every ConfigMap key, and every Secret key by name. See Secrets. If a copied workload mounts an Azure Key Vault SecretProviderClass, the Key Vault (CSI) secrets card is here too. See convert Key Vault secrets to Kubernetes Secrets. Press Apply the review and re-plan to move on.

  5. Preflight against the target.

    Every finding, with a severity of blocker, warning or info, and the list of what will be copied and what was skipped and why. Any blocker refuses submission. Two kinds of blocker ask a decision of you: missing dependencies and the Azure workload identity step.

  6. Review & submit.

    The YAML of every object as it will be committed, a note for the approver, and Submit for approval.

A workload that is copied without what it references does not start. So the plan checks every reference each copied workload’s pod template makes:

  • every ConfigMap and Secret it uses, and the key it reads, from env, envFrom, volumes and projected volumes;
  • every SecretProviderClass a Secrets Store CSI volume mounts, and its nodePublishSecretRef Secret;
  • every PersistentVolumeClaim it mounts;
  • its ServiceAccount, unless it is default, which Kubernetes creates in every namespace.

Each one must be in the copy, or already exist in the target. If it is neither, the plan is blocked, and the finding names the workload, the reference, and what the pods would do without it:

Missing reference What the finding says the pods do
ConfigMap or Secret, read as an env var its containers fail with CreateContainerConfigError
ConfigMap or Secret, mounted as a volume its pods stay ContainerCreating with FailedMount
SecretProviderClass its pods stay ContainerCreating with FailedMount
PersistentVolumeClaim its pods stay Pending (the claim does not exist)
ServiceAccount its ReplicaSet cannot create pods (serviceaccount not found)

A reference marked optional in the pod spec, and every imagePullSecrets entry, is a warning rather than a blocker, because Kubernetes starts the pod without it. The finding then ends Every reference to it is optional, so the pods start without it.

One missing object used by several workloads is one finding, not several.

A blocked dependency reads like this, with Fix one of: naming three ways out:

Deployment/checkout-api needs Secret "checkout-db", but the Secret kind is not selected for this copy, and shop-uat does not have it. Without it its containers fail with CreateContainerConfigError. Fix one of: select the Secret kind; create Secret "checkout-db" in shop-uat before approving; or skip Deployment/checkout-api (untick it in this finding). Referenced by: Deployment/checkout-api: env DB_PASSWORD (container checkout-api).
  1. Include it. The first option depends on why it is missing: include it (you left it out of this copy), select the Secret kind (its kind is not selected), create it in shop-staging and re-plan (the source does not have it either), or let Seylo read and write SecretProviderClasses (...) and re-plan (Seylo could not read that kind; the brackets name the agent upgrade. See permissions for SecretProviderClasses and PVCs).
  2. Create it in the target yourself, then plan again. This is the right fix for a reference the target should own, such as a database credential that must differ. It clears the block only if Seylo can read that kind in the target. If it cannot, the finding says whether shop-uat already has it could not be checked, and stays a blocker.
  3. Skip that workload. Under the finding, press Skip Deployment/checkout-api. The plan runs again without it. Workloads you skipped are listed under Left out in this review:, and Include Deployment/checkout-api again puts one back.

A key that the copy does not create is its own finding, dependency-key-missing. It happens when a key a workload reads is set to skip in the Secret review. It quotes the kubelet’s ("couldn't find key"), and its fix starts Fix: set that key to carry over or new value in the review instead of skip.

The plan reads every key name of every Secret it copies. It never reads a value for the plan, and no step of the wizard shows one.

For each key you choose what the target gets:

Choice What the target gets
carry-over The source value, copied at apply time, where the transport allows it
new-value A value you type now. It is encrypted into the approval request and scrubbed from it after apply
blank The key, created empty. The application will not work until it is filled in
skip No key at all

carry-over is the default where it is possible, and blank where it is not. Whether it is possible depends on how Seylo reaches the two clusters, and between two agent-managed clusters it never is. How a copy carries a Secret value explains each transport.

Keys whose names look like connection strings or addresses (DATABASE_URL, REDIS_HOST) carry a connection? badge and are listed together. Seylo cannot see their values, so it cannot tell whether a carried-over value still points at the source’s database. Tick the ones the target needs a different value for, and type the new value.

A pod that reads Azure Key Vault through workload identity authenticates as a managed identity. Azure trusts its token only from the exact ServiceAccount subject it was told about, system:serviceaccount:<namespace>:<serviceaccount>, and that subject names the namespace. A copy into a new namespace therefore needs a new federated credential on the managed identity. That is a change in Azure.

Seylo does not make it. It makes no Azure call and holds no Azure credentials. Instead, the plan detects the setup, blocks until you acknowledge the step, and gives you the commands to run.

It is detected from any one of:

  • a SecretProviderClass with provider azure and a clientID parameter;
  • a ServiceAccount annotated azure.workload.identity/client-id;
  • a pod template labelled azure.workload.identity/use: "true".

The finding is workload-identity-federation. Under it, in a box titled Run this yourself (Seylo does not), are the commands for each ServiceAccount the copied pods run as in the target. Seylo fills the client ID and the target subject from the copied objects. What only you know stays a <placeholder>. For checkout-api copied into shop-uat:

az identity list --query "[?clientId=='<client-id>'].{name:name, resourceGroup:resourceGroup}" -o table
az identity federated-credential create --name seylo-shop-uat-checkout-api --identity-name <identity-name> --resource-group <identity-resource-group> --issuer "$(az aks show --name <aks-cluster-name> --resource-group <aks-resource-group> --query oidcIssuerProfile.issuerUrl -o tsv)" --subject system:serviceaccount:shop-uat:checkout-api --audience api://AzureADTokenExchange

The first command finds the identity’s name and resource group. The plan shows the real client ID when it can read it from the copied objects, and <client-id> only when it cannot. The commands above are single lines that work in PowerShell and bash alike, once the placeholders are filled.

If you pointed the SecretProviderClass at another vault in 3. Transforms, a third command, with that vault’s name filled in, grants the identity read on it:

MSYS_NO_PATHCONV=1 az role assignment create --role "Key Vault Secrets User" --assignee <client-id> --scope "$(az keyvault show --name <vault-name> --query id -o tsv)"

MSYS_NO_PATHCONV=1 stops Git Bash on Windows from turning the vault’s resource ID, which starts with /subscriptions/, into a file path. Other bash shells ignore it. In PowerShell, leave it out and start at az.

Run them, then tick I have made the Azure change above (or will before this is approved). Seylo cannot make it: it holds no Azure credentials. The plan runs again, and the finding becomes a warning that ends ACKNOWLEDGED: the person submitting says it is done or will be before approval. The approver sees the same step and its commands on the request.

Seylo cannot check that you ran them. If you did not, the pods stay ContainerCreating because the Key Vault mount fails.

Two other Azure setups are reported, not blocked:

  • usePodIdentity: "true" (aad-pod-identity) is a warning: check that the AzureIdentityBinding covers the target namespace.
  • useVMManagedIdentity: "true" is info: the node’s identity is not bound to a namespace, so nothing changes in Azure.

Convert Key Vault (CSI) secrets to Kubernetes Secrets

Section titled “Convert Key Vault (CSI) secrets to Kubernetes Secrets”

A workload that mounts an Azure Key Vault SecretProviderClass starts only on a cluster with the Secrets Store CSI driver, the Azure provider and an identity Azure trusts. If the target has none of those, you can convert the SecretProviderClass into an ordinary Kubernetes Secret for this copy, so the copy runs on any cluster.

It is off by default, and you opt in per SecretProviderClass, per copy.

In 4. Config review, the Key Vault (CSI) secrets card lists every SecretProviderClass a copied workload mounts, with the workloads that mount it. Each has two choices:

  • Keep Key Vault (CSI), the default. The SecretProviderClass is copied as a definition, and the pods fetch values from the vault themselves.
  • Convert to a Kubernetes Secret. The plan runs again with it converted.

Before you choose, the card says what converting would create: a Secret named <secretproviderclass>-secrets, with one key per file the CSI volume produces, mounted at the same paths. If that name is taken in the target, Seylo adds -2, -3 and so on.

On approval, for each SecretProviderClass you convert:

  • the Secret is created in the target;
  • each workload’s CSI volume becomes a secret volume with the same volume name and the same files, so every mount path stays the same;
  • an env var that read a Secret the SecretProviderClass syncs is pointed at the new Secret. A workload that loads such a Secret whole, with envFrom, is converted only when the Secret’s keys are exactly the file names. Otherwise the variable names would change, and the plan blocks it;
  • the SecretProviderClass, and the Secrets it syncs, are left out of the copy and listed as skipped, with the reason.

Conversion reads the file names from the Azure Key Vault provider’s objects parameter. Another provider, or an objects value Seylo cannot read, cannot be converted: the card shows cannot be converted with the reason, and only Keep Key Vault (CSI) is available.

Seylo never reads a value out of Key Vault or out of a pod. Each file shows where its value will come from:

Badge Meaning
carried in-cluster The SecretProviderClass syncs this file to a Kubernetes Secret that exists in the source, and the copy is within one agent-managed cluster. The agent copies the value inside your cluster. It never reaches Seylo
you enter Only you can supply it. Type it in the Value of field for that key, marked write-only. The field is write-only: the value is encrypted into the approval request, scrubbed after the apply, and never shown again

For a carried file you can still choose enter a new value instead of carry over in-cluster. A converted file cannot be left blank or set to skip: an empty file would be mounted where the application expects a secret.

Carrying a value whose key name changes, or that comes from more than one Secret, needs agent 0.2.6. On an older agent the file says Carrying "…" from …/… needs the agent to rename the key (or read more than one Secret) in-cluster, which agent 0.2.6 adds; and asks you to upgrade the agent or enter the value.

As soon as one SecretProviderClass is set to convert, the card shows this warning, before anything is submitted or approved:

Converting a Key Vault (CSI) SecretProviderClass to a Kubernetes Secret makes the copy a SNAPSHOT: the values are written once, so a rotation in Key Vault will NOT reach it. Key Vault's access policies and its audit log no longer apply to these values - they live in the target cluster's Secret store, where anyone with read access to Secrets in that namespace can read them, and whether they are encrypted at rest depends on that cluster's configuration (etcd encryption / a KMS provider). Recommended for dev/test copies, not for production.

The copy is blocked, with the finding csi-conversion-unacknowledged, until you tick I understand: this copy is a snapshot. Key Vault rotations will not reach it, and its values live in the target cluster’s Secret store. The finding then becomes the warning csi-conversion-snapshot, which stays on the plan.

The approver sees Key Vault (CSI) converted to Kubernetes Secrets in this copy:, followed by each SecretProviderClass, the Secret it becomes and, for each key, whether the value is carried in-cluster or entered, then the same warning. Key names only, never a value. The audit row names each conversion.

Nothing is removed from the copied workloads. For a converted workload, the federated-credential step in Azure Key Vault and workload identity becomes a warning rather than a blocker, because its pods start without Key Vault. An application that calls Azure itself still needs the federated credential.

Conversion is offered only in the copy wizard. A coding-agent token has no copy tool, and it cannot accept the snapshot trade-off on a person’s behalf.

Submit locks once a copy is submitted: the button reads Submitted. A second copy into the same target namespace, while the first is still pending, is refused:

A copy into acme-staging/shop-uat is already waiting for approval as request … (submitted …). Two copies into one namespace would overwrite each other. Approve or reject that request first; to change what it copies, reject it and submit this one again.

To change a pending copy, reject it and submit the new one.

If Argo CD, Flux or Helm owns an object the copy would write in the target, the copy is refused, naming the owner, with a message beginning Refusing to change.

When the plan finds a controller-owned object among those it copies, step 6. Review & submit shows A resource in the target is owned by another controller with the box I understand the copy may be reverted or claimed, and want to proceed. Tick it and submit again. The acknowledgement is recorded on the request and in the audit log.

The box is shown only for an owner the plan found. If the refusal names an object and no box is shown, change that object at its controller’s source, or deselect its kind in 2. Kinds to copy.

Ownership labels and annotations from Helm, Argo CD and Flux are stripped from every copy, so the source’s controller cannot claim or revert it. The copy is not reconciled by that controller either.

The copy plans again from a fresh read of the source. If a blocker has appeared since you submitted, nothing is written:

Nothing was applied: re-planned at approval, … preflight check(s) now FAIL: …

Otherwise it:

  1. Creates the namespace first. The API server’s answer decides, not a list: Namespace shop-uat created (the API server answered 201). or Namespace shop-uat already existed (the API server answered AlreadyExists).
  2. Applies in dependency order, so no pod starts before what it reads exists: Namespace, ServiceAccounts, ConfigMaps, Secrets, SecretProviderClasses, PersistentVolumeClaims, Services, Deployments, StatefulSets, DaemonSets, CronJobs, Jobs, Ingresses, HorizontalPodAutoscalers, PodDisruptionBudgets, NetworkPolicies.
  3. Reads every object back, and for a Secret, only its key names.
  4. Re-checks every workload’s references in the target. A reference that is missing there fails the copy. It is not reported as verified.
  5. Watches the rollout of the copied workloads, as for a release.
  6. Scrubs any typed Secret values from the request, whether the copy succeeded or not.
You see What it means
Applied and verified against the live cluster (…) Every object read back as written, and every reference the copied workloads make is present in the target
Verification failed: NOT READY TO START: … A workload’s reference is missing in the target after the apply. The copy is failed, and the message names the workload and the reference
Applied. Verification was INCONCLUSIVE for: Could not read … Seylo could not read a kind in the target, so whether those pods can start is unknown. It is not reported as verified
Rollout FAILED for checkout-api: … A copied component had already failed for a reason Kubernetes retries forever, such as CreateContainerConfigError. Reported as failed when the watch times out: waiting longer will not help
Applied, but the rollout was STILL PROGRESSING when the …s watch timeout expired Nothing had failed, but the pods were not ready when Seylo stopped watching. Reported as Partial

Permissions for SecretProviderClasses and PVCs

Section titled “Permissions for SecretProviderClasses and PVCs”

On an agent-managed cluster, the copy reads and writes through the agent, so it can copy only what the agent’s Kubernetes RBAC allows.

From agent 0.2.6, the chart’s defaults are enough. The agent reads SecretProviderClasses and PersistentVolumeClaims, and tier 1 creates and patches them. It never deletes either. You do not need rbac.genericRead, rbac.write.destructive or any customResources entry to copy them.

What can still stop a workload that mounts one:

  • An agent older than 0.2.6. Its default grant cannot read either kind, so the plan blocks the workload. The first fix names the upgrade and the command:

    helm upgrade seylo-agent oci://ghcr.io/seylo-inc/charts/seylo-agent --namespace seylo-agent --reset-then-reuse-values

    --reset-then-reuse-values needs Helm 3.14 or later; see upgrade the agent for older Helm.

    Read upgrading to 0.2.6 before you run it: one value may need changing first.

  • Posture read-only or deploy, or rbac.write.tier1.enabled=false. The agent reads both kinds but cannot create them, because tier 1 is off. Turn tier 1 on, or create the object in the target yourself and plan again, or skip the workload.

Upgrade the cluster the copy reads from. If the target is a different agent-managed cluster, its agent does the writing, so upgrade that one too.

Creating the object in the target yourself helps only if Seylo can read that kind in the target. If it cannot, the plan cannot see your object, and the way out is to skip the workload.

An agent can be installed with a fixed list of namespaces. It then refuses every namespace outside that list, whatever Seylo allows. Seylo cannot widen that list remotely, by design. That list is how you cap Seylo from inside your own cluster, and Seylo cannot lift it.

If the target is outside the list, step 1 says so before you review anything:

The agent's install limits it to namespaces [shop, shop-staging], so it would refuse "shop-uat". Seylo cannot widen that list. To let Seylo manage every namespace it allows, remove it (cluster-wide RBAC installs): helm upgrade seylo-agent oci://ghcr.io/seylo-inc/charts/seylo-agent --namespace seylo-agent --reset-then-reuse-values --set watchNamespaces=null - or add "shop-uat" to it.

A cluster-wide agent installed without a list needs nothing on the cluster for a new or copied namespace. See a fixed list on a cluster-wide install.

You see What it means What to do
This copy cannot be submitted because … preflight check(s) FAILED against the target: At least one blocker stands Fix each one. The finding says how
dependency-missing, or … needs Secret "…", but … A copied workload references something neither copied nor in the target Include it, create it in the target, or press Skip …. See the three fixes
dependency-key-missing, or … WITHOUT it (it will have: …) A key a workload reads is set to skip Set that key to carry-over or new-value
dependency-keys-unknown, or … could not read which keys … has, so whether the clone starts cannot be known The plan could not read the Secret’s key names Re-plan. If it persists, create the Secret in the target yourself and deselect it
Seylo could not read SecretProviderClasses in … The agent’s RBAC cannot list them, usually because the agent is older than 0.2.6 Upgrade the agent. See permissions for SecretProviderClasses and PVCs
csi-conversion-unacknowledged A SecretProviderClass is set to convert, and the snapshot warning is not ticked Read the warning, then tick I understand: this copy is a snapshot…, or choose Keep Key Vault (CSI). See accept the snapshot warning
csi-conversion-blocked, or SecretProviderClass … cannot be converted to a Kubernetes Secret: Seylo cannot tell which files the volume produces, a file name cannot be a Secret key, no Secret name is free, or a workload’s env cannot be pointed at the new Secret The message gives the reason and the two ways out: choose Keep Key Vault (CSI) for it, or skip its workloads
A converted file needs a value: an empty or missing key would mount an empty file. blank or skip was chosen for a converted file, so it is treated as new-value Enter the value, choose Keep Key Vault (CSI) for that SecretProviderClass, or skip its workloads
workload-identity-federation The pods use Azure workload identity, and the target namespace needs a federated credential Run the commands, then tick the acknowledgement. See Azure Key Vault
agent-install-scope, or The agent's install limits it to namespaces […], so it would refuse "…" or The agent is limited to namespaces […] by its own install The agent was installed with a fixed list that excludes the target Remove the list or add the namespace, with the helm upgrade in the message. Seylo cannot do it for you
A copy into … is already waiting for approval as request … Another copy into this target is pending Approve or reject that request first
A replacement value is missing for … Secret key(s) set to 'new value': A key is set to new-value with no value typed Type the value, or change the key to carry-over or blank
Carry-over is NOT available between two agent-managed clusters: The value would have to leave the source cluster Type a new-value, or create the key blank and fill it in later
Carry-over inside this cluster needs agent 0.2.3 or later, … The agent is too old to copy a value inside the cluster Upgrade the agent, or choose new-value or blank
Nothing would be copied from …: no object of the selected kinds was found there. Nothing was submitted. The source has nothing of the kinds you selected Check the source namespace and the kinds
Refusing to change … A controller owns an object the copy would write in the target Tick the acknowledgement on 6. Review & submit, or change it at the controller’s source
Nothing was applied: re-planned at approval, … preflight check(s) now FAIL: The source or target changed after you submitted. Each failing check follows, with its code in brackets, such as [dependency-missing] Fix the findings and submit the copy again
Verification failed: NOT READY TO START: … A reference is missing in the target after the apply Create what it names in the target, then restart the workload
Rollout FAILED for …: CreateContainerConfigError … A pod reads a ConfigMap or Secret key that does not exist The message names the pod and container. Add the key in the target
Pods stay ContainerCreating with FailedMount on a Key Vault volume The federated credential is missing or wrong Check the az identity federated-credential create subject names the target namespace and ServiceAccount