From 35ad4948cf1537465bcf620c7040bbd652b65404 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Tue, 30 Jun 2026 15:59:36 +0500 Subject: [PATCH 1/2] [proposal] Per-cluster etcd: retire the tenant etcd module Fold etcd into the kubernetes app so each tenant Kubernetes cluster provisions and owns its own etcd and Kamaji DataStore. End state is one etcd per one Kubernetes cluster: consumers can start a cluster with no admin pre-step, control-plane state is physically isolated per cluster, and no etcd runs unless a cluster needs it. Co-Authored-By: Claude Opus 4.8 (1M context) Signed-off-by: Myasnikov Daniil --- design-proposals/etcd-per-cluster/README.md | 214 ++++++++++++++++++++ 1 file changed, 214 insertions(+) create mode 100644 design-proposals/etcd-per-cluster/README.md diff --git a/design-proposals/etcd-per-cluster/README.md b/design-proposals/etcd-per-cluster/README.md new file mode 100644 index 0000000..96012b1 --- /dev/null +++ b/design-proposals/etcd-per-cluster/README.md @@ -0,0 +1,214 @@ +# Per-cluster etcd for tenant Kubernetes + +- **Title:** `Per-cluster etcd: retire the tenant etcd module and bind etcd to the Kubernetes app` +- **Author(s):** `@myasnikovdaniil` +- **Date:** `2026-06-30` +- **Status:** Draft + +## Overview + +Today a tenant Kubernetes cluster does not own its datastore. etcd is a **tenant module** — an admin sets `spec.etcd: true` on a `Tenant`, which deploys one etcd cluster in that tenant's namespace and publishes a reference (`_namespace.etcd`) that is inherited down the whole tenant subtree. Every `Kubernetes` cluster created in that subtree points its Kamaji control plane at that **one shared etcd**. The result is two problems: a consumer who can only create a `Kubernetes` app cannot bring a cluster up until an admin has pre-provisioned etcd on an ancestor tenant (consumers cannot edit tenant parameters), and all clusters that do come up share a single etcd that was never meant to back more than one control plane. + +This proposal **retires etcd as a tenant module and folds it into the `kubernetes` app**: each `Kubernetes` cluster provisions and owns its own etcd and its own Kamaji `DataStore`. The end state is **one etcd per one Kubernetes cluster**, etcd lifecycle bound to cluster lifecycle (delete the cluster → its etcd is gone), and no etcd running anywhere unless a cluster needs it. This costs some per-cluster resource overhead, which we accept and make tunable — production users who care about isolation already pay it by carving a child tenant per cluster today. + +## Scope and related proposals + +- This proposal changes the `kubernetes` app (`packages/apps/kubernetes`), the `tenant` app (`packages/apps/tenant`), the etcd chart (`packages/extra/etcd`), the etcd tenant-module definition (`packages/system/etcd-rd`), and the cluster-bootstrap defaults (`packages/system/cozystack-basics`). It does **not** touch Kamaji, the etcd-operator, or the Cluster API stack. +- **Deferred / out of scope:** replacing etcd with a SQL-backed Kamaji datastore driver (kine/PostgreSQL) per cluster — a legitimate way to cut the three-replica etcd overhead, but an orthogonal datastore-driver decision that can layer on top of the per-cluster model proposed here. +- Related in spirit to other "an app declares all of its own dependencies" cleanups; no hard ordering dependency on another proposal. + +## Context + +A tenant Kubernetes control plane in Cozystack runs as a Kamaji `TenantControlPlane` (rendered via a CAPI `KamajiControlPlane`) on the management cluster. Kamaji does not run etcd itself — it points each control plane at a Kamaji `DataStore`, and Cozystack supplies that datastore from a separately deployed etcd. The wiring today: + +- **etcd is a tenant module.** `packages/apps/tenant/values.yaml` exposes `etcd: false`. When a `Tenant` sets it true, `packages/apps/tenant/templates/etcd.yaml` renders a `HelmRelease` (labeled `internal.cozystack.io/tenantmodule: "true"`) for the etcd chart in the tenant namespace. +- **The etcd chart owns the datastore.** `packages/extra/etcd` renders an `EtcdCluster` (`etcd.aenix.io/v1alpha1`, managed by the always-installed etcd-operator), its cert-manager CA/issuers/certs, and a Kamaji `DataStore` named after the **namespace** pointing at `etcd..svc:2379` (`packages/extra/etcd/templates/datastore.yaml:5-9`). `DataStore` is a **cluster-scoped** resource (`packages/system/kamaji/charts/kamaji/crds/kamaji.clastix.io_datastores.yaml:16`). +- **The reference propagates down the tenant tree.** `tenant-root` ships a hardcoded `_namespace.etcd: tenant-root` in its `cozystack-values` Secret (`packages/system/cozystack-basics/templates/cozystack-values-secret.yaml`). For nested tenants, `packages/apps/tenant/templates/namespace.yaml:22-25` inherits `_namespace.etcd` from the parent and only overrides it (to the current namespace) when *this* tenant sets `spec.etcd: true`. So every tenant in a subtree resolves `_namespace.etcd` to the nearest ancestor that owns an etcd. +- **The Kubernetes app consumes the shared reference.** `packages/apps/kubernetes/templates/cluster.yaml:21` reads `$etcd := .Values._namespace.etcd`. If it is empty, the chart renders **only** a `-awaiting-etcd` ConfigMap beacon and nothing else (`cluster.yaml:168-186`). If it is set, the chart renders the cluster with `dataStoreName: "{{ $etcd }}"` on the `KamajiControlPlane` (`cluster.yaml:291`). +- **One etcd, many control planes.** Because the `DataStore` is cluster-scoped and the reference is shared subtree-wide, every `KamajiControlPlane` in the subtree carries the **same** `dataStoreName`. Kamaji multiplexes them onto the one etcd by minting a per-control-plane connection Secret (`-datastore-config`) with its own credentials and key prefix — the isolation is logical (a shared Raft group, shared disk, shared `quota-backend-bytes`, shared CA), not physical. + +### The problem + +- *"I have access to my tenant. I create a `Kubernetes` app and it just sits there showing `awaiting-etcd`. I can't fix it — turning etcd on is a field on the Tenant, and I don't own the Tenant."* The most common first-run experience for a delegated consumer is a cluster that cannot start, blocked on a parameter they have no RBAC to set. +- *"Two teams each spun up a cluster in our tenant and now a compaction storm on one team's control plane is stalling the other's API server."* A single etcd sized and tuned once (`quota-backend-bytes`, compaction, snapshot count, PVC size, replica count) backs an unbounded number of control planes. There is no per-cluster performance envelope. +- *"Our security review flagged that all our clusters' control-plane state lives in one etcd behind one CA, reachable from every namespace in the subtree."* One etcd shared across trust boundaries is a single blast radius: one CA, one Service, one store; a noisy or compromised control plane is one logical prefix away from the others. This is not a production-grade isolation story. +- *"To get real isolation I create a separate child tenant for every cluster just so each gets its own etcd."* The isolation-conscious workaround already exists in the field — it is exactly the per-cluster etcd this proposal makes the default, minus the tenant-sprawl ceremony. + +## Goals + +- A consumer who can create a `Kubernetes` app can bring up a working cluster **with no admin pre-step** — no ancestor `spec.etcd: true`, no `awaiting-etcd` wait state. +- **One etcd per one Kubernetes cluster.** Each cluster's control-plane state is physically isolated in its own etcd cluster with its own CA, Service, PVCs, and tuning. +- etcd lifecycle is **bound to the cluster**: creating the `Kubernetes` app creates its etcd; deleting the app deletes its etcd, DataStore, and connection Secret. +- **No idle etcd.** With zero `Kubernetes` apps in a namespace, no `EtcdCluster` workload, PVCs, or DataStore exist there. (The cluster-wide etcd-operator stays; it is lightweight platform infrastructure and provisions nothing on its own.) +- Per-cluster etcd sizing is **tunable** from the `Kubernetes` app values (replicas, PVC size, resources), so the overhead is a knob, not a fixed tax. +- Existing shared-etcd clusters keep running across the upgrade and migrate on a controlled, documented path with no surprise data loss. + +### Non-goals + +- **Not** changing Kamaji, the etcd-operator, or the CAPI/KubeVirt provisioning path. Only *where* the etcd + DataStore come from changes. +- **Not** removing the etcd-operator or the etcd chart. The operator remains a default platform package; the chart is reused as the per-cluster etcd implementation rather than deleted. +- **Not** introducing a SQL/kine datastore driver to reduce overhead (deferred, see Scope). +- **Not** preserving standalone "etcd-as-a-tenant-app" as a first-class product surface. etcd ceases to be a tenant module; see Open questions for the standalone-etcd case. +- **Not** auto-migrating live control-plane data without operator opt-in — data movement between datastores is an explicit, gated step. + +## Design + +### Before / after + +```mermaid +flowchart TD + subgraph today[Today: shared tenant etcd] + direction TB + AdminT["Admin sets spec.etcd: true
on an ancestor Tenant"] --> ETM["etcd tenant module
(1 EtcdCluster + 1 DataStore)"] + ETM --> DS["DataStore: namespace
(cluster-scoped)"] + K1["Kubernetes app A"] -->|dataStoreName: shared| DS + K2["Kubernetes app B"] -->|dataStoreName: shared| DS + K3["Kubernetes app C"] -->|dataStoreName: shared| DS + end + + subgraph proposed[Proposed: per-cluster etcd] + direction TB + KA["Kubernetes app A"] --> EA["EtcdCluster A + DataStore A"] + KB["Kubernetes app B"] --> EB["EtcdCluster B + DataStore B"] + KC["Kubernetes app C"] --> EC["EtcdCluster C + DataStore C"] + end +``` + +### 1. The Kubernetes app provisions its own etcd + +The `kubernetes` chart renders, in the cluster's own namespace, the three things the etcd tenant module renders today — but scoped to the single cluster: + +1. an `EtcdCluster` (`etcd.aenix.io/v1alpha1`) named `-etcd`; +2. the cert-manager `Issuer`/CA/`Certificate` set backing that etcd's peer, server, and client TLS; +3. a cluster-scoped Kamaji `DataStore` (see naming below) pointing at `-etcd..svc:2379`. + +The `KamajiControlPlane` then sets `dataStoreName` to **its own** DataStore instead of the inherited shared reference: + +```yaml +# packages/apps/kubernetes/templates/cluster.yaml (after) +spec: + # ... + dataStoreName: {{ include "kubernetes.datastoreName" . }} # was: "{{ $etcd }}" +``` + +To avoid duplicating the etcd templates, the `kubernetes` chart **reuses the existing `packages/extra/etcd` chart** as the implementation — preferred as a declared subchart/dependency, with the etcd values block (below) threaded into it; the fallback is a small shared library of the EtcdCluster + cert templates imported by both charts. Either way `packages/extra/etcd` remains the single source of truth for "how a Cozystack etcd is shaped." + +### 2. New `etcd` values block on the Kubernetes app + +The per-cluster etcd is tunable from the `Kubernetes` CR, defaulting to the current tenant-module defaults so existing sizing is preserved: + +```yaml +# packages/apps/kubernetes/values.yaml (new section) +## @param {Etcd} etcd - Control-plane etcd datastore for this cluster. +etcd: + ## @field {int} replicas=3 - etcd replicas. Set 1 for non-HA / dev clusters. + replicas: 3 + ## @field {quantity} size=4Gi - PVC size per replica. + size: 4Gi + ## @field {string} [storageClass] - StorageClass for etcd PVCs. Empty = cluster default. + storageClass: "" + ## @field {Resources} resources - CPU/memory per replica. + resources: + cpu: 1000m + memory: 512Mi +``` + +This turns the overhead into an explicit, per-cluster decision: a production cluster keeps `replicas: 3`; a throwaway dev cluster can drop to `replicas: 1` and a smaller PVC. + +### 3. DataStore naming (cluster-scoped uniqueness) + +`DataStore` is cluster-scoped, so a per-cluster name **must** be globally unique across the management cluster. Today's name (the bare namespace) is no longer sufficient because two clusters can live in one namespace. The name becomes `-` (with a short hash suffix only if length limits bite): + +```yaml +# packages/extra/etcd/templates/datastore.yaml (after, when used per-cluster) +metadata: + name: {{ .Release.Namespace }}-{{ .Release.Name }} +spec: + driver: etcd + endpoints: + - {{ .Release.Name }}-etcd.{{ .Release.Namespace }}.svc:2379 +``` + +The matching `kubernetes.datastoreName` helper computes the identical value so the control plane and its datastore always agree. + +### 4. Removing the shared-etcd gate + +The `awaiting-etcd` beacon and the `_namespace.etcd` dependency disappear from the normal path. Because the `kubernetes` app now provisions its own datastore, the `{{- if not $etcd }}` branch in `cluster.yaml:168` is no longer how a cluster waits — it renders its etcd and proceeds. During the compatibility window (Rollout) the chart still **honors an explicitly-set legacy `_namespace.etcd`** for clusters that have not yet migrated, so the switch is opt-in per cluster. + +### 5. Retiring the tenant module + +Once clusters are migrated: + +- Remove `etcd` from `packages/apps/tenant/values.yaml` and delete `packages/apps/tenant/templates/etcd.yaml`. +- Drop the etcd tenant-module definition (`packages/system/etcd-rd`) from the tenant app catalog and remove `cozystack.etcd-application` from the tenant application source list (`packages/core/platform/sources/tenant-application.yaml`). +- Remove the hardcoded `etcd: tenant-root` and the `_namespace.etcd` propagation in `cozystack-basics` and `apps/tenant/templates/namespace.yaml`, plus the `cozy-lib.ns-etcd` helper (`packages/library/cozy-lib/templates/_cozyconfig.tpl:89-95`) and the `namespace.cozystack.io/etcd` NetworkPolicy label, after confirming no other app consumes them. + +The etcd-operator (`packages/core/platform/templates/bundles/system.yaml`) and the etcd chart stay. + +## User-facing changes + +- **Consumers:** creating a `Kubernetes` app now yields a working cluster with no admin pre-step. A new `etcd` block on the `Kubernetes` CR exposes replicas, size, storageClass, and resources. The `awaiting-etcd` status disappears. +- **Admins:** `spec.etcd` on a `Tenant` is deprecated then removed; etcd is no longer something to pre-provision per tenant. The etcd entry leaves the app catalog as a tenant module. +- **Dashboard/observability:** each cluster shows its own etcd `WorkloadMonitor` and metrics instead of one shared etcd per subtree. +- **Docs:** the kubernetes app README already states each cluster gets "a dedicated etcd cluster ... using etcd-operator" — the docs become *true by construction*. Update the tenant docs to drop the etcd module and add the per-cluster etcd sizing guidance and the migration runbook. + +## Upgrade and rollback compatibility + +- **Phased and opt-in.** New clusters self-provision etcd immediately. Existing clusters keep using their shared `_namespace.etcd` DataStore until explicitly migrated, because the chart honors a legacy reference during the compatibility window (Design §4). No cluster breaks on upgrade. +- **Migration of an existing cluster** moves control-plane data from its prefix in the shared etcd to its new dedicated etcd, then flips `dataStoreName`. Kamaji supports datastore migration for a control plane; the exact mechanism for the CAPI `KamajiControlPlane` variant (live migration vs. snapshot/restore) is called out under Open questions and Testing and must be proven before the migration step is automated. Where it can be scripted idempotently, it runs through the existing platform numbered-migration hook framework (`packages/core/platform/images/migrations`, which already has etcd-touching migrations such as `16` and `22`); otherwise it is a documented, operator-gated runbook. +- **Rollback (pre-migration):** revert the `kubernetes` app to consume `_namespace.etcd`; nothing was moved, shared etcd still has the data. +- **Rollback (post-migration):** a cluster that has moved to its own etcd would have to migrate its data *back* to roll back — flag this as effectively one-way per cluster. The dedicated etcd retains a snapshot to make a reverse migration possible but non-trivial. + +## Security + +- **Removes a shared trust boundary.** Each cluster gets its own etcd CA, Service, certificates, and PVCs. A control plane can no longer reach another cluster's datastore: today every control plane in a subtree dials the same `etcd..svc` (often cross-namespace, since the DataStore is cluster-scoped and the owner namespace is an ancestor); after this change the endpoint is the cluster's own in-namespace etcd. The existing `policy.cozystack.io/allow-to-etcd` control-plane pod label (`cluster.yaml:334`) is retargeted to the cluster's own etcd so NetworkPolicy stays tight. +- **No new tenant-supplied trust surface.** etcd config is bounded by the existing `Kubernetes` CR schema (replicas/size/resources); a consumer cannot point the control plane at an arbitrary external datastore. +- **Blast radius shrinks from subtree to cluster.** A compromised or resource-exhausted etcd now affects exactly one Kubernetes cluster. +- **Secrets:** per-cluster CA/peer/server/client Secrets are issued by cert-manager exactly as the tenant module does today, just one set per cluster. + +## Failure and edge cases + +- **Two `Kubernetes` apps in one namespace** → each renders its own `-etcd` and `-` DataStore; no name collision (the previous single shared `DataStore: ` could not represent two clusters distinctly). +- **DataStore name exceeds the cluster-scoped name length limit** → fall back to `--`; the helper enforces the limit deterministically. +- **Cluster deletion** → the delete hook (`packages/apps/kubernetes/templates/delete.yaml`) must additionally delete the `EtcdCluster`, the cluster-scoped `DataStore`, and the per-cluster cert Secrets, *after* the existing `-datastore-config` finalizer cleanup (delete.yaml already handles Kamaji's datastore-secret finalizer, issue #3062). Ordering: drain the control plane → strip the datastore-config finalizer → remove EtcdCluster + DataStore. +- **etcd not Ready when Kamaji reconciles** → the control plane stays not-Ready and Flux retries, the same self-healing loop as today's asynchronous datastore readiness; no hard failure. +- **Migration interrupted mid-flight** → the new etcd retains its snapshot and the old shared etcd still holds the source prefix; the migration is re-runnable and the cluster keeps serving from whichever `dataStoreName` is currently set. +- **Legacy reference still set after migration** → the cluster's own DataStore wins; the stale `_namespace.etcd` is ignored and removed during retirement. + +## Testing + +- **Helm unit tests** (`packages/apps/kubernetes/tests`): assert the cluster renders `EtcdCluster -etcd`, the per-cluster `DataStore` with the unique name and correct endpoint, and `dataStoreName` referencing it. Repurpose the existing `values-ci-no-etcd.yaml` scenario: "no inherited etcd" must now produce a fully-rendered cluster with its own etcd rather than the `awaiting-etcd` beacon. Add a two-clusters-in-one-namespace test asserting two distinct etcd + DataStore names. +- **etcd chart tests** (`packages/extra/etcd/tests`): assert the chart renders correctly under the per-cluster naming (release-scoped EtcdCluster/Service and `-` DataStore) as well as the legacy namespace-scoped name during the compatibility window. +- **e2e (`hack/e2e-apps/`):** create a `Kubernetes` app in a tenant with no ancestor etcd and assert it reaches Ready with its own etcd; create two clusters in one namespace and assert independent etcd Pods/PVCs; delete a cluster and assert its etcd, DataStore, and Secrets are gone and the namespace terminates cleanly. +- **Migration e2e (gating, must pass before automating the migration step):** stand up a shared-etcd cluster on the old path, run the migration, and assert the control plane stays continuously available (or within a documented brief window) and that data (a sentinel object) survives the datastore switch. + +## Rollout + +1. **Phase 1 — self-provisioned etcd, opt-in.** Add the `etcd` values block and per-cluster etcd/DataStore rendering to the `kubernetes` app behind a default that self-provisions for new clusters while still honoring a legacy `_namespace.etcd` reference for existing ones. Extend `delete.yaml` to reap the per-cluster etcd/DataStore. Ship docs for the new sizing knobs. +2. **Phase 2 — migration path.** Land and prove the shared→dedicated data migration (Kamaji datastore migration or snapshot/restore), as a numbered-migration hook where idempotent or a gated runbook otherwise. Provide a dashboard/CLI signal of which clusters are still on the shared etcd. +3. **Phase 3 — deprecate the tenant module.** Mark `Tenant.spec.etcd` and the etcd tenant-module catalog entry deprecated; new tenants no longer offer it. Stop hardcoding `etcd: tenant-root`. +4. **Phase 4 — remove plumbing.** Once telemetry shows no cluster on the shared etcd, delete `apps/tenant/templates/etcd.yaml`, the `etcd` tenant value, the `_namespace.etcd` propagation and `cozy-lib.ns-etcd` helper, and the `namespace.cozystack.io/etcd` label. Keep the etcd-operator and the etcd chart. + +## Open questions + +1. **Kamaji datastore migration mechanics for `KamajiControlPlane`.** Does the CAPI control-plane provider support a live datastore switch (Kamaji migrating data when `dataStoreName` changes), or must Phase 2 use etcd snapshot/restore of the per-control-plane prefix? This determines whether migration is zero-downtime and whether it can be a numbered-migration hook. +2. **Standalone etcd-as-an-app.** Is anyone using the etcd tenant module as a standalone datastore unrelated to Kubernetes? If so, keep `packages/extra/etcd` available as an `extra` app (decoupled from the k8s datastore role) rather than removing its catalog presence entirely. +3. **Default replicas.** Keep `replicas: 3` as the safe production default, or default to `1` for the smallest footprint and document the HA upgrade? The shared module defaulted to 3; this proposal keeps 3 but the per-cluster multiplier makes the default worth confirming. +4. **Subchart vs. shared library.** Consume `packages/extra/etcd` as a declared subchart of `kubernetes`, or factor the EtcdCluster + cert templates into a shared library both charts import? Affects values plumbing and the build. +5. **Quota interaction.** Per-cluster etcd now counts against tenant `resourceQuotas`. Should the `kubernetes` app surface the etcd footprint in its sizing/NOTES so consumers see the full cost up front? + +## Alternatives considered + +- **Keep etcd shared but one etcd per tenant (status quo, better documented).** Rejected: still N clusters on 1 etcd within a tenant, so it solves neither the performance-isolation nor the self-service problem. It only renames the workaround. +- **Require the consumer to create a separate `etcd` app before the Kubernetes app.** Rejected: worse UX than today (two apps, explicit ordering, a dangling etcd if the cluster is deleted) and re-exposes a tenant-module concept we are trying to remove. Folding etcd into the cluster keeps it one app and one lifecycle. +- **Per-cluster SQL/kine datastore instead of etcd.** A real way to cut the three-replica etcd overhead, but it is an orthogonal datastore-driver change layered on Kamaji and is deferred (Scope). The per-cluster *ownership* model here is a prerequisite for it either way. +- **Automatic, unconditional migration of all shared-etcd clusters on upgrade.** Rejected: moving live control-plane data is too risky to do without operator opt-in. Migration is gated and per-cluster, with the shared etcd retained until each cluster has moved. +- **Delete the etcd chart and inline everything into the kubernetes app.** Rejected: duplicates the EtcdCluster + cert-manager shapes and the etcd backup/strategy integration. Reusing `packages/extra/etcd` keeps one source of truth. + +--- + + From 6e1ceef3dc18582d3e5d70404cc52c465b739d8b Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Fri, 21 Aug 2026 12:57:56 +0500 Subject: [PATCH 2/2] docs(etcd-per-cluster): reframe as a control-plane app change and close all open questions Second draft of the per-cluster etcd proposal. The framing changes and every open question from the first review round is answered. Reframing: the proposal is about completing packages/apps/kubernetes, which since the control-plane / node-pool split (community#8) is the control-plane app. Every control-plane component is rendered by that chart except the one holding all of its state. Retiring the etcd tenant-module wiring is a consequence of fixing that, not the goal. This makes the relationship to community#39 explicit: #39 owns where packages live and how modules declare themselves, this owns what the control-plane app contains. Disjoint files, neither blocks the other. Resolved questions, replacing the five open ones: - Migration: the CAPI provider does re-project dataStoreName onto TenantControlPlane.spec.dataStore on every reconcile, so a live switch works, but it costs a read-only freeze window and a kubelet restart on every tenant worker node. Migration is therefore an operator-initiated maintenance runbook, never an unattended platform migration. - Standalone etcd: survives. Only the Tenant.spec.etcd bool and the _namespace.etcd propagation retire; the chart and its ApplicationDefinition stay. - Default replicas: 3. - Reuse mechanism: shared named templates in cozy-lib, not a subchart. The etcd chart hard-fails unless the release is named etcd, a subchart inherits the parent release name, and the literal etcd-*-tls names are pinned by an immutable spec.tls on adopted clusters. - Footprint: surfaced in NOTES.txt and the app README. Also folded in: the minimal values surface from cozystack#3179 (only etcd.replicas, with the StorageClass derived from it at platform level), etcd-operator v1alpha2 and the Available condition throughout, a name-length section with the real 53-character release cap and the operator's derived names, a delete hook that reaps a per-cluster etcd but never a shared one, corrected source citations, and a migration slot of 56+ rather than the stale 49. Assisted-By: Claude Signed-off-by: Myasnikov Daniil --- design-proposals/etcd-per-cluster/README.md | 295 ++++++++++++-------- 1 file changed, 176 insertions(+), 119 deletions(-) diff --git a/design-proposals/etcd-per-cluster/README.md b/design-proposals/etcd-per-cluster/README.md index 96012b1..ece0dcf 100644 --- a/design-proposals/etcd-per-cluster/README.md +++ b/design-proposals/etcd-per-cluster/README.md @@ -1,55 +1,69 @@ -# Per-cluster etcd for tenant Kubernetes +# Per-cluster etcd: the Kubernetes control-plane app owns its datastore -- **Title:** `Per-cluster etcd: retire the tenant etcd module and bind etcd to the Kubernetes app` +- **Title:** `Per-cluster etcd: give each Kubernetes control plane its own datastore` - **Author(s):** `@myasnikovdaniil` -- **Date:** `2026-06-30` -- **Status:** Draft +- **Date:** `2026-06-30` (revised `2026-08-21`) +- **Status:** Review + + ## Overview -Today a tenant Kubernetes cluster does not own its datastore. etcd is a **tenant module** — an admin sets `spec.etcd: true` on a `Tenant`, which deploys one etcd cluster in that tenant's namespace and publishes a reference (`_namespace.etcd`) that is inherited down the whole tenant subtree. Every `Kubernetes` cluster created in that subtree points its Kamaji control plane at that **one shared etcd**. The result is two problems: a consumer who can only create a `Kubernetes` app cannot bring a cluster up until an admin has pre-provisioned etcd on an ancestor tenant (consumers cannot edit tenant parameters), and all clusters that do come up share a single etcd that was never meant to back more than one control plane. +A Kubernetes control plane needs a datastore. Since the control-plane / node-pool split ([`kubernetes-nodes-split`](https://github.com/cozystack/community/pull/8), Accepted), `packages/apps/kubernetes` **is** the control-plane app — it renders the Kamaji control plane, its certificates, its konnectivity, its CSI and cloud-controller satellites, and its OIDC wiring, while worker pools live in `packages/apps/kubernetes-nodes`. Everything a control plane is made of is in that chart except the one component holding all of its state. etcd is instead supplied from outside, by an admin flipping `spec.etcd: true` on an ancestor `Tenant`. + +This proposal completes the control-plane app: **`apps/kubernetes` renders its own `EtcdCluster` and its own Kamaji `DataStore`, one per cluster.** A cluster's datastore becomes as much a part of the cluster as its API server — created with it, sized with it, deleted with it, isolated to it. The end state is one etcd per one Kubernetes cluster, and no etcd running anywhere that no cluster needs. -This proposal **retires etcd as a tenant module and folds it into the `kubernetes` app**: each `Kubernetes` cluster provisions and owns its own etcd and its own Kamaji `DataStore`. The end state is **one etcd per one Kubernetes cluster**, etcd lifecycle bound to cluster lifecycle (delete the cluster → its etcd is gone), and no etcd running anywhere unless a cluster needs it. This costs some per-cluster resource overhead, which we accept and make tunable — production users who care about isolation already pay it by carving a child tenant per cluster today. +Two consequences follow, and they are consequences rather than goals. Consumers stop being blocked on an admin pre-step, because a `Kubernetes` app no longer depends on a parameter its creator has no RBAC to set. And the tenant-module *wiring* for etcd — the `Tenant.spec.etcd` bool and the `_namespace.etcd` reference propagated down the tenant tree — has nothing left to serve, so it retires. The etcd chart itself and the standalone etcd app both survive; see Resolved question 2. ## Scope and related proposals -- This proposal changes the `kubernetes` app (`packages/apps/kubernetes`), the `tenant` app (`packages/apps/tenant`), the etcd chart (`packages/extra/etcd`), the etcd tenant-module definition (`packages/system/etcd-rd`), and the cluster-bootstrap defaults (`packages/system/cozystack-basics`). It does **not** touch Kamaji, the etcd-operator, or the Cluster API stack. -- **Deferred / out of scope:** replacing etcd with a SQL-backed Kamaji datastore driver (kine/PostgreSQL) per cluster — a legitimate way to cut the three-replica etcd overhead, but an orthogonal datastore-driver decision that can layer on top of the per-cluster model proposed here. -- Related in spirit to other "an app declares all of its own dependencies" cleanups; no hard ordering dependency on another proposal. +The important framing point first, because the first review round read this proposal as a competing packaging change: + +- **[community#39](https://github.com/cozystack/community/pull/39) — Fold `extra` into `apps`.** **Companion, not competitor.** #39 answers *where packages live and how a module declares itself* — it retires the `packages/extra` directory and turns "hidden from the catalog", "one per tenant", and "shared down the tenant tree" into declarative `ApplicationDefinition` capabilities. This proposal answers a different question: *what the Kubernetes control-plane app is made of.* It adds one component to one chart. The two touch disjoint files (#39 touches directory paths, `-rd` definitions, and the Tenant chart's generic capability plumbing; this proposal touches `apps/kubernetes` templates plus the etcd-specific keys in the Tenant chart), and neither blocks the other. If both land, #39 relocates the etcd chart along with everything else in `extra/` and this proposal's shared template follows it; if only one lands, both still make sense on their own. The one place they meet is that etcd is the single tenant module whose *sharing semantics we intentionally drop* — #39 keeps sharing for monitoring, seaweedfs, ingress, and gateway, which is exactly why etcd is out of its scope and in this one. +- **[community#8](https://github.com/cozystack/community/pull/8) — Control-plane / node-pool split (Accepted, `design-proposals/kubernetes-nodes-split/`).** This proposal builds directly on it. Because worker pools moved out to `apps/kubernetes-nodes`, `apps/kubernetes` is now unambiguously the control-plane chart, and "the control plane owns its datastore" is a coherent statement about it rather than a statement about a chart that also happens to manage VMs. +- **[cozystack#3179](https://github.com/cozystack/cozystack/issues/3179) — Expose only `etcd.replicas`.** Filed out of the first review round to capture the minimal-API argument. **Adopted here** (Design §2), which is what #3179 asked for; the issue is gated on this proposal and can close with it. +- **[cozystack#2859](https://github.com/cozystack/cozystack/pull/2859) / [#3270](https://github.com/cozystack/cozystack/pull/3270) — etcd-operator v1alpha2 (both merged).** The first draft was written against `etcd.aenix.io/v1alpha1`. That is now historical; this revision targets `etcd-operator.cozystack.io/v1alpha2`, the `Available` readiness condition, and the `secretRef` TLS shape throughout. No dependency remains to declare — the dependency landed. +- **[community#33](https://github.com/cozystack/community/pull/33) — ComputePlane (`packages/extra/computeplane`).** Adjacent but independent. ComputePlane wraps `apps/kubernetes`, so it inherits whatever datastore behaviour that chart has, and its README documents the `awaiting-etcd` state that Design §4 removes. It needs a docs update in Phase 3, nothing more. +- **Deferred / out of scope:** a SQL-backed Kamaji datastore driver (kine/PostgreSQL) per cluster. It is a real way to cut the three-replica etcd cost, but it is an orthogonal driver choice that layers on top of per-cluster *ownership* — the model proposed here is a prerequisite for it either way. ## Context -A tenant Kubernetes control plane in Cozystack runs as a Kamaji `TenantControlPlane` (rendered via a CAPI `KamajiControlPlane`) on the management cluster. Kamaji does not run etcd itself — it points each control plane at a Kamaji `DataStore`, and Cozystack supplies that datastore from a separately deployed etcd. The wiring today: +A tenant Kubernetes control plane runs on the management cluster as a Kamaji `TenantControlPlane`, rendered through a CAPI `KamajiControlPlane`. Kamaji does not run etcd; it points each control plane at a Kamaji `DataStore`, and Cozystack supplies that datastore from a separately deployed etcd. The wiring today: -- **etcd is a tenant module.** `packages/apps/tenant/values.yaml` exposes `etcd: false`. When a `Tenant` sets it true, `packages/apps/tenant/templates/etcd.yaml` renders a `HelmRelease` (labeled `internal.cozystack.io/tenantmodule: "true"`) for the etcd chart in the tenant namespace. -- **The etcd chart owns the datastore.** `packages/extra/etcd` renders an `EtcdCluster` (`etcd.aenix.io/v1alpha1`, managed by the always-installed etcd-operator), its cert-manager CA/issuers/certs, and a Kamaji `DataStore` named after the **namespace** pointing at `etcd..svc:2379` (`packages/extra/etcd/templates/datastore.yaml:5-9`). `DataStore` is a **cluster-scoped** resource (`packages/system/kamaji/charts/kamaji/crds/kamaji.clastix.io_datastores.yaml:16`). -- **The reference propagates down the tenant tree.** `tenant-root` ships a hardcoded `_namespace.etcd: tenant-root` in its `cozystack-values` Secret (`packages/system/cozystack-basics/templates/cozystack-values-secret.yaml`). For nested tenants, `packages/apps/tenant/templates/namespace.yaml:22-25` inherits `_namespace.etcd` from the parent and only overrides it (to the current namespace) when *this* tenant sets `spec.etcd: true`. So every tenant in a subtree resolves `_namespace.etcd` to the nearest ancestor that owns an etcd. -- **The Kubernetes app consumes the shared reference.** `packages/apps/kubernetes/templates/cluster.yaml:21` reads `$etcd := .Values._namespace.etcd`. If it is empty, the chart renders **only** a `-awaiting-etcd` ConfigMap beacon and nothing else (`cluster.yaml:168-186`). If it is set, the chart renders the cluster with `dataStoreName: "{{ $etcd }}"` on the `KamajiControlPlane` (`cluster.yaml:291`). -- **One etcd, many control planes.** Because the `DataStore` is cluster-scoped and the reference is shared subtree-wide, every `KamajiControlPlane` in the subtree carries the **same** `dataStoreName`. Kamaji multiplexes them onto the one etcd by minting a per-control-plane connection Secret (`-datastore-config`) with its own credentials and key prefix — the isolation is logical (a shared Raft group, shared disk, shared `quota-backend-bytes`, shared CA), not physical. +- **etcd is a tenant module.** `packages/apps/tenant/values.yaml:8-9` exposes `etcd: false`. When a `Tenant` sets it true, `packages/apps/tenant/templates/etcd.yaml` renders a `HelmRelease` for the etcd chart in that tenant's namespace, labelled `internal.cozystack.io/tenantmodule: "true"`. +- **The etcd chart owns the datastore.** `packages/extra/etcd` renders an `EtcdCluster` (`etcd-operator.cozystack.io/v1alpha2`, `templates/etcd-cluster.yaml:11`), its cert-manager CA/issuers/certificates, and a Kamaji `DataStore` named after the **namespace** pointing at `etcd..svc:2379` (`templates/datastore.yaml:5-8`). `DataStore` is **cluster-scoped** (`packages/system/kamaji/charts/kamaji/crds/kamaji.clastix.io_datastores.yaml:16`). +- **Every name in that chart is the literal string `etcd`.** `EtcdCluster/etcd`; the Secrets `etcd-ca-tls`, `etcd-client-tls`, `etcd-server-tls`, `etcd-peer-tls`; the operator-owned Service `etcd`. The chart enforces this: `templates/check-release-name.yaml` calls `fail` unless `.Release.Name == .Chart.Name`. These names are load-bearing rather than cosmetic — `spec.tls` is immutable in v1alpha2 (CRD CEL `self.tls == oldSelf.tls`) and `etcd-migrate` adopts legacy clusters into exactly this `secretRef` shape, so renaming them would make every post-adoption reconcile fail. Design §1 works with this constraint instead of against it. +- **The reference propagates down the tenant tree.** `tenant-root` ships a hardcoded `_namespace.etcd: tenant-root` in its `cozystack-values` Secret (`packages/system/cozystack-basics/templates/cozystack-values-secret.yaml:17`). For nested tenants, `packages/apps/tenant/templates/namespace.yaml:28-30` inherits `_namespace.etcd` from the parent and overrides it with the current tenant only when *this* tenant sets `spec.etcd: true`; the resolved value is written to the namespace label `namespace.cozystack.io/etcd` (`namespace.yaml:86`) and into descendants' `_namespace.etcd` (`namespace.yaml:120`). Consumers read it through `cozy-lib.ns-etcd` (`packages/library/cozy-lib/templates/_cozyconfig.tpl:116-118`). So every tenant in a subtree resolves `_namespace.etcd` to the nearest ancestor that owns an etcd. +- **The control-plane app consumes the shared reference.** `packages/apps/kubernetes/templates/cluster.yaml:27` reads `$etcd := .Values._namespace.etcd`. If it is empty the chart renders **only** a `-awaiting-etcd` ConfigMap beacon and nothing else (`cluster.yaml:226-244`); `templates/ingress.yaml` and `hack/admin-kubeconfig-invariant.bats` key off the same state. If it is set, the `KamajiControlPlane` carries `dataStoreName: "{{ $etcd }}"` (`cluster.yaml:422`). +- **One etcd, many control planes.** Because the `DataStore` is cluster-scoped and the reference is shared subtree-wide, every `KamajiControlPlane` in a subtree carries the **same** `dataStoreName`. Kamaji multiplexes them onto the one etcd by minting a per-control-plane connection Secret (`-datastore-config`) with its own credentials and key prefix. The isolation is logical — one Raft group, one disk, one `quotaBackendBytes`, one CA — not physical. ### The problem -- *"I have access to my tenant. I create a `Kubernetes` app and it just sits there showing `awaiting-etcd`. I can't fix it — turning etcd on is a field on the Tenant, and I don't own the Tenant."* The most common first-run experience for a delegated consumer is a cluster that cannot start, blocked on a parameter they have no RBAC to set. -- *"Two teams each spun up a cluster in our tenant and now a compaction storm on one team's control plane is stalling the other's API server."* A single etcd sized and tuned once (`quota-backend-bytes`, compaction, snapshot count, PVC size, replica count) backs an unbounded number of control planes. There is no per-cluster performance envelope. -- *"Our security review flagged that all our clusters' control-plane state lives in one etcd behind one CA, reachable from every namespace in the subtree."* One etcd shared across trust boundaries is a single blast radius: one CA, one Service, one store; a noisy or compromised control plane is one logical prefix away from the others. This is not a production-grade isolation story. -- *"To get real isolation I create a separate child tenant for every cluster just so each gets its own etcd."* The isolation-conscious workaround already exists in the field — it is exactly the per-cluster etcd this proposal makes the default, minus the tenant-sprawl ceremony. +- *"I have access to my tenant. I create a `Kubernetes` app and it just sits there saying `awaiting-etcd`. I can't fix it — etcd is a field on the Tenant, and I don't own the Tenant."* The most common first-run experience for a delegated consumer is a cluster that cannot start, blocked on a parameter they have no RBAC to set. +- *"Two teams each spun up a cluster in our tenant, and now a compaction storm on one team's control plane is stalling the other's API server."* One etcd, sized and tuned exactly once, backs an unbounded number of control planes. There is no per-cluster performance envelope, and nothing about a cluster's own configuration can give it one. +- *"Our security review flagged that every cluster's control-plane state lives in one etcd behind one CA."* One etcd shared across trust boundaries is one blast radius. A noisy or compromised control plane is one logical key prefix away from its neighbours. +- *"To get real isolation we create a separate child tenant per cluster, just so each cluster gets its own etcd."* The workaround already exists in the field. It is precisely the per-cluster etcd this proposal makes the default, minus the tenant-sprawl ceremony. +- And the plain structural version, independent of any user story: **the control-plane chart does not contain the control plane's state.** Every other control-plane component is rendered by `apps/kubernetes` and versioned with it. The datastore is the one exception, and every problem above is downstream of that exception. ## Goals -- A consumer who can create a `Kubernetes` app can bring up a working cluster **with no admin pre-step** — no ancestor `spec.etcd: true`, no `awaiting-etcd` wait state. -- **One etcd per one Kubernetes cluster.** Each cluster's control-plane state is physically isolated in its own etcd cluster with its own CA, Service, PVCs, and tuning. -- etcd lifecycle is **bound to the cluster**: creating the `Kubernetes` app creates its etcd; deleting the app deletes its etcd, DataStore, and connection Secret. -- **No idle etcd.** With zero `Kubernetes` apps in a namespace, no `EtcdCluster` workload, PVCs, or DataStore exist there. (The cluster-wide etcd-operator stays; it is lightweight platform infrastructure and provisions nothing on its own.) -- Per-cluster etcd sizing is **tunable** from the `Kubernetes` app values (replicas, PVC size, resources), so the overhead is a knob, not a fixed tax. -- Existing shared-etcd clusters keep running across the upgrade and migrate on a controlled, documented path with no surprise data loss. +- `apps/kubernetes` renders its own `EtcdCluster` and its own cluster-scoped `DataStore`, and points its `KamajiControlPlane` at them. One etcd per one Kubernetes cluster. +- etcd lifecycle is **bound to the cluster**: creating the app creates its etcd; deleting the app reaps its etcd, `DataStore`, certificates, and connection Secret. +- **No idle etcd.** With zero `Kubernetes` apps in a namespace, no `EtcdCluster`, PVC, or `DataStore` exists there. (The cluster-wide etcd-operator stays; it provisions nothing on its own.) +- A consumer who can create a `Kubernetes` app gets a working cluster with **no admin pre-step** — no ancestor `spec.etcd: true`, no `awaiting-etcd` wait state. +- Each cluster's control-plane state is **physically isolated**: its own Raft group, disk, quota, CA, and Service. +- The user-facing surface added is **one field** (`etcd.replicas`), matching the convention the chart already follows for its other infrastructure satellites. +- Existing shared-etcd clusters keep running across the upgrade, indefinitely and without action, and migrate only when an operator chooses to migrate them. +- The etcd chart stays a single source of truth for "how a Cozystack etcd is shaped", with no duplicated `EtcdCluster` or cert-manager templates. ### Non-goals -- **Not** changing Kamaji, the etcd-operator, or the CAPI/KubeVirt provisioning path. Only *where* the etcd + DataStore come from changes. -- **Not** removing the etcd-operator or the etcd chart. The operator remains a default platform package; the chart is reused as the per-cluster etcd implementation rather than deleted. -- **Not** introducing a SQL/kine datastore driver to reduce overhead (deferred, see Scope). -- **Not** preserving standalone "etcd-as-a-tenant-app" as a first-class product surface. etcd ceases to be a tenant module; see Open questions for the standalone-etcd case. -- **Not** auto-migrating live control-plane data without operator opt-in — data movement between datastores is an explicit, gated step. +- **Not** changing Kamaji, the etcd-operator, the CAPI stack, or the KubeVirt provisioning path. Only *where the datastore comes from* changes. +- **Not** removing the etcd chart or the standalone etcd app (Resolved question 2). +- **Not** introducing a SQL/kine datastore driver (deferred, see Scope). +- **Not** migrating live control-plane data automatically. Data movement is operator-initiated, per cluster, and never a side effect of an upgrade (Resolved question 1). +- **Not** changing package layout, directory boundaries, or how modules declare themselves — that is [#39](https://github.com/cozystack/community/pull/39). +- **Not** re-litigating the control-plane / node-pool split; this proposal assumes it. ## Design @@ -57,153 +71,196 @@ A tenant Kubernetes control plane in Cozystack runs as a Kamaji `TenantControlPl ```mermaid flowchart TD - subgraph today[Today: shared tenant etcd] + subgraph today["Today: datastore supplied from outside the app"] direction TB - AdminT["Admin sets spec.etcd: true
on an ancestor Tenant"] --> ETM["etcd tenant module
(1 EtcdCluster + 1 DataStore)"] - ETM --> DS["DataStore: namespace
(cluster-scoped)"] - K1["Kubernetes app A"] -->|dataStoreName: shared| DS - K2["Kubernetes app B"] -->|dataStoreName: shared| DS - K3["Kubernetes app C"] -->|dataStoreName: shared| DS + AdminT["Admin sets spec.etcd: true
on an ancestor Tenant"] --> ETM["etcd tenant module
1 EtcdCluster + 1 DataStore"] + ETM --> DS["DataStore: <namespace>
(cluster-scoped)"] + K1["kubernetes app A"] -->|"dataStoreName: shared"| DS + K2["kubernetes app B"] -->|"dataStoreName: shared"| DS + K3["kubernetes app C"] -->|"dataStoreName: shared"| DS end - subgraph proposed[Proposed: per-cluster etcd] + subgraph proposed["Proposed: datastore is part of the control-plane app"] direction TB - KA["Kubernetes app A"] --> EA["EtcdCluster A + DataStore A"] - KB["Kubernetes app B"] --> EB["EtcdCluster B + DataStore B"] - KC["Kubernetes app C"] --> EC["EtcdCluster C + DataStore C"] + KA["kubernetes app A"] --> EA["EtcdCluster A
+ DataStore A"] + KB["kubernetes app B"] --> EB["EtcdCluster B
+ DataStore B"] + KC["kubernetes app C"] --> EC["EtcdCluster C
+ DataStore C"] end ``` -### 1. The Kubernetes app provisions its own etcd +### 1. One shared template, two callers -The `kubernetes` chart renders, in the cluster's own namespace, the three things the etcd tenant module renders today — but scoped to the single cluster: +`apps/kubernetes` needs the same three shapes the etcd chart already renders — an `EtcdCluster`, its cert-manager issuer/CA/certificate set, and a Kamaji `DataStore` — but under per-cluster names. The etcd chart cannot be consumed as a subchart to get them, for three concrete reasons established in Context: `check-release-name.yaml` hard-fails unless the release is named `etcd`; a subchart is rendered with the *parent's* `.Release.Name`, so it would fail that guard and then produce parent-named objects anyway; and the literal `etcd-{ca,client,server,peer}-tls` names cannot simply be made release-scoped, because adopted legacy clusters reference them through an immutable `spec.tls`. + +So the shapes move into a **named template in `cozy-lib`, parameterised by name**, and both charts call it. `packages/library/cozy-lib` is already symlinked into both charts (`packages/apps/kubernetes/charts/cozy-lib` is a symlink, as is `packages/extra/etcd/charts/cozy-lib`), so there is no new dependency, no `charts/` vendoring step, and no build change: + +```gotemplate +{{/* packages/library/cozy-lib/templates/_etcd.tpl */}} +{{- define "cozy-lib.etcd.cluster" -}} +{{/* args: (list ) */}} +{{- end }} +{{- define "cozy-lib.etcd.certificates" -}}{{- end }} +{{- define "cozy-lib.etcd.datastore" -}}{{- end }} +``` -1. an `EtcdCluster` (`etcd.aenix.io/v1alpha1`) named `-etcd`; -2. the cert-manager `Issuer`/CA/`Certificate` set backing that etcd's peer, server, and client TLS; -3. a cluster-scoped Kamaji `DataStore` (see naming below) pointing at `-etcd..svc:2379`. +`packages/extra/etcd` calls them with `"etcd"` and keeps rendering byte-identical output — same `EtcdCluster/etcd`, same Secret names, same `DataStore/`, same adoption compatibility, same immutable `spec.tls`. Nothing about the standalone chart or an adopted cluster changes. `apps/kubernetes` calls them with its own per-cluster name (§3) and gets a fresh cluster, where there is no adopted object to stay shape-compatible with and the names are therefore free. -The `KamajiControlPlane` then sets `dataStoreName` to **its own** DataStore instead of the inherited shared reference: +The `KamajiControlPlane` then points at its own datastore: ```yaml -# packages/apps/kubernetes/templates/cluster.yaml (after) -spec: - # ... - dataStoreName: {{ include "kubernetes.datastoreName" . }} # was: "{{ $etcd }}" +# packages/apps/kubernetes/templates/cluster.yaml:422 (after) + dataStoreName: {{ include "kubernetes.etcdName" . | quote }} # was: "{{ $etcd }}" ``` -To avoid duplicating the etcd templates, the `kubernetes` chart **reuses the existing `packages/extra/etcd` chart** as the implementation — preferred as a declared subchart/dependency, with the etcd values block (below) threaded into it; the fallback is a small shared library of the EtcdCluster + cert templates imported by both charts. Either way `packages/extra/etcd` remains the single source of truth for "how a Cozystack etcd is shaped." +Extracting the templates is mechanically the largest piece of work in this proposal and is behaviour-neutral by construction: Phase 1 lands the extraction alone, and the etcd chart's existing helm-unittest suite (`packages/extra/etcd/tests/`) proves the rendered output did not move before any per-cluster code is written. -### 2. New `etcd` values block on the Kubernetes app +### 2. One new field: `etcd.replicas` -The per-cluster etcd is tunable from the `Kubernetes` CR, defaulting to the current tenant-module defaults so existing sizing is preserved: +The `Kubernetes` CR gains exactly one etcd knob: ```yaml # packages/apps/kubernetes/values.yaml (new section) ## @param {Etcd} etcd - Control-plane etcd datastore for this cluster. etcd: - ## @field {int} replicas=3 - etcd replicas. Set 1 for non-HA / dev clusters. + ## @field {int} replicas=3 - etcd replicas backing this cluster's control plane. + ## 3 = HA via Raft quorum. 1 = single-member, non-HA; the chart selects a + ## replicated StorageClass so the lone member survives node loss at the + ## storage layer. Values other than 1 and 3 are rejected. replicas: 3 - ## @field {quantity} size=4Gi - PVC size per replica. - size: 4Gi - ## @field {string} [storageClass] - StorageClass for etcd PVCs. Empty = cluster default. - storageClass: "" - ## @field {Resources} resources - CPU/memory per replica. - resources: - cpu: 1000m - memory: 512Mi ``` -This turns the overhead into an explicit, per-cluster decision: a production cluster keeps `replicas: 3`; a throwaway dev cluster can drop to `replicas: 1` and a smaller PVC. +`size`, `resources`, and `storageClass` are deliberately **not** exposed, and this is the point argued in review and captured in [#3179](https://github.com/cozystack/cozystack/issues/3179). The chart already renders three infrastructure satellites with hardcoded resources and no values surface — `templates/kccm/manager.yaml`, `templates/cluster-autoscaler/deployment.yaml`, `templates/csi/deploy.yaml` — and draws a clean line between the control plane the user configures (`controlPlane.*.resourcesPreset`) and the machinery underneath it that the user does not. Per-cluster etcd belongs on the second side of that line: it is a control-plane implementation detail, not a control-plane user surface. Field data from the first review round supports this — measured on a real tenant cluster, the satellites already cost ~1360m requested CPU and ~3.85Gi requested memory, of which `kcsi-controller` alone (760m CPU, 3200Mi memory limits across six-plus sidecars) is heavier than three etcd replicas combined, and none of it is tunable. Adding a full sizing block for etcd alone would break the convention and add UX surface for a knob nobody has asked to turn. -### 3. DataStore naming (cluster-scoped uniqueness) +The internal values follow today's etcd chart defaults exactly (`packages/extra/etcd/values.yaml`), so per-cluster sizing starts where shared sizing is today: -`DataStore` is cluster-scoped, so a per-cluster name **must** be globally unique across the management cluster. Today's name (the bare namespace) is no longer sufficient because two clusters can live in one namespace. The name becomes `-` (with a short hash suffix only if length limits bite): +- `size: 4Gi`, with `quotaBackendBytes` derived at 95% as the chart already does. Ample for one Kamaji datastore. +- `resources: {cpu: 1000m, memory: 512Mi}`, with the existing VPA (`packages/extra/etcd/templates/vpa.yaml`) handling growth. +- `version` tracks the etcd chart, so a single bump moves both callers. +- `storageClass` is **derived from `replicas`**, resolved at platform level rather than per app: `replicas: 3` selects a local-style class, since etcd replicates through Raft and storage-layer replication underneath it is wasted write amplification; `replicas: 1` selects a replicated/DRBD-style class, so the single member survives node loss. The two class names come from `_cluster` values (e.g. `etcd-storageclass-local` and `etcd-storageclass-replicated`) rendered into `cozystack-values` the way the existing `_cluster` keys are, with an unset key falling back to the namespace's default StorageClass — the current behaviour, since the etcd chart's `storageClass` defaults to `""` today. An operator retunes their fleet in one place; a tenant does not make storage-class decisions at all. -```yaml -# packages/extra/etcd/templates/datastore.yaml (after, when used per-cluster) -metadata: - name: {{ .Release.Namespace }}-{{ .Release.Name }} -spec: - driver: etcd - endpoints: - - {{ .Release.Name }}-etcd.{{ .Release.Namespace }}.svc:2379 -``` +This surface is a strict subset of the fuller `{replicas, size, storageClass, resources}` block the first draft proposed, so if a production case ever needs `size` or `resources`, widening is purely additive and breaks nothing. + +`replicas` is validated to `1` or `3`. Even numbers are actively harmful for a Raft quorum (`2` tolerates zero failures while doubling cost and halving availability), and `5+` has no plausible justification for a single tenant control plane's key space. + +### 3. Names, and the length budget + +Two clusters can live in one namespace, so the per-cluster names must be release-scoped rather than namespace-scoped. The base name is `-etcd`, and derived names are: + +| Object | Name | Owner | +|---|---|---| +| `EtcdCluster` | `-etcd` | this chart | +| Native Service (client + peer endpoint) | `-etcd` | etcd-operator | +| Member pods | `-etcd-` | etcd-operator | +| TLS Secrets | `-etcd-{server,operator-client,peer}-tls` | etcd-operator / cert-manager | +| `DataStore` (cluster-scoped) | `-` | this chart | +| Datastore endpoint | `-etcd..svc:2379` | — | -The matching `kubernetes.datastoreName` helper computes the identical value so the control plane and its datastore always agree. +The endpoint is the operator's native Service, which is always named after the cluster — the operator's own comment is explicit that "the operator's native headless Service is always cluster.Name" (`controllers/helpers.go:137`). The `DataStore` name must be globally unique because the resource is cluster-scoped, and `-` gives that: namespaces are unique, and release names are unique within one. + +The length budget is real but bounded, and worth writing down because a review round raised it. The aggregated API caps a Helm release name at 53 characters (`maxHelmReleaseName`, `pkg/registry/apps/application/rest.go:1297-1303`), enforced as prefix-plus-name at admission. So `-etcd` is at most 58 characters, which clears the 63-character DNS label limit for the Service. The one place it does not clear on its own is the member pods: the operator creates members with `GenerateName: cluster.Name + "-"` (`controllers/etcdcluster_controller.go:419`), and 58 plus a separator plus a generated suffix can exceed 63, which would produce an invalid pod hostname at the far end of the range. The chart therefore bounds the name itself rather than assuming: `kubernetes.etcdName` truncates to a budget that leaves room for the operator's longest derived name and appends a short deterministic hash of the untruncated value when it truncates. Same input, same name, every render. The `DataStore` name is a DNS subdomain (253) and is never the binding constraint. ### 4. Removing the shared-etcd gate -The `awaiting-etcd` beacon and the `_namespace.etcd` dependency disappear from the normal path. Because the `kubernetes` app now provisions its own datastore, the `{{- if not $etcd }}` branch in `cluster.yaml:168` is no longer how a cluster waits — it renders its etcd and proceeds. During the compatibility window (Rollout) the chart still **honors an explicitly-set legacy `_namespace.etcd`** for clusters that have not yet migrated, so the switch is opt-in per cluster. +`$etcd` and the `awaiting-etcd` beacon leave the normal path. The chart no longer waits for an external reference, because it renders the datastore itself. + +The beacon and the `_namespace.etcd` read are not deleted immediately. Through the compatibility window (Rollout Phase 1-2) the chart honours an explicitly-set `_namespace.etcd` and keeps using the shared `DataStore` when it finds one, so a cluster that exists today continues exactly as it is, indefinitely, until an operator migrates it. New clusters — including any created in a namespace whose ancestors do provide a shared etcd — provision their own. Three other consumers key off the beacon and are updated with it: `packages/apps/kubernetes/templates/ingress.yaml`, `hack/admin-kubeconfig-invariant.bats`, and the ComputePlane README's description of the wait state. -### 5. Retiring the tenant module +### 5. Retiring the tenant-module wiring -Once clusters are migrated: +Once no cluster is on a shared etcd, the etcd-specific plumbing has no consumer left: -- Remove `etcd` from `packages/apps/tenant/values.yaml` and delete `packages/apps/tenant/templates/etcd.yaml`. -- Drop the etcd tenant-module definition (`packages/system/etcd-rd`) from the tenant app catalog and remove `cozystack.etcd-application` from the tenant application source list (`packages/core/platform/sources/tenant-application.yaml`). -- Remove the hardcoded `etcd: tenant-root` and the `_namespace.etcd` propagation in `cozystack-basics` and `apps/tenant/templates/namespace.yaml`, plus the `cozy-lib.ns-etcd` helper (`packages/library/cozy-lib/templates/_cozyconfig.tpl:89-95`) and the `namespace.cozystack.io/etcd` NetworkPolicy label, after confirming no other app consumes them. +- `etcd` leaves `packages/apps/tenant/values.yaml`, and `packages/apps/tenant/templates/etcd.yaml` is deleted. +- `_namespace.etcd` propagation goes: the inherit/override block at `namespace.yaml:28-30`, the `namespace.cozystack.io/etcd` label at `:86`, the `_namespace` key at `:120`, the hardcoded `etcd: tenant-root` at `cozystack-values-secret.yaml:17`, and the `cozy-lib.ns-etcd` helper (`_cozyconfig.tpl:116-118`) — after confirming no other app reads it. +- The etcd entry leaves the *tenant-module* catalog: `cozystack.etcd-application` drops out of the tenant application source list. The etcd `ApplicationDefinition` itself stays; see Resolved question 2. -The etcd-operator (`packages/core/platform/templates/bundles/system.yaml`) and the etcd chart stay. +The etcd-operator and the etcd chart both stay. Note that all of this is etcd-specific plumbing, not the generic module machinery — the generic side is [#39](https://github.com/cozystack/community/pull/39)'s subject, and these edits neither help nor hinder it. ## User-facing changes -- **Consumers:** creating a `Kubernetes` app now yields a working cluster with no admin pre-step. A new `etcd` block on the `Kubernetes` CR exposes replicas, size, storageClass, and resources. The `awaiting-etcd` status disappears. -- **Admins:** `spec.etcd` on a `Tenant` is deprecated then removed; etcd is no longer something to pre-provision per tenant. The etcd entry leaves the app catalog as a tenant module. -- **Dashboard/observability:** each cluster shows its own etcd `WorkloadMonitor` and metrics instead of one shared etcd per subtree. -- **Docs:** the kubernetes app README already states each cluster gets "a dedicated etcd cluster ... using etcd-operator" — the docs become *true by construction*. Update the tenant docs to drop the etcd module and add the per-cluster etcd sizing guidance and the migration runbook. +- **Consumers:** creating a `Kubernetes` app yields a working cluster with no admin pre-step. `awaiting-etcd` disappears. One new field, `etcd.replicas`, defaulting to `3`. +- **Admins:** `Tenant.spec.etcd` is deprecated and then removed; etcd stops being something to pre-provision per tenant. Existing clusters are untouched until deliberately migrated. Two new platform-level `_cluster` keys select the etcd StorageClasses fleet-wide. +- **Resource accounting:** a cluster's etcd now counts against its own tenant's `resourceQuotas` rather than an ancestor's. The chart's `NOTES.txt` and the app README state the footprint up front — three replicas at 1000m/512Mi each and a 4Gi PVC each at the default — so the cost is visible before creation, not discovered afterwards (Resolved question 5). +- **Dashboard and observability:** each cluster gets its own etcd `WorkloadMonitor`, `PodScrape`, and alert rules instead of one shared set per subtree, so etcd metrics finally attribute to a cluster. +- **Docs:** `packages/apps/kubernetes/README.md` already claims each cluster gets "a dedicated etcd cluster ... using the cozystack etcd-operator (`etcd-operator.cozystack.io/v1alpha2`)". Today that sentence is aspirational. This proposal makes it true, which is a small but honest sign that per-cluster etcd is the shape people already assume. Tenant docs drop the etcd module and gain the migration runbook. ## Upgrade and rollback compatibility -- **Phased and opt-in.** New clusters self-provision etcd immediately. Existing clusters keep using their shared `_namespace.etcd` DataStore until explicitly migrated, because the chart honors a legacy reference during the compatibility window (Design §4). No cluster breaks on upgrade. -- **Migration of an existing cluster** moves control-plane data from its prefix in the shared etcd to its new dedicated etcd, then flips `dataStoreName`. Kamaji supports datastore migration for a control plane; the exact mechanism for the CAPI `KamajiControlPlane` variant (live migration vs. snapshot/restore) is called out under Open questions and Testing and must be proven before the migration step is automated. Where it can be scripted idempotently, it runs through the existing platform numbered-migration hook framework (`packages/core/platform/images/migrations`, which already has etcd-touching migrations such as `16` and `22`); otherwise it is a documented, operator-gated runbook. -- **Rollback (pre-migration):** revert the `kubernetes` app to consume `_namespace.etcd`; nothing was moved, shared etcd still has the data. -- **Rollback (post-migration):** a cluster that has moved to its own etcd would have to migrate its data *back* to roll back — flag this as effectively one-way per cluster. The dedicated etcd retains a snapshot to make a reverse migration possible but non-trivial. +**Nothing migrates on upgrade.** New clusters self-provision. Existing clusters keep resolving `_namespace.etcd` and keep their shared `DataStore`, for as long as the operator leaves them alone. There is no automatic data movement anywhere in this proposal, and the platform never restarts a tenant's kubelets on its own initiative. + +**Migrating an existing cluster** is an operator-initiated, per-cluster, scheduled-maintenance operation. The mechanism is a live datastore switch, and unlike the first draft this is now established rather than assumed — see Resolved question 1 for the evidence. The runbook: + +1. Set `etcd.replicas` and let the chart create the cluster's own `EtcdCluster`, certificates, and `DataStore` alongside the shared one. Wait for `Available=True`. +2. Take a snapshot of the source etcd. This is the rollback path and is not optional. +3. Flip the cluster's `dataStoreName` to its own `DataStore`. Kamaji puts the `TenantControlPlane` into **read-only / freezing mode** while it copies, rejecting writes through an admission controller with an explicit "in freezing mode due to a maintenance mode" message. The default budget is five minutes, adjustable via the `kamaji.clastix.io/migration-timeout` annotation. +4. **Restart `kubelet.service` on every worker node of that tenant cluster.** Kamaji's own guide requires this to complete the procedure. It is the single largest cost in the runbook and the reason this is a maintenance operation rather than a platform migration: rolling a tenant's kubelets is a tenant-visible availability event that the platform must not perform unannounced. Plan it as a rolling restart within a maintenance window. +5. Verify a canary object written before the switch is readable after it, then retain the source prefix and the snapshot for a defined cool-off period before reclaiming. + +**Rollback.** Before migration, trivially: revert the chart and the cluster is still on the shared etcd, because nothing moved. After migration, one-way in practice — reversing means running the same freeze-and-kubelet-restart procedure in the other direction against the retained snapshot. Step 2's snapshot exists precisely so that this is possible-but-deliberate rather than impossible. + +**Numbered platform migrations.** None are needed to move data, and using one would be wrong for the reasons in step 4. If a numbered migration is added at all it is advisory — reporting which clusters still resolve a shared `_namespace.etcd`, so Phase 4 has a real signal instead of a guess. Any such migration claims **56 or higher**: `main` currently ships migrations through `55` with `targetVersion: 56`. The slot must be re-checked against `main` and against every active release branch at implementation time, not taken from this document — the first draft's "migration 49" was already stale when it was written, and bumping `targetVersion` without diffing the slot across branches is a known way to silently skip a migration. ## Security -- **Removes a shared trust boundary.** Each cluster gets its own etcd CA, Service, certificates, and PVCs. A control plane can no longer reach another cluster's datastore: today every control plane in a subtree dials the same `etcd..svc` (often cross-namespace, since the DataStore is cluster-scoped and the owner namespace is an ancestor); after this change the endpoint is the cluster's own in-namespace etcd. The existing `policy.cozystack.io/allow-to-etcd` control-plane pod label (`cluster.yaml:334`) is retargeted to the cluster's own etcd so NetworkPolicy stays tight. -- **No new tenant-supplied trust surface.** etcd config is bounded by the existing `Kubernetes` CR schema (replicas/size/resources); a consumer cannot point the control plane at an arbitrary external datastore. -- **Blast radius shrinks from subtree to cluster.** A compromised or resource-exhausted etcd now affects exactly one Kubernetes cluster. -- **Secrets:** per-cluster CA/peer/server/client Secrets are issued by cert-manager exactly as the tenant module does today, just one set per cluster. +- **Removes a shared trust boundary.** Each cluster gets its own etcd CA, Service, certificates, and PVCs. Today every control plane in a subtree dials the same `etcd..svc` — frequently cross-namespace, because the `DataStore` is cluster-scoped and its owner is an ancestor tenant. Afterwards, a control plane dials only its own in-namespace etcd, and the `policy.cozystack.io/allow-to-etcd` pod label (`cluster.yaml:464`) retargets to it so the NetworkPolicy stays tight rather than subtree-wide. +- **Blast radius shrinks from subtree to cluster.** A compromised or resource-exhausted etcd affects exactly one Kubernetes cluster. A compaction storm no longer has neighbours. +- **No new tenant-supplied trust surface.** The only new input is an integer restricted to `1` or `3`. There is no way to point a control plane at an arbitrary external datastore, and narrowing the surface to one integer (Design §2) is what guarantees that rather than merely discouraging it. +- **Secrets:** per-cluster CA / server / peer / client Secrets are issued by cert-manager exactly as the tenant module does today, one set per cluster instead of one per subtree. Kamaji's per-control-plane `-datastore-config` Secret is unchanged. +- **Residual risk during migration:** between steps 1 and 5 a cluster's data exists in two places. The retained source prefix and snapshot are a deliberate availability trade and should be reclaimed on a defined schedule rather than kept indefinitely. ## Failure and edge cases -- **Two `Kubernetes` apps in one namespace** → each renders its own `-etcd` and `-` DataStore; no name collision (the previous single shared `DataStore: ` could not represent two clusters distinctly). -- **DataStore name exceeds the cluster-scoped name length limit** → fall back to `--`; the helper enforces the limit deterministically. -- **Cluster deletion** → the delete hook (`packages/apps/kubernetes/templates/delete.yaml`) must additionally delete the `EtcdCluster`, the cluster-scoped `DataStore`, and the per-cluster cert Secrets, *after* the existing `-datastore-config` finalizer cleanup (delete.yaml already handles Kamaji's datastore-secret finalizer, issue #3062). Ordering: drain the control plane → strip the datastore-config finalizer → remove EtcdCluster + DataStore. -- **etcd not Ready when Kamaji reconciles** → the control plane stays not-Ready and Flux retries, the same self-healing loop as today's asynchronous datastore readiness; no hard failure. -- **Migration interrupted mid-flight** → the new etcd retains its snapshot and the old shared etcd still holds the source prefix; the migration is re-runnable and the cluster keeps serving from whichever `dataStoreName` is currently set. -- **Legacy reference still set after migration** → the cluster's own DataStore wins; the stale `_namespace.etcd` is ignored and removed during retirement. +- **Two `Kubernetes` apps in one namespace** → each renders `-etcd` and `DataStore/-`; no collision. The old single `DataStore/` could not name two clusters distinctly at all. +- **A long release name** → `kubernetes.etcdName` truncates deterministically with a hash suffix (§3), so the operator's derived pod names stay inside 63 characters. Deterministic means a re-render never renames a live object. +- **Cluster deletion, per-cluster etcd** → the delete hook additionally reaps the `EtcdCluster`, the cluster-scoped `DataStore`, and the per-cluster certificate Secrets, after the existing Step 4b that strips Kamaji's `finalizer.kamaji.clastix.io/datastore-secret` from `-datastore-config` (`templates/delete.yaml`). Ordering: drain the control plane → let the `TenantControlPlane` delete → clear the datastore-secret finalizer → remove `EtcdCluster` and `DataStore`. Removing etcd before Kamaji has finished finalising would leave the controller dialling a datastore that no longer answers. +- **Cluster deletion, legacy shared etcd** → **the hook must not touch the shared `EtcdCluster` or `DataStore`.** Deleting a legacy cluster must leave them exactly as they were, because they belong to an ancestor tenant's module and back every other cluster in the subtree. The hook branches on whether *this release* owns a per-cluster `DataStore`, which is the same condition §4 uses to choose a datastore in the first place, so the two cannot disagree: own the datastore, reap it; inherit it, leave it alone. This case gets a dedicated helm-unittest assertion, because getting it wrong destroys other tenants' clusters and the blast radius justifies a test that exists purely to pin the negative. +- **etcd not `Available` when Kamaji reconciles** → the control plane stays not-ready and Flux retries. Same asynchronous-readiness loop as today, no hard failure. Note the condition is `Available` on v1alpha2, not `Ready`. +- **`replicas: 2` or another even value** → rejected by schema validation with an explanatory message, rather than silently building a quorum that tolerates no failures. +- **Migration interrupted mid-flight** → the source prefix and the step-2 snapshot both still exist; the cluster keeps serving from whichever `dataStoreName` is currently set. Re-runnable. +- **Kubelet restart skipped after migration** → worker nodes keep talking to a control plane whose datastore moved underneath it. This is the failure mode step 4 exists to prevent; the runbook treats the restart as part of the migration, not as follow-up housekeeping. +- **A stale `_namespace.etcd` left set after migration** → the cluster's own `DataStore` wins. The stale reference is inert and is removed in Phase 4. +- **Tenant quota too small for the new etcd** → the `EtcdCluster`'s pods fail admission against the namespace `ResourceQuota` and the cluster does not come up, with the quota error surfaced on status. Publishing the footprint in `NOTES.txt` is what turns this from a surprise into a precondition. ## Testing -- **Helm unit tests** (`packages/apps/kubernetes/tests`): assert the cluster renders `EtcdCluster -etcd`, the per-cluster `DataStore` with the unique name and correct endpoint, and `dataStoreName` referencing it. Repurpose the existing `values-ci-no-etcd.yaml` scenario: "no inherited etcd" must now produce a fully-rendered cluster with its own etcd rather than the `awaiting-etcd` beacon. Add a two-clusters-in-one-namespace test asserting two distinct etcd + DataStore names. -- **etcd chart tests** (`packages/extra/etcd/tests`): assert the chart renders correctly under the per-cluster naming (release-scoped EtcdCluster/Service and `-` DataStore) as well as the legacy namespace-scoped name during the compatibility window. -- **e2e (`hack/e2e-apps/`):** create a `Kubernetes` app in a tenant with no ancestor etcd and assert it reaches Ready with its own etcd; create two clusters in one namespace and assert independent etcd Pods/PVCs; delete a cluster and assert its etcd, DataStore, and Secrets are gone and the namespace terminates cleanly. -- **Migration e2e (gating, must pass before automating the migration step):** stand up a shared-etcd cluster on the old path, run the migration, and assert the control plane stays continuously available (or within a documented brief window) and that data (a sentinel object) survives the datastore switch. +- **`cozy-lib` extraction (Phase 1 gate):** the existing `packages/extra/etcd/tests/` suite must pass unchanged against the extracted templates, proving the standalone chart's rendered output is byte-identical. This is the guard that makes the largest mechanical change in the proposal safe. +- **Helm unit tests, `packages/apps/kubernetes/tests/`:** assert the chart renders `EtcdCluster/-etcd`, the certificate set, `DataStore/-` with the `-etcd..svc:2379` endpoint, and a `dataStoreName` that matches that `DataStore` exactly. Repurpose `tests/values-ci-no-etcd.yaml`: "no inherited etcd" must now render a complete cluster with its own etcd instead of the `awaiting-etcd` beacon. Add two clusters in one namespace and assert two distinct etcd and `DataStore` names. Assert a maximum-length release name produces names inside 63 characters, and that re-rendering it yields the same names. Assert `replicas: 2` is rejected. +- **Delete-hook tests, `tests/delete_hook_test.yaml`:** one case asserting a per-cluster etcd and `DataStore` are reaped in the documented order; one asserting a legacy cluster's shared `EtcdCluster` and `DataStore` are **not** touched. +- **e2e, `hack/e2e-apps/`:** create a `Kubernetes` app in a tenant with no ancestor etcd and assert it reaches ready with its own etcd; create two clusters in one namespace and assert independent etcd pods and PVCs; delete one and assert its etcd, `DataStore`, and Secrets are gone, the other is unaffected, and the namespace terminates cleanly. +- **Migration e2e, gating Phase 2:** stand up a cluster on the shared path, run the full runbook including the kubelet restart, and assert a canary object written before the switch is readable after it, that the freeze window stays inside the configured timeout, and that worker nodes rejoin. This is the test that decides whether the runbook ships as documented; the underlying mechanism is established (Resolved question 1) but its behaviour on a Cozystack cluster with Talos workers is not yet measured, and an operator running the runbook offered to help exercise it on a production-shaped fleet. +- **Quota interaction:** a namespace whose `ResourceQuota` cannot fit the etcd surfaces a clear quota error rather than a mute pending cluster. ## Rollout -1. **Phase 1 — self-provisioned etcd, opt-in.** Add the `etcd` values block and per-cluster etcd/DataStore rendering to the `kubernetes` app behind a default that self-provisions for new clusters while still honoring a legacy `_namespace.etcd` reference for existing ones. Extend `delete.yaml` to reap the per-cluster etcd/DataStore. Ship docs for the new sizing knobs. -2. **Phase 2 — migration path.** Land and prove the shared→dedicated data migration (Kamaji datastore migration or snapshot/restore), as a numbered-migration hook where idempotent or a gated runbook otherwise. Provide a dashboard/CLI signal of which clusters are still on the shared etcd. -3. **Phase 3 — deprecate the tenant module.** Mark `Tenant.spec.etcd` and the etcd tenant-module catalog entry deprecated; new tenants no longer offer it. Stop hardcoding `etcd: tenant-root`. -4. **Phase 4 — remove plumbing.** Once telemetry shows no cluster on the shared etcd, delete `apps/tenant/templates/etcd.yaml`, the `etcd` tenant value, the `_namespace.etcd` propagation and `cozy-lib.ns-etcd` helper, and the `namespace.cozystack.io/etcd` label. Keep the etcd-operator and the etcd chart. +1. **Phase 1 — extract, no behaviour change.** Move the `EtcdCluster`, certificate, and `DataStore` shapes into `cozy-lib` named templates; `packages/extra/etcd` calls them and renders byte-identical output, proven by its existing test suite. Nothing user-visible ships. +2. **Phase 2 — per-cluster etcd for new clusters.** `apps/kubernetes` renders its own etcd, certificates, and `DataStore`, adds `etcd.replicas` and the platform-level StorageClass keys, extends the delete hook with both branches, and keeps honouring an explicit `_namespace.etcd` for clusters that already have one. Ship the sizing and quota notes. Land and prove the migration runbook, including the kubelet-restart step, before documenting it as supported. +3. **Phase 3 — deprecate.** Mark `Tenant.spec.etcd` and the etcd tenant-module catalog entry deprecated; new tenants stop offering it. Stop shipping the hardcoded `_namespace.etcd: tenant-root`. Update the ComputePlane docs and anything else describing `awaiting-etcd`. Optionally add the advisory migration that reports clusters still on a shared etcd. +4. **Phase 4 — remove plumbing.** Once that signal shows no cluster on a shared etcd, delete `apps/tenant/templates/etcd.yaml`, the `etcd` tenant value, the `_namespace.etcd` propagation, the `cozy-lib.ns-etcd` helper, and the `namespace.cozystack.io/etcd` label. Keep the etcd-operator, the etcd chart, and the standalone etcd app. + +Phases 1 and 2 are the proposal. Phases 3 and 4 are bookkeeping that can trail by a release or more, and Phase 4 in particular is gated on evidence rather than on a date. + +## Resolved questions + +The first draft left five questions open. All five are answered here; the reasoning is recorded because the answers are the substance of this revision. + +**1. Does the CAPI `KamajiControlPlane` support a live datastore switch?** **Yes.** The Kamaji control-plane provider projects `KamajiControlPlane.spec.dataStoreName` onto `TenantControlPlane.spec.dataStore` from inside a `controllerutil.CreateOrUpdate` mutate function (`controllers/kamajicontrolplane_controller_tcp.go`, `if kcp.Spec.DataStoreName != "" { tcp.Spec.DataStore = kcp.Spec.DataStoreName }`), so it re-projects on **every** reconcile rather than only at creation, and the field carries no immutability marker. Changing it therefore reaches Kamaji's own datastore-migration flow. That flow is not free: Kamaji's guide states the control plane "is put in read-only mode to avoid misalignments between source and destination datastores", with a five-minute default budget under `kamaji.clastix.io/migration-timeout`, and requires "restarting the `kubelet.service` on all the tenant worker nodes" afterwards. **Decision:** migration is an operator-initiated, per-cluster, scheduled-maintenance runbook — never an unattended numbered platform migration, because the platform must not roll a tenant's kubelets unannounced. A numbered migration, if used at all, only reports which clusters remain. This also settles the related worry that #2859's choice of snapshot-and-adopt was evidence against the live path: #2859 was adopting *operator API versions* for existing etcd clusters, an unrelated problem, so it says nothing about datastore migration either way. + +**2. Is anyone using etcd as a standalone app, and does it survive?** **It survives, and the question does not need answering first.** Only the *tenant-module wiring* retires — the `Tenant.spec.etcd` bool and the `_namespace.etcd` propagation. The chart and its `ApplicationDefinition` remain, so etcd stays installable as an ordinary app for any datastore purpose unrelated to a Kubernetes control plane, and no existing standalone deployment is disturbed. Deciding *which directory* it lives in and *how* it declares itself is [#39](https://github.com/cozystack/community/pull/39)'s question, not this one — which is exactly the division of labour described in Scope. + +**3. Default replicas: 1 or 3?** **3.** Defaulting to `1` would ship non-HA control planes to anyone who does not read the field, which is the wrong direction for a safe default. A user who wants a throwaway cluster can set `replicas: 1` explicitly and gets a replicated StorageClass underneath it as a partial safety net (§2). -## Open questions +**4. Subchart or shared library?** **Shared library — named templates in `cozy-lib`, parameterised by name.** The subchart route is blocked by three concrete facts, not by taste: `packages/extra/etcd/templates/check-release-name.yaml` calls `fail` unless `.Release.Name == .Chart.Name`; a subchart renders with the parent's release name, so it fails that guard and would then emit parent-named objects; and the literal `etcd-*` Secret names cannot be made release-scoped for existing clusters, because `etcd-migrate` adopted them into an immutable `spec.tls` that references those exact names. The library route sidesteps all three, needs no `Chart.yaml` dependency or vendoring step because `cozy-lib` is already symlinked into both charts, and keeps one source of truth for the etcd shape. Cost: `cozy-lib` gains resource-emitting templates alongside its helpers, which is a mild widening of what that chart is for. Accepted, and worth it against the alternative of duplicating the `EtcdCluster` and cert-manager shapes in two places. -1. **Kamaji datastore migration mechanics for `KamajiControlPlane`.** Does the CAPI control-plane provider support a live datastore switch (Kamaji migrating data when `dataStoreName` changes), or must Phase 2 use etcd snapshot/restore of the per-control-plane prefix? This determines whether migration is zero-downtime and whether it can be a numbered-migration hook. -2. **Standalone etcd-as-an-app.** Is anyone using the etcd tenant module as a standalone datastore unrelated to Kubernetes? If so, keep `packages/extra/etcd` available as an `extra` app (decoupled from the k8s datastore role) rather than removing its catalog presence entirely. -3. **Default replicas.** Keep `replicas: 3` as the safe production default, or default to `1` for the smallest footprint and document the HA upgrade? The shared module defaulted to 3; this proposal keeps 3 but the per-cluster multiplier makes the default worth confirming. -4. **Subchart vs. shared library.** Consume `packages/extra/etcd` as a declared subchart of `kubernetes`, or factor the EtcdCluster + cert templates into a shared library both charts import? Affects values plumbing and the build. -5. **Quota interaction.** Per-cluster etcd now counts against tenant `resourceQuotas`. Should the `kubernetes` app surface the etcd footprint in its sizing/NOTES so consumers see the full cost up front? +**5. Should the etcd footprint be surfaced?** **Yes** — in `NOTES.txt` and the app README, stated in concrete numbers rather than as a caveat (§ User-facing changes). Per-cluster etcd counts against the tenant's own quota, and with a minimal values surface this is a documentation change rather than an API one. ## Alternatives considered -- **Keep etcd shared but one etcd per tenant (status quo, better documented).** Rejected: still N clusters on 1 etcd within a tenant, so it solves neither the performance-isolation nor the self-service problem. It only renames the workaround. -- **Require the consumer to create a separate `etcd` app before the Kubernetes app.** Rejected: worse UX than today (two apps, explicit ordering, a dangling etcd if the cluster is deleted) and re-exposes a tenant-module concept we are trying to remove. Folding etcd into the cluster keeps it one app and one lifecycle. -- **Per-cluster SQL/kine datastore instead of etcd.** A real way to cut the three-replica etcd overhead, but it is an orthogonal datastore-driver change layered on Kamaji and is deferred (Scope). The per-cluster *ownership* model here is a prerequisite for it either way. -- **Automatic, unconditional migration of all shared-etcd clusters on upgrade.** Rejected: moving live control-plane data is too risky to do without operator opt-in. Migration is gated and per-cluster, with the shared etcd retained until each cluster has moved. -- **Delete the etcd chart and inline everything into the kubernetes app.** Rejected: duplicates the EtcdCluster + cert-manager shapes and the etcd backup/strategy integration. Reusing `packages/extra/etcd` keeps one source of truth. +- **Keep etcd shared, one per tenant, better documented.** Rejected: N clusters on one etcd within a tenant solves neither the isolation nor the self-service problem. It renames the workaround. +- **Require the consumer to create an `etcd` app before the `Kubernetes` app.** Rejected: worse than today — two apps, an explicit ordering the user must know, a dangling etcd when the cluster is deleted, and it re-exposes the very concept being retired. Also, it would leave the control-plane chart still not containing the control plane's state, which is the actual defect. +- **Expose the full `{replicas, size, storageClass, resources}` block** (the first draft's design). Rejected on review: it breaks the chart's existing convention that infrastructure satellites are hardcoded and only the control plane proper is tunable, and it adds four knobs for a component the user should not be sizing. The minimal surface is a strict subset, so growing into the fuller block later needs no breaking change ([#3179](https://github.com/cozystack/cozystack/issues/3179)). +- **Per-cluster SQL/kine datastore instead of etcd.** A genuine way to cut the three-replica cost, but an orthogonal driver decision that layers on top of per-cluster ownership rather than replacing it. Deferred. +- **Automatic migration of every shared-etcd cluster on upgrade.** Rejected twice over: moving live control-plane data without opt-in is unsafe, and the kubelet restart on every tenant worker node makes it an availability event the platform has no business triggering on its own. +- **Delete the etcd chart and inline everything into `apps/kubernetes`.** Rejected: duplicates the `EtcdCluster` and cert-manager shapes, orphans the standalone use case, and breaks the backup/snapshot integration built around the chart. +- **Wait for [#39](https://github.com/cozystack/community/pull/39) and treat etcd as one more capability migration.** Rejected, and this is the framing correction this revision makes. #39 generalises how modules are declared and where they live; the modules it covers are the ones whose *sharing must be preserved*. etcd is the one module whose sharing is the defect. Folding it into #39 would mean either holding a control-plane fix behind a repository-wide reorganisation, or expressing "this component stops being shared and moves inside another app" as a capability flag, which it is not. The two proposals are companions: one moves packages, the other completes an app. ---