Adding one capability silently strips the runtime class defaults, breaking stock images

[scotwells]

What a customer hits

A customer has a working sandbox workload on the general-purpose runtime class. They need one extra capability, so they add the obvious thing:

securityContext:
  capabilities:
    add: [NET_ADMIN]

They get NET_ADMIN — and silently lose the ten capabilities the class was granting them, including CHOWN, SETGID, SETUID, and NET_BIND_SERVICE. A stock nginx or Postgres image then fails at runtime: its entrypoint chowns data directories and its workers drop privileges. The failure surfaces far from the one-line YAML change that caused it, and the field they edited is named add, which in Kubernetes is additive.

It gets worse: per datum-labs/kata-provider#18 the resulting crash-loop still reports Available=True, so the customer gets no signal from the platform at all.

Found while validating #310.

Evidence

Verified against the staging datum-cloud project control plane with kubectl apply --dry-run=server (real webhooks, nothing created).

Container with no securityContext — admission stamps:

capabilities:
  add: [CHOWN, DAC_OVERRIDE, FOWNER, FSETID, KILL, NET_BIND_SERVICE, SETFCAP, SETGID, SETPCAP, SETUID]
  drop: [ALL]
allowPrivilegeEscalation: false
seccompProfile: {type: RuntimeDefault}

Same container stating capabilities: {add: [NET_ADMIN]} — admission stores:

capabilities:
  add: [NET_ADMIN]
  drop: [ALL]

All ten defaults are gone. A second probe confirms there is no merge anywhere: input add: [NET_ADMIN, CHOWN] stores exactly those two.

Reproduction

Against a control plane with the RuntimeClasses feature gate enabled and the general-purpose class published:

apiVersion: compute.datumapis.com/v1alpha
kind: Workload
metadata:
  name: cap-repro
spec:
  template:
    spec:
      runtime:
        class: general-purpose
        resources:
          instanceType: datumcloud/d1-standard-2
        sandbox:
          containers:
            - name: app
              image: docker.io/library/nginx:latest
              # probe 2: uncomment
              # securityContext:
              #   capabilities:
              #     add: [NET_ADMIN]
      networkInterfaces:
        - network:
            name: default
  placements:
    - name: us
      locations:
        - name: gcp-us-south1-a
      scaleSettings:
        minReplicas: 1

kubectl apply --dry-run=server -f workload.yaml -o json

Run once as written, once with the securityContext block uncommented, and compare the stored securityContext on the container.

The design decision this has to overturn

This behaviour is deliberate today. The reasoning we published was that a customer statement is the whole answer for the capabilities it covers: merging the class default into a stated set would put back the invisible injection the runtime class surface was meant to replace, so a container that states capabilities keeps exactly those.

That rationale no longer holds. The concern it protects against is the platform adding capabilities invisibly — but admission now stamps the resolved result onto the stored workload, so a merged set would be fully visible to the customer in their own spec, alongside the drop: [ALL] that is already appended to stated sets for exactly this legibility reason. Stamping already solves invisibility. Merging costs nothing in legibility, and replacement costs the customer a working deployment.

Expected

A stated capability should be additive to the class default, or the platform must make the replacement explicit to the customer.

[scotwells]

Recommendation: make add additive, with an explicit precedence order

A customer who needs one more capability should get one more capability. Today they get one capability total, and a stock image that worked yesterday crash-loops with no signal. That is the outcome to fix, and the field is already named add.

The direction is additive — but "additive" on its own is not a specification, and the obvious phrasing of it is wrong in a dangerous way. That has to be settled first.

Getting the rule right: drop: [ALL] plus add must mean exactly the adds

The tempting shorthand is "drop: [ALL] with no add is how you ask for zero." That makes ALL inert the moment add is non-empty, and it breaks the single most common hardened manifest in Kubernetes:

capabilities:
  drop: [ALL]
  add: [NET_BIND_SERVICE]

This is the canonical Pod Security Standards restricted idiom. Upstream it means NET_BIND_SERVICE and nothing else. Under the naive additive rule it would resolve to the ten class defaults plus NET_BIND_SERVICE — eleven capabilities, ten of which the customer explicitly asked to drop. It fails silently, and it fails in the permissive direction, on a manifest written by exactly the customer who was being careful. Any additive proposal that does not answer this is not safe to ship.

The precedence order

1. drop: [ALL] clears the class default.
2. Otherwise, start from the class default.
3. Add whatever the customer lists in add.
4. Remove any capability the customer names in drop. A named drop always wins.

The customer writes	They get
nothing	the class default
add: [NET_ADMIN]	the class default plus NET_ADMIN
drop: [CHOWN, SETUID]	the class default minus those two
drop: [ALL]	no capabilities
drop: [ALL], add: [NET_BIND_SERVICE]	exactly NET_BIND_SERVICE
add: [CHOWN], drop: [CHOWN]	not CHOWN — a named drop wins, as it does when the container runs

This is one rule, it reads the way the field names read, and it matches what an operator coming from Kubernetes already expects. "Exactly these capabilities and nothing else" remains fully expressible — drop: [ALL] plus the adds you want — so nothing is given up by making add additive.

Two properties make it safe:

• It gets the restricted idiom right. drop: [ALL] plus one add resolves to exactly that one capability, as the Kubernetes Pod Security Standards intend.
• It is stable on re-apply. A stored workload already lists its resolved capabilities in add alongside drop: [ALL], so re-submitting the stored object — a copy-paste, a GitOps re-apply — resolves to exactly the same set rather than growing or collapsing. That matters because the platform writes the resolved result back onto the workload.

A named drop winning is not a stylistic choice. When the platform runs a container, a capability named in both lists is already left out. If the stored workload resolved an overlap the other way, it would advertise a capability the running container does not have — breaking the promise that the stored workload states what actually runs.

This is the same rule #313 proposes for overlapping add and drop. The two must be settled together, so the stored form and the running container follow one rule.

What this looks like from the customer's side

They write:

securityContext:
  capabilities:
    add: [NET_ADMIN]

Today, their stored workload comes back as:

capabilities:
  add: [NET_ADMIN]
  drop: [ALL]

Under this proposal, it comes back as:

capabilities:
  add: [CHOWN, DAC_OVERRIDE, FOWNER, FSETID, KILL, NET_ADMIN, NET_BIND_SERVICE,
        SETFCAP, SETGID, SETPCAP, SETUID]
  drop: [ALL]

nginx keeps working, NET_ADMIN is there, and the full resolved set is visible in their own spec — which is the point of stamping.

This is a contract change to drop, not a bug fix

drop is published today as "accepted for compatibility with Kubernetes manifests" — deliberately inert boilerplate. Making it the primary way to narrow privilege changes what we promised, and we should say so plainly rather than dress it up as restoring documented behaviour. It is the right change, but it is a change.

The narrow complaint about drop does stand on its own: dropping a single capability today wipes all ten class defaults. That is wrong under any reading, and it gets fixed by the same precedence order.

The rationale this overturns

The original argument was that merging would "put back the invisible injection this replaces." That held before stamping existed. It does not hold now: the merged set is written onto the customer's stored workload, so nothing is invisible — the customer reads exactly what their container runs with. And we already accept this: drop: [ALL] is appended to a customer's stated set for precisely the legibility reason, with a note that leaving it off "would hide it from the customer most likely to be reading it." The same reasoning applied to add gives the opposite answer from the one shipped. The asymmetry is the bug.

A hazard this change creates, and what has to ship with it

Once add is additive, every stamped sandbox container carries the class's full default set — not just the capabilities the customer named. That makes an existing sharp edge fleet-wide.

If a runtime class ever stops granting a capability that is currently in its default set, every workload already stamped in that class is holding a capability the class no longer permits. Those workloads become permanently unupdatable — and because the platform's own cleanup path is itself an update, they become undeletable. Today that hazard only reaches containers that explicitly named the withdrawn capability. After this change it would reach every container in the class.

So this cannot ship alone. It has to ship with a rule that lets a workload keep an unchanged security configuration on update: if the customer is not touching their security settings, an existing workload stays valid even when the class has since narrowed. The platform already does exactly this in two other places — it grandfathers container images, and it grandfathers workloads created while the runtime class surface was switched off. This is the same guarantee, applied to capabilities.

Options considered

Additive add, subtractive drop, with explicit precedence. Matches the field names, matches Kubernetes intuition, keeps the result fully legible via stamping, and keeps the narrow case expressible. Recommended.

Reject at admission instead of merging. Rather than warning, refuse any stated add that omits capabilities the class grants by default — an error naming the missing defaults and pointing at drop as the way to say you really meant it. This deserves a straight answer, because it is genuinely strong: unlike a warning it is lost nowhere, it fails at apply time next to the edit that caused it, and it never resolves a manifest to something the customer did not write. It also has no grandfathering hazard and no drift.

I still prefer additive-with-precedence, for two reasons. First, the error is only correct while the class default is the right answer for that container; the moment a customer legitimately wants a narrower set they must write a drop whose only purpose is to silence us, which trains people to append boilerplate rather than think. Second — and this is the deciding one — the error rejects the canonical restricted manifest, or forces us to special-case it, and every special case in an admission rule is a future silent misresolution. A rule that resolves is easier to reason about than a rule that refuses, provided the resolution is visible, and stamping makes it visible.

Warn at admission, keep replacement. A warning listing the ten capabilities being dropped is real and cheap, but it only reaches a human watching a terminal — it is lost in CI, GitOps, and any controller-driven apply, which is how most workloads get created. Good as a complement, not as the fix.

Docs only. This turns a platform behaviour that contradicts both Kubernetes convention and our own published field documentation into a thing customers are expected to have read. The failure still lands at runtime, far from the edit, and per datum-labs/kata-provider#18 still reports Available=True. Not viable on its own.

On severity — being accurate about who this breaks

Worth being precise, because it is easy to overstate. A customer editing an existing stored workload already sees all ten defaults listed in their spec, and appending to that list works correctly today. Nothing regresses in place, and no running workload changes behaviour.

The break lands on someone writing add from scratch in a fresh manifest — a new workload, a new environment, a copied example. That is real and it is common, and the failure is a silent runtime crash-loop far from the edit. But it is not a running workload degrading on its own, and the case is strong enough without that claim.

The GitOps cost

Under merge, a customer's stored object carries capability entries their source file does not. For anyone reconciling from Git, that is a permanent drift signal on every sandbox container — a resource that reports as out of sync forever, by design. That is a genuine cost, and it cuts against the argument used above to dismiss warnings: GitOps pipelines swallow warnings, but they surface drift loudly and continuously.

I do not think it outweighs a silently broken deployment, but it is the strongest objection to merging and we should expect to hear about it. It is also the cost the admission-error option avoids entirely.

Migration and already-stamped workloads

Nothing is re-stamped and no migration runs. Resolution happens only at creation, so every workload already stored keeps the exact set written on it:

• Workloads stamped with the full default set (the common case) are unaffected — their stored set is already the merged answer.
• A workload stamped with a truncated set stays truncated. Editing it will not re-resolve it.

That leaves a genuinely uncomfortable property, and it should be stated rather than buried: the same source manifest means different things depending on when the workload was created. A file with add: [NET_ADMIN] yields one capability before the change and eleven after, and nothing on the stored object records which semantics produced it. Ordinary recreation is enough to flip it — a delete-and-apply, a namespace rebuild, a restore from backup — and the flip is silent and in the permissive direction, widening privilege by ten capabilities without anyone editing anything.

Because of that, the resolved object should record which semantics resolved it. A stamped workload should carry the capability-resolution behaviour it was created under, so that support, audit, and the customer can all answer "why does this object have eleven capabilities when the file says one" without guessing at creation timestamps. It also gives us a clean way to find the truncated workloads instead of inferring them.

Blast radius is otherwise small and known: the runtime class surface is on in staging and off in production, so production has zero exposure. We should enumerate the affected staging workloads and either recreate them or notify the owners directly rather than relying on the behaviour change to heal them.