Instance failure reasons never reach the workload

[scotwells]

Customer impact

When a customer's instances are failing, their Workload should say why. Today it cannot. Whatever reason an Instance reports — InstanceCrashing, ConfigurationError, or an image that could not be pulled — the WorkloadDeployment and Workload above it never see it. When nothing is ready, the customer is told "Instances are being provisioned", indefinitely, for a workload that will never start.

This is wider than crashes:

• Image pull failures. An Instance correctly reports that its image could not be pulled, yet the Workload would still say provisioning. Staging evidence of an image failure "surfacing" (Crash-looping instances report Available=True forever datum-labs/kata-provider#18) was at the Instance level only.
• Configuration errors. The provider refuses an already-stored Instance whose configuration it cannot run. That reason is lost in the same way.
• Crashes. No provider reports a crash after the container has started yet (Crash-looping instances report Available=True forever datum-labs/kata-provider#18). Once one does, the reason is dropped on its way up.

The bug

The WorkloadDeployment's status is not built from its Instances' reasons. It considers the cell's location, network readiness, referenced data, the number of Instances blocked on quota, and — whenever there are replicas — the generic "Instances are being provisioned". It counts how many Instances are ready but never looks at why the rest are not.

So an Instance's failure reason never becomes a candidate for the deployment's status. Ranking InstanceCrashing or ConfigurationError higher would change nothing, because they never enter the ranking. What is missing is a path that carries an Instance's failure reason up to the WorkloadDeployment and the Workload.

What the customer sees

A workload whose only container exits on startup, in staging:

Available=True   AvailablePlacementFound   At least one available placement was found

After the provider is fixed but without a rollup path, this would become "Instances are being provisioned" — still not the real cause.

Relationship to #310

#310 requires that a workload missing a permission its image needs reports an error "instead of restarting on a loop". Any version of that needs the failure to reach the Workload, so this issue is a prerequisite.

Secondary observation

A WorkloadDeployment was observed reporting ReplicasReady=True alongside the message "0/1 replicas available", on a workload whose image could not be pulled. Likely related to how replica readiness is computed; not diagnosed.

Related

• Crash-looping instances report Available=True forever datum-labs/kata-provider#18
• Customers cannot see or set the security configuration their instances run with #310

[scotwells]

Proposal: carry Instance failure reasons up to the Workload

This is the compute side of one customer problem. datum-labs/kata-provider#18 makes the Instance report a crash truthfully; this issue makes that reason, and every other Instance failure reason, reach the Workload. Neither is enough alone: a fixed provider still shows "Instances are being provisioned" here, and a rollup path has no crash to carry until the provider reports one.

What the customer sees

Today, a workload whose only container exits on startup:

Available=True   AvailablePlacementFound   At least one available placement was found

After both fixes, when every Instance is failing:

Available=False  InstanceCrashing
1 of 1 instances is repeatedly exiting and restarting: container "app" has
exited and restarted 3 times without staying up for 5 minutes. Its last exit
was with code 1.

The same path fixes image failures, which today would also read as provisioning:

Available=False  ImageUnavailable
1 of 1 instances could not pull its image.

These messages are in a condition, which kubectl get workload does not show as a column. The customer sees Available=False there and must read the full object for the reason.

1. Carry the reason upward

The WorkloadDeployment should count its Instances by failure reason — crash-looping, misconfigured, image unavailable — and, when no replica is ready, report the most actionable of those reasons, with the count and one representative Instance message. The Workload then carries that reason from its placements the same way.

The rule for choosing: a reason the customer must fix outranks a reason that means the platform is still working. Instance failure reasons therefore outrank "Instances are being provisioned". Spec-level rejections that stop Instances from being placed at all still come first, since then nothing is running to fail.

2. Partial failure

A serving workload will not be marked down. A WorkloadDeployment stays available while any replica is ready, and a Workload stays available while any placement is. That stays as it is.

The flip side: with 1 of 3 Instances crash-looping, or one placement fully down, the customer sees an available Workload and no hint of trouble. The examples above cover only the case where everything is failing. Partial failure should surface as a degraded signal without changing availability. The cheapest first step is to add it to the available message (for example, "2/3 instances are ready; 1 is repeatedly exiting and restarting"). A dedicated condition is a separate API decision.

3. The capability hint: later, and heavily caveated

We considered adding a hint from the customer's own spec when a startup crash runs with fewer capabilities than the runtime class grants by default. It is worth keeping only with these limits, and it is not part of the first fix:

• Startup crashes only. A container that ran for hours and then crashed is not short of capabilities.
• It would fire on hardened manifests. Under the rule proposed in Adding one capability silently strips the runtime class defaults, breaking stock images #312, drop: [ALL] plus one add is the recommended hardened shape, so the hint would fire on the most security-conscious workloads and nudge them toward more privilege. The wording must say plainly that this is not evidence of the cause.
• The comparison is unstable. Workloads created before Adding one capability silently strips the runtime class defaults, breaking stock images #312 can both lack defaults and carry extra capabilities. Class defaults can also change later: if a default is withdrawn, the hint could suggest a capability the platform now refuses; if one is added, every older workload suddenly looks narrower.
• It cannot name the crashed container. Container identity does not reach the Workload, so the hint can only list which containers in the template run without the defaults.

If built, the wording would be close to:

Containers "app" and "sidecar" run without capabilities that runtime class "general-purpose" currently grants by default: CHOWN, SETGID, SETUID. If the image needs one, add it to the container's capabilities. This is not evidence that capabilities caused the crash.

4. Fix the contradiction

The observed ReplicasReady=True alongside "0/1 replicas available" should be fixed with this work.

What we will not do, and what may help

We will not read or scrape tenant logs and match them into conditions. That turns customer data into platform status, the strings vary by image, and a false match would send customers after the wrong fix.

Some useful signals need no log reading. They come from the container status the provider already receives (see datum-labs/kata-provider#18):

• Out of memory. Reported specifically, and actionable: raise the memory limit.
• Killed by a signal vs exited with an error. These can be told apart and described differently.
• A termination message written by the application. Kubernetes lets a container write a short reason to its termination-message file, which arrives with normal container status. An image can state its own failure reason, and it can travel up as the representative message.

We have not yet verified that the VM-isolated runtime passes these through reliably. That check comes before promising them.

On #310

#310 describes a workload missing a permission its image needs, which would otherwise restart on a loop, reporting a configuration error instead. That scenario is not met, and this proposal does not meet it. A capability the class refuses is already rejected when the workload is submitted, but that is a different case: a refused request never loops.

Nothing observable identifies a missing capability as the cause of a crash. A dropped capability makes a call fail with "operation not permitted", and the process exits like any other misconfigured application. The realistic target for #310 is a truthful crashing state that reaches the Workload, plus, at most, the caveated hint above. #310's test plan should be restated to that target.