Workload image update gets stuck if the current instance isn't in a healthy state

[mdobush]

Confirmed symptom: if the workload's current instance isn't in a healthy/Ready state when you deploy a new image, datumctl compute deploy <name> --image=<new> updates the Workload spec successfully (workloads describe shows the new image), but the underlying Instance never gets reconciled to match it — it stays frozen in whatever unhealthy state it was already in, indefinitely. (Updating a healthy workload's image works fine and rolls out normally — this only reproduces when the instance is already unhealthy at the moment of the update.)

Repro steps

1. Dockerfile

FROM busybox:stable-musl
COPY --chmod=755 run.sh /run.sh
ENTRYPOINT ["/bin/sh", "/run.sh"]

2. run.sh (two variants, same file, different sleep duration)

#!/bin/sh
echo "starting: $(date -u +%Y-%m-%dT%H:%M:%SZ), will sleep 30s then exit 55"
sleep 30
echo "exiting with code 55 at $(date -u +%Y-%m-%dT%H:%M:%SZ)"
exit 55

(the other variant is identical with sleep 5)

3. Commands

# Build + push both variants
datumctl compute build --push --output ghcr.io/mdobush/compute-echo-server:exit55-30s .   # sleep 30
datumctl compute build --push --output ghcr.io/mdobush/compute-echo-server:exit55 .        # sleep 5

# Deploy the 30s variant, let it run to completion
datumctl compute deploy exit55-test --image=ghcr.io/mdobush/compute-echo-server:exit55-30s --city=DFW --min=1 -y

# Wait until the instance transitions out of Ready (observed ~5.6 minutes after
# the process actually exited — not immediate)

# Now try to update the image while the workload is unhealthy:
datumctl compute deploy exit55-test --image=ghcr.io/mdobush/compute-echo-server:exit55 --city=DFW --min=1 -y
# --> hangs past the CLI's 120s watch timeout, never completes rollout
Observed state (10+ minutes stuck)
Health       Unavailable — InstancesProvisioning
    us-central-1 ready: 0/1        Blocked — Instances are being provisioned
Container    Image: ghcr.io/mdobush/compute-echo-server:exit55-30s   <- Workload spec DID update

Instance's own conditions, unchanged since before the update attempt:

{
  "type": "Available", "status": "False",
  "reason": "Stopping", "message": "Instance has stopped",
  "lastTransitionTime": "2026-09-24T11:36:21Z"   <- same timestamp as pre-update
}

Result: updated command stuck. new image can't reach the destination

Expected Result: New image applied