Skip to content

fix(deploy): Replace=true alone did not work — Force=true is what recreates - #125

Merged
izzywdev merged 1 commit into
masterfrom
fix/migrate-job-force-recreate
Aug 31, 2026
Merged

fix(deploy): Replace=true alone did not work — Force=true is what recreates#125
izzywdev merged 1 commit into
masterfrom
fix/migrate-job-force-recreate

Conversation

@izzywdev

Copy link
Copy Markdown
Owner

Follow-up to #122, which did not fix #110. Reopening the substance of that issue.

Verified against the live cluster after merging #122

NAME                SYNC        HEALTH     REV
fuzekeys-platform   OutOfSync   Degraded   c620f4a287abb61023b163ad72ed83f5249e5c72

Argo picked up the fix (that revision is the fix commit) and still failed — with a different and worse error:

error when replacing: Job.batch "fuzekeys-migrate" is invalid:
  spec.selector: Required value,
  `selector` does not match template `labels`,
  spec.selector: Invalid value: null: field is immutable

Issue #110 auto-closed on the Closes #110 keyword one second after the merge. That was the keyword firing, not the cluster recovering.

Why #122 was wrong

Argo's Replace=true means kubectl replace — an in-place swap. I assumed it meant delete-and-recreate. It doesn't.

Replacing a Job still hits spec.template immutability, and additionally fails on spec.selector: Kubernetes auto-generates that selector with a controller-uid the chart never declares, so the replacement manifest has no selector and the existing one can't be changed. One immutability error became two.

The actual fix

Force=true — it makes the operation kubectl replace --force, i.e. DELETE then recreate, which is the only operation that can legally change an immutable field. Replace=true is kept because Force acts on the replace path.

Safe here for the reason the template already documents: alembic upgrade head is a no-op when the DB is at head.

Sync-wave ordering is untouched (config -2migrate -1workloads 0), so the Job still runs after its ConfigMap and Secret and before the workloads — which is why this is still not a Helm pre-upgrade hook (that maps to PreSync and would run before the wave -2 ConfigMap).

Verification

Rendered: sync-options: Replace=true,Force=true present alongside sync-wave: "-1". Chart 0.1.120.1.13.

Not yet verified against the cluster. That's what the next Argo sync decides — and given #122, this one gets checked with cluster-query before anyone calls it fixed.

🤖 Generated with Claude Code

…reates

Follow-up to #122, which did NOT fix #110. Verified against the live cluster
after merge: fuzekeys-platform is still OutOfSync/Degraded, now at revision
c620f4a (the fix commit), with a DIFFERENT and worse error:

    error when replacing: Job.batch "fuzekeys-migrate" is invalid:
      spec.selector: Required value,
      `selector` does not match template `labels`,
      spec.selector: Invalid value: null: field is immutable

Root cause of the bad fix: Argo's `Replace=true` means `kubectl replace` — an
IN-PLACE swap, not delete-and-recreate. Replacing a Job still hits spec.template
immutability, and additionally fails on spec.selector, because Kubernetes
auto-generates that selector with a controller-uid the chart never declares. So
Replace=true turned one immutability error into two.

`Force=true` is the half that matters: it makes the operation
`kubectl replace --force`, i.e. DELETE then recreate — the only operation that
can legally change an immutable field. Safe here for the reason the template
already documents: `alembic upgrade head` is a no-op at head.

The sync-wave ordering (config -2 -> migrate -1 -> workloads 0) is unchanged, so
the Job still runs after its ConfigMap and Secret and before the workloads.

Verified by render: both annotations present, sync-options now Replace=true,Force=true.
NOT yet verified against the cluster — that is what the next Argo sync decides,
and given #122 this one gets checked before it is called fixed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session-Id: fe3aa738-cb24-4315-a29b-29834ddea892
@github-actions
github-actions Bot enabled auto-merge (squash) August 31, 2026 16:19
@codecov

codecov Bot commented Aug 31, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 41.06%. Comparing base (c620f4a) to head (70b0153).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #125   +/-   ##
=======================================
  Coverage   41.06%   41.06%           
=======================================
  Files          81       81           
  Lines        6171     6171           
  Branches      250      250           
=======================================
  Hits         2534     2534           
  Misses       3634     3634           
  Partials        3        3           
Flag Coverage Δ
backend 41.06% <ø> (ø)
frontend 41.06% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@izzywdev
izzywdev merged commit 4001a7a into master Aug 31, 2026
33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant