fix(deploy): Replace=true alone did not work — Force=true is what recreates - #125
Merged
Conversation
…reates Follow-up to #122, which did NOT fix #110. Verified against the live cluster after merge: fuzekeys-platform is still OutOfSync/Degraded, now at revision c620f4a (the fix commit), with a DIFFERENT and worse error: error when replacing: Job.batch "fuzekeys-migrate" is invalid: spec.selector: Required value, `selector` does not match template `labels`, spec.selector: Invalid value: null: field is immutable Root cause of the bad fix: Argo's `Replace=true` means `kubectl replace` — an IN-PLACE swap, not delete-and-recreate. Replacing a Job still hits spec.template immutability, and additionally fails on spec.selector, because Kubernetes auto-generates that selector with a controller-uid the chart never declares. So Replace=true turned one immutability error into two. `Force=true` is the half that matters: it makes the operation `kubectl replace --force`, i.e. DELETE then recreate — the only operation that can legally change an immutable field. Safe here for the reason the template already documents: `alembic upgrade head` is a no-op at head. The sync-wave ordering (config -2 -> migrate -1 -> workloads 0) is unchanged, so the Job still runs after its ConfigMap and Secret and before the workloads. Verified by render: both annotations present, sync-options now Replace=true,Force=true. NOT yet verified against the cluster — that is what the next Argo sync decides, and given #122 this one gets checked before it is called fixed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session-Id: fe3aa738-cb24-4315-a29b-29834ddea892
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #125 +/- ##
=======================================
Coverage 41.06% 41.06%
=======================================
Files 81 81
Lines 6171 6171
Branches 250 250
=======================================
Hits 2534 2534
Misses 3634 3634
Partials 3 3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #122, which did not fix #110. Reopening the substance of that issue.
Verified against the live cluster after merging #122
Argo picked up the fix (that revision is the fix commit) and still failed — with a different and worse error:
Issue #110 auto-closed on the
Closes #110keyword one second after the merge. That was the keyword firing, not the cluster recovering.Why #122 was wrong
Argo's
Replace=truemeanskubectl replace— an in-place swap. I assumed it meant delete-and-recreate. It doesn't.Replacing a Job still hits
spec.templateimmutability, and additionally fails onspec.selector: Kubernetes auto-generates that selector with acontroller-uidthe chart never declares, so the replacement manifest has no selector and the existing one can't be changed. One immutability error became two.The actual fix
Force=true— it makes the operationkubectl replace --force, i.e. DELETE then recreate, which is the only operation that can legally change an immutable field.Replace=trueis kept because Force acts on the replace path.Safe here for the reason the template already documents:
alembic upgrade headis a no-op when the DB is at head.Sync-wave ordering is untouched (
config -2→migrate -1→workloads 0), so the Job still runs after its ConfigMap and Secret and before the workloads — which is why this is still not a Helmpre-upgradehook (that maps to PreSync and would run before the wave-2ConfigMap).Verification
Rendered:
sync-options: Replace=true,Force=truepresent alongsidesync-wave: "-1". Chart0.1.12→0.1.13.Not yet verified against the cluster. That's what the next Argo sync decides — and given #122, this one gets checked with
cluster-querybefore anyone calls it fixed.🤖 Generated with Claude Code