fix(deploy): stop the migrate Job failing every sync on an immutable field - #122
Merged
Conversation
…field Closes #110. `fuzekeys-platform` has been OutOfSync/Degraded since 2026-08-24 with: Job.batch "fuzekeys-migrate" is invalid: spec.template: ... field is immutable (retried 5 times) A Job's `spec.template` is immutable in Kubernetes, and `fuzekeys.labels` stamps `helm.sh/chart` and `app.kubernetes.io/version` onto the POD TEMPLATE. So every chart version bump mutates an immutable field and the apply is rejected. This is not an unlucky upgrade — it is guaranteed on EVERY release, which is why the app never recovered on its own. `Replace=true` makes Argo delete and recreate the Job rather than patch it. That is safe here for the reason the template already documents: `alembic upgrade head` is a no-op when the DB is at head. Deliberately NOT the Helm-hook pattern used by vault-bootstrap-job.yaml in this same chart. Argo maps `helm.sh/hook: pre-upgrade` to PreSync, which runs before the entire sync — including before the wave -2 ConfigMap and the Secret this Job reads via `envFrom`. The migration would then run against absent or stale config. The existing sync-wave ordering (config -2 -> migrate -1 -> workloads 0) is what makes that config available, so it stays. Verified by rendering: `helm template` emits both `sync-wave: "-1"` and `sync-options: Replace=true` on the Job. Not yet verified against the live cluster — that happens when Argo syncs this. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session-Id: fe3aa738-cb24-4315-a29b-29834ddea892
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #122 +/- ##
=======================================
Coverage 41.06% 41.06%
=======================================
Files 81 81
Lines 6171 6171
Branches 250 252 +2
=======================================
Hits 2534 2534
Misses 3634 3634
Partials 3 3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #110.
The failure
fuzekeys-platformhas beenOutOfSync/Degradedsince 2026-08-24:Why it can never recover on its own
A Job's
spec.templateis immutable in Kubernetes.fuzekeys.labelsstampshelm.sh/chartandapp.kubernetes.io/versiononto the pod template, so every chart version bump mutates an immutable field. The apply is rejected, Argo retries 5 times, the Application goes Degraded.This isn't an unlucky upgrade — it's guaranteed on every release. That's why it has been stuck for a week rather than self-healing.
The fix
argocd.argoproj.io/sync-options: Replace=true— Argo deletes and recreates the Job instead of patching it. Safe here for the reason the template already documents:alembic upgrade headis a no-op when the DB is already at head.Why not the Helm-hook pattern used by
vault-bootstrap-job.yamlThat sibling template in this same chart solves its own recreation problem with
helm.sh/hook: post-install,post-upgrade+hook-delete-policy: before-hook-creation. Copying it here would be wrong:Argo maps
helm.sh/hook: pre-upgradeto PreSync, which runs before the entire sync — including before the wave-2ConfigMap and the Secret this Job consumes viaenvFrom. The migration would then run against absent or stale config.The existing sync-wave ordering (
config -2→migrate -1→workloads 0) is precisely what makes that config available, so it stays.Replace=truefixes the immutability without touching the ordering.Verification
helm templaterenders both annotations on the Job:Not verified against the live cluster — that happens when Argo syncs this. The check that it worked is
fuzekeys-platformreturning to Synced/Healthy.Chart version bumped
0.1.11→0.1.12.🤖 Generated with Claude Code