feat(tls): an instance asks for its certificate as part of provisioning, and is not 'ready' without it - #4457
Conversation
…ithout it A provisioning run renders its values from the deployment RECORD. The cert-manager issuer lived only as a hand-written annotation in checked-in overlays, so a record without it rendered an ingress naming a TLS secret that nothing issued — and the controller served the fallback certificate, another instance's. pearl.meshweaver.cloud was in that state for nine hours on 2026-09-15 with a healthy portal behind it, and its Provision could not have fixed it either: step 15/18 died on certificates.cert-manager.io "pearl-tls" is forbidden: User "system:serviceaccount:memex-ops:hosting-operator" cannot get certificates so TLS and all three verification steps never ran. - contract: IngressSpec.ClusterIssuer, DEFAULTED to letsencrypt-prod, so an instance declared with a host and a TLS secret asks for its certificate with no further call — and a record written before the field gets one too, because an absent JSON member leaves the initializer standing. A hand-written annotation still wins; 'none' is the named opt-out. WithIngress takes it. - chart: the portal ingress renders the issuer from that value, and REFUSES to render a tlsSecret with no issuer reaching it at all. - operator: hosting-tls takes --issuer, waits for the Certificate to be Ready AND the Secret to carry a key pair, and names which of the two failed; hosting-verify reads the served certificate back and tells a wrong certificate apart from a dead application. - rbac: the operator may get/create/patch cert-manager Certificates. The coverage check now sees that grant, because hosting-tls reads the object explicitly — a Certificate created through 'kubectl apply -f -' is invisible to it, which is how this reached main ungranted. Tests: 2 contract tests (the default, and that it never overrides a decision), 7 operator tests incl. the fallback-certificate and wildcard shapes, 2 chart render fixtures and 1 chart refusal control. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical and moderate review findings block approval.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds record-driven TLS issuer configuration and stronger certificate provisioning and verification for hosted instances.
Changes:
- Adds default and opt-out
ClusterIssuersupport. - Updates Helm ingress rendering and operator certificate handling.
- Adds RBAC, tests, fixtures, and documentation.
File summaries
| File | Description |
|---|---|
test/MeshWeaver.Deployment.Contract.Test/FluentBuilderTest.cs |
Tests issuer defaults and opt-out behavior. |
src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-15-a-new-instance-gets-its-own-certificate.md |
Documents the TLS provisioning fix. |
src/MeshWeaver.Documentation/Data/Architecture/ConfiguringAnInstanceFromAspire.md |
Documents issuer configuration and ingress parity. |
src/MeshWeaver.Deployment.Contract/InstanceShape.cs |
Adds the ClusterIssuer contract field. |
src/MeshWeaver.Deployment.Contract/DeploymentRecordExtensions.cs |
Extends WithIngress with issuer configuration. |
deploy/helm/values.yaml |
Adds chart issuer configuration. |
deploy/helm/templates/memex-portal/ingress.yaml |
Renders issuer annotations and refuses missing issuers. |
deploy/aks/scripts/testdata/values.ingress-preprovisioned.yaml |
Adds a pre-provisioned TLS fixture. |
deploy/aks/scripts/testdata/values.ingress-no-issuer.yaml |
Adds a missing-issuer refusal fixture. |
deploy/aks/scripts/testdata/values.ingress-issuer.yaml |
Adds an issuer-backed ingress fixture. |
deploy/aks/scripts/check-chart-invariants.sh |
Adds ingress invariant checks. |
deploy/aks/operator/test/stubs/verify/openssl |
Stubs certificate verification. |
deploy/aks/operator/test/stubs/verify/curl |
Stubs endpoint verification. |
deploy/aks/operator/test/run-tests.sh |
Adds operator TLS regression tests. |
deploy/aks/operator/test/check-rbac-coverage.sh |
Checks cert-manager resource usage. |
deploy/aks/operator/bin/hosting-verify |
Validates the served certificate. |
deploy/aks/operator/bin/hosting-tls |
Creates and waits for cert-manager Certificates. |
deploy/aks/manifests/hosting-operator/operator-rbac.yaml |
Grants Certificate permissions. |
Review details
Suppressed comments (7)
deploy/aks/operator/test/check-rbac-coverage.sh:53
- Mapping
certificatehere only exposeshosting-tls'sget certificatescall. The Certificate's create/update/patch operations still usekubectl apply -f -, which this checker explicitly skips, so the new rule can lose write verbs whilecheck-rbac-coverage.shremains green and provisioning fails. Add a checked declaration for that write surface or make the manifest inspectable by the checker.
certificate|certificates|cert) echo "cert-manager.io certificates" ;;
deploy/helm/templates/memex-portal/ingress.yaml:6
- This refusal also runs for the tracked LocalColima overlay, which enables ingress and sets
tlsSecret: memex-portal-tlsbut has noclusterIssueror hand-written issuer annotation.memex-locallayers that overlay and creates the mkcert Secret before Helm, somemex-local upnow fails at render instead of installing; mark that overlay asclusterIssuer: none(and cover the real overlay in the invariant inputs) for the pre-provisioned local shape.
{{- if and $tls (not $issuer) (not $handWritten) }}
{{- fail (printf "ingress.tlsSecret is %q but NO issuer reaches the ingress. Set ingress.clusterIssuer (the fleet's is letsencrypt-prod), or ingress.clusterIssuer=none when that Secret is created by other means. An ingress naming a TLS secret nobody issues serves ANOTHER host's certificate: pearl.meshweaver.cloud served CN=memex.meshweaver.cloud for nine hours on 2026-09-15." $tls) }}
deploy/helm/templates/memex-portal/ingress.yaml:33
- The new positive fixture only goes through generic deployment invariants;
check-chart-invariants.pyhas no ingress/cluster-issuer assertion. Removing the emission on line 33 would therefore still leave this 'issued by cert-manager' combination green. Add a positive assertion for the rendered issuer (and thenonemarker/override behavior).
cert-manager.io/cluster-issuer: {{ $issuer | quote }}
src/MeshWeaver.Deployment.Contract/DeploymentRecordExtensions.cs:281
- The core
WithIngressnow advertisesclusterIssuer, but the public Aspire adapter atmemex/aspire/Memex.Aspire.Hosting/MemexHostingExtensions.cs:199still exposes only the old parameters and never forwards an issuer. Aspire callers therefore cannot select a custom issuer or thenoneopt-out even though this contract and the documentation say they can; update the adapter and its parity coverage.
string? clusterIssuer = null)
src/MeshWeaver.Deployment.Contract/InstanceShape.cs:318
- The new
[Description]is used as the property label, but it has no German[Translation]. This makes this newly exposed setting render in English for German viewers; add thedetranslation beside the description.
[Description("cert-manager ClusterIssuer for the TLS secret — 'none' opts out")]
src/MeshWeaver.Documentation/Data/Architecture/ConfiguringAnInstanceFromAspire.md:183
- The parity row now documents
clusterIssueras an argument toWithIngress, but the Aspire adapter atmemex/aspire/Memex.Aspire.Hosting/MemexHostingExtensions.cs:199still exposes only the five old parameters and forwards no issuer. Aspire callers therefore cannot select a non-default issuer or thenoneopt-out despite this documentation and the contract API. Update the mirror in the same change.
| `WithIngress(className, tlsSecret, annotations, sessionAffinity, affinityCookie, clusterIssuer)` | `Ingress.ClassName`, `Ingress.TlsSecret`, `Ingress.Annotations`, `Ingress.ClusterIssuer`, `Ingress.SessionAffinity.Enabled`, `Ingress.SessionAffinity.CookieName` | `ingress` | the issuer DEFAULTS to Let's Encrypt production, so a host with a TLS secret asks for its certificate without the call; the opt-out value is the literal none |
src/MeshWeaver.Documentation/Data/Architecture/ConfiguringAnInstanceFromAspire.md:183
- The new property is missing from the Labels appendix immediately below this table:
IngressSpeclistsClassName,TlsSecret,Annotations, andSessionAffinity, but notClusterIssuer. That appendix is the documented German-label source while this contract lacks declaration translations, so the new record field is omitted from the contract/renderer documentation. Add the English and German label row forClusterIssuer.
| `WithIngress(className, tlsSecret, annotations, sessionAffinity, affinityCookie, clusterIssuer)` | `Ingress.ClassName`, `Ingress.TlsSecret`, `Ingress.Annotations`, `Ingress.ClusterIssuer`, `Ingress.SessionAffinity.Enabled`, `Ingress.SessionAffinity.CookieName` | `ingress` | the issuer DEFAULTS to Let's Encrypt production, so a host with a TLS secret asks for its certificate without the call; the opt-out value is the literal none |
- Files reviewed: 16/18 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| {{- $handWritten := hasKey (.Values.ingress.annotations | default dict) "cert-manager.io/cluster-issuer" }} | ||
| {{- if and $tls (not $issuer) (not $handWritten) }} |
There was a problem hiding this comment.
Agreed, and this is the repo's own recurring defect shape rather than a nitpick — a key present with an EMPTY value is invisible to a presence check.
{{- $handWritten := hasKey (.Values.ingress.annotations | default dict) "cert-manager.io/cluster-issuer" }}
{{- if and $tls (not $issuer) (not $handWritten) }}
{{- fail (printf "ingress.tlsSecret is %q but NO issuer reaches the ingress. ..." $tls) }}
hasKey answers about the key, and $handWritten is then used as though it answered about the issuer. So
ingress:
tlsSecret: pearl-tls
annotations:
cert-manager.io/cluster-issuer: ""renders happily: $handWritten is true, the refusal is skipped, and cert-manager is handed an empty issuer name — which is exactly the state the refusal exists to prevent. The failure it then produces is the one in the fail message: the host is served the controller's fallback certificate, i.e. pearl.meshweaver.cloud presenting CN=memex.meshweaver.cloud for nine hours on 2026-09-15.
Worth noting the same bypass is reachable from the record route too, not only a hand-written overlay: an annotations entry rendered from a record field that happens to be empty produces the same empty-valued key.
The check should be on the value:
{{- $handWritten := (get (.Values.ingress.annotations | default dict) "cert-manager.io/cluster-issuer") | default "" }}
{{- if and $tls (not $issuer) (not $handWritten) }}
and this deserves a fixture in check-chart-invariants.sh beside the existing values.ingress-no-issuer.yaml refusal — one whose annotation is present-but-empty. Without it the invariant passes on a case it does not actually cover, which is the "a gate that checked nothing" shape AGENTS.md calls out. I ran the gate locally on this branch after merging main: 14 combinations render and all 5 refusal controls hold, so the present-but-empty case is genuinely unexercised today rather than merely untested.
| public static DeploymentContent WithIngress( | ||
| this DeploymentContent d, | ||
| string? className = null, | ||
| string? tlsSecret = null, | ||
| IEnumerable<KeyValuePair<string, string>>? annotations = null, | ||
| bool? sessionAffinity = null, | ||
| string? affinityCookie = null) | ||
| string? affinityCookie = null, | ||
| string? clusterIssuer = null) |
There was a problem hiding this comment.
The mechanism is right — appending an optional parameter is source-compatible and binary-breaking, because the CLR resolves WithIngress by full signature and the 5-argument one stops existing. Worth recording what I measured, because it changes how urgent this is and points at a different (real) gap.
Measured blast radius — small. Every caller is in-tree and gets recompiled together:
memex/aspire/Memex.Aspire.Hosting/MemexHostingExtensions.cs:199test/MeshWeaver.Deployment.Contract.Test/FluentBuilderTest.csandRendererParityTest.cs:169
and MeshWeaver.Deployment.Contract is not a standalone NuGet id — it rides inside the published Aspire package's own lib/ folder, so the contract and its one external caller ship as a version-matched pair rather than being upgradable independently. In-mesh C# is recompiled from source at runtime, so it is source-compatibility that governs there, not this.
But Public surface (binary compatibility) being green is not evidence against you. It is green on this head, and an optional-parameter addition is exactly the kind of change it evidently does not model — so the gate's silence here says nothing either way, and reading it as a clearance would be the mistake.
The gap I would actually fix is next door. The Aspire wrapper kept its five parameters:
public static IResourceBuilder<MemexPortalResource> WithIngress(this IResourceBuilder<MemexPortalResource> p,
string? className = null, string? tlsSecret = null, IEnumerable<KeyValuePair<string, string>>? annotations = null,
bool? sessionAffinity = null, string? affinityCookie = null)
=> p.Configure(r => r.WithIngress(className, tlsSecret, annotations, sessionAffinity, affinityCookie));so an AppHost cannot name an issuer through the documented surface at all, while ConfiguringAnInstanceFromAspire.md:184 already advertises the six-argument form. Since the record defaults to IngressSpec.DefaultClusterIssuer, the common case still works and the gap only bites the staging-issuer and explicit-opt-out cases — which are precisely the two this PR adds.
So: adding the six-argument overload alongside the preserved five-argument one costs nothing and closes the binary question outright, and the wrapper should gain clusterIssuer in the same change so the documented surface is reachable.
Clears the DIRTY merge state so the PR gets a real pull_request run again; its check-runs dated from 2026-09-15, ~520 commits behind main. One conflicted file, deploy/aks/scripts/check-chart-invariants.sh, three hunks: - COMBOS and REFUSALS: additive on both sides, kept both. This branch adds the cert-manager issuer / pre-provisioned-TLS fixtures and the no-issuer refusal; main added the Actions-executor, blank-maintainer and bake-gate-armed fixtures and the misspelled-executor refusal. - The header's invariant list was a genuine collision: both sides added an invariant numbered 18. Main's (the operator EXECUTOR) keeps 18 — it is already referenced by number at two places in the body — and this branch's ingress-issuer invariant becomes 19. Its number appeared only in the header, so nothing else moved. Verified by running the gate, not by reading it: all 14 values combinations render a self-consistent deployment and all 5 refusal controls hold, with both sides' new fixtures and both sides' new refusals exercised by name. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
| resource idx | next --- idx |
what that --- actually is |
slice len | contains the literal | |
|---|---|---|---|---|---|
main |
573 | 1183 | # ---- Next. (a comment) |
610 | ✅ true |
| this branch | 1247 | 1449 | # ---- Which certificate… (a comment) |
202 | ❌ false |
So the guard has never once found a real document separator — on main it stops inside the # ---- Next.js React GUI comment banner, and passes only because that banner happens to sit after the values reference. This PR adds a # ---- Which certificate this host is served with --- banner inside the annotations: block, i.e. before the reference, and the slice is cut to 202 characters ending at annotations:\n # .
The guard's own summary anticipates this class and still gets it wrong — it asserts both markers up front so that a refactor does not "surface as an opaque ArgumentOutOfRangeException, which says nothing about what an author has to fix" — but a silently-truncated slice is worse than that exception: it reports a missing value that is present.
The fix belongs in the guard, not here: a YAML document separator is a line that is exactly ---, so the search should be line-anchored ("\n---\n", or a line-wise scan), not a bare substring. That also retires the latent version of this on main, where the same bug is one comment-banner edit away from firing on an unrelated PR.
I have deliberately not edited the test — a guard asserting something that is not true is a finding to report, not a line to change, and the change touches a guard that covers main as well.
Left as a draft, unmerged and un-undrafted, as it was.
🤖 Generated with Claude Code
… three hyphens
IngressAuthCookieHeaderBudgetGuard sliced the portal Ingress resource with
IndexOf("---"), which matches inside a comment. It therefore never once found
a real YAML document separator:
main first hit '# ---- Next.' 610 chars in, AFTER
.Values.ingress.annotations => passed BY LUCK
this branch first hit '# ---- Which certificate' 202 chars in, BEFORE it
=> reported a value that is present as MISSING
The annotations are still taken from .Values.ingress.annotations (read into
$ann and rendered with toYaml); only the guard's predicate was wrong. A guard
that passes by luck is not a guard, and this false positive costs an author a
hunt for a defect that is not there.
NextDocumentSeparator matches ^---[^\S\n]*(?:#[^\n]*)?$ multiline, so a CRLF
ending, trailing spaces and a trailing # comment are all accepted while four
hyphens and an indented --- are not.
TheDocumentSeparatorIsALine_NotAnyThreeHyphens is the control: it pins the
banner shape that broke it, the four-hyphen and indented cases, each legal
spelling of a real separator, and that the search honours its start offset.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
ℹ️ Merge-queue steward: no action — removed from the queue with reason |
# Conflicts: # deploy/aks/scripts/check-chart-invariants.sh
|
Trunk merge after #4436 landed: both sides had appended fixture lines to |
A provisioning run renders its values from the deployment record. The cert-manager issuer lived only as a hand-written annotation in checked-in overlays, so a record without it rendered an ingress naming a TLS secret that nothing issued — the controller then served the fallback: another instance's certificate.
pearl.meshweaver.cloudservedCN=memex.meshweaver.cloudfor nine hours on 2026-09-15 with a healthy portal behind it, and its own Provision could not have fixed it either: step 15/18 died onso TLS and all three verification steps never ran.
What changes
IngressSpec.ClusterIssuer, defaulted to Let's Encrypt production. An instance declared with a host and a TLS secret asks for its certificate with no further call, and a record written before the field gets one too — an absent JSON member leaves the initializer standing. A hand-written annotation still wins; the literalnoneis the named opt-out.WithIngresstakes it.tlsSecretwith no issuer reaching it.hosting-tlstakes--issuer, waits for the Certificate to be Ready and the Secret to hold a key pair, and names which of the two failed.hosting-verifyreads the served certificate back: a wrong certificate is reported as a wrong certificate, not as thirty failed attempts that read like a dead application.cert-manager.ioCertificates.Why the grant was missing
check-rbac-coverage.shreadskubectl <verb> <resource>with literal resources and says so plainly: it cannot seekubectl apply -f -.hosting-tlscreated its Certificate exactly that way, so the script reached main without its rule and nothing was red. It now reads the object back explicitly — which it needs anyway for the Ready condition — so the grant is covered by the check from here on.Verification, run locally
check-chart-invariants.sh: 11 combinations, 4 refusal controls, all green — including two new fixtures (a public host issued by cert-manager, a pre-provisioned secret) and the new refusal control (a TLS secret with no issuer).operator/test/run-tests.sh: 471 passed, 0 failed, including 7 new ones — the fallback-certificate shape, the no-certificate shape, the wildcard shape, and that only a correct certificate lets the failure blame the application.check-rbac-coverage.sh: 36 kubectl calls, 15 grants, 0 missing.Not in this PR
The renderer half (MeshWeaver.Plugins: put the record's issuer into the rendered ingress values, refuse a TLS step for a record that names none) is a separate PR and needs this contract field sealed first, the same shape as #4417/#1919/#4453.
Deploy note: the new ClusterRole rule reaches the cluster when Systemorph/Memex's
helm-releaselane next deploys memex — that lane re-applies the operator's ClusterRole from this chart. Until thenhosting-tlsstays Forbidden.🤖 Generated with Claude Code