Scaffold hardened Axum application Docker recipes

[ChristianPavilonis]

Summary

• Scaffold a hardened Axum application Dockerfile and .dockerignore, using the application's lockfile and declared Rust toolchain, pinned GNU/Bookworm inputs, CA trust, numeric non-root identity, exec startup, and explicit container binding.
• Extend this PR with Axum: validate multi-platform containers and provide an application publishing workflow #399 preparation: native amd64/arm64 container CI, verified OCI artifact handoff, and a copyable application-owned publishing recipe that preserves the exact tested image bytes.
• Add Axum: document Docker, Kubernetes, and managed-cloud deployments #400 preparation: Docker/Compose guidance and executed packaging smoke, plus a Kubernetes guide draft selecting kind and Envoy Gateway with Gateway API. Runnable Kubernetes manifests and cluster/ECS/Fargate execution remain outstanding.
• Record an AWS-first reference deployment and acceptance matrix linking Add AWS configuration and secret-store providers for self-hosted applications #410/Add native AWS store providers #414 provider work to this packaging/deployment contract. The planned AWS Kubernetes profile includes AppConfig Agent, Secrets Manager and DynamoDB after accepted provider integration; ECR is a proposed publishing option, not an implemented or tested destination.
• Keep framework packaging scans report-only while preserving strict application-release verification. No external image publication, production deployment, runtime changes, new revision flag, or dependency resolver is included.

Scope and issue relationships

Part of #391.

Closes #398.

The existing #398 closing association remains, but this PR is still draft. Packaging evidence does not satisfy its remaining production-integration acceptance or justify merging before those requirements are resolved.

Related to #399 and #400. This PR implements their preparation slices, not their complete acceptance criteria; neither issue should close on this merge.

Related to #410 and draft PR #414 for AWS provider integration. This PR adds reference design and deployment guidance only, not provider implementation, DynamoDB/identity certification, real AWS acceptance or a closing association for #410.

Issue	Work delivered here	Still outstanding
#398	Hardened Docker scaffolding, portable-consumer instructions, constrained container checks	Accepted-runtime required-config failure, durable restart, readiness, and bounded SIGTERM acceptance
#399	Native amd64/arm64 execution, tested-artifact handoff/index checks, app-owned GHCR release recipe	Controlled GHCR publication/attestation verification, strict release acceptance, and accepted-runtime final-image cases
#400	Docker/Compose guide, no-store example and executed packaging smoke; Kubernetes guide draft with local, AWS-provider and single-writer redb profiles; selected ECS/Fargate target	Production Docker/Compose cases, runnable manifests and disposable kind/Envoy Gateway verification, and authorized stateless ECS/Fargate deployment
#410	AWS reference design linking independent config/secrets/KV selection, least-privilege credentials and acceptance requirements	Provider implementation/integration belongs to #414; real resource/IAM/version/DynamoDB evidence remains separately gated

Applications own their release artifacts, registry approval, credentials, configuration, secrets, storage policy, and deployment. Trusted Server's actual image, asset/config integration, installer, and releases remain downstream.

Changes

Area	Change
CLI generator/scaffold and root templates	Conditional Dockerfile/.dockerignore rendering, shared Rust pin, toolchain checks, safe README image tags, runtime package upgrades, and regression tests
scripts/prepare_generated_axum_container.sh	Generate disposable container-probe, deliberately stage framework sources inside its context, preserve dependency features, and prepare one application-owned lockfile for both architectures
scripts/test_generated_axum_container.sh	Context/filesystem/ELF checks, native architecture assertions, UID/read-only-root/capability/mount checks, HTTP and controlled positive/negative TLS; retained-image mode preserves borrowed images
.github/workflows/axum-container.yml	Read-only preparation, native Ubuntu amd64/arm64 execution, exact same-run/attempt artifact selection, separate-job import, and localhost runtime-index verification
scripts/test_axum_container_handoff.sh and examples/container-publishing/	OCI manifest/blob/input verification, digest-preserving transfer/copy, subject-bound scans, SBOM/provenance records, and inactive app-owned GHCR recipe with a separate protected publisher
examples/axum-deployment/compose.yaml and deployment smoke	No-store digest-based execution, UID/GID 10001, read-only root, dropped capabilities, no-new-privileges, localhost publication, stop/start and teardown
Axum, publishing and deployment guides	Build/release ownership, operator config/storage/proxy boundaries, security policies, rollback, and explicit runtime/Kubernetes/Fargate acceptance gaps
docs/guide/deploying-axum-kubernetes.md and documentation navigation	Draft kind/Envoy Gateway recipe, illustrative security/Gateway/HTTPRoute YAML, actual probe and proxy boundaries, single-writer storage, shutdown/replacement/rollback, troubleshooting and an execution matrix
docs/specs/aws-reference-deployment.md	AWS-first Fargate/AppConfig/Secrets Manager design, optional application-state DynamoDB and proposed ECR paths, 22 acceptance cases, compatible image/config/secret release identity, IAM ownership and explicit authorization gates
Existing static/security gates and helper tests	Include new workflow, external recipe, scripts, verifier and Compose inputs in applicable checks

The CI image is a generated disposable container-probe application, not app-demo, Trusted Server, or a public EdgeZero application image. Test-only source staging does not change scaffold dependency selection or replace the original standalone-consumer proof using portable Git dependencies.

CI and recorded security status

The documentation-only update is pushed at 7f846ae02cf2ff334673c59ccd2f6eef076139ba. Current-head hosted CI still requires verification. Container/runtime evidence below was recorded at implementation head 68894a05f6f4a8c49a8a2b7bd7fee2bed24efcfe; it is not a claim that those jobs have rerun at the documentation head.

• Container preparation run: preparation, native amd64, native arm64, and handoff all passed. Native application images execute on their matching hosted runners; no QEMU substitution is used.
• Both variants passed constrained packaging/HTTP/TLS and no-store Compose checks. Retained OCI artifacts survived cross-job import and localhost index assembly without rebuilding.
• At that recorded implementation head, formatting, native tests, provider Clippy, docs formatting, static/security checks, Cloudflare/Spin WASM tests, and action smoke checks passed.
• At that implementation checkpoint, the Fastly WASM test job failed. Fastly unit tests passed, but the reusable HTTP smoke failed for custom-b with missing partial committed response. The previous run passed; intermittence is suspected, not confirmed. Investigate and verify current-head results rather than treat it as an image-scan failure.
• GitHub currently reports conflicts with main; branch integration and affected validation remain outstanding.

Framework packaging/handoff uses SCAN_POLICY=report-only. Scan execution, malformed reports and artifact-integrity failures still block; vulnerability findings are retained and reported, not waived. The publisher and release-record verification remain strict by default and reject High/Critical findings, including unfixed findings, unless an independently trusted owner-reviewed scoped exception is supplied. Report-only policy cannot authorize GHCR publication.

Bookworm package upgrades cleared the eight findings with available fixes in the recorded scan. The latest recorded image scans report 50 High and 1 Critical finding, with no available fixes reported. Packaging passes under report-only; strict release verification rejects the same image. No vulnerability exceptions have been authorized. The image scanner found no language-specific files, so the SBOM is not independent certification of every statically linked Rust dependency.

Documentation update validation

For the documentation-only change in 7f846ae0:

• Scoped Prettier checks passed for both deployment guides, the AWS reference draft and VitePress navigation.
• npm run lint and npm run build passed at the unchanged documentation inputs. The Kubernetes page rendered and local Markdown links resolved.
• Both embedded Kubernetes YAML fragments parsed with the installed Prettier YAML parser. This is syntax evidence, not Kubernetes schema/server validation.
• The AWS reference review clarified that runtime roles cannot publish images, deploy config, mutate secrets or change infrastructure, while approved-table DynamoDB state writes are permitted.
• cargo test -p edgezero-cli passed before commit: 207 unit tests and one external-consumer test passed; the generated-workspace build test remained ignored.
• git diff --check and staged-diff checks passed. Only four documentation/navigation files changed.
• No Kubernetes controller, cluster, AWS service, registry publication or deployed upgrade/rollback was exercised. The guide explicitly identifies runnable manifests, pins and runtime/deployment checks as remaining work.

Recorded local validation

These are executed results from their recorded implementation checkpoints, not a claim that every local command was rerun at the current head or that all CI is green.

• Workspace tests, strict all-feature Clippy, formatting, native adapter feature checks, Spin WASM check, generated-consumer host/typed-CLI/three-WASM builds, and locked app-demo tests passed during implementation.
• Docs install/lint/format/build, shell syntax/ShellCheck, actionlint, action-reference checks, Zizmor, and existing deploy-action contracts passed at the preparation checkpoint.
• Final focused verification covered 27 Python helper tests, CLI tests, formatting, real-image report-only acceptance, and strict rejection of that same image.
• Original local-build and retained-image smokes passed on native amd64, including filesystem/native libraries, mount permissions, published HTTP, and TLS success/untrusted-CA/wrong-hostname cases.
• Original standalone consumer used manually prepared public Git dependencies at 98930917, its own committed Cargo.lock, and no out-of-context dependencies. Its unchanged generated Dockerfile passed the native amd64 smoke.
• A locked app-owned Node asset stage ran before Rust, embedded generated bytes, and left Node/npm out of the runtime image. Node is not required for generic consumers.

Remaining acceptance

• Integrate accepted PR feat: implement portable outbound HTTP and request lifecycle contracts #275 contracts and the owning Axum: define a production server lifecycle with bounded graceful shutdown #392/Axum: make production configuration and local storage explicit #396 lifecycle/configuration work, then reuse residual Axum: enforce configurable buffered request-body limits #393/Axum: preserve incremental streaming with bounded backpressure #394/Axum: reuse outbound clients and preserve proxy HTTP semantics #395/Axum: add safe production diagnostics and trusted-proxy handling #397 HTTP and diagnostics checks through the actual final images.
• Prove required-input startup failure before readiness, mounted-state persistence/recovery, competing-writer rejection, and bounded SIGTERM/draining on both native architectures. Current Compose stop/start and the development run_app entrypoint do not prove these properties.
• Obtain a policy-compliant release candidate or explicit owner-approved scoped expiring exceptions; execute separately authorized disposable app-owned GHCR publication and attestation verification without rebuilding tested artifacts.
• Deliver and execute the planned plain Kubernetes manifests in a disposable kind cluster with Envoy Gateway, including probes, single-writer storage, routed HTTPS/proxy trust, shutdown, upgrades and rollback. Local results do not qualify EKS or real AWS services.
• Integrate accepted AppConfig Agent, Secrets Manager and DynamoDB providers for the separate AWS Kubernetes profile. Real IAM/resource/version/table acceptance requires separate authorization; ordinary KV proof does not certify Trusted Server identity/CAS. The proposed ECR transfer and metadata-verification path remains unimplemented and untested.
• Exercise the selected first managed target, stateless ECS/Fargate, only with explicit account/resource/credential/budget/cleanup approval. No cloud deployment has occurred.

Packaging preparation is not production certification. Preserve accepted HTTP protocol limits and BestEffort/Unsupported distinctions. No externally published image, GHCR/ECR acceptance, Kubernetes/EKS acceptance, Fargate acceptance, real AWS provider acceptance, identity/CAS certification, end-client-delivery guarantee, or multi-writer embedded-store support is claimed.