feat: migrate to Crossplane v2 and upgrade Backstage to 1.54.4 with Crossplane/Kyverno/Argo CD plugins

[koorikla]

Hi! Thanks for this project — it's a genuinely useful reference stack. I ran it locally, hit a few things, and ended up doing the Crossplane v2 migration. Opening this in case it's useful; happy to split it up, drop parts, or rework anything.

There were no open issues covering this, so nothing to link.

Why

Crossplane v2 removes claims. On apiextensions.crossplane.io/v2, claimNames is explicitly deprecated — "Claims aren't supported in apiextensions.crossplane.io/v2" — and a namespaced XR becomes the user-facing API instead. The stack currently models everything around XQueueClaim, so moving to v2 changes the API surface rather than just a version number.

What changed

Crossplane 1.19.0 → 2.4.0, claims → namespaced XRs

• XRD is now apiextensions.crossplane.io/v2 with scope: Namespaced; a team creates an XQueue directly in its namespace.
• crossplane/claims/ → crossplane/xrs/, each XR carrying metadata.namespace.
• crossplane/namespaces/ creates the team-a/team-b tenant namespaces in an earlier sync wave — Argo CD's CreateNamespace=true only covers an Application's own destination namespace, not where tenant XRs land.

A knock-on effect: a namespaced XR can't compose a cluster-scoped managed resource. provider-aws-sqs:v2 ships every kind twice — cluster-scoped under sqs.aws.upbound.io and namespaced under sqs.aws.m.upbound.io — so the composition uses the namespaced Queue, which Crossplane places in the XR's own namespace. Those reference a ClusterProviderConfig (group aws.m.upbound.io) so one LocalStack config is shared by every tenant namespace.

go-templating replaces patch-and-transform, with function-auto-ready added — patch-and-transform derived XR readiness implicitly, and a go-templating-only pipeline needs it explicitly.

provider-kubernetes dropped — unused by any composition. Its ProviderConfig, DeploymentRuntimeConfig and cluster-admin binding go with it.

Backstage 1.47.3 → 1.54.4 and the hand-written entity provider replaced. packages/backend/src/crossplane-entity-provider.ts queries the xqueuesclaim plural in a hardcoded namespace, which returns 403 after the migration — the aggregated crossplane-view role only grants access to kinds that exist. @terasky/backstage-plugin-kubernetes-ingestor handles v2 natively (it branches on the XRD's spec.scope) and additionally generates a scaffolder template and an API entity per XRD, so the platform API is discoverable without hand-maintaining either. Added alongside it: crossplane-resources-frontend, kyverno-policy-reports, the two permissions backends, and RoadieHQ's Argo CD plugin — the ingestor already annotates entities with argocd/app-name, which is exactly what it resolves.

Bug fixes found while getting make up to complete

These are independent of the migration and each one broke a clean run:

• .bootstrap/argocd/up.sh — the port-forward guard used lsof to test whether something held port 8080. The rollout restart immediately above kills the existing forward, but the dying process still owns the socket, so the guard skipped starting a replacement and the following argocd login failed against a forward that was already gone. Now probes the port and waits until it actually accepts connections.
• .bootstrap/backstage/up.sh — export $(cat .env | xargs) splits a trailing # comment into bare words that export rejects. Because the secret apply was chained with &&, that failure skipped creating the secret instead of stopping the script, leaving Backstage running with no GitHub token and nothing reporting it. Also echo "$TOKEN" | base64 bakes a trailing newline into the secret, and GNU base64 line-wraps past 76 chars, so longer tokens were corrupted on Linux.
• .bootstrap/kyverno/up.sh — the readiness loop counted every container reporting ready=false, including the kyverno-migrate-resources Job pod, which stays Succeeded/ready=false forever. Once that job had run the wait could never pass, so it timed out with Kyverno perfectly healthy — and the policies were never applied.
• argocd/apps/localstack/app.yaml — localstack/localstack:latest now resolves to the 2026.x unified image, which exits 55 (License activation failed) without a LOCALSTACK_AUTH_TOKEN. Pinned to 4.14.0, the last community release that starts unlicensed. Pinned via helm.parameters because valueFiles doesn't resolve against the repo for a chart-repo source — it reads the chart's own values.yaml, so argocd/apps/localstack/values.yaml was being silently ignored. Worth knowing: argocd/apps/kyverno/values.yaml is ignored for the same reason.
• Two permanently OutOfSync Applications — backstage-app failed every sync with illegal base64 data at input byte 0, because the source path contains backstage-secrets with GITHUB_TOKEN: <placeholder>; Argo CD was repeatedly trying to overwrite the real token with a value the API server rejects. The secret template moved outside the Application's path. kyverno-app reported 11 CRDs OutOfSync forever because the chart renders metadata.labels: {} and Kubernetes drops the empty map — added an ignoreDifferences for it.

Verification

Everything was exercised on a local kind cluster via make up, which now exits 0:

• Crossplane v2.4.0; XRD apiextensions.crossplane.io/v2, scope=Namespaced
• provider-aws-sqs + upbound-provider-family-aws and both functions Healthy
• Both XQueues SYNCED=True READY=True, each composing a namespaced Queue in its own namespace, backed by real LocalStack queues at the right regions (team-a → sqs.eu-north-1…, team-b → sqs.us-east-2…)
• Kyverno denies an XQueue with maxMessageSize: 999999 at admission — a bound only the policy enforces, not the XRD schema
• Backstage, Argo CD, Crossview and LocalStack all serving; the ingestor registers the XRs and generates the template and API entity per XRD

The composition was also checked with crossplane render on both branches of every conditional, including the case where all optional fields are omitted.

Notes

• location moved from oneOf: [{pattern: ^EU$}, {pattern: ^US$}] to enum: [EU, US]. Same accepted values, but neither oneOf branch carried a type, so the scaffolder form generated from the schema failed with "Unsupported field schema for field root_location". The hand-written template hid this by hardcoding the enum in its own parameter schema.
• Upper bounds on maxMessageSize/visibilityTimeoutSeconds are still deliberately absent from the XRD — Kyverno enforces those, and that split is what demonstrates admission policy catching what the CRD schema doesn't.
• kubernetesIngestor.components.enabled must stay true: setting it false doesn't filter workloads, it short-circuits the whole provider and removes what it already tracked. System namespaces are excluded via excludedNamespaces instead.
• This targets LocalStack as the repo already does; I haven't tested against real AWS.