fix: back up with aws s3 sync instead of a Docker Hub pull [sc-71258]

[NJordan72]

Ticket: SC-71258 · Audit issue: narrative-io/mintlify-docs#824 · Related: SC-71065

The failure

Every run of this workflow has failed since 2026-09-12 — here, and in every repository that calls it. narrative-io/mintlify-docs has taken no offsite copy for four days:

Run	Date (UTC)	Conclusion
34568221941	2026-09-11 06:01	success — mirrored 864 objects
34677061640	2026-09-12 06:01	failure
34741752872	2026-09-13 06:01	failure
34811795212	2026-09-14 06:02	failure
34935151207	2026-09-15 06:01	failure

peter-evans/s3-backup is a Docker action whose Dockerfile:1 is FROM minio/mc:RELEASE.2022-05-04T06-07-55Z. The self-hosted runner rebuilds that image on every run, and Docker Hub now refuses the anonymous pull of that 2022 tag:

#2 [internal] load metadata for docker.io/minio/mc:RELEASE.2022-05-04T06-07-55Z
#2 ERROR: pull access denied, repository does not exist or may require authorization:
   server message: insufficient_scope: authorization failed
##[error]Docker build failed with exit code 1

The image build is step 3 of 8, so actions/checkout, Configure AWS credentials and S3 Backup itself are all skipped and nothing is written:

1 Set up job                            -> success
2 Set up runner                         -> success
3 Build peter-evans/s3-backup@59efd330  -> failure
4 Run actions/checkout@ee0669bd         -> skipped
5 Configure AWS credentials             -> skipped
6 Get current date                      -> skipped
7 S3 Backup ()                          -> skipped

No commit landed in any of these repositories on 2026-09-12. The failure is external to the workflow, which is why it arrived silently.

The fix

Drop the action and use the AWS CLI directly.

Authenticating the pull (docker/login-action) would fix today's symptom and keep the dependency; forking the action onto a current minio/mc tag defers it to the next tag Docker Hub stops serving anonymously. Neither is worth carrying for one sync.

The CLI is already installed on these runners and already reads the credentials Configure AWS credentials exports two steps up — narrative-marketplace-backend's control-plane-workflow-dsl-docs.yml does bare aws s3 cp on runs-on: self-hosted behind the same configure-aws-credentials@e6de0542 step, and its last five runs all succeeded. That also makes the ACCESS_KEY_ID / SECRET_ACCESS_KEY env plumbing (and its comment about which outputs the credentials action really exposes) unnecessary, so it goes too.

Behaviour is unchanged: --delete is mc's --remove, and --overwrite has no counterpart because sync replaces anything whose size or mtime differs and the target prefix is date-stamped, so each run writes into an empty prefix regardless. aws s3 sync . includes .git/, as mc mirror did — which is what persist-credentials: false on the checkout is guarding.

The one addition is the trailing aws s3 ls --recursive --summarize, which puts an object count in the log. AUD-OPS-05 asks whether a scheduled backup "did the thing its name promises"; until now that could only be answered with an AWS credential, and the audit that filed this has none.

Verification

• actionlint (which shellchecks the new run: block) — clean across all four workflows
• zizmor --persona=regular — No findings to report, no suppression added
• Behaviour itself can only be confirmed by a real run on a self-hosted runner with the backup role. After merge, gh workflow run backup-daily.yml -R narrative-io/common-github exercises this repository's own caller; the run should conclude success, its S3 Backup step should not be skipped, and the object count should print.

Fixes narrative-io/mintlify-docs#824

🤖 Generated with Claude Code

[NJordan72]

Root cause confirmed: the image was withdrawn, not gated

Filed as SC-71258, related to SC-71065 — which hit the sibling image minio/minio in the backend test suite on 2026-09-14. Same namespace, same window, same signature. MinIO withdrew their images from Docker Hub and moved the community images to quay.io.

Verified directly rather than inferred from the error string:

check	minio/mc	control (library/alpine)
anonymous pull-scope manifest	401 insufficient_scope, no ratelimit headers	200, ratelimit-remaining: 98;w=3600
hub.docker.com/v2/repositories/...	404 {"message":"object not found"}	200
quay.io	200, public, tags begin 2025	n/a

Both requests left the same IP. A rate limit is a 429 carrying quota headers — this never reaches a quota check, and alpine pulled fine alongside it with 98/100 remaining.

This rules out one of the alternatives in the filed issue. Adding docker/login-action would not fix it: authentication is not the problem, the repository is gone. Repointing to quay.io/minio/mc (SC-71065's route) would work, but keeps a container dependency for a single sync.

The failure is org-wide, not one repository

Every repository that calls this workflow. Run history across all 88 callers:

Date (UTC)	success	failure
2026-09-08	76	0
2026-09-09	76	0
2026-09-10	77	0
2026-09-11	79	0
2026-09-12	0	79
2026-09-13	0	80
2026-09-14	0	80
2026-09-15	0	83

82 distinct repositories have failed at least once since 2026-09-12, and zero have succeeded. The org has taken no offsite copy for four consecutive mornings.

Note that merging this alone fixes nothing: each caller pins this workflow by SHA, so the rollout is 88 SHA bumps. That is tracked in SC-71258 and is the bulk of the work.

🤖 Generated with Claude Code

[mbabic]

Closing in favour of a central backup: narrative-security's github-backup.yml (narrative-io/narrative-security#110) now backs up every repository as a git bundle, so the shared backup.yml is being removed rather than fixed. Thanks for tracking down the Docker Hub root cause.