Skip to content

feat(remote-cache): add self-hosted public cache server - #718

Draft
fengmk2 wants to merge 15 commits into
mainfrom
feat/public-remote-cache
Draft

fengmk2 wants to merge 15 commits into
mainfrom
feat/public-remote-cache

Conversation

@fengmk2

@fengmk2 fengmk2 commented Sep 12, 2026

Copy link
Copy Markdown
Member

Add packages/remote-cache for the server in #716, using Cloudflare Workers, primary D1 metadata, and a private R2 bucket. The server supports public reads and GitHub Actions OpenID Connect checks for writes. Stores use streaming uploads, atomic publication, storage limits, and automatic data expiry.

One workflow deploys related changes to a persistent staging Worker, D1 database, and R2 bucket. Internal PRs, main pushes, and manual runs share these resources. The workflow runs deployments and smoke tests in sequence, then updates PR comments with the tested revision and manual instructions. Each deployment replaces the previous revision. Closing a PR keeps staging available. Setup uses repository secrets and variables. Repository maintainers can configure staging without a GitHub environment.

The e2e plan defines automated checks and manual exercises. PR runs check public reads and rejected writes. Pushes to main also check authorized uploads, multipart storage, replacement, concurrency, and quotas. Local tests repeat smoke checks against reused storage and retain maximum-payload and scheduled-handler coverage. Real Cron execution and maximum payloads on Cloudflare require separate release exercises.

/fetch returns only { kind: "fallback", key } for fallback matches and does not read R2. Exact matches return the value and blob_id. A missing or unreadable exact value returns 503. Fetch misses and unavailable blobs return plain-text 404, as specified in the local RFC.

Setup rejects incompatible origins and repositories before it changes existing policies or saved configuration. Each Worker has separate rate-limit counters that remain stable across revisions. Workers Free CPU support remains unverified. The guide recommends Workers Paid for the full payload limits.

Motivation

Maintainers need a cache service in their own Cloudflare account. Developers and fork contributors must reuse public task results without login. Only trusted jobs on main should publish those results. Reviewers need one persistent staging environment and a clear verification result after each related change.

@socket-security

socket-security Bot commented Sep 12, 2026

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Sep 12, 2026

Copy link
Copy Markdown

fspy benchmark

linux

dynamic/launch             change  +1.89%  [ -7.56% .. +13.22%]  overhead  +287.93%
dynamic/access             change  +0.13%  [-12.85% .. +34.83%]  overhead    +8.04%
dynamic/access-relative    change  -5.01%  [-28.87% .. +26.71%]  overhead   +48.78%
dynamic/access-contended   change  -0.91%  [-16.30% .. +13.55%]  overhead   +10.63%
static/launch              change  -0.86%  [-11.23% .. +10.77%]  overhead  +675.76%
static/access              change  +0.08%  [ -2.74% ..  +8.09%]  overhead  +817.01%
static/access-relative     change  +0.02%  [ -1.80% ..  +1.53%]  overhead +1305.30%
static/access-contended    change  -0.10%  [ -3.50% ..  +4.21%]  overhead +3281.11%

macos

dynamic/launch             change  -0.00%  [ -4.43% ..  +3.65%]  overhead  +222.76%
dynamic/access             change  -1.01%  [-13.15% ..  +5.47%]  overhead    +3.10%
dynamic/access-relative    change  +0.38%  [ -2.80% ..  +3.19%]  overhead  +263.54%
dynamic/access-contended   change  +3.39%  [ -4.13% ..  +8.55%]  overhead    +3.81%

windows

dynamic/launch             change  +0.08%  [ -7.07% ..  +9.42%]  overhead   +24.25%
dynamic/access             change  +0.52%  [ -1.95% ..  +4.39%]  overhead    +2.17%
dynamic/access-relative    change  -0.17%  [ -8.16% ..  +3.24%]  overhead    +0.71%
dynamic/access-contended   change  +1.32%  [ -2.07% ..  +7.49%]  overhead    +2.56%

@github-actions

github-actions Bot commented Sep 12, 2026

Copy link
Copy Markdown

Remote cache staging

Commit: e835e654fc41e3c6fdadc3602b66e6bbb8c80ced

Cloudflare staging deployment and smoke tests passed. You can now perform manual verification.

Endpoint: https://vp-cache-ci-staging.voidzero-docs.workers.dev/projects/manual

Download the remote-cache-e2e-35074425522-1 artifact from the workflow run. It contains manual-fetch.cbor, manual-manifest.json, and report.json.

PR checks cover public reads and rejected writes. Main-branch push checks also cover authorized HTTP stores. This staging endpoint is shared by all PRs and main. A later deployment replaces its code and manual fixture. Compare the deployment ID in the response with the artifact before manual verification. Closing this PR does not remove staging.

Manual checks and complete e2e plan.

fengmk2 and others added 8 commits September 14, 2026 23:14
Co-authored-by: GPT-6 Codex <codex@openai.com>
Co-authored-by: GPT-6 Codex <codex@openai.com>
Co-authored-by: GPT-6 Codex <codex@openai.com>
Co-authored-by: GPT-6 Codex <codex@openai.com>
Co-authored-by: GPT-6 Codex <codex@openai.com>
Copy RFC #716 from c201f8e and adjust relative paths.

Co-authored-by: GPT-6 Codex <codex@openai.com>
Co-authored-by: GPT-6 Codex <codex@openai.com>
Co-authored-by: GPT-6 Codex <codex@openai.com>
@fengmk2
fengmk2 force-pushed the feat/public-remote-cache branch from 194d39f to e461d47 Compare September 14, 2026 15:16
fengmk2 and others added 5 commits September 16, 2026 00:04
Co-authored-by: GPT-6 Codex <codex@openai.com>
Co-authored-by: GPT-6 Codex <codex@openai.com>
Co-authored-by: GPT-6 Codex <codex@openai.com>
Remove the obsolete cache size study and its RFC link.

Co-authored-by: GPT-6 Codex <codex@openai.com>
@fengmk2

fengmk2 commented Sep 16, 2026

Copy link
Copy Markdown
Member Author

Deploy prompt:

Refer to the deployment instructions at https://github.com/voidzero-dev/vite-task/blob/feat/public-remote-cache/packages/remote-cache/docs/self-hosting.md to deploy a remote cache service for the current repo.

fengmk2 and others added 2 commits September 16, 2026 16:33
Co-authored-by: GPT-6 Codex <codex@openai.com>
Co-authored-by: GPT-6 Codex <codex@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant