Skip to content

feat(policy)!: require protobuf-shaped policy YAML - #3401

Open
pimlock wants to merge 10 commits into
mainfrom
3388-public-policy-proto/pimlock
Open

feat(policy)!: require protobuf-shaped policy YAML#3401
pimlock wants to merge 10 commits into
mainfrom
3388-public-policy-proto/pimlock

Conversation

@pimlock

@pimlock pimlock commented Sep 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Make openshell.policy.v1.PolicyDocument the public authored policy contract for policy YAML and gateway APIs while retaining SandboxPolicy as the private effective/runtime and durable model.

This is a breaking change. Policy YAML now uses protobuf field shapes directly: endpoints require ports, matcher oneofs use glob or any.values, and runtime-only authority fields are not authorable.

Related Issue

Closes #3388

Follow-up: #3476 tracks the pre-existing internal any-binary merge-semantics inconsistency, which is deliberately outside this contract migration.

Changes

  • Add the public openshell.policy.v1.PolicyDocument schema with vendored buf.validate annotations and strict bounded YAML/JSON conversion.
  • Lower public policies explicitly into the existing internal runtime model and project base/effective policies explicitly at gateway and curated SDK boundaries.
  • Keep durable sandbox, policy-revision, and provider-profile payloads internal and compatible; canonicalize historical duplicate set values when projecting old records.
  • Enforce portable constraints in Rust, Go, Python, and TypeScript, including incremental fragments and complete merged/profile documents before persistence.
  • Update the Rust, Go, Python, and TypeScript SDK surfaces, generated bindings, examples, provider profiles, E2E fixtures, public docs, architecture docs, and policy-generation skill.
  • Harden TypeScript runtime-policy projection for untrusted __proto__ and constructor parameter keys.

Design gists

These gists record the investigation that led to the public/internal split and the protobuf-shaped YAML migration:

Testing

  • mise run pre-commit
  • mise run test:rust (full workspace, server, integration, doc, and example suites)
  • mise run test:python (256 passed)
  • mise run go:ci
  • mise run sdk:ts:ci (125 passed)
  • mise run docs and mise run proto:lint
  • mise run e2e (MCP 3/3, Python 54 passed/81 skipped, full Rust E2E green)
  • Clean Python 3.11 wheel install/validation smoke test
  • Local Docker external-driver lifecycle with the explicit baseline policy (create, exec, stop/start persistence, delete)
  • Local Podman external-driver lifecycle against the migrated community-image fixture
  • Branch Checks (12 of 12 jobs passed)
  • Branch E2E Checks (52 passed, 6 intentionally skipped, 0 failed; Docker, Podman, Kubernetes, VM, and tmachine)
  • Focused regressions for portable fragment validation, aggregate merge limits, provider-import atomicity, durable legacy rewrites, and JavaScript prototype-shaped keys

mise run test and mise run ci reach the unchanged mainline test:packaging-assets failure: deploy/deb/control.in starts with SPDX comments that dpkg-deb rejects. The branch does not modify that file or the packaging test. All task-relevant gates above pass.

Checklist

  • Tests cover public/internal conversion, validation, SDK, and durable-storage boundaries.
  • User-facing and architecture documentation explain the new contract and breaking YAML shapes.
  • Related skills and examples are synchronized.
  • The commit uses Conventional Commits, includes the breaking-change footer, and is DCO signed off.

@pimlock
feat(policy): prototype public policy protobuf boundary
1496ad8 
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 16, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@pimlock pimlock mentioned this pull request Sep 16, 2026
Open
32 tasks
@pimlock
feat(policy)!: require protobuf-shaped policy YAML
5d52cf5 
BREAKING CHANGE: Policy YAML must use the openshell.policy.v1 field shape; legacy scalar matcher and tool shorthands are no longer accepted.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@pimlock pimlock changed the title feat(policy): prototype public policy protobuf boundary feat(policy)!: require protobuf-shaped policy YAML Sep 18, 2026
@pimlock
chore: merge main into policy proto branch
19330eb 
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@github-actions

github-actions Bot commented Sep 18, 2026

Copy link
Copy Markdown

🌿 Preview your docs: https://nvidia-preview-pr-3401.docs.buildwithfern.com/openshell

@pimlock
fix(policy): enforce parsing and size boundaries
5ae1592 
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@pimlock
Merge remote-tracking branch 'origin/main' into 3388-public-policy-pr…
3a058c5 
…oto/pimlock

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@pimlock pimlock added the test:e2e Requires end-to-end coverage label Sep 18, 2026
@pimlock

pimlock commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 3a058c5

@github-actions

github-actions Bot commented Sep 18, 2026

Copy link
Copy Markdown

Label test:e2e applied, but pull-request/3401 does not exist yet. A maintainer needs to comment /ok to test 3a058c5184e4291f27e3c0d1c09c272e17f1e1e1 to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

@pimlock
chore: merge main into policy proto branch
f9c76ac 
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@pimlock
feat(policy)!: finalize public policy protobuf contract
aca0575 
BREAKING CHANGE: Policy YAML and public policy APIs now use the protobuf-shaped openshell.policy.v1.PolicyDocument contract.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@pimlock

pimlock commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test aca0575

@pimlock pimlock added test:e2e Requires end-to-end coverage and removed test:e2e Requires end-to-end coverage labels Sep 19, 2026
@github-actions

github-actions Bot commented Sep 19, 2026

Copy link
Copy Markdown

Label test:e2e applied for aca0575. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@pimlock
fix(build): provide protoc in Nix development shell
6b6d4fb 
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@pimlock

pimlock commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 6b6d4fb

@pimlock
test(e2e): migrate community policy fixtures
3b79984 
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@pimlock

pimlock commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 3b79984

@pimlock
test(e2e): retain sandbox filesystem baseline
b838549 
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@pimlock

pimlock commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test b838549

1 similar comment
@pimlock

pimlock commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test b838549

@pimlock
pimlock marked this pull request as ready for review September 19, 2026 08:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mrunalp mrunalp Awaiting requested review from mrunalp mrunalp is a code owner
@sjenning sjenning Awaiting requested review from sjenning sjenning is a code owner
@derekwaynecarr derekwaynecarr Awaiting requested review from derekwaynecarr derekwaynecarr is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat(policy)!: add a public policy protobuf and keep runtime policy internal

1 participant

@pimlock