feat(policy)!: require protobuf-shaped policy YAML

.gitignore

@@ -107,6 +107,8 @@ cython_debug/
 # Generated Python protobuf stubs (keep package marker)
 python/openshell/_proto/*
 !python/openshell/_proto/__init__.py
+python/buf/validate/*
+!python/buf/validate/__init__.py
 
 # =============================================================================
 # IDE / Editor

Cargo.toml

@@ -24,6 +24,7 @@ tonic-prost-build = "0.14"
 prost = "0.14"
 prost-types = "0.14"
 prost-reflect = { version = "0.16.5", features = ["serde"] }
+prost-protovalidate = { version = "=0.6.0", default-features = false, features = ["reflect"] }
 
 # HTTP server
 axum = { version = "0.8", features = ["ws"] }

architecture/gateway.md

@@ -397,6 +397,16 @@ The storage schema is intentionally narrow:
 
 Public RPC contracts and durable protobuf formats have separate ownership. The `openshell.v1.OpenShell` service's request and response roots, streaming flags, and transitive message closure come from the public descriptor set generated by `openshell-core`. The `public_and_durable_schema_inventories_are_complete` test in `openshell-server` owns the counts and fingerprints and requires this inventory to be reviewed whenever it changes. Compute-driver, credential-driver, gateway-interceptor, and supervisor-middleware services are compiled contracts for internal extension boundaries.
 
+Authored policy is an explicit conversion boundary. Public RPCs use
+`openshell.policy.v1.PolicyDocument`; persistence, composition, and supervisor
+configuration use `openshell.sandbox.v1.SandboxPolicy`. The gateway validates
+and lowers a `PolicyDocument` before storing the internal message, and projects
+the internal message for authored API responses. The shared
+`GetSandboxConfig` supervisor RPC is the exception: its wire response carries
+the effective internal policy required for enforcement, and curated SDKs
+project that field to `PolicyDocument` before exposing it. Existing policy rows
+therefore keep their wire format and require no data migration.
+
 `ReportEndpointStatus` is a sandbox-authenticated public gateway RPC. Its request, response, and `EndpointObservation` messages belong only to the public closure. `EndpointStatus` and `EndpointResult` also belong to the durable closure because `Sandbox.status.endpoint_statuses` persists them. The repeated status field uses a new wire tag; stored sandboxes without it decode with an empty endpoint list and retain their lifecycle fields. A fixed payload encoded with the earlier sandbox schema verifies that no database rewrite is required.
 
 Allow and deny append requests carry `L7RuleTarget` to declare the rule, endpoint, and complete affected scope. The removed `host` and `port` fields remain reserved by number and name, and requests without a target are rejected. These mutation requests are not persisted formats.
@@ -464,11 +474,12 @@ Stored settings JSON also carries per-key change IDs and commit timestamps,
 including deletion tombstones. Legacy values acquire stable source identities
 on read; a subsequent write preserves them. These clocks distinguish effective
 edits from no-op writes without treating status updates as configuration edits.
-With timestamp types, deletion outcomes, and optional mutation request IDs, the
-admission contract brings the public closure to 298 messages and 21 enums, the
-durable closure to 92 messages and 16 enums, and their overlap to 80 messages
-and 16 enums. Mutation request IDs extend public request fields without adding
-messages to these closures or changing the durable protobuf schema.
+With the authored policy contract, timestamp types, deletion outcomes, and
+optional mutation request IDs, the public closure contains 327 messages and 24
+enums, the durable closure contains 92 messages and 19 enums, and their overlap
+contains 75 messages and 19 enums. Mutation request IDs extend public request
+fields without adding messages to these closures or changing the durable
+protobuf schema.
 
 | Dual-purpose encoded root | Current decision |
 |---|---|
@@ -479,20 +490,21 @@ messages to these closures or changing the durable protobuf schema.
 | `WorkspaceMember` | Defer a storage twin; govern its complete dependency closure as durable. |
 | `SshSession` | Defer a storage twin; govern its complete dependency closure as durable. |
 | `ServiceEndpoint` | Defer a storage twin; govern its complete dependency closure as durable. |
-| `SandboxPolicy` | Defer a storage twin; govern its complete dependency closure as durable. |
 | `ConfigUpdateOperation` | Persist the common historical outcome within `StoredConfigUpdateOperation`; govern its complete dependency closure as durable. |
 | `ProviderMutationReceipt` | Persist the immutable provider projection within `StoredConfigUpdateOperation`; govern its complete dependency closure as durable. |
 | `ProviderReadinessReason` | Persist only the closed snapshot failure category within the operation; govern its enum values as durable. |
 
-The public/storage overlap is deliberate for the current format. Storage twins
-for the public roots are deferred: introducing them would require a broad
+The public/storage overlap is deliberate for the remaining roots. Storage twins
+for those public roots are deferred: introducing them would require a broad
 conversion boundary, and Prost does not retain unknown fields through a
-decode-and-reencode conversion. Each root therefore carries a reviewed decision
-to remain dual-purpose, and its complete transitive dependency closure is also
-a durable format. Important embedded dependencies include `ObjectMeta`,
-`ProviderProfile`, `CredentialHandle`, `SandboxPolicy`, and
-`NetworkPolicyRule`. Global and sandbox settings additionally store an encoded
-`SandboxPolicy` inside their JSON envelope.
+decode-and-reencode conversion. Each root in the table therefore carries a
+reviewed decision to remain dual-purpose, and its complete transitive dependency
+closure is also a durable format. Important shared dependencies include
+`ObjectMeta`, `ProviderProfile`, and `CredentialHandle`. Policy is the exception:
+`PolicyDocument` is public-only, while the separately owned internal
+`SandboxPolicy` and `NetworkPolicyRule` remain durable. Global and sandbox
+settings additionally store an encoded internal `SandboxPolicy` inside their
+JSON envelope.
 
 Public API compatibility and storage compatibility are reviewed independently:
 

architecture/security-policy.md

@@ -41,6 +41,55 @@ before any consumer-specific projection runs. There is no permissive parsing
 profile: unsupported policy fields always invalidate the document. Middleware `config`, query and persisted-query names, and recursive MCP
 parameter names are open user-data maps rather than schema extensions.
 
+The generated `openshell.policy.v1` package is the language-neutral authored
+contract used by policy inputs and curated SDK policy surfaces. The gateway
+lowers that message at ingress into the internal `openshell.sandbox.v1` runtime
+policy and projects stored or effective runtime policy back to the public
+message at authored API boundaries. Compute drivers, the supervisor, policy
+history payloads, and merge execution continue to use the internal
+representation. `GetSandboxConfig` is also the authenticated supervisor's
+configuration RPC, so its wire response necessarily carries the effective
+internal policy; curated SDKs project that field to `PolicyDocument` before
+returning it to callers. Runtime-derived fields such as advisor and provider
+provenance have no public authored field and their internal wire numbers are
+reserved in the authored endpoint message.
+
+Existing policy revision rows, sandbox records, and provider-profile records
+remain internal and retain their current wire encoding. The gateway uses
+private storage envelopes whose field numbers mirror the historical records,
+then projects their policy fields to the public schema on read. No clean
+database or policy-data migration is required for this boundary. Mixed-version
+gateway rollouts remain unsupported because the public RPC contract changes.
+
+Policy YAML uses the public protobuf field shape directly. The schema crate
+performs bounded YAML decoding and descriptor-driven protobuf conversion; it
+does not rewrite legacy spellings. Scalar matchers, scalar MCP `tool` values,
+and other YAML-only shorthands are breaking changes and must be rewritten as
+their message or oneof forms. Protobuf cannot distinguish omitted repeated
+fields from empty lists, so both select the same default where the policy
+language defines one. Contextual validation remains explicit code because
+rules such as endpoint protocol, credential binding, and provider composition
+depend on more than one message; generated field validation is not an
+enforcement substitute.
+
+The portable field constraints use the vendored
+`proto/buf/validate/validate.proto` schema. Go, Python, and TypeScript execute
+those annotations with the official Protovalidate runtimes. Rust uses the
+exact-pinned `prost-protovalidate` reflection runtime because Buf does not
+publish an official Rust implementation; the dependency remains isolated in
+`openshell-policy-schema`, and the policy rules deliberately avoid CEL. Tests
+assert the same rule IDs in all four languages so runtime drift is visible.
+Cross-field policy semantics still run after portable field validation.
+
+Typed YAML fields reject explicit `null`; omission expresses absence. Null is
+accepted only as user data below middleware `config`, whose protobuf type is
+`google.protobuf.Struct`. Public endpoints expose only the non-empty `ports`
+list. Field number 2 and the name `port` stay reserved in the public message so
+the legacy scalar cannot reappear accidentally. The internal runtime policy
+keeps its scalar field for stored-data and rollout compatibility; lowering
+promotes a one-item public list into that scalar, and public projection always
+emits the effective repeated list.
+
 Before applying Landlock, the supervisor enriches baseline filesystem paths that
 the runtime needs. Missing baseline paths are skipped so one absent runtime path
 does not weaken the whole ruleset. When GPU devices are present, GPU baseline

buf.yaml

@@ -13,6 +13,9 @@ modules:
 lint:
   use:
     - STANDARD
+  # This file is a vendored upstream dependency, not an OpenShell API.
+  ignore:
+    - proto/buf/validate/validate.proto
   except:
     # Flat proto/ layout: all files live in one directory with nested
     # packages (openshell.v1, openshell.sandbox.v1, ...). Adopting these
@@ -30,3 +33,5 @@ lint:
 breaking:
   use:
     - FILE
+  ignore:
+    - proto/buf/validate/validate.proto

crates/openshell-cli/src/main.rs

@@ -1968,7 +1968,10 @@ enum PolicyCommands {
         #[arg(long = "rule-name")]
         rule_name: Option<String>,
 
-        /// Explicitly declare that the target rule for L7 appends allows any binary.
+        /// Select the incremental update API's internal any-binary target state.
+        ///
+        /// This is not an authored policy-file state; omitted or empty public
+        /// binary lists match no executable.
         #[arg(long, conflicts_with = "binaries")]
         any_binary: bool,