feat(mxc): add Windows host proxy for MXC sandbox network egress - #3163
Open
araza008 wants to merge 31 commits into
Open
feat(mxc): add Windows host proxy for MXC sandbox network egress#3163araza008 wants to merge 31 commits into
araza008 wants to merge 31 commits into
Conversation
Add a Windows MXC ETW->OCSF audit trail in openshell-driver-mxc: a real-time Sandboxing-provider ETW consumer that decodes events (TDH), attributes each to an OpenShell sandbox_id, and maps them to OCSF (lifecycle 6002, config 5019, process 1007, finding 2004). cp6 Phase 1 - durable OCSF JSONL audit-file parity with Linux: - openshell-ocsf: add emit_ocsf_event_routed (populates the event-bridge thread-local AND stamps sandbox_id+message in one dispatch) plus public set/clear_current_event; OS-aware device (Device::windows/for_current_os) so device.os.name reflects the host instead of a hardcoded Linux stub. - etw_consumer: emit via the routed emit (previously fired a bare info! that never populated the bridge, so the structured event was dropped). - openshell-server: install OcsfJsonlLayer over a synchronous daily-rotated appender (durable under force-kill), gated by OPENSHELL_OCSF_JSON, path via %PROGRAMDATA%\OpenShell\logs (override OPENSHELL_OCSF_LOG_DIR). - device.hostname now resolves to the real gateway machine name. Box-proven on 7F203-MXC-001: JSONL lines == shorthand OCSF rows, all valid OCSF JSON, per-sandbox attribution intact, disabled state writes nothing. Signed-off-by: Akber Raza <akberr@nvidia.com>
Close the last three ETW->OCSF gaps so the audit trail covers the full set of events the Sandboxing provider emits (12/12): - ProcessLaunched -> Process Activity [1007] "Launch" (confirmed start; carries the real processId/threadId, the twin of CreateProcessInSandbox which only has the request + command line). - SandboxProxyConfigured -> Device Config State Change [5019] (the one network-plane setup event; surfaces proxyPort, "no proxy" when 0). - SandboxConsoleReferencePlumbed -> Device Config State Change [5019] (console-handle plumbing). map_config_state now handles the full config/hardening/setup family and carries proxyPort/hasConsoleReference/creationFlags as unmapped fields. Verified on 7F203-MXC-001: 11/12 event types emit OCSF without a proxy (SandboxProxyConfigured requires proxy config to fire). Signed-off-by: Akber Raza <akberr@nvidia.com>
Address CodeRabbit review on !31: - Prevent stale ETW attribution on a delete/launch race: register the wxc-exec pid while holding the registry lock, and bail if the sandbox entry is already gone. Previously the attribution key could be seeded after `delete` had removed the sandbox, leaving a stale key that could misroute later Sandboxing ETW events to a dead sandbox_id. Lock order (registry -> attribution) matches the delete path, so no deadlock. - Add unit tests for the new Device::windows and Device::for_current_os constructors to harden Windows/Linux OCSF device parity. Signed-off-by: Akber Raza <akberr@nvidia.com>
Addresses two ETW->OCSF attribution review items (Shailendra NVIDIA#1, NVIDIA#2). NVIDIA#2 early-event loss: ETW delivers the sandbox create/config burst the instant wxc-exec starts, which can beat the driver's register_launch (now under the registry lock post-Ready). process_event previously dropped anything unresolved, losing the racing burst. Add a bounded, time-bounded pending buffer (PENDING_MAX=4096, PENDING_TTL=5s): unresolved events are held and replayed once attribution lands, aged-out ones dropped. Consumer switched to a timed recv_timeout(200ms) so the buffer is re-driven after each event and on a tick. Emit path factored into shared emit_resolved(). NVIDIA#1 attribution collisions: a Windows PID is recycled after exit and a command line is commonly identical across sandboxes. register_launch now rebinds by_pid on reuse and clears the stale last_pid_sid hint (warns if the PID still pointed at a different, leaked sandbox); command line is held in by_cmd only while unique and demoted to a new ambiguous_cmds set on a second owner, so a duplicate command refuses to resolve rather than misroute. Unit tests: buffer replay (direct + cross-link), buffer bound, PID-reuse rebind, duplicate-cmd non-resolution. Box-verified on 7F203-MXC-001 (5 sandboxes, identical cmd -> 5 isolated sandbox_ids, 50/50 OCSF/JSONL, BuffersLost=0). Signed-off-by: Akber Raza <akberr@nvidia.com>
Review item NVIDIA#3 (Shailendra): add a PRIVACY NOTE on map_process_launch stating cmd_line is copied verbatim into OCSF process.cmd_line with no redaction, so secrets/PII on a command line land unredacted in the durable audit trail (deliberate audit-fidelity trade-off; treat the log as sensitive). Redaction is owned by an upstream privacy layer, not this path; no general audit-output PII scrubber exists today (openshell_core::secrets [CREDENTIAL] redaction is scoped to the proxy HTTP-target logging, a separate egress path). Signed-off-by: Akber Raza <akberr@nvidia.com>
…s real status Review item NVIDIA#4 (Shailendra): start_session previously returned Ok(EtwSession) as soon as the pump thread was spawned, but OpenTraceW ran later inside that thread; if it failed we still handed back a live-looking session and logged 'consumer started' (silent failure = false audit coverage). Split the two Win32 calls instead of adding a channel handshake (avoids any lost-wakeup/hang risk): the quick, synchronous OpenTraceW now runs on the caller thread (open_trace), and only the blocking ProcessTrace runs on the pump thread (run_trace). start_session returns Err if OpenTraceW fails (reclaiming the boxed Sender so the consumer disconnects, stopping the session, joining the consumer) and returns Ok/logs 'started' only once capture is genuinely open. Opened handle + LoggerName buffer + boxed Sender are carried to the pump via a Send OpenedTrace so they outlive ProcessTrace. Box-verified on 7F203-MXC-001: consumer started=True, failed-to-start=False, 50 OCSF rows / 50 JSONL, BuffersLost=0 (no regression to capture/emit). Signed-off-by: Akber Raza <akberr@nvidia.com>
CodeRabbit flagged that drain_resolved() re-resolved buffered events against the live by_pid map, so if Windows recycled a wxc-exec PID within PENDING_TTL a stale event from the dead sandbox could be emitted under the new owner. Stamp each by_pid registration with its Instant and add resolve_replay(), used only on the buffered/replay path. It (a) never falls back to the recycle-/ambiguity-prone by_cmd or last_pid_sid keys, and (b) trusts a PID match only when the registration is not newer than the buffered event by more than REPLAY_PID_GRACE (2s) - a recycled PID's registration lands well outside that window, so the stale event ages out instead of misattributing. The legitimate NVIDIA#2 seed race (registration lands ~immediately) still replays. Adds unit tests for the recycle-refusal, in-grace acceptance, and weak-fallback exclusion. Signed-off-by: Akber Raza <akberr@nvidia.com>
…DIA#4) start_session already returns Err on OpenTraceW failure (runs on the caller thread since e41a770), closing the first half of Shailendra's NVIDIA#4. This closes the second half: ProcessTrace's result was discarded, so if capture died mid-run the backend had no way to know. Add a shared CaptureHealth (stopped/stopping/exit_code) between the pump thread and EtwSession. run_trace now records ProcessTrace's WIN32_ERROR and, when the pump returns without a deliberate stop, logs at ERROR that MXC OCSF capture is no longer running. EtwSession::stop() sets `stopping` before teardown so a normal shutdown isn't misreported, and EtwSession::is_capture_alive() exposes the state for status/diagnostics. Box-verified on 7F203-MXC-001: 5 sandboxes, 50 attributed OCSF rows, JSONL parity 50/50, BuffersLost=0, clean start/stop (no false failure). Signed-off-by: Akber Raza <akberr@nvidia.com>
…figured message Add a runnable OCSF audit-trail example under examples/ (run-ocsf-audit.ps1, mxc-ocsf-audit.toml, ocsf-audit.yaml, README) that spins up sandboxes with the in-process ETW consumer and egress proxy on, emitting a full OCSF JSONL audit trail across all four classes (6002/5019/1007/2004). Fix SandboxProxyConfigured mapping to log "MXC sandbox proxy configured" instead of a misleading "(no proxy)" when the provider reports proxyPort=0; the event's presence already indicates proxy configuration. Verified on-box: 26 events, all mapped ETW event types present. Signed-off-by: Akber Raza <akberr@nvidia.com>
Improve the ETW to OCSF audit-trail example output and make it safe to ship.
Report:
- Add an event-type coverage count ("N of M expected event types fired");
the denominator auto-adjusts (8 with proxy on, 7 with -NoProxy).
- Split the checklist into expected event types vs anomaly findings
(ActivityError/FallbackError), which are reported separately and not
counted toward coverage (a clean run may emit none).
- Verdict is now coverage-based (all expected types must fire) instead of
the looser "at least 3 OCSF classes".
- Call out the absolute path to the durable OCSF JSONL log prominently.
Client-safety:
- Default -ShareOut to empty (no auto-copy); pass -ShareOut a UNC path to
opt in. Removes a hardcoded internal share path from a published example.
- Drop internal-team wording ("Hand that zip back for evaluation", "BUNDLE:")
in favor of neutral "Results bundle:".
- Update README-ocsf-audit.txt to match the opt-in -ShareOut behavior.
Verified on both MXC boxes: 7F203-MXC-001 (base-container) -> PASS, 8 of 8
event types, 26 OCSF events across 4 classes; 7F203-MXC-003 (AppContainer
fallback) -> reduced set as expected, clean output.
Signed-off-by: Akber Raza <akberr@nvidia.com>
…ling and allocation
… tests for binary path and SHA256 hash
… error handling for hosts file reading
…ng metadata handling
…est exclusion list
Generate per-sandbox TLS state for the MXC host proxy so HTTPS L7 enforcement can use the same MITM path as Linux. Grant generated CA material to the MXC process and inject standard trust env vars, while matching Linux behavior by disabling TLS termination on CA setup failure and relying on proxy fail-closed handling.
The MXC e2e harness never actually exercised the fs scenarios: it started the gateway once and patched agent_command per scenario AFTERWARDS, so the running gateway kept launching the default demo agent (not shipped in the kit) and every fs scenario failed with CreateProcessW error:2. It also scored on the `sandbox create` exit code (non-zero due to the harmless interactive attach), wrote sandbox records to the persistent gateway DB (leaving orphans that collided on later runs), and its deny scenarios never proved denial. Changes: - Start a FRESH gateway per scenario so each scenario's agent_command is actually loaded (root cause of CreateProcessW error:2). - Score by on-disk artifact / expected outcome, not `sandbox create` exit. - Real deny assertions: a control write to a granted path must succeed (proves the agent ran) while the denied write must be absent. fs-empty probes an ungranted out-of-share path (share_dir is mapped rw by design). - Run the gateway on an ephemeral in-memory DB (sqlite::memory:) so the harness never writes to the persistent store and cannot leave orphan sandbox records; also use unique per-run sandbox names + pre-delete. - Fix the process_container probe: use a real cwd + absolute cmd.exe (canonical wxc-exec does not expand %TEMP% -> 0x8007010B). - Fix summary counts (@() so a single FAIL is counted and exit is non-zero). Verified PASS=4 FAIL=0 on 7F203-MXC-003 (no BaseContainer velocity keys) using a canonical wxc-exec build (AppContainer fallback). Signed-off-by: Akber Raza <akberr@nvidia.com>
MXC process.timeout is wall-clock ms (wire.rs). The 10 value meant 10ms, which the base-container tier (7F203-MXC-001/.181) enforced strictly and timed the probe out. AppContainer path (.18/-003) happened to slip under it. Bump to 30000ms so the process_container preflight probe is reliable across both tiers. Signed-off-by: Akber Raza <akberr@nvidia.com>
Signed-off-by: Shailendra Singh <shailendras@nvidia.com> Signed-off-by: Akber Raza <akberr@nvidia.com>
Three fixes to support the release wxc-exec binary (BaseContainer dispatcher) in addition to mxc-fixes-env-vars: 1. Seed process env from host (driver.rs) ProcessContainer starts with a completely blank environment -- no PATH, SystemRoot, or anything. Seed the process env from the gateway host environment so the agent binary can locate DLLs and run. Skip internal Windows drive-letter variables (keys starting with '=') which cause CreateProcessW to return ERROR_ENVVAR_NOT_FOUND. User agent_env entries and TLS CA vars are applied as overrides on top of the host env. 2. Remove TLS readonly_paths grant (driver.rs) The release wxc-exec (BaseContainer dispatcher) requires write-DAC permission on every path in readonly_paths to set up AppContainer ACLs. Adding the proxy's temp TLS directory caused a DACL error and exit -1. The CA cert paths remain available to the agent via TLS env vars. 3. Remove allowedHosts from network JSON (mxc.rs) The release wxc-exec rejects network.allowedHosts / network.blockedHosts on Windows with "not yet supported". Removed the loopback exemption attempt (127.0.0.1, ::1, localhost) from the network section. Intra-container loopback works natively in the release binary without it -- the spawner can connect to the server at 127.0.0.1:22000 directly. Additional changes: - mxc-ws-agent.rs: add relay-debug.txt error capture and relay-ready.txt marker for reliable timing of host client connections. - mxc-ws-gateway.toml: debug = true for JSON config dump during diagnosis. - run-ws-agent-test.ps1: default port changed to 17670 (gateway default); relay-ready.txt polling before ws-echo to avoid connecting before the spawner has established the proxy bridge. Signed-off-by: Prashant Khodade <pkhodade@nvidia.com> Signed-off-by: Akber Raza <akberr@nvidia.com>
Four robustness/correctness fixes from CodeRabbit: 1. Start-Gw: kill the spawned gateway before the "did not start within 30s" throw. If the process is alive but never binds the port, $gw is not yet assigned in the caller, so the finally block cannot reap it -> orphan gateway holding the port for the next run. 2. create-fail scoring: a non-zero `sandbox create` exit alone is not proof of a policy rejection (gateway-registration/transport/fixture errors also exit non-zero and would false-pass). PASS now requires a genuine rejection signal (network / invalid_argument / network_policies) AND that it is not an infrastructure failure; other non-zero exits go to FAIL with output captured. 3. deny scenarios (ControlTarget path): snapshot the deny target AFTER Wait-File lands the control artifact, so a late denied write (enforcement regression racing the control write) can no longer be recorded as PASS. 4. -KeepRunning: break out of the scenario loop after the first scenario so a later scenario does not start a second gateway on the same port (previously a reliable port collision instead of a usable debug mode). Re-verified PASS=4 FAIL=0 on both boxes (7F203-MXC-001 base-container and 7F203-MXC-003 AppContainer fallback); network-policy-rejected correctly scores as "policy rejection". Signed-off-by: Akber Raza <akberr@nvidia.com>
Mirror the sibling run-*.ps1 scripts by collecting every run's logs into a timestamped results-e2e-<stamp>\ folder and zipping it. The bundle contains the console transcript, per-scenario gateway stdout/stderr, the exact TOML rendered for each scenario, the policy fixture used, and a summary.txt with the verdict table. Per-scenario gateway logs now land in gateway.<scenario>.log/.err.log inside the bundle instead of a single fixed gateway.e2e.log in the script directory. Wrap pre-flight, mode setup, scenario definitions, and the scenario loop in a single try/catch/finally so the finally always writes the summary, stops the transcript, and zips the bundle -- even on a pre-flight failure. The existing per-scenario gateway-cleanup try/finally stays nested inside. All scenario logic, scoring rules, and comments are preserved. Signed-off-by: Prashant S Khodade <pkhodade@nvidia.com> Signed-off-by: Akber Raza <akberr@nvidia.com>
- Require -Scenario when -KeepRunning: the loop breaks after the first scenario, so a full-suite run would execute only one scenario yet still report the suite as PASS. Fail fast so a partial run can't be mislabeled complete. - Start-Transcript now runs inside the guarded try block with a $transcriptStarted flag; Stop-Transcript is only called when it actually started, so a Start-Transcript failure still yields the results bundle. - Wrap the -Scenario filter in @() so a single exact match stays an array (reliable .Count and a proper array for the scenario loop on PS 5.1). Signed-off-by: Prashant S Khodade <pkhodade@nvidia.com> Signed-off-by: Akber Raza <akberr@nvidia.com>
…paced paths Start-Process -ArgumentList does not quote array elements, so launching the gateway with a bare --config <path> token split on any space in the install path (e.g. C:\Users\First Last\...), and clap rejected the fragment with 'unrecognized subcommand'. Every MXC example launcher that started the gateway hit this when the kit was unzipped under a path containing a space. Pass the config path through the OPENSHELL_GATEWAY_CONFIG env var (which the gateway already reads via clap) and drop the --config token. Env vars carry spaces safely. Affected: run-ocsf-audit, run-mxc-e2e, run-demo, run-inference-test, run-ollama-test. run-mtls-test was not affected (its launch passes no config path). Root-caused and fix-verified on 7F203-MXC-003 from a spaced path. Signed-off-by: Akber Raza <akberr@nvidia.com>
…forever ComposedPhase::new (introduced in 662dee6) determines SandboxPhase::Ready by requiring a live supervisor session (session_connected == true). For backends that have no in-sandbox supervisor (e.g. MXC), session_connected is always false, so the public phase was permanently stuck at Provisioning even after the driver reported Ready=True. The CLI watch loop in sandbox_create blocks until it observes a Provisioning -> Ready transition, so it would spin until the 300-second idle timeout fired -- appearing as a hang to the user. Fix: thread supports_interactive_session (already stored on ComputeRuntime as has_supervisor) through apply_driver_snapshot into ComposedPhase::new. When has_supervisor is false the backend phase passes through directly as the authoritative readiness signal, matching the pre-662dee68 behaviour for MXC. Also guard backend_ready_without_session with has_supervisor so supervisorless backends do not emit the misleading SupervisorNotConnected status condition. Regression introduced by: 662dee6 refactor(compute): make sandbox readiness gateway-owned across all drivers (NVIDIA#2153) Signed-off-by: Prashant S Khodade <pkhodade@nvidia.com> Signed-off-by: Akber Raza <akberr@nvidia.com>
araza008
requested review from
a team,
derekwaynecarr,
mrunalp and
sjenning
as code owners
September 3, 2026 16:14
Contributor
Author
|
I have read the DCO document and I hereby sign the DCO. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Windows MXC host-proxy support for governed sandbox egress, including TLS state generation and L7 enforcement. It also adds ETW-to-OCSF auditing, improves ProcessContainer compatibility, and hardens the Windows MXC E2E harness.
Related Issue
No issue linked yet.
Changes
wxc-execimplementations.Provisioning.run-mxc-e2e.ps1command invocation, cleanup, scoring, and result collection.Testing
mise run --skip-tools windows:cipassedgit diff --check origin/main...HEADpasses.Checklist