feat(ocsf): carry structured events on sandbox log stream

[krishicks]

Summary

Sandbox log streams now retain complete OCSF event payloads, allowing exporters and API consumers to use the original structured security and audit records without reconstructing them from display text.

openshell logs continues to show readable shorthand, including across supported mixed-version deployments. Ordinary log lines and malformed or older structured payloads still display their original message instead of being dropped.

Related Issue

#1055

Changes

Testing

• mise run pre-commit passes
• Unit tests added/updated
• E2E tests added/updated (if applicable)

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[johntmyers]

Review findings

• [P1] Preserve the fallback message for mixed-version gateways. For OCSF events, LogPushLayer now sets message to an empty string and carries the content only in the new ocsf_json protobuf field (log_push.rs:54-91). A pre-feat(ocsf): carry structured events on sandbox log stream #3104 gateway ignores unknown field 8 and publishes the empty fallback, so OCSF entries appear blank when a newer supervisor reconnects to an older gateway during a rollout or rollback. Supervisor images can be pinned independently, so this mixed-version pairing is supported and realistic. Please keep message populated with the shorthand representation and treat ocsf_json as additive metadata, or negotiate support before clearing the fallback.

• [P2] Expose the structured payload through the curated Go SDK. The protobuf gains ocsf_json, but types.LogLine has no corresponding field (types/log.go:9-21) and LogLineFromProto does not copy GetOcsfJson() (converter/log.go:18-25). Exporters using the curated API therefore still cannot retain the complete event. Please add a deep-copied byte field or parsed OCSF representation and converter coverage.

[krishicks]

@johntmyers

Supervisor images can be pinned independently, so this mixed-version pairing is supported and realistic.

Is this really the case? I know previously the supervisor was always using the latest image, but do we make any guarantees about using a newer supervisor with an older gateway?

[krishicks]

@johntmyers

Expose the structured payload through the curated Go SDK.

I think this should be done as a separate PR. Thoughts?

[johntmyers]

@johntmyers

Supervisor images can be pinned independently, so this mixed-version pairing is supported and realistic.

Is this really the case? I know previously the supervisor was always using the latest image, but do we make any guarantees about using a newer supervisor with an older gateway?

I don't think we explicitly prevent this case. Maybe for pre-0.1.0 this doesn't matter. But I think the review agent was orienting around HA rolling upgrades where the mismatch could happen.

[zanetworker]

The ocsf_json field creates a second consumption path for structured OCSF events at the gateway. In #2717 I add a configurable schema version downgrade that strips fields/profiles for SIEMs targeting older OCSF versions (1.1, 1.3). Currently the downgrade runs at the sandbox JSONL layer. With ocsf_json now traveling to the gateway, should the bytes be pre-downgraded, or should the gateway apply its own downgrade pass? If a future exporter consumes structured events at the gateway, it would need the downgraded version for SIEM compatibility.

[sylvesterkaczmarek]

publish_security_log_event() computes the shorthand first but returns without emitting anything if to_json_line() fails. The receiving side deliberately keeps the text fallback when structured OCSF is bad, so the sender should preserve that contract and send the shorthand with empty ocsf_json on serialization failure instead of dropping the security event entirely. A forced-serialization-error test would pin this.

[krishicks]

@zanetworker My current thinking is that doing the downgrade at the sandbox is not sufficient, or scalable, because we would need to support many potential downgrades, and there can be multiple destinations for OCSF events, including OTLP, a rotated log file, and others. Throwing away information at the sandbox to satisfy a single SIEM schema is suboptimal, I think.

Someone at NVIDIA built a pretty comprehensive solution that correlates information from gateways and sandboxes, including OCSF events but also spans and other logs. This solution makes me think we should keep the gateway and sandbox implementation as simple as possible and rely on other tooling (like that solution) to do transformation of the data for various consumers. The architecture of that solution today looks like this:

flowchart LR
    A[OpenShell files] --> E[Exporter]
    B[OpenShell read-only gRPC] --> E
    C[Gateway and Relay OTLP] --> E
    D[Kubernetes context] --> E
    E --> P[Identify, validate, redact, correlate]
    P --> Q1[Persistent CloudEvents queue]
    P --> Q2[Persistent OTLP queues]
    P --> R[Redacted recovery archive]
    Q1 --> S[SIEM, security, data lake, policy system]
    Q2 --> O[Observability backend]

Loading

I spiked having different kinds of exporters built into the gateway and I feel it expands the scope of the gateway too much, particularly given how critical OCSF events are. Instead, I think we should keep it simple and have the gateway create a durable record of OCSF events using the latest supported schema version and use something like the above tooling to read that stream, do any other relevant correlation, as well as necessary PII redaction, and forward it on to external consumers. That means all the queuing and retrying behavior that we'd want would exist in the other tool, not within the gateway.

What do you think?