Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 25 additions & 3 deletions architecture/sandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,9 +173,31 @@ The sandbox reserves `SIGUSR2` with a non-restarting no-op handler for these
broker threads; startup rejects a conflicting handler. This signal disposition
is process-global kernel state, while registrations and cancellation state are
owned by the broker. Workload exec resets the caught handler to its default.
This sandbox runtime requires Linux 6.2 or newer for Landlock ABI v3 and treats
`SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV` as mandatory so cancelled
notifications cannot race task-memory writes.
This sandbox runtime requires Landlock ABI v3 (Linux 6.2, or an equivalent
vendor backport). The seccomp listener is installed in one of two cancellation
modes, and the launch confirmation enforces the invariant
`cancellation || task_memory_writes_disabled`:

- **Killable** (`SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV`, Linux 5.19+): the
notified workload thread waits kill-only, so a non-fatal signal cannot resume
a mediated syscall between notification validation and the broker's result
write. Full mediation, including task-memory output writes.
- **LegacyReadOnly** (kernels < 5.19, e.g. RHEL 9.x / 5.14): the flag is
unavailable (`EINVAL`), so the listener falls back to a plain notifier and the
broker refuses every task-memory *output* write to stay cancellation-safe.
Concretely, in this mode `getpeername`, `accept`/`accept4` **with a non-null
peer-address argument**, and `sendmmsg` paths that write per-message lengths
fail closed with `EOPNOTSUPP`. `accept` with a null address, and socket
creation, `connect`, `bind`, `listen`, `sendto`, and `sendmsg` continue to
work — they use copied inputs, scalar responses, or atomic `ADDFD_SEND`, none
of which write into workload memory. Some server workloads whose accept
wrappers request the peer address will therefore not run until the kernel
provides `WAIT_KILLABLE_RECV` (a distribution backport); outbound-oriented
workloads are unaffected.

Input mediation, DNS/TCP authorization, and outer-fence enforcement are
identical in both modes. The selected mode is emitted in the sandbox
qualification output (`seccomp_listener_mode`).

DNS uses an exact sandbox-local resolver at `127.0.0.53:53`. The driver sets the
nameserver and permits an unprivileged bind to port 53. UDP and TCP DNS requests
Expand Down
14 changes: 13 additions & 1 deletion crates/openshell-isolation-interface/src/contract.rs
Original file line number Diff line number Diff line change
Expand Up @@ -423,6 +423,10 @@ pub struct SeccompEvidence {
pub task_memory_read: bool,
pub task_memory_write: bool,
pub cancellation: bool,
/// The broker disables every task-memory *output* write (legacy plain
/// listener on kernels < 5.19). Paired with `cancellation` in the
/// launch-safety invariant: one of the two must hold.
pub task_memory_writes_disabled: bool,
}

/// Driver-owned evidence that the mandatory outer network fence is installed.
Expand Down Expand Up @@ -562,7 +566,15 @@ impl SandboxConfirmEvidence {
&& self.seccomp.proc_fd_identity
&& self.seccomp.task_memory_read
&& self.seccomp.task_memory_write
&& self.seccomp.cancellation
// Cancellation-safety invariant: `cancellation || writes_disabled`.
// Either the listener keeps the notified workload thread kill-only
// (WAIT_KILLABLE_RECV / cancellation, Linux 5.19+), or the broker
// disables every task-memory output write (legacy plain listener on
// kernels < 5.19). One of the two MUST hold so a signal-resumed
// syscall can never race a privileged write into workload memory.
// We do not remove cancellation unconditionally; we require the
// legacy read-only guarantee in its place.
&& (self.seccomp.cancellation || self.seccomp.task_memory_writes_disabled)
&& self.landlock_abi >= 3
&& self.landlock_allow_deny
&& self.udp_dns_round_trip
Expand Down
156 changes: 147 additions & 9 deletions crates/openshell-isolation-interface/src/linux/seccomp_notify.rs
Original file line number Diff line number Diff line change
Expand Up @@ -177,13 +177,52 @@ impl NotificationProbeReport {
}
}

/// Cancellation posture a listener was installed with.
///
/// `WAIT_KILLABLE_RECV` (Linux 5.19+) keeps the notified workload thread in a
/// kill-only wait so a non-fatal signal cannot resume the mediated syscall
/// after the broker has validated the notification. A plain listener has no
/// such guarantee, so it runs read-only: the broker must refuse every
/// task-memory *output* write to stay cancellation-safe.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ListenerMode {
/// Modern kernel: `WAIT_KILLABLE_RECV` active; full mediation including
/// task-memory output writes.
Killable,
/// Legacy kernel (< 5.19): plain listener; task-memory output writes are
/// disabled so a resumed syscall cannot race a broker write.
LegacyReadOnly,
}

impl ListenerMode {
/// Stable identifier for qualification output and diagnostics.
#[must_use]
pub fn as_str(self) -> &'static str {
match self {
Self::Killable => "killable",
Self::LegacyReadOnly => "legacy_read_only",
}
}
}

/// Owned listener returned by `SECCOMP_FILTER_FLAG_NEW_LISTENER`.
pub struct NotificationListener {
fd: OwnedFd,
wait_killable_recv: bool,
}

impl NotificationListener {
/// Construct a listener from an already-owned notification descriptor in a
/// specific mode. Intended for tests that must exercise the legacy
/// read-only fail-closed paths without a `< 5.19` kernel.
#[must_use]
pub fn from_fd_with_mode(fd: OwnedFd, mode: ListenerMode) -> Self {
Self {
fd,
wait_killable_recv: matches!(mode, ListenerMode::Killable),
}
}

/// Raw listener descriptor for readiness integration and diagnostics.
#[must_use]
pub fn as_raw_fd(&self) -> RawFd {
Expand All @@ -196,6 +235,23 @@ impl NotificationListener {
self.wait_killable_recv
}

/// The cancellation mode this listener was installed with.
#[must_use]
pub fn mode(&self) -> ListenerMode {
if self.wait_killable_recv {
ListenerMode::Killable
} else {
ListenerMode::LegacyReadOnly
}
}

/// Whether broker task-memory output writes are disabled for this listener.
/// True exactly in `LegacyReadOnly` mode (no `WAIT_KILLABLE_RECV`).
#[must_use]
pub fn writes_disabled(&self) -> bool {
matches!(self.mode(), ListenerMode::LegacyReadOnly)
}

/// Receive the next kernel notification.
pub fn receive(&self) -> io::Result<Notification> {
let mut raw = RawNotification::default();
Expand Down Expand Up @@ -223,6 +279,34 @@ impl NotificationListener {
Ok(())
}

/// Write broker-produced output into the notified task's memory, closing
/// the validation-to-write race that a plain listener cannot.
///
/// In `Killable` mode `WAIT_KILLABLE_RECV` keeps the notified workload
/// thread in a kill-only wait, so a non-fatal signal cannot resume the
/// mediated syscall between `validate_id` and this write. In
/// `LegacyReadOnly` mode (kernels < 5.19) there is no such guarantee: a
/// resumed syscall could repurpose the target buffer while the privileged
/// broker writes through the captured tid and pointer — via `/proc/<tid>/mem`
/// even into pages the workload has since made read-only. There is no way to
/// close that window without the flag, so this fails closed (`EOPNOTSUPP`)
/// rather than racing. Callers must route every task-memory *output* write
/// through this method; input reads never write workload memory and are
/// unaffected.
pub fn write_task_output(
&self,
id: u64,
tid: u32,
address: u64,
data: &[u8],
) -> io::Result<()> {
if self.writes_disabled() {
return Err(io::Error::from_raw_os_error(libc::EOPNOTSUPP));
}
self.validate_id(id)?;
crate::linux::task_memory::write_exact(tid, address, data)
}

/// Return a successful scalar result to the notifying syscall.
pub fn respond_value(&self, id: u64, value: i64) -> io::Result<()> {
self.validate_id(id)?;
Expand Down Expand Up @@ -330,16 +414,22 @@ pub fn install_listener(syscalls: &[i64]) -> io::Result<NotificationListener> {
verify_notification_sizes()?;
set_no_new_privileges()?;

install_listener_with_flags(syscalls, true).map_err(|error| {
if error.raw_os_error() == Some(libc::EINVAL) {
io::Error::new(
io::ErrorKind::Unsupported,
"seccomp WAIT_KILLABLE_RECV is required (Linux 5.19 or newer)",
)
} else {
error
// WAIT_KILLABLE_RECV (Linux 5.19+) keeps the *notified workload thread* in
// a kill-only wait while the broker services its syscall, so a non-fatal
// signal cannot resume the syscall and repurpose its buffers underneath a
// pending broker write. Kernels older than 5.19 (for example RHEL 9.x /
// 5.14 nodes) reject the flag with EINVAL. Rather than refusing to start
// there, fall back to a plain listener so the sandbox boots; the resulting
// listener records `wait_killable_recv = false`, and the broker then fails
// closed on every task-memory output write (see `write_task_output`)
// instead of racing them. Input mediation is unaffected.
match install_listener_with_flags(syscalls, true) {
Ok(listener) => Ok(listener),
Err(error) if error.raw_os_error() == Some(libc::EINVAL) => {
Comment thread
johntmyers marked this conversation as resolved.
install_listener_with_flags(syscalls, false)
}
})
Err(error) => Err(error),
}
}

/// Install the capability-free workload networking listener on the calling
Expand Down Expand Up @@ -935,4 +1025,52 @@ mod tests {
.expect_err("zero errno must fail");
assert_eq!(error.kind(), io::ErrorKind::InvalidInput);
}

#[test]
fn plain_listener_fails_closed_on_output_write() {
// A LegacyReadOnly listener (kernels < 5.19) must refuse every
// task-memory output write rather than race a resumed syscall. The
// guard short-circuits before touching the descriptor or workload
// memory, so a dup of stderr is a sufficient stand-in.
// SAFETY: dup takes one valid descriptor and returns a new descriptor
// or a negative error without modifying memory.
let duplicated = unsafe { libc::dup(libc::STDERR_FILENO) };
assert!(duplicated >= 0, "duplicate stderr for validation test");
// SAFETY: successful dup returned a new owned descriptor.
let listener = NotificationListener::from_fd_with_mode(
unsafe { OwnedFd::from_raw_fd(duplicated) },
ListenerMode::LegacyReadOnly,
);
assert!(listener.writes_disabled());
assert_eq!(listener.mode(), ListenerMode::LegacyReadOnly);
let error = listener
.write_task_output(1, 0, 0, &[0_u8; 4])
.expect_err("plain listener must reject output writes");
assert_eq!(error.raw_os_error(), Some(libc::EOPNOTSUPP));
}

#[test]
fn real_plain_listener_disables_output_writes() {
// Deliberately install a plain NEW_LISTENER (no WAIT_KILLABLE_RECV)
// even on a modern CI kernel and prove the broker write path fails
// closed on the real listener object.
set_no_new_privileges().expect("no_new_privs for listener install");
let listener = install_listener_with_flags(&[libc::SYS_getppid], false)
.expect("install plain listener");
assert_eq!(listener.mode(), ListenerMode::LegacyReadOnly);
assert!(listener.writes_disabled());
let error = listener
.write_task_output(1, 0, 0, &[0_u8; 4])
.expect_err("plain listener must reject output writes");
assert_eq!(error.raw_os_error(), Some(libc::EOPNOTSUPP));
}

#[test]
fn killable_listener_enables_output_writes() {
set_no_new_privileges().expect("no_new_privs for listener install");
let listener = install_listener_with_flags(&[libc::SYS_getppid], true)
.expect("install killable listener");
assert_eq!(listener.mode(), ListenerMode::Killable);
assert!(!listener.writes_disabled());
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -395,6 +395,7 @@ fn confirmation_evidence() -> SandboxConfirmEvidence {
task_memory_read: true,
task_memory_write: true,
cancellation: true,
task_memory_writes_disabled: false,
},
landlock_abi: 3,
landlock_allow_deny: true,
Expand Down Expand Up @@ -537,8 +538,43 @@ async fn one_driver_runs_both_backends() {

#[test]
fn confirmation_constructor_rejects_incomplete_evidence() {
let mut evidence = confirmation_evidence();
// A missing containment-critical primitive (the notification listener
// itself) must be rejected. `cancellation` is intentionally NOT in this set:
// it reflects WAIT_KILLABLE_RECV (Linux 5.19+) and is unavailable on older
// kernels, where the sandbox degrades to a plain listener.
evidence.seccomp.new_listener = false;
let result = ConfirmedBoundary::try_new(
Box::new(MockReady::<Primary> { _k: PhantomData }),
evidence,
&workload_identity(),
);
assert!(matches!(result, Err(BackendError::Confirm(_))));
}

#[test]
fn confirmation_accepts_legacy_read_only_listener() {
// Cancellation-safety invariant: cancellation OR writes-disabled. A legacy
// plain listener (< 5.19) has no cancellation but disables broker output
// writes, so it satisfies the invariant and must be accepted.
let mut evidence = confirmation_evidence();
evidence.seccomp.cancellation = false;
evidence.seccomp.task_memory_writes_disabled = true;
let result = ConfirmedBoundary::try_new(
Box::new(MockReady::<Primary> { _k: PhantomData }),
evidence,
&workload_identity(),
);
assert!(result.is_ok());
}

#[test]
fn confirmation_rejects_plain_listener_with_writes_enabled() {
// The unsafe configuration this PR must never admit: no cancellation AND
// task-memory output writes still enabled (the cancellation race).
let mut evidence = confirmation_evidence();
evidence.seccomp.cancellation = false;
evidence.seccomp.task_memory_writes_disabled = false;
let result = ConfirmedBoundary::try_new(
Box::new(MockReady::<Primary> { _k: PhantomData }),
evidence,
Expand Down
1 change: 1 addition & 0 deletions crates/openshell-sandbox-backend/src/runtime.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2580,6 +2580,7 @@ mod tests {
task_memory_read: true,
task_memory_write: true,
cancellation: true,
task_memory_writes_disabled: false,
},
landlock_abi: 3,
landlock_allow_deny: true,
Expand Down
1 change: 1 addition & 0 deletions crates/openshell-sandbox/src/boundary_server.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4277,6 +4277,7 @@ mod linux {
task_memory_read: true,
task_memory_write: true,
cancellation: true,
task_memory_writes_disabled: false,
},
landlock_abi: 6,
landlock_allow_deny: true,
Expand Down
14 changes: 14 additions & 0 deletions crates/openshell-sandbox/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,11 @@ struct QualificationReport {
tcp_allow_round_trip: bool,
tcp_deny_round_trip: bool,
wait_killable_recv: bool,
/// Selected seccomp listener cancellation mode: `killable` (>= 5.19) or
/// `legacy_read_only` (< 5.19, broker output writes disabled).
seccomp_listener_mode: &'static str,
/// Whether the broker disables task-memory output writes (legacy mode).
task_memory_writes_disabled: bool,
}

#[cfg(target_os = "linux")]
Expand Down Expand Up @@ -237,6 +242,12 @@ fn qualify_runtime() -> Result<(openshell_sandbox::RuntimeQualification, Qualifi
tcp_allow_round_trip: true,
tcp_deny_round_trip: true,
wait_killable_recv: notification.wait_killable_recv,
seccomp_listener_mode: if notification.wait_killable_recv {
"killable"
} else {
"legacy_read_only"
},
task_memory_writes_disabled: !notification.wait_killable_recv,
};
let qualification = openshell_sandbox::RuntimeQualification {
seccomp: openshell_isolation_interface::contract::SeccompEvidence {
Expand All @@ -249,6 +260,9 @@ fn qualify_runtime() -> Result<(openshell_sandbox::RuntimeQualification, Qualifi
task_memory_read: notification.task_memory_copy(),
task_memory_write: notification.task_memory_copy(),
cancellation: notification.wait_killable_recv,
// Legacy plain listener (< 5.19) disables broker output writes;
// satisfies the `cancellation || writes_disabled` launch invariant.
task_memory_writes_disabled: !notification.wait_killable_recv,
Comment on lines 262 to +265

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Under which conditions would cancellation and task_memory_writes_disabled not have different values?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@elezar you're right: today task_memory_writes_disabled is just !cancellation, so the two are always opposite and the cancellation || task_memory_writes_disabled check is tautological.

Instead of dropping the field, I'd make it a real independent attestation. At qualification the sandbox installs a LegacyReadOnly listener and checks that its own broker guard (write_task_output) actually fails closed with EOPNOTSUPP, then reports that result. It becomes true only when the listener is legacy and the guard is verified, so if the guard ever regressed and started writing again, the field drops to false while cancellation is already false, and validate() rejects the boundary instead of admitting the race. The two can then genuinely diverge, and the invariant has teeth. It's ~20 lines in qualify_runtime (prototype on explore/seccomp-writes-disabled-selftest).

I can fix it in this PR, or in a follow-up PR if you'd rather land the current change first.

Since this touches the cancellation-safety contract you reviewed: @mrunalp @johntmyers, does the direction work for you? The alternative is collapsing to a single ListenerMode, which makes the unsafe state unrepresentable but leaves no confirmation cross-check. I'm leaning toward the attestation since it keeps validate() meaningful.

},
landlock_abi,
landlock_allow_deny: true,
Expand Down
Loading
Loading