HostProcess containers with Hypervisor Isolation by rawahars · Pull Request #2531 · microsoft/hcsshim

rawahars · 2025-10-07T20:58:24Z

Summary

This PR adds support for running HostProcess containers with hypervisor isolation. These HPCs will run as privileged processes within the UVM. Few key aspects of this feature are-

This feature will be applicable for WS2025 and later.
A pod needs to be marked privileged by passing in annotation 'microsoft.com/hostprocess-container'. This ensures that the UVM can support HPCs alongside normal containers.
Any container which has 'microsoft.com/hostprocess-container' annotation will be launched as HostProcess Container within the UVM.
Other containers will be launched as normal process-isolated containers within the UVM.
GCS will create the HPCs based on the specs. The behavior is similar to existing HPCs for process-isolated cases.
Annotations for Inherit user will lead to the HPC running as System user within UVM.
Annotation for custom rootfs will lead to the container filesystem being virtualized in a directory with that custom name. If not provided, C:\hpc is used by default.

Primary changes

Refactored pod sandbox logic to use a centralized updateConfigForHostProcessContainer method for handling privileged container configuration and annotation passthrough. This method enforces correct sandbox/container combinations and properly sets user and annotation values for HostProcess containers.
Added new fields to the container and storage schema (ContainerType and ContainerRootPath) to support HostProcess container identification and custom root filesystem locations. These are set during container creation when an isolated job container is detected.
Introduced handleProcessArgsForIsolatedJobContainer to ensure that isolated job containers have their process arguments correctly prefixed with cmd /c when necessary, supporting both CommandLine and Args fields. This logic is invoked during container and exec creation.
Improved container creation flow to distinguish between job containers running on the host and those inside a UVM, ensuring the right creation path is used.

Testing Enhancements

Added comprehensive unit tests for both HostProcess container configuration updates and process argument handling, covering a wide range of scenarios and edge cases to ensure robust behavior.
Tests have been added to cover most of the newly added logic.

jterry75 · 2025-10-31T17:05:58Z

+	// For Isolated Job containers, we need special handling wherein if the command line
+	// is not specified, then we add 'cmd /C' by default.
+	if oci.IsIsolatedJobContainer(s) {
+		handleProcessArgsForIsolatedJobContainer(s, s.Process)


Why is this part of it special for HPC? I dont get it, this is true for all Windows containers right? At this point its just another container

It's specific to HostProcess containers because of when the process is joined to the silo.

For a normal WCOW container (process- or Hyper-V-isolated), the guest creates the workload process from inside the container silo, so CreateProcess resolves the image name and PATH using the silo's filesystem view — the bind-mounted rootfs, the container's System32, etc. A bare echo / dir or a binary that lives only in the rootfs works fine because the silo is already in scope at creation time.

For a HostProcess container the process is created outside the silo and only assigned to the silo's job object after the fact. That means image-name resolution happens against the launcher's view (host/UVM System32, Windows, host PATH), not the container's rootfs. Shell builtins (echo, set, redirection, …) and any rootfs-only executable would fail with ERROR_FILE_NOT_FOUND.

Wrapping with cmd /c sidesteps this: cmd.exe is guaranteed to exist under the launcher's System32, and once it's joined to the silo it runs inside the container's filesystem view, where it can resolve builtins and rootfs-relative paths normally. Regular WCOW containers don't need this because they never have that "created outside, joined after" gap.

This commit adds support for running HostProcess containers with hypervisor isolated workloads. Therefore, these HPCs will run as privileged processes within the UVM. Some of the key aspects of this feature are- - A pod needs to be marked privileged by passing in annotation 'microsoft.com/hostprocess-container' - Any container which has 'microsoft.com/hostprocess-container' annotation will be launched as HostProcess Container. - Other containers will be launched as normal process-isolated containers within the UVM - HPC within UVM will not have access to any network since the sandbox (UVM) does not have any network. - GCS will create the HPCs based on the specs. The behaviour is similar to existing HPCs for process-isolated case. - Annotations for Inherit user will lead to the HPC running as System user within UVM. - Annotation for custom rootfs will lead to the container filesystem being virtualized in a directory with that custom name. If not provided, `C:\hpc` is used by default. Signed-off-by: Harsh Rawat <harshrawat@microsoft.com>

This commit adds thorough unit tests for: - IsJobContainer - IsIsolatedJobContainer These tests explicitly verify the expected behavior for Windows HostProcess containers across isolation modes and OS targets. Signed-off-by: Harsh Rawat <harshrawat@microsoft.com>

This commit adds thorough unit tests for the updateConfigForHostProcessContainer method. We test the following functionality- - Reject privileged container in unprivileged sandbox (isolated HPC) — return error when container requests microsoft.com/hostprocess-container=true but the pod lacks it. - Allow privileged container in privileged sandbox (isolated HPC) — ensure no unexpected mutations when no passthrough annotations are present. - Block normal process-isolated container in process-isolated HPC pod — enforce constraint that all containers in a process HPC pod must also be job containers (error on mixing). - Allow normal hypervisor-isolated container in hypervisor-isolated HPC pod. - Passthrough annotations to privileged containers — propagate HostProcessInheritUser and HostProcessRootfsLocation from pod → container for: hypervisor-isolated HPC pods with privileged containers, and process-isolated HPC pods with privileged containers. - No passthrough for normal containers — verify annotations aren’t propagated when the container is non-privileged. - Set SYSTEM user for hypervisor-isolated privileged containers — when HostProcessInheritUser=true, set Process.User.Username to NT AUTHORITY\SYSTEM. - Do not change user for normal containers — ensure container user remains unchanged even if the pod has HostProcessInheritUser=true. - Force HostProcessInheritUser to "false" for normal containers — in hypervisor-isolated privileged pods, if a non-privileged container sets the inherit annotation, flip it to "false". Signed-off-by: Harsh Rawat <harshrawat@microsoft.com>

Signed-off-by: Harsh Rawat <harshrawat@microsoft.com>

rawahars requested a review from a team as a code owner October 7, 2025 20:58

rawahars force-pushed the hpc_inboc_hcs_for_hypervisor_isolation branch from 76a7277 to b56ed8c Compare October 7, 2025 21:02

rawahars changed the title ~~Hpc inboc hcs for hypervisor isolation~~ HostProcess containers with Hypervisor Isolation Oct 7, 2025

jterry75 reviewed Oct 7, 2025

View reviewed changes

Comment thread internal/hcsoci/hcsdoc_wcow.go Outdated

rawahars force-pushed the hpc_inboc_hcs_for_hypervisor_isolation branch from b56ed8c to e1e1b82 Compare October 8, 2025 07:40

anmaxvl self-assigned this Oct 9, 2025