compose: runtime-agnostic orchestrator (opt-in)

README.md

@@ -26,6 +26,64 @@ shelling out.
 between minor versions until v1.0.0. The `events` channel surface is
 explicitly experimental.
 
+### Backends
+
+The container backend is pluggable. Pick one at engine construction time:
+
+- **`runtime/docker`** — Docker Engine over `moby/moby/client`. Default
+  choice; requires a reachable Docker daemon socket.
+- **`runtime/applecontainer`** — Apple's `container` runtime on
+  darwin/arm64 (macOS 15+). Talks to Apple's apiserver through an
+  embedded Swift bridge (`libACBridge.dylib`, dlopen'd at runtime).
+  Lets you run devcontainers on Apple Silicon without Docker
+  Desktop.
+
+Both backends implement the same `runtime.Runtime` interface — the
+engine, feature pipeline, lifecycle, and compose paths don't care which
+one you wire in.
+
+#### Apple-container gotchas
+
+Things specific to `runtime/applecontainer` that don't apply to the
+Docker backend. None of these are bugs in this library; they reflect
+the current state of Apple's `container` runtime (0.12.x).
+
+- **Daemon + builder lifecycle is manual.** Run `container system start`
+  once per boot; `container builder start` once per machine before any
+  `build`-source devcontainer or `features` install. Engine surfaces a
+  typed `runtime.BuilderUnavailableError` with a remediation hint when
+  the builder is down.
+- **Image-store credentials are separate from Docker's.** Pulls from
+  private registries require `container registry login <host>` before
+  Up; `~/.docker/config.json` is not consulted.
+- **Multi-arch base images.** Apple's BuildKit shim ships amd64-only,
+  so the builder VM always runs amd64 (Rosetta on Apple Silicon).
+  *Output* images target the host platform (`arm64`) by default —
+  feature builds produce native arm64 images that run without Rosetta.
+  If your `FROM` base image is amd64-only, the resulting image is also
+  amd64 and the runtime container needs Rosetta-for-Linux to boot.
+- **No amd64 default kernel.** Running amd64 containers requires
+  installing the amd64 kernel explicitly:
+  `container system kernel set --tar <url-to-amd64-tarball> --arch amd64`.
+  Don't use `--force` without `--arch`; it can overwrite the arm64
+  default registration.
+- **Port forwarding (`forwardPorts` / compose `ports:`) is parsed but
+  not actuated.** Apple's networking model differs from Docker's
+  host-port-publish; on this backend ports are silently dropped today.
+- **Compose feature gates that depend on upstream apple/container
+  fixes are refused at Plan-validate time** with typed errors:
+  `service_healthy` / `service_completed_successfully` (no healthcheck
+  surface yet — apple/container #1502, #1501), namespace sharing
+  modes (`network_mode: service:<x>`, `pid:`, `ipc:` — architectural,
+  VM-per-container), shared named volumes (ext4 single-mount —
+  apple/container #889). `restart:` policies are silently ignored with
+  a one-shot warning (apple/container #286).
+- **Service-name DNS** is not native on the project network
+  (apple/container #856). The compose orchestrator patches `/etc/hosts`
+  in each running container after each level so `depends_on`-declared
+  peers resolve; intra-level peers without an explicit edge can lose
+  a first lookup. Documented limitation.
+
 ### Spec compliance
 
 Status of each [Dev Containers spec](https://containers.dev/implementors/json_reference/)
@@ -37,7 +95,7 @@ field/behavior the library covers. Legend: ✅ acted on · ⚠️ parsed but not
 | --- | --- | --- |
 | `image` | ✅ | Pull, run, exec |
 | `build` (`dockerfile`, `context`, `args`, `target`, `cacheFrom`) | ✅ | User Dockerfile + features layered atop |
-| `dockerComposeFile`, `service`, `runServices` | ✅ | compose-go parse, shell-out to `docker compose` |
+| `dockerComposeFile`, `service`, `runServices` | ✅ | compose-go parse + either shell-out to `docker compose` (default) or an in-process orchestrator (`EngineOptions.ComposeBackend = ComposeBackendNative`). The native orchestrator drives any `Runtime` implementing the compose primitives — works against both `runtime/docker` and `runtime/applecontainer`. |
 
 **Container config**
 
@@ -110,8 +168,15 @@ go get github.com/crunchloop/devcontainer
 Requires:
 
 - Go 1.25+
-- Docker daemon socket reachable
-- Docker Compose v2 plugin (only for `dockerComposeFile` source)
+- A container backend, one of:
+  - **Docker:** daemon socket reachable; Docker Compose v2 plugin
+    only when running `dockerComposeFile` projects under the default
+    shellout backend (skip the plugin if you opt into
+    `ComposeBackendNative`).
+  - **Apple `container`:** macOS 15+ on Apple Silicon, `container
+    system start` already up. Swift toolchain only if you're
+    building the bridge from source — releases embed the
+    pre-built dylib.
 
 ## Quick start
 
@@ -159,6 +224,16 @@ func main() {
 }
 ```
 
+To target Apple's `container` runtime instead of Docker, swap the
+backend import — the rest of the engine code is unchanged:
+
+```go
+import "github.com/crunchloop/devcontainer/runtime/applecontainer"
+
+rt, err := applecontainer.New(ctx, applecontainer.Options{})
+// ... pass to devcontainer.New the same way
+```
+
 Runnable end-to-end examples in [`examples/`](examples/):
 
 - [`image-source/`](examples/image-source/) — minimal image-only devcontainer
@@ -186,10 +261,11 @@ func (*Engine) Down(ctx, *Workspace, DownOptions) error
 Sub-packages:
 
 - `config` — devcontainer.json parsing, merging, host-context substitution
-- `runtime` — container backend abstraction (`Runtime`, `ComposeRuntime`)
+- `runtime` — container backend abstraction (`Runtime`, `ComposeRuntime`, capabilities, network/volume/list primitives)
 - `runtime/docker` — Docker Engine API implementation (uses `moby/moby/client`)
+- `runtime/applecontainer` — Apple `container` implementation; darwin/arm64 only, cgo-linked Swift bridge
 - `feature` — feature resolution (OCI / HTTPS / local), DAG ordering, dockerfile generation
-- `compose` — `dockerComposeFile` parsing via `compose-spec/compose-go`, override-file generation
+- `compose` — `dockerComposeFile` parsing via `compose-spec/compose-go`, plus a runtime-agnostic in-process orchestrator (`Orchestrator`, `Plan`, topological + health gating) used when `ComposeBackendNative` is selected
 
 ## Tests
 
@@ -204,6 +280,12 @@ pulls public images from GHCR, builds Dockerfiles, runs feature install
 scripts, drives `docker compose up/down`. Skipped automatically if a
 Docker daemon isn't reachable.
 
+Apple-container integration tests are tagged
+`integration && darwin && arm64` and run against a live `container`
+apiserver — skipped when the daemon isn't running. CI runs both the
+Linux + Docker suite and a `macos-26` job that builds the Swift
+bridge and runs the applecontainer unit tests.
+
 ## Contributing
 
 See [CONTRIBUTING.md](CONTRIBUTING.md). Bug reports welcome via

applecontainer-bridge/Sources/ACBridge/lifecycle.swift

@@ -20,6 +20,13 @@ private struct RunSpecJSON: Decodable {
     var env: [String]?
     var labels: [String: String]?
     var mounts: [MountJSON]?
+    // Network IDs the container should be attached to. Empty / nil
+    // means "no explicit attachment" — apple's apiserver auto-joins
+    // the built-in default network when the field is unset. The
+    // compose orchestrator passes <project>_default here so its
+    // services land on the project network it created via
+    // NetworkClient.create.
+    var networks: [String]?
     var initProcess: Bool?
     var capAdd: [String]?
     var overrideCommand: Bool?
@@ -73,24 +80,83 @@ private func runContainer(spec: RunSpecJSON) async throws {
 
     // The image must already be in the local content store. Pull is
     // PR-F's job; here we assume the caller has done it.
+    // Resolve the platform on the cached image first, then re-fetch
+    // for that platform so the daemon has the correct snapshot
+    // staged. apple/container's containerConfigFromFlags uses
+    // ClientImage.fetch (not get) for exactly this reason: get
+    // returns the index entry but doesn't ensure a per-platform
+    // snapshot is present, and ContainerClient().create rejects
+    // missing snapshots as "does not support required platforms".
+    // Resolve the image's platform. For multi-arch images, prefer
+    // the host's `.current`; for single-arch images (commonly
+    // amd64-only when the publisher only builds on x86 CI), fall
+    // back to whatever the image actually carries. Then stage the
+    // per-platform snapshot the apiserver requires before create —
+    // mirrors apple/container CLI's containerConfigFromFlags path.
     let img = try await ClientImage.get(reference: spec.image)
-    let imageConfig = try await img.config(for: .current).config
+    let platform = try await resolvePlatform(for: img)
+    try await img.getCreateSnapshot(platform: platform, progressUpdate: nil)
+    let imageConfig = try await img.config(for: platform).config
 
     let process = try buildProcessConfiguration(spec: spec, imageConfig: imageConfig)
 
     var cfg = ContainerConfiguration(id: spec.id, image: img.description, process: process)
-    cfg.platform = .current
+    cfg.platform = platform
     cfg.labels = spec.labels ?? [:]
     cfg.mounts = try (spec.mounts ?? []).map(toFilesystem)
     cfg.capAdd = spec.capAdd ?? []
     cfg.useInit = spec.initProcess ?? false
+    // Enable Rosetta when running an amd64 container on an arm64
+    // host. Without this flag the apiserver rejects amd64 containers
+    // with "unsupported: platform linux/amd64". Mirrors
+    // apple/container CLI's containerConfigFromFlags auto-enabling
+    // of rosetta for the same case. Subject to host's Rosetta-for-
+    // Linux being installed and Virtualization.framework allowing
+    // its use — neither is universally available, and an
+    // unsupported host surfaces as VZErrorDomain Code=1 at bootstrap.
+    let host = ContainerizationOCI.Platform.current
+    if host.architecture == "arm64" && platform.architecture == "amd64" {
+        cfg.rosetta = true
+    }
+    // Attach explicitly to any networks the caller requested. The
+    // hostname per attachment defaults to the container id, matching
+    // apple/container CLI's behavior. Empty Networks => no override:
+    // the apiserver attaches to the built-in default automatically.
+    if let nets = spec.networks, !nets.isEmpty {
+        cfg.networks = nets.map {
+            AttachmentConfiguration(network: $0, options: AttachmentOptions(hostname: spec.id))
+        }
+    }
+
+    // Kernel selection: always use the host platform. For amd64
+    // containers on arm64 hosts, the VM still runs an arm64 kernel
+    // and Apple's Rosetta translates amd64 userland binaries
+    // (cfg.rosetta=true, set below). Mirrors apple/container's CLI:
+    // the kernel is host-arch; the container's platform only
+    // influences Rosetta enablement and image manifest selection.
+    let hostSysPlatform: SystemPlatform = .linuxArm
+    let kernel = try await ClientKernel.getDefaultKernel(for: hostSysPlatform)
+    // Stage the init image for the host platform (.current).
+    // The init binary runs in the VM's pid 1 slot — apple's
+    // apiserver wires up a translation when the container's
+    // platform differs (Rosetta on Apple silicon). Mirrors the
+    // CLI's containerConfigFromFlags: it always fetches init for
+    // .current regardless of the container's platform.
+    let initImageRef = ClientImage.initImageRef
+    let initImg = try await ClientImage.fetch(
+        reference: initImageRef,
+        platform: .current,
+        scheme: .auto,
+        progressUpdate: nil
+    )
+    try await initImg.getCreateSnapshot(platform: .current, progressUpdate: nil)
 
-    let kernel = try await ClientKernel.getDefaultKernel(for: .current)
     let options = ContainerCreateOptions(autoRemove: false)
     try await ContainerClient().create(
         configuration: cfg,
         options: options,
-        kernel: kernel
+        kernel: kernel,
+        initImage: initImageRef
     )
 }
 
@@ -192,6 +258,33 @@ private func toFilesystem(_ m: MountJSON) throws -> Filesystem {
     }
 }
 
+// resolvePlatform picks a platform descriptor the image actually
+// supports. Default preference is the host's `.current`; if the
+// image's index doesn't carry that variant (common case: amd64-only
+// images on Apple silicon), fall back to the first variant the
+// image's index declares. Falls back to .current if the image
+// store can't surface an index (legacy single-manifest images).
+private func resolvePlatform(for img: ClientImage) async throws -> ContainerizationOCI.Platform {
+    let current = ContainerizationOCI.Platform.current
+    do {
+        let index = try await img.index()
+        for desc in index.manifests {
+            if let p = desc.platform, p.architecture == current.architecture && p.os == current.os {
+                return current
+            }
+        }
+        for desc in index.manifests {
+            if let p = desc.platform {
+                return p
+            }
+        }
+    } catch {
+        // Single-manifest image or other index lookup error;
+        // .current is the right default to try.
+    }
+    return current
+}
+
 private enum BridgeError: LocalizedError {
     case invalidArgument(String)
 

applecontainer-bridge/Sources/ACBridge/list.swift

@@ -0,0 +1,97 @@
+import ContainerAPIClient
+import ContainerizationError
+import ContainerizationOCI
+import Foundation
+
+// Compose orchestrator primitives — listing containers / images and
+// removing images. Apple's surface lacks server-side label
+// filtering (probe R1b confirmed `container list` has no --filter
+// in 0.12.3), so these exports enumerate full result sets and
+// expect the Go layer to filter. The result-set size for a single
+// project is small enough that the overhead is negligible.
+
+private let listTimeoutSeconds = 15
+
+// ContainerListItem is the projection per container the Go side
+// consumes from ListContainers. Mirrors the runtime.Container shape
+// the docker backend returns from its own ContainerList.
+private struct ContainerListItem: Encodable {
+    let id: String
+    let name: String
+    let image: String
+    let state: String
+    let labels: [String: String]
+}
+
+private struct ContainerListData: Encodable {
+    let containers: [ContainerListItem]
+}
+
+@_cdecl("ac_list_containers")
+public func ac_list_containers() -> UnsafePointer<CChar>? {
+    return runSync(timeoutSeconds: listTimeoutSeconds) {
+        do {
+            // ContainerListFilters.all enumerates every container
+            // regardless of state. The Go side applies label-based
+            // filtering after this returns.
+            let snaps = try await ContainerClient().list(filters: .all)
+            let items = snaps.map { snap -> ContainerListItem in
+                let cfg = snap.configuration
+                return ContainerListItem(
+                    id: cfg.id,
+                    name: cfg.id,
+                    image: cfg.image.reference,
+                    state: snap.status.rawValue,
+                    labels: cfg.labels
+                )
+            }
+            return encodeOK(ContainerListData(containers: items))
+        } catch {
+            return encodeErr(error)
+        }
+    }
+}
+
+// ImageListItem mirrors runtime.ImageRef. Tags slice carries the
+// image's user-facing reference; ID is the manifest digest.
+private struct ImageListItem: Encodable {
+    let id: String
+    let tags: [String]
+}
+
+private struct ImageListData: Encodable {
+    let images: [ImageListItem]
+}
+
+@_cdecl("ac_list_images")
+public func ac_list_images() -> UnsafePointer<CChar>? {
+    return runSync(timeoutSeconds: listTimeoutSeconds) {
+        do {
+            let imgs = try await ClientImage.list()
+            let items = imgs.map { img -> ImageListItem in
+                ImageListItem(
+                    id: img.description.digest,
+                    tags: [img.description.reference]
+                )
+            }
+            return encodeOK(ImageListData(images: items))
+        } catch {
+            return encodeErr(error)
+        }
+    }
+}
+
+@_cdecl("ac_remove_image")
+public func ac_remove_image(_ refPtr: UnsafePointer<CChar>?) -> UnsafePointer<CChar>? {
+    guard let ref = readCString(refPtr) else { return dupNullArgErr("reference") }
+    return runSync(timeoutSeconds: listTimeoutSeconds) {
+        do {
+            try await ClientImage.delete(reference: ref, garbageCollect: true)
+            return "{\"ok\":true}"
+        } catch let e as ContainerizationError where e.code == .notFound {
+            return "{\"ok\":true}"
+        } catch {
+            return encodeErr(error)
+        }
+    }
+}