From 3fc43200f436762963e4ad9cb2fb66c99b75e864 Mon Sep 17 00:00:00 2001 From: Pratyush Singhal Date: Tue, 6 Oct 2026 16:17:43 -0700 Subject: [PATCH 01/11] feat(sandbox): add a Sail Sailbox VM sandbox provider Adds `sail`, a built-in VM sandbox provider for Sail Research Sailboxes. A Sailbox boots from Sail's devbox image (Docker and Compose v2, root), so the gateway, agent, deploy_sandbox and image-build paths run on it unchanged through VmSandbox. - Config: [sandbox.providers.sail.config] api_key (secret:/env: ref, required), app, min_size, auto_sleep (off by default), auto_sleep_min_idle_seconds, runtime_threads. - Resources map to the smallest Sail size covering the CPU; memory and disk are ceilings rounded up into the size's range. - Ports are public *.sail.box listeners; reconnect restores them and the applied egress policy (fail closed when unrepresentable). - Egress allowlists of hostnames and IPv4 CIDRs, up to Sail's 128 entries; signed download hosts are added under a per-Sailbox lock. - Exec streams output with bounded backpressure; a lost host maps to exit -1 so exec_script retries. - A create cancelled by its caller terminates the Sailbox it yields. - Attribution goes in the Sailbox name and an agent_env.sail_sandbox_started log event keyed by sailbox_id. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- THIRD_PARTY_NOTICES.md | 51 +++ pyproject.toml | 1 + src/agent_env/cli/a2a_agent/deploy.py | 2 +- src/agent_env/cli/env/deploy.py | 2 +- src/agent_env/providers/__init__.py | 2 + .../providers/sandbox_providers/__init__.py | 2 + .../sandbox_providers/sail/__init__.py | 6 + .../providers/sandbox_providers/sail/_sdk.py | 60 +++ .../sandbox_providers/sail/provider.py | 315 ++++++++++++++ .../sandbox_providers/sail/sandbox.py | 282 +++++++++++++ .../sandbox_providers/sandbox_provider.py | 1 + tst/integration/env/gateway/gateway_test.py | 9 + .../env/task_step/task_steps_test.py | 10 + .../sail_sandbox_smoke_test.py | 87 ++++ tst/unit/config/test_explain_and_sources.py | 5 +- .../env_gateway_provider_test.py | 22 + .../sandbox_providers/egress_hosts_test.py | 2 +- .../network_policy_stamping_test.py | 14 + .../sail_sandbox_provider_test.py | 398 ++++++++++++++++++ .../sandbox_providers/sail_sandbox_test.py | 327 ++++++++++++++ .../test_config_sandbox_providers.py | 34 ++ tst/unit/test_capabilities.py | 15 + tst/util/capabilities.py | 14 +- uv.lock | 27 ++ 25 files changed, 1677 insertions(+), 13 deletions(-) create mode 100644 src/agent_env/providers/sandbox_providers/sail/__init__.py create mode 100644 src/agent_env/providers/sandbox_providers/sail/_sdk.py create mode 100644 src/agent_env/providers/sandbox_providers/sail/provider.py create mode 100644 src/agent_env/providers/sandbox_providers/sail/sandbox.py create mode 100644 tst/integration/providers/sandbox_providers/sail_sandbox_smoke_test.py create mode 100644 tst/unit/providers/sandbox_providers/sail_sandbox_provider_test.py create mode 100644 tst/unit/providers/sandbox_providers/sail_sandbox_test.py diff --git a/AGENTS.md b/AGENTS.md index 9135ed2b..202c2ee0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -82,7 +82,7 @@ and deserialized through a registry. | `task/`, `task_step/` | A `Task` holds its `TaskStep`s inline; `Task.run()` executes them as a DAG. `depends_on` (None means all prior steps) gates a step, independent steps run concurrently, `fail_task_on_error` makes a failure fatal or tolerated, `retry_config` rolls a failed span back through the step journal and re-dispatches it. Built-in steps live in `task_step/task_steps/` (`deploy_env`, `deploy_agent`, `prompt_agent`, the verifiers under `verifiers/`, and more). | | `store/` | Four store ABCs with local and cloud implementations: `DocumentStore` (SQLite, MongoDB), `ObjectStore` (filesystem, S3, Cloud Storage), `ImageStore` (local OCI registry, ECR), `SecretStore` (env vars or file, AWS Secrets Manager, Google Cloud Secret Manager). `VersionedEntityStore` implements the shared versioned get/put logic, `QueryBuilder` is the immutable chained query API, `store/base.py` holds the error types. A new backend must pass the conformance kits in `tst/store/`. | | `config/` | The `Config` singleton (`get_config`, `configure`, `reset_config`) in `config/runtime.py`, file discovery in `config/loader.py`, and `load_impl`, which resolves `module:Class` pointers. `agent_env.store` re-exports the config names for compatibility. | -| `providers/` | `providers/sandbox_providers/` holds the sandbox providers `local`, `modal`, `modal_vm`, `e2b`; `[sandbox] default` and `agent_default` accept a comma-separated fallback chain. `providers/env_providers/` holds the environment providers: `EnvironmentProvider` (an env's containers and state store) and `EnvironmentGatewayProvider`, which renders a docker-compose for the gateway and its MCP servers inside the sandbox; `providers/env_state/` holds env-state providers (`local_postgres` built in). | +| `providers/` | `providers/sandbox_providers/` holds the sandbox providers `local`, `modal`, `modal_vm`, `e2b`, `sail`; `[sandbox] default` and `agent_default` accept a comma-separated fallback chain. `providers/env_providers/` holds the environment providers: `EnvironmentProvider` (an env's containers and state store) and `EnvironmentGatewayProvider`, which renders a docker-compose for the gateway and its MCP servers inside the sandbox; `providers/env_state/` holds env-state providers (`local_postgres` built in). | | `a2a_agent/` | The `A2AAgent` entity (`a2a_agent`), its stores and the validator steps. The protocol package provides the agent-side framework. | | `runner/` | The `[runner]` seam: `Runner.submit()` returns `(run_id, instance_id)`; `LocalRunner` is built in. | | `explorer/` | Optional local web UI: `agent-env up`, needs the `explorer` extra, binds loopback `:8234`. | diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index 7eb436bb..c32d66fc 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -128,6 +128,13 @@ The runtime dependencies of `agentenv-framework` and its `explorer` and `gcp` ex - Source: - [License text 12](#license-text-12) +### cloudpickle 3.1.2 + +- License: BSD-3-Clause +- Author: The cloudpickle developer team +- Source: +- [License text 70](#license-text-70) + ### colorama 0.4.6 - License: BSD @@ -660,6 +667,13 @@ The runtime dependencies of `agentenv-framework` and its `explorer` and `gcp` ex - Source: - [License text 1](#license-text-1), [License text 42](#license-text-42) +### sail 0.12.8 + +- License: Apache-2.0 +- Author: Sail +- Source: +- [License text 1](#license-text-1) + ### shellingham 1.5.4 - License: ISC @@ -7663,3 +7677,40 @@ OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. ***************************************************************************** */ ``` + +### License text 70 + +```text +This module was extracted from the `cloud` package, developed by +PiCloud, Inc. + +Copyright (c) 2015, Cloudpickle contributors. +Copyright (c) 2012, Regents of the University of California. +Copyright (c) 2009 PiCloud, Inc. http://www.picloud.com. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + * Neither the name of the University of California, Berkeley nor the + names of its contributors may be used to endorse or promote + products derived from this software without specific prior written + permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +``` diff --git a/pyproject.toml b/pyproject.toml index 5dbdcb12..b3adb772 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -47,6 +47,7 @@ dependencies = [ "uvicorn>=0.47", # 0.47 adds ssl_context_factory, how the local grant server gets its TLS context "modal>=1.5.2.dev22", # dev22 adds _experimental_create(tags=); dev2 added VM-mode V2 (vm_runtime) "e2b~=2.46.4", # AsyncSandbox/AsyncTemplate APIs validated against the 2.46 line + "sail~=0.12.8", # Sailbox exec/fs/listener/egress APIs validated against the 0.12 line ] [project.optional-dependencies] diff --git a/src/agent_env/cli/a2a_agent/deploy.py b/src/agent_env/cli/a2a_agent/deploy.py index 5c6ae687..2b9e4721 100644 --- a/src/agent_env/cli/a2a_agent/deploy.py +++ b/src/agent_env/cli/a2a_agent/deploy.py @@ -18,7 +18,7 @@ @click.option("--ttl-seconds", type=click.IntRange(min=MIN_TTL_SECONDS, max=MAX_TTL_SECONDS), default=DEFAULT_TTL_SECONDS, help=f"VM lifetime in seconds (default {DEFAULT_TTL_SECONDS})") @click.option("--sandbox", default=None, - help="Sandbox backend(s): a built-in (modal, modal_vm, e2b, local) " + help="Sandbox backend(s): a built-in (modal, modal_vm, e2b, sail, local) " "or a name from [sandbox.providers] in .agentenv/config.toml; comma-separated for a " "fallback chain. Defaults to [sandbox].agent_default (else local) when omitted.") def deploy(agent_id: str, agent_version: int | None, env_var_pairs: tuple[str, ...], ttl_seconds: int, sandbox: str): diff --git a/src/agent_env/cli/env/deploy.py b/src/agent_env/cli/env/deploy.py index 770e5e38..39b949e1 100644 --- a/src/agent_env/cli/env/deploy.py +++ b/src/agent_env/cli/env/deploy.py @@ -19,7 +19,7 @@ @click.option("--gateway-mode", type=click.Choice([m.value for m in GatewayMode], case_sensitive=False), default=GatewayMode.PERFORMANCE.value, help="Gateway mode (performance or consistent)") @click.option("--sandbox", default=None, - help="Sandbox backend(s): a built-in (modal, modal_vm, e2b, local) " + help="Sandbox backend(s): a built-in (modal, modal_vm, e2b, sail, local) " "or a name from [sandbox.providers] in .agentenv/config.toml; comma-separated for a " "fallback chain. Defaults to [sandbox].default (else local) when omitted.") @click.option("--service-db", "service_db_env_id", default=None, diff --git a/src/agent_env/providers/__init__.py b/src/agent_env/providers/__init__.py index 3c403dcb..25af89da 100644 --- a/src/agent_env/providers/__init__.py +++ b/src/agent_env/providers/__init__.py @@ -19,6 +19,7 @@ from agent_env.providers.sandbox_providers.modal_sandbox import ModalSandbox, ModalSandboxProvider from agent_env.providers.sandbox_providers.modal_vm_sandbox import ModalVmSandbox, ModalVmSandboxProvider from agent_env.providers.sandbox_providers.e2b import E2BSandbox, E2BSandboxProvider +from agent_env.providers.sandbox_providers.sail import SailSandbox, SailSandboxProvider __all__ = [ "ChainedSandboxProvider", @@ -29,6 +30,7 @@ "ModalSandbox", "ModalSandboxProvider", "ModalVmSandbox", "ModalVmSandboxProvider", "E2BSandbox", "E2BSandboxProvider", + "SailSandbox", "SailSandboxProvider", "WebsiteConfig", "build_env_provider", "build_sandbox_provider", "get_sandbox_provider", "set_sandbox_provider", "reset_sandbox_provider", diff --git a/src/agent_env/providers/sandbox_providers/__init__.py b/src/agent_env/providers/sandbox_providers/__init__.py index b0f13b2a..7848868d 100644 --- a/src/agent_env/providers/sandbox_providers/__init__.py +++ b/src/agent_env/providers/sandbox_providers/__init__.py @@ -14,6 +14,7 @@ from agent_env.providers.sandbox_providers.modal_sandbox import ModalSandbox, ModalSandboxProvider from agent_env.providers.sandbox_providers.modal_vm_sandbox import ModalVmSandbox, ModalVmSandboxProvider from agent_env.providers.sandbox_providers.e2b import E2BSandbox, E2BSandboxProvider +from agent_env.providers.sandbox_providers.sail import SailSandbox, SailSandboxProvider __all__ = [ "ChainedSandboxProvider", "Sandbox", "VmSandbox", @@ -22,6 +23,7 @@ "ModalSandbox", "ModalSandboxProvider", "ModalVmSandbox", "ModalVmSandboxProvider", "E2BSandbox", "E2BSandboxProvider", + "SailSandbox", "SailSandboxProvider", "build_sandbox_provider", "get_sandbox_provider", "set_sandbox_provider", "reset_sandbox_provider", "get_env_sandbox_provider", "set_env_sandbox_provider", "reset_env_sandbox_provider", diff --git a/src/agent_env/providers/sandbox_providers/sail/__init__.py b/src/agent_env/providers/sandbox_providers/sail/__init__.py new file mode 100644 index 00000000..da11a516 --- /dev/null +++ b/src/agent_env/providers/sandbox_providers/sail/__init__.py @@ -0,0 +1,6 @@ +"""Sail Research Sailbox sandbox provider package.""" + +from agent_env.providers.sandbox_providers.sail.provider import SailSandboxProvider +from agent_env.providers.sandbox_providers.sail.sandbox import SailSandbox + +__all__ = ["SailSandbox", "SailSandboxProvider"] diff --git a/src/agent_env/providers/sandbox_providers/sail/_sdk.py b/src/agent_env/providers/sandbox_providers/sail/_sdk.py new file mode 100644 index 00000000..3f3499b7 --- /dev/null +++ b/src/agent_env/providers/sandbox_providers/sail/_sdk.py @@ -0,0 +1,60 @@ +"""The Sail SDK, imported on first use and authenticated with the provider's configured key. + +The Python SDK takes its key only from ``SAIL_API_KEY``, read once when it builds its process-wide +client. The key is set for that one build and the variable restored, so subprocesses never inherit it +and a process holds one Sail key. +""" + +from __future__ import annotations + +import hashlib +import os +import threading +from types import ModuleType +from typing import Any + +from agent_env.config.errors import ConfigError + +API_KEY_ENV = "SAIL_API_KEY" +RUNTIME_THREADS_ENV = "SAIL_RUNTIME_THREADS" + +_lock = threading.Lock() +_installed_key: str | None = None +_apps: dict[str, Any] = {} + + +def _fingerprint(api_key: str) -> str: + return hashlib.sha256(api_key.encode()).hexdigest() + + +def connect(api_key: str, app_name: str, *, runtime_threads: int | None = None, sdk: ModuleType | Any | None = None) -> tuple[Any, Any]: + """The SDK module and the Sail App ``app_name`` (minted if missing), authenticating on first use. + + Blocking: call it through ``asyncio.to_thread``. + """ + global _installed_key + if sdk is None: + import sail as sdk + fingerprint = _fingerprint(api_key) + with _lock: + if _installed_key is None: + previous = os.environ.get(API_KEY_ENV) + os.environ[API_KEY_ENV] = api_key + if runtime_threads is not None: + os.environ[RUNTIME_THREADS_ENV] = str(runtime_threads) + try: + sdk.reset_transports() + _apps[app_name] = sdk.App.find(name=app_name, mint_if_missing=True) + finally: + if previous is None: + os.environ.pop(API_KEY_ENV, None) + else: + os.environ[API_KEY_ENV] = previous + _installed_key = fingerprint + elif _installed_key != fingerprint: + raise ConfigError( + "a process can use one Sail API key: another [sandbox.providers.sail] key is already in use" + ) + elif app_name not in _apps: + _apps[app_name] = sdk.App.find(name=app_name, mint_if_missing=True) + return sdk, _apps[app_name] diff --git a/src/agent_env/providers/sandbox_providers/sail/provider.py b/src/agent_env/providers/sandbox_providers/sail/provider.py new file mode 100644 index 00000000..a6840bba --- /dev/null +++ b/src/agent_env/providers/sandbox_providers/sail/provider.py @@ -0,0 +1,315 @@ +"""Sail Research Sailbox VM sandbox provider.""" + +from __future__ import annotations + +import asyncio +import logging +import math +import re +import uuid +from typing import Any, ClassVar, Self + +from agent_env.attribution import PIPELINE_STEP_KEY, RUN_ID_KEY, Attribution +from agent_env.config.errors import ConfigError +from agent_env.providers.sandbox_providers.sail import _sdk +from agent_env.providers.sandbox_providers.sail.sandbox import ( + MAX_ALLOWLIST_ENTRIES, + SailSandbox, + egress_document, + policy_from_document, +) +from agent_env.providers.sandbox_providers.sandbox import NetworkPolicy, NetworkPolicyUnsupportedError +from agent_env.providers.sandbox_providers.sandbox_provider import ( + SANDBOX_MODE_VM, + SandboxProvider, + apply_default_attribution, +) + +logger = logging.getLogger(__name__) + +SANDBOX_STARTED_EVENT = "agent_env.sail_sandbox_started" + +#: (size, vCPU, (min, max) memory GiB, (min, max) disk GiB), smallest first. Memory and disk are +#: ceilings, not reservations: Sail bills observed usage. +_SIZES: tuple[tuple[str, int, tuple[int, int], tuple[int, int]], ...] = ( + ("s", 1, (2, 64), (8, 128)), + ("m", 4, (8, 128), (32, 512)), + ("l", 8, (16, 256), (64, 1024)), +) +_SIZE_NAMES = tuple(size[0] for size in _SIZES) +_LISTENER_TIMEOUT = 60 +_LISTENER_POLL_INTERVAL = 1 +_MAX_NAME_LENGTH = 128 +_REAP_ATTEMPTS = 3 + +_reapers: set[asyncio.Task] = set() + + +def sailbox_shape(cpu: float, memory_mb: int, disk_size_gb: float, *, min_size: str = "s") -> tuple[str, int, int]: + """The smallest size at or above ``min_size`` covering the request, and its memory and disk ceilings + in GiB, each rounded up to whole GiB and into the size's range.""" + memory_gib = math.ceil(memory_mb / 1024) + disk_gib = math.ceil(disk_size_gb) + for name, vcpu, (memory_min, memory_max), (disk_min, disk_max) in _SIZES[_SIZE_NAMES.index(min_size):]: + if cpu <= vcpu and memory_gib <= memory_max and disk_gib <= disk_max: + return name, max(memory_gib, memory_min), max(disk_gib, disk_min) + raise ValueError( + f"no Sailbox size fits cpu={cpu}, memory={memory_mb}MiB, disk={disk_size_gb}GB " + "(largest is l: 8 vCPU, 256 GiB memory, 1024 GiB disk)" + ) + + +def sailbox_name(attribution: Attribution) -> str: + """``ae-`` plus the attribution values in key order, slugged: for people and ``list(search=)``.""" + slugs = [re.sub(r"[^A-Za-z0-9]+", "-", str(attribution[key])).strip("-") for key in sorted(attribution)] + return "-".join(["ae", uuid.uuid4().hex[:8], *filter(None, slugs)])[:_MAX_NAME_LENGTH].rstrip("-") + + +async def _reap(sailbox: Any) -> None: + for attempt in range(_REAP_ATTEMPTS): + try: + await sailbox.terminate.aio() + logger.info("Terminated Sailbox %s, created after its caller was cancelled", sailbox.sailbox_id) + return + except Exception as exc: # noqa: BLE001 - every failure is retried, then reported + logger.warning("Terminating orphaned Sailbox %s failed (attempt %s): %s", sailbox.sailbox_id, attempt + 1, exc) + await asyncio.sleep(2 ** attempt) + logger.error( + "Orphaned Sailbox %s is still running after %s termination attempts; it stops at its max lifetime", + sailbox.sailbox_id, _REAP_ATTEMPTS, + ) + + +def _terminate_orphan(create: asyncio.Future) -> None: + if create.cancelled() or create.exception() is not None: + return + reaper = asyncio.ensure_future(_reap(create.result())) + _reapers.add(reaper) + reaper.add_done_callback(_reapers.discard) + + +async def _create_or_reclaim(create: Any) -> Any: + """Await a Sailbox create; if the caller is cancelled first, terminate the Sailbox it yields, which + would otherwise keep running with no handle.""" + task = asyncio.ensure_future(create) + try: + return await asyncio.shield(task) + except asyncio.CancelledError: + task.add_done_callback(_terminate_orphan) + raise + + +class SailSandboxProvider(SandboxProvider): + """Docker-capable Sailboxes. ``api_key`` comes from resolved provider config (a ``secret:`` reference) + and never reaches a workload.""" + + EGRESS_HOSTS: ClassVar[tuple[str, ...]] = ("*.sail.box",) + + def __init__( + self, + *, + api_key: str, + app: str = "agent-env", + min_size: str = "s", + auto_sleep: bool = False, + auto_sleep_min_idle_seconds: int | None = None, + runtime_threads: int | None = None, + sdk: Any | None = None, + ): + self._api_key = api_key + self._app_name = app + self._min_size = min_size + self._auto_sleep = auto_sleep or auto_sleep_min_idle_seconds is not None + self._auto_sleep_min_idle_seconds = auto_sleep_min_idle_seconds + self._runtime_threads = runtime_threads + self._sdk = sdk + self._app: Any | None = None + + def __repr__(self) -> str: + return f"SailSandboxProvider(app={self._app_name!r})" + + @classmethod + def from_config(cls, **config: Any) -> Self: + section = "[sandbox.providers.sail.config]" + api_key = config.get("api_key") + if not isinstance(api_key, str) or not api_key.strip(): + raise ConfigError(f"{section} requires a non-empty 'api_key' (e.g. \"secret:sail_api_key\")") + app = config.get("app", "agent-env") + if not isinstance(app, str) or not app.strip(): + raise ConfigError(f"{section} 'app' must be a non-empty string") + if config.get("min_size", "s") not in _SIZE_NAMES: + raise ConfigError(f"{section} 'min_size' must be one of {list(_SIZE_NAMES)}") + if not isinstance(config.get("auto_sleep", False), bool): + raise ConfigError(f"{section} 'auto_sleep' must be true or false") + for key, (low, high) in {"auto_sleep_min_idle_seconds": (1, 3600), "runtime_threads": (1, 256)}.items(): + value = config.get(key) + if value is not None and (isinstance(value, bool) or not isinstance(value, int) or not low <= value <= high): + raise ConfigError(f"{section} '{key}' must be an integer from {low} to {high}") + unknown = set(config) - {"api_key", "app", "min_size", "auto_sleep", "auto_sleep_min_idle_seconds", "runtime_threads"} + if unknown: + raise ConfigError(f"{section} has unknown key(s): {sorted(unknown)}") + return cls(**config) + + @classmethod + def supports_network_policy(cls, policy: NetworkPolicy) -> bool: + """Allow-all, or an allowlist of hostnames, IPv4 addresses and IPv4 CIDRs within Sail's entry limit.""" + if not policy.restricts_egress: + return True + return ( + len(policy.allow_hosts) + len(policy.allow_cidrs) <= MAX_ALLOWLIST_ENTRIES + and not any(":" in entry for entry in (*policy.allow_hosts, *policy.allow_cidrs)) + ) + + async def _connect(self) -> tuple[Any, Any]: + if self._app is None: + self._sdk, self._app = await asyncio.to_thread( + _sdk.connect, self._api_key, self._app_name, runtime_threads=self._runtime_threads, sdk=self._sdk, + ) + return self._sdk, self._app + + def _auto_sleep_setting(self, sdk: Any) -> Any: + if not self._auto_sleep: + return sdk.AutoSleep.never() + if self._auto_sleep_min_idle_seconds is not None: + return sdk.AutoSleep.not_before(self._auto_sleep_min_idle_seconds) + return sdk.AutoSleep.default() + + async def create_vm( + self, + *, + image: str | None = None, + boot_mode: str | None = None, + cpu: float = 1.0, + memory: int = 8192, + disk_size_gb: float = 10, + timeout: int = 3600 * 2, + exposed_ports: list[int] | None = None, + setup_for_gateway: bool = True, + attribution: Attribution | None = None, + network_policy: NetworkPolicy | None = None, + ) -> SailSandbox: + """Create a Sailbox from the devbox image; ``timeout`` is its hard maximum lifetime.""" + if image is not None: + raise ValueError("the Sail provider boots its own Docker-capable image; image overrides are unsupported") + del boot_mode + size, memory_gib, disk_gib = sailbox_shape(cpu, memory, disk_size_gb, min_size=self._min_size) + effective_policy = self.effective_network_policy(network_policy) + if not self.supports_network_policy(effective_policy): + raise NetworkPolicyUnsupportedError( + f"Sail enforces allow-all or up to {MAX_ALLOWLIST_ENTRIES} hostname/IPv4 allowlist entries, " + f"not {effective_policy.to_dict()}" + ) + resolved_attribution = { + key: str(value) for key, value in apply_default_attribution(dict(attribution or {})).items() if value is not None + } + ports = list(dict.fromkeys(exposed_ports or [])) + sdk, app = await self._connect() + raw = await _create_or_reclaim(sdk.Sailbox.create.aio( + app=app, + image=sdk.Image.devbox("amd64"), + name=sailbox_name(resolved_attribution), + size=size, + memory_limit_gib=memory_gib, + disk_limit_gib=disk_gib, + max_lifetime_seconds=timeout, + ingress_ports=ports, + auto_sleep=self._auto_sleep_setting(sdk), + egress_policy=egress_document(effective_policy), + )) + try: + if raw.status in ("failed", "create_failed"): + raise RuntimeError(f"Sailbox {raw.sailbox_id} failed to start: {raw.error_message}") + sandbox = SailSandbox( + raw, sdk=sdk, tunnel_urls=await self._tunnel_urls(raw, ports), network_policy=effective_policy, + ) + sandbox.mode = SANDBOX_MODE_VM + logger.info( + "Sail sandbox started: sailbox_id=%s app=%s size=%s memory=%sGiB disk=%sGiB attribution=%s", + raw.sailbox_id, self._app_name, size, memory_gib, disk_gib, resolved_attribution, + extra={ + "event": SANDBOX_STARTED_EVENT, + "sail_sailbox_id": raw.sailbox_id, + "sail_app_name": self._app_name, + "sail_attribution": resolved_attribution, + PIPELINE_STEP_KEY: resolved_attribution.get(PIPELINE_STEP_KEY), + RUN_ID_KEY: resolved_attribution.get(RUN_ID_KEY), + "size": size, + "memory_limit_gib": memory_gib, + "disk_limit_gib": disk_gib, + }, + ) + if setup_for_gateway: + await sandbox.setup_vm_for_gateway(ports) + return sandbox + except BaseException: + try: + await raw.terminate.aio() + except Exception as cleanup_error: # noqa: BLE001 - cleanup must not mask the create failure + logger.warning("Failed to terminate Sailbox %s after setup failure: %s", raw.sailbox_id, cleanup_error) + raise + + @staticmethod + async def _tunnel_urls(raw: Any, ports: list[int]) -> dict[int, str]: + """Each exposed port's public URL, polling until Sail has routed all of them.""" + loop = asyncio.get_running_loop() + deadline = loop.time() + _LISTENER_TIMEOUT + while True: + urls = { + listener.guest_port: listener.endpoint.url + for listener in await raw.listeners.aio() + if listener.endpoint is not None and getattr(listener.endpoint, "url", None) + } + missing = [port for port in ports if port not in urls] + if not missing: + return {port: urls[port] for port in ports} + if loop.time() >= deadline: + raise RuntimeError(f"Sailbox {raw.sailbox_id} has no public URL for port(s) {missing} after {_LISTENER_TIMEOUT}s") + await asyncio.sleep(_LISTENER_POLL_INTERVAL) + + async def create_sandbox( + self, + *, + image_name: str, + port: int, + env: dict[str, str], + cpu: float = 1.0, + memory: int = 8192, + disk_size_gb: float = 10, + timeout: int = 3600 * 2, + attribution: Attribution | None = None, + network_policy: NetworkPolicy | None = None, + ) -> SailSandbox: + """A bare VM: the caller loads and starts ``image_name`` in it, as on the other VM providers.""" + del image_name, env + return await self.create_vm( + cpu=cpu, memory=memory, disk_size_gb=disk_size_gb, timeout=timeout, + exposed_ports=[port], attribution=attribution, network_policy=network_policy, + ) + + async def create_container(self, **kwargs: Any) -> SailSandbox: + """The inherited login-pull-run, then the registry credentials removed from the VM disk, which Sail + checkpoints for host-failure recovery.""" + sandbox = await super().create_container(**kwargs) + try: + await sandbox.exec_script("rm -f /root/.docker/config.json") + except BaseException: + await sandbox.terminate() + raise + return sandbox + + async def get_sandbox(self, sandbox_id: str) -> SailSandbox: + sdk, _ = await self._connect() + raw = await sdk.Sailbox.get.aio(sandbox_id) + tunnel_urls = { + listener.guest_port: listener.endpoint.url + for listener in await raw.listeners.aio() + if listener.endpoint is not None and getattr(listener.endpoint, "url", None) + } + applied = getattr(raw, "egress_policy", None) + policy = policy_from_document(getattr(applied, "document", None)) if getattr(applied, "policy_id", None) is None else None + if policy is None: + logger.warning( + "Sailbox %s has an egress policy agent-env can't represent (%r); image loading will fail closed", + sandbox_id, applied, + ) + return SailSandbox(raw, sdk=sdk, tunnel_urls=tunnel_urls, network_policy=policy) diff --git a/src/agent_env/providers/sandbox_providers/sail/sandbox.py b/src/agent_env/providers/sandbox_providers/sail/sandbox.py new file mode 100644 index 00000000..942f75cb --- /dev/null +++ b/src/agent_env/providers/sandbox_providers/sail/sandbox.py @@ -0,0 +1,282 @@ +"""A Sailbox (a Sail Research Linux VM with Docker) presented through the :class:`VmSandbox` contract.""" + +from __future__ import annotations + +import asyncio +import logging +import shlex +import time +import uuid +import weakref +from ipaddress import ip_network +from typing import Any, Optional +from urllib.parse import urlparse + +from agent_env.config import get_config +from agent_env.providers.sandbox_providers.sandbox import CURL_RETRY_FLAGS, NetworkMode, NetworkPolicy, VmSandbox +from agent_env.providers.sandbox_providers.sandbox_provider import SANDBOX_MODE_VM + +logger = logging.getLogger(__name__) + +#: Sail's limit on entries in one egress allowlist. +MAX_ALLOWLIST_ENTRIES = 128 + +# One lock per Sailbox and event loop, shared by every handle to it while an update is in flight. +_policy_locks: weakref.WeakValueDictionary[tuple[asyncio.AbstractEventLoop, str], asyncio.Lock] = ( + weakref.WeakValueDictionary() +) + + +def egress_document(policy: NetworkPolicy) -> dict[str, Any]: + """``policy`` as a Sail egress-policy document: ``{}`` allows everything.""" + if policy.mode is NetworkMode.ALLOW_ALL: + return {} + return {"allowlist": [*policy.allow_hosts, *policy.allow_cidrs]} + + +def policy_from_document(document: Any) -> NetworkPolicy | None: + """The agent-env policy a Sail egress document applies, or None when it can't be represented.""" + if not isinstance(document, dict): + return None + if not document: + return NetworkPolicy() + entries = document.get("allowlist") + if set(document) != {"allowlist"} or not isinstance(entries, list) or not all(isinstance(e, str) and e for e in entries): + return None + hosts, cidrs = [], [] + for entry in entries: + try: + ip_network(entry, strict=False) + except ValueError: + hosts.append(entry) + else: + cidrs.append(entry) + return NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=tuple(hosts), allow_cidrs=tuple(cidrs)) + + +def _allows_host(policy: NetworkPolicy, host: str) -> bool: + """Whether an allowlist entry already admits ``host``: an exact match, or a ``*.domain`` it sits under.""" + return any(entry == host or (entry.startswith("*.") and host.endswith(entry[1:])) for entry in policy.allow_hosts) + + +class _BytesReader: + def __init__(self, value: bytes): + self._value = value + + async def read(self) -> bytes: + return self._value + + +class _CompletedProcess: + """An exec that failed before it produced output, in the process shape ``VmSandbox`` reads.""" + + def __init__(self, stdout: bytes, stderr: bytes, exit_code: int): + self.stdout = _BytesReader(stdout) + self.stderr = _BytesReader(stderr) + self._exit_code = exit_code + + async def wait(self) -> int: + return self._exit_code + + +class _Stream: + """One output stream, pumped from the moment the exec starts so no byte is dropped: iterate it to + stream (``collect_artifacts`` does), or ``read()`` it whole. The queue is bounded, so a slow reader + pauses the command rather than growing memory; a stream nobody claims is drained by ``wait()``.""" + + _MAX_CHUNKS = 64 + + def __init__(self, chunks): + self._queue: asyncio.Queue[bytes | None] = asyncio.Queue(maxsize=self._MAX_CHUNKS) + self.claimed = False + self.error: BaseException | None = None + self.pump = asyncio.ensure_future(self._pump(chunks)) + + async def _pump(self, chunks) -> None: + try: + async for chunk in chunks: + await self._queue.put(chunk) + except Exception as exc: # noqa: BLE001 - re-raised to the reader and by wait() + self.error = exc + finally: + await self._queue.put(None) + + async def _chunks(self): + while (chunk := await self._queue.get()) is not None: + yield chunk + if self.error is not None: + raise self.error + + def __aiter__(self): + self.claimed = True + return self._chunks() + + async def read(self) -> bytes: + return b"".join([chunk async for chunk in self]) + + async def drain(self) -> None: + if not self.claimed: + self.claimed = True + async for _ in self._chunks(): + pass + + +class _SailProcess: + """A running Sail exec; a lost host or transport while it runs is exit -1, which ``exec_script`` retries.""" + + def __init__(self, process: Any, sdk: Any): + self._process = process + self._transient = (sdk.SailboxHostLostError, sdk.TransportError) + self.stdout = _Stream(process.stdout_bytes) + self.stderr = _Stream(process.stderr_bytes) + + async def wait(self) -> int: + await asyncio.gather(self.stdout.drain(), self.stderr.drain(), return_exceptions=True) + await asyncio.gather(self.stdout.pump, self.stderr.pump) + try: + exit_code = int((await self._process.wait()).exit_code) + except self._transient: + return -1 + for error in (self.stdout.error, self.stderr.error): + if isinstance(error, self._transient): + return -1 + if error is not None: + raise error + return exit_code + + +class SailSandbox(VmSandbox): + """A Sailbox from the Docker-capable devbox image; commands run as root.""" + + type = "sail" + _DOCKER_PROBE_TIMEOUT = 10 + + def __init__(self, sailbox: Any, *, sdk: Any, tunnel_urls: dict[int, str], network_policy: NetworkPolicy | None): + self._sailbox = sailbox + self._sdk = sdk + self.sandbox_id = sailbox.sailbox_id + self.tunnel_urls = tunnel_urls + self.vnc_url = None + self.mode = SANDBOX_MODE_VM + self.network_policy = network_policy + + async def terminate(self) -> None: + try: + await self._sailbox.terminate.aio() + except self._sdk.NotFoundError: + logger.info("Sailbox %s was already gone at terminate", self.sandbox_id) + + async def exec(self, *command: str) -> _SailProcess | _CompletedProcess: + """Run argv to completion. A leading ``sudo`` is dropped (commands already run as root, and the + guest's hostname doesn't resolve, so sudo warns on every call); a lost host or transport maps to + exit -1, which ``exec_script`` retries.""" + return await self._run(*command) + + async def _run(self, *command: str, timeout: Optional[int] = None) -> _SailProcess | _CompletedProcess: + argv = list(command[1:] if command[:1] == ("sudo",) else command) + try: + process = await self._sailbox.exec.aio( + argv, timeout=timeout, output_mode="pipe", idempotency_key=uuid.uuid4().hex, + ) + except (self._sdk.SailboxHostLostError, self._sdk.TransportError) as exc: + return _CompletedProcess(b"", f"{type(exc).__name__}: {exc}".encode(), -1) + return _SailProcess(process, self._sdk) + + async def _exec_with_output(self, *command: str, timeout: int) -> tuple[int, str, str]: + process = await self._run(*command, timeout=timeout) + stdout, stderr = await asyncio.gather(process.stdout.read(), process.stderr.read()) + return await process.wait(), stdout.decode(errors="replace"), stderr.decode(errors="replace") + + async def wait_for_vm(self) -> None: + """Poll ``docker info`` until the daemon answers, starting dockerd once if it is not running.""" + deadline = time.monotonic() + self._VM_READY_TIMEOUT + started = False + attempts = 0 + detail = "" + while True: + attempts += 1 + exit_code, _, stderr = await self._exec_with_output( + "docker", "info", "--format", "{{.ServerVersion}}", timeout=self._DOCKER_PROBE_TIMEOUT, + ) + if exit_code == 0: + logger.info("Docker ready in Sailbox %s after %s probe(s)", self.sandbox_id, attempts) + return + detail = stderr.strip() + if time.monotonic() >= deadline: + break + if not started: + await self.exec_script("pgrep -x dockerd > /dev/null || (nohup dockerd > /var/log/dockerd.log 2>&1 &)") + started = True + await asyncio.sleep(self._VM_READY_POLL_INTERVAL) + raise RuntimeError( + f"Docker not ready in Sailbox {self.sandbox_id} after {self._VM_READY_TIMEOUT}s ({attempts} probes): {detail[-500:]}" + ) + + async def setup_vm_for_gateway(self, exposed_ports: Optional[list[int]] = None) -> None: + """Wait for Docker and require Compose v2. Sail routes exposed ports itself, so no firewall rules.""" + await self.wait_for_vm() + exit_code, stdout, stderr = await self.exec_with_output("docker", "compose", "version") + if exit_code != 0: + raise RuntimeError(f"Sailbox {self.sandbox_id} has no Docker Compose v2: {(stderr or stdout).strip()}") + + async def _write_bytes_to_vm_path(self, data: bytes, vm_path: str) -> None: + await self._sailbox.fs.write.aio(vm_path, data) + + async def apply_network_policy(self, policy: NetworkPolicy) -> None: + """Replace the Sailbox's egress policy; applies to new connections.""" + await self._sailbox.set_egress_policy.aio(egress_document(policy)) + self.network_policy = policy + + def _known_policy(self, purpose: str) -> NetworkPolicy: + if self.network_policy is None: + raise RuntimeError( + f"Cannot {purpose} in Sailbox {self.sandbox_id}: its applied egress policy is unknown, " + "so the signed download hosts cannot be added safely" + ) + return self.network_policy + + async def _allow_download_hosts(self, urls: list[str | None], purpose: str) -> None: + """Add the hosts of signed download ``urls`` to the Sailbox's applied egress policy. Hosts the cached + policy already admits need nothing (it only ever lags the applied one); a new host re-reads the + applied policy under a per-Sailbox lock, so concurrent downloads don't drop each other's hosts. + Refuses before exceeding Sail's limit.""" + hosts = {parsed.hostname for url in urls if url and (parsed := urlparse(url)).hostname} + cached = self._known_policy(purpose) + if not cached.restricts_egress or all(_allows_host(cached, host) for host in hosts): + return + lock = _policy_locks.setdefault((asyncio.get_running_loop(), self.sandbox_id), asyncio.Lock()) + async with lock: + applied = (await self._sdk.Sailbox.get.aio(self.sandbox_id)).egress_policy + self.network_policy = policy_from_document(applied.document) if applied and applied.policy_id is None else None + policy = self._known_policy(purpose) + if not policy.restricts_egress: + return + missing = sorted(host for host in hosts if not _allows_host(policy, host)) + if not missing: + return + if len(policy.allow_hosts) + len(policy.allow_cidrs) + len(missing) > MAX_ALLOWLIST_ENTRIES: + raise RuntimeError( + f"Cannot {purpose} in Sailbox {self.sandbox_id}: adding {missing} would exceed " + f"Sail's {MAX_ALLOWLIST_ENTRIES}-entry egress allowlist" + ) + await self.apply_network_policy(policy.with_hosts(missing)) + + async def load_docker_images(self, artifacts: list) -> None: + """Load images, first adding their signed-download hosts to a restrictive policy.""" + if not artifacts: + return + self._known_policy("load Docker images") + signed_urls = await self._signed_image_urls(artifacts) + await self._allow_download_hosts(signed_urls, "load Docker images") + await self._load_docker_images(artifacts, signed_urls) + + async def _download_object_to_vm(self, object_url: str, vm_path: str) -> None: + """Download through a signed URL whose host a restrictive policy now allows, or stream the bytes + through Sail's filesystem API when the store cannot sign one.""" + object_store = get_config().get_object_store() + signed = await asyncio.to_thread(object_store.signed_get_url, object_url) + if signed is None: + await self._write_unsigned_object(object_store, object_url, vm_path) + return + await self._allow_download_hosts([signed], "download an object") + await self.exec_script(f"curl -fsSL {CURL_RETRY_FLAGS} {shlex.quote(signed)} -o {shlex.quote(vm_path)}") diff --git a/src/agent_env/providers/sandbox_providers/sandbox_provider.py b/src/agent_env/providers/sandbox_providers/sandbox_provider.py index a6aa7747..c9fdc335 100644 --- a/src/agent_env/providers/sandbox_providers/sandbox_provider.py +++ b/src/agent_env/providers/sandbox_providers/sandbox_provider.py @@ -222,6 +222,7 @@ def effective_network_policy(cls, policy: Optional[NetworkPolicy]) -> NetworkPol "modal": "agent_env.providers.sandbox_providers.modal_sandbox:ModalSandboxProvider", "modal_vm": "agent_env.providers.sandbox_providers.modal_vm_sandbox:ModalVmSandboxProvider", "e2b": "agent_env.providers.sandbox_providers.e2b:E2BSandboxProvider", + "sail": "agent_env.providers.sandbox_providers.sail:SailSandboxProvider", "local": "agent_env.providers.sandbox_providers.local_sandbox:LocalSandboxProvider", } diff --git a/tst/integration/env/gateway/gateway_test.py b/tst/integration/env/gateway/gateway_test.py index c299eae0..6f2594d0 100644 --- a/tst/integration/env/gateway/gateway_test.py +++ b/tst/integration/env/gateway/gateway_test.py @@ -446,11 +446,17 @@ def agentenv_website_env() -> WebsiteEnv: id="modal_vm", marks=skip_without_remote_sandbox("modal_vm"), ), + pytest.param( + "sail", + id="sail", + marks=skip_without_remote_sandbox("sail"), + ), ]) def sandbox_provider(request): from agent_env.providers import ( ModalSandboxProvider, ModalVmSandboxProvider, + build_sandbox_provider, reset_env_sandbox_provider, reset_sandbox_provider, set_env_sandbox_provider, @@ -463,6 +469,9 @@ def sandbox_provider(request): elif request.param == "modal_vm": set_sandbox_provider(ModalVmSandboxProvider()) set_env_sandbox_provider(ModalVmSandboxProvider()) + elif request.param == "sail": + set_sandbox_provider(build_sandbox_provider("sail")) + set_env_sandbox_provider(build_sandbox_provider("sail")) try: yield request.param finally: diff --git a/tst/integration/env/task_step/task_steps_test.py b/tst/integration/env/task_step/task_steps_test.py index 340a9208..1911a340 100644 --- a/tst/integration/env/task_step/task_steps_test.py +++ b/tst/integration/env/task_step/task_steps_test.py @@ -274,11 +274,17 @@ def echo_agent() -> A2AAgent: id="modal_vm", marks=skip_without_remote_sandbox("modal_vm"), ), + pytest.param( + "sail", + id="sail", + marks=skip_without_remote_sandbox("sail"), + ), ]) def sandbox_provider(request): from agent_env.providers import ( ModalSandboxProvider, ModalVmSandboxProvider, + build_sandbox_provider, reset_agent_sandbox_provider, reset_env_sandbox_provider, reset_sandbox_provider, @@ -295,6 +301,10 @@ def sandbox_provider(request): set_sandbox_provider(ModalVmSandboxProvider()) set_env_sandbox_provider(ModalVmSandboxProvider()) set_agent_sandbox_provider(ModalVmSandboxProvider()) + elif request.param == "sail": + set_sandbox_provider(build_sandbox_provider("sail")) + set_env_sandbox_provider(build_sandbox_provider("sail")) + set_agent_sandbox_provider(build_sandbox_provider("sail")) try: yield request.param finally: diff --git a/tst/integration/providers/sandbox_providers/sail_sandbox_smoke_test.py b/tst/integration/providers/sandbox_providers/sail_sandbox_smoke_test.py new file mode 100644 index 00000000..40e3f617 --- /dev/null +++ b/tst/integration/providers/sandbox_providers/sail_sandbox_smoke_test.py @@ -0,0 +1,87 @@ +"""Sail Sailbox smoke test. + +Runs when the resolved config can build the ``sail`` sandbox provider (``[sandbox.providers.sail.config]`` +with ``api_key = "secret:sail_api_key"``), the ``remote_sandbox`` capability, and skips with the declared +reason otherwise. The test never reads or prints the API key. +""" + +from __future__ import annotations + +import hashlib + +import httpx +import pytest +import pytest_asyncio + +from agent_env.providers.sandbox_providers.sandbox import NetworkMode, NetworkPolicy +from agent_env.providers.sandbox_providers.sandbox_provider import build_sandbox_provider +from tst.util.capabilities import skip_without_remote_sandbox + +_PORT = 8080 + +pytestmark = [ + pytest.mark.integration, + pytest.mark.int_test_slow, + pytest.mark.asyncio, + skip_without_remote_sandbox("sail"), +] + + +@pytest.fixture(scope="module") +def sail_provider(): + return build_sandbox_provider("sail") + + +@pytest_asyncio.fixture(scope="module") +async def sail_sandbox(sail_provider): + sandbox = await sail_provider.create_vm(cpu=1.0, memory=2048, exposed_ports=[_PORT], timeout=900) + try: + yield sandbox + finally: + await sandbox.terminate() + + +async def test_sailbox_runs_docker_serves_its_port_and_reconnects(sail_provider, sail_sandbox): + assert (sail_sandbox.type, sail_sandbox.mode) == ("sail", "vm") + + await sail_sandbox.exec_script( + f"docker run -d --name web -p {_PORT}:80 public.ecr.aws/nginx/nginx:alpine > /dev/null" + ) + url = sail_sandbox.tunnel_urls[_PORT] + assert url.startswith("https://") and url.endswith(".sail.box") + async with httpx.AsyncClient(timeout=30) as client: + for _ in range(30): + response = await client.get(url) + if response.status_code == 200: + break + assert response.status_code == 200 + assert "nginx" in response.text + + data = bytes(range(256)) * 4096 + await sail_sandbox.write_host_file(data, "/tmp/agent-env-smoke/blob.bin") + exit_code, digest, stderr = await sail_sandbox.exec_with_output("sha256sum", "/tmp/agent-env-smoke/blob.bin") + assert exit_code == 0, stderr + assert digest.split()[0] == hashlib.sha256(data).hexdigest() + + reconnected = await sail_provider.get_sandbox(sail_sandbox.sandbox_id) + assert reconnected.sandbox_id == sail_sandbox.sandbox_id + assert reconnected.tunnel_urls == sail_sandbox.tunnel_urls + assert reconnected.network_policy == NetworkPolicy() + + +async def test_an_allowlist_is_enforced_for_containers_in_the_sailbox(sail_provider): + policy = NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",)) + sandbox = await sail_provider.create_vm(cpu=1.0, memory=2048, exposed_ports=[], timeout=600, network_policy=policy) + try: + assert "*.sail.box" in sandbox.network_policy.allow_hosts + exit_code, out, stderr = await sandbox.exec_with_output( + "bash", "-c", + "curl -s -m 10 -o /dev/null -w '%{http_code}' https://pypi.org/simple/; echo; " + "curl -s -m 10 -o /dev/null https://example.com && echo example-reachable || echo example-blocked", + ) + assert exit_code == 0, stderr + assert out.split() == ["200", "example-blocked"] + reconnected = await sail_provider.get_sandbox(sandbox.sandbox_id) + assert reconnected.network_policy == sandbox.network_policy + finally: + await sandbox.terminate() diff --git a/tst/unit/config/test_explain_and_sources.py b/tst/unit/config/test_explain_and_sources.py index d2c33924..624a5839 100644 --- a/tst/unit/config/test_explain_and_sources.py +++ b/tst/unit/config/test_explain_and_sources.py @@ -160,8 +160,9 @@ def test_sources_says_how_the_file_was_found(config_file, tmp_path, monkeypatch) assert "walking up" in next(s for s in sources() if s.kind == KIND_FILE).detail -# Read for identity, never for configuration, so it is not a layer. -_NOT_CONFIGURATION = {"USER"} +# Read for identity (USER), or set from [sandbox.providers.sail.config] for the Sail SDK to read +# (SAIL_*), never configuration agent-env takes, so none is a layer. +_NOT_CONFIGURATION = {"USER", "SAIL_API_KEY", "SAIL_RUNTIME_THREADS"} def _env_vars_read_by(root): diff --git a/tst/unit/providers/env_providers/env_gateway_provider_test.py b/tst/unit/providers/env_providers/env_gateway_provider_test.py index 3cf1b74d..8fd8790d 100644 --- a/tst/unit/providers/env_providers/env_gateway_provider_test.py +++ b/tst/unit/providers/env_providers/env_gateway_provider_test.py @@ -548,6 +548,28 @@ async def test_modal_vm_provider_routes_to_vm_path_not_containers(): gp._deploy_via_containers.assert_not_awaited() +@pytest.mark.asyncio +async def test_sail_provider_routes_to_vm_path_not_containers(): + """A Sailbox is a Docker-capable VM: the gateway deploys onto it with docker-compose.""" + from agent_env.env.gateway import GatewayMode + from agent_env.providers.sandbox_providers.sail import SailSandboxProvider + + gp = EnvironmentGatewayProvider() + gp._deploy_via_vm = AsyncMock(return_value="VM_RESULT") + gp._deploy_via_containers = AsyncMock(return_value="CONTAINER_RESULT") + + result = await gp.create_gateway( + sandbox_provider=SailSandboxProvider(api_key="sail-test-key"), + mcp_servers=[MCPServerConfig(image="mcp-a", environment_name="a")], + mcp_server_images=[MagicMock(image_name="mcp-a")], + gateway_mode=GatewayMode.PERFORMANCE, + ttl_seconds=60, + disk_size_gb=10, + ) + assert result == "VM_RESULT" + gp._deploy_via_containers.assert_not_awaited() + + @pytest.mark.asyncio async def test_build_local_store_no_services(): """No services at all (and website_configs left unset, as the container path calls it) must diff --git a/tst/unit/providers/sandbox_providers/egress_hosts_test.py b/tst/unit/providers/sandbox_providers/egress_hosts_test.py index 6221ce85..7024b69d 100644 --- a/tst/unit/providers/sandbox_providers/egress_hosts_test.py +++ b/tst/unit/providers/sandbox_providers/egress_hosts_test.py @@ -95,4 +95,4 @@ def test_the_builtin_floor_is_pinned(builtin_registry): """Every restricted sandbox gets these, so an empty allowlist is not total denial. Pinned rather than counted: a new entry should be argued for in review. Platforms registered from config add theirs on top.""" - assert set(all_sandbox_egress_hosts()) == {"*.modal.host", "*.w.modal.host", "*.e2b.app"} + assert set(all_sandbox_egress_hosts()) == {"*.modal.host", "*.w.modal.host", "*.e2b.app", "*.sail.box"} diff --git a/tst/unit/providers/sandbox_providers/network_policy_stamping_test.py b/tst/unit/providers/sandbox_providers/network_policy_stamping_test.py index f18454ec..68f0f943 100644 --- a/tst/unit/providers/sandbox_providers/network_policy_stamping_test.py +++ b/tst/unit/providers/sandbox_providers/network_policy_stamping_test.py @@ -8,6 +8,7 @@ test_every_builtin_backend_is_covered until someone drives it here. """ +from types import SimpleNamespace from unittest.mock import AsyncMock, MagicMock, patch import pytest @@ -16,6 +17,8 @@ from agent_env.providers.sandbox_providers.local_sandbox import LocalSandboxProvider from agent_env.providers.sandbox_providers.modal_sandbox import ModalSandboxProvider from agent_env.providers.sandbox_providers.modal_vm_sandbox import ModalVmSandboxProvider +from agent_env.providers.sandbox_providers.sail import _sdk as sail_sdk +from agent_env.providers.sandbox_providers.sail.provider import SailSandboxProvider from agent_env.providers.sandbox_providers.sandbox import Sandbox from agent_env.providers.sandbox_providers.sandbox_provider import _BUILTIN_SANDBOX_PROVIDERS @@ -73,10 +76,21 @@ async def _e2b() -> Sandbox: return await provider.create_vm(exposed_ports=[], setup_for_gateway=False) +async def _sail() -> Sandbox: + sailbox = MagicMock(sailbox_id="sb_test", status="running") + sailbox.listeners.aio = AsyncMock(return_value=[]) + sdk = MagicMock() + sdk.Sailbox.create.aio = AsyncMock(return_value=sailbox) + with patch.object(sail_sdk, "connect", return_value=(sdk, SimpleNamespace(id="app_test"))): + provider = SailSandboxProvider(api_key="test-key", sdk=sdk) + return await provider.create_vm(exposed_ports=[], setup_for_gateway=False) + + FACTORIES = { "modal": _modal, "modal_vm": _modal_vm, "e2b": _e2b, + "sail": _sail, "local": _local, } diff --git a/tst/unit/providers/sandbox_providers/sail_sandbox_provider_test.py b/tst/unit/providers/sandbox_providers/sail_sandbox_provider_test.py new file mode 100644 index 00000000..b27be084 --- /dev/null +++ b/tst/unit/providers/sandbox_providers/sail_sandbox_provider_test.py @@ -0,0 +1,398 @@ +"""Unit tests for the Sail Sailbox provider; the SDK is a fake, never imported.""" + +import asyncio +import logging +import os +import sys +from types import SimpleNamespace +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from agent_env.config.errors import ConfigError +from agent_env.providers.sandbox_providers.sail import _sdk +from agent_env.providers.sandbox_providers.sail import provider as provider_module +from agent_env.providers.sandbox_providers.sail.provider import ( + SANDBOX_STARTED_EVENT, + SailSandboxProvider, + sailbox_name, + sailbox_shape, +) +from agent_env.providers.sandbox_providers.sail.sandbox import SailSandbox +from agent_env.providers.sandbox_providers.sandbox import NetworkMode, NetworkPolicy, NetworkPolicyUnsupportedError + + +class _SdkError(Exception): + pass + + +def _listener(port, url=True): + endpoint = SimpleNamespace(url=f"https://sb-1-{port}.sail.box") if url else None + return SimpleNamespace(guest_port=port, endpoint=endpoint) + + +def _sailbox(ports=(), status="running"): + sailbox = MagicMock(sailbox_id="sb_1", status=status, error_message="no capacity") + sailbox.listeners.aio = AsyncMock(return_value=[_listener(port) for port in ports]) + sailbox.terminate.aio = AsyncMock() + sailbox.egress_policy = SimpleNamespace(policy_id=None, document={}) + return sailbox + + +def _fake_sdk(sailbox): + sdk = SimpleNamespace( + App=SimpleNamespace(find=MagicMock(return_value=SimpleNamespace(id="app_1"))), + Sailbox=SimpleNamespace( + create=SimpleNamespace(aio=AsyncMock(return_value=sailbox)), + get=SimpleNamespace(aio=AsyncMock(return_value=sailbox)), + ), + Image=SimpleNamespace(devbox=MagicMock(return_value="devbox-amd64")), + AutoSleep=SimpleNamespace( + never=lambda: "never", default=lambda: "default", not_before=lambda seconds: f"not_before:{seconds}" + ), + reset_transports=MagicMock(), + NotFoundError=_SdkError, + SailboxHostLostError=_SdkError, + TransportError=_SdkError, + ) + return sdk + + +@pytest.fixture(autouse=True) +def fresh_key_state(monkeypatch): + monkeypatch.setattr(_sdk, "_installed_key", None) + monkeypatch.setattr(_sdk, "_apps", {}) + monkeypatch.delenv(_sdk.API_KEY_ENV, raising=False) + monkeypatch.delenv(_sdk.RUNTIME_THREADS_ENV, raising=False) + + +@pytest.fixture +def setup(monkeypatch): + setup = AsyncMock() + monkeypatch.setattr(SailSandbox, "setup_vm_for_gateway", setup) + return setup + + +def _provider(sdk, **config): + return SailSandboxProvider(api_key="sail-secret", sdk=sdk, **config) + + +@pytest.mark.parametrize( + ("config", "message"), + [ + ({}, "requires a non-empty 'api_key'"), + ({"api_key": " "}, "requires a non-empty 'api_key'"), + ({"api_key": "k", "app": ""}, "'app' must be a non-empty string"), + ({"api_key": "k", "min_size": "xl"}, "'min_size' must be one of"), + ({"api_key": "k", "auto_sleep": "yes"}, "'auto_sleep' must be true or false"), + ({"api_key": "k", "auto_sleep_min_idle_seconds": 0}, "from 1 to 3600"), + ({"api_key": "k", "auto_sleep_min_idle_seconds": True}, "from 1 to 3600"), + ({"api_key": "k", "runtime_threads": 257}, "from 1 to 256"), + ({"api_key": "k", "region": "us"}, "unknown key"), + ], +) +def test_from_config_rejects_invalid_config(config, message): + with pytest.raises(ConfigError, match=message): + SailSandboxProvider.from_config(**config) + + +def test_construction_neither_imports_the_sdk_nor_sets_the_key(monkeypatch): + monkeypatch.delitem(sys.modules, "sail", raising=False) + SailSandboxProvider.from_config(api_key="sail-secret") + assert "sail" not in sys.modules + assert _sdk.API_KEY_ENV not in os.environ + + +@pytest.mark.parametrize( + ("cpu", "memory", "disk", "min_size", "shape"), + [ + (1.0, 8192, 10, "s", ("s", 8, 10)), + (0.5, 1024, 1, "s", ("s", 2, 8)), + (2.0, 4096, 10, "s", ("m", 8, 32)), + (1.0, 100 * 1024, 10, "s", ("m", 100, 32)), + (1.0, 2048, 600, "s", ("l", 16, 600)), + (1.0, 1500, 10.2, "m", ("m", 8, 32)), + ], +) +def test_shape_is_the_smallest_covering_size_with_ceilings_rounded_up(cpu, memory, disk, min_size, shape): + assert sailbox_shape(cpu, memory, disk, min_size=min_size) == shape + + +@pytest.mark.parametrize(("cpu", "memory", "disk"), [(9, 8192, 10), (1, 300 * 1024, 10), (1, 8192, 2000)]) +def test_a_request_no_size_fits_is_refused(cpu, memory, disk): + with pytest.raises(ValueError, match="no Sailbox size fits"): + sailbox_shape(cpu, memory, disk) + + +def test_name_carries_slugged_attribution_in_key_order_within_128_chars(): + name = sailbox_name({"run_id": "inst-9f3c", "project_id": "p/123", "team": "env pod"}) + prefix, random, *rest = name.split("-", 2) + assert (prefix, len(random)) == ("ae", 8) + assert name.endswith("p-123-inst-9f3c-env-pod") + assert len(sailbox_name({"k": "x" * 300})) == 128 + + +@pytest.mark.asyncio +async def test_create_vm_sends_the_shape_lifetime_ports_and_policy(setup): + sailbox = _sailbox(ports=[8080, 9000]) + sdk = _fake_sdk(sailbox) + + sandbox = await _provider(sdk).create_vm( + cpu=2.0, memory=4096, disk_size_gb=20, timeout=900, exposed_ports=[8080, 9000, 8080], + attribution={"run_id": "inst-1"}, + ) + + kwargs = sdk.Sailbox.create.aio.await_args.kwargs + assert kwargs["app"].id == "app_1" + assert kwargs["image"] == "devbox-amd64" + sdk.Image.devbox.assert_called_once_with("amd64") + assert kwargs["name"].startswith("ae-") and kwargs["name"].endswith("-inst-1") + assert {k: kwargs[k] for k in ("size", "memory_limit_gib", "disk_limit_gib", "max_lifetime_seconds")} == { + "size": "m", "memory_limit_gib": 8, "disk_limit_gib": 32, "max_lifetime_seconds": 900, + } + assert kwargs["ingress_ports"] == [8080, 9000] + assert kwargs["auto_sleep"] == "never" + assert kwargs["egress_policy"] == {} + assert "api_key" not in kwargs and "env" not in kwargs + assert isinstance(sandbox, SailSandbox) + assert (sandbox.type, sandbox.mode, sandbox.sandbox_id) == ("sail", "vm", "sb_1") + assert sandbox.tunnel_urls == {8080: "https://sb-1-8080.sail.box", 9000: "https://sb-1-9000.sail.box"} + assert sandbox.network_policy == NetworkPolicy() + setup.assert_awaited_once_with([8080, 9000]) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("config", "expected"), + [({}, "never"), ({"auto_sleep": True}, "default"), ({"auto_sleep_min_idle_seconds": 600}, "not_before:600")], +) +async def test_auto_sleep_is_off_unless_configured(setup, config, expected): + sdk = _fake_sdk(_sailbox()) + await _provider(sdk, **config).create_vm(exposed_ports=[]) + assert sdk.Sailbox.create.aio.await_args.kwargs["auto_sleep"] == expected + + +@pytest.mark.asyncio +async def test_allowlist_becomes_a_sail_allowlist_with_the_platform_floor(setup): + sdk = _fake_sdk(_sailbox()) + policy = NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",), allow_cidrs=("10.0.0.0/8",)) + + sandbox = await _provider(sdk).create_vm(exposed_ports=[], network_policy=policy) + + entries = sdk.Sailbox.create.aio.await_args.kwargs["egress_policy"]["allowlist"] + assert entries[0] == "pypi.org" and entries[-1] == "10.0.0.0/8" + assert {"*.sail.box", "*.e2b.app"} <= set(entries) + assert sandbox.network_policy.allow_cidrs == ("10.0.0.0/8",) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "policy", + [ + NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_cidrs=("2001:db8::/32",)), + NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=tuple(f"h{i}.example" for i in range(130))), + ], +) +async def test_an_unenforceable_policy_is_refused_before_provisioning(policy): + sdk = _fake_sdk(_sailbox()) + assert SailSandboxProvider.supports_network_policy(policy) is False + with pytest.raises(NetworkPolicyUnsupportedError): + await _provider(sdk).create_vm(exposed_ports=[], network_policy=policy) + sdk.Sailbox.create.aio.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_a_failed_sailbox_is_terminated_and_reported(): + sailbox = _sailbox(status="failed") + with pytest.raises(RuntimeError, match="failed to start: no capacity"): + await _provider(_fake_sdk(sailbox)).create_vm(exposed_ports=[]) + sailbox.terminate.aio.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_a_setup_failure_terminates_the_sailbox(setup): + sailbox = _sailbox() + setup.side_effect = RuntimeError("docker never came up") + with pytest.raises(RuntimeError, match="docker never came up"): + await _provider(_fake_sdk(sailbox)).create_vm(exposed_ports=[]) + sailbox.terminate.aio.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_tunnel_urls_wait_until_every_port_is_routed(setup, monkeypatch): + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail.provider._LISTENER_POLL_INTERVAL", 0) + sailbox = _sailbox() + sailbox.listeners.aio = AsyncMock(side_effect=[[_listener(8080, url=False)], [_listener(8080)]]) + sandbox = await _provider(_fake_sdk(sailbox)).create_vm(exposed_ports=[8080]) + assert sandbox.tunnel_urls == {8080: "https://sb-1-8080.sail.box"} + + +@pytest.mark.asyncio +async def test_a_port_that_never_routes_fails_the_create(setup, monkeypatch): + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail.provider._LISTENER_POLL_INTERVAL", 0) + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail.provider._LISTENER_TIMEOUT", 0) + sailbox = _sailbox() + with pytest.raises(RuntimeError, match=r"no public URL for port\(s\) \[8080\]"): + await _provider(_fake_sdk(sailbox)).create_vm(exposed_ports=[8080]) + sailbox.terminate.aio.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_a_cancelled_create_terminates_the_sailbox_it_produces(): + sailbox = _sailbox() + sdk = _fake_sdk(sailbox) + created = asyncio.Event() + + async def slow_create(**_kwargs): + await created.wait() + return sailbox + + sdk.Sailbox.create.aio = slow_create + caller = asyncio.ensure_future(_provider(sdk).create_vm(exposed_ports=[])) + await asyncio.sleep(0.01) + caller.cancel() + with pytest.raises(asyncio.CancelledError): + await caller + created.set() + for _ in range(5): + await asyncio.sleep(0) + sailbox.terminate.aio.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_a_failing_orphan_termination_is_retried_then_reported(monkeypatch, caplog): + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail.provider.asyncio.sleep", AsyncMock()) + sailbox = _sailbox() + sailbox.terminate.aio = AsyncMock(side_effect=RuntimeError("api down")) + + await provider_module._reap(sailbox) + + assert sailbox.terminate.aio.await_count == provider_module._REAP_ATTEMPTS + assert "Orphaned Sailbox sb_1 is still running" in caplog.text + + +@pytest.mark.asyncio +async def test_create_logs_the_attribution_join_event_without_the_key(setup, caplog): + caplog.set_level(logging.INFO, logger="agent_env.providers.sandbox_providers.sail.provider") + await _provider(_fake_sdk(_sailbox())).create_vm(exposed_ports=[], attribution={"run_id": "inst-1", "team": "t"}) + + (record,) = [r for r in caplog.records if getattr(r, "event", None) == SANDBOX_STARTED_EVENT] + assert record.sail_sailbox_id == "sb_1" + assert record.sail_app_name == "agent-env" + assert record.sail_attribution == {"run_id": "inst-1", "team": "t"} + assert record.run_id == "inst-1" + assert "sail-secret" not in caplog.text + + +@pytest.mark.asyncio +async def test_create_sandbox_is_a_bare_vm_that_ignores_image_and_env(setup): + sdk = _fake_sdk(_sailbox(ports=[8080])) + await _provider(sdk).create_sandbox(image_name="registry.example/agent:1", port=8080, env={"TOKEN": "workload-token-value"}) + kwargs = sdk.Sailbox.create.aio.await_args.kwargs + assert kwargs["ingress_ports"] == [8080] + assert "workload-token-value" not in repr(kwargs) + + +@pytest.mark.asyncio +async def test_image_overrides_are_refused(): + with pytest.raises(ValueError, match="image overrides are unsupported"): + await _provider(_fake_sdk(_sailbox())).create_vm(image="ubuntu:22.04") + + +@pytest.mark.asyncio +async def test_create_container_removes_the_registry_login_from_the_vm(monkeypatch): + sandbox = MagicMock(spec=SailSandbox) + sandbox.exec_script = AsyncMock() + monkeypatch.setattr( + "agent_env.providers.sandbox_providers.sandbox_provider.SandboxProvider.create_container", + AsyncMock(return_value=sandbox), + ) + result = await _provider(_fake_sdk(_sailbox())).create_container(image_name="r/i:1", port=80, env={}) + assert result is sandbox + sandbox.exec_script.assert_awaited_once_with("rm -f /root/.docker/config.json") + + +@pytest.mark.asyncio +async def test_get_sandbox_restores_ports_and_the_applied_policy(): + sailbox = _sailbox(ports=[8080]) + sailbox.egress_policy = SimpleNamespace(policy_id=None, document={"allowlist": ["pypi.org", "10.0.0.0/8"]}) + sdk = _fake_sdk(sailbox) + + sandbox = await _provider(sdk).get_sandbox("sb_1") + + sdk.Sailbox.get.aio.assert_awaited_once_with("sb_1") + assert sandbox.tunnel_urls == {8080: "https://sb-1-8080.sail.box"} + assert sandbox.network_policy == NetworkPolicy( + mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",), allow_cidrs=("10.0.0.0/8",) + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "applied", + [ + None, + SimpleNamespace(policy_id="ep_1", document={}), + SimpleNamespace(policy_id=None, document={"allowlist": ["a.example"], "rules": []}), + ], +) +async def test_get_sandbox_leaves_an_unrepresentable_policy_unknown(applied): + sailbox = _sailbox() + sailbox.egress_policy = applied + sandbox = await _provider(_fake_sdk(sailbox)).get_sandbox("sb_1") + assert sandbox.network_policy is None + + +def test_the_key_is_set_only_while_the_sdk_builds_its_client(): + sdk = _fake_sdk(_sailbox()) + seen = {} + + def find(**kwargs): + seen["key"] = os.environ.get(_sdk.API_KEY_ENV) + seen["threads"] = os.environ.get(_sdk.RUNTIME_THREADS_ENV) + return SimpleNamespace(id="app_1") + + sdk.App.find = MagicMock(side_effect=find) + _sdk.connect("sail-secret", "agent-env", runtime_threads=16, sdk=sdk) + + assert seen == {"key": "sail-secret", "threads": "16"} + assert _sdk.API_KEY_ENV not in os.environ + sdk.reset_transports.assert_called_once() + + +def test_an_operator_key_is_overridden_for_the_build_and_then_restored(monkeypatch): + monkeypatch.setenv(_sdk.API_KEY_ENV, "operator-key") + sdk = _fake_sdk(_sailbox()) + sdk.App.find = MagicMock(side_effect=lambda **_: os.environ[_sdk.API_KEY_ENV]) + + _, app = _sdk.connect("sail-secret", "agent-env", sdk=sdk) + + assert app == "sail-secret" + assert os.environ[_sdk.API_KEY_ENV] == "operator-key" + + +def test_the_key_is_installed_once_per_process_and_apps_are_cached(): + sdk = _fake_sdk(_sailbox()) + for _ in range(3): + _sdk.connect("sail-secret", "agent-env", sdk=sdk) + _sdk.connect("sail-secret", "other-app", sdk=sdk) + + sdk.reset_transports.assert_called_once() + assert [c.kwargs["name"] for c in sdk.App.find.call_args_list] == ["agent-env", "other-app"] + + +def test_a_second_key_in_the_same_process_is_refused(): + sdk = _fake_sdk(_sailbox()) + _sdk.connect("sail-secret", "agent-env", sdk=sdk) + with pytest.raises(ConfigError, match="one Sail API key"): + _sdk.connect("another-key", "agent-env", sdk=sdk) + + +def test_a_rejected_key_can_be_retried(): + sdk = _fake_sdk(_sailbox()) + sdk.App.find = MagicMock(side_effect=[PermissionError("Invalid API key"), SimpleNamespace(id="app_1")]) + with pytest.raises(PermissionError): + _sdk.connect("sail-secret", "agent-env", sdk=sdk) + assert _sdk.API_KEY_ENV not in os.environ + _sdk.connect("sail-secret", "agent-env", sdk=sdk) + assert sdk.reset_transports.call_count == 2 diff --git a/tst/unit/providers/sandbox_providers/sail_sandbox_test.py b/tst/unit/providers/sandbox_providers/sail_sandbox_test.py new file mode 100644 index 00000000..d666b5b2 --- /dev/null +++ b/tst/unit/providers/sandbox_providers/sail_sandbox_test.py @@ -0,0 +1,327 @@ +"""Unit tests for the Sailbox adapter over a fake SDK Sailbox.""" + +import asyncio +from types import SimpleNamespace +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from agent_env.providers.sandbox_providers.sail.sandbox import SailSandbox, egress_document, policy_from_document +from agent_env.providers.sandbox_providers.sandbox import NetworkMode, NetworkPolicy + + +class _HostLost(Exception): + pass + + +class _NotFound(Exception): + pass + + +_SDK = SimpleNamespace(SailboxHostLostError=_HostLost, TransportError=_HostLost, NotFoundError=_NotFound) +_ALLOWLIST = NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",), allow_cidrs=("10.0.0.0/8",)) + + +async def _chunks(*parts: bytes): + for part in parts: + yield part + + +def _process(stdout=b"", stderr=b"", exit_code=0): + process = MagicMock() + process.stdout_bytes = _chunks(stdout[:3], stdout[3:]) + process.stderr_bytes = _chunks(stderr) + process.wait = AsyncMock(return_value=SimpleNamespace(exit_code=exit_code)) + return process + + +def _sandbox(*outcomes, policy=NetworkPolicy()): + sailbox = MagicMock(sailbox_id="sb_1") + sailbox.exec.aio = AsyncMock(side_effect=list(outcomes)) + sailbox.terminate.aio = AsyncMock() + sailbox.fs.write.aio = AsyncMock() + applied = {"document": egress_document(policy) if policy is not None else {"rules": []}} + + async def set_policy(document): + await asyncio.sleep(0) + applied["document"] = document + + sailbox.set_egress_policy.aio = AsyncMock(side_effect=set_policy) + sdk = SimpleNamespace(**vars(_SDK), Sailbox=SimpleNamespace(get=SimpleNamespace(aio=AsyncMock( + side_effect=lambda _id: SimpleNamespace(egress_policy=SimpleNamespace(policy_id=None, document=applied["document"])) + )))) + return SailSandbox(sailbox, sdk=sdk, tunnel_urls={}, network_policy=policy), sailbox + + +@pytest.mark.asyncio +async def test_exec_returns_exact_bytes_and_exit_code_and_drops_a_leading_sudo(): + binary = bytes(range(256)) + sandbox, sailbox = _sandbox(_process(stdout=binary, stderr=b"warn\n", exit_code=7)) + + process = await sandbox.exec("sudo", "bash", "-c", "echo 'a b'", "sudo") + + assert (await process.stdout.read(), await process.stderr.read(), await process.wait()) == (binary, b"warn\n", 7) + args, kwargs = sailbox.exec.aio.await_args + assert args == (["bash", "-c", "echo 'a b'", "sudo"],) + assert kwargs["output_mode"] == "pipe" + assert kwargs["timeout"] is None + assert len(kwargs["idempotency_key"]) == 32 + + +@pytest.mark.asyncio +async def test_stdout_streams_chunk_by_chunk_for_large_reads(): + sandbox, _ = _sandbox(_process(stdout=b"abcdefgh")) + process = await sandbox.exec("cat", "big") + assert [chunk async for chunk in process.stdout] == [b"abc", b"defgh"] + assert await process.wait() == 0 + + +@pytest.mark.asyncio +async def test_a_host_lost_while_the_command_runs_is_exit_minus_one(): + process = _process(stdout=b"partial") + process.wait = AsyncMock(side_effect=_HostLost("migrated")) + sandbox, _ = _sandbox(process) + assert (await sandbox.exec_with_output("true"))[0] == -1 + + +@pytest.mark.asyncio +async def test_output_nobody_reads_is_drained_by_wait_rather_than_blocking_the_command(): + process = MagicMock() + + async def many_chunks(): + for _ in range(1000): + yield b"x" * 1024 + + process.stdout_bytes = many_chunks() + process.stderr_bytes = _chunks(b"") + process.wait = AsyncMock(return_value=SimpleNamespace(exit_code=0)) + sandbox, _ = _sandbox(process) + + assert await asyncio.wait_for((await sandbox.exec("yes")).wait(), timeout=5) == 0 + + +async def _failing_chunks(error): + yield b"partial" + raise error + + +@pytest.mark.asyncio +async def test_a_broken_stream_is_raised_not_reported_as_success(): + process = _process() + process.stdout_bytes = _failing_chunks(ValueError("stream corrupted")) + sandbox, _ = _sandbox(process) + running = await sandbox.exec("cat", "f") + with pytest.raises(ValueError, match="stream corrupted"): + await running.stdout.read() + with pytest.raises(ValueError, match="stream corrupted"): + await running.wait() + + +@pytest.mark.asyncio +async def test_a_stream_cut_by_a_migration_is_exit_minus_one(): + process = _process() + process.stdout_bytes = _failing_chunks(_HostLost("moved")) + sandbox, _ = _sandbox(process) + running = await sandbox.exec("cat", "f") + with pytest.raises(_HostLost): + await running.stdout.read() + assert await running.wait() == -1 + + +@pytest.mark.asyncio +async def test_each_exec_gets_its_own_idempotency_key(): + sandbox, sailbox = _sandbox(_process(), _process()) + await sandbox.exec("true") + await sandbox.exec("true") + keys = [call.kwargs["idempotency_key"] for call in sailbox.exec.aio.await_args_list] + assert keys[0] != keys[1] + + +@pytest.mark.asyncio +async def test_a_lost_host_is_exit_minus_one_which_exec_script_retries(monkeypatch): + monkeypatch.setattr("agent_env.providers.sandbox_providers.sandbox.asyncio.sleep", AsyncMock()) + sandbox, sailbox = _sandbox(_HostLost("host gone"), _process(stdout=b"ok\n")) + + assert await sandbox.exec_script("echo ok", max_retries=1) == "ok\n" + assert sailbox.exec.aio.await_count == 2 + + +@pytest.mark.asyncio +async def test_other_sdk_errors_propagate(): + sandbox, _ = _sandbox(PermissionError("Invalid API key")) + with pytest.raises(PermissionError): + await sandbox.exec("true") + + +@pytest.mark.asyncio +async def test_wait_for_vm_starts_dockerd_once_when_it_is_not_running(monkeypatch): + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail.sandbox.asyncio.sleep", AsyncMock()) + sandbox, sailbox = _sandbox( + _process(stderr=b"Cannot connect to the Docker daemon", exit_code=1), + _process(), + _process(stderr=b"still starting", exit_code=1), + _process(stdout=b"20.10.24\n"), + ) + + await sandbox.wait_for_vm() + + commands = [call.args[0] for call in sailbox.exec.aio.await_args_list] + assert commands[0][:2] == ["docker", "info"] + assert commands[1][:2] == ["bash", "-c"] and "nohup dockerd" in commands[1][2] + assert [c[:2] for c in commands[2:]] == [["docker", "info"], ["docker", "info"]] + assert sailbox.exec.aio.await_args_list[0].kwargs["timeout"] == SailSandbox._DOCKER_PROBE_TIMEOUT + + +@pytest.mark.asyncio +async def test_wait_for_vm_reports_the_last_error_when_docker_never_answers(monkeypatch): + monkeypatch.setattr(SailSandbox, "_VM_READY_TIMEOUT", 0) + sandbox, _ = _sandbox(_process(stderr=b"daemon down", exit_code=1)) + with pytest.raises(RuntimeError, match="Docker not ready in Sailbox sb_1.*daemon down"): + await sandbox.wait_for_vm() + + +@pytest.mark.asyncio +async def test_setup_requires_compose_v2(monkeypatch): + monkeypatch.setattr(SailSandbox, "wait_for_vm", AsyncMock()) + sandbox, _ = _sandbox(_process(stderr=b"docker: 'compose' is not a docker command", exit_code=1)) + with pytest.raises(RuntimeError, match="no Docker Compose v2"): + await sandbox.setup_vm_for_gateway([8080]) + + +@pytest.mark.asyncio +async def test_host_files_are_written_with_the_native_filesystem_api(): + sandbox, sailbox = _sandbox(_process()) + await sandbox.write_host_file(b"\x00\x01" * 100_000, "/opt/data/blob.bin") + sailbox.fs.write.aio.assert_awaited_once_with("/opt/data/blob.bin", b"\x00\x01" * 100_000) + + +@pytest.mark.asyncio +async def test_terminate_tolerates_an_already_deleted_sailbox(): + sandbox, sailbox = _sandbox() + sailbox.terminate.aio.side_effect = _NotFound("sailbox not found") + await sandbox.terminate() + + +@pytest.mark.asyncio +async def test_image_loading_fails_closed_when_the_policy_is_unknown(): + sandbox, _ = _sandbox(policy=None) + with pytest.raises(RuntimeError, match="applied egress policy is unknown"): + await sandbox.load_docker_images([object()]) + + +@pytest.mark.asyncio +async def test_image_loading_widens_an_allowlist_with_the_signed_download_hosts(monkeypatch): + sandbox, sailbox = _sandbox(policy=_ALLOWLIST) + monkeypatch.setattr( + SailSandbox, "_signed_image_urls", + AsyncMock(return_value=["https://bucket.s3.amazonaws.com/a?sig=1", None]), + ) + load = AsyncMock() + monkeypatch.setattr(SailSandbox, "_load_docker_images", load) + + await sandbox.load_docker_images(["a", "b"]) + + sailbox.set_egress_policy.aio.assert_awaited_once_with( + {"allowlist": ["pypi.org", "bucket.s3.amazonaws.com", "10.0.0.0/8"]} + ) + assert sandbox.network_policy.allow_hosts == ("pypi.org", "bucket.s3.amazonaws.com") + load.assert_awaited_once_with(["a", "b"], ["https://bucket.s3.amazonaws.com/a?sig=1", None]) + + +@pytest.mark.asyncio +async def test_image_loading_leaves_an_allow_all_policy_alone(monkeypatch): + sandbox, sailbox = _sandbox() + monkeypatch.setattr(SailSandbox, "_signed_image_urls", AsyncMock(return_value=["https://x.example/a"])) + monkeypatch.setattr(SailSandbox, "_load_docker_images", AsyncMock()) + await sandbox.load_docker_images(["a"]) + sailbox.set_egress_policy.aio.assert_not_awaited() + sandbox._sdk.Sailbox.get.aio.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_widening_past_sails_allowlist_limit_is_refused(monkeypatch): + full = NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=tuple(f"h{i}.example" for i in range(128))) + sandbox, sailbox = _sandbox(policy=full) + monkeypatch.setattr(SailSandbox, "_signed_image_urls", AsyncMock(return_value=["https://bucket.example/a"])) + with pytest.raises(RuntimeError, match="exceed Sail's 128-entry egress allowlist"): + await sandbox.load_docker_images(["a"]) + sailbox.set_egress_policy.aio.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_a_host_under_an_allowed_wildcard_needs_no_new_entry(monkeypatch): + full = NetworkPolicy( + mode=NetworkMode.ALLOWLIST, allow_hosts=("*.s3.amazonaws.com", *(f"h{i}.example" for i in range(127))) + ) + sandbox, sailbox = _sandbox(policy=full) + monkeypatch.setattr(SailSandbox, "_signed_image_urls", AsyncMock(return_value=["https://bucket.s3.amazonaws.com/a"])) + monkeypatch.setattr(SailSandbox, "_load_docker_images", AsyncMock()) + await sandbox.load_docker_images(["a"]) + sailbox.set_egress_policy.aio.assert_not_awaited() + sandbox._sdk.Sailbox.get.aio.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_concurrent_downloads_through_separate_handles_keep_each_others_hosts(): + sandbox, sailbox = _sandbox(policy=_ALLOWLIST) + other_handle = SailSandbox(sailbox, sdk=sandbox._sdk, tunnel_urls={}, network_policy=_ALLOWLIST) + + await asyncio.gather( + sandbox._allow_download_hosts(["https://a.example/x"], "download"), + other_handle._allow_download_hosts(["https://b.example/y"], "download"), + ) + + assert set(sailbox.set_egress_policy.aio.await_args.args[0]["allowlist"]) >= {"a.example", "b.example"} + + +@pytest.mark.asyncio +async def test_a_host_already_allowed_is_not_reapplied(monkeypatch): + sandbox, sailbox = _sandbox(policy=_ALLOWLIST) + monkeypatch.setattr(SailSandbox, "_signed_image_urls", AsyncMock(return_value=["https://pypi.org/a"])) + monkeypatch.setattr(SailSandbox, "_load_docker_images", AsyncMock()) + await sandbox.load_docker_images(["a"]) + sailbox.set_egress_policy.aio.assert_not_awaited() + + +def _object_store(monkeypatch, signed): + store = MagicMock(signed_get_url=MagicMock(return_value=signed), get=MagicMock(return_value=b"\x00payload")) + monkeypatch.setattr( + "agent_env.providers.sandbox_providers.sail.sandbox.get_config", lambda: MagicMock(get_object_store=lambda: store) + ) + return store + + +@pytest.mark.asyncio +async def test_a_signed_object_download_allows_its_host_first(monkeypatch): + _object_store(monkeypatch, "https://bucket.s3.amazonaws.com/f?sig=1") + sandbox, sailbox = _sandbox(_process(), policy=_ALLOWLIST) + + await sandbox.load_s3_file("s3://bucket/f", "/tmp/f") + + sailbox.set_egress_policy.aio.assert_awaited_once_with( + {"allowlist": ["pypi.org", "bucket.s3.amazonaws.com", "10.0.0.0/8"]} + ) + script = sailbox.exec.aio.await_args.args[0][2] + assert script.startswith("curl -fsSL") and "'https://bucket.s3.amazonaws.com/f?sig=1'" in script + + +@pytest.mark.asyncio +async def test_an_unsignable_object_is_streamed_through_the_filesystem_api(monkeypatch): + _object_store(monkeypatch, None) + sandbox, sailbox = _sandbox(policy=None) + await sandbox.load_s3_file("file:///store/f", "/tmp/f") + sailbox.fs.write.aio.assert_awaited_once_with("/tmp/f", b"\x00payload") + sailbox.exec.aio.assert_not_awaited() + + +@pytest.mark.parametrize("policy", [NetworkPolicy(), _ALLOWLIST, NetworkPolicy(mode=NetworkMode.ALLOWLIST)]) +def test_egress_documents_round_trip(policy): + assert policy_from_document(egress_document(policy)) == policy + + +@pytest.mark.parametrize( + "document", + [None, {"no_network": True}, {"allowlist": ["a.example"], "blocked": ["b.example"]}, {"allowlist": "a.example"}], +) +def test_documents_agent_env_cannot_represent_are_unknown(document): + assert policy_from_document(document) is None diff --git a/tst/unit/providers/sandbox_providers/test_config_sandbox_providers.py b/tst/unit/providers/sandbox_providers/test_config_sandbox_providers.py index 8244561f..635dd78a 100644 --- a/tst/unit/providers/sandbox_providers/test_config_sandbox_providers.py +++ b/tst/unit/providers/sandbox_providers/test_config_sandbox_providers.py @@ -1,6 +1,7 @@ """Unit tests for config.toml-declared custom sandbox providers + the deploy-time type guard.""" import asyncio +import os import textwrap import pytest @@ -10,6 +11,7 @@ from agent_env.config import reset_config from agent_env.providers.sandbox_providers.e2b.provider import E2BSandboxProvider from agent_env.providers.sandbox_providers.local_sandbox import LocalSandboxProvider +from agent_env.providers.sandbox_providers.sail.provider import SailSandboxProvider from agent_env.providers.sandbox_providers.sandbox import Sandbox from agent_env.providers.sandbox_providers.sandbox_provider import ( SandboxProvider, @@ -213,6 +215,38 @@ def test_e2b_missing_base_template_is_a_config_error(monkeypatch, tmp_path): build_sandbox_provider("e2b") +def test_sail_builtin_receives_interpolated_key_without_touching_the_sdk(monkeypatch, tmp_path): + cfg = _write_config(tmp_path, """ + [sandbox.providers.sail.config] + api_key = "env:SAIL_TEST_API_KEY" + app = "agent-env-test" + auto_sleep_min_idle_seconds = 600 + """) + monkeypatch.setenv("AGENT_ENV_CONFIG", str(cfg)) + monkeypatch.setenv("SAIL_TEST_API_KEY", "resolved-sail-key") + monkeypatch.delenv("SAIL_API_KEY", raising=False) + + provider = build_sandbox_provider("sail") + + assert isinstance(provider, SailSandboxProvider) + assert provider._api_key == "resolved-sail-key" + assert provider._app_name == "agent-env-test" + assert provider._auto_sleep_min_idle_seconds == 600 + assert "SAIL_API_KEY" not in os.environ + assert "resolved-sail-key" not in repr(provider) + + +def test_sail_missing_api_key_is_a_config_error(monkeypatch, tmp_path): + cfg = _write_config(tmp_path, """ + [sandbox.providers.sail.config] + app = "agent-env-test" + """) + monkeypatch.setenv("AGENT_ENV_CONFIG", str(cfg)) + + with pytest.raises(ConfigError, match="requires a non-empty 'api_key'"): + build_sandbox_provider("sail") + + def test_builtin_config_reaches_chain_members(monkeypatch, tmp_path): cfg = _write_config(tmp_path, """ [sandbox.providers.modal.config] diff --git a/tst/unit/test_capabilities.py b/tst/unit/test_capabilities.py index 81c5816b..40c21b20 100644 --- a/tst/unit/test_capabilities.py +++ b/tst/unit/test_capabilities.py @@ -165,3 +165,18 @@ def test_e2b_present_when_the_provider_builds(monkeypatch): def test_e2b_other_failures_read_as_present_rather_than_absent(monkeypatch): _fake_e2b_builder(monkeypatch, RuntimeError("api key rejected")) assert capabilities.remote_sandbox_is_available("e2b") is True + + +@pytest.mark.parametrize( + ("outcome", "available"), + [(ConfigError("[sandbox.providers.sail.config] requires a non-empty 'api_key'"), False), (object(), True), (RuntimeError("boom"), True)], +) +def test_sail_availability_follows_the_provider_build(monkeypatch, outcome, available): + def build(spec): + assert spec == "sail" + if isinstance(outcome, Exception): + raise outcome + return outcome + + monkeypatch.setattr("agent_env.providers.sandbox_providers.sandbox_provider.build_sandbox_provider", build) + assert capabilities.remote_sandbox_is_available("sail") is available diff --git a/tst/util/capabilities.py b/tst/util/capabilities.py index e276b661..74182919 100644 --- a/tst/util/capabilities.py +++ b/tst/util/capabilities.py @@ -19,7 +19,7 @@ #: Only the agent-driving paths need a model endpoint. MODEL_ENDPOINT = "model_endpoint_configured" -#: The remote sandbox providers (``modal``, ``modal_vm``, ``e2b``) need credentials the resolved +#: The remote sandbox providers (``modal``, ``modal_vm``, ``e2b``, ``sail``) need credentials the resolved #: config may not carry; the local default never does. REMOTE_SANDBOX = "remote_sandbox" @@ -75,21 +75,21 @@ def skip_without_model_endpoint() -> pytest.MarkDecorator: def remote_sandbox_is_available(provider: str) -> bool: - """Whether the resolved config can build the ``modal`` / ``modal_vm`` / ``e2b`` sandbox + """Whether the resolved config can build the ``modal`` / ``modal_vm`` / ``e2b`` / ``sail`` sandbox provider, credentials included. - Modal: any failure to resolve the credentials answers False. E2B: building the provider - resolves ``[sandbox.providers.e2b.config]`` and its ``secret:`` references; only a + Modal: any failure to resolve the credentials answers False. E2B and Sail: building the provider + resolves ``[sandbox.providers..config]`` and its ``secret:`` references; only a ``ConfigError`` (absent or incomplete config) answers False, any other failure answers True and lets the test fail on the real problem, as ``model_endpoint_is_configured`` does.""" - if provider not in ("modal", "modal_vm", "e2b"): + if provider not in ("modal", "modal_vm", "e2b", "sail"): raise ValueError(f"unknown remote sandbox provider {provider!r}") - if provider == "e2b": + if provider in ("e2b", "sail"): from agent_env.config.errors import ConfigError from agent_env.providers.sandbox_providers.sandbox_provider import build_sandbox_provider try: - build_sandbox_provider("e2b") + build_sandbox_provider(provider) except ConfigError: return False except Exception: diff --git a/uv.lock b/uv.lock index 810e71d5..b654d527 100644 --- a/uv.lock +++ b/uv.lock @@ -62,6 +62,7 @@ dependencies = [ { name = "pymongo" }, { name = "python-multipart" }, { name = "pyyaml" }, + { name = "sail" }, { name = "starlette" }, { name = "uvicorn" }, ] @@ -131,6 +132,7 @@ requires-dist = [ { name = "python-multipart", specifier = ">=0.0.13" }, { name = "pyyaml", specifier = ">=6.0" }, { name = "requests", marker = "extra == 'gcp'", specifier = ">=2.31" }, + { name = "sail", specifier = "~=0.12.8" }, { name = "starlette", specifier = ">=0.40" }, { name = "uvicorn", specifier = ">=0.47" }, { name = "uvicorn", marker = "extra == 'explorer'", specifier = ">=0.29" }, @@ -623,6 +625,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/58/50/6c0d534c5f134586a8e1ba4e330569e32f057e33372ae556463212fb4cd3/click-8.5.0-py3-none-any.whl", hash = "sha256:255bc9599cf7748b4b1a446ccc735421bd08a2ae529a8b88597d3de5664ee360", size = 125251, upload-time = "2026-08-26T13:33:12.928Z" }, ] +[[package]] +name = "cloudpickle" +version = "3.1.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/27/fb/576f067976d320f5f0114a8d9fa1215425441bb35627b1993e5afd8111e5/cloudpickle-3.1.2.tar.gz", hash = "sha256:7fda9eb655c9c230dab534f1983763de5835249750e85fbcef43aaa30a9a2414", size = 22330, upload-time = "2025-11-03T09:25:26.604Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/88/39/799be3f2f0f38cc727ee3b4f1445fe6d5e4133064ec2e4115069418a5bb6/cloudpickle-3.1.2-py3-none-any.whl", hash = "sha256:9acb47f6afd73f60dc1df93bb801b472f05ff42fa6c84167d25cb206be1fbf4a", size = 22228, upload-time = "2025-11-03T09:25:25.534Z" }, +] + [[package]] name = "colorama" version = "0.4.6" @@ -2886,6 +2897,22 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/46/5f/4c174edad94f82de888ac00a5ddd8d07b35609b6c94f0bdf4d74af57703e/s3transfer-0.19.0-py3-none-any.whl", hash = "sha256:777cc2415536f1debadb5c2ef7779275d0fc0fe0e042411cdd6caebeb2685262", size = 90101, upload-time = "2026-06-16T19:44:50.439Z" }, ] +[[package]] +name = "sail" +version = "0.12.8" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "cloudpickle" }, + { name = "protobuf" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/51/63/8bb772e93ee9bdeab0a68a37b88b567015590468c9474b878e6f3c6fc7db/sail-0.12.8-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:43d12688928065e5dde17e72e7fea65bf49733df0a6f9819bb4760ef91d4b667", size = 15361157, upload-time = "2026-10-06T17:37:52.987Z" }, + { url = "https://files.pythonhosted.org/packages/4c/ee/b138d5fe3be97e3b6285214a76bdf2881eeb31afbab11f2f69618cd2a9ea/sail-0.12.8-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:aac5b20e428b52da2dac65534f06a0af267c67e7dc821bcc160ec7f1557c4696", size = 14649143, upload-time = "2026-10-06T17:37:56.09Z" }, + { url = "https://files.pythonhosted.org/packages/06/d1/8a2c5baf5a3ee9edee66ea65aebe1ee5237401b8b13bf8a3737622d9f094/sail-0.12.8-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:8a8dbd1f3659941caf1567859ed7285cedb2b8d5bf64c18395fe6daec635ff91", size = 16525128, upload-time = "2026-10-06T17:37:58.789Z" }, + { url = "https://files.pythonhosted.org/packages/b0/d8/11778c650c67d479f0bea86082a6edbca44b66a6520807798a3509892559/sail-0.12.8-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:9589e149a8cf1f6847042b658b17e3764b1e1366ee5b44c47a83578d496c8022", size = 16246879, upload-time = "2026-10-06T17:38:01.375Z" }, + { url = "https://files.pythonhosted.org/packages/1e/5d/6070a931a2ff9247ad8c548be7b74929262e8dee2fe613bf460dac313a97/sail-0.12.8-cp39-abi3-win_amd64.whl", hash = "sha256:c9602dcc17c69b2d783433d62d207ea30e009c4470627554abc1a5ce3a533d82", size = 13017169, upload-time = "2026-10-06T17:38:04.133Z" }, +] + [[package]] name = "setuptools" version = "84.0.0" From f57cc321156c3787981a80e6b71b7de065a1c56b Mon Sep 17 00:00:00 2001 From: Pratyush Singhal Date: Tue, 6 Oct 2026 16:46:44 -0700 Subject: [PATCH 02/11] fix(sandbox): address greptile review findings - create_container terminates the VM when cancelled during login, pull or run (the cleanup caught Exception, not CancelledError), so a cancelled deploy no longer leaves a billed VM behind. - A Sail output stream cut by a lost host or transport ends instead of raising from read(), so wait() reports exit -1 and exec_script retries. - SAIL_RUNTIME_THREADS is restored after the SDK builds its client, like SAIL_API_KEY. Co-Authored-By: Claude Opus 5.5 --- .../providers/sandbox_providers/sail/_sdk.py | 22 ++++++----- .../sandbox_providers/sail/sandbox.py | 13 ++++--- .../sandbox_providers/sandbox_provider.py | 2 +- .../sail_sandbox_provider_test.py | 12 +++--- .../sandbox_providers/sail_sandbox_test.py | 16 ++++++-- .../sandbox_provider_test.py | 39 +++++++++++++++++++ 6 files changed, 80 insertions(+), 24 deletions(-) diff --git a/src/agent_env/providers/sandbox_providers/sail/_sdk.py b/src/agent_env/providers/sandbox_providers/sail/_sdk.py index 3f3499b7..95fcb2d1 100644 --- a/src/agent_env/providers/sandbox_providers/sail/_sdk.py +++ b/src/agent_env/providers/sandbox_providers/sail/_sdk.py @@ -1,8 +1,8 @@ """The Sail SDK, imported on first use and authenticated with the provider's configured key. -The Python SDK takes its key only from ``SAIL_API_KEY``, read once when it builds its process-wide -client. The key is set for that one build and the variable restored, so subprocesses never inherit it -and a process holds one Sail key. +The Python SDK takes its key (and its thread-pool size) only from the environment, read once when it +builds its process-wide client. Both are set for that one build and restored, so subprocesses never +inherit them and a process holds one Sail key. """ from __future__ import annotations @@ -38,18 +38,20 @@ def connect(api_key: str, app_name: str, *, runtime_threads: int | None = None, fingerprint = _fingerprint(api_key) with _lock: if _installed_key is None: - previous = os.environ.get(API_KEY_ENV) - os.environ[API_KEY_ENV] = api_key + overrides = {API_KEY_ENV: api_key} if runtime_threads is not None: - os.environ[RUNTIME_THREADS_ENV] = str(runtime_threads) + overrides[RUNTIME_THREADS_ENV] = str(runtime_threads) + previous = {name: os.environ.get(name) for name in overrides} + os.environ.update(overrides) try: sdk.reset_transports() _apps[app_name] = sdk.App.find(name=app_name, mint_if_missing=True) finally: - if previous is None: - os.environ.pop(API_KEY_ENV, None) - else: - os.environ[API_KEY_ENV] = previous + for name, value in previous.items(): + if value is None: + os.environ.pop(name, None) + else: + os.environ[name] = value _installed_key = fingerprint elif _installed_key != fingerprint: raise ConfigError( diff --git a/src/agent_env/providers/sandbox_providers/sail/sandbox.py b/src/agent_env/providers/sandbox_providers/sail/sandbox.py index 942f75cb..89b63213 100644 --- a/src/agent_env/providers/sandbox_providers/sail/sandbox.py +++ b/src/agent_env/providers/sandbox_providers/sail/sandbox.py @@ -82,11 +82,14 @@ async def wait(self) -> int: class _Stream: """One output stream, pumped from the moment the exec starts so no byte is dropped: iterate it to stream (``collect_artifacts`` does), or ``read()`` it whole. The queue is bounded, so a slow reader - pauses the command rather than growing memory; a stream nobody claims is drained by ``wait()``.""" + pauses the command rather than growing memory; a stream nobody claims is drained by ``wait()``. A + transient failure just ends the stream, leaving ``wait()`` to report exit -1 for ``exec_script`` to + retry; any other is raised to the reader.""" _MAX_CHUNKS = 64 - def __init__(self, chunks): + def __init__(self, chunks, transient: tuple[type[BaseException], ...]): + self._transient = transient self._queue: asyncio.Queue[bytes | None] = asyncio.Queue(maxsize=self._MAX_CHUNKS) self.claimed = False self.error: BaseException | None = None @@ -104,7 +107,7 @@ async def _pump(self, chunks) -> None: async def _chunks(self): while (chunk := await self._queue.get()) is not None: yield chunk - if self.error is not None: + if self.error is not None and not isinstance(self.error, self._transient): raise self.error def __aiter__(self): @@ -127,8 +130,8 @@ class _SailProcess: def __init__(self, process: Any, sdk: Any): self._process = process self._transient = (sdk.SailboxHostLostError, sdk.TransportError) - self.stdout = _Stream(process.stdout_bytes) - self.stderr = _Stream(process.stderr_bytes) + self.stdout = _Stream(process.stdout_bytes, self._transient) + self.stderr = _Stream(process.stderr_bytes, self._transient) async def wait(self) -> int: await asyncio.gather(self.stdout.drain(), self.stderr.drain(), return_exceptions=True) diff --git a/src/agent_env/providers/sandbox_providers/sandbox_provider.py b/src/agent_env/providers/sandbox_providers/sandbox_provider.py index c9fdc335..61099bcb 100644 --- a/src/agent_env/providers/sandbox_providers/sandbox_provider.py +++ b/src/agent_env/providers/sandbox_providers/sandbox_provider.py @@ -123,7 +123,7 @@ async def create_container( await self._start_container(sandbox, image_name=image_name, port=port, env=env) sandbox.mode = SANDBOX_MODE_CONTAINER return sandbox - except Exception: + except BaseException: try: await sandbox.terminate() except Exception: diff --git a/tst/unit/providers/sandbox_providers/sail_sandbox_provider_test.py b/tst/unit/providers/sandbox_providers/sail_sandbox_provider_test.py index b27be084..a3a4b500 100644 --- a/tst/unit/providers/sandbox_providers/sail_sandbox_provider_test.py +++ b/tst/unit/providers/sandbox_providers/sail_sandbox_provider_test.py @@ -357,18 +357,20 @@ def find(**kwargs): assert seen == {"key": "sail-secret", "threads": "16"} assert _sdk.API_KEY_ENV not in os.environ + assert _sdk.RUNTIME_THREADS_ENV not in os.environ sdk.reset_transports.assert_called_once() -def test_an_operator_key_is_overridden_for_the_build_and_then_restored(monkeypatch): +def test_operator_settings_are_overridden_for_the_build_and_then_restored(monkeypatch): monkeypatch.setenv(_sdk.API_KEY_ENV, "operator-key") + monkeypatch.setenv(_sdk.RUNTIME_THREADS_ENV, "4") sdk = _fake_sdk(_sailbox()) - sdk.App.find = MagicMock(side_effect=lambda **_: os.environ[_sdk.API_KEY_ENV]) + sdk.App.find = MagicMock(side_effect=lambda **_: (os.environ[_sdk.API_KEY_ENV], os.environ[_sdk.RUNTIME_THREADS_ENV])) - _, app = _sdk.connect("sail-secret", "agent-env", sdk=sdk) + _, app = _sdk.connect("sail-secret", "agent-env", runtime_threads=16, sdk=sdk) - assert app == "sail-secret" - assert os.environ[_sdk.API_KEY_ENV] == "operator-key" + assert app == ("sail-secret", "16") + assert (os.environ[_sdk.API_KEY_ENV], os.environ[_sdk.RUNTIME_THREADS_ENV]) == ("operator-key", "4") def test_the_key_is_installed_once_per_process_and_apps_are_cached(): diff --git a/tst/unit/providers/sandbox_providers/sail_sandbox_test.py b/tst/unit/providers/sandbox_providers/sail_sandbox_test.py index d666b5b2..932bdfdd 100644 --- a/tst/unit/providers/sandbox_providers/sail_sandbox_test.py +++ b/tst/unit/providers/sandbox_providers/sail_sandbox_test.py @@ -118,16 +118,26 @@ async def test_a_broken_stream_is_raised_not_reported_as_success(): @pytest.mark.asyncio -async def test_a_stream_cut_by_a_migration_is_exit_minus_one(): +async def test_a_stream_cut_by_a_migration_ends_and_waits_to_exit_minus_one(): process = _process() process.stdout_bytes = _failing_chunks(_HostLost("moved")) sandbox, _ = _sandbox(process) running = await sandbox.exec("cat", "f") - with pytest.raises(_HostLost): - await running.stdout.read() + assert await running.stdout.read() == b"partial" assert await running.wait() == -1 +@pytest.mark.asyncio +async def test_exec_script_retries_a_command_whose_output_stream_was_cut(monkeypatch): + monkeypatch.setattr("agent_env.providers.sandbox_providers.sandbox.asyncio.sleep", AsyncMock()) + cut = _process() + cut.stdout_bytes = _failing_chunks(_HostLost("moved")) + sandbox, sailbox = _sandbox(cut, _process(stdout=b"loaded\n")) + + assert await sandbox.exec_script("docker load < image.tar", max_retries=1) == "loaded\n" + assert sailbox.exec.aio.await_count == 2 + + @pytest.mark.asyncio async def test_each_exec_gets_its_own_idempotency_key(): sandbox, sailbox = _sandbox(_process(), _process()) diff --git a/tst/unit/providers/sandbox_providers/sandbox_provider_test.py b/tst/unit/providers/sandbox_providers/sandbox_provider_test.py index 2457527a..3265e2ec 100644 --- a/tst/unit/providers/sandbox_providers/sandbox_provider_test.py +++ b/tst/unit/providers/sandbox_providers/sandbox_provider_test.py @@ -119,6 +119,45 @@ async def create_sandbox(self, **kwargs): assert result.mode == "container" +@pytest.mark.asyncio +async def test_a_create_container_cancelled_during_the_pull_terminates_the_vm(): + pulling = asyncio.Event() + + class _HangingVm(VmSandbox): + type = "fake-vm" + + def __init__(self): + self.sandbox_id = "vm-fake" + self.tunnel_urls = {} + self.vnc_url = None + self.mode = "vm" + self.terminated = False + + async def terminate(self) -> None: + self.terminated = True + + async def exec_script(self, script: str) -> str: + pulling.set() + await asyncio.Event().wait() + return "" + + vm = _HangingVm() + + class _VmStyleProvider(SandboxProvider): + async def create_vm(self, **kwargs): + return vm + + async def create_sandbox(self, **kwargs): + raise NotImplementedError + + task = asyncio.ensure_future(_VmStyleProvider().create_container(image_name="nginx:latest", port=8080, env={})) + await pulling.wait() + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + assert vm.terminated + + @pytest.mark.asyncio async def test_create_container_mints_the_registry_login_off_the_event_loop(): """A remote registry's login is a network round trip, an IAM token exchange for one.""" From 778e859bd58b8d2847dcec22a63efb28fae1319e Mon Sep 17 00:00:00 2001 From: Pratyush Singhal Date: Tue, 6 Oct 2026 16:56:10 -0700 Subject: [PATCH 03/11] fix(gateway): never parse a trajectory read whose exec transport failed - read_trajectory retries a read that exited -1 (lost exec transport, possibly partial output) and raises after three attempts instead of handing a partial tool-call history to the agent checks. Other exit codes keep their behaviour, so a gateway with no history is still []. - SailSandbox.exec_with_output returns no stdout for a command whose output stream was cut, so a caller that skips the exit code can't mistake partial output for the whole. Co-Authored-By: Claude Opus 5.5 --- .../env_providers/env_gateway_provider.py | 17 +++++++--- .../sandbox_providers/sail/sandbox.py | 8 +++++ .../env_gateway_provider_test.py | 32 +++++++++++++++++++ .../sandbox_providers/sail_sandbox_test.py | 8 +++++ 4 files changed, 61 insertions(+), 4 deletions(-) diff --git a/src/agent_env/providers/env_providers/env_gateway_provider.py b/src/agent_env/providers/env_providers/env_gateway_provider.py index 291d4b11..36a60252 100644 --- a/src/agent_env/providers/env_providers/env_gateway_provider.py +++ b/src/agent_env/providers/env_providers/env_gateway_provider.py @@ -46,6 +46,8 @@ logger = logging.getLogger(__name__) +_TRAJECTORY_READ_ATTEMPTS = 3 + # MCP server health-check budget. Some servers wait on an upstream before they bind; a server # that is up passes its first probe, so the long start period only delays failing a broken one. MCP_HC_RETRIES = 30 @@ -987,14 +989,21 @@ async def _wait_for_gateway(self, sandbox: VmSandbox, port: int, timeout: int = return False async def read_trajectory(self, sandbox: VmSandbox) -> list[dict]: - """Read trajectory JSONL from gateway container.""" + """Read trajectory JSONL from gateway container. A read whose exec transport failed (exit -1) may + hold only part of the history, so it is retried and then raised, never parsed.""" container_id = await self._get_container_id(sandbox, GATEWAY_SERVICE_NAME) if not container_id: return [] - exit_code, stdout, stderr = await sandbox.exec_with_output( - "sudo", "docker", "exec", container_id, "cat", "/var/log/agentenv/trajectory.jsonl" - ) + for attempt in range(_TRAJECTORY_READ_ATTEMPTS): + exit_code, stdout, stderr = await sandbox.exec_with_output( + "sudo", "docker", "exec", container_id, "cat", "/var/log/agentenv/trajectory.jsonl" + ) + if exit_code != -1: + break + logger.warning(f"Trajectory read lost its exec transport (attempt {attempt + 1}): {stderr[-200:]}") + else: + raise RuntimeError(f"Could not read the gateway trajectory: exec transport failed {_TRAJECTORY_READ_ATTEMPTS} times") events = [] for line in stdout.strip().split("\n"): if line: diff --git a/src/agent_env/providers/sandbox_providers/sail/sandbox.py b/src/agent_env/providers/sandbox_providers/sail/sandbox.py index 89b63213..53b575f5 100644 --- a/src/agent_env/providers/sandbox_providers/sail/sandbox.py +++ b/src/agent_env/providers/sandbox_providers/sail/sandbox.py @@ -175,6 +175,14 @@ async def exec(self, *command: str) -> _SailProcess | _CompletedProcess: exit -1, which ``exec_script`` retries.""" return await self._run(*command) + async def exec_with_output(self, *args: str) -> tuple[int, str, str]: + """``(exit_code, stdout, stderr)``; a command whose output stream was cut (exit -1) returns no stdout, + so a caller that skips the exit code can't mistake partial output for the whole.""" + process = await self.exec(*args) + stdout, stderr = await asyncio.gather(process.stdout.read(), process.stderr.read()) + exit_code = await process.wait() + return exit_code, "" if exit_code == -1 else stdout.decode(), stderr.decode() + async def _run(self, *command: str, timeout: Optional[int] = None) -> _SailProcess | _CompletedProcess: argv = list(command[1:] if command[:1] == ("sudo",) else command) try: diff --git a/tst/unit/providers/env_providers/env_gateway_provider_test.py b/tst/unit/providers/env_providers/env_gateway_provider_test.py index 8fd8790d..ed67affe 100644 --- a/tst/unit/providers/env_providers/env_gateway_provider_test.py +++ b/tst/unit/providers/env_providers/env_gateway_provider_test.py @@ -548,6 +548,38 @@ async def test_modal_vm_provider_routes_to_vm_path_not_containers(): gp._deploy_via_containers.assert_not_awaited() +@pytest.mark.asyncio +async def test_read_trajectory_retries_a_read_whose_transport_failed(): + gp = EnvironmentGatewayProvider() + gp._get_container_id = AsyncMock(return_value="gw") + sandbox = MagicMock() + sandbox.exec_with_output = AsyncMock(side_effect=[(-1, "", "host lost"), (0, '{"a": 1}\n{"b": 2}\n', "")]) + + assert await gp.read_trajectory(sandbox) == [{"a": 1}, {"b": 2}] + assert sandbox.exec_with_output.await_count == 2 + + +@pytest.mark.asyncio +async def test_read_trajectory_raises_rather_than_parse_a_read_that_keeps_failing(): + gp = EnvironmentGatewayProvider() + gp._get_container_id = AsyncMock(return_value="gw") + sandbox = MagicMock() + sandbox.exec_with_output = AsyncMock(return_value=(-1, '{"a": 1}\n', "host lost")) + + with pytest.raises(RuntimeError, match="exec transport failed 3 times"): + await gp.read_trajectory(sandbox) + + +@pytest.mark.asyncio +async def test_read_trajectory_of_a_gateway_with_no_history_yet_is_empty(): + gp = EnvironmentGatewayProvider() + gp._get_container_id = AsyncMock(return_value="gw") + sandbox = MagicMock() + sandbox.exec_with_output = AsyncMock(return_value=(1, "", "cat: /var/log/agentenv/trajectory.jsonl: No such file")) + + assert await gp.read_trajectory(sandbox) == [] + + @pytest.mark.asyncio async def test_sail_provider_routes_to_vm_path_not_containers(): """A Sailbox is a Docker-capable VM: the gateway deploys onto it with docker-compose.""" diff --git a/tst/unit/providers/sandbox_providers/sail_sandbox_test.py b/tst/unit/providers/sandbox_providers/sail_sandbox_test.py index 932bdfdd..253ffa38 100644 --- a/tst/unit/providers/sandbox_providers/sail_sandbox_test.py +++ b/tst/unit/providers/sandbox_providers/sail_sandbox_test.py @@ -127,6 +127,14 @@ async def test_a_stream_cut_by_a_migration_ends_and_waits_to_exit_minus_one(): assert await running.wait() == -1 +@pytest.mark.asyncio +async def test_exec_with_output_never_returns_the_partial_stdout_of_a_cut_stream(): + process = _process(stderr=b"moved\n") + process.stdout_bytes = _failing_chunks(_HostLost("moved")) + sandbox, _ = _sandbox(process) + assert await sandbox.exec_with_output("cat", "trajectory.jsonl") == (-1, "", "moved\n") + + @pytest.mark.asyncio async def test_exec_script_retries_a_command_whose_output_stream_was_cut(monkeypatch): monkeypatch.setattr("agent_env.providers.sandbox_providers.sandbox.asyncio.sleep", AsyncMock()) From d81ec1d0de1f9331a3505baffbe02768633d730c Mon Sep 17 00:00:00 2001 From: Pratyush Singhal Date: Tue, 6 Oct 2026 18:06:35 -0700 Subject: [PATCH 04/11] feat(sail): keep the agent's model key out of the Sailbox Sail injects the key into the agent's model requests as they leave the Sailbox, so neither the VM disk (which Sail checkpoints) nor its memory ever holds it. On by default; [sandbox.providers.sail.config] inject_model_key = false passes the key in as before. - create_sandbox stores the env's LITELLM_API_KEY as a Sail secret named from its SHA-256 (one key, one secret; keys never share one) and creates the Sailbox with a saved egress policy whose rules set it as the authorization and x-api-key headers for LITELLM_BASE_URL's host, which a restricted allowlist also admits. An http endpoint is refused. - Every command and file the SailSandbox sends is scrubbed of the key, so docker run/exec -e carry a placeholder. - A docker shim on the Sailbox gives every container it runs or creates the VM's CA bundle, which holds the CA Sail terminates those requests' TLS with (SSL_CERT_FILE, REQUESTS_CA_BUNDLE, NODE_EXTRA_CA_CERTS, CURL_CA_BUNDLE). - Widening the allowlist replaces the saved policy, keeping its rules. - Terminate deletes the policy, then the secret unless another Sailbox's policy still names it; a failed or cancelled create cleans up too. - Reconnect restores the injection from the applied policy, and scrubs the configured [model] key when it is the injected one. Co-Authored-By: Claude Opus 5.5 --- .../sandbox_providers/sail/model_key.py | 113 ++++++ .../sandbox_providers/sail/provider.py | 150 +++++--- .../sandbox_providers/sail/sandbox.py | 97 +++++- .../sail_sandbox_smoke_test.py | 34 ++ .../sandbox_providers/sail_model_key_test.py | 328 ++++++++++++++++++ .../sail_sandbox_provider_test.py | 8 +- 6 files changed, 665 insertions(+), 65 deletions(-) create mode 100644 src/agent_env/providers/sandbox_providers/sail/model_key.py create mode 100644 tst/unit/providers/sandbox_providers/sail_model_key_test.py diff --git a/src/agent_env/providers/sandbox_providers/sail/model_key.py b/src/agent_env/providers/sandbox_providers/sail/model_key.py new file mode 100644 index 00000000..d579cb60 --- /dev/null +++ b/src/agent_env/providers/sandbox_providers/sail/model_key.py @@ -0,0 +1,113 @@ +"""Keeping the model key out of a Sailbox: Sail adds it to the agent's model requests as they leave. + +The key is stored as a Sail secret named from its SHA-256, so one key is one secret and different keys never +share one. A saved egress policy on the Sailbox sets it as the auth headers of requests to the model endpoint's +host. Inside the Sailbox the key is replaced by a placeholder, and containers start trusting the CA Sail +terminates those requests' TLS with. +""" + +from __future__ import annotations + +import hashlib +import json +import re +from dataclasses import dataclass, field +from typing import Any, Mapping +from urllib.parse import urlparse + +SECRET_PREFIX = "AGENTENV_LITELLM_" +PLACEHOLDER = "sail-injected-model-key" +KEY_ENV = "LITELLM_API_KEY" +BASE_URL_ENV = "LITELLM_BASE_URL" + +#: The Sailbox's CA bundle, which includes the CA Sail intercepts TLS with, and where containers see it. +VM_CA_BUNDLE = "/etc/ssl/certs/ca-certificates.crt" +CONTAINER_CA_BUNDLE = "/etc/ssl/certs/sailbox-ca-bundle.crt" +CONTAINER_TRUST_ENV = { + "SSL_CERT_FILE": CONTAINER_CA_BUNDLE, + "REQUESTS_CA_BUNDLE": CONTAINER_CA_BUNDLE, + "NODE_EXTRA_CA_CERTS": CONTAINER_CA_BUNDLE, + "CURL_CA_BUNDLE": CONTAINER_CA_BUNDLE, +} +#: Ahead of /usr/bin on the Sailbox's PATH, so every ``docker run`` / ``create`` passes through it. +DOCKER_SHIM_PATH = "/usr/local/bin/docker" + +_SECRET_REF = re.compile(r"\$\{secrets\.([A-Za-z0-9_]+)\}") + + +def secret_name(key: str) -> str: + return SECRET_PREFIX + hashlib.sha256(key.encode()).hexdigest()[:32].upper() + + +def docker_shim() -> str: + """A ``docker`` wrapper that gives every container it runs or creates the Sailbox's CA bundle.""" + trust = " ".join(f"-e {name}={value}" for name, value in CONTAINER_TRUST_ENV.items()) + flags = f"-v {VM_CA_BUNDLE}:{CONTAINER_CA_BUNDLE}:ro {trust}" + return f"""#!/bin/sh +case "$1 $2" in + "container run"*|"container create"*) sub="$1 $2"; shift 2; exec /usr/bin/docker $sub {flags} "$@" ;; +esac +case "$1" in + run|create) sub="$1"; shift; exec /usr/bin/docker "$sub" {flags} "$@" ;; +esac +exec /usr/bin/docker "$@" +""" + + +@dataclass +class ModelKeyInjection: + """The model endpoint host, the Sail secret holding the key, and the saved policy applying it. + ``key`` is None on a reconnected Sailbox unless the configured key is the one injected.""" + + host: str + secret: str + key: str | None = field(default=None, repr=False) + policy_id: str | None = None + + @classmethod + def for_env(cls, env: Mapping[str, str]) -> ModelKeyInjection | None: + """The injection for an agent ``env`` carrying a model key, or None when it carries none.""" + key = env.get(KEY_ENV) + if not key or key == PLACEHOLDER: + return None + base_url = env.get(BASE_URL_ENV) or "" + parsed = urlparse(base_url) + if parsed.scheme != "https" or not parsed.hostname: + raise ValueError( + f"Sail injects the model key only into HTTPS requests, but {BASE_URL_ENV} is {base_url!r}; " + "use an https endpoint or set inject_model_key = false in [sandbox.providers.sail.config]" + ) + return cls(host=parsed.hostname, secret=secret_name(key), key=key) + + @classmethod + def from_document(cls, document: Any, policy_id: str | None) -> ModelKeyInjection | None: + """The injection a saved policy document applies, or None when its rules aren't one.""" + rules = document.get("rules") if isinstance(document, dict) else None + if not isinstance(rules, dict) or len(rules) != 1: + return None + (host,) = rules + names = set(_SECRET_REF.findall(json.dumps(rules))) + if len(names) != 1: + return None + (name,) = names + injection = cls(host=host, secret=name, policy_id=policy_id) + return injection if name.startswith(SECRET_PREFIX) and rules == injection.rules() else None + + def rules(self) -> dict[str, Any]: + """Sail egress rules setting the key as both auth headers model endpoints read.""" + ref = f"${{secrets.{self.secret}}}" + return {self.host: [{"request": {"set": {"headers": {"authorization": f"Bearer {ref}", "x-api-key": ref}}}}]} + + def matches(self, other: ModelKeyInjection | None) -> bool: + return other is not None and (other.host, other.secret) == (self.host, self.secret) + + def recover_key(self, candidate: str | None) -> None: + """Take ``candidate`` as the key when it is the one this injection's secret was named from.""" + if candidate and secret_name(candidate) == self.secret: + self.key = candidate + + def scrub(self, text: str) -> str: + return text.replace(self.key, PLACEHOLDER) if self.key else text + + def scrub_bytes(self, data: bytes) -> bytes: + return data.replace(self.key.encode(), PLACEHOLDER.encode()) if self.key else data diff --git a/src/agent_env/providers/sandbox_providers/sail/provider.py b/src/agent_env/providers/sandbox_providers/sail/provider.py index a6840bba..9eee58d6 100644 --- a/src/agent_env/providers/sandbox_providers/sail/provider.py +++ b/src/agent_env/providers/sandbox_providers/sail/provider.py @@ -7,16 +7,18 @@ import math import re import uuid -from typing import Any, ClassVar, Self +from typing import Any, Callable, ClassVar, Self from agent_env.attribution import PIPELINE_STEP_KEY, RUN_ID_KEY, Attribution +from agent_env.config import get_config from agent_env.config.errors import ConfigError from agent_env.providers.sandbox_providers.sail import _sdk +from agent_env.providers.sandbox_providers.sail.model_key import ModelKeyInjection from agent_env.providers.sandbox_providers.sail.sandbox import ( MAX_ALLOWLIST_ENTRIES, SailSandbox, + delete_saved_policy, egress_document, - policy_from_document, ) from agent_env.providers.sandbox_providers.sandbox import NetworkPolicy, NetworkPolicyUnsupportedError from agent_env.providers.sandbox_providers.sandbox_provider import ( @@ -65,43 +67,44 @@ def sailbox_name(attribution: Attribution) -> str: return "-".join(["ae", uuid.uuid4().hex[:8], *filter(None, slugs)])[:_MAX_NAME_LENGTH].rstrip("-") -async def _reap(sailbox: Any) -> None: +async def _reap(sandbox: SailSandbox) -> None: for attempt in range(_REAP_ATTEMPTS): try: - await sailbox.terminate.aio() - logger.info("Terminated Sailbox %s, created after its caller was cancelled", sailbox.sailbox_id) + await sandbox.terminate() + logger.info("Terminated Sailbox %s, created after its caller was cancelled", sandbox.sandbox_id) return except Exception as exc: # noqa: BLE001 - every failure is retried, then reported - logger.warning("Terminating orphaned Sailbox %s failed (attempt %s): %s", sailbox.sailbox_id, attempt + 1, exc) + logger.warning("Terminating orphaned Sailbox %s failed (attempt %s): %s", sandbox.sandbox_id, attempt + 1, exc) await asyncio.sleep(2 ** attempt) logger.error( "Orphaned Sailbox %s is still running after %s termination attempts; it stops at its max lifetime", - sailbox.sailbox_id, _REAP_ATTEMPTS, + sandbox.sandbox_id, _REAP_ATTEMPTS, ) -def _terminate_orphan(create: asyncio.Future) -> None: - if create.cancelled() or create.exception() is not None: - return - reaper = asyncio.ensure_future(_reap(create.result())) - _reapers.add(reaper) - reaper.add_done_callback(_reapers.discard) +async def _create_or_reclaim(create: Any, wrap: Callable[[Any], SailSandbox]) -> Any: + """Await a Sailbox create; if the caller is cancelled first, terminate (``wrap``ped, so its model-key + policy goes too) the Sailbox it yields, which would otherwise keep running with no handle.""" + task = asyncio.ensure_future(create) + def terminate_orphan(done: asyncio.Future) -> None: + if done.cancelled() or done.exception() is not None: + return + reaper = asyncio.ensure_future(_reap(wrap(done.result()))) + _reapers.add(reaper) + reaper.add_done_callback(_reapers.discard) -async def _create_or_reclaim(create: Any) -> Any: - """Await a Sailbox create; if the caller is cancelled first, terminate the Sailbox it yields, which - would otherwise keep running with no handle.""" - task = asyncio.ensure_future(create) try: return await asyncio.shield(task) except asyncio.CancelledError: - task.add_done_callback(_terminate_orphan) + task.add_done_callback(terminate_orphan) raise class SailSandboxProvider(SandboxProvider): """Docker-capable Sailboxes. ``api_key`` comes from resolved provider config (a ``secret:`` reference) - and never reaches a workload.""" + and never reaches a workload. With ``inject_model_key`` (the default) neither does an agent's model key: + Sail adds it to the agent's requests to the model endpoint (see ``model_key``).""" EGRESS_HOSTS: ClassVar[tuple[str, ...]] = ("*.sail.box",) @@ -114,6 +117,7 @@ def __init__( auto_sleep: bool = False, auto_sleep_min_idle_seconds: int | None = None, runtime_threads: int | None = None, + inject_model_key: bool = True, sdk: Any | None = None, ): self._api_key = api_key @@ -122,6 +126,7 @@ def __init__( self._auto_sleep = auto_sleep or auto_sleep_min_idle_seconds is not None self._auto_sleep_min_idle_seconds = auto_sleep_min_idle_seconds self._runtime_threads = runtime_threads + self._inject_model_key = inject_model_key self._sdk = sdk self._app: Any | None = None @@ -139,13 +144,16 @@ def from_config(cls, **config: Any) -> Self: raise ConfigError(f"{section} 'app' must be a non-empty string") if config.get("min_size", "s") not in _SIZE_NAMES: raise ConfigError(f"{section} 'min_size' must be one of {list(_SIZE_NAMES)}") - if not isinstance(config.get("auto_sleep", False), bool): - raise ConfigError(f"{section} 'auto_sleep' must be true or false") + for key in ("auto_sleep", "inject_model_key"): + if not isinstance(config.get(key, False), bool): + raise ConfigError(f"{section} '{key}' must be true or false") for key, (low, high) in {"auto_sleep_min_idle_seconds": (1, 3600), "runtime_threads": (1, 256)}.items(): value = config.get(key) if value is not None and (isinstance(value, bool) or not isinstance(value, int) or not low <= value <= high): raise ConfigError(f"{section} '{key}' must be an integer from {low} to {high}") - unknown = set(config) - {"api_key", "app", "min_size", "auto_sleep", "auto_sleep_min_idle_seconds", "runtime_threads"} + unknown = set(config) - { + "api_key", "app", "min_size", "auto_sleep", "auto_sleep_min_idle_seconds", "runtime_threads", "inject_model_key", + } if unknown: raise ConfigError(f"{section} has unknown key(s): {sorted(unknown)}") return cls(**config) @@ -192,8 +200,28 @@ async def create_vm( if image is not None: raise ValueError("the Sail provider boots its own Docker-capable image; image overrides are unsupported") del boot_mode + return await self._create( + cpu=cpu, memory=memory, disk_size_gb=disk_size_gb, timeout=timeout, exposed_ports=exposed_ports, + setup_for_gateway=setup_for_gateway, attribution=attribution, network_policy=network_policy, injection=None, + ) + + async def _create( + self, + *, + cpu: float, + memory: int, + disk_size_gb: float, + timeout: int, + exposed_ports: list[int] | None, + setup_for_gateway: bool, + attribution: Attribution | None, + network_policy: NetworkPolicy | None, + injection: ModelKeyInjection | None, + ) -> SailSandbox: size, memory_gib, disk_gib = sailbox_shape(cpu, memory, disk_size_gb, min_size=self._min_size) effective_policy = self.effective_network_policy(network_policy) + if injection is not None: + effective_policy = effective_policy.with_hosts([injection.host]) if not self.supports_network_policy(effective_policy): raise NetworkPolicyUnsupportedError( f"Sail enforces allow-all or up to {MAX_ALLOWLIST_ENTRIES} hostname/IPv4 allowlist entries, " @@ -204,28 +232,41 @@ async def create_vm( } ports = list(dict.fromkeys(exposed_ports or [])) sdk, app = await self._connect() - raw = await _create_or_reclaim(sdk.Sailbox.create.aio( - app=app, - image=sdk.Image.devbox("amd64"), - name=sailbox_name(resolved_attribution), - size=size, - memory_limit_gib=memory_gib, - disk_limit_gib=disk_gib, - max_lifetime_seconds=timeout, - ingress_ports=ports, - auto_sleep=self._auto_sleep_setting(sdk), - egress_policy=egress_document(effective_policy), - )) + egress: Any = egress_document(effective_policy) + if injection is not None: + await sdk.Secret.set.aio(injection.secret, injection.key) + egress = await sdk.EgressPolicy.create.aio(f"agentenv-{uuid.uuid4().hex}", egress_document(effective_policy, injection)) + injection.policy_id = egress.id + + def wrap(raw: Any) -> SailSandbox: + return SailSandbox(raw, sdk=sdk, tunnel_urls={}, network_policy=effective_policy, injection=injection) + + try: + raw = await _create_or_reclaim(sdk.Sailbox.create.aio( + app=app, + image=sdk.Image.devbox("amd64"), + name=sailbox_name(resolved_attribution), + size=size, + memory_limit_gib=memory_gib, + disk_limit_gib=disk_gib, + max_lifetime_seconds=timeout, + ingress_ports=ports, + auto_sleep=self._auto_sleep_setting(sdk), + egress_policy=egress, + ), wrap) + except Exception: + if injection is not None: + await delete_saved_policy(sdk, injection.policy_id) + raise + sandbox = wrap(raw) try: if raw.status in ("failed", "create_failed"): raise RuntimeError(f"Sailbox {raw.sailbox_id} failed to start: {raw.error_message}") - sandbox = SailSandbox( - raw, sdk=sdk, tunnel_urls=await self._tunnel_urls(raw, ports), network_policy=effective_policy, - ) + sandbox.tunnel_urls = await self._tunnel_urls(raw, ports) sandbox.mode = SANDBOX_MODE_VM logger.info( - "Sail sandbox started: sailbox_id=%s app=%s size=%s memory=%sGiB disk=%sGiB attribution=%s", - raw.sailbox_id, self._app_name, size, memory_gib, disk_gib, resolved_attribution, + "Sail sandbox started: sailbox_id=%s app=%s size=%s memory=%sGiB disk=%sGiB model_key_injected=%s attribution=%s", + raw.sailbox_id, self._app_name, size, memory_gib, disk_gib, injection is not None, resolved_attribution, extra={ "event": SANDBOX_STARTED_EVENT, "sail_sailbox_id": raw.sailbox_id, @@ -240,10 +281,12 @@ async def create_vm( ) if setup_for_gateway: await sandbox.setup_vm_for_gateway(ports) + if injection is not None: + await sandbox.install_container_trust() return sandbox except BaseException: try: - await raw.terminate.aio() + await sandbox.terminate() except Exception as cleanup_error: # noqa: BLE001 - cleanup must not mask the create failure logger.warning("Failed to terminate Sailbox %s after setup failure: %s", raw.sailbox_id, cleanup_error) raise @@ -279,11 +322,13 @@ async def create_sandbox( attribution: Attribution | None = None, network_policy: NetworkPolicy | None = None, ) -> SailSandbox: - """A bare VM: the caller loads and starts ``image_name`` in it, as on the other VM providers.""" - del image_name, env - return await self.create_vm( - cpu=cpu, memory=memory, disk_size_gb=disk_size_gb, timeout=timeout, - exposed_ports=[port], attribution=attribution, network_policy=network_policy, + """A bare VM the caller loads and starts ``image_name`` in, as on the other VM providers. A model key in + ``env`` is injected by Sail rather than passed in (unless ``inject_model_key`` is off).""" + del image_name + injection = ModelKeyInjection.for_env(env) if self._inject_model_key else None + return await self._create( + cpu=cpu, memory=memory, disk_size_gb=disk_size_gb, timeout=timeout, exposed_ports=[port], + setup_for_gateway=True, attribution=attribution, network_policy=network_policy, injection=injection, ) async def create_container(self, **kwargs: Any) -> SailSandbox: @@ -298,6 +343,8 @@ async def create_container(self, **kwargs: Any) -> SailSandbox: return sandbox async def get_sandbox(self, sandbox_id: str) -> SailSandbox: + """Reconnect, restoring ports, the applied egress policy and any model-key injection. The key itself + is recovered only when it is the configured ``[model]`` key, so the reconnected handle scrubs it too.""" sdk, _ = await self._connect() raw = await sdk.Sailbox.get.aio(sandbox_id) tunnel_urls = { @@ -306,10 +353,17 @@ async def get_sandbox(self, sandbox_id: str) -> SailSandbox: if listener.endpoint is not None and getattr(listener.endpoint, "url", None) } applied = getattr(raw, "egress_policy", None) - policy = policy_from_document(getattr(applied, "document", None)) if getattr(applied, "policy_id", None) is None else None - if policy is None: + injection = ModelKeyInjection.from_document(getattr(applied, "document", None), getattr(applied, "policy_id", None)) + if injection is not None: + try: + injection.recover_key(get_config().get_litellm_api_key()) + except Exception: # noqa: BLE001 - no configured key to recover; the handle just can't scrub it + pass + sandbox = SailSandbox(raw, sdk=sdk, tunnel_urls=tunnel_urls, network_policy=None, injection=injection) + sandbox.network_policy = sandbox.adopt_applied_policy(applied) + if sandbox.network_policy is None: logger.warning( "Sailbox %s has an egress policy agent-env can't represent (%r); image loading will fail closed", sandbox_id, applied, ) - return SailSandbox(raw, sdk=sdk, tunnel_urls=tunnel_urls, network_policy=policy) + return sandbox diff --git a/src/agent_env/providers/sandbox_providers/sail/sandbox.py b/src/agent_env/providers/sandbox_providers/sail/sandbox.py index 53b575f5..2c22ef7f 100644 --- a/src/agent_env/providers/sandbox_providers/sail/sandbox.py +++ b/src/agent_env/providers/sandbox_providers/sail/sandbox.py @@ -13,6 +13,7 @@ from urllib.parse import urlparse from agent_env.config import get_config +from agent_env.providers.sandbox_providers.sail.model_key import DOCKER_SHIM_PATH, ModelKeyInjection, docker_shim from agent_env.providers.sandbox_providers.sandbox import CURL_RETRY_FLAGS, NetworkMode, NetworkPolicy, VmSandbox from agent_env.providers.sandbox_providers.sandbox_provider import SANDBOX_MODE_VM @@ -27,17 +28,25 @@ ) -def egress_document(policy: NetworkPolicy) -> dict[str, Any]: - """``policy`` as a Sail egress-policy document: ``{}`` allows everything.""" - if policy.mode is NetworkMode.ALLOW_ALL: - return {} - return {"allowlist": [*policy.allow_hosts, *policy.allow_cidrs]} +def egress_document(policy: NetworkPolicy, injection: ModelKeyInjection | None = None) -> dict[str, Any]: + """``policy`` as a Sail egress-policy document (``{}`` allows everything), with ``injection``'s rules.""" + document: dict[str, Any] = {} + if policy.mode is NetworkMode.ALLOWLIST: + document["allowlist"] = [*policy.allow_hosts, *policy.allow_cidrs] + if injection is not None: + document["rules"] = injection.rules() + return document -def policy_from_document(document: Any) -> NetworkPolicy | None: - """The agent-env policy a Sail egress document applies, or None when it can't be represented.""" +def policy_from_document(document: Any, injection: ModelKeyInjection | None = None) -> NetworkPolicy | None: + """The agent-env policy a Sail egress document applies, or None when it can't be represented. Rules are + representable only as ``injection``'s, which add headers and leave egress alone.""" if not isinstance(document, dict): return None + document = dict(document) + rules = document.pop("rules", None) + if rules is not None and (injection is None or rules != injection.rules()): + return None if not document: return NetworkPolicy() entries = document.get("allowlist") @@ -59,6 +68,18 @@ def _allows_host(policy: NetworkPolicy, host: str) -> bool: return any(entry == host or (entry.startswith("*.") and host.endswith(entry[1:])) for entry in policy.allow_hosts) +async def delete_saved_policy(sdk: Any, policy_id: str | None) -> None: + """Delete a saved egress policy, best effort: a leftover one names a secret but holds no value.""" + if policy_id is None: + return + try: + await (await sdk.EgressPolicy.get.aio(policy_id)).delete.aio() + except sdk.NotFoundError: + pass + except Exception as exc: # noqa: BLE001 - reported, never masks the caller's outcome + logger.warning("Could not delete egress policy %s: %s", policy_id, exc) + + class _BytesReader: def __init__(self, value: bytes): self._value = value @@ -149,14 +170,19 @@ async def wait(self) -> int: class SailSandbox(VmSandbox): - """A Sailbox from the Docker-capable devbox image; commands run as root.""" + """A Sailbox from the Docker-capable devbox image; commands run as root. With a model-key ``injection``, + the key's value never enters the Sailbox: every command and file is scrubbed of it.""" type = "sail" _DOCKER_PROBE_TIMEOUT = 10 - def __init__(self, sailbox: Any, *, sdk: Any, tunnel_urls: dict[int, str], network_policy: NetworkPolicy | None): + def __init__( + self, sailbox: Any, *, sdk: Any, tunnel_urls: dict[int, str], network_policy: NetworkPolicy | None, + injection: ModelKeyInjection | None = None, + ): self._sailbox = sailbox self._sdk = sdk + self._injection = injection self.sandbox_id = sailbox.sailbox_id self.tunnel_urls = tunnel_urls self.vnc_url = None @@ -164,10 +190,32 @@ def __init__(self, sailbox: Any, *, sdk: Any, tunnel_urls: dict[int, str], netwo self.network_policy = network_policy async def terminate(self) -> None: + """Terminate the Sailbox, then delete its model-key policy and, unless another Sailbox's policy + still names it, the key's secret.""" try: await self._sailbox.terminate.aio() except self._sdk.NotFoundError: logger.info("Sailbox %s was already gone at terminate", self.sandbox_id) + if self._injection is not None: + await self._delete_policy(self._injection.policy_id) + await self._delete_secret(self._injection.secret) + + async def _delete_policy(self, policy_id: str | None) -> None: + await delete_saved_policy(self._sdk, policy_id) + + async def _delete_secret(self, name: str) -> None: + try: + await (await self._sdk.Secret.get.aio(name)).delete.aio() + except self._sdk.SecretInUseError: + logger.info("Model-key secret %s is still used by another Sailbox; leaving it", name) + except self._sdk.NotFoundError: + pass + except Exception as exc: # noqa: BLE001 - the next Sailbox using this key re-sets it; report and go on + logger.warning("Could not delete model-key secret %s: %s", name, exc) + + async def install_container_trust(self) -> None: + """Have every container started on this Sailbox trust the CA Sail injects the model key behind.""" + await self._sailbox.fs.write.aio(DOCKER_SHIM_PATH, docker_shim(), mode=0o755) async def exec(self, *command: str) -> _SailProcess | _CompletedProcess: """Run argv to completion. A leading ``sudo`` is dropped (commands already run as root, and the @@ -185,6 +233,8 @@ async def exec_with_output(self, *args: str) -> tuple[int, str, str]: async def _run(self, *command: str, timeout: Optional[int] = None) -> _SailProcess | _CompletedProcess: argv = list(command[1:] if command[:1] == ("sudo",) else command) + if self._injection is not None: + argv = [self._injection.scrub(arg) for arg in argv] try: process = await self._sailbox.exec.aio( argv, timeout=timeout, output_mode="pipe", idempotency_key=uuid.uuid4().hex, @@ -231,13 +281,35 @@ async def setup_vm_for_gateway(self, exposed_ports: Optional[list[int]] = None) raise RuntimeError(f"Sailbox {self.sandbox_id} has no Docker Compose v2: {(stderr or stdout).strip()}") async def _write_bytes_to_vm_path(self, data: bytes, vm_path: str) -> None: + if self._injection is not None: + data = self._injection.scrub_bytes(data) await self._sailbox.fs.write.aio(vm_path, data) async def apply_network_policy(self, policy: NetworkPolicy) -> None: - """Replace the Sailbox's egress policy; applies to new connections.""" - await self._sailbox.set_egress_policy.aio(egress_document(policy)) + """Replace the Sailbox's egress policy; applies to new connections. A model-key injection needs a + saved policy (only those can name a secret), so a new one replaces the old, which is deleted.""" + if self._injection is None: + await self._sailbox.set_egress_policy.aio(egress_document(policy)) + else: + saved = await self._sdk.EgressPolicy.create.aio(f"agentenv-{uuid.uuid4().hex}", egress_document(policy, self._injection)) + await self._sailbox.set_egress_policy.aio(saved) + previous, self._injection.policy_id = self._injection.policy_id, saved.id + await self._delete_policy(previous) self.network_policy = policy + def adopt_applied_policy(self, applied: Any) -> NetworkPolicy | None: + """The agent-env policy Sail reports applied (``Sailbox.egress_policy``), or None when it can't be + represented: a saved policy counts only when it carries this Sailbox's model-key injection.""" + if applied is None: + return None + document = getattr(applied, "document", None) + if getattr(applied, "policy_id", None) is not None: + saved = ModelKeyInjection.from_document(document, applied.policy_id) + if self._injection is None or not self._injection.matches(saved): + return None + self._injection.policy_id = applied.policy_id + return policy_from_document(document, self._injection) + def _known_policy(self, purpose: str) -> NetworkPolicy: if self.network_policy is None: raise RuntimeError( @@ -257,8 +329,7 @@ async def _allow_download_hosts(self, urls: list[str | None], purpose: str) -> N return lock = _policy_locks.setdefault((asyncio.get_running_loop(), self.sandbox_id), asyncio.Lock()) async with lock: - applied = (await self._sdk.Sailbox.get.aio(self.sandbox_id)).egress_policy - self.network_policy = policy_from_document(applied.document) if applied and applied.policy_id is None else None + self.network_policy = self.adopt_applied_policy((await self._sdk.Sailbox.get.aio(self.sandbox_id)).egress_policy) policy = self._known_policy(purpose) if not policy.restricts_egress: return diff --git a/tst/integration/providers/sandbox_providers/sail_sandbox_smoke_test.py b/tst/integration/providers/sandbox_providers/sail_sandbox_smoke_test.py index 40e3f617..9fda2f68 100644 --- a/tst/integration/providers/sandbox_providers/sail_sandbox_smoke_test.py +++ b/tst/integration/providers/sandbox_providers/sail_sandbox_smoke_test.py @@ -8,11 +8,14 @@ from __future__ import annotations import hashlib +import json +import secrets import httpx import pytest import pytest_asyncio +from agent_env.providers.sandbox_providers.sail.model_key import PLACEHOLDER, secret_name from agent_env.providers.sandbox_providers.sandbox import NetworkMode, NetworkPolicy from agent_env.providers.sandbox_providers.sandbox_provider import build_sandbox_provider from tst.util.capabilities import skip_without_remote_sandbox @@ -85,3 +88,34 @@ async def test_an_allowlist_is_enforced_for_containers_in_the_sailbox(sail_provi assert reconnected.network_policy == sandbox.network_policy finally: await sandbox.terminate() + + +_ECHO_HOST = "httpbin.org" +_CLIENT = ( + "import json, os, urllib.request; " + f"r = urllib.request.Request('https://{_ECHO_HOST}/headers', headers={{'Authorization': 'Bearer ' + os.environ['LITELLM_API_KEY']}}); " + "print(json.loads(urllib.request.urlopen(r, timeout=20).read())['headers']['Authorization'])" +) + + +async def test_an_agents_model_key_is_injected_by_sail_and_never_enters_the_sailbox(sail_provider): + """A throwaway key against a header-echo host stands in for the model endpoint.""" + key = f"sk-agentenv-smoke-{secrets.token_hex(16)}" + env = {"LITELLM_API_KEY": key, "LITELLM_BASE_URL": f"https://{_ECHO_HOST}/v1"} + sandbox = await sail_provider.create_sandbox(image_name="unused", port=_PORT, env=env, cpu=1.0, memory=2048, timeout=900) + try: + flags = " ".join(f"-e {name}='{value}'" for name, value in env.items()) + await sandbox.exec_script(f"docker run -d --name agent-api {flags} public.ecr.aws/docker/library/python:3.12-slim sleep 600 > /dev/null") + exit_code, sent, stderr = await sandbox.exec_with_output("docker", "exec", "agent-api", "python", "-c", _CLIENT) + assert exit_code == 0, stderr + assert sent.strip() == f"Bearer {key}" + + _, inside, _ = await sandbox.exec_with_output("docker", "exec", "agent-api", "printenv", "LITELLM_API_KEY") + assert inside.strip() == PLACEHOLDER + _, config, _ = await sandbox.exec_with_output("docker", "inspect", "agent-api") + assert key not in config and PLACEHOLDER in json.dumps(json.loads(config)[0]["Config"]["Env"]) + finally: + await sandbox.terminate() + sdk = sandbox._sdk + with pytest.raises(sdk.NotFoundError): + await sdk.Secret.get.aio(secret_name(key)) diff --git a/tst/unit/providers/sandbox_providers/sail_model_key_test.py b/tst/unit/providers/sandbox_providers/sail_model_key_test.py new file mode 100644 index 00000000..0d4c2b4e --- /dev/null +++ b/tst/unit/providers/sandbox_providers/sail_model_key_test.py @@ -0,0 +1,328 @@ +"""Model-key injection on Sailboxes: the key stays at Sail, the Sailbox only ever holds a placeholder.""" + +import logging +import subprocess +from types import SimpleNamespace +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from agent_env.providers.sandbox_providers.sail import _sdk +from agent_env.providers.sandbox_providers.sail import model_key +from agent_env.providers.sandbox_providers.sail.model_key import ( + DOCKER_SHIM_PATH, + PLACEHOLDER, + ModelKeyInjection, + docker_shim, + secret_name, +) +from agent_env.providers.sandbox_providers.sail.provider import SailSandboxProvider +from agent_env.providers.sandbox_providers.sail.sandbox import SailSandbox +from agent_env.providers.sandbox_providers.sandbox import NetworkMode, NetworkPolicy + +_KEY = "sk-live-model-key-0123456789" +_ENV = {"LITELLM_API_KEY": _KEY, "LITELLM_BASE_URL": "https://llm.example.com/v1", "A2A_PORT": "8000"} + + +class _SecretInUse(Exception): + pass + + +class _NotFound(Exception): + pass + + +class _HostLost(Exception): + pass + + +def _fake_sdk(sailbox): + saved = [] + + async def create_policy(name, document): + policy = MagicMock(id=f"ep_{len(saved) + 1}", document=document) + policy.name = name + policy.delete.aio = AsyncMock() + saved.append(policy) + return policy + + secret = MagicMock() + secret.delete.aio = AsyncMock() + sdk = SimpleNamespace( + App=SimpleNamespace(find=MagicMock(return_value=SimpleNamespace(id="app_1"))), + Sailbox=SimpleNamespace( + create=SimpleNamespace(aio=AsyncMock(return_value=sailbox)), + get=SimpleNamespace(aio=AsyncMock(return_value=sailbox)), + ), + Image=SimpleNamespace(devbox=MagicMock(return_value="devbox-amd64")), + AutoSleep=SimpleNamespace(never=lambda: "never", default=lambda: "default", not_before=lambda s: s), + Secret=SimpleNamespace(set=SimpleNamespace(aio=AsyncMock()), get=SimpleNamespace(aio=AsyncMock(return_value=secret))), + EgressPolicy=SimpleNamespace( + create=SimpleNamespace(aio=AsyncMock(side_effect=create_policy)), + get=SimpleNamespace(aio=AsyncMock(side_effect=lambda policy_id: next(p for p in saved if p.id == policy_id))), + ), + reset_transports=MagicMock(), + NotFoundError=_NotFound, + SecretInUseError=_SecretInUse, + SailboxHostLostError=_HostLost, + TransportError=_HostLost, + ) + return sdk, saved, secret + + +def _sailbox(ports=(8000,)): + sailbox = MagicMock(sailbox_id="sb_1", status="running") + sailbox.listeners.aio = AsyncMock(return_value=[ + SimpleNamespace(guest_port=p, endpoint=SimpleNamespace(url=f"https://sb-1-{p}.sail.box")) for p in ports + ]) + sailbox.terminate.aio = AsyncMock() + sailbox.fs.write.aio = AsyncMock() + sailbox.set_egress_policy.aio = AsyncMock() + return sailbox + + +@pytest.fixture(autouse=True) +def fresh_key_state(monkeypatch): + monkeypatch.setattr(_sdk, "_installed_key", None) + monkeypatch.setattr(_sdk, "_apps", {}) + + +@pytest.fixture(autouse=True) +def no_vm_setup(monkeypatch): + monkeypatch.setattr(SailSandbox, "setup_vm_for_gateway", AsyncMock()) + + +def _process(stdout=b""): + async def chunks(data): + yield data + + process = MagicMock() + process.stdout_bytes = chunks(stdout) + process.stderr_bytes = chunks(b"") + process.wait = AsyncMock(return_value=SimpleNamespace(exit_code=0)) + return process + + +def test_secrets_are_named_from_the_key_so_keys_never_share_one(): + assert secret_name(_KEY) == secret_name(_KEY) + assert secret_name(_KEY) != secret_name(_KEY + "x") + assert secret_name(_KEY).startswith("AGENTENV_LITELLM_") and _KEY not in secret_name(_KEY) + + +def test_an_env_without_a_model_key_injects_nothing(): + assert ModelKeyInjection.for_env({"LITELLM_BASE_URL": "https://llm.example.com"}) is None + assert ModelKeyInjection.for_env({"LITELLM_API_KEY": PLACEHOLDER}) is None + + +@pytest.mark.parametrize("base_url", ["http://llm.internal:4000", "", "llm.example.com"]) +def test_injection_needs_an_https_model_endpoint(base_url): + with pytest.raises(ValueError, match="only into HTTPS requests"): + ModelKeyInjection.for_env({"LITELLM_API_KEY": _KEY, "LITELLM_BASE_URL": base_url}) + + +def test_rules_set_both_auth_headers_from_the_secret_and_never_hold_the_key(): + injection = ModelKeyInjection.for_env(_ENV) + ref = f"${{secrets.{secret_name(_KEY)}}}" + assert injection.rules() == { + "llm.example.com": [{"request": {"set": {"headers": {"authorization": f"Bearer {ref}", "x-api-key": ref}}}}] + } + assert _KEY not in repr(injection) and _KEY not in str(injection.rules()) + + +def test_an_injection_is_recovered_from_its_saved_policy_only(): + injection = ModelKeyInjection.for_env(_ENV) + recovered = ModelKeyInjection.from_document({"allowlist": ["x.example"], "rules": injection.rules()}, "ep_1") + assert injection.matches(recovered) and recovered.policy_id == "ep_1" and recovered.key is None + foreign = {"rules": {"api.github.com": [{"request": {"set": {"headers": {"authorization": "Bearer ${secrets.GITHUB_TOKEN}"}}}}]}} + assert ModelKeyInjection.from_document(foreign, "ep_2") is None + assert ModelKeyInjection.from_document({"allowlist": []}, "ep_3") is None + + +def test_the_key_is_recovered_only_when_it_named_the_secret(): + recovered = ModelKeyInjection.from_document({"rules": ModelKeyInjection.for_env(_ENV).rules()}, "ep_1") + recovered.recover_key("some-other-key") + assert recovered.key is None + recovered.recover_key(_KEY) + assert recovered.scrub(f"-e LITELLM_API_KEY='{_KEY}'") == f"-e LITELLM_API_KEY='{PLACEHOLDER}'" + + +@pytest.mark.parametrize( + ("args", "expected_prefix"), + [ + (["run", "-d", "--name", "agent-api", "img"], ["run", "-v"]), + (["create", "--name", "agent-api", "img"], ["create", "-v"]), + (["container", "run", "img"], ["container", "run", "-v"]), + (["exec", "agent-api", "ls"], ["exec", "agent-api", "ls"]), + (["load"], ["load"]), + ], +) +def test_the_docker_shim_adds_the_ca_bundle_only_to_containers_it_starts(tmp_path, args, expected_prefix): + shim = tmp_path / "docker" + shim.write_text(docker_shim().replace("/usr/bin/docker", "printf '%s\\n'")) + shim.chmod(0o755) + argv = subprocess.run([str(shim), *args], capture_output=True, text=True, check=True).stdout.split("\n")[:-1] + assert argv[: len(expected_prefix)] == expected_prefix + trusts = f"{model_key.VM_CA_BUNDLE}:{model_key.CONTAINER_CA_BUNDLE}:ro" in argv + assert trusts is (expected_prefix[-1] == "-v") + if trusts: + assert "NODE_EXTRA_CA_CERTS=/etc/ssl/certs/sailbox-ca-bundle.crt" in argv + assert argv[-len(args) + (2 if args[0] == "container" else 1):] == args[(2 if args[0] == "container" else 1):] + + +@pytest.mark.asyncio +async def test_an_agent_sailbox_gets_its_key_through_a_secret_and_a_saved_policy(caplog): + caplog.set_level(logging.DEBUG) + sailbox = _sailbox() + sdk, saved, _ = _fake_sdk(sailbox) + + sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + + sdk.Secret.set.aio.assert_awaited_once_with(secret_name(_KEY), _KEY) + (policy,) = saved + assert policy.document == {"rules": ModelKeyInjection.for_env(_ENV).rules()} + kwargs = sdk.Sailbox.create.aio.await_args.kwargs + assert kwargs["egress_policy"] is policy + assert _KEY not in repr(kwargs) + sailbox.fs.write.aio.assert_awaited_once_with(DOCKER_SHIM_PATH, docker_shim(), mode=0o755) + assert sandbox._injection.policy_id == policy.id + assert _KEY not in caplog.text + + +@pytest.mark.asyncio +async def test_a_restricted_agent_sailbox_allows_the_model_endpoint_it_injects_into(): + sdk, saved, _ = _fake_sdk(_sailbox()) + restricted = NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",)) + + sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( + image_name="agent:1", port=8000, env=_ENV, network_policy=restricted, + ) + + assert "llm.example.com" in saved[0].document["allowlist"] + assert "llm.example.com" in sandbox.network_policy.allow_hosts + + +@pytest.mark.asyncio +async def test_injection_can_be_turned_off_to_pass_the_key_in(): + sdk, saved, _ = _fake_sdk(_sailbox()) + sandbox = await SailSandboxProvider(api_key="sail-key", inject_model_key=False, sdk=sdk).create_sandbox( + image_name="agent:1", port=8000, env={**_ENV, "LITELLM_BASE_URL": "http://llm.internal:4000"}, + ) + sdk.Secret.set.aio.assert_not_awaited() + assert saved == [] and sandbox._injection is None + assert sdk.Sailbox.create.aio.await_args.kwargs["egress_policy"] == {} + + +@pytest.mark.asyncio +async def test_a_plain_endpoint_is_refused_before_anything_is_created(): + sdk, saved, _ = _fake_sdk(_sailbox()) + with pytest.raises(ValueError, match="only into HTTPS requests"): + await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( + image_name="agent:1", port=8000, env={**_ENV, "LITELLM_BASE_URL": "http://llm.internal:4000"}, + ) + sdk.Secret.set.aio.assert_not_awaited() + sdk.Sailbox.create.aio.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_a_failed_create_deletes_the_policy_it_made(): + sdk, saved, _ = _fake_sdk(_sailbox()) + sdk.Sailbox.create.aio = AsyncMock(side_effect=RuntimeError("no capacity")) + with pytest.raises(RuntimeError, match="no capacity"): + await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + saved[0].delete.aio.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_gateway_and_plain_vms_never_inject(): + sdk, saved, _ = _fake_sdk(_sailbox(ports=())) + sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_vm(exposed_ports=[]) + assert saved == [] and sandbox._injection is None + + +@pytest.mark.asyncio +async def test_commands_and_files_carry_the_placeholder_never_the_key(): + sailbox = _sailbox() + sailbox.exec.aio = AsyncMock(return_value=_process()) + sdk, _, _ = _fake_sdk(sailbox) + sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + + await sandbox.exec_script(f"docker run -d -e LITELLM_API_KEY='{_KEY}' -e ANTHROPIC_API_KEY={_KEY} agent:1") + await sandbox.write_host_file(f"key: {_KEY}\n".encode(), "/opt/agent/config.yaml") + + sent = [arg for call in sailbox.exec.aio.await_args_list for arg in call.args[0]] + (script,) = [arg for arg in sent if "docker run" in arg] + assert script.count(PLACEHOLDER) == 2 + assert not any(_KEY in arg for arg in sent) + assert sailbox.fs.write.aio.await_args.args == ("/opt/agent/config.yaml", f"key: {PLACEHOLDER}\n".encode()) + + +@pytest.mark.asyncio +async def test_terminate_deletes_the_policy_then_the_secret(): + sailbox = _sailbox() + sdk, saved, secret = _fake_sdk(sailbox) + sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + + await sandbox.terminate() + + sailbox.terminate.aio.assert_awaited_once() + saved[0].delete.aio.assert_awaited_once() + sdk.Secret.get.aio.assert_awaited_once_with(secret_name(_KEY)) + secret.delete.aio.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_a_secret_another_sailbox_still_uses_is_left_in_place(caplog): + caplog.set_level(logging.INFO) + sdk, _, secret = _fake_sdk(_sailbox()) + secret.delete.aio = AsyncMock(side_effect=_SecretInUse("in use")) + sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + + await sandbox.terminate() + + assert "still used by another Sailbox" in caplog.text + + +@pytest.mark.asyncio +async def test_widening_an_injected_sailbox_replaces_its_saved_policy(): + sailbox = _sailbox() + sdk, saved, _ = _fake_sdk(sailbox) + sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( + image_name="agent:1", port=8000, env=_ENV, + network_policy=NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",)), + ) + + await sandbox.apply_network_policy(sandbox.network_policy.with_hosts(["bucket.example"])) + + first, second = saved + assert "bucket.example" in second.document["allowlist"] and second.document["rules"] == first.document["rules"] + sailbox.set_egress_policy.aio.assert_awaited_once_with(second) + first.delete.aio.assert_awaited_once() + assert sandbox._injection.policy_id == second.id + + +@pytest.mark.asyncio +async def test_reconnect_restores_the_injection_and_scrubs_the_configured_key(monkeypatch): + injection = ModelKeyInjection.for_env(_ENV) + sailbox = _sailbox() + sailbox.egress_policy = SimpleNamespace(policy_id="ep_9", document={"allowlist": ["llm.example.com"], "rules": injection.rules()}) + sailbox.exec.aio = AsyncMock(return_value=_process()) + sdk, _, _ = _fake_sdk(sailbox) + monkeypatch.setattr( + "agent_env.providers.sandbox_providers.sail.provider.get_config", lambda: MagicMock(get_litellm_api_key=lambda: _KEY) + ) + + sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).get_sandbox("sb_1") + + assert sandbox.network_policy == NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("llm.example.com",)) + assert sandbox._injection.policy_id == "ep_9" + await sandbox.exec_with_output("docker", "exec", "-e", f"LITELLM_API_KEY={_KEY}", "agent-api", "pytest") + assert sailbox.exec.aio.await_args.args[0][3] == f"LITELLM_API_KEY={PLACEHOLDER}" + + +@pytest.mark.asyncio +async def test_reconnect_treats_someone_elses_saved_policy_as_unknown(): + sailbox = _sailbox() + sailbox.egress_policy = SimpleNamespace(policy_id="ep_7", document={"allowlist": ["a.example"]}) + sdk, _, _ = _fake_sdk(sailbox) + sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).get_sandbox("sb_1") + assert sandbox.network_policy is None and sandbox._injection is None diff --git a/tst/unit/providers/sandbox_providers/sail_sandbox_provider_test.py b/tst/unit/providers/sandbox_providers/sail_sandbox_provider_test.py index a3a4b500..a9603e22 100644 --- a/tst/unit/providers/sandbox_providers/sail_sandbox_provider_test.py +++ b/tst/unit/providers/sandbox_providers/sail_sandbox_provider_test.py @@ -262,12 +262,12 @@ async def slow_create(**_kwargs): @pytest.mark.asyncio async def test_a_failing_orphan_termination_is_retried_then_reported(monkeypatch, caplog): monkeypatch.setattr("agent_env.providers.sandbox_providers.sail.provider.asyncio.sleep", AsyncMock()) - sailbox = _sailbox() - sailbox.terminate.aio = AsyncMock(side_effect=RuntimeError("api down")) + sandbox = MagicMock(sandbox_id="sb_1") + sandbox.terminate = AsyncMock(side_effect=RuntimeError("api down")) - await provider_module._reap(sailbox) + await provider_module._reap(sandbox) - assert sailbox.terminate.aio.await_count == provider_module._REAP_ATTEMPTS + assert sandbox.terminate.await_count == provider_module._REAP_ATTEMPTS assert "Orphaned Sailbox sb_1 is still running" in caplog.text From 4f20a0a8ae5bb843c9bba0e2357e21a52cc9ec4e Mon Sep 17 00:00:00 2001 From: Pratyush Singhal Date: Tue, 6 Oct 2026 18:15:34 -0700 Subject: [PATCH 05/11] fix(sail): address greptile review findings - A handle reconnected in the process that injected a key recovers that key (kept in process memory, never persisted), so it keeps scrubbing an agent's own key even when it isn't the configured [model] key. - A create that fails, or is cancelled before the Sailbox create starts, deletes the model-key secret along with its policy; one abandoned in flight releases both once the create settles, if it fails. - A policy swap that fails or is cancelled deletes the replacement policy and keeps tracking the one still applied. Co-Authored-By: Claude Opus 5.5 --- .../sandbox_providers/sail/model_key.py | 14 ++- .../sandbox_providers/sail/provider.py | 39 ++++--- .../sandbox_providers/sail/sandbox.py | 42 ++++--- .../sandbox_providers/sail_model_key_test.py | 106 +++++++++++++++++- 4 files changed, 163 insertions(+), 38 deletions(-) diff --git a/src/agent_env/providers/sandbox_providers/sail/model_key.py b/src/agent_env/providers/sandbox_providers/sail/model_key.py index d579cb60..65cb4615 100644 --- a/src/agent_env/providers/sandbox_providers/sail/model_key.py +++ b/src/agent_env/providers/sandbox_providers/sail/model_key.py @@ -34,6 +34,10 @@ _SECRET_REF = re.compile(r"\$\{secrets\.([A-Za-z0-9_]+)\}") +# Keys this process has injected, by secret name, so a handle reconnected here can still scrub its own +# agent's key. Process memory only: the keys are already held here, and nothing is persisted. +_injected_keys: dict[str, str] = {} + def secret_name(key: str) -> str: return SECRET_PREFIX + hashlib.sha256(key.encode()).hexdigest()[:32].upper() @@ -77,7 +81,9 @@ def for_env(cls, env: Mapping[str, str]) -> ModelKeyInjection | None: f"Sail injects the model key only into HTTPS requests, but {BASE_URL_ENV} is {base_url!r}; " "use an https endpoint or set inject_model_key = false in [sandbox.providers.sail.config]" ) - return cls(host=parsed.hostname, secret=secret_name(key), key=key) + injection = cls(host=parsed.hostname, secret=secret_name(key), key=key) + _injected_keys[injection.secret] = key + return injection @classmethod def from_document(cls, document: Any, policy_id: str | None) -> ModelKeyInjection | None: @@ -102,8 +108,10 @@ def matches(self, other: ModelKeyInjection | None) -> bool: return other is not None and (other.host, other.secret) == (self.host, self.secret) def recover_key(self, candidate: str | None) -> None: - """Take ``candidate`` as the key when it is the one this injection's secret was named from.""" - if candidate and secret_name(candidate) == self.secret: + """Recover the key this injection's secret was named from: one this process injected, else + ``candidate`` when it is that key.""" + self.key = _injected_keys.get(self.secret) + if self.key is None and candidate and secret_name(candidate) == self.secret: self.key = candidate def scrub(self, text: str) -> str: diff --git a/src/agent_env/providers/sandbox_providers/sail/provider.py b/src/agent_env/providers/sandbox_providers/sail/provider.py index 9eee58d6..08b968ed 100644 --- a/src/agent_env/providers/sandbox_providers/sail/provider.py +++ b/src/agent_env/providers/sandbox_providers/sail/provider.py @@ -7,7 +7,7 @@ import math import re import uuid -from typing import Any, Callable, ClassVar, Self +from typing import Any, Awaitable, Callable, ClassVar, Self from agent_env.attribution import PIPELINE_STEP_KEY, RUN_ID_KEY, Attribution from agent_env.config import get_config @@ -17,8 +17,8 @@ from agent_env.providers.sandbox_providers.sail.sandbox import ( MAX_ALLOWLIST_ENTRIES, SailSandbox, - delete_saved_policy, egress_document, + release_injection, ) from agent_env.providers.sandbox_providers.sandbox import NetworkPolicy, NetworkPolicyUnsupportedError from agent_env.providers.sandbox_providers.sandbox_provider import ( @@ -82,15 +82,22 @@ async def _reap(sandbox: SailSandbox) -> None: ) -async def _create_or_reclaim(create: Any, wrap: Callable[[Any], SailSandbox]) -> Any: - """Await a Sailbox create; if the caller is cancelled first, terminate (``wrap``ped, so its model-key - policy goes too) the Sailbox it yields, which would otherwise keep running with no handle.""" +async def _create_or_reclaim( + create: Any, wrap: Callable[[Any], SailSandbox], release: Callable[[], Awaitable[None]] | None = None, +) -> Any: + """Await a Sailbox create. If the caller is cancelled first, terminate (``wrap``ped, so its model-key + policy and secret go too) the Sailbox it yields, which would otherwise keep running with no handle; if + that create then fails, ``release`` what it was given.""" task = asyncio.ensure_future(create) def terminate_orphan(done: asyncio.Future) -> None: if done.cancelled() or done.exception() is not None: + cleanup = release() if release is not None else None + else: + cleanup = _reap(wrap(done.result())) + if cleanup is None: return - reaper = asyncio.ensure_future(_reap(wrap(done.result()))) + reaper = asyncio.ensure_future(cleanup) _reapers.add(reaper) reaper.add_done_callback(_reapers.discard) @@ -233,15 +240,18 @@ async def _create( ports = list(dict.fromkeys(exposed_ports or [])) sdk, app = await self._connect() egress: Any = egress_document(effective_policy) - if injection is not None: - await sdk.Secret.set.aio(injection.secret, injection.key) - egress = await sdk.EgressPolicy.create.aio(f"agentenv-{uuid.uuid4().hex}", egress_document(effective_policy, injection)) - injection.policy_id = egress.id def wrap(raw: Any) -> SailSandbox: return SailSandbox(raw, sdk=sdk, tunnel_urls={}, network_policy=effective_policy, injection=injection) + release = (lambda: release_injection(sdk, injection)) if injection is not None else None + creating = False try: + if injection is not None: + await sdk.Secret.set.aio(injection.secret, injection.key) + egress = await sdk.EgressPolicy.create.aio(f"agentenv-{uuid.uuid4().hex}", egress_document(effective_policy, injection)) + injection.policy_id = egress.id + creating = True raw = await _create_or_reclaim(sdk.Sailbox.create.aio( app=app, image=sdk.Image.devbox("amd64"), @@ -253,10 +263,11 @@ def wrap(raw: Any) -> SailSandbox: ingress_ports=ports, auto_sleep=self._auto_sleep_setting(sdk), egress_policy=egress, - ), wrap) - except Exception: - if injection is not None: - await delete_saved_policy(sdk, injection.policy_id) + ), wrap, release) + except BaseException as exc: + # A create cancelled in flight is _create_or_reclaim's to clean up once it settles. + if release is not None and not (creating and isinstance(exc, asyncio.CancelledError)): + await release() raise sandbox = wrap(raw) try: diff --git a/src/agent_env/providers/sandbox_providers/sail/sandbox.py b/src/agent_env/providers/sandbox_providers/sail/sandbox.py index 2c22ef7f..5d5a74a7 100644 --- a/src/agent_env/providers/sandbox_providers/sail/sandbox.py +++ b/src/agent_env/providers/sandbox_providers/sail/sandbox.py @@ -80,6 +80,24 @@ async def delete_saved_policy(sdk: Any, policy_id: str | None) -> None: logger.warning("Could not delete egress policy %s: %s", policy_id, exc) +async def delete_secret(sdk: Any, name: str) -> None: + """Delete a model-key secret unless another Sailbox's policy still names it (Sail refuses then).""" + try: + await (await sdk.Secret.get.aio(name)).delete.aio() + except sdk.SecretInUseError: + logger.info("Model-key secret %s is still used by another Sailbox; leaving it", name) + except sdk.NotFoundError: + pass + except Exception as exc: # noqa: BLE001 - the next Sailbox using this key re-sets it; report and go on + logger.warning("Could not delete model-key secret %s: %s", name, exc) + + +async def release_injection(sdk: Any, injection: ModelKeyInjection) -> None: + """Delete an injection's saved policy, then its secret, in the order Sail requires.""" + await delete_saved_policy(sdk, injection.policy_id) + await delete_secret(sdk, injection.secret) + + class _BytesReader: def __init__(self, value: bytes): self._value = value @@ -197,21 +215,7 @@ async def terminate(self) -> None: except self._sdk.NotFoundError: logger.info("Sailbox %s was already gone at terminate", self.sandbox_id) if self._injection is not None: - await self._delete_policy(self._injection.policy_id) - await self._delete_secret(self._injection.secret) - - async def _delete_policy(self, policy_id: str | None) -> None: - await delete_saved_policy(self._sdk, policy_id) - - async def _delete_secret(self, name: str) -> None: - try: - await (await self._sdk.Secret.get.aio(name)).delete.aio() - except self._sdk.SecretInUseError: - logger.info("Model-key secret %s is still used by another Sailbox; leaving it", name) - except self._sdk.NotFoundError: - pass - except Exception as exc: # noqa: BLE001 - the next Sailbox using this key re-sets it; report and go on - logger.warning("Could not delete model-key secret %s: %s", name, exc) + await release_injection(self._sdk, self._injection) async def install_container_trust(self) -> None: """Have every container started on this Sailbox trust the CA Sail injects the model key behind.""" @@ -292,9 +296,13 @@ async def apply_network_policy(self, policy: NetworkPolicy) -> None: await self._sailbox.set_egress_policy.aio(egress_document(policy)) else: saved = await self._sdk.EgressPolicy.create.aio(f"agentenv-{uuid.uuid4().hex}", egress_document(policy, self._injection)) - await self._sailbox.set_egress_policy.aio(saved) + try: + await self._sailbox.set_egress_policy.aio(saved) + except BaseException: + await delete_saved_policy(self._sdk, saved.id) + raise previous, self._injection.policy_id = self._injection.policy_id, saved.id - await self._delete_policy(previous) + await delete_saved_policy(self._sdk, previous) self.network_policy = policy def adopt_applied_policy(self, applied: Any) -> NetworkPolicy | None: diff --git a/tst/unit/providers/sandbox_providers/sail_model_key_test.py b/tst/unit/providers/sandbox_providers/sail_model_key_test.py index 0d4c2b4e..23f507cc 100644 --- a/tst/unit/providers/sandbox_providers/sail_model_key_test.py +++ b/tst/unit/providers/sandbox_providers/sail_model_key_test.py @@ -1,5 +1,6 @@ """Model-key injection on Sailboxes: the key stays at Sail, the Sailbox only ever holds a placeholder.""" +import asyncio import logging import subprocess from types import SimpleNamespace @@ -85,6 +86,7 @@ def _sailbox(ports=(8000,)): def fresh_key_state(monkeypatch): monkeypatch.setattr(_sdk, "_installed_key", None) monkeypatch.setattr(_sdk, "_apps", {}) + monkeypatch.setattr(model_key, "_injected_keys", {}) @pytest.fixture(autouse=True) @@ -138,8 +140,16 @@ def test_an_injection_is_recovered_from_its_saved_policy_only(): assert ModelKeyInjection.from_document({"allowlist": []}, "ep_3") is None -def test_the_key_is_recovered_only_when_it_named_the_secret(): +def test_a_key_injected_by_this_process_is_recovered_even_when_it_is_not_the_configured_one(): recovered = ModelKeyInjection.from_document({"rules": ModelKeyInjection.for_env(_ENV).rules()}, "ep_1") + recovered.recover_key("the-configured-model-key") + assert recovered.key == _KEY + + +def test_in_another_process_only_the_configured_key_is_recovered(monkeypatch): + rules = ModelKeyInjection.for_env(_ENV).rules() + monkeypatch.setattr(model_key, "_injected_keys", {}) + recovered = ModelKeyInjection.from_document({"rules": rules}, "ep_1") recovered.recover_key("some-other-key") assert recovered.key is None recovered.recover_key(_KEY) @@ -224,12 +234,60 @@ async def test_a_plain_endpoint_is_refused_before_anything_is_created(): @pytest.mark.asyncio -async def test_a_failed_create_deletes_the_policy_it_made(): - sdk, saved, _ = _fake_sdk(_sailbox()) +async def test_a_failed_create_deletes_the_policy_and_secret_it_made(): + sdk, saved, secret = _fake_sdk(_sailbox()) sdk.Sailbox.create.aio = AsyncMock(side_effect=RuntimeError("no capacity")) with pytest.raises(RuntimeError, match="no capacity"): await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) saved[0].delete.aio.assert_awaited_once() + secret.delete.aio.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_a_create_cancelled_while_saving_the_policy_deletes_the_secret(): + sdk, _, secret = _fake_sdk(_sailbox()) + started = asyncio.Event() + + async def hang(name, document): + started.set() + await asyncio.Event().wait() + + sdk.EgressPolicy.create.aio = AsyncMock(side_effect=hang) + task = asyncio.ensure_future( + SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + ) + await started.wait() + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + secret.delete.aio.assert_awaited_once() + sdk.Sailbox.create.aio.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_a_create_abandoned_in_flight_that_then_fails_releases_its_policy_and_secret(): + sdk, saved, secret = _fake_sdk(_sailbox()) + started, fail = asyncio.Event(), asyncio.Event() + + async def create(**_kwargs): + started.set() + await fail.wait() + raise RuntimeError("no capacity") + + sdk.Sailbox.create.aio = create + task = asyncio.ensure_future( + SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + ) + await started.wait() + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + saved[0].delete.aio.assert_not_awaited() + fail.set() + for _ in range(10): + await asyncio.sleep(0) + saved[0].delete.aio.assert_awaited_once() + secret.delete.aio.assert_awaited_once() @pytest.mark.asyncio @@ -301,12 +359,32 @@ async def test_widening_an_injected_sailbox_replaces_its_saved_policy(): @pytest.mark.asyncio -async def test_reconnect_restores_the_injection_and_scrubs_the_configured_key(monkeypatch): +async def test_a_failed_policy_swap_deletes_the_replacement_and_keeps_the_old_one(): + sailbox = _sailbox() + sdk, saved, _ = _fake_sdk(sailbox) + sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( + image_name="agent:1", port=8000, env=_ENV, + network_policy=NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",)), + ) + sailbox.set_egress_policy.aio = AsyncMock(side_effect=RuntimeError("api down")) + + with pytest.raises(RuntimeError, match="api down"): + await sandbox.apply_network_policy(sandbox.network_policy.with_hosts(["bucket.example"])) + + first, second = saved + second.delete.aio.assert_awaited_once() + first.delete.aio.assert_not_awaited() + assert sandbox._injection.policy_id == first.id + + +@pytest.mark.asyncio +async def test_reconnect_in_another_process_scrubs_the_configured_key(monkeypatch): injection = ModelKeyInjection.for_env(_ENV) sailbox = _sailbox() sailbox.egress_policy = SimpleNamespace(policy_id="ep_9", document={"allowlist": ["llm.example.com"], "rules": injection.rules()}) sailbox.exec.aio = AsyncMock(return_value=_process()) sdk, _, _ = _fake_sdk(sailbox) + monkeypatch.setattr(model_key, "_injected_keys", {}) monkeypatch.setattr( "agent_env.providers.sandbox_providers.sail.provider.get_config", lambda: MagicMock(get_litellm_api_key=lambda: _KEY) ) @@ -326,3 +404,23 @@ async def test_reconnect_treats_someone_elses_saved_policy_as_unknown(): sdk, _, _ = _fake_sdk(sailbox) sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).get_sandbox("sb_1") assert sandbox.network_policy is None and sandbox._injection is None + + +@pytest.mark.asyncio +async def test_reconnect_in_the_deploying_process_scrubs_the_agents_own_key(monkeypatch): + agent_key = "sk-agent-override-key-42" + sailbox = _sailbox() + sdk, _, _ = _fake_sdk(sailbox) + provider = SailSandboxProvider(api_key="sail-key", sdk=sdk) + created = await provider.create_sandbox(image_name="agent:1", port=8000, env={**_ENV, "LITELLM_API_KEY": agent_key}) + (policy,) = [c.args[1] for c in sdk.EgressPolicy.create.aio.await_args_list] + sailbox.egress_policy = SimpleNamespace(policy_id=created._injection.policy_id, document=policy) + sailbox.exec.aio = AsyncMock(return_value=_process()) + monkeypatch.setattr( + "agent_env.providers.sandbox_providers.sail.provider.get_config", lambda: MagicMock(get_litellm_api_key=lambda: _KEY) + ) + + reconnected = await provider.get_sandbox("sb_1") + await reconnected.exec_with_output("docker", "exec", "-e", f"LITELLM_API_KEY={agent_key}", "agent-api", "pytest") + + assert sailbox.exec.aio.await_args.args[0][3] == f"LITELLM_API_KEY={PLACEHOLDER}" From 8a1b1e97a05bf400eb9125b59a748596618a130b Mon Sep 17 00:00:00 2001 From: Pratyush Singhal Date: Wed, 7 Oct 2026 08:48:09 -0700 Subject: [PATCH 06/11] fix(sail): harden model-key injection after review - Refuse rather than send: a Sailbox (with inject_model_key on) refuses any command or file write assigning LITELLM_API_KEY/ANTHROPIC_API_KEY a value other than the placeholder, so a key Sail doesn't inject (an agent deployed into an existing sandbox, or one a reconnected handle can't recover) never reaches its disk. Shell scripts are read as the shell reads them; other arguments and file bytes are literal. - Scrub the key in every form agent-env writes it: shlex-quoted, escaped inside single quotes, and raw. - Never delete the key's secret: it is named from the key, so a launch can no longer remove one a concurrent launch (here or in another process) still needs. The Sailbox's saved policy is still deleted. - Keys are held per launch and Sailbox in this process and forgotten once none needs them. - Policy changes and terminate are serialized per Sailbox; terminate deletes the policy actually applied. - Saved policies are named before creation, so one whose create response is lost is found by name and deleted, with retries. - The gateway's container lookup retries a lost exec transport and raises rather than read it as no container. Co-Authored-By: Claude Opus 5.5 --- .../env_providers/env_gateway_provider.py | 15 +- .../sandbox_providers/sail/model_key.py | 66 +++- .../sandbox_providers/sail/provider.py | 22 +- .../sandbox_providers/sail/sandbox.py | 127 +++++-- .../sail_sandbox_smoke_test.py | 8 +- .../env_gateway_provider_test.py | 19 + .../sandbox_providers/sail_model_key_test.py | 332 ++++++++++++++++-- 7 files changed, 511 insertions(+), 78 deletions(-) diff --git a/src/agent_env/providers/env_providers/env_gateway_provider.py b/src/agent_env/providers/env_providers/env_gateway_provider.py index 36a60252..ea1308a4 100644 --- a/src/agent_env/providers/env_providers/env_gateway_provider.py +++ b/src/agent_env/providers/env_providers/env_gateway_provider.py @@ -1011,10 +1011,17 @@ async def read_trajectory(self, sandbox: VmSandbox) -> list[dict]: return events async def _get_container_id(self, sandbox: VmSandbox, compose_service: str) -> str | None: - """Get container ID for a docker-compose service (including exited containers).""" - exit_code, stdout, stderr = await sandbox.exec_with_output( - "sudo", "docker", "compose", "-f", DOCKER_COMPOSE_PATH, "ps", "-a", "-q", compose_service - ) + """Get container ID for a docker-compose service (including exited containers). A lookup whose + exec transport failed (exit -1) is retried, then raised: it can't tell "no container" apart.""" + for attempt in range(_TRAJECTORY_READ_ATTEMPTS): + exit_code, stdout, stderr = await sandbox.exec_with_output( + "sudo", "docker", "compose", "-f", DOCKER_COMPOSE_PATH, "ps", "-a", "-q", compose_service + ) + if exit_code != -1: + break + logger.warning(f"Container lookup for {compose_service} lost its exec transport (attempt {attempt + 1}): {stderr[-200:]}") + else: + raise RuntimeError(f"Could not look up the {compose_service} container: exec transport failed {_TRAJECTORY_READ_ATTEMPTS} times") container_id = stdout.strip() return container_id if container_id else None diff --git a/src/agent_env/providers/sandbox_providers/sail/model_key.py b/src/agent_env/providers/sandbox_providers/sail/model_key.py index 65cb4615..c4c3d128 100644 --- a/src/agent_env/providers/sandbox_providers/sail/model_key.py +++ b/src/agent_env/providers/sandbox_providers/sail/model_key.py @@ -1,7 +1,8 @@ """Keeping the model key out of a Sailbox: Sail adds it to the agent's model requests as they leave. The key is stored as a Sail secret named from its SHA-256, so one key is one secret and different keys never -share one. A saved egress policy on the Sailbox sets it as the auth headers of requests to the model endpoint's +share one. The secret is never deleted, so no launch can remove one another, concurrent or in another +process, still needs; a Sailbox's saved policy, which only names it, is deleted with the Sailbox. A saved egress policy on the Sailbox sets it as the auth headers of requests to the model endpoint's host. Inside the Sailbox the key is replaced by a placeholder, and containers start trusting the CA Sail terminates those requests' TLS with. """ @@ -11,6 +12,7 @@ import hashlib import json import re +import shlex from dataclasses import dataclass, field from typing import Any, Mapping from urllib.parse import urlparse @@ -33,16 +35,42 @@ DOCKER_SHIM_PATH = "/usr/local/bin/docker" _SECRET_REF = re.compile(r"\$\{secrets\.([A-Za-z0-9_]+)\}") - -# Keys this process has injected, by secret name, so a handle reconnected here can still scrub its own -# agent's key. Process memory only: the keys are already held here, and nothing is persisted. +# A model-key env var assignment and the whole shell word assigned: quoted, escaped and bare parts alike. +_KEY_ASSIGNMENT = re.compile( + r"\b(?:LITELLM_API_KEY|ANTHROPIC_API_KEY)=((?:'[^']*'|\"(?:[^\"\\]|\\.)*\"|\\.|[^\s'\"\\])*)" +) +# A whole word that is one variable reference a shell expands: $NAME, ${NAME}, "$NAME" or "${NAME}". +_REFERENCE = re.compile(r'^"?\$(?:[A-Za-z_][A-Za-z0-9_]*|\{[A-Za-z_][A-Za-z0-9_]*\})"?$') + +# By secret name: the key, so a handle reconnected here can still scrub its agent's key, and the launches +# and Sailboxes in this process that need it, so the key is forgotten once none does. Process memory only. _injected_keys: dict[str, str] = {} +_holders: dict[str, set[str]] = {} def secret_name(key: str) -> str: return SECRET_PREFIX + hashlib.sha256(key.encode()).hexdigest()[:32].upper() +def carries_model_key(text: str, *, shell: bool) -> bool: + """Whether ``text`` assigns a model-key env var a value other than the placeholder. In a ``shell`` script + the assigned word is decoded as the shell would (quotes and escapes), and a word that is wholly one + unquoted or double-quoted variable reference is not a value; in any other text, every value is literal.""" + for word in _KEY_ASSIGNMENT.findall(text): + if shell and _REFERENCE.match(word): + continue + if shell: + try: + value = "".join(shlex.split(word)) + except ValueError: + return True + else: + value = word + if value and value != PLACEHOLDER: + return True + return False + + def docker_shim() -> str: """A ``docker`` wrapper that gives every container it runs or creates the Sailbox's CA bundle.""" trust = " ".join(f"-e {name}={value}" for name, value in CONTAINER_TRUST_ENV.items()) @@ -81,9 +109,7 @@ def for_env(cls, env: Mapping[str, str]) -> ModelKeyInjection | None: f"Sail injects the model key only into HTTPS requests, but {BASE_URL_ENV} is {base_url!r}; " "use an https endpoint or set inject_model_key = false in [sandbox.providers.sail.config]" ) - injection = cls(host=parsed.hostname, secret=secret_name(key), key=key) - _injected_keys[injection.secret] = key - return injection + return cls(host=parsed.hostname, secret=secret_name(key), key=key) @classmethod def from_document(cls, document: Any, policy_id: str | None) -> ModelKeyInjection | None: @@ -107,6 +133,20 @@ def rules(self) -> dict[str, Any]: def matches(self, other: ModelKeyInjection | None) -> bool: return other is not None and (other.host, other.secret) == (self.host, self.secret) + def hold(self, holder: str) -> None: + """Record ``holder`` (a launch or a Sailbox id) as needing this injection's secret.""" + if self.key is not None: + _injected_keys[self.secret] = self.key + _holders.setdefault(self.secret, set()).add(holder) + + def release(self, holder: str) -> None: + """Drop ``holder``, forgetting the key once no launch or Sailbox in this process needs it.""" + holders = _holders.get(self.secret, set()) + holders.discard(holder) + if not holders: + _holders.pop(self.secret, None) + _injected_keys.pop(self.secret, None) + def recover_key(self, candidate: str | None) -> None: """Recover the key this injection's secret was named from: one this process injected, else ``candidate`` when it is that key.""" @@ -114,8 +154,16 @@ def recover_key(self, candidate: str | None) -> None: if self.key is None and candidate and secret_name(candidate) == self.secret: self.key = candidate + def _encodings(self) -> list[str]: + """The key as agent-env writes it into commands: shell-quoted, inside single quotes, and raw.""" + return [shlex.quote(self.key), self.key.replace("'", "'\\''"), self.key] + def scrub(self, text: str) -> str: - return text.replace(self.key, PLACEHOLDER) if self.key else text + for encoded in self._encodings() if self.key else (): + text = text.replace(encoded, PLACEHOLDER) + return text def scrub_bytes(self, data: bytes) -> bytes: - return data.replace(self.key.encode(), PLACEHOLDER.encode()) if self.key else data + for encoded in self._encodings() if self.key else (): + data = data.replace(encoded.encode(), PLACEHOLDER.encode()) + return data diff --git a/src/agent_env/providers/sandbox_providers/sail/provider.py b/src/agent_env/providers/sandbox_providers/sail/provider.py index 08b968ed..14e7c427 100644 --- a/src/agent_env/providers/sandbox_providers/sail/provider.py +++ b/src/agent_env/providers/sandbox_providers/sail/provider.py @@ -17,7 +17,9 @@ from agent_env.providers.sandbox_providers.sail.sandbox import ( MAX_ALLOWLIST_ENTRIES, SailSandbox, + create_saved_policy, egress_document, + policy_name, release_injection, ) from agent_env.providers.sandbox_providers.sandbox import NetworkPolicy, NetworkPolicyUnsupportedError @@ -241,15 +243,24 @@ async def _create( sdk, app = await self._connect() egress: Any = egress_document(effective_policy) + launch = f"launch-{uuid.uuid4().hex}" + def wrap(raw: Any) -> SailSandbox: - return SailSandbox(raw, sdk=sdk, tunnel_urls={}, network_policy=effective_policy, injection=injection) + sandbox = SailSandbox( + raw, sdk=sdk, tunnel_urls={}, network_policy=effective_policy, injection=injection, + refuse_model_keys=self._inject_model_key, + ) + if injection is not None: + injection.release(launch) + return sandbox - release = (lambda: release_injection(sdk, injection)) if injection is not None else None + release = (lambda: release_injection(sdk, injection, launch)) if injection is not None else None creating = False try: if injection is not None: + injection.hold(launch) await sdk.Secret.set.aio(injection.secret, injection.key) - egress = await sdk.EgressPolicy.create.aio(f"agentenv-{uuid.uuid4().hex}", egress_document(effective_policy, injection)) + egress = await create_saved_policy(sdk, policy_name(), egress_document(effective_policy, injection)) injection.policy_id = egress.id creating = True raw = await _create_or_reclaim(sdk.Sailbox.create.aio( @@ -370,7 +381,10 @@ async def get_sandbox(self, sandbox_id: str) -> SailSandbox: injection.recover_key(get_config().get_litellm_api_key()) except Exception: # noqa: BLE001 - no configured key to recover; the handle just can't scrub it pass - sandbox = SailSandbox(raw, sdk=sdk, tunnel_urls=tunnel_urls, network_policy=None, injection=injection) + sandbox = SailSandbox( + raw, sdk=sdk, tunnel_urls=tunnel_urls, network_policy=None, injection=injection, + refuse_model_keys=self._inject_model_key, + ) sandbox.network_policy = sandbox.adopt_applied_policy(applied) if sandbox.network_policy is None: logger.warning( diff --git a/src/agent_env/providers/sandbox_providers/sail/sandbox.py b/src/agent_env/providers/sandbox_providers/sail/sandbox.py index 5d5a74a7..5ca40eaf 100644 --- a/src/agent_env/providers/sandbox_providers/sail/sandbox.py +++ b/src/agent_env/providers/sandbox_providers/sail/sandbox.py @@ -13,7 +13,12 @@ from urllib.parse import urlparse from agent_env.config import get_config -from agent_env.providers.sandbox_providers.sail.model_key import DOCKER_SHIM_PATH, ModelKeyInjection, docker_shim +from agent_env.providers.sandbox_providers.sail.model_key import ( + DOCKER_SHIM_PATH, + ModelKeyInjection, + carries_model_key, + docker_shim, +) from agent_env.providers.sandbox_providers.sandbox import CURL_RETRY_FLAGS, NetworkMode, NetworkPolicy, VmSandbox from agent_env.providers.sandbox_providers.sandbox_provider import SANDBOX_MODE_VM @@ -21,6 +26,8 @@ #: Sail's limit on entries in one egress allowlist. MAX_ALLOWLIST_ENTRIES = 128 +_CLEANUP_ATTEMPTS = 3 +_SHELLS = frozenset({"bash", "sh"}) # One lock per Sailbox and event loop, shared by every handle to it while an update is in flight. _policy_locks: weakref.WeakValueDictionary[tuple[asyncio.AbstractEventLoop, str], asyncio.Lock] = ( @@ -68,34 +75,60 @@ def _allows_host(policy: NetworkPolicy, host: str) -> bool: return any(entry == host or (entry.startswith("*.") and host.endswith(entry[1:])) for entry in policy.allow_hosts) -async def delete_saved_policy(sdk: Any, policy_id: str | None) -> None: - """Delete a saved egress policy, best effort: a leftover one names a secret but holds no value.""" - if policy_id is None: - return +class ModelKeyRefusedError(RuntimeError): + """A command or file would have carried a model key Sail doesn't inject for this Sailbox into it.""" + + +def policy_name() -> str: + return f"agentenv-{uuid.uuid4().hex}" + + +async def create_saved_policy(sdk: Any, name: str, document: dict[str, Any]) -> Any: + """Create a saved egress policy. If the create fails or is cancelled, the policy Sail may still have made + is found by ``name`` and deleted, so a lost response can't leave one naming a secret behind.""" + try: + return await sdk.EgressPolicy.create.aio(name, document) + except BaseException: + await delete_policy_named(sdk, name) + raise + + +async def delete_policy_named(sdk: Any, name: str) -> None: + """Delete the saved policy called ``name``, if Sail made one, retrying through a brief outage; one Sail + stays unreachable for is logged by name so it can be swept (``sail egress-policy list``).""" + for attempt in range(_CLEANUP_ATTEMPTS): + try: + for summary in await sdk.EgressPolicy.list.aio(search=name): + if summary.name == name: + await _delete_policy_by_id(sdk, summary.id) + return + except Exception as exc: # noqa: BLE001 - retried, then reported; never masks the caller's outcome + logger.warning("Looking up egress policy %s for cleanup failed (attempt %s): %s", name, attempt + 1, exc) + await asyncio.sleep(2 ** attempt) + logger.error("Egress policy %s may be left in Sail, naming a model-key secret; delete it by name", name) + + +async def _delete_policy_by_id(sdk: Any, policy_id: str) -> None: try: await (await sdk.EgressPolicy.get.aio(policy_id)).delete.aio() except sdk.NotFoundError: pass - except Exception as exc: # noqa: BLE001 - reported, never masks the caller's outcome - logger.warning("Could not delete egress policy %s: %s", policy_id, exc) -async def delete_secret(sdk: Any, name: str) -> None: - """Delete a model-key secret unless another Sailbox's policy still names it (Sail refuses then).""" +async def delete_saved_policy(sdk: Any, policy_id: str | None) -> None: + """Delete a saved egress policy, best effort: a leftover one names a secret but holds no value.""" + if policy_id is None: + return try: - await (await sdk.Secret.get.aio(name)).delete.aio() - except sdk.SecretInUseError: - logger.info("Model-key secret %s is still used by another Sailbox; leaving it", name) - except sdk.NotFoundError: - pass - except Exception as exc: # noqa: BLE001 - the next Sailbox using this key re-sets it; report and go on - logger.warning("Could not delete model-key secret %s: %s", name, exc) + await _delete_policy_by_id(sdk, policy_id) + except Exception as exc: # noqa: BLE001 - reported, never masks the caller's outcome + logger.warning("Could not delete egress policy %s: %s", policy_id, exc) -async def release_injection(sdk: Any, injection: ModelKeyInjection) -> None: - """Delete an injection's saved policy, then its secret, in the order Sail requires.""" +async def release_injection(sdk: Any, injection: ModelKeyInjection, holder: str) -> None: + """Delete ``holder``'s saved policy; the secret it names stays (see ``model_key``).""" + injection.release(holder) await delete_saved_policy(sdk, injection.policy_id) - await delete_secret(sdk, injection.secret) class _BytesReader: @@ -189,18 +222,22 @@ async def wait(self) -> int: class SailSandbox(VmSandbox): """A Sailbox from the Docker-capable devbox image; commands run as root. With a model-key ``injection``, - the key's value never enters the Sailbox: every command and file is scrubbed of it.""" + the key's value never enters the Sailbox: every command and file is scrubbed of it. With + ``refuse_model_keys``, one that still carries a model key (one Sail doesn't inject here) is refused.""" type = "sail" _DOCKER_PROBE_TIMEOUT = 10 def __init__( self, sailbox: Any, *, sdk: Any, tunnel_urls: dict[int, str], network_policy: NetworkPolicy | None, - injection: ModelKeyInjection | None = None, + injection: ModelKeyInjection | None = None, refuse_model_keys: bool = False, ): self._sailbox = sailbox self._sdk = sdk self._injection = injection + self._refuse_model_keys = refuse_model_keys + if injection is not None: + injection.hold(sailbox.sailbox_id) self.sandbox_id = sailbox.sailbox_id self.tunnel_urls = tunnel_urls self.vnc_url = None @@ -208,14 +245,28 @@ def __init__( self.network_policy = network_policy async def terminate(self) -> None: - """Terminate the Sailbox, then delete its model-key policy and, unless another Sailbox's policy - still names it, the key's secret.""" + """Terminate the Sailbox, then delete its model-key policy: the one applied, which another handle may + have replaced.""" + if self._injection is None: + await self._terminate_sailbox() + return + async with self._policy_lock(): + try: + self.adopt_applied_policy((await self._sdk.Sailbox.get.aio(self.sandbox_id)).egress_policy) + except Exception as exc: # noqa: BLE001 - fall back to the policy this handle last applied + logger.info("Could not read Sailbox %s's applied policy before terminate: %s", self.sandbox_id, exc) + await self._terminate_sailbox() + await release_injection(self._sdk, self._injection, self.sandbox_id) + + async def _terminate_sailbox(self) -> None: try: await self._sailbox.terminate.aio() except self._sdk.NotFoundError: logger.info("Sailbox %s was already gone at terminate", self.sandbox_id) - if self._injection is not None: - await release_injection(self._sdk, self._injection) + + def _policy_lock(self) -> asyncio.Lock: + """The lock every handle to this Sailbox in this event loop holds to change or tear down its policy.""" + return _policy_locks.setdefault((asyncio.get_running_loop(), self.sandbox_id), asyncio.Lock()) async def install_container_trust(self) -> None: """Have every container started on this Sailbox trust the CA Sail injects the model key behind.""" @@ -239,6 +290,11 @@ async def _run(self, *command: str, timeout: Optional[int] = None) -> _SailProce argv = list(command[1:] if command[:1] == ("sudo",) else command) if self._injection is not None: argv = [self._injection.scrub(arg) for arg in argv] + if self._refuse_model_keys and any( + carries_model_key(arg, shell=index > 0 and argv[index - 1] == "-c" and argv[0] in _SHELLS) + for index, arg in enumerate(argv) + ): + raise self._refusal() try: process = await self._sailbox.exec.aio( argv, timeout=timeout, output_mode="pipe", idempotency_key=uuid.uuid4().hex, @@ -287,15 +343,29 @@ async def setup_vm_for_gateway(self, exposed_ports: Optional[list[int]] = None) async def _write_bytes_to_vm_path(self, data: bytes, vm_path: str) -> None: if self._injection is not None: data = self._injection.scrub_bytes(data) + if self._refuse_model_keys and carries_model_key(data.decode(errors="replace"), shell=False): + raise self._refusal() await self._sailbox.fs.write.aio(vm_path, data) + def _refusal(self) -> ModelKeyRefusedError: + return ModelKeyRefusedError( + f"Refusing to send a model key into Sailbox {self.sandbox_id}: Sail injects model keys only for an agent " + "deployed on its own Sail sandbox, so this one would land on the Sailbox's disk. Deploy the agent on its " + "own sandbox, or set inject_model_key = false in [sandbox.providers.sail.config] to pass keys in." + ) + async def apply_network_policy(self, policy: NetworkPolicy) -> None: """Replace the Sailbox's egress policy; applies to new connections. A model-key injection needs a saved policy (only those can name a secret), so a new one replaces the old, which is deleted.""" + async with self._policy_lock(): + await self._apply_policy(policy) + + async def _apply_policy(self, policy: NetworkPolicy) -> None: + """``apply_network_policy`` for a caller already holding ``_policy_lock``.""" if self._injection is None: await self._sailbox.set_egress_policy.aio(egress_document(policy)) else: - saved = await self._sdk.EgressPolicy.create.aio(f"agentenv-{uuid.uuid4().hex}", egress_document(policy, self._injection)) + saved = await create_saved_policy(self._sdk, policy_name(), egress_document(policy, self._injection)) try: await self._sailbox.set_egress_policy.aio(saved) except BaseException: @@ -335,8 +405,7 @@ async def _allow_download_hosts(self, urls: list[str | None], purpose: str) -> N cached = self._known_policy(purpose) if not cached.restricts_egress or all(_allows_host(cached, host) for host in hosts): return - lock = _policy_locks.setdefault((asyncio.get_running_loop(), self.sandbox_id), asyncio.Lock()) - async with lock: + async with self._policy_lock(): self.network_policy = self.adopt_applied_policy((await self._sdk.Sailbox.get.aio(self.sandbox_id)).egress_policy) policy = self._known_policy(purpose) if not policy.restricts_egress: @@ -349,7 +418,7 @@ async def _allow_download_hosts(self, urls: list[str | None], purpose: str) -> N f"Cannot {purpose} in Sailbox {self.sandbox_id}: adding {missing} would exceed " f"Sail's {MAX_ALLOWLIST_ENTRIES}-entry egress allowlist" ) - await self.apply_network_policy(policy.with_hosts(missing)) + await self._apply_policy(policy.with_hosts(missing)) async def load_docker_images(self, artifacts: list) -> None: """Load images, first adding their signed-download hosts to a restrictive policy.""" diff --git a/tst/integration/providers/sandbox_providers/sail_sandbox_smoke_test.py b/tst/integration/providers/sandbox_providers/sail_sandbox_smoke_test.py index 9fda2f68..24652518 100644 --- a/tst/integration/providers/sandbox_providers/sail_sandbox_smoke_test.py +++ b/tst/integration/providers/sandbox_providers/sail_sandbox_smoke_test.py @@ -99,7 +99,8 @@ async def test_an_allowlist_is_enforced_for_containers_in_the_sailbox(sail_provi async def test_an_agents_model_key_is_injected_by_sail_and_never_enters_the_sailbox(sail_provider): - """A throwaway key against a header-echo host stands in for the model endpoint.""" + """A throwaway key against a header-echo host stands in for the model endpoint. The provider leaves the + key's secret in Sail; this test deletes its own.""" key = f"sk-agentenv-smoke-{secrets.token_hex(16)}" env = {"LITELLM_API_KEY": key, "LITELLM_BASE_URL": f"https://{_ECHO_HOST}/v1"} sandbox = await sail_provider.create_sandbox(image_name="unused", port=_PORT, env=env, cpu=1.0, memory=2048, timeout=900) @@ -116,6 +117,5 @@ async def test_an_agents_model_key_is_injected_by_sail_and_never_enters_the_sail assert key not in config and PLACEHOLDER in json.dumps(json.loads(config)[0]["Config"]["Env"]) finally: await sandbox.terminate() - sdk = sandbox._sdk - with pytest.raises(sdk.NotFoundError): - await sdk.Secret.get.aio(secret_name(key)) + secret = await sandbox._sdk.Secret.get.aio(secret_name(key)) + await secret.delete.aio() diff --git a/tst/unit/providers/env_providers/env_gateway_provider_test.py b/tst/unit/providers/env_providers/env_gateway_provider_test.py index ed67affe..cd57fbd9 100644 --- a/tst/unit/providers/env_providers/env_gateway_provider_test.py +++ b/tst/unit/providers/env_providers/env_gateway_provider_test.py @@ -570,6 +570,25 @@ async def test_read_trajectory_raises_rather_than_parse_a_read_that_keeps_failin await gp.read_trajectory(sandbox) +@pytest.mark.asyncio +async def test_a_container_lookup_whose_transport_keeps_failing_raises_rather_than_read_as_no_container(): + gp = EnvironmentGatewayProvider() + sandbox = MagicMock() + sandbox.exec_with_output = AsyncMock(return_value=(-1, "", "host lost")) + + with pytest.raises(RuntimeError, match="Could not look up the .* container: exec transport failed 3 times"): + await gp.read_trajectory(sandbox) + + +@pytest.mark.asyncio +async def test_a_container_lookup_retries_a_failed_transport(): + gp = EnvironmentGatewayProvider() + sandbox = MagicMock() + sandbox.exec_with_output = AsyncMock(side_effect=[(-1, "", "host lost"), (0, "gw\n", ""), (0, '{"a": 1}\n', "")]) + + assert await gp.read_trajectory(sandbox) == [{"a": 1}] + + @pytest.mark.asyncio async def test_read_trajectory_of_a_gateway_with_no_history_yet_is_empty(): gp = EnvironmentGatewayProvider() diff --git a/tst/unit/providers/sandbox_providers/sail_model_key_test.py b/tst/unit/providers/sandbox_providers/sail_model_key_test.py index 23f507cc..641092f7 100644 --- a/tst/unit/providers/sandbox_providers/sail_model_key_test.py +++ b/tst/unit/providers/sandbox_providers/sail_model_key_test.py @@ -2,6 +2,7 @@ import asyncio import logging +import shlex import subprocess from types import SimpleNamespace from unittest.mock import AsyncMock, MagicMock @@ -14,21 +15,23 @@ DOCKER_SHIM_PATH, PLACEHOLDER, ModelKeyInjection, + carries_model_key, docker_shim, secret_name, ) from agent_env.providers.sandbox_providers.sail.provider import SailSandboxProvider -from agent_env.providers.sandbox_providers.sail.sandbox import SailSandbox +from agent_env.providers.sandbox_providers.sail.sandbox import ( + ModelKeyRefusedError, + SailSandbox, + create_saved_policy, + delete_policy_named, +) from agent_env.providers.sandbox_providers.sandbox import NetworkMode, NetworkPolicy _KEY = "sk-live-model-key-0123456789" _ENV = {"LITELLM_API_KEY": _KEY, "LITELLM_BASE_URL": "https://llm.example.com/v1", "A2A_PORT": "8000"} -class _SecretInUse(Exception): - pass - - class _NotFound(Exception): pass @@ -61,10 +64,10 @@ async def create_policy(name, document): EgressPolicy=SimpleNamespace( create=SimpleNamespace(aio=AsyncMock(side_effect=create_policy)), get=SimpleNamespace(aio=AsyncMock(side_effect=lambda policy_id: next(p for p in saved if p.id == policy_id))), + list=SimpleNamespace(aio=AsyncMock(side_effect=lambda search: [p for p in saved if p.name == search])), ), reset_transports=MagicMock(), NotFoundError=_NotFound, - SecretInUseError=_SecretInUse, SailboxHostLostError=_HostLost, TransportError=_HostLost, ) @@ -87,6 +90,7 @@ def fresh_key_state(monkeypatch): monkeypatch.setattr(_sdk, "_installed_key", None) monkeypatch.setattr(_sdk, "_apps", {}) monkeypatch.setattr(model_key, "_injected_keys", {}) + monkeypatch.setattr(model_key, "_holders", {}) @pytest.fixture(autouse=True) @@ -141,7 +145,9 @@ def test_an_injection_is_recovered_from_its_saved_policy_only(): def test_a_key_injected_by_this_process_is_recovered_even_when_it_is_not_the_configured_one(): - recovered = ModelKeyInjection.from_document({"rules": ModelKeyInjection.for_env(_ENV).rules()}, "ep_1") + injected = ModelKeyInjection.for_env(_ENV) + injected.hold("sb_1") + recovered = ModelKeyInjection.from_document({"rules": injected.rules()}, "ep_1") recovered.recover_key("the-configured-model-key") assert recovered.key == _KEY @@ -234,17 +240,18 @@ async def test_a_plain_endpoint_is_refused_before_anything_is_created(): @pytest.mark.asyncio -async def test_a_failed_create_deletes_the_policy_and_secret_it_made(): +async def test_a_failed_create_deletes_its_policy_but_never_the_secret(): sdk, saved, secret = _fake_sdk(_sailbox()) sdk.Sailbox.create.aio = AsyncMock(side_effect=RuntimeError("no capacity")) with pytest.raises(RuntimeError, match="no capacity"): await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) saved[0].delete.aio.assert_awaited_once() - secret.delete.aio.assert_awaited_once() + secret.delete.aio.assert_not_awaited() + assert secret_name(_KEY) not in model_key._injected_keys @pytest.mark.asyncio -async def test_a_create_cancelled_while_saving_the_policy_deletes_the_secret(): +async def test_a_create_cancelled_while_saving_the_policy_forgets_the_key(): sdk, _, secret = _fake_sdk(_sailbox()) started = asyncio.Event() @@ -260,12 +267,13 @@ async def hang(name, document): task.cancel() with pytest.raises(asyncio.CancelledError): await task - secret.delete.aio.assert_awaited_once() sdk.Sailbox.create.aio.assert_not_awaited() + secret.delete.aio.assert_not_awaited() + assert secret_name(_KEY) not in model_key._injected_keys @pytest.mark.asyncio -async def test_a_create_abandoned_in_flight_that_then_fails_releases_its_policy_and_secret(): +async def test_a_create_abandoned_in_flight_that_then_fails_deletes_its_policy(): sdk, saved, secret = _fake_sdk(_sailbox()) started, fail = asyncio.Event(), asyncio.Event() @@ -287,7 +295,7 @@ async def create(**_kwargs): for _ in range(10): await asyncio.sleep(0) saved[0].delete.aio.assert_awaited_once() - secret.delete.aio.assert_awaited_once() + secret.delete.aio.assert_not_awaited() @pytest.mark.asyncio @@ -315,7 +323,7 @@ async def test_commands_and_files_carry_the_placeholder_never_the_key(): @pytest.mark.asyncio -async def test_terminate_deletes_the_policy_then_the_secret(): +async def test_terminate_deletes_the_policy_and_keeps_the_secret(): sailbox = _sailbox() sdk, saved, secret = _fake_sdk(sailbox) sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) @@ -324,20 +332,8 @@ async def test_terminate_deletes_the_policy_then_the_secret(): sailbox.terminate.aio.assert_awaited_once() saved[0].delete.aio.assert_awaited_once() - sdk.Secret.get.aio.assert_awaited_once_with(secret_name(_KEY)) - secret.delete.aio.assert_awaited_once() - - -@pytest.mark.asyncio -async def test_a_secret_another_sailbox_still_uses_is_left_in_place(caplog): - caplog.set_level(logging.INFO) - sdk, _, secret = _fake_sdk(_sailbox()) - secret.delete.aio = AsyncMock(side_effect=_SecretInUse("in use")) - sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) - - await sandbox.terminate() - - assert "still used by another Sailbox" in caplog.text + secret.delete.aio.assert_not_awaited() + assert secret_name(_KEY) not in model_key._injected_keys @pytest.mark.asyncio @@ -424,3 +420,283 @@ async def test_reconnect_in_the_deploying_process_scrubs_the_agents_own_key(monk await reconnected.exec_with_output("docker", "exec", "-e", f"LITELLM_API_KEY={agent_key}", "agent-api", "pytest") assert sailbox.exec.aio.await_args.args[0][3] == f"LITELLM_API_KEY={PLACEHOLDER}" + + +@pytest.mark.parametrize( + ("text", "shell", "carries"), + [ + (f"docker run -e LITELLM_API_KEY='{_KEY}' img", True, True), + (f"docker exec -e ANTHROPIC_API_KEY={_KEY} c pytest", True, True), + (f"- LITELLM_API_KEY={_KEY}", False, True), + ("docker run -e LITELLM_API_KEY='$test-secret' img", True, True), + ("docker run -e LITELLM_API_KEY=''\\''sk-secret' img", True, True), + ("docker run -e LITELLM_API_KEY=$test-secret img", True, True), + ('docker run -e LITELLM_API_KEY="$LITELLM_API_KEY-x" img', True, True), + ("LITELLM_API_KEY=$test-secret", False, True), + ("LITELLM_API_KEY=$LITELLM_API_KEY", False, True), + (f"-e LITELLM_API_KEY='{PLACEHOLDER}'", True, False), + (f"-e LITELLM_API_KEY={PLACEHOLDER}", False, False), + ('-e LITELLM_API_KEY="$LITELLM_API_KEY"', True, False), + ("-e LITELLM_API_KEY=${LITELLM_API_KEY}", True, False), + ("-e LITELLM_API_KEY=$LITELLM_API_KEY", True, False), + ("echo LITELLM_API_KEY=", True, False), + ("docker run -e OTHER=1 img", True, False), + ], +) +def test_a_model_key_assignment_is_detected_as_the_shell_reads_it(text, shell, carries): + assert carries_model_key(text, shell=shell) is carries + + +def test_the_agents_own_escaping_of_a_key_is_seen_through(): + key = "'sk-starts-with-a-quote" + escaped = key.replace("'", "'\\''") + assert carries_model_key(f"docker run -e LITELLM_API_KEY='{escaped}' img", shell=True) + + +@pytest.mark.asyncio +async def test_a_dollar_value_in_a_plain_argument_is_a_literal_key_and_refused(): + sailbox = _sailbox(ports=()) + sailbox.exec.aio = AsyncMock(return_value=_process()) + sdk, _, _ = _fake_sdk(sailbox) + sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_vm(exposed_ports=[]) + with pytest.raises(ModelKeyRefusedError): + await sandbox.exec_with_output("docker", "exec", "-e", "LITELLM_API_KEY=$looks-like-a-var", "agent-api", "true") + await sandbox.exec_with_output("bash", "-c", 'docker exec -e LITELLM_API_KEY="$LITELLM_API_KEY" agent-api true') + + +@pytest.mark.asyncio +async def test_a_sailbox_without_injection_refuses_a_model_key_rather_than_take_it(): + sailbox = _sailbox(ports=()) + sailbox.exec.aio = AsyncMock(return_value=_process()) + sdk, _, _ = _fake_sdk(sailbox) + sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_vm(exposed_ports=[]) + sailbox.exec.aio.reset_mock() + + with pytest.raises(ModelKeyRefusedError, match="Deploy the agent on its own sandbox"): + await sandbox.exec_script(f"docker run -d --name agent-api -e LITELLM_API_KEY='{_KEY}' agent:1") + with pytest.raises(ModelKeyRefusedError): + await sandbox.write_host_file(f"LITELLM_API_KEY={_KEY}\n".encode(), "/opt/agent/.env") + assert not any(_KEY in arg for call in sailbox.exec.aio.await_args_list for arg in call.args[0]) + sailbox.fs.write.aio.assert_not_awaited() + + await sandbox.exec_script('docker run -e LITELLM_API_KEY="$LITELLM_API_KEY" agent:1') + + +@pytest.mark.asyncio +async def test_with_injection_off_keys_pass_through_as_before(): + sailbox = _sailbox(ports=()) + sailbox.exec.aio = AsyncMock(return_value=_process()) + sdk, _, _ = _fake_sdk(sailbox) + sandbox = await SailSandboxProvider(api_key="sail-key", inject_model_key=False, sdk=sdk).create_vm(exposed_ports=[]) + await sandbox.exec_script(f"docker run -e LITELLM_API_KEY='{_KEY}' agent:1") + assert _KEY in sailbox.exec.aio.await_args.args[0][2] + + +@pytest.mark.asyncio +async def test_a_reconnected_handle_that_cannot_recover_the_key_refuses_it(monkeypatch): + sailbox = _sailbox() + sailbox.egress_policy = SimpleNamespace(policy_id="ep_9", document={"rules": ModelKeyInjection.for_env(_ENV).rules()}) + sailbox.exec.aio = AsyncMock(return_value=_process()) + sdk, _, _ = _fake_sdk(sailbox) + monkeypatch.setattr(model_key, "_injected_keys", {}) + monkeypatch.setattr( + "agent_env.providers.sandbox_providers.sail.provider.get_config", lambda: MagicMock(get_litellm_api_key=lambda: "other") + ) + sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).get_sandbox("sb_1") + with pytest.raises(ModelKeyRefusedError): + await sandbox.exec_with_output("docker", "exec", "-e", f"LITELLM_API_KEY={_KEY}", "agent-api", "pytest") + sailbox.exec.aio.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_terminate_deletes_the_policy_actually_applied_even_if_another_handle_replaced_it(): + sailbox = _sailbox() + sdk, saved, _ = _fake_sdk(sailbox) + sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + replacement = await sdk.EgressPolicy.create.aio("agentenv-replacement", saved[0].document) + sailbox.egress_policy = SimpleNamespace(policy_id=replacement.id, document=replacement.document) + + await sandbox.terminate() + + replacement.delete.aio.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_a_policy_create_whose_response_is_lost_is_found_by_name_and_deleted(): + sdk, saved, _ = _fake_sdk(_sailbox()) + created = sdk.EgressPolicy.create.aio.side_effect + + async def create_then_lose(name, document): + await created(name, document) + raise TimeoutError("response lost") + + sdk.EgressPolicy.create.aio = AsyncMock(side_effect=create_then_lose) + with pytest.raises(TimeoutError): + await create_saved_policy(sdk, "agentenv-lost", {"rules": {}}) + saved[0].delete.aio.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_the_key_is_remembered_until_the_last_box_here_that_needs_it_is_gone(): + first, second = _sailbox(), _sailbox() + second.sailbox_id = "sb_2" + sdk, _, secret = _fake_sdk(first) + sdk.Sailbox.create.aio = AsyncMock(side_effect=[first, second]) + provider = SailSandboxProvider(api_key="sail-key", sdk=sdk) + one = await provider.create_sandbox(image_name="agent:1", port=8000, env=_ENV) + two = await provider.create_sandbox(image_name="agent:1", port=8000, env=_ENV) + + await one.terminate() + assert model_key._injected_keys[secret_name(_KEY)] == _KEY + await two.terminate() + assert secret_name(_KEY) not in model_key._injected_keys + secret.delete.aio.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_a_failed_launch_never_touches_the_secret_a_concurrent_launch_with_the_same_key_needs(): + sailbox = _sailbox() + sdk, _, secret = _fake_sdk(sailbox) + first_waiting, go = asyncio.Event(), asyncio.Event() + calls = 0 + + async def create(**_kwargs): + nonlocal calls + calls += 1 + if calls == 1: + first_waiting.set() + await go.wait() + return sailbox + raise RuntimeError("no capacity") + + sdk.Sailbox.create.aio = create + provider = SailSandboxProvider(api_key="sail-key", sdk=sdk) + pending = asyncio.ensure_future(provider.create_sandbox(image_name="agent:1", port=8000, env=_ENV)) + await asyncio.wait_for(first_waiting.wait(), timeout=5) + with pytest.raises(RuntimeError, match="no capacity"): + await provider.create_sandbox(image_name="agent:1", port=8000, env=_ENV) + assert model_key._injected_keys[secret_name(_KEY)] == _KEY + go.set() + sandbox = await asyncio.wait_for(pending, timeout=5) + await sandbox.terminate() + secret.delete.aio.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_terminate_waits_for_a_policy_swap_in_flight_on_another_handle(): + sailbox = _sailbox() + sdk, saved, _ = _fake_sdk(sailbox) + provider = SailSandboxProvider(api_key="sail-key", sdk=sdk) + sandbox = await provider.create_sandbox( + image_name="agent:1", port=8000, env=_ENV, + network_policy=NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",)), + ) + sailbox.egress_policy = SimpleNamespace(policy_id=saved[0].id, document=saved[0].document) + other = SailSandbox(sailbox, sdk=sdk, tunnel_urls={}, network_policy=sandbox.network_policy, + injection=ModelKeyInjection.from_document(saved[0].document, saved[0].id)) + swapping, release = asyncio.Event(), asyncio.Event() + + async def slow_set(policy): + swapping.set() + await release.wait() + sailbox.egress_policy = SimpleNamespace(policy_id=policy.id, document=policy.document) + + sailbox.set_egress_policy.aio = AsyncMock(side_effect=slow_set) + sdk.Sailbox.get.aio = AsyncMock(side_effect=lambda _id: sailbox) + swap = asyncio.ensure_future(other._allow_download_hosts(["https://bucket.example/a"], "download")) + await asyncio.wait_for(swapping.wait(), timeout=5) + teardown = asyncio.ensure_future(sandbox.terminate()) + await asyncio.sleep(0.01) + sailbox.terminate.aio.assert_not_awaited() + release.set() + await asyncio.wait_for(asyncio.gather(swap, teardown), timeout=5) + + replacement = saved[-1] + replacement.delete.aio.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_lost_response_cleanup_retries_through_a_brief_outage(monkeypatch, caplog): + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail.sandbox.asyncio.sleep", AsyncMock()) + sdk, saved, _ = _fake_sdk(_sailbox()) + await sdk.EgressPolicy.create.aio("agentenv-lost", {"rules": {}}) + listing = sdk.EgressPolicy.list.aio.side_effect + sdk.EgressPolicy.list.aio = AsyncMock(side_effect=[RuntimeError("down"), listing(search="agentenv-lost")]) + + await delete_policy_named(sdk, "agentenv-lost") + + saved[0].delete.aio.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_lost_response_cleanup_that_never_reaches_sail_names_the_policy_to_sweep(monkeypatch, caplog): + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail.sandbox.asyncio.sleep", AsyncMock()) + sdk, _, _ = _fake_sdk(_sailbox()) + sdk.EgressPolicy.list.aio = AsyncMock(side_effect=RuntimeError("down")) + + await delete_policy_named(sdk, "agentenv-lost") + + assert sdk.EgressPolicy.list.aio.await_count == 3 + assert "Egress policy agentenv-lost may be left in Sail" in caplog.text + + +@pytest.mark.asyncio +async def test_lost_response_cleanup_retries_a_failed_delete_too(monkeypatch): + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail.sandbox.asyncio.sleep", AsyncMock()) + sdk, saved, _ = _fake_sdk(_sailbox()) + await sdk.EgressPolicy.create.aio("agentenv-lost", {"rules": {}}) + saved[0].delete.aio = AsyncMock(side_effect=[RuntimeError("down"), None]) + + await delete_policy_named(sdk, "agentenv-lost") + + assert saved[0].delete.aio.await_count == 2 + + +@pytest.mark.asyncio +async def test_a_direct_policy_update_waits_for_terminate_on_another_handle(): + sailbox = _sailbox() + sdk, saved, _ = _fake_sdk(sailbox) + sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( + image_name="agent:1", port=8000, env=_ENV, + network_policy=NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",)), + ) + reading, release = asyncio.Event(), asyncio.Event() + + async def slow_get(_id): + reading.set() + await release.wait() + return SimpleNamespace(egress_policy=SimpleNamespace(policy_id=saved[0].id, document=saved[0].document)) + + sdk.Sailbox.get.aio = AsyncMock(side_effect=slow_get) + other = SailSandbox(sailbox, sdk=sdk, tunnel_urls={}, network_policy=sandbox.network_policy, + injection=ModelKeyInjection.from_document(saved[0].document, saved[0].id)) + teardown = asyncio.ensure_future(sandbox.terminate()) + await asyncio.wait_for(reading.wait(), timeout=5) + update = asyncio.ensure_future(other.apply_network_policy(sandbox.network_policy.with_hosts(["late.example"]))) + await asyncio.sleep(0.01) + assert len(saved) == 1 + release.set() + await asyncio.wait_for(asyncio.gather(teardown, update), timeout=5) + sailbox.terminate.aio.assert_awaited_once() + assert sailbox.set_egress_policy.aio.await_count == 1 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("quoting", ["agent", "shlex"]) +async def test_a_key_with_an_apostrophe_is_scrubbed_in_its_shell_encoded_form(quoting): + + + key = "sk-a'b-quoted-key" + sailbox = _sailbox() + sailbox.exec.aio = AsyncMock(return_value=_process()) + sdk, _, _ = _fake_sdk(sailbox) + sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( + image_name="agent:1", port=8000, env={**_ENV, "LITELLM_API_KEY": key}, + ) + encoded = "'" + key.replace("'", "'\\''") + "'" if quoting == "agent" else shlex.quote(key) + + await sandbox.exec_script(f"docker run -d --name agent-api -e LITELLM_API_KEY={encoded} agent:1") + + script = sailbox.exec.aio.await_args.args[0][2] + assert "sk-a" not in script and PLACEHOLDER in script + assert shlex.split(script.split("LITELLM_API_KEY=", 1)[1])[0] == PLACEHOLDER From fdfa1a959197f05b3fb047e888c423f62f755cf5 Mon Sep 17 00:00:00 2001 From: Pratyush Singhal Date: Wed, 7 Oct 2026 09:21:43 -0700 Subject: [PATCH 07/11] fix(gateway): never let a deploy-time log lookup stop a healthy deploy The gateway and sidecar log dumps after compose up go through _log_service, which reports a container lookup that keeps losing its exec transport instead of raising it into the deploy. read_trajectory still raises, since there a failed lookup must not read as no history. Co-Authored-By: Claude Opus 5.5 --- .../env_providers/env_gateway_provider.py | 32 +++++++++++-------- .../env_gateway_provider_test.py | 19 +++++++++++ 2 files changed, 38 insertions(+), 13 deletions(-) diff --git a/src/agent_env/providers/env_providers/env_gateway_provider.py b/src/agent_env/providers/env_providers/env_gateway_provider.py index ea1308a4..d39162dc 100644 --- a/src/agent_env/providers/env_providers/env_gateway_provider.py +++ b/src/agent_env/providers/env_providers/env_gateway_provider.py @@ -757,12 +757,7 @@ async def _stand_up(_spec): # Print gateway logs for debugging logger.info("Gateway logs:") - gateway_container_id = await self._get_container_id(sandbox, GATEWAY_SERVICE_NAME) - if gateway_container_id: - exit_code, logs, stderr = await sandbox.exec_with_output( "sudo", "docker", "logs", gateway_container_id) - logger.info(f" stdout:\n{logs}") - if stderr: - logger.info(f" stderr:\n{stderr}") + await self._log_service(sandbox, GATEWAY_SERVICE_NAME) if not gateway_ready: raise RuntimeError("Gateway did not become ready in time") @@ -772,13 +767,8 @@ async def _stand_up(_spec): if self._needs_local_postgres else [] ) for name in sidecar_names: - container_id = await self._get_container_id(sandbox, name) - if container_id: - exit_code, logs, stderr = await sandbox.exec_with_output("sudo", "docker", "logs", container_id) - logger.info(f"{name} logs:\n stdout:\n{logs}") - if stderr: - logger.info(f" stderr:\n{stderr}") - else: + logger.info(f"{name} logs:") + if not await self._log_service(sandbox, name): logger.warning(f"{name} container not found") gateway_url = sandbox.tunnel_urls.get(gateway_port) @@ -1010,6 +1000,22 @@ async def read_trajectory(self, sandbox: VmSandbox) -> list[dict]: events.append(json.loads(line)) return events + async def _log_service(self, sandbox: VmSandbox, compose_service: str) -> bool: + """Log a compose service's container output; False when it has no container. Diagnostics only, so + a lookup that keeps losing its exec transport is reported, never raised into the deploy.""" + try: + container_id = await self._get_container_id(sandbox, compose_service) + except RuntimeError as e: + logger.warning(f"Skipping {compose_service} logs: {e}") + return True + if not container_id: + return False + exit_code, logs, stderr = await sandbox.exec_with_output("sudo", "docker", "logs", container_id) + logger.info(f" stdout:\n{logs}") + if stderr: + logger.info(f" stderr:\n{stderr}") + return True + async def _get_container_id(self, sandbox: VmSandbox, compose_service: str) -> str | None: """Get container ID for a docker-compose service (including exited containers). A lookup whose exec transport failed (exit -1) is retried, then raised: it can't tell "no container" apart.""" diff --git a/tst/unit/providers/env_providers/env_gateway_provider_test.py b/tst/unit/providers/env_providers/env_gateway_provider_test.py index cd57fbd9..7703c9d4 100644 --- a/tst/unit/providers/env_providers/env_gateway_provider_test.py +++ b/tst/unit/providers/env_providers/env_gateway_provider_test.py @@ -1694,3 +1694,22 @@ def test_every_mcp_server_gets_the_same_healthcheck_budget(): assert " timeout: 5s" in block assert " retries: 30" in block assert " start_period: 60s" in block + + +@pytest.mark.asyncio +async def test_a_log_lookup_that_keeps_losing_its_transport_never_stops_a_deploy(caplog): + gp = EnvironmentGatewayProvider() + sandbox = MagicMock() + sandbox.exec_with_output = AsyncMock(return_value=(-1, "", "host lost")) + + assert await gp._log_service(sandbox, "gateway") is True + assert "Skipping gateway logs" in caplog.text + + +@pytest.mark.asyncio +async def test_a_service_without_a_container_is_reported_as_missing(): + gp = EnvironmentGatewayProvider() + sandbox = MagicMock() + sandbox.exec_with_output = AsyncMock(return_value=(0, "", "")) + + assert await gp._log_service(sandbox, "pgweb") is False From 0abf4b24ea077a7ed0bfb3211f2c80d503a2585f Mon Sep 17 00:00:00 2001 From: Pratyush Singhal Date: Wed, 7 Oct 2026 09:57:22 -0700 Subject: [PATCH 08/11] fix(sail): stream objects a store can't sign through Sail's filesystem API main now pushes such objects onto a VM as base64 chunks over exec, reading them through ObjectStore.open. A Sailbox takes them through its own filesystem API instead, streamed a chunk at a time with no base64 or per-chunk exec, which also keeps them out of the model-key guards meant for agent-env's own commands. Co-Authored-By: Claude Opus 5.5 --- .../providers/sandbox_providers/sail/sandbox.py | 10 ++++++++++ .../sandbox_providers/sail_sandbox_test.py | 17 +++++++++++++++-- 2 files changed, 25 insertions(+), 2 deletions(-) diff --git a/src/agent_env/providers/sandbox_providers/sail/sandbox.py b/src/agent_env/providers/sandbox_providers/sail/sandbox.py index 5ca40eaf..dcf5f4db 100644 --- a/src/agent_env/providers/sandbox_providers/sail/sandbox.py +++ b/src/agent_env/providers/sandbox_providers/sail/sandbox.py @@ -3,6 +3,7 @@ from __future__ import annotations import asyncio +import contextlib import logging import shlex import time @@ -27,6 +28,7 @@ #: Sail's limit on entries in one egress allowlist. MAX_ALLOWLIST_ENTRIES = 128 _CLEANUP_ATTEMPTS = 3 +_STREAM_CHUNK_BYTES = 8 * 1024 * 1024 _SHELLS = frozenset({"bash", "sh"}) # One lock per Sailbox and event loop, shared by every handle to it while an update is in flight. @@ -340,6 +342,14 @@ async def setup_vm_for_gateway(self, exposed_ports: Optional[list[int]] = None) if exit_code != 0: raise RuntimeError(f"Sailbox {self.sandbox_id} has no Docker Compose v2: {(stderr or stdout).strip()}") + async def _write_unsigned_object(self, object_store: Any, object_url: str, vm_path: str) -> None: + """Stream an object the store can't sign straight onto the VM host through Sail's filesystem API, + a chunk at a time, instead of base64 over exec.""" + with contextlib.closing(await asyncio.to_thread(object_store.open, object_url)) as source: + async with await self._sailbox.fs.write_stream.aio(vm_path) as writer: + while chunk := await asyncio.to_thread(source.read, _STREAM_CHUNK_BYTES): + await writer.write(chunk) + async def _write_bytes_to_vm_path(self, data: bytes, vm_path: str) -> None: if self._injection is not None: data = self._injection.scrub_bytes(data) diff --git a/tst/unit/providers/sandbox_providers/sail_sandbox_test.py b/tst/unit/providers/sandbox_providers/sail_sandbox_test.py index 253ffa38..64ff58f1 100644 --- a/tst/unit/providers/sandbox_providers/sail_sandbox_test.py +++ b/tst/unit/providers/sandbox_providers/sail_sandbox_test.py @@ -1,6 +1,7 @@ """Unit tests for the Sailbox adapter over a fake SDK Sailbox.""" import asyncio +import io from types import SimpleNamespace from unittest.mock import AsyncMock, MagicMock @@ -302,7 +303,10 @@ async def test_a_host_already_allowed_is_not_reapplied(monkeypatch): def _object_store(monkeypatch, signed): - store = MagicMock(signed_get_url=MagicMock(return_value=signed), get=MagicMock(return_value=b"\x00payload")) + store = MagicMock( + signed_get_url=MagicMock(return_value=signed), + open=MagicMock(side_effect=lambda _url: io.BytesIO(b"\x00payload")), + ) monkeypatch.setattr( "agent_env.providers.sandbox_providers.sail.sandbox.get_config", lambda: MagicMock(get_object_store=lambda: store) ) @@ -326,9 +330,18 @@ async def test_a_signed_object_download_allows_its_host_first(monkeypatch): @pytest.mark.asyncio async def test_an_unsignable_object_is_streamed_through_the_filesystem_api(monkeypatch): _object_store(monkeypatch, None) + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail.sandbox._STREAM_CHUNK_BYTES", 4) sandbox, sailbox = _sandbox(policy=None) + writer = MagicMock(write=AsyncMock()) + stream = MagicMock(__aenter__=AsyncMock(return_value=writer), __aexit__=AsyncMock(return_value=False)) + sailbox.fs.write_stream.aio = AsyncMock(return_value=stream) + await sandbox.load_s3_file("file:///store/f", "/tmp/f") - sailbox.fs.write.aio.assert_awaited_once_with("/tmp/f", b"\x00payload") + + sailbox.fs.write_stream.aio.assert_awaited_once_with("/tmp/f") + assert b"".join(call.args[0] for call in writer.write.await_args_list) == b"\x00payload" + assert [len(call.args[0]) for call in writer.write.await_args_list] == [4, 4] + stream.__aexit__.assert_awaited_once() sailbox.exec.aio.assert_not_awaited() From b04e2ad34468e19ca21fe5fe1dc2e6d10968603e Mon Sep 17 00:00:00 2001 From: Pratyush Singhal Date: Wed, 7 Oct 2026 10:40:10 -0700 Subject: [PATCH 09/11] refactor(sandbox): name the Sail provider sail_vm, like modal_vm The provider, its sandbox type and its config table are now sail_vm ([sandbox.providers.sail_vm.config], --sandbox sail_vm), the package sandbox_providers/sail_vm, and the classes SailVmSandbox and SailVmSandboxProvider, matching modal_vm for a VM-mode provider. THIRD_PARTY_NOTICES: sail ships no license file and its source isn't public, so its entry says so and points at PyPI, as for other packages without one, rather than linking a generic Apache text. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- THIRD_PARTY_NOTICES.md | 4 +- src/agent_env/cli/a2a_agent/deploy.py | 2 +- src/agent_env/cli/env/deploy.py | 2 +- src/agent_env/providers/__init__.py | 4 +- .../providers/sandbox_providers/__init__.py | 4 +- .../sandbox_providers/sail/__init__.py | 6 -- .../sandbox_providers/sail_vm/__init__.py | 6 ++ .../{sail => sail_vm}/_sdk.py | 0 .../{sail => sail_vm}/model_key.py | 2 +- .../{sail => sail_vm}/provider.py | 38 ++++----- .../{sail => sail_vm}/sandbox.py | 8 +- .../sandbox_providers/sandbox_provider.py | 2 +- tst/integration/env/gateway/gateway_test.py | 12 +-- .../env/task_step/task_steps_test.py | 14 ++-- ..._test.py => sail_vm_sandbox_smoke_test.py} | 10 +-- tst/unit/config/test_explain_and_sources.py | 2 +- .../env_gateway_provider_test.py | 4 +- .../network_policy_stamping_test.py | 8 +- ..._key_test.py => sail_vm_model_key_test.py} | 80 +++++++++---------- ...st.py => sail_vm_sandbox_provider_test.py} | 36 ++++----- ...andbox_test.py => sail_vm_sandbox_test.py} | 36 ++++----- .../test_config_sandbox_providers.py | 12 +-- tst/unit/test_capabilities.py | 6 +- tst/util/capabilities.py | 8 +- 25 files changed, 154 insertions(+), 154 deletions(-) delete mode 100644 src/agent_env/providers/sandbox_providers/sail/__init__.py create mode 100644 src/agent_env/providers/sandbox_providers/sail_vm/__init__.py rename src/agent_env/providers/sandbox_providers/{sail => sail_vm}/_sdk.py (100%) rename src/agent_env/providers/sandbox_providers/{sail => sail_vm}/model_key.py (99%) rename src/agent_env/providers/sandbox_providers/{sail => sail_vm}/provider.py (94%) rename src/agent_env/providers/sandbox_providers/{sail => sail_vm}/sandbox.py (99%) rename tst/integration/providers/sandbox_providers/{sail_sandbox_smoke_test.py => sail_vm_sandbox_smoke_test.py} (93%) rename tst/unit/providers/sandbox_providers/{sail_model_key_test.py => sail_vm_model_key_test.py} (88%) rename tst/unit/providers/sandbox_providers/{sail_sandbox_provider_test.py => sail_vm_sandbox_provider_test.py} (93%) rename tst/unit/providers/sandbox_providers/{sail_sandbox_test.py => sail_vm_sandbox_test.py} (89%) diff --git a/AGENTS.md b/AGENTS.md index 202c2ee0..76d8338a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -82,7 +82,7 @@ and deserialized through a registry. | `task/`, `task_step/` | A `Task` holds its `TaskStep`s inline; `Task.run()` executes them as a DAG. `depends_on` (None means all prior steps) gates a step, independent steps run concurrently, `fail_task_on_error` makes a failure fatal or tolerated, `retry_config` rolls a failed span back through the step journal and re-dispatches it. Built-in steps live in `task_step/task_steps/` (`deploy_env`, `deploy_agent`, `prompt_agent`, the verifiers under `verifiers/`, and more). | | `store/` | Four store ABCs with local and cloud implementations: `DocumentStore` (SQLite, MongoDB), `ObjectStore` (filesystem, S3, Cloud Storage), `ImageStore` (local OCI registry, ECR), `SecretStore` (env vars or file, AWS Secrets Manager, Google Cloud Secret Manager). `VersionedEntityStore` implements the shared versioned get/put logic, `QueryBuilder` is the immutable chained query API, `store/base.py` holds the error types. A new backend must pass the conformance kits in `tst/store/`. | | `config/` | The `Config` singleton (`get_config`, `configure`, `reset_config`) in `config/runtime.py`, file discovery in `config/loader.py`, and `load_impl`, which resolves `module:Class` pointers. `agent_env.store` re-exports the config names for compatibility. | -| `providers/` | `providers/sandbox_providers/` holds the sandbox providers `local`, `modal`, `modal_vm`, `e2b`, `sail`; `[sandbox] default` and `agent_default` accept a comma-separated fallback chain. `providers/env_providers/` holds the environment providers: `EnvironmentProvider` (an env's containers and state store) and `EnvironmentGatewayProvider`, which renders a docker-compose for the gateway and its MCP servers inside the sandbox; `providers/env_state/` holds env-state providers (`local_postgres` built in). | +| `providers/` | `providers/sandbox_providers/` holds the sandbox providers `local`, `modal`, `modal_vm`, `e2b`, `sail_vm`; `[sandbox] default` and `agent_default` accept a comma-separated fallback chain. `providers/env_providers/` holds the environment providers: `EnvironmentProvider` (an env's containers and state store) and `EnvironmentGatewayProvider`, which renders a docker-compose for the gateway and its MCP servers inside the sandbox; `providers/env_state/` holds env-state providers (`local_postgres` built in). | | `a2a_agent/` | The `A2AAgent` entity (`a2a_agent`), its stores and the validator steps. The protocol package provides the agent-side framework. | | `runner/` | The `[runner]` seam: `Runner.submit()` returns `(run_id, instance_id)`; `LocalRunner` is built in. | | `explorer/` | Optional local web UI: `agent-env up`, needs the `explorer` extra, binds loopback `:8234`. | diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index c32d66fc..c5ce6975 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -671,8 +671,8 @@ The runtime dependencies of `agentenv-framework` and its `explorer` and `gcp` ex - License: Apache-2.0 - Author: Sail -- Source: -- [License text 1](#license-text-1) +- Source: +- License text: not shipped with the package; see its source ### shellingham 1.5.4 diff --git a/src/agent_env/cli/a2a_agent/deploy.py b/src/agent_env/cli/a2a_agent/deploy.py index 2b9e4721..c835c582 100644 --- a/src/agent_env/cli/a2a_agent/deploy.py +++ b/src/agent_env/cli/a2a_agent/deploy.py @@ -18,7 +18,7 @@ @click.option("--ttl-seconds", type=click.IntRange(min=MIN_TTL_SECONDS, max=MAX_TTL_SECONDS), default=DEFAULT_TTL_SECONDS, help=f"VM lifetime in seconds (default {DEFAULT_TTL_SECONDS})") @click.option("--sandbox", default=None, - help="Sandbox backend(s): a built-in (modal, modal_vm, e2b, sail, local) " + help="Sandbox backend(s): a built-in (modal, modal_vm, e2b, sail_vm, local) " "or a name from [sandbox.providers] in .agentenv/config.toml; comma-separated for a " "fallback chain. Defaults to [sandbox].agent_default (else local) when omitted.") def deploy(agent_id: str, agent_version: int | None, env_var_pairs: tuple[str, ...], ttl_seconds: int, sandbox: str): diff --git a/src/agent_env/cli/env/deploy.py b/src/agent_env/cli/env/deploy.py index 39b949e1..a65bfd7c 100644 --- a/src/agent_env/cli/env/deploy.py +++ b/src/agent_env/cli/env/deploy.py @@ -19,7 +19,7 @@ @click.option("--gateway-mode", type=click.Choice([m.value for m in GatewayMode], case_sensitive=False), default=GatewayMode.PERFORMANCE.value, help="Gateway mode (performance or consistent)") @click.option("--sandbox", default=None, - help="Sandbox backend(s): a built-in (modal, modal_vm, e2b, sail, local) " + help="Sandbox backend(s): a built-in (modal, modal_vm, e2b, sail_vm, local) " "or a name from [sandbox.providers] in .agentenv/config.toml; comma-separated for a " "fallback chain. Defaults to [sandbox].default (else local) when omitted.") @click.option("--service-db", "service_db_env_id", default=None, diff --git a/src/agent_env/providers/__init__.py b/src/agent_env/providers/__init__.py index 25af89da..a1ac301a 100644 --- a/src/agent_env/providers/__init__.py +++ b/src/agent_env/providers/__init__.py @@ -19,7 +19,7 @@ from agent_env.providers.sandbox_providers.modal_sandbox import ModalSandbox, ModalSandboxProvider from agent_env.providers.sandbox_providers.modal_vm_sandbox import ModalVmSandbox, ModalVmSandboxProvider from agent_env.providers.sandbox_providers.e2b import E2BSandbox, E2BSandboxProvider -from agent_env.providers.sandbox_providers.sail import SailSandbox, SailSandboxProvider +from agent_env.providers.sandbox_providers.sail_vm import SailVmSandbox, SailVmSandboxProvider __all__ = [ "ChainedSandboxProvider", @@ -30,7 +30,7 @@ "ModalSandbox", "ModalSandboxProvider", "ModalVmSandbox", "ModalVmSandboxProvider", "E2BSandbox", "E2BSandboxProvider", - "SailSandbox", "SailSandboxProvider", + "SailVmSandbox", "SailVmSandboxProvider", "WebsiteConfig", "build_env_provider", "build_sandbox_provider", "get_sandbox_provider", "set_sandbox_provider", "reset_sandbox_provider", diff --git a/src/agent_env/providers/sandbox_providers/__init__.py b/src/agent_env/providers/sandbox_providers/__init__.py index 7848868d..b41c4b13 100644 --- a/src/agent_env/providers/sandbox_providers/__init__.py +++ b/src/agent_env/providers/sandbox_providers/__init__.py @@ -14,7 +14,7 @@ from agent_env.providers.sandbox_providers.modal_sandbox import ModalSandbox, ModalSandboxProvider from agent_env.providers.sandbox_providers.modal_vm_sandbox import ModalVmSandbox, ModalVmSandboxProvider from agent_env.providers.sandbox_providers.e2b import E2BSandbox, E2BSandboxProvider -from agent_env.providers.sandbox_providers.sail import SailSandbox, SailSandboxProvider +from agent_env.providers.sandbox_providers.sail_vm import SailVmSandbox, SailVmSandboxProvider __all__ = [ "ChainedSandboxProvider", "Sandbox", "VmSandbox", @@ -23,7 +23,7 @@ "ModalSandbox", "ModalSandboxProvider", "ModalVmSandbox", "ModalVmSandboxProvider", "E2BSandbox", "E2BSandboxProvider", - "SailSandbox", "SailSandboxProvider", + "SailVmSandbox", "SailVmSandboxProvider", "build_sandbox_provider", "get_sandbox_provider", "set_sandbox_provider", "reset_sandbox_provider", "get_env_sandbox_provider", "set_env_sandbox_provider", "reset_env_sandbox_provider", diff --git a/src/agent_env/providers/sandbox_providers/sail/__init__.py b/src/agent_env/providers/sandbox_providers/sail/__init__.py deleted file mode 100644 index da11a516..00000000 --- a/src/agent_env/providers/sandbox_providers/sail/__init__.py +++ /dev/null @@ -1,6 +0,0 @@ -"""Sail Research Sailbox sandbox provider package.""" - -from agent_env.providers.sandbox_providers.sail.provider import SailSandboxProvider -from agent_env.providers.sandbox_providers.sail.sandbox import SailSandbox - -__all__ = ["SailSandbox", "SailSandboxProvider"] diff --git a/src/agent_env/providers/sandbox_providers/sail_vm/__init__.py b/src/agent_env/providers/sandbox_providers/sail_vm/__init__.py new file mode 100644 index 00000000..25402fc6 --- /dev/null +++ b/src/agent_env/providers/sandbox_providers/sail_vm/__init__.py @@ -0,0 +1,6 @@ +"""Sail Research Sailbox sandbox provider package.""" + +from agent_env.providers.sandbox_providers.sail_vm.provider import SailVmSandboxProvider +from agent_env.providers.sandbox_providers.sail_vm.sandbox import SailVmSandbox + +__all__ = ["SailVmSandbox", "SailVmSandboxProvider"] diff --git a/src/agent_env/providers/sandbox_providers/sail/_sdk.py b/src/agent_env/providers/sandbox_providers/sail_vm/_sdk.py similarity index 100% rename from src/agent_env/providers/sandbox_providers/sail/_sdk.py rename to src/agent_env/providers/sandbox_providers/sail_vm/_sdk.py diff --git a/src/agent_env/providers/sandbox_providers/sail/model_key.py b/src/agent_env/providers/sandbox_providers/sail_vm/model_key.py similarity index 99% rename from src/agent_env/providers/sandbox_providers/sail/model_key.py rename to src/agent_env/providers/sandbox_providers/sail_vm/model_key.py index c4c3d128..d952e3d8 100644 --- a/src/agent_env/providers/sandbox_providers/sail/model_key.py +++ b/src/agent_env/providers/sandbox_providers/sail_vm/model_key.py @@ -107,7 +107,7 @@ def for_env(cls, env: Mapping[str, str]) -> ModelKeyInjection | None: if parsed.scheme != "https" or not parsed.hostname: raise ValueError( f"Sail injects the model key only into HTTPS requests, but {BASE_URL_ENV} is {base_url!r}; " - "use an https endpoint or set inject_model_key = false in [sandbox.providers.sail.config]" + "use an https endpoint or set inject_model_key = false in [sandbox.providers.sail_vm.config]" ) return cls(host=parsed.hostname, secret=secret_name(key), key=key) diff --git a/src/agent_env/providers/sandbox_providers/sail/provider.py b/src/agent_env/providers/sandbox_providers/sail_vm/provider.py similarity index 94% rename from src/agent_env/providers/sandbox_providers/sail/provider.py rename to src/agent_env/providers/sandbox_providers/sail_vm/provider.py index 14e7c427..09fe23c7 100644 --- a/src/agent_env/providers/sandbox_providers/sail/provider.py +++ b/src/agent_env/providers/sandbox_providers/sail_vm/provider.py @@ -12,11 +12,11 @@ from agent_env.attribution import PIPELINE_STEP_KEY, RUN_ID_KEY, Attribution from agent_env.config import get_config from agent_env.config.errors import ConfigError -from agent_env.providers.sandbox_providers.sail import _sdk -from agent_env.providers.sandbox_providers.sail.model_key import ModelKeyInjection -from agent_env.providers.sandbox_providers.sail.sandbox import ( +from agent_env.providers.sandbox_providers.sail_vm import _sdk +from agent_env.providers.sandbox_providers.sail_vm.model_key import ModelKeyInjection +from agent_env.providers.sandbox_providers.sail_vm.sandbox import ( MAX_ALLOWLIST_ENTRIES, - SailSandbox, + SailVmSandbox, create_saved_policy, egress_document, policy_name, @@ -31,7 +31,7 @@ logger = logging.getLogger(__name__) -SANDBOX_STARTED_EVENT = "agent_env.sail_sandbox_started" +SANDBOX_STARTED_EVENT = "agent_env.sail_vm_sandbox_started" #: (size, vCPU, (min, max) memory GiB, (min, max) disk GiB), smallest first. Memory and disk are #: ceilings, not reservations: Sail bills observed usage. @@ -69,7 +69,7 @@ def sailbox_name(attribution: Attribution) -> str: return "-".join(["ae", uuid.uuid4().hex[:8], *filter(None, slugs)])[:_MAX_NAME_LENGTH].rstrip("-") -async def _reap(sandbox: SailSandbox) -> None: +async def _reap(sandbox: SailVmSandbox) -> None: for attempt in range(_REAP_ATTEMPTS): try: await sandbox.terminate() @@ -85,7 +85,7 @@ async def _reap(sandbox: SailSandbox) -> None: async def _create_or_reclaim( - create: Any, wrap: Callable[[Any], SailSandbox], release: Callable[[], Awaitable[None]] | None = None, + create: Any, wrap: Callable[[Any], SailVmSandbox], release: Callable[[], Awaitable[None]] | None = None, ) -> Any: """Await a Sailbox create. If the caller is cancelled first, terminate (``wrap``ped, so its model-key policy and secret go too) the Sailbox it yields, which would otherwise keep running with no handle; if @@ -110,7 +110,7 @@ def terminate_orphan(done: asyncio.Future) -> None: raise -class SailSandboxProvider(SandboxProvider): +class SailVmSandboxProvider(SandboxProvider): """Docker-capable Sailboxes. ``api_key`` comes from resolved provider config (a ``secret:`` reference) and never reaches a workload. With ``inject_model_key`` (the default) neither does an agent's model key: Sail adds it to the agent's requests to the model endpoint (see ``model_key``).""" @@ -140,11 +140,11 @@ def __init__( self._app: Any | None = None def __repr__(self) -> str: - return f"SailSandboxProvider(app={self._app_name!r})" + return f"SailVmSandboxProvider(app={self._app_name!r})" @classmethod def from_config(cls, **config: Any) -> Self: - section = "[sandbox.providers.sail.config]" + section = "[sandbox.providers.sail_vm.config]" api_key = config.get("api_key") if not isinstance(api_key, str) or not api_key.strip(): raise ConfigError(f"{section} requires a non-empty 'api_key' (e.g. \"secret:sail_api_key\")") @@ -204,7 +204,7 @@ async def create_vm( setup_for_gateway: bool = True, attribution: Attribution | None = None, network_policy: NetworkPolicy | None = None, - ) -> SailSandbox: + ) -> SailVmSandbox: """Create a Sailbox from the devbox image; ``timeout`` is its hard maximum lifetime.""" if image is not None: raise ValueError("the Sail provider boots its own Docker-capable image; image overrides are unsupported") @@ -226,7 +226,7 @@ async def _create( attribution: Attribution | None, network_policy: NetworkPolicy | None, injection: ModelKeyInjection | None, - ) -> SailSandbox: + ) -> SailVmSandbox: size, memory_gib, disk_gib = sailbox_shape(cpu, memory, disk_size_gb, min_size=self._min_size) effective_policy = self.effective_network_policy(network_policy) if injection is not None: @@ -245,8 +245,8 @@ async def _create( launch = f"launch-{uuid.uuid4().hex}" - def wrap(raw: Any) -> SailSandbox: - sandbox = SailSandbox( + def wrap(raw: Any) -> SailVmSandbox: + sandbox = SailVmSandbox( raw, sdk=sdk, tunnel_urls={}, network_policy=effective_policy, injection=injection, refuse_model_keys=self._inject_model_key, ) @@ -287,7 +287,7 @@ def wrap(raw: Any) -> SailSandbox: sandbox.tunnel_urls = await self._tunnel_urls(raw, ports) sandbox.mode = SANDBOX_MODE_VM logger.info( - "Sail sandbox started: sailbox_id=%s app=%s size=%s memory=%sGiB disk=%sGiB model_key_injected=%s attribution=%s", + "Sail VM sandbox started: sailbox_id=%s app=%s size=%s memory=%sGiB disk=%sGiB model_key_injected=%s attribution=%s", raw.sailbox_id, self._app_name, size, memory_gib, disk_gib, injection is not None, resolved_attribution, extra={ "event": SANDBOX_STARTED_EVENT, @@ -343,7 +343,7 @@ async def create_sandbox( timeout: int = 3600 * 2, attribution: Attribution | None = None, network_policy: NetworkPolicy | None = None, - ) -> SailSandbox: + ) -> SailVmSandbox: """A bare VM the caller loads and starts ``image_name`` in, as on the other VM providers. A model key in ``env`` is injected by Sail rather than passed in (unless ``inject_model_key`` is off).""" del image_name @@ -353,7 +353,7 @@ async def create_sandbox( setup_for_gateway=True, attribution=attribution, network_policy=network_policy, injection=injection, ) - async def create_container(self, **kwargs: Any) -> SailSandbox: + async def create_container(self, **kwargs: Any) -> SailVmSandbox: """The inherited login-pull-run, then the registry credentials removed from the VM disk, which Sail checkpoints for host-failure recovery.""" sandbox = await super().create_container(**kwargs) @@ -364,7 +364,7 @@ async def create_container(self, **kwargs: Any) -> SailSandbox: raise return sandbox - async def get_sandbox(self, sandbox_id: str) -> SailSandbox: + async def get_sandbox(self, sandbox_id: str) -> SailVmSandbox: """Reconnect, restoring ports, the applied egress policy and any model-key injection. The key itself is recovered only when it is the configured ``[model]`` key, so the reconnected handle scrubs it too.""" sdk, _ = await self._connect() @@ -381,7 +381,7 @@ async def get_sandbox(self, sandbox_id: str) -> SailSandbox: injection.recover_key(get_config().get_litellm_api_key()) except Exception: # noqa: BLE001 - no configured key to recover; the handle just can't scrub it pass - sandbox = SailSandbox( + sandbox = SailVmSandbox( raw, sdk=sdk, tunnel_urls=tunnel_urls, network_policy=None, injection=injection, refuse_model_keys=self._inject_model_key, ) diff --git a/src/agent_env/providers/sandbox_providers/sail/sandbox.py b/src/agent_env/providers/sandbox_providers/sail_vm/sandbox.py similarity index 99% rename from src/agent_env/providers/sandbox_providers/sail/sandbox.py rename to src/agent_env/providers/sandbox_providers/sail_vm/sandbox.py index dcf5f4db..a27bac93 100644 --- a/src/agent_env/providers/sandbox_providers/sail/sandbox.py +++ b/src/agent_env/providers/sandbox_providers/sail_vm/sandbox.py @@ -14,7 +14,7 @@ from urllib.parse import urlparse from agent_env.config import get_config -from agent_env.providers.sandbox_providers.sail.model_key import ( +from agent_env.providers.sandbox_providers.sail_vm.model_key import ( DOCKER_SHIM_PATH, ModelKeyInjection, carries_model_key, @@ -222,12 +222,12 @@ async def wait(self) -> int: return exit_code -class SailSandbox(VmSandbox): +class SailVmSandbox(VmSandbox): """A Sailbox from the Docker-capable devbox image; commands run as root. With a model-key ``injection``, the key's value never enters the Sailbox: every command and file is scrubbed of it. With ``refuse_model_keys``, one that still carries a model key (one Sail doesn't inject here) is refused.""" - type = "sail" + type = "sail_vm" _DOCKER_PROBE_TIMEOUT = 10 def __init__( @@ -361,7 +361,7 @@ def _refusal(self) -> ModelKeyRefusedError: return ModelKeyRefusedError( f"Refusing to send a model key into Sailbox {self.sandbox_id}: Sail injects model keys only for an agent " "deployed on its own Sail sandbox, so this one would land on the Sailbox's disk. Deploy the agent on its " - "own sandbox, or set inject_model_key = false in [sandbox.providers.sail.config] to pass keys in." + "own sandbox, or set inject_model_key = false in [sandbox.providers.sail_vm.config] to pass keys in." ) async def apply_network_policy(self, policy: NetworkPolicy) -> None: diff --git a/src/agent_env/providers/sandbox_providers/sandbox_provider.py b/src/agent_env/providers/sandbox_providers/sandbox_provider.py index 61099bcb..a3732c17 100644 --- a/src/agent_env/providers/sandbox_providers/sandbox_provider.py +++ b/src/agent_env/providers/sandbox_providers/sandbox_provider.py @@ -222,7 +222,7 @@ def effective_network_policy(cls, policy: Optional[NetworkPolicy]) -> NetworkPol "modal": "agent_env.providers.sandbox_providers.modal_sandbox:ModalSandboxProvider", "modal_vm": "agent_env.providers.sandbox_providers.modal_vm_sandbox:ModalVmSandboxProvider", "e2b": "agent_env.providers.sandbox_providers.e2b:E2BSandboxProvider", - "sail": "agent_env.providers.sandbox_providers.sail:SailSandboxProvider", + "sail_vm": "agent_env.providers.sandbox_providers.sail_vm:SailVmSandboxProvider", "local": "agent_env.providers.sandbox_providers.local_sandbox:LocalSandboxProvider", } diff --git a/tst/integration/env/gateway/gateway_test.py b/tst/integration/env/gateway/gateway_test.py index 74a86ec7..c4d609f0 100644 --- a/tst/integration/env/gateway/gateway_test.py +++ b/tst/integration/env/gateway/gateway_test.py @@ -447,9 +447,9 @@ def agentenv_website_env() -> WebsiteEnv: marks=skip_without_remote_sandbox("modal_vm"), ), pytest.param( - "sail", - id="sail", - marks=skip_without_remote_sandbox("sail"), + "sail_vm", + id="sail_vm", + marks=skip_without_remote_sandbox("sail_vm"), ), ]) def sandbox_provider(request): @@ -469,9 +469,9 @@ def sandbox_provider(request): elif request.param == "modal_vm": set_sandbox_provider(ModalVmSandboxProvider()) set_env_sandbox_provider(ModalVmSandboxProvider()) - elif request.param == "sail": - set_sandbox_provider(build_sandbox_provider("sail")) - set_env_sandbox_provider(build_sandbox_provider("sail")) + elif request.param == "sail_vm": + set_sandbox_provider(build_sandbox_provider("sail_vm")) + set_env_sandbox_provider(build_sandbox_provider("sail_vm")) try: yield request.param finally: diff --git a/tst/integration/env/task_step/task_steps_test.py b/tst/integration/env/task_step/task_steps_test.py index 1911a340..c351f045 100644 --- a/tst/integration/env/task_step/task_steps_test.py +++ b/tst/integration/env/task_step/task_steps_test.py @@ -275,9 +275,9 @@ def echo_agent() -> A2AAgent: marks=skip_without_remote_sandbox("modal_vm"), ), pytest.param( - "sail", - id="sail", - marks=skip_without_remote_sandbox("sail"), + "sail_vm", + id="sail_vm", + marks=skip_without_remote_sandbox("sail_vm"), ), ]) def sandbox_provider(request): @@ -301,10 +301,10 @@ def sandbox_provider(request): set_sandbox_provider(ModalVmSandboxProvider()) set_env_sandbox_provider(ModalVmSandboxProvider()) set_agent_sandbox_provider(ModalVmSandboxProvider()) - elif request.param == "sail": - set_sandbox_provider(build_sandbox_provider("sail")) - set_env_sandbox_provider(build_sandbox_provider("sail")) - set_agent_sandbox_provider(build_sandbox_provider("sail")) + elif request.param == "sail_vm": + set_sandbox_provider(build_sandbox_provider("sail_vm")) + set_env_sandbox_provider(build_sandbox_provider("sail_vm")) + set_agent_sandbox_provider(build_sandbox_provider("sail_vm")) try: yield request.param finally: diff --git a/tst/integration/providers/sandbox_providers/sail_sandbox_smoke_test.py b/tst/integration/providers/sandbox_providers/sail_vm_sandbox_smoke_test.py similarity index 93% rename from tst/integration/providers/sandbox_providers/sail_sandbox_smoke_test.py rename to tst/integration/providers/sandbox_providers/sail_vm_sandbox_smoke_test.py index 24652518..c0906a96 100644 --- a/tst/integration/providers/sandbox_providers/sail_sandbox_smoke_test.py +++ b/tst/integration/providers/sandbox_providers/sail_vm_sandbox_smoke_test.py @@ -1,6 +1,6 @@ """Sail Sailbox smoke test. -Runs when the resolved config can build the ``sail`` sandbox provider (``[sandbox.providers.sail.config]`` +Runs when the resolved config can build the ``sail_vm`` sandbox provider (``[sandbox.providers.sail_vm.config]`` with ``api_key = "secret:sail_api_key"``), the ``remote_sandbox`` capability, and skips with the declared reason otherwise. The test never reads or prints the API key. """ @@ -15,7 +15,7 @@ import pytest import pytest_asyncio -from agent_env.providers.sandbox_providers.sail.model_key import PLACEHOLDER, secret_name +from agent_env.providers.sandbox_providers.sail_vm.model_key import PLACEHOLDER, secret_name from agent_env.providers.sandbox_providers.sandbox import NetworkMode, NetworkPolicy from agent_env.providers.sandbox_providers.sandbox_provider import build_sandbox_provider from tst.util.capabilities import skip_without_remote_sandbox @@ -26,13 +26,13 @@ pytest.mark.integration, pytest.mark.int_test_slow, pytest.mark.asyncio, - skip_without_remote_sandbox("sail"), + skip_without_remote_sandbox("sail_vm"), ] @pytest.fixture(scope="module") def sail_provider(): - return build_sandbox_provider("sail") + return build_sandbox_provider("sail_vm") @pytest_asyncio.fixture(scope="module") @@ -45,7 +45,7 @@ async def sail_sandbox(sail_provider): async def test_sailbox_runs_docker_serves_its_port_and_reconnects(sail_provider, sail_sandbox): - assert (sail_sandbox.type, sail_sandbox.mode) == ("sail", "vm") + assert (sail_sandbox.type, sail_sandbox.mode) == ("sail_vm", "vm") await sail_sandbox.exec_script( f"docker run -d --name web -p {_PORT}:80 public.ecr.aws/nginx/nginx:alpine > /dev/null" diff --git a/tst/unit/config/test_explain_and_sources.py b/tst/unit/config/test_explain_and_sources.py index 624a5839..2e894796 100644 --- a/tst/unit/config/test_explain_and_sources.py +++ b/tst/unit/config/test_explain_and_sources.py @@ -160,7 +160,7 @@ def test_sources_says_how_the_file_was_found(config_file, tmp_path, monkeypatch) assert "walking up" in next(s for s in sources() if s.kind == KIND_FILE).detail -# Read for identity (USER), or set from [sandbox.providers.sail.config] for the Sail SDK to read +# Read for identity (USER), or set from [sandbox.providers.sail_vm.config] for the Sail SDK to read # (SAIL_*), never configuration agent-env takes, so none is a layer. _NOT_CONFIGURATION = {"USER", "SAIL_API_KEY", "SAIL_RUNTIME_THREADS"} diff --git a/tst/unit/providers/env_providers/env_gateway_provider_test.py b/tst/unit/providers/env_providers/env_gateway_provider_test.py index 7703c9d4..380eaaa6 100644 --- a/tst/unit/providers/env_providers/env_gateway_provider_test.py +++ b/tst/unit/providers/env_providers/env_gateway_provider_test.py @@ -603,14 +603,14 @@ async def test_read_trajectory_of_a_gateway_with_no_history_yet_is_empty(): async def test_sail_provider_routes_to_vm_path_not_containers(): """A Sailbox is a Docker-capable VM: the gateway deploys onto it with docker-compose.""" from agent_env.env.gateway import GatewayMode - from agent_env.providers.sandbox_providers.sail import SailSandboxProvider + from agent_env.providers.sandbox_providers.sail_vm import SailVmSandboxProvider gp = EnvironmentGatewayProvider() gp._deploy_via_vm = AsyncMock(return_value="VM_RESULT") gp._deploy_via_containers = AsyncMock(return_value="CONTAINER_RESULT") result = await gp.create_gateway( - sandbox_provider=SailSandboxProvider(api_key="sail-test-key"), + sandbox_provider=SailVmSandboxProvider(api_key="sail-test-key"), mcp_servers=[MCPServerConfig(image="mcp-a", environment_name="a")], mcp_server_images=[MagicMock(image_name="mcp-a")], gateway_mode=GatewayMode.PERFORMANCE, diff --git a/tst/unit/providers/sandbox_providers/network_policy_stamping_test.py b/tst/unit/providers/sandbox_providers/network_policy_stamping_test.py index 68f0f943..0b801a53 100644 --- a/tst/unit/providers/sandbox_providers/network_policy_stamping_test.py +++ b/tst/unit/providers/sandbox_providers/network_policy_stamping_test.py @@ -17,8 +17,8 @@ from agent_env.providers.sandbox_providers.local_sandbox import LocalSandboxProvider from agent_env.providers.sandbox_providers.modal_sandbox import ModalSandboxProvider from agent_env.providers.sandbox_providers.modal_vm_sandbox import ModalVmSandboxProvider -from agent_env.providers.sandbox_providers.sail import _sdk as sail_sdk -from agent_env.providers.sandbox_providers.sail.provider import SailSandboxProvider +from agent_env.providers.sandbox_providers.sail_vm import _sdk as sail_sdk +from agent_env.providers.sandbox_providers.sail_vm.provider import SailVmSandboxProvider from agent_env.providers.sandbox_providers.sandbox import Sandbox from agent_env.providers.sandbox_providers.sandbox_provider import _BUILTIN_SANDBOX_PROVIDERS @@ -82,7 +82,7 @@ async def _sail() -> Sandbox: sdk = MagicMock() sdk.Sailbox.create.aio = AsyncMock(return_value=sailbox) with patch.object(sail_sdk, "connect", return_value=(sdk, SimpleNamespace(id="app_test"))): - provider = SailSandboxProvider(api_key="test-key", sdk=sdk) + provider = SailVmSandboxProvider(api_key="test-key", sdk=sdk) return await provider.create_vm(exposed_ports=[], setup_for_gateway=False) @@ -90,7 +90,7 @@ async def _sail() -> Sandbox: "modal": _modal, "modal_vm": _modal_vm, "e2b": _e2b, - "sail": _sail, + "sail_vm": _sail, "local": _local, } diff --git a/tst/unit/providers/sandbox_providers/sail_model_key_test.py b/tst/unit/providers/sandbox_providers/sail_vm_model_key_test.py similarity index 88% rename from tst/unit/providers/sandbox_providers/sail_model_key_test.py rename to tst/unit/providers/sandbox_providers/sail_vm_model_key_test.py index 641092f7..5cd1e92c 100644 --- a/tst/unit/providers/sandbox_providers/sail_model_key_test.py +++ b/tst/unit/providers/sandbox_providers/sail_vm_model_key_test.py @@ -9,9 +9,9 @@ import pytest -from agent_env.providers.sandbox_providers.sail import _sdk -from agent_env.providers.sandbox_providers.sail import model_key -from agent_env.providers.sandbox_providers.sail.model_key import ( +from agent_env.providers.sandbox_providers.sail_vm import _sdk +from agent_env.providers.sandbox_providers.sail_vm import model_key +from agent_env.providers.sandbox_providers.sail_vm.model_key import ( DOCKER_SHIM_PATH, PLACEHOLDER, ModelKeyInjection, @@ -19,10 +19,10 @@ docker_shim, secret_name, ) -from agent_env.providers.sandbox_providers.sail.provider import SailSandboxProvider -from agent_env.providers.sandbox_providers.sail.sandbox import ( +from agent_env.providers.sandbox_providers.sail_vm.provider import SailVmSandboxProvider +from agent_env.providers.sandbox_providers.sail_vm.sandbox import ( ModelKeyRefusedError, - SailSandbox, + SailVmSandbox, create_saved_policy, delete_policy_named, ) @@ -95,7 +95,7 @@ def fresh_key_state(monkeypatch): @pytest.fixture(autouse=True) def no_vm_setup(monkeypatch): - monkeypatch.setattr(SailSandbox, "setup_vm_for_gateway", AsyncMock()) + monkeypatch.setattr(SailVmSandbox, "setup_vm_for_gateway", AsyncMock()) def _process(stdout=b""): @@ -191,7 +191,7 @@ async def test_an_agent_sailbox_gets_its_key_through_a_secret_and_a_saved_policy sailbox = _sailbox() sdk, saved, _ = _fake_sdk(sailbox) - sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) sdk.Secret.set.aio.assert_awaited_once_with(secret_name(_KEY), _KEY) (policy,) = saved @@ -209,7 +209,7 @@ async def test_a_restricted_agent_sailbox_allows_the_model_endpoint_it_injects_i sdk, saved, _ = _fake_sdk(_sailbox()) restricted = NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",)) - sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( image_name="agent:1", port=8000, env=_ENV, network_policy=restricted, ) @@ -220,7 +220,7 @@ async def test_a_restricted_agent_sailbox_allows_the_model_endpoint_it_injects_i @pytest.mark.asyncio async def test_injection_can_be_turned_off_to_pass_the_key_in(): sdk, saved, _ = _fake_sdk(_sailbox()) - sandbox = await SailSandboxProvider(api_key="sail-key", inject_model_key=False, sdk=sdk).create_sandbox( + sandbox = await SailVmSandboxProvider(api_key="sail-key", inject_model_key=False, sdk=sdk).create_sandbox( image_name="agent:1", port=8000, env={**_ENV, "LITELLM_BASE_URL": "http://llm.internal:4000"}, ) sdk.Secret.set.aio.assert_not_awaited() @@ -232,7 +232,7 @@ async def test_injection_can_be_turned_off_to_pass_the_key_in(): async def test_a_plain_endpoint_is_refused_before_anything_is_created(): sdk, saved, _ = _fake_sdk(_sailbox()) with pytest.raises(ValueError, match="only into HTTPS requests"): - await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( + await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( image_name="agent:1", port=8000, env={**_ENV, "LITELLM_BASE_URL": "http://llm.internal:4000"}, ) sdk.Secret.set.aio.assert_not_awaited() @@ -244,7 +244,7 @@ async def test_a_failed_create_deletes_its_policy_but_never_the_secret(): sdk, saved, secret = _fake_sdk(_sailbox()) sdk.Sailbox.create.aio = AsyncMock(side_effect=RuntimeError("no capacity")) with pytest.raises(RuntimeError, match="no capacity"): - await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) saved[0].delete.aio.assert_awaited_once() secret.delete.aio.assert_not_awaited() assert secret_name(_KEY) not in model_key._injected_keys @@ -261,7 +261,7 @@ async def hang(name, document): sdk.EgressPolicy.create.aio = AsyncMock(side_effect=hang) task = asyncio.ensure_future( - SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) ) await started.wait() task.cancel() @@ -284,7 +284,7 @@ async def create(**_kwargs): sdk.Sailbox.create.aio = create task = asyncio.ensure_future( - SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) ) await started.wait() task.cancel() @@ -301,7 +301,7 @@ async def create(**_kwargs): @pytest.mark.asyncio async def test_gateway_and_plain_vms_never_inject(): sdk, saved, _ = _fake_sdk(_sailbox(ports=())) - sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_vm(exposed_ports=[]) + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_vm(exposed_ports=[]) assert saved == [] and sandbox._injection is None @@ -310,7 +310,7 @@ async def test_commands_and_files_carry_the_placeholder_never_the_key(): sailbox = _sailbox() sailbox.exec.aio = AsyncMock(return_value=_process()) sdk, _, _ = _fake_sdk(sailbox) - sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) await sandbox.exec_script(f"docker run -d -e LITELLM_API_KEY='{_KEY}' -e ANTHROPIC_API_KEY={_KEY} agent:1") await sandbox.write_host_file(f"key: {_KEY}\n".encode(), "/opt/agent/config.yaml") @@ -326,7 +326,7 @@ async def test_commands_and_files_carry_the_placeholder_never_the_key(): async def test_terminate_deletes_the_policy_and_keeps_the_secret(): sailbox = _sailbox() sdk, saved, secret = _fake_sdk(sailbox) - sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) await sandbox.terminate() @@ -340,7 +340,7 @@ async def test_terminate_deletes_the_policy_and_keeps_the_secret(): async def test_widening_an_injected_sailbox_replaces_its_saved_policy(): sailbox = _sailbox() sdk, saved, _ = _fake_sdk(sailbox) - sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( image_name="agent:1", port=8000, env=_ENV, network_policy=NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",)), ) @@ -358,7 +358,7 @@ async def test_widening_an_injected_sailbox_replaces_its_saved_policy(): async def test_a_failed_policy_swap_deletes_the_replacement_and_keeps_the_old_one(): sailbox = _sailbox() sdk, saved, _ = _fake_sdk(sailbox) - sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( image_name="agent:1", port=8000, env=_ENV, network_policy=NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",)), ) @@ -382,10 +382,10 @@ async def test_reconnect_in_another_process_scrubs_the_configured_key(monkeypatc sdk, _, _ = _fake_sdk(sailbox) monkeypatch.setattr(model_key, "_injected_keys", {}) monkeypatch.setattr( - "agent_env.providers.sandbox_providers.sail.provider.get_config", lambda: MagicMock(get_litellm_api_key=lambda: _KEY) + "agent_env.providers.sandbox_providers.sail_vm.provider.get_config", lambda: MagicMock(get_litellm_api_key=lambda: _KEY) ) - sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).get_sandbox("sb_1") + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).get_sandbox("sb_1") assert sandbox.network_policy == NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("llm.example.com",)) assert sandbox._injection.policy_id == "ep_9" @@ -398,7 +398,7 @@ async def test_reconnect_treats_someone_elses_saved_policy_as_unknown(): sailbox = _sailbox() sailbox.egress_policy = SimpleNamespace(policy_id="ep_7", document={"allowlist": ["a.example"]}) sdk, _, _ = _fake_sdk(sailbox) - sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).get_sandbox("sb_1") + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).get_sandbox("sb_1") assert sandbox.network_policy is None and sandbox._injection is None @@ -407,13 +407,13 @@ async def test_reconnect_in_the_deploying_process_scrubs_the_agents_own_key(monk agent_key = "sk-agent-override-key-42" sailbox = _sailbox() sdk, _, _ = _fake_sdk(sailbox) - provider = SailSandboxProvider(api_key="sail-key", sdk=sdk) + provider = SailVmSandboxProvider(api_key="sail-key", sdk=sdk) created = await provider.create_sandbox(image_name="agent:1", port=8000, env={**_ENV, "LITELLM_API_KEY": agent_key}) (policy,) = [c.args[1] for c in sdk.EgressPolicy.create.aio.await_args_list] sailbox.egress_policy = SimpleNamespace(policy_id=created._injection.policy_id, document=policy) sailbox.exec.aio = AsyncMock(return_value=_process()) monkeypatch.setattr( - "agent_env.providers.sandbox_providers.sail.provider.get_config", lambda: MagicMock(get_litellm_api_key=lambda: _KEY) + "agent_env.providers.sandbox_providers.sail_vm.provider.get_config", lambda: MagicMock(get_litellm_api_key=lambda: _KEY) ) reconnected = await provider.get_sandbox("sb_1") @@ -458,7 +458,7 @@ async def test_a_dollar_value_in_a_plain_argument_is_a_literal_key_and_refused() sailbox = _sailbox(ports=()) sailbox.exec.aio = AsyncMock(return_value=_process()) sdk, _, _ = _fake_sdk(sailbox) - sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_vm(exposed_ports=[]) + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_vm(exposed_ports=[]) with pytest.raises(ModelKeyRefusedError): await sandbox.exec_with_output("docker", "exec", "-e", "LITELLM_API_KEY=$looks-like-a-var", "agent-api", "true") await sandbox.exec_with_output("bash", "-c", 'docker exec -e LITELLM_API_KEY="$LITELLM_API_KEY" agent-api true') @@ -469,7 +469,7 @@ async def test_a_sailbox_without_injection_refuses_a_model_key_rather_than_take_ sailbox = _sailbox(ports=()) sailbox.exec.aio = AsyncMock(return_value=_process()) sdk, _, _ = _fake_sdk(sailbox) - sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_vm(exposed_ports=[]) + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_vm(exposed_ports=[]) sailbox.exec.aio.reset_mock() with pytest.raises(ModelKeyRefusedError, match="Deploy the agent on its own sandbox"): @@ -487,7 +487,7 @@ async def test_with_injection_off_keys_pass_through_as_before(): sailbox = _sailbox(ports=()) sailbox.exec.aio = AsyncMock(return_value=_process()) sdk, _, _ = _fake_sdk(sailbox) - sandbox = await SailSandboxProvider(api_key="sail-key", inject_model_key=False, sdk=sdk).create_vm(exposed_ports=[]) + sandbox = await SailVmSandboxProvider(api_key="sail-key", inject_model_key=False, sdk=sdk).create_vm(exposed_ports=[]) await sandbox.exec_script(f"docker run -e LITELLM_API_KEY='{_KEY}' agent:1") assert _KEY in sailbox.exec.aio.await_args.args[0][2] @@ -500,9 +500,9 @@ async def test_a_reconnected_handle_that_cannot_recover_the_key_refuses_it(monke sdk, _, _ = _fake_sdk(sailbox) monkeypatch.setattr(model_key, "_injected_keys", {}) monkeypatch.setattr( - "agent_env.providers.sandbox_providers.sail.provider.get_config", lambda: MagicMock(get_litellm_api_key=lambda: "other") + "agent_env.providers.sandbox_providers.sail_vm.provider.get_config", lambda: MagicMock(get_litellm_api_key=lambda: "other") ) - sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).get_sandbox("sb_1") + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).get_sandbox("sb_1") with pytest.raises(ModelKeyRefusedError): await sandbox.exec_with_output("docker", "exec", "-e", f"LITELLM_API_KEY={_KEY}", "agent-api", "pytest") sailbox.exec.aio.assert_not_awaited() @@ -512,7 +512,7 @@ async def test_a_reconnected_handle_that_cannot_recover_the_key_refuses_it(monke async def test_terminate_deletes_the_policy_actually_applied_even_if_another_handle_replaced_it(): sailbox = _sailbox() sdk, saved, _ = _fake_sdk(sailbox) - sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) replacement = await sdk.EgressPolicy.create.aio("agentenv-replacement", saved[0].document) sailbox.egress_policy = SimpleNamespace(policy_id=replacement.id, document=replacement.document) @@ -542,7 +542,7 @@ async def test_the_key_is_remembered_until_the_last_box_here_that_needs_it_is_go second.sailbox_id = "sb_2" sdk, _, secret = _fake_sdk(first) sdk.Sailbox.create.aio = AsyncMock(side_effect=[first, second]) - provider = SailSandboxProvider(api_key="sail-key", sdk=sdk) + provider = SailVmSandboxProvider(api_key="sail-key", sdk=sdk) one = await provider.create_sandbox(image_name="agent:1", port=8000, env=_ENV) two = await provider.create_sandbox(image_name="agent:1", port=8000, env=_ENV) @@ -570,7 +570,7 @@ async def create(**_kwargs): raise RuntimeError("no capacity") sdk.Sailbox.create.aio = create - provider = SailSandboxProvider(api_key="sail-key", sdk=sdk) + provider = SailVmSandboxProvider(api_key="sail-key", sdk=sdk) pending = asyncio.ensure_future(provider.create_sandbox(image_name="agent:1", port=8000, env=_ENV)) await asyncio.wait_for(first_waiting.wait(), timeout=5) with pytest.raises(RuntimeError, match="no capacity"): @@ -586,13 +586,13 @@ async def create(**_kwargs): async def test_terminate_waits_for_a_policy_swap_in_flight_on_another_handle(): sailbox = _sailbox() sdk, saved, _ = _fake_sdk(sailbox) - provider = SailSandboxProvider(api_key="sail-key", sdk=sdk) + provider = SailVmSandboxProvider(api_key="sail-key", sdk=sdk) sandbox = await provider.create_sandbox( image_name="agent:1", port=8000, env=_ENV, network_policy=NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",)), ) sailbox.egress_policy = SimpleNamespace(policy_id=saved[0].id, document=saved[0].document) - other = SailSandbox(sailbox, sdk=sdk, tunnel_urls={}, network_policy=sandbox.network_policy, + other = SailVmSandbox(sailbox, sdk=sdk, tunnel_urls={}, network_policy=sandbox.network_policy, injection=ModelKeyInjection.from_document(saved[0].document, saved[0].id)) swapping, release = asyncio.Event(), asyncio.Event() @@ -617,7 +617,7 @@ async def slow_set(policy): @pytest.mark.asyncio async def test_lost_response_cleanup_retries_through_a_brief_outage(monkeypatch, caplog): - monkeypatch.setattr("agent_env.providers.sandbox_providers.sail.sandbox.asyncio.sleep", AsyncMock()) + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail_vm.sandbox.asyncio.sleep", AsyncMock()) sdk, saved, _ = _fake_sdk(_sailbox()) await sdk.EgressPolicy.create.aio("agentenv-lost", {"rules": {}}) listing = sdk.EgressPolicy.list.aio.side_effect @@ -630,7 +630,7 @@ async def test_lost_response_cleanup_retries_through_a_brief_outage(monkeypatch, @pytest.mark.asyncio async def test_lost_response_cleanup_that_never_reaches_sail_names_the_policy_to_sweep(monkeypatch, caplog): - monkeypatch.setattr("agent_env.providers.sandbox_providers.sail.sandbox.asyncio.sleep", AsyncMock()) + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail_vm.sandbox.asyncio.sleep", AsyncMock()) sdk, _, _ = _fake_sdk(_sailbox()) sdk.EgressPolicy.list.aio = AsyncMock(side_effect=RuntimeError("down")) @@ -642,7 +642,7 @@ async def test_lost_response_cleanup_that_never_reaches_sail_names_the_policy_to @pytest.mark.asyncio async def test_lost_response_cleanup_retries_a_failed_delete_too(monkeypatch): - monkeypatch.setattr("agent_env.providers.sandbox_providers.sail.sandbox.asyncio.sleep", AsyncMock()) + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail_vm.sandbox.asyncio.sleep", AsyncMock()) sdk, saved, _ = _fake_sdk(_sailbox()) await sdk.EgressPolicy.create.aio("agentenv-lost", {"rules": {}}) saved[0].delete.aio = AsyncMock(side_effect=[RuntimeError("down"), None]) @@ -656,7 +656,7 @@ async def test_lost_response_cleanup_retries_a_failed_delete_too(monkeypatch): async def test_a_direct_policy_update_waits_for_terminate_on_another_handle(): sailbox = _sailbox() sdk, saved, _ = _fake_sdk(sailbox) - sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( image_name="agent:1", port=8000, env=_ENV, network_policy=NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",)), ) @@ -668,7 +668,7 @@ async def slow_get(_id): return SimpleNamespace(egress_policy=SimpleNamespace(policy_id=saved[0].id, document=saved[0].document)) sdk.Sailbox.get.aio = AsyncMock(side_effect=slow_get) - other = SailSandbox(sailbox, sdk=sdk, tunnel_urls={}, network_policy=sandbox.network_policy, + other = SailVmSandbox(sailbox, sdk=sdk, tunnel_urls={}, network_policy=sandbox.network_policy, injection=ModelKeyInjection.from_document(saved[0].document, saved[0].id)) teardown = asyncio.ensure_future(sandbox.terminate()) await asyncio.wait_for(reading.wait(), timeout=5) @@ -690,7 +690,7 @@ async def test_a_key_with_an_apostrophe_is_scrubbed_in_its_shell_encoded_form(qu sailbox = _sailbox() sailbox.exec.aio = AsyncMock(return_value=_process()) sdk, _, _ = _fake_sdk(sailbox) - sandbox = await SailSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( image_name="agent:1", port=8000, env={**_ENV, "LITELLM_API_KEY": key}, ) encoded = "'" + key.replace("'", "'\\''") + "'" if quoting == "agent" else shlex.quote(key) diff --git a/tst/unit/providers/sandbox_providers/sail_sandbox_provider_test.py b/tst/unit/providers/sandbox_providers/sail_vm_sandbox_provider_test.py similarity index 93% rename from tst/unit/providers/sandbox_providers/sail_sandbox_provider_test.py rename to tst/unit/providers/sandbox_providers/sail_vm_sandbox_provider_test.py index a9603e22..3a1bb9ba 100644 --- a/tst/unit/providers/sandbox_providers/sail_sandbox_provider_test.py +++ b/tst/unit/providers/sandbox_providers/sail_vm_sandbox_provider_test.py @@ -10,15 +10,15 @@ import pytest from agent_env.config.errors import ConfigError -from agent_env.providers.sandbox_providers.sail import _sdk -from agent_env.providers.sandbox_providers.sail import provider as provider_module -from agent_env.providers.sandbox_providers.sail.provider import ( +from agent_env.providers.sandbox_providers.sail_vm import _sdk +from agent_env.providers.sandbox_providers.sail_vm import provider as provider_module +from agent_env.providers.sandbox_providers.sail_vm.provider import ( SANDBOX_STARTED_EVENT, - SailSandboxProvider, + SailVmSandboxProvider, sailbox_name, sailbox_shape, ) -from agent_env.providers.sandbox_providers.sail.sandbox import SailSandbox +from agent_env.providers.sandbox_providers.sail_vm.sandbox import SailVmSandbox from agent_env.providers.sandbox_providers.sandbox import NetworkMode, NetworkPolicy, NetworkPolicyUnsupportedError @@ -69,12 +69,12 @@ def fresh_key_state(monkeypatch): @pytest.fixture def setup(monkeypatch): setup = AsyncMock() - monkeypatch.setattr(SailSandbox, "setup_vm_for_gateway", setup) + monkeypatch.setattr(SailVmSandbox, "setup_vm_for_gateway", setup) return setup def _provider(sdk, **config): - return SailSandboxProvider(api_key="sail-secret", sdk=sdk, **config) + return SailVmSandboxProvider(api_key="sail-secret", sdk=sdk, **config) @pytest.mark.parametrize( @@ -93,12 +93,12 @@ def _provider(sdk, **config): ) def test_from_config_rejects_invalid_config(config, message): with pytest.raises(ConfigError, match=message): - SailSandboxProvider.from_config(**config) + SailVmSandboxProvider.from_config(**config) def test_construction_neither_imports_the_sdk_nor_sets_the_key(monkeypatch): monkeypatch.delitem(sys.modules, "sail", raising=False) - SailSandboxProvider.from_config(api_key="sail-secret") + SailVmSandboxProvider.from_config(api_key="sail-secret") assert "sail" not in sys.modules assert _sdk.API_KEY_ENV not in os.environ @@ -154,8 +154,8 @@ async def test_create_vm_sends_the_shape_lifetime_ports_and_policy(setup): assert kwargs["auto_sleep"] == "never" assert kwargs["egress_policy"] == {} assert "api_key" not in kwargs and "env" not in kwargs - assert isinstance(sandbox, SailSandbox) - assert (sandbox.type, sandbox.mode, sandbox.sandbox_id) == ("sail", "vm", "sb_1") + assert isinstance(sandbox, SailVmSandbox) + assert (sandbox.type, sandbox.mode, sandbox.sandbox_id) == ("sail_vm", "vm", "sb_1") assert sandbox.tunnel_urls == {8080: "https://sb-1-8080.sail.box", 9000: "https://sb-1-9000.sail.box"} assert sandbox.network_policy == NetworkPolicy() setup.assert_awaited_once_with([8080, 9000]) @@ -195,7 +195,7 @@ async def test_allowlist_becomes_a_sail_allowlist_with_the_platform_floor(setup) ) async def test_an_unenforceable_policy_is_refused_before_provisioning(policy): sdk = _fake_sdk(_sailbox()) - assert SailSandboxProvider.supports_network_policy(policy) is False + assert SailVmSandboxProvider.supports_network_policy(policy) is False with pytest.raises(NetworkPolicyUnsupportedError): await _provider(sdk).create_vm(exposed_ports=[], network_policy=policy) sdk.Sailbox.create.aio.assert_not_awaited() @@ -220,7 +220,7 @@ async def test_a_setup_failure_terminates_the_sailbox(setup): @pytest.mark.asyncio async def test_tunnel_urls_wait_until_every_port_is_routed(setup, monkeypatch): - monkeypatch.setattr("agent_env.providers.sandbox_providers.sail.provider._LISTENER_POLL_INTERVAL", 0) + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail_vm.provider._LISTENER_POLL_INTERVAL", 0) sailbox = _sailbox() sailbox.listeners.aio = AsyncMock(side_effect=[[_listener(8080, url=False)], [_listener(8080)]]) sandbox = await _provider(_fake_sdk(sailbox)).create_vm(exposed_ports=[8080]) @@ -229,8 +229,8 @@ async def test_tunnel_urls_wait_until_every_port_is_routed(setup, monkeypatch): @pytest.mark.asyncio async def test_a_port_that_never_routes_fails_the_create(setup, monkeypatch): - monkeypatch.setattr("agent_env.providers.sandbox_providers.sail.provider._LISTENER_POLL_INTERVAL", 0) - monkeypatch.setattr("agent_env.providers.sandbox_providers.sail.provider._LISTENER_TIMEOUT", 0) + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail_vm.provider._LISTENER_POLL_INTERVAL", 0) + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail_vm.provider._LISTENER_TIMEOUT", 0) sailbox = _sailbox() with pytest.raises(RuntimeError, match=r"no public URL for port\(s\) \[8080\]"): await _provider(_fake_sdk(sailbox)).create_vm(exposed_ports=[8080]) @@ -261,7 +261,7 @@ async def slow_create(**_kwargs): @pytest.mark.asyncio async def test_a_failing_orphan_termination_is_retried_then_reported(monkeypatch, caplog): - monkeypatch.setattr("agent_env.providers.sandbox_providers.sail.provider.asyncio.sleep", AsyncMock()) + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail_vm.provider.asyncio.sleep", AsyncMock()) sandbox = MagicMock(sandbox_id="sb_1") sandbox.terminate = AsyncMock(side_effect=RuntimeError("api down")) @@ -273,7 +273,7 @@ async def test_a_failing_orphan_termination_is_retried_then_reported(monkeypatch @pytest.mark.asyncio async def test_create_logs_the_attribution_join_event_without_the_key(setup, caplog): - caplog.set_level(logging.INFO, logger="agent_env.providers.sandbox_providers.sail.provider") + caplog.set_level(logging.INFO, logger="agent_env.providers.sandbox_providers.sail_vm.provider") await _provider(_fake_sdk(_sailbox())).create_vm(exposed_ports=[], attribution={"run_id": "inst-1", "team": "t"}) (record,) = [r for r in caplog.records if getattr(r, "event", None) == SANDBOX_STARTED_EVENT] @@ -301,7 +301,7 @@ async def test_image_overrides_are_refused(): @pytest.mark.asyncio async def test_create_container_removes_the_registry_login_from_the_vm(monkeypatch): - sandbox = MagicMock(spec=SailSandbox) + sandbox = MagicMock(spec=SailVmSandbox) sandbox.exec_script = AsyncMock() monkeypatch.setattr( "agent_env.providers.sandbox_providers.sandbox_provider.SandboxProvider.create_container", diff --git a/tst/unit/providers/sandbox_providers/sail_sandbox_test.py b/tst/unit/providers/sandbox_providers/sail_vm_sandbox_test.py similarity index 89% rename from tst/unit/providers/sandbox_providers/sail_sandbox_test.py rename to tst/unit/providers/sandbox_providers/sail_vm_sandbox_test.py index 64ff58f1..e60027a7 100644 --- a/tst/unit/providers/sandbox_providers/sail_sandbox_test.py +++ b/tst/unit/providers/sandbox_providers/sail_vm_sandbox_test.py @@ -7,7 +7,7 @@ import pytest -from agent_env.providers.sandbox_providers.sail.sandbox import SailSandbox, egress_document, policy_from_document +from agent_env.providers.sandbox_providers.sail_vm.sandbox import SailVmSandbox, egress_document, policy_from_document from agent_env.providers.sandbox_providers.sandbox import NetworkMode, NetworkPolicy @@ -51,7 +51,7 @@ async def set_policy(document): sdk = SimpleNamespace(**vars(_SDK), Sailbox=SimpleNamespace(get=SimpleNamespace(aio=AsyncMock( side_effect=lambda _id: SimpleNamespace(egress_policy=SimpleNamespace(policy_id=None, document=applied["document"])) )))) - return SailSandbox(sailbox, sdk=sdk, tunnel_urls={}, network_policy=policy), sailbox + return SailVmSandbox(sailbox, sdk=sdk, tunnel_urls={}, network_policy=policy), sailbox @pytest.mark.asyncio @@ -174,7 +174,7 @@ async def test_other_sdk_errors_propagate(): @pytest.mark.asyncio async def test_wait_for_vm_starts_dockerd_once_when_it_is_not_running(monkeypatch): - monkeypatch.setattr("agent_env.providers.sandbox_providers.sail.sandbox.asyncio.sleep", AsyncMock()) + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail_vm.sandbox.asyncio.sleep", AsyncMock()) sandbox, sailbox = _sandbox( _process(stderr=b"Cannot connect to the Docker daemon", exit_code=1), _process(), @@ -188,12 +188,12 @@ async def test_wait_for_vm_starts_dockerd_once_when_it_is_not_running(monkeypatc assert commands[0][:2] == ["docker", "info"] assert commands[1][:2] == ["bash", "-c"] and "nohup dockerd" in commands[1][2] assert [c[:2] for c in commands[2:]] == [["docker", "info"], ["docker", "info"]] - assert sailbox.exec.aio.await_args_list[0].kwargs["timeout"] == SailSandbox._DOCKER_PROBE_TIMEOUT + assert sailbox.exec.aio.await_args_list[0].kwargs["timeout"] == SailVmSandbox._DOCKER_PROBE_TIMEOUT @pytest.mark.asyncio async def test_wait_for_vm_reports_the_last_error_when_docker_never_answers(monkeypatch): - monkeypatch.setattr(SailSandbox, "_VM_READY_TIMEOUT", 0) + monkeypatch.setattr(SailVmSandbox, "_VM_READY_TIMEOUT", 0) sandbox, _ = _sandbox(_process(stderr=b"daemon down", exit_code=1)) with pytest.raises(RuntimeError, match="Docker not ready in Sailbox sb_1.*daemon down"): await sandbox.wait_for_vm() @@ -201,7 +201,7 @@ async def test_wait_for_vm_reports_the_last_error_when_docker_never_answers(monk @pytest.mark.asyncio async def test_setup_requires_compose_v2(monkeypatch): - monkeypatch.setattr(SailSandbox, "wait_for_vm", AsyncMock()) + monkeypatch.setattr(SailVmSandbox, "wait_for_vm", AsyncMock()) sandbox, _ = _sandbox(_process(stderr=b"docker: 'compose' is not a docker command", exit_code=1)) with pytest.raises(RuntimeError, match="no Docker Compose v2"): await sandbox.setup_vm_for_gateway([8080]) @@ -232,11 +232,11 @@ async def test_image_loading_fails_closed_when_the_policy_is_unknown(): async def test_image_loading_widens_an_allowlist_with_the_signed_download_hosts(monkeypatch): sandbox, sailbox = _sandbox(policy=_ALLOWLIST) monkeypatch.setattr( - SailSandbox, "_signed_image_urls", + SailVmSandbox, "_signed_image_urls", AsyncMock(return_value=["https://bucket.s3.amazonaws.com/a?sig=1", None]), ) load = AsyncMock() - monkeypatch.setattr(SailSandbox, "_load_docker_images", load) + monkeypatch.setattr(SailVmSandbox, "_load_docker_images", load) await sandbox.load_docker_images(["a", "b"]) @@ -250,8 +250,8 @@ async def test_image_loading_widens_an_allowlist_with_the_signed_download_hosts( @pytest.mark.asyncio async def test_image_loading_leaves_an_allow_all_policy_alone(monkeypatch): sandbox, sailbox = _sandbox() - monkeypatch.setattr(SailSandbox, "_signed_image_urls", AsyncMock(return_value=["https://x.example/a"])) - monkeypatch.setattr(SailSandbox, "_load_docker_images", AsyncMock()) + monkeypatch.setattr(SailVmSandbox, "_signed_image_urls", AsyncMock(return_value=["https://x.example/a"])) + monkeypatch.setattr(SailVmSandbox, "_load_docker_images", AsyncMock()) await sandbox.load_docker_images(["a"]) sailbox.set_egress_policy.aio.assert_not_awaited() sandbox._sdk.Sailbox.get.aio.assert_not_awaited() @@ -261,7 +261,7 @@ async def test_image_loading_leaves_an_allow_all_policy_alone(monkeypatch): async def test_widening_past_sails_allowlist_limit_is_refused(monkeypatch): full = NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=tuple(f"h{i}.example" for i in range(128))) sandbox, sailbox = _sandbox(policy=full) - monkeypatch.setattr(SailSandbox, "_signed_image_urls", AsyncMock(return_value=["https://bucket.example/a"])) + monkeypatch.setattr(SailVmSandbox, "_signed_image_urls", AsyncMock(return_value=["https://bucket.example/a"])) with pytest.raises(RuntimeError, match="exceed Sail's 128-entry egress allowlist"): await sandbox.load_docker_images(["a"]) sailbox.set_egress_policy.aio.assert_not_awaited() @@ -273,8 +273,8 @@ async def test_a_host_under_an_allowed_wildcard_needs_no_new_entry(monkeypatch): mode=NetworkMode.ALLOWLIST, allow_hosts=("*.s3.amazonaws.com", *(f"h{i}.example" for i in range(127))) ) sandbox, sailbox = _sandbox(policy=full) - monkeypatch.setattr(SailSandbox, "_signed_image_urls", AsyncMock(return_value=["https://bucket.s3.amazonaws.com/a"])) - monkeypatch.setattr(SailSandbox, "_load_docker_images", AsyncMock()) + monkeypatch.setattr(SailVmSandbox, "_signed_image_urls", AsyncMock(return_value=["https://bucket.s3.amazonaws.com/a"])) + monkeypatch.setattr(SailVmSandbox, "_load_docker_images", AsyncMock()) await sandbox.load_docker_images(["a"]) sailbox.set_egress_policy.aio.assert_not_awaited() sandbox._sdk.Sailbox.get.aio.assert_not_awaited() @@ -283,7 +283,7 @@ async def test_a_host_under_an_allowed_wildcard_needs_no_new_entry(monkeypatch): @pytest.mark.asyncio async def test_concurrent_downloads_through_separate_handles_keep_each_others_hosts(): sandbox, sailbox = _sandbox(policy=_ALLOWLIST) - other_handle = SailSandbox(sailbox, sdk=sandbox._sdk, tunnel_urls={}, network_policy=_ALLOWLIST) + other_handle = SailVmSandbox(sailbox, sdk=sandbox._sdk, tunnel_urls={}, network_policy=_ALLOWLIST) await asyncio.gather( sandbox._allow_download_hosts(["https://a.example/x"], "download"), @@ -296,8 +296,8 @@ async def test_concurrent_downloads_through_separate_handles_keep_each_others_ho @pytest.mark.asyncio async def test_a_host_already_allowed_is_not_reapplied(monkeypatch): sandbox, sailbox = _sandbox(policy=_ALLOWLIST) - monkeypatch.setattr(SailSandbox, "_signed_image_urls", AsyncMock(return_value=["https://pypi.org/a"])) - monkeypatch.setattr(SailSandbox, "_load_docker_images", AsyncMock()) + monkeypatch.setattr(SailVmSandbox, "_signed_image_urls", AsyncMock(return_value=["https://pypi.org/a"])) + monkeypatch.setattr(SailVmSandbox, "_load_docker_images", AsyncMock()) await sandbox.load_docker_images(["a"]) sailbox.set_egress_policy.aio.assert_not_awaited() @@ -308,7 +308,7 @@ def _object_store(monkeypatch, signed): open=MagicMock(side_effect=lambda _url: io.BytesIO(b"\x00payload")), ) monkeypatch.setattr( - "agent_env.providers.sandbox_providers.sail.sandbox.get_config", lambda: MagicMock(get_object_store=lambda: store) + "agent_env.providers.sandbox_providers.sail_vm.sandbox.get_config", lambda: MagicMock(get_object_store=lambda: store) ) return store @@ -330,7 +330,7 @@ async def test_a_signed_object_download_allows_its_host_first(monkeypatch): @pytest.mark.asyncio async def test_an_unsignable_object_is_streamed_through_the_filesystem_api(monkeypatch): _object_store(monkeypatch, None) - monkeypatch.setattr("agent_env.providers.sandbox_providers.sail.sandbox._STREAM_CHUNK_BYTES", 4) + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail_vm.sandbox._STREAM_CHUNK_BYTES", 4) sandbox, sailbox = _sandbox(policy=None) writer = MagicMock(write=AsyncMock()) stream = MagicMock(__aenter__=AsyncMock(return_value=writer), __aexit__=AsyncMock(return_value=False)) diff --git a/tst/unit/providers/sandbox_providers/test_config_sandbox_providers.py b/tst/unit/providers/sandbox_providers/test_config_sandbox_providers.py index 635dd78a..5d233b8a 100644 --- a/tst/unit/providers/sandbox_providers/test_config_sandbox_providers.py +++ b/tst/unit/providers/sandbox_providers/test_config_sandbox_providers.py @@ -11,7 +11,7 @@ from agent_env.config import reset_config from agent_env.providers.sandbox_providers.e2b.provider import E2BSandboxProvider from agent_env.providers.sandbox_providers.local_sandbox import LocalSandboxProvider -from agent_env.providers.sandbox_providers.sail.provider import SailSandboxProvider +from agent_env.providers.sandbox_providers.sail_vm.provider import SailVmSandboxProvider from agent_env.providers.sandbox_providers.sandbox import Sandbox from agent_env.providers.sandbox_providers.sandbox_provider import ( SandboxProvider, @@ -217,7 +217,7 @@ def test_e2b_missing_base_template_is_a_config_error(monkeypatch, tmp_path): def test_sail_builtin_receives_interpolated_key_without_touching_the_sdk(monkeypatch, tmp_path): cfg = _write_config(tmp_path, """ - [sandbox.providers.sail.config] + [sandbox.providers.sail_vm.config] api_key = "env:SAIL_TEST_API_KEY" app = "agent-env-test" auto_sleep_min_idle_seconds = 600 @@ -226,9 +226,9 @@ def test_sail_builtin_receives_interpolated_key_without_touching_the_sdk(monkeyp monkeypatch.setenv("SAIL_TEST_API_KEY", "resolved-sail-key") monkeypatch.delenv("SAIL_API_KEY", raising=False) - provider = build_sandbox_provider("sail") + provider = build_sandbox_provider("sail_vm") - assert isinstance(provider, SailSandboxProvider) + assert isinstance(provider, SailVmSandboxProvider) assert provider._api_key == "resolved-sail-key" assert provider._app_name == "agent-env-test" assert provider._auto_sleep_min_idle_seconds == 600 @@ -238,13 +238,13 @@ def test_sail_builtin_receives_interpolated_key_without_touching_the_sdk(monkeyp def test_sail_missing_api_key_is_a_config_error(monkeypatch, tmp_path): cfg = _write_config(tmp_path, """ - [sandbox.providers.sail.config] + [sandbox.providers.sail_vm.config] app = "agent-env-test" """) monkeypatch.setenv("AGENT_ENV_CONFIG", str(cfg)) with pytest.raises(ConfigError, match="requires a non-empty 'api_key'"): - build_sandbox_provider("sail") + build_sandbox_provider("sail_vm") def test_builtin_config_reaches_chain_members(monkeypatch, tmp_path): diff --git a/tst/unit/test_capabilities.py b/tst/unit/test_capabilities.py index 40c21b20..3aecc9d8 100644 --- a/tst/unit/test_capabilities.py +++ b/tst/unit/test_capabilities.py @@ -169,14 +169,14 @@ def test_e2b_other_failures_read_as_present_rather_than_absent(monkeypatch): @pytest.mark.parametrize( ("outcome", "available"), - [(ConfigError("[sandbox.providers.sail.config] requires a non-empty 'api_key'"), False), (object(), True), (RuntimeError("boom"), True)], + [(ConfigError("[sandbox.providers.sail_vm.config] requires a non-empty 'api_key'"), False), (object(), True), (RuntimeError("boom"), True)], ) def test_sail_availability_follows_the_provider_build(monkeypatch, outcome, available): def build(spec): - assert spec == "sail" + assert spec == "sail_vm" if isinstance(outcome, Exception): raise outcome return outcome monkeypatch.setattr("agent_env.providers.sandbox_providers.sandbox_provider.build_sandbox_provider", build) - assert capabilities.remote_sandbox_is_available("sail") is available + assert capabilities.remote_sandbox_is_available("sail_vm") is available diff --git a/tst/util/capabilities.py b/tst/util/capabilities.py index 74182919..ef86c8f6 100644 --- a/tst/util/capabilities.py +++ b/tst/util/capabilities.py @@ -19,7 +19,7 @@ #: Only the agent-driving paths need a model endpoint. MODEL_ENDPOINT = "model_endpoint_configured" -#: The remote sandbox providers (``modal``, ``modal_vm``, ``e2b``, ``sail``) need credentials the resolved +#: The remote sandbox providers (``modal``, ``modal_vm``, ``e2b``, ``sail_vm``) need credentials the resolved #: config may not carry; the local default never does. REMOTE_SANDBOX = "remote_sandbox" @@ -75,16 +75,16 @@ def skip_without_model_endpoint() -> pytest.MarkDecorator: def remote_sandbox_is_available(provider: str) -> bool: - """Whether the resolved config can build the ``modal`` / ``modal_vm`` / ``e2b`` / ``sail`` sandbox + """Whether the resolved config can build the ``modal`` / ``modal_vm`` / ``e2b`` / ``sail_vm`` sandbox provider, credentials included. Modal: any failure to resolve the credentials answers False. E2B and Sail: building the provider resolves ``[sandbox.providers..config]`` and its ``secret:`` references; only a ``ConfigError`` (absent or incomplete config) answers False, any other failure answers True and lets the test fail on the real problem, as ``model_endpoint_is_configured`` does.""" - if provider not in ("modal", "modal_vm", "e2b", "sail"): + if provider not in ("modal", "modal_vm", "e2b", "sail_vm"): raise ValueError(f"unknown remote sandbox provider {provider!r}") - if provider in ("e2b", "sail"): + if provider in ("e2b", "sail_vm"): from agent_env.config.errors import ConfigError from agent_env.providers.sandbox_providers.sandbox_provider import build_sandbox_provider From 5ee577a6e795ae05d6ff18e4465657c4467a4e91 Mon Sep 17 00:00:00 2001 From: Pratyush Singhal Date: Wed, 7 Oct 2026 11:21:08 -0700 Subject: [PATCH 10/11] refactor(sail_vm): keep the provider in sandbox_providers, as an optional extra Review follow-ups for the sail_vm provider. - sail is an optional extra (pip install 'agentenv-framework[sail]'); dev includes it. The package's __init__ imports nothing and the registry points at sail_vm.provider, so core never loads the SDK and selecting sail_vm without the extra names the extra to install. - The source change stays in sandbox_providers/: the CLI help and the providers/__init__ exports are reverted, and the gateway exec-transport fixes leave this PR for their own. - sail_vm/README.md documents install, config, sizing, networking, model-key injection, attribution and caveats. - Guard: shell words are decoded as the shell reads them (a quoted or escaped name is still a key), a value passes only when it is wholly one variable reference, and YAML/JSON KEY: value is caught in files. Keys under 16 characters are refused, since scrubbing replaces every occurrence. - Policies: every saved policy for a Sailbox shares a name prefix, so teardown from any handle in any process deletes them all; a swap whose outcome Sail can't report keeps both ids, one that applied is settled even when cancelled, and a create whose response is lost terminates the Sailbox it made (by name, with retries) before releasing its policy. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- THIRD_PARTY_NOTICES.md | 2 +- pyproject.toml | 7 +- src/agent_env/cli/a2a_agent/deploy.py | 2 +- src/agent_env/cli/env/deploy.py | 2 +- src/agent_env/providers/__init__.py | 2 - .../env_providers/env_gateway_provider.py | 64 ++---- .../providers/sandbox_providers/__init__.py | 2 - .../sandbox_providers/sail_vm/README.md | 111 ++++++++++ .../sandbox_providers/sail_vm/__init__.py | 7 +- .../sandbox_providers/sail_vm/_sdk.py | 6 +- .../sandbox_providers/sail_vm/model_key.py | 71 +++++-- .../sandbox_providers/sail_vm/provider.py | 37 +++- .../sandbox_providers/sail_vm/sandbox.py | 58 ++++-- .../sandbox_providers/sandbox_provider.py | 2 +- .../env_gateway_provider_test.py | 74 +------ .../sail_vm_model_key_test.py | 191 +++++++++++++++++- .../sail_vm_sandbox_provider_test.py | 15 +- .../test_optional_extras_stay_optional.py | 15 +- uv.lock | 11 +- 20 files changed, 503 insertions(+), 178 deletions(-) create mode 100644 src/agent_env/providers/sandbox_providers/sail_vm/README.md diff --git a/AGENTS.md b/AGENTS.md index 76d8338a..c920f419 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -82,7 +82,7 @@ and deserialized through a registry. | `task/`, `task_step/` | A `Task` holds its `TaskStep`s inline; `Task.run()` executes them as a DAG. `depends_on` (None means all prior steps) gates a step, independent steps run concurrently, `fail_task_on_error` makes a failure fatal or tolerated, `retry_config` rolls a failed span back through the step journal and re-dispatches it. Built-in steps live in `task_step/task_steps/` (`deploy_env`, `deploy_agent`, `prompt_agent`, the verifiers under `verifiers/`, and more). | | `store/` | Four store ABCs with local and cloud implementations: `DocumentStore` (SQLite, MongoDB), `ObjectStore` (filesystem, S3, Cloud Storage), `ImageStore` (local OCI registry, ECR), `SecretStore` (env vars or file, AWS Secrets Manager, Google Cloud Secret Manager). `VersionedEntityStore` implements the shared versioned get/put logic, `QueryBuilder` is the immutable chained query API, `store/base.py` holds the error types. A new backend must pass the conformance kits in `tst/store/`. | | `config/` | The `Config` singleton (`get_config`, `configure`, `reset_config`) in `config/runtime.py`, file discovery in `config/loader.py`, and `load_impl`, which resolves `module:Class` pointers. `agent_env.store` re-exports the config names for compatibility. | -| `providers/` | `providers/sandbox_providers/` holds the sandbox providers `local`, `modal`, `modal_vm`, `e2b`, `sail_vm`; `[sandbox] default` and `agent_default` accept a comma-separated fallback chain. `providers/env_providers/` holds the environment providers: `EnvironmentProvider` (an env's containers and state store) and `EnvironmentGatewayProvider`, which renders a docker-compose for the gateway and its MCP servers inside the sandbox; `providers/env_state/` holds env-state providers (`local_postgres` built in). | +| `providers/` | `providers/sandbox_providers/` holds the sandbox providers `local`, `modal`, `modal_vm`, `e2b`, `sail_vm` (the `sail` extra); `[sandbox] default` and `agent_default` accept a comma-separated fallback chain. `providers/env_providers/` holds the environment providers: `EnvironmentProvider` (an env's containers and state store) and `EnvironmentGatewayProvider`, which renders a docker-compose for the gateway and its MCP servers inside the sandbox; `providers/env_state/` holds env-state providers (`local_postgres` built in). | | `a2a_agent/` | The `A2AAgent` entity (`a2a_agent`), its stores and the validator steps. The protocol package provides the agent-side framework. | | `runner/` | The `[runner]` seam: `Runner.submit()` returns `(run_id, instance_id)`; `LocalRunner` is built in. | | `explorer/` | Optional local web UI: `agent-env up`, needs the `explorer` extra, binds loopback `:8234`. | diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index c5ce6975..51e2fb0c 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -14,7 +14,7 @@ marks, or contributor names to endorse or promote Scale AI, or related products. ## Python dependencies -The runtime dependencies of `agentenv-framework` and its `explorer` and `gcp` extras, at the versions resolved in `uv.lock`. Development-only dependencies are not listed. +The runtime dependencies of `agentenv-framework` and its `explorer`, `gcp` and `sail` extras, at the versions resolved in `uv.lock`. Development-only dependencies are not listed. ### a2a-sdk 0.3.26 diff --git a/pyproject.toml b/pyproject.toml index 0228e4f5..312ddeb6 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -47,7 +47,6 @@ dependencies = [ "uvicorn>=0.47", # 0.47 adds ssl_context_factory, how the local grant server gets its TLS context "modal>=1.5.2.dev22", # dev22 adds _experimental_create(tags=); dev2 added VM-mode V2 (vm_runtime) "e2b~=2.46.4", # AsyncSandbox/AsyncTemplate APIs validated against the 2.46 line - "sail~=0.12.8", # Sailbox exec/fs/listener/egress APIs validated against the 0.12 line ] [project.optional-dependencies] @@ -65,8 +64,12 @@ gcp = [ "google-cloud-storage>=3.0", "requests>=2.31", ] +# The sail_vm sandbox provider (Sail Research Sailboxes), selected with --sandbox sail_vm. +sail = [ + "sail~=0.12.8", # Sailbox exec/fs/listener/egress APIs validated against the 0.12 line +] dev = [ - "agentenv-framework[explorer,gcp]", # so the explorer and GCP backend tests run in CI + "agentenv-framework[explorer,gcp,sail]", # so the explorer, GCP backend and Sail provider tests run in CI "griffelib==2.3.0", # the plugin API check; exact, since what it reports as a break changes between releases "moto>=5.0.0", "psycopg2-binary>=2.9.0", # the gateway server module's driver; outside its container only the tests import it diff --git a/src/agent_env/cli/a2a_agent/deploy.py b/src/agent_env/cli/a2a_agent/deploy.py index c835c582..5c6ae687 100644 --- a/src/agent_env/cli/a2a_agent/deploy.py +++ b/src/agent_env/cli/a2a_agent/deploy.py @@ -18,7 +18,7 @@ @click.option("--ttl-seconds", type=click.IntRange(min=MIN_TTL_SECONDS, max=MAX_TTL_SECONDS), default=DEFAULT_TTL_SECONDS, help=f"VM lifetime in seconds (default {DEFAULT_TTL_SECONDS})") @click.option("--sandbox", default=None, - help="Sandbox backend(s): a built-in (modal, modal_vm, e2b, sail_vm, local) " + help="Sandbox backend(s): a built-in (modal, modal_vm, e2b, local) " "or a name from [sandbox.providers] in .agentenv/config.toml; comma-separated for a " "fallback chain. Defaults to [sandbox].agent_default (else local) when omitted.") def deploy(agent_id: str, agent_version: int | None, env_var_pairs: tuple[str, ...], ttl_seconds: int, sandbox: str): diff --git a/src/agent_env/cli/env/deploy.py b/src/agent_env/cli/env/deploy.py index a65bfd7c..770e5e38 100644 --- a/src/agent_env/cli/env/deploy.py +++ b/src/agent_env/cli/env/deploy.py @@ -19,7 +19,7 @@ @click.option("--gateway-mode", type=click.Choice([m.value for m in GatewayMode], case_sensitive=False), default=GatewayMode.PERFORMANCE.value, help="Gateway mode (performance or consistent)") @click.option("--sandbox", default=None, - help="Sandbox backend(s): a built-in (modal, modal_vm, e2b, sail_vm, local) " + help="Sandbox backend(s): a built-in (modal, modal_vm, e2b, local) " "or a name from [sandbox.providers] in .agentenv/config.toml; comma-separated for a " "fallback chain. Defaults to [sandbox].default (else local) when omitted.") @click.option("--service-db", "service_db_env_id", default=None, diff --git a/src/agent_env/providers/__init__.py b/src/agent_env/providers/__init__.py index a1ac301a..3c403dcb 100644 --- a/src/agent_env/providers/__init__.py +++ b/src/agent_env/providers/__init__.py @@ -19,7 +19,6 @@ from agent_env.providers.sandbox_providers.modal_sandbox import ModalSandbox, ModalSandboxProvider from agent_env.providers.sandbox_providers.modal_vm_sandbox import ModalVmSandbox, ModalVmSandboxProvider from agent_env.providers.sandbox_providers.e2b import E2BSandbox, E2BSandboxProvider -from agent_env.providers.sandbox_providers.sail_vm import SailVmSandbox, SailVmSandboxProvider __all__ = [ "ChainedSandboxProvider", @@ -30,7 +29,6 @@ "ModalSandbox", "ModalSandboxProvider", "ModalVmSandbox", "ModalVmSandboxProvider", "E2BSandbox", "E2BSandboxProvider", - "SailVmSandbox", "SailVmSandboxProvider", "WebsiteConfig", "build_env_provider", "build_sandbox_provider", "get_sandbox_provider", "set_sandbox_provider", "reset_sandbox_provider", diff --git a/src/agent_env/providers/env_providers/env_gateway_provider.py b/src/agent_env/providers/env_providers/env_gateway_provider.py index d39162dc..291d4b11 100644 --- a/src/agent_env/providers/env_providers/env_gateway_provider.py +++ b/src/agent_env/providers/env_providers/env_gateway_provider.py @@ -46,8 +46,6 @@ logger = logging.getLogger(__name__) -_TRAJECTORY_READ_ATTEMPTS = 3 - # MCP server health-check budget. Some servers wait on an upstream before they bind; a server # that is up passes its first probe, so the long start period only delays failing a broken one. MCP_HC_RETRIES = 30 @@ -757,7 +755,12 @@ async def _stand_up(_spec): # Print gateway logs for debugging logger.info("Gateway logs:") - await self._log_service(sandbox, GATEWAY_SERVICE_NAME) + gateway_container_id = await self._get_container_id(sandbox, GATEWAY_SERVICE_NAME) + if gateway_container_id: + exit_code, logs, stderr = await sandbox.exec_with_output( "sudo", "docker", "logs", gateway_container_id) + logger.info(f" stdout:\n{logs}") + if stderr: + logger.info(f" stderr:\n{stderr}") if not gateway_ready: raise RuntimeError("Gateway did not become ready in time") @@ -767,8 +770,13 @@ async def _stand_up(_spec): if self._needs_local_postgres else [] ) for name in sidecar_names: - logger.info(f"{name} logs:") - if not await self._log_service(sandbox, name): + container_id = await self._get_container_id(sandbox, name) + if container_id: + exit_code, logs, stderr = await sandbox.exec_with_output("sudo", "docker", "logs", container_id) + logger.info(f"{name} logs:\n stdout:\n{logs}") + if stderr: + logger.info(f" stderr:\n{stderr}") + else: logger.warning(f"{name} container not found") gateway_url = sandbox.tunnel_urls.get(gateway_port) @@ -979,55 +987,25 @@ async def _wait_for_gateway(self, sandbox: VmSandbox, port: int, timeout: int = return False async def read_trajectory(self, sandbox: VmSandbox) -> list[dict]: - """Read trajectory JSONL from gateway container. A read whose exec transport failed (exit -1) may - hold only part of the history, so it is retried and then raised, never parsed.""" + """Read trajectory JSONL from gateway container.""" container_id = await self._get_container_id(sandbox, GATEWAY_SERVICE_NAME) if not container_id: return [] - for attempt in range(_TRAJECTORY_READ_ATTEMPTS): - exit_code, stdout, stderr = await sandbox.exec_with_output( - "sudo", "docker", "exec", container_id, "cat", "/var/log/agentenv/trajectory.jsonl" - ) - if exit_code != -1: - break - logger.warning(f"Trajectory read lost its exec transport (attempt {attempt + 1}): {stderr[-200:]}") - else: - raise RuntimeError(f"Could not read the gateway trajectory: exec transport failed {_TRAJECTORY_READ_ATTEMPTS} times") + exit_code, stdout, stderr = await sandbox.exec_with_output( + "sudo", "docker", "exec", container_id, "cat", "/var/log/agentenv/trajectory.jsonl" + ) events = [] for line in stdout.strip().split("\n"): if line: events.append(json.loads(line)) return events - async def _log_service(self, sandbox: VmSandbox, compose_service: str) -> bool: - """Log a compose service's container output; False when it has no container. Diagnostics only, so - a lookup that keeps losing its exec transport is reported, never raised into the deploy.""" - try: - container_id = await self._get_container_id(sandbox, compose_service) - except RuntimeError as e: - logger.warning(f"Skipping {compose_service} logs: {e}") - return True - if not container_id: - return False - exit_code, logs, stderr = await sandbox.exec_with_output("sudo", "docker", "logs", container_id) - logger.info(f" stdout:\n{logs}") - if stderr: - logger.info(f" stderr:\n{stderr}") - return True - async def _get_container_id(self, sandbox: VmSandbox, compose_service: str) -> str | None: - """Get container ID for a docker-compose service (including exited containers). A lookup whose - exec transport failed (exit -1) is retried, then raised: it can't tell "no container" apart.""" - for attempt in range(_TRAJECTORY_READ_ATTEMPTS): - exit_code, stdout, stderr = await sandbox.exec_with_output( - "sudo", "docker", "compose", "-f", DOCKER_COMPOSE_PATH, "ps", "-a", "-q", compose_service - ) - if exit_code != -1: - break - logger.warning(f"Container lookup for {compose_service} lost its exec transport (attempt {attempt + 1}): {stderr[-200:]}") - else: - raise RuntimeError(f"Could not look up the {compose_service} container: exec transport failed {_TRAJECTORY_READ_ATTEMPTS} times") + """Get container ID for a docker-compose service (including exited containers).""" + exit_code, stdout, stderr = await sandbox.exec_with_output( + "sudo", "docker", "compose", "-f", DOCKER_COMPOSE_PATH, "ps", "-a", "-q", compose_service + ) container_id = stdout.strip() return container_id if container_id else None diff --git a/src/agent_env/providers/sandbox_providers/__init__.py b/src/agent_env/providers/sandbox_providers/__init__.py index b41c4b13..b0f13b2a 100644 --- a/src/agent_env/providers/sandbox_providers/__init__.py +++ b/src/agent_env/providers/sandbox_providers/__init__.py @@ -14,7 +14,6 @@ from agent_env.providers.sandbox_providers.modal_sandbox import ModalSandbox, ModalSandboxProvider from agent_env.providers.sandbox_providers.modal_vm_sandbox import ModalVmSandbox, ModalVmSandboxProvider from agent_env.providers.sandbox_providers.e2b import E2BSandbox, E2BSandboxProvider -from agent_env.providers.sandbox_providers.sail_vm import SailVmSandbox, SailVmSandboxProvider __all__ = [ "ChainedSandboxProvider", "Sandbox", "VmSandbox", @@ -23,7 +22,6 @@ "ModalSandbox", "ModalSandboxProvider", "ModalVmSandbox", "ModalVmSandboxProvider", "E2BSandbox", "E2BSandboxProvider", - "SailVmSandbox", "SailVmSandboxProvider", "build_sandbox_provider", "get_sandbox_provider", "set_sandbox_provider", "reset_sandbox_provider", "get_env_sandbox_provider", "set_env_sandbox_provider", "reset_env_sandbox_provider", diff --git a/src/agent_env/providers/sandbox_providers/sail_vm/README.md b/src/agent_env/providers/sandbox_providers/sail_vm/README.md new file mode 100644 index 00000000..749e7205 --- /dev/null +++ b/src/agent_env/providers/sandbox_providers/sail_vm/README.md @@ -0,0 +1,111 @@ +# `sail_vm` sandbox provider + +Runs agent-env sandboxes on [Sail Research Sailboxes](https://docs.sailresearch.com/sailboxes). These are +Linux VMs booted from Sail's `devbox` image, which ships Docker and Compose v2 and runs as root. Like +`modal_vm` and `e2b`, it is a VM provider: agent-env's docker-in-VM flows run on it unchanged. That covers +the gateway's docker-compose, an agent's `docker run`, image loading and artifact collection. + +## Install + +The Sail SDK is an optional extra: + +```bash +pip install 'agentenv-framework[sail]' +``` + +Without it, selecting `sail_vm` fails with a `ConfigError` that names this extra. Nothing else needs it. + +## Configure + +Put the Sail API key in your secret store, then add the provider to `.agentenv/config.toml`: + +```toml +[sandbox.providers.sail_vm.config] +api_key = "secret:sail_api_key" # required; env:SAIL_API_KEY for local development +app = "agent-env" # the Sail App every Sailbox belongs to +min_size = "s" # the smallest Sailbox size to pick: s, m or l +auto_sleep = false # let Sail sleep idle Sailboxes; off by default +# auto_sleep_min_idle_seconds = 600 # 1-3600; turns auto_sleep on +# runtime_threads = 16 # the SDK's network thread pool (1-256) +inject_model_key = true # keep an agent's model key out of its Sailbox (below) +``` + +Use it for a run with `agent-env run --sandbox sail_vm`, or by default with: + +```toml +[sandbox] +default = "sail_vm" +agent_default = "sail_vm" +``` + +A process uses one Sail API key: the SDK reads it from `SAIL_API_KEY` when it builds its process-wide +client. The provider sets that variable only for that one build and then restores it. Workloads never +see the key. + +## Resources and lifetime + +- **Size:** a Sailbox's size fixes its vCPU (`s`, `m`, `l` = 1, 4, 8). The provider picks the smallest + size, no smaller than `min_size`, that covers the requested CPU. +- **Memory and disk:** these are ceilings, not reservations, since Sail bills observed usage. Requests + are rounded up to whole GiB, into the size's range: memory 2–64, 8–128 or 16–256 GiB, and disk 8–128, + 32–512 or 64–1024 GiB. +- **Refused requests:** a request no size can meet is refused before anything is created. +- **Lifetime:** the sandbox's `timeout` is the Sailbox's hard maximum lifetime. + +## Networking + +- **Ports:** each exposed port gets a public URL, `https://sb--.sail.box`, which fills + `tunnel_urls`. It serves HTTP and WebSocket, with no platform authentication, like Modal's tunnels. +- **Egress:** allow-all, or an allowlist of hostnames, `*.domain` wildcards, IPv4 addresses and IPv4 + CIDRs, up to Sail's 128 entries. IPv6 entries are refused. The provider adds the hosts of signed + download URLs to an allowlist before it loads images or objects. +- **Reconnect:** `get_sandbox` restores the ports and the applied egress policy. A policy it can't + represent leaves `network_policy` unknown, and image loading then fails closed. + +## The agent's model key + +With `inject_model_key` on, an agent's `LITELLM_API_KEY` never enters its Sailbox: + +1. The key is stored as a Sail secret named `AGENTENV_LITELLM_`, one per distinct key. +2. The Sailbox is created with a saved egress policy. On requests to `LITELLM_BASE_URL`'s host, the policy + sets the `authorization: Bearer …` and `x-api-key` headers from that secret. Sail adds the key as each + request leaves the box. +3. Inside the box, every command and file agent-env sends carries the placeholder + `sail-injected-model-key` in place of the key. +4. A `docker` shim on the box gives every container the VM's CA bundle (`SSL_CERT_FILE`, + `REQUESTS_CA_BUNDLE`, `NODE_EXTRA_CA_CERTS`, `CURL_CA_BUNDLE`). Sail terminates TLS for the model host + with its own CA, so containers need it to trust those requests. + +Requirements and behaviour: + +- **Endpoint:** the model endpoint must be HTTPS and reachable from Sail. +- **Key length:** keys shorter than 16 characters are refused, because scrubbing replaces the key + wherever it appears. +- **Other keys are refused:** a Sailbox refuses any command or file that would carry a model key Sail + doesn't inject for it. One example is an agent deployed into a sandbox it didn't create. +- **Turning it off:** set `inject_model_key = false` to pass keys in as other providers do. +- **Cleanup:** terminate deletes the Sailbox's policy. The secret is kept, so a concurrent launch with the + same key never loses it. To remove secrets for retired keys, sweep `AGENTENV_LITELLM_*` with + `sail secret list`. + +## Attribution and cost + +Sailboxes have no labels. The provider names each one `ae--`, which you can +find with the SDK's `Sailbox.list(search=...)`. It also logs one `agent_env.sail_vm_sandbox_started` event per box, +with `sailbox_id`, the App and the full attribution. To attribute cost, join Sail's per-Sailbox spend +(`GET /sailboxes/spend`) on `sailbox_id`. + +## Things to know + +- **Automatic checkpoints:** Sail checkpoints every Sailbox's disk for host-failure recovery, and this + can't be turned off. Anything a workload's container env holds lands there, apart from the injected + model key. With the S3 object store, keep `share_credentials` off for Sail runs. +- **Docker-in-Docker:** containers an agent starts with its own Docker-in-Docker don't get the CA bundle. +- **No `SAIL_MODE`:** the provider talks to Sail's production endpoints. + +## Testing + +`tst/integration/providers/sandbox_providers/sail_vm_sandbox_smoke_test.py` runs against a real Sail +account when `[sandbox.providers.sail_vm.config]` resolves. Otherwise it skips with +`agentenv-capability-missing: remote_sandbox`. `gateway_test.py` and `task_steps_test.py` include a +`sail_vm` case. diff --git a/src/agent_env/providers/sandbox_providers/sail_vm/__init__.py b/src/agent_env/providers/sandbox_providers/sail_vm/__init__.py index 25402fc6..480242d3 100644 --- a/src/agent_env/providers/sandbox_providers/sail_vm/__init__.py +++ b/src/agent_env/providers/sandbox_providers/sail_vm/__init__.py @@ -1,6 +1 @@ -"""Sail Research Sailbox sandbox provider package.""" - -from agent_env.providers.sandbox_providers.sail_vm.provider import SailVmSandboxProvider -from agent_env.providers.sandbox_providers.sail_vm.sandbox import SailVmSandbox - -__all__ = ["SailVmSandbox", "SailVmSandboxProvider"] +"""Sail Research Sailbox VM sandbox provider (the ``sail`` extra); see README.md.""" diff --git a/src/agent_env/providers/sandbox_providers/sail_vm/_sdk.py b/src/agent_env/providers/sandbox_providers/sail_vm/_sdk.py index 95fcb2d1..e76ccc69 100644 --- a/src/agent_env/providers/sandbox_providers/sail_vm/_sdk.py +++ b/src/agent_env/providers/sandbox_providers/sail_vm/_sdk.py @@ -1,4 +1,4 @@ -"""The Sail SDK, imported on first use and authenticated with the provider's configured key. +"""The Sail SDK (the ``sail`` extra), authenticated with the provider's configured key. The Python SDK takes its key (and its thread-pool size) only from the environment, read once when it builds its process-wide client. Both are set for that one build and restored, so subprocesses never @@ -13,6 +13,8 @@ from types import ModuleType from typing import Any +import sail + from agent_env.config.errors import ConfigError API_KEY_ENV = "SAIL_API_KEY" @@ -34,7 +36,7 @@ def connect(api_key: str, app_name: str, *, runtime_threads: int | None = None, """ global _installed_key if sdk is None: - import sail as sdk + sdk = sail fingerprint = _fingerprint(api_key) with _lock: if _installed_key is None: diff --git a/src/agent_env/providers/sandbox_providers/sail_vm/model_key.py b/src/agent_env/providers/sandbox_providers/sail_vm/model_key.py index d952e3d8..e86cc282 100644 --- a/src/agent_env/providers/sandbox_providers/sail_vm/model_key.py +++ b/src/agent_env/providers/sandbox_providers/sail_vm/model_key.py @@ -13,11 +13,13 @@ import json import re import shlex +import uuid from dataclasses import dataclass, field from typing import Any, Mapping from urllib.parse import urlparse SECRET_PREFIX = "AGENTENV_LITELLM_" +POLICY_PREFIX = "agentenv-" PLACEHOLDER = "sail-injected-model-key" KEY_ENV = "LITELLM_API_KEY" BASE_URL_ENV = "LITELLM_BASE_URL" @@ -36,11 +38,17 @@ _SECRET_REF = re.compile(r"\$\{secrets\.([A-Za-z0-9_]+)\}") # A model-key env var assignment and the whole shell word assigned: quoted, escaped and bare parts alike. -_KEY_ASSIGNMENT = re.compile( - r"\b(?:LITELLM_API_KEY|ANTHROPIC_API_KEY)=((?:'[^']*'|\"(?:[^\"\\]|\\.)*\"|\\.|[^\s'\"\\])*)" -) -# A whole word that is one variable reference a shell expands: $NAME, ${NAME}, "$NAME" or "${NAME}". +_KEY_NAMES = ("LITELLM_API_KEY", "ANTHROPIC_API_KEY") +# A shell word: quoted, escaped and bare parts, concatenated. +_SHELL_WORD = re.compile(r"(?:'[^']*'|\"(?:[^\"\\]|\\.)*\"|\\.|[^\s'\"\\;|&<>()])+") +# An assigned value, as written, that is wholly one variable reference a shell expands: $NAME, ${NAME}, "$NAME", "${NAME}". _REFERENCE = re.compile(r'^"?\$(?:[A-Za-z_][A-Za-z0-9_]*|\{[A-Za-z_][A-Za-z0-9_]*\})"?$') +# In text that isn't a shell script: KEY=value, or a YAML / JSON KEY: value. +_LITERAL_ASSIGNMENT = re.compile( + r"\b(?:LITELLM_API_KEY|ANTHROPIC_API_KEY)['\"]?(?:=|\s*:\s*)['\"]?([^\s'\",}#]*)" +) +#: Shorter keys are refused: scrubbing replaces every occurrence, which for a short one hits unrelated text. +MIN_KEY_LENGTH = 16 # By secret name: the key, so a handle reconnected here can still scrub its agent's key, and the launches # and Sailboxes in this process that need it, so the key is forgotten once none does. Process memory only. @@ -54,23 +62,44 @@ def secret_name(key: str) -> str: def carries_model_key(text: str, *, shell: bool) -> bool: """Whether ``text`` assigns a model-key env var a value other than the placeholder. In a ``shell`` script - the assigned word is decoded as the shell would (quotes and escapes), and a word that is wholly one - unquoted or double-quoted variable reference is not a value; in any other text, every value is literal.""" - for word in _KEY_ASSIGNMENT.findall(text): - if shell and _REFERENCE.match(word): + each word is decoded as the shell would (quotes and escapes, in the name too), and a value that is wholly + one unquoted or double-quoted variable reference isn't one; in other text, every value is literal.""" + if not shell: + return any(value and value != PLACEHOLDER for value in _LITERAL_ASSIGNMENT.findall(text)) + for word in _SHELL_WORD.findall(text): + try: + decoded = "".join(shlex.split(word)) + except ValueError: + return True + name, assigned, value = decoded.partition("=") + if not assigned or name not in _KEY_NAMES or not value or value == PLACEHOLDER: continue - if shell: - try: - value = "".join(shlex.split(word)) - except ValueError: - return True - else: - value = word - if value and value != PLACEHOLDER: + if not _REFERENCE.match(_written_value(word)): return True return False +def _written_value(word: str) -> str: + """The part of a shell word after its first ``=`` outside single quotes and escapes, as written.""" + quote = None + index = 0 + while index < len(word): + char = word[index] + if quote == "'": + if char == "'": + quote = None + elif char == "=": + return word[index + 1:] + elif char == "\\": + index += 1 + elif char == '"': + quote = None if quote == '"' else '"' + elif char == "'": + quote = "'" + index += 1 + return "" + + def docker_shim() -> str: """A ``docker`` wrapper that gives every container it runs or creates the Sailbox's CA bundle.""" trust = " ".join(f"-e {name}={value}" for name, value in CONTAINER_TRUST_ENV.items()) @@ -95,6 +124,11 @@ class ModelKeyInjection: secret: str key: str | None = field(default=None, repr=False) policy_id: str | None = None + #: Saved policies a swap may or may not have applied (Sail couldn't say), deleted once it can. + unsettled_policy_ids: list[str] = field(default_factory=list) + #: Every saved policy made for one Sailbox is named under this prefix, so teardown from any handle, in + #: any process, finds them all; a reconnected handle takes it from the applied policy's name. + policy_prefix: str = field(default_factory=lambda: f"{POLICY_PREFIX}{uuid.uuid4().hex[:12]}") @classmethod def for_env(cls, env: Mapping[str, str]) -> ModelKeyInjection | None: @@ -109,6 +143,11 @@ def for_env(cls, env: Mapping[str, str]) -> ModelKeyInjection | None: f"Sail injects the model key only into HTTPS requests, but {BASE_URL_ENV} is {base_url!r}; " "use an https endpoint or set inject_model_key = false in [sandbox.providers.sail_vm.config]" ) + if len(key) < MIN_KEY_LENGTH: + raise ValueError( + f"Sail model-key injection needs a {KEY_ENV} of at least {MIN_KEY_LENGTH} characters, to scrub it " + "safely; set inject_model_key = false in [sandbox.providers.sail_vm.config] to pass it in" + ) return cls(host=parsed.hostname, secret=secret_name(key), key=key) @classmethod diff --git a/src/agent_env/providers/sandbox_providers/sail_vm/provider.py b/src/agent_env/providers/sandbox_providers/sail_vm/provider.py index 09fe23c7..27116548 100644 --- a/src/agent_env/providers/sandbox_providers/sail_vm/provider.py +++ b/src/agent_env/providers/sandbox_providers/sail_vm/provider.py @@ -84,6 +84,23 @@ async def _reap(sandbox: SailVmSandbox) -> None: ) +async def _terminate_named(sdk: Any, app: Any, name: str) -> bool: + """Terminate every live Sailbox in ``app`` called ``name`` (one a create made though it reported failing), + retrying through a brief outage; False when that couldn't be confirmed.""" + for attempt in range(_REAP_ATTEMPTS): + try: + for box in await sdk.Sailbox.list.aio(app_id=app, search=name): + if box.name == name and box.status not in ("terminated", "terminating"): + await box.terminate.aio() + logger.info("Terminated Sailbox %s, created though its create reported a failure", box.sailbox_id) + return True + except Exception as exc: # noqa: BLE001 - retried, then reported + logger.warning("Checking for a Sailbox %s left by a failed create failed (attempt %s): %s", name, attempt + 1, exc) + await asyncio.sleep(2 ** attempt) + logger.error("A Sailbox %s may be left by a failed create; it stops at its max lifetime", name) + return False + + async def _create_or_reclaim( create: Any, wrap: Callable[[Any], SailVmSandbox], release: Callable[[], Awaitable[None]] | None = None, ) -> Any: @@ -254,19 +271,31 @@ def wrap(raw: Any) -> SailVmSandbox: injection.release(launch) return sandbox - release = (lambda: release_injection(sdk, injection, launch)) if injection is not None else None + name = sailbox_name(resolved_attribution) + + async def release() -> None: + """Undo a create that failed or was abandoned: terminate any Sailbox Sail made under ``name`` (its + response may have been lost), then release the injection's policy.""" + terminated = await _terminate_named(sdk, app, name) + if injection is not None: + if terminated: + await release_injection(sdk, injection, launch) + else: + injection.release(launch) + logger.error("Egress policy %s stays with the unconfirmed Sailbox %s; delete it by name", injection.policy_id, name) + creating = False try: if injection is not None: injection.hold(launch) await sdk.Secret.set.aio(injection.secret, injection.key) - egress = await create_saved_policy(sdk, policy_name(), egress_document(effective_policy, injection)) + egress = await create_saved_policy(sdk, policy_name(injection), egress_document(effective_policy, injection)) injection.policy_id = egress.id creating = True raw = await _create_or_reclaim(sdk.Sailbox.create.aio( app=app, image=sdk.Image.devbox("amd64"), - name=sailbox_name(resolved_attribution), + name=name, size=size, memory_limit_gib=memory_gib, disk_limit_gib=disk_gib, @@ -277,7 +306,7 @@ def wrap(raw: Any) -> SailVmSandbox: ), wrap, release) except BaseException as exc: # A create cancelled in flight is _create_or_reclaim's to clean up once it settles. - if release is not None and not (creating and isinstance(exc, asyncio.CancelledError)): + if not (creating and isinstance(exc, asyncio.CancelledError)): await release() raise sandbox = wrap(raw) diff --git a/src/agent_env/providers/sandbox_providers/sail_vm/sandbox.py b/src/agent_env/providers/sandbox_providers/sail_vm/sandbox.py index a27bac93..d709981e 100644 --- a/src/agent_env/providers/sandbox_providers/sail_vm/sandbox.py +++ b/src/agent_env/providers/sandbox_providers/sail_vm/sandbox.py @@ -16,6 +16,7 @@ from agent_env.config import get_config from agent_env.providers.sandbox_providers.sail_vm.model_key import ( DOCKER_SHIM_PATH, + POLICY_PREFIX, ModelKeyInjection, carries_model_key, docker_shim, @@ -81,8 +82,8 @@ class ModelKeyRefusedError(RuntimeError): """A command or file would have carried a model key Sail doesn't inject for this Sailbox into it.""" -def policy_name() -> str: - return f"agentenv-{uuid.uuid4().hex}" +def policy_name(injection: ModelKeyInjection) -> str: + return f"{injection.policy_prefix}-{uuid.uuid4().hex[:12]}" async def create_saved_policy(sdk: Any, name: str, document: dict[str, Any]) -> Any: @@ -95,13 +96,14 @@ async def create_saved_policy(sdk: Any, name: str, document: dict[str, Any]) -> raise -async def delete_policy_named(sdk: Any, name: str) -> None: - """Delete the saved policy called ``name``, if Sail made one, retrying through a brief outage; one Sail - stays unreachable for is logged by name so it can be swept (``sail egress-policy list``).""" +async def delete_policy_named(sdk: Any, name: str, *, family: bool = False) -> None: + """Delete the saved policy called ``name`` (with ``family``, every one named ``name-…``), if Sail made + any, retrying through a brief outage; one Sail stays unreachable for is logged by name so it can be + swept (``sail egress-policy list``).""" for attempt in range(_CLEANUP_ATTEMPTS): try: for summary in await sdk.EgressPolicy.list.aio(search=name): - if summary.name == name: + if summary.name == name or (family and summary.name.startswith(f"{name}-")): await _delete_policy_by_id(sdk, summary.id) return except Exception as exc: # noqa: BLE001 - retried, then reported; never masks the caller's outcome @@ -258,6 +260,9 @@ async def terminate(self) -> None: except Exception as exc: # noqa: BLE001 - fall back to the policy this handle last applied logger.info("Could not read Sailbox %s's applied policy before terminate: %s", self.sandbox_id, exc) await self._terminate_sailbox() + await delete_policy_named(self._sdk, self._injection.policy_prefix, family=True) + for policy_id in self._injection.unsettled_policy_ids: + await delete_saved_policy(self._sdk, policy_id) await release_injection(self._sdk, self._injection, self.sandbox_id) async def _terminate_sailbox(self) -> None: @@ -375,16 +380,42 @@ async def _apply_policy(self, policy: NetworkPolicy) -> None: if self._injection is None: await self._sailbox.set_egress_policy.aio(egress_document(policy)) else: - saved = await create_saved_policy(self._sdk, policy_name(), egress_document(policy, self._injection)) + saved = await create_saved_policy(self._sdk, policy_name(self._injection), egress_document(policy, self._injection)) + previous = self._injection.policy_id try: await self._sailbox.set_egress_policy.aio(saved) - except BaseException: - await delete_saved_policy(self._sdk, saved.id) - raise - previous, self._injection.policy_id = self._injection.policy_id, saved.id - await delete_saved_policy(self._sdk, previous) + except BaseException as exc: + known, applied = await self._applied_policy_id() + if not known: + self._injection.unsettled_policy_ids.append(saved.id) + raise + if applied != saved.id: + await delete_saved_policy(self._sdk, saved.id) + raise + logger.info("Sailbox %s took policy %s though setting it reported a failure", self.sandbox_id, saved.id) + await self._settle(saved.id, previous) + if not isinstance(exc, Exception): + raise + else: + await self._settle(saved.id, previous) self.network_policy = policy + async def _applied_policy_id(self) -> tuple[bool, str | None]: + """Whether Sail could say which saved policy is applied, and its id.""" + try: + return True, getattr((await self._sdk.Sailbox.get.aio(self.sandbox_id)).egress_policy, "policy_id", None) + except Exception as exc: # noqa: BLE001 - an unknown answer is kept unknown, never guessed + logger.info("Could not read Sailbox %s's applied policy: %s", self.sandbox_id, exc) + return False, None + + async def _settle(self, applied: str, *superseded: str | None) -> None: + """Track ``applied`` as this Sailbox's policy and delete the others it replaced, settled ones included.""" + stale = [*superseded, *self._injection.unsettled_policy_ids] + self._injection.policy_id, self._injection.unsettled_policy_ids = applied, [] + for policy_id in dict.fromkeys(stale): + if policy_id != applied: + await delete_saved_policy(self._sdk, policy_id) + def adopt_applied_policy(self, applied: Any) -> NetworkPolicy | None: """The agent-env policy Sail reports applied (``Sailbox.egress_policy``), or None when it can't be represented: a saved policy counts only when it carries this Sailbox's model-key injection.""" @@ -396,6 +427,9 @@ def adopt_applied_policy(self, applied: Any) -> NetworkPolicy | None: if self._injection is None or not self._injection.matches(saved): return None self._injection.policy_id = applied.policy_id + name = getattr(applied, "name", None) + if isinstance(name, str) and name.startswith(POLICY_PREFIX) and "-" in name[len(POLICY_PREFIX):]: + self._injection.policy_prefix = name.rsplit("-", 1)[0] return policy_from_document(document, self._injection) def _known_policy(self, purpose: str) -> NetworkPolicy: diff --git a/src/agent_env/providers/sandbox_providers/sandbox_provider.py b/src/agent_env/providers/sandbox_providers/sandbox_provider.py index a3732c17..68009580 100644 --- a/src/agent_env/providers/sandbox_providers/sandbox_provider.py +++ b/src/agent_env/providers/sandbox_providers/sandbox_provider.py @@ -222,7 +222,7 @@ def effective_network_policy(cls, policy: Optional[NetworkPolicy]) -> NetworkPol "modal": "agent_env.providers.sandbox_providers.modal_sandbox:ModalSandboxProvider", "modal_vm": "agent_env.providers.sandbox_providers.modal_vm_sandbox:ModalVmSandboxProvider", "e2b": "agent_env.providers.sandbox_providers.e2b:E2BSandboxProvider", - "sail_vm": "agent_env.providers.sandbox_providers.sail_vm:SailVmSandboxProvider", + "sail_vm": "agent_env.providers.sandbox_providers.sail_vm.provider:SailVmSandboxProvider", "local": "agent_env.providers.sandbox_providers.local_sandbox:LocalSandboxProvider", } diff --git a/tst/unit/providers/env_providers/env_gateway_provider_test.py b/tst/unit/providers/env_providers/env_gateway_provider_test.py index 380eaaa6..233cbc7e 100644 --- a/tst/unit/providers/env_providers/env_gateway_provider_test.py +++ b/tst/unit/providers/env_providers/env_gateway_provider_test.py @@ -549,61 +549,10 @@ async def test_modal_vm_provider_routes_to_vm_path_not_containers(): @pytest.mark.asyncio -async def test_read_trajectory_retries_a_read_whose_transport_failed(): - gp = EnvironmentGatewayProvider() - gp._get_container_id = AsyncMock(return_value="gw") - sandbox = MagicMock() - sandbox.exec_with_output = AsyncMock(side_effect=[(-1, "", "host lost"), (0, '{"a": 1}\n{"b": 2}\n', "")]) - - assert await gp.read_trajectory(sandbox) == [{"a": 1}, {"b": 2}] - assert sandbox.exec_with_output.await_count == 2 - - -@pytest.mark.asyncio -async def test_read_trajectory_raises_rather_than_parse_a_read_that_keeps_failing(): - gp = EnvironmentGatewayProvider() - gp._get_container_id = AsyncMock(return_value="gw") - sandbox = MagicMock() - sandbox.exec_with_output = AsyncMock(return_value=(-1, '{"a": 1}\n', "host lost")) - - with pytest.raises(RuntimeError, match="exec transport failed 3 times"): - await gp.read_trajectory(sandbox) - - -@pytest.mark.asyncio -async def test_a_container_lookup_whose_transport_keeps_failing_raises_rather_than_read_as_no_container(): - gp = EnvironmentGatewayProvider() - sandbox = MagicMock() - sandbox.exec_with_output = AsyncMock(return_value=(-1, "", "host lost")) - - with pytest.raises(RuntimeError, match="Could not look up the .* container: exec transport failed 3 times"): - await gp.read_trajectory(sandbox) - - -@pytest.mark.asyncio -async def test_a_container_lookup_retries_a_failed_transport(): - gp = EnvironmentGatewayProvider() - sandbox = MagicMock() - sandbox.exec_with_output = AsyncMock(side_effect=[(-1, "", "host lost"), (0, "gw\n", ""), (0, '{"a": 1}\n', "")]) - - assert await gp.read_trajectory(sandbox) == [{"a": 1}] - - -@pytest.mark.asyncio -async def test_read_trajectory_of_a_gateway_with_no_history_yet_is_empty(): - gp = EnvironmentGatewayProvider() - gp._get_container_id = AsyncMock(return_value="gw") - sandbox = MagicMock() - sandbox.exec_with_output = AsyncMock(return_value=(1, "", "cat: /var/log/agentenv/trajectory.jsonl: No such file")) - - assert await gp.read_trajectory(sandbox) == [] - - -@pytest.mark.asyncio -async def test_sail_provider_routes_to_vm_path_not_containers(): +async def test_sail_vm_provider_routes_to_vm_path_not_containers(): """A Sailbox is a Docker-capable VM: the gateway deploys onto it with docker-compose.""" from agent_env.env.gateway import GatewayMode - from agent_env.providers.sandbox_providers.sail_vm import SailVmSandboxProvider + from agent_env.providers.sandbox_providers.sail_vm.provider import SailVmSandboxProvider gp = EnvironmentGatewayProvider() gp._deploy_via_vm = AsyncMock(return_value="VM_RESULT") @@ -1694,22 +1643,3 @@ def test_every_mcp_server_gets_the_same_healthcheck_budget(): assert " timeout: 5s" in block assert " retries: 30" in block assert " start_period: 60s" in block - - -@pytest.mark.asyncio -async def test_a_log_lookup_that_keeps_losing_its_transport_never_stops_a_deploy(caplog): - gp = EnvironmentGatewayProvider() - sandbox = MagicMock() - sandbox.exec_with_output = AsyncMock(return_value=(-1, "", "host lost")) - - assert await gp._log_service(sandbox, "gateway") is True - assert "Skipping gateway logs" in caplog.text - - -@pytest.mark.asyncio -async def test_a_service_without_a_container_is_reported_as_missing(): - gp = EnvironmentGatewayProvider() - sandbox = MagicMock() - sandbox.exec_with_output = AsyncMock(return_value=(0, "", "")) - - assert await gp._log_service(sandbox, "pgweb") is False diff --git a/tst/unit/providers/sandbox_providers/sail_vm_model_key_test.py b/tst/unit/providers/sandbox_providers/sail_vm_model_key_test.py index 5cd1e92c..db731bae 100644 --- a/tst/unit/providers/sandbox_providers/sail_vm_model_key_test.py +++ b/tst/unit/providers/sandbox_providers/sail_vm_model_key_test.py @@ -57,6 +57,7 @@ async def create_policy(name, document): Sailbox=SimpleNamespace( create=SimpleNamespace(aio=AsyncMock(return_value=sailbox)), get=SimpleNamespace(aio=AsyncMock(return_value=sailbox)), + list=SimpleNamespace(aio=AsyncMock(return_value=[])), ), Image=SimpleNamespace(devbox=MagicMock(return_value="devbox-amd64")), AutoSleep=SimpleNamespace(never=lambda: "never", default=lambda: "default", not_before=lambda s: s), @@ -64,7 +65,7 @@ async def create_policy(name, document): EgressPolicy=SimpleNamespace( create=SimpleNamespace(aio=AsyncMock(side_effect=create_policy)), get=SimpleNamespace(aio=AsyncMock(side_effect=lambda policy_id: next(p for p in saved if p.id == policy_id))), - list=SimpleNamespace(aio=AsyncMock(side_effect=lambda search: [p for p in saved if p.name == search])), + list=SimpleNamespace(aio=AsyncMock(side_effect=lambda search: [p for p in saved if search in p.name])), ), reset_transports=MagicMock(), NotFoundError=_NotFound, @@ -331,7 +332,7 @@ async def test_terminate_deletes_the_policy_and_keeps_the_secret(): await sandbox.terminate() sailbox.terminate.aio.assert_awaited_once() - saved[0].delete.aio.assert_awaited_once() + saved[0].delete.aio.assert_awaited() secret.delete.aio.assert_not_awaited() assert secret_name(_KEY) not in model_key._injected_keys @@ -700,3 +701,189 @@ async def test_a_key_with_an_apostrophe_is_scrubbed_in_its_shell_encoded_form(qu script = sailbox.exec.aio.await_args.args[0][2] assert "sk-a" not in script and PLACEHOLDER in script assert shlex.split(script.split("LITELLM_API_KEY=", 1)[1])[0] == PLACEHOLDER + + +@pytest.mark.parametrize( + ("text", "shell", "carries"), + [ + ("docker run -e 'LITELLM_API_KEY'=other-secret img", True, True), + ('docker run -e "LITELLM_API_KEY=other-secret" img', True, True), + ('docker run -e "LITELLM_API_KEY=$LITELLM_API_KEY" img', True, False), + ("LITELLM_API_KEY: other-secret", False, True), + ('{"LITELLM_API_KEY": "sk-x"}', False, True), + (f'{{"LITELLM_API_KEY": "{PLACEHOLDER}"}}', False, False), + ("os.environ['LITELLM_API_KEY']", False, False), + ], +) +def test_a_quoted_name_and_yaml_or_json_forms_are_detected(text, shell, carries): + assert carries_model_key(text, shell=shell) is carries + + +def test_a_key_too_short_to_scrub_safely_is_refused(): + with pytest.raises(ValueError, match="at least 16 characters"): + ModelKeyInjection.for_env({**_ENV, "LITELLM_API_KEY": "docker"}) + + +@pytest.mark.asyncio +async def test_a_swap_whose_response_was_lost_but_applied_keeps_the_replacement_and_deletes_the_old(): + sailbox = _sailbox() + sdk, saved, _ = _fake_sdk(sailbox) + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( + image_name="agent:1", port=8000, env=_ENV, + network_policy=NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",)), + ) + + async def applied_then_lost(policy): + sailbox.egress_policy = SimpleNamespace(policy_id=policy.id, document=policy.document) + raise TimeoutError("response lost") + + sailbox.set_egress_policy.aio = AsyncMock(side_effect=applied_then_lost) + sdk.Sailbox.get.aio = AsyncMock(side_effect=lambda _id: sailbox) + + await sandbox.apply_network_policy(sandbox.network_policy.with_hosts(["bucket.example"])) + + first, second = saved + first.delete.aio.assert_awaited_once() + second.delete.aio.assert_not_awaited() + assert sandbox._injection.policy_id == second.id + + +@pytest.mark.parametrize( + ("text", "carries"), + [ + ("docker run -e LITELLM_API_K\\EY=other-secret img", True), + ("docker run -e LITELLM_API_KEY=other-secret=$HOME img", True), + ('docker run -e "LITELLM_API_KEY=$X-x" img', True), + ("docker run -e 'LITELLM_API_KEY=$X' img", True), + ('docker run -e "LITELLM_API_KEY=$LITELLM_API_KEY" img', False), + ], +) +def test_names_are_decoded_and_only_a_whole_reference_passes(text, carries): + assert carries_model_key(text, shell=True) is carries + + +async def _restricted_injected_sandbox(): + sailbox = _sailbox() + sdk, saved, _ = _fake_sdk(sailbox) + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( + image_name="agent:1", port=8000, env=_ENV, + network_policy=NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",)), + ) + return sandbox, sailbox, sdk, saved + + +@pytest.mark.asyncio +async def test_a_swap_whose_outcome_sail_cannot_report_keeps_both_policies_until_terminate(): + sandbox, sailbox, sdk, saved = await _restricted_injected_sandbox() + sailbox.set_egress_policy.aio = AsyncMock(side_effect=TimeoutError("response lost")) + sdk.Sailbox.get.aio = AsyncMock(side_effect=RuntimeError("api down")) + + with pytest.raises(TimeoutError): + await sandbox.apply_network_policy(sandbox.network_policy.with_hosts(["bucket.example"])) + + first, second = saved + first.delete.aio.assert_not_awaited() + second.delete.aio.assert_not_awaited() + assert sandbox._injection.unsettled_policy_ids == [second.id] + await sandbox.terminate() + first.delete.aio.assert_awaited() + second.delete.aio.assert_awaited() + + +@pytest.mark.asyncio +async def test_a_cancelled_swap_that_applied_still_settles_and_re_raises_the_cancellation(): + sandbox, sailbox, sdk, saved = await _restricted_injected_sandbox() + + async def applied_then_cancelled(policy): + sailbox.egress_policy = SimpleNamespace(policy_id=policy.id, document=policy.document) + raise asyncio.CancelledError + + sailbox.set_egress_policy.aio = AsyncMock(side_effect=applied_then_cancelled) + sdk.Sailbox.get.aio = AsyncMock(side_effect=lambda _id: sailbox) + + with pytest.raises(asyncio.CancelledError): + await sandbox.apply_network_policy(sandbox.network_policy.with_hosts(["bucket.example"])) + + first, second = saved + first.delete.aio.assert_awaited_once() + assert sandbox._injection.policy_id == second.id + + +@pytest.mark.asyncio +async def test_teardown_through_a_fresh_handle_deletes_every_policy_made_for_the_box(): + sandbox, sailbox, sdk, saved = await _restricted_injected_sandbox() + sailbox.set_egress_policy.aio = AsyncMock(side_effect=TimeoutError("response lost")) + sdk.Sailbox.get.aio = AsyncMock(side_effect=RuntimeError("api down")) + with pytest.raises(TimeoutError): + await sandbox.apply_network_policy(sandbox.network_policy.with_hosts(["bucket.example"])) + first, second = saved + assert first.name.rsplit("-", 1)[0] == second.name.rsplit("-", 1)[0] == sandbox._injection.policy_prefix + + sailbox.egress_policy = SimpleNamespace(policy_id=first.id, name=first.name, document=first.document) + sdk.Sailbox.get.aio = AsyncMock(side_effect=lambda _id: sailbox) + fresh = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).get_sandbox("sb_1") + await fresh.terminate() + + first.delete.aio.assert_awaited() + second.delete.aio.assert_awaited() + + +@pytest.mark.asyncio +async def test_a_create_whose_response_was_lost_terminates_the_box_sail_made_then_deletes_its_policy(): + sailbox = _sailbox() + sdk, saved, _ = _fake_sdk(sailbox) + made = MagicMock(sailbox_id="sb_lost", status="running") + made.terminate.aio = AsyncMock() + gone = MagicMock(sailbox_id="sb_old", status="terminated") + gone.terminate.aio = AsyncMock() + + async def create_then_lose(**kwargs): + made.name = gone.name = kwargs["name"] + raise TimeoutError("response lost") + + sdk.Sailbox.create.aio = AsyncMock(side_effect=create_then_lose) + sdk.Sailbox.list.aio = AsyncMock(side_effect=lambda **kw: [made, gone] if kw["search"] == made.name else []) + + with pytest.raises(TimeoutError): + await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + + made.terminate.aio.assert_awaited_once() + gone.terminate.aio.assert_not_awaited() + saved[0].delete.aio.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_a_lost_create_is_reclaimed_through_a_brief_outage_before_its_policy_goes(monkeypatch): + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail_vm.provider.asyncio.sleep", AsyncMock()) + sdk, saved, _ = _fake_sdk(_sailbox()) + made = MagicMock(sailbox_id="sb_lost", status="running") + made.terminate.aio = AsyncMock(side_effect=[RuntimeError("api down"), None]) + + async def create_then_lose(**kwargs): + made.name = kwargs["name"] + raise TimeoutError("response lost") + + sdk.Sailbox.create.aio = AsyncMock(side_effect=create_then_lose) + sdk.Sailbox.list.aio = AsyncMock(side_effect=lambda **kw: [made]) + + with pytest.raises(TimeoutError): + await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + + assert made.terminate.aio.await_count == 2 + saved[0].delete.aio.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_an_unconfirmed_lost_create_keeps_its_policy_rather_than_fail_deleting_it(monkeypatch, caplog): + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail_vm.provider.asyncio.sleep", AsyncMock()) + sdk, saved, _ = _fake_sdk(_sailbox()) + sdk.Sailbox.create.aio = AsyncMock(side_effect=TimeoutError("response lost")) + sdk.Sailbox.list.aio = AsyncMock(side_effect=RuntimeError("api down")) + + with pytest.raises(TimeoutError): + await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + + assert sdk.Sailbox.list.aio.await_count == 3 + saved[0].delete.aio.assert_not_awaited() + assert "stays with the unconfirmed Sailbox" in caplog.text + assert secret_name(_KEY) not in model_key._injected_keys diff --git a/tst/unit/providers/sandbox_providers/sail_vm_sandbox_provider_test.py b/tst/unit/providers/sandbox_providers/sail_vm_sandbox_provider_test.py index 3a1bb9ba..bce5ff88 100644 --- a/tst/unit/providers/sandbox_providers/sail_vm_sandbox_provider_test.py +++ b/tst/unit/providers/sandbox_providers/sail_vm_sandbox_provider_test.py @@ -3,6 +3,7 @@ import asyncio import logging import os +import subprocess import sys from types import SimpleNamespace from unittest.mock import AsyncMock, MagicMock @@ -45,6 +46,7 @@ def _fake_sdk(sailbox): Sailbox=SimpleNamespace( create=SimpleNamespace(aio=AsyncMock(return_value=sailbox)), get=SimpleNamespace(aio=AsyncMock(return_value=sailbox)), + list=SimpleNamespace(aio=AsyncMock(return_value=[])), ), Image=SimpleNamespace(devbox=MagicMock(return_value="devbox-amd64")), AutoSleep=SimpleNamespace( @@ -96,10 +98,17 @@ def test_from_config_rejects_invalid_config(config, message): SailVmSandboxProvider.from_config(**config) -def test_construction_neither_imports_the_sdk_nor_sets_the_key(monkeypatch): - monkeypatch.delitem(sys.modules, "sail", raising=False) +def test_importing_core_or_the_provider_package_never_loads_the_sail_sdk(): + script = ( + "import sys, agent_env, agent_env.providers, agent_env.providers.sandbox_providers.sail_vm; " + "print('sail' in sys.modules)" + ) + out = subprocess.run([sys.executable, "-c", script], capture_output=True, text=True, check=True).stdout.strip() + assert out == "False" + + +def test_construction_never_sets_the_key(): SailVmSandboxProvider.from_config(api_key="sail-secret") - assert "sail" not in sys.modules assert _sdk.API_KEY_ENV not in os.environ diff --git a/tst/unit/test_optional_extras_stay_optional.py b/tst/unit/test_optional_extras_stay_optional.py index 0bc91f80..b40e59ba 100644 --- a/tst/unit/test_optional_extras_stay_optional.py +++ b/tst/unit/test_optional_extras_stay_optional.py @@ -24,9 +24,12 @@ def find_spec(self, fullname, path=None, target=None): sys.meta_path.insert(0, Refuse()) """ -# Not core: the explorer is the explorer extra's own package, the gateway runs only in its -# container image, and the code runner is a script that reads its arguments on import. -_NOT_CORE = ("agent_env.explorer", "agent_env.env.gateway", "agent_env.task_step.task_steps.run_code_runner") +# Not core: the explorer and the sail_vm provider are their extras' own packages, the gateway runs +# only in its container image, and the code runner is a script that reads its arguments on import. +_NOT_CORE = ( + "agent_env.explorer", "agent_env.providers.sandbox_providers.sail_vm", "agent_env.env.gateway", + "agent_env.task_step.task_steps.run_code_runner", +) _IMPORT_CORE = _BLOCKING + """ import ast, pkgutil @@ -123,3 +126,9 @@ def test_core_and_every_store_module_import_without_the_extra(extra): def test_a_gcp_impl_without_the_extra_names_the_extra_to_install(impl, base): message = _run(_LOAD_IMPL, _only_in("gcp"), impl, base) assert "pip install 'agentenv-framework[gcp]'" in message, message + + +def test_the_sail_vm_provider_without_the_extra_names_the_extra_to_install(): + impl = "agent_env.providers.sandbox_providers.sail_vm.provider:SailVmSandboxProvider" + message = _run(_LOAD_IMPL, _only_in("sail"), impl, "agent_env.providers.sandbox_providers.sandbox_provider:SandboxProvider") + assert "pip install 'agentenv-framework[sail]'" in message, message diff --git a/uv.lock b/uv.lock index e979aaf0..babbcf6f 100644 --- a/uv.lock +++ b/uv.lock @@ -62,7 +62,6 @@ dependencies = [ { name = "pymongo" }, { name = "python-multipart" }, { name = "pyyaml" }, - { name = "sail" }, { name = "starlette" }, { name = "uvicorn" }, ] @@ -84,6 +83,7 @@ dev = [ { name = "pytest-timeout" }, { name = "pytest-xdist" }, { name = "requests" }, + { name = "sail" }, { name = "uvicorn" }, ] explorer = [ @@ -97,11 +97,14 @@ gcp = [ { name = "google-cloud-storage" }, { name = "requests" }, ] +sail = [ + { name = "sail" }, +] [package.metadata] requires-dist = [ { name = "a2a-sdk", specifier = "==0.3.26" }, - { name = "agentenv-framework", extras = ["explorer", "gcp"], marker = "extra == 'dev'" }, + { name = "agentenv-framework", extras = ["explorer", "gcp", "sail"], marker = "extra == 'dev'" }, { name = "agentenv-framework-protocol", editable = "packages/agentenv-protocol" }, { name = "boto3", specifier = ">=1.35.0" }, { name = "certifi", specifier = ">=2024.2.2" }, @@ -132,12 +135,12 @@ requires-dist = [ { name = "python-multipart", specifier = ">=0.0.13" }, { name = "pyyaml", specifier = ">=6.0" }, { name = "requests", marker = "extra == 'gcp'", specifier = ">=2.31" }, - { name = "sail", specifier = "~=0.12.8" }, + { name = "sail", marker = "extra == 'sail'", specifier = "~=0.12.8" }, { name = "starlette", specifier = ">=0.40" }, { name = "uvicorn", specifier = ">=0.47" }, { name = "uvicorn", marker = "extra == 'explorer'", specifier = ">=0.29" }, ] -provides-extras = ["explorer", "gcp", "dev"] +provides-extras = ["explorer", "gcp", "sail", "dev"] [[package]] name = "agentenv-framework-protocol" From b6c60fa7d96ed4bdf1f98e196fc17bf6d3072ecb Mon Sep 17 00:00:00 2001 From: Pratyush Singhal Date: Wed, 7 Oct 2026 11:48:51 -0700 Subject: [PATCH 11/11] docs(sail_vm): walk through setup with complete config examples Set up in four steps (install the extra, get a key, a complete config.toml, a first run with its output), a reference table of every config key, and examples for a shared deployment, per-run model keys, a sandbox with restricted egress, auto-sleep, a non-HTTPS model endpoint, a fallback chain and the Python API. Co-Authored-By: Claude Opus 5.5 --- .../sandbox_providers/sail_vm/README.md | 159 ++++++++++++++++-- 1 file changed, 143 insertions(+), 16 deletions(-) diff --git a/src/agent_env/providers/sandbox_providers/sail_vm/README.md b/src/agent_env/providers/sandbox_providers/sail_vm/README.md index 749e7205..fc007ce9 100644 --- a/src/agent_env/providers/sandbox_providers/sail_vm/README.md +++ b/src/agent_env/providers/sandbox_providers/sail_vm/README.md @@ -5,43 +5,170 @@ Linux VMs booted from Sail's `devbox` image, which ships Docker and Compose v2 a `modal_vm` and `e2b`, it is a VM provider: agent-env's docker-in-VM flows run on it unchanged. That covers the gateway's docker-compose, an agent's `docker run`, image loading and artifact collection. -## Install +## Set up + +### 1. Install the extra The Sail SDK is an optional extra: ```bash -pip install 'agentenv-framework[sail]' +pip install 'agentenv-framework[sail]' # or: uv add 'agentenv-framework[sail]' ``` Without it, selecting `sail_vm` fails with a `ConfigError` that names this extra. Nothing else needs it. -## Configure +### 2. Get a Sail API key -Put the Sail API key in your secret store, then add the provider to `.agentenv/config.toml`: +Create a key in the [Sail dashboard](https://app.sailresearch.com). For local work, export it: -```toml -[sandbox.providers.sail_vm.config] -api_key = "secret:sail_api_key" # required; env:SAIL_API_KEY for local development -app = "agent-env" # the Sail App every Sailbox belongs to -min_size = "s" # the smallest Sailbox size to pick: s, m or l -auto_sleep = false # let Sail sleep idle Sailboxes; off by default -# auto_sleep_min_idle_seconds = 600 # 1-3600; turns auto_sleep on -# runtime_threads = 16 # the SDK's network thread pool (1-256) -inject_model_key = true # keep an agent's model key out of its Sailbox (below) +```bash +export SAIL_API_KEY=sk_... ``` -Use it for a run with `agent-env run --sandbox sail_vm`, or by default with: +For a shared deployment, put it in your secret store instead, for example as `sail_api_key`, and reference +it with `secret:sail_api_key`. Never put the key itself in `config.toml`. + +### 3. Configure + +A complete `.agentenv/config.toml` that runs every environment and agent on Sailboxes, with the local stores: ```toml [sandbox] -default = "sail_vm" -agent_default = "sail_vm" +default = "sail_vm" # environments and sandboxes +agent_default = "sail_vm" # agents + +[sandbox.providers.sail_vm.config] +api_key = "env:SAIL_API_KEY" # or "secret:sail_api_key" + +# The model endpoint agents call. Sail injects its key into their requests (see below), so it must be HTTPS. +[model] +base_url = "https://litellm.example.com" +api_key = "env:LITELLM_API_KEY" ``` +To keep the local default and use Sail per run instead, add only the `[sandbox.providers.sail_vm.config]` +table and pass `--sandbox sail_vm`. `agent-env config show` prints the file in effect and masks the key. + +### 4. Run something + +The bundled `hello` task deploys a Sailbox, loads a file into it and checks it, with no model needed: + +```console +$ agent-env run hello --sandbox sail_vm +[tasks/hello.json] step 1/3 box (deploy_sandbox) +[tasks/hello.json] step 1/3 box done in 2.4s +[tasks/hello.json] step 2/3 load (load_artifact) +[tasks/hello.json] step 3/3 hello (verify_sandbox) +[tasks/hello.json] passed in 3.1s + +Tasks: + tasks/hello.json v1: passed (hello: 1), 3.1s + +Tore down 1 sandbox. +``` + +`agent-env -v run …` also logs each Sailbox as it starts +(`Sail VM sandbox started: sailbox_id=sb_… app=agent-env size=s …`). + +## Configuration reference + +All keys go under `[sandbox.providers.sail_vm.config]`. Unknown keys are refused. + +| Key | Default | Meaning | +|---|---|---| +| `api_key` | required | The Sail API key, as an `env:` or `secret:` reference. | +| `app` | `"agent-env"` | The Sail App every Sailbox belongs to; Sail groups and bills by App. | +| `min_size` | `"s"` | The smallest Sailbox size to pick: `s`, `m` or `l`. | +| `auto_sleep` | `false` | Let Sail sleep an idle Sailbox; the first request after waking waits a few seconds. | +| `auto_sleep_min_idle_seconds` | unset | 1–3600 seconds of idleness before Sail may sleep a Sailbox; turns `auto_sleep` on. | +| `runtime_threads` | the SDK's own | The size of the SDK's network thread pool (1–256). | +| `inject_model_key` | `true` | Keep an agent's model key out of its Sailbox (see "The agent's model key"). | + A process uses one Sail API key: the SDK reads it from `SAIL_API_KEY` when it builds its process-wide client. The provider sets that variable only for that one build and then restores it. Workloads never see the key. +## Examples + +**A shared deployment.** The key comes from the secret store, and costs are grouped under their own App: + +```toml +[sandbox] +default = "sail_vm" +agent_default = "sail_vm" +attribution = { team = "env-pod", project_id = "env:PROJECT_ID?unassigned" } + +[sandbox.providers.sail_vm.config] +api_key = "secret:sail_api_key" +app = "agent-env-prod" +min_size = "m" + +[model] +base_url = "https://litellm.example.com" +api_key = "secret:litellm_api_key" +``` + +**A per-run model key.** A run's override key is injected the same way as the configured one, so short-lived +per-run keys never reach a Sailbox either: + +```bash +agent-env task run --id my-task --agent-sandbox sail_vm --env-sandbox sail_vm \ + --litellm-api-key "$RUN_SCOPED_KEY" --judge-litellm-api-key "$JUDGE_SCOPED_KEY" +``` + +**A sandbox with restricted egress.** In a task, `deploy_sandbox` and `deploy_agent` take a +`network_policy`; Sail enforces it for the VM and its containers: + +```json +{"id": "box", "type": "deploy_sandbox", "sandbox_name": "box", "sandbox_mode": "vm", "sandbox_type": "sail_vm", + "network_policy": {"mode": "allowlist", "allow_hosts": ["pypi.org", "*.github.com"], "allow_cidrs": ["10.0.0.0/8"]}} +``` + +**Long, mostly idle runs.** Let Sail sleep a Sailbox after 10 idle minutes; it wakes on traffic or a command: + +```toml +[sandbox.providers.sail_vm.config] +api_key = "secret:sail_api_key" +auto_sleep_min_idle_seconds = 600 +``` + +**An internal, non-HTTPS model endpoint.** Injection needs HTTPS, so pass the key into the Sailbox as other +providers do: + +```toml +[sandbox.providers.sail_vm.config] +api_key = "secret:sail_api_key" +inject_model_key = false +``` + +**A fallback chain.** Try Sail first and fall back to E2B when a Sailbox can't be created in time: + +```toml +[sandbox] +default = "sail_vm,e2b" +``` + +**From Python.** Build the configured provider and create a VM directly: + +```python +import asyncio + +from agent_env.providers.sandbox_providers.sandbox_provider import build_sandbox_provider + + +async def main() -> None: + provider = build_sandbox_provider("sail_vm") + sandbox = await provider.create_vm(cpu=1, memory=2048, exposed_ports=[8080], timeout=900) + try: + print(await sandbox.exec_with_output("docker", "info", "--format", "{{.ServerVersion}}")) + print(sandbox.tunnel_urls[8080]) # https://sb--8080.sail.box + finally: + await sandbox.terminate() + + +asyncio.run(main()) +``` + ## Resources and lifetime - **Size:** a Sailbox's size fixes its vCPU (`s`, `m`, `l` = 1, 4, 8). The provider picks the smallest