Migrate Nomad to FastMCP v4 beta

[awadell1]

Summary

• migrate Nomad to FastMCP 4.0.0b1 and MCP 2.x
• defer MCP tracing and propagation to FastMCP while retaining Nomad's domain metrics and child spans
• enable optional background execution for SciFM tools
• add configurable in-memory or Redis-backed Docket task storage
• derive Docket worker concurrency from tool queue and device pipeline capacity
• reserve Nomad queue capacity before Docket dispatches a task
• retain 25% per-tool queue headroom for foreground requests
• return a visible Server busy tool error at Nomad's defensive queue limit
• document Docket's 15-minute post-completion result retention

Configuration

New tool_manager settings:

• task_backend_url, default memory://
• task_min_concurrency, default 10
• device_queue_depth, default 2

Changed setting:

• max_pending_per_tool is now finite and non-nullable, defaulting to 65,536

Validation

• uv run ruff check src test
• uv run ruff format --check src test
• uv run pytest - 371 passed
• make -C docs html

Release note

This PR intentionally targets the FastMCP v4 beta. Keep it as a draft and do not merge until FastMCP v4 is officially released.

[BrennanTM]

Looked into the failing Demo Image check; it reproduces locally off this branch's lockfiles and the mechanism is narrow. The smoke test's tools/call is a direct (non-task) invocation, and FastMCP's dependency resolver treats any parameter default that is a Dependency instance as resolvable on that path too. The two task markers that build_torch_module_fastmcp_tool places in the tool signature are both worker-only: docket's ConcurrencyLimit.__aenter__ reads current_execution unconditionally, so with no docket worker on the direct path the resolver gets a LookupError and wraps it as the RuntimeError: Failed to resolve dependency '_task_concurrency' for fn in the CI log.

A few things worth knowing from the investigation:

• It is not a v4 regression or an upstream break: fastmcp 3.4.2's resolver handles these defaults identically (verified with a minimal probe; the resolver code is byte-identical between 3.4.2 and 4.0.0b1). The failure arrives with this branch because the markers arrive with this branch.
• Nothing shipped upstream fixes it: the 4.0.0b2 train fails identically with pydocket held at 0.20.1, and pydocket 0.24.1's __aenter__ still hard-requires worker context.
• _task_admission is a second, quieter instance of the same pattern. Under normal load it happens to work outside workers (docket and fastmcp share uncalled_for's _Depends.stack ContextVar, so the cleanup registration lands), but when the per-tool queue is full, AdmissionBlocked(current_execution.get(), ...) raises the same LookupError, and the client would get an internal RuntimeError for _task_admission instead of the graceful Server busy error the manager produces on its own. The six-way burst below exercises this queue-full path end to end at max_pending_per_tool: 1 with the fix in place.
• Direct calls are worth keeping first-class here: the tools are registered with task_config=TaskConfig(mode="optional"), and under the tasks extension (SEP-2663) task augmentation is per-request, so plain tools/call is sanctioned client behavior for these tools.
• Related but out of scope here: with the tool manager disabled, the unmanaged registration path builds the wrapper with _task_concurrency: ConcurrencyLimit | None = None, and pydantic rejects that annotation at call time, so direct calls fail on that path too, just differently. Flagging it so it does not surprise anyone later; happy to cover it in a follow-up.

#11 makes both markers worker-scoped: the concurrency limit subclasses ConcurrencyLimit and no-ops unless current_execution is set, and DocketQueueAdmission.__aenter__ returns before reserving when no worker owns the execution, which hands backpressure to _enqueue_request's own Server busy path (the same manager-side limit that governs direct calls on main). Worker-side enforcement is unchanged. With the fix, the exact smoke call returns real inference on this branch, a six-way burst at queue cap 1 degrades gracefully with no reservation leak, and the suite is green apart from one sandbox test that fails on my machine with or without the change. Three regression tests are included; the first is a direct run() of a built tool, which is the smoke failure's exact call shape.

Two honest caveats: the observability smoke has never executed on this branch (the job dies before reaching it), so its first run will be new information about the migration rather than about this fix; and the in-worker delegation is covered at logic level, not with a live docket round trip. This is your migration and your call: merge the stacked PR, cherry-pick it, or just take the findings. If you think the underlying sharp edge deserves an upstream report (task-sanctioned signature markers being unconditionally resolved on direct calls, with no way to scope them to task mode), I am glad to write that up too.