Add configurable Mason server and durable runtime modes

[shivam5]

What changed

• Adds databricks_mason.AgentApp, the Mason HTTP server for foreground, streaming, and background invocations. AgentApp() uses process-local state; AgentApp(durable_runtime=True) adds Lakebase persistence, heartbeats, and crash recovery.
• Simplifies mason init to three user-facing modes for both LangGraph and OpenAI:

Command	Server	Runtime state
mason init my-agent	Mason AgentApp	Durable Lakebase runtime after deploy (default)
mason init my-agent --no-durable-runtime	Mason AgentApp	Process-local background/event state
mason init my-agent --server custom	Plain FastAPI	Foreground-only; no Mason AgentApp

• Removes the previous automatic-recovery CLI/configuration knob. Recovery is a capability of the durable runtime: the generated app registers @app.on_recovery only when DURABLE_RUNTIME is enabled.
• Adds minimal custom FastAPI templates for LangGraph and OpenAI while retaining the full existing agents, tools, HITL, sessions, memory, and chat UI in the Mason templates.
• Reuses databricks_mason.workspace_client() and workspace_headers() in custom templates for account-host workspace routing without using AgentApp.
• Provisions/reuses a Lakebase durability store only when generated agent.toml contains [durability] enabled = true. The process-local and custom modes do not provision it.
• Keeps runtime/main.py thin: configure the agent, construct the app, register hooks, and install the optional UI.

Validation

Validated through commit fe86e628e6f3297b056504674c458a964af9afea.

• Mason package: 400 passed
• ruff format, ruff check, ty check, uv build, and git diff --check: passed
• Generated and synced all six framework/mode combinations using the installed mason command
• Every generated project passed its own tests and package build
• Every generated project started with mason dev and completed a real model invocation using explicit profile e2-dogfood

Framework	Mode	Generated tests	Live local invocation
LangGraph	Durable Mason	26 passed, 1 skipped	HTTP 200, langgraph durable ok
LangGraph	Process-local Mason	26 passed, 1 skipped	HTTP 200, langgraph-local ok
LangGraph	Custom FastAPI	2 passed	HTTP 200, langgraph-custom ok
OpenAI	Durable Mason	27 passed, 1 skipped	HTTP 200, openai-durable ok
OpenAI	Process-local Mason	27 passed, 1 skipped	HTTP 200, openai-local ok
OpenAI	Custom FastAPI	2 passed	HTTP 200, openai-custom ok

Install and scaffold all six modes

git clone https://github.com/shivam5/databricks-ai-bridge.git
cd databricks-ai-bridge
git checkout poc/durable-existing-template-migration

export BRIDGE=$PWD
export PROFILE=e2-dogfood

uv tool install --force --editable "$BRIDGE/integrations/mason"

mason init /tmp/langgraph-durable \
  --framework langgraph --profile "$PROFILE" \
  --repo https://github.com/shivam5/databricks-ai-bridge.git \
  --ref poc/durable-existing-template-migration

mason init /tmp/langgraph-local \
  --framework langgraph --no-durable-runtime --profile "$PROFILE" \
  --repo https://github.com/shivam5/databricks-ai-bridge.git \
  --ref poc/durable-existing-template-migration

mason init /tmp/langgraph-custom \
  --framework langgraph --server custom --profile "$PROFILE" \
  --repo https://github.com/shivam5/databricks-ai-bridge.git \
  --ref poc/durable-existing-template-migration

mason init /tmp/openai-durable \
  --framework openai --profile "$PROFILE" \
  --repo https://github.com/shivam5/databricks-ai-bridge.git \
  --ref poc/durable-existing-template-migration

mason init /tmp/openai-local \
  --framework openai --no-durable-runtime --profile "$PROFILE" \
  --repo https://github.com/shivam5/databricks-ai-bridge.git \
  --ref poc/durable-existing-template-migration

mason init /tmp/openai-custom \
  --framework openai --server custom --profile "$PROFILE" \
  --repo https://github.com/shivam5/databricks-ai-bridge.git \
  --ref poc/durable-existing-template-migration

Run tests, builds, and local servers

For each generated directory:

cd /tmp/<generated-directory>
uv sync
uv run pytest -q
uv build
mason --profile "$PROFILE" dev

Mason server request:

export INVOCATION_ID=$(uuidgen)
curl -sS http://localhost:8000/api/invocations \
  -H 'Content-Type: application/json' \
  -d "{\"id\":\"$INVOCATION_ID\",\"input\":[{\"role\":\"user\",\"content\":\"Reply exactly: mode ok\"}]}"

Custom FastAPI request:

curl -sS http://localhost:8000/invocations \
  -H 'Content-Type: application/json' \
  -d '{"input":[{"role":"user","content":"Reply exactly: mode ok"}]}'

Deployed durable-runtime evidence

The durable LangGraph path was deployed to e2-dogfood with one instance and an app-owned Lakebase database:

App: mason-pr550-durable-lg-0908
Database: projects/mason-pr550-durable-lg-0908-durability/branches/production/databases/databricks-postgres
Schema: databricks_mason_runtime_5752c6d7ed88

Validated foreground, foreground SSE, background polling, background persisted SSE, UUID idempotency, multi-turn sessions, and HITL. A background invocation was then interrupted by stopping the app and recovered after restart:

Before stop:  status=ACTIVE,    attempt=1
After start:  status=COMPLETED, attempt=2, output=recovered-ok

Lakebase retained the attempt-1 lifecycle/tool-call events and added attempt-2 run.started, streamed deltas/messages, and run.completed. The OpenAI durable path was also deployed and validated for foreground and SSE responses.

[elainewang-db]

qq: should we use these flag or just specify something like --template which fills in integrations/mason/templates/TEMPLATE ?

[shivam5]

I would not prefer going with template because then the developer has to read our templates, reason about them, and then select one.
If we are going with --template, then we should remove the all --framework, etc knobs. But would not prefer that.

[elainewang-db]

hmm that makes sense, but then we need to add new flags for new templates. I feel like it's fine to keep the other templates less discoverable with documentation for the --template flag, especially if we want to push new users to use our sdk provided server.

[shivam5]

yeah that does make sense, but i would argue we should not expose a lot of templates as well.
It violates the principal that mason --help should be self sufficient.
I would defer adding --templates to a later pr, when we really need it.

[elainewang-db]

then can we remove the --durability and --server flags too? the other templates can just serve as examples until we have a need to expose them.

[shivam5]

makes sense, will remove the durability for now. I think --server can be of value since custom serves the cuj of just porting existing agent and deploying on runtime