feat(dashboard): quantify cases and preserve project journeys

.env.example

@@ -39,6 +39,8 @@ MCP_RATE_LIMIT_WINDOW_SECONDS=
 # running contextual-orchestrator to turn the channels on.
 ORCHESTRATOR_BASE_URL=
 ORCHESTRATOR_API_KEY=
+SOURCE_RESEARCH_MAXIMUM_LEADS=
+SOURCE_RESEARCH_MAXIMUM_RESULTS=
 
 # GitHub workflows inject the canonical provider names from masked secrets.
 # Non-GitHub Compose runs also accept the operator's ~/.env compatibility
@@ -49,7 +51,6 @@ LLM_GATEWAY_API_URL=
 # Compatibility alias; LLM_GATEWAY_API_URL wins when both are set.
 LLM_GATEWAY_URL=
 LLM_GATEWAY_API_KEY=
-LLM_GATEWAY_EMBEDDING_MODEL=
 LLM_API_GATEWAY=
 LLM_API_KEY=
 CALDAV_BASE_URL=

AGENTS.md

@@ -21,6 +21,14 @@ documents unless an ADR explicitly promotes a decision from them --
 to its governing ADR. Update research notes as literature changes; never
 use them to introduce an untracked architecture decision.
 
+Customer-facing copy must help the reader take the next product action. Do
+not expose implementation boundaries, provider or package names, schema
+versions, internal status or reason codes, environment variables, transport
+setup, hashes, or developer remediation instructions as explanatory UI copy.
+Keep that evidence in governed audit and administrator surfaces and logs;
+translate a customer-visible state into the source, decision, retry, or
+administrator action the reader can actually take.
+
 ## Hard rule: no real data in repository artifacts
 
 This repository ships **synthetic fixtures only** (`lineageweave/fixtures.py`)
@@ -188,8 +196,10 @@ contextual-orchestrator owns model discovery and selection.
 
 `NullEmbeddingClient`, `NullAdjudicationClient`,
 `NullKeymanExtractionClient`, `NullEntityRelationshipClient`,
-`NullPostSummaryClient`, `NullPostChatClient`, and
-`NullCommitmentExtractionClient` (and any new channel client you add)
+`NullPostSummaryClient`, `NullPostChatClient`,
+`NullCommitmentExtractionClient`, `NullRelationVerificationClient`,
+`NullClaimVerificationClient`, and `NullSourceResearchClient`
+(and any new channel client you add)
 must set `available = False` and make their channel dropped +
 renormalized (`reconstruct.active_weights`), never silently return a
 placeholder score, invented Keyman, guessed relationship, fabricated
@@ -202,6 +212,14 @@ adjudication does -- never a raw LLM API. Demo TEPP seed goes through
 envelope is Failed (`tepp_not_available` / `tepp_result_not_persisted`),
 never a fabricated theta or a local psychometric substitute.
 
+Public source-reference research (ADR 0274) is a post-scoped write action
+on existing semantic units or image regions. Only `visibility_code=public`
+posts may send lead text to SearXNG or retrieve a result URL. Private posts
+fail closed without egress. Redirects and non-global targets are rejected.
+Unavailable search, retrieval, or adjudication is `research_unavailable`,
+never a fabricated supported/refuted judgment. Global Ask public
+verification (ADR 0215) still never fetches result URLs.
+
 The lineage `text` channel follows [ADR 0190](docs/adr/0190-lineage-text-channel-embedding-swap.md):
 when an embedding provider is configured, `reconstruct()` precomputes
 batched label embeddings once per reconstruction and scores cosine
@@ -237,6 +255,20 @@ vector degrades that pair back to difflib; it never fabricates a score.
 
 ## Tests
 
+### Isolated Compose lifecycle
+
+- The canonical standalone Compose project is `lineageweave` (ADR 0224).
+- A test, review, or stacked-PR environment may use an explicit isolated
+  project name only while that environment is needed. Once its stated test or
+  review objective has succeeded, preserve the relevant evidence and port any
+  required behavior into the canonical Compose contract, then run `docker
+  compose -p <exact-project> down` so its containers and network do not become
+  a second production-looking stack.
+- Never use `down -v` or otherwise delete named volumes without separate,
+  explicit authorization. Resolve the exact project from Compose labels before
+  cleanup; never target a glob, directory root, or another agent's active
+  environment.
+
 ```bash
 # backend extra compiles fast-mlsirm's PyO3 core -- needs rustc 1.97.1
 # (see backend/Dockerfile). Without it, pip falls over at build time.

ARCHITECTURE.md

@@ -67,7 +67,7 @@ flowchart LR
 | `models.py` | `Record`, `Edge`, `Tree` -- source-agnostic data shapes |
 | `channels.py` | Independent `[0, 1]` scoring functions |
 | `chunking.py` | Splits a document into meaning-identifiable units (paragraph, sentence, DOM, conversation-turn) plus embedded-image extraction, in document order |
-| `embedding_client.py` | Pluggable text-embedding channel (`Null` default, `OpenAiCompatible` real impl) + `chunked_max_similarity` |
+| `embedding_client.py` | Provider-neutral contextual-orchestrator embedding transport and strict vector-envelope validation; no local similarity arithmetic |
 | `adjudication_client.py` | Pluggable LLM-judgment channel (`Null` default, `ContextualOrchestrator` real impl) |
 | `image_content.py` | Pluggable vision channel: OCR + object recognition/tagging for embedded images (`Null` default, `OpenAiCompatibleVisionClient` real impl). The product popup (`frontend/src/PostBody.tsx`) renders each `data:image` payload in document order so the buyer sees the picture, not the base64 string; GET does not call the vision client. |
 | `tepp_client.py` | TEPP's published `AnalysisRunRequest` wire contract, pluggable transport |
@@ -172,6 +172,10 @@ real-provider LLM tests).
 provider (`docker/keycloak/realm-export.json` seeds a `lineageweave-demo`
 realm with synthetic demo accounts carrying `corp_code` / `pu_code` as
 custom token claims -- see [README](README.md#local-product-stack-docker-compose)).
+ADR 0224 fixes the default project name to `lineageweave` and keeps the
+migration, SearXNG, contextual-orchestrator, backend, worker, and frontend in
+that same project. Test stacks use an explicit disposable `-p` name; they never
+replace a canonical service with a container built from another worktree.
 `scripts/smoke_test_oidc.py` proves the round-trip is real: it logs in as
 the synthetic demo user, fetches Keycloak's live JWKS, and cryptographically
 verifies the returned JWT's RS256 signature rather than just checking for an
@@ -462,6 +466,14 @@ name, confirmed the events on the activity endpoint, and independently
 confirmed the stream's existence and length with `valkey-cli` directly
 against the `valkey` container.
 
+Post-content ingestion uses the same transport with a stronger durability
+boundary (ADR 0098): PostgreSQL owns each job and Valkey only wakes the worker.
+`POST /api/post-content/backfill` is a `post_admin`-gated producer for one
+1--200-row eligible page. It commits jobs before publishing, returns HTTP 202
+without running semantic providers, and reports wake-ups that the worker's
+bounded recovery sweep must republish. `FOR UPDATE SKIP LOCKED` partitions
+concurrent operator calls without a second scheduler or an in-memory task.
+
 ## Phase 5c: customer commitment derivation and the calendar
 
 The brief asked for two separate-sounding things: issues auto-registered
@@ -805,6 +817,18 @@ HTML-wrapped, base64-image-embedded version of the existing
 people through the live `/extract-keymen` endpoint
 (`test_extract_keymen_normalizes_html_and_embedded_image_content`).
 
+## Evidence-operations lifecycle projection
+
+ADR 0206's Dashboard persists a semantic classification separately from its
+facts and observed milestones. `operations_case_milestone` binds a closed XES-
+style activity code to an exact evidence span, evidence-post digest, observed
+instant, and named source clock; `operations_case_missing_milestone` records an
+unsupported required endpoint without fabricating one. The Dashboard pairs
+only the three declared start/end definitions for claim investigation, rebid
+response, and handover. Both endpoints yield `end - start`; a cited start plus
+a missing end is open with nullable elapsed time. API projection rechecks
+current ABAC for focal and evidence posts before returning either span.
+
 ## Phase 6d: external search verification for Ontology relation inferences
 
 The brief requires an external web/internal search agent to check the
@@ -854,6 +878,19 @@ against a deliberately fabricated one in the same request, asserting
 the former comes back `verify_corroborated` with a real evidence URL
 and the latter `verify_uncorroborated` with none.
 
+## Phase 6e: post-scoped source-reference research
+
+Issue #611's remaining ADR 0133 criterion is a different workflow from
+relation verification and from Global Ask snippet verification (ADR 0215).
+A public post may send an existing semantic unit or image-region excerpt
+to self-hosted SearXNG, retrieve one cited public page under SSRF and
+redirect rejection, and ask contextual-orchestrator to judge in
+`mode="verify"`. Private posts fail closed without egress. Citations
+persist to `source_research_citation` (migration 0236, ADR 0274). The
+reader next action is to open the cited public resource and compare it
+with the highlighted passage or image detail. Global Ask still never
+fetches result URLs.
+
 ## Phase 7: R&R's named actor is a PROV-O Agent, not always a person
 
 `post_summary.py`'s R&R extraction forced every named actor into a

CHANGELOG.d/2.19.0-post-scoped-source-reference-research.md

@@ -0,0 +1,17 @@
+# 2.19.0 — Post-scoped source-reference research
+
+## Added
+
+- Public posts can research a highlighted passage or image detail against a
+  cited public page (ADR 0274, remaining ADR 0133 / issue #611). The workflow
+  reuses self-hosted SearXNG, retrieves one public HTTP(S) target with
+  redirects disabled and non-global addresses rejected, and judges through
+  contextual-orchestrator `mode=verify`. Private posts fail closed without
+  egress. Deployments must set both source-research resource budgets explicitly;
+  otherwise the channel remains unavailable. Citations persist in 3NF
+  `source_research_citation`.
+- Reader next action: open the cited public resource, then compare it with
+  the highlighted passage or image detail. Supported or refuted judgments
+  without a cited URL downgrade to not enough information. Missing search,
+  retrieval, or adjudication is `research_unavailable`, never a fabricated
+  score.

CHANGELOG.d/2.20.0-backend-contract-regressions.md

@@ -1,3 +1,3 @@
 ### Fixed
 
-- Restored the runtime-only TEPP credential setting, kept API integration fixtures collision-free, aligned asynchronous Ask tests with semantic retrieval, replaced deprecated FastAPI 422 constants, and moved Starlette integration tests to its supported `httpx2` transport.
+- Restored the runtime-only TEPP credential setting, kept API integration fixtures collision-free, aligned asynchronous Ask tests with semantic retrieval, replaced deprecated FastAPI 422 constants, moved Starlette integration tests to its supported `httpx2` transport, and adopted the SPDX license expression required by current packaging metadata.

CHANGELOG.d/2.21.2-orchestrator-promotion-preflight.md

@@ -0,0 +1,5 @@
+### Fixed
+
+- Fail closed before recreating the canonical contextual-orchestrator when the
+  exact candidate cannot authenticate a structured-capable model through the
+  currently configured gateway endpoint.

CHANGELOG.md

@@ -6,7 +6,71 @@ All notable changes to this project are documented here. Format follows
 
 ## [Unreleased]
 
-### Added
+- Evidence-rich API responses now use standard HTTP gzip content negotiation,
+  and the Dashboard k6 gate requires and verifies that negotiated transfer.
+  The exact payload measured 96,415 bytes uncompressed and 10,282 bytes on the
+  wire; its 130.43 ms five-VU maximum keeps the 20 ms release gate open
+  (ADR 0272).
+
+- Global Ask now has a consumer-selective durable worker in the canonical
+  Compose project. Per-consumer PostgreSQL advisory leases prevent overlapping
+  queue owners, while stopping post-content consumption no longer suspends Ask
+  or starts any post-content recovery from the Ask worker (ADR 0279).
+
+- Post-content recovery now keys every initial attempt, retry, and stale lease
+  by its exact eligibility instant, so work that becomes due after the durable
+  cursor advances is reached without waiting for a full ledger wrap.
+
+- Global Ask public verification now admits only bounded, provenance-bearing
+  persisted claims attached to exact cited public posts; missing admission
+  fails closed without token-overlap egress.
+
+### Added
+
+- Product extraction now uses a strict receipt-bearing orchestrator schema,
+  revalidates the focal source and complete typed-target digest under a shared
+  replacement lock, and continues after an independent operations-stage
+  failure. Typed operations/project targets are admitted only from current
+  evidence-bearing assertions; catalog misses remain misses without guessed
+  identities, and a later admission delay cannot hide an ordinary stage
+  failure (ADR 0228).
+
+- Derived Voice classification now uses one strict contextual-orchestrator
+  receipt over the exact focal Post body, admits all twelve governed Voice
+  concepts as evidence-supported multi-label assertions, and preserves the
+  imported source category separately. Exact spans and source digests fail
+  closed; valid empty results retain a completion receipt, and operations-case
+  failures no longer suppress the independent Voice stage (ADR 0244).
+
+- Governed product-catalog provisioning now accepts only explicit product
+  master rows with authorized source-record provenance, a canonical payload
+  digest, and source-linked aliases. Replays are idempotent, contradictory
+  definitions fail closed, and unresolved Post evidence tells the reader the
+  next catalog action without creating identities from model output, keywords,
+  fuzzy matches, or generic source categories.
+
+- Temporal topic influence now has a durable external-production path: the
+  worker binds the exact completed TEPP artifact, posterior draws, and
+  business-unit/PU/team/person memberships into a content-addressed request,
+  then persists only a complete, converged, identified, parity-passed
+  fast-mlsirm result. Missing owner transport, partial rows, or digest mismatch
+  remains unavailable without local scoring. Time-valid membership slices
+  remain distinct; incomplete evidence enters an event-woken awaiting state;
+  expired work is reclaimed only from its declared request/lease contract,
+  whose lease must strictly exceed the request timeout for persistence; and
+  every terminal transition matches a unique lease token. Evidence changed
+  during computation releases a fresh request automatically.
+  LineageWeave-owned request and membership bytes and producer-owned result
+  bytes are SHA-256 verified before parsing, so admission never depends on
+  cross-language JSON reserialization. Other
+  retries use only an exact remote delay or an explicit operator requeue
+  (ADR 0210).
+
+- Evidence Operations now presents cited claim, rebid, handover, external,
+  product, and Voice evidence with explicit unavailable states and source-open
+  actions. Durable analysis and bounded backfill run only in the dedicated
+  backend worker, while the web process remains responsive; topic-context
+  results fail closed when their authorized provenance is incomplete.
 
 - Period leftover pairs now caption leftover-map graphic-display pair
   segments with persisted leftover-map distance `d` (ADR 0271 /
@@ -268,6 +332,21 @@ All notable changes to this project are documented here. Format follows
 
 ### Fixed
 
+- Product requirement identifiers are now unique and authoritative; duplicated
+  occupational-taxonomy sections were consolidated without changing their
+  evidence, measurement, or provider boundaries.
+- Public-claim provenance validation now selects its sole UUID binding without
+  calling PostgreSQL's unavailable `min(uuid)` aggregate, so fresh schema
+  replays accept valid provenance-bound public claims.
+- Post-content wake-up recovery now advances through every ready ledger page
+  with a deterministic keyset, while Valkey trims only entries already
+  consumed by the worker. Large backfills can no longer replay the same first
+  page until a later queued record starves or its unread wake-up is trimmed.
+- Post-content ingestion now keeps its bounded Valkey reader and durable-ledger
+  recovery sweep live while the single provider pipeline awaits a slow
+  structured operation. Provider work remains serial; PostgreSQL claim and
+  expected-attempt fencing, restart recovery, and the existing retry budget are
+  unchanged (ADR 0098).
 - Full-corpus Event Lineage rebuilds now count candidate pairs before provider
   work and omit the optional LLM channel above the 5,000-pair ADR budget,
   preventing millions of synchronous orchestrator calls while retaining one

Makefile

@@ -2,7 +2,7 @@
 
 # Keep provider credentials outside the repository. Compose interpolation must
 # read the same home env file as the orchestrator container's env_file.
-COMPOSE := docker compose --env-file "$$HOME/.env"
+COMPOSE := COMPOSE_FILE=docker-compose.yml docker compose --env-file "$$HOME/.env"
 
 up:
 	$(COMPOSE) up -d
@@ -21,7 +21,7 @@ ps:
 # Keycloak's live JWKS, and asserts the corp_code/pu_code claims. See
 # scripts/smoke_test_oidc.py.
 smoke:
-	uv run --locked python scripts/smoke_test_oidc.py
+	uv run --locked --extra dev python scripts/smoke_test_oidc.py
 
 # Seeds synthetic corp/account/post rows keyed to the actual Keycloak demo
 # users' real subject ids, plus Valkey ticket_created events so Activity

README.md

@@ -123,6 +123,16 @@ compatibility aliases only. `ORCHESTRATOR_BASE_URL` and
 `ORCHESTRATOR_API_KEY` are separate, internal
 LineageWeave-to-orchestrator settings.
 
+The Compose file declares `lineageweave` as its canonical default project, so
+the same eight-service synthetic stack is addressed from the repository and
+temporary worktrees. Isolated tests may override it explicitly with `-p`; use
+`docker compose down` without `-v` when retiring such a project so its named
+volumes remain recoverable (ADR 0224).
+The bundled Keycloak realm is the standalone/local/dev/test fallback only.
+Setting `KEYVERSE_ISSUER` selects central Keyverse for both backend and
+frontend and activates the fail-closed claim binding in ADR 0156; the two
+issuers are never combined as authorization authorities.
+
 Postgres and Keycloak are built (`docker/postgres-init/`, `docker/keycloak/`)
 rather than bind-mounted, so the keycloak database's init script and the
 realm seed ship inside the images themselves -- portable to any Docker host
@@ -148,6 +158,12 @@ no re-typed copy.
 `backend/` is a FastAPI app talking directly to that database (`asyncpg`,
 no ORM, no file DB) and to Keycloak's live JWKS for OIDC verification:
 
+The API does not consume durable jobs. Canonical Compose makes `backend`
+depend on the progress-healthy `backend-worker`, so `docker compose up backend`
+starts both. Any non-Compose deployment must co-deploy
+`python -m backend.app.worker` and gate API readiness on that worker service;
+`/healthz` is process liveness only.
+
 ```bash
 make up
 make seed   # scripts/seed_demo_data.py: inserts synthetic corp/account/post
@@ -168,6 +184,11 @@ docker compose --profile mcp up mcp
 # Streamable HTTP resource: http://localhost:18001/mcp
 ```
 
+For client initialization, tool arguments, durable status handling, and quota
+recovery, see the [MCP manual](docs/manuals/mcp-manual.md). Workspace users can
+start with the [user guide](docs/manuals/user-guide.md); deployment and incident
+procedures are in the [operations manual](docs/manuals/operations-manual.md).
+
 `GET /api/posts`, `GET /api/posts/{post_id}`,
 `GET /api/posts/{post_id}/keymen`, `GET /api/keymen/{person_id}/related`,
 `GET /api/posts/{post_id}/affiliate-tree`,