Jev conversation-to-document workflow

.env.example

@@ -22,6 +22,14 @@ HARNESS=copilot-sdk
 BACKGROUND_JOBS=on
 WEB_RESEARCH=on
 
+# Optional conversation-derived cards prototype; independent of AGENT.
+CONVERSATION_PLAN=off
+JEV_MODEL=jev-latest
+# Per Jev request, in milliseconds (100–60000; default 30000).
+JEV_TIMEOUT_MS=30000
+# Required only when CONVERSATION_PLAN=on. Keep this key on the server.
+JEV_API_KEY=
+
 # GitHub App sign-in.
 # Remote VM mode overrides this local origin in child processes; see docs/exe-dev.md.
 APP_ORIGIN=http://127.0.0.1:8787

AGENTS.md

@@ -100,9 +100,10 @@ external implementation runs are durable.
 - **The dialect is an allowlist.** Document MDX is parsed and rendered, never
   evaluated. Keep imports, exports, expressions, raw HTML, and unknown JSX out.
 - **Records own decisions.** Question answers and accepted comment decisions live
-  in sidecar records. Their document components are projections. Planner
-  operations protect those projections, but browser CRDT validation does not
-  yet cross-check them against records; do not treat the projection as authority.
+  in sidecar records. Their document components are projections. Browser CRDT
+  validation allows unchanged projections to move but rejects their creation,
+  removal or alteration. Domain operations update records and projections together;
+  do not treat the projection as authority.
 - **Admission is not authorization.** Optional user and organization lists admit
   an identity. Browser routes, sockets, and Planner tools separately recheck the
   App installation and repository role.
@@ -139,13 +140,12 @@ replayed automatically because it may already have made durable tool changes.
 
 ## Questions, comments, and anchors
 
-Question submission currently claims the shared draft with `claimSubmit()`,
-projects the answer and authoritative record inside the room lock, calls
-`Store.stage()`, persists the document or sidecar, and invokes the returned
-finalizer. `stage()` removes the open draft before persistence, and the failure
-path does not restore it after a storage error. Treat this as a known durability
-gap: a proper two-phase refactor must retain or restore the draft until the
-fenced commit succeeds.
+Question submission claims the shared draft with `claimSubmit()`, then stages
+the document, authoritative record, and copied question maps inside the room
+lock. Publish the candidate only after its fenced commit and invoke the returned
+`Store.stage()` finalizer afterward. A failed commit releases the claim and keeps
+the accepted draft available for retry. Planner asks and cancellation follow the
+same persistence-before-publication ordering.
 
 A decision's definition is frozen except for appended options. Any writer may
 send `question:option` to add one while the question is open (at most 10 options,
@@ -160,8 +160,9 @@ renders and derives.
 Anchors combine Yjs relative positions with canonical block digests. A position
 survives surrounding edits; a digest can recover one unique block after a move
 or epoch replacement. Ambiguous matches must orphan rather than guess. The safe
-ordering is to rebase against the old document before a server-authored edit;
-the current Planner path reconciles first and is a known recovery gap.
+ordering is to rebase against the old document before a server-authored edit.
+Scoped decision-prose jobs follow that ordering. The ordinary `edit_plan` tool
+still reconciles first and remains a recovery gap.
 
 The browser starts a comment from a bounded quote locator, selected length,
 offset hint, and block indices, not an unbounded copy of selected text. The
@@ -279,9 +280,9 @@ so merge ranges from all mounted editors before replacing a registry entry.
   after `send()` loses edits on disconnect.
 - Rebuilds and reconnects must replay unacknowledged updates only when the epoch
   is still compatible.
-- Idle eviction removes a room registry entry before its asynchronous close and
-  checkpoint completes. Until that lifecycle is serialized, avoid opening a
-  replacement room during close and test revision conflicts around eviction.
+- Room opening and closing share the document lifecycle lock. Opening waits for
+  an active close; eviction forgets the room only after runtime shutdown and
+  persistence complete. Preserve that ordering.
 - Sidecar restoration currently drops an invalid optional implementation graph
   instead of rejecting the whole sidecar. Do not generalize that fail-open
   behavior to other durable fields.

README.md

@@ -89,6 +89,10 @@ and tool vocabulary remain optimized for planning.
   credentials in an OS vault or an explicitly consented local file. Documents,
   transcripts, decisions, research request staging, background-job inputs and
   artifacts, and token-free session records are stored in PostgreSQL.
+- Optional conversation-derived cards default to off. With `CONVERSATION_PLAN=on`,
+  current and recent Chat messages and selected decision context are sent to
+  TypeSafe's Jev service for interpretation. This uses the server's `JEV_API_KEY`,
+  separately from the Planner's harness credentials, and continues with `AGENT=off`.
 - One Chopin process may write to a database at a time. Horizontal application
   scaling and zero-downtime rolling deployment are not supported.
 

apps/server/src/agent/card-tool-context.ts

@@ -0,0 +1,10 @@
+import { jobToolContext } from "./job-tool-context";
+import type { Context as JobContext } from "./job-tool-context";
+import type { DocumentRoom } from "./tools";
+
+export type Context = JobContext & { currentMemberRequest?: () => { text: string } | undefined };
+export type CardToolRoom = DocumentRoom & Pick<Context, "currentMemberRequest">;
+
+export function cardToolContext(room: CardToolRoom): Context {
+	return { ...jobToolContext(room), currentMemberRequest: room.currentMemberRequest };
+}

apps/server/src/agent/decision-prose-1.test.ts

@@ -0,0 +1,53 @@
+import { expect, test } from "bun:test";
+import * as room from "../plan/room";
+import * as Service from "../plan/service";
+import * as Questions from "../questions/service";
+import { createDecisionProseFixture } from "./decision-prose.test-fixtures";
+
+let contexts: ReturnType<typeof createDecisionProseFixture>["contexts"];
+let fixture = createDecisionProseFixture(() => contexts, value => {
+	contexts = value;
+});
+contexts = fixture.contexts;
+let { CARD, WIDGET, opened, job, proseTool, input } = fixture;
+
+test("write_decision_prose commits one staged insert and publishes anchors and metadata afterward", async () => {
+	let context = await opened();
+	job(context);
+	let tool = proseTool(context);
+	let before = room.project(context.plan.document);
+	let response = await tool.handler(input(context), {} as never);
+	expect(JSON.parse(String(response))).toEqual({
+		ok: true,
+		title: "Which authentication approach?",
+		mode: "insert",
+		revision: 1,
+	});
+	expect(room.project(context.plan.document)).toBe(
+		before.replace("Context.\n\n", "Context.\n\nWe chose GitHub Apps.\n\n"),
+	);
+	expect(context.plan.document.seq).toBe(1);
+	expect(context.broadcasts.filter(frame => frame.kind === "plan:update")).toHaveLength(1);
+	expect(context.broadcasts.filter(frame => frame.kind === "plan:anchors")).toHaveLength(1);
+	expect(context.broadcasts.filter(frame => frame.kind === "question:meta")).toHaveLength(1);
+	expect(tool.anchors()).toBe(1);
+	expect(tool.changes()).toBe(1);
+	expect(Questions.prose(context.plan)[0]).toMatchObject({ widget: CARD, orphaned: false });
+	expect(room.matchesAnchor(context.plan.document, context.plan.records.get(CARD)!.prose![0]!, 1))
+		.toBe(true);
+	let loaded = (await context.storage.collaboration.load(context.channel.id, context.now))!;
+	expect((await Service.readStored(loaded)).source).toBe(room.project(context.plan.document));
+});
+
+test("write_decision_prose uses one durable delta for a first-card two-step insertion", async () => {
+	let context = await opened(WIDGET);
+	job(context);
+	let before = room.project(context.plan.document);
+	let result = await proseTool(context).handler(input(context), {} as never);
+	expect(JSON.parse(String(result)).ok).toBe(true);
+	expect(room.project(context.plan.document)).toBe(`We chose GitHub Apps.\n\n${before}`);
+	expect(context.plan.document.seq).toBe(1);
+	expect(context.broadcasts.filter(frame => frame.kind === "plan:update")).toHaveLength(1);
+	expect(room.matchesAnchor(context.plan.document, context.plan.records.get(CARD)!.prose![0]!, 0))
+		.toBe(true);
+});

apps/server/src/agent/decision-prose-2.test.ts

@@ -0,0 +1,98 @@
+import { expect, test } from "bun:test";
+import { $createParagraphNode, $getRoot, $isParagraphNode } from "lexical";
+import * as Y from "yjs";
+import * as room from "../plan/room";
+import * as Service from "../plan/service";
+import { createDecisionProseFixture } from "./decision-prose.test-fixtures";
+
+let contexts: ReturnType<typeof createDecisionProseFixture>["contexts"];
+let fixture = createDecisionProseFixture(() => contexts, value => {
+	contexts = value;
+});
+contexts = fixture.contexts;
+let { CARD, WIDGET, OTHER_WIDGET, opened, job, proseTool, input } = fixture;
+
+test("write_decision_prose addresses a first card past live leading and inter-card carets", async () => {
+	let context = await opened(`${WIDGET}\n${OTHER_WIDGET}`);
+	context.plan.document.editor.update(() => {
+		let children = $getRoot().getChildren();
+		children[0]!.insertBefore($createParagraphNode());
+		children[1]!.insertBefore($createParagraphNode());
+	}, { discrete: true });
+	await room.settle();
+	let caretKeys: string[] = [];
+	context.plan.document.editor.getEditorState().read(() => {
+		caretKeys = $getRoot().getChildren()
+			.filter(node => $isParagraphNode(node) && node.getChildrenSize() === 0)
+			.map(node => node.getKey());
+	});
+	expect(caretKeys).toHaveLength(2);
+	let relatives = caretKeys.map(key => {
+		let type = context.plan.document.binding.collabNodeMap.get(key)?.getSharedType();
+		if (!type) throw new Error("caret has no shared identity");
+		return Y.createRelativePositionFromTypeIndex(type, 0, -1);
+	});
+	let peer = await room.restore(
+		context.plan.document.epoch,
+		Y.encodeStateAsUpdate(context.plan.document.doc),
+		room.project(context.plan.document),
+		[],
+	);
+	expect(room.digests(context.plan.document)).toHaveLength(2);
+	expect(room.questionnaireIndex(context.plan.document, CARD)).toBe(0);
+	job(context);
+	let response = await proseTool(context).handler(input(context), {} as never);
+	expect(JSON.parse(String(response)).ok).toBe(true);
+	expect(room.project(context.plan.document)).toStartWith(
+		"We chose GitHub Apps.\n\n<Questionnaire",
+	);
+	expect(room.questionnaireIndex(context.plan.document, CARD)).toBe(1);
+	expect(room.matchesAnchor(context.plan.document, context.plan.records.get(CARD)!.prose![0]!, 0))
+		.toBe(true);
+	context.plan.document.editor.getEditorState().read(() => {
+		let current = new Set($getRoot().getChildren().map(node => node.getKey()));
+		for (let key of caretKeys) expect(current.has(key)).toBe(true);
+	});
+	let update = context.broadcasts.find(frame => frame.kind === "plan:update")?.update;
+	if (typeof update !== "string") throw new Error("missing persisted document delta");
+	Y.applyUpdate(peer.doc, Buffer.from(update, "base64"), "remote");
+	await room.settle();
+	let absolutes = relatives.map(relative =>
+		Y.createAbsolutePositionFromRelativePosition(relative, peer.doc, false)
+	);
+	for (let absolute of absolutes) expect(absolute).not.toBeNull();
+	peer.editor.getEditorState().read(() => {
+		let empty = $getRoot().getChildren().filter(node =>
+			$isParagraphNode(node) && node.getChildrenSize() === 0
+		);
+		expect(empty).toHaveLength(2);
+		for (let absolute of absolutes) {
+			expect(
+				empty.some(node =>
+					peer.binding.collabNodeMap.get(node.getKey())?.getSharedType() === absolute?.type
+				),
+			).toBe(true);
+		}
+	});
+	peer.doc.destroy();
+	let expected = room.project(context.plan.document);
+	await Service.close(context.plan);
+	contexts = contexts.filter(item => item !== context);
+	let reopened = await Service.open(context.channel.id, context.backend, context.server);
+	contexts.push({ ...context, plan: reopened });
+	expect(room.project(reopened.document)).toBe(expected);
+	expect(room.matchesAnchor(reopened.document, reopened.records.get(CARD)!.prose![0]!, 0))
+		.toBe(true);
+});
+
+test("write_decision_prose replaces the anchored duplicate, leaving its twin alone", async () => {
+	let context = await opened(`Same.\n\nSame.\n\n${WIDGET}`);
+	context.plan.records.get(CARD)!.prose = [
+		room.anchorAt(context.plan.document, 1, room.digests(context.plan.document)[1]!),
+	];
+	job(context);
+	let result = await proseTool(context).handler(input(context, "New prose."), {} as never);
+	expect(JSON.parse(String(result)).mode).toBe("replace");
+	expect(room.project(context.plan.document)).toStartWith("Same.\n\nNew prose.\n\n");
+	expect(context.plan.records.get(CARD)!.prose).toHaveLength(1);
+});

apps/server/src/agent/decision-prose-3.test.ts

@@ -0,0 +1,48 @@
+import { expect, spyOn, test } from "bun:test";
+import { $getRoot } from "lexical";
+import * as room from "../plan/room";
+import { createDecisionProseFixture } from "./decision-prose.test-fixtures";
+
+let contexts: ReturnType<typeof createDecisionProseFixture>["contexts"];
+let fixture = createDecisionProseFixture(() => contexts, value => {
+	contexts = value;
+});
+contexts = fixture.contexts;
+let { CARD, WIDGET, opened, job, proseTool, input } = fixture;
+
+test("write_decision_prose replaces uniquely moved prose at its new position", async () => {
+	let context = await opened(`Other context.\n\nOld decision prose.\n\n${WIDGET}`);
+	context.plan.records.get(CARD)!.prose = [
+		room.anchorAt(context.plan.document, 1, room.digests(context.plan.document)[1]!),
+	];
+	context.plan.document.editor.update(() => {
+		let children = $getRoot().getChildren();
+		children[0]!.insertBefore(children[1]!);
+	}, { discrete: true });
+	await room.settle();
+	job(context);
+	let result = await proseTool(context).handler(input(context, "New prose."), {} as never);
+	expect(JSON.parse(String(result)).mode).toBe("replace");
+	expect(room.project(context.plan.document)).toStartWith("New prose.\n\nOther context.\n\n");
+});
+
+test("write_decision_prose rejects missing or stale job generation and invalid text", async () => {
+	let context = await opened();
+	let tool = proseTool(context);
+	let before = room.project(context.plan.document);
+	let errors = spyOn(console, "error").mockImplementation(() => {});
+	try {
+		expect(String(await tool.handler(input(context), {} as never)))
+			.toContain("only available to a background Planner job");
+		job(context, 2);
+		expect(String(await tool.handler(input(context), {} as never))).toContain("generation");
+		job(context);
+		for (let text of ["# Heading", "One.\n\nTwo.", "Text <Badge />"]) {
+			expect(String(await tool.handler(input(context, text), {} as never))).toStartWith("Error:");
+		}
+		expect(room.project(context.plan.document)).toBe(before);
+		expect(context.plan.chat.jobOutput).toBeUndefined();
+	} finally {
+		errors.mockRestore();
+	}
+});

apps/server/src/agent/decision-prose-4.test.ts

@@ -0,0 +1,70 @@
+import { expect, spyOn, test } from "bun:test";
+import * as room from "../plan/room";
+import type { ConversationPlan } from "@chopin/protocol";
+import { createDecisionProseFixture } from "./decision-prose.test-fixtures";
+
+let contexts: ReturnType<typeof createDecisionProseFixture>["contexts"];
+let fixture = createDecisionProseFixture(() => contexts, value => {
+	contexts = value;
+});
+contexts = fixture.contexts;
+let { CARD, OPTION, opened, job, proseTool, input } = fixture;
+
+test("write_decision_prose fences a replaced job after an awaited staging read", async () => {
+	let context = await opened();
+	job(context);
+	let before = room.project(context.plan.document);
+	let original = room.restore;
+	let entered = Promise.withResolvers<void>();
+	let release = Promise.withResolvers<void>();
+	let held = spyOn(room, "restore").mockImplementation(async (...args) => {
+		entered.resolve();
+		await release.promise;
+		return original(...args);
+	});
+	let errors = spyOn(console, "error").mockImplementation(() => {});
+	try {
+		let action = proseTool(context).handler(input(context), {} as never);
+		await entered.promise;
+		context.plan.chat.job = { ...context.plan.chat.job!, id: "newer" } as ConversationPlan.Job;
+		release.resolve();
+		expect(String(await action)).toContain("background Planner job changed");
+		expect(room.project(context.plan.document)).toBe(before);
+		expect(context.plan.document.seq).toBe(0);
+		expect(context.plan.chat.jobOutput).toBeUndefined();
+	} finally {
+		errors.mockRestore();
+		held.mockRestore();
+	}
+});
+
+test("write_decision_prose fences a newer saved decision after an awaited staging read", async () => {
+	let context = await opened();
+	job(context);
+	let before = room.project(context.plan.document);
+	let original = room.restore;
+	let entered = Promise.withResolvers<void>();
+	let release = Promise.withResolvers<void>();
+	let held = spyOn(room, "restore").mockImplementation(async (...args) => {
+		entered.resolve();
+		await release.promise;
+		return original(...args);
+	});
+	let errors = spyOn(console, "error").mockImplementation(() => {});
+	try {
+		let action = proseTool(context).handler(input(context), {} as never);
+		await entered.promise;
+		let record = context.plan.records.get(CARD)!;
+		context.plan.records.set(CARD, {
+			...record,
+			history: [{ choices: [OPTION], owner: "mina", at: 1_758_645_000 }],
+		});
+		release.resolve();
+		expect(String(await action)).toContain("background Planner job changed");
+		expect(room.project(context.plan.document)).toBe(before);
+		expect(context.plan.chat.jobOutput).toBeUndefined();
+	} finally {
+		errors.mockRestore();
+		held.mockRestore();
+	}
+});