-
Notifications
You must be signed in to change notification settings - Fork 667
feat(sessions): deliver durable Stop directly to the runner #6503
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
39 commits
Select commit
Hold shift + click to select a range
4bdcad8
feat(api): record a session command, and stamp when a turn started
mmabrouk 6287072
feat(api): reach the runner directly to cancel a turn
mmabrouk ace9f45
feat(api): add POST /sessions/{session_id}/cancel and the outcome route
mmabrouk a4c5652
feat(runner): accept a cancel command and stop the turn it names
mmabrouk deb51f1
feat(web): point the desktop Stop button at the cancel route
mmabrouk ba52d3a
fix(sessions): make the Stop actually reach the run, and settle it
mmabrouk 059b0a5
docs(sessions): record what the durable Stop slice built and verified
mmabrouk 9c42f0a
fix(sessions): label the control-plane abort, and drop the replica ce…
mmabrouk 04ebb96
docs(sessions): update the slice record for the rebase and the census…
mmabrouk fecca18
fix(sessions): make Stop settlement write the stream row, not only Redis
mmabrouk bb2d14b
fix(web): key the liveness polls on running, not on the alive set
mmabrouk 14c4eff
fix(sessions): compare a Stop's expectation against the target it res…
mmabrouk c5ccd22
fix(sessions): settle a Stop outcome whether the row is pending or cl…
mmabrouk 5e466af
fix(sessions): a Stop that lost the race must not destroy the warm sa…
mmabrouk 78861bb
feat(sessions): gate durable stop and late output
mmabrouk 5b06659
fix(sessions): preserve legacy cancel contract
mmabrouk 442db48
fix(auth): narrow session control exemption
mmabrouk 76db4d7
fix(sessions): preserve legacy cancel response shape
mmabrouk 4889c3d
fix(sessions): persist cancelled interaction records
mmabrouk 113cad4
fix(sessions): preserve idempotent Stop targets
mmabrouk d6e93c9
fix(auth): compare runner tokens as bytes
mmabrouk 290820f
fix(sessions): keep cancellation publishing fail-open
mmabrouk 28f4b98
fix(sessions): validate direct cancel responses
mmabrouk 8d6d019
fix(runner): harden durable Stop delivery
mmabrouk 24aab68
fix(frontend): route session Stop through Fern
mmabrouk e13661e
fix(frontend): capture Stop target before unlocking sends
mmabrouk f184867
fix(mobile): react to session liveness when polling gates
mmabrouk f43cfef
style(sessions): trim liveness rationale comments
mmabrouk cd21450
docs(sessions): record direct delivery as version one
mmabrouk d91ed34
fix(runner): preserve admitted stop handle
mmabrouk 347cb5c
fix(runner): honor stop during pause teardown
mmabrouk 0791279
fix(api): replay cancel idempotency before targeting
mmabrouk 1c202ae
fix(entities): normalize legacy stop response
mmabrouk adcc987
fix(frontend): abort locally while resolving stop target
mmabrouk 00fae4b
test(runner): preserve cancel fixture type
mmabrouk 56531f6
fix(frontend): pin stop target before queued sends
mmabrouk bac1a42
fix(frontend): pin durable Stop to the streamed turn
mmabrouk d4c4e43
fix(frontend): keep new turns unpinned before metadata
mmabrouk 3935e0c
fix(sessions): make unfenced cancel running-only
mmabrouk File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
164 changes: 164 additions & 0 deletions
164
api/oss/databases/postgres/migrations/core_oss/versions/oss000000022_add_session_commands.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,164 @@ | ||
| """add session commands, and the two session_streams columns a Stop needs | ||
|
|
||
| A user Stop reached the runner only through the absence of a Redis lock, discovered on the next | ||
| heartbeat up to 30 seconds later. Nothing recorded that a Stop had been asked for, so a Stop | ||
| against an unreachable runner was simply lost and no execution ever reached a terminal outcome | ||
| anyone could read. | ||
|
|
||
| `session_commands` is that record. One row per durable request to change an execution. `state` | ||
| is where the COMMAND is (pending, claimed, applied, obsolete); `outcome` is what happened to the | ||
| EXECUTION (stopped, not_running, superseded_by_newer_turn, failed, lost). The two are separate | ||
| columns because they answer different questions and settle at different times. | ||
|
|
||
| Two columns join `session_streams`: | ||
|
|
||
| * `stopping_turn_id` names the execution an accepted Stop is waiting on, written in the same | ||
| transaction as the command insert and cleared at settlement. | ||
| * `turn_started_at` records when the row's current `turn_id` started. Nothing else could serve | ||
| the stale-Stop guard: `updated_at` is the heartbeat timestamp and moves every 30 seconds, | ||
| runner-minted turn ids are uuid4 and carry no time, the Redis lock value is a bare turn id | ||
| that a Lua compare reads whole, and the `session_turns` append is fire-and-forget so a | ||
| running turn may have no row at all. | ||
|
|
||
| Both are nullable and backfill to NULL. A row written before this migration yields no | ||
| comparison, and the guard then does not fire — deliberately, because a guard that refused every | ||
| Stop it could not verify would break the common case to protect a rare one. | ||
|
|
||
| Revision ID: oss000000022 | ||
| Revises: oss000000021 | ||
| Create Date: 2026-09-02 23:30:00.000000 | ||
|
|
||
| """ | ||
|
|
||
| from typing import Sequence, Union | ||
|
|
||
| from alembic import op | ||
| import sqlalchemy as sa | ||
| from sqlalchemy.dialects import postgresql | ||
|
|
||
|
|
||
| revision: str = "oss000000022" | ||
| down_revision: Union[str, None] = "oss000000021" | ||
| branch_labels: Union[str, Sequence[str], None] = None | ||
| depends_on: Union[str, Sequence[str], None] = None | ||
|
|
||
|
|
||
| def upgrade() -> None: | ||
| op.create_table( | ||
| "session_commands", | ||
| sa.Column("id", sa.UUID(as_uuid=True), nullable=False), | ||
| sa.Column("project_id", sa.UUID(as_uuid=True), nullable=False), | ||
| sa.Column("session_id", sa.String(), nullable=False), | ||
| sa.Column("kind", sa.String(), nullable=False), | ||
| sa.Column("target_turn_id", sa.String(), nullable=True), | ||
| sa.Column("expected_turn_id", sa.String(), nullable=True), | ||
| sa.Column("state", sa.String(), nullable=False), | ||
| sa.Column("claimed_by", sa.String(), nullable=True), | ||
| sa.Column("claim_expires_at", sa.TIMESTAMP(timezone=True), nullable=True), | ||
| sa.Column( | ||
| "claim_count", | ||
| sa.Integer(), | ||
| server_default="0", | ||
| nullable=False, | ||
| ), | ||
| sa.Column("outcome", sa.String(), nullable=True), | ||
| sa.Column("idempotency_key", sa.String(), nullable=True), | ||
| sa.Column("settled_at", sa.TIMESTAMP(timezone=True), nullable=True), | ||
| sa.Column("data", sa.JSON(), nullable=True), | ||
| sa.Column( | ||
| "flags", | ||
| postgresql.JSONB(none_as_null=True), | ||
| nullable=True, | ||
| ), | ||
| sa.Column( | ||
| "tags", | ||
| postgresql.JSONB(none_as_null=True), | ||
| nullable=True, | ||
| ), | ||
| sa.Column("meta", sa.JSON(), nullable=True), | ||
| sa.Column( | ||
| "created_at", | ||
| sa.TIMESTAMP(timezone=True), | ||
| server_default=sa.func.current_timestamp(), | ||
| nullable=True, | ||
| ), | ||
| sa.Column("updated_at", sa.TIMESTAMP(timezone=True), nullable=True), | ||
| sa.Column("deleted_at", sa.TIMESTAMP(timezone=True), nullable=True), | ||
| sa.Column("created_by_id", sa.UUID(as_uuid=True), nullable=True), | ||
| sa.Column("updated_by_id", sa.UUID(as_uuid=True), nullable=True), | ||
| sa.Column("deleted_by_id", sa.UUID(as_uuid=True), nullable=True), | ||
| sa.CheckConstraint("kind IN ('cancel')", name="ck_session_commands_kind"), | ||
| sa.CheckConstraint( | ||
| "state IN ('pending', 'claimed', 'applied', 'obsolete')", | ||
| name="ck_session_commands_state", | ||
| ), | ||
| sa.ForeignKeyConstraint( | ||
| ["project_id"], | ||
| ["projects.id"], | ||
| ondelete="CASCADE", | ||
| ), | ||
| sa.PrimaryKeyConstraint("project_id", "id"), | ||
| sa.UniqueConstraint( | ||
| "project_id", | ||
| "session_id", | ||
| "idempotency_key", | ||
| name="uq_session_commands_idempotency", | ||
| ), | ||
| ) | ||
| # One open command per target execution, enforced by the database because admission's | ||
| # read-then-insert races itself: two Stops in the same instant both find no open command. | ||
| op.create_index( | ||
| "uq_session_commands_open_target", | ||
| "session_commands", | ||
| ["project_id", "session_id", "kind", "target_turn_id"], | ||
| unique=True, | ||
| postgresql_where=sa.text( | ||
| "state IN ('pending', 'claimed') AND deleted_at IS NULL" | ||
| ), | ||
| ) | ||
| op.create_index( | ||
| "ix_session_commands_open", | ||
| "session_commands", | ||
| ["project_id", "session_id", "created_at"], | ||
| postgresql_where=sa.text( | ||
| "state IN ('pending', 'claimed') AND deleted_at IS NULL" | ||
| ), | ||
| ) | ||
| op.create_index( | ||
| "ix_session_commands_claims", | ||
| "session_commands", | ||
| ["claim_expires_at"], | ||
| postgresql_where=sa.text("state = 'claimed' AND deleted_at IS NULL"), | ||
| ) | ||
| op.create_index( | ||
| "ix_session_commands_project_session", | ||
| "session_commands", | ||
| ["project_id", "session_id", "created_at"], | ||
| ) | ||
| # The runner reports an outcome with the command id alone; it holds no project credential, | ||
| # so that read cannot use the primary key's leading column. | ||
| op.create_index( | ||
| "ix_session_commands_id", | ||
| "session_commands", | ||
| ["id"], | ||
| ) | ||
|
|
||
| op.add_column( | ||
| "session_streams", | ||
| sa.Column("stopping_turn_id", sa.String(), nullable=True), | ||
| ) | ||
| op.add_column( | ||
| "session_streams", | ||
| sa.Column("turn_started_at", sa.TIMESTAMP(timezone=True), nullable=True), | ||
| ) | ||
|
|
||
|
|
||
| def downgrade() -> None: | ||
| op.drop_column("session_streams", "turn_started_at") | ||
| op.drop_column("session_streams", "stopping_turn_id") | ||
| op.drop_index("ix_session_commands_id", table_name="session_commands") | ||
| op.drop_index("ix_session_commands_project_session", table_name="session_commands") | ||
| op.drop_index("ix_session_commands_claims", table_name="session_commands") | ||
| op.drop_index("ix_session_commands_open", table_name="session_commands") | ||
| op.drop_index("uq_session_commands_open_target", table_name="session_commands") | ||
| op.drop_table("session_commands") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.