feat!: platform relations by name - #323
Open
aaaaahaaaaa wants to merge 17 commits into
Open
Conversation
…ry write; self-edge and cross-instance guards By Digitl
…es resolved through the store By Digitl
…nd_relation tools By Digitl
…ins read-only upstreams By Digitl
…nd em-dashes By Digitl
…on agent The toolkit's two write functions were reachable from no surface: the MCP server is deliberately read-only, and the ADK agent never registered them, so a relation could only be rebound through the app. They now sit in the collection agent's tool list beside its other write tools, re-exported from the toolkit package as its whole write surface, with the read-only guard on the MCP server's tool list widened to catch them. The collection instruction and the update_component copy said rebinding a relation happens in the app; both now point at the tools instead. By Digitl
…tion duplicates and drift are tested A source whose class dropped one of its assets kept the child row, and the hydrator read that child's relations against the vocabulary the narrowed class resolves: the rows the dropped asset legitimately holds then read as "relations its class does not declare", failing the whole source with a misleading HydrationError. The child set is now checked against the parent's own declaration before any child's relations are read, so the drift is reported as drift, naming the child and its source. Two hydration guards gain the tests they lacked: the new drift path, and two rows under a single-valued relation (hand-inserted, since the store's own writes repoint such a name instead of accumulating). Also: the components module docstring named a RelationMixin that does not exist (the store composes a RelationStore), remove() now states that it guards no undeclared name on purpose, so drifted rows can be cleared, and the emptiness guard in _sync_relations says why creation may leave a non-optional relation unbound while an update may not. By Digitl
The last deferred test of the branch: a definition declaring only destinations, handed a connection_id, reports that the connection fits no relation of the source rather than binding it anywhere. Sweep of what the earlier tasks left: the executor module docstring's remaining em-dash (on the one line the rewrite did not touch), a test name still saying "sibling upstreams" where the store binds sibling relations, and the migration 017 docstring, which listed neither destination among the types _PLURAL maps nor the empty-name rows its defensive DELETE drops. By Digitl
The clause the relation tools were appended to used a pair of em-dashes; parentheses carry the same aside. By Digitl
… deleted Deletion of a relation's target is governed by on_delete alone; optional governs only whether a relation may be left unbound or empty. An optional data() upstream parameter already tolerates a missing leg, so the inferred relation now declares on_delete="detach" and no longer blocks deleting the upstream through the platform's delete guard. A parameter naming a component class keeps the default block: a connection or config a read consumes stays undeletable while it is bound, optional or not. By Digitl
The delete guard treated any optional relation as detaching, so a destination in use (declared optional=True, on_delete="block") could be deleted from under the sources writing to it, and every optional consumption relation with it. _relation_detaches now reads on_delete only; an unresolvable name still blocks, keeping the guard fail-closed. optional keeps its own job, which unbind still honors: whether a relation may be left unbound or emptied. By Digitl
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 1 gave every component link one primitive,
Relation, keyed by name instead of the oldfive-mechanism vocabulary (
relation_types,RelationDefinition,resource_types,ResourceRef,depends_on,Dependency,RelationSlot). This PR carries that rename through the platform: thedatabase rows, the store, the hydrator, the HTTP API, the toolkit, the agent, and the scheduler all
address a relation by its declared name, not by a
(type, slot)pair.component_relationsrows are now(src_id, name, dst_id, ...), primary keyed on that triple.RelationStoreapplies the same acceptance rule the coreRelationdescriptor enforces (single- vsmany-valued,
on_deleteblock-or-detach), so a bind or unbind that core would refuse is refused atthe store too, not just in memory. The hydrator mirrors the core serialisation rule when it loads a
component graph back into
Relationinstances: a parented target becomes a{ref: id}the firsttime it is seen and a bare reference afterwards, same as
to_spec()writes it. It also failsclosed on a source whose class dropped one of its assets: the child row's own relations are
declared by the class it fell out of, so the child set is checked against the parent's declaration
first and the drift is reported as drift, naming the child, instead of surfacing as an undeclared
relation name on it. The HTTP API,
interloper-toolkit,interloper-agentandinterloper-schedulerall read and write relations byname end to end: no caller anywhere still asks for a
typeor aslot.Because a relation is now addressable by name alone, one generic pair of write tools covers every
kind:
bind_relation/unbind_relationare registered on the agent's Collection specialist (andre-exported from
interloper_toolkitas its whole write surface), so rebinding a source'sconnection or a job's targets no longer has to happen in the app. The MCP server stays deliberately
read-only, and its tool-surface test now guards the
bind_/unbind_prefixes too.What changes on the wire
RelationResponseand the request bodies'relationsfield are keyed byname(
{name: [RelationEntry, ...]}), not bytype/slot.RelationResponsecarriessrc_kindalongside
dst_kindso the app's graph and upstream views can filter asset-to-asset rows withouta second lookup.
GET /components/relations. Filters byname,src_kind,dst_kind(wastype).DELETE /components/{id}/relations/{name}/{dst_id}. Addresses the relation to remove byname(wastype/slotin the path).relationsJSON. A catalog entry'srelationsdict (fromcatalog.dump()/GET /catalog/{key}) is the sameRelationshape core declares: keyed by name, carryingkey,many,optional,on_delete, with no separate resource/dependency vocabulary left todistinguish.
Migration
Revision 017 is rewritten in place; it never shipped, so there is no revision 018. It adds a
namecolumn, backfills it from the old
(type, slot)pair (resource/dependency/upstreamcopyslotverbatim;destination/target/watchbecome their plural field name), then dropstypeand
slotand repoints the primary key and indexes at(src_id, name, dst_id). The downgrade is themirror: it re-adds
type/slot, maps every asset-kind row back totype = 'upstream'(a pre-017dependencytype is never recovered, since that rename never shipped), and restores the old primarykey, the partial unique index on resource slots, and the
type-keyed indexes._PLURALmatches inboth directions by
namealone.The round trip surfaced one real defect:
create_all()always provisionscomponent_relationsfromthe current model, so a freshly provisioned database already has the
namecolumn before Alembicruns at all, and the original
upgrade()unconditionally tried to add it again.upgrade()nowchecks for the
typecolumn first and no-ops when it is already gone, matching the idempotencyconvention every other additive-column migration in this package already follows (008, 015).
Run migration 017 before the api, scheduler and worker images roll (they read
name). Update theinterloper-manifestsrepo in lockstep: its YAMLdependencies:keys become relation names.Stack
Stacked on PR #321 (phase 1: core relation model). Base branch is
feat/many-upstreams-core;retarget this branch to
mainonce #321 merges. Phases 3 (matcher) and 4 (app) follow, each stackedon the one before.
Verification
interloper_scratch, never theshared dev database):
db init(create_all + upgrade head) now succeeds on a genuinely freshdatabase;
db downgrade 016produces the old(type, slot)shape; three rows were inserted inthat shape (
resource/connection,destination/``,upstream/`orders`); `db upgrade head`produced `name` values `connection`, `destinations`, `orders` as expected; `db downgrade 016`
reproduced `(destination, '')`, `(resource, connection)`, `(upstream, orders)`; `db upgrade head`
a second time succeeded cleanly. The scratch database was dropped afterward.
uv run --frozen ruff check: all checks passed.uv run --frozen ty check: all checks passed.uv run --frozen pytest -q: 2752 passed, 0 failed(interloper-core 1474, interloper-db 452, interloper-api 388, interloper-assets 89,
interloper-scheduler 107, interloper-google-cloud 123, interloper-slack 23, interloper-toolkit 16,
interloper-agent 17, interloper-pandas 37, interloper-mcp 8, interloper-k8s 8, interloper-app 6,
interloper-docker 4).
slot,.type ==,type="upstream",type="watch",relation_types,resource_types,RelationDefinition,Dependency,depends_on) overinterloper-db,interloper-api,interloper-toolkit,interloper-agent,interloper-scheduler,interloper-slack,interloper-mcp: every hit reviewed. One stale test name found and fixed(
interloper-slack/tests/test_hook.py:test_declares_a_connection_slotrenamed totest_declares_a_connection_relation, since the relation it asserts on is no longer a "slot").Every other hit is a legitimate unrelated sense: Alembic's own
depends_onrevision-metadatafield (every migration file), a queue/quota "slot" (reservation, concurrency, dispatch), a
module-level test fixture "slot", and the migration 017 file itself, which is the definition of
the old vocabulary, not a residue of it.
git diff 9f1c2b84..HEAD | grep '^+' | grep -c (the em-dash character)was 5 beforethe sweep, 0 after. Fixed: two docstrings in
interloper-api/tests/routes/test_components.py(
TestListRelations,TestComponentResponseRelations), the rewritten half ofinterloper-scheduler/src/interloper_scheduler/executor.py's module docstring, and twoArgs:entries in
interloper-db/src/interloper_db/store/components.py(create/update, bothdescribing the
relationsparameter) that the sweep instructions did not name but theverification command caught. Counting added lines missed one em-dash the branch left in place:
the executor docstring's first sentence spans a line the rewrite did not touch, so its em-dash
never showed up in the diff. It is gone now, and the sweep is 0 over the whole branch.
Follow-ups
Both tests deferred during the tasks (the two-rows-on-a-single-relation
HydrationError, and asource with zero connection relations handed a
connection_id) are written; nothing is leftdeferred. Carried forward for a later PR: relation reads still fan out per destination without
ComponentRelation.dsteager-loaded,bind_siblingstakes no row lock (its rows are the ones nouniqueness index covers), and
RelationEdge.param_namekeeps a name the relation model no longeruses, which is an MCP consumer break to time deliberately. Three smaller ones: the catalog wire
repeats each relation's name inside its own value (phase 1 output, to align when phase 4 consumes
it), the agent's "requires a X as Y" error can interpolate a list of keys, and a
resolve_fetch_fieldcall on a non-component path raisesAttributeErrorinto a 500.Added after the phase 4 walk
Two commits landed here after the app walk showed a used destination could be deleted:
a8f720c3fix(core)!: an optional inferred upstream (x: il.Upstream | None) is declaredon_delete="detach", so an optional upstream may vanish; resource-class parameters keepblock.99df39defix(db)!:on_deletealone decides whether a referrer blocks a delete;optionalgoverns unbinding and emptiness only.destinations(optional,on_delete="block") therefore blocks deleting a destination in use, as before the refactor. Tests cover both knobs independently.By Digitl