Skip to content

feat!: platform relations by name - #323

Open
aaaaahaaaaa wants to merge 17 commits into
feat/many-upstreams-corefrom
feat/relation-model-platform
Open

feat!: platform relations by name#323
aaaaahaaaaa wants to merge 17 commits into
feat/many-upstreams-corefrom
feat/relation-model-platform

Conversation

@aaaaahaaaaa

@aaaaahaaaaa aaaaahaaaaa commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Phase 1 gave every component link one primitive, Relation, keyed by name instead of the old
five-mechanism vocabulary (relation_types, RelationDefinition, resource_types, ResourceRef,
depends_on, Dependency, RelationSlot). This PR carries that rename through the platform: the
database rows, the store, the hydrator, the HTTP API, the toolkit, the agent, and the scheduler all
address a relation by its declared name, not by a (type, slot) pair.

component_relations rows are now (src_id, name, dst_id, ...), primary keyed on that triple.
RelationStore applies the same acceptance rule the core Relation descriptor enforces (single- vs
many-valued, on_delete block-or-detach), so a bind or unbind that core would refuse is refused at
the store too, not just in memory. The hydrator mirrors the core serialisation rule when it loads a
component graph back into Relation instances: a parented target becomes a {ref: id} the first
time it is seen and a bare reference afterwards, same as to_spec() writes it. It also fails
closed on a source whose class dropped one of its assets: the child row's own relations are
declared by the class it fell out of, so the child set is checked against the parent's declaration
first and the drift is reported as drift, naming the child, instead of surfacing as an undeclared
relation name on it. The HTTP API,
interloper-toolkit, interloper-agent and interloper-scheduler all read and write relations by
name end to end: no caller anywhere still asks for a type or a slot.

Because a relation is now addressable by name alone, one generic pair of write tools covers every
kind: bind_relation / unbind_relation are registered on the agent's Collection specialist (and
re-exported from interloper_toolkit as its whole write surface), so rebinding a source's
connection or a job's targets no longer has to happen in the app. The MCP server stays deliberately
read-only, and its tool-surface test now guards the bind_/unbind_ prefixes too.

What changes on the wire

  • API models. RelationResponse and the request bodies' relations field are keyed by name
    ({name: [RelationEntry, ...]}), not by type/slot. RelationResponse carries src_kind
    alongside dst_kind so the app's graph and upstream views can filter asset-to-asset rows without
    a second lookup.
  • GET /components/relations. Filters by name, src_kind, dst_kind (was type).
  • DELETE /components/{id}/relations/{name}/{dst_id}. Addresses the relation to remove by
    name (was type/slot in the path).
  • Catalog relations JSON. A catalog entry's relations dict (from catalog.dump() /
    GET /catalog/{key}) is the same Relation shape core declares: keyed by name, carrying key,
    many, optional, on_delete, with no separate resource/dependency vocabulary left to
    distinguish.

Migration

Revision 017 is rewritten in place; it never shipped, so there is no revision 018. It adds a name
column, backfills it from the old (type, slot) pair (resource/dependency/upstream copy
slot verbatim; destination/target/watch become their plural field name), then drops type
and slot and repoints the primary key and indexes at (src_id, name, dst_id). The downgrade is the
mirror: it re-adds type/slot, maps every asset-kind row back to type = 'upstream' (a pre-017
dependency type is never recovered, since that rename never shipped), and restores the old primary
key, the partial unique index on resource slots, and the type-keyed indexes. _PLURAL matches in
both directions by name alone.

The round trip surfaced one real defect: create_all() always provisions component_relations from
the current model, so a freshly provisioned database already has the name column before Alembic
runs at all, and the original upgrade() unconditionally tried to add it again. upgrade() now
checks for the type column first and no-ops when it is already gone, matching the idempotency
convention every other additive-column migration in this package already follows (008, 015).

Run migration 017 before the api, scheduler and worker images roll (they read name). Update the
interloper-manifests repo in lockstep: its YAML dependencies: keys become relation names.

Stack

Stacked on PR #321 (phase 1: core relation model). Base branch is feat/many-upstreams-core;
retarget this branch to main once #321 merges. Phases 3 (matcher) and 4 (app) follow, each stacked
on the one before.

Verification

  • Throwaway-database round trip against a real local Postgres (interloper_scratch, never the
    shared dev database): db init (create_all + upgrade head) now succeeds on a genuinely fresh
    database; db downgrade 016 produces the old (type, slot) shape; three rows were inserted in
    that shape (resource/connection, destination/``, upstream/`orders`); `db upgrade head`
    produced `name` values `connection`, `destinations`, `orders` as expected; `db downgrade 016`
    reproduced `(destination, '')`, `(resource, connection)`, `(upstream, orders)`; `db upgrade head`
    a second time succeeded cleanly. The scratch database was dropped afterward.
  • uv run --frozen ruff check: all checks passed.
  • uv run --frozen ty check: all checks passed.
  • uv run --frozen pytest -q: 2752 passed, 0 failed
    (interloper-core 1474, interloper-db 452, interloper-api 388, interloper-assets 89,
    interloper-scheduler 107, interloper-google-cloud 123, interloper-slack 23, interloper-toolkit 16,
    interloper-agent 17, interloper-pandas 37, interloper-mcp 8, interloper-k8s 8, interloper-app 6,
    interloper-docker 4).
  • Retired-name sweep (slot, .type ==, type="upstream", type="watch", relation_types,
    resource_types, RelationDefinition, Dependency, depends_on) over interloper-db,
    interloper-api, interloper-toolkit, interloper-agent, interloper-scheduler,
    interloper-slack, interloper-mcp: every hit reviewed. One stale test name found and fixed
    (interloper-slack/tests/test_hook.py: test_declares_a_connection_slot renamed to
    test_declares_a_connection_relation, since the relation it asserts on is no longer a "slot").
    Every other hit is a legitimate unrelated sense: Alembic's own depends_on revision-metadata
    field (every migration file), a queue/quota "slot" (reservation, concurrency, dispatch), a
    module-level test fixture "slot", and the migration 017 file itself, which is the definition of
    the old vocabulary, not a residue of it.
  • Em-dash sweep: git diff 9f1c2b84..HEAD | grep '^+' | grep -c (the em-dash character) was 5 before
    the sweep, 0 after. Fixed: two docstrings in interloper-api/tests/routes/test_components.py
    (TestListRelations, TestComponentResponseRelations), the rewritten half of
    interloper-scheduler/src/interloper_scheduler/executor.py's module docstring, and two Args:
    entries in interloper-db/src/interloper_db/store/components.py (create/update, both
    describing the relations parameter) that the sweep instructions did not name but the
    verification command caught. Counting added lines missed one em-dash the branch left in place:
    the executor docstring's first sentence spans a line the rewrite did not touch, so its em-dash
    never showed up in the diff. It is gone now, and the sweep is 0 over the whole branch.

Follow-ups

Both tests deferred during the tasks (the two-rows-on-a-single-relation HydrationError, and a
source with zero connection relations handed a connection_id) are written; nothing is left
deferred. Carried forward for a later PR: relation reads still fan out per destination without
ComponentRelation.dst eager-loaded, bind_siblings takes no row lock (its rows are the ones no
uniqueness index covers), and RelationEdge.param_name keeps a name the relation model no longer
uses, which is an MCP consumer break to time deliberately. Three smaller ones: the catalog wire
repeats each relation's name inside its own value (phase 1 output, to align when phase 4 consumes
it), the agent's "requires a X as Y" error can interpolate a list of keys, and a
resolve_fetch_field call on a non-component path raises AttributeError into a 500.

Added after the phase 4 walk

Two commits landed here after the app walk showed a used destination could be deleted:

  • a8f720c3 fix(core)!: an optional inferred upstream (x: il.Upstream | None) is declared on_delete="detach", so an optional upstream may vanish; resource-class parameters keep block.
  • 99df39de fix(db)!: on_delete alone decides whether a referrer blocks a delete; optional governs unbinding and emptiness only. destinations (optional, on_delete="block") therefore blocks deleting a destination in use, as before the refactor. Tests cover both knobs independently.

By Digitl

…ry write; self-edge and cross-instance guards

By Digitl
…on agent

The toolkit's two write functions were reachable from no surface: the MCP
server is deliberately read-only, and the ADK agent never registered them,
so a relation could only be rebound through the app. They now sit in the
collection agent's tool list beside its other write tools, re-exported from
the toolkit package as its whole write surface, with the read-only guard on
the MCP server's tool list widened to catch them.

The collection instruction and the update_component copy said rebinding a
relation happens in the app; both now point at the tools instead.

By Digitl
…tion duplicates and drift are tested

A source whose class dropped one of its assets kept the child row, and the
hydrator read that child's relations against the vocabulary the narrowed
class resolves: the rows the dropped asset legitimately holds then read as
"relations its class does not declare", failing the whole source with a
misleading HydrationError. The child set is now checked against the
parent's own declaration before any child's relations are read, so the
drift is reported as drift, naming the child and its source.

Two hydration guards gain the tests they lacked: the new drift path, and
two rows under a single-valued relation (hand-inserted, since the store's
own writes repoint such a name instead of accumulating).

Also: the components module docstring named a RelationMixin that does not
exist (the store composes a RelationStore), remove() now states that it
guards no undeclared name on purpose, so drifted rows can be cleared, and
the emptiness guard in _sync_relations says why creation may leave a
non-optional relation unbound while an update may not.

By Digitl
The last deferred test of the branch: a definition declaring only
destinations, handed a connection_id, reports that the connection fits no
relation of the source rather than binding it anywhere.

Sweep of what the earlier tasks left: the executor module docstring's
remaining em-dash (on the one line the rewrite did not touch), a test name
still saying "sibling upstreams" where the store binds sibling relations,
and the migration 017 docstring, which listed neither destination among
the types _PLURAL maps nor the empty-name rows its defensive DELETE drops.

By Digitl
The clause the relation tools were appended to used a pair of em-dashes;
parentheses carry the same aside.

By Digitl
… deleted

Deletion of a relation's target is governed by on_delete alone; optional
governs only whether a relation may be left unbound or empty. An optional
data() upstream parameter already tolerates a missing leg, so the inferred
relation now declares on_delete="detach" and no longer blocks deleting the
upstream through the platform's delete guard.

A parameter naming a component class keeps the default block: a connection
or config a read consumes stays undeletable while it is bound, optional or
not.

By Digitl
The delete guard treated any optional relation as detaching, so a
destination in use (declared optional=True, on_delete="block") could be
deleted from under the sources writing to it, and every optional
consumption relation with it. _relation_detaches now reads on_delete only;
an unresolvable name still blocks, keeping the guard fail-closed.

optional keeps its own job, which unbind still honors: whether a relation
may be left unbound or emptied.

By Digitl
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant