Skip to content

feat: campaign matcher source, the first fan-in over every campaigns asset - #324

Open
aaaaahaaaaa wants to merge 9 commits into
feat/relation-model-platformfrom
feat/campaign-matcher
Open

feat: campaign matcher source, the first fan-in over every campaigns asset#324
aaaaahaaaaa wants to merge 9 commits into
feat/relation-model-platformfrom
feat/campaign-matcher

Conversation

@aaaaahaaaaa

Copy link
Copy Markdown
Contributor

Summary

CampaignMatcher is a new interloper-assets source with one asset, campaign_matches,
declaring relations={"campaigns": il.Relation("asset", "*.campaigns", many=True)}: a many-valued
wildcard relation that binds every campaigns asset a DAG holds, from any source, without either
connector knowing the matcher exists. It is the first asset in the codebase to fan a whole class of
upstreams into one downstream node.

The matching logic itself is a placeholder, documented as such in the module docstring: it
lower-cases and strips each upstream campaign's name into a canonical_name and always reports
similarity: 1.0. It exists only to prove the fan-in mechanics; a real fuzzy-matching strategy is
future work. Since different connectors name the same fields differently (Facebook: id/name;
TikTok: campaign_id/campaign_name), a small helper tries both spellings in turn and falls back
to an empty string rather than filtering a row that matches neither.

The fan-in is declared exactly once, on the asset's relations= kwarg; wiring it into a DAG is
ordinary construction (il.DAG(fb, tt, matcher)), an explicit bind("campaigns", ...), or a
manifest where each connector overrides its campaigns asset to materializable: false and
campaign_matches names both by {ref: ...}. docs/guide/dependencies.md ("Many upstreams" /
"Fan-in across sources") and examples/campaign_matcher.yaml cover both forms.

What was proven

A platform run built the manifest end to end: two connectors (Facebook, TikTok) each held one
campaigns asset, both bound by name into the matcher's campaigns relation with no shared
vocabulary beyond the wildcard key. Both connector assets ran read-only
(materializable: false); only the matcher's own asset actually materialized. The run produced
four rows, one per upstream campaign, each carrying the originating connector's source_key and
source_id. The manifest (examples/campaign_matcher.yaml) builds into a runnable DAG from a
YAML file with no Python.

Fixes found by the run

  • CSV and file destinations now signal a missing scope as DataNotFoundError.
    CSVDestination already raised it; FileDestination._read_scope still raised a bare
    FileNotFoundError, so the "an upstream leg with no data resolves to None" contract
    (asset/base.py, matching on DataNotFoundError) never engaged for a file-backed destination.
    Both now agree.
  • The matcher reads both campaign field spellings. The placeholder matching logic first read
    only id/name; TikTok's campaign_id/campaign_name schema produced empty matches. _first
    now tries both spellings for id and name.
  • Dev seed binds the demo job target by relation name, matching the platform's by-name wiring
    instead of the retired (type, slot) shape, so make dev builds again.

Stack

Stacked on PR #323 (phase 2, platform relations by name). Base branch is
feat/relation-model-platform; retarget as the stack merges. Phase 4 (app) follows, stacked on
this one.

Verification

  • uv run --frozen ruff check: all checks passed.
  • uv run --frozen ty check: all checks passed.
  • uv run --frozen pytest -q: 2761 passed, 0 failed
    (interloper-core 1476, interloper-db 452, interloper-api 388, interloper-google-cloud 123,
    interloper-scheduler 107, interloper-assets 96, interloper-pandas 37, interloper-slack 23,
    interloper-agent 17, interloper-toolkit 16, interloper-mcp 8, interloper-k8s 8,
    interloper-app 6, interloper-docker 4).

Follow-ups

  • DatabaseDestination._read_scope has no existence check at all (unlike CSV and file); it needs a
    driver-level probe rather than a path check, and is left for its own change.
  • The run summary's denominator counts read-only nodes: runner/state.py:160 divides by
    len(self.dag.operations) (every operation, materializable or not), while :325 marks read-only
    operations SKIPPED rather than QUEUED. A fan-in run over two read-only connectors and one
    materializing matcher reports "1/3 succeeded" even when everything that could run did.
  • A CI guard that builds every examples/*.yaml (construct the DAG, don't execute it) would have
    caught the manifest's earlier unquoted ${VAR} YAML error before review.
  • _first's falsy fallback ("" treated the same as absent) is pinned by a test now, but if the
    placeholder matching logic outlives phase 3, it is worth revisiting whether an empty string
    should still count as "no match" the way None does.

By Digitl

@codecov

codecov Bot commented Sep 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

… arrive as None

CSVDestination._read_csv raised a bare FileNotFoundError for a missing
file, so Asset._read_upstreams (which only converts a DataNotFoundError
cause into a None leg) never caught it and the whole asset failed
instead of warning. Raise DataNotFoundError instead, matching
MemoryDestination and the BigQuery/GCS destinations.

By Digitl
…schemas

_match read row["id"]/row["name"], but TikTok's campaigns rows carry
campaign_id/campaign_name instead, so its fanned-in rows came out
blank. Try campaign_id/campaign_name before id/name (first present,
non-empty value) so every connector's campaigns schema matches
correctly.

By Digitl
_ensure_demo_job still passed relations={"target": [(source_id, "")]},
a shape from before ComponentStore.create moved to
dict[str, list[UUID]] keyed by relation name. Job declares "targets"
(plural), not "target", so make dev / dev-reset traceback with
ConfigError on a fresh database. Pass relations={"targets": [source_id]}.

By Digitl
…icate block

docs/guide/specs.md's manifest example left ${GCP_KEY} and ${FB_TOKEN}
unquoted inside YAML flow mappings, so yaml.safe_load choked on the bare
`{` indicator; both are now quoted, matching examples/campaign_matcher.yaml's
block style elsewhere on the page. Verified by extracting the block to a
scratch file, exporting dummy values, and building the DAG from it.

docs/guide/dependencies.md's "Fan-in across sources" section reprinted the
campaign_matches decorator block already shown under "Many upstreams"; it
now refers back to that one copy in prose. It also named an import path
that isn't importable (the campaign_matcher package's __init__ is empty);
the fix points at interloper_assets.CampaignMatcher instead, the source's
actual export.

examples/campaign_matcher.yaml's header now leads with the --dry-run
invocation, mirroring examples/job.yaml, so a reader validates the plan
before firing a real BigQuery job with placeholder credentials.

By Digitl
campaign_matches was the only schema in interloper-assets with no
Field(description=...), so BigQuery would push undocumented columns.
Each of the seven fields now carries a one-sentence description in the
connectors' style.

_first treats any falsy value as absent, not just a missing key; a
comment now states that this is deliberate given the shipped connector
schemas type these fields as str | None, so "" falls through the same
as None. A new test pins the resulting behaviour: a leg whose rows
carry neither the Facebook nor the TikTok spelling still emits a row,
with empty campaign_id and canonical_name rather than being filtered.

By Digitl
FileDestination._read_scope raised a bare FileNotFoundError for a
missing scope, unlike CSVDestination which already raises
DataNotFoundError so a missing upstream leg resolves to None instead
of the run failing outright. Same fix here: the two existing tests
that named the missing path now assert DataNotFoundError.

DatabaseDestination._read_scope has no existence check at all; that
needs a driver-level probe and is left for a follow-up, not folded in
here.

By Digitl
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant