feat: campaign matcher source, the first fan-in over every campaigns asset - #324
Open
aaaaahaaaaa wants to merge 9 commits into
Open
feat: campaign matcher source, the first fan-in over every campaigns asset#324aaaaahaaaaa wants to merge 9 commits into
aaaaahaaaaa wants to merge 9 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…tream over every campaigns asset By Digitl
… arrive as None CSVDestination._read_csv raised a bare FileNotFoundError for a missing file, so Asset._read_upstreams (which only converts a DataNotFoundError cause into a None leg) never caught it and the whole asset failed instead of warning. Raise DataNotFoundError instead, matching MemoryDestination and the BigQuery/GCS destinations. By Digitl
…schemas _match read row["id"]/row["name"], but TikTok's campaigns rows carry campaign_id/campaign_name instead, so its fanned-in rows came out blank. Try campaign_id/campaign_name before id/name (first present, non-empty value) so every connector's campaigns schema matches correctly. By Digitl
_ensure_demo_job still passed relations={"target": [(source_id, "")]},
a shape from before ComponentStore.create moved to
dict[str, list[UUID]] keyed by relation name. Job declares "targets"
(plural), not "target", so make dev / dev-reset traceback with
ConfigError on a fresh database. Pass relations={"targets": [source_id]}.
By Digitl
…icate block
docs/guide/specs.md's manifest example left ${GCP_KEY} and ${FB_TOKEN}
unquoted inside YAML flow mappings, so yaml.safe_load choked on the bare
`{` indicator; both are now quoted, matching examples/campaign_matcher.yaml's
block style elsewhere on the page. Verified by extracting the block to a
scratch file, exporting dummy values, and building the DAG from it.
docs/guide/dependencies.md's "Fan-in across sources" section reprinted the
campaign_matches decorator block already shown under "Many upstreams"; it
now refers back to that one copy in prose. It also named an import path
that isn't importable (the campaign_matcher package's __init__ is empty);
the fix points at interloper_assets.CampaignMatcher instead, the source's
actual export.
examples/campaign_matcher.yaml's header now leads with the --dry-run
invocation, mirroring examples/job.yaml, so a reader validates the plan
before firing a real BigQuery job with placeholder credentials.
By Digitl
campaign_matches was the only schema in interloper-assets with no Field(description=...), so BigQuery would push undocumented columns. Each of the seven fields now carries a one-sentence description in the connectors' style. _first treats any falsy value as absent, not just a missing key; a comment now states that this is deliberate given the shipped connector schemas type these fields as str | None, so "" falls through the same as None. A new test pins the resulting behaviour: a leg whose rows carry neither the Facebook nor the TikTok spelling still emits a row, with empty campaign_id and canonical_name rather than being filtered. By Digitl
FileDestination._read_scope raised a bare FileNotFoundError for a missing scope, unlike CSVDestination which already raises DataNotFoundError so a missing upstream leg resolves to None instead of the run failing outright. Same fix here: the two existing tests that named the missing path now assert DataNotFoundError. DatabaseDestination._read_scope has no existence check at all; that needs a driver-level probe and is left for a follow-up, not folded in here. By Digitl
aaaaahaaaaa
force-pushed
the
feat/campaign-matcher
branch
from
September 8, 2026 05:48
1c56d0e to
38326f2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CampaignMatcheris a newinterloper-assetssource with one asset,campaign_matches,declaring
relations={"campaigns": il.Relation("asset", "*.campaigns", many=True)}: a many-valuedwildcard relation that binds every
campaignsasset a DAG holds, from any source, without eitherconnector knowing the matcher exists. It is the first asset in the codebase to fan a whole class of
upstreams into one downstream node.
The matching logic itself is a placeholder, documented as such in the module docstring: it
lower-cases and strips each upstream campaign's name into a
canonical_nameand always reportssimilarity: 1.0. It exists only to prove the fan-in mechanics; a real fuzzy-matching strategy isfuture work. Since different connectors name the same fields differently (Facebook:
id/name;TikTok:
campaign_id/campaign_name), a small helper tries both spellings in turn and falls backto an empty string rather than filtering a row that matches neither.
The fan-in is declared exactly once, on the asset's
relations=kwarg; wiring it into a DAG isordinary construction (
il.DAG(fb, tt, matcher)), an explicitbind("campaigns", ...), or amanifest where each connector overrides its
campaignsasset tomaterializable: falseandcampaign_matchesnames both by{ref: ...}.docs/guide/dependencies.md("Many upstreams" /"Fan-in across sources") and
examples/campaign_matcher.yamlcover both forms.What was proven
A platform run built the manifest end to end: two connectors (Facebook, TikTok) each held one
campaignsasset, both bound by name into the matcher'scampaignsrelation with no sharedvocabulary beyond the wildcard key. Both connector assets ran read-only
(
materializable: false); only the matcher's own asset actually materialized. The run producedfour rows, one per upstream campaign, each carrying the originating connector's
source_keyandsource_id. The manifest (examples/campaign_matcher.yaml) builds into a runnableDAGfrom aYAML file with no Python.
Fixes found by the run
DataNotFoundError.CSVDestinationalready raised it;FileDestination._read_scopestill raised a bareFileNotFoundError, so the "an upstream leg with no data resolves toNone" contract(
asset/base.py, matching onDataNotFoundError) never engaged for a file-backed destination.Both now agree.
only
id/name; TikTok'scampaign_id/campaign_nameschema produced empty matches._firstnow tries both spellings for id and name.
instead of the retired
(type, slot)shape, somake devbuilds again.Stack
Stacked on PR #323 (phase 2, platform relations by name). Base branch is
feat/relation-model-platform; retarget as the stack merges. Phase 4 (app) follows, stacked onthis one.
Verification
uv run --frozen ruff check: all checks passed.uv run --frozen ty check: all checks passed.uv run --frozen pytest -q: 2761 passed, 0 failed(interloper-core 1476, interloper-db 452, interloper-api 388, interloper-google-cloud 123,
interloper-scheduler 107, interloper-assets 96, interloper-pandas 37, interloper-slack 23,
interloper-agent 17, interloper-toolkit 16, interloper-mcp 8, interloper-k8s 8,
interloper-app 6, interloper-docker 4).
Follow-ups
DatabaseDestination._read_scopehas no existence check at all (unlike CSV and file); it needs adriver-level probe rather than a path check, and is left for its own change.
runner/state.py:160divides bylen(self.dag.operations)(every operation, materializable or not), while:325marks read-onlyoperations
SKIPPEDrather thanQUEUED. A fan-in run over two read-only connectors and onematerializing matcher reports "1/3 succeeded" even when everything that could run did.
examples/*.yaml(construct the DAG, don't execute it) would havecaught the manifest's earlier unquoted
${VAR}YAML error before review._first's falsy fallback (""treated the same as absent) is pinned by a test now, but if theplaceholder matching logic outlives phase 3, it is worth revisiting whether an empty string
should still count as "no match" the way
Nonedoes.By Digitl