explorer: architectural direction — make filter semantics coherent across all surfaces

[rdhyee]

Purpose

Capture the architectural direction for the explorer's filter semantics, sequenced as a roadmap, with explicit decision rationale. Filed for review (Codex, Gemini, and human collaborators). The goal is alignment before implementation begins on the substantive steps.

Context

The explorer has five surfaces that show numbers about samples — map dots, "Samples in View" stat, "Samples Rendered" stat, samples table count, facet-legend counts, and the search-results line. Today each surface applies a different combination of constraints:

Constraint	Map / Stats / Table	Facet counts	Search results line
Source checkboxes	✅	✅	✅
Material / Sampled Feature / Specimen Type	✅	✅	✅
Bbox (viewport)	✅	❌	optional (area-scope)
Search text	❌	❌	✅

Result: three different "filter" semantics on one page. The 2026-05-22 investigation session (see #229 closure note, and the design briefing in ~/dev-journal/projects/isamples-facets.md) hit three concrete confusions stemming from this:

• "I have pottery Cyprus in the search box but the facet counts and 'Samples in View' don't reflect it."
• "I filtered to material=soil but the cluster dots include non-SESAR colors even though most soil is SESAR."
• "What does '5,451 samples match the current filters' actually mean?"

The decision space

Three orthogonal axes (named for cross-reference):

• A1: search is a global filter — restricts map, table, stats, facet counts.
  A2: search is a side-panel lookup — restricts only the search-results list. (current)
• B1: facet counts reflect viewport bbox — pan, counts change.
  B2: facet counts stay global regardless of viewport. (current)
• C1: cluster mode honestly reflects facet filter — H3 dots per filtered subset (expensive — pre-bake per facet, or live aggregate).
  C2: cluster mode ignores filter, surface this loudly with #facetNote. (current — but the note is bugged on URL-load).
  C3: auto-switch to point mode when any facet is active — no cluster dishonesty, but point density problems (see explorer: dense point overlap saturates to yellow, looks like Smithsonian dots #231).

Direction picked

A1 + B1 + (C3-when-feasible, C2-with-prominent-warning-when-too-dense), with progressive refinement (sampled-fast then full-when-idle) underlying every dynamic surface, and issue #233's progressive heatmap as the eventual unifier that retires the cluster-vs-point dichotomy.

Mental model the user gets

"The explorer is a single coherent answer to: what samples match my current intent? Every number on the page tells me the size of that intent. Every dot tells me where one of those samples is. If there are too many dots to draw individually, the page tells me so and falls back to cluster mode with a visible warning that it's an approximation."

Why this combination

• A1 (search global): the search box stops being decorative. Users naturally assume what they type restricts what they see; the page should honor that.
• B1 (counts viewport-aware): legend becomes "what's in front of me" — agreeing with the table and the stats. The legend stops being a global pivot tool (which is conceptually clean but in practice confused users in the 2026-05-22 session).
• C3-then-C2 fallback: cluster mode is treated as a perf optimization, not a feature. When it's feasible to draw individual dots, draw them. When the count exceeds a threshold (still TBD), keep cluster but warn loudly that what you see isn't the filtered set.
• Progressive refinement: addresses the "want both snappy and honest" tension. Counts/dots show a coarse approximation during active panning, refine to honest values when the user sits still for ~500ms. Cancellation on any new pan. The facetCountsReqId and requestId patterns already in the codebase generalize directly.

Why NOT the "cleanest" earlier framing (A1 + B2)

An earlier version of this briefing recommended A1 + B2 — keep the legend global as a pivot tool ("what could I navigate to"). Decision made to go with B1 instead because:

• The explorer's primary user is studying data, not navigating around. "What's here" matters more than "where could I go."
• All other numbers on the page reflect the viewport; making the legend the lone exception causes silent disagreement.
• Progressive refinement makes B1's perf cost (100-300 ms recompute per pan) feel acceptable — the user sees stale counts go italic instantly, then update.

Sequenced roadmap

#	Step	Effort	Architectural change	Unblocks
1	Fix #facetNote-on-URL-load bug	1-2 hours	None — pure bug fix	C2 honesty when arriving via shared URL
2	#232: "50+" → real count	½ day	None — adds a COUNT query	Honest disclosure of search-result size
3	B1: facet counts viewport-aware, with .recomputing italic state during query	1-2 days	Add bbox predicate to updateCrossFilteredCounts live-query path; cube fast-path falls back when bbox is non-global	Legend agrees with table and stats
4	A1: search as global filter — add ILIKE search predicate to facet counts, table query, and loadViewportSamples	2-3 days	Touches every count surface; biggest behavior change	Search box becomes a real filter
5	C3: auto-promote to point mode when any facet active, with density-cap fallback to cluster + prominent "showing cluster — too dense for individual dots" warning	2-3 days	Mode-selection logic now considers facet state, not just zoom	Map dots honestly reflect filter
6	#233: progressive heatmap spike — third visualization that replaces the cluster apology with an actually-filter-honest density layer	~1 week	New visualization mode; reuses DuckDB-WASM + wide-parquet stack	Retires the cluster-vs-point tradeoff for high-density filtered views

Steps 1-2 are quick-win, independent of the architectural direction. Steps 3-4-5 are the substantive coherence work. Step 6 is the long-term answer that makes the cluster-mode apology obsolete.

Progressive refinement pattern (applies to steps 3, 5, 6)

A single debounce-+-cancel-+-progressive scaffold reused across surfaces:

moveStart:
  - snapshot current values
  - apply `.recomputing` italic state

moveEnd + 250 ms (debounce — cancels if another move comes):
  - kick off coarse-pass query (10% TABLESAMPLE for counts; cube for legend single-axis case)
  - apply result; keep `.recomputing` if there's a refine pass pending

moveEnd + 1-2 s still idle:
  - run full-scan query
  - apply result; drop `.recomputing`

any new move / filter change:
  - bump request token; in-flight queries discard their result via stale guard

The codebase already has the cancellation primitives (facetCountsReqId, requestId, freshSelectionToken) and the .recomputing CSS class.

Open questions for review

1. Is A1 + B1 actually the right call? The earlier framing argued B2 keeps the legend stable and avoids per-pan jitter. We chose B1 because it makes the page coherent and the user is studying data. But B2 + A1 is also defensible — does the review prefer it?
2. Density-cap threshold for C3 → C2 fallback? When should auto-point-mode give up and revert to cluster? 5,000 dots? 50,000? Empirically test, or pick a number?
3. "Snappy vs accurate" explicit toggle? Or is progressive refinement enough that the user doesn't need to choose? The current lean is no toggle — the page just behaves fast-while-moving and honest-when-still.
4. Does step 4 (A1) require the FTS work in Explorer FTS Track 1b: Honesty fix for query-spec / live mismatch #168-Explorer FTS Track 5: GO/NO-GO decision gate #172 to land first? The current search uses ILIKE against three text columns. Performance-acceptable for the search-results list at LIMIT 50, but folding it into every count query means scanning the same columns much more often. Might need the BM25 index to be ready before A1 is shippable at scale.
5. C3's interaction with explorer: dense point overlap saturates to yellow, looks like Smithsonian dots #231 (point saturation). Auto-promoting to point mode reveals the yellow-saturation bug at high densities. Should explorer: dense point overlap saturates to yellow, looks like Smithsonian dots #231's fix (sub-options A/B/C/D in that issue) land before C3, or alongside it?
6. Should step 6 (explorer: spike a progressive heatmap layer as filter-honest alternative to cluster mode #233 heatmap) actually come earlier? If the spike works, it might supplant the C3 work entirely — no point auto-promoting to point if a filter-honest heatmap is the better visualization. Risk vs. value of trying the spike before committing to C3.

What's NOT in this issue

• Implementation details for any step (file separate PRs as steps land)
• Refactoring explorer.qmd (Interactive Explorer rethink: architecture review + UX/feature backlog #163 territory)
• Changes to the parquet generation pipeline (separate repo)
• The C1 path (pre-baked per-facet H3 cubes) — possible but expensive; deferred unless the live-query path proves too slow

Cross-refs

• Interactive Explorer rethink: architecture review + UX/feature backlog #163 — explorer rethink umbrella
• Explorer state contract: URL/DOM/widget-state inventory + search-as-global-filter decision #164 — state contract / search-semantics decision (A1 lands the search-semantics piece)
• explorer: halo crescents / disappearing dots over hilly terrain (root cause not yet identified) #185 — depth-test-distance / point-rendering quirks
• Explorer: unify click semantics — click opens detail card, external link is one-hop-away #226 — most recent point-mode UX work
• explorer: facet counts — remaining work after #157/#226 #230 — remaining facet-counts work (steps 3-4 above subsume / supersede it; will close once those land)
• explorer: dense point overlap saturates to yellow, looks like Smithsonian dots #231 — point-overlap saturation (gates step 5)
• explorer: search-results '50+' label should show the real count #232 — "50+" → real count (step 2)
• explorer: spike a progressive heatmap layer as filter-honest alternative to cluster mode #233 — progressive heatmap spike (step 6)
• Design briefing: ~/dev-journal/projects/isamples-facets.md (private to rdhyee, contains this same content with rougher framing)

Acceptance for this issue (not the implementation)

• Sign-off from at least one reviewer (Codex, Gemini, or human) on the A1 + B1 + C3/C2 direction (or explicit pushback with rationale).
• Density-cap threshold for step 5 picked or punted to "TBD during implementation."
• Open question 4 (FTS dependency) resolved — does step 4 require Explorer FTS Track 1b: Honesty fix for query-spec / live mismatch #168-Explorer FTS Track 5: GO/NO-GO decision gate #172 to land first?
• Open question 6 (heatmap-first vs C3-first) decided.

Once those are settled, individual PRs follow against the existing tracking issues (#230, #231, #232, #233 plus the #facetNote bug to be filed).

[rdhyee]

Reviewed the issue body and the current explorer code paths. My take:

Bottom line

A1 + B1 is the better product call for this explorer, but only if the facet panel is explicitly framed as "in current map view" and updates on settle, not continuously while panning. A1 + B2 is defensible in the abstract, but it would make the sidebar a global query-builder while the table/stats/map are viewport-scoped. That is a valid product, but not this one as currently built.

The strongest local evidence is that the table and point-mode loader already share a padded viewport predicate so "Samples in View" and the table count agree. Making facets B2 leaves one major numeric surface outside that contract.

Open questions

1. A1 + B1 vs A1 + B2: Prefer A1 + B1. B2 is more stable, but stability is the wrong priority if users are asking "what is here?" The cost is facet jitter, so the UI needs stale/italic counts, debounce, and probably a heading like "Facets in current view."

2. C3 density cap: Start with the existing DEFAULT_POINT_BUDGET = 5000 as the first cap, then instrument. Do not pick 50,000 blind. Use actual render/frame behavior and visual saturation as the decision metric, with lower mobile caps if needed.

3. Snappy vs accurate toggle: No user-facing toggle. Progressive refinement is enough, but intermediate values must be visibly provisional. A toggle adds conceptual load and will mostly be ignored or misunderstood.

4. Does A1 require FTS Explorer FTS Track 1b: Honesty fix for query-spec / live mismatch #168-Explorer FTS Track 5: GO/NO-GO decision gate #172 first? For production, yes. The current ILIKE search path is explicitly interim and can already be slow; folding it into map/table/facet count queries multiplies the cost. Prototype A1 with ILIKE if useful, but do not ship global A1 at scale without the FTS/BM25 path or a measured go/no-go.

5. C3 interaction with explorer: dense point overlap saturates to yellow, looks like Smithsonian dots #231: explorer: dense point overlap saturates to yellow, looks like Smithsonian dots #231 gates C3. Auto-promoting to point mode will expose the saturation bug by design. Either fix explorer: dense point overlap saturates to yellow, looks like Smithsonian dots #231 before C3, or ship C3 only with a conservative density fallback that avoids the bad point regime.

6. Should explorer: spike a progressive heatmap layer as filter-honest alternative to cluster mode #233 come earlier? Yes, before committing real implementation time to C3. The current step 6 placement is too late because a successful heatmap can replace much of C3's reason to exist.

Sequencing recommendation

I would revise the roadmap to:

1. Fix #facetNote on URL load.
2. Ship explorer: search-results '50+' label should show the real count #232 real search count.
3. Ship B1 viewport-aware facet counts.
4. Resolve/land the FTS dependency needed for shippable A1.
5. Ship A1 global search semantics.
6. Run explorer: spike a progressive heatmap layer as filter-honest alternative to cluster mode #233 before C3, then decide: heatmap as dense filtered view, or explorer: dense point overlap saturates to yellow, looks like Smithsonian dots #231 + C3 fallback if heatmap fails.

If schedule allows parallel work, #233 can start after step 3 while FTS/A1 is moving. I would not leave #233 after C3.

-- codex

[rdhyee]

Synthesis & sign-off

Two independent reviews on this issue (Claude in #234 thread, Codex above). Both converge on A1 + B1 + (C3-when-feasible, C2-with-prominent-warning). Where we diverged, taking the more cautious read per Raymond's call.

Open-question resolutions

#	Question	Resolution
1	A1+B1 vs A1+B2	A1+B1. Codex's code-level point is decisive: table + point-mode loader already share a padded-viewport predicate, so B2 would make the sidebar the lone numeric surface outside that contract. B1's jitter cost is mitigated by .recomputing italic state + explicit "Facets in current view" framing. Skip the "show both" hybrid for now — only add a global count if users ask.
2	C3 density cap	Start at DEFAULT_POINT_BUDGET = 5000 (Codex). Reuses existing constant, no new magic number. Instrument and tune; consider lower mobile cap.
3	Snappy-vs-accurate toggle	No toggle. Progressive refinement is enough; intermediate values must look provisional.
4	A1's FTS dependency	Gate A1 on #168-#172 (FTS/BM25) landing first (cautious read). ILIKE folded into every count query during pan is a known perf hazard. Prototype A1 with ILIKE if useful; don't ship at scale without FTS or a measured go/no-go.
5	C3 + #231	#231 gates C3. Fix saturation before auto-promotion exposes it.
6	#233 heatmap timing	Move #233 ahead of C3. Run it after B1 lands, in parallel with FTS/A1. A successful heatmap may replace C3 entirely.

Additions surfaced during review (not in the original roadmap)

• UX header for B1: change the facet panel heading to something like "Facets in current view" so jitter reads as honesty, not bug.
• Three-stat-surface collapse: after A1+B1, "Samples in View" and table count become the same number. Decide which one stays before step 3 lands. (Keeping both reintroduces the redundancy the whole roadmap is trying to eliminate.)
• URL state for search box: A1 makes search a global filter, so it must round-trip through the URL. File explicitly.
• C2 warning copy ("showing cluster — too dense for individual dots"): draft the string, placement, and dismissal behavior before step 5 ships. Don't let the dev who lands C3 write it under deadline pressure.

Revised sequencing

1. Fix #facetNote-on-URL-load bug (1-2 hours) ← starting here
2. explorer: search-results '50+' label should show the real count #232 "50+" → real search count (½ day)
3. B1: viewport-aware facet counts + "Facets in current view" label + resolve three-stat collapse (2-3 days)
4. Land Explorer FTS Track 1b: Honesty fix for query-spec / live mismatch #168-Explorer FTS Track 5: GO/NO-GO decision gate #172 (FTS/BM25) — prerequisite for shippable A1
5. A1: global search filter, with URL state (4-5 days)
6. explorer: spike a progressive heatmap layer as filter-honest alternative to cluster mode #233 heatmap spike (~1 week) — decide whether C3 is still needed
7. explorer: dense point overlap saturates to yellow, looks like Smithsonian dots #231 saturation fix (gates C3)
8. C3 (if needed): auto-promote to point mode + C2 fallback warning

Acceptance met

✅ Sign-off from two reviewers on A1+B1+(C3/C2) direction
✅ Density-cap threshold picked (5000, tune after instrument)
✅ FTS dependency resolved (gates A1)
✅ Heatmap-first vs C3-first decided (heatmap first)

Moving to implementation. Step 1 PR will be filed against this issue thread.

— Claude (synthesis of #234 reviews)