Skip to content

Replace the search index with Pagefind - #3371

Open
enf0rc3 wants to merge 9 commits into
mainfrom
willlaugesen/docs-search-pagefind
Open

Replace the search index with Pagefind#3371
enf0rc3 wants to merge 9 commits into
mainfrom
willlaugesen/docs-search-pagefind

Conversation

@enf0rc3

@enf0rc3 enf0rc3 commented Aug 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Replaces the hand-rolled search.json index with Pagefind, behind the SearchEngine seam the overlay already talks to. Pagefind indexes the HTML the build emits, so article body text is searchable for the first time — previously only titles, headings and extracted keywords were.

Build

src/integrations/pagefind-index.ts runs on astro:build:done and indexes dist/docs/**/*.html into dist/docs/pagefind/.

  • Registered before pruneDist() in astro.config.mjs, which deletes every top-level entry outside its allowlist.
  • Indexes from dist/docs, so stored URLs are relative to the /docs/ prefix the site is proxied under.
  • includeCharacters: '.#{}<>+$_' keeps technical punctuation in the index, so <head>, .nupkg and #{Octopus.Environment.Name} are findable as written. Pagefind strips punctuation from both index and query by default, and <head> matched nothing at all.
  • Logs how many pages were indexed and warns below 1,000. A broken body scope is a silent failure otherwise: search still works, it is just full of the wrong pages.

Pagefind's Node API is imported statically, because astro:build:done fires after Vite's module runner has closed and a dynamic import from inside the hook cannot resolve.

What gets indexed

Default.astro decides:

  • data-pagefind-body on the <article>, which keeps sidebar and footer text out of every page's terms and excludes the 1,409 redirect stubs for free, since they never reach this layout.
  • data-pagefind-filter supplies the facet tabs, matching classify() in search-engine.ts.
  • data-pagefind-meta carries the breadcrumb the result row prints.
  • data-pagefind-sort carries a date key, so a recency sort is available in the index without a rebuild. Nothing in the overlay offers that control yet.
  • data-pagefind-ignore="all" on the page-action and markdown-link blocks, which are identical on every page.
  • ArticleHeader.astro takes a searchWeight prop, which lifts an h1 to 10 on pages three segments deep or shallower.

src/plugins/pagefind-image-attrs.js puts data-pagefind-index-attrs on every <img> carrying alt or title text. A screenshot's alt text is often the only place a UI label appears in prose. Image.astro carries the attribute itself, because raw <img> tags are passed through as HTML and never become hast nodes.

Query side

src/scripts/search-engine-pagefind.ts:

  • metaWeights: { title: 8, trail: 0.5 }. The breadcrumb is derived from the URL, so every page under /docs/projects/ matches a search for "projects" as well as the Projects page does. Demoting it is worth seven points of Success@5.
  • A reorder that puts a page the query names — by title or last URL segment — ahead of the pages inside it, with a per-segment depth discount behind that. data-pagefind-weight alone measured as a no-op; this is worth about twenty points of Success@5.
  • A score floor of 8 on the top result, so a query with no answer returns empty. Pagefind ranks whatever shares a few letters and returns it as confidently as a real hit. Calibrated against every term in the search log: the weakest genuine query (cli) scores 9.5, keyboard mash 6.0. It suppresses 17 of the 33 terms with no right answer and costs none of the 57 that have one.
  • Up to two matched headings under each of the leading three results, each its own role="option", so a reader can arrow onto a section and land there.
  • excerptLength: 20, sized to the single line the result row gives it.
  • highlightParam: 'highlight' appends the query to result URLs. Nothing reads it yet.
  • Facet counts come from the unfiltered match set, so the tab strip keeps showing what the other tabs hold while one is selected.

SearchEngine gains eager and preload. Pagefind sets eager: true — 118KB of runtime and WASM, warmed on page load — and preloads the chunks a query needs on each keystroke, ahead of the debounce.

Overlay

docs-search.ts:

  • Three character minimum query length, applied in the debounce and in the ?q= deep-link path. One and two character prefixes match a large share of the index, which for a chunked engine means fetching a large share of it: 975ms and 837ms, against 342ms at three characters.
  • A role="status" live region reports the result count and the empty state. The combobox roles said a listbox existed and which row was active; nothing said what came back. It sits outside the search body, which is hidden until there is a query — a region placed inside would leave and re-enter the accessibility tree on every open and drop the first announcement.

Removed

search.json.ts, search-engine-legacy.ts, the Porter stemmer, the string helpers and the synonym map: 645 lines. Drops the html-to-text and keyword-extractor dependencies, adds pagefind as a dev dependency.

Tests

tests/api-page.spec.ts read /docs/search.json to check the under-construction API reference stays out of search. It now asks through the overlay, searching accounts — a word naming pages both inside and outside the reference, so an empty list cannot pass the test by accident.

Notes

  • Azure Blob and Front Door serve .pf_fragment, .pf_index, .pf_meta and .pf_filter correctly. Pagefind reads them as bytes, so the empty Content-Type is fine, and there is no .wasm file to configure — the wasm ships as wasm.en.pagefind.
  • Pagefind's options() merges rather than replaces, so anything recalibrating the score floor has to start from a clean page or every score comes back depressed.

Known gaps

  • Thirty results, with no paging and nothing saying the list is capped.
  • Cold first search is slower than the old engine's. Warming on page load covers most of it.
  • The Integrations facet has no content: all 153 pages under /docs/api-and-integration/ are redirect stubs. The tab hides itself, so the mapping needs a real home or the tab should go.
  • Plausible.astro listens for a searched event that nothing fires, so there is still no search analytics.

@enf0rc3 enf0rc3 mentioned this pull request Aug 17, 2026
Closed
@enf0rc3
enf0rc3 force-pushed the willlaugesen/docs-search-pagefind branch from 5ac3f67 to 6cbd53f Compare August 17, 2026 21:23
@enf0rc3
enf0rc3 force-pushed the willlaugesen/docs-search-pagefind branch 3 times, most recently from 0d065ca to 1c011db Compare August 17, 2026 22:38
@team-marketing-branch-protections

team-marketing-branch-protections commented Aug 17, 2026

Copy link
Copy Markdown

Pull request environment is available at https://stoctodocspr3371.z22.web.core.windows.net.

You can view the ephemeral environment status in Octopus Deploy.

This environment will be automatically deprovisioned when the pull request is closed, or after 7 days of inactivity.

@enf0rc3

enf0rc3 commented Aug 17, 2026

Copy link
Copy Markdown
Contributor Author

Verified on the ephemeral environment

https://stoctodocspr3371.z22.web.core.windows.net/docs — this closes the one open risk from the plan.

Every Pagefind file type serves. pagefind.js as application/javascript, pagefind-entry.json as application/json, the wasm as application/x-gzip (Azure content-sniffs it, which is harmless because Pagefind reads it as bytes rather than streaming-compiling). The index chunks and fragments load too. Nothing was skipped by the upload step and no MIME allowlist was needed.

Search is correct on real hosting. tentacle → 248 results, All 248 / Docs 183 / API 14 / CLI 51 — identical to local. Deployed pagefind-entry.json reports page_count: 1251, matching the build. Cold first search 2.31s over the internet.

Result links stay on the host. Zero absolute hrefs; clicking a result landed on stoctodocspr3371.../docs/octopus-rest-api/tentacle.exe-command-line/.

One thing to know about the hosting

Azure Blob static website does no compression — Accept-Encoding: gzip returns everything with no Content-Encoding. It barely matters here, because Pagefind's .pf_* chunks are already compressed binary and pagefind.js is 45 KB. It matters a great deal for #3372, which pays 11.4 MB uncompressed rather than the 2.4 MB I estimated. Details in that PR.

Bake-off result

Legacy Pagefind Orama
First search, over the wire 1,860 KB 278 KB 11,989 KB
Each later search 0 KB 40–160 KB 0 KB
Cold on staging — 2.31s 2.16s
At 1.6 Mbps (arithmetic) — ~1.4s ~60s
Typo tolerance no no no (did not work)
Hosting risk — disproven none

Both feel the same on a fast connection. The difference is what happens below that, and payload was the defect this work set out to fix.

Recommend merging this and closing #3372.

@enf0rc3
enf0rc3 force-pushed the willlaugesen/docs-search-pagefind branch 4 times, most recently from ca3d278 to d9d2279 Compare August 18, 2026 05:37
enf0rc3 and others added 3 commits August 19, 2026 08:55
@enf0rc3 @claude
Spike: Pagefind
e2d5177 
Indexes the built HTML at the end of the build and puts Pagefind behind the
SearchEngine seam. Body text is indexed for the first time, so a phrase that
appears in an article but not its title or headings is now findable.

The index goes to dist/docs/pagefind, before pruneDist would delete it, and is
built from dist/docs so stored URLs match the /docs/ prefix the client sets as
its basePath.

data-pagefind-body on the article is what scopes indexing and, as a side effect,
excludes all 1,409 redirect stubs: they render through a minimal layout that has
no article at all. 1,250 pages are indexed, matching the eligible count the
markdown emitter reports.

Removes the search.json endpoint and the client scoring it fed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@enf0rc3 @claude
Add pagefind to the project dictionary
6c634d2 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@enf0rc3 @claude
Preload while typing, warm on page load, and set the excerpt length
49885ce 
Three configuration corrections from the Pagefind audit. Its indexing and
ranking were already idiomatic, so none of this touches either.

preload() now runs on each keystroke ahead of the debounced search, which is
what Pagefind documents as the way to fetch the chunks a query needs while the
reader is still typing.

Warming moves to page load for Pagefind only. That is 118KB of runtime and WASM
against a cold first result measured at 2239ms. Which side of that trade is
right differs by engine, so `SearchEngine` carries an `eager` flag and the
engine states its own answer rather than the overlay assuming one — Orama pays
for its whole index on every navigation and must stay lazy.

excerptLength drops from its 30-word default to 20, to fit the single line the
result row gives it.

Sub-results are deliberately not used. They render heading-scoped matches with
their own anchors, which is real, but the corpus fights it: the CLI tree has 511
headings across only 49 distinct titles, 213 of them "Learn more" and 176
"Examples", and the API tree 291 across 100. Only documentation pages are
distinctive, at 4160 titles across 5302 headings. Suppressing the boilerplate
needs a threshold nobody has justified yet, and the audit is clear that
sub-results move neither Success@1 nor Success@5 — so this stays out of a
comparison it cannot decide, and can be added later against whichever engine
wins.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@enf0rc3
enf0rc3 force-pushed the willlaugesen/docs-search-pagefind branch from d9d2279 to 49885ce Compare August 18, 2026 20:59
enf0rc3 and others added 6 commits August 19, 2026 09:55
@enf0rc3 @claude
Use the rest of what Pagefind offers
91bc6e9 
Six capabilities that were available and unused.

Indexing. A depth-derived data-pagefind-weight on the page header, so a bare
section name ranks the section above the pages inside it — the largest single
source of missed traffic on the search terms readers actually type. Image alt text
becomes searchable via data-pagefind-index-attrs, which has no inheritance and so
needs a hast plugin to reach every markdown image. A date sort key is indexed, and
a frontmatter title is registered as fallback metadata for any page whose heading
comes back empty.

Querying. highlightParam so result links carry the query for the destination page
to highlight, and sub_results rendered as their own options beneath their page, so
a match inside a long page can be arrowed onto and lands on that heading.

The weight is the one worth measuring: applied as a client-side reorder it took
Pagefind from 42% to 62% weighted Success@5 on real search terms, and from 73% to
95% on top-visited pages. At index time it has no top-30 reach limit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@enf0rc3 @claude
Reorder Pagefind's results, and stop sections crowding them out
f2308fa 
Built and measured locally, which corrected two things.

data-pagefind-weight on its own did nothing. Scaling it with depth scored 42%
weighted Success@5 on real search terms against the deployed 42%, because plus or
minus a point around Pagefind's default h1 weight of 7 is far too weak a lever.
It now boosts shallow pages to 10 and leaves deep pages at the default, and the
reordering that does the work lives in the engine: the same name-then-depth rule
the Orama worker uses, with the same constant.

Three matched headings across thirty results made two thirds of the list headings,
which buried the pages a reader is choosing between. Only the leading three
results carry them now, two each — 15% of rows rather than 67%.

Real search terms go 42% to 62% Success@5 and 19% to 43% at rank one.
Top-visited pages go 73% to 95% and 37% to 82%.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@enf0rc3 @claude
Index technical punctuation, weight the metadata, ignore properly
02722c7 
Three gaps, all measured on a local build.

Pagefind strips punctuation from index and query alike, so `<head>` matched nothing
and `Octopus.Action.Package` was indistinguishable from prose using those words.
includeCharacters keeps `.#{}<>+$_`, chosen from what appears in this corpus and in
the search logs. Indexing is additive — both `head` and `<head>` are stored — so
ordinary searches are untouched and the index grew 1%, 8.8MB to 8.9MB. `<head>`
went from 0 results to 55, and `octopus.server.exe` now leads with the
octopus.server.exe reference page.

metaWeights sets title to 8 and trail to 0.5. The breadcrumb is derived from the
URL, so every page under /docs/projects/ carries "Projects" and matched a search
for it as well as the Projects page did. It belongs in the result row, not the
ranking.

Both ignore blocks move to `all`. Bare ignore still lets a title or metadata be
read out of the block, and neither holds anything worth reading.

Real search terms: 62% to 81% Success@5, 43% to 52% at rank one.
Top-visited pages: 95% to 98%, and 82% to 89%.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@enf0rc3 @claude
Ask the overlay whether the API reference is findable
b1685e1 
The test read /docs/search.json, which this branch deletes along with the rest of
the legacy engine, so it was parsing a 404 page as JSON.

It now drives the search overlay instead. The two spikes ship indexes of different
shapes — one JSON document, one directory of compressed chunks — and neither is
readable the way the old one was, but what has to hold is the same either way: a
reader searching a word the API reference is full of must not be sent into it.

`accounts` names pages both inside the API reference and outside it, so the result
list is never empty. A query that matched nothing would pass without proving
anything.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@enf0rc3 @claude
Stop searching on one and two characters, and on nothing at all
39539d2 
Two changes, both measured on a local build.

Searches now need three characters. Almost the whole of Pagefind's keystroke lag
lives in the two keystrokes nobody could act on anyway: one character costs 975ms
and two cost 837ms, against 342ms at three and under 200ms beyond. On a desktop
this takes p95 from 927ms to 347ms, and the median from 346ms to 200ms.

A search whose best match scores under 8 now returns nothing. Pagefind has no
notion of a query it cannot answer — asked for `sssieddqxsx`, the most-typed term
in the log, it offered three security articles. Calibrated against every logged
term: the weakest genuine query scores 9.5, the mash 6.0. The floor suppresses 17
of the 33 terms with no right answer and costs none of the 57 that have one.

Relevance is unchanged either way: 81% Success@5 on real terms and 98% on
top-visited pages, the same as before. Nonsense returning nothing went from 0% to
76%.

The floor was first set at 6, which did nothing, because it was calibrated while
earlier sweep parameters were still active — `options()` merges rather than
replaces, and every score came back depressed. calibrate-score-floor.mjs starts
from a clean page for that reason.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@enf0rc3 @claude
Tell a screen reader what the search found
1993ecf 
The overlay carried the combobox roles without the one part that speaks. A
screen reader was told a listbox existed and which row was active, and never
how many results arrived or that none had.

The live region sits outside the body, which is hidden until there is a query.
Inside it, the region would leave and re-enter the accessibility tree on every
open, and a region that has only just appeared does not reliably announce the
first thing written to it. It is clipped rather than hidden for the same
reason: `display: none` would silence it.

It speaks from `render`, which runs once per settled query, so it keeps pace
with the results rather than with the keyboard. Below the three character
minimum it says nothing, matching the panel's own waiting state.

Pagefind's own Component UI does this and ours did not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@enf0rc3 enf0rc3 changed the title Spike: Pagefind Replace the search index with Pagefind Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@steve-fenton-octopus steve-fenton-octopus Awaiting requested review from steve-fenton-octopus steve-fenton-octopus is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@enf0rc3 @team-marketing-branch-protections