Make Search take a URL, the web, and news - #23
Merged
Conversation
The Search tab could only do full-text over indexed transcripts, which is the
narrowest of the three things someone arrives at that box wanting to do. Now
one input works out what you meant:
- a URL -> fetch that page and describe it
- a phrase -> transcripts and the open web together (Auto), or one of
transcripts / web / news on demand
Pasting a link gives back title, publisher and source tier, author, date, the
phrases the page keeps using, the tickers it names (linked to a stored report
where one exists) and any feeds it advertises. Retrieval reuses the news
pipeline's fetcher unchanged, so a pasted URL obeys the same rules as an
ingested one: robots.txt honoured, blocking publishers reported as blocked,
no body text invented.
Searching gives back the things a list of links does not tell you: Google's
related searches, the phrases recurring across the titles that came back, and
the niches the results cluster into. All clickable, so following a thread
never means retyping it.
Phrases and niches score by *document frequency* — how many separate results
contain a phrase, not how often it occurs — so one long article repeating its
own keyword twelve times cannot invent a trend. Where two phrases cover the
same documents the longer one wins, so the chip row is not the same idea at
three lengths. Deterministic and free: no LLM.
Two costs shaped the design.
Web/news search is a ValueSERP credit per page out of a bucket shared with
other properties, so identical queries are cached (30 min) and only a real
credit counts against the per-IP throttle. `num=10` with `page` walked, never
`num=100` — the offset upstream is `(page - 1) * num`, so a large `num` puts
page 2 past the end of a truncated result set and pagination dies after one
page. Pages are fetched concurrently because one takes 3-34s. An empty bucket
answers 402 with a plain message rather than a generic failure, and a missing
key answers 503 rather than pretending — the tab still searches transcripts
and still parses a URL without one.
`/api/parse` fetches a URL chosen by an anonymous caller, which is a server-side
request forgery vector unless the target is checked first. It is: http(s) only,
standard ports only, and the hostname must resolve exclusively to public
addresses, so a name pointing at 169.254.169.254 or 10.0.0.5 is refused before
any request is made.
Also: api() now surfaces the server's own explanation instead of throwing a
bare status, so "out of credits" reaches the reader instead of "failed (402)".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WQQYeLjnzqv5n39KLu3gdK
The tab's central decision — is this a URL, a phrase, or a phrase in a mode I
picked — is made in the browser and is invisible to any API test. This drives
the real index.html and app.js in jsdom, the way the crypto and watchlist tabs
are already covered.
Two properties get asserted directly because they are the ones that will
actually bite:
- a 402 from the metered web search must not take the free transcript
results down with it
- the server's own explanation has to reach the reader, so "credits are
exhausted" is what shows rather than "failed (402)"
Writing it turned up that the tickers found in a parsed page were <button>s,
so there was nothing to copy, middle-click or share. They are <a
href="/stocks/SYM"> now; the document-level `.tlink` handler still intercepts
an ordinary click, so the behaviour is unchanged and the affordances are not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WQQYeLjnzqv5n39KLu3gdK
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Search tab could only run full-text over indexed transcripts, which is the narrowest of the three things someone arrives at that box wanting to do. Now one input works out what you meant.
Paste a URL → the page is read back to you: title, publisher and source tier, author, date, the phrases it keeps using, the tickers it names (linked to a stored report where one exists), and any feeds it advertises. Retrieval reuses the news pipeline's fetcher unchanged, so a pasted URL obeys the same rules as an ingested one —
robots.txthonoured, blocking publishers reported as blocked, no body text invented.Type a phrase → Auto searches transcripts and the open web together; Transcripts, Web, News and Parse URL are there when you want one specifically, with a time window for the metered ones. Alongside the results you get what a list of links does not tell you: Google's related searches, the recurring phrases shared across the titles that came back, and the niches the results cluster into. All clickable, so following a thread never means retyping it.
New
GET /api/web?q=&kind=web|news&pages=&time=— results, related searches, phrases, niches, publisher mixGET /api/parse?url=— one page, describedsrc/research/{serp,page,phrases,routes}.ts;test/research.test.ts(28) andtest/dashboard-search.test.ts(19)Design notes
Phrases and niches are deterministic, not generated. They score by document frequency — how many separate results contain a phrase, not how often it occurs — so one long article repeating its own keyword twelve times cannot invent a trend. Where two phrases cover the same documents the longer one wins, so the chip row is not the same idea at three lengths. No LLM, free to run on every search.
ValueSERP is metered, and the pagination is a trap.
numis the stride, not the page size — the upstream offset is(page - 1) * num, so a largenumputs page 2 past the end of a truncated result set and pagination dies after one page. This sendsnum=10and walkspage, concurrently, because one page takes 3–34s. Identical queries are cached for 30 min and only a real credit counts against the per-IP throttle. An empty bucket answers 402 with a plain message and a missing key answers 503 rather than pretending — without a key the tab still searches transcripts and still parses a URL./api/parsefetches a URL chosen by an anonymous caller, which is an SSRF vector unless the target is checked first. It is: http(s) only, standard ports only, and the hostname must resolve exclusively to public addresses, so a name pointing at169.254.169.254or10.0.0.5is refused before any request is made.Also:
api()now surfaces the server's own explanation instead of throwing a bare status, so "out of credits" reaches the reader instead of "failed (402)".Verified
tsc --noEmitclean; 587 pass / 1 fail, and that failure (dashboard-watchlist, arange=1Yassertion) reproduces onorigin/main's frontend files — it predates this branch./api/parseend to end on real pages (metadata, feeds, headings, phrases extracted;theregister.comcorrectly refused as disallowed by robots.txt), the SSRF guard refusinglocalhostand169.254.169.254,/api/webanswering 503 with no key and 402 against the live API.The shared ValueSERP bucket is at 0 credits. 43,871 used in August against a 25k plan; it resets 2026-09-13. Until then
/api/webwill answer 402 on every request and the Web/News parts of the tab will show "credits are exhausted" — the transcripts and URL-parsing halves work regardless. The same key already serves crawlproof.com and rssamplifier.com, so advis0r is a third consumer of a bucket that is already over-subscribed; the plan probably needs raising, or the other consumers throttling, before this feature has budget to run on.VALUESERP_API_KEYis already set on the advis0r.com Railway service (the news pipeline uses it), so nothing needs configuring to deploy this — it just has no budget to spend until the reset.🤖 Generated with Claude Code
https://claude.ai/code/session_01WQQYeLjnzqv5n39KLu3gdK