Repository navigation
feat(app): serve a root-domain robots.txt - #551
Merged
Merged
Conversation
https://docs.page/robots.txt returned 404. Per-repo robots files exist only at /{owner}/{repo}/robots.txt, so search and AI crawlers hitting the root domain got no crawl policy and no sitemap pointer. Adds a static App Router route serving an explicit "Allow: /" policy plus a Sitemap directive, with cache policy via a new ROOT_ROBOTS_TXT_CACHE_HEADERS constant. All crawlers, AI crawlers included, are deliberately allowed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y5HaatshAzKUdXWYUbAC4a
|
🚅 Deployed to the docs.page-pr-551 environment in docs.page
|
railway-app
Bot
temporarily deployed
to
docs.page / docs.page-pr-551
September 8, 2026 10:28
Destroyed
9 of 15 tasks
|
|
claude Bot
pushed a commit
that referenced
this pull request
Sep 21, 2026
Brings the branch up to date with main after #550, #551, #552 and #553 landed. `app/src/proxy.ts` auto-merged: the root robots.txt and root sitemap.xml cache-header constants from main sit alongside this branch's ROOT_MCP_SERVER_CARD_CACHE_HEADERS, each still consumed by its own route. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y5HaatshAzKUdXWYUbAC4a
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested via Slack thread
One of five PRs replacing #530, which bundled all five root-domain discovery fixes into a single change.
Summary
Before:
https://docs.page/robots.txtreturns a 404. Per-repo robots files exist only under each hosted repository's own path, so a search or AI crawler arriving at the root domain is told nothing about what it may crawl and is given no pointer to a sitemap.After: that same URL returns a plain-text policy that allows every crawler and points at the root sitemap:
In short: the root domain gets an explicit crawl policy and a sitemap pointer instead of a 404. All crawlers, AI crawlers included, are deliberately allowed — there are no AI-specific blocks.
How: a new static App Router route at
app/src/app/robots.txt/route.tsholds the policy as a build-time constant, mirroring the existing root llms.txt route. Its cache policy comes from a newROOT_ROBOTS_TXT_CACHE_HEADERSconstant inapp/src/proxy.ts(day-long edge TTL, hourly browser revalidation).Scope
app/(hosted site, MCP, Ask AI)packages/cli/packages/mdx-bundler/docs/(product documentation)Type of change
Test plan
biome ci .clean (the check CI runs)bun test— 140 pass, 0 failtsc --noEmitinapp/— cleanNotes for reviewers
Sitemap:directive here points athttps://docs.page/sitemap.xml, which is added by feat(app): serve a root-domain sitemap.xml #550. Until feat(app): serve a root-domain sitemap.xml #550 lands, this file advertises a sitemap URL that 404s. Suggested merge order is feat(app): serve a root-domain sitemap.xml #550 first, then this PR.main, not stacked on feat(app): serve a root-domain sitemap.xml #550, so this is independently mergeable either way.🤖 Generated with Claude Code
https://claude.ai/code/session_01Y5HaatshAzKUdXWYUbAC4a
Generated by Claude Code