From c1e519ff85e0329da93aae1386722783879d5139 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Tue, 1 Sep 2026 14:45:09 -0400 Subject: [PATCH 01/20] docs(topics): tag-tree topics design spec Rebuild /topics/ to concept-parity: tag-hierarchy tree nav, KG-concept enrichment, dynamic-slug CAP serve path, search-box fix. Draft for review. --- .../2026-09-01-tag-tree-topics-design.md | 154 ++++++++++++++++++ 1 file changed, 154 insertions(+) create mode 100644 docs/superpowers/specs/2026-09-01-tag-tree-topics-design.md diff --git a/docs/superpowers/specs/2026-09-01-tag-tree-topics-design.md b/docs/superpowers/specs/2026-09-01-tag-tree-topics-design.md new file mode 100644 index 000000000..067f7a17f --- /dev/null +++ b/docs/superpowers/specs/2026-09-01-tag-tree-topics-design.md @@ -0,0 +1,154 @@ +# Tag-Tree Topics — Design + +**Date:** 2026-09-01 +**Status:** Draft for review +**Issue/context:** `/topics/` is broken in production: every topic detail URL 404s, detail pages that render show no concepts, and the topics search box submits to a dead endpoint. This design rebuilds `/topics/` as a first-class subsystem with the same technical quality as `/concepts/`. + +--- + +## 1. Problem statement (why topics is broken today) + +Three independent defects, all confirmed against the live site and the repo: + +1. **Every topic detail page 404s.** The approuter has a full dynamic per-slug serve path for concepts (`^/concepts/(.*)$` → `/content/concepts/$1`, served from HANA and rendered by `srv/lib/concept-detail-render.js`), but for topics **only the index route exists** (`^/topics/?$` → `/content/pages/topics/`). There is **no `/topics/(.*)` route**, so `/topics//` — for any slug — falls through to a bare 404 (verified: the 404 carries no `x-content-source` header, i.e. it never reaches CAP). The Hugo-baked topic stubs are neither in the content-publish scope (`srv/lib/page-key-map.js` `IN_SCOPE_PAGES` lists only `page-topics`, the index) nor routed. + +2. **Detail pages show no concepts.** Topic detail (`hugo/layouts/topics/single.html`) reads `concepts[]` from the `/build/topics-gallery` payload (`srv/lib/build-topics-gallery.js`). That array is a `KgCommunity` (concept-vertex membership) → `Concepts.name` join. When the join returns nothing (empty/stale `KgCommunity` rows or slug-case mismatch), `concepts[]` is `[]` and the `{{ with $c.concepts }}` guard omits the whole section — while `memberCount`/`tutorialCount`, read straight off the `TopicClusters` row, still claim "N concepts". Counts and content come from different sources and disagree. + +3. **`/search/?q=` does nothing.** There is no `/search/` HTML page. `/search/` is the CAP `SearchService` **OData** endpoint. The topics search form (`hugo/layouts/topics/list.html:23`) does a full-page GET to `/search/?q=…`, landing on the OData service root, which ignores `?q=` (OData uses `$search=`) and renders no UI. The working search UI is the navigator island at `/tutorial-navigator/` (reads `?q=` via `hugo-apps/src/navigator/urlSync.ts`). + +Additional structural cause of link rot: the `/topics/` index currently links to ~501 slugs derived from **KG Louvain community labels**, which are **LLM-generated and non-deterministic** — they change on every KG rebuild, so previously valid `/topics//` links stop existing. This is orthogonal to the routing gap but compounds it. + +--- + +## 2. Goals & non-goals + +**Goals** +- Topic detail pages resolve (200) with the same dynamic-slug serve architecture as concepts. +- Topics are keyed by **stable, source-of-truth slugs** (SAP's central software tag hierarchy), so links do not rot. +- Detail pages show **real, deterministically-populated concepts** plus the topic's tutorials. +- `/topics/` presents a **tree navigation experience** over the tag hierarchy (facet → product → sub-product). +- The topics search box reaches the working search UI. +- Legacy AEM tag URLs resurrect **where the tag still has tutorials**; otherwise they redirect gracefully. + +**Non-goals (YAGNI / deferred)** +- Curated editorial "learning-path" topics with hand-ordered concepts (`orderConcepts`, `orderMode: path`). The existing `TopicClusters`/`topics_gallery` learning-path UX can layer on top later; it is not required for parity and is not carried forward as the primary model. +- KG Louvain communities as a public URL key (retained only for the homepage band, unchanged). +- Resurrecting legacy URLs for retired products that have no current tutorials (impossible from a current-catalog source of truth). +- A client-side-only tree (rejected for SEO/parity — see §5). + +--- + +## 3. Topic model + +A **topic = a live SAP tag** (a tag currently applied to ≥1 tutorial). Tags come from SAP's central software hierarchy and are the source of truth already consumed by this platform, so slugs are stable and externally meaningful. + +Tag data (`/build/tags`, cached to `hugo/data/tags.json`) is `group>value`, 8 facet groups, 143 live tags, with a deeper `--` sub-level: + +- Facets: `software-product` (90), `software-product-function` (16), `programming-tool` (12), `topic` (11), `tutorial` (8), `deprecated-concepts` (3), `operating-system` (2), `type` (1). +- Deep example: `software-product-function>sap-hana-cloud--data-lake` → `sap-hana-cloud` → `data-lake`. + +**Natural tree:** `facet → product → sub-product`. The hierarchy *is* the clustering — deterministic, no LLM. + +### Slug scheme +- Topic slug = the flattened leaf value: `sap-hana-cloud--data-lake` → `sap-hana-cloud-data-lake`; `>` and `--` both collapse to `-`. +- Matches the legacy AEM leaf-slug shape (`sap-hana-smart-data-streaming-development`), which is what enables legacy-URL resurrection. +- **Collision handling:** build-time check across all leaves. On collision, facet-qualify the loser (`-`). Expected rare; asserted in a unit test so a future tag import that introduces a collision fails loudly. + +--- + +## 4. Data model — CAP builders + +Two new builder endpoints, both fed by existing sources; both fail-open (return empty payload + `error` field, matching the `build-topics-gallery.js` posture). + +### 4.1 `/build/topics-tree` (index payload) +Builds the navigation tree: +``` +{ tree: [ { facet, label, children: [ { slug, label, tutorialCount, conceptCount, + children: [ …sub-product leaves… ] } ] } ], + buildAt, error } +``` +- Nodes/leaves from `/build/tags` (live tags only → every node has content). +- Human labels from `/build/tag-labels`. +- `tutorialCount` per tag from the existing tag→tutorial index. +- `conceptCount` per tag from the concept-enrichment join (§4.2), so the tree can show concept density. + +### 4.2 `/build/topics/:slug` (detail payload) +``` +{ slug, label, facet, rationale?, + tutorials: [ { slug, title, level, time, href, isNew } ], + concepts: [ { slug, name, rank } ], // deterministic — see below + relatedTags: [ { slug, label } ], // siblings + children in the tree + buildAt, error } +``` +- **Concepts populate deterministically** as: concepts linked (via existing `TutorialConceptLinks`) to the set of tutorials carrying this tag, deduped, ranked by `loadRankMaps()` conceptRank. This replaces the empty `KgCommunity` membership join that causes defect #2. Concepts link out to the existing `/concepts//` pages. +- `tutorials` from the tag→tutorial index (same source the navigator/facets use), lowercased slugs (`tutorialsTableInfo` helper) to avoid the known slug-case pitfall. +- `relatedTags` = tree siblings + children, for cross-navigation. + +### 4.3 Reuse note +`build-topics-gallery.js`, `TopicClusters`, and `topics_gallery.json` are **retired from the `/topics/` path** (the homepage `topic_clusters.json` band is untouched). We delete the two committed stub pages (`hugo/content/topics/btp-basics.md`, `cap-fundamentals.md`) and the `fetch-topics-gallery.ts` stub-generation. Any code deletion sweeps orphaned tests in the same commit. + +--- + +## 5. Tree navigation UI — server-rendered + progressive island + +Matches the concepts-page pattern (server-rendered HTML + a small enhancing island), for SEO and no-JS resilience. **Critically, the index is CAP-rendered, not Hugo-static** — the original topics breakage came from Hugo-baked pages that no route served. `/concepts/` is fully CAP-rendered (`srv/lib/concept-list-page.js` behind `/content/concepts-index`); topics matches that exactly. + +- A new **`srv/lib/topic-list-page.js`** (mirroring `concept-list-page.js`), served at `/content/topics-index`, renders the full `facet → product → sub-product` tree from the `/build/topics-tree` payload as **semantic nested `
    `** with native `
    /` disclosure — fully functional without JS, crawlable — wrapped in the shared chrome (`srv/lib/chrome-shell.js`). Published as a HANA content blob like the concepts index. +- A **small Vue island** (`hugo-apps/src/topics-tree/`) progressively enhances: type-ahead filter, expand/collapse-all, deep-link to an expanded node. Mount is inert until JS loads (same posture as `concepts-filter.js`). Built into the island manifest; the CAP-rendered HTML embeds the hashed island `'); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/unit/topic-list-page.test.js --project unit` +Expected: FAIL — module not found. + +- [ ] **Step 3: Write minimal implementation** + +```js +// srv/lib/topic-list-page.js +import { readFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { gzipSync } from 'node:zlib'; +import { createHash } from 'node:crypto'; +import cds from '@sap/cds'; +import { createShellLoader, ShellMarkerError, composeShell } from './chrome-shell.js'; +import { buildTopicsTreePayload } from './topics-query.js'; +import { setContentCacheHeaders } from './edge-cache-headers.js'; + +const _dir = dirname(fileURLToPath(import.meta.url)); +let _islandManifest; +function islandSrc(name) { + if (!_islandManifest) { + try { _islandManifest = JSON.parse(readFileSync(join(_dir, 'island-manifest.json'), 'utf8')); } + catch { _islandManifest = {}; } + } + return _islandManifest[name] ?? `/js/${name}.js`; +} + +const DEFAULT_NAMESPACE = 'com.sap.developers.ims'; +const HANA_TABLE = 'com_sap_developers_ims_ContentFiles'; +const HANA_CURRENT_TABLE = 'com_sap_developers_ims_ContentCurrent'; + +const TOPICS_STYLE = ``; + +function esc(s) { + return String(s).replace(/&/g, '&').replace(//g, '>').replace(/"/g, '"'); +} +function jsonForScript(obj) { + return JSON.stringify(obj).replace(/${node.tutorialCount ?? 0} tutorials · ${node.conceptCount ?? 0} concepts` + : ''; + const label = node.slug + ? `${esc(node.label)}${count}` + : `${esc(node.label)}`; + if (node.children && node.children.length) { + return `
  • ${label}
      ${node.children.map(renderNode).join('')}
    • `; + } + return `
  • ${label}
    • `; +} + +export function renderTopicListBody(model) { + const facets = (model.tree || []).map(f => + `
  • ${esc(f.label)}
      ${f.children.map(renderNode).join('')}
    • `, + ).join(''); + const data = jsonForScript({ tree: model.tree || [] }); + return `${TOPICS_STYLE} + + +`; +} + +export async function buildTopicListModel(db, _deps = {}) { + const payload = await buildTopicsTreePayload(db); + return { tree: payload.tree || [], version: null }; +} + +export function createTopicListPage({ namespace = DEFAULT_NAMESPACE } = {}) { + const NS = namespace; + const { ContentManifest } = cds.entities(NS); + async function getActiveVersion(db) { + const row = await db.run(SELECT.one.from(ContentManifest).columns('version').where({ status: 'ACTIVE' })).catch(() => null); + return row?.version ?? null; + } + const shellLoader = createShellLoader({ + namespace: NS, hanaTableName: HANA_TABLE, hanaCurrentTableName: HANA_CURRENT_TABLE, + getActiveVersion: () => cds.connect.to('db').then(getActiveVersion), + }); + let cache = null; // { version, gz, etag } + + async function topicsIndexHandler(req, res) { + try { + const db = await cds.connect.to('db'); + const version = await getActiveVersion(db); + if (cache && cache.version === version) { + if (req.headers['if-none-match'] === cache.etag) { res.status(304).end(); return; } + setContentCacheHeaders(res, { slug: 'topics' }); + res.set('Content-Encoding', 'gzip').set('ETag', cache.etag).set('X-Content-Source', 'db-current').type('html').send(cache.gz); + return; + } + const model = await buildTopicListModel(db); + model.version = version; + const body = renderTopicListBody(model); + const meta = { kind: 'topics-index', slug: 'topics', title: 'Explore topics', description: 'Browse SAP developer topics by product hierarchy.' }; + const shell = await shellLoader.get(); + if (!shell) throw new ShellMarkerError('shell unavailable'); + const html = composeShell(shell, body, meta); + const gz = gzipSync(Buffer.from(html, 'utf8')); + const etag = `"${createHash('sha256').update(gz).digest('hex').slice(0, 32)}"`; + cache = { version, gz, etag }; + if (req.headers['if-none-match'] === etag) { res.status(304).end(); return; } + setContentCacheHeaders(res, { slug: 'topics' }); + res.set('Content-Encoding', 'gzip').set('ETag', etag).set('X-Content-Source', 'db-current').type('html').send(gz); + } catch (err) { + if (cache) { + res.set('Content-Encoding', 'gzip').set('X-Content-Source', 'db-stale').type('html').send(cache.gz); + return; + } + res.status(503).type('text/plain').send('topics index unavailable'); + } + } + return { topicsIndexHandler, _invalidate() { cache = null; shellLoader.invalidate?.(); } }; +} + +export const { topicsIndexHandler } = createTopicListPage(); +export default topicsIndexHandler; +``` + +> **NOTE for implementer:** the cache/etag/shell-loader boilerplate above is transcribed from `concept-list-page.js:239-334`. **Diff your version against that source** and adopt any details you missed (exact `setContentCacheHeaders` args, `getActiveVersion` source table, `ContentManifest` query shape). Do NOT invent behavior the concepts handler doesn't have. + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/unit/topic-list-page.test.js --project unit` +Expected: PASS (2 cases). + +- [ ] **Step 5: Add chrome-shell cases + verify concepts untouched** + +Edit `srv/lib/chrome-shell.js`: in `canonicalUrlFor` (kind switch ~`:75-89`) add `case 'topics-index': return '/topics/';` and `case 'topic': return \`/topics/${meta.slug}/\`;`. Add the identical two cases to `buildBreadcrumbJsonLd` (~`:99-137`). + +- [ ] **Step 6: Commit** + +```bash +git add srv/lib/topic-list-page.js srv/lib/chrome-shell.js test/unit/topic-list-page.test.js +git commit -m "feat(topics): CAP-rendered tree index (topic-list-page) + chrome-shell topic canonical/breadcrumb" +``` + +--- + +## Task 5: CAP detail renderer (`srv/lib/topic-detail-render.js`) + +**Files:** +- Create: `srv/lib/topic-detail-render.js` +- Test: extend `test/unit/topics-query.test.js` (add a render block) or new `test/unit/topic-detail-render.test.js`. + +**Interfaces:** +- Consumes: topic detail payload shape from Task 2. +- Produces: `renderTopicDetail(topic): { body: string, contentHash: string }` — `body` supplies its own `
    ` (the shell has only the marker). `topic` = `{ slug, label, facet, tutorials, concepts, relatedTags }`. + +**Reference to mirror:** `srv/lib/concept-detail-render.js:51` (`renderConceptDetail` → `{body, contentHash}`, `body = \`
    ${…}
    \``, `contentHash = sha256(body)`, throws if key fields missing). + +- [ ] **Step 1: Write the failing test** + +```js +// test/unit/topic-detail-render.test.js +import { describe, it, expect } from 'vitest'; +import { renderTopicDetail } from '../../srv/lib/topic-detail-render.js'; + +describe('renderTopicDetail', () => { + const topic = { + slug: 'sap-hana-cloud', label: 'SAP HANA Cloud', facet: 'software-product', + tutorials: [{ slug: 'hana-intro', title: 'HANA Intro', level: 'Beginner', time: 15, href: '/tutorials/hana-intro/', isNew: true }], + concepts: [{ slug: 'in-memory-database', name: 'In-Memory Database', rank: 0.9 }], + relatedTags: [{ slug: 'sap-hana-cloud-data-lake', label: 'Data Lake' }], + }; + it('renders a
    body with breadcrumb, tutorials, concepts, related tags', () => { + const { body, contentHash } = renderTopicDetail(topic); + expect(body.startsWith('
    ')).toBe(true); + expect(body.endsWith('
    ')).toBe(true); + expect(body).toContain('SAP HANA Cloud'); + expect(body).toContain('href="/tutorials/hana-intro/"'); + expect(body).toContain('href="/concepts/in-memory-database/"'); + expect(body).toContain('href="/topics/sap-hana-cloud-data-lake/"'); + expect(contentHash).toMatch(/^[a-f0-9]{64}$/); + }); + it('throws when slug or label missing', () => { + expect(() => renderTopicDetail({ slug: '', label: 'x' })).toThrow(); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/unit/topic-detail-render.test.js --project unit` +Expected: FAIL — module not found. + +- [ ] **Step 3: Write minimal implementation** + +```js +// srv/lib/topic-detail-render.js +import { createHash } from 'node:crypto'; + +function esc(s) { + return String(s ?? '').replace(/&/g, '&').replace(//g, '>').replace(/"/g, '"'); +} + +export function renderTopicDetail(topic) { + if (!topic?.slug || !topic?.label) throw new Error('renderTopicDetail: slug and label required'); + const tutorials = (topic.tutorials || []).map(t => ` +
  • + ${esc(t.title)} + ${t.isNew ? 'NEW' : ''} + ${t.level ? `${esc(t.level)}` : ''} +
    • `).join(''); + const concepts = (topic.concepts || []).map(c => ` +
  • ${esc(c.name)}
    • `).join(''); + const related = (topic.relatedTags || []).map(r => ` +
  • ${esc(r.label)}
    • `).join(''); + + const conceptsSection = concepts + ? `
    +

    Concepts in this topic

    +
      ${concepts}
    +
    ` + : ''; + const relatedSection = related + ? `
    + +
      ${related}
    +
    ` + : ''; + + const body = `
    +
    + +
    +

    ${esc(topic.label)}

    +

    ${esc(topic.facet)}

    +
    +
    +

    Tutorials

    +
      ${tutorials || '
    • No tutorials yet.
      • '}
    +
    + ${conceptsSection} + ${relatedSection} +
    +
    `; + const contentHash = createHash('sha256').update(body).digest('hex'); + return { body, contentHash }; +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/unit/topic-detail-render.test.js --project unit` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add srv/lib/topic-detail-render.js test/unit/topic-detail-render.test.js +git commit -m "feat(topics): topic detail body renderer" +``` + +--- + +## Task 6: Blob publisher (`srv/lib/publish-topics.js`) + +**Files:** +- Create: `srv/lib/publish-topics.js` + +**Interfaces:** +- Consumes: `loadLiveTags`, `buildTopicDetailPayload` (Task 2); `renderTopicDetail` (Task 5); `composeShell` (chrome-shell). +- Produces: `renderTopicsIntoSession({ db, sessionId, helpers, priorHashes = {}, shell, deps = {} }): Promise<{ topicsSeen, topicsChanged, topicsSkipped, topicsErrored, durationMs }>`. Key = `topic-${slug}`. Meta = `{ kind: 'topic', slug, title: label, description }`. + +**Reference to mirror:** `srv/lib/publish-concepts.js:78-171` — per-item loop, `composeShell` wrap, `sha256(fullDoc)`, delta-skip on `priorHashes[key] === contentHash`, `gzipSync(fullDoc).toString('base64')`, `MAX_ERROR_RATE = 0.05` abort, `BATCH_SIZE = 20` append via `helpers.appendToSession`, `loadPriorTopicHashes` filtering `slug.startsWith('topic-')`. + +- [ ] **Step 1: Write the implementation** + +```js +// srv/lib/publish-topics.js +import { gzipSync } from 'node:zlib'; +import { createHash } from 'node:crypto'; +import { loadLiveTags, buildTopicDetailPayload } from './topics-query.js'; +import { renderTopicDetail } from './topic-detail-render.js'; +import { composeShell } from './chrome-shell.js'; + +const BATCH_SIZE = 20; +const MAX_ERROR_RATE = 0.05; +const META_DESC_MAX = 160; + +function topicMetaDescription(topic) { + const n = topic.tutorials?.length ?? 0; + const c = topic.concepts?.length ?? 0; + return `${topic.label}: ${n} tutorials and ${c} concepts on developers.sap.com.`.slice(0, META_DESC_MAX); +} + +export async function renderTopicsIntoSession({ db, sessionId, helpers, priorHashes = {}, shell, deps = {} }) { + const start = Date.now(); + const live = deps.loadLiveTags ? await deps.loadLiveTags(db) : await loadLiveTags(db); + let seen = 0, changed = 0, skipped = 0, errored = 0; + let batch = []; + const flush = async () => { + if (!batch.length) return; + await helpers.appendToSession({ sessionId, files: Object.fromEntries(batch) }); + batch = []; + }; + for (const tag of live) { + seen++; + try { + const topic = await buildTopicDetailPayload(db, tag.slug); + if (topic.notFound || topic.error) { errored++; continue; } + const { body } = renderTopicDetail(topic); + const meta = { kind: 'topic', slug: topic.slug, title: topic.label, description: topicMetaDescription(topic) }; + const fullDoc = composeShell(shell, body, meta); + const key = `topic-${topic.slug}`; + const contentHash = createHash('sha256').update(fullDoc).digest('hex'); + if (priorHashes[key] === contentHash) { skipped++; continue; } + batch.push([key, gzipSync(Buffer.from(fullDoc, 'utf8')).toString('base64')]); + changed++; + if (batch.length >= BATCH_SIZE) await flush(); + } catch { + errored++; + if (seen > 20 && errored / seen > MAX_ERROR_RATE) throw new Error('renderTopicsIntoSession: error rate exceeded'); + } + } + await flush(); + return { topicsSeen: seen, topicsChanged: changed, topicsSkipped: skipped, topicsErrored: errored, durationMs: Date.now() - start }; +} +``` + +> **NOTE for implementer:** wiring `renderTopicsIntoSession` into the actual publish orchestration (the POST `/content/publish/render-*` chain and `loadPriorConceptHashes` equivalent) mirrors `publish-concepts.js:159-194`. Locate where `renderConceptsIntoSession` is invoked during a publish run and register a sibling `renderTopicsIntoSession` call in the same place, loading prior hashes filtered by `slug.startsWith('topic-')`. Confirm the `shell`/`helpers`/`sessionId` objects passed to concepts are reused verbatim. + +- [ ] **Step 2: Commit** + +```bash +git add srv/lib/publish-topics.js +git commit -m "feat(topics): per-slug HANA blob publisher (renderTopicsIntoSession)" +``` + +--- + +## Task 7: Dynamic-slug discovery in publish (`page-key-map.js` + `publish-content.ts`) + +**Files:** +- Modify: `srv/lib/page-key-map.js` (add `TOPIC_KEY_PREFIX`, `isTopicKey`, `discoverTopicPages`) +- Modify: `scripts/publish-content.ts:~1058-1071` (call site) +- Test: extend `test/unit` page-key-map coverage if present; else add `test/unit/page-key-map-topics.test.js`. + +**Interfaces:** +- Produces: `TOPIC_KEY_PREFIX = 'topic-'`; `isTopicKey(key): boolean`; `discoverTopicPages(hugoDir): Map`. + +> **DESIGN NOTE:** topic detail blobs are produced server-side by `renderTopicsIntoSession` (Task 6), NOT from Hugo files on disk — there is no `hugo/.../topic-/index.html`. So `discoverTopicPages` is only relevant if topics are published as file-backed blobs. Since topics are **server-rendered at publish time**, the canonical publish path is the `renderTopicsIntoSession` server call (Task 6), invoked in the publish orchestration alongside `renderConceptsIntoSession`. **Confirm which mechanism the concepts subsystem actually uses at publish time** (recon shows concepts use the *server-side* `renderConceptsIntoSession`, not `discoverPageFiles`). If concepts are server-rendered, SKIP the `discoverTopicPages` file-walker entirely and rely solely on Task 6's server call — this task then reduces to registering that call. + +- [ ] **Step 1: Determine the actual publish mechanism** + +Read `scripts/publish-content.ts` around the concepts publish step and `srv/lib/publish-concepts.js:159-194`. Decide: +- **(A) Server-rendered** (expected): concepts blobs are generated by a POST to `/content/publish/render-concepts`. → Register a sibling topics render call; **do not** add `discoverTopicPages`. Skip Steps 2-4 below. +- **(B) File-backed**: concepts index/detail come from disk via `discoverPageFiles`. → Add `discoverTopicPages` (Steps 2-4). + +- [ ] **Step 2 (only if B): Add discovery helper** + +```js +// srv/lib/page-key-map.js — near AUTHOR/ADVOCATE prefixes +export const TOPIC_KEY_PREFIX = 'topic-'; +export const isTopicKey = (key) => key.startsWith(TOPIC_KEY_PREFIX); + +export function discoverTopicPages(hugoDir) { + const out = new Map(); + const base = path.join(hugoDir, 'topics'); + let entries = []; + try { entries = fs.readdirSync(base, { withFileTypes: true }); } catch { return out; } + for (const e of entries) { + if (!e.isDirectory()) continue; + const slug = e.name; + if (!/^[a-z0-9][a-z0-9-]*$/.test(slug)) continue; + const idx = path.join(base, slug, 'index.html'); + if (fs.existsSync(idx)) out.set(`${TOPIC_KEY_PREFIX}${slug}`, idx); + } + return out; +} +``` + +- [ ] **Step 3 (only if B): Wire the call site** in `scripts/publish-content.ts` next to the author/advocate discovery block: + +```ts +const topics = discoverTopicPages(opts.hugoDir); +for (const [key, absPath] of topics) tutorials.set(key, absPath); +``` + +- [ ] **Step 4 (only if B): Test + commit.** + +- [ ] **Step 5 (path A): Register the server render call** + +Add the `renderTopicsIntoSession` invocation to the publish orchestration alongside the concepts call (mirror `publish-concepts.js`'s `createRenderConcepts`/handler registration; expose `POST /content/publish/render-topics` in `srv/server.js` if concepts has an analogous route). Load prior hashes filtering `slug.startsWith('topic-')`. + +- [ ] **Step 6: Commit** + +```bash +git add srv/lib/page-key-map.js scripts/publish-content.ts srv/server.js +git commit -m "feat(topics): register topic-blob publish step in content publish path" +``` + +--- + +## Task 8: Register CAP routes (`srv/server.js`) + +**Files:** +- Modify: `srv/server.js` + +**Interfaces:** +- Consumes: `buildTopicsTreeHandler`, `buildTopicDetailHandler` (Task 3); `topicsIndexHandler`, `createTopicListPage` (Task 4); `buildTopicDetailPayload`, `resolveTopicBySlug` (Task 2); `renderTopicDetail` (Task 5); shell loader + serve helpers already used by `/content/concepts/:slug`. + +**Reference to mirror:** imports `server.js:15-37`; `/build/*` registrations `:296-306`; `/content/concepts/:slug` wrapper `:503-528`; `/content/concepts-index` `:533`. + +- [ ] **Step 1: Add imports** (top of `srv/server.js`, with sibling imports) + +```js +import { buildTopicsTreeHandler, buildTopicDetailHandler } from './lib/build-topics.js'; +import { topicsIndexHandler } from './lib/topic-list-page.js'; +import { resolveTopicBySlug, buildTopicDetailPayload } from './lib/topics-query.js'; +import { renderTopicDetail } from './lib/topic-detail-render.js'; +``` + +- [ ] **Step 2: Register build feeds** (next to `:298-300`) + +```js +app.get('/build/topics-tree', buildTopicsTreeHandler); +app.get('/build/topics/:slug', buildTopicDetailHandler); +``` + +- [ ] **Step 3: Register content index** (next to `:533`) + +```js +app.get('/content/topics-index', topicsIndexHandler); +``` + +- [ ] **Step 4: Register content detail** — serves the published `topic-` blob, with legacy/retired redirect fallthrough. Mirror the `/content/concepts/:slug` wrapper (`:503-528`) that rewrites `req.params.slug = \`concept-${lower}\`` then delegates to `serveHandler`. For topics, resolve legacy/retired slugs to a 301 before the blob lookup: + +```js +app.get('/content/topics/:slug', async (req, res, next) => { + let raw = String(req.params.slug || '').replace(/\.html$/, ''); + const lower = raw.toLowerCase(); + if (raw !== lower) { res.redirect(301, `/topics/${lower}/`); return; } + // legacy -N / retired resolution + try { + const db = await cds.connect.to('db'); + const { tag, redirectTo } = await resolveTopicBySlug(db, lower); + if (!tag && redirectTo) { res.redirect(301, redirectTo); return; } + if (tag && redirectTo) { res.redirect(301, redirectTo); return; } + } catch { /* fail-open to blob lookup */ } + // delegate to the shared blob serve handler with topic- key prefix (same as concepts) + req.params.slug = `topic-${lower}`; + return serveHandler(req, res, next); +}); +``` + +> **NOTE for implementer:** `serveHandler` is the same content-store serve function `/content/concepts/:slug` delegates to (`server.js:526-527`). Confirm its exact name/signature at that line and match it. If a served `topic-` blob is missing (never published), `serveHandler` will 404 — acceptable; the resolve step above catches *known-live-but-unpublished* only insofar as it doesn't redirect them. For a fully fail-open detail path, when `serveHandler` would 404 AND the slug resolves to a live tag, fall back to on-the-fly render: +> ```js +> // optional fallback inside a wrapper around serveHandler's 404 +> const payload = await buildTopicDetailPayload(db, lower); +> if (!payload.notFound) { /* renderTopicDetail + composeShell + send */ } +> ``` +> Decide based on whether concepts has an equivalent live-render fallback; prefer parity. + +- [ ] **Step 5: Verify build feeds live** + +```bash +cds watch & # or npm run dev:hybrid for real HANA +sleep 8 +curl -s http://localhost:4004/build/topics-tree | jq '.tree | length, .error' +curl -s http://localhost:4004/build/topics/sap-hana-cloud | jq '{tutorials: (.tutorials|length), concepts: (.concepts|length), error}' +``` +Expected: tree length ≥ 1, `error: null`; detail returns tutorials/concepts arrays (may be empty in a bare in-memory DB — verify against seeded/hybrid data). + +- [ ] **Step 6: Commit** + +```bash +git add srv/server.js +git commit -m "feat(topics): register /build/topics-tree, /build/topics/:slug, /content/topics-index, /content/topics/:slug" +``` + +--- + +## Task 9: Approuter routes + search redirect (`approuter/xs-app.json`) + +**Files:** +- Modify: `approuter/xs-app.json` + +**Reference:** concepts index route `:546-550`, detail route `:552-556`, `/build/*` allow-list `:399`, current `/topics/` route `:585`, `/search/` OData route `:375-376`, catch-all `:605`. + +- [ ] **Step 1: Add `topics-tree` to the `/build/*` allow-list alternation** (`:399`) — insert `|topics-tree` into the group (keep `topics-gallery` until Task 11 retires it, or replace it): + +```json +{ "source": "^/build/(breadcrumb-context|catalog|co-completions|concepts|homepage-shelves|kg-stats|mission|my-progress|navigator|repo-catalog|slug-mapping|tag-labels|topics-gallery|topics-tree|topics)(/.*)?(\\?.*)?$", "target": "/build/$1$2$3", "destination": "srv-api", "authenticationType": "none" } +``` + +> `topics` (bare) covers `/build/topics/:slug`; `topics-tree` covers the index feed. Ensure `topics` does not shadow `topics-tree` — regex alternation is ordered but both are matched as whole path segments by the `(/.*)?` boundary, so list `topics-tree` before `topics`. + +- [ ] **Step 2: Replace the `page-topics` index route** (`:585`) with the dynamic index route, and add the detail route immediately after. Both MUST precede the catch-all `^(.*)$` (`:605`): + +```json +{ "source": "^/topics/?(\\?.*)?$", "target": "/content/topics-index$1", "destination": "srv-api", "authenticationType": "none" }, +{ "source": "^/topics/(.*)$", "target": "/content/topics/$1", "destination": "srv-api", "authenticationType": "none" } +``` + +- [ ] **Step 3: Add the `/search/` → `/tutorial-navigator/` redirect, scoped to bare `/search/` + query only** so it does NOT shadow the OData `SearchService` routes (`^/search/(.*)$` at `:375-376`). Place this redirect BEFORE the OData `/search/` route: + +```json +{ "source": "^/search/?(\\?.*)?$", "target": "/tutorial-navigator/$1", "status": 301, "authenticationType": "none" } +``` + +> Verify `/search/SearchableItems` etc. still route to `srv-api` (the `^/search/(.*)$` route must remain and must be reached for non-empty paths). Test both after deploy. + +- [ ] **Step 4: Validate JSON** + +Run: `jq . approuter/xs-app.json > /dev/null && echo OK` +Expected: `OK` (no parse error). + +- [ ] **Step 5: Commit** + +```bash +git add approuter/xs-app.json +git commit -m "feat(topics): approuter dynamic /topics routes + /search 301 to navigator" +``` + +--- + +## Task 10: Progressive-enhancement island (`hugo-apps/src/topics-tree/`) + +**Files:** +- Create: `hugo-apps/src/topics-tree/main.ts`, `hugo-apps/src/topics-tree/App.vue` +- Modify: `hugo-apps/vite.config.ts:~316` (add entry) + +**Interfaces:** +- Consumes: server-embedded ` +`; +} + +/** + * Pure-ish data assembly for the topics list page. + * @param {object} db CDS db service + * @returns {Promise<{tree, version}>} version is null here; the handler stamps it. + */ +export async function buildTopicListModel(db, _deps = {}) { + const payload = await buildTopicsTreePayload(db); + return { tree: payload.tree || [], version: null }; +} + +/** + * Factory — builds an Express handler bound to a namespace, owning its own + * getActiveVersion + shellLoader + version-keyed gzip cache. + */ +export function createTopicListPage({ namespace = DEFAULT_NAMESPACE, deps = {} } = {}) { + const hanaTableName = () => `${namespace.replace(/\./g, '_').toUpperCase()}_CONTENTFILES`; + + async function getActiveVersion() { + const { ContentManifest } = cds.entities(namespace); + const [row] = await SELECT.from(ContentManifest) + .where({ status: 'ACTIVE' }) + .columns('version'); + return row?.version ?? null; + } + + const shellLoader = createShellLoader({ namespace, hanaTableName, getActiveVersion }); + + // Module-level version-keyed cache: { version, gzip, etag }. + let cache = null; + + function fallbackShellCompose(body, meta) { + const s = escapeHtml; + return `` + + `${s(meta.title)}` + + `` + + `` + + `
    ${body}
    `; + } + + async function topicsIndexHandler(req, res) { + const started = Date.now(); + try { + const db = await cds.connect.to('db'); + const version = await getActiveVersion(); + + // Cache hit — same active manifest version. + if (cache && cache.version === version) { + metrics.counter('topic_list_cache_hits'); + const ifNoneMatch = req.headers['if-none-match']; + if (ifNoneMatch && ifNoneMatch === cache.etag) return res.status(304).end(); + res.setHeader('Content-Type', 'text/html; charset=utf-8'); + res.setHeader('Content-Encoding', 'gzip'); + setContentCacheHeaders(res, { slug: 'topics' }); + res.setHeader('ETag', cache.etag); + res.setHeader('X-Content-Source', 'memcache'); + return res.status(200).send(cache.gzip); + } + metrics.counter('topic_list_cache_misses'); + + const model = await buildTopicListModel(db, deps); + model.version = version; + const body = renderTopicListBody(model); + const meta = { kind: 'topics-index', slug: 'topics', title: 'Topics', description: 'Browse SAP developer topics by product and technology hierarchy.' }; + + let html; + try { + const shell = await shellLoader.get(); + if (!shell) throw new ShellMarkerError('shell unavailable'); + html = composeShell(shell, body, meta); + } catch (err) { + console.warn('[content/topics-index] chrome shell missing — degraded rendering:', err.message); + html = fallbackShellCompose(body, meta); + } + + const gzip = gzipSync(Buffer.from(html, 'utf-8')); + const etag = `"${version ?? 'none'}"`; + cache = { version, gzip, etag }; + + metrics.observe('topic_list_render_ms', Date.now() - started); + const ifNoneMatch = req.headers['if-none-match']; + if (ifNoneMatch && ifNoneMatch === etag) return res.status(304).end(); + res.setHeader('Content-Type', 'text/html; charset=utf-8'); + res.setHeader('Content-Encoding', 'gzip'); + setContentCacheHeaders(res, { slug: 'topics' }); + res.setHeader('ETag', etag); + res.setHeader('X-Content-Source', 'fresh'); + return res.status(200).send(gzip); + } catch (err) { + metrics.counter('topic_list_query_failure'); + console.error('[content/topics-index] build failed:', err?.message); + // Serve last-known-good if we have any, even across versions. + if (cache) { + res.setHeader('Content-Type', 'text/html; charset=utf-8'); + res.setHeader('Content-Encoding', 'gzip'); + res.setHeader('Cache-Control', 'public, max-age=60'); + res.setHeader('ETag', cache.etag); + res.setHeader('X-Content-Source', 'stale'); + return res.status(200).send(cache.gzip); + } + res.setHeader('Content-Type', 'text/html; charset=utf-8'); + return res.status(503).send('

    Topics temporarily unavailable

    Please try again shortly.

    '); + } + } + + return { topicsIndexHandler, _invalidate() { cache = null; shellLoader.invalidate?.(); } }; +} + +const _default = createTopicListPage(); +export const topicsIndexHandler = _default.topicsIndexHandler; +export default topicsIndexHandler; diff --git a/test/unit/topic-list-page.test.js b/test/unit/topic-list-page.test.js new file mode 100644 index 000000000..ba066d6d6 --- /dev/null +++ b/test/unit/topic-list-page.test.js @@ -0,0 +1,32 @@ +// test/unit/topic-list-page.test.js +import { describe, it, expect } from 'vitest'; +import { renderTopicListBody } from '../../srv/lib/topic-list-page.js'; + +describe('renderTopicListBody', () => { + const model = { + version: 'v1', + tree: [{ + facet: 'software-product', label: 'Software Product', + children: [ + { segment: 'sap-hana-cloud', slug: 'sap-hana-cloud', label: 'SAP HANA Cloud', tutorialCount: 3, conceptCount: 5, children: [ + { segment: 'data-lake', slug: 'sap-hana-cloud-data-lake', label: 'Data Lake', tutorialCount: 1, conceptCount: 2, children: [] }, + ] }, + ], + }], + }; + it('renders nested details/ul with topic links and no-JS disclosure', () => { + const html = renderTopicListBody(model); + expect(html).toContain(' { + const html = renderTopicListBody(model); + expect(html).toContain('id="topics-tree-data"'); + expect(html).toMatch(/'); + }); +}); From 52139bad344144c92bca70d2e7e864b05206a875 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Tue, 1 Sep 2026 15:42:40 -0400 Subject: [PATCH 09/20] feat(topics): topic detail body renderer --- srv/lib/topic-detail-render.js | 56 +++++++++++++++++++++++++++ test/unit/topic-detail-render.test.js | 24 ++++++++++++ 2 files changed, 80 insertions(+) create mode 100644 srv/lib/topic-detail-render.js create mode 100644 test/unit/topic-detail-render.test.js diff --git a/srv/lib/topic-detail-render.js b/srv/lib/topic-detail-render.js new file mode 100644 index 000000000..48b388a3b --- /dev/null +++ b/srv/lib/topic-detail-render.js @@ -0,0 +1,56 @@ +import { createHash } from 'node:crypto'; + +function esc(s) { + return String(s ?? '').replace(/&/g, '&').replace(//g, '>').replace(/"/g, '"'); +} + +export function renderTopicDetail(topic) { + if (!topic?.slug || !topic?.label) throw new Error('renderTopicDetail: slug and label required'); + const tutorials = (topic.tutorials || []).map(t => ` +
  • + ${esc(t.title)} + ${t.isNew ? 'NEW' : ''} + ${t.level ? `${esc(t.level)}` : ''} +
    • `).join(''); + const concepts = (topic.concepts || []).map(c => ` +
  • ${esc(c.name)}
    • `).join(''); + const related = (topic.relatedTags || []).map(r => ` +
  • ${esc(r.label)}
    • `).join(''); + + const conceptsSection = concepts + ? `
    +

    Concepts in this topic

    +
      ${concepts}
    +
    ` + : ''; + const relatedSection = related + ? `
    + +
      ${related}
    +
    ` + : ''; + + const body = `
    +
    + +
    +

    ${esc(topic.label)}

    +

    ${esc(topic.facet)}

    +
    +
    +

    Tutorials

    +
      ${tutorials || '
    • No tutorials yet.
      • '}
    +
    + ${conceptsSection} + ${relatedSection} +
    +
    `; + const contentHash = createHash('sha256').update(body).digest('hex'); + return { body, contentHash }; +} diff --git a/test/unit/topic-detail-render.test.js b/test/unit/topic-detail-render.test.js new file mode 100644 index 000000000..091c30f72 --- /dev/null +++ b/test/unit/topic-detail-render.test.js @@ -0,0 +1,24 @@ +import { describe, it, expect } from 'vitest'; +import { renderTopicDetail } from '../../srv/lib/topic-detail-render.js'; + +describe('renderTopicDetail', () => { + const topic = { + slug: 'sap-hana-cloud', label: 'SAP HANA Cloud', facet: 'software-product', + tutorials: [{ slug: 'hana-intro', title: 'HANA Intro', level: 'Beginner', time: 15, href: '/tutorials/hana-intro/', isNew: true }], + concepts: [{ slug: 'in-memory-database', name: 'In-Memory Database', rank: 0.9 }], + relatedTags: [{ slug: 'sap-hana-cloud-data-lake', label: 'Data Lake' }], + }; + it('renders a
    body with breadcrumb, tutorials, concepts, related tags', () => { + const { body, contentHash } = renderTopicDetail(topic); + expect(body.startsWith('
    ')).toBe(true); + expect(body.endsWith('
    ')).toBe(true); + expect(body).toContain('SAP HANA Cloud'); + expect(body).toContain('href="/tutorials/hana-intro/"'); + expect(body).toContain('href="/concepts/in-memory-database/"'); + expect(body).toContain('href="/topics/sap-hana-cloud-data-lake/"'); + expect(contentHash).toMatch(/^[a-f0-9]{64}$/); + }); + it('throws when slug or label missing', () => { + expect(() => renderTopicDetail({ slug: '', label: 'x' })).toThrow(); + }); +}); From 92b5b4ef7191ab861798336d4ee6a3003e0e7694 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Tue, 1 Sep 2026 15:45:41 -0400 Subject: [PATCH 10/20] fix(topics): esc() single-quote + explicit utf-8 hash encoding for concept-detail-render parity --- srv/lib/topic-detail-render.js | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/srv/lib/topic-detail-render.js b/srv/lib/topic-detail-render.js index 48b388a3b..59216a3ef 100644 --- a/srv/lib/topic-detail-render.js +++ b/srv/lib/topic-detail-render.js @@ -1,7 +1,7 @@ import { createHash } from 'node:crypto'; function esc(s) { - return String(s ?? '').replace(/&/g, '&').replace(//g, '>').replace(/"/g, '"'); + return String(s ?? '').replace(/&/g, '&').replace(//g, '>').replace(/"/g, '"').replace(/'/g, '''); } export function renderTopicDetail(topic) { @@ -51,6 +51,6 @@ export function renderTopicDetail(topic) { ${relatedSection}
    `; - const contentHash = createHash('sha256').update(body).digest('hex'); + const contentHash = createHash('sha256').update(body, 'utf-8').digest('hex'); return { body, contentHash }; } From 831d4565a62b0e37f59f80f15274e5a26b4ce873 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Tue, 1 Sep 2026 15:48:29 -0400 Subject: [PATCH 11/20] feat(topics): per-slug HANA blob publisher (renderTopicsIntoSession) --- srv/lib/publish-topics.js | 168 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 168 insertions(+) create mode 100644 srv/lib/publish-topics.js diff --git a/srv/lib/publish-topics.js b/srv/lib/publish-topics.js new file mode 100644 index 000000000..7d2ea4952 --- /dev/null +++ b/srv/lib/publish-topics.js @@ -0,0 +1,168 @@ +// srv/lib/publish-topics.js +// +// Task 6 of the tag-tree-topics plan. A session-scoped publish phase that +// renders every live topic tag and appends `topic-` BLOBs to an open +// publish session. Each BLOB is a FULL HTML document: the topic BODY +// (renderTopicDetail) composed into the __shell__ chrome via composeShell, +// mirroring the same pattern used by publish-concepts.js for concept pages. +// +// Dark launch: no publish-content.ts caller yet (Task 7 wires it). + +import cds from '@sap/cds'; +import { gzipSync } from 'node:zlib'; +import { createHash } from 'node:crypto'; +import { loadLiveTags, buildTopicDetailPayload } from './topics-query.js'; +import { renderTopicDetail } from './topic-detail-render.js'; +import { composeShell, createShellLoader } from './chrome-shell.js'; +import { createSessionHelpers } from './content-publish-session.js'; +import * as metrics from './metrics.js'; + +const DEFAULT_NAMESPACE = 'com.sap.developers.ims'; +const BATCH_SIZE = 20; +const MAX_ERROR_RATE = 0.05; // >5% of topics erroring aborts the phase + +const META_DESC_MAX = 160; // Google truncates around 155–160 chars. + +export function topicMetaDescription(topic) { + const n = topic.tutorials?.length ?? 0; + const c = topic.concepts?.length ?? 0; + return `${topic.label}: ${n} tutorial${n === 1 ? '' : 's'} and ${c} concept${c === 1 ? '' : 's'} on developers.sap.com.`.slice(0, META_DESC_MAX); +} + +/** + * Render every live topic into an open publish session. + * + * @param {object} args + * @param {object} args.db CDS db service (passed to loadLiveTags / buildTopicDetailPayload). + * @param {string} args.sessionId Open publish session id. + * @param {object} args.helpers { appendToSession } (from createSessionHelpers). + * @param {object} args.priorHashes Map of `topic-` → prior stored + * contentHash (full-doc sha256) for delta skip. + * @param {{before:string,after:string}} args.shell Parsed __shell__ halves. + * @param {object} [args.deps] { loadLiveTags, buildTopicDetailPayload } — defaults to real. + * @returns {Promise<{topicsSeen,topicsChanged,topicsSkipped,topicsErrored,durationMs}>} + */ +export async function renderTopicsIntoSession({ db, sessionId, helpers, priorHashes = {}, shell, deps = {} }) { + const started = Date.now(); + if (!shell || typeof shell.before !== 'string' || typeof shell.after !== 'string') { + throw new Error('render-topics: shell unavailable — __shell__ sidecar not yet published'); + } + const _loadLiveTags = deps.loadLiveTags || loadLiveTags; + const _buildTopicDetailPayload = deps.buildTopicDetailPayload || buildTopicDetailPayload; + const live = await _loadLiveTags(db); + const topicsSeen = live.length; + + // Render + delta-filter into a flat file map first, so we can enforce the + // error threshold before writing anything. + const changedFiles = {}; // key → base64(gzip(full doc)) + let topicsSkipped = 0; + let topicsErrored = 0; + + for (const tag of live) { + const key = `topic-${tag.slug}`; + try { + const topic = await _buildTopicDetailPayload(db, tag.slug); + if (topic.notFound || topic.error) { + topicsErrored++; + metrics.counter('topic_render_error'); + console.error(`[render-topics] topic "${tag.slug}" payload returned error/notFound`); + continue; + } + const { body } = renderTopicDetail(topic); + const meta = { + kind: 'topic', + slug: topic.slug, + title: topic.label, + description: topicMetaDescription(topic), + }; + const fullDoc = composeShell(shell, body, meta); + const contentHash = createHash('sha256').update(fullDoc, 'utf-8').digest('hex'); + if (priorHashes[key] === contentHash) { + topicsSkipped++; + continue; + } + changedFiles[key] = gzipSync(Buffer.from(fullDoc, 'utf-8')).toString('base64'); + } catch (err) { + topicsErrored++; + metrics.counter('topic_render_error'); + console.error(`[render-topics] topic "${tag.slug}" render failed — carrying forward prior BLOB: ${err.message}`); + } + } + + // Corrupt-run guard: if a large fraction of topics errored, abort so the + // caller can roll the session back rather than commit a degraded corpus. + if (topicsSeen > 0 && topicsErrored / topicsSeen > MAX_ERROR_RATE) { + throw new Error(`render-topics: error rate too high (${topicsErrored}/${topicsSeen}) — aborting phase`); + } + + // Append in batches of BATCH_SIZE. + const keys = Object.keys(changedFiles); + for (let i = 0; i < keys.length; i += BATCH_SIZE) { + const slice = keys.slice(i, i + BATCH_SIZE); + const files = {}; + for (const k of slice) files[k] = changedFiles[k]; + await helpers.appendToSession({ sessionId, files }); + } + + const durationMs = Date.now() - started; + const topicsChanged = keys.length; + metrics.observe('topic_render_ms', durationMs); + metrics.counter('topics_rendered_total', topicsChanged); + metrics.counter('topics_skipped_total', topicsSkipped); + return { topicsSeen, topicsChanged, topicsSkipped, topicsErrored, durationMs }; +} + +/** + * Express handler factory for POST /content/publish/render-topics. Bound to + * a namespace; owns getActiveVersion + shellLoader + prior-hash lookup. + */ +export function createRenderTopics({ namespace = DEFAULT_NAMESPACE } = {}) { + const hanaTableName = () => `${namespace.replace(/\./g, '_').toUpperCase()}_CONTENTFILES`; + + async function getActiveVersion() { + const { ContentManifest } = cds.entities(namespace); + const [row] = await SELECT.from(ContentManifest).where({ status: 'ACTIVE' }).columns('version'); + return row?.version ?? null; + } + + const shellLoader = createShellLoader({ namespace, hanaTableName, getActiveVersion }); + const helpers = createSessionHelpers({ namespace }); + + // Prior stored hashes for topic-* slugs at the current ACTIVE version, for + // delta skip. Same source as hashesHandler. + async function loadPriorTopicHashes() { + const activeVersion = await getActiveVersion(); + if (activeVersion === null) return {}; + const { ContentFiles } = cds.entities(namespace); + const rows = await SELECT.from(ContentFiles) + .where({ version: activeVersion }) + .columns('slug', 'contentHash'); + const out = {}; + for (const r of rows) { + if (typeof r.slug === 'string' && r.slug.startsWith('topic-')) out[r.slug] = r.contentHash; + } + return out; + } + + async function renderTopicsHandler(req, res) { + const sessionId = req.body?.sessionId; + if (!sessionId) return res.status(400).json({ error: 'sessionId is required' }); + try { + const db = await cds.connect.to('db'); + const shell = await shellLoader.get(); // { before, after, version } | null + const priorHashes = await loadPriorTopicHashes(); + const counts = await renderTopicsIntoSession({ db, sessionId, helpers, priorHashes, shell }); + return res.status(200).json(counts); + } catch (err) { + console.error('[render-topics] phase failed:', err?.message); + metrics.counter('topic_render_batch_failure'); + // Leave the session for the caller to abort — do not commit here. + return res.status(500).json({ error: err?.message || 'render-topics failed' }); + } + } + + return { renderTopicsHandler }; +} + +const _default = createRenderTopics(); +export const renderTopicsHandler = _default.renderTopicsHandler; From 4c506c692af9e99218fde5d96935c804b8d8fdec Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Tue, 1 Sep 2026 15:56:08 -0400 Subject: [PATCH 12/20] feat(topics): register topic-blob publish step in content publish path --- scripts/lib/publish-client.ts | 20 ++++++++++++++++++++ scripts/publish-content.ts | 21 ++++++++++++++++++++- srv/server.js | 4 ++++ 3 files changed, 44 insertions(+), 1 deletion(-) diff --git a/scripts/lib/publish-client.ts b/scripts/lib/publish-client.ts index 121e60855..a1e5e053e 100644 --- a/scripts/lib/publish-client.ts +++ b/scripts/lib/publish-client.ts @@ -100,6 +100,26 @@ export async function renderConceptsPhase(i: RenderConceptsInput): Promise` BLOBs to the session (delta skips unchanged). + * Mirrors renderConceptsPhase exactly: must run AFTER the __shell__ sidecar is + * present and BEFORE commit. + */ +export async function renderTopicsPhase(i: RenderTopicsInput): Promise { + return postJson(`${i.baseUrl}/content/publish/render-topics`, i.apiKey, { sessionId: i.sessionId }); +} + export async function abortSession({ baseUrl, apiKey, sessionId, reason }: { baseUrl: string; apiKey: string; sessionId: string; reason?: string; }): Promise<{ aborted: boolean }> { diff --git a/scripts/publish-content.ts b/scripts/publish-content.ts index 9e4e520f0..e1bd2535e 100644 --- a/scripts/publish-content.ts +++ b/scripts/publish-content.ts @@ -5,7 +5,7 @@ import { gzipSync } from 'node:zlib'; import { userInfo, hostname } from 'node:os'; import { parse as parseYaml } from 'yaml'; import { parseChannel, type Channel } from './fetch-tutorials.js'; -import { beginSession, appendBatch, commitSession, abortSession, fetchRemoteHashes, fetchRemoteSourceHashes, renderConceptsPhase } from './lib/publish-client.js'; +import { beginSession, appendBatch, commitSession, abortSession, fetchRemoteHashes, fetchRemoteSourceHashes, renderConceptsPhase, renderTopicsPhase } from './lib/publish-client.js'; import { withRetry, formatErrorChain } from './lib/publish-retry.js'; import { chunk, runConcurrent } from './lib/publish-batcher.js'; import { collectCodeCheckSpecs, publishCodeCheckSpecs } from './lib/publish-codecheck.js'; @@ -1245,6 +1245,25 @@ async function main() { await abortSession({ baseUrl: opts.baseUrl, apiKey: opts.apiKey, sessionId: begin.sessionId, reason: 'render-concepts failed' }); process.exit(1); } + // tag-tree-topics Task 7 — render topic detail pages alongside concepts. + // Same guard (prod-only, full-publish): POST /content/publish/render-topics + // is not registered on srv-qa (topics are a prod-only content surface). + try { + const rt = await withRetry( + () => renderTopicsPhase({ baseUrl: opts.baseUrl, apiKey: opts.apiKey, sessionId: begin.sessionId }), + { + attempts: 3, backoffMs: [1000, 3000, 9000], + onAttemptFail: (attempt, err, willRetry) => { + console.error(`[publish-content] render-topics failed (attempt ${attempt}/3): ${formatErrorChain(err)}${willRetry ? ' — retrying' : ''}`); + }, + } + ); + log(`render-topics: ${rt.topicsChanged} changed, ${rt.topicsSkipped} skipped, ${rt.topicsErrored} errored of ${rt.topicsSeen} (${rt.durationMs} ms)`); + } catch (err) { + console.error(`[publish-content] render-topics failed permanently: ${formatErrorChain(err)}`); + await abortSession({ baseUrl: opts.baseUrl, apiKey: opts.apiKey, sessionId: begin.sessionId, reason: 'render-topics failed' }); + process.exit(1); + } } let commit; diff --git a/srv/server.js b/srv/server.js index 1835c59b6..ab0705d9d 100644 --- a/srv/server.js +++ b/srv/server.js @@ -37,6 +37,7 @@ import { bumpCacheGeneration } from './lib/content-cache-coherence.js'; import { conceptsIndexHandler } from './lib/concept-list-page.js'; import { puzzlePageHandler, puzzleIndexHandler } from './lib/puzzle-page.js'; import { renderConceptsHandler } from './lib/publish-concepts.js'; +import { renderTopicsHandler } from './lib/publish-topics.js'; import { repoCatalogReadHandler, repoCatalogWriteHandler } from './lib/repo-catalog.js'; import { modelJsonHandler } from './lib/model-json-handler.js'; import { kgStatsHandler } from './routes/kg-stats.js'; @@ -573,6 +574,9 @@ cds.on('bootstrap', (app) => { // (Thread B). Dark launch: no publish-content.ts caller yet (Task 5 wires // it). Auth like the other publish routes. app.post('/content/publish/render-concepts', express.json({ limit: '1mb' }), contentAuthMiddleware, renderConceptsHandler); + // tag-tree-topics Task 7 — render topic detail pages alongside concepts. + // Auth and sequencing mirror render-concepts exactly. + app.post('/content/publish/render-topics', express.json({ limit: '1mb' }), contentAuthMiddleware, renderTopicsHandler); app.post('/content/publish/commit', express.json({ limit: '1mb' }), contentAuthMiddleware, commitHandler); app.post('/content/publish/abort', express.json({ limit: '1mb' }), contentAuthMiddleware, abortHandler); From b2d2a808dcfaa622cab90cc86137411e1cf04951 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Tue, 1 Sep 2026 16:09:27 -0400 Subject: [PATCH 13/20] feat(topics): register /build/topics-tree, /build/topics/:slug, /content/topics-index, /content/topics/:slug --- srv/server.js | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/srv/server.js b/srv/server.js index ab0705d9d..81520d364 100644 --- a/srv/server.js +++ b/srv/server.js @@ -38,6 +38,9 @@ import { conceptsIndexHandler } from './lib/concept-list-page.js'; import { puzzlePageHandler, puzzleIndexHandler } from './lib/puzzle-page.js'; import { renderConceptsHandler } from './lib/publish-concepts.js'; import { renderTopicsHandler } from './lib/publish-topics.js'; +import { buildTopicsTreeHandler, buildTopicDetailHandler } from './lib/build-topics.js'; +import { topicsIndexHandler } from './lib/topic-list-page.js'; +import { resolveTopicBySlug } from './lib/topics-query.js'; import { repoCatalogReadHandler, repoCatalogWriteHandler } from './lib/repo-catalog.js'; import { modelJsonHandler } from './lib/model-json-handler.js'; import { kgStatsHandler } from './routes/kg-stats.js'; @@ -299,6 +302,8 @@ cds.on('bootstrap', (app) => { app.get('/build/concepts', buildConceptsHandler); app.get('/build/topic-clusters', buildTopicClustersHandler); app.get('/build/topics-gallery', buildTopicsGalleryHandler); + app.get('/build/topics-tree', buildTopicsTreeHandler); + app.get('/build/topics/:slug', buildTopicDetailHandler); app.get('/graph/explore-data', exploreDataHandler); app.get('/graph/clusters-data', clustersDataHandler); app.get('/graph/path', graphPathHandler); @@ -532,6 +537,35 @@ cds.on('bootstrap', (app) => { // here yet (the /concepts/?$ flip lands in Task 5). Public, no auth — like // serveHandler. app.get('/content/concepts-index', conceptsIndexHandler); + // #topics-scale Task 8 — CAP-served /topics/ detail pages (published topic- + // BLOBs) and the /topics/ index page. Mirrors the /content/concepts/:slug + + // /content/concepts-index pattern exactly: same serveHandler delegation, same + // two-arg call signature, no live-render fallback (concepts has none). + // + // Topics adds a legacy/retired slug resolution step: resolveTopicBySlug checks + // whether the slug is a renamed/retired tag and returns a redirectTo URL when it + // is. Both !tag+redirectTo and tag+redirectTo cases 301 — the redirect takes + // precedence whenever it is set, matching the brief spec. + app.get('/content/topics/:slug', async (req, res) => { + const raw = String(req.params.slug || '').replace(/\.html$/, ''); + const lower = raw.toLowerCase(); + // Mixed-case or .html-stripped → 301 to canonical lowercase /topics/ + if (raw !== lower) { + res.redirect(301, `/topics/${lower}/`); + return; + } + // Legacy / retired slug resolution → 301 to canonical replacement + try { + const db = await cds.connect.to('db'); + const { redirectTo } = await resolveTopicBySlug(db, lower); + if (redirectTo) { res.redirect(301, redirectTo); return; } + } catch { /* fail-open to blob lookup */ } + // Delegate to shared blob serve handler with topic- key prefix (same as concepts) + req.params.slug = `topic-${lower}`; + return serveHandler(req, res); + }); + // Topics index page: /topics/ → CAP SSR (mirrors /content/concepts-index). + app.get('/content/topics-index', topicsIndexHandler); // #1914 — CAP-served puzzle solver pages (/puzzles//, dynamic slug). // The page is a thin island shell composed into the __shell__ chrome; the // `puzzle` island fetches grid/clue data from /puzzle-api at runtime. Serving From 098ff317833c94c8542915e3c5d7617aa181847d Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Tue, 1 Sep 2026 16:14:35 -0400 Subject: [PATCH 14/20] =?UTF-8?q?fix(topics):=20mirror=20concepts=20redire?= =?UTF-8?q?ct=20block=20in=20/content/topics/:slug=20=E2=80=94=20preserve?= =?UTF-8?q?=20query,=20cache-control,=20slug=20validation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- srv/server.js | 34 +++++++++++++++++++++++++++------- 1 file changed, 27 insertions(+), 7 deletions(-) diff --git a/srv/server.js b/srv/server.js index 81520d364..48b505005 100644 --- a/srv/server.js +++ b/srv/server.js @@ -547,20 +547,40 @@ cds.on('bootstrap', (app) => { // is. Both !tag+redirectTo and tag+redirectTo cases 301 — the redirect takes // precedence whenever it is set, matching the brief spec. app.get('/content/topics/:slug', async (req, res) => { - const raw = String(req.params.slug || '').replace(/\.html$/, ''); + const raw = String(req.params.slug || ''); + // Strip .html suffix → 301 to canonical /topics/ + if (/\.html$/i.test(raw)) { + const stripped = raw.replace(/\.html$/i, '').toLowerCase(); + if (/^[a-z0-9][a-z0-9-]*$/.test(stripped)) { + const qIdx = req.url.indexOf('?'); + const query = qIdx >= 0 ? req.url.slice(qIdx) : ''; + res.setHeader('Location', `/topics/${stripped}${query}`); + res.setHeader('Cache-Control', 'public, max-age=3600'); + return res.status(301).end(); + } + } + // Mixed-case → 301 to canonical lowercase /topics/ const lower = raw.toLowerCase(); - // Mixed-case or .html-stripped → 301 to canonical lowercase /topics/ - if (raw !== lower) { - res.redirect(301, `/topics/${lower}/`); - return; + if (raw && raw !== lower && /^[a-z0-9][a-z0-9-]*$/.test(lower)) { + const qIdx = req.url.indexOf('?'); + const query = qIdx >= 0 ? req.url.slice(qIdx) : ''; + res.setHeader('Location', `/topics/${lower}${query}`); + res.setHeader('Cache-Control', 'public, max-age=3600'); + return res.status(301).end(); } // Legacy / retired slug resolution → 301 to canonical replacement try { const db = await cds.connect.to('db'); const { redirectTo } = await resolveTopicBySlug(db, lower); - if (redirectTo) { res.redirect(301, redirectTo); return; } + if (redirectTo) { + const qIdx = req.url.indexOf('?'); + const query = qIdx >= 0 ? req.url.slice(qIdx) : ''; + res.setHeader('Location', `${redirectTo}${query}`); + res.setHeader('Cache-Control', 'public, max-age=3600'); + return res.status(301).end(); + } } catch { /* fail-open to blob lookup */ } - // Delegate to shared blob serve handler with topic- key prefix (same as concepts) + // Canonical form — delegate to serveHandler with the topic- prefix. req.params.slug = `topic-${lower}`; return serveHandler(req, res); }); From 025a45dd514dd845a79934620154035a77e79466 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Tue, 1 Sep 2026 16:20:06 -0400 Subject: [PATCH 15/20] feat(topics): approuter dynamic /topics routes + /search 301 to navigator --- approuter/xs-app.json | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/approuter/xs-app.json b/approuter/xs-app.json index 0ed2cb08a..c1128a70a 100644 --- a/approuter/xs-app.json +++ b/approuter/xs-app.json @@ -371,6 +371,7 @@ "destination": "srv-api", "authenticationType": "xsuaa" }, + { "source": "^/search/?(\\?.*)?$", "target": "/tutorial-navigator/$1", "status": 301, "authenticationType": "none" }, { "source": "^/search/(.*)$", "target": "/search/$1", @@ -396,7 +397,7 @@ "authenticationType": "none" }, { - "source": "^/build/(breadcrumb-context|catalog|co-completions|concepts|homepage-shelves|kg-stats|mission|my-progress|navigator|repo-catalog|slug-mapping|tag-labels|topics-gallery)(/.*)?(\\?.*)?$", + "source": "^/build/(breadcrumb-context|catalog|co-completions|concepts|homepage-shelves|kg-stats|mission|my-progress|navigator|repo-catalog|slug-mapping|tag-labels|topics-gallery|topics-tree|topics)(/.*)?(\\?.*)?$", "target": "/build/$1$2$3", "destination": "srv-api", "authenticationType": "none" @@ -582,7 +583,8 @@ "cacheControl": "public, max-age=3600" }, { "source": "^/browse/?(\\?.*)?$", "target": "/content/pages/browse/$1", "destination": "srv-api", "authenticationType": "none" }, - { "source": "^/topics/?(\\?.*)?$", "target": "/content/pages/topics/$1", "destination": "srv-api", "authenticationType": "none" }, + { "source": "^/topics/?(\\?.*)?$", "target": "/content/topics-index$1", "destination": "srv-api", "authenticationType": "none" }, + { "source": "^/topics/(.*)$", "target": "/content/topics/$1", "destination": "srv-api", "authenticationType": "none" }, { "source": "^/tutorial-navigator/?(\\?.*)?$", "target": "/content/pages/tutorial-navigator/$1", "destination": "srv-api", "authenticationType": "none" }, { "source": "^/developer-advocates/?(\\?.*)?$", "target": "/content/pages/developer-advocates/$1", "destination": "srv-api", "authenticationType": "none" }, { "source": "^/developer-advocates/([^/?]+)/?(\\?.*)?$", "target": "/content/developer-advocates/$1", "destination": "srv-api", "authenticationType": "none" }, From bd8c7355ad797d1eaaaf6e96a51184ccc6957de8 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Tue, 1 Sep 2026 16:28:12 -0400 Subject: [PATCH 16/20] feat(topics): topics-tree progressive-enhancement island (filter + deep-link) --- hugo-apps/src/topics-tree/main.ts | 70 +++++++++++++++++++++++++++++++ hugo-apps/vite.config.ts | 1 + 2 files changed, 71 insertions(+) create mode 100644 hugo-apps/src/topics-tree/main.ts diff --git a/hugo-apps/src/topics-tree/main.ts b/hugo-apps/src/topics-tree/main.ts new file mode 100644 index 000000000..059ce5f50 --- /dev/null +++ b/hugo-apps/src/topics-tree/main.ts @@ -0,0 +1,70 @@ +// hugo-apps/src/topics-tree/main.ts +// +// Progressive enhancement over the server-rendered
    /