From 076a61b255038ae030ff68183bc7e1c95187006f Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 11:17:16 -0400 Subject: [PATCH 01/71] docs: slim CLAUDE.md Top Gotchas to headline+link, relocate detail to gotchas.md Cuts per-session CLAUDE.md load ~59% (36.5k->15.1k chars). Verbose gotcha paragraphs moved verbatim into tutorials-ims-gotchas.md 'Top Gotchas - full detail'; items with dedicated reference docs (cds-caching, hcql, cap-ai, e2e-coverage) link straight to those. No content deleted. --- CLAUDE.md | 84 ++++++------------- .../reference/tutorials-ims-gotchas.md | 45 ++++++++++ 2 files changed, 70 insertions(+), 59 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 659deac10..f9197520d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -108,62 +108,28 @@ Subsystem one-liners: - **Admin-UI changes need a FULL deploy + are bundle-gated** — the admin apps (`app/admin/*` + `app/admin-shell/`) are raw-copied into the approuter's `static/admin-ui/` by the MTA's approuter builder during `mbt build`. A `--skip-build` deploy (reuses a stale mtar), a module-scoped `cf deploy -m tutorials-srv`, or an mtar packaged before the change landed will silently ship a **stale admin UI** even though the fix is on `main` (this is why PR #1331/#1345's value-help fix looked "not deployed" on DEV). `npm run deploy` now runs **Step 3.5** (`scripts/check-shipped-admin-bundle.cjs`) which cracks the mtar and diffs the shipped admin component files against source, failing the deploy on drift. Rule: deploy admin-UI changes with a full `npm run deploy -- --env ` (NO `--skip-build`, NO `-m` scoping), and never bypass Step 3.5. -## Top Gotchas (rest in [tutorials-ims-gotchas.md](docs/developers/reference/tutorials-ims-gotchas.md)) - -- **Fresh worktree setup needs `npm run setup` after `npm install`** — global npmrc has `ignore-scripts=true`. Without it, `hugo-apps/node_modules` won't be populated and `better-sqlite3`'s native binding won't build. Symptoms: hugo-apps tests fail resolving `@mediapipe/tasks-vision`, `npm test` hangs. - -- **`ignore-scripts=true` silences `postbuild:apps` — build artifacts wired into it are NOT produced by local `npm run build:all`** — the global npmrc `ignore-scripts=true` (see above) means npm **lifecycle hooks never fire**. The `postbuild:apps` hook is where the #1604 island-fingerprint step (`build:island-manifest`, which writes `hugo/data/island_manifest.json`) and 8 static guards live. During a local `build:all`, none of them run. Symptom class: fresh JS/CSS **compiles** (Vite emits `navigator-.js`) but is **never referenced** — `hugo/layouts/partials/island-src.html` falls back to the unhashed `/js/.js`, Hugo bakes the stale path, and the approuter ships old bundles sitting next to the new ones. **Merged fixes look "not deployed" even though the deploy succeeded.** CI dodges this because `deploy.yml`/`unit-tests.yml` run `npm run postbuild:apps` as an **explicit step** (see deploy.yml:217-223 comment). Fix (2026-08-10): `build:all` now calls `npm run build:island-manifest` **explicitly** (not via the hook), and `scripts/deploy-mta.cjs` Step 2.5 fails the deploy if `hugo/public/index.html` bakes only unhashed island paths while a Vite manifest exists. Rule: any build **artifact** (not just a guard) needed for a correct ship must be an explicit step in `build:all`, never left to a `post*`/`pre*` lifecycle hook. - -- **`build:page-fallback` is an explicit `build:all` step (NOT a lifecycle hook)** — `scripts/build-page-fallback.cjs` copies in-scope page snapshots from `hugo/public` into `srv/page-fallback/.` after `build:hugo` runs. Because `ignore-scripts=true` silences all `pre*`/`post*` hooks (see above), it is wired as an explicit `npm run build:page-fallback` in the `build:all` chain, positioned right after `build:hugo`. If you add a new in-scope page to `IN_SCOPE_PAGES` in `srv/lib/page-key-map.js`, the snapshot is picked up automatically on the next full build. Snapshots are gitignored (`srv/page-fallback/*` except `.gitkeep`); the directory is committed empty and populated at build time. - -- **`hugo/content/tutorials/` is entirely generated** — Never edit; overwritten by `npm run fetch-tutorials`. Edit `scripts/parsers/` or source tutorials in `sap-tutorials` org. - -- **Group/Mission completions are rollup-derived (issue #1934)** — the CAP rewrite never carried over the legacy IMS TUTORIAL→GROUP→MISSION rollup, so GROUP/MISSION `TaskRecords` stopped being created at the 2026-08-10 cutover. `srv/lib/completion-rollup.js` recomputes parent group(s)/mission(s) after any TUTORIAL/PUZZLE/CHECKPOINT/PETOBERFEST completion — called from `_updateTutorialProgress`, `resetTutorialProgress`, the CHECKPOINT edge of `createTaskRecord` (developer-service), `puzzle-service`, and `petoberfest-upload`. Slot model: alt-groups (`#172`) collapse to one slot where **any branch satisfies**; a nested GROUP slot needs all its tutorials. Records key on `(user_ID, taskLegacyId=.legacyId, taskType)` and are upserted (SELECT-then-UPDATE-or-INSERT) with `stampSubmissionId` so they carry the NGDS dedup key. NGDS auto-send fires on the → COMPLETED edge (GROUP/MISSION are the only NGDS-eligible rollup types). The orchestrator never throws into the completion tx. Backfill (post-cutover only): `scripts/backfill-group-mission-completions.mjs` (bulk, `--dry-run`/`--since`/`--user`, no NGDS send) then `scripts/backfill-ngds-send.mjs` (rate-limited, resumable via `ImsConfig 'ngds.backfill.cursor'`, honors env=prod + kill-switch + epoch + canonical-sapId; receiver dedups on `submissionIdCompleted`). Pre-cutover completions are intentionally NOT re-minted (legacy IMS credited them; the NGDS epoch guard suppresses them). `completion-rollup.js` is NOT a `content-store.js` dependency → no `srv-qa` `cp` entry needed. - -- **Never run `publish-content` from a workstation** — Use `gh workflow run rebuild-content.yml`. Workstation publishes skip CI validation; the server-side no-revert guard catches the worst stale-cache regressions but not everything. - -- **Tutorial slugs are lowercase canonical** — Hugo emits lowercase; read path 301-redirects mixed-case. Write path lowercases via `tutorialsTableInfo` helper. Never compare slugs to publish payload without `.toLowerCase()`. Mismatches manifest as "0 steps" on group SSR. - -- **`TutorialMeta` and `Tutorials/Missions/Groups` slugs are unique** — `@assert.unique.slug` + `@assert.unique.tutorial`. New write paths MUST upsert on slug (SELECT-then-UPDATE-or-INSERT). Canonical pattern at `srv/lib/content-publish-session.js:285` / `:349`. Hybrid tests guard. - -- **Never SELECT a HANA BLOB alongside metadata in a single CDS QL query** — LOB locators expire before consumption when mixed with non-BLOB columns. Use raw `db.run()` for BLOB retrieval on HANA (`srv/lib/content-store.js` + `srv/lib/embedding-query.js`). CDS QL works on SQLite for unit tests. - -- **`CONTENT_API_KEY` env var required** for `POST /content/publish` and `/content/rollback`. Missing → 401. Set locally when testing publish. - -- **Content served from mutable `ContentCurrent` (Option B, #2017 / Workstream D), not the old `ContentFiles` snapshot-per-version** — three env flags gate the migration, all default OFF, flip in order: `CONTENT_DELTA_WRITE_ENABLED` (publish dual-writes changed slugs → `ContentCurrent` + append-only `ContentHistory`, fail-safe) → **seed** via a full force rebuild (`-f mode=full -f force-publish=true` dual-writes all slugs; no separate migration) → `CONTENT_DELTA_READ_ENABLED` (serve/readers hit `ContentCurrent`, per-slug fallback to `ContentFiles`) → `CONTENT_DELTA_SKIP_CARRYFORWARD` (publish skips `carryForwardUnchanged` → **O(changed) publish**; rollback then **replays `ContentHistory`** into `ContentCurrent`, not clear+fallback). Measured DEV: publish commit ~62s→973ms (PROD carry-forward was ~95s @ 11k files). Serve source header `X-Content-Source: db-current` (ContentCurrent) vs `db` (legacy). Revert = flip `SKIP_CARRYFORWARD` off + `cf restart`. Flags in `srv/lib/feature-flags/registry.js`; readers via `resolveContentBlob` (`srv/lib/content-store.js`). LOB reads stay raw `db.run()`. Cutover cleanup (delete `carryForwardUnchanged` + `ContentFiles`) deferred to a release after PROD soak. - -- **GitHub Actions secret is `DISPATCH_TOKEN`, not `GITHUB_DISPATCH_TOKEN`** — GH reserves the `GITHUB_` prefix. The runtime env var is `GITHUB_DISPATCH_TOKEN`, read by `srv/lib/rebuild-trigger.js`. - -- **`rebuild-content.yml` auto-infers `mode=slug-targeted`** when a `slug` input is set — don't pass `-f mode=slug-targeted`. Wall-clock: catalog-only ~5min, slug-targeted ~2min, full ~10min. - -- **Alert saves do NOT trigger rebuilds** — Alerts are runtime-served. Cache-bust on save is the only freshness mechanism; up-to-60s admin-to-visitor delay expected. - -- **NGDS auto-send is PROD-only + DB-gated (double gate)** — automatic push of task completions to NGDS (badging/gamification) fires from `srv/lib/ngds-autosend.js#maybeAutoSendCompletion`, called at the two completion transition points in `srv/developer-service.js` (`_updateTutorialProgress` + `createTaskRecord`). Sends ONLY when BOTH gates pass: (1) CF `space_name==='prod'` (`resolveDeployEnvironment`, not spoofable headers) AND (2) `ImsConfig` key `ngds.autosend.enabled==='true'` (admin kill-switch via `AdminService.toggleNgdsAutoSend`; 60s flag cache, busted on toggle). Edge-only (fires on the transition → COMPLETED, never on repeat saves) and task-type-allowlisted to TUTORIAL/GROUP/MISSION (legacy parity — PUZZLE/PETOBERFEST/STEP never sent). Fails CLOSED (config read error → disabled) and never throws into the completion tx (a send fault queues in `NGDSFailedMessages` for the 2h retry job). Bulk recompute (raw HANA MERGE) + migration (raw SQL) bypass the service layer, so they cannot flood NGDS. Payload shape itself is the legacy `MessageModel` (#1473). Default OFF in every env; enable in PROD via the admin toggle. Status: `AdminService.getNgdsAutoSendConfig()` returns `{enabled, environment, effective}`. - -- **`@cap-js/ai` plugin (issue #959, PR 2 of 2)** — adopted for RPT-1 recommendations on `@Common.ValueList` fields. Auto-hooks every such field in Fiori draft-enabled admin UIs. Local `cds watch` uses `AICore-mocked` (no recommendations, zero AI Core quota); hybrid/production use `AICore-btp` against the `aicore` VCAP binding. Per-field opt-out: `@UI.RecommendationState: 0`. Reference: [docs/developers/reference/cap-ai-plugin.md](docs/developers/reference/cap-ai-plugin.md). - -- **`KG_PAGERANK_ENABLED` env var (issue #916)** — when `'true'`, `rankNeighborhood` in `srv/knowledge-graph-service.js` multiplicatively blends per-tutorial PageRank (`weight *= 1 + α × normPR`) into all three tutorial-targeted arms (`prerequisitesOf`, `sharedConcepts`, `whatToLearnNext`) and sorts `teaches` by concept-side PageRank. Default off. Scores recomputed nightly at 03:53 UTC by `srv/jobs/kg-pagerank-job.js` — PageRank runs in **Node.js** (not HANA GraphScript — that engine ships no PageRank primitive) over `KG_PG_VERTICES_V` + `KG_PG_EDGES_V`, materialized into `ConceptRank`/`TutorialRank` sidecars. Fail-opens on every fault path (missing sidecars, HANA hiccup, empty maps → multiplier collapses to 1.0). Toggle: `cf set-env tutorials-srv KG_PAGERANK_ENABLED true && cf restart tutorials-srv`. Blend strength via `KG_PAGERANK_ALPHA` (default `1.0` → weights grow at most 2×). - -- **`KG_WCC_ISOLATION_THRESHOLD` env var (issue #918)** — nightly `srv/jobs/kg-wcc-job.js` runs at 04:07 UTC and materializes rows into `KgIsolation` for concept + tutorial vertices whose weakly-connected-component size ≤ threshold. Default `1`; `0` empties the table on the next run (effectively disables the "Isolated" red-badge column on the admin Concepts + Tutorials LRs). Compute is Node.js union-find over `KG_PG_VERTICES_V` + `KG_PG_EDGES_V` — same reason as #916 that HANA GraphScript ships no WCC primitive (SCC yes, WCC no; enumerated by the #916 Task 0 probe). Fail-quiet at read time: the `after('READ')` decorators on `KnowledgeGraphService.Concepts` and `AdminService.Tutorials` catch any SELECT throw and leave `isolated` unset — Fiori renders `null` boolean as no badge. Toggle: `cf set-env tutorials-srv KG_WCC_ISOLATION_THRESHOLD 2 && cf restart tutorials-srv` (or `0` to disable). - -- **`KG_ONDEMAND_ENABLED` env var / `KnowledgeGraphSettings.onDemandExtractionEnabled` (issue #948)** — when `true`, `expandSearchConcepts` fire-and-forgets an enqueue on zero-seed queries; a new 2-minute cron `kg-ondemand-drain` cosine-ranks the corpus and extracts concepts from top-K tutorials via `extractConceptsFromTutorial`. Coalesces near-duplicate queries; per-user (default 3/hr) and global (default 20/hr) rate-limit caps. Default OFF. Env knobs: `KG_ONDEMAND_USER_MAX_PER_HOUR`, `KG_ONDEMAND_GLOBAL_MAX_PER_HOUR`, `KG_ONDEMAND_DRAIN_BATCH` (default 3), `KG_ONDEMAND_TUTORIALS_PER_REQ` (default 5), `KG_ONDEMAND_MAX_ATTEMPTS` (default 3). Admin surface: `/admin-ui/#kgOnDemand`. Drain uses try/finally to recover stuck RUNNING rows on UPDATE failure. Toggle: flip `KnowledgeGraphSettings.onDemandExtractionEnabled=true` at `/admin-ui/#kg-settings` (or `cf set-env tutorials-srv KG_ONDEMAND_ENABLED true && cf restart tutorials-srv`). - -- **KG community detection (issue #917)** — Louvain community detection over `KG_PG_WORKSPACE` runs nightly at 03:57 UTC (`srv/jobs/kg-communities-job.js`) via HANA GraphScript `Communities_Louvain` in `db/src/procedures/KG_LOUVAIN_GRAPH.hdbprocedure`. Memberships materialize into the `KgCommunity` sidecar (`db/knowledge-graph-communities.cds`). Admin surface: `/admin-ui/#kgCommunities` renders a FE List Report (aggregated summary) + Object Page over `AdminService.KgCommunities` and `AdminService.KgCommunityMembers`. `promoteCommunityToMission(communityId, missionSlug, title)` action (SuperAdmin-gated) drafts a `Missions` row + `CompletionPaths` + `CompletionPathItems` sorted `Tutorials.title ASC`, with `Missions.sourceKgCommunityId` set so already-promoted communities can be filtered out. Nightly job fail-opens; empty sidecar renders as FE "No data", never a 500. **No env flag** — tile is always visible to XSUAA `Tutorial.Author` scope (Task 10 skipped: no shell-config precedent). **DEV-only in v1**; PROD rollout deferred. Metrics: `kg_communities_{duration_ms,count,max_size,failures}`. - -- **HCQL protocol adapter (issue #995)** — CAP 10 beta feature. `@hcql` annotation on 9 read-heavy services (AdminService, AuthorService, AnalyticsService, ExportsService, ConsolidationService, KnowledgeGraphService, HomepageService, SearchService, DeveloperService) in `srv/hcql-enablement.cds` exposes each service at its existing OData URL to also accept CQN `SELECT` bodies (HCQL and OData share URLs; dispatch is by request-body shape). Auth inherited from existing `@readonly`/`@requires`. Writes intentionally unsupported (beta not stable cross-runtime). **Runtime hazard:** CAP 10.0.3 exits the process on malformed CQN — do not expose to untrusted clients until CAP hardens the adapter. Kill switch: delete `srv/hcql-enablement.cds`, `cds build --production`, redeploy. Full reference: [docs/developers/reference/hcql-support.md](docs/developers/reference/hcql-support.md). - -- **`KG_RETIRE_ORPHANS_ENABLED` / `KG_RETIRE_ORPHANS_AGE_DAYS` (issue #1115)** — nightly `srv/jobs/kg-retire-orphans-job.js` at 04:37 UTC flips `Concepts.status` ACTIVE→RETIRED for concepts with zero links across all 10 link tables and `firstSeenAt` older than `KG_RETIRE_ORPHANS_AGE_DAYS` (default 14). RETIRED falls out of every read path (all filter `status='ACTIVE'` positively). First-run retirement **ramps** rather than purging instantly (~1 concept on the first run, climbing toward the full zero-link backlog as each concept ages past the grace window) — because on current data all zero-link orphans were minted within the last ~7-22 days. Reversible: `cf set-env tutorials-srv KG_RETIRE_ORPHANS_ENABLED false` (off) or bulk `UPDATE Concepts SET status='ACTIVE' WHERE status='RETIRED'` (data revert). Companion fix: on-demand extraction (#948) is now **link-only** — it attaches existing concepts (0.7 floor) but never mints, and a re-proposed retired slug is reactivated in-tx by `kg-merge-on-write.js` (`retiredBySlug` + `action:'reactivated'`). - -- **KG community peers + community labeling (issue #1126)** — `communityPeersEnabled` on `ChatSettings` (default `false`) gates the `findCommunityPeers` Joule tool (`srv/lib/kg/joule-tool-community-peers.js`). When enabled, the tool accepts a `tutorial_slug`, looks up the anchor's `communityFingerprint` in `KgCommunity`, and returns sibling tutorials from the same Louvain cluster plus the LLM-generated cluster label from `KgCommunityLabel`. Nightly `kg-community-labels` job (`srv/jobs/kg-community-label-job.js`) runs at 04:12 UTC (after Louvain at 03:57) and LLM-names each community with ≥ 2 tutorials. Identity key is `communityFingerprint` (String(64)); skip-key is `memberSlugsHash` (SHA-256 of sorted slugs) — stable member sets incur zero LLM calls. Daily budget is `communityLabelLlmBudgetPerDay` on `ChatSettings` (default 50); a fresh install ramps the backlog over several nights. Fail-open per community; overall throw → scheduler chassis logs `FAILED`. Toggle: `communityPeersEnabled` is a `ChatSettings` column (NOT an env var — nothing reads a `KG_COMMUNITY_PEERS_ENABLED`), enabled by an Admin via `PATCH /admin/ChatSettings()` on the AdminService singleton — the same entity the `/admin-ui/#joule` Joule Settings page edits (note that page's fixed field list does not yet surface this flag, so a direct PATCH is the enablement path until it is added). **DEV-only until PROD Louvain data verifies.** Metrics: `kg_community_label_{duration_ms,labeled,skipped,failures}`. - -- **`KG_COMMUNITY_WEIGHT` env var (issue #1171)** — when `> 0`, `SearchService.before('READ')` appends a SECOND additive rank term `+ KG_COMMUNITY_WEIGHT * (case slug when '' then 1.0 else 0 end)` alongside the existing concept-overlap `KG_WEIGHT` (#945). Peers are tutorials sharing a Louvain `communityFingerprint` (#917/#1126) with the top-`COMMUNITY_TOP_K` (5) concept-overlap hits. Default `0` (OFF) → `buildCommunityRankFragment` in `srv/lib/search-kg-signal.js` short-circuits before any DB fetch and the rank SQL is byte-identical to the #945 formula. **Only fires when `ChatSettings.searchKgRerankEnabled=true`** — the community term is computed inside the same `before('READ')` gate as the KG signal, so setting `KG_COMMUNITY_WEIGHT > 0` while KG rerank is off has no effect. Fail-open (any DB throw → term collapses to `''`). Membership fetched packet-safe (≤5 fingerprints `.in()`, members capped 200, filtered in Node — `cqn-where-in-hana-packet-cap`). Regression harness + churn report at `test/harness/community-rank-churn*`; do NOT enable in any env before the ON-vs-OFF churn is hand-reviewed. Toggle: `cf set-env tutorials-srv KG_COMMUNITY_WEIGHT 1.5 && cf restart tutorials-srv` (with `searchKgRerankEnabled=true`). - -- **`KG_COMMUNITY_COVERAGE_NUDGE_THRESHOLD` env var (issue #1172)** — the `after('READ','KgCommunities')` decorator in `srv/admin-service.js` computes, per community at read time, mission-coverage % + dominant published mission + orphan-tutorial count (helper: `srv/lib/kg-community-coverage.js`) and populates virtual fields on `AdminService.KgCommunities`. Coverage is **published-missions-only** and the % denominator is **tutorial members only** (concept/tag-only communities render N/A, not 0%). `coverageHigh` (`>= threshold`, default **70**) is the single server-computed flag driving both the LR criticality badge and the FE promote-time `MessageBox.warning` ("~X% already in — extend instead?") in `app/admin/kgCommunities/webapp/ext/KgCommunityActionsController.controller.js`. **Fail-quiet** in its own try/catch (separate from `topConceptSlugs`): any throw → warn-log, fields unset, no badge, never a 500 (mirrors #918). No new job/table/migration — computed live. Packet-safe: the covered-slug `.in()` is chunked at 500 ([[cqn-where-in-hana-packet-cap]]). SuperAdmin gate on `promoteCommunityToMission` unchanged; the nudge is advisory. Override: `cf set-env tutorials-srv KG_COMMUNITY_COVERAGE_NUDGE_THRESHOLD 80 && cf restart tutorials-srv`. DEV-only until the #1126 PROD Louvain rollout lands (no `KgCommunity` data in PROD → empty LR, no nudges). -- **Cluster-level Q&A in Joule (issue #1173)** — `describeCommunity` Joule tool (`srv/lib/kg/joule-tool-describe-community.js`) answers "what's the AI cluster?" / "everything around RAP" by resolving a free-text topic to a labeled Louvain community. **LLM-side matching:** `communityCatalogLayer` in `srv/lib/chat-context.js` injects the labeled-cluster catalog (from `KgCommunityLabel`, cached ~5min, cap 40) into the learner system prompt **only when `communityPeersEnabled` is true**; the model passes the chosen label as `matched_label`, and `matchLabel` (`srv/lib/kg/community-label-match.js`, pure) does case-insensitive exact match + token-overlap fallback + ambiguity detection. Reuses the existing `communityPeersEnabled` flag (NO new flag/schema), the `community-peers-cards` SSE frame + `renderCommunityPeersCards` render path, and the extracted `resolveCommunityMembers` helper (`srv/lib/kg/community-members.js`, also used by `findCommunityPeers`). Fail-open throughout (never 500). **Gotcha:** `buildSystemPromptLines` in `chat-orchestrator.js` is DEAD at runtime — `buildSystemPrompt` (chat-context.js) never calls it; the live guidance ships via `communityCatalogLayer`. DEV-only until PROD Louvain data verifies (same posture as #1126). -- **`test:e2e` is post-deploy only, not on PRs** — the five Playwright admin-UI specs (issue #1338) run in the `e2e` CI job after `smoke-test`. They self-skip when `SMOKE_BASE_URL`/`PLAYWRIGHT_BASE_URL` is absent so `npm test` is never affected. Auth uses `SMOKE_TECH_USER`/`SMOKE_TECH_PASSWORD` (same secrets as smoke) via Basic auth against the approuter — verified against current main; the XSUAA routes short-circuit their IDP redirect for a provisioned tech user. Runbook: `test/e2e/README.md`. Selector gotcha: served tutorials render `
`+`

`, NOT `
` — verified against DEV DOM on 2026-07-27. - -- **cds-caching CDS-DB store + metrics (issue #1179 → re-enabled #1182 → metrics disabled #1215 → metrics re-enabled #1222)** — the shared `caching` service (`cds.requires.caching`) uses `store: "cds"` in the `[hybrid]`/`[production]` profiles (base stays `store: "memory"` for local `cds watch` + unit tests). Gives multi-instance CF coherence (a `deleteByTag` bust on one instance invalidates all). **`metrics.enabled` is ON again (#1222)** — #1215 disabled it because on cds-caching ≤2.0.1 it threw `Wrong input for INT type` on every HANA flush (the plugin read back the hourly row via a flattened table-name SELECT → UPPERCASE column keys → `existingHourly.hits` is `undefined` → `undefined + n = NaN` → hdb INT-bind throw), so counters never accumulated (22 hourly rows stuck at 0). **cds-caching 2.0.2 fixes it** ([mikezaschka/cds-caching#27](https://github.com/mikezaschka/cds-caching/issues/27)): readback via the resolved CSN entity (`SELECT.one.from(Metrics)`) + `Number(existingHourly.) || 0` coercion in `StatisticsPersistenceManager.js`. Re-enable is **config-only** (one-way): the plugin auto-persists `METRICSENABLED=1` on connect when config says `enabled:true` (`CachingService.js:133-135`) — no manual SQL to turn it on (the #1215 *disable* needed a `UPDATE ...SET METRICSENABLED=0` because nothing writes it back to 0). After deploy, clear the stale zeroed rows once per env: `DELETE FROM "PLUGIN_CDS_CACHING_METRICS" WHERE "cache"='caching'` (+ `KEYMETRICS`). Guards: `test/unit/caching-metrics-enabled.test.js` (renamed from `-disabled`) + `test/hybrid/caching-cds-store-boot.test.js` now assert metrics ON. **#1179 crash history:** the first srv deploy carrying `store: "cds"` crash-looped on CF (`Duplicate definition of artifact`) and was reverted (PR #1207); #1182 re-enables it with the resolve-guard fix below. **CF resolve-guard fix (#1182), two required halves:** (1) the srv nodejs `build.task` `model` list also includes `cds-caching/db/cache-store` + `cds-caching/db/statistics`, baking the four `plugin.cds_caching.*` entities into `srv/csn.json` (+4 defs, 0 views lost); (2) `srv/lib/strip-precompiled-plugin-roots.js` (called at top of `srv/server.js`, after `cds.plugins`, before model resolve) strips the plugin's runtime `env.roots` push when a precompiled `srv/csn.json` is present — otherwise those 2 extra roots tip CF's `resolve.many(env.roots)` past `length===1`, re-merging every `requires[].model` onto the precompiled csn → crash. Gated on `srv/csn.json` existence: strips in CF production, no-op in hybrid `cds watch` (compiles from source) and dev/unit (`store:memory` pushes nothing). Baking alone or stripping alone is insufficient — both are load-bearing. Reproducible locally from `gen/srv` (not the source tree — that always collapses to the single csn). **Metrics HDI gotcha:** this project's **explicit `build.tasks` list in `.cdsrc.json` suppresses cds's auto-registration of the plugin's build task** — so the four tables (`CacheStore` + `Caches`/`Metrics`/`KeyMetrics`) do NOT emit unless you add them by hand: `{ "for": "cds-caching" }` (emits `CacheStore.hdbtable`) + `{ "for": "hana", "src": "db", "dest": "db", "options": { "model": ["cds-caching/db/statistics"] } }` (the 3 metrics tables — model must be `statistics`-only; adding `db` to a `db`-dest task's model drops all ~247 service `.hdbview`s). QA container (`tutorials-hana-qa`) intentionally gets none — srv-qa doesn't wire caching. Metrics OData management API deliberately NOT registered (write actions). Test harness: unit workers get a stable memory-store config via `cds_requires_caching_*` env vars in `vitest.config.ts` (NOT a setupFiles that imports `@sap/cds` — that installs getter-only `SELECT`/`INSERT` globals and breaks tests assigning `globalThis.SELECT`) to close the fork-pool boot race (#1177). Full reference: [docs/developers/reference/cds-caching-store.md](docs/developers/reference/cds-caching-store.md). - -- **User-facing UI changes want a committed e2e spec** — a per-PR unit test can't catch a cross-PR seam (that's how #1366's value-help widening shipped dead behind #1353's custom dialog; see #1371). Changing `app/admin/**`, `app/**/webapp/**`, `hugo/layouts/**`, or `hugo-apps/**` triggers an *advisory* PR nudge (`.github/workflows/e2e-coverage-nudge.yml`) to add/update a `test/e2e/` spec. It never blocks merge; the existing post-DEV-deploy `e2e` CI job is where coverage is actually exercised. Pattern: [docs/developers/reference/e2e-coverage-pattern.md](docs/developers/reference/e2e-coverage-pattern.md). - -- **Freshness detector grounding needs the corpus-embedding backfill** — the `checkFreshness`/`freshness-scan` engine cosine-searches `ApiDocs`/`Samples` embeddings. Those columns are populated by `srv/jobs/freshness-corpus-embedding-job.js` (nightly `17 3` + on-demand `runJob`). Until it runs in an env, grounding returns nothing and every API-obsolescence claim degrades to `confidence: Low` (fail-open, by design). LLM calls use the SAP AI SDK directly (`@sap-ai-sdk/orchestration`, forced tool-call), NOT `@cap-js/ai`; unit tests inject `globalThis.__FRESHNESS_TEST_IMPL__`. Bulk scan gated by `FRESHNESS_SCAN_ENABLED` (default OFF). **Tutorial markdown is sourced from `ContentFiles.sourceContent` via `getTutorialSource(slug)` in `srv/lib/content-store.js` — NOT from `Steps.description`** (Steps are never populated with step markdown; reading Steps would yield nothing). Findings carry a **global `codeBlockIndex`** across the whole-tutorial markdown — per-step attribution is deferred because the persisted source is not split per step. +## Top Gotchas + +The load-bearing few. **Full detail for every relocated item → [tutorials-ims-gotchas.md](docs/developers/reference/tutorials-ims-gotchas.md)** ("Top Gotchas — full detail" section); items with their own reference doc link straight to it. + +- **Fresh worktree: run `npm run setup` after `npm install`** — global npmrc `ignore-scripts=true` skips native builds; without it `hugo-apps/node_modules` is empty and `better-sqlite3` won't build (tests hang / fail resolving `@mediapipe/tasks-vision`). +- **`ignore-scripts=true` silences all `pre*`/`post*` hooks** — local `build:all` does NOT fire `postbuild:apps`; island-manifest + `build:page-fallback` are wired as explicit `build:all` steps. Rely on the hook and merged JS/CSS ships dead (unhashed island paths) → "not deployed" despite a green deploy. → gotchas.md "Build artifacts & lifecycle hooks". +- **`hugo/content/tutorials/` is entirely generated** — never edit; `fetch-tutorials` overwrites. Edit `scripts/parsers/` or source repos. +- **Group/Mission completions are rollup-derived (#1934)** — `srv/lib/completion-rollup.js` recomputes parents after any TUTORIAL/PUZZLE/CHECKPOINT/PETOBERFEST completion; upserts on `(user_ID, taskLegacyId, taskType)`; NOT a `content-store.js` dep. → gotchas.md "Completions rollup". +- **Never run `publish-content` from a workstation** — use `gh workflow run rebuild-content.yml`; workstation publishes skip CI validation. +- **Tutorial slugs are lowercase canonical** — always `.toLowerCase()` before comparing to publish payload; mismatch = "0 steps" on group SSR. +- **`TutorialMeta` + `Tutorials/Missions/Groups` slugs are unique** — new write paths MUST upsert on slug (SELECT-then-UPDATE-or-INSERT); pattern at `srv/lib/content-publish-session.js:285`/`:349`. +- **Never SELECT a HANA BLOB alongside metadata in one CDS QL query** — LOB locators expire; use raw `db.run()` (`srv/lib/content-store.js`, `srv/lib/embedding-query.js`). CDS QL is fine on SQLite unit tests. +- **`CONTENT_API_KEY` env var required** for `POST /content/publish` + `/content/rollback` (missing → 401). +- **Content served from mutable `ContentCurrent` (Option B, #2017)** — 3 env flags gate it (all default OFF, flip in order); rollback replays `ContentHistory`. Serve header `X-Content-Source: db-current` vs `db`. → gotchas.md "Content model — mutable ContentCurrent". +- **GitHub Actions secret is `DISPATCH_TOKEN`** (GH reserves `GITHUB_`) — runtime var `GITHUB_DISPATCH_TOKEN`, read by `srv/lib/rebuild-trigger.js`. +- **`rebuild-content.yml` auto-infers `mode=slug-targeted`** when `slug` is set — don't pass mode. Wall-clock: catalog ~5m, slug ~2m, full ~10m. +- **Alert saves do NOT trigger rebuilds** — runtime-served; cache-bust on save only, up-to-60s delay. +- **NGDS auto-send is PROD-only + DB-gated (double gate)** — fires only when CF `space_name==='prod'` AND `ImsConfig ngds.autosend.enabled==='true'`; edge-only, fails closed, never throws into the completion tx, allowlisted to TUTORIAL/GROUP/MISSION. → gotchas.md "NGDS auto-send". +- **`@cap-js/ai` for RPT-1 ValueList recommendations (#959)** — `AICore-mocked` locally, `AICore-btp` in hybrid/prod; per-field opt-out `@UI.RecommendationState: 0`. Ref: [cap-ai-plugin.md](docs/developers/reference/cap-ai-plugin.md). +- **Knowledge-graph feature flags (`KG_*`), all default OFF + DEV-only, all fail-open** — PageRank #916, WCC isolation #918, on-demand extraction #948, Louvain communities #917, orphan retirement #1115, community peers/labels #1126, community search weight #1171, coverage nudge #1172, cluster Q&A #1173. Toggles, nightly jobs, and fail-open specifics → gotchas.md "Knowledge graph feature flags". +- **HCQL protocol adapter (#995, CAP 10 beta)** — `@hcql` on 9 read services accepts CQN `SELECT` bodies at existing OData URLs. **CAP 10.0.3 exits the process on malformed CQN — do not expose to untrusted clients.** Kill: delete `srv/hcql-enablement.cds` + rebuild. Ref: [hcql-support.md](docs/developers/reference/hcql-support.md). +- **cds-caching CDS-DB store + metrics ON (#1222)** — `store:"cds"` in hybrid/prod, `memory` in base/unit. CF resolve-guard needs BOTH the baked csn entities AND `srv/lib/strip-precompiled-plugin-roots.js` (both load-bearing). Ref: [cds-caching-store.md](docs/developers/reference/cds-caching-store.md). +- **User-facing UI changes want a committed e2e spec** — advisory PR nudge on `app/**`/`hugo/**` changes; real coverage runs in the post-DEV-deploy `e2e` job. Ref: [e2e-coverage-pattern.md](docs/developers/reference/e2e-coverage-pattern.md). +- **`test:e2e` is post-deploy only, not on PRs** — self-skips without `SMOKE_BASE_URL`. Served tutorials render `
`+`

`, NOT `
`. Runbook: `test/e2e/README.md`. +- **Freshness detector grounding needs the corpus-embedding backfill** — until `srv/jobs/freshness-corpus-embedding-job.js` runs, every API-obsolescence claim degrades to `confidence: Low`. Tutorial source from `ContentFiles.sourceContent` via `getTutorialSource(slug)`, NOT `Steps.description`. → gotchas.md "Freshness detector". diff --git a/docs/developers/reference/tutorials-ims-gotchas.md b/docs/developers/reference/tutorials-ims-gotchas.md index 9fcb47ddc..235204168 100644 --- a/docs/developers/reference/tutorials-ims-gotchas.md +++ b/docs/developers/reference/tutorials-ims-gotchas.md @@ -102,3 +102,48 @@ Cross-references: ## Tutorial Navigator - **Navigator "Featured" rail is curated via `/admin-ui/#/operations` → Featured Tasks** — draft CRUD (pick items by title via `FeaturedTaskCandidates` value-help, unique per item, order defaults to next integer); SSR from `browse.json`'s `featured[]` array (mission-curated or first-6-missions fallback when empty); live-rehydrated from `GET /build/featured` (ETag/304, 60s server cache, mixed tutorial/mission/group types); cache busts automatically on `FeaturedTasks` save/delete via `resetFeaturedCache()`. + +--- + +# Top Gotchas — full detail (relocated from CLAUDE.md) + +These paragraphs used to live inline in `CLAUDE.md`'s "Top Gotchas" section. They were moved here verbatim to keep `CLAUDE.md` lean; the headline + load-bearing one-liner + a link back to this section remain in `CLAUDE.md`. + +## Build artifacts & lifecycle hooks + +- **`ignore-scripts=true` silences `postbuild:apps` — build artifacts wired into it are NOT produced by local `npm run build:all`** — the global npmrc `ignore-scripts=true` means npm **lifecycle hooks never fire**. The `postbuild:apps` hook is where the #1604 island-fingerprint step (`build:island-manifest`, which writes `hugo/data/island_manifest.json`) and 8 static guards live. During a local `build:all`, none of them run. Symptom class: fresh JS/CSS **compiles** (Vite emits `navigator-.js`) but is **never referenced** — `hugo/layouts/partials/island-src.html` falls back to the unhashed `/js/.js`, Hugo bakes the stale path, and the approuter ships old bundles sitting next to the new ones. **Merged fixes look "not deployed" even though the deploy succeeded.** CI dodges this because `deploy.yml`/`unit-tests.yml` run `npm run postbuild:apps` as an **explicit step** (see deploy.yml:217-223 comment). Fix (2026-08-10): `build:all` now calls `npm run build:island-manifest` **explicitly** (not via the hook), and `scripts/deploy-mta.cjs` Step 2.5 fails the deploy if `hugo/public/index.html` bakes only unhashed island paths while a Vite manifest exists. Rule: any build **artifact** (not just a guard) needed for a correct ship must be an explicit step in `build:all`, never left to a `post*`/`pre*` lifecycle hook. +- **`build:page-fallback` is an explicit `build:all` step (NOT a lifecycle hook)** — `scripts/build-page-fallback.cjs` copies in-scope page snapshots from `hugo/public` into `srv/page-fallback/.` after `build:hugo` runs. Because `ignore-scripts=true` silences all `pre*`/`post*` hooks, it is wired as an explicit `npm run build:page-fallback` in the `build:all` chain, positioned right after `build:hugo`. If you add a new in-scope page to `IN_SCOPE_PAGES` in `srv/lib/page-key-map.js`, the snapshot is picked up automatically on the next full build. Snapshots are gitignored (`srv/page-fallback/*` except `.gitkeep`); the directory is committed empty and populated at build time. + +## Completions rollup (issue #1934) + +- **Group/Mission completions are rollup-derived** — the CAP rewrite never carried over the legacy IMS TUTORIAL→GROUP→MISSION rollup, so GROUP/MISSION `TaskRecords` stopped being created at the 2026-08-10 cutover. `srv/lib/completion-rollup.js` recomputes parent group(s)/mission(s) after any TUTORIAL/PUZZLE/CHECKPOINT/PETOBERFEST completion — called from `_updateTutorialProgress`, `resetTutorialProgress`, the CHECKPOINT edge of `createTaskRecord` (developer-service), `puzzle-service`, and `petoberfest-upload`. Slot model: alt-groups (`#172`) collapse to one slot where **any branch satisfies**; a nested GROUP slot needs all its tutorials. Records key on `(user_ID, taskLegacyId=.legacyId, taskType)` and are upserted (SELECT-then-UPDATE-or-INSERT) with `stampSubmissionId` so they carry the NGDS dedup key. NGDS auto-send fires on the → COMPLETED edge (GROUP/MISSION are the only NGDS-eligible rollup types). The orchestrator never throws into the completion tx. Backfill (post-cutover only): `scripts/backfill-group-mission-completions.mjs` (bulk, `--dry-run`/`--since`/`--user`, no NGDS send) then `scripts/backfill-ngds-send.mjs` (rate-limited, resumable via `ImsConfig 'ngds.backfill.cursor'`, honors env=prod + kill-switch + epoch + canonical-sapId; receiver dedups on `submissionIdCompleted`). Pre-cutover completions are intentionally NOT re-minted (legacy IMS credited them; the NGDS epoch guard suppresses them). `completion-rollup.js` is NOT a `content-store.js` dependency → no `srv-qa` `cp` entry needed. + +## Content model — mutable ContentCurrent (Option B, #2017 / Workstream D) + +- **Content served from mutable `ContentCurrent`, not the old `ContentFiles` snapshot-per-version** — three env flags gate the migration, all default OFF, flip in order: `CONTENT_DELTA_WRITE_ENABLED` (publish dual-writes changed slugs → `ContentCurrent` + append-only `ContentHistory`, fail-safe) → **seed** via a full force rebuild (`-f mode=full -f force-publish=true` dual-writes all slugs; no separate migration) → `CONTENT_DELTA_READ_ENABLED` (serve/readers hit `ContentCurrent`, per-slug fallback to `ContentFiles`) → `CONTENT_DELTA_SKIP_CARRYFORWARD` (publish skips `carryForwardUnchanged` → **O(changed) publish**; rollback then **replays `ContentHistory`** into `ContentCurrent`, not clear+fallback). Measured DEV: publish commit ~62s→973ms (PROD carry-forward was ~95s @ 11k files). Serve source header `X-Content-Source: db-current` (ContentCurrent) vs `db` (legacy). Revert = flip `SKIP_CARRYFORWARD` off + `cf restart`. Flags in `srv/lib/feature-flags/registry.js`; readers via `resolveContentBlob` (`srv/lib/content-store.js`). LOB reads stay raw `db.run()`. Cutover cleanup (delete `carryForwardUnchanged` + `ContentFiles`) deferred to a release after PROD soak. + +## NGDS auto-send + +- **NGDS auto-send is PROD-only + DB-gated (double gate)** — automatic push of task completions to NGDS (badging/gamification) fires from `srv/lib/ngds-autosend.js#maybeAutoSendCompletion`, called at the two completion transition points in `srv/developer-service.js` (`_updateTutorialProgress` + `createTaskRecord`). Sends ONLY when BOTH gates pass: (1) CF `space_name==='prod'` (`resolveDeployEnvironment`, not spoofable headers) AND (2) `ImsConfig` key `ngds.autosend.enabled==='true'` (admin kill-switch via `AdminService.toggleNgdsAutoSend`; 60s flag cache, busted on toggle). Edge-only (fires on the transition → COMPLETED, never on repeat saves) and task-type-allowlisted to TUTORIAL/GROUP/MISSION (legacy parity — PUZZLE/PETOBERFEST/STEP never sent). Fails CLOSED (config read error → disabled) and never throws into the completion tx (a send fault queues in `NGDSFailedMessages` for the 2h retry job). Bulk recompute (raw HANA MERGE) + migration (raw SQL) bypass the service layer, so they cannot flood NGDS. Payload shape itself is the legacy `MessageModel` (#1473). Default OFF in every env; enable in PROD via the admin toggle. Status: `AdminService.getNgdsAutoSendConfig()` returns `{enabled, environment, effective}`. + +## Knowledge graph feature flags + +All default OFF and DEV-only unless noted. Toggles fail-open on every fault path. + +- **`KG_PAGERANK_ENABLED` (issue #916)** — when `'true'`, `rankNeighborhood` in `srv/knowledge-graph-service.js` multiplicatively blends per-tutorial PageRank (`weight *= 1 + α × normPR`) into all three tutorial-targeted arms (`prerequisitesOf`, `sharedConcepts`, `whatToLearnNext`) and sorts `teaches` by concept-side PageRank. Scores recomputed nightly at 03:53 UTC by `srv/jobs/kg-pagerank-job.js` — PageRank runs in **Node.js** (not HANA GraphScript — that engine ships no PageRank primitive) over `KG_PG_VERTICES_V` + `KG_PG_EDGES_V`, materialized into `ConceptRank`/`TutorialRank` sidecars. Fail-opens on every fault path (missing sidecars, HANA hiccup, empty maps → multiplier collapses to 1.0). Toggle: `cf set-env tutorials-srv KG_PAGERANK_ENABLED true && cf restart tutorials-srv`. Blend strength via `KG_PAGERANK_ALPHA` (default `1.0` → weights grow at most 2×). +- **`KG_WCC_ISOLATION_THRESHOLD` (issue #918)** — nightly `srv/jobs/kg-wcc-job.js` runs at 04:07 UTC and materializes rows into `KgIsolation` for concept + tutorial vertices whose weakly-connected-component size ≤ threshold. Default `1`; `0` empties the table on the next run (effectively disables the "Isolated" red-badge column on the admin Concepts + Tutorials LRs). Compute is Node.js union-find over `KG_PG_VERTICES_V` + `KG_PG_EDGES_V` — same reason as #916 that HANA GraphScript ships no WCC primitive (SCC yes, WCC no). Fail-quiet at read time: the `after('READ')` decorators on `KnowledgeGraphService.Concepts` and `AdminService.Tutorials` catch any SELECT throw and leave `isolated` unset — Fiori renders `null` boolean as no badge. Toggle: `cf set-env tutorials-srv KG_WCC_ISOLATION_THRESHOLD 2 && cf restart tutorials-srv` (or `0` to disable). +- **`KG_ONDEMAND_ENABLED` / `KnowledgeGraphSettings.onDemandExtractionEnabled` (issue #948)** — when `true`, `expandSearchConcepts` fire-and-forgets an enqueue on zero-seed queries; a new 2-minute cron `kg-ondemand-drain` cosine-ranks the corpus and extracts concepts from top-K tutorials via `extractConceptsFromTutorial`. Coalesces near-duplicate queries; per-user (default 3/hr) and global (default 20/hr) rate-limit caps. Env knobs: `KG_ONDEMAND_USER_MAX_PER_HOUR`, `KG_ONDEMAND_GLOBAL_MAX_PER_HOUR`, `KG_ONDEMAND_DRAIN_BATCH` (default 3), `KG_ONDEMAND_TUTORIALS_PER_REQ` (default 5), `KG_ONDEMAND_MAX_ATTEMPTS` (default 3). Admin surface: `/admin-ui/#kgOnDemand`. Drain uses try/finally to recover stuck RUNNING rows on UPDATE failure. On-demand extraction is now **link-only** (#1115) — it attaches existing concepts (0.7 floor) but never mints. Toggle: flip `onDemandExtractionEnabled=true` at `/admin-ui/#kg-settings` (or `cf set-env tutorials-srv KG_ONDEMAND_ENABLED true && cf restart tutorials-srv`). +- **KG community detection (issue #917)** — Louvain community detection over `KG_PG_WORKSPACE` runs nightly at 03:57 UTC (`srv/jobs/kg-communities-job.js`) via HANA GraphScript `Communities_Louvain` in `db/src/procedures/KG_LOUVAIN_GRAPH.hdbprocedure`. Memberships materialize into the `KgCommunity` sidecar (`db/knowledge-graph-communities.cds`). Admin surface: `/admin-ui/#kgCommunities` renders a FE List Report (aggregated summary) + Object Page over `AdminService.KgCommunities` and `AdminService.KgCommunityMembers`. `promoteCommunityToMission(communityId, missionSlug, title)` action (SuperAdmin-gated) drafts a `Missions` row + `CompletionPaths` + `CompletionPathItems` sorted `Tutorials.title ASC`, with `Missions.sourceKgCommunityId` set so already-promoted communities can be filtered out. Nightly job fail-opens; empty sidecar renders as FE "No data", never a 500. **No env flag** — tile is always visible to XSUAA `Tutorial.Author` scope. **DEV-only in v1**; PROD rollout deferred. Metrics: `kg_communities_{duration_ms,count,max_size,failures}`. +- **`KG_RETIRE_ORPHANS_ENABLED` / `KG_RETIRE_ORPHANS_AGE_DAYS` (issue #1115)** — nightly `srv/jobs/kg-retire-orphans-job.js` at 04:37 UTC flips `Concepts.status` ACTIVE→RETIRED for concepts with zero links across all 10 link tables and `firstSeenAt` older than `KG_RETIRE_ORPHANS_AGE_DAYS` (default 14). RETIRED falls out of every read path (all filter `status='ACTIVE'` positively). First-run retirement **ramps** rather than purging instantly. Reversible: `cf set-env tutorials-srv KG_RETIRE_ORPHANS_ENABLED false` (off) or bulk `UPDATE Concepts SET status='ACTIVE' WHERE status='RETIRED'` (data revert). A re-proposed retired slug is reactivated in-tx by `kg-merge-on-write.js` (`retiredBySlug` + `action:'reactivated'`). +- **KG community peers + community labeling (issue #1126)** — `communityPeersEnabled` on `ChatSettings` (default `false`) gates the `findCommunityPeers` Joule tool (`srv/lib/kg/joule-tool-community-peers.js`). When enabled, the tool accepts a `tutorial_slug`, looks up the anchor's `communityFingerprint` in `KgCommunity`, and returns sibling tutorials from the same Louvain cluster plus the LLM-generated cluster label from `KgCommunityLabel`. Nightly `kg-community-labels` job (`srv/jobs/kg-community-label-job.js`) runs at 04:12 UTC (after Louvain at 03:57) and LLM-names each community with ≥ 2 tutorials. Identity key is `communityFingerprint` (String(64)); skip-key is `memberSlugsHash` (SHA-256 of sorted slugs) — stable member sets incur zero LLM calls. Daily budget is `communityLabelLlmBudgetPerDay` on `ChatSettings` (default 50). Fail-open per community. Toggle: `communityPeersEnabled` is a `ChatSettings` column (NOT an env var), enabled by an Admin via `PATCH /admin/ChatSettings()` on the AdminService singleton (the `/admin-ui/#joule` Joule Settings page edits the same entity but does not yet surface this flag). **DEV-only until PROD Louvain data verifies.** Metrics: `kg_community_label_{duration_ms,labeled,skipped,failures}`. +- **`KG_COMMUNITY_WEIGHT` (issue #1171)** — when `> 0`, `SearchService.before('READ')` appends a SECOND additive rank term `+ KG_COMMUNITY_WEIGHT * (case slug when '' then 1.0 else 0 end)` alongside the existing concept-overlap `KG_WEIGHT` (#945). Peers are tutorials sharing a Louvain `communityFingerprint` (#917/#1126) with the top-`COMMUNITY_TOP_K` (5) concept-overlap hits. Default `0` (OFF) → `buildCommunityRankFragment` in `srv/lib/search-kg-signal.js` short-circuits before any DB fetch and the rank SQL is byte-identical to the #945 formula. **Only fires when `ChatSettings.searchKgRerankEnabled=true`**. Fail-open (any DB throw → term collapses to `''`). Membership fetched packet-safe (≤5 fingerprints `.in()`, members capped 200, filtered in Node). Regression harness + churn report at `test/harness/community-rank-churn*`; do NOT enable in any env before the ON-vs-OFF churn is hand-reviewed. Toggle: `cf set-env tutorials-srv KG_COMMUNITY_WEIGHT 1.5 && cf restart tutorials-srv` (with `searchKgRerankEnabled=true`). +- **`KG_COMMUNITY_COVERAGE_NUDGE_THRESHOLD` (issue #1172)** — the `after('READ','KgCommunities')` decorator in `srv/admin-service.js` computes, per community at read time, mission-coverage % + dominant published mission + orphan-tutorial count (helper: `srv/lib/kg-community-coverage.js`) and populates virtual fields on `AdminService.KgCommunities`. Coverage is **published-missions-only** and the % denominator is **tutorial members only** (concept/tag-only communities render N/A, not 0%). `coverageHigh` (`>= threshold`, default **70**) is the single server-computed flag driving both the LR criticality badge and the FE promote-time `MessageBox.warning` ("~X% already in — extend instead?"). **Fail-quiet** in its own try/catch: any throw → warn-log, fields unset, no badge, never a 500. No new job/table/migration — computed live. Packet-safe: the covered-slug `.in()` is chunked at 500. SuperAdmin gate on `promoteCommunityToMission` unchanged; the nudge is advisory. Override: `cf set-env tutorials-srv KG_COMMUNITY_COVERAGE_NUDGE_THRESHOLD 80 && cf restart tutorials-srv`. DEV-only until the #1126 PROD Louvain rollout lands. +- **Cluster-level Q&A in Joule (issue #1173)** — `describeCommunity` Joule tool (`srv/lib/kg/joule-tool-describe-community.js`) answers "what's the AI cluster?" / "everything around RAP" by resolving a free-text topic to a labeled Louvain community. **LLM-side matching:** `communityCatalogLayer` in `srv/lib/chat-context.js` injects the labeled-cluster catalog (from `KgCommunityLabel`, cached ~5min, cap 40) into the learner system prompt **only when `communityPeersEnabled` is true**; the model passes the chosen label as `matched_label`, and `matchLabel` (`srv/lib/kg/community-label-match.js`, pure) does case-insensitive exact match + token-overlap fallback + ambiguity detection. Reuses the existing `communityPeersEnabled` flag (NO new flag/schema), the `community-peers-cards` SSE frame + `renderCommunityPeersCards` render path, and the extracted `resolveCommunityMembers` helper (`srv/lib/kg/community-members.js`). Fail-open throughout (never 500). **Gotcha:** `buildSystemPromptLines` in `chat-orchestrator.js` is DEAD at runtime — `buildSystemPrompt` (chat-context.js) never calls it; the live guidance ships via `communityCatalogLayer`. DEV-only until PROD Louvain data verifies. + +## HCQL protocol adapter (issue #995) + +- **HCQL protocol adapter** — CAP 10 beta feature. `@hcql` annotation on 9 read-heavy services (AdminService, AuthorService, AnalyticsService, ExportsService, ConsolidationService, KnowledgeGraphService, HomepageService, SearchService, DeveloperService) in `srv/hcql-enablement.cds` exposes each service at its existing OData URL to also accept CQN `SELECT` bodies (HCQL and OData share URLs; dispatch is by request-body shape). Auth inherited from existing `@readonly`/`@requires`. Writes intentionally unsupported (beta not stable cross-runtime). **Runtime hazard:** CAP 10.0.3 exits the process on malformed CQN — do not expose to untrusted clients until CAP hardens the adapter. Kill switch: delete `srv/hcql-enablement.cds`, `cds build --production`, redeploy. Full reference: [hcql-support.md](hcql-support.md). + +## Freshness detector + +- **Freshness detector grounding needs the corpus-embedding backfill** — the `checkFreshness`/`freshness-scan` engine cosine-searches `ApiDocs`/`Samples` embeddings. Those columns are populated by `srv/jobs/freshness-corpus-embedding-job.js` (nightly `17 3` + on-demand `runJob`). Until it runs in an env, grounding returns nothing and every API-obsolescence claim degrades to `confidence: Low` (fail-open, by design). LLM calls use the SAP AI SDK directly (`@sap-ai-sdk/orchestration`, forced tool-call), NOT `@cap-js/ai`; unit tests inject `globalThis.__FRESHNESS_TEST_IMPL__`. Bulk scan gated by `FRESHNESS_SCAN_ENABLED` (default OFF). **Tutorial markdown is sourced from `ContentFiles.sourceContent` via `getTutorialSource(slug)` in `srv/lib/content-store.js` — NOT from `Steps.description`** (Steps are never populated with step markdown; reading Steps would yield nothing). Findings carry a **global `codeBlockIndex`** across the whole-tutorial markdown — per-step attribution is deferred because the persisted source is not split per step. From 929547d238ccbc786130a1dda421cc3de53197a1 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 11:26:06 -0400 Subject: [PATCH 02/71] fix(docs): wrap bare in backticks so VitePress/Vue build passes The relocated KG coverage-nudge gotcha carried a bare token. VitePress compiles docs/**/*.md as Vue SFCs, so the unclosed angle-bracket parsed as an HTML tag and failed 'Deploy Docs to GitHub Pages'. Harmless in CLAUDE.md (not VitePress-built). --- docs/developers/reference/tutorials-ims-gotchas.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developers/reference/tutorials-ims-gotchas.md b/docs/developers/reference/tutorials-ims-gotchas.md index 235204168..bb4a08286 100644 --- a/docs/developers/reference/tutorials-ims-gotchas.md +++ b/docs/developers/reference/tutorials-ims-gotchas.md @@ -137,7 +137,7 @@ All default OFF and DEV-only unless noted. Toggles fail-open on every fault path - **`KG_RETIRE_ORPHANS_ENABLED` / `KG_RETIRE_ORPHANS_AGE_DAYS` (issue #1115)** — nightly `srv/jobs/kg-retire-orphans-job.js` at 04:37 UTC flips `Concepts.status` ACTIVE→RETIRED for concepts with zero links across all 10 link tables and `firstSeenAt` older than `KG_RETIRE_ORPHANS_AGE_DAYS` (default 14). RETIRED falls out of every read path (all filter `status='ACTIVE'` positively). First-run retirement **ramps** rather than purging instantly. Reversible: `cf set-env tutorials-srv KG_RETIRE_ORPHANS_ENABLED false` (off) or bulk `UPDATE Concepts SET status='ACTIVE' WHERE status='RETIRED'` (data revert). A re-proposed retired slug is reactivated in-tx by `kg-merge-on-write.js` (`retiredBySlug` + `action:'reactivated'`). - **KG community peers + community labeling (issue #1126)** — `communityPeersEnabled` on `ChatSettings` (default `false`) gates the `findCommunityPeers` Joule tool (`srv/lib/kg/joule-tool-community-peers.js`). When enabled, the tool accepts a `tutorial_slug`, looks up the anchor's `communityFingerprint` in `KgCommunity`, and returns sibling tutorials from the same Louvain cluster plus the LLM-generated cluster label from `KgCommunityLabel`. Nightly `kg-community-labels` job (`srv/jobs/kg-community-label-job.js`) runs at 04:12 UTC (after Louvain at 03:57) and LLM-names each community with ≥ 2 tutorials. Identity key is `communityFingerprint` (String(64)); skip-key is `memberSlugsHash` (SHA-256 of sorted slugs) — stable member sets incur zero LLM calls. Daily budget is `communityLabelLlmBudgetPerDay` on `ChatSettings` (default 50). Fail-open per community. Toggle: `communityPeersEnabled` is a `ChatSettings` column (NOT an env var), enabled by an Admin via `PATCH /admin/ChatSettings()` on the AdminService singleton (the `/admin-ui/#joule` Joule Settings page edits the same entity but does not yet surface this flag). **DEV-only until PROD Louvain data verifies.** Metrics: `kg_community_label_{duration_ms,labeled,skipped,failures}`. - **`KG_COMMUNITY_WEIGHT` (issue #1171)** — when `> 0`, `SearchService.before('READ')` appends a SECOND additive rank term `+ KG_COMMUNITY_WEIGHT * (case slug when '' then 1.0 else 0 end)` alongside the existing concept-overlap `KG_WEIGHT` (#945). Peers are tutorials sharing a Louvain `communityFingerprint` (#917/#1126) with the top-`COMMUNITY_TOP_K` (5) concept-overlap hits. Default `0` (OFF) → `buildCommunityRankFragment` in `srv/lib/search-kg-signal.js` short-circuits before any DB fetch and the rank SQL is byte-identical to the #945 formula. **Only fires when `ChatSettings.searchKgRerankEnabled=true`**. Fail-open (any DB throw → term collapses to `''`). Membership fetched packet-safe (≤5 fingerprints `.in()`, members capped 200, filtered in Node). Regression harness + churn report at `test/harness/community-rank-churn*`; do NOT enable in any env before the ON-vs-OFF churn is hand-reviewed. Toggle: `cf set-env tutorials-srv KG_COMMUNITY_WEIGHT 1.5 && cf restart tutorials-srv` (with `searchKgRerankEnabled=true`). -- **`KG_COMMUNITY_COVERAGE_NUDGE_THRESHOLD` (issue #1172)** — the `after('READ','KgCommunities')` decorator in `srv/admin-service.js` computes, per community at read time, mission-coverage % + dominant published mission + orphan-tutorial count (helper: `srv/lib/kg-community-coverage.js`) and populates virtual fields on `AdminService.KgCommunities`. Coverage is **published-missions-only** and the % denominator is **tutorial members only** (concept/tag-only communities render N/A, not 0%). `coverageHigh` (`>= threshold`, default **70**) is the single server-computed flag driving both the LR criticality badge and the FE promote-time `MessageBox.warning` ("~X% already in — extend instead?"). **Fail-quiet** in its own try/catch: any throw → warn-log, fields unset, no badge, never a 500. No new job/table/migration — computed live. Packet-safe: the covered-slug `.in()` is chunked at 500. SuperAdmin gate on `promoteCommunityToMission` unchanged; the nudge is advisory. Override: `cf set-env tutorials-srv KG_COMMUNITY_COVERAGE_NUDGE_THRESHOLD 80 && cf restart tutorials-srv`. DEV-only until the #1126 PROD Louvain rollout lands. +- **`KG_COMMUNITY_COVERAGE_NUDGE_THRESHOLD` (issue #1172)** — the `after('READ','KgCommunities')` decorator in `srv/admin-service.js` computes, per community at read time, mission-coverage % + dominant published mission + orphan-tutorial count (helper: `srv/lib/kg-community-coverage.js`) and populates virtual fields on `AdminService.KgCommunities`. Coverage is **published-missions-only** and the % denominator is **tutorial members only** (concept/tag-only communities render N/A, not 0%). `coverageHigh` (`>= threshold`, default **70**) is the single server-computed flag driving both the LR criticality badge and the FE promote-time `MessageBox.warning` ("~X% already in `` — extend instead?"). **Fail-quiet** in its own try/catch: any throw → warn-log, fields unset, no badge, never a 500. No new job/table/migration — computed live. Packet-safe: the covered-slug `.in()` is chunked at 500. SuperAdmin gate on `promoteCommunityToMission` unchanged; the nudge is advisory. Override: `cf set-env tutorials-srv KG_COMMUNITY_COVERAGE_NUDGE_THRESHOLD 80 && cf restart tutorials-srv`. DEV-only until the #1126 PROD Louvain rollout lands. - **Cluster-level Q&A in Joule (issue #1173)** — `describeCommunity` Joule tool (`srv/lib/kg/joule-tool-describe-community.js`) answers "what's the AI cluster?" / "everything around RAP" by resolving a free-text topic to a labeled Louvain community. **LLM-side matching:** `communityCatalogLayer` in `srv/lib/chat-context.js` injects the labeled-cluster catalog (from `KgCommunityLabel`, cached ~5min, cap 40) into the learner system prompt **only when `communityPeersEnabled` is true**; the model passes the chosen label as `matched_label`, and `matchLabel` (`srv/lib/kg/community-label-match.js`, pure) does case-insensitive exact match + token-overlap fallback + ambiguity detection. Reuses the existing `communityPeersEnabled` flag (NO new flag/schema), the `community-peers-cards` SSE frame + `renderCommunityPeersCards` render path, and the extracted `resolveCommunityMembers` helper (`srv/lib/kg/community-members.js`). Fail-open throughout (never 500). **Gotcha:** `buildSystemPromptLines` in `chat-orchestrator.js` is DEAD at runtime — `buildSystemPrompt` (chat-context.js) never calls it; the live guidance ships via `communityCatalogLayer`. DEV-only until PROD Louvain data verifies. ## HCQL protocol adapter (issue #995) From 6780272a92799bd175170e35b35c926762b5668c Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 12:35:10 -0400 Subject: [PATCH 03/71] docs(admin): design spec for Tutorials OP enhancements (categories/contributors/validation/KG/media) --- ...-tutorials-admin-op-enhancements-design.md | 135 ++++++++++++++++++ 1 file changed, 135 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-31-tutorials-admin-op-enhancements-design.md diff --git a/docs/superpowers/specs/2026-08-31-tutorials-admin-op-enhancements-design.md b/docs/superpowers/specs/2026-08-31-tutorials-admin-op-enhancements-design.md new file mode 100644 index 000000000..d7a5564a7 --- /dev/null +++ b/docs/superpowers/specs/2026-08-31-tutorials-admin-op-enhancements-design.md @@ -0,0 +1,135 @@ +# Tutorials Admin Object Page — Enhancements Design + +**Date:** 2026-08-31 +**Status:** Draft for review +**Scope:** `AdminService.Tutorials` Object Page (Fiori Elements) + supporting CDS model, fetch/publish pipeline, and object-store exposure. + +## Motivation + +The Tutorials admin Object Page has three facets that render **empty tables** despite being wired correctly end-to-end, plus a large body of persisted data that is never surfaced. Investigation (four research passes) found a single recurring root cause for the empties: **the admin UI is correctly wired, but the fetch/publish pipeline never populates the backing entity with data it already holds.** This design closes those gaps and adds high-value read-only facets from data we already persist. + +Example tutorial used for reference: `Tutorials(ID=1350bd47-09eb-5656-a9b9-bdc2575441cb)` on `developers.sap.com/admin-ui/`. + +## Decisions locked (from brainstorming) + +1. **Categories** — one-time backfill **plus** self-healing at publish time. +2. **Validation rules** — **persist** the full parsed `rules.vr` rule set at publish time into a new read-only entity (not on-demand GitHub parse). +3. **Contributors** — populate the **full git-derived contributor list** (up to 10: login, name, email, avatar), each linking to `github.com/`. +4. **Media** — surface object-store items with **rich detail** (explicit user ask). + +## Open decisions (need a call — recommendations inline) + +- **D1 — Object Store prod binding gap.** `package.json` sets `attachments.kind: s3` for `[production]`, but **`mta.yaml` declares no `objectstore` resource and `tutorials-srv` binds none**. Media may be DB-backed (or unpopulated) in prod today. **Recommendation:** the Media *facet* (read-only projection + download links) works regardless of backing store, so ship it; treat the S3 binding as a **separate ops investigation** tracked out of this design. Confirm. +- **D2 — Byte size / image dimensions.** Not persisted today (`buffer.length` is used only for a guard; `probe-image-size` is a devDep but unused in the ingest path). To show size/dimensions we need a small schema + ingest change. **Recommendation:** add `byteSize : Integer64` (cheap, already in memory at ingest) now; defer width/height (needs probe wiring) unless you want it. Confirm which. + +## Workstreams + +Five independently shippable workstreams. Suggested phasing at the end. + +--- + +### WS1 — Categories: populate + self-heal + +**Problem:** The Categories facet reads `TutorialCategories` (AI-classifier-derived junction, cosine ≥ 0.32 / LLM tiebreak). Publish/migration create `Tutorials` rows via **direct `INSERT.into(Tutorials)`** (`srv/lib/content-publish-session.js:748-762`), bypassing the CAP `after('CREATE','Tutorials')` classifier hook (`srv/handlers/categories-after-hooks.js:44-53`). So classification never fires for published tutorials. + +**Design:** +1. **Backfill (one-time):** confirm category seed embeddings exist (admin `embedAllSeeds`), then run `scripts/backfill-categories.cjs` against the target DB. +2. **Self-heal (pipeline):** after `upsertTutorialMetadata`/`linkTutorialAuthorship` in `content-publish-session.js` (~`:198-210`), iterate the already-collected touched `tutorialIds` and call the exported `classifyAndPersist('tutorial', id).catch(warn)` **fire-and-forget** — mirroring the swallow-and-warn dispatcher in `categories-after-hooks.js:34-40`. Never throw into the publish tx. + +**No schema change.** `TutorialCategories`, its projection (`srv/admin-service.cds:174`), `@UI.LineItem`, and the facet already exist. + +**Risk:** classification is LLM-backed on the tiebreak path; publishing many slugs could fan out calls. Mitigate by keeping it fire-and-forget and reusing the existing threshold/short-circuit (most slugs resolve on cosine alone; skips write nothing). + +**Testing:** unit — publish a tutorial through the session, assert `classifyAndPersist` invoked with the new `tutorial_ID`; hybrid — publish + assert `TutorialCategories` rows exist for the slug. + +--- + +### WS2 — Contributors: map git list + GitHub links + +**Problem:** `TutorialContributors` is fully wired (entity `db/schema.cds:451-457`, projection `srv/admin-service.cds:234`, `@UI.LineItem` + facet `app/admin-annotations.cds:715-725,767`) but never populated per-tutorial: IMS migration inserts rows with `tutorial_ID = NULL` (`scripts/migrate-from-hana.js:360`); publish only SELECTs/UPDATEs (`linkTutorialAuthorship`), never INSERTs. The rich git contributor list (`{login, name, email, avatarUrl}`) exists in scope at fetch time (`scripts/fetch-tutorials.ts` ~`:936-948`) and flows to Hugo frontmatter (`render-frontmatter.ts:119`) but never into CAP. + +**Design:** +1. **Schema:** extend `TutorialContributors` with `login : String(255)`, `avatarUrl : String(1024)`, `profileUrl : String(1024)` (derive `profileUrl = https://github.com/`). Keep existing `name`, `email`, `role`, `user`. Add via `cds build --production` migration (never hand-author `.hdbmigrationtable`); register the entity in `db/persistence.cds` if a fresh table/columns need journaling. +2. **Sidecar (fetch):** beside the validate-answer write (`fetch-tutorials.ts:1043`), write `.contributors.json` = `{ slug, contributors: contributors.slice(0,10).map(...) }` using the in-scope array (same `writeFileSync(join(CACHE_DIR, ...))` idiom). +3. **Publish aux step:** mirror `publishValidateAnswerSpecs` — a new client collector in `publish-content.ts` globs `*.contributors.json` and POSTs `{ slug, contributors }` per slug (non-fatal aux step). +4. **Server upsert:** new `srv/lib/contributors-publish.js` following the **REPLACE-per-slug inside `cds.tx()`** pattern from `validate-answer-spec-publish.js`: resolve tutorial by lowercased slug → `DELETE.from(TutorialContributors).where({tutorial_ID})` → `INSERT` the new set. Apply the `entity_not_in_model` fail-fast guard and skip on `channel === 'qa'` if the QA namespace lacks the entity. +5. **UI:** add `login` (as GitHub link via `@Communication.Contact` or a `@UI.LineItem` cell with `@Common.SemanticObject`/URL), `avatarUrl`, and keep name/email/role. Render the avatar + external link to `github.com/`, mirroring the live page (`tutorial-author.html`). + +**srv-qa cp-list:** add `srv/lib/contributors-publish.js` to the `cp` command in `.deploy/mta.yaml` (module `tutorials-srv-qa`, ~`:175`). + +**Testing:** unit — server upsert replaces rows for a slug, leaves other slugs untouched; parser — sidecar shape; hybrid — publish + assert contributor rows with `login`/`avatarUrl` link to the tutorial. + +--- + +### WS3 — Validation: surface ALL rules.vr rules + +**Problem:** The Validation Questions facet is bound to `ValidateAnswerSpecs`, which by design persists **only AI-graded** questions (`collectAiGradedSpecs` drops non-AI at `scripts/parsers/rules.ts:320`; `correctAnswer` is server-only for AI grading). `rules.vr` *is* fully fetched and parsed for every tutorial, but the plain client-graded MCQ/exact-match rules are never persisted. + +**Design:** +1. **New entity** `TutorialValidationRules` (read-only in admin), key `(tutorial, stepNumber, questionId)`, fields: `questionText`, `ruleType`, `questionType` (MCQ/TEXT), `choiceMode` (single/multiple), `options` (LargeString JSON or child rows), `correctAnswer`, `aiGrading : Boolean`. Register in `db/persistence.cds`. + - *Note:* MCQ correct answers for client-graded rules are already public (they ship in Hugo frontmatter `TutorialStep.validation`), so showing them in the admin is not a new leak. AI-graded reference answers remain in `ValidateAnswerSpecs` as today. +2. **Sidecar (fetch):** write `.validation-rules.json` = `{ slug, rules: [...] }` from the **full** `validationMap` (available at `fetch-tutorials.ts:975`), not the AI-filtered subset. Carry `type`, `options`, `choiceMode`, `correctAnswers` in addition to the AI-spec fields. +3. **Publish + server upsert:** same REPLACE-per-slug pattern (new `srv/lib/validation-rules-publish.js`; add to srv-qa cp-list). +4. **UI:** new facet "All Validation Rules" (read-only `@UI.LineItem`) with columns: step, question, type, ruleType, grading (AI/client), correct answer. Keep the existing "Validation Questions" (AI) facet, or relabel it "AI-Graded Validation" for clarity. Confirm whether to keep both or replace. + +**Testing:** parser — sidecar carries all rule types incl. non-AI; server upsert REPLACE semantics; hybrid — publish a tutorial with mixed AI + MCQ rules, assert both appear in `TutorialValidationRules` and only AI ones in `ValidateAnswerSpecs`. + +--- + +### WS4 — Knowledge-Graph facet + +**Problem:** The richest hidden dataset. `TutorialConceptLinks` (teaches/extends + confidence, `db/knowledge-graph.cds:61-70`), `TutorialRank` (PageRank, `:196-200`), `KgCommunity`/`KgCommunityLabel`, and `CoCompletions` ("users who did A also did B") are **not exposed on AdminService** (they live on `KnowledgeGraphService`). + +**Design (read-only, additive):** +1. Expose on `AdminService` as `@readonly` projections + associations from `Tutorials`: + - `conceptLinks` (teaches/prerequisites with confidence) — association already injected on the db entity (`db/knowledge-graph.cds:94-97`). + - `TutorialRank` — surface score as a virtual/flattened field or a small FieldGroup. + - Community label — via `KgCommunity`/`KgCommunityLabel` (KgCommunityMembers already exposed `srv/admin-service.cds:1249`). + - Co-completed neighbors — new read-only projection on `CoCompletions`. + - Use `@cds.redirection.target: false` on pick-list-style projections to avoid stealing redirects (pattern at `srv/admin-service.cds:114,120`). +2. **UI:** a "Knowledge Graph" facet: LineItem of concepts taught (concept, predicate, confidence), a prerequisites list, PageRank + community label in a FieldGroup, and a co-completed-tutorials LineItem. + +**No pipeline change** — these are populated by existing nightly jobs. **Fail-open reads** (mirror existing KG decorators that leave fields unset on SELECT throw). + +**Testing:** unit — projections resolve and are `@readonly`; hybrid — a tutorial with concept links shows them. + +--- + +### WS5 — Media facet + Freshness report header + +**Problem:** `TutorialImages` (`db/tutorial-images.cds`) and `TutorialAssets` (`db/tutorial-assets.cds`) are **not exposed on any service**. `FreshnessReport` is exposed (`srv/admin-service.cds:129`) but the OP only facets findings, not the report header. + +**Design — Media:** +1. **Expose** `@readonly` projections of `TutorialImages` and `TutorialAssets` on `AdminService`, reachable from `Tutorials` (association on matching `slug`, `channel='prod'`). The `content : Composition of many Attachments` child auto-exposes with `@cap-js/attachments` annotations (`@Core.MediaType`, `@Core.ContentDisposition`, `@UI.MediaResource`) → native Fiori **download/preview link**. +2. **Surfaceable detail (persisted today):** `sourceUrl` (GitHub raw URL, render as external link), `contentHash` (sha-256), `mimeType`, `channel`, child `filename` + `createdAt` + `status`. +3. **Optional (D2):** add `byteSize : Integer64` to the schema and persist `buffer.length` at ingest (`srv/lib/image-ingest-handler.js` ~`:77`); width/height deferred unless requested. +4. **UI:** "Media" facet with two tables (Images, Assets): thumbnail/download link, filename, mime, size (if added), source URL link, content hash, channel. + +**Design — Freshness header:** +- Add a "Freshness Report" FieldGroup on the OP sourcing the latest `FreshnessReport` for the tutorial: `runAt`, `model`, `cost`, `status` (QUEUED/RUNNING/DONE/FAILED), `openHighCount`, `error`. Sits above the existing findings facet. + +**Testing:** unit — projections `@readonly`, media child annotations present; hybrid — a tutorial with ingested images shows rows + a working download link; freshness header reflects the latest report row. + +--- + +## Cross-cutting constraints + +- **srv-qa cp-list:** every new `srv/lib/*.js` (contributors-publish, validation-rules-publish) and any new dep MUST be appended to the `cp` command in `.deploy/mta.yaml` (`tutorials-srv-qa`, ~`:175`), or QA boot fails with `MODULE_NOT_FOUND`. +- **Schema/migration:** WS2 + WS3 (+ optional WS5 byteSize) add columns/entities → `cds build --production`, register in `db/persistence.cds` (`@cds.persistence.journal`), never hand-author `.hdbmigrationtable`. Run `npx cds deploy --to sqlite::memory:` before committing db changes. +- **BLOB reads:** any raw content read stays `db.run()` (never mix LOB + metadata in one CDS QL query). +- **QA namespace:** new publish routes need the `entity_not_in_model` guard + skip on `channel === 'qa'` if the QA CDS model lacks the entity. +- **Feature-flag registry:** if any new `*Settings` Boolean column is introduced, register it (guard requirement); none currently planned. +- **PR flow:** feature branch → PR targeting **DEV** (never main). + +## Suggested phasing + +- **Phase 1 (fixes your 3 empties):** WS1 (categories), WS2 (contributors), WS3 (validation). Highest user-visible payoff; shares the sidecar/publish pattern. +- **Phase 2 (new visibility):** WS5 (media + freshness header) — you flagged media as the most exciting. +- **Phase 3:** WS4 (KG facet) — additive, read-only, lowest risk but largest surface. + +Each workstream is independently shippable; phases can overlap. TDD throughout (unit + hybrid guards per workstream). + +## Out of scope (this design) + +- The S3/object-store prod binding investigation (D1) — separate ops task. +- Learner submission pass-rate analytics, learning-path membership facet, audit-field surfacing — noted in the inventory as future candidates, not included here to keep scope focused. From abfe44637006b7cb98eae2807e7e3cb1a07f0919 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 12:44:06 -0400 Subject: [PATCH 04/71] docs(admin): record approved decisions (D1-D3) in OP enhancements spec --- ...26-08-31-tutorials-admin-op-enhancements-design.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/superpowers/specs/2026-08-31-tutorials-admin-op-enhancements-design.md b/docs/superpowers/specs/2026-08-31-tutorials-admin-op-enhancements-design.md index d7a5564a7..799197bd5 100644 --- a/docs/superpowers/specs/2026-08-31-tutorials-admin-op-enhancements-design.md +++ b/docs/superpowers/specs/2026-08-31-tutorials-admin-op-enhancements-design.md @@ -17,10 +17,11 @@ Example tutorial used for reference: `Tutorials(ID=1350bd47-09eb-5656-a9b9-bdc25 3. **Contributors** — populate the **full git-derived contributor list** (up to 10: login, name, email, avatar), each linking to `github.com/`. 4. **Media** — surface object-store items with **rich detail** (explicit user ask). -## Open decisions (need a call — recommendations inline) +## Resolved decisions (approved 2026-08-31) -- **D1 — Object Store prod binding gap.** `package.json` sets `attachments.kind: s3` for `[production]`, but **`mta.yaml` declares no `objectstore` resource and `tutorials-srv` binds none**. Media may be DB-backed (or unpopulated) in prod today. **Recommendation:** the Media *facet* (read-only projection + download links) works regardless of backing store, so ship it; treat the S3 binding as a **separate ops investigation** tracked out of this design. Confirm. -- **D2 — Byte size / image dimensions.** Not persisted today (`buffer.length` is used only for a guard; `probe-image-size` is a devDep but unused in the ingest path). To show size/dimensions we need a small schema + ingest change. **Recommendation:** add `byteSize : Integer64` (cheap, already in memory at ingest) now; defer width/height (needs probe wiring) unless you want it. Confirm which. +- **D1 — Object Store prod binding gap.** `package.json` sets `attachments.kind: s3` for `[production]`, but **`mta.yaml` declares no `objectstore` resource and `tutorials-srv` binds none**. **Resolved:** ship the Media facet (works regardless of backing store); the S3 binding is a **separate ops task**, out of scope here. +- **D2 — Byte size / image dimensions.** **Resolved:** add `byteSize : Integer64` and persist `buffer.length` at ingest now; **defer** width/height (needs `probe-image-size` wiring). +- **D3 — WS3 validation UI.** **Resolved:** keep BOTH facets — relabel the existing AI-only facet "AI-Graded Validation", add the new "All Validation Rules" facet alongside it. ## Workstreams @@ -70,7 +71,7 @@ Five independently shippable workstreams. Suggested phasing at the end. - *Note:* MCQ correct answers for client-graded rules are already public (they ship in Hugo frontmatter `TutorialStep.validation`), so showing them in the admin is not a new leak. AI-graded reference answers remain in `ValidateAnswerSpecs` as today. 2. **Sidecar (fetch):** write `.validation-rules.json` = `{ slug, rules: [...] }` from the **full** `validationMap` (available at `fetch-tutorials.ts:975`), not the AI-filtered subset. Carry `type`, `options`, `choiceMode`, `correctAnswers` in addition to the AI-spec fields. 3. **Publish + server upsert:** same REPLACE-per-slug pattern (new `srv/lib/validation-rules-publish.js`; add to srv-qa cp-list). -4. **UI:** new facet "All Validation Rules" (read-only `@UI.LineItem`) with columns: step, question, type, ruleType, grading (AI/client), correct answer. Keep the existing "Validation Questions" (AI) facet, or relabel it "AI-Graded Validation" for clarity. Confirm whether to keep both or replace. +4. **UI:** new facet "All Validation Rules" (read-only `@UI.LineItem`) with columns: step, question, type, ruleType, grading (AI/client), correct answer. **Keep both facets** (D3): relabel the existing AI facet "AI-Graded Validation"; the new all-rules facet sits alongside it. **Testing:** parser — sidecar carries all rule types incl. non-AI; server upsert REPLACE semantics; hybrid — publish a tutorial with mixed AI + MCQ rules, assert both appear in `TutorialValidationRules` and only AI ones in `ValidateAnswerSpecs`. @@ -102,7 +103,7 @@ Five independently shippable workstreams. Suggested phasing at the end. **Design — Media:** 1. **Expose** `@readonly` projections of `TutorialImages` and `TutorialAssets` on `AdminService`, reachable from `Tutorials` (association on matching `slug`, `channel='prod'`). The `content : Composition of many Attachments` child auto-exposes with `@cap-js/attachments` annotations (`@Core.MediaType`, `@Core.ContentDisposition`, `@UI.MediaResource`) → native Fiori **download/preview link**. 2. **Surfaceable detail (persisted today):** `sourceUrl` (GitHub raw URL, render as external link), `contentHash` (sha-256), `mimeType`, `channel`, child `filename` + `createdAt` + `status`. -3. **Optional (D2):** add `byteSize : Integer64` to the schema and persist `buffer.length` at ingest (`srv/lib/image-ingest-handler.js` ~`:77`); width/height deferred unless requested. +3. **Byte size (D2):** add `byteSize : Integer64` to the schema and persist `buffer.length` at ingest (`srv/lib/image-ingest-handler.js` ~`:77`); width/height deferred. 4. **UI:** "Media" facet with two tables (Images, Assets): thumbnail/download link, filename, mime, size (if added), source URL link, content hash, channel. **Design — Freshness header:** From 7f569ac2cbdfc1f937b4edd4169fd5824c9ff17e Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 12:48:33 -0400 Subject: [PATCH 05/71] docs(admin): Phase 1 implementation plan (WS1-3 empty facets) --- ...-tutorials-admin-op-phase1-empty-facets.md | 1215 +++++++++++++++++ 1 file changed, 1215 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-31-tutorials-admin-op-phase1-empty-facets.md diff --git a/docs/superpowers/plans/2026-08-31-tutorials-admin-op-phase1-empty-facets.md b/docs/superpowers/plans/2026-08-31-tutorials-admin-op-phase1-empty-facets.md new file mode 100644 index 000000000..b2ee34f24 --- /dev/null +++ b/docs/superpowers/plans/2026-08-31-tutorials-admin-op-phase1-empty-facets.md @@ -0,0 +1,1215 @@ +# Tutorials Admin OP — Phase 1 (Empty Facets) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Populate the three empty Tutorials Object Page facets — Categories, Contributors, Validation Questions — by closing the fetch/publish pipeline gaps that leave their backing entities unpopulated. + +**Architecture:** Reuse the existing "fetch-sidecar → publish aux step → server REPLACE-per-slug inside `cds.tx()`" pattern (proven by `validate-answer-spec-publish.js`). Contributors and full validation rules get new `..json` cache sidecars written in the same `fetch-tutorials.ts` per-tutorial pass, POSTed by new client collectors in `publish-content.ts`, and upserted by new `srv/lib/*-publish.js` handlers. Categories are fixed by a fire-and-forget `classifyAndPersist('tutorial', id)` call after publish upserts a tutorial (publish bypasses the CAP `after('CREATE')` hook), plus a one-time backfill. + +**Tech Stack:** SAP CAP (Node.js, CDS), HANA (prod) / SQLite (unit), Fiori Elements annotations (`app/admin-annotations.cds`), Vitest (unit + hybrid projects). + +**Spec:** `docs/superpowers/specs/2026-08-31-tutorials-admin-op-enhancements-design.md` + +## Global Constraints + +- **Slugs are lowercase-canonical** — lowercase in both sidecar filename and JSON body; server resolves tutorial by lowercased slug. +- **REPLACE-per-slug** — publishing slug A must never touch slug B's rows. DELETE-by-`tutorial_ID` then INSERT inside `cds.tx()`. +- **srv-qa cp-list** — every new `srv/lib/*.js` MUST be appended to the `cp` command in `.deploy/mta.yaml` module `tutorials-srv-qa` (~line 175), or QA boot fails with `MODULE_NOT_FOUND`. +- **QA namespace guard** — new publish routes apply the `entity_not_in_model` fail-fast guard and skip the aux step on `channel === 'qa'` if the QA CDS model lacks the entity. +- **Never throw into the publish/completion tx** — fire-and-forget classification must `.catch(warn)`. +- **Schema changes** — `cds build --production`; register new persisted entities in `db/persistence.cds` (`@cds.persistence.journal`); never hand-author `.hdbmigrationtable`; run `npx cds deploy --to sqlite::memory:` before committing db changes. +- **BLOB reads stay raw `db.run()`** — never mix LOB + metadata in one CDS QL query (not expected in Phase 1). +- **Aux publish steps are non-fatal** — a sidecar publish failure warns, never fails the deploy. +- **Tests:** unit via `npm test` (in-memory SQLite); hybrid via `npm run test:hybrid` (real HANA, `--project hybrid`). Bare `vitest ` skips hybrid setup. +- **PR targets DEV, never main.** + +--- + +## File Structure + +**WS1 — Categories self-heal + backfill:** +- Modify: `srv/lib/content-publish-session.js` — add fire-and-forget `classifyAndPersist` loop over touched `tutorialIds` after the metadata/authorship block. +- Test: `test/unit/publish-category-selfheal.test.js`, `test/hybrid/publish-categories.test.js`. +- Ops: run `scripts/backfill-categories.cjs` (existing). + +**WS2 — Contributors:** +- Modify: `db/schema.cds` — add `login`/`avatarUrl`/`profileUrl` to `TutorialContributors`. +- Modify: `db/persistence.cds` — ensure `TutorialContributors` journaled. +- Modify: `scripts/fetch-tutorials.ts` — write `.contributors.json` sidecar. +- Create: `scripts/publish/publish-contributors.ts` (client collector), wired into `scripts/publish-content.ts`. +- Create: `srv/lib/contributors-publish.js` (server REPLACE handler). +- Modify: `srv/server.js` — mount the publish route. +- Modify: `app/admin-annotations.cds` — add `login` (GitHub link) + `avatarUrl` to `TutorialContributors` `@UI.LineItem`. +- Modify: `.deploy/mta.yaml` — add `contributors-publish.js` to srv-qa cp-list. +- Test: `test/unit/contributors-publish.test.js`, `test/hybrid/publish-contributors.test.js`. + +**WS3 — All validation rules:** +- Modify: `db/schema.cds` — add `TutorialValidationRules` entity. +- Modify: `db/persistence.cds` — journal `TutorialValidationRules`. +- Modify: `srv/admin-service.cds` — projection + `validationRules` association on `Tutorials`. +- Modify: `scripts/parsers/rules.ts` — add `collectAllRules()` alongside `collectAiGradedSpecs()`. +- Modify: `scripts/fetch-tutorials.ts` — write `.validation-rules.json` sidecar. +- Create: `scripts/publish/publish-validation-rules.ts` + wire into `publish-content.ts`. +- Create: `srv/lib/validation-rules-publish.js`. +- Modify: `srv/server.js` — mount route. +- Modify: `app/admin-annotations.cds` — relabel AI facet "AI-Graded Validation"; add "All Validation Rules" facet + LineItem. +- Modify: `.deploy/mta.yaml` — add `validation-rules-publish.js` to srv-qa cp-list. +- Test: `test/unit/collect-all-rules.test.js`, `test/unit/validation-rules-publish.test.js`, `test/hybrid/publish-validation-rules.test.js`. + +--- + +## WS1 — Categories: self-heal at publish + backfill + +### Task 1: Fire-and-forget classification after publish upsert + +**Files:** +- Modify: `srv/lib/content-publish-session.js` (~line 198-210, after `upsertTutorialMetadata`/`linkTutorialAuthorship`; `tutorialIds` already collected/returned per research) +- Test: `test/unit/publish-category-selfheal.test.js` + +**Interfaces:** +- Consumes: `classifyAndPersist(kind, id, _opts?)` — named export from `srv/lib/category-classifier.js:127`. +- Produces: nothing new; side effect is `TutorialCategories` rows for published tutorials. + +- [ ] **Step 1: Write the failing test** + +```js +// test/unit/publish-category-selfheal.test.js +import { describe, it, expect, vi, beforeEach } from 'vitest' + +// Mock the classifier so the test asserts invocation without real embeddings/LLM. +const classifySpy = vi.fn().mockResolvedValue(undefined) +vi.mock('../../srv/lib/category-classifier.js', () => ({ + classifyAndPersist: classifySpy, +})) + +import { classifyTouchedTutorials } from '../../srv/lib/content-publish-session.js' + +describe('publish category self-heal', () => { + beforeEach(() => classifySpy.mockClear()) + + it('classifies every touched tutorial id, fire-and-forget', async () => { + await classifyTouchedTutorials(['id-a', 'id-b']) + expect(classifySpy).toHaveBeenCalledTimes(2) + expect(classifySpy).toHaveBeenCalledWith('tutorial', 'id-a') + expect(classifySpy).toHaveBeenCalledWith('tutorial', 'id-b') + }) + + it('never rejects even if a classification throws', async () => { + classifySpy.mockRejectedValueOnce(new Error('boom')) + await expect(classifyTouchedTutorials(['id-a'])).resolves.toBeUndefined() + }) +}) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/unit/publish-category-selfheal.test.js --project unit` +Expected: FAIL — `classifyTouchedTutorials` is not exported. + +- [ ] **Step 3: Implement the helper + call site** + +In `srv/lib/content-publish-session.js`, add near the top (after existing imports): + +```js +const { classifyAndPersist } = require('./category-classifier.js') + +// Exported for unit testing; classifies touched tutorials without ever throwing +// into the publish tx (publish bypasses the CAP after('CREATE') classifier hook). +async function classifyTouchedTutorials(tutorialIds) { + await Promise.all( + (tutorialIds || []).map((id) => + Promise.resolve() + .then(() => classifyAndPersist('tutorial', id)) + .catch((e) => console.warn('[publish] category classify skipped', id, e?.message)), + ), + ) +} +module.exports.classifyTouchedTutorials = classifyTouchedTutorials +``` + +> Match the file's existing module system. Research shows `category-classifier.js` uses ES named exports; if `content-publish-session.js` is CommonJS, use dynamic `import()` inside the helper instead of top-level `require`. Verify the first two lines of `content-publish-session.js` before choosing. + +Then at the post-metadata call site (~line 198-210, where `tutorialIds` is in scope): + +```js +// Fire-and-forget: keep categories populated for publish-created tutorials. +classifyTouchedTutorials(tutorialIds) +``` + +(No `await` — must not block or fail the publish.) + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/unit/publish-category-selfheal.test.js --project unit` +Expected: PASS (both cases). + +- [ ] **Step 5: Commit** + +```bash +git add srv/lib/content-publish-session.js test/unit/publish-category-selfheal.test.js +git commit -m "feat(publish): self-heal categories via classifyAndPersist after upsert (#WS1)" +``` + +### Task 2: Hybrid guard — published tutorial gets categories + +**Files:** +- Test: `test/hybrid/publish-categories.test.js` + +**Interfaces:** +- Consumes: real `AdminService` + publish session against HANA (via `cds bind --exec`). + +- [ ] **Step 1: Write the hybrid test** + +```js +// test/hybrid/publish-categories.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' + +describe('publish populates categories (hybrid)', () => { + let db + beforeAll(async () => { db = await cds.connect.to('db') }) + + it('a freshly published tutorial has >= 0 category rows and no orphan write errors', async () => { + // Precondition: category seed embeddings must exist in this env. + const { Categories } = cds.entities('com.sap.developers.ims') + const seeds = await db.run(SELECT.from(Categories)) + expect(seeds.length).toBeGreaterThan(0) // else run embedAllSeeds first + + // Assert the classifier is reachable and idempotent for a known slug. + // (Use a slug known to exist in the bound DB.) + const { Tutorials, TutorialCategories } = cds.entities('com.sap.developers.ims') + const t = await db.run(SELECT.one.from(Tutorials).columns('ID', 'slug')) + expect(t).toBeTruthy() + const rows = await db.run(SELECT.from(TutorialCategories).where({ tutorial_ID: t.ID })) + expect(Array.isArray(rows)).toBe(true) + }) +}) +``` + +- [ ] **Step 2: Run the hybrid test** + +Run: `npm run test:hybrid -- test/hybrid/publish-categories.test.js` +Expected: PASS if seed embeddings exist. If `seeds.length === 0`, run the `embedAllSeeds` admin action first, then re-run. + +- [ ] **Step 3: Commit** + +```bash +git add test/hybrid/publish-categories.test.js +git commit -m "test(publish): hybrid guard for category population (#WS1)" +``` + +### Task 3: One-time backfill (ops step — documented, not code) + +**Files:** none (uses existing `scripts/backfill-categories.cjs`). + +- [ ] **Step 1: Confirm seed embeddings exist** — via admin action `embedAllSeeds` on the target env, or query `Categories` seed rows. +- [ ] **Step 2: Dry-run then run backfill** + +Run (against bound env): `node scripts/backfill-categories.cjs --dry-run` then without the flag. +Expected: rows inserted into `TutorialCategories` for previously-empty tutorials. + +- [ ] **Step 3: Spot-check in admin UI** — open the reference tutorial's Categories facet; confirm rows render. + +--- + +## WS2 — Contributors: map git list + GitHub links + +### Task 4: Schema — add GitHub columns to TutorialContributors + +**Files:** +- Modify: `db/schema.cds:451-457` (`TutorialContributors`) +- Modify: `db/persistence.cds` (ensure journaled) +- Test: `test/unit/schema-contributors.test.js` + +**Interfaces:** +- Produces: `TutorialContributors` now has `login : String(255)`, `avatarUrl : String(1024)`, `profileUrl : String(1024)`. + +- [ ] **Step 1: Write the failing test** + +```js +// test/unit/schema-contributors.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' + +describe('TutorialContributors schema', () => { + let m + beforeAll(async () => { m = await cds.load('*') }) + it('has GitHub link columns', () => { + const e = m.definitions['com.sap.developers.ims.TutorialContributors'] + expect(e.elements.login).toBeTruthy() + expect(e.elements.avatarUrl).toBeTruthy() + expect(e.elements.profileUrl).toBeTruthy() + }) +}) +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `npx vitest run test/unit/schema-contributors.test.js --project unit` +Expected: FAIL — elements undefined. + +- [ ] **Step 3: Add the columns** + +In `db/schema.cds`, extend the `TutorialContributors` entity body: + +```cds +entity TutorialContributors : cuid, LegacyKeyed { + tutorial : Association to Tutorials; + name : String(255); + email : String(255); + role : String(50); + login : String(255); // GitHub handle + avatarUrl : String(1024); // https://github.com/.png + profileUrl : String(1024); // https://github.com/ + user : Association to Users; +} +``` + +Confirm `db/persistence.cds` journals `TutorialContributors` (add `@cds.persistence.journal` registration entry if a new column set requires a migration table — follow the existing entries' shape). + +- [ ] **Step 4: Verify schema + deploy dry-run** + +Run: `npx vitest run test/unit/schema-contributors.test.js --project unit` +Then: `npx cds deploy --to sqlite::memory:` +Expected: test PASS; deploy succeeds with no errors. + +- [ ] **Step 5: Build migration + commit** + +```bash +npx cds build --production +git add db/schema.cds db/persistence.cds db/src/gen test/unit/schema-contributors.test.js +git commit -m "feat(db): add GitHub link columns to TutorialContributors (#WS2)" +``` + +### Task 5: Fetch sidecar — write `.contributors.json` + +**Files:** +- Modify: `scripts/fetch-tutorials.ts` (~line 1043, beside the validate-answer sidecar write; `contributors` array in scope from ~`:936-948`) +- Test: `test/unit/contributors-sidecar.test.js` + +**Interfaces:** +- Produces: cache file `.contributors.json` = `{ slug, contributors: Array<{login,name,email,avatarUrl}> }` (max 10). + +- [ ] **Step 1: Write the failing test** (extract a pure helper to keep it testable) + +```js +// test/unit/contributors-sidecar.test.js +import { describe, it, expect } from 'vitest' +import { buildContributorsSidecar } from '../../scripts/parsers/contributors-sidecar' + +describe('buildContributorsSidecar', () => { + it('lowercases slug and caps at 10', () => { + const contribs = Array.from({ length: 12 }, (_, i) => ({ + login: `u${i}`, name: `N${i}`, email: `${i}@x.com`, avatarUrl: `a${i}`, + })) + const out = buildContributorsSidecar('My-Slug', contribs) + expect(out.slug).toBe('my-slug') + expect(out.contributors).toHaveLength(10) + expect(out.contributors[0]).toEqual({ login: 'u0', name: 'N0', email: '0@x.com', avatarUrl: 'a0' }) + }) + it('returns null when no contributors', () => { + expect(buildContributorsSidecar('s', [])).toBeNull() + }) +}) +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `npx vitest run test/unit/contributors-sidecar.test.js --project unit` +Expected: FAIL — module not found. + +- [ ] **Step 3: Implement the helper + wire the write** + +Create `scripts/parsers/contributors-sidecar.ts`: + +```ts +export interface SidecarContributor { login: string; name: string; email: string; avatarUrl: string } +export interface ContributorsSidecar { slug: string; contributors: SidecarContributor[] } + +export function buildContributorsSidecar( + slug: string, + contributors: Array>, +): ContributorsSidecar | null { + if (!contributors || contributors.length === 0) return null + return { + slug: slug.toLowerCase(), + contributors: contributors.slice(0, 10).map((c) => ({ + login: c.login ?? '', name: c.name ?? '', email: c.email ?? '', avatarUrl: c.avatarUrl ?? '', + })), + } +} +``` + +In `scripts/fetch-tutorials.ts`, beside the validate-answer write (~`:1043`): + +```ts +import { buildContributorsSidecar } from './parsers/contributors-sidecar' +// ... +const contribSidecar = buildContributorsSidecar(t.slug, contributors) +if (contribSidecar) { + writeFileSync( + join(CACHE_DIR, `${t.slug.toLowerCase()}.contributors.json`), + JSON.stringify(contribSidecar, null, 2), + ) +} +``` + +- [ ] **Step 4: Run to verify it passes** + +Run: `npx vitest run test/unit/contributors-sidecar.test.js --project unit` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add scripts/parsers/contributors-sidecar.ts scripts/fetch-tutorials.ts test/unit/contributors-sidecar.test.js +git commit -m "feat(fetch): write contributors sidecar from git contributor list (#WS2)" +``` + +### Task 6: Server handler — REPLACE contributors per slug + +**Files:** +- Create: `srv/lib/contributors-publish.js` +- Modify: `srv/server.js` (mount route, mirror validate-answer mount) +- Modify: `.deploy/mta.yaml` (~line 175, srv-qa cp-list) +- Test: `test/unit/contributors-publish.test.js` + +**Interfaces:** +- Consumes: POST body `{ slug, contributors: [{login,name,email,avatarUrl}] }`. +- Produces: `publishContributors(req, res)` Express handler; REPLACE-by-`tutorial_ID` in `TutorialContributors`. + +- [ ] **Step 1: Write the failing test** (mirror `validate-answer-spec-publish` test shape, in-memory SQLite) + +```js +// test/unit/contributors-publish.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +import { replaceContributorsForSlug } from '../../srv/lib/contributors-publish.js' + +describe('replaceContributorsForSlug', () => { + let db + beforeAll(async () => { + await cds.test('serve', '--in-memory').in(process.cwd()) + db = await cds.connect.to('db') + }) + + it('replaces rows for the slug and derives profileUrl', async () => { + const { Tutorials, TutorialContributors } = cds.entities('com.sap.developers.ims') + const ID = cds.utils.uuid() + await db.run(INSERT.into(Tutorials).entries({ ID, slug: 'demo', title: 'Demo' })) + + await replaceContributorsForSlug(db, 'DEMO', [ + { login: 'octocat', name: 'Octo Cat', email: 'o@x.com', avatarUrl: 'https://github.com/octocat.png' }, + ]) + let rows = await db.run(SELECT.from(TutorialContributors).where({ tutorial_ID: ID })) + expect(rows).toHaveLength(1) + expect(rows[0].login).toBe('octocat') + expect(rows[0].profileUrl).toBe('https://github.com/octocat') + + // Second publish REPLACES, does not append. + await replaceContributorsForSlug(db, 'demo', [ + { login: 'hubot', name: 'Hubot', email: 'h@x.com', avatarUrl: 'https://github.com/hubot.png' }, + ]) + rows = await db.run(SELECT.from(TutorialContributors).where({ tutorial_ID: ID })) + expect(rows).toHaveLength(1) + expect(rows[0].login).toBe('hubot') + }) +}) +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `npx vitest run test/unit/contributors-publish.test.js --project unit` +Expected: FAIL — module not found. + +- [ ] **Step 3: Implement the handler** + +Create `srv/lib/contributors-publish.js` (mirror `validate-answer-spec-publish.js` structure — namespace guard, per-slug REPLACE in `cds.tx()`): + +```js +const cds = require('@sap/cds') +const NS = 'com.sap.developers.ims' +const MAX_FIELD_BYTES = 2000 + +function githubProfileUrl(login) { + return login ? `https://github.com/${login}` : null +} + +// Core, unit-testable: REPLACE all contributor rows for one slug. +async function replaceContributorsForSlug(db, slug, contributors) { + const { Tutorials, TutorialContributors } = cds.entities(NS) + const lcSlug = String(slug || '').toLowerCase() + const tut = await db.run(SELECT.one.from(Tutorials).columns('ID').where({ slug: lcSlug })) + if (!tut) return { ok: false, reason: 'tutorial_not_found', slug: lcSlug } + + const entries = (contributors || []) + .filter((c) => c && (c.login || c.name || c.email)) + .slice(0, 10) + .map((c) => ({ + ID: cds.utils.uuid(), + tutorial_ID: tut.ID, + login: (c.login || '').slice(0, 255), + name: (c.name || '').slice(0, 255), + email: (c.email || '').slice(0, 255), + avatarUrl: (c.avatarUrl || '').slice(0, 1024), + profileUrl: githubProfileUrl(c.login), + })) + + await cds.tx(async (tx) => { + await tx.run(DELETE.from(TutorialContributors).where({ tutorial_ID: tut.ID })) + if (entries.length) await tx.run(INSERT.into(TutorialContributors).entries(entries)) + }) + return { ok: true, slug: lcSlug, count: entries.length } +} + +// Express handler mirroring validate-answer-spec-publish route. +async function publishContributors(req, res) { + try { + const { slug, contributors } = req.body || {} + if (!slug || !Array.isArray(contributors)) { + return res.status(400).json({ error: 'bad_request', detail: 'expected { slug, contributors[] }' }) + } + let entities + try { entities = cds.entities(NS) } catch { entities = null } + if (!entities || !entities.TutorialContributors) { + return res.status(409).json({ error: 'entity_not_in_model' }) + } + const db = await cds.connect.to('db') + const result = await replaceContributorsForSlug(db, slug, contributors) + if (!result.ok) return res.status(404).json(result) + return res.json(result) + } catch (e) { + return res.status(500).json({ error: 'internal', detail: e?.message }) + } +} + +module.exports = { replaceContributorsForSlug, publishContributors, githubProfileUrl } +``` + +Mount in `srv/server.js` beside the validate-answer route (guard with the same `CONTENT_API_KEY` middleware the other publish routes use): + +```js +const { publishContributors } = require('./lib/contributors-publish.js') +app.post('/content/publish-contributors', requireContentApiKey, express.json({ limit: '1mb' }), publishContributors) +``` + +> Verify the exact auth-middleware name and JSON body parser used by the existing `/content/publish` + validate-answer routes and match it. + +- [ ] **Step 4: Run to verify it passes** + +Run: `npx vitest run test/unit/contributors-publish.test.js --project unit` +Expected: PASS (both replace + derive assertions). + +- [ ] **Step 5: Add to srv-qa cp-list** + +In `.deploy/mta.yaml`, module `tutorials-srv-qa` `cp` command (~line 175), append `../../srv/lib/contributors-publish.js` to the `srv/lib/` copy segment. + +- [ ] **Step 6: Commit** + +```bash +git add srv/lib/contributors-publish.js srv/server.js .deploy/mta.yaml test/unit/contributors-publish.test.js +git commit -m "feat(publish): server REPLACE handler for TutorialContributors (#WS2)" +``` + +### Task 7: Client publish step + wire into publish-content + +**Files:** +- Create: `scripts/publish/publish-contributors.ts` +- Modify: `scripts/publish-content.ts` (~line 1327-1352, beside `publishValidateAnswerSpecs`; non-fatal aux step) +- Test: `test/unit/publish-contributors-client.test.js` + +**Interfaces:** +- Consumes: cache dir globbed for `*.contributors.json`; POSTs each to `/content/publish-contributors`. +- Produces: `publishContributors({ cacheDir, baseUrl, apiKey })`. + +- [ ] **Step 1: Write the failing test** (mock `fetch`) + +```js +// test/unit/publish-contributors-client.test.js +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest' +import { mkdtempSync, writeFileSync } from 'node:fs' +import { join } from 'node:path' +import { tmpdir } from 'node:os' +import { publishContributors } from '../../scripts/publish/publish-contributors' + +describe('publishContributors client', () => { + let dir + beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), 'contrib-')) + writeFileSync(join(dir, 'demo.contributors.json'), + JSON.stringify({ slug: 'demo', contributors: [{ login: 'octocat', name: 'O', email: 'o@x', avatarUrl: 'a' }] })) + global.fetch = vi.fn().mockResolvedValue({ ok: true, json: async () => ({ ok: true, count: 1 }) }) + }) + afterEach(() => { vi.restoreAllMocks() }) + + it('POSTs each sidecar to the endpoint', async () => { + const res = await publishContributors({ cacheDir: dir, baseUrl: 'http://x', apiKey: 'k' }) + expect(global.fetch).toHaveBeenCalledTimes(1) + const [url, opts] = global.fetch.mock.calls[0] + expect(url).toBe('http://x/content/publish-contributors') + expect(JSON.parse(opts.body).slug).toBe('demo') + expect(res.published).toBe(1) + }) +}) +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `npx vitest run test/unit/publish-contributors-client.test.js --project unit` +Expected: FAIL — module not found. + +- [ ] **Step 3: Implement + wire** + +Create `scripts/publish/publish-contributors.ts`: + +```ts +import { readdirSync, readFileSync } from 'node:fs' +import { join } from 'node:path' + +export async function publishContributors(opts: { cacheDir: string; baseUrl: string; apiKey: string }) { + const { cacheDir, baseUrl, apiKey } = opts + const files = readdirSync(cacheDir).filter((f) => f.endsWith('.contributors.json')) + let published = 0 + for (const f of files) { + const body = readFileSync(join(cacheDir, f), 'utf8') + const res = await fetch(`${baseUrl}/content/publish-contributors`, { + method: 'POST', + headers: { 'content-type': 'application/json', 'x-api-key': apiKey }, + body, + }) + if (res.ok) published += 1 + else console.warn(`[publish-contributors] ${f} -> ${res.status}`) + } + return { published, total: files.length } +} +``` + +> Match the exact auth header name the existing publish client uses (research: validate-answer client — confirm `x-api-key` vs `authorization`). + +In `scripts/publish-content.ts`, beside `publishValidateAnswerSpecs` (~`:1327`), add a non-fatal aux step, skipping QA channel: + +```ts +if (channel !== 'qa') { + try { + const r = await publishContributors({ cacheDir: CACHE_DIR, baseUrl, apiKey }) + console.log(`[publish] contributors: ${r.published}/${r.total}`) + } catch (e) { + console.warn('[publish] contributors step failed (non-fatal)', (e as Error).message) + } +} +``` + +- [ ] **Step 4: Run to verify it passes** + +Run: `npx vitest run test/unit/publish-contributors-client.test.js --project unit` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add scripts/publish/publish-contributors.ts scripts/publish-content.ts test/unit/publish-contributors-client.test.js +git commit -m "feat(publish): non-fatal client step to publish contributors sidecars (#WS2)" +``` + +### Task 8: UI — GitHub-linked Contributors LineItem + +**Files:** +- Modify: `app/admin-annotations.cds:715-725` (`TutorialContributors` `@UI.LineItem`) +- Test: `test/unit/annotations-contributors.test.js` + +**Interfaces:** +- Consumes: `TutorialContributors.login`/`avatarUrl`/`profileUrl` (Task 4). + +- [ ] **Step 1: Write the failing test** + +```js +// test/unit/annotations-contributors.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' + +describe('Contributors LineItem', () => { + let m + beforeAll(async () => { m = await cds.load('*') }) + it('LineItem includes login column', () => { + const e = m.definitions['AdminService.TutorialContributors'] + const li = e['@UI.LineItem'] + const values = li.map((x) => x.Value?.['='] || x.Value) + expect(values).toContain('login') + }) +}) +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `npx vitest run test/unit/annotations-contributors.test.js --project unit` +Expected: FAIL — `login` not in LineItem. + +- [ ] **Step 3: Add columns + GitHub link** + +In `app/admin-annotations.cds`, extend the `TutorialContributors` `@UI.LineItem` (add `login` and render it as an external link to `profileUrl`): + +```cds +annotate AdminService.TutorialContributors with @( + UI.LineItem: [ + { Value: name, Label: 'Name' }, + { Value: login, Label: 'GitHub', @HTML5.LinkTarget: '_blank' }, + { Value: email, Label: 'Email' }, + { Value: role, Label: 'Role' } + ] +); +annotate AdminService.TutorialContributors with { + login @Common.Text: profileUrl @Common.TextArrangement: #TextOnly; +}; +``` + +> Preferred: make `login` a link via a `DataFieldWithUrl` pointing at `profileUrl` so the cell navigates to `github.com/`: +> ```cds +> { $Type: 'UI.DataFieldWithUrl', Value: login, Url: profileUrl, Label: 'GitHub' } +> ``` +> Use whichever renders as a clickable GitHub link in the current FE version; verify against the running admin UI. + +- [ ] **Step 4: Run to verify it passes** + +Run: `npx vitest run test/unit/annotations-contributors.test.js --project unit` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add app/admin-annotations.cds test/unit/annotations-contributors.test.js +git commit -m "feat(admin-ui): GitHub-linked login column on Contributors table (#WS2)" +``` + +### Task 9: Hybrid guard — publish links contributors to tutorial + +**Files:** +- Test: `test/hybrid/publish-contributors.test.js` + +- [ ] **Step 1: Write the hybrid test** + +```js +// test/hybrid/publish-contributors.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +import { replaceContributorsForSlug } from '../../srv/lib/contributors-publish.js' + +describe('contributors publish (hybrid)', () => { + let db + beforeAll(async () => { db = await cds.connect.to('db') }) + it('links contributor rows to an existing tutorial by slug', async () => { + const { Tutorials, TutorialContributors } = cds.entities('com.sap.developers.ims') + const t = await db.run(SELECT.one.from(Tutorials).columns('ID', 'slug')) + expect(t).toBeTruthy() + await replaceContributorsForSlug(db, t.slug, [ + { login: 'octocat', name: 'Octo', email: 'o@x.com', avatarUrl: 'https://github.com/octocat.png' }, + ]) + const rows = await db.run(SELECT.from(TutorialContributors).where({ tutorial_ID: t.ID, login: 'octocat' })) + expect(rows.length).toBe(1) + expect(rows[0].profileUrl).toBe('https://github.com/octocat') + // cleanup + await db.run(DELETE.from(TutorialContributors).where({ tutorial_ID: t.ID, login: 'octocat' })) + }) +}) +``` + +- [ ] **Step 2: Run** `npm run test:hybrid -- test/hybrid/publish-contributors.test.js` — Expected: PASS. +- [ ] **Step 3: Commit** + +```bash +git add test/hybrid/publish-contributors.test.js +git commit -m "test(publish): hybrid guard for contributor linking (#WS2)" +``` + +--- + +## WS3 — All validation rules + +### Task 10: Schema — `TutorialValidationRules` entity + +**Files:** +- Modify: `db/schema.cds` (add entity near `ValidateAnswerSpecs` ~`:865`) +- Modify: `db/persistence.cds` (journal it) +- Test: `test/unit/schema-validation-rules.test.js` + +**Interfaces:** +- Produces: `com.sap.developers.ims.TutorialValidationRules` with key `(tutorial, stepNumber, questionId)`, fields `questionText`, `ruleType`, `questionType`, `choiceMode`, `options` (LargeString JSON), `correctAnswer`, `aiGrading : Boolean`. + +- [ ] **Step 1: Write the failing test** + +```js +// test/unit/schema-validation-rules.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +describe('TutorialValidationRules schema', () => { + let m + beforeAll(async () => { m = await cds.load('*') }) + it('exists with expected elements', () => { + const e = m.definitions['com.sap.developers.ims.TutorialValidationRules'] + expect(e).toBeTruthy() + for (const k of ['stepNumber','questionId','questionText','ruleType','questionType','choiceMode','options','correctAnswer','aiGrading']) + expect(e.elements[k]).toBeTruthy() + }) +}) +``` + +- [ ] **Step 2: Run to verify it fails** — `npx vitest run test/unit/schema-validation-rules.test.js --project unit` → FAIL. + +- [ ] **Step 3: Add the entity** + +In `db/schema.cds`: + +```cds +entity TutorialValidationRules { + key tutorial : Association to Tutorials; + key stepNumber : Integer; + key questionId : String(100); + questionText : String(2000); + ruleType : String(50); // single-choice | multiple-choice | regex | exact-match | ... + questionType : String(20); // MCQ | TEXT + choiceMode : String(20); // single | multiple | null + options : LargeString; // JSON array of option strings (MCQ) or null + correctAnswer: LargeString; // reference answer (client-graded) or null when aiGrading + aiGrading : Boolean default false; +} +``` + +Register in `db/persistence.cds` mirroring the existing entries' `@cds.persistence.journal` shape. + +- [ ] **Step 4: Verify + deploy dry-run** — `npx vitest run test/unit/schema-validation-rules.test.js --project unit` then `npx cds deploy --to sqlite::memory:` → PASS + clean deploy. + +- [ ] **Step 5: Build migration + commit** + +```bash +npx cds build --production +git add db/schema.cds db/persistence.cds db/src/gen test/unit/schema-validation-rules.test.js +git commit -m "feat(db): add TutorialValidationRules entity for all rules.vr rules (#WS3)" +``` + +### Task 11: Parser — `collectAllRules()` + +**Files:** +- Modify: `scripts/parsers/rules.ts` (add beside `collectAiGradedSpecs` ~`:312`) +- Test: `test/unit/collect-all-rules.test.js` + +**Interfaces:** +- Consumes: `validationMap`, `ruleTypeByStepAndId`, `correctAnswerByStepAndId` (from `parseRulesVrEnriched`). +- Produces: `collectAllRules(map, ruleTypeMap, answerMap) => Array<{ stepNumber, questionId, questionText, ruleType, questionType, choiceMode, options, correctAnswer, aiGrading }>`. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/unit/collect-all-rules.test.ts +import { describe, it, expect } from 'vitest' +import { collectAllRules } from '../../scripts/parsers/rules' + +describe('collectAllRules', () => { + it('includes non-AI MCQ rules with options + correctAnswer', () => { + const map = new Map([[1, [ + { id: 'validate-1', question: 'Pick one', type: 'QUESTION_TYPE_MCQ', options: ['A','B'], choiceMode: 'single', correctAnswer: 'A' }, + { id: 'validate-1b', question: 'AI graded', type: 'QUESTION_TYPE_TEXT', aiGrading: true }, + ]]]) + const ruleTypeMap = new Map([['1::validate-1', 'single-choice'], ['1::validate-1b', 'regex']]) + const answerMap = new Map([['1::validate-1', 'A']]) + const rows = collectAllRules(map, ruleTypeMap, answerMap) + expect(rows).toHaveLength(2) + const mcq = rows.find((r) => r.questionId === 'validate-1') + expect(mcq.aiGrading).toBe(false) + expect(mcq.questionType).toBe('MCQ') + expect(JSON.parse(mcq.options)).toEqual(['A','B']) + expect(mcq.correctAnswer).toBe('A') + const ai = rows.find((r) => r.questionId === 'validate-1b') + expect(ai.aiGrading).toBe(true) + expect(ai.correctAnswer).toBeNull() + }) +}) +``` + +> Confirm the exact key format of `ruleTypeByStepAndId`/`correctAnswerByStepAndId` in `rules.ts` (research indicated a `step::id` style). Adjust the test's key strings to match the real format before implementing. + +- [ ] **Step 2: Run to verify it fails** — `npx vitest run test/unit/collect-all-rules.test.ts --project unit` → FAIL. + +- [ ] **Step 3: Implement `collectAllRules`** + +In `scripts/parsers/rules.ts` (adapt key access to the confirmed map format): + +```ts +export interface AllRuleRow { + stepNumber: number; questionId: string; questionText: string; + ruleType: string; questionType: 'MCQ' | 'TEXT'; choiceMode: string | null; + options: string | null; correctAnswer: string | null; aiGrading: boolean; +} + +export function collectAllRules( + map: Map, + ruleTypeByStepAndId: Map, + correctAnswerByStepAndId: Map, +): AllRuleRow[] { + const rows: AllRuleRow[] = [] + for (const [stepNumber, questions] of map.entries()) { + for (const q of questions) { + const key = `${stepNumber}::${q.id}` + const isMcq = q.type === 'QUESTION_TYPE_MCQ' + const ai = Boolean((q as any).aiGrading) + rows.push({ + stepNumber, + questionId: q.id, + questionText: q.question, + ruleType: ruleTypeByStepAndId.get(key) ?? '', + questionType: isMcq ? 'MCQ' : 'TEXT', + choiceMode: (q as any).choiceMode ?? null, + options: isMcq && (q as any).options ? JSON.stringify((q as any).options) : null, + correctAnswer: ai ? null : (correctAnswerByStepAndId.get(key) ?? (q as any).correctAnswer ?? null), + aiGrading: ai, + }) + } + } + return rows +} +``` + +- [ ] **Step 4: Run to verify it passes** — `npx vitest run test/unit/collect-all-rules.test.ts --project unit` → PASS. +- [ ] **Step 5: Commit** + +```bash +git add scripts/parsers/rules.ts test/unit/collect-all-rules.test.ts +git commit -m "feat(parser): collectAllRules for full rules.vr rule set (#WS3)" +``` + +### Task 12: Fetch sidecar — `.validation-rules.json` + +**Files:** +- Modify: `scripts/fetch-tutorials.ts` (~`:975` where `validationMap` etc. are destructured; write beside other sidecars ~`:1043`) +- Test: covered by Task 11 helper + Task 13 server test; add a small write-path assertion. + +- [ ] **Step 1: Wire the sidecar write** + +```ts +import { collectAllRules } from './parsers/rules' +// ... +const allRules = collectAllRules(validationMap, ruleTypeByStepAndId, correctAnswerByStepAndId) +if (allRules.length > 0) { + writeFileSync( + join(CACHE_DIR, `${t.slug.toLowerCase()}.validation-rules.json`), + JSON.stringify({ slug: t.slug.toLowerCase(), rules: allRules }, null, 2), + ) +} +``` + +- [ ] **Step 2: Sanity build** — run `npm run fetch-tutorials` for a small subset if a `--slug`/limit flag exists, or type-check: `npx tsc --noEmit -p tsconfig.json` (confirm project has this). Expected: no type errors. +- [ ] **Step 3: Commit** + +```bash +git add scripts/fetch-tutorials.ts +git commit -m "feat(fetch): write validation-rules sidecar (all rule types) (#WS3)" +``` + +### Task 13: Server handler — REPLACE validation rules per slug + +**Files:** +- Create: `srv/lib/validation-rules-publish.js` +- Modify: `srv/server.js` (mount route) +- Modify: `.deploy/mta.yaml` (srv-qa cp-list ~line 175) +- Test: `test/unit/validation-rules-publish.test.js` + +**Interfaces:** +- Consumes: `{ slug, rules: AllRuleRow[] }`. +- Produces: `replaceValidationRulesForSlug(db, slug, rules)` + `publishValidationRules(req, res)`. + +- [ ] **Step 1: Write the failing test** + +```js +// test/unit/validation-rules-publish.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +import { replaceValidationRulesForSlug } from '../../srv/lib/validation-rules-publish.js' + +describe('replaceValidationRulesForSlug', () => { + let db + beforeAll(async () => { await cds.test('serve', '--in-memory').in(process.cwd()); db = await cds.connect.to('db') }) + it('replaces all-rule rows for a slug', async () => { + const { Tutorials, TutorialValidationRules } = cds.entities('com.sap.developers.ims') + const ID = cds.utils.uuid() + await db.run(INSERT.into(Tutorials).entries({ ID, slug: 'vr-demo', title: 'VR' })) + await replaceValidationRulesForSlug(db, 'VR-DEMO', [ + { stepNumber: 1, questionId: 'validate-1', questionText: 'Q', ruleType: 'single-choice', questionType: 'MCQ', choiceMode: 'single', options: '["A","B"]', correctAnswer: 'A', aiGrading: false }, + ]) + let rows = await db.run(SELECT.from(TutorialValidationRules).where({ tutorial_ID: ID })) + expect(rows).toHaveLength(1) + expect(rows[0].aiGrading).toBe(false) + await replaceValidationRulesForSlug(db, 'vr-demo', []) + rows = await db.run(SELECT.from(TutorialValidationRules).where({ tutorial_ID: ID })) + expect(rows).toHaveLength(0) + }) +}) +``` + +- [ ] **Step 2: Run to verify it fails** — `npx vitest run test/unit/validation-rules-publish.test.js --project unit` → FAIL. + +- [ ] **Step 3: Implement the handler** (mirror `contributors-publish.js`) + +```js +const cds = require('@sap/cds') +const NS = 'com.sap.developers.ims' + +async function replaceValidationRulesForSlug(db, slug, rules) { + const { Tutorials, TutorialValidationRules } = cds.entities(NS) + const lcSlug = String(slug || '').toLowerCase() + const tut = await db.run(SELECT.one.from(Tutorials).columns('ID').where({ slug: lcSlug })) + if (!tut) return { ok: false, reason: 'tutorial_not_found', slug: lcSlug } + const entries = (rules || []).map((r) => ({ + tutorial_ID: tut.ID, + stepNumber: r.stepNumber, + questionId: String(r.questionId).slice(0, 100), + questionText: (r.questionText || '').slice(0, 2000), + ruleType: (r.ruleType || '').slice(0, 50), + questionType: (r.questionType || '').slice(0, 20), + choiceMode: r.choiceMode || null, + options: r.options || null, + correctAnswer: r.correctAnswer ?? null, + aiGrading: Boolean(r.aiGrading), + })) + await cds.tx(async (tx) => { + await tx.run(DELETE.from(TutorialValidationRules).where({ tutorial_ID: tut.ID })) + if (entries.length) await tx.run(INSERT.into(TutorialValidationRules).entries(entries)) + }) + return { ok: true, slug: lcSlug, count: entries.length } +} + +async function publishValidationRules(req, res) { + try { + const { slug, rules } = req.body || {} + if (!slug || !Array.isArray(rules)) return res.status(400).json({ error: 'bad_request' }) + let entities; try { entities = cds.entities(NS) } catch { entities = null } + if (!entities || !entities.TutorialValidationRules) return res.status(409).json({ error: 'entity_not_in_model' }) + const db = await cds.connect.to('db') + const result = await replaceValidationRulesForSlug(db, slug, rules) + return res.status(result.ok ? 200 : 404).json(result) + } catch (e) { return res.status(500).json({ error: 'internal', detail: e?.message }) } +} + +module.exports = { replaceValidationRulesForSlug, publishValidationRules } +``` + +Mount in `srv/server.js`: + +```js +const { publishValidationRules } = require('./lib/validation-rules-publish.js') +app.post('/content/publish-validation-rules', requireContentApiKey, express.json({ limit: '4mb' }), publishValidationRules) +``` + +- [ ] **Step 4: Run to verify it passes** — `npx vitest run test/unit/validation-rules-publish.test.js --project unit` → PASS. +- [ ] **Step 5: Add to srv-qa cp-list** — append `../../srv/lib/validation-rules-publish.js` to the `.deploy/mta.yaml` srv-qa `cp` command (~line 175). +- [ ] **Step 6: Commit** + +```bash +git add srv/lib/validation-rules-publish.js srv/server.js .deploy/mta.yaml test/unit/validation-rules-publish.test.js +git commit -m "feat(publish): server REPLACE handler for TutorialValidationRules (#WS3)" +``` + +### Task 14: Client publish step for validation rules + +**Files:** +- Create: `scripts/publish/publish-validation-rules.ts` +- Modify: `scripts/publish-content.ts` (beside contributors aux step) +- Test: `test/unit/publish-validation-rules-client.test.js` + +**Interfaces:** +- Produces: `publishValidationRules({ cacheDir, baseUrl, apiKey })` — globs `*.validation-rules.json`, POSTs to `/content/publish-validation-rules`. + +- [ ] **Step 1: Write the failing test** (mirror Task 7 client test, glob `*.validation-rules.json`, endpoint `/content/publish-validation-rules`). + +```js +// test/unit/publish-validation-rules-client.test.js +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest' +import { mkdtempSync, writeFileSync } from 'node:fs' +import { join } from 'node:path'; import { tmpdir } from 'node:os' +import { publishValidationRules } from '../../scripts/publish/publish-validation-rules' + +describe('publishValidationRules client', () => { + let dir + beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), 'vr-')) + writeFileSync(join(dir, 'demo.validation-rules.json'), + JSON.stringify({ slug: 'demo', rules: [{ stepNumber: 1, questionId: 'validate-1' }] })) + global.fetch = vi.fn().mockResolvedValue({ ok: true, json: async () => ({ ok: true }) }) + }) + afterEach(() => vi.restoreAllMocks()) + it('POSTs each sidecar', async () => { + const res = await publishValidationRules({ cacheDir: dir, baseUrl: 'http://x', apiKey: 'k' }) + expect(global.fetch).toHaveBeenCalledTimes(1) + expect(global.fetch.mock.calls[0][0]).toBe('http://x/content/publish-validation-rules') + expect(res.published).toBe(1) + }) +}) +``` + +- [ ] **Step 2: Run to verify it fails** — → FAIL (module not found). + +- [ ] **Step 3: Implement + wire** (copy `publish-contributors.ts`, swap glob suffix + endpoint): + +```ts +import { readdirSync, readFileSync } from 'node:fs' +import { join } from 'node:path' +export async function publishValidationRules(opts: { cacheDir: string; baseUrl: string; apiKey: string }) { + const { cacheDir, baseUrl, apiKey } = opts + const files = readdirSync(cacheDir).filter((f) => f.endsWith('.validation-rules.json')) + let published = 0 + for (const f of files) { + const res = await fetch(`${baseUrl}/content/publish-validation-rules`, { + method: 'POST', headers: { 'content-type': 'application/json', 'x-api-key': apiKey }, + body: readFileSync(join(cacheDir, f), 'utf8'), + }) + if (res.ok) published += 1; else console.warn(`[publish-validation-rules] ${f} -> ${res.status}`) + } + return { published, total: files.length } +} +``` + +In `scripts/publish-content.ts`, add a non-fatal aux step (skip `channel === 'qa'`), mirroring Task 7. + +- [ ] **Step 4: Run to verify it passes** — → PASS. +- [ ] **Step 5: Commit** + +```bash +git add scripts/publish/publish-validation-rules.ts scripts/publish-content.ts test/unit/publish-validation-rules-client.test.js +git commit -m "feat(publish): non-fatal client step to publish validation-rules sidecars (#WS3)" +``` + +### Task 15: Service projection + association + UI facets + +**Files:** +- Modify: `srv/admin-service.cds` (projection + association on `Tutorials`) +- Modify: `app/admin-annotations.cds` (relabel AI facet; add "All Validation Rules" facet + LineItem) +- Test: `test/unit/annotations-validation-rules.test.js` + +**Interfaces:** +- Consumes: `TutorialValidationRules` (Task 10). +- Produces: `AdminService.TutorialValidationRules` (read-only) + `Tutorials.validationRules` association. + +- [ ] **Step 1: Write the failing test** + +```js +// test/unit/annotations-validation-rules.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +describe('validation rules exposure + facet', () => { + let m + beforeAll(async () => { m = await cds.load('*') }) + it('AdminService exposes TutorialValidationRules read-only', () => { + expect(m.definitions['AdminService.TutorialValidationRules']).toBeTruthy() + }) + it('Tutorials has validationRules association', () => { + expect(m.definitions['AdminService.Tutorials'].elements.validationRules).toBeTruthy() + }) + it('OP facets include an All Validation Rules facet', () => { + const facets = m.definitions['AdminService.Tutorials']['@UI.Facets'] + const ids = facets.map((f) => f.ID) + expect(ids).toContain('AllValidationRulesFacet') + }) +}) +``` + +- [ ] **Step 2: Run to verify it fails** — → FAIL. + +- [ ] **Step 3: Implement projection + association + facets** + +In `srv/admin-service.cds`: + +```cds +@readonly entity TutorialValidationRules as projection on ims.TutorialValidationRules; +``` + +Add to the `Tutorials` projection body (beside `validationSpecs` ~`:66`): + +```cds +validationRules : Association to many TutorialValidationRules on validationRules.tutorial = $self; +``` + +In `app/admin-annotations.cds`: +1. Relabel the existing AI facet (`ValidationSpecsFacet`, ~`:957`) `Label: 'AI-Graded Validation'`. +2. Add a LineItem + facet: + +```cds +annotate AdminService.TutorialValidationRules with @( + UI.LineItem: [ + { Value: stepNumber, Label: 'Step' }, + { Value: questionText, Label: 'Question' }, + { Value: questionType, Label: 'Type' }, + { Value: ruleType, Label: 'Rule' }, + { Value: aiGrading, Label: 'AI-Graded' }, + { Value: correctAnswer, Label: 'Correct Answer' } + ] +); +``` + +Add to the winning `@UI.Facets` block (~`:948-974`): + +```cds +{ $Type: 'UI.ReferenceFacet', Label: 'All Validation Rules', ID: 'AllValidationRulesFacet', Target: 'validationRules/@UI.LineItem' }, +``` + +- [ ] **Step 4: Run to verify it passes** — `npx vitest run test/unit/annotations-validation-rules.test.js --project unit` → PASS. Then `npx cds deploy --to sqlite::memory:` → clean. +- [ ] **Step 5: Commit** + +```bash +git add srv/admin-service.cds app/admin-annotations.cds test/unit/annotations-validation-rules.test.js +git commit -m "feat(admin-ui): All Validation Rules facet + relabel AI facet (#WS3)" +``` + +### Task 16: Hybrid guard — publish populates all rules + +**Files:** +- Test: `test/hybrid/publish-validation-rules.test.js` + +- [ ] **Step 1: Write the hybrid test** — publish a slug via `replaceValidationRulesForSlug` against HANA, assert both AI and non-AI rows land in `TutorialValidationRules`, and that AI rows still exist in `ValidateAnswerSpecs` (unchanged). Clean up after. + +```js +// test/hybrid/publish-validation-rules.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +import { replaceValidationRulesForSlug } from '../../srv/lib/validation-rules-publish.js' +describe('validation rules publish (hybrid)', () => { + let db; beforeAll(async () => { db = await cds.connect.to('db') }) + it('lands mixed AI + client rules for an existing slug', async () => { + const { Tutorials, TutorialValidationRules } = cds.entities('com.sap.developers.ims') + const t = await db.run(SELECT.one.from(Tutorials).columns('ID','slug')) + await replaceValidationRulesForSlug(db, t.slug, [ + { stepNumber: 99, questionId: 'vr-test-a', questionText: 'client', ruleType: 'single-choice', questionType: 'MCQ', choiceMode: 'single', options: '["A"]', correctAnswer: 'A', aiGrading: false }, + { stepNumber: 99, questionId: 'vr-test-b', questionText: 'ai', ruleType: 'regex', questionType: 'TEXT', choiceMode: null, options: null, correctAnswer: null, aiGrading: true }, + ]) + const rows = await db.run(SELECT.from(TutorialValidationRules).where({ tutorial_ID: t.ID, stepNumber: 99 })) + expect(rows.length).toBe(2) + await db.run(DELETE.from(TutorialValidationRules).where({ tutorial_ID: t.ID, stepNumber: 99 })) + }) +}) +``` + +- [ ] **Step 2: Run** `npm run test:hybrid -- test/hybrid/publish-validation-rules.test.js` → PASS. +- [ ] **Step 3: Commit** + +```bash +git add test/hybrid/publish-validation-rules.test.js +git commit -m "test(publish): hybrid guard for all-rules population (#WS3)" +``` + +--- + +## Final verification (whole phase) + +- [ ] Run full unit suite: `npm test` — Expected: all green. +- [ ] Run affected hybrid tests: `npm run test:hybrid -- test/hybrid/publish-categories.test.js test/hybrid/publish-contributors.test.js test/hybrid/publish-validation-rules.test.js` (requires `cf login` + `cds bind`). +- [ ] `npx cds deploy --to sqlite::memory:` clean (schema sanity). +- [ ] Confirm both new `srv/lib/*-publish.js` files are in the `.deploy/mta.yaml` srv-qa `cp` list. +- [ ] Open a PR targeting **DEV** summarizing WS1-WS3 + the one-time category backfill ops step. + +## Post-deploy validation (DEV) + +- [ ] After DEV deploy + a content publish, open the reference tutorial's OP: + - Categories facet shows rows (post-backfill + self-heal). + - Contributors table shows the git contributor list, each `login` linking to `github.com/`. + - "All Validation Rules" facet shows every rule; "AI-Graded Validation" still shows the AI subset. + +## Self-review notes (author) + +- **Spec coverage:** WS1 (Tasks 1-3), WS2 (Tasks 4-9), WS3 (Tasks 10-16) map to spec §WS1-WS3. WS4 (KG) and WS5 (media+freshness) are deferred to separate plans per the spec's phasing. +- **Assumptions flagged for the implementer to verify against live code before writing:** (a) module system of `content-publish-session.js` (CommonJS vs ESM) for Task 1; (b) exact auth-middleware + header name used by existing publish routes (Tasks 6/7/13/14); (c) exact key format of `ruleTypeByStepAndId`/`correctAnswerByStepAndId` (Task 11); (d) the FE link idiom that renders a clickable GitHub link (Task 8). These are grounded by research but must be confirmed at the touched lines. From 158f7e6da80061b5273eade1131f9174ee2b8ab5 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 13:03:33 -0400 Subject: [PATCH 06/71] docs(admin): Phase 2 (media/freshness) + Phase 3 (KG) implementation plans --- ...torials-admin-op-phase2-media-freshness.md | 371 ++++++++++++++++++ ...torials-admin-op-phase3-knowledge-graph.md | 268 +++++++++++++ 2 files changed, 639 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-31-tutorials-admin-op-phase2-media-freshness.md create mode 100644 docs/superpowers/plans/2026-08-31-tutorials-admin-op-phase3-knowledge-graph.md diff --git a/docs/superpowers/plans/2026-08-31-tutorials-admin-op-phase2-media-freshness.md b/docs/superpowers/plans/2026-08-31-tutorials-admin-op-phase2-media-freshness.md new file mode 100644 index 000000000..452cfd33d --- /dev/null +++ b/docs/superpowers/plans/2026-08-31-tutorials-admin-op-phase2-media-freshness.md @@ -0,0 +1,371 @@ +# Tutorials Admin OP — Phase 2 (Media + Freshness Header) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Surface per-tutorial object-store items (images + assets) with rich detail and a download/preview link, and add a Freshness report facet (last-run/model/cost/status) to the Tutorials Object Page. + +**Architecture:** `TutorialImages`/`TutorialAssets` are persisted but exposed on no service. Add `@readonly` projections on `AdminService` reachable from `Tutorials` via slug/channel associations; the `content : Composition of many Attachments` child auto-exposes with `@cap-js/attachments` media annotations (`@Core.MediaType`/`@Core.ContentDisposition`/`@UI.MediaResource`) so Fiori renders a native download link. Persist `byteSize` at image/asset ingest (already in memory as `buffer.length`). `FreshnessReport` is already exposed — add an association from `Tutorials` and a sorted LineItem facet. + +**Tech Stack:** SAP CAP (Node.js, CDS), `@cap-js/attachments` 4.0.0, Fiori Elements annotations, Vitest. + +**Spec:** `docs/superpowers/specs/2026-08-31-tutorials-admin-op-enhancements-design.md` (§WS5, decisions D1/D2) + +## Global Constraints + +- **D1 (prod S3 binding):** the Media facet works regardless of backing store; the missing `objectstore` resource in `mta.yaml` is a **separate ops task**, out of scope here. Do not add the S3 binding in this plan. +- **D2:** add `byteSize : Integer64`; **defer** width/height. +- **Read-only exposure** — projections are `@readonly`; use `@cds.redirection.target: false` (pattern at `srv/admin-service.cds:114,120`) to avoid stealing association redirects. +- **BLOB reads stay raw `db.run()`** — never mix LOB + metadata in one CDS QL query. +- **Schema changes** — `cds build --production`; register in `db/persistence.cds`; `npx cds deploy --to sqlite::memory:` before committing. +- **Associations on `slug`** — `TutorialImages`/`TutorialAssets` join `Tutorials` on `slug` (unmanaged `on`), filtered `channel = 'prod'`. +- **Tests:** `npm test` (unit), `npm run test:hybrid` (real HANA). PR targets DEV. + +## File Structure + +- Modify: `db/tutorial-images.cds`, `db/tutorial-assets.cds` — add `byteSize`. +- Modify: `db/persistence.cds` — journal (if needed for new column). +- Modify: `srv/lib/image-ingest-handler.js` (~`:77`) + the asset ingest handler — persist `byteSize`. +- Modify: `srv/admin-service.cds` — `@readonly` projections + `images`/`assets`/`freshnessReports` associations on `Tutorials`. +- Modify: `app/admin-annotations.cds` — Media facet (2 LineItems w/ download link + external sourceUrl) + Freshness Reports facet. +- Test: `test/unit/schema-media.test.js`, `test/unit/annotations-media.test.js`, `test/unit/annotations-freshness-facet.test.js`, `test/hybrid/media-exposure.test.js`. + +--- + +### Task 1: Schema — add `byteSize` to images + assets + +**Files:** +- Modify: `db/tutorial-images.cds:7-16`, `db/tutorial-assets.cds:6-16` +- Modify: `db/persistence.cds` +- Test: `test/unit/schema-media.test.js` + +**Interfaces:** +- Produces: `TutorialImages.byteSize : Integer64`, `TutorialAssets.byteSize : Integer64`. + +- [ ] **Step 1: Write the failing test** + +```js +// test/unit/schema-media.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +describe('media byteSize', () => { + let m; beforeAll(async () => { m = await cds.load('*') }) + it('images + assets have byteSize', () => { + expect(m.definitions['com.sap.developers.ims.TutorialImages'].elements.byteSize).toBeTruthy() + expect(m.definitions['com.sap.developers.ims.TutorialAssets'].elements.byteSize).toBeTruthy() + }) +}) +``` + +- [ ] **Step 2: Run to verify it fails** — `npx vitest run test/unit/schema-media.test.js --project unit` → FAIL. + +- [ ] **Step 3: Add the column** (both files): + +```cds +byteSize : Integer64; // original byte length captured at ingest +``` + +Register in `db/persistence.cds` if the column needs a migration-table entry (follow existing shape). + +- [ ] **Step 4: Verify + deploy dry-run** — test PASS; `npx cds deploy --to sqlite::memory:` clean. +- [ ] **Step 5: Build + commit** + +```bash +npx cds build --production +git add db/tutorial-images.cds db/tutorial-assets.cds db/persistence.cds db/src/gen test/unit/schema-media.test.js +git commit -m "feat(db): add byteSize to TutorialImages/TutorialAssets (#WS5)" +``` + +### Task 2: Persist `byteSize` at ingest + +**Files:** +- Modify: `srv/lib/image-ingest-handler.js` (~`:60-77`, where `contentHash`/`mimeType` are computed and `imageStore.put` is called) +- Modify: the asset ingest handler (per research: `srv/lib/attachment-source-handler.js:118` region / the asset ingest analog — confirm exact put call) +- Modify: `srv/lib/image-store.cjs` + `srv/lib/attachment-store.cjs` — accept/persist `byteSize` on the parent row +- Test: `test/unit/ingest-bytesize.test.js` + +**Interfaces:** +- Consumes: `buffer` at ingest. +- Produces: parent `TutorialImages`/`TutorialAssets` row carries `byteSize = buffer.length`. + +- [ ] **Step 1: Write the failing test** (unit-level against the store put, in-memory SQLite) + +```js +// test/unit/ingest-bytesize.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +const store = require('../../srv/lib/image-store.cjs') + +describe('image store persists byteSize', () => { + beforeAll(async () => { await cds.test('serve', '--in-memory').in(process.cwd()) }) + it('stores buffer length as byteSize', async () => { + const db = await cds.connect.to('db') + const buf = Buffer.from('hello world') + await store.put('https://raw.example/img.png', { buffer: buf, mimeType: 'image/png', contentHash: 'abc', slug: 'demo', channel: 'prod', byteSize: buf.length }) + const { TutorialImages } = cds.entities('com.sap.developers.ims') + const row = await db.run(SELECT.one.from(TutorialImages).where({ sourceUrl: 'https://raw.example/img.png' })) + expect(row.byteSize).toBe(11) + }) +}) +``` + +> Confirm `image-store.cjs` `put()` signature + how it writes the parent row before finalizing the test; adapt arg shape to the real API. + +- [ ] **Step 2: Run to verify it fails** — → FAIL (byteSize null/undefined). + +- [ ] **Step 3: Implement** + +In `srv/lib/image-ingest-handler.js`, pass `byteSize: buffer.length` into the `imageStore.put(...)` options (~`:77`). In `srv/lib/image-store.cjs`, include `byteSize` in the parent-row INSERT/UPSERT. Mirror both for assets (`attachment-store.cjs` + the asset ingest handler). + +- [ ] **Step 4: Run to verify it passes** — → PASS. +- [ ] **Step 5: Commit** + +```bash +git add srv/lib/image-ingest-handler.js srv/lib/image-store.cjs srv/lib/attachment-store.cjs test/unit/ingest-bytesize.test.js +git commit -m "feat(media): persist byteSize at image/asset ingest (#WS5)" +``` + +> If the asset ingest handler lives in a distinct file, add it to this commit and to the srv-qa cp-list if not already present. + +### Task 3: Expose read-only Media projections + associations + +**Files:** +- Modify: `srv/admin-service.cds` +- Test: `test/unit/media-exposure.test.js` + +**Interfaces:** +- Produces: `AdminService.TutorialImages`, `AdminService.TutorialAssets` (`@readonly`); `Tutorials.images`, `Tutorials.assets` associations (join on `slug`, `channel='prod'`). + +- [ ] **Step 1: Write the failing test** + +```js +// test/unit/media-exposure.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +describe('media exposure', () => { + let m; beforeAll(async () => { m = await cds.load('*') }) + it('exposes images + assets read-only', () => { + expect(m.definitions['AdminService.TutorialImages']).toBeTruthy() + expect(m.definitions['AdminService.TutorialAssets']).toBeTruthy() + }) + it('Tutorials has images + assets associations', () => { + const t = m.definitions['AdminService.Tutorials'].elements + expect(t.images).toBeTruthy() + expect(t.assets).toBeTruthy() + }) +}) +``` + +- [ ] **Step 2: Run to verify it fails** — → FAIL. + +- [ ] **Step 3: Implement** + +In `srv/admin-service.cds`: + +```cds +@readonly @cds.redirection.target: false entity TutorialImages as projection on ims.TutorialImages; +@readonly @cds.redirection.target: false entity TutorialAssets as projection on ims.TutorialAssets; +``` + +Add to the `Tutorials` projection body: + +```cds +images : Association to many TutorialImages on images.slug = $self.slug and images.channel = 'prod'; +assets : Association to many TutorialAssets on assets.slug = $self.slug and assets.channel = 'prod'; +``` + +> The `content` Attachments composition auto-exposes when reachable from an exposed entity (@cap-js/attachments relies on this). + +- [ ] **Step 4: Run to verify it passes** — test PASS; `npx cds deploy --to sqlite::memory:` clean. +- [ ] **Step 5: Commit** + +```bash +git add srv/admin-service.cds test/unit/media-exposure.test.js +git commit -m "feat(admin): read-only Media projections + Tutorials associations (#WS5)" +``` + +### Task 4: UI — Media facet with download link + detail + +**Files:** +- Modify: `app/admin-annotations.cds` +- Test: `test/unit/annotations-media.test.js` + +**Interfaces:** +- Consumes: `AdminService.TutorialImages/TutorialAssets` + their `content` media child. + +- [ ] **Step 1: Write the failing test** + +```js +// test/unit/annotations-media.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +describe('Media facet', () => { + let m; beforeAll(async () => { m = await cds.load('*') }) + it('OP facets include Media', () => { + const ids = m.definitions['AdminService.Tutorials']['@UI.Facets'].map((f) => f.ID) + expect(ids).toContain('MediaImagesFacet') + expect(ids).toContain('MediaAssetsFacet') + }) + it('image LineItem shows sourceUrl + byteSize + mimeType', () => { + const li = m.definitions['AdminService.TutorialImages']['@UI.LineItem'] + const vals = li.map((x) => x.Value?.['='] || x.Value) + for (const c of ['sourceUrl','byteSize','mimeType','contentHash']) expect(vals).toContain(c) + }) +}) +``` + +- [ ] **Step 2: Run to verify it fails** — → FAIL. + +- [ ] **Step 3: Implement** + +In `app/admin-annotations.cds`, add LineItems (render `sourceUrl` as external link via `DataFieldWithUrl`; the media download comes from the auto-exposed `content` child's ready-made annotations — a nested facet on `content` gives the download link): + +```cds +annotate AdminService.TutorialImages with @( + UI.LineItem: [ + { $Type: 'UI.DataFieldWithUrl', Value: sourceUrl, Url: sourceUrl, Label: 'Source (GitHub)' }, + { Value: mimeType, Label: 'Type' }, + { Value: byteSize, Label: 'Bytes' }, + { Value: contentHash, Label: 'Hash' }, + { Value: channel, Label: 'Channel' } + ] +); +annotate AdminService.TutorialAssets with @( + UI.LineItem: [ + { Value: filename, Label: 'File' }, + { $Type: 'UI.DataFieldWithUrl', Value: sourceUrl, Url: sourceUrl, Label: 'Source (GitHub)' }, + { Value: mimeType, Label: 'Type' }, + { Value: byteSize, Label: 'Bytes' }, + { Value: contentHash, Label: 'Hash' } + ] +); +``` + +Add facets to the winning `@UI.Facets` block: + +```cds +{ $Type: 'UI.ReferenceFacet', Label: 'Images', ID: 'MediaImagesFacet', Target: 'images/@UI.LineItem' }, +{ $Type: 'UI.ReferenceFacet', Label: 'Assets', ID: 'MediaAssetsFacet', Target: 'assets/@UI.LineItem' }, +``` + +> The @cap-js/attachments `content` child ships its own `@UI.LineItem` with a media download link. To surface a clickable download/preview, optionally add a nested facet targeting `images/content/@UI.LineItem` once verified against the running FE version. + +- [ ] **Step 4: Run to verify it passes** — → PASS; `npx cds deploy --to sqlite::memory:` clean. +- [ ] **Step 5: Commit** + +```bash +git add app/admin-annotations.cds test/unit/annotations-media.test.js +git commit -m "feat(admin-ui): Media facets (images/assets) with source link + byte size (#WS5)" +``` + +### Task 5: Freshness Reports facet (header) + +**Files:** +- Modify: `srv/admin-service.cds` — add `freshnessReports` association on `Tutorials` +- Modify: `app/admin-annotations.cds` — LineItem + facet, sorted by `runAt` desc via `@UI.PresentationVariant` +- Test: `test/unit/annotations-freshness-facet.test.js` + +**Interfaces:** +- Consumes: `AdminService.FreshnessReport` (already exposed `srv/admin-service.cds:129`). +- Produces: `Tutorials.freshnessReports` association + `FreshnessReportsFacet`. + +- [ ] **Step 1: Write the failing test** + +```js +// test/unit/annotations-freshness-facet.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +describe('Freshness reports facet', () => { + let m; beforeAll(async () => { m = await cds.load('*') }) + it('Tutorials has freshnessReports association', () => { + expect(m.definitions['AdminService.Tutorials'].elements.freshnessReports).toBeTruthy() + }) + it('OP facets include FreshnessReportsFacet', () => { + const ids = m.definitions['AdminService.Tutorials']['@UI.Facets'].map((f) => f.ID) + expect(ids).toContain('FreshnessReportsFacet') + }) +}) +``` + +- [ ] **Step 2: Run to verify it fails** — → FAIL. + +- [ ] **Step 3: Implement** + +In `srv/admin-service.cds` `Tutorials` projection: + +```cds +freshnessReports : Association to many FreshnessReport on freshnessReports.tutorial = $self; +``` + +In `app/admin-annotations.cds`: + +```cds +annotate AdminService.FreshnessReport with @( + UI.LineItem: [ + { Value: runAt, Label: 'Run At' }, + { Value: status, Label: 'Status' }, + { Value: model, Label: 'Model' }, + { Value: cost, Label: 'Cost' }, + { Value: openHighCount, Label: 'Open High' }, + { Value: error, Label: 'Error' } + ], + UI.PresentationVariant: { SortOrder: [{ Property: runAt, Descending: true }], Visualizations: ['@UI.LineItem'] } +); +``` + +Add to `@UI.Facets` (place above the existing Freshness findings facet): + +```cds +{ $Type: 'UI.ReferenceFacet', Label: 'Freshness Reports', ID: 'FreshnessReportsFacet', Target: 'freshnessReports/@UI.PresentationVariant' }, +``` + +- [ ] **Step 4: Run to verify it passes** — → PASS; `npx cds deploy --to sqlite::memory:` clean. +- [ ] **Step 5: Commit** + +```bash +git add srv/admin-service.cds app/admin-annotations.cds test/unit/annotations-freshness-facet.test.js +git commit -m "feat(admin-ui): Freshness Reports facet (runAt/model/cost/status) (#WS5)" +``` + +### Task 6: Hybrid guard — media exposure resolves + +**Files:** +- Test: `test/hybrid/media-exposure.test.js` + +- [ ] **Step 1: Write the hybrid test** + +```js +// test/hybrid/media-exposure.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +describe('media exposure (hybrid)', () => { + let admin; beforeAll(async () => { admin = await cds.connect.to('AdminService') }) + it('reads images for a tutorial without LOB errors', async () => { + const t = await admin.run(SELECT.one.from('AdminService.Tutorials').columns('ID','slug')) + expect(t).toBeTruthy() + // metadata-only read (no BLOB mix) + const imgs = await admin.run(SELECT.from('AdminService.TutorialImages').columns('ID','sourceUrl','mimeType','byteSize').where({ slug: t.slug })) + expect(Array.isArray(imgs)).toBe(true) + }) +}) +``` + +- [ ] **Step 2: Run** `npm run test:hybrid -- test/hybrid/media-exposure.test.js` → PASS. +- [ ] **Step 3: Commit** + +```bash +git add test/hybrid/media-exposure.test.js +git commit -m "test(media): hybrid guard for media exposure (#WS5)" +``` + +--- + +## Final verification + +- [ ] `npm test` all green; `npx cds deploy --to sqlite::memory:` clean. +- [ ] Backfill media if needed: `npm run backfill-images` (+ assets) against the env so rows exist. +- [ ] DEV post-deploy: reference tutorial OP shows Images + Assets tables (source link, type, bytes, hash) and a Freshness Reports table sorted newest-first. + +## Self-review notes + +- **Spec coverage:** WS5 §Media (Tasks 1-4, 6) + §Freshness header (Task 5). D1 respected (no S3 binding change). D2 respected (byteSize yes, dimensions no). +- **Verify against live code:** `image-store.cjs`/`attachment-store.cjs` `put()` signatures (Task 2); the asset ingest handler path; FE `DataFieldWithUrl` + media-child download idiom for the current FE version (Task 4). diff --git a/docs/superpowers/plans/2026-08-31-tutorials-admin-op-phase3-knowledge-graph.md b/docs/superpowers/plans/2026-08-31-tutorials-admin-op-phase3-knowledge-graph.md new file mode 100644 index 000000000..2fec35da6 --- /dev/null +++ b/docs/superpowers/plans/2026-08-31-tutorials-admin-op-phase3-knowledge-graph.md @@ -0,0 +1,268 @@ +# Tutorials Admin OP — Phase 3 (Knowledge-Graph Facet) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Surface per-tutorial knowledge-graph data on the Tutorials Object Page — concepts taught (with confidence), PageRank importance, community label, and co-completed neighbor tutorials — all read-only from data already populated by nightly jobs. + +**Architecture:** These entities (`TutorialConceptLinks`, `TutorialRank`, `CoCompletions`, `KgCommunity`/`KgCommunityLabel`) live on `KnowledgeGraphService`, not `AdminService`. Add `@readonly` projections on `AdminService` + associations from `Tutorials`, then a "Knowledge Graph" facet group. Purely additive and read-only; no pipeline changes. Reads fail-open (any SELECT throw leaves fields unset — FE renders nothing rather than 500), mirroring the existing KG `after('READ')` decorators. + +**Tech Stack:** SAP CAP (Node.js, CDS), Fiori Elements annotations, Vitest. KG data populated by existing nightly jobs (PageRank #916, WCC #918, Louvain #917, community labels #1126). + +**Spec:** `docs/superpowers/specs/2026-08-31-tutorials-admin-op-enhancements-design.md` (§WS4) + +## Global Constraints + +- **Read-only, additive** — `@readonly` projections; `@cds.redirection.target: false` (pattern `srv/admin-service.cds:114,120`) to avoid stealing redirects. +- **Fail-open reads** — if a KG projection can throw at read time, guard with an `after('READ')` that leaves fields unset (mirror `KnowledgeGraphService.Concepts`/`AdminService.Tutorials` isolation decorators). Pure associations/projections need no decorator. +- **No new env flags, no schema, no jobs** — data already materialized. +- **DEV-only until PROD KG data verifies** — same posture as #1126 (PROD Louvain/community data may be sparse; empty facets render as FE "No data", never a 500). +- **Namespace** — KG entities are in the `com.sap.developers.ims` model (via `db/knowledge-graph*.cds`); some carry `@cds.autoexpose:false` (e.g. `TutorialRank`) requiring explicit projection. +- **Tests:** `npm test` (unit), `npm run test:hybrid`. PR targets DEV. + +## File Structure + +- Modify: `srv/admin-service.cds` — `@readonly` projections (`TutorialConceptLinks`, `TutorialRank`, `CoCompletions`) + associations (`conceptLinks`, `rank`, `coCompletions`) on `Tutorials`; community label reachable via existing `KgCommunityMembers` (`:1249`) join. +- Modify: `srv/admin-service.js` — optional fail-open `after('READ','Tutorials')` if a computed field is added (community label flatten). +- Modify: `app/admin-annotations.cds` — Knowledge Graph facet group (concepts taught LineItem, prerequisites, PageRank + community FieldGroup, co-completed neighbors LineItem). +- Test: `test/unit/kg-exposure.test.js`, `test/unit/annotations-kg.test.js`, `test/hybrid/kg-facet.test.js`. + +--- + +### Task 1: Expose read-only KG projections + associations + +**Files:** +- Modify: `srv/admin-service.cds` +- Test: `test/unit/kg-exposure.test.js` + +**Interfaces:** +- Consumes: `ims.TutorialConceptLinks` (`db/knowledge-graph.cds:61-70`, predicate teaches/extends + confidence), `ims.TutorialRank` (`:196-200`, PageRank score, `@cds.autoexpose:false`), `ims.CoCompletions` (`:158-162`, weighted A→B). +- Produces: `AdminService.TutorialConceptLinks`, `AdminService.TutorialRank`, `AdminService.CoCompletions` (`@readonly`); `Tutorials.conceptLinks` (many), `Tutorials.rank` (one), `Tutorials.coCompletions` (many). + +- [ ] **Step 1: Write the failing test** + +```js +// test/unit/kg-exposure.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +describe('KG exposure on AdminService', () => { + let m; beforeAll(async () => { m = await cds.load('*') }) + it('exposes concept links, rank, co-completions read-only', () => { + expect(m.definitions['AdminService.TutorialConceptLinks']).toBeTruthy() + expect(m.definitions['AdminService.TutorialRank']).toBeTruthy() + expect(m.definitions['AdminService.CoCompletions']).toBeTruthy() + }) + it('Tutorials carries conceptLinks / rank / coCompletions', () => { + const t = m.definitions['AdminService.Tutorials'].elements + expect(t.conceptLinks).toBeTruthy() + expect(t.rank).toBeTruthy() + expect(t.coCompletions).toBeTruthy() + }) +}) +``` + +- [ ] **Step 2: Run to verify it fails** — `npx vitest run test/unit/kg-exposure.test.js --project unit` → FAIL. + +> **Before implementing:** confirm exact entity + element names in `db/knowledge-graph.cds` — the join columns on `TutorialConceptLinks` (which side is the tutorial: `tutorial`/`source`), `TutorialRank` key/score field name (`score`/`pagerank`), and `CoCompletions` columns (`tutorialA`/`tutorialB`/`weight`). Adapt the `on` conditions below to the real names. + +- [ ] **Step 3: Implement** + +In `srv/admin-service.cds`: + +```cds +@readonly @cds.redirection.target: false entity TutorialConceptLinks as projection on ims.TutorialConceptLinks; +@readonly @cds.redirection.target: false entity TutorialRank as projection on ims.TutorialRank; +@readonly @cds.redirection.target: false entity CoCompletions as projection on ims.CoCompletions; +``` + +Add to the `Tutorials` projection body (adapt `on` to confirmed column names): + +```cds +conceptLinks : Association to many TutorialConceptLinks on conceptLinks.tutorial = $self; +rank : Association to one TutorialRank on rank.tutorial = $self; +coCompletions : Association to many CoCompletions on coCompletions.tutorialA = $self; +``` + +> If `Tutorials.conceptLinks` is already injected on the db entity (`db/knowledge-graph.cds:94-97`), the `*` in the projection may already carry it — in that case only expose the target projection and drop the redundant association line. Verify with the failing test after each change. + +- [ ] **Step 4: Run to verify it passes** — test PASS; `npx cds deploy --to sqlite::memory:` clean. +- [ ] **Step 5: Commit** + +```bash +git add srv/admin-service.cds test/unit/kg-exposure.test.js +git commit -m "feat(admin): read-only KG projections + Tutorials associations (#WS4)" +``` + +### Task 2: Community label reachable on the OP + +**Files:** +- Modify: `srv/admin-service.cds` (association to community membership/label) + optional `srv/admin-service.js` fail-open decorator +- Test: `test/unit/kg-community-link.test.js` + +**Interfaces:** +- Consumes: `AdminService.KgCommunityMembers` (already exposed `:1249`), `KgCommunityLabel` (`db/knowledge-graph-communities.cds:76-83`, keyed by `communityFingerprint`). +- Produces: a way to show the tutorial's community label on the OP — either an association `communityMembership` on `Tutorials` (many, filtered to this tutorial's slug) rendered as a small LineItem, or a flattened `virtual communityLabel` populated fail-open in `after('READ','Tutorials')`. + +- [ ] **Step 1: Write the failing test** + +```js +// test/unit/kg-community-link.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +describe('community label reachable', () => { + let m; beforeAll(async () => { m = await cds.load('*') }) + it('Tutorials exposes community membership or virtual label', () => { + const t = m.definitions['AdminService.Tutorials'].elements + expect(t.communityMembership || t.communityLabel).toBeTruthy() + }) +}) +``` + +- [ ] **Step 2: Run to verify it fails** — → FAIL. + +- [ ] **Step 3: Implement (prefer the association — no compute, lowest risk)** + +In `srv/admin-service.cds`, add an association from `Tutorials` to the exposed community-member rows for this tutorial's slug (adapt column names to `KgCommunityMembers`): + +```cds +communityMembership : Association to many KgCommunityMembers on communityMembership.memberSlug = $self.slug; +``` + +> If `KgCommunityMembers` does not carry a `memberSlug`/label directly, add a `virtual communityLabel : String` and populate it fail-open in `srv/admin-service.js` `after('READ','Tutorials')`: +> ```js +> srv.after('READ', 'Tutorials', async (rows) => { +> try { /* look up KgCommunity by slug → KgCommunityLabel; set r.communityLabel */ } +> catch (e) { /* fail-open: leave unset */ } +> }) +> ``` +> Choose the association path if the columns allow; only fall back to the virtual+decorator if a join isn't expressible. + +- [ ] **Step 4: Run to verify it passes** — → PASS; `npx cds deploy --to sqlite::memory:` clean. +- [ ] **Step 5: Commit** + +```bash +git add srv/admin-service.cds srv/admin-service.js test/unit/kg-community-link.test.js +git commit -m "feat(admin): expose community membership/label on Tutorials (#WS4)" +``` + +### Task 3: UI — Knowledge Graph facet group + +**Files:** +- Modify: `app/admin-annotations.cds` +- Test: `test/unit/annotations-kg.test.js` + +**Interfaces:** +- Consumes: Task 1 + Task 2 associations/projections. + +- [ ] **Step 1: Write the failing test** + +```js +// test/unit/annotations-kg.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +describe('Knowledge Graph facets', () => { + let m; beforeAll(async () => { m = await cds.load('*') }) + it('OP facets include KG facets', () => { + const ids = m.definitions['AdminService.Tutorials']['@UI.Facets'].map((f) => f.ID) + expect(ids).toContain('ConceptsTaughtFacet') + expect(ids).toContain('CoCompletionsFacet') + }) + it('concept links LineItem shows predicate + confidence', () => { + const li = m.definitions['AdminService.TutorialConceptLinks']['@UI.LineItem'] + const vals = li.map((x) => x.Value?.['='] || x.Value) + expect(vals).toContain('predicate') + expect(vals).toContain('confidence') + }) +}) +``` + +- [ ] **Step 2: Run to verify it fails** — → FAIL. + +- [ ] **Step 3: Implement** + +In `app/admin-annotations.cds` (adapt element names to confirmed schema): + +```cds +annotate AdminService.TutorialConceptLinks with @( + UI.LineItem: [ + { Value: concept_ID, Label: 'Concept' }, + { Value: predicate, Label: 'Relation' }, // teaches | extends + { Value: confidence, Label: 'Confidence' } + ] +); +annotate AdminService.CoCompletions with @( + UI.LineItem: [ + { Value: tutorialB_ID, Label: 'Also Completed' }, + { Value: weight, Label: 'Weight' } + ] +); +``` + +Add a "Knowledge Graph" FieldGroup for PageRank + community, and facets to the winning `@UI.Facets` block: + +```cds +annotate AdminService.Tutorials with @( + UI.FieldGroup #KnowledgeGraph: { Data: [ + { Value: rank.score, Label: 'PageRank' }, + { Value: communityLabel, Label: 'Community' } // or a nested membership ref + ]} +); +// facets: +{ $Type: 'UI.ReferenceFacet', Label: 'Knowledge Graph', ID: 'KgFieldsFacet', Target: '@UI.FieldGroup#KnowledgeGraph' }, +{ $Type: 'UI.ReferenceFacet', Label: 'Concepts Taught', ID: 'ConceptsTaughtFacet', Target: 'conceptLinks/@UI.LineItem' }, +{ $Type: 'UI.ReferenceFacet', Label: 'Co-Completed', ID: 'CoCompletionsFacet', Target: 'coCompletions/@UI.LineItem' }, +``` + +> `rank.score` path-navigation in a FieldGroup requires the to-one `rank` association from Task 1; if FE rejects the deep path, expose a flattened `virtual pageRank : Decimal` on `Tutorials` populated fail-open in `after('READ')` instead. + +- [ ] **Step 4: Run to verify it passes** — → PASS; `npx cds deploy --to sqlite::memory:` clean. +- [ ] **Step 5: Commit** + +```bash +git add app/admin-annotations.cds test/unit/annotations-kg.test.js +git commit -m "feat(admin-ui): Knowledge Graph facet (concepts/PageRank/community/co-completions) (#WS4)" +``` + +### Task 4: Hybrid guard — KG facet reads fail-open + +**Files:** +- Test: `test/hybrid/kg-facet.test.js` + +- [ ] **Step 1: Write the hybrid test** + +```js +// test/hybrid/kg-facet.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +describe('KG facet reads (hybrid)', () => { + let admin; beforeAll(async () => { admin = await cds.connect.to('AdminService') }) + it('reads a tutorial with KG associations expanded without error', async () => { + const t = await admin.run(SELECT.one.from('AdminService.Tutorials').columns('ID','slug')) + expect(t).toBeTruthy() + const links = await admin.run(SELECT.from('AdminService.TutorialConceptLinks').where({ tutorial_ID: t.ID })) + expect(Array.isArray(links)).toBe(true) // may be empty in DEV — that's fine + const co = await admin.run(SELECT.from('AdminService.CoCompletions').limit(1)) + expect(Array.isArray(co)).toBe(true) + }) +}) +``` + +- [ ] **Step 2: Run** `npm run test:hybrid -- test/hybrid/kg-facet.test.js` → PASS (empty arrays acceptable). +- [ ] **Step 3: Commit** + +```bash +git add test/hybrid/kg-facet.test.js +git commit -m "test(kg): hybrid guard for KG facet reads (#WS4)" +``` + +--- + +## Final verification + +- [ ] `npm test` all green; `npx cds deploy --to sqlite::memory:` clean. +- [ ] DEV post-deploy: reference tutorial OP shows a Knowledge Graph facet — concepts taught with confidence, PageRank + community label, co-completed neighbors. Empty sections render as FE "No data" (never a 500). +- [ ] PR targets DEV. Note: DEV-only until PROD Louvain/community data verifies (#1126 posture). + +## Self-review notes + +- **Spec coverage:** WS4 concepts/prerequisites (Task 1/3), PageRank (Task 1/3), community label (Task 2/3), co-completed neighbors (Task 1/3). +- **Verify against live code before writing each task:** exact `TutorialConceptLinks` join column (`tutorial` vs `source`) + `predicate`/`confidence` names; `TutorialRank` score field name; `CoCompletions` column names (`tutorialA`/`tutorialB`/`weight`); whether `Tutorials.conceptLinks` is already carried via the db-entity injection (`db/knowledge-graph.cds:94-97`); whether `KgCommunityMembers` exposes a slug/label to associate on. Each is grounded by the inventory research but must be confirmed at the touched lines. Prefer associations over computed virtuals; only add a fail-open `after('READ')` decorator where a join isn't expressible. From 5f74833a10e574db1f5036091e81642f0b5ace87 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 13:07:29 -0400 Subject: [PATCH 07/71] feat(publish): self-heal categories via classifyAndPersist after upsert (#WS1) --- srv/lib/content-publish-session.js | 15 ++++++++++++ test/unit/publish-category-selfheal.test.js | 27 +++++++++++++++++++++ 2 files changed, 42 insertions(+) create mode 100644 test/unit/publish-category-selfheal.test.js diff --git a/srv/lib/content-publish-session.js b/srv/lib/content-publish-session.js index ce3b12bfd..7a5f487b7 100644 --- a/srv/lib/content-publish-session.js +++ b/srv/lib/content-publish-session.js @@ -11,6 +11,19 @@ import { tutorialsTableInfo } from './_tutorials-table.js'; import { logPipelineStart, logPipelineEnd, logPipelineItem } from './pipeline-log.js'; import { resolveTutorialAuthor } from './resolve-tutorial-author.js'; import { isDeltaWrite, isDeltaSkipCarryForward } from './content-delta-flags.js'; +import { classifyAndPersist } from './category-classifier.js'; + +// Exported for unit testing; classifies touched tutorials without ever throwing +// into the publish tx (publish bypasses the CAP after('CREATE') classifier hook). +export async function classifyTouchedTutorials(tutorialIds) { + await Promise.all( + (tutorialIds || []).map((id) => + Promise.resolve() + .then(() => classifyAndPersist('tutorial', id)) + .catch((e) => console.warn('[publish] category classify skipped', id, e?.message)), + ), + ); +} const LOG = cds.log('content-publish'); const LOCK_NAME = 'content-publish'; @@ -208,6 +221,8 @@ export function createSessionHelpers({ namespace }) { } catch (err) { LOG.warn('linkTutorialAuthorship failed; skipping', err); } + // Fire-and-forget: keep categories populated for publish-created tutorials. + classifyTouchedTutorials(tutorialIds); } if (Object.keys(bodyTexts).length > 0) { await upsertBodyTexts(namespace, bodyTexts); diff --git a/test/unit/publish-category-selfheal.test.js b/test/unit/publish-category-selfheal.test.js new file mode 100644 index 000000000..cc66dbea7 --- /dev/null +++ b/test/unit/publish-category-selfheal.test.js @@ -0,0 +1,27 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' + +// vi.mock is hoisted to the top of the file by Vitest, so the factory runs +// before any const declarations. vi.hoisted() lets us declare classifySpy in +// a way that is also hoisted, keeping the reference valid inside the factory. +const classifySpy = vi.hoisted(() => vi.fn().mockResolvedValue(undefined)) +vi.mock('../../srv/lib/category-classifier.js', () => ({ + classifyAndPersist: classifySpy, +})) + +import { classifyTouchedTutorials } from '../../srv/lib/content-publish-session.js' + +describe('publish category self-heal', () => { + beforeEach(() => classifySpy.mockClear()) + + it('classifies every touched tutorial id, fire-and-forget', async () => { + await classifyTouchedTutorials(['id-a', 'id-b']) + expect(classifySpy).toHaveBeenCalledTimes(2) + expect(classifySpy).toHaveBeenCalledWith('tutorial', 'id-a') + expect(classifySpy).toHaveBeenCalledWith('tutorial', 'id-b') + }) + + it('never rejects even if a classification throws', async () => { + classifySpy.mockRejectedValueOnce(new Error('boom')) + await expect(classifyTouchedTutorials(['id-a'])).resolves.toBeUndefined() + }) +}) From 170965e18d6658df5b066901d87ae35271c71bfa Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 13:11:42 -0400 Subject: [PATCH 08/71] test(publish): hybrid guard for category population (#WS1) --- test/hybrid/publish-categories.test.js | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 test/hybrid/publish-categories.test.js diff --git a/test/hybrid/publish-categories.test.js b/test/hybrid/publish-categories.test.js new file mode 100644 index 000000000..aca607160 --- /dev/null +++ b/test/hybrid/publish-categories.test.js @@ -0,0 +1,26 @@ +// test/hybrid/publish-categories.test.js +// Hybrid guard: verifies published tutorials populate categories from the classifier. +// Requires: real HANA + cds bind --exec, and seed embeddings in Categories table. + +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' + +describe('publish populates categories (hybrid)', () => { + let db + beforeAll(async () => { db = await cds.connect.to('db') }) + + it('a freshly published tutorial has >= 0 category rows and no orphan write errors', async () => { + // Precondition: category seed embeddings must exist in this env. + const { Categories } = cds.entities('com.sap.developers.ims') + const seeds = await db.run(SELECT.from(Categories)) + expect(seeds.length).toBeGreaterThan(0) // else run embedAllSeeds first + + // Assert the classifier is reachable and idempotent for a known slug. + // (Use a slug known to exist in the bound DB.) + const { Tutorials, TutorialCategories } = cds.entities('com.sap.developers.ims') + const t = await db.run(SELECT.one.from(Tutorials).columns('ID', 'slug')) + expect(t).toBeTruthy() + const rows = await db.run(SELECT.from(TutorialCategories).where({ tutorial_ID: t.ID })) + expect(Array.isArray(rows)).toBe(true) + }) +}) From 67a5369ab669f7982b1fa94d226e3dcb47fc2bb0 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 13:17:50 -0400 Subject: [PATCH 09/71] feat(db): add GitHub link columns to TutorialContributors (#WS2) --- db/last-dev/csn.json | 15 +++++++++++++++ db/schema.cds | 3 +++ test/unit/schema-contributors.test.js | 14 ++++++++++++++ 3 files changed, 32 insertions(+) create mode 100644 test/unit/schema-contributors.test.js diff --git a/db/last-dev/csn.json b/db/last-dev/csn.json index c574809bc..5b4c3460d 100644 --- a/db/last-dev/csn.json +++ b/db/last-dev/csn.json @@ -1075,6 +1075,21 @@ "length": 50, "@cds.persistence.name": "ROLE" }, + "login": { + "type": "cds.String", + "length": 255, + "@cds.persistence.name": "LOGIN" + }, + "avatarUrl": { + "type": "cds.String", + "length": 1024, + "@cds.persistence.name": "AVATARURL" + }, + "profileUrl": { + "type": "cds.String", + "length": 1024, + "@cds.persistence.name": "PROFILEURL" + }, "user_ID": { "type": "cds.String", "length": 36, diff --git a/db/schema.cds b/db/schema.cds index 07ffce7df..f80399cc8 100644 --- a/db/schema.cds +++ b/db/schema.cds @@ -453,6 +453,9 @@ entity TutorialContributors : cuid, LegacyKeyed { name : String(255); email : String(255); role : String(50); + login : String(255); // GitHub handle + avatarUrl : String(1024); // https://github.com/.png + profileUrl : String(1024); // https://github.com/ user : Association to Users; } diff --git a/test/unit/schema-contributors.test.js b/test/unit/schema-contributors.test.js new file mode 100644 index 000000000..be6d96357 --- /dev/null +++ b/test/unit/schema-contributors.test.js @@ -0,0 +1,14 @@ +// test/unit/schema-contributors.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' + +describe('TutorialContributors schema', () => { + let m + beforeAll(async () => { m = await cds.load('*') }) + it('has GitHub link columns', () => { + const e = m.definitions['com.sap.developers.ims.TutorialContributors'] + expect(e.elements.login).toBeTruthy() + expect(e.elements.avatarUrl).toBeTruthy() + expect(e.elements.profileUrl).toBeTruthy() + }) +}) From 809a433fc3dba54b0f6081673218496824c3d0ab Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 13:28:09 -0400 Subject: [PATCH 10/71] fix(db): emit HANA migration=3 for TutorialContributors GitHub columns (#WS2) Root cause: db/last-dev/csn.json (beforeImage for cds compile.to.hana diff) was updated in the prior commit, so cds build --production saw zero diff and emitted no migration. Workaround: restored csn.json to HEAD~1 baseline, ran cds build --production with the second hana/dest:db task temporarily removed (cds-caching/db/statistics clobber hazard), copied gen/ result to db/src/, restored .cdsrc.json, re-ran deploy to regenerate csn.json correctly. --- ...developers.ims.TutorialContributors.hdbmigrationtable | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/db/src/com.sap.developers.ims.TutorialContributors.hdbmigrationtable b/db/src/com.sap.developers.ims.TutorialContributors.hdbmigrationtable index 4d27bb3be..8d7511e59 100644 --- a/db/src/com.sap.developers.ims.TutorialContributors.hdbmigrationtable +++ b/db/src/com.sap.developers.ims.TutorialContributors.hdbmigrationtable @@ -1,4 +1,4 @@ -== version=2 +== version=3 COLUMN TABLE com_sap_developers_ims_TutorialContributors ( ID NVARCHAR(36) NOT NULL, legacyId INTEGER, @@ -6,10 +6,17 @@ COLUMN TABLE com_sap_developers_ims_TutorialContributors ( name NVARCHAR(255), email NVARCHAR(255), role NVARCHAR(50), + login NVARCHAR(255), + avatarUrl NVARCHAR(1024), + profileUrl NVARCHAR(1024), user_ID NVARCHAR(36), PRIMARY KEY(ID) ) +== migration=3 +-- generated by cds-compiler version 7.0.1 +ALTER TABLE com_sap_developers_ims_TutorialContributors ADD (login NVARCHAR(255), avatarUrl NVARCHAR(1024), profileUrl NVARCHAR(1024)); + == migration=2 -- generated by cds-compiler version 6.9.0 ALTER TABLE com_sap_developers_ims_TutorialContributors ADD (user_ID NVARCHAR(36)); From 6c2047681bdc32a2c6745d22dba33b08a2b2dbc3 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 13:33:20 -0400 Subject: [PATCH 11/71] feat(fetch): write contributors sidecar from git contributor list (#WS2) --- scripts/fetch-tutorials.ts | 9 +++++++++ scripts/parsers/contributors-sidecar.ts | 15 +++++++++++++++ test/unit/contributors-sidecar.test.js | 17 +++++++++++++++++ 3 files changed, 41 insertions(+) create mode 100644 scripts/parsers/contributors-sidecar.ts create mode 100644 test/unit/contributors-sidecar.test.js diff --git a/scripts/fetch-tutorials.ts b/scripts/fetch-tutorials.ts index 57edb7a1b..d8e146211 100644 --- a/scripts/fetch-tutorials.ts +++ b/scripts/fetch-tutorials.ts @@ -31,6 +31,7 @@ import type { CatalogTutorialMeta, CategoryMeta, Mission, MissionHierarchy, Hier import { QUESTION_TYPE_TEXT } from './parsers/types.js' import { advocateLoginToSlug, type AuthorTutorialRow } from './parsers/author-index.js' import { writeAuthorPages } from './lib/author-pages-writer.js' +import { buildContributorsSidecar } from './parsers/contributors-sidecar.js' const __dirname = dirname(fileURLToPath(import.meta.url)) @@ -1076,6 +1077,14 @@ async function main() { } } + const contribSidecar = buildContributorsSidecar(t.slug, contributors) + if (contribSidecar) { + writeFileSync( + join(CACHE_DIR, `${t.slug.toLowerCase()}.contributors.json`), + JSON.stringify(contribSidecar, null, 2), + ) + } + const rawNavSlugs = [...new Set([frontmatter.primary_tag ?? '', ...(frontmatter.tags ?? [])])] .map(s => s.replace(/\\/g, '')).filter(s => s.length > 0) diff --git a/scripts/parsers/contributors-sidecar.ts b/scripts/parsers/contributors-sidecar.ts new file mode 100644 index 000000000..f9eed7f84 --- /dev/null +++ b/scripts/parsers/contributors-sidecar.ts @@ -0,0 +1,15 @@ +export interface SidecarContributor { login: string; name: string; email: string; avatarUrl: string } +export interface ContributorsSidecar { slug: string; contributors: SidecarContributor[] } + +export function buildContributorsSidecar( + slug: string, + contributors: Array>, +): ContributorsSidecar | null { + if (!contributors || contributors.length === 0) return null + return { + slug: slug.toLowerCase(), + contributors: contributors.slice(0, 10).map((c) => ({ + login: c.login ?? '', name: c.name ?? '', email: c.email ?? '', avatarUrl: c.avatarUrl ?? '', + })), + } +} diff --git a/test/unit/contributors-sidecar.test.js b/test/unit/contributors-sidecar.test.js new file mode 100644 index 000000000..76339a8b9 --- /dev/null +++ b/test/unit/contributors-sidecar.test.js @@ -0,0 +1,17 @@ +import { describe, it, expect } from 'vitest' +import { buildContributorsSidecar } from '../../scripts/parsers/contributors-sidecar' + +describe('buildContributorsSidecar', () => { + it('lowercases slug and caps at 10', () => { + const contribs = Array.from({ length: 12 }, (_, i) => ({ + login: `u${i}`, name: `N${i}`, email: `${i}@x.com`, avatarUrl: `a${i}`, + })) + const out = buildContributorsSidecar('My-Slug', contribs) + expect(out.slug).toBe('my-slug') + expect(out.contributors).toHaveLength(10) + expect(out.contributors[0]).toEqual({ login: 'u0', name: 'N0', email: '0@x.com', avatarUrl: 'a0' }) + }) + it('returns null when no contributors', () => { + expect(buildContributorsSidecar('s', [])).toBeNull() + }) +}) From 27386a5bade652cc22770b42616cfb6a5412e04d Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 13:39:56 -0400 Subject: [PATCH 12/71] feat(publish): server REPLACE handler for TutorialContributors (#WS2) --- .deploy/mta.yaml | 2 +- srv/lib/contributors-publish.js | 80 ++++++++++++++++++++++++++ srv/server.js | 9 +++ test/unit/contributors-publish.test.js | 35 +++++++++++ 4 files changed, 125 insertions(+), 1 deletion(-) create mode 100644 srv/lib/contributors-publish.js create mode 100644 test/unit/contributors-publish.test.js diff --git a/.deploy/mta.yaml b/.deploy/mta.yaml index 1463ec2ad..192afa606 100644 --- a/.deploy/mta.yaml +++ b/.deploy/mta.yaml @@ -172,7 +172,7 @@ modules: - cp -r ../../hugo/assets ./hugo/assets - cp -r ../../hugo/data ./hugo/data - cp -r ../../hugo/i18n ./hugo/i18n - - bash -c "mkdir -p srv/jobs && mkdir -p srv/handlers && mkdir -p srv/lib/branch && mkdir -p srv/lib/runtime-config && mkdir -p srv/lib/prompts && mkdir -p srv/lib/kg && mkdir -p srv/mcp/prompts && cp ../../srv/lib/branch/condition.js ../../srv/lib/branch/engine.js ../../srv/lib/branch/ranker.js ../../srv/lib/branch/user-state.js ../../srv/lib/branch/loaders.js ../../srv/lib/branch/mission-detail.js ../../srv/lib/branch/slug-key.js ../../srv/lib/branch/decide.js ../../srv/lib/branch/joule-tool.js ../../srv/lib/branch/branch-telemetry.js ../../srv/lib/branch/group-by-alt.js ../../srv/lib/branch/profile-fields.js ../../srv/lib/branch/profile-override.js srv/lib/branch/ && cp ../../srv/lib/runtime-config/kg-settings.js ../../srv/lib/runtime-config/ui-events-settings.js ../../srv/lib/runtime-config/search-settings.js ../../srv/lib/runtime-config/navigator-settings.js ../../srv/lib/runtime-config/display-settings.js ../../srv/lib/runtime-config/tenant-settings.js ../../srv/lib/runtime-config/alert-settings.js srv/lib/runtime-config/ && cp ../../srv/lib/kg/on-demand-enqueue.js ../../srv/lib/kg/on-demand-cosine-rank.js srv/lib/kg/ && cp ../../srv/lib/credstore.js ../../srv/lib/secret-resolver.js ../../srv/lib/content-store.js ../../srv/lib/content-delta-flags.js ../../srv/lib/content-cache-coherence.js ../../srv/lib/edge-cache-headers.js ../../srv/lib/content-publish-session.js ../../srv/lib/resolve-tutorial-author.js ../../srv/lib/_tutorials-table.js ../../srv/lib/catalog-renderer.js ../../srv/lib/catalog-data.js ../../srv/lib/catalog-mission-hierarchy.js ../../srv/lib/chrome-shell.js ../../srv/lib/pipeline-log.js ../../srv/lib/legacy-id.js ../../srv/lib/embedding-pipeline.js ../../srv/lib/step-text-extractor.js ../../srv/lib/embedding-client.js ../../srv/lib/step-vectors.js ../../srv/lib/user-progress.js ../../srv/lib/co-completion.js ../../srv/lib/tutorial-centroid.js ../../srv/lib/tag-label-map.js ../../srv/lib/code-check-tool.js ../../srv/lib/code-check-prompt.js ../../srv/lib/code-check-handler.js ../../srv/lib/code-check-llm.js ../../srv/lib/code-check-step-loader.js ../../srv/lib/code-check-spec-publish.js ../../srv/lib/validate-answer-spec-publish.js ../../srv/lib/category-classifier.js ../../srv/lib/category-classifier-llm.js ../../srv/lib/category-seed-embeddings.js ../../srv/lib/build-catalog-categories.js ../../srv/lib/chat-settings-resolver.js ../../srv/lib/kg-extract.js ../../srv/lib/kg-queries.js ../../srv/lib/kg-projection.js ../../srv/lib/kg-similarity.js ../../srv/lib/kg-cycles.js ../../srv/lib/kg-graph-rebuild.js ../../srv/lib/kg-sparql-client.js ../../srv/lib/kg-merge-pair.js ../../srv/lib/kg-concept-loader.js ../../srv/lib/kg-neighborhood-cache.js ../../srv/lib/kg-neighborhood-merge.js ../../srv/lib/kg-neighborhood-full-helpers.js ../../srv/lib/kg-other-resources-loader.js ../../srv/lib/kg-stamp-meta-text.js ../../srv/lib/kg-tutorial-teaches-map.js ../../srv/lib/kg-resource-type-config.js ../../srv/lib/kg-meta-formatters.js ../../srv/lib/discovery-mission-categories.js ../../srv/lib/external-content-ttl.js ../../srv/lib/recompute-tutorial-progress-bulk-sql.js ../../srv/lib/youtube-fetcher.js ../../srv/lib/homepage-events-merger.js ../../srv/lib/homepage-rss-fetcher.js ../../srv/lib/rss-parse.js ../../srv/lib/community-blogs-fetcher.js ../../srv/lib/community-blog-source-defaults.js ../../srv/lib/community-blogs-classifier.js ../../srv/lib/safe-fetch.js ../../srv/lib/curl-transport.js ../../srv/lib/khoros-transport.js ../../srv/lib/explainer-generator.js ../../srv/lib/_token-cost.js ../../srv/lib/metrics.js ../../srv/lib/alerting.js ../../srv/lib/relevance-classifier.js ../../srv/lib/relevance-seed-embeddings.js ../../srv/lib/relevance-keyword-rules.js ../../srv/lib/canonicalize-link.js ../../srv/lib/detect-language-en.js ../../srv/lib/kg-community-coverage.js ../../srv/lib/page-key-map.js ../../srv/lib/page-fallback.js ../../srv/lib/task-record-submission-id.js ../../srv/lib/image-store.cjs ../../srv/lib/image-ingest.cjs ../../srv/lib/image-source-handler.js ../../srv/lib/img-cdn-fetch.cjs ../../srv/lib/img-cdn-retry.cjs ../../srv/lib/image-warm-utils.js ../../srv/lib/attachment-store.cjs ../../srv/lib/attachment-ingest.cjs ../../srv/lib/attachment-mime.cjs ../../srv/lib/attachment-warm-utils.js ../../srv/lib/attachment-source-handler.js ../../srv/lib/attachment-ingest-handler.js ../../srv/lib/island-manifest.json srv/lib/ && mkdir -p srv/lib/feature-flags && cp ../../srv/lib/feature-flags/db-flags.js ../../srv/lib/feature-flags/registry.js srv/lib/feature-flags/ && cp ../../srv/handlers/categories-after-hooks.js ../../srv/handlers/completion-path-items-altgroup.js srv/handlers/ && mkdir -p srv && cp ../../srv/content-moderation-service.js srv/ && cp ../../srv/jobs/consolidate-concepts-job.js ../../srv/jobs/extract-concepts-job.js ../../srv/jobs/job-lock.js ../../srv/jobs/secret-expiry-check.js ../../srv/jobs/homepage-link-health.js ../../srv/jobs/kg-ondemand-job.js ../../srv/jobs/community-blogs-fetch-job.js ../../srv/jobs/community-blogs-classify-job.js ../../srv/jobs/fetch-news-job.js srv/jobs/ && cp ../../srv/lib/prompts/explainer-verb.md ../../srv/lib/prompts/explainer-shelf.md ../../srv/lib/prompts/explainer-shelf-entry.md ../../srv/lib/prompts/community-blogs-classifier.md srv/lib/prompts/ && cp ../../srv/mcp/prompts/summarize_mission_for_beginner.md ../../srv/mcp/prompts/generate_lab_exercise.md ../../srv/mcp/prompts/explain_concept.md ../../srv/mcp/prompts/suggest_learning_path.md srv/mcp/prompts/" + - bash -c "mkdir -p srv/jobs && mkdir -p srv/handlers && mkdir -p srv/lib/branch && mkdir -p srv/lib/runtime-config && mkdir -p srv/lib/prompts && mkdir -p srv/lib/kg && mkdir -p srv/mcp/prompts && cp ../../srv/lib/branch/condition.js ../../srv/lib/branch/engine.js ../../srv/lib/branch/ranker.js ../../srv/lib/branch/user-state.js ../../srv/lib/branch/loaders.js ../../srv/lib/branch/mission-detail.js ../../srv/lib/branch/slug-key.js ../../srv/lib/branch/decide.js ../../srv/lib/branch/joule-tool.js ../../srv/lib/branch/branch-telemetry.js ../../srv/lib/branch/group-by-alt.js ../../srv/lib/branch/profile-fields.js ../../srv/lib/branch/profile-override.js srv/lib/branch/ && cp ../../srv/lib/runtime-config/kg-settings.js ../../srv/lib/runtime-config/ui-events-settings.js ../../srv/lib/runtime-config/search-settings.js ../../srv/lib/runtime-config/navigator-settings.js ../../srv/lib/runtime-config/display-settings.js ../../srv/lib/runtime-config/tenant-settings.js ../../srv/lib/runtime-config/alert-settings.js srv/lib/runtime-config/ && cp ../../srv/lib/kg/on-demand-enqueue.js ../../srv/lib/kg/on-demand-cosine-rank.js srv/lib/kg/ && cp ../../srv/lib/credstore.js ../../srv/lib/secret-resolver.js ../../srv/lib/content-store.js ../../srv/lib/content-delta-flags.js ../../srv/lib/content-cache-coherence.js ../../srv/lib/edge-cache-headers.js ../../srv/lib/content-publish-session.js ../../srv/lib/resolve-tutorial-author.js ../../srv/lib/_tutorials-table.js ../../srv/lib/catalog-renderer.js ../../srv/lib/catalog-data.js ../../srv/lib/catalog-mission-hierarchy.js ../../srv/lib/chrome-shell.js ../../srv/lib/pipeline-log.js ../../srv/lib/legacy-id.js ../../srv/lib/embedding-pipeline.js ../../srv/lib/step-text-extractor.js ../../srv/lib/embedding-client.js ../../srv/lib/step-vectors.js ../../srv/lib/user-progress.js ../../srv/lib/co-completion.js ../../srv/lib/tutorial-centroid.js ../../srv/lib/tag-label-map.js ../../srv/lib/code-check-tool.js ../../srv/lib/code-check-prompt.js ../../srv/lib/code-check-handler.js ../../srv/lib/code-check-llm.js ../../srv/lib/code-check-step-loader.js ../../srv/lib/code-check-spec-publish.js ../../srv/lib/validate-answer-spec-publish.js ../../srv/lib/category-classifier.js ../../srv/lib/category-classifier-llm.js ../../srv/lib/category-seed-embeddings.js ../../srv/lib/build-catalog-categories.js ../../srv/lib/chat-settings-resolver.js ../../srv/lib/kg-extract.js ../../srv/lib/kg-queries.js ../../srv/lib/kg-projection.js ../../srv/lib/kg-similarity.js ../../srv/lib/kg-cycles.js ../../srv/lib/kg-graph-rebuild.js ../../srv/lib/kg-sparql-client.js ../../srv/lib/kg-merge-pair.js ../../srv/lib/kg-concept-loader.js ../../srv/lib/kg-neighborhood-cache.js ../../srv/lib/kg-neighborhood-merge.js ../../srv/lib/kg-neighborhood-full-helpers.js ../../srv/lib/kg-other-resources-loader.js ../../srv/lib/kg-stamp-meta-text.js ../../srv/lib/kg-tutorial-teaches-map.js ../../srv/lib/kg-resource-type-config.js ../../srv/lib/kg-meta-formatters.js ../../srv/lib/discovery-mission-categories.js ../../srv/lib/external-content-ttl.js ../../srv/lib/recompute-tutorial-progress-bulk-sql.js ../../srv/lib/youtube-fetcher.js ../../srv/lib/homepage-events-merger.js ../../srv/lib/homepage-rss-fetcher.js ../../srv/lib/rss-parse.js ../../srv/lib/community-blogs-fetcher.js ../../srv/lib/community-blog-source-defaults.js ../../srv/lib/community-blogs-classifier.js ../../srv/lib/safe-fetch.js ../../srv/lib/curl-transport.js ../../srv/lib/khoros-transport.js ../../srv/lib/explainer-generator.js ../../srv/lib/_token-cost.js ../../srv/lib/metrics.js ../../srv/lib/alerting.js ../../srv/lib/relevance-classifier.js ../../srv/lib/relevance-seed-embeddings.js ../../srv/lib/relevance-keyword-rules.js ../../srv/lib/canonicalize-link.js ../../srv/lib/detect-language-en.js ../../srv/lib/kg-community-coverage.js ../../srv/lib/page-key-map.js ../../srv/lib/page-fallback.js ../../srv/lib/task-record-submission-id.js ../../srv/lib/image-store.cjs ../../srv/lib/image-ingest.cjs ../../srv/lib/image-source-handler.js ../../srv/lib/img-cdn-fetch.cjs ../../srv/lib/img-cdn-retry.cjs ../../srv/lib/image-warm-utils.js ../../srv/lib/attachment-store.cjs ../../srv/lib/attachment-ingest.cjs ../../srv/lib/attachment-mime.cjs ../../srv/lib/attachment-warm-utils.js ../../srv/lib/attachment-source-handler.js ../../srv/lib/attachment-ingest-handler.js ../../srv/lib/contributors-publish.js ../../srv/lib/island-manifest.json srv/lib/ && mkdir -p srv/lib/feature-flags && cp ../../srv/lib/feature-flags/db-flags.js ../../srv/lib/feature-flags/registry.js srv/lib/feature-flags/ && cp ../../srv/handlers/categories-after-hooks.js ../../srv/handlers/completion-path-items-altgroup.js srv/handlers/ && mkdir -p srv && cp ../../srv/content-moderation-service.js srv/ && cp ../../srv/jobs/consolidate-concepts-job.js ../../srv/jobs/extract-concepts-job.js ../../srv/jobs/job-lock.js ../../srv/jobs/secret-expiry-check.js ../../srv/jobs/homepage-link-health.js ../../srv/jobs/kg-ondemand-job.js ../../srv/jobs/community-blogs-fetch-job.js ../../srv/jobs/community-blogs-classify-job.js ../../srv/jobs/fetch-news-job.js srv/jobs/ && cp ../../srv/lib/prompts/explainer-verb.md ../../srv/lib/prompts/explainer-shelf.md ../../srv/lib/prompts/explainer-shelf-entry.md ../../srv/lib/prompts/community-blogs-classifier.md srv/lib/prompts/ && cp ../../srv/mcp/prompts/summarize_mission_for_beginner.md ../../srv/mcp/prompts/generate_lab_exercise.md ../../srv/mcp/prompts/explain_concept.md ../../srv/mcp/prompts/suggest_learning_path.md srv/mcp/prompts/" - bash -c "node -e \"const p=require('./package.json'); p.dependencies=Object.assign(p.dependencies||{},{cheerio:'^1.2.0','@sap-ai-sdk/foundation-models':'^2.10.0'}); require('fs').writeFileSync('./package.json', JSON.stringify(p,null,2));\"" properties: EXPOSE_CAP_UI: false diff --git a/srv/lib/contributors-publish.js b/srv/lib/contributors-publish.js new file mode 100644 index 000000000..60a576505 --- /dev/null +++ b/srv/lib/contributors-publish.js @@ -0,0 +1,80 @@ +// srv/lib/contributors-publish.js +// Handler for POST /content/publish-contributors. +// +// Bearer auth is delegated to `contentAuthMiddleware` from +// srv/lib/content-store.js — same shape as /content/validate-answer-specs +// and /content/code-check-specs: 503 when CONTENT_API_KEY is unset, +// 401 on missing Bearer header, 403 on wrong key, with timing-safe comparison. +// This handler runs ONLY after auth has succeeded. +// +// Accepts `{ slug, contributors: [{login,name,email,avatarUrl}] }` and +// REPLACE-per-slug: DELETEs all TutorialContributors rows for that tutorial +// then INSERTs the new set atomically inside cds.tx(). Publishing slug A +// never touches slug B's rows. + +import cds from '@sap/cds' + +const NS = 'com.sap.developers.ims' + +export function githubProfileUrl(login) { + return login ? `https://github.com/${login}` : null +} + +/** + * Core, unit-testable: REPLACE all contributor rows for one slug. + * @param {object} db – connected CDS db service (cds.connect.to('db')) + * @param {string} slug – tutorial slug (case-insensitive) + * @param {Array} contributors – array of {login,name,email,avatarUrl} + */ +export async function replaceContributorsForSlug(db, slug, contributors) { + const { Tutorials, TutorialContributors } = cds.entities(NS) + const lcSlug = String(slug || '').toLowerCase() + const tut = await db.run(SELECT.one.from(Tutorials).columns('ID').where({ slug: lcSlug })) + if (!tut) return { ok: false, reason: 'tutorial_not_found', slug: lcSlug } + + const entries = (contributors || []) + .filter((c) => c && (c.login || c.name || c.email)) + .slice(0, 10) + .map((c) => ({ + ID: cds.utils.uuid(), + tutorial_ID: tut.ID, + login: (c.login || '').slice(0, 255), + name: (c.name || '').slice(0, 255), + email: (c.email || '').slice(0, 255), + avatarUrl: (c.avatarUrl || '').slice(0, 1024), + profileUrl: githubProfileUrl(c.login), + })) + + await cds.tx(async (tx) => { + await tx.run(DELETE.from(TutorialContributors).where({ tutorial_ID: tut.ID })) + if (entries.length) await tx.run(INSERT.into(TutorialContributors).entries(entries)) + }) + return { ok: true, slug: lcSlug, count: entries.length } +} + +/** + * Express handler mirroring the validate-answer-specs route shape. + * Mounted in server.js with contentAuthMiddleware + express.json(). + */ +export async function publishContributors(req, res) { + try { + const { slug, contributors } = req.body || {} + if (!slug || !Array.isArray(contributors)) { + return res.status(400).json({ error: 'bad_request', detail: 'expected { slug, contributors[] }' }) + } + + // entity_not_in_model guard (QA namespace safety — mirrors validate-answer-spec-publish.js). + let entities + try { entities = cds.entities(NS) } catch { entities = null } + if (!entities || !entities.TutorialContributors) { + return res.status(409).json({ error: 'entity_not_in_model' }) + } + + const db = await cds.connect.to('db') + const result = await replaceContributorsForSlug(db, slug, contributors) + if (!result.ok) return res.status(404).json(result) + return res.json(result) + } catch (e) { + return res.status(500).json({ error: 'internal', detail: e?.message }) + } +} diff --git a/srv/server.js b/srv/server.js index 0367a3558..635f83679 100644 --- a/srv/server.js +++ b/srv/server.js @@ -65,6 +65,7 @@ import { defaultCallModel } from './lib/code-check-llm.js'; import { defaultLoadStepText } from './lib/code-check-step-loader.js'; import { codeCheckSpecPublishHandler } from './lib/code-check-spec-publish.js'; import { publishValidateAnswerSpecs } from './lib/validate-answer-spec-publish.js'; +import { publishContributors } from './lib/contributors-publish.js'; import { resolveSearchSettings } from './lib/runtime-config/search-settings.js'; import { resolveTenantSettings } from './lib/runtime-config/tenant-settings.js'; import { makeValidateAnswerHandler } from './lib/validate-answer-handler.js'; @@ -606,6 +607,14 @@ cds.on('bootstrap', (app) => { publishValidateAnswerSpecs ); + // REPLACE-per-slug handler for TutorialContributors (WS2 #task-6). + // Same auth guard and body parser as the validate-answer-specs sibling. + app.post('/content/publish-contributors', + express.json({ limit: '1mb' }), + contentAuthMiddleware, + publishContributors + ); + // Tutorial feedback bridge. Express handler (rather than letting CAP expose // the action over OData) so we can derive the originating client IP from // X-Forwarded-For and inject it into req.data via AsyncLocalStorage + a diff --git a/test/unit/contributors-publish.test.js b/test/unit/contributors-publish.test.js new file mode 100644 index 000000000..c20558d0a --- /dev/null +++ b/test/unit/contributors-publish.test.js @@ -0,0 +1,35 @@ +// test/unit/contributors-publish.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import path from 'node:path' +import cds from '@sap/cds' +import { replaceContributorsForSlug } from '../../srv/lib/contributors-publish.js' + +describe('replaceContributorsForSlug', () => { + let db + beforeAll(async () => { + await cds.deploy(path.join(process.cwd(), 'db', 'schema.cds')).to('sqlite::memory:') + db = cds.db + }) + + it('replaces rows for the slug and derives profileUrl', async () => { + const { Tutorials, TutorialContributors } = cds.entities('com.sap.developers.ims') + const ID = cds.utils.uuid() + await db.run(INSERT.into(Tutorials).entries({ ID, slug: 'demo', title: 'Demo' })) + + await replaceContributorsForSlug(db, 'DEMO', [ + { login: 'octocat', name: 'Octo Cat', email: 'o@x.com', avatarUrl: 'https://github.com/octocat.png' }, + ]) + let rows = await db.run(SELECT.from(TutorialContributors).where({ tutorial_ID: ID })) + expect(rows).toHaveLength(1) + expect(rows[0].login).toBe('octocat') + expect(rows[0].profileUrl).toBe('https://github.com/octocat') + + // Second publish REPLACES, does not append. + await replaceContributorsForSlug(db, 'demo', [ + { login: 'hubot', name: 'Hubot', email: 'h@x.com', avatarUrl: 'https://github.com/hubot.png' }, + ]) + rows = await db.run(SELECT.from(TutorialContributors).where({ tutorial_ID: ID })) + expect(rows).toHaveLength(1) + expect(rows[0].login).toBe('hubot') + }) +}) From 48ffdceb711ed89a5eac438b98eea7924c481de2 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 13:45:13 -0400 Subject: [PATCH 13/71] feat(publish): non-fatal client step to publish contributors sidecars (#WS2) --- scripts/publish-content.ts | 15 ++++ scripts/publish/publish-contributors.ts | 70 +++++++++++++++++++ test/unit/publish-contributors-client.test.js | 25 +++++++ 3 files changed, 110 insertions(+) create mode 100644 scripts/publish/publish-contributors.ts create mode 100644 test/unit/publish-contributors-client.test.js diff --git a/scripts/publish-content.ts b/scripts/publish-content.ts index 9761957db..0fb886206 100644 --- a/scripts/publish-content.ts +++ b/scripts/publish-content.ts @@ -10,6 +10,7 @@ import { withRetry, formatErrorChain } from './lib/publish-retry.js'; import { chunk, runConcurrent } from './lib/publish-batcher.js'; import { collectCodeCheckSpecs, publishCodeCheckSpecs } from './lib/publish-codecheck.js'; import { publishValidateAnswerSpecs } from './lib/publish-validate-answer.js'; +import { publishContributors } from './publish/publish-contributors.js'; import { computeOrphans, enforceCap, formatStepSummary } from './lib/purge-orphans.js'; import { discoverPageFiles, discoverAuthorPages, discoverAdvocatePages } from '../srv/lib/page-key-map.js'; @@ -1351,6 +1352,20 @@ async function main() { } } + // --- contributors sidecar publish (non-fatal auxiliary step, issue #WS2) --- + // QA channel skips: srv-qa has no ContributorCache entity, POST would 404/500. + if (channel === 'qa') { + log('[publish-contributors] skipped (channel=qa)'); + } else { + try { + const cacheDir = join(process.cwd(), '.tutorial-cache'); + const r = await publishContributors({ cacheDir, baseUrl: opts.baseUrl, apiKey: opts.apiKey }); + log(`[publish-contributors] published ${r.published}/${r.total}`); + } catch (err) { + console.error('[publish-content] contributors publish failed (non-fatal):', formatErrorChain(err)); + } + } + // --- auto-verify --- log('Verifying server state matches local...'); let postRemote: Record; diff --git a/scripts/publish/publish-contributors.ts b/scripts/publish/publish-contributors.ts new file mode 100644 index 000000000..337893c8d --- /dev/null +++ b/scripts/publish/publish-contributors.ts @@ -0,0 +1,70 @@ +// scripts/publish/publish-contributors.ts +// Non-fatal auxiliary publish step for issue #WS2. +// Walks `cacheDir` for `*.contributors.json` sidecar files emitted by +// scripts/fetch-tutorials.ts and POSTs each one to +// /content/publish-contributors (Task 6 REPLACE handler). +// +// Auth: CONTENT_API_KEY via contentAuthMiddleware (Authorization: Bearer). +// Failures are NON-FATAL — captured and returned to the caller. + +import { readdirSync, readFileSync } from 'node:fs' +import { join } from 'node:path' + +const SUFFIX = '.contributors.json' + +/** + * Walk cacheDir for *.contributors.json sidecar files, + * POST each one to /content/publish-contributors. + * + * @param opts.cacheDir Tutorial cache dir (e.g. .tutorial-cache) + * @param opts.baseUrl CAP base URL + * @param opts.apiKey CONTENT_API_KEY value + * @returns { published, total } + */ +export async function publishContributors(opts: { + cacheDir: string + baseUrl: string + apiKey: string +}): Promise<{ published: number; total: number }> { + const { cacheDir, baseUrl, apiKey } = opts + let files: string[] + try { + files = readdirSync(cacheDir).filter((f) => f.endsWith(SUFFIX)) + } catch { + return { published: 0, total: 0 } + } + + let published = 0 + for (const f of files) { + const filePath = join(cacheDir, f) + let raw: string + try { + raw = readFileSync(filePath, 'utf8') + } catch { + continue + } + + let res: Response + try { + res = await fetch(`${baseUrl}/content/publish-contributors`, { + method: 'POST', + headers: { + 'authorization': `Bearer ${apiKey}`, + 'content-type': 'application/json', + }, + body: raw, + }) + } catch (err) { + console.warn(`[publish-contributors] network error for ${f}:`, (err as Error).message) + continue + } + + if (res.ok) { + published += 1 + } else { + console.warn(`[publish-contributors] ${f} -> ${res.status}`) + } + } + + return { published, total: files.length } +} diff --git a/test/unit/publish-contributors-client.test.js b/test/unit/publish-contributors-client.test.js new file mode 100644 index 000000000..19ca2bf5d --- /dev/null +++ b/test/unit/publish-contributors-client.test.js @@ -0,0 +1,25 @@ +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest' +import { mkdtempSync, writeFileSync } from 'node:fs' +import { join } from 'node:path' +import { tmpdir } from 'node:os' +import { publishContributors } from '../../scripts/publish/publish-contributors' + +describe('publishContributors client', () => { + let dir + beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), 'contrib-')) + writeFileSync(join(dir, 'demo.contributors.json'), + JSON.stringify({ slug: 'demo', contributors: [{ login: 'octocat', name: 'O', email: 'o@x', avatarUrl: 'a' }] })) + global.fetch = vi.fn().mockResolvedValue({ ok: true, json: async () => ({ ok: true, count: 1 }) }) + }) + afterEach(() => { vi.restoreAllMocks() }) + + it('POSTs each sidecar to the endpoint', async () => { + const res = await publishContributors({ cacheDir: dir, baseUrl: 'http://x', apiKey: 'k' }) + expect(global.fetch).toHaveBeenCalledTimes(1) + const [url, opts] = global.fetch.mock.calls[0] + expect(url).toBe('http://x/content/publish-contributors') + expect(JSON.parse(opts.body).slug).toBe('demo') + expect(res.published).toBe(1) + }) +}) From 5e9192c9621f8a4160cdc7e4502ccb1564404cb6 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 13:48:25 -0400 Subject: [PATCH 14/71] feat(admin-ui): GitHub-linked login column on Contributors table (#WS2) --- app/admin-annotations.cds | 2 ++ test/unit/annotations-contributors.test.js | 13 +++++++++++++ 2 files changed, 15 insertions(+) create mode 100644 test/unit/annotations-contributors.test.js diff --git a/app/admin-annotations.cds b/app/admin-annotations.cds index caf86fb23..4c3aef516 100644 --- a/app/admin-annotations.cds +++ b/app/admin-annotations.cds @@ -714,12 +714,14 @@ annotate AdminService.Tutorials with { annotate AdminService.TutorialContributors with { name @Common.Label: 'Name'; + login @Common.Label: 'GitHub'; email @Common.Label: 'Email'; role @Common.Label: 'Role'; }; annotate AdminService.TutorialContributors with @UI.LineItem: [ { Value: name }, + { $Type: 'UI.DataFieldWithUrl', Value: login, Url: profileUrl, Label: 'GitHub' }, { Value: email }, { Value: role } ]; diff --git a/test/unit/annotations-contributors.test.js b/test/unit/annotations-contributors.test.js new file mode 100644 index 000000000..7e896f710 --- /dev/null +++ b/test/unit/annotations-contributors.test.js @@ -0,0 +1,13 @@ +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' + +describe('Contributors LineItem', () => { + let m + beforeAll(async () => { m = await cds.load('*') }) + it('LineItem includes login column', () => { + const e = m.definitions['AdminService.TutorialContributors'] + const li = e['@UI.LineItem'] + const values = li.map((x) => x.Value?.['='] || x.Value) + expect(values).toContain('login') + }) +}) From 710449011d92a2b02c9303f3ff91c4fa69d08756 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 13:50:53 -0400 Subject: [PATCH 15/71] test(publish): hybrid guard for contributor linking (#WS2) --- test/hybrid/publish-contributors.test.js | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 test/hybrid/publish-contributors.test.js diff --git a/test/hybrid/publish-contributors.test.js b/test/hybrid/publish-contributors.test.js new file mode 100644 index 000000000..2368ae948 --- /dev/null +++ b/test/hybrid/publish-contributors.test.js @@ -0,0 +1,23 @@ +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +import { replaceContributorsForSlug } from '../../srv/lib/contributors-publish.js' + +cds.test('serve', '--project', '.', '--profile', 'hybrid') + +describe('contributors publish (hybrid)', () => { + let db + beforeAll(async () => { db = await cds.connect.to('db') }) + it('links contributor rows to an existing tutorial by slug', async () => { + const { Tutorials, TutorialContributors } = cds.entities('com.sap.developers.ims') + const t = await db.run(SELECT.one.from(Tutorials).columns('ID', 'slug')) + expect(t).toBeTruthy() + await replaceContributorsForSlug(db, t.slug, [ + { login: 'octocat', name: 'Octo', email: 'o@x.com', avatarUrl: 'https://github.com/octocat.png' }, + ]) + const rows = await db.run(SELECT.from(TutorialContributors).where({ tutorial_ID: t.ID, login: 'octocat' })) + expect(rows.length).toBe(1) + expect(rows[0].profileUrl).toBe('https://github.com/octocat') + // cleanup + await db.run(DELETE.from(TutorialContributors).where({ tutorial_ID: t.ID, login: 'octocat' })) + }) +}) From 28472568d659fdbf925c386db6e5f2ac618dd93b Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 13:56:16 -0400 Subject: [PATCH 16/71] feat(db): add TutorialValidationRules entity for all rules.vr rules (#WS3) --- db/last-dev/csn.json | 60 +++++++++++++++++++ db/persistence.cds | 1 + db/schema.cds | 17 ++++++ ....TutorialValidationRules.hdbmigrationtable | 14 +++++ test/unit/schema-validation-rules.test.js | 13 ++++ 5 files changed, 105 insertions(+) create mode 100644 db/src/com.sap.developers.ims.TutorialValidationRules.hdbmigrationtable create mode 100644 test/unit/schema-validation-rules.test.js diff --git a/db/last-dev/csn.json b/db/last-dev/csn.json index 5b4c3460d..474df9979 100644 --- a/db/last-dev/csn.json +++ b/db/last-dev/csn.json @@ -3630,6 +3630,66 @@ } }, "@cds.persistence.name": "COM_SAP_DEVELOPERS_IMS_CONTENTMANIFEST" + }, + "com.sap.developers.ims.TutorialValidationRules": { + "kind": "entity", + "@cds.persistence.journal": true, + "elements": { + "tutorial_ID": { + "type": "cds.String", + "length": 36, + "@odata.foreignKey4": "tutorial", + "key": true, + "@cds.persistence.name": "TUTORIAL_ID" + }, + "stepNumber": { + "key": true, + "type": "cds.Integer", + "@cds.persistence.name": "STEPNUMBER" + }, + "questionId": { + "key": true, + "type": "cds.String", + "length": 100, + "@cds.persistence.name": "QUESTIONID" + }, + "questionText": { + "type": "cds.String", + "length": 2000, + "@cds.persistence.name": "QUESTIONTEXT" + }, + "ruleType": { + "type": "cds.String", + "length": 50, + "@cds.persistence.name": "RULETYPE" + }, + "questionType": { + "type": "cds.String", + "length": 20, + "@cds.persistence.name": "QUESTIONTYPE" + }, + "choiceMode": { + "type": "cds.String", + "length": 20, + "@cds.persistence.name": "CHOICEMODE" + }, + "options": { + "type": "cds.LargeString", + "@cds.persistence.name": "OPTIONS" + }, + "correctAnswer": { + "type": "cds.LargeString", + "@cds.persistence.name": "CORRECTANSWER" + }, + "aiGrading": { + "type": "cds.Boolean", + "default": { + "val": false + }, + "@cds.persistence.name": "AIGRADING" + } + }, + "@cds.persistence.name": "COM_SAP_DEVELOPERS_IMS_TUTORIALVALIDATIONRULES" } }, "meta": { diff --git a/db/persistence.cds b/db/persistence.cds index 97d8c91c8..1ba6e23f8 100644 --- a/db/persistence.cds +++ b/db/persistence.cds @@ -50,3 +50,4 @@ annotate ims.CatGameAwards with @cds.persistence.journal; // #2042 Hit-the-Cat // #805 — Observability annotate ims.MetricSnapshots with @cds.persistence.journal; annotate ims.PublishTimings with @cds.persistence.journal; +annotate ims.TutorialValidationRules with @cds.persistence.journal; diff --git a/db/schema.cds b/db/schema.cds index f80399cc8..e1d48af79 100644 --- a/db/schema.cds +++ b/db/schema.cds @@ -875,6 +875,23 @@ entity ValidateAnswerSpecs : managed { aiGrading : Boolean default false; } +// Full parsed rules.vr rule set for a tutorial, persisted at publish time. +// Unlike ValidateAnswerSpecs (AI-graded only), this holds ALL rule types so +// the admin Validation Questions facet can display the complete rule set. +// options and correctAnswer are JSON-serialised; null when not applicable. +entity TutorialValidationRules { + key tutorial : Association to Tutorials; + key stepNumber : Integer; + key questionId : String(100); + questionText : String(2000); + ruleType : String(50); // single-choice | multiple-choice | regex | exact-match | ... + questionType : String(20); // MCQ | TEXT + choiceMode : String(20); // single | multiple | null + options : LargeString; // JSON array of option strings (MCQ) or null + correctAnswer: LargeString; // reference answer (client-graded) or null when aiGrading + aiGrading : Boolean default false; +} + // Every learner submission. Drives offline grader-quality evaluation. // 'verdict' allows 'error' as a server-side outcome value (the LLM JSON // schema only emits 'pass' | 'partial' | 'fail'). diff --git a/db/src/com.sap.developers.ims.TutorialValidationRules.hdbmigrationtable b/db/src/com.sap.developers.ims.TutorialValidationRules.hdbmigrationtable new file mode 100644 index 000000000..99ba6938b --- /dev/null +++ b/db/src/com.sap.developers.ims.TutorialValidationRules.hdbmigrationtable @@ -0,0 +1,14 @@ +== version=1 +COLUMN TABLE com_sap_developers_ims_TutorialValidationRules ( + tutorial_ID NVARCHAR(36) NOT NULL, + stepNumber INTEGER NOT NULL, + questionId NVARCHAR(100) NOT NULL, + questionText NVARCHAR(2000), + ruleType NVARCHAR(50), + questionType NVARCHAR(20), + choiceMode NVARCHAR(20), + options NCLOB, + correctAnswer NCLOB, + aiGrading BOOLEAN DEFAULT FALSE, + PRIMARY KEY(tutorial_ID, stepNumber, questionId) +) diff --git a/test/unit/schema-validation-rules.test.js b/test/unit/schema-validation-rules.test.js new file mode 100644 index 000000000..43f892399 --- /dev/null +++ b/test/unit/schema-validation-rules.test.js @@ -0,0 +1,13 @@ +// test/unit/schema-validation-rules.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +describe('TutorialValidationRules schema', () => { + let m + beforeAll(async () => { m = await cds.load('*') }) + it('exists with expected elements', () => { + const e = m.definitions['com.sap.developers.ims.TutorialValidationRules'] + expect(e).toBeTruthy() + for (const k of ['stepNumber','questionId','questionText','ruleType','questionType','choiceMode','options','correctAnswer','aiGrading']) + expect(e.elements[k]).toBeTruthy() + }) +}) From 5e32029c6fe761a6b157e3028644962e9fed5c6f Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 14:00:17 -0400 Subject: [PATCH 17/71] feat(parser): collectAllRules for full rules.vr rule set (#WS3) --- scripts/parsers/rules.ts | 50 +++++++++++++++++++++++++++++ test/unit/collect-all-rules.test.js | 24 ++++++++++++++ 2 files changed, 74 insertions(+) create mode 100644 test/unit/collect-all-rules.test.js diff --git a/scripts/parsers/rules.ts b/scripts/parsers/rules.ts index b2791a961..0018db098 100644 --- a/scripts/parsers/rules.ts +++ b/scripts/parsers/rules.ts @@ -294,6 +294,56 @@ function parseChoiceOptions(content: string): { options: string[]; correctAnswer export { parseCodeCheckBlocks } from './codecheck.js' +export interface AllRuleRow { + stepNumber: number + questionId: string + questionText: string + ruleType: string + questionType: 'MCQ' | 'TEXT' + choiceMode: string | null + options: string | null + correctAnswer: string | null + aiGrading: boolean +} + +/** + * Collect ALL validation rules across all steps for a tutorial — both + * AI-graded and client-graded (MCQ single/multiple, exact-match text). + * + * Parallel to collectAiGradedSpecs but returns every emitted question so the + * publish pipeline can persist a full all-rules sidecar and the admin UI can + * surface a complete rule facet (issue WS3). + * + * Anti-leak contract preserved: correctAnswer is null for AI-graded rows — + * the reference answer stays server-side in ValidateAnswerSpecs only. + */ +export function collectAllRules( + map: Map, + ruleTypeByStepAndId: Map, + correctAnswerByStepAndId: Map, +): AllRuleRow[] { + const rows: AllRuleRow[] = [] + for (const [stepNumber, questions] of map.entries()) { + for (const q of questions) { + const key = `${stepNumber}:${q.id}` + const isMcq = q.type === QUESTION_TYPE_MCQ + const ai = Boolean((q as any).aiGrading) + rows.push({ + stepNumber, + questionId: q.id, + questionText: q.question, + ruleType: ruleTypeByStepAndId.get(key) ?? '', + questionType: isMcq ? 'MCQ' : 'TEXT', + choiceMode: (q as any).choiceMode ?? null, + options: isMcq && (q as any).options ? JSON.stringify((q as any).options) : null, + correctAnswer: ai ? null : (correctAnswerByStepAndId.get(key) ?? (q as any).correctAnswer ?? null), + aiGrading: ai, + }) + } + } + return rows +} + export interface AiGradedSpec { stepNumber: number questionId: string diff --git a/test/unit/collect-all-rules.test.js b/test/unit/collect-all-rules.test.js new file mode 100644 index 000000000..389a95395 --- /dev/null +++ b/test/unit/collect-all-rules.test.js @@ -0,0 +1,24 @@ +import { describe, it, expect } from 'vitest' +import { collectAllRules } from '../../scripts/parsers/rules.js' + +describe('collectAllRules', () => { + it('includes non-AI MCQ rules with options + correctAnswer', () => { + const map = new Map([[1, [ + { id: 'validate-1', question: 'Pick one', type: 'multiple-choice', options: ['A','B'], choiceMode: 'single', correctAnswer: 'A' }, + { id: 'validate-1b', question: 'AI graded', type: 'text', aiGrading: true }, + ]]]) + // Single colon key format: `${stepNumber}:${q.id}` (confirmed in rules.ts line 244) + const ruleTypeMap = new Map([['1:validate-1', 'single-choice'], ['1:validate-1b', 'regex']]) + const answerMap = new Map([['1:validate-1', 'A']]) + const rows = collectAllRules(map, ruleTypeMap, answerMap) + expect(rows).toHaveLength(2) + const mcq = rows.find((r) => r.questionId === 'validate-1') + expect(mcq.aiGrading).toBe(false) + expect(mcq.questionType).toBe('MCQ') + expect(JSON.parse(mcq.options)).toEqual(['A','B']) + expect(mcq.correctAnswer).toBe('A') + const ai = rows.find((r) => r.questionId === 'validate-1b') + expect(ai.aiGrading).toBe(true) + expect(ai.correctAnswer).toBeNull() + }) +}) From 464296adc3bfb32b73f29d2f0ad73480a9e86d79 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 14:03:25 -0400 Subject: [PATCH 18/71] feat(fetch): write validation-rules sidecar (all rule types) (#WS3) --- scripts/fetch-tutorials.ts | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/scripts/fetch-tutorials.ts b/scripts/fetch-tutorials.ts index d8e146211..ba3fb4402 100644 --- a/scripts/fetch-tutorials.ts +++ b/scripts/fetch-tutorials.ts @@ -15,7 +15,7 @@ import { navEntriesBySlug, SIDECAR_VERSION, } from './lib/content-cache.js' -import { parseRulesVrEnriched, collectAiGradedSpecs } from './parsers/rules.js' +import { parseRulesVrEnriched, collectAiGradedSpecs, collectAllRules } from './parsers/rules.js' import { expandAiAuthoredQuestions, populateAiAuthoredSiblingMaps, type ExpandStats } from './lib/expand-ai-authored.js' import { loadAiQuizCache, saveAiQuizCache } from './lib/ai-quiz-cache.js' import { callQuizModel } from '../srv/lib/ai-quiz-llm.js' @@ -1043,6 +1043,16 @@ async function main() { writeFileSync(validateSidecarPath, JSON.stringify({ slug: t.slug.toLowerCase(), specs: aiGradedSpecs }, null, 2)) } + // Write full validation-rules sidecar (all rule types, all steps). + // Consumed by the publish pipeline to upsert TutorialValidationRules rows. + const allRules = collectAllRules(validationMap, ruleTypeByStepAndId, correctAnswerByStepAndId) + if (allRules.length > 0) { + writeFileSync( + join(CACHE_DIR, `${t.slug.toLowerCase()}.validation-rules.json`), + JSON.stringify({ slug: t.slug.toLowerCase(), rules: allRules }, null, 2), + ) + } + // [#208] Anti-leak strip: AI-authored text questions had correctAnswer // restored on validationMap so populateAiAuthoredSiblingMaps (above) // could mirror it into correctAnswerByStepAndId, which collectAiGradedSpecs From bbba5c3e60dc18be9be482281eeb1b154294b3c1 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 14:07:31 -0400 Subject: [PATCH 19/71] feat(publish): server REPLACE handler for TutorialValidationRules (#WS3) --- .deploy/mta.yaml | 2 +- srv/lib/validation-rules-publish.js | 75 ++++++++++++++++++++++ srv/server.js | 9 +++ test/unit/validation-rules-publish.test.js | 28 ++++++++ 4 files changed, 113 insertions(+), 1 deletion(-) create mode 100644 srv/lib/validation-rules-publish.js create mode 100644 test/unit/validation-rules-publish.test.js diff --git a/.deploy/mta.yaml b/.deploy/mta.yaml index 192afa606..c76a7b2b5 100644 --- a/.deploy/mta.yaml +++ b/.deploy/mta.yaml @@ -172,7 +172,7 @@ modules: - cp -r ../../hugo/assets ./hugo/assets - cp -r ../../hugo/data ./hugo/data - cp -r ../../hugo/i18n ./hugo/i18n - - bash -c "mkdir -p srv/jobs && mkdir -p srv/handlers && mkdir -p srv/lib/branch && mkdir -p srv/lib/runtime-config && mkdir -p srv/lib/prompts && mkdir -p srv/lib/kg && mkdir -p srv/mcp/prompts && cp ../../srv/lib/branch/condition.js ../../srv/lib/branch/engine.js ../../srv/lib/branch/ranker.js ../../srv/lib/branch/user-state.js ../../srv/lib/branch/loaders.js ../../srv/lib/branch/mission-detail.js ../../srv/lib/branch/slug-key.js ../../srv/lib/branch/decide.js ../../srv/lib/branch/joule-tool.js ../../srv/lib/branch/branch-telemetry.js ../../srv/lib/branch/group-by-alt.js ../../srv/lib/branch/profile-fields.js ../../srv/lib/branch/profile-override.js srv/lib/branch/ && cp ../../srv/lib/runtime-config/kg-settings.js ../../srv/lib/runtime-config/ui-events-settings.js ../../srv/lib/runtime-config/search-settings.js ../../srv/lib/runtime-config/navigator-settings.js ../../srv/lib/runtime-config/display-settings.js ../../srv/lib/runtime-config/tenant-settings.js ../../srv/lib/runtime-config/alert-settings.js srv/lib/runtime-config/ && cp ../../srv/lib/kg/on-demand-enqueue.js ../../srv/lib/kg/on-demand-cosine-rank.js srv/lib/kg/ && cp ../../srv/lib/credstore.js ../../srv/lib/secret-resolver.js ../../srv/lib/content-store.js ../../srv/lib/content-delta-flags.js ../../srv/lib/content-cache-coherence.js ../../srv/lib/edge-cache-headers.js ../../srv/lib/content-publish-session.js ../../srv/lib/resolve-tutorial-author.js ../../srv/lib/_tutorials-table.js ../../srv/lib/catalog-renderer.js ../../srv/lib/catalog-data.js ../../srv/lib/catalog-mission-hierarchy.js ../../srv/lib/chrome-shell.js ../../srv/lib/pipeline-log.js ../../srv/lib/legacy-id.js ../../srv/lib/embedding-pipeline.js ../../srv/lib/step-text-extractor.js ../../srv/lib/embedding-client.js ../../srv/lib/step-vectors.js ../../srv/lib/user-progress.js ../../srv/lib/co-completion.js ../../srv/lib/tutorial-centroid.js ../../srv/lib/tag-label-map.js ../../srv/lib/code-check-tool.js ../../srv/lib/code-check-prompt.js ../../srv/lib/code-check-handler.js ../../srv/lib/code-check-llm.js ../../srv/lib/code-check-step-loader.js ../../srv/lib/code-check-spec-publish.js ../../srv/lib/validate-answer-spec-publish.js ../../srv/lib/category-classifier.js ../../srv/lib/category-classifier-llm.js ../../srv/lib/category-seed-embeddings.js ../../srv/lib/build-catalog-categories.js ../../srv/lib/chat-settings-resolver.js ../../srv/lib/kg-extract.js ../../srv/lib/kg-queries.js ../../srv/lib/kg-projection.js ../../srv/lib/kg-similarity.js ../../srv/lib/kg-cycles.js ../../srv/lib/kg-graph-rebuild.js ../../srv/lib/kg-sparql-client.js ../../srv/lib/kg-merge-pair.js ../../srv/lib/kg-concept-loader.js ../../srv/lib/kg-neighborhood-cache.js ../../srv/lib/kg-neighborhood-merge.js ../../srv/lib/kg-neighborhood-full-helpers.js ../../srv/lib/kg-other-resources-loader.js ../../srv/lib/kg-stamp-meta-text.js ../../srv/lib/kg-tutorial-teaches-map.js ../../srv/lib/kg-resource-type-config.js ../../srv/lib/kg-meta-formatters.js ../../srv/lib/discovery-mission-categories.js ../../srv/lib/external-content-ttl.js ../../srv/lib/recompute-tutorial-progress-bulk-sql.js ../../srv/lib/youtube-fetcher.js ../../srv/lib/homepage-events-merger.js ../../srv/lib/homepage-rss-fetcher.js ../../srv/lib/rss-parse.js ../../srv/lib/community-blogs-fetcher.js ../../srv/lib/community-blog-source-defaults.js ../../srv/lib/community-blogs-classifier.js ../../srv/lib/safe-fetch.js ../../srv/lib/curl-transport.js ../../srv/lib/khoros-transport.js ../../srv/lib/explainer-generator.js ../../srv/lib/_token-cost.js ../../srv/lib/metrics.js ../../srv/lib/alerting.js ../../srv/lib/relevance-classifier.js ../../srv/lib/relevance-seed-embeddings.js ../../srv/lib/relevance-keyword-rules.js ../../srv/lib/canonicalize-link.js ../../srv/lib/detect-language-en.js ../../srv/lib/kg-community-coverage.js ../../srv/lib/page-key-map.js ../../srv/lib/page-fallback.js ../../srv/lib/task-record-submission-id.js ../../srv/lib/image-store.cjs ../../srv/lib/image-ingest.cjs ../../srv/lib/image-source-handler.js ../../srv/lib/img-cdn-fetch.cjs ../../srv/lib/img-cdn-retry.cjs ../../srv/lib/image-warm-utils.js ../../srv/lib/attachment-store.cjs ../../srv/lib/attachment-ingest.cjs ../../srv/lib/attachment-mime.cjs ../../srv/lib/attachment-warm-utils.js ../../srv/lib/attachment-source-handler.js ../../srv/lib/attachment-ingest-handler.js ../../srv/lib/contributors-publish.js ../../srv/lib/island-manifest.json srv/lib/ && mkdir -p srv/lib/feature-flags && cp ../../srv/lib/feature-flags/db-flags.js ../../srv/lib/feature-flags/registry.js srv/lib/feature-flags/ && cp ../../srv/handlers/categories-after-hooks.js ../../srv/handlers/completion-path-items-altgroup.js srv/handlers/ && mkdir -p srv && cp ../../srv/content-moderation-service.js srv/ && cp ../../srv/jobs/consolidate-concepts-job.js ../../srv/jobs/extract-concepts-job.js ../../srv/jobs/job-lock.js ../../srv/jobs/secret-expiry-check.js ../../srv/jobs/homepage-link-health.js ../../srv/jobs/kg-ondemand-job.js ../../srv/jobs/community-blogs-fetch-job.js ../../srv/jobs/community-blogs-classify-job.js ../../srv/jobs/fetch-news-job.js srv/jobs/ && cp ../../srv/lib/prompts/explainer-verb.md ../../srv/lib/prompts/explainer-shelf.md ../../srv/lib/prompts/explainer-shelf-entry.md ../../srv/lib/prompts/community-blogs-classifier.md srv/lib/prompts/ && cp ../../srv/mcp/prompts/summarize_mission_for_beginner.md ../../srv/mcp/prompts/generate_lab_exercise.md ../../srv/mcp/prompts/explain_concept.md ../../srv/mcp/prompts/suggest_learning_path.md srv/mcp/prompts/" + - bash -c "mkdir -p srv/jobs && mkdir -p srv/handlers && mkdir -p srv/lib/branch && mkdir -p srv/lib/runtime-config && mkdir -p srv/lib/prompts && mkdir -p srv/lib/kg && mkdir -p srv/mcp/prompts && cp ../../srv/lib/branch/condition.js ../../srv/lib/branch/engine.js ../../srv/lib/branch/ranker.js ../../srv/lib/branch/user-state.js ../../srv/lib/branch/loaders.js ../../srv/lib/branch/mission-detail.js ../../srv/lib/branch/slug-key.js ../../srv/lib/branch/decide.js ../../srv/lib/branch/joule-tool.js ../../srv/lib/branch/branch-telemetry.js ../../srv/lib/branch/group-by-alt.js ../../srv/lib/branch/profile-fields.js ../../srv/lib/branch/profile-override.js srv/lib/branch/ && cp ../../srv/lib/runtime-config/kg-settings.js ../../srv/lib/runtime-config/ui-events-settings.js ../../srv/lib/runtime-config/search-settings.js ../../srv/lib/runtime-config/navigator-settings.js ../../srv/lib/runtime-config/display-settings.js ../../srv/lib/runtime-config/tenant-settings.js ../../srv/lib/runtime-config/alert-settings.js srv/lib/runtime-config/ && cp ../../srv/lib/kg/on-demand-enqueue.js ../../srv/lib/kg/on-demand-cosine-rank.js srv/lib/kg/ && cp ../../srv/lib/credstore.js ../../srv/lib/secret-resolver.js ../../srv/lib/content-store.js ../../srv/lib/content-delta-flags.js ../../srv/lib/content-cache-coherence.js ../../srv/lib/edge-cache-headers.js ../../srv/lib/content-publish-session.js ../../srv/lib/resolve-tutorial-author.js ../../srv/lib/_tutorials-table.js ../../srv/lib/catalog-renderer.js ../../srv/lib/catalog-data.js ../../srv/lib/catalog-mission-hierarchy.js ../../srv/lib/chrome-shell.js ../../srv/lib/pipeline-log.js ../../srv/lib/legacy-id.js ../../srv/lib/embedding-pipeline.js ../../srv/lib/step-text-extractor.js ../../srv/lib/embedding-client.js ../../srv/lib/step-vectors.js ../../srv/lib/user-progress.js ../../srv/lib/co-completion.js ../../srv/lib/tutorial-centroid.js ../../srv/lib/tag-label-map.js ../../srv/lib/code-check-tool.js ../../srv/lib/code-check-prompt.js ../../srv/lib/code-check-handler.js ../../srv/lib/code-check-llm.js ../../srv/lib/code-check-step-loader.js ../../srv/lib/code-check-spec-publish.js ../../srv/lib/validate-answer-spec-publish.js ../../srv/lib/category-classifier.js ../../srv/lib/category-classifier-llm.js ../../srv/lib/category-seed-embeddings.js ../../srv/lib/build-catalog-categories.js ../../srv/lib/chat-settings-resolver.js ../../srv/lib/kg-extract.js ../../srv/lib/kg-queries.js ../../srv/lib/kg-projection.js ../../srv/lib/kg-similarity.js ../../srv/lib/kg-cycles.js ../../srv/lib/kg-graph-rebuild.js ../../srv/lib/kg-sparql-client.js ../../srv/lib/kg-merge-pair.js ../../srv/lib/kg-concept-loader.js ../../srv/lib/kg-neighborhood-cache.js ../../srv/lib/kg-neighborhood-merge.js ../../srv/lib/kg-neighborhood-full-helpers.js ../../srv/lib/kg-other-resources-loader.js ../../srv/lib/kg-stamp-meta-text.js ../../srv/lib/kg-tutorial-teaches-map.js ../../srv/lib/kg-resource-type-config.js ../../srv/lib/kg-meta-formatters.js ../../srv/lib/discovery-mission-categories.js ../../srv/lib/external-content-ttl.js ../../srv/lib/recompute-tutorial-progress-bulk-sql.js ../../srv/lib/youtube-fetcher.js ../../srv/lib/homepage-events-merger.js ../../srv/lib/homepage-rss-fetcher.js ../../srv/lib/rss-parse.js ../../srv/lib/community-blogs-fetcher.js ../../srv/lib/community-blog-source-defaults.js ../../srv/lib/community-blogs-classifier.js ../../srv/lib/safe-fetch.js ../../srv/lib/curl-transport.js ../../srv/lib/khoros-transport.js ../../srv/lib/explainer-generator.js ../../srv/lib/_token-cost.js ../../srv/lib/metrics.js ../../srv/lib/alerting.js ../../srv/lib/relevance-classifier.js ../../srv/lib/relevance-seed-embeddings.js ../../srv/lib/relevance-keyword-rules.js ../../srv/lib/canonicalize-link.js ../../srv/lib/detect-language-en.js ../../srv/lib/kg-community-coverage.js ../../srv/lib/page-key-map.js ../../srv/lib/page-fallback.js ../../srv/lib/task-record-submission-id.js ../../srv/lib/image-store.cjs ../../srv/lib/image-ingest.cjs ../../srv/lib/image-source-handler.js ../../srv/lib/img-cdn-fetch.cjs ../../srv/lib/img-cdn-retry.cjs ../../srv/lib/image-warm-utils.js ../../srv/lib/attachment-store.cjs ../../srv/lib/attachment-ingest.cjs ../../srv/lib/attachment-mime.cjs ../../srv/lib/attachment-warm-utils.js ../../srv/lib/attachment-source-handler.js ../../srv/lib/attachment-ingest-handler.js ../../srv/lib/contributors-publish.js ../../srv/lib/validation-rules-publish.js ../../srv/lib/island-manifest.json srv/lib/ && mkdir -p srv/lib/feature-flags && cp ../../srv/lib/feature-flags/db-flags.js ../../srv/lib/feature-flags/registry.js srv/lib/feature-flags/ && cp ../../srv/handlers/categories-after-hooks.js ../../srv/handlers/completion-path-items-altgroup.js srv/handlers/ && mkdir -p srv && cp ../../srv/content-moderation-service.js srv/ && cp ../../srv/jobs/consolidate-concepts-job.js ../../srv/jobs/extract-concepts-job.js ../../srv/jobs/job-lock.js ../../srv/jobs/secret-expiry-check.js ../../srv/jobs/homepage-link-health.js ../../srv/jobs/kg-ondemand-job.js ../../srv/jobs/community-blogs-fetch-job.js ../../srv/jobs/community-blogs-classify-job.js ../../srv/jobs/fetch-news-job.js srv/jobs/ && cp ../../srv/lib/prompts/explainer-verb.md ../../srv/lib/prompts/explainer-shelf.md ../../srv/lib/prompts/explainer-shelf-entry.md ../../srv/lib/prompts/community-blogs-classifier.md srv/lib/prompts/ && cp ../../srv/mcp/prompts/summarize_mission_for_beginner.md ../../srv/mcp/prompts/generate_lab_exercise.md ../../srv/mcp/prompts/explain_concept.md ../../srv/mcp/prompts/suggest_learning_path.md srv/mcp/prompts/" - bash -c "node -e \"const p=require('./package.json'); p.dependencies=Object.assign(p.dependencies||{},{cheerio:'^1.2.0','@sap-ai-sdk/foundation-models':'^2.10.0'}); require('fs').writeFileSync('./package.json', JSON.stringify(p,null,2));\"" properties: EXPOSE_CAP_UI: false diff --git a/srv/lib/validation-rules-publish.js b/srv/lib/validation-rules-publish.js new file mode 100644 index 000000000..ba7f934d1 --- /dev/null +++ b/srv/lib/validation-rules-publish.js @@ -0,0 +1,75 @@ +// srv/lib/validation-rules-publish.js +// Handler for POST /content/publish-validation-rules. +// +// Bearer auth is delegated to `contentAuthMiddleware` from +// srv/lib/content-store.js — same shape as /content/publish-contributors: +// 503 when CONTENT_API_KEY is unset, 401 on missing Bearer header, 403 on +// wrong key, with timing-safe comparison. +// +// Accepts `{ slug, rules: AllRuleRow[] }` and REPLACE-per-slug: +// DELETEs all TutorialValidationRules rows for that tutorial then INSERTs +// the new set atomically inside cds.tx(). Publishing slug A never touches +// slug B's rows. + +import cds from '@sap/cds' + +const NS = 'com.sap.developers.ims' + +/** + * Core, unit-testable: REPLACE all validation-rule rows for one slug. + * @param {object} db – connected CDS db service (cds.connect.to('db')) + * @param {string} slug – tutorial slug (case-insensitive) + * @param {Array} rules – array of rule objects from the sidecar JSON + */ +export async function replaceValidationRulesForSlug(db, slug, rules) { + const { Tutorials, TutorialValidationRules } = cds.entities(NS) + const lcSlug = String(slug || '').toLowerCase() + const tut = await db.run(SELECT.one.from(Tutorials).columns('ID').where({ slug: lcSlug })) + if (!tut) return { ok: false, reason: 'tutorial_not_found', slug: lcSlug } + + const entries = (rules || []).map((r) => ({ + tutorial_ID: tut.ID, + stepNumber: r.stepNumber, + questionId: String(r.questionId).slice(0, 100), + questionText: (r.questionText || '').slice(0, 2000), + ruleType: (r.ruleType || '').slice(0, 50), + questionType: (r.questionType || '').slice(0, 20), + choiceMode: r.choiceMode || null, + options: r.options || null, + correctAnswer: r.correctAnswer ?? null, + aiGrading: Boolean(r.aiGrading), + })) + + await cds.tx(async (tx) => { + await tx.run(DELETE.from(TutorialValidationRules).where({ tutorial_ID: tut.ID })) + if (entries.length) await tx.run(INSERT.into(TutorialValidationRules).entries(entries)) + }) + return { ok: true, slug: lcSlug, count: entries.length } +} + +/** + * Express handler mirroring the publish-contributors route shape. + * Mounted in server.js with contentAuthMiddleware + express.json(). + */ +export async function publishValidationRules(req, res) { + try { + const { slug, rules } = req.body || {} + if (!slug || !Array.isArray(rules)) { + return res.status(400).json({ error: 'bad_request', detail: 'expected { slug, rules[] }' }) + } + + // entity_not_in_model guard (QA namespace safety — mirrors contributors-publish.js). + let entities + try { entities = cds.entities(NS) } catch { entities = null } + if (!entities || !entities.TutorialValidationRules) { + return res.status(409).json({ error: 'entity_not_in_model' }) + } + + const db = await cds.connect.to('db') + const result = await replaceValidationRulesForSlug(db, slug, rules) + if (!result.ok) return res.status(404).json(result) + return res.json(result) + } catch (e) { + return res.status(500).json({ error: 'internal', detail: e?.message }) + } +} diff --git a/srv/server.js b/srv/server.js index 635f83679..65a36abce 100644 --- a/srv/server.js +++ b/srv/server.js @@ -66,6 +66,7 @@ import { defaultLoadStepText } from './lib/code-check-step-loader.js'; import { codeCheckSpecPublishHandler } from './lib/code-check-spec-publish.js'; import { publishValidateAnswerSpecs } from './lib/validate-answer-spec-publish.js'; import { publishContributors } from './lib/contributors-publish.js'; +import { publishValidationRules } from './lib/validation-rules-publish.js'; import { resolveSearchSettings } from './lib/runtime-config/search-settings.js'; import { resolveTenantSettings } from './lib/runtime-config/tenant-settings.js'; import { makeValidateAnswerHandler } from './lib/validate-answer-handler.js'; @@ -615,6 +616,14 @@ cds.on('bootstrap', (app) => { publishContributors ); + // REPLACE-per-slug handler for TutorialValidationRules (WS3 #task-13). + // Same auth guard and body parser shape as publish-contributors. + app.post('/content/publish-validation-rules', + express.json({ limit: '4mb' }), + contentAuthMiddleware, + publishValidationRules + ); + // Tutorial feedback bridge. Express handler (rather than letting CAP expose // the action over OData) so we can derive the originating client IP from // X-Forwarded-For and inject it into req.data via AsyncLocalStorage + a diff --git a/test/unit/validation-rules-publish.test.js b/test/unit/validation-rules-publish.test.js new file mode 100644 index 000000000..d3016ae54 --- /dev/null +++ b/test/unit/validation-rules-publish.test.js @@ -0,0 +1,28 @@ +// test/unit/validation-rules-publish.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import path from 'node:path' +import cds from '@sap/cds' +import { replaceValidationRulesForSlug } from '../../srv/lib/validation-rules-publish.js' + +describe('replaceValidationRulesForSlug', () => { + let db + beforeAll(async () => { + await cds.deploy(path.join(process.cwd(), 'db', 'schema.cds')).to('sqlite::memory:') + db = cds.db + }) + + it('replaces all-rule rows for a slug', async () => { + const { Tutorials, TutorialValidationRules } = cds.entities('com.sap.developers.ims') + const ID = cds.utils.uuid() + await db.run(INSERT.into(Tutorials).entries({ ID, slug: 'vr-demo', title: 'VR' })) + await replaceValidationRulesForSlug(db, 'VR-DEMO', [ + { stepNumber: 1, questionId: 'validate-1', questionText: 'Q', ruleType: 'single-choice', questionType: 'MCQ', choiceMode: 'single', options: '["A","B"]', correctAnswer: 'A', aiGrading: false }, + ]) + let rows = await db.run(SELECT.from(TutorialValidationRules).where({ tutorial_ID: ID })) + expect(rows).toHaveLength(1) + expect(rows[0].aiGrading).toBe(false) + await replaceValidationRulesForSlug(db, 'vr-demo', []) + rows = await db.run(SELECT.from(TutorialValidationRules).where({ tutorial_ID: ID })) + expect(rows).toHaveLength(0) + }) +}) From 20185926c39d5c769968b3aacbed3373b7c3edb3 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 14:11:39 -0400 Subject: [PATCH 20/71] feat(publish): non-fatal client step to publish validation-rules sidecars (#WS3) --- scripts/publish-content.ts | 15 ++++ scripts/publish/publish-validation-rules.ts | 70 +++++++++++++++++++ .../publish-validation-rules-client.test.js | 21 ++++++ 3 files changed, 106 insertions(+) create mode 100644 scripts/publish/publish-validation-rules.ts create mode 100644 test/unit/publish-validation-rules-client.test.js diff --git a/scripts/publish-content.ts b/scripts/publish-content.ts index 0fb886206..9e4e520f0 100644 --- a/scripts/publish-content.ts +++ b/scripts/publish-content.ts @@ -11,6 +11,7 @@ import { chunk, runConcurrent } from './lib/publish-batcher.js'; import { collectCodeCheckSpecs, publishCodeCheckSpecs } from './lib/publish-codecheck.js'; import { publishValidateAnswerSpecs } from './lib/publish-validate-answer.js'; import { publishContributors } from './publish/publish-contributors.js'; +import { publishValidationRules } from './publish/publish-validation-rules.js'; import { computeOrphans, enforceCap, formatStepSummary } from './lib/purge-orphans.js'; import { discoverPageFiles, discoverAuthorPages, discoverAdvocatePages } from '../srv/lib/page-key-map.js'; @@ -1366,6 +1367,20 @@ async function main() { } } + // --- validation-rules sidecar publish (non-fatal auxiliary step, issue #WS3) --- + // QA channel skips: srv-qa has no ValidationRules entity, POST would 404/500. + if (channel === 'qa') { + log('[publish-validation-rules] skipped (channel=qa)'); + } else { + try { + const cacheDir = join(process.cwd(), '.tutorial-cache'); + const r = await publishValidationRules({ cacheDir, baseUrl: opts.baseUrl, apiKey: opts.apiKey }); + log(`[publish-validation-rules] published ${r.published}/${r.total}`); + } catch (err) { + console.error('[publish-content] validation-rules publish failed (non-fatal):', formatErrorChain(err)); + } + } + // --- auto-verify --- log('Verifying server state matches local...'); let postRemote: Record; diff --git a/scripts/publish/publish-validation-rules.ts b/scripts/publish/publish-validation-rules.ts new file mode 100644 index 000000000..5c1efcc39 --- /dev/null +++ b/scripts/publish/publish-validation-rules.ts @@ -0,0 +1,70 @@ +// scripts/publish/publish-validation-rules.ts +// Non-fatal auxiliary publish step for issue #WS3. +// Walks `cacheDir` for `*.validation-rules.json` sidecar files emitted by +// scripts/fetch-tutorials.ts and POSTs each one to +// /content/publish-validation-rules (Task 13 REPLACE handler). +// +// Auth: CONTENT_API_KEY via contentAuthMiddleware (Authorization: Bearer). +// Failures are NON-FATAL — captured and returned to the caller. + +import { readdirSync, readFileSync } from 'node:fs' +import { join } from 'node:path' + +const SUFFIX = '.validation-rules.json' + +/** + * Walk cacheDir for *.validation-rules.json sidecar files, + * POST each one to /content/publish-validation-rules. + * + * @param opts.cacheDir Tutorial cache dir (e.g. .tutorial-cache) + * @param opts.baseUrl CAP base URL + * @param opts.apiKey CONTENT_API_KEY value + * @returns { published, total } + */ +export async function publishValidationRules(opts: { + cacheDir: string + baseUrl: string + apiKey: string +}): Promise<{ published: number; total: number }> { + const { cacheDir, baseUrl, apiKey } = opts + let files: string[] + try { + files = readdirSync(cacheDir).filter((f) => f.endsWith(SUFFIX)) + } catch { + return { published: 0, total: 0 } + } + + let published = 0 + for (const f of files) { + const filePath = join(cacheDir, f) + let raw: string + try { + raw = readFileSync(filePath, 'utf8') + } catch { + continue + } + + let res: Response + try { + res = await fetch(`${baseUrl}/content/publish-validation-rules`, { + method: 'POST', + headers: { + 'authorization': `Bearer ${apiKey}`, + 'content-type': 'application/json', + }, + body: raw, + }) + } catch (err) { + console.warn(`[publish-validation-rules] network error for ${f}:`, (err as Error).message) + continue + } + + if (res.ok) { + published += 1 + } else { + console.warn(`[publish-validation-rules] ${f} -> ${res.status}`) + } + } + + return { published, total: files.length } +} diff --git a/test/unit/publish-validation-rules-client.test.js b/test/unit/publish-validation-rules-client.test.js new file mode 100644 index 000000000..2e110e33c --- /dev/null +++ b/test/unit/publish-validation-rules-client.test.js @@ -0,0 +1,21 @@ +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest' +import { mkdtempSync, writeFileSync } from 'node:fs' +import { join } from 'node:path'; import { tmpdir } from 'node:os' +import { publishValidationRules } from '../../scripts/publish/publish-validation-rules' + +describe('publishValidationRules client', () => { + let dir + beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), 'vr-')) + writeFileSync(join(dir, 'demo.validation-rules.json'), + JSON.stringify({ slug: 'demo', rules: [{ stepNumber: 1, questionId: 'validate-1' }] })) + global.fetch = vi.fn().mockResolvedValue({ ok: true, json: async () => ({ ok: true }) }) + }) + afterEach(() => vi.restoreAllMocks()) + it('POSTs each sidecar', async () => { + const res = await publishValidationRules({ cacheDir: dir, baseUrl: 'http://x', apiKey: 'k' }) + expect(global.fetch).toHaveBeenCalledTimes(1) + expect(global.fetch.mock.calls[0][0]).toBe('http://x/content/publish-validation-rules') + expect(res.published).toBe(1) + }) +}) From 72b0a989f9a6c95bf15d6a1d2cfdf5b29d3bc627 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 14:15:26 -0400 Subject: [PATCH 21/71] feat(admin-ui): All Validation Rules facet + relabel AI facet (#WS3) --- app/admin-annotations.cds | 17 ++++++++++++++++- srv/admin-service.cds | 2 ++ test/unit/annotations-validation-rules.test.js | 17 +++++++++++++++++ 3 files changed, 35 insertions(+), 1 deletion(-) create mode 100644 test/unit/annotations-validation-rules.test.js diff --git a/app/admin-annotations.cds b/app/admin-annotations.cds index 4c3aef516..8c97cea5f 100644 --- a/app/admin-annotations.cds +++ b/app/admin-annotations.cds @@ -883,6 +883,19 @@ annotate AdminService.ValidateAnswerSpecs with @UI: { } }; +// TutorialValidationRules — all validation rules for a tutorial (step, question, +// type, rule, AI-grading flag, correct answer). Joined via `validationRules` association. +annotate AdminService.TutorialValidationRules with @( + UI.LineItem: [ + { Value: stepNumber, Label: 'Step' }, + { Value: questionText, Label: 'Question' }, + { Value: questionType, Label: 'Type' }, + { Value: ruleType, Label: 'Rule' }, + { Value: aiGrading, Label: 'AI-Graded' }, + { Value: correctAnswer, Label: 'Correct Answer' } + ] +); + // CodeCheckSpecs — per-step code-check specs (goal + reference solution). // Joined via `codeCheckSpecs` association. annotate AdminService.CodeCheckSpecs with { @@ -956,8 +969,10 @@ annotate AdminService.Tutorials with @UI: { { $Type: 'UI.ReferenceFacet', Label: 'Contributors', ID: 'ContributorsFacet', Target: 'contributors/@UI.LineItem' }, { $Type: 'UI.ReferenceFacet', Label: 'Completion Stats', ID: 'CompletionStatsFacet', Target: 'completionStats/@UI.FieldGroup#Stats' }, - { $Type: 'UI.ReferenceFacet', Label: 'Validation Questions', ID: 'ValidationSpecsFacet', + { $Type: 'UI.ReferenceFacet', Label: 'AI-Graded Validation', ID: 'ValidationSpecsFacet', Target: 'validationSpecs/@UI.LineItem' }, + { $Type: 'UI.ReferenceFacet', Label: 'All Validation Rules', ID: 'AllValidationRulesFacet', + Target: 'validationRules/@UI.LineItem' }, { $Type: 'UI.ReferenceFacet', Label: 'Code-Check Specs', ID: 'CodeCheckSpecsFacet', Target: 'codeCheckSpecs/@UI.LineItem' }, { $Type: 'UI.ReferenceFacet', Label: 'AI-Author Requests', ID: 'AiRequestsFacet', diff --git a/srv/admin-service.cds b/srv/admin-service.cds index 3a872801d..a12e3995b 100644 --- a/srv/admin-service.cds +++ b/srv/admin-service.cds @@ -64,6 +64,7 @@ service AdminService { // Specs use the existing tutorial Association FK; submissions and stats // join by slug because they predate the FK pattern. validationSpecs : Association to many ValidateAnswerSpecs on validationSpecs.tutorial = $self, + validationRules : Association to many TutorialValidationRules on validationRules.tutorial = $self, validationSubmissions : Association to many ValidateAnswerSubmissions on validationSubmissions.tutorialSlug = slug, codeCheckSpecs : Association to many CodeCheckSpecs on codeCheckSpecs.tutorial = $self, codeCheckSubmissions : Association to many CodeCheckSubmissions on codeCheckSubmissions.tutorialSlug = slug, @@ -618,6 +619,7 @@ service AdminService { @readonly entity CodeCheckSubmissions as projection on ims.CodeCheckSubmissions; @readonly entity AuthorAiRequests as projection on ims.AuthorAiRequests; @readonly entity TutorialCompletionStats as projection on ims.TutorialCompletionStats; + @readonly entity TutorialValidationRules as projection on ims.TutorialValidationRules; // Issue #622 — read-only recipient list for the "Last Chance Emails" // admin section. Powers the dropdown for sendLastChanceEmail and the diff --git a/test/unit/annotations-validation-rules.test.js b/test/unit/annotations-validation-rules.test.js new file mode 100644 index 000000000..ed20f1b8f --- /dev/null +++ b/test/unit/annotations-validation-rules.test.js @@ -0,0 +1,17 @@ +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +describe('validation rules exposure + facet', () => { + let m + beforeAll(async () => { m = await cds.load('*') }) + it('AdminService exposes TutorialValidationRules read-only', () => { + expect(m.definitions['AdminService.TutorialValidationRules']).toBeTruthy() + }) + it('Tutorials has validationRules association', () => { + expect(m.definitions['AdminService.Tutorials'].elements.validationRules).toBeTruthy() + }) + it('OP facets include an All Validation Rules facet', () => { + const facets = m.definitions['AdminService.Tutorials']['@UI.Facets'] + const ids = facets.map((f) => f.ID) + expect(ids).toContain('AllValidationRulesFacet') + }) +}) From 6969b246070eb3d8c96e3c85bd77f53a80c47ba3 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 14:18:42 -0400 Subject: [PATCH 22/71] test(publish): hybrid guard for all-rules population (#WS3) --- test/hybrid/publish-validation-rules.test.js | 31 ++++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 test/hybrid/publish-validation-rules.test.js diff --git a/test/hybrid/publish-validation-rules.test.js b/test/hybrid/publish-validation-rules.test.js new file mode 100644 index 000000000..22e5e32ed --- /dev/null +++ b/test/hybrid/publish-validation-rules.test.js @@ -0,0 +1,31 @@ +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +import { replaceValidationRulesForSlug } from '../../srv/lib/validation-rules-publish.js' + +cds.test('serve', '--project', '.', '--profile', 'hybrid') + +describe('validation rules publish (hybrid)', () => { + let db + beforeAll(async () => { db = await cds.connect.to('db') }) + it('lands mixed AI + client rules for an existing slug', async () => { + const { Tutorials, TutorialValidationRules, ValidateAnswerSpecs } = cds.entities('com.sap.developers.ims') + const t = await db.run(SELECT.one.from(Tutorials).columns('ID', 'slug')) + expect(t).toBeTruthy() + try { + // Assert ValidateAnswerSpecs isolation: query count before + const beforeCount = await db.run(SELECT.from(ValidateAnswerSpecs).where({ tutorial_ID: t.ID, stepNumber: 99 }).columns(c => c`count(*) as cnt`)).then(r => r[0]?.cnt ?? 0) + await replaceValidationRulesForSlug(db, t.slug, [ + { stepNumber: 99, questionId: 'vr-test-a', questionText: 'client', ruleType: 'single-choice', questionType: 'MCQ', choiceMode: 'single', options: '["A"]', correctAnswer: 'A', aiGrading: false }, + { stepNumber: 99, questionId: 'vr-test-b', questionText: 'ai', ruleType: 'regex', questionType: 'TEXT', choiceMode: null, options: null, correctAnswer: null, aiGrading: true }, + ]) + const rows = await db.run(SELECT.from(TutorialValidationRules).where({ tutorial_ID: t.ID, stepNumber: 99 })) + expect(rows.length).toBe(2) + // Assert ValidateAnswerSpecs isolation: count after should match before + const afterCount = await db.run(SELECT.from(ValidateAnswerSpecs).where({ tutorial_ID: t.ID, stepNumber: 99 }).columns(c => c`count(*) as cnt`)).then(r => r[0]?.cnt ?? 0) + expect(afterCount).toBe(beforeCount) + } finally { + // cleanup + await db.run(DELETE.from(TutorialValidationRules).where({ tutorial_ID: t.ID, stepNumber: 99 })) + } + }) +}) From 47f10e3fec377cd621922c2747ab105986dd5323 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 15:01:31 -0400 Subject: [PATCH 23/71] fix(db): align last-dev/csn.json with canonical cds build ordering The migration-clobber workaround (temporarily removing the 2nd hana build task) emitted TutorialValidationRules at a different position in db/last-dev/csn.json than the canonical two-task `cds build --production` that CI's cds-build-staging check runs. Content is byte-identical (pure reordering); regenerated via canonical build. Migration tables unchanged. --- db/last-dev/csn.json | 120 +++++++++++++++++++++---------------------- 1 file changed, 60 insertions(+), 60 deletions(-) diff --git a/db/last-dev/csn.json b/db/last-dev/csn.json index 474df9979..ea9751704 100644 --- a/db/last-dev/csn.json +++ b/db/last-dev/csn.json @@ -1472,6 +1472,66 @@ }, "@cds.persistence.name": "COM_SAP_DEVELOPERS_IMS_DEVELOPERENVIRONMENTLINKS" }, + "com.sap.developers.ims.TutorialValidationRules": { + "kind": "entity", + "@cds.persistence.journal": true, + "elements": { + "tutorial_ID": { + "type": "cds.String", + "length": 36, + "@odata.foreignKey4": "tutorial", + "key": true, + "@cds.persistence.name": "TUTORIAL_ID" + }, + "stepNumber": { + "key": true, + "type": "cds.Integer", + "@cds.persistence.name": "STEPNUMBER" + }, + "questionId": { + "key": true, + "type": "cds.String", + "length": 100, + "@cds.persistence.name": "QUESTIONID" + }, + "questionText": { + "type": "cds.String", + "length": 2000, + "@cds.persistence.name": "QUESTIONTEXT" + }, + "ruleType": { + "type": "cds.String", + "length": 50, + "@cds.persistence.name": "RULETYPE" + }, + "questionType": { + "type": "cds.String", + "length": 20, + "@cds.persistence.name": "QUESTIONTYPE" + }, + "choiceMode": { + "type": "cds.String", + "length": 20, + "@cds.persistence.name": "CHOICEMODE" + }, + "options": { + "type": "cds.LargeString", + "@cds.persistence.name": "OPTIONS" + }, + "correctAnswer": { + "type": "cds.LargeString", + "@cds.persistence.name": "CORRECTANSWER" + }, + "aiGrading": { + "type": "cds.Boolean", + "default": { + "val": false + }, + "@cds.persistence.name": "AIGRADING" + } + }, + "@cds.persistence.name": "COM_SAP_DEVELOPERS_IMS_TUTORIALVALIDATIONRULES" + }, "com.sap.developers.ims.Puzzles": { "kind": "entity", "@assert.unique.slug": [ @@ -3630,66 +3690,6 @@ } }, "@cds.persistence.name": "COM_SAP_DEVELOPERS_IMS_CONTENTMANIFEST" - }, - "com.sap.developers.ims.TutorialValidationRules": { - "kind": "entity", - "@cds.persistence.journal": true, - "elements": { - "tutorial_ID": { - "type": "cds.String", - "length": 36, - "@odata.foreignKey4": "tutorial", - "key": true, - "@cds.persistence.name": "TUTORIAL_ID" - }, - "stepNumber": { - "key": true, - "type": "cds.Integer", - "@cds.persistence.name": "STEPNUMBER" - }, - "questionId": { - "key": true, - "type": "cds.String", - "length": 100, - "@cds.persistence.name": "QUESTIONID" - }, - "questionText": { - "type": "cds.String", - "length": 2000, - "@cds.persistence.name": "QUESTIONTEXT" - }, - "ruleType": { - "type": "cds.String", - "length": 50, - "@cds.persistence.name": "RULETYPE" - }, - "questionType": { - "type": "cds.String", - "length": 20, - "@cds.persistence.name": "QUESTIONTYPE" - }, - "choiceMode": { - "type": "cds.String", - "length": 20, - "@cds.persistence.name": "CHOICEMODE" - }, - "options": { - "type": "cds.LargeString", - "@cds.persistence.name": "OPTIONS" - }, - "correctAnswer": { - "type": "cds.LargeString", - "@cds.persistence.name": "CORRECTANSWER" - }, - "aiGrading": { - "type": "cds.Boolean", - "default": { - "val": false - }, - "@cds.persistence.name": "AIGRADING" - } - }, - "@cds.persistence.name": "COM_SAP_DEVELOPERS_IMS_TUTORIALVALIDATIONRULES" } }, "meta": { From 645ab88759d741ef2ef951a4b22350a439ea42ce Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 15:14:43 -0400 Subject: [PATCH 24/71] feat(db): add byteSize to TutorialImages/TutorialAssets (#WS5) --- db/tutorial-assets.cds | 1 + db/tutorial-images.cds | 1 + test/unit/schema-media.test.js | 9 +++++++++ 3 files changed, 11 insertions(+) create mode 100644 test/unit/schema-media.test.js diff --git a/db/tutorial-assets.cds b/db/tutorial-assets.cds index d030f5a54..a07a94de7 100644 --- a/db/tutorial-assets.cds +++ b/db/tutorial-assets.cds @@ -11,6 +11,7 @@ entity TutorialAssets { channel : String(8); // 'prod' | 'qa' contentHash : String(64); // sha-256 of stored bytes mimeType : String(128); + byteSize : Integer64; // original byte length captured at ingest filename : String(255); // for Content-Disposition content : Composition of many Attachments; } diff --git a/db/tutorial-images.cds b/db/tutorial-images.cds index 91a2f799d..903da7199 100644 --- a/db/tutorial-images.cds +++ b/db/tutorial-images.cds @@ -12,5 +12,6 @@ entity TutorialImages { channel : String(8); // 'prod' | 'qa' contentHash : String(64); // sha-256 of the stored original mimeType : String(128); + byteSize : Integer64; // original byte length captured at ingest content : Composition of many Attachments; } diff --git a/test/unit/schema-media.test.js b/test/unit/schema-media.test.js new file mode 100644 index 000000000..47c967e7e --- /dev/null +++ b/test/unit/schema-media.test.js @@ -0,0 +1,9 @@ +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +describe('media byteSize', () => { + let m; beforeAll(async () => { m = await cds.load('*') }) + it('images + assets have byteSize', () => { + expect(m.definitions['com.sap.developers.ims.TutorialImages'].elements.byteSize).toBeTruthy() + expect(m.definitions['com.sap.developers.ims.TutorialAssets'].elements.byteSize).toBeTruthy() + }) +}) From 90a0d1fa14852a2ea231959bb45229e6d940f894 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 15:25:01 -0400 Subject: [PATCH 25/71] feat(media): persist byteSize at image/asset ingest (#WS5) --- srv/lib/attachment-ingest-handler.js | 2 +- srv/lib/attachment-store.cjs | 5 +-- srv/lib/image-ingest-handler.js | 2 +- srv/lib/image-store.cjs | 5 +-- test/unit/ingest-bytesize.test.js | 46 ++++++++++++++++++++++++++++ 5 files changed, 54 insertions(+), 6 deletions(-) create mode 100644 test/unit/ingest-bytesize.test.js diff --git a/srv/lib/attachment-ingest-handler.js b/srv/lib/attachment-ingest-handler.js index 137acaf36..5fab5e81b 100644 --- a/srv/lib/attachment-ingest-handler.js +++ b/srv/lib/attachment-ingest-handler.js @@ -84,7 +84,7 @@ export async function attachmentIngestHandler(req, res) { return res.status(200).json({ action: 'unchanged', contentHash }) } } - await attachmentStore.put(u, { buffer, mimeType, contentHash, slug, channel, filename }) + await attachmentStore.put(u, { buffer, mimeType, contentHash, slug, channel, filename, byteSize: buffer.length }) return res.status(200).json({ action: 'stored', contentHash }) } catch (err) { LOG.error('[attachment-ingest] store put failed for', u, '-', err.message) diff --git a/srv/lib/attachment-store.cjs b/srv/lib/attachment-store.cjs index 89e574b1e..0ed1dd025 100644 --- a/srv/lib/attachment-store.cjs +++ b/srv/lib/attachment-store.cjs @@ -35,14 +35,15 @@ async function head(sourceUrl) { }) } -async function put(sourceUrl, { buffer, mimeType, contentHash, slug, channel, filename }) { +async function put(sourceUrl, { buffer, mimeType, contentHash, slug, channel, filename, byteSize }) { return withCtx(async () => { const { TutorialAssets } = cds.entities('com.sap.developers.ims') // delete-then-insert avoids NonUpdatableProperties:[content] 409 on overwrite await remove(sourceUrl) const parentID = cds.utils.uuid() const name = filename || sourceUrl.split('/').pop() - await INSERT.into(TutorialAssets).entries({ ID: parentID, sourceUrl, slug, channel, contentHash, mimeType, filename: name }) + const bs = byteSize != null ? byteSize : (Buffer.isBuffer(buffer) ? buffer.length : null) + await INSERT.into(TutorialAssets).entries({ ID: parentID, sourceUrl, slug, channel, contentHash, mimeType, filename: name, byteSize: bs }) const AttachmentsSrv = await cds.connect.to('attachments') await AttachmentsSrv.put(linkedContent(), { ID: cds.utils.uuid(), diff --git a/srv/lib/image-ingest-handler.js b/srv/lib/image-ingest-handler.js index 04683e29a..6f7bff8db 100644 --- a/srv/lib/image-ingest-handler.js +++ b/srv/lib/image-ingest-handler.js @@ -74,7 +74,7 @@ export async function imageIngestHandler(req, res) { return res.status(200).json({ action: 'unchanged', contentHash }) } } - await imageStore.put(u, { buffer, mimeType, contentHash, slug, channel }) + await imageStore.put(u, { buffer, mimeType, contentHash, slug, channel, byteSize: buffer.length }) return res.status(200).json({ action: 'stored', contentHash }) } catch (err) { LOG.error('[image-ingest] store put failed for', u, '-', err.message) diff --git a/srv/lib/image-store.cjs b/srv/lib/image-store.cjs index ce18ca779..34a092bfc 100644 --- a/srv/lib/image-store.cjs +++ b/srv/lib/image-store.cjs @@ -37,13 +37,14 @@ async function head(sourceUrl) { }) } -async function put(sourceUrl, { buffer, mimeType, contentHash, slug, channel }) { +async function put(sourceUrl, { buffer, mimeType, contentHash, slug, channel, byteSize }) { return withCtx(async () => { const { TutorialImages } = cds.entities('com.sap.developers.ims') // R5: delete-then-insert avoids the NonUpdatableProperties:[content] 409 on overwrite await remove(sourceUrl) const parentID = cds.utils.uuid() - await INSERT.into(TutorialImages).entries({ ID: parentID, sourceUrl, slug, channel, contentHash, mimeType }) + const bs = byteSize != null ? byteSize : (Buffer.isBuffer(buffer) ? buffer.length : null) + await INSERT.into(TutorialImages).entries({ ID: parentID, sourceUrl, slug, channel, contentHash, mimeType, byteSize: bs }) const AttachmentsSrv = await cds.connect.to('attachments') await AttachmentsSrv.put(linkedContent(), { ID: cds.utils.uuid(), diff --git a/test/unit/ingest-bytesize.test.js b/test/unit/ingest-bytesize.test.js new file mode 100644 index 000000000..ba19bc0f7 --- /dev/null +++ b/test/unit/ingest-bytesize.test.js @@ -0,0 +1,46 @@ +// test/unit/ingest-bytesize.test.js +// +// Verifies that byteSize = buffer.length is persisted on the parent row when +// image-store.cjs / attachment-store.cjs put() is called with a byteSize option. +// +// Run: npx vitest run --project unit test/unit/ingest-bytesize.test.js +// +// Boot full CAP server in-memory (SQLite). cds.test registers its own +// beforeAll/afterAll hooks at this scope so all it() blocks run after boot. + +import { describe, it, expect } from 'vitest' +import cds from '@sap/cds' +import { createRequire } from 'node:module' + +const require = createRequire(import.meta.url) + +cds.test('serve', '--project', '.', '--in-memory') + +const imageStore = require('../../srv/lib/image-store.cjs') +const attachmentStore = require('../../srv/lib/attachment-store.cjs') + +describe('image store persists byteSize', () => { + it('stores buffer.length as byteSize on the TutorialImages row', async () => { + const buf = Buffer.from('hello world') + await imageStore.put('https://raw.example/img.png', { + buffer: buf, mimeType: 'image/png', contentHash: 'abc123', + slug: 'demo', channel: 'prod', byteSize: buf.length, + }) + const { TutorialImages } = cds.entities('com.sap.developers.ims') + const row = await SELECT.one.from(TutorialImages).where({ sourceUrl: 'https://raw.example/img.png' }) + expect(Number(row.byteSize)).toBe(11) + }) +}) + +describe('attachment store persists byteSize', () => { + it('stores buffer.length as byteSize on the TutorialAssets row', async () => { + const buf = Buffer.from('hello world') + await attachmentStore.put('https://raw.example/file.txt', { + buffer: buf, mimeType: 'text/plain', contentHash: 'def456', + slug: 'demo', channel: 'prod', filename: 'file.txt', byteSize: buf.length, + }) + const { TutorialAssets } = cds.entities('com.sap.developers.ims') + const row = await SELECT.one.from(TutorialAssets).where({ sourceUrl: 'https://raw.example/file.txt' }) + expect(Number(row.byteSize)).toBe(11) + }) +}) From 9d3310b4e4c1b9bb4d796b7cd18f4fdd15ef4627 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 15:28:45 -0400 Subject: [PATCH 26/71] feat(admin): read-only Media projections + Tutorials associations (#WS5) --- srv/admin-service.cds | 12 ++++++++++++ test/unit/media-exposure.test.js | 14 ++++++++++++++ 2 files changed, 26 insertions(+) create mode 100644 test/unit/media-exposure.test.js diff --git a/srv/admin-service.cds b/srv/admin-service.cds index a12e3995b..c817804d2 100644 --- a/srv/admin-service.cds +++ b/srv/admin-service.cds @@ -9,6 +9,8 @@ using from '../db/homepage-featured'; using from '../db/views'; using from '../db/devtoberfest-analytics'; using from '../db/mcp-pats'; +using from '../db/tutorial-images'; +using from '../db/tutorial-assets'; using from '../app/admin-annotations'; using { external.devtoberfest as external_dtf } from '../db/external/devtoberfest'; @@ -109,6 +111,10 @@ service AdminService { virtual openHighCount : Integer, // populated in after('READ','Tutorials') — Task 8 virtual freshnessStatus : String, virtual freshnessCriticality : Integer, + // Media facet (task-3): object-store images + assets for this tutorial. + // Filtered to channel='prod' so the admin OP shows prod-published media. + images : Association to many TutorialImages on images.slug = $self.slug and images.channel = 'prod', + assets : Association to many TutorialAssets on assets.slug = $self.slug and assets.channel = 'prod', }; // Filtered picklist for redirectTo value help — only ACTIVE tutorials can be redirect targets @readonly @@ -621,6 +627,12 @@ service AdminService { @readonly entity TutorialCompletionStats as projection on ims.TutorialCompletionStats; @readonly entity TutorialValidationRules as projection on ims.TutorialValidationRules; + // Media facet (task-3): read-only projections of the object-store image and + // asset metadata entities. The content Attachments composition auto-exposes + // when the parent is reachable from an exposed entity (@cap-js/attachments). + @readonly @cds.redirection.target: false entity TutorialImages as projection on ims.TutorialImages; + @readonly @cds.redirection.target: false entity TutorialAssets as projection on ims.TutorialAssets; + // Issue #622 — read-only recipient list for the "Last Chance Emails" // admin section. Powers the dropdown for sendLastChanceEmail and the // preview list for sendLastChanceEmailsAllDormant. One row per diff --git a/test/unit/media-exposure.test.js b/test/unit/media-exposure.test.js new file mode 100644 index 000000000..e5de747cc --- /dev/null +++ b/test/unit/media-exposure.test.js @@ -0,0 +1,14 @@ +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +describe('media exposure', () => { + let m; beforeAll(async () => { m = await cds.load('*') }) + it('exposes images + assets read-only', () => { + expect(m.definitions['AdminService.TutorialImages']).toBeTruthy() + expect(m.definitions['AdminService.TutorialAssets']).toBeTruthy() + }) + it('Tutorials has images + assets associations', () => { + const t = m.definitions['AdminService.Tutorials'].elements + expect(t.images).toBeTruthy() + expect(t.assets).toBeTruthy() + }) +}) From 99f52355624eb69c93b9a610892cb74b8fb6a756 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 15:32:37 -0400 Subject: [PATCH 27/71] feat(admin-ui): Media facets (images/assets) with source link + byte size (#WS5) --- app/admin-annotations.cds | 22 ++++++++++++++++++++++ test/unit/annotations-media.test.js | 15 +++++++++++++++ 2 files changed, 37 insertions(+) create mode 100644 test/unit/annotations-media.test.js diff --git a/app/admin-annotations.cds b/app/admin-annotations.cds index 8c97cea5f..4cfbb6e24 100644 --- a/app/admin-annotations.cds +++ b/app/admin-annotations.cds @@ -979,6 +979,8 @@ annotate AdminService.Tutorials with @UI: { Target: 'aiRequests/@UI.LineItem' }, { $Type: 'UI.ReferenceFacet', ID: 'FreshnessFacet', Label: 'Freshness', Target: 'freshnessFindings/@UI.LineItem' }, + { $Type: 'UI.ReferenceFacet', Label: 'Images', ID: 'MediaImagesFacet', Target: 'images/@UI.LineItem' }, + { $Type: 'UI.ReferenceFacet', Label: 'Assets', ID: 'MediaAssetsFacet', Target: 'assets/@UI.LineItem' }, { $Type: 'UI.CollectionFacet', ID: 'Feedback', Label: 'Feedback', Facets: [ { $Type: 'UI.ReferenceFacet', ID: 'FeedbackSummary', Target: 'feedbackSummary/@UI.FieldGroup#FeedbackSummary', @@ -4395,3 +4397,23 @@ annotate AdminService.FreshnessFinding with { suggestedFix @UI.MultiLineText; evidence @UI.MultiLineText; }; + +// --- Media facets: TutorialImages + TutorialAssets (Task 4) --- +annotate AdminService.TutorialImages with @( + UI.LineItem: [ + { $Type: 'UI.DataFieldWithUrl', Value: sourceUrl, Url: sourceUrl, Label: 'Source (GitHub)' }, + { Value: mimeType, Label: 'Type' }, + { Value: byteSize, Label: 'Bytes' }, + { Value: contentHash, Label: 'Hash' }, + { Value: channel, Label: 'Channel' } + ] +); +annotate AdminService.TutorialAssets with @( + UI.LineItem: [ + { Value: filename, Label: 'File' }, + { $Type: 'UI.DataFieldWithUrl', Value: sourceUrl, Url: sourceUrl, Label: 'Source (GitHub)' }, + { Value: mimeType, Label: 'Type' }, + { Value: byteSize, Label: 'Bytes' }, + { Value: contentHash, Label: 'Hash' } + ] +); diff --git a/test/unit/annotations-media.test.js b/test/unit/annotations-media.test.js new file mode 100644 index 000000000..d5e9a426a --- /dev/null +++ b/test/unit/annotations-media.test.js @@ -0,0 +1,15 @@ +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +describe('Media facet', () => { + let m; beforeAll(async () => { m = await cds.load('*') }) + it('OP facets include Media', () => { + const ids = m.definitions['AdminService.Tutorials']['@UI.Facets'].map((f) => f.ID) + expect(ids).toContain('MediaImagesFacet') + expect(ids).toContain('MediaAssetsFacet') + }) + it('image LineItem shows sourceUrl + byteSize + mimeType', () => { + const li = m.definitions['AdminService.TutorialImages']['@UI.LineItem'] + const vals = li.map((x) => x.Value?.['='] || x.Value) + for (const c of ['sourceUrl','byteSize','mimeType','contentHash']) expect(vals).toContain(c) + }) +}) From 3ec94a292279bebc20c7bee5539f6e79737a1fbb Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 15:36:19 -0400 Subject: [PATCH 28/71] feat(admin-ui): Freshness Reports facet (runAt/model/cost/status) (#WS5) --- app/admin-annotations.cds | 15 +++++++++++++++ srv/admin-service.cds | 1 + test/unit/annotations-freshness-facet.test.js | 13 +++++++++++++ 3 files changed, 29 insertions(+) create mode 100644 test/unit/annotations-freshness-facet.test.js diff --git a/app/admin-annotations.cds b/app/admin-annotations.cds index 4cfbb6e24..7e71172b9 100644 --- a/app/admin-annotations.cds +++ b/app/admin-annotations.cds @@ -977,6 +977,7 @@ annotate AdminService.Tutorials with @UI: { Target: 'codeCheckSpecs/@UI.LineItem' }, { $Type: 'UI.ReferenceFacet', Label: 'AI-Author Requests', ID: 'AiRequestsFacet', Target: 'aiRequests/@UI.LineItem' }, + { $Type: 'UI.ReferenceFacet', Label: 'Freshness Reports', ID: 'FreshnessReportsFacet', Target: 'freshnessReports/@UI.PresentationVariant' }, { $Type: 'UI.ReferenceFacet', ID: 'FreshnessFacet', Label: 'Freshness', Target: 'freshnessFindings/@UI.LineItem' }, { $Type: 'UI.ReferenceFacet', Label: 'Images', ID: 'MediaImagesFacet', Target: 'images/@UI.LineItem' }, @@ -4367,6 +4368,20 @@ annotate AdminService.TopicClustersAdmin with @( // --- Tutorial Freshness Detector (spec 2026-08-22-tutorial-freshness-detector) --- // Surfaces per-finding analysis rows on the Tutorials Object Page and wires the +// Freshness Reports — report-level header (spec 2026-08-31 task-5). +// PresentationVariant sorts newest-first so the latest run appears at the top. +annotate AdminService.FreshnessReport with @( + UI.LineItem: [ + { Value: runAt, Label: 'Run At' }, + { Value: status, Label: 'Status' }, + { Value: model, Label: 'Model' }, + { Value: cost, Label: 'Cost' }, + { Value: openHighCount, Label: 'Open High' }, + { Value: error, Label: 'Error' } + ], + UI.PresentationVariant: { SortOrder: [{ Property: runAt, Descending: true }], Visualizations: ['@UI.LineItem'] } +); + // Set Disposition action. Criticality paths delegate to the virtual // `confidenceCriticality` field (computed by after('READ','FreshnessFinding') // in admin-service.js). diff --git a/srv/admin-service.cds b/srv/admin-service.cds index c817804d2..f93fcddd9 100644 --- a/srv/admin-service.cds +++ b/srv/admin-service.cds @@ -108,6 +108,7 @@ service AdminService { virtual mainPreviewLabel : String, // Freshness detector (spec 2026-08-22) freshnessFindings : Association to many FreshnessFinding on freshnessFindings.tutorial.ID = ID, + freshnessReports : Association to many FreshnessReport on freshnessReports.tutorial = $self, virtual openHighCount : Integer, // populated in after('READ','Tutorials') — Task 8 virtual freshnessStatus : String, virtual freshnessCriticality : Integer, diff --git a/test/unit/annotations-freshness-facet.test.js b/test/unit/annotations-freshness-facet.test.js new file mode 100644 index 000000000..aff71dbed --- /dev/null +++ b/test/unit/annotations-freshness-facet.test.js @@ -0,0 +1,13 @@ +// test/unit/annotations-freshness-facet.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +describe('Freshness reports facet', () => { + let m; beforeAll(async () => { m = await cds.load('*') }) + it('Tutorials has freshnessReports association', () => { + expect(m.definitions['AdminService.Tutorials'].elements.freshnessReports).toBeTruthy() + }) + it('OP facets include FreshnessReportsFacet', () => { + const ids = m.definitions['AdminService.Tutorials']['@UI.Facets'].map((f) => f.ID) + expect(ids).toContain('FreshnessReportsFacet') + }) +}) From b4d88978f7ce7f1128733dd2380c3919a6a71fda Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 15:39:38 -0400 Subject: [PATCH 29/71] test(media): hybrid guard for media exposure (#WS5) --- test/hybrid/media-exposure.test.js | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 test/hybrid/media-exposure.test.js diff --git a/test/hybrid/media-exposure.test.js b/test/hybrid/media-exposure.test.js new file mode 100644 index 000000000..30cf7ed6d --- /dev/null +++ b/test/hybrid/media-exposure.test.js @@ -0,0 +1,16 @@ +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' + +cds.test('serve', '--project', '.', '--profile', 'hybrid') + +describe('media exposure (hybrid)', () => { + let admin + beforeAll(async () => { admin = await cds.connect.to('AdminService') }) + it('reads images for a tutorial without LOB errors', async () => { + const t = await admin.run(SELECT.one.from('AdminService.Tutorials').columns('ID', 'slug')) + expect(t).toBeTruthy() + // metadata-only read (no BLOB mix — never select 'content') + const imgs = await admin.run(SELECT.from('AdminService.TutorialImages').columns('ID', 'sourceUrl', 'mimeType', 'byteSize').where({ slug: t.slug })) + expect(Array.isArray(imgs)).toBe(true) + }) +}) From b2807cf20adde7af64354abb9ecfe8bfcad52987 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 15:55:33 -0400 Subject: [PATCH 30/71] feat(admin): read-only KG projections + Tutorials associations (#WS4) - Expose TutorialConceptLinks, TutorialRank, CoCompletions as @readonly @cds.redirection.target:false projections on AdminService - Add rank (to-one on slug) + coCompletions (to-many on sourceSlug) associations to AdminService.Tutorials projection; conceptLinks already carried by '*' via db/knowledge-graph.cds extend entity base.Tutorials - Unit test: test/unit/kg-exposure.test.js (2 tests, all green) - sqlite deploy clean (pre-existing warnings only) --- srv/admin-service.cds | 9 +++++++++ test/unit/kg-exposure.test.js | 17 +++++++++++++++++ 2 files changed, 26 insertions(+) create mode 100644 test/unit/kg-exposure.test.js diff --git a/srv/admin-service.cds b/srv/admin-service.cds index f93fcddd9..17ff36e69 100644 --- a/srv/admin-service.cds +++ b/srv/admin-service.cds @@ -116,6 +116,10 @@ service AdminService { // Filtered to channel='prod' so the admin OP shows prod-published media. images : Association to many TutorialImages on images.slug = $self.slug and images.channel = 'prod', assets : Association to many TutorialAssets on assets.slug = $self.slug and assets.channel = 'prod', + // KG facet (task-1): rank + co-completions associations. + // conceptLinks already carried by '*' (db/knowledge-graph.cds extends base.Tutorials). + rank : Association to one TutorialRank on rank.slug = $self.slug, + coCompletions : Association to many CoCompletions on coCompletions.sourceSlug = $self.slug, }; // Filtered picklist for redirectTo value help — only ACTIVE tutorials can be redirect targets @readonly @@ -634,6 +638,11 @@ service AdminService { @readonly @cds.redirection.target: false entity TutorialImages as projection on ims.TutorialImages; @readonly @cds.redirection.target: false entity TutorialAssets as projection on ims.TutorialAssets; + // KG facet (task-1): read-only projections used by the Tutorials admin OP KG tab. + @readonly @cds.redirection.target: false entity TutorialConceptLinks as projection on ims.TutorialConceptLinks; + @readonly @cds.redirection.target: false entity TutorialRank as projection on ims.TutorialRank; + @readonly @cds.redirection.target: false entity CoCompletions as projection on ims.CoCompletions; + // Issue #622 — read-only recipient list for the "Last Chance Emails" // admin section. Powers the dropdown for sendLastChanceEmail and the // preview list for sendLastChanceEmailsAllDormant. One row per diff --git a/test/unit/kg-exposure.test.js b/test/unit/kg-exposure.test.js new file mode 100644 index 000000000..1246b8c50 --- /dev/null +++ b/test/unit/kg-exposure.test.js @@ -0,0 +1,17 @@ +// test/unit/kg-exposure.test.js +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' +describe('KG exposure on AdminService', () => { + let m; beforeAll(async () => { m = await cds.load('*') }) + it('exposes concept links, rank, co-completions read-only', () => { + expect(m.definitions['AdminService.TutorialConceptLinks']).toBeTruthy() + expect(m.definitions['AdminService.TutorialRank']).toBeTruthy() + expect(m.definitions['AdminService.CoCompletions']).toBeTruthy() + }) + it('Tutorials carries conceptLinks / rank / coCompletions', () => { + const t = m.definitions['AdminService.Tutorials'].elements + expect(t.conceptLinks).toBeTruthy() + expect(t.rank).toBeTruthy() + expect(t.coCompletions).toBeTruthy() + }) +}) From 4957afc946a77619fe11451fb8528c73c404be13 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 16:00:14 -0400 Subject: [PATCH 31/71] feat(admin): expose community membership/label on Tutorials (#WS4) Add communityMembership association on AdminService.Tutorials pointing to KgCommunityMembers (projection on ims.KgCommunity) joined by slug. Fully-qualified target name (AdminService.KgCommunityMembers) required because the entity is in a later extend-service block. --- srv/admin-service.cds | 7 +++++++ test/unit/kg-community-link.test.js | 13 +++++++++++++ 2 files changed, 20 insertions(+) create mode 100644 test/unit/kg-community-link.test.js diff --git a/srv/admin-service.cds b/srv/admin-service.cds index 17ff36e69..d57df23c8 100644 --- a/srv/admin-service.cds +++ b/srv/admin-service.cds @@ -120,6 +120,13 @@ service AdminService { // conceptLinks already carried by '*' (db/knowledge-graph.cds extends base.Tutorials). rank : Association to one TutorialRank on rank.slug = $self.slug, coCompletions : Association to many CoCompletions on coCompletions.sourceSlug = $self.slug, + // KG facet (task-2): community membership — links this tutorial to its + // Louvain-detected community rows (slug-based join). From the OP, a $expand + // on communityMembership exposes communityId + communityFingerprint, which + // can be used to fetch the label from KgCommunityLabel. No schema change: + // KgCommunityMembers is an @readonly projection on ims.KgCommunity, which + // already carries the slug column. + communityMembership : Association to many AdminService.KgCommunityMembers on communityMembership.slug = $self.slug, }; // Filtered picklist for redirectTo value help — only ACTIVE tutorials can be redirect targets @readonly diff --git a/test/unit/kg-community-link.test.js b/test/unit/kg-community-link.test.js new file mode 100644 index 000000000..2406fa4fd --- /dev/null +++ b/test/unit/kg-community-link.test.js @@ -0,0 +1,13 @@ +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' + +describe('community label reachable', () => { + let m + beforeAll(async () => { + m = await cds.load('*') + }) + it('Tutorials exposes community membership or virtual label', () => { + const t = m.definitions['AdminService.Tutorials'].elements + expect(t.communityMembership || t.communityLabel).toBeTruthy() + }) +}) From fb994af48891b4f77e1e237617bd04811a7356d9 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 16:06:29 -0400 Subject: [PATCH 32/71] feat(admin-ui): Knowledge Graph facet (concepts/PageRank/community/co-completions) (#WS4) --- app/admin-annotations.cds | 41 ++++++++++++++++++++++++++++++++ test/unit/annotations-kg.test.js | 20 ++++++++++++++++ 2 files changed, 61 insertions(+) create mode 100644 test/unit/annotations-kg.test.js diff --git a/app/admin-annotations.cds b/app/admin-annotations.cds index 7e71172b9..fa1251dcc 100644 --- a/app/admin-annotations.cds +++ b/app/admin-annotations.cds @@ -982,6 +982,9 @@ annotate AdminService.Tutorials with @UI: { Target: 'freshnessFindings/@UI.LineItem' }, { $Type: 'UI.ReferenceFacet', Label: 'Images', ID: 'MediaImagesFacet', Target: 'images/@UI.LineItem' }, { $Type: 'UI.ReferenceFacet', Label: 'Assets', ID: 'MediaAssetsFacet', Target: 'assets/@UI.LineItem' }, + { $Type: 'UI.ReferenceFacet', Label: 'Knowledge Graph', ID: 'KgFieldsFacet', Target: '@UI.FieldGroup#KnowledgeGraph' }, + { $Type: 'UI.ReferenceFacet', Label: 'Concepts Taught', ID: 'ConceptsTaughtFacet', Target: 'conceptLinks/@UI.LineItem' }, + { $Type: 'UI.ReferenceFacet', Label: 'Co-Completed', ID: 'CoCompletionsFacet', Target: 'coCompletions/@UI.LineItem' }, { $Type: 'UI.CollectionFacet', ID: 'Feedback', Label: 'Feedback', Facets: [ { $Type: 'UI.ReferenceFacet', ID: 'FeedbackSummary', Target: 'feedbackSummary/@UI.FieldGroup#FeedbackSummary', @@ -4432,3 +4435,41 @@ annotate AdminService.TutorialAssets with @( { Value: contentHash, Label: 'Hash' } ] ); + +// --- Knowledge Graph facets (task-3) --- +// TutorialConceptLinks LineItem: concept FK, predicate (teaches|extends), confidence score. +annotate AdminService.TutorialConceptLinks with @( + UI.LineItem: [ + { Value: concept_ID, Label: 'Concept' }, + { Value: predicate, Label: 'Relation' }, + { Value: confidence, Label: 'Confidence' } + ] +); + +// CoCompletions LineItem: target tutorial slug + co-completion score. +annotate AdminService.CoCompletions with @( + UI.LineItem: [ + { Value: targetSlug, Label: 'Also Completed' }, + { Value: score, Label: 'Score' } + ] +); + +// KgCommunityMembers LineItem: community id + fingerprint for the OP facet. +annotate AdminService.KgCommunityMembers with @( + UI.LineItem: [ + { Value: communityId, Label: 'Community ID' }, + { Value: communityFingerprint, Label: 'Fingerprint' }, + { Value: vertexType, Label: 'Type' } + ] +); + +// FieldGroup for PageRank score — shown in KgFieldsFacet on Tutorials OP. +// rank is a to-one Association (slug-joined) added in task-1. +annotate AdminService.Tutorials with @( + UI.FieldGroup #KnowledgeGraph: { + Label: 'Knowledge Graph', + Data: [ + { $Type: 'UI.DataField', Value: rank.score, Label: 'PageRank Score' } + ] + } +); diff --git a/test/unit/annotations-kg.test.js b/test/unit/annotations-kg.test.js new file mode 100644 index 000000000..cbb4ecd9f --- /dev/null +++ b/test/unit/annotations-kg.test.js @@ -0,0 +1,20 @@ +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' + +describe('Knowledge Graph facets', () => { + let m + beforeAll(async () => { m = await cds.load('*') }) + + it('OP facets include KG facets', () => { + const ids = m.definitions['AdminService.Tutorials']['@UI.Facets'].map((f) => f.ID) + expect(ids).toContain('ConceptsTaughtFacet') + expect(ids).toContain('CoCompletionsFacet') + }) + + it('concept links LineItem shows predicate + confidence', () => { + const li = m.definitions['AdminService.TutorialConceptLinks']['@UI.LineItem'] + const vals = li.map((x) => x.Value?.['='] || x.Value) + expect(vals).toContain('predicate') + expect(vals).toContain('confidence') + }) +}) From 81fbf949224f2b6c1f9f789798f401635abbd29c Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 16:09:21 -0400 Subject: [PATCH 33/71] fix(admin-ui): add KgCommunityFacet to Tutorials OP (communityMembership wiring) (#WS4) --- app/admin-annotations.cds | 1 + test/unit/annotations-kg.test.js | 1 + 2 files changed, 2 insertions(+) diff --git a/app/admin-annotations.cds b/app/admin-annotations.cds index fa1251dcc..30d7542ca 100644 --- a/app/admin-annotations.cds +++ b/app/admin-annotations.cds @@ -985,6 +985,7 @@ annotate AdminService.Tutorials with @UI: { { $Type: 'UI.ReferenceFacet', Label: 'Knowledge Graph', ID: 'KgFieldsFacet', Target: '@UI.FieldGroup#KnowledgeGraph' }, { $Type: 'UI.ReferenceFacet', Label: 'Concepts Taught', ID: 'ConceptsTaughtFacet', Target: 'conceptLinks/@UI.LineItem' }, { $Type: 'UI.ReferenceFacet', Label: 'Co-Completed', ID: 'CoCompletionsFacet', Target: 'coCompletions/@UI.LineItem' }, + { $Type: 'UI.ReferenceFacet', Label: 'Community', ID: 'KgCommunityFacet', Target: 'communityMembership/@UI.LineItem' }, { $Type: 'UI.CollectionFacet', ID: 'Feedback', Label: 'Feedback', Facets: [ { $Type: 'UI.ReferenceFacet', ID: 'FeedbackSummary', Target: 'feedbackSummary/@UI.FieldGroup#FeedbackSummary', diff --git a/test/unit/annotations-kg.test.js b/test/unit/annotations-kg.test.js index cbb4ecd9f..e436acf56 100644 --- a/test/unit/annotations-kg.test.js +++ b/test/unit/annotations-kg.test.js @@ -9,6 +9,7 @@ describe('Knowledge Graph facets', () => { const ids = m.definitions['AdminService.Tutorials']['@UI.Facets'].map((f) => f.ID) expect(ids).toContain('ConceptsTaughtFacet') expect(ids).toContain('CoCompletionsFacet') + expect(ids).toContain('KgCommunityFacet') }) it('concept links LineItem shows predicate + confidence', () => { From ee93c1ac0af694fa63453b0f10e234b0b747069d Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 16:12:49 -0400 Subject: [PATCH 34/71] test(kg): hybrid guard for KG facet reads (#WS4) --- test/hybrid/kg-facet.test.js | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 test/hybrid/kg-facet.test.js diff --git a/test/hybrid/kg-facet.test.js b/test/hybrid/kg-facet.test.js new file mode 100644 index 000000000..69673d6ec --- /dev/null +++ b/test/hybrid/kg-facet.test.js @@ -0,0 +1,17 @@ +import { describe, it, expect, beforeAll } from 'vitest' +import cds from '@sap/cds' + +cds.test('serve', '--project', '.', '--profile', 'hybrid') + +describe('KG facet reads (hybrid)', () => { + let admin + beforeAll(async () => { admin = await cds.connect.to('AdminService') }) + it('reads a tutorial with KG associations expanded without error', async () => { + const t = await admin.run(SELECT.one.from('AdminService.Tutorials').columns('ID', 'slug')) + expect(t).toBeTruthy() + const links = await admin.run(SELECT.from('AdminService.TutorialConceptLinks').where({ tutorial_ID: t.ID })) + expect(Array.isArray(links)).toBe(true) // may be empty in DEV — that's fine + const co = await admin.run(SELECT.from('AdminService.CoCompletions').limit(1)) + expect(Array.isArray(co)).toBe(true) + }) +}) From 2532ae8de0c85e95037308dd67bfe28259bb0cc8 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 16:52:06 -0400 Subject: [PATCH 35/71] fix(ci): allowlist srv-only publish-contributors + publish-validation-rules routes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit WS2/WS3 added POST /content/publish-contributors and /content/publish-validation-rules to srv/server.js plus channel=qa skips in publish-content.ts, but did not update the route-drift guard's ALLOWLIST_ONLY_ON_SRV. Both are intentionally srv-only (srv-qa has no ContributorCache/ValidationRules entity — POST would 404/500, and no QA runtime reader), same rationale as validate-answer-specs (#1375). Fixes the failing 'Static build guards (postbuild:apps)' CI step on DEV. --- scripts/check-srv-qa-route-drift.ts | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/scripts/check-srv-qa-route-drift.ts b/scripts/check-srv-qa-route-drift.ts index d7907329b..4cfaed6e9 100644 --- a/scripts/check-srv-qa-route-drift.ts +++ b/scripts/check-srv-qa-route-drift.ts @@ -102,6 +102,18 @@ const ALLOWLIST_ONLY_ON_SRV: Record = { '(com.sap.developers.ims.qa) does not load, and srv-qa has no runtime reader of ' + 'ValidateAnswerSpecs (author preview re-parses rules.vr live). The publish CLI skips this ' + 'step for channel=qa. Re-evaluate only if QA gains a runtime /api/validate-answer surface.', + 'POST /content/publish-contributors': + 'Contributor sidecar publish (#WS2). Same rationale as validate-answer-specs above: srv-qa ' + + 'has no ContributorCache entity (the QA model com.sap.developers.ims.qa does not load it), ' + + 'so a POST would 404/500, and there is no QA runtime reader. The publish CLI already skips ' + + 'this step for channel=qa (scripts/publish-content.ts: "[publish-contributors] skipped"). ' + + 'Re-evaluate only if QA gains a contributor reader.', + 'POST /content/publish-validation-rules': + 'Validation-rules sidecar publish (#WS3). Same rationale as publish-contributors above: ' + + 'srv-qa has no ValidationRules entity, so a POST would 404/500, and there is no QA runtime ' + + 'reader. The publish CLI already skips this step for channel=qa ' + + '(scripts/publish-content.ts: "[publish-validation-rules] skipped"). Re-evaluate only if ' + + 'QA gains a validation-rules reader.', 'GET /content/authors/:login': 'CAP-served /authors/{login}/ pages (#1659 Phase C) — a public prod content surface that ' + 'aggregates across published tutorials, not tutorial-draft author preview. Same rationale ' + From c9eeb27b5beeae35fa0ab2c077be509cab429338 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 17:22:35 -0400 Subject: [PATCH 36/71] fix(srv-qa): lazy-import category-classifier-llm to unbreak srv-qa boot MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 1's category self-heal added a static import of category-classifier into content-publish-session.js (loaded at srv-qa boot). category-classifier statically imported category-classifier-llm, which requires @sap-ai-sdk/orchestration — a package NOT in the stripped srv-qa container (only @sap-ai-sdk/foundation-models is). srv-qa crash-looped at boot with ERR_MODULE_NOT_FOUND, failing the MTA deploy. Fix at the root: import category-classifier-llm lazily inside the LLM fallback path (which srv-qa never reaches). category-classifier.js now loads without resolving the AI SDK, so any boot-time importer is safe. No behavior change on the main srv (LLM path still runs when reached). --- srv/lib/category-classifier.js | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/srv/lib/category-classifier.js b/srv/lib/category-classifier.js index 22f72f979..a328e637f 100644 --- a/srv/lib/category-classifier.js +++ b/srv/lib/category-classifier.js @@ -13,7 +13,11 @@ import cds from '@sap/cds'; import { getSeedEmbeddings, embedAdHoc } from './category-seed-embeddings.js'; -import { classifyViaLlm } from './category-classifier-llm.js'; +// category-classifier-llm (→ @sap-ai-sdk/orchestration) is imported LAZILY inside the +// LLM fallback path below. A static import here resolves @sap-ai-sdk/orchestration at +// module load, which crash-loops the srv-qa container (that package is not installed in +// the stripped srv-qa deps — only @sap-ai-sdk/foundation-models is). srv-qa never reaches +// the LLM path, so deferring the import keeps srv-qa boot free of the missing dependency. const LOG = cds.log('category-classifier'); @@ -157,6 +161,7 @@ export async function classifyAndPersist(kind, id, _opts = {}) { if (!assigned) { path = 'llm'; try { + const { classifyViaLlm } = await import('./category-classifier-llm.js'); const { assigned: llmAssigned } = await classifyViaLlm({ title: item.raw.title, description: item.raw.description, From f3077ccac0b8ef69c579ed629d77ff9c5fc2ac47 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Mon, 31 Aug 2026 21:04:51 -0400 Subject: [PATCH 37/71] fix(publish): gate category self-heal to avoid OOM on bulk publishes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 1's category self-heal fired classifyTouchedTutorials(tutorialIds) as an unbounded Promise.all on every append batch. Fine for incremental publishes (a few slugs) — but a full publish touches ~1700 tutorials, so it spawned ~1700 concurrent classifications (each an AI-Core embedding call) → memory spike → tutorials-srv OOM-killed (exit 137) mid /content/publish/append, failing the whole publish. Fix: skip self-heal for bulk publishes (> 25 touched tutorials; those are (re)classified by scripts/backfill-categories.cjs), and classify incremental publishes SEQUENTIALLY instead of a fan-out Promise.all. No behavior change for normal incremental publishes; categories for full rebuilds come from the backfill. --- srv/lib/content-publish-session.js | 34 ++++++++++++++++----- test/unit/publish-category-selfheal.test.js | 12 ++++++++ 2 files changed, 39 insertions(+), 7 deletions(-) diff --git a/srv/lib/content-publish-session.js b/srv/lib/content-publish-session.js index 7a5f487b7..79ef5dd96 100644 --- a/srv/lib/content-publish-session.js +++ b/srv/lib/content-publish-session.js @@ -13,16 +13,36 @@ import { resolveTutorialAuthor } from './resolve-tutorial-author.js'; import { isDeltaWrite, isDeltaSkipCarryForward } from './content-delta-flags.js'; import { classifyAndPersist } from './category-classifier.js'; +// Incremental publishes above this many touched tutorials are treated as "bulk" +// and skip the fire-and-forget self-heal (categories are (re)populated in bulk by +// scripts/backfill-categories.cjs instead). Rationale: a full rebuild touches +// ~all tutorials; firing an unbounded classification per tutorial (each does an +// AI-Core embedding call) OOM-killed tutorials-srv mid-publish. See memory +// srv-category-selfheal-bulk-publish-oom. +const SELF_HEAL_MAX_TUTORIALS = 25; + // Exported for unit testing; classifies touched tutorials without ever throwing // into the publish tx (publish bypasses the CAP after('CREATE') classifier hook). +// Bulk publishes are skipped (backfill script handles them); incremental publishes +// classify SEQUENTIALLY — never a fan-out Promise.all — so the classifier can't +// stampede the srv / AI Core during a publish. export async function classifyTouchedTutorials(tutorialIds) { - await Promise.all( - (tutorialIds || []).map((id) => - Promise.resolve() - .then(() => classifyAndPersist('tutorial', id)) - .catch((e) => console.warn('[publish] category classify skipped', id, e?.message)), - ), - ); + const ids = tutorialIds || []; + if (ids.length === 0) return; + if (ids.length > SELF_HEAL_MAX_TUTORIALS) { + console.warn( + `[publish] category self-heal skipped for bulk publish (${ids.length} tutorials > ${SELF_HEAL_MAX_TUTORIALS}); ` + + `run scripts/backfill-categories.cjs to (re)classify in bulk`, + ); + return; + } + for (const id of ids) { + try { + await classifyAndPersist('tutorial', id); + } catch (e) { + console.warn('[publish] category classify skipped', id, e?.message); + } + } } const LOG = cds.log('content-publish'); diff --git a/test/unit/publish-category-selfheal.test.js b/test/unit/publish-category-selfheal.test.js index cc66dbea7..8ff589d3a 100644 --- a/test/unit/publish-category-selfheal.test.js +++ b/test/unit/publish-category-selfheal.test.js @@ -24,4 +24,16 @@ describe('publish category self-heal', () => { classifySpy.mockRejectedValueOnce(new Error('boom')) await expect(classifyTouchedTutorials(['id-a'])).resolves.toBeUndefined() }) + + it('skips self-heal for a bulk publish (> 25 tutorials) so it cannot stampede', async () => { + const bulk = Array.from({ length: 26 }, (_, i) => `id-${i}`) + await classifyTouchedTutorials(bulk) + expect(classifySpy).not.toHaveBeenCalled() + }) + + it('still classifies an at-threshold incremental publish (25 tutorials)', async () => { + const ids = Array.from({ length: 25 }, (_, i) => `id-${i}`) + await classifyTouchedTutorials(ids) + expect(classifySpy).toHaveBeenCalledTimes(25) + }) }) From d310d32f81423a551dfcfaccfd35da31dc51d7b0 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Tue, 1 Sep 2026 13:06:45 -0400 Subject: [PATCH 38/71] fix(categories): resolve namespaced entities so backfill works on HANA + seed classifier descriptions The category backfill (scripts/backfill-categories.cjs) and classifier persist() queried bare short-name entity strings ('Tutorials', 'TutorialCategories'). Those resolve on local SQLite but emit unqualified TUTORIALS / TUTORIALCATEGORIES SQL against HANA (real tables are COM_SAP_DEVELOPERS_IMS_*), so the backfill died on its first query via `cds bind --exec` and only its SQLite unit test ever passed. - backfill: resolve ims entity OBJECTS via a linked model (resolveImsEntities, falling back to cds.linked(cds.load('*')) when cds.model is unset in the standalone runner), and SELECT with explicit .columns('ID'). - classifier persist(): resolve the junction via cds.entities(ns)[junction] instead of a bare string (loadItemText already resolved this way). Second, independent blocker: Category.seedDescription was unseeded (CSV ships only ID/slug/label/sortOrder), so the embedding classifier degraded to LLM-only. Add baseline seed texts (category-seed-descriptions-defaults.js) and an idempotent, non-destructive boot-seed (seed-category-descriptions.js, wired into cds.on('served'), VITEST-gated, fills only empty rows). Kept out of CSV so deploys can't full-replace the admin-editable column. Tests: unit (defaults integrity + idempotency + non-destructive) and hybrid (load+link resolution reaches the namespaced HANA tables; seed runs on HANA). --- scripts/backfill-categories.cjs | 28 ++++++- srv/lib/category-classifier.js | 11 ++- .../category-seed-descriptions-defaults.js | 65 +++++++++++++++ srv/lib/seed-category-descriptions.js | 64 +++++++++++++++ srv/server.js | 20 +++++ ...ackfill-categories-hana-resolution.test.js | 63 ++++++++++++++ test/unit/seed-category-descriptions.test.js | 82 +++++++++++++++++++ 7 files changed, 330 insertions(+), 3 deletions(-) create mode 100644 srv/lib/category-seed-descriptions-defaults.js create mode 100644 srv/lib/seed-category-descriptions.js create mode 100644 test/hybrid/backfill-categories-hana-resolution.test.js create mode 100644 test/unit/seed-category-descriptions.test.js diff --git a/scripts/backfill-categories.cjs b/scripts/backfill-categories.cjs index b0c2d4a13..f3c783092 100644 --- a/scripts/backfill-categories.cjs +++ b/scripts/backfill-categories.cjs @@ -40,6 +40,28 @@ function kindToEntity(kind) { return map[kind]; } +/** + * Resolve the ims namespace entities robustly across contexts. + * + * In a standalone `cds bind --exec` runner the model may not be linked into + * the cds.entities() globals (cds.model unset), so fall back to explicitly + * loading + linking (same gotcha as srv/lib/seed-poc-puzzle.js). Returns the + * RESOLVED entity objects — SELECT/DELETE/INSERT against a bare short-name + * STRING ('Tutorials') does NOT resolve to the namespaced HANA table + * (COM_SAP_DEVELOPERS_IMS_TUTORIALS) and dies with "Could not find table/view + * TUTORIALS"; it only ever worked against local SQLite. + * + * @param {typeof import('@sap/cds')} cds + * @returns {Promise>} + */ +async function resolveImsEntities(cds) { + if (typeof cds.entities === 'function' && cds.model) { + return cds.entities('com.sap.developers.ims'); + } + const linked = cds.linked(await cds.load('*')); + return linked.entities('com.sap.developers.ims'); +} + /** * Run a batch of items concurrently using Promise.allSettled. * @param {Array} items @@ -64,6 +86,10 @@ async function main(argv) { // Connect to database const db = await cds.connect.to('db'); + // Resolve namespaced entity objects once (see resolveImsEntities). Passing a + // bare short-name string to SELECT.from fails against HANA. + const ims = await resolveImsEntities(cds); + // Dynamic import for ESM module const { classifyAndPersist } = await import('../srv/lib/category-classifier.js'); @@ -79,7 +105,7 @@ async function main(argv) { // Fetch all IDs ordered const rows = await db.run( - SELECT.from(entityName).columns('ID').orderBy('ID') + SELECT.from(ims[entityName]).columns('ID').orderBy('ID') ); let items = rows; diff --git a/srv/lib/category-classifier.js b/srv/lib/category-classifier.js index a328e637f..5f6777bad 100644 --- a/srv/lib/category-classifier.js +++ b/srv/lib/category-classifier.js @@ -100,15 +100,22 @@ function pickEmbeddingResult(scored) { async function persist(kind, itemId, assigned) { const cfg = KIND_TO_ENTITY[kind]; + // Resolve the junction to its namespaced entity object. A bare short-name + // string ('TutorialCategories') does NOT resolve to the HANA table + // (COM_SAP_DEVELOPERS_IMS_TUTORIALCATEGORIES) when this runs outside a + // service handler (e.g. the standalone backfill via `cds bind --exec`) — + // it emits bare `TUTORIALCATEGORIES` SQL and dies. loadItemText already + // resolves this way, so cds.entities() is known-good by the time we get here. + const junction = cds.entities('com.sap.developers.ims')[cfg.junction]; await cds.tx(async (tx) => { - await tx.run(DELETE.from(cfg.junction).where({ [cfg.fk]: itemId })); + await tx.run(DELETE.from(junction).where({ [cfg.fk]: itemId })); if (assigned.length === 0) return; const rows = assigned.map(a => ({ [cfg.fk]: itemId, category_ID: a.ID, score: a.score ?? 1.0, })); - await tx.run(INSERT.into(cfg.junction).entries(rows)); + await tx.run(INSERT.into(junction).entries(rows)); }); } diff --git a/srv/lib/category-seed-descriptions-defaults.js b/srv/lib/category-seed-descriptions-defaults.js new file mode 100644 index 000000000..49c552ced --- /dev/null +++ b/srv/lib/category-seed-descriptions-defaults.js @@ -0,0 +1,65 @@ +// srv/lib/category-seed-descriptions-defaults.js +// +// Single source of truth for the baseline Category.seedDescription texts. +// +// These paragraphs are what the category classifier EMBEDS and cosine-compares +// against each tutorial/mission/group's (title + description + primaryTag). They +// are intentionally keyword-rich and product-named so the embedding path +// (srv/lib/category-classifier.js) can classify without falling back to the LLM. +// +// The Categories reference rows themselves come from +// db/data/com.sap.developers.ims-Categories.csv (ID/slug/label/sortOrder only — +// no seedDescription column, so seeds are NOT shipped via CSV; that would +// full-replace the admin-editable column on every deploy). Instead these are +// seeded idempotently + non-destructively at boot by +// ./seed-category-descriptions.js, which fills ONLY rows whose seedDescription +// is empty — admin edits made at /admin-ui/ are preserved. +// +// Keyed by Categories.slug (stable) so a re-ordered/re-IDed CSV can't misalign. + +export const CATEGORY_SEED_DESCRIPTIONS = { + 'app-dev-automation': + 'Building business applications and extensions with the SAP Cloud Application ' + + 'Programming Model (CAP), SAP Build and SAP Build Process Automation, low-code and ' + + 'pro-code development, workflow and business process automation, the ABAP RESTful ' + + 'Application Programming Model (RAP), side-by-side extensions, SAP Business Application ' + + 'Studio, and developer tooling for creating, deploying, and automating apps and services.', + + 'data-analytics': + 'Working with data using SAP HANA Cloud, SAP Datasphere, and SAP Analytics Cloud: data ' + + 'modeling, SQL and calculation views, data federation and replication, business ' + + 'intelligence, reporting, dashboards, stories, data warehousing, and analytical models.', + + 'extended-planning': + 'Financial and operational planning, budgeting, forecasting, and analysis with SAP ' + + 'Analytics Cloud planning, SAP Datasphere, and extended planning and analysis (xP&A): ' + + 'predictive planning, allocations, value driver trees, and enterprise performance management.', + + 'integration': + 'Connecting systems and services with SAP Integration Suite: Cloud Integration, API ' + + 'Management, Open Connectors, event-driven integration with SAP Event Mesh and Advanced ' + + 'Event Mesh, EDI and B2B, destinations, connectivity, and integrating SAP with ' + + 'third-party applications.', + + 'artificial-intelligence': + 'Building intelligent applications with SAP AI Core, the Generative AI Hub, SAP Business ' + + 'AI, and Joule: machine learning, large language models, embeddings and ' + + 'retrieval-augmented generation (RAG), document information extraction, orchestration, ' + + 'and AI-powered automation and copilots.', + + 'frontend-ux': + 'Creating user interfaces and experiences with SAPUI5, SAP Fiori and Fiori Elements, UI5 ' + + 'Web Components, SAP Build Apps, HTML, CSS and JavaScript, React and Vue front ends, ' + + 'responsive design, theming, and building engaging developer and end-user experiences.', + + 'cloud-operations': + 'Operating and administering SAP BTP, the Cloud Foundry and Kyma runtimes: deployment ' + + 'with multitarget applications (MTA), CI/CD pipelines and DevOps, security and ' + + 'authentication with XSUAA, monitoring, logging and alerting, subaccounts, entitlements, ' + + 'and cloud lifecycle management.', + + 'abap-core': + 'ABAP programming and ABAP Cloud development: clean core extensibility, SAP S/4HANA and ' + + 'on-premise systems, RAP and CDS views in ABAP, the ABAP Development Tools (ADT) in ' + + 'Eclipse, released (tier-1) APIs, and core ERP business logic and data models.', +}; diff --git a/srv/lib/seed-category-descriptions.js b/srv/lib/seed-category-descriptions.js new file mode 100644 index 000000000..32c5586e1 --- /dev/null +++ b/srv/lib/seed-category-descriptions.js @@ -0,0 +1,64 @@ +// srv/lib/seed-category-descriptions.js +// +// Idempotent, NON-DESTRUCTIVE boot-seed for Category.seedDescription. +// +// Categories.csv ships ID/slug/label/sortOrder only — the admin-editable +// seedDescription column is deliberately absent (a CSV column would +// full-replace admin edits on every deploy; see CLAUDE.md "CSV changes wipe +// admin-editable columns"). So the baseline seed texts live in +// ./category-seed-descriptions-defaults.js and are applied here, filling ONLY +// rows whose seedDescription is currently empty/null. Rows an author has +// already edited (non-empty) are never touched. +// +// Called from: +// - srv/server.js cds.on('served') (guarded by globalThis sentinel + VITEST gate) +// - test/unit + test/hybrid (via dynamic import, passing a db override) + +import cds from '@sap/cds'; +import { CATEGORY_SEED_DESCRIPTIONS } from './category-seed-descriptions-defaults.js'; + +const NAMESPACE = 'com.sap.developers.ims'; + +/** + * Seed missing Category.seedDescription values idempotently. + * + * @param {object} [dbOverride] — already-connected cds db (tests). When omitted, + * connects via cds.connect.to('db'). + * @returns {Promise<{updated: number, total: number}>} + * updated = rows whose empty seedDescription we filled this run + * total = category rows examined + */ +export async function seedCategoryDescriptions(dbOverride) { + const db = dbOverride ?? await cds.connect.to('db'); + + // Resolve Categories robustly across contexts (booted server / cds.test → + // cds.entities() installed; standalone `cds bind --exec` → model not linked + // into globals, so load+link). Same gotcha as seed-poc-puzzle.js. + let Categories; + if (typeof cds.entities === 'function' && cds.model) { + ({ Categories } = cds.entities(NAMESPACE)); + } else { + const linked = cds.linked(await cds.load('*')); + ({ Categories } = linked.entities(NAMESPACE)); + } + + // Explicit columns: a bare SELECT emits `SELECT *`, which HANA cannot infer + // when the entity comes from a separately-linked model (standalone path). + const rows = await db.run( + SELECT.from(Categories).columns('ID', 'slug', 'seedDescription') + ); + + let updated = 0; + for (const row of rows) { + const seed = CATEGORY_SEED_DESCRIPTIONS[row.slug]; + if (!seed) continue; // slug not in defaults → leave alone + const current = (row.seedDescription ?? '').trim(); + if (current) continue; // author-authored / already seeded → preserve + await db.run( + UPDATE(Categories).set({ seedDescription: seed }).where({ ID: row.ID }) + ); + updated++; + } + + return { updated, total: rows.length }; +} diff --git a/srv/server.js b/srv/server.js index 65a36abce..1835c59b6 100644 --- a/srv/server.js +++ b/srv/server.js @@ -1437,6 +1437,26 @@ cds.on('served', async () => { } } + // Seed baseline Category.seedDescription texts (tunes the embedding-based + // category classifier). Idempotent + non-destructive: fills only rows whose + // seedDescription is empty, never overwrites admin edits. Kept out of CSV so + // deploys can't full-replace the admin-editable column. VITEST-gated: seeding + // descriptions flips the classifier's embedding path ON, which would leak + // into unit tests asserting the LLM/skip fallback (see memory + // db-flag-boot-seed-leaks-into-vitest-harness). Non-fatal. + if (!process.env.VITEST && !globalThis.__categorySeedDescriptionsSeeded) { + globalThis.__categorySeedDescriptionsSeeded = true; + try { + const { seedCategoryDescriptions } = await import('./lib/seed-category-descriptions.js'); + const result = await seedCategoryDescriptions(cds.db); + if (result.updated > 0) { + console.log(`[category-seed-descriptions] seeded ${result.updated}/${result.total} category descriptions`); + } + } catch (err) { + console.warn('[category-seed-descriptions] seed failed (non-fatal):', err.message); + } + } + app.get('/auth/user', contextMw, authMw, async (req, res) => { // #1268: coarse deploy environment (DEV/PROD/QA/LOCAL) for the admin // header. Derived from the CF space name — safe to expose to anonymous diff --git a/test/hybrid/backfill-categories-hana-resolution.test.js b/test/hybrid/backfill-categories-hana-resolution.test.js new file mode 100644 index 000000000..772d20aaf --- /dev/null +++ b/test/hybrid/backfill-categories-hana-resolution.test.js @@ -0,0 +1,63 @@ +import { describe, it, expect } from 'vitest'; +import cds from '@sap/cds'; +import { isSafeForWrites } from './_guard.js'; + +// Runs against real HANA via `cds bind --exec` + the hybrid profile. +// The seed is non-destructive (fills only empty seedDescription); still gated +// behind isSafeForWrites() so it can never touch a prod container. +const RUN = process.env.HYBRID_TESTS === 'true' && isSafeForWrites(); + +cds.test('serve', '--project', '.', '--profile', 'hybrid'); + +const NS = 'com.sap.developers.ims'; + +(RUN ? describe : describe.skip)('hybrid: category backfill entity resolution against real HANA', () => { + // The backfill (scripts/backfill-categories.cjs) and classifier persist() + // regressed because they queried bare short-name strings ('Tutorials', + // 'TutorialCategories'), which resolve on local SQLite but emit bare + // TUTORIALS / TUTORIALCATEGORIES SQL against HANA (real tables are + // COM_SAP_DEVELOPERS_IMS_*). The fix resolves entity OBJECTS via a linked + // model. This test drives that exact load+link path (the standalone + // `cds bind --exec` branch, where cds.model is unset) against real HANA. + + it('load+link resolution reaches the namespaced HANA tables (no "table not found")', async () => { + const db = await cds.connect.to('db'); + const linked = cds.linked(await cds.load('*')); + const ents = linked.entities(NS); + + for (const name of ['Tutorials', 'Missions', 'Groups', 'TutorialCategories', 'MissionCategories', 'GroupCategories']) { + const ent = ents[name]; + expect(ent, `${name} must resolve from the linked model`).toBeTruthy(); + // Fully-qualified so the emitted SQL targets COM_SAP_DEVELOPERS_IMS_, + // not a bare unqualified table. + expect(ent.name).toBe(`${NS}.${name}`); + // The real assertion: a resolved-object SELECT executes on HANA. A bare + // short-name string here would throw "Could not find table/view ". + await expect( + db.run(SELECT.from(ent).columns('ID').limit(1)), + ).resolves.toBeDefined(); + } + }); + + it('seedCategoryDescriptions runs against real HANA and leaves all seeds populated', async () => { + const { seedCategoryDescriptions } = await import('../../srv/lib/seed-category-descriptions.js'); + const { CATEGORY_SEED_DESCRIPTIONS } = await import('../../srv/lib/category-seed-descriptions-defaults.js'); + const db = await cds.connect.to('db'); + + const res = await seedCategoryDescriptions(db); + expect(res.total).toBeGreaterThan(0); + expect(res.updated).toBeGreaterThanOrEqual(0); // may already be seeded from a prior run + + // Non-destructive + idempotent: after seeding, every category that has a + // default must carry a non-empty seedDescription, and a second run is a no-op. + const { Categories } = cds.linked(await cds.load('*')).entities(NS); + const rows = await db.run(SELECT.from(Categories).columns('slug', 'seedDescription')); + for (const row of rows) { + if (CATEGORY_SEED_DESCRIPTIONS[row.slug]) { + expect((row.seedDescription ?? '').trim().length).toBeGreaterThan(0); + } + } + const again = await seedCategoryDescriptions(db); + expect(again.updated).toBe(0); + }); +}); diff --git a/test/unit/seed-category-descriptions.test.js b/test/unit/seed-category-descriptions.test.js new file mode 100644 index 000000000..bfa539348 --- /dev/null +++ b/test/unit/seed-category-descriptions.test.js @@ -0,0 +1,82 @@ +import { describe, it, expect, beforeAll } from 'vitest'; +import cds from '@sap/cds'; +import { seedCategoryDescriptions } from '../../srv/lib/seed-category-descriptions.js'; +import { CATEGORY_SEED_DESCRIPTIONS } from '../../srv/lib/category-seed-descriptions-defaults.js'; + +cds.test('serve', '--project', '.', '--in-memory'); + +// NOTE: the boot seed (srv/server.js) is VITEST-gated, so category +// seedDescriptions are NOT auto-filled here — every test drives +// seedCategoryDescriptions(db) explicitly. Categories rows themselves come from +// db/data/com.sap.developers.ims-Categories.csv (ID/slug/label/sortOrder), which +// loads into the in-memory DB; their seedDescription starts null. + +describe('CATEGORY_SEED_DESCRIPTIONS (defaults integrity)', () => { + it('has non-empty descriptions and keys match the shipped category slugs', async () => { + const db = await cds.connect.to('db'); + const { Categories } = cds.entities('com.sap.developers.ims'); + const rows = await db.run(SELECT.from(Categories).columns('slug')); + const csvSlugs = new Set(rows.map((r) => r.slug)); + + const defaultSlugs = Object.keys(CATEGORY_SEED_DESCRIPTIONS); + expect(defaultSlugs.length).toBe(rows.length); // one default per shipped category + for (const slug of defaultSlugs) { + expect(csvSlugs.has(slug), `default slug '${slug}' not in Categories.csv`).toBe(true); + const text = CATEGORY_SEED_DESCRIPTIONS[slug]; + expect(typeof text).toBe('string'); + expect(text.trim().length).toBeGreaterThan(40); // rich enough to embed + } + }); +}); + +describe('seedCategoryDescriptions (idempotent, non-destructive boot seed)', () => { + let db; + let Categories; + beforeAll(async () => { + db = await cds.connect.to('db'); + ({ Categories } = cds.entities('com.sap.developers.ims')); + }); + + it('fills every empty seedDescription on first run', async () => { + const res = await seedCategoryDescriptions(db); + expect(res.total).toBe(Object.keys(CATEGORY_SEED_DESCRIPTIONS).length); + expect(res.updated).toBe(res.total); // all started empty + + const rows = await db.run(SELECT.from(Categories).columns('slug', 'seedDescription')); + for (const row of rows) { + expect(row.seedDescription).toBe(CATEGORY_SEED_DESCRIPTIONS[row.slug]); + } + }); + + it('is idempotent — a second run updates nothing', async () => { + const res = await seedCategoryDescriptions(db); + expect(res.updated).toBe(0); + }); + + it('never overwrites an admin-edited seedDescription', async () => { + const edited = 'ADMIN EDITED SEED — do not clobber'; + const target = Object.keys(CATEGORY_SEED_DESCRIPTIONS)[0]; + await db.run(UPDATE(Categories).set({ seedDescription: edited }).where({ slug: target })); + + const res = await seedCategoryDescriptions(db); + expect(res.updated).toBe(0); // nothing empty → nothing changed + + const row = await db.run( + SELECT.one.from(Categories).columns('seedDescription').where({ slug: target }), + ); + expect(row.seedDescription).toBe(edited); // untouched + }); + + it('re-seeds a description that was cleared back to empty (self-heals)', async () => { + const target = Object.keys(CATEGORY_SEED_DESCRIPTIONS)[1]; + await db.run(UPDATE(Categories).set({ seedDescription: '' }).where({ slug: target })); + + const res = await seedCategoryDescriptions(db); + expect(res.updated).toBe(1); + + const row = await db.run( + SELECT.one.from(Categories).columns('seedDescription').where({ slug: target }), + ); + expect(row.seedDescription).toBe(CATEGORY_SEED_DESCRIPTIONS[target]); + }); +}); From b1319112e3345f8c4e4912115bb4c361722bcfa0 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Tue, 1 Sep 2026 13:11:24 -0400 Subject: [PATCH 39/71] fix(categories): pass resolved embedding model to embed() in seed-embeddings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit category-seed-embeddings.js called embed() with no model arg in loadAll, recomputeStale, and embedAdHoc, so AzureOpenAiEmbeddingClient(undefined) threw "Cannot read properties of undefined (reading 'modelName')" on every classify — the embedding path was dead and every item fell back to LLM-only, even with seedDescriptions populated. Same #2001 class of bug as the freshness pipeline. Resolve the model via resolveEmbeddingSettings() like every other embed() caller (relevance-seed-embeddings.js is the direct analog), memoized at module level so a bulk backfill doesn't re-read ChatSettings per item. Verified against DEV: path=embedding now fires. Both category-seed-embeddings.js and chat-settings-resolver.js are already in the srv-qa cp list, and the resolver imports only @sap/cds → no srv-qa boot-crash risk. --- srv/lib/category-seed-embeddings.js | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/srv/lib/category-seed-embeddings.js b/srv/lib/category-seed-embeddings.js index a621ae7ba..4f064ae29 100644 --- a/srv/lib/category-seed-embeddings.js +++ b/srv/lib/category-seed-embeddings.js @@ -18,17 +18,35 @@ import cds from '@sap/cds'; import { embed } from './embedding-client.js'; +import { resolveEmbeddingSettings } from './chat-settings-resolver.js'; const LOG = cds.log('category-seed-embeddings'); let _cache = null; // Map | null let _stale = new Set(); // IDs marked invalid; recomputed on next getSeedEmbeddings() let _loadingPromise = null; // Promise | null — in-flight loader +let _modelPromise = null; // Promise | null — memoized embedding model name /** Test-only — resets module state between tests. */ export function _resetCache() { _cache = null; _stale = new Set(); _loadingPromise = null; + _modelPromise = null; +} + +/** + * Resolve (once, memoized) the embedding model name. embed() REQUIRES a model + * — passing undefined constructs AzureOpenAiEmbeddingClient(undefined), which + * throws "Cannot read properties of undefined (reading 'modelName')" (the + * #2001 class of bug: embed callers must pass a resolved model). The model + * rarely changes; memoize so a bulk backfill (thousands of embedAdHoc calls) + * doesn't re-read ChatSettings per item. Cleared by _resetCache() in tests. + */ +function getEmbeddingModel() { + if (!_modelPromise) { + _modelPromise = resolveEmbeddingSettings().then((s) => s.model); + } + return _modelPromise; } /** @@ -44,7 +62,7 @@ async function loadAll() { LOG.warn('No categories with seedDescription found — classifier will fall back to LLM for everything'); return new Map(); } - const vectors = await embed(usable.map(r => r.seedDescription)); + const vectors = await embed(usable.map(r => r.seedDescription), await getEmbeddingModel()); const m = new Map(); for (let i = 0; i < usable.length; i++) { m.set(usable[i].ID, vectors[i]); @@ -69,7 +87,7 @@ async function recomputeStale(staleIds) { const rows = await SELECT.from(Categories).columns('ID', 'seedDescription'); const targets = rows.filter(r => staleIds.has(r.ID) && r.seedDescription && r.seedDescription.trim().length > 0); if (targets.length === 0) return; - const vectors = await embed(targets.map(r => r.seedDescription)); + const vectors = await embed(targets.map(r => r.seedDescription), await getEmbeddingModel()); for (let i = 0; i < targets.length; i++) { _cache.set(targets[i].ID, vectors[i]); } @@ -142,6 +160,6 @@ export async function embedAdHoc(text) { if (!text || !text.trim()) { throw new Error('embedAdHoc: empty text'); } - const [vec] = await embed([text]); + const [vec] = await embed([text], await getEmbeddingModel()); return vec; } From 3b811e1e771373ee4b41f18e8a1fdae4c65981e8 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Tue, 1 Sep 2026 14:33:47 -0400 Subject: [PATCH 40/71] feat(admin-ui): make Tutorials OP facets readable (concepts/community/images) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three Object-Page facets on the Tutorials admin UI showed raw keys or blanks. Per Tom's feedback: - Concepts Taught: concept_ID rendered a bare GUID. Annotate the `concept` association with @Common.Text: concept.name (#TextOnly) so the FK shows the human-readable concept name. - Community: membership facet showed a numeric communityId + opaque fingerprint. Expose KgCommunityLabelInfo (projection on the existing KgCommunityLabel) + a `labelInfo` select-list association joined on communityFingerprint, and annotate communityId with @Common.Text: labelInfo.label (#TextFirst). LineItem now shows Community (labelled) + member slug + type; fingerprint dropped. Label is blank until the Louvain labeling job (#1126) runs in an env — member slug is the always-present fallback. - Images: "Bytes" looked empty on legacy rows and there was no link to the stored/served image. Add virtual thumbUrl/viewUrl to the TutorialImages projection, filled in after('READ') from sourceUrl (thumbUrl → approuter /img-cdn resize/WebP preview; viewUrl → CAP /content/image-source raw original, self-heals from the object store). LineItem now leads with an inline thumbnail (@UI.IsImageURL) + a "Image (served)" link alongside the GitHub source and the mime/bytes/hash/channel store details. Guarded by test/unit/admin-facet-readability.test.js (CSN annotation guards + served URL-enrichment check incl. null-sourceUrl skip). EDMX verified: all three annotations land on concept_ID, communityId, thumbUrl. Admin-UI change — needs a FULL deploy (no --skip-build / -m scoping) to view. --- app/admin-annotations.cds | 28 ++++-- srv/admin-service.cds | 25 ++++- srv/admin-service.js | 15 +++ test/unit/admin-facet-readability.test.js | 108 ++++++++++++++++++++++ 4 files changed, 167 insertions(+), 9 deletions(-) create mode 100644 test/unit/admin-facet-readability.test.js diff --git a/app/admin-annotations.cds b/app/admin-annotations.cds index 30d7542ca..7d5a38e86 100644 --- a/app/admin-annotations.cds +++ b/app/admin-annotations.cds @@ -4420,13 +4420,18 @@ annotate AdminService.FreshnessFinding with { // --- Media facets: TutorialImages + TutorialAssets (Task 4) --- annotate AdminService.TutorialImages with @( UI.LineItem: [ + { Value: thumbUrl, Label: 'Preview' }, + { $Type: 'UI.DataFieldWithUrl', Value: viewUrl, Url: viewUrl, Label: 'Image (served)' }, { $Type: 'UI.DataFieldWithUrl', Value: sourceUrl, Url: sourceUrl, Label: 'Source (GitHub)' }, { Value: mimeType, Label: 'Type' }, { Value: byteSize, Label: 'Bytes' }, { Value: contentHash, Label: 'Hash' }, { Value: channel, Label: 'Channel' } ] -); +) { + // Render thumbUrl inline as an image thumbnail rather than as raw text. + thumbUrl @UI.IsImageURL; +}; annotate AdminService.TutorialAssets with @( UI.LineItem: [ { Value: filename, Label: 'File' }, @@ -4445,7 +4450,12 @@ annotate AdminService.TutorialConceptLinks with @( { Value: predicate, Label: 'Relation' }, { Value: confidence, Label: 'Confidence' } ] -); +) { + // Show the human-readable concept name instead of the raw GUID FK. + // Annotate the association (not concept_ID) so the compiler propagates the + // text to the generated foreign key — mirrors the primaryTagRef precedent. + concept @Common.Text: concept.name @Common.TextArrangement: #TextOnly; +}; // CoCompletions LineItem: target tutorial slug + co-completion score. annotate AdminService.CoCompletions with @( @@ -4455,14 +4465,18 @@ annotate AdminService.CoCompletions with @( ] ); -// KgCommunityMembers LineItem: community id + fingerprint for the OP facet. +// KgCommunityMembers LineItem: community (with LLM label) + member slug + type. annotate AdminService.KgCommunityMembers with @( UI.LineItem: [ - { Value: communityId, Label: 'Community ID' }, - { Value: communityFingerprint, Label: 'Fingerprint' }, - { Value: vertexType, Label: 'Type' } + { Value: communityId, Label: 'Community' }, + { Value: slug, Label: 'Member' }, + { Value: vertexType, Label: 'Type' } ] -); +) { + // Prefix the numeric id with the LLM-generated cluster label when present + // (#1126); falls back to the bare id where labeling hasn't run yet. + communityId @Common.Text: labelInfo.label @Common.TextArrangement: #TextFirst; +}; // FieldGroup for PageRank score — shown in KgFieldsFacet on Tutorials OP. // rank is a to-one Association (slug-joined) added in task-1. diff --git a/srv/admin-service.cds b/srv/admin-service.cds index d57df23c8..9e3bf132e 100644 --- a/srv/admin-service.cds +++ b/srv/admin-service.cds @@ -642,7 +642,14 @@ service AdminService { // Media facet (task-3): read-only projections of the object-store image and // asset metadata entities. The content Attachments composition auto-exposes // when the parent is reachable from an exposed entity (@cap-js/attachments). - @readonly @cds.redirection.target: false entity TutorialImages as projection on ims.TutorialImages; + @readonly @cds.redirection.target: false entity TutorialImages as projection on ims.TutorialImages { + *, + // Served-image URLs (filled in after('READ','TutorialImages')): the + // tutorial-system's own rendered output, not the GitHub source. + // thumbUrl → approuter resize/WebP CDN preview; viewUrl → raw original from CAP. + virtual thumbUrl : String, + virtual viewUrl : String + }; @readonly @cds.redirection.target: false entity TutorialAssets as projection on ims.TutorialAssets; // KG facet (task-1): read-only projections used by the Tutorials admin OP KG tab. @@ -1275,9 +1282,23 @@ extend service AdminService with { virtual null as coverageHigh : Boolean, }; + // Human-readable Louvain community label (#1126), keyed by + // communityFingerprint. Exposed read-only so the OP membership facet can + // surface the LLM-generated cluster name instead of the numeric communityId. + @readonly @cds.redirection.target: false + entity KgCommunityLabelInfo as projection on ims.KgCommunityLabel; + // OP-facing memberships. Rows keyed to (communityId, vertexKey). + // `labelInfo` is an unmanaged select-list association joining to + // KgCommunityLabelInfo on the shared communityFingerprint so the facet can + // show labelInfo.label (blank until the Louvain labeling job runs in the + // target env — member slug is the always-present fallback column). @readonly - entity KgCommunityMembers as projection on ims.KgCommunity; + entity KgCommunityMembers as projection on ims.KgCommunity { + *, + labelInfo : Association to KgCommunityLabelInfo + on labelInfo.communityFingerprint = communityFingerprint + }; // Drafts a Mission from the community's tutorial members, ordered A→Z. // Curator finishes the draft in the Missions LR (write description, diff --git a/srv/admin-service.js b/srv/admin-service.js index 8e5610965..107492368 100644 --- a/srv/admin-service.js +++ b/srv/admin-service.js @@ -2195,6 +2195,21 @@ export default class AdminService extends cds.ApplicationService { for (const row of list) if (row) row.confidenceCriticality = map[row.confidence] ?? 0; }); + // Media facet: surface the tutorial-system's own served-image URLs so the + // admin OP shows a live preview + a link to the rendered image (not just the + // GitHub source). thumbUrl → approuter resize/WebP CDN; viewUrl → raw + // original from CAP (self-heals from the object store on a miss). Both are + // root-relative so they resolve against the approuter origin serving /admin-ui/. + this.after('READ', 'TutorialImages', (rows) => { + const list = Array.isArray(rows) ? rows : [rows]; + for (const row of list) { + if (!row || !row.sourceUrl) continue; + const u = encodeURIComponent(row.sourceUrl); + row.thumbUrl = `/img-cdn?u=${u}`; + row.viewUrl = `/content/image-source?u=${u}`; + } + }); + // Guard: only SuperAdmin can change the published flag in either direction // (publish OR unpublish). The CREATE exemption permits the runtime's // draft-activation flow, where the activation payload echoes published=false diff --git a/test/unit/admin-facet-readability.test.js b/test/unit/admin-facet-readability.test.js new file mode 100644 index 000000000..cf2f9d64b --- /dev/null +++ b/test/unit/admin-facet-readability.test.js @@ -0,0 +1,108 @@ +// test/unit/admin-facet-readability.test.js +// +// Guards the three Tutorials Object-Page facet-readability fixes: +// 1. Concepts Taught — concept_ID FK shows the concept NAME (@Common.Text +// → concept/name, #TextOnly) instead of a raw GUID. +// 2. Community — communityId shows the LLM cluster LABEL when present +// (@Common.Text → labelInfo/label, #TextFirst), member slug as fallback. +// 3. Images — TutorialImages exposes virtual thumbUrl/viewUrl +// (the tutorial-system's own served-image URLs); thumbUrl is @UI.IsImageURL +// and the after('READ') handler fills both from sourceUrl. +// +// Two layers: CSN annotation guards (structural, so a future annotation edit +// can't silently revert), plus a served runtime check of the URL enrichment. +// +// Auth: username='admin', password='admin' (mocked-auth convention). + +import { describe, it, expect, beforeAll } from 'vitest'; +import cds from '@sap/cds'; + +const project = cds.test('serve', '--project', '.', '--in-memory'); +const ADMIN_AUTH = { auth: { username: 'admin', password: 'admin' } }; +const IMAGES = 'com.sap.developers.ims.TutorialImages'; + +describe('admin facet readability — CSN annotation guards', () => { + let csn; + beforeAll(async () => { + csn = await cds.load(['srv', 'app', 'db']); + }); + + it('TutorialConceptLinks.concept carries @Common.Text: concept.name (#TextOnly)', () => { + const el = csn.definitions['AdminService.TutorialConceptLinks']?.elements?.concept; + expect(el, 'concept association should exist').toBeTruthy(); + expect(el['@Common.Text']?.['=']).toBe('concept.name'); + expect(el['@Common.TextArrangement']?.['#']).toBe('TextOnly'); + }); + + it('KgCommunityMembers.communityId carries @Common.Text: labelInfo.label (#TextFirst)', () => { + const el = csn.definitions['AdminService.KgCommunityMembers']?.elements?.communityId; + expect(el, 'communityId should exist').toBeTruthy(); + expect(el['@Common.Text']?.['=']).toBe('labelInfo.label'); + expect(el['@Common.TextArrangement']?.['#']).toBe('TextFirst'); + }); + + it('KgCommunityMembers exposes the labelInfo association to KgCommunityLabelInfo', () => { + const el = csn.definitions['AdminService.KgCommunityMembers']?.elements?.labelInfo; + expect(el, 'labelInfo association should exist').toBeTruthy(); + expect(el.target).toBe('AdminService.KgCommunityLabelInfo'); + }); + + it('TutorialImages exposes virtual thumbUrl/viewUrl; thumbUrl is @UI.IsImageURL', () => { + const els = csn.definitions['AdminService.TutorialImages']?.elements ?? {}; + expect(els.thumbUrl?.virtual).toBe(true); + expect(els.viewUrl?.virtual).toBe(true); + expect(els.thumbUrl?.['@UI.IsImageURL']).toBe(true); + }); +}); + +describe('after(READ, TutorialImages) — served-image URL enrichment', () => { + const SRC = 'https://github.com/sap-tutorials/Tutorials/raw/master/tutorials/foo/img/a b.png'; + + beforeAll(async () => { + await cds.run( + INSERT.into(IMAGES).entries( + { + ID: cds.utils.uuid(), + slug: 'facet-readability-test', + channel: 'prod', + sourceUrl: SRC, + mimeType: 'image/png', + byteSize: 1234, + contentHash: 'abc', + }, + { + // Null sourceUrl → handler must skip enrichment (no crash, no URLs). + ID: cds.utils.uuid(), + slug: 'facet-readability-nosrc', + channel: 'prod', + sourceUrl: null, + } + ) + ); + }); + + it('fills thumbUrl (/img-cdn) and viewUrl (/content/image-source) from sourceUrl', async () => { + const res = await project.get( + `/admin/TutorialImages?$filter=slug eq 'facet-readability-test'`, + ADMIN_AUTH + ); + expect(res.status).toBe(200); + const row = res.data.value[0]; + expect(row).toBeTruthy(); + const enc = encodeURIComponent(SRC); + expect(row.thumbUrl).toBe(`/img-cdn?u=${enc}`); + expect(row.viewUrl).toBe(`/content/image-source?u=${enc}`); + }); + + it('leaves thumbUrl/viewUrl unset when sourceUrl is null (no crash)', async () => { + const res = await project.get( + `/admin/TutorialImages?$filter=slug eq 'facet-readability-nosrc'`, + ADMIN_AUTH + ); + expect(res.status).toBe(200); + const row = res.data.value[0]; + expect(row).toBeTruthy(); + expect(row.thumbUrl == null).toBe(true); + expect(row.viewUrl == null).toBe(true); + }); +}); From c1e519ff85e0329da93aae1386722783879d5139 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Tue, 1 Sep 2026 14:45:09 -0400 Subject: [PATCH 41/71] docs(topics): tag-tree topics design spec Rebuild /topics/ to concept-parity: tag-hierarchy tree nav, KG-concept enrichment, dynamic-slug CAP serve path, search-box fix. Draft for review. --- .../2026-09-01-tag-tree-topics-design.md | 154 ++++++++++++++++++ 1 file changed, 154 insertions(+) create mode 100644 docs/superpowers/specs/2026-09-01-tag-tree-topics-design.md diff --git a/docs/superpowers/specs/2026-09-01-tag-tree-topics-design.md b/docs/superpowers/specs/2026-09-01-tag-tree-topics-design.md new file mode 100644 index 000000000..067f7a17f --- /dev/null +++ b/docs/superpowers/specs/2026-09-01-tag-tree-topics-design.md @@ -0,0 +1,154 @@ +# Tag-Tree Topics — Design + +**Date:** 2026-09-01 +**Status:** Draft for review +**Issue/context:** `/topics/` is broken in production: every topic detail URL 404s, detail pages that render show no concepts, and the topics search box submits to a dead endpoint. This design rebuilds `/topics/` as a first-class subsystem with the same technical quality as `/concepts/`. + +--- + +## 1. Problem statement (why topics is broken today) + +Three independent defects, all confirmed against the live site and the repo: + +1. **Every topic detail page 404s.** The approuter has a full dynamic per-slug serve path for concepts (`^/concepts/(.*)$` → `/content/concepts/$1`, served from HANA and rendered by `srv/lib/concept-detail-render.js`), but for topics **only the index route exists** (`^/topics/?$` → `/content/pages/topics/`). There is **no `/topics/(.*)` route**, so `/topics//` — for any slug — falls through to a bare 404 (verified: the 404 carries no `x-content-source` header, i.e. it never reaches CAP). The Hugo-baked topic stubs are neither in the content-publish scope (`srv/lib/page-key-map.js` `IN_SCOPE_PAGES` lists only `page-topics`, the index) nor routed. + +2. **Detail pages show no concepts.** Topic detail (`hugo/layouts/topics/single.html`) reads `concepts[]` from the `/build/topics-gallery` payload (`srv/lib/build-topics-gallery.js`). That array is a `KgCommunity` (concept-vertex membership) → `Concepts.name` join. When the join returns nothing (empty/stale `KgCommunity` rows or slug-case mismatch), `concepts[]` is `[]` and the `{{ with $c.concepts }}` guard omits the whole section — while `memberCount`/`tutorialCount`, read straight off the `TopicClusters` row, still claim "N concepts". Counts and content come from different sources and disagree. + +3. **`/search/?q=` does nothing.** There is no `/search/` HTML page. `/search/` is the CAP `SearchService` **OData** endpoint. The topics search form (`hugo/layouts/topics/list.html:23`) does a full-page GET to `/search/?q=…`, landing on the OData service root, which ignores `?q=` (OData uses `$search=`) and renders no UI. The working search UI is the navigator island at `/tutorial-navigator/` (reads `?q=` via `hugo-apps/src/navigator/urlSync.ts`). + +Additional structural cause of link rot: the `/topics/` index currently links to ~501 slugs derived from **KG Louvain community labels**, which are **LLM-generated and non-deterministic** — they change on every KG rebuild, so previously valid `/topics//` links stop existing. This is orthogonal to the routing gap but compounds it. + +--- + +## 2. Goals & non-goals + +**Goals** +- Topic detail pages resolve (200) with the same dynamic-slug serve architecture as concepts. +- Topics are keyed by **stable, source-of-truth slugs** (SAP's central software tag hierarchy), so links do not rot. +- Detail pages show **real, deterministically-populated concepts** plus the topic's tutorials. +- `/topics/` presents a **tree navigation experience** over the tag hierarchy (facet → product → sub-product). +- The topics search box reaches the working search UI. +- Legacy AEM tag URLs resurrect **where the tag still has tutorials**; otherwise they redirect gracefully. + +**Non-goals (YAGNI / deferred)** +- Curated editorial "learning-path" topics with hand-ordered concepts (`orderConcepts`, `orderMode: path`). The existing `TopicClusters`/`topics_gallery` learning-path UX can layer on top later; it is not required for parity and is not carried forward as the primary model. +- KG Louvain communities as a public URL key (retained only for the homepage band, unchanged). +- Resurrecting legacy URLs for retired products that have no current tutorials (impossible from a current-catalog source of truth). +- A client-side-only tree (rejected for SEO/parity — see §5). + +--- + +## 3. Topic model + +A **topic = a live SAP tag** (a tag currently applied to ≥1 tutorial). Tags come from SAP's central software hierarchy and are the source of truth already consumed by this platform, so slugs are stable and externally meaningful. + +Tag data (`/build/tags`, cached to `hugo/data/tags.json`) is `group>value`, 8 facet groups, 143 live tags, with a deeper `--` sub-level: + +- Facets: `software-product` (90), `software-product-function` (16), `programming-tool` (12), `topic` (11), `tutorial` (8), `deprecated-concepts` (3), `operating-system` (2), `type` (1). +- Deep example: `software-product-function>sap-hana-cloud--data-lake` → `sap-hana-cloud` → `data-lake`. + +**Natural tree:** `facet → product → sub-product`. The hierarchy *is* the clustering — deterministic, no LLM. + +### Slug scheme +- Topic slug = the flattened leaf value: `sap-hana-cloud--data-lake` → `sap-hana-cloud-data-lake`; `>` and `--` both collapse to `-`. +- Matches the legacy AEM leaf-slug shape (`sap-hana-smart-data-streaming-development`), which is what enables legacy-URL resurrection. +- **Collision handling:** build-time check across all leaves. On collision, facet-qualify the loser (`-`). Expected rare; asserted in a unit test so a future tag import that introduces a collision fails loudly. + +--- + +## 4. Data model — CAP builders + +Two new builder endpoints, both fed by existing sources; both fail-open (return empty payload + `error` field, matching the `build-topics-gallery.js` posture). + +### 4.1 `/build/topics-tree` (index payload) +Builds the navigation tree: +``` +{ tree: [ { facet, label, children: [ { slug, label, tutorialCount, conceptCount, + children: [ …sub-product leaves… ] } ] } ], + buildAt, error } +``` +- Nodes/leaves from `/build/tags` (live tags only → every node has content). +- Human labels from `/build/tag-labels`. +- `tutorialCount` per tag from the existing tag→tutorial index. +- `conceptCount` per tag from the concept-enrichment join (§4.2), so the tree can show concept density. + +### 4.2 `/build/topics/:slug` (detail payload) +``` +{ slug, label, facet, rationale?, + tutorials: [ { slug, title, level, time, href, isNew } ], + concepts: [ { slug, name, rank } ], // deterministic — see below + relatedTags: [ { slug, label } ], // siblings + children in the tree + buildAt, error } +``` +- **Concepts populate deterministically** as: concepts linked (via existing `TutorialConceptLinks`) to the set of tutorials carrying this tag, deduped, ranked by `loadRankMaps()` conceptRank. This replaces the empty `KgCommunity` membership join that causes defect #2. Concepts link out to the existing `/concepts//` pages. +- `tutorials` from the tag→tutorial index (same source the navigator/facets use), lowercased slugs (`tutorialsTableInfo` helper) to avoid the known slug-case pitfall. +- `relatedTags` = tree siblings + children, for cross-navigation. + +### 4.3 Reuse note +`build-topics-gallery.js`, `TopicClusters`, and `topics_gallery.json` are **retired from the `/topics/` path** (the homepage `topic_clusters.json` band is untouched). We delete the two committed stub pages (`hugo/content/topics/btp-basics.md`, `cap-fundamentals.md`) and the `fetch-topics-gallery.ts` stub-generation. Any code deletion sweeps orphaned tests in the same commit. + +--- + +## 5. Tree navigation UI — server-rendered + progressive island + +Matches the concepts-page pattern (server-rendered HTML + a small enhancing island), for SEO and no-JS resilience. **Critically, the index is CAP-rendered, not Hugo-static** — the original topics breakage came from Hugo-baked pages that no route served. `/concepts/` is fully CAP-rendered (`srv/lib/concept-list-page.js` behind `/content/concepts-index`); topics matches that exactly. + +- A new **`srv/lib/topic-list-page.js`** (mirroring `concept-list-page.js`), served at `/content/topics-index`, renders the full `facet → product → sub-product` tree from the `/build/topics-tree` payload as **semantic nested `
    `** with native `
    /` disclosure — fully functional without JS, crawlable — wrapped in the shared chrome (`srv/lib/chrome-shell.js`). Published as a HANA content blob like the concepts index. +- A **small Vue island** (`hugo-apps/src/topics-tree/`) progressively enhances: type-ahead filter, expand/collapse-all, deep-link to an expanded node. Mount is inert until JS loads (same posture as `concepts-filter.js`). Built into the island manifest; the CAP-rendered HTML embeds the hashed island `'); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/unit/topic-list-page.test.js --project unit` +Expected: FAIL — module not found. + +- [ ] **Step 3: Write minimal implementation** + +```js +// srv/lib/topic-list-page.js +import { readFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { gzipSync } from 'node:zlib'; +import { createHash } from 'node:crypto'; +import cds from '@sap/cds'; +import { createShellLoader, ShellMarkerError, composeShell } from './chrome-shell.js'; +import { buildTopicsTreePayload } from './topics-query.js'; +import { setContentCacheHeaders } from './edge-cache-headers.js'; + +const _dir = dirname(fileURLToPath(import.meta.url)); +let _islandManifest; +function islandSrc(name) { + if (!_islandManifest) { + try { _islandManifest = JSON.parse(readFileSync(join(_dir, 'island-manifest.json'), 'utf8')); } + catch { _islandManifest = {}; } + } + return _islandManifest[name] ?? `/js/${name}.js`; +} + +const DEFAULT_NAMESPACE = 'com.sap.developers.ims'; +const HANA_TABLE = 'com_sap_developers_ims_ContentFiles'; +const HANA_CURRENT_TABLE = 'com_sap_developers_ims_ContentCurrent'; + +const TOPICS_STYLE = ``; + +function esc(s) { + return String(s).replace(/&/g, '&').replace(//g, '>').replace(/"/g, '"'); +} +function jsonForScript(obj) { + return JSON.stringify(obj).replace(/${node.tutorialCount ?? 0} tutorials · ${node.conceptCount ?? 0} concepts` + : ''; + const label = node.slug + ? `${esc(node.label)}${count}` + : `${esc(node.label)}`; + if (node.children && node.children.length) { + return `
  • ${label}
      ${node.children.map(renderNode).join('')}
    • `; + } + return `
  • ${label}
    • `; +} + +export function renderTopicListBody(model) { + const facets = (model.tree || []).map(f => + `
  • ${esc(f.label)}
      ${f.children.map(renderNode).join('')}
    • `, + ).join(''); + const data = jsonForScript({ tree: model.tree || [] }); + return `${TOPICS_STYLE} + + +`; +} + +export async function buildTopicListModel(db, _deps = {}) { + const payload = await buildTopicsTreePayload(db); + return { tree: payload.tree || [], version: null }; +} + +export function createTopicListPage({ namespace = DEFAULT_NAMESPACE } = {}) { + const NS = namespace; + const { ContentManifest } = cds.entities(NS); + async function getActiveVersion(db) { + const row = await db.run(SELECT.one.from(ContentManifest).columns('version').where({ status: 'ACTIVE' })).catch(() => null); + return row?.version ?? null; + } + const shellLoader = createShellLoader({ + namespace: NS, hanaTableName: HANA_TABLE, hanaCurrentTableName: HANA_CURRENT_TABLE, + getActiveVersion: () => cds.connect.to('db').then(getActiveVersion), + }); + let cache = null; // { version, gz, etag } + + async function topicsIndexHandler(req, res) { + try { + const db = await cds.connect.to('db'); + const version = await getActiveVersion(db); + if (cache && cache.version === version) { + if (req.headers['if-none-match'] === cache.etag) { res.status(304).end(); return; } + setContentCacheHeaders(res, { slug: 'topics' }); + res.set('Content-Encoding', 'gzip').set('ETag', cache.etag).set('X-Content-Source', 'db-current').type('html').send(cache.gz); + return; + } + const model = await buildTopicListModel(db); + model.version = version; + const body = renderTopicListBody(model); + const meta = { kind: 'topics-index', slug: 'topics', title: 'Explore topics', description: 'Browse SAP developer topics by product hierarchy.' }; + const shell = await shellLoader.get(); + if (!shell) throw new ShellMarkerError('shell unavailable'); + const html = composeShell(shell, body, meta); + const gz = gzipSync(Buffer.from(html, 'utf8')); + const etag = `"${createHash('sha256').update(gz).digest('hex').slice(0, 32)}"`; + cache = { version, gz, etag }; + if (req.headers['if-none-match'] === etag) { res.status(304).end(); return; } + setContentCacheHeaders(res, { slug: 'topics' }); + res.set('Content-Encoding', 'gzip').set('ETag', etag).set('X-Content-Source', 'db-current').type('html').send(gz); + } catch (err) { + if (cache) { + res.set('Content-Encoding', 'gzip').set('X-Content-Source', 'db-stale').type('html').send(cache.gz); + return; + } + res.status(503).type('text/plain').send('topics index unavailable'); + } + } + return { topicsIndexHandler, _invalidate() { cache = null; shellLoader.invalidate?.(); } }; +} + +export const { topicsIndexHandler } = createTopicListPage(); +export default topicsIndexHandler; +``` + +> **NOTE for implementer:** the cache/etag/shell-loader boilerplate above is transcribed from `concept-list-page.js:239-334`. **Diff your version against that source** and adopt any details you missed (exact `setContentCacheHeaders` args, `getActiveVersion` source table, `ContentManifest` query shape). Do NOT invent behavior the concepts handler doesn't have. + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/unit/topic-list-page.test.js --project unit` +Expected: PASS (2 cases). + +- [ ] **Step 5: Add chrome-shell cases + verify concepts untouched** + +Edit `srv/lib/chrome-shell.js`: in `canonicalUrlFor` (kind switch ~`:75-89`) add `case 'topics-index': return '/topics/';` and `case 'topic': return \`/topics/${meta.slug}/\`;`. Add the identical two cases to `buildBreadcrumbJsonLd` (~`:99-137`). + +- [ ] **Step 6: Commit** + +```bash +git add srv/lib/topic-list-page.js srv/lib/chrome-shell.js test/unit/topic-list-page.test.js +git commit -m "feat(topics): CAP-rendered tree index (topic-list-page) + chrome-shell topic canonical/breadcrumb" +``` + +--- + +## Task 5: CAP detail renderer (`srv/lib/topic-detail-render.js`) + +**Files:** +- Create: `srv/lib/topic-detail-render.js` +- Test: extend `test/unit/topics-query.test.js` (add a render block) or new `test/unit/topic-detail-render.test.js`. + +**Interfaces:** +- Consumes: topic detail payload shape from Task 2. +- Produces: `renderTopicDetail(topic): { body: string, contentHash: string }` — `body` supplies its own `
    ` (the shell has only the marker). `topic` = `{ slug, label, facet, tutorials, concepts, relatedTags }`. + +**Reference to mirror:** `srv/lib/concept-detail-render.js:51` (`renderConceptDetail` → `{body, contentHash}`, `body = \`
    ${…}
    \``, `contentHash = sha256(body)`, throws if key fields missing). + +- [ ] **Step 1: Write the failing test** + +```js +// test/unit/topic-detail-render.test.js +import { describe, it, expect } from 'vitest'; +import { renderTopicDetail } from '../../srv/lib/topic-detail-render.js'; + +describe('renderTopicDetail', () => { + const topic = { + slug: 'sap-hana-cloud', label: 'SAP HANA Cloud', facet: 'software-product', + tutorials: [{ slug: 'hana-intro', title: 'HANA Intro', level: 'Beginner', time: 15, href: '/tutorials/hana-intro/', isNew: true }], + concepts: [{ slug: 'in-memory-database', name: 'In-Memory Database', rank: 0.9 }], + relatedTags: [{ slug: 'sap-hana-cloud-data-lake', label: 'Data Lake' }], + }; + it('renders a
    body with breadcrumb, tutorials, concepts, related tags', () => { + const { body, contentHash } = renderTopicDetail(topic); + expect(body.startsWith('
    ')).toBe(true); + expect(body.endsWith('
    ')).toBe(true); + expect(body).toContain('SAP HANA Cloud'); + expect(body).toContain('href="/tutorials/hana-intro/"'); + expect(body).toContain('href="/concepts/in-memory-database/"'); + expect(body).toContain('href="/topics/sap-hana-cloud-data-lake/"'); + expect(contentHash).toMatch(/^[a-f0-9]{64}$/); + }); + it('throws when slug or label missing', () => { + expect(() => renderTopicDetail({ slug: '', label: 'x' })).toThrow(); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/unit/topic-detail-render.test.js --project unit` +Expected: FAIL — module not found. + +- [ ] **Step 3: Write minimal implementation** + +```js +// srv/lib/topic-detail-render.js +import { createHash } from 'node:crypto'; + +function esc(s) { + return String(s ?? '').replace(/&/g, '&').replace(//g, '>').replace(/"/g, '"'); +} + +export function renderTopicDetail(topic) { + if (!topic?.slug || !topic?.label) throw new Error('renderTopicDetail: slug and label required'); + const tutorials = (topic.tutorials || []).map(t => ` +
  • + ${esc(t.title)} + ${t.isNew ? 'NEW' : ''} + ${t.level ? `${esc(t.level)}` : ''} +
    • `).join(''); + const concepts = (topic.concepts || []).map(c => ` +
  • ${esc(c.name)}
    • `).join(''); + const related = (topic.relatedTags || []).map(r => ` +
  • ${esc(r.label)}
    • `).join(''); + + const conceptsSection = concepts + ? `
    +

    Concepts in this topic

    +
      ${concepts}
    +
    ` + : ''; + const relatedSection = related + ? `
    + +
      ${related}
    +
    ` + : ''; + + const body = `
    +
    + +
    +

    ${esc(topic.label)}

    +

    ${esc(topic.facet)}

    +
    +
    +

    Tutorials

    +
      ${tutorials || '
    • No tutorials yet.
      • '}
    +
    + ${conceptsSection} + ${relatedSection} +
    +
    `; + const contentHash = createHash('sha256').update(body).digest('hex'); + return { body, contentHash }; +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/unit/topic-detail-render.test.js --project unit` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add srv/lib/topic-detail-render.js test/unit/topic-detail-render.test.js +git commit -m "feat(topics): topic detail body renderer" +``` + +--- + +## Task 6: Blob publisher (`srv/lib/publish-topics.js`) + +**Files:** +- Create: `srv/lib/publish-topics.js` + +**Interfaces:** +- Consumes: `loadLiveTags`, `buildTopicDetailPayload` (Task 2); `renderTopicDetail` (Task 5); `composeShell` (chrome-shell). +- Produces: `renderTopicsIntoSession({ db, sessionId, helpers, priorHashes = {}, shell, deps = {} }): Promise<{ topicsSeen, topicsChanged, topicsSkipped, topicsErrored, durationMs }>`. Key = `topic-${slug}`. Meta = `{ kind: 'topic', slug, title: label, description }`. + +**Reference to mirror:** `srv/lib/publish-concepts.js:78-171` — per-item loop, `composeShell` wrap, `sha256(fullDoc)`, delta-skip on `priorHashes[key] === contentHash`, `gzipSync(fullDoc).toString('base64')`, `MAX_ERROR_RATE = 0.05` abort, `BATCH_SIZE = 20` append via `helpers.appendToSession`, `loadPriorTopicHashes` filtering `slug.startsWith('topic-')`. + +- [ ] **Step 1: Write the implementation** + +```js +// srv/lib/publish-topics.js +import { gzipSync } from 'node:zlib'; +import { createHash } from 'node:crypto'; +import { loadLiveTags, buildTopicDetailPayload } from './topics-query.js'; +import { renderTopicDetail } from './topic-detail-render.js'; +import { composeShell } from './chrome-shell.js'; + +const BATCH_SIZE = 20; +const MAX_ERROR_RATE = 0.05; +const META_DESC_MAX = 160; + +function topicMetaDescription(topic) { + const n = topic.tutorials?.length ?? 0; + const c = topic.concepts?.length ?? 0; + return `${topic.label}: ${n} tutorials and ${c} concepts on developers.sap.com.`.slice(0, META_DESC_MAX); +} + +export async function renderTopicsIntoSession({ db, sessionId, helpers, priorHashes = {}, shell, deps = {} }) { + const start = Date.now(); + const live = deps.loadLiveTags ? await deps.loadLiveTags(db) : await loadLiveTags(db); + let seen = 0, changed = 0, skipped = 0, errored = 0; + let batch = []; + const flush = async () => { + if (!batch.length) return; + await helpers.appendToSession({ sessionId, files: Object.fromEntries(batch) }); + batch = []; + }; + for (const tag of live) { + seen++; + try { + const topic = await buildTopicDetailPayload(db, tag.slug); + if (topic.notFound || topic.error) { errored++; continue; } + const { body } = renderTopicDetail(topic); + const meta = { kind: 'topic', slug: topic.slug, title: topic.label, description: topicMetaDescription(topic) }; + const fullDoc = composeShell(shell, body, meta); + const key = `topic-${topic.slug}`; + const contentHash = createHash('sha256').update(fullDoc).digest('hex'); + if (priorHashes[key] === contentHash) { skipped++; continue; } + batch.push([key, gzipSync(Buffer.from(fullDoc, 'utf8')).toString('base64')]); + changed++; + if (batch.length >= BATCH_SIZE) await flush(); + } catch { + errored++; + if (seen > 20 && errored / seen > MAX_ERROR_RATE) throw new Error('renderTopicsIntoSession: error rate exceeded'); + } + } + await flush(); + return { topicsSeen: seen, topicsChanged: changed, topicsSkipped: skipped, topicsErrored: errored, durationMs: Date.now() - start }; +} +``` + +> **NOTE for implementer:** wiring `renderTopicsIntoSession` into the actual publish orchestration (the POST `/content/publish/render-*` chain and `loadPriorConceptHashes` equivalent) mirrors `publish-concepts.js:159-194`. Locate where `renderConceptsIntoSession` is invoked during a publish run and register a sibling `renderTopicsIntoSession` call in the same place, loading prior hashes filtered by `slug.startsWith('topic-')`. Confirm the `shell`/`helpers`/`sessionId` objects passed to concepts are reused verbatim. + +- [ ] **Step 2: Commit** + +```bash +git add srv/lib/publish-topics.js +git commit -m "feat(topics): per-slug HANA blob publisher (renderTopicsIntoSession)" +``` + +--- + +## Task 7: Dynamic-slug discovery in publish (`page-key-map.js` + `publish-content.ts`) + +**Files:** +- Modify: `srv/lib/page-key-map.js` (add `TOPIC_KEY_PREFIX`, `isTopicKey`, `discoverTopicPages`) +- Modify: `scripts/publish-content.ts:~1058-1071` (call site) +- Test: extend `test/unit` page-key-map coverage if present; else add `test/unit/page-key-map-topics.test.js`. + +**Interfaces:** +- Produces: `TOPIC_KEY_PREFIX = 'topic-'`; `isTopicKey(key): boolean`; `discoverTopicPages(hugoDir): Map`. + +> **DESIGN NOTE:** topic detail blobs are produced server-side by `renderTopicsIntoSession` (Task 6), NOT from Hugo files on disk — there is no `hugo/.../topic-/index.html`. So `discoverTopicPages` is only relevant if topics are published as file-backed blobs. Since topics are **server-rendered at publish time**, the canonical publish path is the `renderTopicsIntoSession` server call (Task 6), invoked in the publish orchestration alongside `renderConceptsIntoSession`. **Confirm which mechanism the concepts subsystem actually uses at publish time** (recon shows concepts use the *server-side* `renderConceptsIntoSession`, not `discoverPageFiles`). If concepts are server-rendered, SKIP the `discoverTopicPages` file-walker entirely and rely solely on Task 6's server call — this task then reduces to registering that call. + +- [ ] **Step 1: Determine the actual publish mechanism** + +Read `scripts/publish-content.ts` around the concepts publish step and `srv/lib/publish-concepts.js:159-194`. Decide: +- **(A) Server-rendered** (expected): concepts blobs are generated by a POST to `/content/publish/render-concepts`. → Register a sibling topics render call; **do not** add `discoverTopicPages`. Skip Steps 2-4 below. +- **(B) File-backed**: concepts index/detail come from disk via `discoverPageFiles`. → Add `discoverTopicPages` (Steps 2-4). + +- [ ] **Step 2 (only if B): Add discovery helper** + +```js +// srv/lib/page-key-map.js — near AUTHOR/ADVOCATE prefixes +export const TOPIC_KEY_PREFIX = 'topic-'; +export const isTopicKey = (key) => key.startsWith(TOPIC_KEY_PREFIX); + +export function discoverTopicPages(hugoDir) { + const out = new Map(); + const base = path.join(hugoDir, 'topics'); + let entries = []; + try { entries = fs.readdirSync(base, { withFileTypes: true }); } catch { return out; } + for (const e of entries) { + if (!e.isDirectory()) continue; + const slug = e.name; + if (!/^[a-z0-9][a-z0-9-]*$/.test(slug)) continue; + const idx = path.join(base, slug, 'index.html'); + if (fs.existsSync(idx)) out.set(`${TOPIC_KEY_PREFIX}${slug}`, idx); + } + return out; +} +``` + +- [ ] **Step 3 (only if B): Wire the call site** in `scripts/publish-content.ts` next to the author/advocate discovery block: + +```ts +const topics = discoverTopicPages(opts.hugoDir); +for (const [key, absPath] of topics) tutorials.set(key, absPath); +``` + +- [ ] **Step 4 (only if B): Test + commit.** + +- [ ] **Step 5 (path A): Register the server render call** + +Add the `renderTopicsIntoSession` invocation to the publish orchestration alongside the concepts call (mirror `publish-concepts.js`'s `createRenderConcepts`/handler registration; expose `POST /content/publish/render-topics` in `srv/server.js` if concepts has an analogous route). Load prior hashes filtering `slug.startsWith('topic-')`. + +- [ ] **Step 6: Commit** + +```bash +git add srv/lib/page-key-map.js scripts/publish-content.ts srv/server.js +git commit -m "feat(topics): register topic-blob publish step in content publish path" +``` + +--- + +## Task 8: Register CAP routes (`srv/server.js`) + +**Files:** +- Modify: `srv/server.js` + +**Interfaces:** +- Consumes: `buildTopicsTreeHandler`, `buildTopicDetailHandler` (Task 3); `topicsIndexHandler`, `createTopicListPage` (Task 4); `buildTopicDetailPayload`, `resolveTopicBySlug` (Task 2); `renderTopicDetail` (Task 5); shell loader + serve helpers already used by `/content/concepts/:slug`. + +**Reference to mirror:** imports `server.js:15-37`; `/build/*` registrations `:296-306`; `/content/concepts/:slug` wrapper `:503-528`; `/content/concepts-index` `:533`. + +- [ ] **Step 1: Add imports** (top of `srv/server.js`, with sibling imports) + +```js +import { buildTopicsTreeHandler, buildTopicDetailHandler } from './lib/build-topics.js'; +import { topicsIndexHandler } from './lib/topic-list-page.js'; +import { resolveTopicBySlug, buildTopicDetailPayload } from './lib/topics-query.js'; +import { renderTopicDetail } from './lib/topic-detail-render.js'; +``` + +- [ ] **Step 2: Register build feeds** (next to `:298-300`) + +```js +app.get('/build/topics-tree', buildTopicsTreeHandler); +app.get('/build/topics/:slug', buildTopicDetailHandler); +``` + +- [ ] **Step 3: Register content index** (next to `:533`) + +```js +app.get('/content/topics-index', topicsIndexHandler); +``` + +- [ ] **Step 4: Register content detail** — serves the published `topic-` blob, with legacy/retired redirect fallthrough. Mirror the `/content/concepts/:slug` wrapper (`:503-528`) that rewrites `req.params.slug = \`concept-${lower}\`` then delegates to `serveHandler`. For topics, resolve legacy/retired slugs to a 301 before the blob lookup: + +```js +app.get('/content/topics/:slug', async (req, res, next) => { + let raw = String(req.params.slug || '').replace(/\.html$/, ''); + const lower = raw.toLowerCase(); + if (raw !== lower) { res.redirect(301, `/topics/${lower}/`); return; } + // legacy -N / retired resolution + try { + const db = await cds.connect.to('db'); + const { tag, redirectTo } = await resolveTopicBySlug(db, lower); + if (!tag && redirectTo) { res.redirect(301, redirectTo); return; } + if (tag && redirectTo) { res.redirect(301, redirectTo); return; } + } catch { /* fail-open to blob lookup */ } + // delegate to the shared blob serve handler with topic- key prefix (same as concepts) + req.params.slug = `topic-${lower}`; + return serveHandler(req, res, next); +}); +``` + +> **NOTE for implementer:** `serveHandler` is the same content-store serve function `/content/concepts/:slug` delegates to (`server.js:526-527`). Confirm its exact name/signature at that line and match it. If a served `topic-` blob is missing (never published), `serveHandler` will 404 — acceptable; the resolve step above catches *known-live-but-unpublished* only insofar as it doesn't redirect them. For a fully fail-open detail path, when `serveHandler` would 404 AND the slug resolves to a live tag, fall back to on-the-fly render: +> ```js +> // optional fallback inside a wrapper around serveHandler's 404 +> const payload = await buildTopicDetailPayload(db, lower); +> if (!payload.notFound) { /* renderTopicDetail + composeShell + send */ } +> ``` +> Decide based on whether concepts has an equivalent live-render fallback; prefer parity. + +- [ ] **Step 5: Verify build feeds live** + +```bash +cds watch & # or npm run dev:hybrid for real HANA +sleep 8 +curl -s http://localhost:4004/build/topics-tree | jq '.tree | length, .error' +curl -s http://localhost:4004/build/topics/sap-hana-cloud | jq '{tutorials: (.tutorials|length), concepts: (.concepts|length), error}' +``` +Expected: tree length ≥ 1, `error: null`; detail returns tutorials/concepts arrays (may be empty in a bare in-memory DB — verify against seeded/hybrid data). + +- [ ] **Step 6: Commit** + +```bash +git add srv/server.js +git commit -m "feat(topics): register /build/topics-tree, /build/topics/:slug, /content/topics-index, /content/topics/:slug" +``` + +--- + +## Task 9: Approuter routes + search redirect (`approuter/xs-app.json`) + +**Files:** +- Modify: `approuter/xs-app.json` + +**Reference:** concepts index route `:546-550`, detail route `:552-556`, `/build/*` allow-list `:399`, current `/topics/` route `:585`, `/search/` OData route `:375-376`, catch-all `:605`. + +- [ ] **Step 1: Add `topics-tree` to the `/build/*` allow-list alternation** (`:399`) — insert `|topics-tree` into the group (keep `topics-gallery` until Task 11 retires it, or replace it): + +```json +{ "source": "^/build/(breadcrumb-context|catalog|co-completions|concepts|homepage-shelves|kg-stats|mission|my-progress|navigator|repo-catalog|slug-mapping|tag-labels|topics-gallery|topics-tree|topics)(/.*)?(\\?.*)?$", "target": "/build/$1$2$3", "destination": "srv-api", "authenticationType": "none" } +``` + +> `topics` (bare) covers `/build/topics/:slug`; `topics-tree` covers the index feed. Ensure `topics` does not shadow `topics-tree` — regex alternation is ordered but both are matched as whole path segments by the `(/.*)?` boundary, so list `topics-tree` before `topics`. + +- [ ] **Step 2: Replace the `page-topics` index route** (`:585`) with the dynamic index route, and add the detail route immediately after. Both MUST precede the catch-all `^(.*)$` (`:605`): + +```json +{ "source": "^/topics/?(\\?.*)?$", "target": "/content/topics-index$1", "destination": "srv-api", "authenticationType": "none" }, +{ "source": "^/topics/(.*)$", "target": "/content/topics/$1", "destination": "srv-api", "authenticationType": "none" } +``` + +- [ ] **Step 3: Add the `/search/` → `/tutorial-navigator/` redirect, scoped to bare `/search/` + query only** so it does NOT shadow the OData `SearchService` routes (`^/search/(.*)$` at `:375-376`). Place this redirect BEFORE the OData `/search/` route: + +```json +{ "source": "^/search/?(\\?.*)?$", "target": "/tutorial-navigator/$1", "status": 301, "authenticationType": "none" } +``` + +> Verify `/search/SearchableItems` etc. still route to `srv-api` (the `^/search/(.*)$` route must remain and must be reached for non-empty paths). Test both after deploy. + +- [ ] **Step 4: Validate JSON** + +Run: `jq . approuter/xs-app.json > /dev/null && echo OK` +Expected: `OK` (no parse error). + +- [ ] **Step 5: Commit** + +```bash +git add approuter/xs-app.json +git commit -m "feat(topics): approuter dynamic /topics routes + /search 301 to navigator" +``` + +--- + +## Task 10: Progressive-enhancement island (`hugo-apps/src/topics-tree/`) + +**Files:** +- Create: `hugo-apps/src/topics-tree/main.ts`, `hugo-apps/src/topics-tree/App.vue` +- Modify: `hugo-apps/vite.config.ts:~316` (add entry) + +**Interfaces:** +- Consumes: server-embedded ` +`; +} + +/** + * Pure-ish data assembly for the topics list page. + * @param {object} db CDS db service + * @returns {Promise<{tree, version}>} version is null here; the handler stamps it. + */ +export async function buildTopicListModel(db, _deps = {}) { + const payload = await buildTopicsTreePayload(db); + return { tree: payload.tree || [], version: null }; +} + +/** + * Factory — builds an Express handler bound to a namespace, owning its own + * getActiveVersion + shellLoader + version-keyed gzip cache. + */ +export function createTopicListPage({ namespace = DEFAULT_NAMESPACE, deps = {} } = {}) { + const hanaTableName = () => `${namespace.replace(/\./g, '_').toUpperCase()}_CONTENTFILES`; + + async function getActiveVersion() { + const { ContentManifest } = cds.entities(namespace); + const [row] = await SELECT.from(ContentManifest) + .where({ status: 'ACTIVE' }) + .columns('version'); + return row?.version ?? null; + } + + const shellLoader = createShellLoader({ namespace, hanaTableName, getActiveVersion }); + + // Module-level version-keyed cache: { version, gzip, etag }. + let cache = null; + + function fallbackShellCompose(body, meta) { + const s = escapeHtml; + return `` + + `${s(meta.title)}` + + `` + + `` + + `
    ${body}
    `; + } + + async function topicsIndexHandler(req, res) { + const started = Date.now(); + try { + const db = await cds.connect.to('db'); + const version = await getActiveVersion(); + + // Cache hit — same active manifest version. + if (cache && cache.version === version) { + metrics.counter('topic_list_cache_hits'); + const ifNoneMatch = req.headers['if-none-match']; + if (ifNoneMatch && ifNoneMatch === cache.etag) return res.status(304).end(); + res.setHeader('Content-Type', 'text/html; charset=utf-8'); + res.setHeader('Content-Encoding', 'gzip'); + setContentCacheHeaders(res, { slug: 'topics' }); + res.setHeader('ETag', cache.etag); + res.setHeader('X-Content-Source', 'memcache'); + return res.status(200).send(cache.gzip); + } + metrics.counter('topic_list_cache_misses'); + + const model = await buildTopicListModel(db, deps); + model.version = version; + const body = renderTopicListBody(model); + const meta = { kind: 'topics-index', slug: 'topics', title: 'Topics', description: 'Browse SAP developer topics by product and technology hierarchy.' }; + + let html; + try { + const shell = await shellLoader.get(); + if (!shell) throw new ShellMarkerError('shell unavailable'); + html = composeShell(shell, body, meta); + } catch (err) { + console.warn('[content/topics-index] chrome shell missing — degraded rendering:', err.message); + html = fallbackShellCompose(body, meta); + } + + const gzip = gzipSync(Buffer.from(html, 'utf-8')); + const etag = `"${version ?? 'none'}"`; + cache = { version, gzip, etag }; + + metrics.observe('topic_list_render_ms', Date.now() - started); + const ifNoneMatch = req.headers['if-none-match']; + if (ifNoneMatch && ifNoneMatch === etag) return res.status(304).end(); + res.setHeader('Content-Type', 'text/html; charset=utf-8'); + res.setHeader('Content-Encoding', 'gzip'); + setContentCacheHeaders(res, { slug: 'topics' }); + res.setHeader('ETag', etag); + res.setHeader('X-Content-Source', 'fresh'); + return res.status(200).send(gzip); + } catch (err) { + metrics.counter('topic_list_query_failure'); + console.error('[content/topics-index] build failed:', err?.message); + // Serve last-known-good if we have any, even across versions. + if (cache) { + res.setHeader('Content-Type', 'text/html; charset=utf-8'); + res.setHeader('Content-Encoding', 'gzip'); + res.setHeader('Cache-Control', 'public, max-age=60'); + res.setHeader('ETag', cache.etag); + res.setHeader('X-Content-Source', 'stale'); + return res.status(200).send(cache.gzip); + } + res.setHeader('Content-Type', 'text/html; charset=utf-8'); + return res.status(503).send('

    Topics temporarily unavailable

    Please try again shortly.

    '); + } + } + + return { topicsIndexHandler, _invalidate() { cache = null; shellLoader.invalidate?.(); } }; +} + +const _default = createTopicListPage(); +export const topicsIndexHandler = _default.topicsIndexHandler; +export default topicsIndexHandler; diff --git a/test/unit/topic-list-page.test.js b/test/unit/topic-list-page.test.js new file mode 100644 index 000000000..ba066d6d6 --- /dev/null +++ b/test/unit/topic-list-page.test.js @@ -0,0 +1,32 @@ +// test/unit/topic-list-page.test.js +import { describe, it, expect } from 'vitest'; +import { renderTopicListBody } from '../../srv/lib/topic-list-page.js'; + +describe('renderTopicListBody', () => { + const model = { + version: 'v1', + tree: [{ + facet: 'software-product', label: 'Software Product', + children: [ + { segment: 'sap-hana-cloud', slug: 'sap-hana-cloud', label: 'SAP HANA Cloud', tutorialCount: 3, conceptCount: 5, children: [ + { segment: 'data-lake', slug: 'sap-hana-cloud-data-lake', label: 'Data Lake', tutorialCount: 1, conceptCount: 2, children: [] }, + ] }, + ], + }], + }; + it('renders nested details/ul with topic links and no-JS disclosure', () => { + const html = renderTopicListBody(model); + expect(html).toContain(' { + const html = renderTopicListBody(model); + expect(html).toContain('id="topics-tree-data"'); + expect(html).toMatch(/'); + }); +}); From 52139bad344144c92bca70d2e7e864b05206a875 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Tue, 1 Sep 2026 15:42:40 -0400 Subject: [PATCH 50/71] feat(topics): topic detail body renderer --- srv/lib/topic-detail-render.js | 56 +++++++++++++++++++++++++++ test/unit/topic-detail-render.test.js | 24 ++++++++++++ 2 files changed, 80 insertions(+) create mode 100644 srv/lib/topic-detail-render.js create mode 100644 test/unit/topic-detail-render.test.js diff --git a/srv/lib/topic-detail-render.js b/srv/lib/topic-detail-render.js new file mode 100644 index 000000000..48b388a3b --- /dev/null +++ b/srv/lib/topic-detail-render.js @@ -0,0 +1,56 @@ +import { createHash } from 'node:crypto'; + +function esc(s) { + return String(s ?? '').replace(/&/g, '&').replace(//g, '>').replace(/"/g, '"'); +} + +export function renderTopicDetail(topic) { + if (!topic?.slug || !topic?.label) throw new Error('renderTopicDetail: slug and label required'); + const tutorials = (topic.tutorials || []).map(t => ` +
  • + ${esc(t.title)} + ${t.isNew ? 'NEW' : ''} + ${t.level ? `${esc(t.level)}` : ''} +
    • `).join(''); + const concepts = (topic.concepts || []).map(c => ` +
  • ${esc(c.name)}
    • `).join(''); + const related = (topic.relatedTags || []).map(r => ` +
  • ${esc(r.label)}
    • `).join(''); + + const conceptsSection = concepts + ? `
    +

    Concepts in this topic

    +
      ${concepts}
    +
    ` + : ''; + const relatedSection = related + ? `
    + +
      ${related}
    +
    ` + : ''; + + const body = `
    +
    + +
    +

    ${esc(topic.label)}

    +

    ${esc(topic.facet)}

    +
    +
    +

    Tutorials

    +
      ${tutorials || '
    • No tutorials yet.
      • '}
    +
    + ${conceptsSection} + ${relatedSection} +
    +
    `; + const contentHash = createHash('sha256').update(body).digest('hex'); + return { body, contentHash }; +} diff --git a/test/unit/topic-detail-render.test.js b/test/unit/topic-detail-render.test.js new file mode 100644 index 000000000..091c30f72 --- /dev/null +++ b/test/unit/topic-detail-render.test.js @@ -0,0 +1,24 @@ +import { describe, it, expect } from 'vitest'; +import { renderTopicDetail } from '../../srv/lib/topic-detail-render.js'; + +describe('renderTopicDetail', () => { + const topic = { + slug: 'sap-hana-cloud', label: 'SAP HANA Cloud', facet: 'software-product', + tutorials: [{ slug: 'hana-intro', title: 'HANA Intro', level: 'Beginner', time: 15, href: '/tutorials/hana-intro/', isNew: true }], + concepts: [{ slug: 'in-memory-database', name: 'In-Memory Database', rank: 0.9 }], + relatedTags: [{ slug: 'sap-hana-cloud-data-lake', label: 'Data Lake' }], + }; + it('renders a
    body with breadcrumb, tutorials, concepts, related tags', () => { + const { body, contentHash } = renderTopicDetail(topic); + expect(body.startsWith('
    ')).toBe(true); + expect(body.endsWith('
    ')).toBe(true); + expect(body).toContain('SAP HANA Cloud'); + expect(body).toContain('href="/tutorials/hana-intro/"'); + expect(body).toContain('href="/concepts/in-memory-database/"'); + expect(body).toContain('href="/topics/sap-hana-cloud-data-lake/"'); + expect(contentHash).toMatch(/^[a-f0-9]{64}$/); + }); + it('throws when slug or label missing', () => { + expect(() => renderTopicDetail({ slug: '', label: 'x' })).toThrow(); + }); +}); From 92b5b4ef7191ab861798336d4ee6a3003e0e7694 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Tue, 1 Sep 2026 15:45:41 -0400 Subject: [PATCH 51/71] fix(topics): esc() single-quote + explicit utf-8 hash encoding for concept-detail-render parity --- srv/lib/topic-detail-render.js | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/srv/lib/topic-detail-render.js b/srv/lib/topic-detail-render.js index 48b388a3b..59216a3ef 100644 --- a/srv/lib/topic-detail-render.js +++ b/srv/lib/topic-detail-render.js @@ -1,7 +1,7 @@ import { createHash } from 'node:crypto'; function esc(s) { - return String(s ?? '').replace(/&/g, '&').replace(//g, '>').replace(/"/g, '"'); + return String(s ?? '').replace(/&/g, '&').replace(//g, '>').replace(/"/g, '"').replace(/'/g, '''); } export function renderTopicDetail(topic) { @@ -51,6 +51,6 @@ export function renderTopicDetail(topic) { ${relatedSection}

`; - const contentHash = createHash('sha256').update(body).digest('hex'); + const contentHash = createHash('sha256').update(body, 'utf-8').digest('hex'); return { body, contentHash }; } From 831d4565a62b0e37f59f80f15274e5a26b4ce873 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Tue, 1 Sep 2026 15:48:29 -0400 Subject: [PATCH 52/71] feat(topics): per-slug HANA blob publisher (renderTopicsIntoSession) --- srv/lib/publish-topics.js | 168 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 168 insertions(+) create mode 100644 srv/lib/publish-topics.js diff --git a/srv/lib/publish-topics.js b/srv/lib/publish-topics.js new file mode 100644 index 000000000..7d2ea4952 --- /dev/null +++ b/srv/lib/publish-topics.js @@ -0,0 +1,168 @@ +// srv/lib/publish-topics.js +// +// Task 6 of the tag-tree-topics plan. A session-scoped publish phase that +// renders every live topic tag and appends `topic-` BLOBs to an open +// publish session. Each BLOB is a FULL HTML document: the topic BODY +// (renderTopicDetail) composed into the __shell__ chrome via composeShell, +// mirroring the same pattern used by publish-concepts.js for concept pages. +// +// Dark launch: no publish-content.ts caller yet (Task 7 wires it). + +import cds from '@sap/cds'; +import { gzipSync } from 'node:zlib'; +import { createHash } from 'node:crypto'; +import { loadLiveTags, buildTopicDetailPayload } from './topics-query.js'; +import { renderTopicDetail } from './topic-detail-render.js'; +import { composeShell, createShellLoader } from './chrome-shell.js'; +import { createSessionHelpers } from './content-publish-session.js'; +import * as metrics from './metrics.js'; + +const DEFAULT_NAMESPACE = 'com.sap.developers.ims'; +const BATCH_SIZE = 20; +const MAX_ERROR_RATE = 0.05; // >5% of topics erroring aborts the phase + +const META_DESC_MAX = 160; // Google truncates around 155–160 chars. + +export function topicMetaDescription(topic) { + const n = topic.tutorials?.length ?? 0; + const c = topic.concepts?.length ?? 0; + return `${topic.label}: ${n} tutorial${n === 1 ? '' : 's'} and ${c} concept${c === 1 ? '' : 's'} on developers.sap.com.`.slice(0, META_DESC_MAX); +} + +/** + * Render every live topic into an open publish session. + * + * @param {object} args + * @param {object} args.db CDS db service (passed to loadLiveTags / buildTopicDetailPayload). + * @param {string} args.sessionId Open publish session id. + * @param {object} args.helpers { appendToSession } (from createSessionHelpers). + * @param {object} args.priorHashes Map of `topic-` → prior stored + * contentHash (full-doc sha256) for delta skip. + * @param {{before:string,after:string}} args.shell Parsed __shell__ halves. + * @param {object} [args.deps] { loadLiveTags, buildTopicDetailPayload } — defaults to real. + * @returns {Promise<{topicsSeen,topicsChanged,topicsSkipped,topicsErrored,durationMs}>} + */ +export async function renderTopicsIntoSession({ db, sessionId, helpers, priorHashes = {}, shell, deps = {} }) { + const started = Date.now(); + if (!shell || typeof shell.before !== 'string' || typeof shell.after !== 'string') { + throw new Error('render-topics: shell unavailable — __shell__ sidecar not yet published'); + } + const _loadLiveTags = deps.loadLiveTags || loadLiveTags; + const _buildTopicDetailPayload = deps.buildTopicDetailPayload || buildTopicDetailPayload; + const live = await _loadLiveTags(db); + const topicsSeen = live.length; + + // Render + delta-filter into a flat file map first, so we can enforce the + // error threshold before writing anything. + const changedFiles = {}; // key → base64(gzip(full doc)) + let topicsSkipped = 0; + let topicsErrored = 0; + + for (const tag of live) { + const key = `topic-${tag.slug}`; + try { + const topic = await _buildTopicDetailPayload(db, tag.slug); + if (topic.notFound || topic.error) { + topicsErrored++; + metrics.counter('topic_render_error'); + console.error(`[render-topics] topic "${tag.slug}" payload returned error/notFound`); + continue; + } + const { body } = renderTopicDetail(topic); + const meta = { + kind: 'topic', + slug: topic.slug, + title: topic.label, + description: topicMetaDescription(topic), + }; + const fullDoc = composeShell(shell, body, meta); + const contentHash = createHash('sha256').update(fullDoc, 'utf-8').digest('hex'); + if (priorHashes[key] === contentHash) { + topicsSkipped++; + continue; + } + changedFiles[key] = gzipSync(Buffer.from(fullDoc, 'utf-8')).toString('base64'); + } catch (err) { + topicsErrored++; + metrics.counter('topic_render_error'); + console.error(`[render-topics] topic "${tag.slug}" render failed — carrying forward prior BLOB: ${err.message}`); + } + } + + // Corrupt-run guard: if a large fraction of topics errored, abort so the + // caller can roll the session back rather than commit a degraded corpus. + if (topicsSeen > 0 && topicsErrored / topicsSeen > MAX_ERROR_RATE) { + throw new Error(`render-topics: error rate too high (${topicsErrored}/${topicsSeen}) — aborting phase`); + } + + // Append in batches of BATCH_SIZE. + const keys = Object.keys(changedFiles); + for (let i = 0; i < keys.length; i += BATCH_SIZE) { + const slice = keys.slice(i, i + BATCH_SIZE); + const files = {}; + for (const k of slice) files[k] = changedFiles[k]; + await helpers.appendToSession({ sessionId, files }); + } + + const durationMs = Date.now() - started; + const topicsChanged = keys.length; + metrics.observe('topic_render_ms', durationMs); + metrics.counter('topics_rendered_total', topicsChanged); + metrics.counter('topics_skipped_total', topicsSkipped); + return { topicsSeen, topicsChanged, topicsSkipped, topicsErrored, durationMs }; +} + +/** + * Express handler factory for POST /content/publish/render-topics. Bound to + * a namespace; owns getActiveVersion + shellLoader + prior-hash lookup. + */ +export function createRenderTopics({ namespace = DEFAULT_NAMESPACE } = {}) { + const hanaTableName = () => `${namespace.replace(/\./g, '_').toUpperCase()}_CONTENTFILES`; + + async function getActiveVersion() { + const { ContentManifest } = cds.entities(namespace); + const [row] = await SELECT.from(ContentManifest).where({ status: 'ACTIVE' }).columns('version'); + return row?.version ?? null; + } + + const shellLoader = createShellLoader({ namespace, hanaTableName, getActiveVersion }); + const helpers = createSessionHelpers({ namespace }); + + // Prior stored hashes for topic-* slugs at the current ACTIVE version, for + // delta skip. Same source as hashesHandler. + async function loadPriorTopicHashes() { + const activeVersion = await getActiveVersion(); + if (activeVersion === null) return {}; + const { ContentFiles } = cds.entities(namespace); + const rows = await SELECT.from(ContentFiles) + .where({ version: activeVersion }) + .columns('slug', 'contentHash'); + const out = {}; + for (const r of rows) { + if (typeof r.slug === 'string' && r.slug.startsWith('topic-')) out[r.slug] = r.contentHash; + } + return out; + } + + async function renderTopicsHandler(req, res) { + const sessionId = req.body?.sessionId; + if (!sessionId) return res.status(400).json({ error: 'sessionId is required' }); + try { + const db = await cds.connect.to('db'); + const shell = await shellLoader.get(); // { before, after, version } | null + const priorHashes = await loadPriorTopicHashes(); + const counts = await renderTopicsIntoSession({ db, sessionId, helpers, priorHashes, shell }); + return res.status(200).json(counts); + } catch (err) { + console.error('[render-topics] phase failed:', err?.message); + metrics.counter('topic_render_batch_failure'); + // Leave the session for the caller to abort — do not commit here. + return res.status(500).json({ error: err?.message || 'render-topics failed' }); + } + } + + return { renderTopicsHandler }; +} + +const _default = createRenderTopics(); +export const renderTopicsHandler = _default.renderTopicsHandler; From 4c506c692af9e99218fde5d96935c804b8d8fdec Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Tue, 1 Sep 2026 15:56:08 -0400 Subject: [PATCH 53/71] feat(topics): register topic-blob publish step in content publish path --- scripts/lib/publish-client.ts | 20 ++++++++++++++++++++ scripts/publish-content.ts | 21 ++++++++++++++++++++- srv/server.js | 4 ++++ 3 files changed, 44 insertions(+), 1 deletion(-) diff --git a/scripts/lib/publish-client.ts b/scripts/lib/publish-client.ts index 121e60855..a1e5e053e 100644 --- a/scripts/lib/publish-client.ts +++ b/scripts/lib/publish-client.ts @@ -100,6 +100,26 @@ export async function renderConceptsPhase(i: RenderConceptsInput): Promise` BLOBs to the session (delta skips unchanged). + * Mirrors renderConceptsPhase exactly: must run AFTER the __shell__ sidecar is + * present and BEFORE commit. + */ +export async function renderTopicsPhase(i: RenderTopicsInput): Promise { + return postJson(`${i.baseUrl}/content/publish/render-topics`, i.apiKey, { sessionId: i.sessionId }); +} + export async function abortSession({ baseUrl, apiKey, sessionId, reason }: { baseUrl: string; apiKey: string; sessionId: string; reason?: string; }): Promise<{ aborted: boolean }> { diff --git a/scripts/publish-content.ts b/scripts/publish-content.ts index 9e4e520f0..e1bd2535e 100644 --- a/scripts/publish-content.ts +++ b/scripts/publish-content.ts @@ -5,7 +5,7 @@ import { gzipSync } from 'node:zlib'; import { userInfo, hostname } from 'node:os'; import { parse as parseYaml } from 'yaml'; import { parseChannel, type Channel } from './fetch-tutorials.js'; -import { beginSession, appendBatch, commitSession, abortSession, fetchRemoteHashes, fetchRemoteSourceHashes, renderConceptsPhase } from './lib/publish-client.js'; +import { beginSession, appendBatch, commitSession, abortSession, fetchRemoteHashes, fetchRemoteSourceHashes, renderConceptsPhase, renderTopicsPhase } from './lib/publish-client.js'; import { withRetry, formatErrorChain } from './lib/publish-retry.js'; import { chunk, runConcurrent } from './lib/publish-batcher.js'; import { collectCodeCheckSpecs, publishCodeCheckSpecs } from './lib/publish-codecheck.js'; @@ -1245,6 +1245,25 @@ async function main() { await abortSession({ baseUrl: opts.baseUrl, apiKey: opts.apiKey, sessionId: begin.sessionId, reason: 'render-concepts failed' }); process.exit(1); } + // tag-tree-topics Task 7 — render topic detail pages alongside concepts. + // Same guard (prod-only, full-publish): POST /content/publish/render-topics + // is not registered on srv-qa (topics are a prod-only content surface). + try { + const rt = await withRetry( + () => renderTopicsPhase({ baseUrl: opts.baseUrl, apiKey: opts.apiKey, sessionId: begin.sessionId }), + { + attempts: 3, backoffMs: [1000, 3000, 9000], + onAttemptFail: (attempt, err, willRetry) => { + console.error(`[publish-content] render-topics failed (attempt ${attempt}/3): ${formatErrorChain(err)}${willRetry ? ' — retrying' : ''}`); + }, + } + ); + log(`render-topics: ${rt.topicsChanged} changed, ${rt.topicsSkipped} skipped, ${rt.topicsErrored} errored of ${rt.topicsSeen} (${rt.durationMs} ms)`); + } catch (err) { + console.error(`[publish-content] render-topics failed permanently: ${formatErrorChain(err)}`); + await abortSession({ baseUrl: opts.baseUrl, apiKey: opts.apiKey, sessionId: begin.sessionId, reason: 'render-topics failed' }); + process.exit(1); + } } let commit; diff --git a/srv/server.js b/srv/server.js index 1835c59b6..ab0705d9d 100644 --- a/srv/server.js +++ b/srv/server.js @@ -37,6 +37,7 @@ import { bumpCacheGeneration } from './lib/content-cache-coherence.js'; import { conceptsIndexHandler } from './lib/concept-list-page.js'; import { puzzlePageHandler, puzzleIndexHandler } from './lib/puzzle-page.js'; import { renderConceptsHandler } from './lib/publish-concepts.js'; +import { renderTopicsHandler } from './lib/publish-topics.js'; import { repoCatalogReadHandler, repoCatalogWriteHandler } from './lib/repo-catalog.js'; import { modelJsonHandler } from './lib/model-json-handler.js'; import { kgStatsHandler } from './routes/kg-stats.js'; @@ -573,6 +574,9 @@ cds.on('bootstrap', (app) => { // (Thread B). Dark launch: no publish-content.ts caller yet (Task 5 wires // it). Auth like the other publish routes. app.post('/content/publish/render-concepts', express.json({ limit: '1mb' }), contentAuthMiddleware, renderConceptsHandler); + // tag-tree-topics Task 7 — render topic detail pages alongside concepts. + // Auth and sequencing mirror render-concepts exactly. + app.post('/content/publish/render-topics', express.json({ limit: '1mb' }), contentAuthMiddleware, renderTopicsHandler); app.post('/content/publish/commit', express.json({ limit: '1mb' }), contentAuthMiddleware, commitHandler); app.post('/content/publish/abort', express.json({ limit: '1mb' }), contentAuthMiddleware, abortHandler); From b2d2a808dcfaa622cab90cc86137411e1cf04951 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Tue, 1 Sep 2026 16:09:27 -0400 Subject: [PATCH 54/71] feat(topics): register /build/topics-tree, /build/topics/:slug, /content/topics-index, /content/topics/:slug --- srv/server.js | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/srv/server.js b/srv/server.js index ab0705d9d..81520d364 100644 --- a/srv/server.js +++ b/srv/server.js @@ -38,6 +38,9 @@ import { conceptsIndexHandler } from './lib/concept-list-page.js'; import { puzzlePageHandler, puzzleIndexHandler } from './lib/puzzle-page.js'; import { renderConceptsHandler } from './lib/publish-concepts.js'; import { renderTopicsHandler } from './lib/publish-topics.js'; +import { buildTopicsTreeHandler, buildTopicDetailHandler } from './lib/build-topics.js'; +import { topicsIndexHandler } from './lib/topic-list-page.js'; +import { resolveTopicBySlug } from './lib/topics-query.js'; import { repoCatalogReadHandler, repoCatalogWriteHandler } from './lib/repo-catalog.js'; import { modelJsonHandler } from './lib/model-json-handler.js'; import { kgStatsHandler } from './routes/kg-stats.js'; @@ -299,6 +302,8 @@ cds.on('bootstrap', (app) => { app.get('/build/concepts', buildConceptsHandler); app.get('/build/topic-clusters', buildTopicClustersHandler); app.get('/build/topics-gallery', buildTopicsGalleryHandler); + app.get('/build/topics-tree', buildTopicsTreeHandler); + app.get('/build/topics/:slug', buildTopicDetailHandler); app.get('/graph/explore-data', exploreDataHandler); app.get('/graph/clusters-data', clustersDataHandler); app.get('/graph/path', graphPathHandler); @@ -532,6 +537,35 @@ cds.on('bootstrap', (app) => { // here yet (the /concepts/?$ flip lands in Task 5). Public, no auth — like // serveHandler. app.get('/content/concepts-index', conceptsIndexHandler); + // #topics-scale Task 8 — CAP-served /topics/ detail pages (published topic- + // BLOBs) and the /topics/ index page. Mirrors the /content/concepts/:slug + + // /content/concepts-index pattern exactly: same serveHandler delegation, same + // two-arg call signature, no live-render fallback (concepts has none). + // + // Topics adds a legacy/retired slug resolution step: resolveTopicBySlug checks + // whether the slug is a renamed/retired tag and returns a redirectTo URL when it + // is. Both !tag+redirectTo and tag+redirectTo cases 301 — the redirect takes + // precedence whenever it is set, matching the brief spec. + app.get('/content/topics/:slug', async (req, res) => { + const raw = String(req.params.slug || '').replace(/\.html$/, ''); + const lower = raw.toLowerCase(); + // Mixed-case or .html-stripped → 301 to canonical lowercase /topics/ + if (raw !== lower) { + res.redirect(301, `/topics/${lower}/`); + return; + } + // Legacy / retired slug resolution → 301 to canonical replacement + try { + const db = await cds.connect.to('db'); + const { redirectTo } = await resolveTopicBySlug(db, lower); + if (redirectTo) { res.redirect(301, redirectTo); return; } + } catch { /* fail-open to blob lookup */ } + // Delegate to shared blob serve handler with topic- key prefix (same as concepts) + req.params.slug = `topic-${lower}`; + return serveHandler(req, res); + }); + // Topics index page: /topics/ → CAP SSR (mirrors /content/concepts-index). + app.get('/content/topics-index', topicsIndexHandler); // #1914 — CAP-served puzzle solver pages (/puzzles//, dynamic slug). // The page is a thin island shell composed into the __shell__ chrome; the // `puzzle` island fetches grid/clue data from /puzzle-api at runtime. Serving From 098ff317833c94c8542915e3c5d7617aa181847d Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Tue, 1 Sep 2026 16:14:35 -0400 Subject: [PATCH 55/71] =?UTF-8?q?fix(topics):=20mirror=20concepts=20redire?= =?UTF-8?q?ct=20block=20in=20/content/topics/:slug=20=E2=80=94=20preserve?= =?UTF-8?q?=20query,=20cache-control,=20slug=20validation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- srv/server.js | 34 +++++++++++++++++++++++++++------- 1 file changed, 27 insertions(+), 7 deletions(-) diff --git a/srv/server.js b/srv/server.js index 81520d364..48b505005 100644 --- a/srv/server.js +++ b/srv/server.js @@ -547,20 +547,40 @@ cds.on('bootstrap', (app) => { // is. Both !tag+redirectTo and tag+redirectTo cases 301 — the redirect takes // precedence whenever it is set, matching the brief spec. app.get('/content/topics/:slug', async (req, res) => { - const raw = String(req.params.slug || '').replace(/\.html$/, ''); + const raw = String(req.params.slug || ''); + // Strip .html suffix → 301 to canonical /topics/ + if (/\.html$/i.test(raw)) { + const stripped = raw.replace(/\.html$/i, '').toLowerCase(); + if (/^[a-z0-9][a-z0-9-]*$/.test(stripped)) { + const qIdx = req.url.indexOf('?'); + const query = qIdx >= 0 ? req.url.slice(qIdx) : ''; + res.setHeader('Location', `/topics/${stripped}${query}`); + res.setHeader('Cache-Control', 'public, max-age=3600'); + return res.status(301).end(); + } + } + // Mixed-case → 301 to canonical lowercase /topics/ const lower = raw.toLowerCase(); - // Mixed-case or .html-stripped → 301 to canonical lowercase /topics/ - if (raw !== lower) { - res.redirect(301, `/topics/${lower}/`); - return; + if (raw && raw !== lower && /^[a-z0-9][a-z0-9-]*$/.test(lower)) { + const qIdx = req.url.indexOf('?'); + const query = qIdx >= 0 ? req.url.slice(qIdx) : ''; + res.setHeader('Location', `/topics/${lower}${query}`); + res.setHeader('Cache-Control', 'public, max-age=3600'); + return res.status(301).end(); } // Legacy / retired slug resolution → 301 to canonical replacement try { const db = await cds.connect.to('db'); const { redirectTo } = await resolveTopicBySlug(db, lower); - if (redirectTo) { res.redirect(301, redirectTo); return; } + if (redirectTo) { + const qIdx = req.url.indexOf('?'); + const query = qIdx >= 0 ? req.url.slice(qIdx) : ''; + res.setHeader('Location', `${redirectTo}${query}`); + res.setHeader('Cache-Control', 'public, max-age=3600'); + return res.status(301).end(); + } } catch { /* fail-open to blob lookup */ } - // Delegate to shared blob serve handler with topic- key prefix (same as concepts) + // Canonical form — delegate to serveHandler with the topic- prefix. req.params.slug = `topic-${lower}`; return serveHandler(req, res); }); From 025a45dd514dd845a79934620154035a77e79466 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Tue, 1 Sep 2026 16:20:06 -0400 Subject: [PATCH 56/71] feat(topics): approuter dynamic /topics routes + /search 301 to navigator --- approuter/xs-app.json | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/approuter/xs-app.json b/approuter/xs-app.json index 0ed2cb08a..c1128a70a 100644 --- a/approuter/xs-app.json +++ b/approuter/xs-app.json @@ -371,6 +371,7 @@ "destination": "srv-api", "authenticationType": "xsuaa" }, + { "source": "^/search/?(\\?.*)?$", "target": "/tutorial-navigator/$1", "status": 301, "authenticationType": "none" }, { "source": "^/search/(.*)$", "target": "/search/$1", @@ -396,7 +397,7 @@ "authenticationType": "none" }, { - "source": "^/build/(breadcrumb-context|catalog|co-completions|concepts|homepage-shelves|kg-stats|mission|my-progress|navigator|repo-catalog|slug-mapping|tag-labels|topics-gallery)(/.*)?(\\?.*)?$", + "source": "^/build/(breadcrumb-context|catalog|co-completions|concepts|homepage-shelves|kg-stats|mission|my-progress|navigator|repo-catalog|slug-mapping|tag-labels|topics-gallery|topics-tree|topics)(/.*)?(\\?.*)?$", "target": "/build/$1$2$3", "destination": "srv-api", "authenticationType": "none" @@ -582,7 +583,8 @@ "cacheControl": "public, max-age=3600" }, { "source": "^/browse/?(\\?.*)?$", "target": "/content/pages/browse/$1", "destination": "srv-api", "authenticationType": "none" }, - { "source": "^/topics/?(\\?.*)?$", "target": "/content/pages/topics/$1", "destination": "srv-api", "authenticationType": "none" }, + { "source": "^/topics/?(\\?.*)?$", "target": "/content/topics-index$1", "destination": "srv-api", "authenticationType": "none" }, + { "source": "^/topics/(.*)$", "target": "/content/topics/$1", "destination": "srv-api", "authenticationType": "none" }, { "source": "^/tutorial-navigator/?(\\?.*)?$", "target": "/content/pages/tutorial-navigator/$1", "destination": "srv-api", "authenticationType": "none" }, { "source": "^/developer-advocates/?(\\?.*)?$", "target": "/content/pages/developer-advocates/$1", "destination": "srv-api", "authenticationType": "none" }, { "source": "^/developer-advocates/([^/?]+)/?(\\?.*)?$", "target": "/content/developer-advocates/$1", "destination": "srv-api", "authenticationType": "none" }, From bd8c7355ad797d1eaaaf6e96a51184ccc6957de8 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Tue, 1 Sep 2026 16:28:12 -0400 Subject: [PATCH 57/71] feat(topics): topics-tree progressive-enhancement island (filter + deep-link) --- hugo-apps/src/topics-tree/main.ts | 70 +++++++++++++++++++++++++++++++ hugo-apps/vite.config.ts | 1 + 2 files changed, 71 insertions(+) create mode 100644 hugo-apps/src/topics-tree/main.ts diff --git a/hugo-apps/src/topics-tree/main.ts b/hugo-apps/src/topics-tree/main.ts new file mode 100644 index 000000000..059ce5f50 --- /dev/null +++ b/hugo-apps/src/topics-tree/main.ts @@ -0,0 +1,70 @@ +// hugo-apps/src/topics-tree/main.ts +// +// Progressive enhancement over the server-rendered
/