diff --git a/.github/reports/audits/changelog-keepachangelog-audit-2026-07-29.md b/.github/reports/audits/changelog-keepachangelog-audit-2026-07-29.md index 9db88f490..c44a95716 100644 --- a/.github/reports/audits/changelog-keepachangelog-audit-2026-07-29.md +++ b/.github/reports/audits/changelog-keepachangelog-audit-2026-07-29.md @@ -15,8 +15,103 @@ stability: stable domain: governance language: en --- ---- ---- +# CHANGELOG.md — Keep a Changelog 1.1.0 conformance audit + +**Date:** 2026-07-29 +**Spec:** [Keep a Changelog 1.1.0](https://keepachangelog.com/en/1.1.0/) +**Scope:** the repository's single `CHANGELOG.md`. The other changelog-named files are tooling, guidelines or fixtures, not changelogs. + +## Already conformant before this audit + +- ISO 8601 dates on version headings (`## [0.6.0] - 2026-06-19`) +- Latest version first, `[Unreleased]` at the top +- Semantic Versioning stated in the header + +## Divergences found + +| # | Divergence | Spec basis | Status | +|---|---|---|---| +| D1 | Type sections not in canonical order in any version | defined order `Added, Changed, Deprecated, Removed, Fixed, Security` | **Fixed** | +| D2 | Duplicate type sections within one version — `0.6.0` had `Added`×3 and `Fixed`×4; `0.5.0` had `Fixed`×3 and `Documentation`×2 | "The same types of changes should be grouped." | **Fixed** | +| D3 | Non-standard types `Documentation` (×4) and `Performance` (×1) | only six types are defined | **Deferred** — needs reclassification | +| D4 | Empty sections carrying filler — `(none identified)`×2, `- [placeholder]`×3 | "Remove empty sections from CHANGELOG, they occupy too much space and create too much noise in the file." | **Fixed** | +| D5 | **Zero link definitions.** Every `[0.6.0]`-style heading rendered as literal text | "Versions and sections should be linkable." | **Partly fixed** | +| D6 | `## Contributors` sat *between* `[Unreleased]` and `[0.6.0]`, inside the version list | version-per-section model | **Fixed** | +| D7 | That placement also broke `extract-pr-entries.cjs` | — | **Fixed** as a consequence of D6 | +| D8 | PR references carried no titles | requested internally, not a spec item | **Fixed** | +| D9 | Four tagged releases have **no changelog entry**: `v0.2.1`, `v0.5.1`, `v0.7.0`, `v1.0.0` | "There should be an entry for every single version." | **Deferred** — needs release history | +| D10 | Two documented versions were **never tagged**: `0.6.0`, `0.1.0`; and two dates disagree with their tags | "The release date of each version is displayed." | **Deferred** — needs a decision | +| D11 | Five references used `/pull/N` URLs for numbers that are **issues** (`#1093`, `#1096`, `#1099`, `#1100`, `#1126`), and `#1145` was cited both ways | accuracy | **Fixed** | + +### D7 in detail — the automation consequence + +`scripts/workflows/changelog/extract-pr-entries.cjs:78` ends the `[Unreleased]` section at the next `^##\s+\[` heading — a *bracketed* one. `## Contributors` is not bracketed, so the extractor ran straight past it and swallowed the Contributors prose into the extracted entries. Reproduced before the fix (143 lines extracted, including the `PR Range:` paragraph); after moving the section below the version list, extraction is clean at 127 lines. + +## What was fixed, and how it was verified + +Applied as two scripted passes rather than hand edits, each with an invariant check: + +1. **Structural** — merged duplicate type sections, ordered canonically, dropped empty/placeholder sections, moved non-version `##` sections below the version list. + *Invariant:* all 462 non-placeholder items preserved. Total list items went 465 → 462, exactly the three `- [placeholder]` items removed intentionally. The two `(none identified)` lines were not list items. +2. **References** — injected 160 PR titles, corrected 6 mislabelled issue URLs, appended 5 version link definitions. + +**Entry format for PR references** is now: + +```markdown +([PR #1392](https://github.com/lightspeedwp/.github/pull/1392) — *feat(...): …*, [#1393](…/issues/1393)) +``` + +The title sits **outside** the link, deliberately. Putting it inside the link text (`[PR #1392 — title](url)`) broke `scripts/validation/changelog-rules.cjs:66`, whose rule is `/\(\[PR #(\d+)\]/` — it requires the bracket to close directly after the digits. That mistake took the validator from 1 error to 59; moving the title out satisfies both the rule and readability. Six titles containing literal `*` or `_` are escaped, so `research/*` no longer terminates the emphasis early. + +### D5 — partly fixed, and why + +Link definitions were added only where a corresponding tag exists: + +``` +[Unreleased], [0.5.0], [0.4.0], [0.3.0], [0.2.0] +``` + +`[0.6.0]` and `[0.1.0]` are **omitted on purpose** — neither has a tag, so any compare range would be fabricated. They stay unlinkable until D9/D10 is resolved. + +Because `v0.2.1` and `v0.5.1` are undocumented (D9), the `[0.3.0]` compare range spans `v0.2.1`'s commits. Documented predecessors were used so the ranges match the document's own narrative; this resolves once D9 does. + +## Deferred — these need a human decision + +### D3 — non-standard change types + +`### Documentation` appears in `0.5.0`, `0.4.0`, `0.2.0`, `0.1.0`; `### Performance` in `0.2.0`. The spec defines only six types. Reclassifying is per-entry judgement — new docs are `Added`, doc revisions are `Changed` — so it was not done mechanically. Options: reclassify entry by entry, or record a deliberate house extension to the spec. + +### D9 / D10 — the version history does not reconcile with the tags + +| | | +|---|---| +| Tagged, no entry | `v0.2.1` (2025-12-18), `v0.5.1` (2026-06-08), `v0.7.0` (2026-06-03), `v1.0.0` (2026-06-03) | +| Entry, never tagged | `0.6.0` (2026-06-19), `0.1.0` (2025-09-25) | +| Date disagreement | `v0.4.0` tag 2026-05-26 vs entry 2026-05-27; `v0.2.0` tag 2025-12-15 vs entry 2025-12-18 | + +`v0.7.0` and `v1.0.0` are both tagged 2026-06-03 — the same day as `v0.5.0` — while `0.6.0` is documented as 2026-06-19, *after* both. The version history itself is tangled, not merely its changelog. Writing entries for the four undocumented releases requires knowing what shipped in each; that cannot be inferred safely and was not guessed. + +### Two outstanding validator errors + +`npm run validate:changelog` reports 2 errors (from 1 before, warnings unchanged at 34): + +- **Pre-existing** — the *GitHub Workflows Consolidation Initiative — Phase 1A* entry opens `([Epic #1227](…)`, so the required `([PR #` token never appears. Unchanged by this audit. +- **Newly revealed, not caused** — the *GitHub Actions workflow hardening* entry cited `#1093`, `#1096`, `#1099`, `#1100` as PRs. All four are **issues**, and all four are "rewrite X Agent for multi-provider support" — unrelated to workflow hardening. Correcting the URLs (D11) removed the only `[PR #` token, exposing that the entry references no PR at all and that its description and references do not match each other. The single candidate PR by title, `#413 fix: command-gate release workflow and harden permissions`, is not plausible given the issue numbers. **Needs whoever wrote it to supply the real reference.** + +Also note the validator is not wired into CI — `checks.yml` runs `npm run test`, which does not include `validate:changelog`. Worth adding once the two errors are resolved, or it will keep drifting. + +## Suggested next steps + +1. Decide D3 — reclassify, or document the extension. +2. Reconcile D9/D10 with whoever owns releases; then add the two missing link definitions. +3. Supply the real PR reference for the *workflow hardening* entry, and restructure the *Phase 1A* entry to lead with its PR. +4. Add `validate:changelog` to CI once green. + +## Reproducing -*Audit generated by the LightSpeedWP Automation Team* +```bash +npm run validate:changelog # repo rules (title/description limits, PR link) +npx markdownlint-cli2 CHANGELOG.md # markdown lint +git tag --list # compare against documented versions +``` diff --git a/CHANGELOG.md b/CHANGELOG.md index 1232fb6ec..4bb870c98 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -35,6 +35,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 (none identified) ### Added +- **Content-parity invariant covers all eight requirement types** — `consistency.json` declares the full taxonomy so any type change breaks the check, which required unifying the list's phrasing across three files. ([PR #1422](https://github.com/lightspeedwp/.github/pull/1422), [#1421](https://github.com/lightspeedwp/.github/issues/1421)) - **Gitleaks secret scanning** — Added `gitleaks-reusable.yml`, an organisation-wide reusable workflow other repositories call via `workflow_call`, plus a `gitleaks.yml` caller running on pull requests into `develop`/`main`. Runs the open-source Gitleaks CLI directly (the `gitleaks-action` wrapper requires a paid licence for organisation repositories). Per-PR runs scan the working tree; `workflow_dispatch` accepts a `full-history` input for on-demand full-history rescans. A baseline full-history scan of this repository returned 50 hits, all verified as placeholder values in documentation and tests, allowlisted in `.gitleaks.toml`. ([PR #1444](https://github.com/lightspeedwp/.github/pull/1444)) - **Repository restructuring Phase 1 — folder consolidation and asset reorganization** — Consolidated internal asset management by moving `scripts/`, `website/`, and `projects/active/` folders to `.github/` directory, and migrated `schema/` to visible `schemas/` folder at repository root for improved portability. Updated 400+ path references across package.json, npm scripts, GitHub workflows, validation scripts, and documentation. All 458 files reorganized with zero data loss and comprehensive path reference validation. ([PR #1446](https://github.com/lightspeedwp/.github/pull/1446) — *refactor: Phase 1 repository restructuring — folder consolidation*, [#1447](https://github.com/lightspeedwp/.github/issues/1447), [#1438](https://github.com/lightspeedwp/.github/issues/1438)) @@ -134,6 +135,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **Restored changelog audit destroyed during a rebase** — The Keep a Changelog conformance audit was cut from 117 lines to 22, losing its entire body. Recovered intact. ([PR #1422](https://github.com/lightspeedwp/.github/pull/1422), [#1421](https://github.com/lightspeedwp/.github/issues/1421)) + +- **`validate-pr-template` rejected cross-repo issue references** — `hasIssueReference` matched only bare `#123`, so `owner/repo#123` and URL forms were wrongly flagged as missing. Regex widened, with tests. ([PR #1422](https://github.com/lightspeedwp/.github/pull/1422), [#1421](https://github.com/lightspeedwp/.github/issues/1421)) + - **`documentation.yml` workflow security hardening and correctness fixes** — Scoped workflow permissions to individual jobs: global demoted to `contents: read`; `regenerate` and `maintain` jobs receive only the write scopes they need. Added `persist-credentials: false` for the read-only audit checkout and `persist-credentials: true` for write jobs. Fixed maintenance report accuracy: per-step outcomes for mermaid, contrast, and staleness tasks are now captured and displayed instead of always showing success. Fixed README resolver over-inclusion: root README is now only added to the regeneration list when subdirectory files change. ([PR #1387](https://github.com/lightspeedwp/.github/pull/1387), [#1386](https://github.com/lightspeedwp/.github/issues/1386), [Epic #1227](https://github.com/lightspeedwp/.github/issues/1227)) - **Changelog automation: Section headers destroyed on merge** — The merge-entries workflow was discarding section headers during deduplication, corrupting changelog structure on every PR merge. Fixed deduplication logic to preserve headers and limited scope to [Unreleased] section only. ([PR #1276](https://github.com/lightspeedwp/.github/pull/1276), [#1275](https://github.com/lightspeedwp/.github/issues/1275)) diff --git a/agents/playwright-testing-agent/AGENT.md b/agents/playwright-testing-agent/AGENT.md index ec69e48fe..ea177a832 100644 --- a/agents/playwright-testing-agent/AGENT.md +++ b/agents/playwright-testing-agent/AGENT.md @@ -88,7 +88,7 @@ PRD / acceptance criteria → requirement IDs → classification (functional flow, content rule, visual rule, accessibility rule, performance rule, analytics or conversion rule, - integration rule, error or empty state) + integration rule, or error or empty state) → human-readable test cases → traceability matrix → REVIEW GATE diff --git a/agents/playwright-testing-agent/consistency.json b/agents/playwright-testing-agent/consistency.json index b79caf356..acf02ffe7 100644 --- a/agents/playwright-testing-agent/consistency.json +++ b/agents/playwright-testing-agent/consistency.json @@ -3,8 +3,8 @@ "sharedPhrases": [ { "name": "requirement-type-taxonomy", - "why": "The approved requirement types are restated in three places. If they drift, an extractor can classify against a list one file does not recognise — which is how performance requirements were being dropped before they became a first-class type.", - "text": "functional flow, content rule, visual rule, accessibility rule, performance rule", + "why": "The approved requirement types are restated in three places. If they drift, an extractor can classify against a list one file does not recognise — which is how performance requirements were being dropped before they became a first-class type. All eight types are declared, so a change to any one of them breaks the check; this requires the three files to phrase the list identically (word forms, Oxford 'or'), which they now do.", + "text": "functional flow, content rule, visual rule, accessibility rule, performance rule, analytics or conversion rule, integration rule, or error or empty state", "files": [ "shared/core-prompt.md", "AGENT.md", diff --git a/agents/playwright-testing-agent/skills/agent-attached/hermes/test-pack-builder/SKILL.md b/agents/playwright-testing-agent/skills/agent-attached/hermes/test-pack-builder/SKILL.md index 631ce6c1d..c9d65fc04 100644 --- a/agents/playwright-testing-agent/skills/agent-attached/hermes/test-pack-builder/SKILL.md +++ b/agents/playwright-testing-agent/skills/agent-attached/hermes/test-pack-builder/SKILL.md @@ -78,7 +78,7 @@ For each requirement: - assign a requirement ID - write one short requirement statement -- classify it as one of: functional flow, content rule, visual rule, accessibility rule, performance rule, analytics/conversion rule, integration rule, or error/empty state +- classify it as one of the following: functional flow, content rule, visual rule, accessibility rule, performance rule, analytics or conversion rule, integration rule, or error or empty state - cite the source evidence that supports it Do not split one criterion into many IDs unless the evidence clearly supports separate traceable requirements. diff --git a/scripts/validation/__tests__/template-helpers.test.js b/scripts/validation/__tests__/template-helpers.test.js index 1447a4cf2..b8a066f87 100644 --- a/scripts/validation/__tests__/template-helpers.test.js +++ b/scripts/validation/__tests__/template-helpers.test.js @@ -113,6 +113,28 @@ describe("hasIssueReference", () => { it("detects issue at document start", () => { expect(hasIssueReference("#999")).toBe(true); }); + + it("detects cross-repo references (owner/repo#number)", () => { + expect(hasIssueReference("Closes lightspeedwp/.github#1393")).toBe(true); + expect(hasIssueReference("Fixes org/repo#456")).toBe(true); + expect(hasIssueReference("lightspeedwp/.github#1087")).toBe(true); + }); + + it("detects cross-repo references with closing keywords", () => { + expect(hasIssueReference("Relates to lightspeedwp/.github#1079")).toBe( + true, + ); + expect(hasIssueReference("resolves org/repo#100")).toBe(true); + }); + + it("detects full GitHub issue/PR URLs", () => { + expect(hasIssueReference("https://github.com/org/repo/issues/123")).toBe( + true, + ); + expect(hasIssueReference("https://github.com/org/repo/pull/456")).toBe( + true, + ); + }); }); describe("hasChangelogEntry", () => { diff --git a/scripts/validation/template-helpers.cjs b/scripts/validation/template-helpers.cjs index ebaa70ea4..7f9e4482e 100644 --- a/scripts/validation/template-helpers.cjs +++ b/scripts/validation/template-helpers.cjs @@ -21,7 +21,10 @@ function sectionBody(body, headingRegex) { function hasIssueReference(sectionText) { const cleaned = stripHtmlComments(sectionText); - return /(?:^|\n)\s*(?:[-*]\s*)?(?:(?:closes|fixes|resolves|relates to)\s+)?#\d+\b/i.test(cleaned); + // Matches same-repo (#123) or cross-repo (owner/repo#123) issue references (optionally preceded by + // GitHub closing keywords) at the start of a line, plus full GitHub issue/PR URLs anywhere in the text. + return /(?:^|\n)\s*(?:[-*]\s*)?(?:(?:closes|fixes|resolves|relates to)\s+)?(?:[\w.-]+\/[\w.-]+)?#\d+\b/i.test(cleaned) + || /https:\/\/github\.com\/[\w.-]+\/[\w.-]+\/(?:issues|pull)\/\d+\b/.test(cleaned); } function hasChangelogEntry(sectionText) {