Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
22c9cc1
Add QA agents for test planning and reporting
keber Sep 2, 2026
180d128
fix(templates): correct ADO config path in generated instructions/rules
keber Sep 2, 2026
b71851f
feat(settings): add initial settings.json for attribution configuration
keber Sep 2, 2026
2e4cf63
fix(skills): repair broken references in qa-spec-generation and qa-au…
keber Sep 8, 2026
0ad2419
feat(skills): add PENDING-BROWSER to the TC origin vocabulary
keber Sep 8, 2026
b028e6f
fix(templates): restore Spanish accents in the four agent templates
keber Sep 8, 2026
ac41102
fix(templates): parameterize the defect ID prefix
keber Sep 8, 2026
61a0e0c
feat(skills): gate Stage 5 on a static check and a smoke run
keber Sep 8, 2026
2449e1e
feat(skills): establish the official suite as the only source of truth
keber Sep 8, 2026
b921584
feat(skills): block defect creation on a known-issues lookup
keber Sep 8, 2026
780ae1a
feat(scripts): add a language-aware accent gate for Markdown
keber Sep 8, 2026
588d271
feat(skills): require doc-generating stages to emit the measurement
keber Sep 8, 2026
9aa073a
fix(templates): keep QA password out of Playwright traces
keber Sep 8, 2026
74d84f5
fix(templates): keep QA password out of traces in auth fixtures
keber Sep 8, 2026
84b6f9e
fix(init): scaffold module specs under qa/01-specifications/
keber Sep 8, 2026
f47fe4f
fix(validate): look for automation specs under e2e/tests/
keber Sep 8, 2026
11f8f5f
fix(validate): root the submodule spec scan at 01-specifications/
keber Sep 8, 2026
3ef1e2b
merge: keep QA password out of Playwright traces
keber Sep 8, 2026
3b1f219
merge: converge spec paths on qa/01-specifications/
keber Sep 8, 2026
13d6553
fix(scripts): replace forbidden typographic characters with ASCII
keber Sep 8, 2026
9d362ca
feat(scripts): add check-forbidden-chars gate
keber Sep 8, 2026
b0e5470
merge: stop propagating forbidden characters and gate against them
keber Sep 8, 2026
89060e2
fix(docs): replace arrows in ASCII diagrams with ASCII equivalents
keber Sep 8, 2026
e8a4c96
fix(skills,templates): replace forbidden typographic characters in prose
keber Sep 8, 2026
b09d623
fix(docs,examples,integrations): replace forbidden typographic charac…
keber Sep 8, 2026
c54e93c
merge: clear forbidden typographic characters from package prose
keber Sep 8, 2026
0ac74b8
fix(docs): replace forbidden typographic characters in README and mig…
keber Sep 8, 2026
9d38630
fix(integrations): scan .ps1 and clear its forbidden characters
keber Sep 8, 2026
12ecea4
fix(templates): load only the needed keys from .env in playwright con…
keber Sep 9, 2026
34d87e1
docs: record the lane heartbeat caller decision
keber Sep 9, 2026
c2e9d94
feat(config): add parallelLanes lane table schema and resolver
keber Sep 9, 2026
a58cc56
feat(automation): add config-driven lane-lock to the scaffold
keber Sep 9, 2026
6f9b7be
feat(automation): add global-setup lane guards
keber Sep 9, 2026
c5868bb
feat(automation): drive scaffold projects and auth from the lane table
keber Sep 9, 2026
9930f5a
docs(instructions): add the BLOCKING lane-lock behavior rule
keber Sep 9, 2026
8c17537
fix(templates): strip BOM and em-dashes from .env.example, scan them
keber Sep 9, 2026
f721957
merge: generalize the parallel scaffold from 2 fixed lanes to N
keber Sep 9, 2026
1bdedcd
fix(ci): match test files with a flat glob
keber Sep 9, 2026
2278cc8
fix(ci): let the shell expand the test glob
keber Sep 9, 2026
6b31681
chore(ci): test on Node 20, 22 and 24, require >=20
keber Sep 9, 2026
e22021a
fix(ci): validate a scaffolded project instead of this repo's qa/
keber Sep 9, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .claude/rules/qa-framework-dev.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
@.github/copilot-instructions.md
7 changes: 7 additions & 0 deletions .claude/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"attribution": {
"commit": "",
"pr":"",
"sessionUrl": false
}
}
15 changes: 13 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: ['18', '20']
node-version: ['20', '22', '24']

steps:
- uses: actions/checkout@v6
Expand All @@ -22,4 +22,15 @@ jobs:

- run: npm test

- run: npm run validate
# validate checks a scaffolded qa/ tree. This repo's own qa/ is gitignored,
# so it never exists on the runner - validating it here checked nothing.
# Scaffold a throwaway project instead: that exercises the real contract,
# which is that what init.js generates passes validate.js.
- name: Scaffold a project and validate it
run: |
set -e
scaffold="$(mktemp -d)"
cd "$scaffold"
node "$GITHUB_WORKSPACE/scripts/init.js"
node "$GITHUB_WORKSPACE/scripts/validate.js"
node "$GITHUB_WORKSPACE/scripts/validate.js" --strict
73 changes: 63 additions & 10 deletions MIGRATION-NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ each pattern to the decoupled package approach.

---

## Pattern A Repo A style (redacted-repo-web)
## Pattern A - Repo A style (redacted-repo-web)

### Characteristics

Expand Down Expand Up @@ -61,7 +61,7 @@ each pattern to the decoupled package approach.

4. **Move existing spec files** (no structural changes needed):
```
qa/07-automation/e2e/ keep your existing .spec.ts files here
qa/07-automation/e2e/ <- keep your existing .spec.ts files here
```

5. **Add missing standard folders**:
Expand All @@ -70,7 +70,7 @@ each pattern to the decoupled package approach.
```
Copy the standards templates:
- `node_modules/keber/qa-framework/templates/defect-report.md`
`qa/00-standards/bug-report-template.md`
-> `qa/00-standards/bug-report-template.md`

6. **Update `package.json` in `qa/07-automation/`**:
Replace the existing `gmoindustrial-qa-e2e` package name with your project name.
Expand All @@ -89,15 +89,15 @@ each pattern to the decoupled package approach.

---

## Pattern B Repo B style (redacted-repo)
## Pattern B - Repo B style (redacted-repo)

### Characteristics

- QA lives under `qa/` inside the main project repo
- Has `00-standards/` with naming-conventions, bug-template, TC-template, test-data-guidelines
- Has `08-azure-integration/` with playwright-azure-reporter, inject-ado-ids.ps1, module-registry.json
- Multi-module structure: 4 modules × 17+ submodules
- Full ADO integration: Plans 22304/22794/22875, WI IDs 2295723034
- Full ADO integration: Plans 22304/22794/22875, WI IDs 22957-23034
- Login: email-based auth

### Migration steps
Expand Down Expand Up @@ -133,15 +133,15 @@ each pattern to the decoupled package approach.
}
```

3. **Keep existing `00-standards/` files** they are compliant with the framework.
3. **Keep existing `00-standards/` files** - they are compliant with the framework.
The framework's `templates/defect-report.md` is a generalization of the existing
bug-report template; no changes required.

4. **Keep existing `08-azure-integration/` files**:
- `module-registry.json` compatible as-is
- `module-registry.json` - compatible as-is
- Replace `inject-ado-ids.ps1` with the generalized version from
`node_modules/keber/qa-framework/integrations/ado-powershell/scripts/inject-ado-ids.ps1`
(optional existing script continues to work)
(optional - existing script continues to work)

5. **Verify `.gitignore`** contains:
```
Expand All @@ -164,10 +164,10 @@ each pattern to the decoupled package approach.
- **TC IDs**: The framework uses `[TC-MODULE-SUB-NNN]` format. Existing TCs with different
formats (e.g., plain numbers like `[TC-001]`) can be migrated by renaming at the next
spec refresh cycle; no immediate change required.
- **ADO WI IDs**: Already injected IDs (`[22957]` prefixes) are compatible the reporter
- **ADO WI IDs**: Already injected IDs (`[22957]` prefixes) are compatible - the reporter
reads the numeric prefix regardless of what follows.
- **storageState files**: Existing `.auth/*.json` files are compatible with the scaffold's
`global-setup.ts` no migration needed.
`global-setup.ts` - no migration needed.
- **Package name**: The automation sub-package (`gmoindustrial-qa-e2e` or similar) is a
private local package; renaming is optional cosmetic change.

Expand Down Expand Up @@ -205,3 +205,56 @@ upgradeable without touching the project's custom Copilot instructions.

The detection marker is the heading `# QA Framework Instructions`, which is consistent
across all previous versions.

---

## Native Claude Code support (no manual per-project work needed)

Previously, projects that used Claude Code instead of (or alongside) GitHub Copilot had
to hand-build their own bridge to the framework's skills and instructions - there was no
generated artifact for Claude Code at all. Two real consumer projects independently built
two incompatible solutions to the same problem (a full skill mirror with a manual sync
script in one case, hand-written thin command wrappers in the other).

`init` and `upgrade` now generate Claude Code artifacts natively, in parallel with the
Copilot ones, with no agent detection required:

- `.claude/commands/qa-{name}.md` - one thin wrapper per skill, generated dynamically from
`.github/skills/`. Each wrapper points back at the corresponding `SKILL.md` as the single
source of truth (`Read the full skill at .github/skills/qa-{name}/SKILL.md FIRST...`).
Zero duplication, zero drift.
- `.claude/rules/qa-framework.md` - the same 11 agent behavior rules and pipeline table as
`.github/instructions/qa-framework.instructions.md`, with skill references expressed as
`/qa-{name}` slash commands. No frontmatter, so it loads unconditionally (equivalent to
`applyTo: '**'`).

Both files are framework-owned and refreshed safely by `npx qa-framework upgrade`, the
same way the Copilot artifacts are. Projects that built a manual bridge to Claude Code
before this version can retire it (custom sync scripts, hand-copied skill mirrors, or
hand-translated `CLAUDE.md`/`.claude/rules/` content) and rely on the generated artifacts
instead.

---

## Optional ANALISIS/PLAN sprint-cycle mode (Claude Code only)

Previously, a project that wanted a sprint-centric manual testing workflow parallel to
the 6-stage pipeline (analysis document, ADO-trace-linked test plan, chat-only QA
advisory, sprint closing results report) had to hand-build its own Claude Code
subagents - one real consumer project did exactly that, with the sprint duration,
manual-testing timebox, date format, and locale all hardcoded to its own conventions.

`init` and `upgrade` now generate this mode natively as four generic, parameterized
subagents under `.claude/agents/` (`qa-analisis.md`, `qa-plan.md`, `qa-asesoria.md`,
`qa-informe-resultados.md`), gated by `integrations.azureDevOps.sprintCycle.enabled` in
`qa-framework.config.json`. It is off by default, requires Azure DevOps integration to
be meaningful (three of the four agents consume ADO work items or an ADO execution
report), and is intentionally Claude-only - these are Claude Code subagents with
per-agent `tools`/`model` frontmatter, a primitive GitHub Copilot has no equivalent for.

Projects that hand-built this mode before this version can retire their local copy and
adopt the generated one, moving their project-specific sprint duration, manual-testing
timebox, date format, and timezone into `integrations.azureDevOps.sprintCycle` in
`qa-framework.config.json`. See [docs/usage-with-agent.md](docs/usage-with-agent.md) and
[docs/installation.md](docs/installation.md) for the full config shape and generated
file list.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ qa/
├── {module}/{submodule}/ <- 6-file spec sets created per module/submodule in config
├── 02-test-plans/ <- Test plans (automated + manual)
├── 03-test-cases/ <- TC-*.md step-by-step docs (manual track; optional if fully automated)
├── 04-test-data/ <- Shared test data factories/seeders (optional per-module data lives in 01-specs)
├── 04-test-data/ <- Shared test data factories/seeders (optional - per-module data lives in 01-specs)
├── 05-test-execution/ <- Execution reports and results
├── 06-defects/open|resolved/ <- Defect tracking
├── 07-automation/ <- Playwright automation code and config
Expand Down
50 changes: 27 additions & 23 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,11 @@

## Design Goals

1. **Decoupled core** the framework works without Playwright, without Azure DevOps, without any specific CI/CD system
2. **Layered optionality** features are added as explicit opt-in integrations, not baked into the core
3. **Agent-first design** every convention exists so that an IDE agent can navigate and produce artifacts predictably
4. **Spec-before-automation** the specification layer is always the source of truth; automation references specs, never the reverse
5. **Parameterization over hardcoding** project-specific values live in `qa-framework.config.json`, not in framework files
1. **Decoupled core** - the framework works without Playwright, without Azure DevOps, without any specific CI/CD system
2. **Layered optionality** - features are added as explicit opt-in integrations, not baked into the core
3. **Agent-first design** - every convention exists so that an IDE agent can navigate and produce artifacts predictably
4. **Spec-before-automation** - the specification layer is always the source of truth; automation references specs, never the reverse
5. **Parameterization over hardcoding** - project-specific values live in `qa-framework.config.json`, not in framework files

---

Expand Down Expand Up @@ -86,6 +86,8 @@
│ │
│ .github/instructions/qa-framework.instructions.md <- generated (framework) │
│ .github/skills/ <- copied from package (framework-owned) │
│ .claude/rules/qa-framework.md <- generated (framework, Claude Code) │
│ .claude/commands/qa-*.md <- generated (framework, Claude Code) │
└─────────────────────────────────────────────────────────────────────────────┘
```

Expand All @@ -98,12 +100,12 @@ Module analysis (agent)
01-specifications/module-X/submodule-Y/
├── 00-inventory.md what exists in the UI
├── 01-business-rules.md RN-* identifiers
├── 02-workflows.md FL-* flowcharts
├── 00-inventory.md <- what exists in the UI
├── 01-business-rules.md <- RN-* identifiers
├── 02-workflows.md <- FL-* flowcharts
├── 03-roles-permissions.md
├── 04-test-data.md
└── 05-test-scenarios.md TC-* identifiers ──────────────────────┐
└── 05-test-scenarios.md <- TC-* identifiers ──────────────────────┐
│ │
▼ ▼
Expand All @@ -120,8 +122,8 @@ Module analysis (agent)
[ADO enabled?]
YES playwright-azure-reporter syncs results to ADO Test Plan
NO 05-test-execution/automated/{date}.md (local report)
YES -> playwright-azure-reporter syncs results to ADO Test Plan
NO -> 05-test-execution/automated/{date}.md (local report)
06-defects/ (if test.skip for known bug)
Expand All @@ -132,18 +134,20 @@ Module analysis (agent)

## Layer Definitions

### Layer 1 Framework Core (mandatory)
### Layer 1 - Framework Core (mandatory)

Installed always. Contains:

- `qa/` directory skeleton (10 folders)
- `.github/skills/` 8 agent skill sets (3-layer model: SKILL.md + references/)
- `00-standards/` naming conventions, templates
- `.github/skills/` - 8 agent skill sets (3-layer model: SKILL.md + references/)
- `00-standards/` - naming conventions, templates
- `QA-STRUCTURE-GUIDE.md`
- `.github/copilot-instructions.md` — generated pipeline sequencer
- `.github/instructions/qa-framework.instructions.md` - generated pipeline sequencer (Copilot)
- `.claude/rules/qa-framework.md` - generated pipeline sequencer (Claude Code, unconditional load)
- `.claude/commands/qa-*.md` - generated thin wrappers, one per skill (Claude Code)
- `qa-framework.config.json` schema

### Layer 2 Playwright Integration (opt-in)
### Layer 2 - Playwright Integration (opt-in)

Installed always (scaffold is always created by `init`):

Expand All @@ -156,7 +160,7 @@ Installed always (scaffold is always created by `init`):
- `qa/07-automation/integration/README.md` (placeholder)
- `qa/07-automation/load/README.md` (placeholder)

### Layer 3 Azure DevOps Integration (opt-in)
### Layer 3 - Azure DevOps Integration (opt-in)

Installed when `integrations.azureDevOps.enabled = true`:

Expand All @@ -173,11 +177,11 @@ Installed when `integrations.azureDevOps.enabled = true`:
```
qa-framework.config.json (project-level, committed to repo)
├── project.* Display values, URLs (non-secret)
├── modules[] Module codes, paths, ADO IDs
├── conventions.* Naming patterns, TC ID format
├── testUsers[] RoleenvVar mapping (NOT credentials)
└── integrations.* Feature flags + integration config
├── project.* -> Display values, URLs (non-secret)
├── modules[] -> Module codes, paths, ADO IDs
├── conventions.* -> Naming patterns, TC ID format
├── testUsers[] -> Role->envVar mapping (NOT credentials)
└── integrations.* -> Feature flags + integration config
└── credentials come from:
.env (local, gitignored)
Expand Down Expand Up @@ -227,6 +231,6 @@ Full rules: [docs/folder-structure-guide.md](folder-structure-guide.md)

1. The primary test runner is Playwright. Other runners (Jest, Cypress) are not excluded but are not provided adapters in v1.0.
2. The target application runs in a browser. Back-end API-only testing is not the primary use case of this framework (though API testing can be added to `07-automation/` as needed).
3. The IDE agent is GitHub Copilot or equivalent. The instructions are written in Markdown and are IDE-agnostic.
3. The IDE agent is GitHub Copilot, Claude Code, or equivalent. The instructions are written in Markdown and are IDE-agnostic; `init`/`upgrade` generate native artifacts for both Copilot (`.github/instructions/`) and Claude Code (`.claude/rules/`, `.claude/commands/`) in parallel, without detecting which agent the consumer uses.
4. The project uses Git. The `qa/` directory lives inside the same repository as the application code (monorepo-friendly).
5. Credentials are always managed via environment variables. There is no fallback to hardcoded credentials in any framework file.
86 changes: 86 additions & 0 deletions docs/decisions/lane-heartbeat-caller.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# Decision: the dispatcher calls the lane heartbeat, not the executor

**Status**: accepted, not implemented
**Date**: 2026-09-08
**Scope**: the N-lane work still pending upstream (FRAMEWORK-FIXES v3 items 1-4, v5 items 1, 2, 4)

---

## Context

Parallel QA lanes exist to work around a constraint of the applications under test:
one account cannot hold two sessions at once, so a second login silently invalidates
the first. A lane lock assigns each concurrent worker its own account.

A lock is only useful if a held lane stays held. The reference implementation
(`lane-lock.js` in the Sispro Exportadora project) frees a lane when it looks stale,
using a composite predicate: the acquisition is older than 20 minutes AND either no
heartbeat was ever sent OR the last heartbeat is itself older than 20 minutes. A
recent heartbeat proves the holder is alive, so the lane is never swept.

That predicate is only as good as whoever calls `heartbeat`.

## The incident that forced the decision

On 2026-08-22 a delegated sub-agent spent over 20 minutes on a legitimate live
investigation - genuine environment congestion, 80 to 90 concurrent browser and node
processes - and never called `heartbeat`. The sweep did exactly what it was written
to do and freed the lane while the sub-agent was still driving that account: the
precise scenario the heartbeat exists to prevent.

No cross-session collision followed, but only because a person noticed the lane was
free with no matching completion report, paused the sub-agent, confirmed by explicit
reply that nothing had run against the browser after the sweep, and re-reserved the
lane. The protection failed by design and was caught by attentiveness.

## Decision

**The dispatcher calls `heartbeat` from its own wait loop.** The executor is not
required to call it and must not be relied upon to.

## Why

The executor option fails for a structural reason, not a discipline one. A delegated
agent has no reason to discover `heartbeat` unless the dispatching prompt tells it,
every single time. That is the same failure this project has now recorded four times
over: a known-issues record that existed and was not consulted, an orthography rule
that was asserted rather than measured, and this heartbeat. A mechanism whose
protection depends on correctly instructing an uninstructed party is not a control.

The dispatcher, by contrast, already knows whether the task is still running - it can
query task state directly. The executor cannot see how long it has been between its
own tool calls. Putting the call where the knowledge already is removes the
instruction step entirely, which is where option 1 actually broke.

The cost is real and accepted: the dispatcher's wait loop becomes more complex, and
the concrete implementation is agent-specific rather than framework-generic.

## Layering

This follows the split established in FRAMEWORK-FIXES v3 section 5.

The agent-agnostic framework states the requirement: a lane must be acquired through
the lock before any worker is dispatched against it, released when the work ends, and
kept alive by the dispatcher while the work is in flight. Each agent's adaptation
prescribes the mechanism. For Claude Code that means a `/loop` wrapper with a
`ScheduleWakeup` fallback and a non-blocking task-state check - primitives that have
no equivalent in other agents, and so must not be written into
`templates/qa-framework.instructions.md`.

## Consequences

- The N-lane scaffold ships `heartbeat` together with the rule naming its caller.
Shipping the sweep without that answer reproduces the 2026-08-22 incident in every
project that adopts it.
- A lane held far longer than the application's session TTL is a signal to verify,
not a defect to ignore. Each project's agent-specific instructions must define how
it verifies a lock is still legitimately held and how it clears a stale one.
- Neither option was implemented in the originating project, so there is no verified
reference implementation of the dispatcher-side loop to lift. It has to be written.

## Sources

- `temp/FRAMEWORK-FIXES-qa-framework-v5.md` item 5, which framed the fork and
recommended this side.
- `temp/FRAMEWORK-FIXES-qa-framework-v3.md` section 5, for the agent-agnostic versus
agent-specific split.
Loading