From 4a2ca23533cbf13040227f5e6b3306cba3b6da25 Mon Sep 17 00:00:00 2001 From: Dylan Nguyen Date: Tue, 1 Sep 2026 15:16:44 -0400 Subject: [PATCH 1/3] docs: add ChallengeAI federal delivery layer --- .challengeai/challenge-ac.md | 89 ++++++++++++++++++ .challengeai/challenge-api.md | 98 ++++++++++++++++++++ .challengeai/challenge-ato.md | 99 ++++++++++++++++++++ .challengeai/challenge-cd.md | 101 +++++++++++++++++++++ .challengeai/challenge-ci.md | 103 +++++++++++++++++++++ .challengeai/challenge-cli.md | 59 ++++++++++++ .challengeai/challenge-ea.md | 124 +++++++++++++++++++++++++ .challengeai/challenge-iac.md | 99 ++++++++++++++++++++ .challengeai/challenge-melt.md | 89 ++++++++++++++++++ .challengeai/challenge-sql.md | 89 ++++++++++++++++++ .challengeai/challenge-tdd.md | 90 ++++++++++++++++++ .challengeai/challenge-ui.md | 108 ++++++++++++++++++++++ .challengeai/federal-context.md | 156 ++++++++++++++++++++++++++++++++ .challengeai/profile.yml | 91 +++++++++++++++++++ CHALLENGEAI.md | 67 ++++++++++++++ 15 files changed, 1462 insertions(+) create mode 100644 .challengeai/challenge-ac.md create mode 100644 .challengeai/challenge-api.md create mode 100644 .challengeai/challenge-ato.md create mode 100644 .challengeai/challenge-cd.md create mode 100644 .challengeai/challenge-ci.md create mode 100644 .challengeai/challenge-cli.md create mode 100644 .challengeai/challenge-ea.md create mode 100644 .challengeai/challenge-iac.md create mode 100644 .challengeai/challenge-melt.md create mode 100644 .challengeai/challenge-sql.md create mode 100644 .challengeai/challenge-tdd.md create mode 100644 .challengeai/challenge-ui.md create mode 100644 .challengeai/federal-context.md create mode 100644 .challengeai/profile.yml create mode 100644 CHALLENGEAI.md diff --git a/.challengeai/challenge-ac.md b/.challengeai/challenge-ac.md new file mode 100644 index 0000000..05cd791 --- /dev/null +++ b/.challengeai/challenge-ac.md @@ -0,0 +1,89 @@ +# ChallengeAC + +Agile artifacts: whether what was asked for can be traced to what was built. + +## Covers + +Acceptance criteria, requirement traceability, ambiguity in what was requested, +and audit readiness of the delivery record. + +## The requirement + +SP 800-53 Rev. 5 System and Services Acquisition (SA-3, system development life +cycle) and Configuration Management (CM-3, change control). An assessor asks how +the team knows it built what was asked for, and how a change was decided. + +## Traceability without ceremony + +The requirement is that a capability can be traced to its implementation and its +test. It is not that a particular artifact exists before work starts. + +A traceability record written as the work lands describes the system, so it +survives an assessment as written. A backlog written ahead of the work describes +intent, which drifts from what shipped and has to be reconciled against reality +before an assessor can use it. Either can satisfy the requirement; only one is +accurate by construction. + +Where the record claims a capability is complete, that claim is checkable. A +partial state written honestly, with a sentence on the shortfall, reads better in +assessment than a completion that an assessor disproves. + +## Change control + +An assessor asking how a change was controlled is asking for a durable record +that shows what changed, who approved it, and how it was verified. A pull +request carrying one concern, with review and conversation resolution required, +answers that. A pull request carrying several makes the record of why any one of +them landed ambiguous. + +## In this repository + +There's no formal traceability record — no requirements table, no linked +issue tracker mapping capability to implementation to test. What exists is +real, structured process discipline for an open-source project of this size: +`.github/pull_request_template.md` requires a description, a change-type +checkbox, a "Related Issues" section (`Fixes #`/`Closes #`/`Resolves #`, +linking the PR to an issue), a changes-made list, and a testing checklist. +`.github/ISSUE_TEMPLATE/` has both bug-report and feature-request forms with +`config.yml` present, giving contributors a structured way to open work +items in the first place — real traceability infrastructure, even without a +formal capability-tracking document behind it. + +**One specific gap in that template, worth naming precisely:** the PR +template's testing checklist asks for `npm run lint` and `npm run build` +explicitly, but not `npm test` — even though `npm test` (vitest) is a real, +required CI job (see ChallengeCI). A contributor following the template +literally could check every box without ever having run the unit tests +locally, even though CI would still catch a failure before merge. + +`CONTRIBUTING.md` (not audited in depth by this tool file — see its own +content directly) describes the expected contribution flow; whether one PR +per concern, review approval, and conversation resolution are actually +enforced by GitHub branch protection isn't verified — see `profile.yml`'s +`gates` section. + +## Evidence + +- A traceability record mapping each capability to its implementation and test, + with honest states rather than aspirational ones. +- Change history, one concern per change, with the verification recorded. +- Review approval and conversation resolution enforced rather than optional. + +The PR and issue templates are real, structured evidence of intent to trace +changes to issues; no formal capability-to-test mapping exists beyond that. + +## Review checklist + +- Does the capability that just landed appear in the traceability record, + naming its implementation and its test? +- Does the record claim completion for something only partly built? +- Does the change record say how it was verified? +- Is more than one concern bundled here, making the record of why it landed + ambiguous? +- Was something ambiguous resolved by asking, or by guessing and documenting the + guess? +- **This repository-specific:** does a PR that changes generator or + sanitization logic actually run `npm test` locally, not just the two + boxes the PR template checklist currently asks for? Add the missing + checkbox, or rely on CI catching it — but don't assume the template alone + ensures it. diff --git a/.challengeai/challenge-api.md b/.challengeai/challenge-api.md new file mode 100644 index 0000000..9e35811 --- /dev/null +++ b/.challengeai/challenge-api.md @@ -0,0 +1,98 @@ +# ChallengeAPI + +The API: its contract, what it accepts, and what it discloses. + +## Covers + +API contracts, input validation, interoperability, versioning, rate limiting, +and federal API governance. + +## The requirement + +- **SP 800-53 Rev. 5** System and Communications Protection (SC) and System and + Information Integrity (SI-10, input validation). +- **The Federal Source Code Policy and API guidance** expect government APIs to + be documented, versioned, and stable for the people who build against them. +- **OMB guidance on open data** expects machine-readable access where the data + is public. + +## The contract + +An API is a promise to people who cannot be consulted before it changes. + +- **Versioned in the path**, moving independently of the product release, so + adding a version does not by itself require a major release of the system. +- **Published as a machine-readable document**, so a client can generate against + it rather than read prose. +- **Validated at the edge of the process.** Every parameter is parsed and + bounded before it reaches a query. Unbounded pagination and unbounded result + sizes are both denial-of-service vectors and cost vectors. +- **Health reported against dependencies**, not just process liveness. A process + that is up and cannot reach its data is not healthy. +- **Rate limited at the edge**, so a burst is refused rather than queued. + +## Errors + +An error names what was wrong with the request without describing the internals +of the system. A stack trace, a driver message, or a query fragment in a +response body is an information disclosure finding. + +## In this repository + +There is no API — no backend, no server-rendered route, nothing this +repository exposes over the network beyond the static built site itself +(see ChallengeIaC). Everything runs in the browser: CSV/JSON parsing +(`papaparse`), chart preview rendering, and code generation are all +client-side, with no request/response cycle to a server this project +controls. + +**The closest analog worth reviewing with this tool's questions in mind is +the generated code output itself** — what the app hands back to the user as +a download or copy-paste, rather than a network response: + +- **Validated at the edge:** real. Uploaded CSV is checked for file + type/extension and a 10MB size cap (`Generator.tsx`) before it's parsed at + all. Every string value in the user's data and options is HTML-escaped + (`BaseGenerator.sanitizeForCodeGeneration`, `Generator.tsx`'s own parallel + copy of the same logic — see ChallengeEA's note on the two implementations) + before it's interpolated into generated code — real, deliberate XSS + prevention on the one thing this app produces that could carry an + injection risk into whatever page or Drupal site the generated code ends + up embedded in. +- **Versioning:** not applicable — there's no published contract to version. + The generated code's shape can change between releases with no + compatibility guarantee to anyone who saved output from an earlier version. +- **Published as a machine-readable document:** not applicable — nothing + here is a contract a client generates against. +- **Health / rate limiting:** not applicable — there's no server-side + process or endpoint to report health or limit request rate on. The only + rate-limit-shaped concern is client-side performance on very large CSV + uploads, bounded by the existing 10MB cap. +- **Errors:** `Generator.tsx` surfaces user-facing error messages (invalid + file type, file too large, parse failures) directly in the UI rather than + as an API response — no internal detail (stack traces, file paths) was + found leaking into any user-visible error message during this review. + +## Evidence + +- The published contract document. +- Route tests exercising the documented behaviour, including failure cases. + +Neither applies in the usual sense. `dataTransform.test.ts` (see +ChallengeTDD) is the closest evidence this repository has for validated +behavior, though it covers data transformation, not the sanitization path +specifically. + +## Review checklist + +- Is every new parameter validated and bounded? +- Does a new route appear in the contract document in the same pull request? +- Does an error response leak an internal detail? +- Does a change alter the shape of an existing response? That is a breaking + change to a published contract, and it belongs behind a new version. +- Is anything personal placed in a URL, where it reaches logs and history? +- **This repository-specific:** does a new code-output generator (a fifth + format alongside the existing four — see ChallengeEA) route every + user-supplied string through `sanitizeForCodeGeneration` before + interpolating it into generated code? Skipping that reintroduces the + exact XSS risk this system's one real security control exists to close. diff --git a/.challengeai/challenge-ato.md b/.challengeai/challenge-ato.md new file mode 100644 index 0000000..790d5d0 --- /dev/null +++ b/.challengeai/challenge-ato.md @@ -0,0 +1,99 @@ +# ChallengeATO + +Authorization readiness: whether this system could be assessed today and whether +the evidence would hold. + +## Covers + +Control traceability, evidence sufficiency, the security package, the privacy +assessment, and bounded penetration testing against a running instance. + +## The requirement + +- **FISMA** requires federal systems to implement an information security + program and to be authorized before operating. +- **NIST SP 800-37 Rev. 2** defines the Risk Management Framework. +- **FIPS 199** categorizes the system as low, moderate, or high impact. The + categorization drives everything downstream, and it is recorded in + `profile.yml`. +- **NIST SP 800-53 Rev. 5** provides the control baseline the categorization + selects. +- **NIST SP 800-53A Rev. 5** provides the procedures an assessor uses. +- **The E-Government Act** requires a Privacy Impact Assessment where a + system handles information **in identifiable form** — not merely + information "about" individuals in aggregate — or initiates a qualifying + electronic collection from ten or more non-federal persons. + +## The package + +A security package is a set, numbered so it can be handed over as one. The +System Security Plan describes the system, its boundary, and how each control is +met. Around it sit the assessment plan and report, the plan of action and +milestones, the risk assessment, the contingency and incident response plans, +configuration management, access control policy, continuous monitoring, and the +privacy impact assessment. + +The boundary described in the SSP is the boundary the system actually +provisions. Anything outside it is inherited from the provider's own +authorization and is named as inherited rather than claimed. + +## In this repository + +No security package exists — no `docs/security/`, no SSP, no FIPS 199 +categorization, no POA&M, no PIA. `profile.yml` records `control_baseline: +null` and `impact_level: null` because none is pursued, which is +appropriate for a tool with no accounts, no persistence, and no PII (see +ChallengeSQL) — not a gap this system needs to close, unlike a system that +actually handles federal data. + +What's worth checking instead is `SECURITY.md`'s own "Security Measures" +list, since it makes specific, checkable claims rather than staying generic: + +| Claim (`SECURITY.md`) | What was actually found | +|---|---| +| "Input Sanitization: All user inputs are sanitized to prevent XSS attacks" | Real. `BaseGenerator.sanitizeString` HTML-entity-escapes (`&`, `<`, `>`, `"`, `'`) every string value before it's embedded into generated code, applied recursively via `sanitizeForCodeGeneration` across the whole data/options object. This claim holds up. | +| "File Upload Validation: Strict file type and size validation for uploads" | Real. `Generator.tsx` checks both MIME type/extension (rejecting anything but `.csv`) and a 10MB size cap before accepting an uploaded file — matches `DataInput.tsx`'s "max 10MB" UI copy exactly. | +| "Content Security Policy: CSP headers to prevent code injection" | Not backed. No `[[headers]]` block in `netlify.toml`, no `` in `index.html`, no CSP configuration found anywhere in this repository. | +| "Dependency Management: Regular updates of dependencies with security patches" | Partially backed by process, not by a schedule: `ci.yml`'s `security` job runs `npm audit` on every push/PR (real, and blocking — see ChallengeCI), which would surface a new advisory quickly. There's no Dependabot or Renovate configuration (no `.github/dependabot.yml`) automating the updates themselves, though — the audit finds problems; nothing here fixes them automatically. | +| "Secure Defaults: Secure configuration by default" | Too generic to check against anything specific. | +| "HTTPS Enforcement: All traffic encrypted in transit" | Real in practice — Netlify serves everything over HTTPS by default — but that's Netlify's platform behavior, not a setting this repository configures or could disable if it wanted to. Reasonable to claim, worth knowing it's inherited rather than enforced by this codebase. | + +`README.md` carries a `Security: SOC 2` badge linking to +`netlify.com/security` — that's **Netlify's own SOC 2 Type II report about +its hosting platform**, not an audit of this application's code or +generated output. Presented alongside this project's own badges without +that distinction, a reader would reasonably assume it describes this +project. See ChallengeUI for the parallel check on the accessibility badge. + +## Evidence + +- Pipeline output produced on every run: scan results, test reports, and a + component inventory. +- A traceability record mapping each capability to its implementation and test. +- A reviewable record of the deployed configuration, so the boundary can be + checked rather than taken on description. + +CI produces real scan output (`npm audit`, on every run — see ChallengeCI) +and a real test run (16 unit tests in `dataTransform.test.ts` — see +ChallengeTDD), which is more than several other repos in this suite +produce. No component inventory (SBOM) and no traceability record exist. + +## Review checklist + +- Does the SSP describe the system as it is now, or as it was at the last + release? +- Is every control claim backed by something a reader can open? +- Does the boundary in the SSP match what is actually provisioned? +- Has the data model started handling personal information the PIA does not + mention? +- **A weakness, POA&M item, or vulnerability goes into a security document only + with explicit approval.** Cataloguing theoretical gaps manufactures a record + of insecurity. State what is implemented; independent assessment produces the + authoritative findings. This repository has no security document to add a + finding to — the table above states what was independently checked + against the actual code, not a fabricated POA&M. +- **This repository-specific:** before adding a new claim to `SECURITY.md` + or a new badge to `README.md`, can it be traced to real, checkable code + or configuration in this repository — and if it describes a third + party's own certification (like Netlify's SOC 2), is that attribution + clear rather than implied to be this project's own? diff --git a/.challengeai/challenge-cd.md b/.challengeai/challenge-cd.md new file mode 100644 index 0000000..adc8915 --- /dev/null +++ b/.challengeai/challenge-cd.md @@ -0,0 +1,101 @@ +# ChallengeCD + +Deployment: whether a release can be made safely and undone quickly. + +## Covers + +Rollout safety, rollback readiness, secrets handling, and the operational +controls around putting a change in front of users. + +## The requirement + +SP 800-53 Rev. 5 Configuration Management (CM-3 change control, CM-5 access +restrictions for change) and Contingency Planning (CP-10 system recovery). An +authorized system must be able to show that changes are controlled and that a +bad change can be reversed. + +## Know where the risk is actually taken + +Every deployment model takes the risk somewhere. What matters is that everyone +knows where, because a branch treated as a safety gate that does not gate +anything is worse than no gate: it produces confidence without protection. + +Where a promotion redeploys code that is already live, the promotion names a +version rather than de-risking anything, and the risk was taken earlier. + +## Rollout and rollback + +- **Health-gated rollout**, so a release that never passes its health check + reverses itself rather than waiting to be noticed. +- **Rollback documented and rehearsed.** A procedure that has never been run is + a hypothesis. +- **Credentials from short-lived federated identity**, so no long-lived key + exists to leak or rotate. + +A change that cannot be rolled back without also reversing a data migration is +a different class of change, and saying so at review time is the point. + +## In this repository + +Deployment is Netlify's own GitHub App integration — a push to `main` +triggers a Netlify build and deploy directly, entirely outside GitHub +Actions (`netlify.toml` supplies the build command and publish directory +only; there's no deploy step in `.github/workflows/ci.yml` — see +ChallengeCI). There's no health-gated rollout: Netlify's default model +replaces the live site once the build succeeds, with no automated check of +the deployed result. Rollback is whatever Netlify's dashboard provides +(redeploying a previous build), undocumented as a procedure anywhere in +this repository. + +The one genuinely clean fact here: `netlify.toml` needs no build-time +secrets at all, and none are configured. A repository-wide search confirms +zero `import.meta.env`/`process.env` references anywhere in `src/`, and no +`.env` or `.env.example` file exists. There's nothing to leak because +nothing is configured that could leak — a real, structural advantage of a +backend-less, static tool, worth crediting explicitly since it's the +opposite finding from other repos in this suite. + +## Secrets + +Secrets reach the running system at start from a secret store, never from the +repository and never from a build artifact. The repository is scanned on every +push and on a schedule, and a verified finding fails the build. + +A value inlined at build time is baked into whatever ships and is readable by +anyone holding the artifact. That can be an acceptable trade, but it is a +decision to make deliberately rather than discover. + +Not applicable — see above. This system holds no secret of any kind, so +there's nothing for this section to review beyond confirming that absence, +which was checked directly rather than assumed. + +## Evidence + +- Deploy runs recorded per change and retained. +- A release runbook carrying the verification commands and the rollback + procedure. +- Whatever the platform keeps as the previous good revision is the rollback + record. + +Netlify retains its own deploy history (visible in its dashboard, not in +this repository), which functions as the rollback record by platform +default. No release runbook exists in this repository. + +## Review checklist + +- Does the deploy watch its own run with a failure exit status? A watch that + returns success whichever way the run ended will let a failed deploy be tagged + as a release. +- Is the run identified by commit rather than by branch? A branch filter asked + seconds after a merge returns the previous run, which is already green. +- Can this change be rolled back without a data migration being reversed? If + not, say so in the pull request. +- Does anything new read a secret at build time rather than at run time? +- **This repository-specific:** if a future feature introduces any + environment variable at all (an API key for a new integration, say), + does it get documented in a new `.env.example`, and — since this app has + no backend today (see ChallengeIaC, ChallengeAPI) — is a server-side + component added at the same time to hold it, rather than reading it + directly in client code the way a purely static app would default to? + It's easier to establish that habit before there's a real secret to + protect than after. diff --git a/.challengeai/challenge-ci.md b/.challengeai/challenge-ci.md new file mode 100644 index 0000000..4e4f5dc --- /dev/null +++ b/.challengeai/challenge-ci.md @@ -0,0 +1,103 @@ +# ChallengeCI + +The pipeline: what it proves, and whether it can be bypassed. + +## Covers + +Workflow hardening, evidence gates, security scanning, and the release controls +that make a merge into a protected branch mean something. + +## The requirement + +SP 800-53 Rev. 5 control families, principally Configuration Management (CM), +System and Information Integrity (SI), and Risk Assessment (RA). An assessor +asks how change is controlled and how flaws are found; a pipeline that gates on +both answers the question with artifacts instead of assertions. + +## Gates and reports + +A gate blocks a merge. A report informs one. Both are useful and they are not +interchangeable, so which is which is a decision rather than an accident of +configuration. + +The gates worth having cover: that the code compiles, conforms and behaves; that +the artifact can actually be produced; that the system works end to end; that +every rendered route passes accessibility; that no dependency carries a high or +critical advisory; and that no verified secret reached the history. + +Which of those block, and which report, is declared in `profile.yml` so the +answer is written down rather than inferred from workflow files. + +## Hardening + +- Workflow permissions are least privilege, declared per workflow rather than + inherited. +- Actions are pinned, and raised on a schedule. +- Deploy credentials come from short-lived federated identity rather than a + long-lived key held in the repository or in secrets. +- The pipeline runs on pull requests from forks without secrets in scope. + +## In this repository + +One workflow, `.github/workflows/ci.yml`, two jobs, both running on every +push and PR to `main` and `dev`: + +- **`test`** — lint, unit tests (`vitest run`), and build, run across two + Node versions (20.x, 22.x). No `continue-on-error` anywhere, so a red + result on any step fails the job. The build artifact (`dist/`) is + retained for 7 days, but only from the 20.x matrix leg. +- **`security`** — `npm audit --audit-level=moderate`, then + `npm audit --audit-level=high --production`. Also no + `continue-on-error` — **this is genuinely stronger than most repos in + this suite**, where a dependency scan is typically advisory + (`continue-on-error: true`). Here, a moderate-or-worse advisory in any + dependency (dev included, from the first `audit` call) fails the job. + +Against this tool's gate list: compile/conform/behave is covered (lint, +test, build). No end-to-end test exists (see ChallengeTDD). No accessibility +check runs despite the README's WCAG badge (see ChallengeUI). The +dependency scan is real and blocking, which covers the "no dependency +carries a high or critical advisory" gate better than most sibling repos. +No secret-scanning tool (gitleaks, trufflehog, GitHub secret scanning +verification) was found configured in this workflow. + +There is no deploy step in this workflow at all — deployment is Netlify's +own GitHub-integration auto-deploy, entirely outside GitHub Actions (see +ChallengeCD), so this workflow never handles a deploy credential. + +Against the hardening checklist specifically: + +- **Workflow permissions:** neither job declares a `permissions:` block — + both run under whatever the repository's default `GITHUB_TOKEN` + permissions are, not an explicit least-privilege grant. +- **Action pinning:** `actions/checkout@v4`, `actions/setup-node@v4`, + `actions/upload-artifact@v4` are pinned to major-version tags, not commit + SHAs. No Dependabot configuration exists to raise them on a schedule (no + `.github/dependabot.yml`). +- **Deploy credentials:** not applicable to this workflow — there is none + here (see above and ChallengeCD). + +## Evidence + +Each run uploads its reports, and they are retained long enough to be asked for. +Coverage is surfaced on the change itself rather than only in a log. + +The `test` job's `dist/` artifact (7-day retention, 20.x leg only) is the +only retained artifact. Neither job uploads a test report, a coverage +report (none is generated — see ChallengeTDD), or an audit report; both +results are visible only in the raw Actions log. + +## Review checklist + +- Does a new job have wider permissions than it needs? +- Is a new action pinned? +- Did a required check get renamed? The name is what branch protection matches, + so renaming one silently stops it gating. +- Does a job that cannot fail still report, so a required check is satisfied + rather than left pending forever? +- Does the gate list in `profile.yml` still match what the pipeline runs? +- **This repository-specific:** does a new dependency introduce a + moderate-or-worse advisory? The `security` job would fail on it today — + confirm that's still true after any future change to the audit-level + flags, since loosening them silently weakens the one blocking security + gate this repository has. diff --git a/.challengeai/challenge-cli.md b/.challengeai/challenge-cli.md new file mode 100644 index 0000000..62a210a --- /dev/null +++ b/.challengeai/challenge-cli.md @@ -0,0 +1,59 @@ +# ChallengeCLI + +The accelerator itself: the layer that carries the standards in this folder into +Claude and Codex so the work arrives shaped by them. + +## Covers + +Cross-runtime operation. The same guidance drives both runtimes, which is why +the federal layer lives in `.challengeai/` and the agent files point at it +rather than restating it. + +## The requirement + +None directly. ChallengeCLI is delivery tooling, and no federal authority +mandates it. It exists so the requirements the other eleven tools cover are +applied while code is written rather than discovered during assessment. + +MetaPhase governs the suite under ISO/IEC 42001, the management-system standard +for artificial intelligence, which is what makes its use in federal delivery +defensible. + +## One source, two runtimes + +Guidance duplicated per runtime drifts, and drift is worse than absence: two +agents then follow two different rule sets while both appear governed. The +federal layer therefore has one home, and each runtime's entry file references +it instead of copying it. + +Where a repository maintains parallel agent files, they are kept in agreement +and that agreement is worth enforcing mechanically rather than by habit. + +## In this repository + +There is no `AGENTS.md` and no `CLAUDE.md` anywhere in this repository as of +this writing — no runtime entry point of any kind, for either Claude Code or +Codex. An agent started directly in this repo today discovers this +`.challengeai/` folder only by being told to look for it or by browsing the +file tree; nothing points here automatically. `README.md` and +`CONTRIBUTING.md` are the only onboarding documents, and neither mentions +ChallengeAI or `.challengeai/`. This is a real, named gap against this +tool's own "one source, two runtimes" principle, worth closing with a root +`AGENTS.md`/`CLAUDE.md` pair as a deliberate maintainer decision — not +something this documentation pass adds unasked. + +## Evidence + +The folder is the evidence. Someone reading `.challengeai/` can see what the +team was held to without interviewing anyone. + +## Review checklist + +- Do the parallel agent files still agree with each other? +- Is anything here duplicated into a runtime-specific location, where the two + copies will drift? +- Has a repository-specific detail leaked into a tool file? It belongs in + `profile.yml`, this file's `In this repository` section, or the repository's + own documentation. +- Does user-facing copy describe ChallengeAI as a feature of the product? It is + how the product was built, and saying otherwise is wrong. diff --git a/.challengeai/challenge-ea.md b/.challengeai/challenge-ea.md new file mode 100644 index 0000000..7b29bd7 --- /dev/null +++ b/.challengeai/challenge-ea.md @@ -0,0 +1,124 @@ +# ChallengeEA + +Enterprise architecture: whether the system fits the environment it has to live +in. + +## Covers + +Federal architecture alignment, governance rigor, interoperability with agency +systems, and the traceability between a mission need and a technical choice. + +## The requirement + +- **The Clinger-Cohen Act** requires agencies to manage IT as a capital + investment, with architecture as part of that discipline. +- **OMB Circular A-130** sets expectations for managing federal information + resources. +- **The Federal Enterprise Architecture Framework** provides the reference + models an agency maps its systems against. +- **The Federal Source Code Policy** governs custom-developed code, including + reuse and, where applicable, release. + +## Reasoning travels with the component + +An architecture record that captures only the decision leaves the next team to +rediscover the constraint that produced it, and rediscovery usually happens by +reversing the decision and hitting the constraint again. + +So each significant choice carries the alternatives that were considered and why +they were not taken, kept next to the description of the component rather than +in a separate decision archive, where somebody changing it will actually +encounter it. + +## In this repository + +There's no `docs/architecture/` or equivalent; `README.md`'s "Project +Structure" section is the closest thing to an architecture record, and it's +accurate as far as it goes (see ChallengeCI, ChallengeUI for where its other +claims fall short). + +The actual shape: a fully client-side React SPA with no backend of any kind +(see ChallengeIaC, ChallengeSQL). The generator's core abstraction is +`src/services/CodeGenerator.ts` dispatching to one of four output-format +generators (`JavaScriptEmbedGenerator`, `StaticHTMLGenerator`, +`DrupalBlockGenerator`, `DrupalControllerGenerator`), all extending a shared +`BaseGenerator` that centralizes HTML-entity sanitization (see ChallengeUI's +sanitization note, ChallengeAPI). A distinct, worth-naming architectural +fact: `src/data/libraries.json` lists six charting/mapping libraries the +generator can target (Chart.js, D3.js, Highcharts, Apache ECharts, +OpenLayers, Leaflet), but only two of them — Chart.js and D3 — are actual +npm dependencies used for the in-app live preview. The other four are +code-generation targets only: the app emits template code referencing them +(typically loaded via CDN in the generated output) without installing, +bundling, or executing them itself. That's a real, deliberate design choice +— it's why `package.json` doesn't list Highcharts/ECharts/OpenLayers/Leaflet +as dependencies despite the README and this system's own `libraries.json` +naming all six as "supported" — but it isn't written down anywhere as a +decision with its reasoning; this file is the first place it's stated +explicitly. + +**One specific duplication worth naming as a "reasoning travels with the +component" case study:** the sanitization logic that closes the XSS risk in +generated code (`sanitizeString`/`sanitizeForCodeGeneration`) exists as +byte-for-byte identical, separately maintained copies in +`BaseGenerator.ts` and directly inline in `Generator.tsx` — not shared via +import. A future fix to one (an escaped character added, a bug found) has +no mechanism forcing it into the other; the two will drift silently unless +someone remembers both exist. See ChallengeAPI for why this specific logic +is security-critical. + +No alternatives-considered record exists for any architectural choice here — +this section is reconstructed from the code, not read from a design +document. + +There is no agency mission need this traces to, and no live integration +with any agency system. + +## Boundaries the architecture has to respect + +- **The authorization boundary** is what the system provisions and controls. A + component added outside it changes the security posture and the + documentation that describes it. +- **Data stays inside the provider boundary** unless a deliberate decision says + otherwise, and that decision is recorded with its reasoning. +- **Services are chosen from what is authorized** at the required impact level, + checked before adoption. + +Not applicable in the ATO sense — see ChallengeIaC for what this system +actually runs on (a static site on Netlify, no authorization boundary of its +own). + +## Interoperability + +Where the system exchanges data with an agency system, the interface is +documented as a contract with a version, and the failure behaviour is +specified. An integration whose failure mode is unspecified becomes an +incident rather than a degraded state. + +No live integration with any external system exists — the app processes +user-supplied CSV/JSON entirely client-side and produces code as output; it +doesn't call out to any API at runtime. + +## Evidence + +- Architecture documentation carrying the design and its reasoning. +- A reviewable record of what is actually provisioned. +- The published interface contract. + +None of the three exists as a dedicated document; the code itself and this +tool file are the closest things to evidence. + +## Review checklist + +- Does this choice have its reasoning recorded next to it? +- Were alternatives considered, and is the reason for not taking them written + down? +- Does a new component sit inside the authorization boundary? +- Does data leave the provider boundary, and was that decided or assumed? +- Does a new integration specify what happens when the other side is down? +- **This repository-specific:** does adding a new output-format generator + extend `BaseGenerator` and route its output through the shared + sanitization path, the way all four existing generators do? A generator + written outside that pattern would reintroduce the XSS risk + `sanitizeForCodeGeneration` exists specifically to close — see + ChallengeAPI. diff --git a/.challengeai/challenge-iac.md b/.challengeai/challenge-iac.md new file mode 100644 index 0000000..819d250 --- /dev/null +++ b/.challengeai/challenge-iac.md @@ -0,0 +1,99 @@ +# ChallengeIaC + +Infrastructure: what is provisioned, what it costs, and where the authorization +boundary falls. + +## Covers + +Infrastructure as code, FedRAMP service selection, the authorization boundary, +and the cost consequences of a topology. + +## The requirement + +- **FedRAMP** authorizes cloud service offerings. Using an authorized service at + the required impact level lets the system inherit the controls that service + already satisfies. +- **SP 800-53 Rev. 5** Configuration Management, applied to infrastructure: the + deployed configuration has to be reviewable and controlled. + +Inheritance is only valid for services inside the authorized boundary at the +authorized level. Checking that before adopting a service is a design step, not +an assessment finding. + +## Declared, not clicked + +Infrastructure that exists because someone configured it by hand is +infrastructure nobody can review, reproduce, or diff. Declaring it makes the +deployed state readable, and makes a change to it something that can be +approved. + +The declaration is checked for syntax and validity as a gate, and the same +declaration is what gets applied, so the reviewed configuration and the running +one are the same artifact. + +## In this repository + +This is the simplest topology in this suite: a static site, built by Vite, +served by Netlify. `netlify.toml` is the entire infrastructure declaration +— a build command, a publish directory, and a single SPA-fallback redirect +rule. No IaC tool of any kind is in use (no Terraform, Bicep, CloudFormation), +and none is needed at this scale — there's no compute to provision beyond +what Netlify's build system already manages, no database, no queue, no +network boundary this repository defines. + +FedRAMP authorization doesn't bind this choice: this system carries no +federal data on an agency's behalf (see ChallengeATO, federal-context.md), +so Netlify's own authorization status is not a live constraint here the way +it would be for a system actually processing federal information. + +## Boundary + +What the system provisions is the boundary. Everything else is inherited from +the provider and is named as inherited rather than claimed as implemented. + +This system provisions nothing beyond what `netlify.toml` declares. The +custom domain (`drupaldata.dev`) and its DNS configuration live entirely in +Netlify's own dashboard, not in this repository. + +## Cost + +Cost-relevant settings are variables, each carrying its reasoning. A default was +usually chosen against a measurement, and keeping the reasoning next to it means +the next person changes it knowingly. + +Topology drives cost more than instance sizing does, and the expensive parts are +usually the ones added without being noticed: an always-on gateway, a +cross-region transfer, a log stream with no retention. + +Nothing here is a cost-relevant variable — a static site with no backend, no +database, and no per-request compute has essentially no cost surface beyond +Netlify's own bandwidth/build-minute pricing, which this repository doesn't +configure or control. + +## Evidence + +- The infrastructure declaration is the record of what is provisioned. +- Validation results from the pipeline. +- Variables and their reasoning. + +`netlify.toml` is real, minimal, and complete evidence of what this +repository provisions — there's genuinely little more to declare at this +system's scale. Nothing in CI validates `netlify.toml` directly; it's only +exercised implicitly by Netlify's own build succeeding or failing at deploy +time. + +## Review checklist + +- Is this service authorized at the required impact level? +- Does this change move anything across the authorization boundary? +- Is a cost-relevant setting hardcoded rather than declared as a variable with + its reasoning? +- Does the running configuration still match what is declared? +- Is a new resource missing a retention or lifecycle setting, leaving it at the + provider default? +- **This repository-specific:** if this system ever grows a backend (see + ChallengeCD's note on future secrets), does that change also add the IaC + declaration this simple static-site topology has never needed? A backend + added the same way the frontend was — configured by hand in a provider + dashboard — reintroduces exactly the "clicked, not declared" gap this + tool file currently has nothing to point at. diff --git a/.challengeai/challenge-melt.md b/.challengeai/challenge-melt.md new file mode 100644 index 0000000..07b9248 --- /dev/null +++ b/.challengeai/challenge-melt.md @@ -0,0 +1,89 @@ +# ChallengeMELT + +Metrics, events, logs, and traces: whether an operator can tell what happened. + +## Covers + +Observability coverage, alerting, audit logging, retention, and runbooks. + +## The requirement + +- **SP 800-53 Rev. 5** Audit and Accountability (AU) is the core family: what is + recorded, protected, retained, and reviewed. +- **The Federal Records Act** governs retention and disposition. Records are kept + on a schedule rather than until storage becomes inconvenient. +- **Incident Response (IR)** depends on this: an incident that cannot be + reconstructed cannot be reported accurately. + +## What gets recorded + +- **Audit records are tamper evident and retained on a schedule**, not rotated + by size. They exist to be read later by somebody investigating something. +- **Application logs carry no personal information and no secrets.** A log line + is a permanent record in an environment where records are discoverable. +- **Absence is monitored as well as failure.** A scheduled job that stops + running produces no errors at all, so the alarm is on the job not having run + rather than on it having failed. + +## Alerting + +An alarm fires on a condition an operator can act on. An alarm nobody acts on +trains people to ignore alarms, which leaves the system worse off than having no +alarm at all. + +Every alarm therefore has an action attached, and that action lives in a runbook +carrying the actual commands rather than describing them. + +## In this repository + +There is no logging, metrics, or tracing of any kind — no server exists to +log anything (see ChallengeIaC, ChallengeSQL), and the client-side app +itself doesn't call out to any logging or analytics service. A +repository-wide search confirms no analytics script, no error-tracking SDK +(Sentry or similar), and no telemetry call anywhere in `src/` or +`index.html`. + +That's structurally appropriate for what this system is: nothing happens +here that would need reconstructing after an incident in the SP 800-53 AU +sense — no accounts to compromise, no data at rest to exfiltrate, no +server-side process to fail silently. The one place an operator might +actually want visibility — whether the client-side sanitization is ever +bypassed, or whether a generated-code download later turns out to have +carried unescaped input — has no instrumentation at all, but that's a +testing gap (see ChallengeTDD) more than an observability one, since +there's no runtime for a log line to exist in. + +There is no scheduled job in this repository, so "absence is monitored" has +nothing to apply to. + +## Retention + +Retention is a decision with a records-schedule basis, recorded where the log +is configured. Changing it is a compliance change, not a cost optimization. + +Not applicable — no log stream exists to have a retention setting. The one +retained artifact in this repository, the CI build output +(`dist/`, 7 days — see ChallengeCI), is a build artifact, not a log. + +## Evidence + +- Alarm definitions declared as code, so they are reviewable. +- Runbooks versioned with the system. +- Retention declared rather than left at a provider default. + +None of the three exists, and none is currently warranted given what this +system actually does. + +## Review checklist + +- Does a new log line carry anything personal, or any secret? +- Does a new scheduled task have an alarm on it not running? +- Does a new alarm have an action, or does it only notify? +- Is retention on a new log stream declared? +- Would this incident be reconstructable from what is recorded today? +- **This repository-specific:** if a future backend is added (see + ChallengeCD, ChallengeIaC's forward-looking notes), does it come with + real logging from day one, rather than shipping the same + "nothing is logged" default that's appropriate for today's fully + client-side app but wouldn't be once a server exists to have something + go wrong on? diff --git a/.challengeai/challenge-sql.md b/.challengeai/challenge-sql.md new file mode 100644 index 0000000..66a90dd --- /dev/null +++ b/.challengeai/challenge-sql.md @@ -0,0 +1,89 @@ +# ChallengeSQL + +The data layer: correctness, performance, and who can reach what. + +## Covers + +Schema design, query correctness and performance, migrations, and database-level +access control. + +## The requirement + +SP 800-53 Rev. 5 Access Control (AC), Audit and Accountability (AU), and System +and Communications Protection (SC-28, protection at rest). Where the data +concerns individuals, the Privacy Act and the PIA govern what may be stored and +for how long. + +## Access + +- **Reached through a server-side layer.** The browser holds no database + credential and issues no query. +- **Least privilege by role.** The application connects as a role holding narrow + per-table grants rather than as the owner, and something asserts those grants + so a migration that widens access fails rather than passing silently. + + Build this early. A role added after the schema has grown means auditing every + table to work out what the application actually needs. +- **Row-level security enabled and forced** on tables that carry it, so the + owner cannot bypass the policy by accident. + +## Migrations + +Migrations are checksummed, and the runner records a hash of each file. + +**An applied migration is never edited, comments included.** Changing one makes +the runner report a mismatch on every subsequent run, and the fix for a bad +migration is another migration. + +## Performance + +Queries that touch a growing table are checked against a plan rather than +against intuition. An index helps only when the planner chooses it, and a join, +a function on a column, or a mismatched type will each quietly defeat one. + +## In this repository + +There is no database of any kind, and no server to hold one against — this +is a fully static, client-only SPA (see ChallengeIaC). Confirmed by a +repository-wide search: no database client, ORM, or connection string +anywhere in the code. + +The only "data" this system handles: + +- **User-supplied CSV/JSON**, uploaded or pasted directly in the browser, + parsed client-side (`papaparse`), held only in React component state + (`Generator.tsx`), and never sent anywhere — the app has no server to send + it to. It disappears when the tab closes or the page reloads. +- **Static configuration files committed to the repo** + (`src/data/chartStyles.json`, `chartTypes.json`, `libraries.json`, + `sampleData/`, `visualizationTypes/`) — read-only, shipped with the + build, carrying no user or personal data. + +Every requirement in this tool file's "Access," "Migrations," and +"Performance" sections is inapplicable for the same reason: there's no +database role to scope, no migration history to checksum, no query plan to +check. This is a genuine, structural non-applicability, not an unfilled gap. + +## Evidence + +- The migration history is the schema history. +- The grants held by the application role, and whatever asserts them, are the + access-control evidence. +- Database tests run against a real migrated database rather than a mock. + +None of this applies. If persistence is ever added to this system, this +tool file's requirements become live and need rewriting from an actual +schema — not before. + +## Review checklist + +- Does this migration edit one that has already been applied? +- Does a new table need row-level security, and is it forced as well as enabled? +- Does the application role get the narrowest grant that works? +- Does a new query have a plan that uses the index it was written for? +- Does a new column hold personal information the PIA does not mention? +- **This repository-specific:** if a future version of this tool adds + server-side persistence (saved visualizations, user accounts, anything), + does `profile.yml`'s `privacy_assessment_required: false` get revisited + in the same change, rather than left stale? Today it's accurate because + nothing is stored anywhere. diff --git a/.challengeai/challenge-tdd.md b/.challengeai/challenge-tdd.md new file mode 100644 index 0000000..3f12715 --- /dev/null +++ b/.challengeai/challenge-tdd.md @@ -0,0 +1,90 @@ +# ChallengeTDD + +The tests: what they prove, and what a green run actually means. + +## Covers + +Test traceability to capabilities, coverage of the paths that matter, and +release readiness. + +## The requirement + +SP 800-53 Rev. 5 System and Services Acquisition (SA-11, developer testing) and +System and Information Integrity (SI-2, flaw remediation). An assessor asks how +the team knows the system works, and expects an answer with artifacts. + +## Breadth, then depth + +Breadth across every kind of test comes first, so no category is missing: logic +in isolation, rendered behaviour, accessibility per route, real queries against +a real migrated database, route behaviour including failure cases, and the whole +system end to end at more than one viewport. + +Depth follows risk. The paths where a defect changes what a user is shown, or +lets someone reach data they should not, earn the most. + +## What a test should assert + +A test that passes against a deliberately broken implementation is worse than no +test, because it reports confidence it has not earned. When adding a regression +test, break the fix on purpose and confirm the test fails. + +A test asserting an implementation detail rather than a behaviour will break on +a harmless refactor and be deleted by whoever it inconveniences, which costs the +coverage it was written for. + +## Traceability + +A capability with no test named against it is either untested or untraceable, +and both are findings. The traceability record maps each capability to its +implementation and its test, which is what an assessor asks for. + +## In this repository + +There's exactly one test file — `src/utils/dataTransform.test.ts`, 16 real +`it()` cases across three `describe` blocks (validation helpers, conversion +helpers, and `convertDataForLibrary`) — covering GeoJSON/Chart.js format +validation and data conversion between chart library formats (Highcharts, +ECharts, D3). It's real, targeted testing of genuinely tricky logic (format +conversion has a lot of edge cases), and it runs in CI on every push/PR +across two Node versions with no `continue-on-error` (see ChallengeCI) — a +real, enforced gate, better than most repos in this suite have. + +**What it doesn't cover is the one place this repository has an actual +security control:** `sanitizeString`/`sanitizeForCodeGeneration` — the +HTML-entity-escaping that prevents XSS in generated code output (see +ChallengeAPI, ChallengeEA) — has zero test coverage, in either of its two +duplicated implementations (`BaseGenerator.ts` and `Generator.tsx`). No test +asserts that a malicious string (`&...` — makes that markup live in the downloaded HTML. Real, working XSS via a URL parameter this claim says is covered. See ChallengeAPI and ChallengeEA for the fix this points to. | +| "File Upload Validation: Strict file type and size validation for uploads" | Partially true. The 10MB size cap is real and strictly enforced. The type check is weaker than "strict," though: `Generator.tsx` accepts a file if its extension is `.csv` **or** its reported MIME type is `text/csv`, `application/csv`, **or `text/plain`** — rejection only happens if both checks fail. `text/plain` is what browsers report for a wide range of non-CSV files, so a non-CSV file reported as `text/plain` passes regardless of extension, and any file merely named `something.csv` passes regardless of its actual MIME type or content. Not "strict," and not "requires both" — it's an either/or check with a permissive MIME option. | | "Content Security Policy: CSP headers to prevent code injection" | Not backed. No `[[headers]]` block in `netlify.toml`, no `` in `index.html`, no CSP configuration found anywhere in this repository. | | "Dependency Management: Regular updates of dependencies with security patches" | Partially backed by process, not by a schedule: `ci.yml`'s `security` job runs `npm audit` on every push/PR (real, and blocking — see ChallengeCI), which would surface a new advisory quickly. There's no Dependabot or Renovate configuration (no `.github/dependabot.yml`) automating the updates themselves, though — the audit finds problems; nothing here fixes them automatically. | | "Secure Defaults: Secure configuration by default" | Too generic to check against anything specific. | diff --git a/.challengeai/challenge-ci.md b/.challengeai/challenge-ci.md index 4e4f5dc..0a7c278 100644 --- a/.challengeai/challenge-ci.md +++ b/.challengeai/challenge-ci.md @@ -56,10 +56,15 @@ push and PR to `main` and `dev`: Against this tool's gate list: compile/conform/behave is covered (lint, test, build). No end-to-end test exists (see ChallengeTDD). No accessibility check runs despite the README's WCAG badge (see ChallengeUI). The -dependency scan is real and blocking, which covers the "no dependency -carries a high or critical advisory" gate better than most sibling repos. -No secret-scanning tool (gitleaks, trufflehog, GitHub secret scanning -verification) was found configured in this workflow. +dependency scan fails its own workflow run on a moderate-or-worse advisory +(no `continue-on-error`), which is a stronger *workflow-level* posture than +most sibling repos have — but "fails the workflow run" and "required to +merge" are different facts, and no branch-protection setting was found +confirming this job actually blocks a merge (see +`profile.yml`'s `documented_but_unverified_branch_protection`). Described +as a failing check, not a confirmed merge gate. No secret-scanning tool +(gitleaks, trufflehog, GitHub secret scanning verification) was found +configured in this workflow. There is no deploy step in this workflow at all — deployment is Netlify's own GitHub-integration auto-deploy, entirely outside GitHub Actions (see diff --git a/.challengeai/challenge-ea.md b/.challengeai/challenge-ea.md index 7b29bd7..96e8a92 100644 --- a/.challengeai/challenge-ea.md +++ b/.challengeai/challenge-ea.md @@ -42,20 +42,26 @@ The actual shape: a fully client-side React SPA with no backend of any kind `src/services/CodeGenerator.ts` dispatching to one of four output-format generators (`JavaScriptEmbedGenerator`, `StaticHTMLGenerator`, `DrupalBlockGenerator`, `DrupalControllerGenerator`), all extending a shared -`BaseGenerator` that centralizes HTML-entity sanitization (see ChallengeUI's -sanitization note, ChallengeAPI). A distinct, worth-naming architectural -fact: `src/data/libraries.json` lists six charting/mapping libraries the +`BaseGenerator` that centralizes HTML-entity sanitization (see ChallengeAPI +for where that sanitization does and doesn't reach). A distinct, +worth-naming architectural fact, corrected after an initial read got this +wrong: `src/data/libraries.json` lists six charting/mapping libraries the generator can target (Chart.js, D3.js, Highcharts, Apache ECharts, -OpenLayers, Leaflet), but only two of them — Chart.js and D3 — are actual -npm dependencies used for the in-app live preview. The other four are -code-generation targets only: the app emits template code referencing them -(typically loaded via CDN in the generated output) without installing, -bundling, or executing them itself. That's a real, deliberate design choice -— it's why `package.json` doesn't list Highcharts/ECharts/OpenLayers/Leaflet -as dependencies despite the README and this system's own `libraries.json` -naming all six as "supported" — but it isn't written down anywhere as a -decision with its reasoning; this file is the first place it's stated -explicitly. +OpenLayers, Leaflet). Only Chart.js is npm-bundled — every other one, +**D3 included**, is dynamically loaded and *executed* at runtime via a +CDN `