diff --git a/.agents/tasks/maven-central-throttling.md b/.agents/tasks/maven-central-throttling.md new file mode 100644 index 00000000..48bf037e --- /dev/null +++ b/.agents/tasks/maven-central-throttling.md @@ -0,0 +1,254 @@ +--- +slug: maven-central-throttling +branch: maven-central-throttling +owner: unassigned +status: draft +started: 2026-08-20 +related-memories: + - projectbuilder-in-memory-caches # lives in `core-jvm-compiler`; promote to `.agents/shared` +--- + +## Goal + +No Spine SDK build — local or CI — fails because Maven Central refuses to +serve us. Remote artifact traffic becomes *per machine, once* instead of +*per build run*, so that our aggregate footprint stays far below Sonatype's +consumption limits, and a throttled repository can no longer abort a build +whose artifacts are already on disk. + +## Context + +### What changed upstream + +In May 2026 Sonatype tightened the consumption limits of Maven Central. +Traffic is now measured per organisation / network / egress address, and +the highest-volume consumers receive `429 Too Many Requests` — CI runner +pools also see `403 Forbidden`. Per the [Sonatype FAQ][faq]: + +- During a block **every** request from that egress fails, including + artifacts that would otherwise be served from the CDN edge cache. +- Repeated requests **extend** the block, up to 24 hours. Retrying harder + is the one reaction guaranteed to make it worse. +- Ephemeral environments and tools that bypass dependency caches are named + explicitly as the traffic pattern that triggers enforcement. + +This is an industry-wide adjustment, not an outage and not a Spine problem: +Gradle, Trivy, Renovate, and TeamCity all carry public issues about it, and +Bitrise now runs dedicated Gradle mirrors for its customers. + +### What we observed (2026-08-19/20, `core-jvm-compiler`) + +- **Local.** `:gradle-plugin:test` failed 17 specs; every failure bottomed + out in `429` for a single POM. The block covered the whole machine — + unrelated builds included — and lasted about three hours. +- **CI.** `Build on Windows` failed 69 seconds in, resolving + `kotlin-stdlib` for the `:buildSrc` classpath: `403 Forbidden` from + `repo.maven.apache.org`, before a line of project code ran. A re-run on + a different runner passed. +- `status.maven.org` reported all systems operational throughout — the + block was ours, and the diagnosis has to start with that assumption. + +### Why our builds are exposed + +Four independent causes, each fixable on its own: + +1. **`ProjectBuilder` cannot reuse the dependency cache — ever.** + `ProjectBuilderImpl` builds its services from `TestGlobalScopeServices`, + which overrides `createCacheFactory` to return + `TestInMemoryCacheFactory`. Every "persistent" cache, the module + metadata store included, is an empty in-memory map in each test JVM. + `withGradleUserHomeDir(...)` does not change this — resolver ids, the + user home, and the on-disk metadata all look correct while resolution + still reports a miss. Any spec that resolves a real dependency graph + therefore re-downloads it in full on **every run**. + +2. **Production plugin code resolves at configuration time.** + Paths such as `CoreJvmCompilerSettings.buildClasspath()` materialise a + project's `compileClasspath` eagerly, which is what drags resolution + into unit tests at all — and, for consumers, into IDE sync. + +3. **A repository *error* aborts resolution.** Unlike a miss, an error + from one repository fails the whole resolution even when an earlier + repository already holds the module. One throttled `mavenCentral()` + entry breaks builds that do not need Central. Several of our TestKit + settings templates also omitted the Spine registry and only worked + because `~/.m2` happened to be warm. + +4. **CI has no consolidation point.** Every job on every runner is a cold + cache facing Central directly, including the `buildSrc` bootstrap that + precedes any project configuration. + +### Reference implementation + +Causes 1 and 3 are already solved in `core-jvm-compiler` (merged in +[core-jvm-compiler#111]). The pattern to generalise: + +| File | Role | +|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------------------| +| `base/src/testFixtures/.../StubResolution.kt` | `forbidNetworkResolution()` puts a stub project in offline mode; `stubRepository` locates the local fixture repo | +| `gradle-plugin/build.gradle.kts` | `stubRepoDeps` configuration + `prepareStubRepo` task mirror the needed artifacts out of the enclosing build's warm cache | +| `gradle-plugin/src/test/.../StandardRepos.kt` | serves that directory first, with `metadataSources { artifact() }` | + +The result was verified under an **active** Central block: the previously +failing specs passed with zero network access. Offline mode is the load- +bearing part — a gap fails loudly with "No cached version available for +offline mode" and names the artifact to add, instead of silently reaching +for the network. + +## Plan + +### Phase 1 — CI stops facing Central cold (`config`) + +- [ ] Audit the JVM workflows `config` distributes: confirm what + `gradle/actions/setup-gradle` caches today and whether the + `buildSrc` bootstrap classpath is covered by it. +- [ ] Evaluate seeding runners from a read-only dependency cache + (`GRADLE_RO_DEP_CACHE`) — designed for exactly this + "ephemeral environment reuses a warm cache" case. +- [ ] Make the re-run policy explicit in the workflow docs: a `403`/`429` + from Central is an infrastructure signal; re-run the job once (a new + runner means a new egress), never loop. +- [ ] (deferred 2026-08-20) Give the Windows CI job a cache to restore from. + Diagnosed and costed; held pending evidence that it is needed. + - `windows-latest` runs in exactly one workflow — `build-on-windows.yml`, + triggered `on: pull_request` only. Every workflow that runs on the + default branch (`build-on-ubuntu`, `publish`, `revalidate-versions`) + is `ubuntu-latest`. + - GitHub Actions caches are readable across branches only from the + default branch, and `gradle/actions/setup-gradle` saves entries only + from the default branch. With no Windows job on `master`, no shared + Windows entry is ever written, so every Windows PR job resolves the + `buildSrc` bootstrap from Central cold. That is precisely the + 2026-08-19 `403` on `kotlin-stdlib`, 69 seconds in. + - Options weighed: (a) a scheduled Windows run on `master` that seeds the + cache and doubles as the safety net; (b) `cache-read-only: false` on the + PR job — same-PR reuse only, and per-PR entries risk LRU-evicting the + Ubuntu entry from the 10 GB repo budget; (c) `GRADLE_RO_DEP_CACHE`, + which still needs a seeded cache and therefore depends on (a); + (d) running Windows CI less often. + - **Decision.** Change nothing for now. `master` builds exist to publish; + their non-publishing part is a safety net, and standing up a Windows + run purely to seed a cache is not worth the spend after a single + incident. Observe CI instead. + - **Re-open when** another Central `403`/`429` fails a CI job. Option (a) + is the first move; the `setup-gradle` read-only default is worth + confirming from a real run's Gradle job summary at that point. + +### Phase 2 — Shared offline stub fixture (`tool-base`) + +- [ ] Promote the `StubResolution` pattern into `plugin-testlib`, so that + `compiler`, `validation`, and `core-jvm-compiler` share one + implementation: offline enforcement, the stub-repository property, + and a reusable `prepareStubRepo`-style task. +- [ ] Document the failure mode in the fixture's KDoc — the + `TestInMemoryCacheFactory` fact is non-obvious and cost this session + several hours to isolate. +- [ ] Migrate `core-jvm-compiler` onto the shared fixture and delete its + local copy. + +#### Considered and set aside: migrating `ProjectBuilder` tests to TestKit + +Examined 2026-08-20 as an alternative to the stub fixture. TestKit would cure +the re-download pathology — it runs a real build with a real on-disk Gradle +user home, so `TestInMemoryCacheFactory` does not apply — but it does not fit +these specs: + +- **CI traffic is not fixed, only moved.** `plugin-testlib` pins the TestKit + dir to `/.gradle-test-kit` (`RootProject.testKitTempDir()`), + which `setup-gradle` does not cache. On an ephemeral runner that cache is + cold every run, so migrated specs would resume per-run full downloads from + Central — through a different directory. Locally it also duplicates the + whole graph once per checkout. +- **Assertion power is lost.** The affected specs are white-box: they assert + on the in-process `Project` model (extensions, task wiring). TestKit is + black-box across the Tooling API process boundary — task outcomes and + output only — and each spec pays real-build startup. +- **No guarantee.** TestKit reduces traffic but leaves it unbounded and + silent; the stub fixture is provably zero-network and fails loudly on + a gap. + +Gradle's docs draw the same line implicitly: `ProjectBuilder` is for +"lightweight, isolated" unit tests, TestKit for behavior "in a real build" — +and say nothing about resolution or caches in `ProjectBuilder` tests, +because in the intended taxonomy such tests never resolve a real graph. +The in-memory-cache behavior is documented nowhere public (searched +2026-08-20); this task's write-up appears to be the only record of it. +Per-test rule going forward: specs asserting on configuration stay on +`ProjectBuilder` + stub fixture; specs needing real build behavior belong +in TestKit anyway. + +### Phase 3 — Repository ordering discipline (`config`, then consumers) + +- [ ] In the shared repository helpers (`standardToSpineSdk()`, + `standardSpineSdkRepositories()`), fix the order as + `mavenLocal()` → Spine registry → Central, so Central is consulted + only for what genuinely lives there. +- [ ] Sweep TestKit settings/build templates across repos for + `pluginManagement` blocks that omit the Spine registry — they resolve + Spine artifacts through Central's 404 path today, and break outright + when Central errors. + +### Phase 4 — Less configuration-time resolution (plugin repos) + +- [ ] Convert eager `Configuration.getFiles()` calls in plugin production + code into lazy `Provider`s resolved at execution time. This shrinks + what tests can trigger, and doubles as configuration-cache + readiness work. + +### Phase 5 — Escalation, only if Phases 1–4 are not enough + +- [ ] Stand up a caching repository manager (Nexus/Artifactory) as the + single Central-facing consumer for the org, injected through an + init script distributed by `config`. Sonatype names repository + managers as the sanctioned consolidation point. Held as an + infrastructure card because it adds operational surface the + test-design fixes avoid. + +### Guardrails + +- [ ] Promote the `projectbuilder-in-memory-caches` memory from + `core-jvm-compiler` into `.agents/shared`, so every repo's agents + inherit it. +- [ ] Add a short "429 discipline" note to the shared guidelines: check + `status.maven.org` first; never poll or retry during a block; a + block is per-egress, so a green build elsewhere does not disprove it. + +## Acceptance + +1. `./gradlew build` in `core-jvm-compiler`, `compiler`, and `validation` + completes with the network firewalled off after one warm-up run. +2. A test suite run twice performs **zero** remote requests on the second + run (verify with `--offline`, or by watching the runner's egress). +3. CI green on a runner whose Gradle cache is cold, without Central + serving the `buildSrc` bootstrap. + +## Diagnostics — is it us? + +```bash +curl -sS -D - -o /dev/null https://repo.maven.apache.org/maven2/\ +com/google/auto/service/auto-service-annotations/1.1.1/\ +auto-service-annotations-1.1.1.pom | head -3 +``` + +`429`/`403` plus `server: cloudflare` and no `Retry-After` is a +consumption block. Cross-check `status.maven.org` to rule out an actual +incident. Then **stop making requests** from that machine and wait. + +## Log + +- 2026-08-20 — drafted from the incident during + [core-jvm-compiler#111]; Phase 2's pattern is already implemented and + proven there, the rest is untouched. +- 2026-08-20 — Phase 1: traced the CI half of the incident to the Windows + job never running on the default branch, so no shared Gradle cache entry + can exist for it. Windows workflow changes **deferred by decision** — + watch CI and revisit on the next Central-caused CI failure. +- 2026-08-20 — Phase 2: considered migrating the affected `ProjectBuilder` + specs to TestKit instead of the stub fixture; set aside (see the note under + Phase 2). Confirmed upstream context: [gradle/gradle#37880] tracks the + repository-error-aborts-resolution behavior (cause 3), and no public Gradle + source documents the `ProjectBuilder` in-memory-cache behavior (cause 1). + +[faq]: https://central.sonatype.org/faq/429-error/ +[core-jvm-compiler#111]: https://github.com/SpineEventEngine/core-jvm-compiler/pull/111 +[gradle/gradle#37880]: https://github.com/gradle/gradle/issues/37880 diff --git a/.gitignore b/.gitignore index 3e1f89ad..b9610402 100644 --- a/.gitignore +++ b/.gitignore @@ -58,6 +58,10 @@ # indices, external-storage toggles) — so `.idea/*.xml` above keeps it ignored. # `./config/pull` (via `migrate`) untracks any copy an earlier pull committed. +# `.idea/kotlinc.xml` is likewise NOT re-included. It records the Kotlin compiler +# settings IDEA rewrites on its own (JVM target, bundled plugin version), so it +# churns per machine and per IDE build — `.idea/*.xml` above keeps it ignored. + # Do not ignore the following IDEA settings !.idea/codeStyleSettings.xml !.idea/codeStyles/ diff --git a/.idea/kotlinc.xml b/.idea/kotlinc.xml index 8394dde8..ea19fbb6 100644 --- a/.idea/kotlinc.xml +++ b/.idea/kotlinc.xml @@ -9,6 +9,6 @@ \ No newline at end of file diff --git a/buildSrc/src/main/kotlin/io/spine/dependency/local/CoreJvmCompiler.kt b/buildSrc/src/main/kotlin/io/spine/dependency/local/CoreJvmCompiler.kt index 8309e478..8678b234 100644 --- a/buildSrc/src/main/kotlin/io/spine/dependency/local/CoreJvmCompiler.kt +++ b/buildSrc/src/main/kotlin/io/spine/dependency/local/CoreJvmCompiler.kt @@ -32,8 +32,8 @@ package io.spine.dependency.local * See [CoreJvm Compiler](https://github.com/SpineEventEngine/core-jvm-compiler). */ @Suppress( - "MemberVisibilityCanBePrivate" /* `pluginLib()` is used by subprojects. */, - "ConstPropertyName", + "MemberVisibilityCanBePrivate" /* The properties are used directly by other subprojects. */, + "ConstPropertyName" /* We use a custom convention for artifact properties. */, "unused" ) object CoreJvmCompiler { @@ -46,12 +46,12 @@ object CoreJvmCompiler { /** * The version used in the build classpath. */ - const val dogfoodingVersion = "2.0.0-SNAPSHOT.082" + const val dogfoodingVersion = "2.0.0-SNAPSHOT.090" /** * The version to be used for integration tests. */ - const val version = "2.0.0-SNAPSHOT.082" + const val version = "2.0.0-SNAPSHOT.090" /** * The ID of the Gradle plugin. @@ -59,22 +59,35 @@ object CoreJvmCompiler { const val pluginId = "io.spine.core-jvm" /** - * The library with the [dogfoodingVersion]. + * The name of the published artifact with the CoreJvm Gradle Plugin. + * + * The POM of this artifact declares a runtime dependency on + * [the Compiler plugins][compilerPluginsArtifact]. */ - val pluginLib = pluginLib(dogfoodingVersion) + const val gradlePluginArtifact = "core-jvm-gradle-plugin" /** - * The name of the published fat JAR artifact. + * The name of the published artifact with the CoreJvm Compiler plugins. */ - const val fatJarArtifact = "core-jvm-plugins" + const val compilerPluginsArtifact = "core-jvm-plugins" /** - * The library with the given [version]. + * The CoreJvm Gradle Plugin library with the [dogfoodingVersion]. */ - fun pluginLib(version: String): String = "$group:core-jvm-plugins:$version" + val gradlePlugin: String = gradlePlugin(dogfoodingVersion) /** - * The artifact reference for forcing in configurations. + * The CoreJvm Gradle Plugin library with the given [version]. */ - val pluginsArtifact: String = pluginLib(version) + fun gradlePlugin(version: String): String = "$group:$gradlePluginArtifact:$version" + + /** + * The library with the CoreJvm Compiler plugins with the [version]. + */ + val compilerPlugins: String = compilerPlugins(version) + + /** + * The library with the CoreJvm Compiler plugins with the given [version]. + */ + fun compilerPlugins(version: String): String = "$group:$compilerPluginsArtifact:$version" } diff --git a/migrate b/migrate index a228d444..8a0fe183 100644 --- a/migrate +++ b/migrate @@ -72,20 +72,26 @@ function initialize() { } echo "Updating IDEA configuration" -# Preserve a project's `.idea/misc.xml` (do not overwrite it). It is project-local -# — the per-project JDK name plus IDEA's own churn — so it is ignored and untracked -# further below; letting this shared `.idea` overlay clobber it with config's copy -# would defeat that (the reset is git-silent, but still resets the consumer's JDK -# on every pull). Config's copy still seeds a consumer that has none yet, carrying -# the shared `EntryPointsManager` / nullness defaults; an existing project-local -# copy wins. Same approach as `module.gradle.kts` below. -DEST_MISC="../.idea/misc.xml" -MISC_PRESERVE_TMP="" -if [ -f "$DEST_MISC" ]; then - echo "Preserving existing \`.idea/misc.xml\`" - MISC_PRESERVE_TMP=$(mktemp -t misc.XXXXXX) - cp -a "$DEST_MISC" "$MISC_PRESERVE_TMP" -fi +# Preserve a project's IDE-managed `.idea` files (do not overwrite them). Both are +# project-local — `misc.xml` carries the per-project JDK name plus IDEA's own churn, +# `kotlinc.xml` the Kotlin compiler settings (JVM target, bundled plugin version) — +# so both are ignored and untracked further below; letting this shared `.idea` overlay +# clobber them with config's copies would defeat that (the reset is git-silent, but +# still resets the consumer's JDK and Kotlin settings on every pull). Config tracks +# neither file, so its copies are whatever the puller's own checkout happens to hold. +# Such a copy still seeds a consumer that has none yet; an existing project-local copy +# wins. Same approach as `module.gradle.kts` below. +IDE_LOCAL_FILES=(.idea/misc.xml .idea/kotlinc.xml) +IDE_LOCAL_TMP=() +for ide_local in "${IDE_LOCAL_FILES[@]}"; do + preserved="" + if [ -f "../$ide_local" ]; then + echo "Preserving existing \`$ide_local\`" + preserved=$(mktemp -t "$(basename "$ide_local")".XXXXXX) + cp -a "../$ide_local" "$preserved" + fi + IDE_LOCAL_TMP+=("$preserved") +done # Preserve a project's `.idea/copyright/profiles_settings.xml` (do not overwrite it). # It selects the default copyright profile, which is project-specific: proprietary @@ -107,12 +113,16 @@ fi cp -R .idea .. -# Restore preserved `.idea/misc.xml`, if any. -if [ -n "$MISC_PRESERVE_TMP" ] && [ -f "$MISC_PRESERVE_TMP" ]; then - echo "Restoring existing \`.idea/misc.xml\`" - cp -a "$MISC_PRESERVE_TMP" "$DEST_MISC" - rm -f "$MISC_PRESERVE_TMP" -fi +# Restore the preserved IDE-managed `.idea` files, if any. +for i in "${!IDE_LOCAL_FILES[@]}"; do + ide_local="${IDE_LOCAL_FILES[$i]}" + preserved="${IDE_LOCAL_TMP[$i]}" + if [ -n "$preserved" ] && [ -f "$preserved" ]; then + echo "Restoring existing \`$ide_local\`" + cp -a "$preserved" "../$ide_local" + rm -f "$preserved" + fi +done # Restore preserved `.idea/copyright/profiles_settings.xml`, if any. Fails closed for # the same reason as the preservation above — ensure the parent dir exists and abort @@ -377,39 +387,48 @@ if git rev-parse --is-inside-work-tree >/dev/null 2>&1; then fi # --------------------------------------------------------------------------- -# Make `.idea/misc.xml` project-local. It carries the per-project JDK name and -# IDEA's own churn (entry-point list indices, external-storage toggles), so a -# tracked copy collides with the consumer's local IDE state on every -# `./config/pull`. Two steps: +# Make the IDE-managed `.idea` project files project-local: +# +# * `.idea/misc.xml` — the per-project JDK name plus IDEA's own churn +# (entry-point list indices, external-storage toggles). +# * `.idea/kotlinc.xml` — the Kotlin compiler settings IDEA rewrites on its own +# (JVM target, bundled Kotlin plugin version). +# +# IDEA rewrites both without the developer touching them, so a tracked copy +# collides with the consumer's local IDE state on every `./config/pull`. Two +# steps per file: # -# 1. Drop any surviving `!.idea/misc.xml` negation. The shared baseline no -# longer rescues the file, but `update-gitignore.sh` preserves a legacy +# 1. Drop any surviving `!` negation. The shared baseline no longer +# rescues these files, but `update-gitignore.sh` preserves a legacy # raw-copied `.gitignore`'s negations into the repo-local block, so the # retired rescue can outlive the baseline and — `.gitignore` being -# last-match-wins — un-ignore the file, leaving it `?? .idea/misc.xml` -# (which `git add -A` re-adds) instead of ignored. The merge already ran -# above; scrub the negation from its result. +# last-match-wins — un-ignore the file, leaving it `?? ` (which +# `git add -A` re-adds) instead of ignored. The merge already ran above; +# scrub the negation from its result. Kept in step with the +# `retired_negations` list in `scripts/update-gitignore.sh`. # 2. Untrack any copy an earlier pull committed. `--cached` keeps the working # file; `--force` overrides `git rm`'s up-to-date check so a consumer that -# staged `misc.xml` IDE churn (its index differing from both HEAD and the -# work tree) is still migrated — otherwise `git rm` fails and, as `migrate` -# runs without `set -e`, silently leaves the file tracked. The `git ls-files` +# staged IDE churn (the index differing from both HEAD and the work tree) +# is still migrated — otherwise `git rm` fails and, as `migrate` runs +# without `set -e`, silently leaves the file tracked. The `git ls-files` # guard makes a re-run a quiet no-op. # --------------------------------------------------------------------------- -if [ -f .gitignore ] && grep -qxF '!.idea/misc.xml' .gitignore; then - echo "Dropping the retired '!.idea/misc.xml' negation from .gitignore" - gi_tmp=$(mktemp ./.gitignore.XXXXXX) - if grep -vxF '!.idea/misc.xml' .gitignore > "$gi_tmp"; then - mv "$gi_tmp" .gitignore - else - rm -f "$gi_tmp" +for ide_file in .idea/misc.xml .idea/kotlinc.xml; do + if [ -f .gitignore ] && grep -qxF "!$ide_file" .gitignore; then + echo "Dropping the retired '!$ide_file' negation from .gitignore" + gi_tmp=$(mktemp ./.gitignore.XXXXXX) + if grep -vxF "!$ide_file" .gitignore > "$gi_tmp"; then + mv "$gi_tmp" .gitignore + else + rm -f "$gi_tmp" + fi fi -fi -if git rev-parse --is-inside-work-tree >/dev/null 2>&1 \ - && git ls-files --error-unmatch .idea/misc.xml >/dev/null 2>&1; then - echo "Untracking '.idea/misc.xml' (now project-local, git-ignored)" - git rm --cached --force --quiet .idea/misc.xml -fi + if git rev-parse --is-inside-work-tree >/dev/null 2>&1 \ + && git ls-files --error-unmatch "$ide_file" >/dev/null 2>&1; then + echo "Untracking '$ide_file' (now project-local, git-ignored)" + git rm --cached --force --quiet "$ide_file" + fi +done # --------------------------------------------------------------------------- # Remove the retired Gradle Wrapper validation workflow. `config` no longer diff --git a/scripts/test-migrate-ide-files.sh b/scripts/test-migrate-ide-files.sh new file mode 100755 index 00000000..d4f34c77 --- /dev/null +++ b/scripts/test-migrate-ide-files.sh @@ -0,0 +1,263 @@ +#!/usr/bin/env bash + +# Copyright 2026, TeamDev. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Redistribution and use in source and/or binary forms, with or without +# modification, must retain the above copyright notice and the following +# disclaimer. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +# Regression checks for `migrate`'s handling of the IDE-managed `.idea` files +# (`misc.xml`, `kotlinc.xml`). IDEA rewrites both on its own, so `migrate` must +# leave a consumer with copies that are UNTRACKED, IGNORED, and carrying the +# consumer's own settings rather than config's. +# +# `migrate` runs WITHOUT `set -e` and is sourced by `pull`, so a broken step here +# fails silently mid-pull: the file just stays tracked, which is indistinguishable +# from "this consumer has not pulled yet". Hence these checks assert the observable +# end state (`git ls-files`, `git check-ignore`, file contents) rather than trusting +# the script's output. +# +# The REAL `migrate` is run against a minimal fake `config/` fixture. Two +# consequences worth knowing before reading a failure: +# +# * `adopt-shared-agents` is STUBBED. The real one reaches out to +# `github.com/SpineEventEngine/agents`, and `migrate` aborts (exit 1) when it +# fails — that would make this suite network-dependent. +# * The fixture omits most files `migrate` copies (`AGENTS.md`, `buildSrc`, the +# workflows, ...), so the run prints `cp: No such file or directory` to stderr. +# That noise is EXPECTED — `migrate` is fail-open by design and continues. The +# run log is captured, and shown only when a check fails. +# +# SCOPE. Two mechanisms conspire to keep these files out of git, and this suite +# pins only one of them end-to-end: +# +# * `migrate` — untracks the files and scrubs the retired `!` negation from the +# merged `.gitignore`. Covered here. +# * `scripts/update-gitignore.sh` — drops the same negation during the merge, +# via its `retired_negations` list. Covered by `scripts/test-update-gitignore.sh`. +# +# The two overlap on purpose: `migrate` scrubs the negation AFTER the merge runs, +# so dropping an entry from `retired_negations` does not break the pull — and this +# suite keeps passing. That is redundancy working, not a blind spot; the entry +# matters when `update-gitignore.sh` is run on its own, which is exactly what the +# sibling suite asserts. Run both. + +set -eo pipefail + +# Resolve the repo's `config` directory from this script's location +# (scripts/ lives directly under it), so the test works regardless of CWD. +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +config_dir="$(cd "$script_dir/.." && pwd)" + +fail=0 +pass() { echo "PASS: $1"; } +f-ail() { echo "FAIL: $1" >&2; fail=1; } + +# --- Pre-flight: everything the fixture copies in must exist. ----------------- +for required in migrate scripts/update-gitignore.sh .gitignore; do + [ -f "$config_dir/$required" ] \ + || { echo "FAIL: cannot find '$required' under $config_dir" >&2; exit 1; } +done + +# The files under test, and the marker each one carries in the fixture. +ide_files=(.idea/misc.xml .idea/kotlinc.xml) + +work="$(mktemp -d)" +trap 'rm -rf "$work"' EXIT +consumer="$work/consumer" +fake_config="$consumer/config" +mkdir -p "$fake_config/scripts" "$fake_config/.idea" "$consumer/.idea" + +# --- The fake `config` the consumer pulls from. ------------------------------- +cp "$config_dir/migrate" "$fake_config/migrate" +cp "$config_dir/scripts/update-gitignore.sh" "$fake_config/scripts/" +cp "$config_dir/.gitignore" "$fake_config/.gitignore" + +# Stub out the network gate (see the header note). +printf '#!/usr/bin/env bash\nexit 0\n' > "$fake_config/adopt-shared-agents" +chmod +x "$fake_config/adopt-shared-agents" + +# Config's own `.idea`. It carries copies of BOTH files under test: config no +# longer tracks them, so a puller's checkout may still hold stray local ones — +# exactly the case the preserve/restore around `cp -R .idea ..` must survive. +# `other.xml` stands for the genuinely shared settings the overlay must deliver. +for f in "${ide_files[@]}"; do + printf 'CONFIG-%s\n' "$(basename "$f")" > "$fake_config/$f" +done +printf 'CONFIG-shared\n' > "$fake_config/.idea/other.xml" + +# --- A legacy consumer: both files COMMITTED, retired negations in place. ----- +git -C "$consumer" init -q +git -C "$consumer" config user.email test@example.com +git -C "$consumer" config user.name test + +for f in "${ide_files[@]}"; do + printf 'CONSUMER-%s\n' "$(basename "$f")" > "$consumer/$f" +done + +# A raw copy of an OLDER baseline: the current baseline plus the `!` re-inclusions +# config shipped until 2025 (`2fbc0303`), plus a genuine consumer line. Without +# the retired-negation filter these `!`s land in the repo-local block, and +# `.gitignore` being last-match-wins they would un-ignore both files. +{ cat "$config_dir/.gitignore" + printf '%s\n' '!.idea/misc.xml' '!.idea/kotlinc.xml' 'my-own-cache/' +} > "$consumer/.gitignore" + +git -C "$consumer" add -A +git -C "$consumer" commit -qm "Legacy consumer with tracked IDE files" + +# Sanity-check the premise: the fixture must really start with both files tracked, +# otherwise every assertion below would pass vacuously. +for f in "${ide_files[@]}"; do + git -C "$consumer" ls-files --error-unmatch "$f" >/dev/null 2>&1 \ + || { echo "FAIL: fixture is broken — '$f' was not tracked before migrate ran" >&2; exit 1; } +done + +# --- Run the real `migrate`, exactly as `pull` does (CWD = `config`). --------- +run_migrate() { + local log="$1" + ( cd "$fake_config" && bash migrate ) > "$log" 2>&1 +} + +log1="$work/migrate-1.log" +if run_migrate "$log1"; then + pass "migrate ran to completion (exit 0)" +else + f-ail "migrate exited non-zero — see the log below" + cat "$log1" >&2 +fi + +cd "$consumer" + +# --- (1) Untracked, but still on disk, and ignored. --------------------------- +for f in "${ide_files[@]}"; do + if git ls-files --error-unmatch "$f" >/dev/null 2>&1; then + f-ail "'$f' is still tracked after migrate" + else + pass "'$f' untracked" + fi + + if [ -f "$f" ]; then + pass "'$f' still present on disk" + else + f-ail "'$f' was deleted from the working tree (must survive \`git rm --cached\`)" + fi + + if git check-ignore -q "$f"; then + pass "'$f' ignored" + else + f-ail "'$f' is NOT ignored — it will come back as '??' and \`git add -A\` will re-add it" + fi +done + +# --- (2) The consumer's own settings survive config's `.idea` overlay. -------- +# A git-silent clobber here would reset every consumer's JDK / Kotlin JVM target +# to whatever the puller happened to have locally. +for f in "${ide_files[@]}"; do + expected="CONSUMER-$(basename "$f")" + if [ "$(cat "$f")" = "$expected" ]; then + pass "'$f' kept the consumer's own content" + else + f-ail "'$f' was clobbered by config's copy (expected '$expected', got '$(cat "$f")')" + fi +done + +# The overlay must still deliver genuinely shared `.idea` files — a preserve step +# that accidentally skipped the copy would also pass every check above. +if [ -f .idea/other.xml ] && [ "$(cat .idea/other.xml)" = "CONFIG-shared" ]; then + pass "shared '.idea/other.xml' delivered by the overlay" +else + f-ail "shared '.idea/other.xml' missing — the '.idea' overlay did not run" +fi + +# --- (3) The retired negations are gone; genuine consumer lines are not. ------ +for f in "${ide_files[@]}"; do + if grep -qxF "!$f" .gitignore; then + f-ail "retired negation '!$f' survived in the merged .gitignore" + else + pass "retired negation '!$f' stripped from the merged .gitignore" + fi +done + +if git check-ignore -q my-own-cache/x; then + pass "genuine consumer entry 'my-own-cache/' preserved" +else + f-ail "genuine consumer entry 'my-own-cache/' was lost by the merge" +fi + +# --- (4) `git add -A` must not resurrect them. -------------------------------- +# This is the failure the whole mechanism exists to prevent: an un-ignored file +# left in the working tree is silently re-committed by the consumer's next +# `git add -A`, and the churn returns. +git add -A +for f in "${ide_files[@]}"; do + if git ls-files --error-unmatch "$f" >/dev/null 2>&1; then + f-ail "'git add -A' re-added '$f' — the ignore did not hold" + else + pass "'git add -A' left '$f' untracked" + fi +done + +# --- (5) No temp files leaked into the consumer. ------------------------------ +# The negation scrub builds `./.gitignore.XXXXXX` next to the target so the +# replacement is an atomic same-directory rename; every one must be consumed. +leaked="$(find . -maxdepth 1 -name '.gitignore.??????' -print)" +if [ -z "$leaked" ]; then + pass "no '.gitignore.XXXXXX' temp files left behind" +else + f-ail "temp files leaked into the consumer: $leaked" +fi + +# --- (6) A second pull is a quiet no-op. -------------------------------------- +# The `git ls-files` / `grep -qxF` guards must make a re-run do nothing: `git rm` +# on an already-untracked path fails, and `migrate` runs without `set -e`. +git commit -qm "Pull: untrack IDE-managed files" + +log2="$work/migrate-2.log" +if run_migrate "$log2"; then + pass "second migrate run completed (exit 0)" +else + f-ail "second migrate run exited non-zero — see the log below" + cat "$log2" >&2 +fi + +for f in "${ide_files[@]}"; do + if grep -q "Untracking '$f'" "$log2"; then + f-ail "second run tried to untrack '$f' again (guard is not idempotent)" + else + pass "second run skipped '$f' (quiet no-op)" + fi +done + +if [ -z "$(git status --porcelain)" ]; then + pass "second run left the working tree clean" +else + f-ail "second run dirtied the working tree:" + git status --porcelain >&2 +fi + +echo +if [ "$fail" -eq 0 ]; then + echo "OK: all migrate IDE-file regression checks passed." +else + echo "FAILED: one or more migrate IDE-file regression checks failed." >&2 + exit 1 +fi diff --git a/scripts/test-update-gitignore.sh b/scripts/test-update-gitignore.sh index c6bb2ae1..079803f5 100755 --- a/scripts/test-update-gitignore.sh +++ b/scripts/test-update-gitignore.sh @@ -80,9 +80,10 @@ git -C "$consumer" config user.name test printf '%s\n' '!gradle-wrapper.jar' printf '%s\n' '!debug.log' printf '%s\n' '*.log' - # A retired baseline negation an earlier first-migration baked into repo-local; - # the steady-state merge must strip it so `.idea/misc.xml` stays ignored. + # Retired baseline negations an earlier first-migration baked into repo-local; + # the steady-state merge must strip both so the `.idea` files stay ignored. printf '%s\n' '!.idea/misc.xml' + printf '%s\n' '!.idea/kotlinc.xml' # A consumer comment that merely BEGINS like the legacy secret trailer, plus a # real entry after it. The legacy match is exact, so this comment is preserved # and the entry survives (regression guard for the start-anchored to-EOF drop). @@ -106,7 +107,7 @@ mkdir -p generated && : > generated/secret.gpg : > info.log : > creds.secret.properties.gpg mkdir -p keep-me-ignored && : > keep-me-ignored/x -mkdir -p .idea && : > .idea/misc.xml +mkdir -p .idea && : > .idea/misc.xml && : > .idea/kotlinc.xml # (1) Every decrypted credential must be ignored despite the repo-local negations. for f in "${secrets[@]}"; do @@ -200,20 +201,22 @@ else f-ail "look-alike legacy comment triggered to-EOF drop; 'keep-me-ignored/' lost" fi -# (3h) A retired baseline negation baked into repo-local (`!.idea/misc.xml`) must be -# stripped so the baseline's `.idea/*.xml` ignore wins. The secret trailer does NOT -# cover it (misc.xml is not a secret glob), so without stripping it would re-expose the -# per-developer IDEA project file. -if git check-ignore -q .idea/misc.xml; then - pass "retired '!.idea/misc.xml' stripped from repo-local (.idea/misc.xml stays ignored)" -else - f-ail "retired '!.idea/misc.xml' survived in repo-local (.idea/misc.xml re-exposed)" -fi -if grep -qxF '!.idea/misc.xml' "$consumer/.gitignore"; then - f-ail "retired '!.idea/misc.xml' still present in the merged .gitignore" -else - pass "retired '!.idea/misc.xml' removed from the merged .gitignore" -fi +# (3h) Retired baseline negations baked into repo-local (`!.idea/misc.xml`, +# `!.idea/kotlinc.xml`) must be stripped so the baseline's `.idea/*.xml` ignore wins. +# The secret trailer does NOT cover them (neither is a secret glob), so without +# stripping they would re-expose the IDE-managed project files. +for ide_file in .idea/misc.xml .idea/kotlinc.xml; do + if git check-ignore -q "$ide_file"; then + pass "retired '!$ide_file' stripped from repo-local ($ide_file stays ignored)" + else + f-ail "retired '!$ide_file' survived in repo-local ($ide_file re-exposed)" + fi + if grep -qxF "!$ide_file" "$consumer/.gitignore"; then + f-ail "retired '!$ide_file' still present in the merged .gitignore" + else + pass "retired '!$ide_file' removed from the merged .gitignore" + fi +done # --- (4) Idempotency: re-running yields byte-identical output. ----------------- cp "$consumer/.gitignore" "$work/first.gitignore" @@ -331,39 +334,47 @@ else diff "$work/c3-first.gitignore" "$consumer3/.gitignore" >&2 || true fi -# --- (8) First-migration retired negation: a legacy raw copy carrying the OLD ------ -# baseline's `!.idea/misc.xml`. The bootstrap keeps consumer negations, so without the -# retired-negation filter this stale `!` lands in repo-local (after the managed block) -# and re-includes `.idea/misc.xml`. It must be stripped; a genuine consumer line beside -# it must still survive. +# --- (8) First-migration retired negations: a legacy raw copy carrying the OLD ----- +# baseline's `!.idea/misc.xml` and `!.idea/kotlinc.xml`. The bootstrap keeps consumer +# negations, so without the retired-negation filter these stale `!`s land in repo-local +# (after the managed block) and re-include the IDE-managed files. Both must be +# stripped; a genuine consumer line beside them must still survive. consumer4="$work/consumer4" mkdir -p "$consumer4/config" cp "$baseline" "$consumer4/config/.gitignore" git -C "$consumer4" init -q git -C "$consumer4" config user.email test@example.com git -C "$consumer4" config user.name test -# Legacy raw copy of an OLDER baseline: current baseline + the retired negation it used -# to carry + a genuine consumer line. No managed markers. -{ cat "$baseline"; printf '%s\n' '!.idea/misc.xml' 'my-own-cache/'; } > "$consumer4/.gitignore" +# Legacy raw copy of an OLDER baseline: current baseline + the retired negations it +# used to carry + a genuine consumer line. No managed markers. +{ cat "$baseline" + printf '%s\n' '!.idea/misc.xml' '!.idea/kotlinc.xml' 'my-own-cache/' +} > "$consumer4/.gitignore" ( cd "$consumer4/config" && bash "$script" ) -mkdir -p "$consumer4/.idea" && : > "$consumer4/.idea/misc.xml" -if git -C "$consumer4" check-ignore -q .idea/misc.xml; then - pass "first-migration strips retired '!.idea/misc.xml' (.idea/misc.xml stays ignored)" -else - f-ail "first-migration kept retired '!.idea/misc.xml' (.idea/misc.xml re-exposed)" -fi +mkdir -p "$consumer4/.idea" +: > "$consumer4/.idea/misc.xml" +: > "$consumer4/.idea/kotlinc.xml" +for ide_file in .idea/misc.xml .idea/kotlinc.xml; do + if git -C "$consumer4" check-ignore -q "$ide_file"; then + pass "first-migration strips retired '!$ide_file' ($ide_file stays ignored)" + else + f-ail "first-migration kept retired '!$ide_file' ($ide_file re-exposed)" + fi +done if git -C "$consumer4" check-ignore -q my-own-cache/x; then pass "first-migration preserved the genuine consumer line beside the filtered negation" else f-ail "first-migration dropped the genuine consumer line" fi -if grep -qxF '!.idea/misc.xml' "$consumer4/.gitignore"; then - f-ail "first-migration left the retired '!.idea/misc.xml' in the merged file" -else - pass "first-migration removed the retired '!.idea/misc.xml' from the merged file" -fi +for ide_file in .idea/misc.xml .idea/kotlinc.xml; do + if grep -qxF "!$ide_file" "$consumer4/.gitignore"; then + f-ail "first-migration left the retired '!$ide_file' in the merged file" + else + pass "first-migration removed the retired '!$ide_file' from the merged file" + fi +done echo if [ "$fail" -eq 0 ]; then diff --git a/scripts/update-gitignore.sh b/scripts/update-gitignore.sh index 3f56dd10..cd24d60c 100755 --- a/scripts/update-gitignore.sh +++ b/scripts/update-gitignore.sh @@ -93,10 +93,16 @@ legacy_secret_label='# --- secret ignores re-asserted last so a repo-local negat # repo-local was written by an earlier first-migration — still carries them. Left in the # repo-local region they sit AFTER the managed baseline and, gitignore being # last-match-wins, re-include a path the current baseline now ignores. Strip them so the -# baseline's ignore wins. First entry: `!.idea/misc.xml` (the current baseline dropped its -# `!.idea/misc.xml` re-inclusion to keep the per-machine IDEA project file untracked). -# Newline-separated; extend as further negations are retired. -retired_negations='!.idea/misc.xml' +# baseline's ignore wins. Both current entries are IDEA project files the baseline once +# re-included and now keeps ignored because the IDE rewrites them on its own: +# `!.idea/misc.xml` (per-project JDK name) and `!.idea/kotlinc.xml` (Kotlin compiler +# settings — JVM target, bundled plugin version). +# Newline-separated; extend as further negations are retired. The list reaches `awk` +# through the ENVIRONMENT, not `-v`: an `-v` assignment runs escape processing and +# cannot carry an embedded newline, so a multi-entry list fails there with +# "newline in string". `ENVIRON[]` passes the value through verbatim. +retired_negations='!.idea/misc.xml +!.idea/kotlinc.xml' # Positive secret patterns from the baseline's Secrets section: the span from the # `# Secrets` header to the closing `!*.gpg`, excluding comments and negations. @@ -149,12 +155,13 @@ if grep -qxF "$base_begin" "$dest"; then # `retired_negations`) that would otherwise re-include a now-ignored path — so an # already-migrated consumer converts to the marker format exactly once, and a # look-alike consumer comment is never dropped. + RETIRED_NEGATIONS="$retired_negations" \ awk -v bb="$base_begin" -v eb="$base_end" \ -v sb="$secret_begin" -v se="$secret_end" \ -v lb="$local_begin" -v le="$local_end" \ - -v lll="$legacy_local_label" -v lsl="$legacy_secret_label" \ - -v rn="$retired_negations" ' - BEGIN { k = split(rn, r, "\n"); for (i = 1; i <= k; i++) if (r[i] != "") retired[r[i]] = 1 } + -v lll="$legacy_local_label" -v lsl="$legacy_secret_label" ' + BEGIN { k = split(ENVIRON["RETIRED_NEGATIONS"], r, "\n") + for (i = 1; i <= k; i++) if (r[i] != "") retired[r[i]] = 1 } $0 == bb { inb = 1; next } inb { if ($0 == eb) inb = 0; next } $0 == sb { ins = 1; next } @@ -183,8 +190,9 @@ else # steady-state path above takes over. `|| true`: a pure raw copy leaves no custom # lines, and `grep -v` exits 1 when it selects nothing — not an error here. { grep -vxF -f <(grep -v '^!' "$src") "$dest" || true; } \ - | awk -v rn="$retired_negations" \ - 'BEGIN { k = split(rn, r, "\n"); for (i = 1; i <= k; i++) if (r[i] != "") retired[r[i]] = 1 } + | RETIRED_NEGATIONS="$retired_negations" awk \ + 'BEGIN { k = split(ENVIRON["RETIRED_NEGATIONS"], r, "\n") + for (i = 1; i <= k; i++) if (r[i] != "") retired[r[i]] = 1 } $0 in retired { next } !seen[$0]++' \ | trim_blank_edges > "$locals" fi