Concurrent cargo builds race on the shared tsjs dist and can embed a partial bundle set

[aram356]

Description

crates/trusted-server-js/build.rs embeds the browser bundles into the Rust binary through one shared directory, crates/trusted-server-js/dist:

1. It runs npm run build (build.rs:73-85), which is node build-all.mjs.
2. build-all.mjs deletes and recreates that directory (crates/trusted-server-js/lib/build-all.mjs:24-29), then writes 13 bundles into it: core first, then the 12 integrations in parallel (:79-84). Rollup 4.57.1 writes each file in place with writeFile.
3. build.rs then lists tsjs-*.js in the same directory (build.rs:87-99), copies each file to OUT_DIR (:127-130), hashes the copies (:142-143) and embeds them with include_str! (:144-149).

Every cargo invocation in a checkout uses this directory. When two build scripts run at the same time, one deletes and rewrites dist while the other lists and copies it.

Nothing downstream notices a short or damaged set:

• build.rs only checks that it found at least one bundle (:112-116). It does not know which modules to expect and accepts empty files.
• The bundle.rs tests compare each embedded bundle with a hash computed from the same copied file (build.rs:143, crates/trusted-server-js/src/bundle.rs:141-153), so an empty bundle passes.
• At runtime an enabled integration without a bundle is skipped on purpose, because Rust-only integrations have none (crates/trusted-server-core/src/integrations/registry.rs:1182-1197). concatenated_module_ids also skips any missing ID, including core (bundle.rs:65-83).
• Nothing watches dist. rerun-if-changed covers only lib (build.rs:18-20), so a bad embed stays in that target directory until a file under lib changes.

The shared dist and the directory scan arrived with per-module builds in #242 (c06ae9cbd, 2026-02-19). Before that, build.rs copied one required tsjs-unified.js.

When two build scripts overlap

Cargo serializes builds that share a target directory and a profile. It does not serialize the others:

• Different target directories do not block each other.
• In one target directory, a dev build and a release build ran at the same time. Neither printed "Blocking waiting for file lock". Two dev builds for different --target triples did block ("Blocking waiting for file lock on artifact directory").

Realistic overlaps:

• fastly compute serve or fastly compute publish builds with --release (fastly.toml:14) while a dev-profile cargo build, cargo check or cargo test-* runs in the same checkout, for example from an editor.
• Any tool with its own target directory, such as an editor configured with a separate target dir or a CARGO_TARGET_DIR override.
• npm run build run by hand or by scripts/integration-tests-browser.sh:62-63 writes the same dist while cargo builds.

The build script reruns only after a change under lib, such as editing TypeScript or switching branches. So both commands must start soon after such a change.

The repository's own CI does not hit this. Each job has its own checkout and runs its cargo steps one after another (.github/workflows/test.yml).

Steps to reproduce

From the repository root, with Node and npm installed:

# Warm up both profiles once.
cargo build -p trusted-server-js
cargo build -p trusted-server-js --release

# Start a dev build, then a release build 0.6 s later, after a change under lib.
for i in 1 2 3 4 5 6 7 8 9 10; do
  touch crates/trusted-server-js/lib/package.json
  cargo build -p trusted-server-js > target/race-debug.log 2>&1 &
  sleep 0.6
  cargo build -p trusted-server-js --release > target/race-release.log 2>&1
  release_exit=$?
  wait $!
  echo "round $i: dev exit=$? release exit=$release_exit"
  grep -h "tsjs: Discovered" target/race-debug.log target/race-release.log | sed 's/.*tsjs: //'
  # Any generated module table that embeds an empty bundle (SHA-256 of empty input):
  grep -l 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855' \
    target/debug/build/trusted-server-js-*/out/tsjs_modules.rs \
    target/release/build/trusted-server-js-*/out/tsjs_modules.rs
done

To see exactly what a build embedded, add crates/trusted-server-js/examples/print_modules.rs:

fn main() {
    let ids = trusted_server_js::all_module_ids();
    println!("embedded {} modules", ids.len());
    for id in ids {
        let bytes = trusted_server_js::module_bundle(id).map_or(0, str::len);
        println!("  {id}: {bytes} bytes");
    }
}

Run it with cargo run -q -p trusted-server-js --example print_modules, adding --release for the release profile. Cargo reuses the existing build script output, so it shows what that profile's last build embedded.

The right delay depends on the machine. It must start the second build script while the first is still building bundles. On the machine I used, a full build script run took about 2 seconds, and delays from 0.3 to 0.9 seconds reproduced it.

Observed at a4e01eb55 (macOS, Node 24.12.0, Rust 1.95.0). I ran this loop three times (30 rounds). Five rounds went wrong:

• In four rounds both builds exited 0, but one of them had embedded a partial set: 2, 3, 12 and 12 modules.
• In one round the dev build failed (exit 101).

Two of the exit-0 rounds, with the Discovered 13 module files lines left out:

round 2: dev exit=0 release exit=0
Discovered 2 module files: ["gpt", "prebid"]
round 9: dev exit=0 release exit=0
Discovered 3 module files: ["gpt", "prebid", "sourcepoint"]

A scripted variant of the same loop stopped at the first bad round. There the dev build exited 0 with 9 modules: no core, creative, didomi or osano. I then built a small example program against that target directory. It prints all_module_ids() and the length of each module_bundle(id), and cargo did not rerun the build script for it:

embedded 9 modules
  datadome: 6488 bytes
  google_tag_manager: 8310 bytes
  gpt: 46560 bytes
  gpt_diagnostics: 53370 bytes
  lockr: 6928 bytes
  permutive: 8157 bytes
  prebid: 39214 bytes
  sourcepoint: 9351 bytes
  testlight: 2084 bytes

The release build is exposed the same way. With the order reversed (release first, dev 0.6 seconds later), the release build exited 0 with Discovered 2 module files: ["gpt", "prebid"] in round 4, and the example program built with --release printed embedded 2 modules.

With two dev builds in two target directories, staggered by 0.3, 0.6, 0.9 and 1.2 seconds for 5 rounds each (20 rounds), I saw:

• 3 builds that exited 0 with a bad set:
  • 11 modules (didomi and testlight missing);
  • 13 modules with an empty gpt bundle;
  • 13 modules with an empty gpt_diagnostics bundle.
• 2 builds that failed with tsjs: no tsjs-*.js files found in .../crates/trusted-server-js/dist. That failure prints 34,013 lines, almost all cargo:rerun-if-changed, with the panic at line 34,011.

A separate 3-round run also failed once with tsjs: npm run build failed - refusing to use stale bundles. I did not capture the Node error for that round.

A reader that lists dist while two builds overlap, the way build.rs does, can see any count from 0 to 13. Across two samples of 9 and 12 seconds every count from 0 to 13 appeared, and in one of them 5,405 of 42,583 reads listed at least one 0-byte bundle.

The fallbacks that skip the build reuse whatever dist holds:

cargo build -p trusted-server-js                     # "Discovered 13 module files"
rm crates/trusted-server-js/dist/tsjs-gpt.js         # stand-in for a stale or partial dist
touch crates/trusted-server-js/lib/package.json
TSJS_SKIP_BUILD=1 cargo build -p trusted-server-js   # exit 0, "Discovered 12 module files"
cargo build -v -p trusted-server-js                  # "Fresh trusted-server-js": still 12 modules

The same stale dist with npm absent from PATH gives warning: tsjs: npm not found; will use existing dist if available, then Discovered 12 module files, and exit 0.

Expected behavior

• A cargo build embeds the complete, freshly built set of 13 non-empty bundles, or it fails with a clear message.
• Concurrent builds in one checkout do not affect each other.
• Skipping the Node build is an explicit choice, and it is checked the same way.

Actual behavior

• Concurrent builds can exit 0 with bundles missing, including core, or with empty bundles. They can also fail with errors that point at the wrong cause.
• TSJS_SKIP_BUILD=1 or a missing npm silently embeds whatever is in dist.
• A bad embed persists until something under lib changes.

Root cause

• Shared, mutable output. The Node build cleans and rewrites crates/trusted-server-js/dist (build-all.mjs:27-29), which every build script reads (build.rs:30, :87-99).
• Discovery without an expected set. build.rs embeds whatever tsjs-*.js files exist at that moment (:87-99), requires only one (:112-116), and never checks for empty files.
• The runtime tolerates missing modules by design (registry.rs:1182-1197, bundle.rs:65-83), so nothing fails later either.

The protection against stale output is partial:

• A failed npm run build panics with "refusing to use stale bundles" (build.rs:81-84), and an empty dist panics (:112-116). Nothing else checks that the embedded set is complete or fresh.
• npm not found: warns and uses the existing dist (:41-45).
• TSJS_SKIP_BUILD=1: uses the existing dist (:23, :48, :62, :74). docs/guide/error-reference.md:615-619 recommends it as a workaround without that caveat.
• node_modules is installed only when it is missing (:47-59). A node_modules that is out of date with package-lock.json is used as is. A failed npm ci only warns (:56-58). Two build scripts that both find no node_modules would both run npm ci in the same directory. I did not reproduce that case.
• There is no rerun-if-env-changed. The build script output has none, so setting or clearing TSJS_SKIP_BUILD or TSJS_TEST does not rerun it.
• rerun-if-changed lists 33,939 paths: lib and everything under it, including node_modules (build.rs:18-20, :201-221).

Impact

• Local builds, including local deploys. fastly compute publish is a release build. A release built while another cargo command compiles can ship without some integrations' browser code, or without core. No build step or test fails. Pages then get a unified bundle without that code.
• Developers get confusing results: missing integrations in a local run, or panics that blame npm run build or an empty dist.
• Likelihood: two build scripts must overlap within about two seconds of each other, right after a change under lib. That is rare per build, but an editor plus a terminal, or a dev plus a release build, makes it reachable.
• CI is not affected.

Proposed fix

1. Give each build script its own output. Add --out-dir <dir> to build-all.mjs, keeping ../dist as the default for npm run build, the Playwright fixtures and CI. build.rs passes a directory under OUT_DIR and embeds from there. This removes the shared cleanup, the discovery race and the copy step.
2. Check the set. build.rs derives the expected modules the same way build-all.mjs:31-42 discovers them: core plus every lib/src/integrations/<name>/index.ts. It fails if any bundle is missing or empty.
3. Make prebuilt bundles explicit. Replace the silent reuse of dist for a missing npm and for TSJS_SKIP_BUILD=1 with an explicit directory variable (for example TSJS_PREBUILT_DIR), validated as in step 2. Without it, fail with instructions.
4. Declare cargo:rerun-if-env-changed for every variable build.rs reads.
5. Detect stale dependencies. Compare lib/package-lock.json with lib/node_modules/.package-lock.json and fail with "run npm ci" rather than reinstalling. An automatic npm ci deletes node_modules under any other build that is running.
6. Optional: narrow rerun-if-changed to the sources, configs and lockfile instead of 33,939 paths. This also keeps a failed build's output readable.

Alternative: a cross-process lock around the whole Node step (std::fs::File::lock is stable in the pinned Rust 1.95.0). It would serialize the build scripts, but keep the shared directory, and it does not cover npm run build run outside cargo. Private output is simpler.

Compatibility: npm run build still writes dist for Playwright and CI. Only the Rust build stops reading it. Anyone who relies on TSJS_SKIP_BUILD=1 moves to the explicit variable.

Done when

• build.rs builds into a directory under OUT_DIR and never reads crates/trusted-server-js/dist.
• build.rs fails when an expected module is missing or empty, and a test covers both cases, for example a validation function run against a directory missing one bundle and a directory with one empty bundle.
• The reproduction loop above, with 20 rounds at each delay from 0.3 to 1.2 seconds and with either build started first, always embeds 13 non-empty modules in both profiles.
• Missing npm, skipped builds and stale node_modules have defined behavior and clear errors, and docs/guide/error-reference.md describes them.
• build.rs declares rerun-if-env-changed for each variable it reads.

Affected area

JS build pipeline

Version

main at a4e01eb

Related

• Add Prebid.js NPM integration with per-module JS builds #242: per-module builds, which introduced the shared dist cleanup and the directory scan.
• build.rs should use npm run ci instead of install to prevent package-lock.json changes in diffs #378 and Use npm ci instead of npm install in build.rs #385: switched build.rs from npm install to npm ci.
• Return a static slice from all_module_ids instead of allocating a Vec #1180 (open): changes the code that build.rs generates. Coordinate the edits.
• Per-request fixed cost: release profile, settings pipeline trim, build-time tsjs hashes #855 (open): build-time tsjs hashes, in the same generated code.
• Define integrations crate split and ordered configuration #1194 (open): design PR whose review found this.