Skip to content

ci: track root causes and restore CI on latest and release/v11 #10027

Description

@reggi

Purpose

This is the umbrella tracking issue for restoring CI and post-release integration on npm/cli's latest and release/v11 branches. It owns the overall failure inventory, coordinates fixes across npm/cli and upstream fixtures/actions, and tracks verification on both branches.

Track issues by root cause, not by error message, job, runtime, or branch. A child issue can contain multiple failures. Where the initiating cause is not yet established, retain an explicitly scoped investigation rather than presenting an infrastructure hypothesis as fact.

Investigated scope

The inventory below covers all 14 failed jobs in the four completed September 22, 2026 branch-head CI and Release runs:

Branch Release Commit Workflow Failed jobs
latest npm 12.1.0 c039090578a5b21a1aa3aba9c96e199feaf1823a Release 9
latest npm 12.1.0 c039090578a5b21a1aa3aba9c96e199feaf1823a CI 1
release/v11 npm 11.20.0 d12b9434dd010b5fb7044c3cc149cdda317813f8 Release 3
release/v11 npm 11.20.0 d12b9434dd010b5fb7044c3cc149cdda317813f8 CI 1

This is a complete inventory for those runs, not a claim that every historical failure has the same causes. The failed Release workflow status must not be equated with failed package publication: these failures occur in downstream integration.

Root-cause and incident owners

Owner issue Group latest Release v11 Release latest CI v11 CI
#10016 Invalid Node nightly source response; publication-race hypothesis remains unproven 1 1 0 0
#10020 Stale binary-split source repository metadata 2 2 0 0
#10024 Stable integration matrix selects Node 23 outside npm 12's engines 2 0 0 0
#10026 CITGM Git dependency is blocked by npm 12's default policy 2 0 0 0
#10023 CITGM native-build option forwarding is rejected by npm 12 2 0 0 0
#10022 Live-registry smoke-test socket reset with retries disabled; initiating cause unresolved 0 0 1 0
#10025 Cygwin provisioning fails before shim tests; installer cause unresolved 0 0 0 1
Total 9 3 1 1

Complete failed-job inventory

Branch/workflow Failed job Owner
latest Release Build nightly #10016
v11 Release Build nightly #10016
latest Release Node 22: binary-split #10020
latest Release Node 23: binary-split #10020
v11 Release Node 22: binary-split #10020
v11 Release Node 23: binary-split #10020
latest Release Node 23: test nodejs #10024
latest Release Node 23: test npm #10024
latest Release Node 22: JSONStream #10026
latest Release Node 23: JSONStream #10026
latest Release Node 22: thread-sleep #10023
latest Release Node 23: thread-sleep #10023
latest CI Smoke Tests #10022
v11 CI Windows Shims Tests #10025

Grouping decisions and important qualifications

The Node 23 warning assertion, npm smoke stderr mismatch, EBADENGINE, subsequent unsatisfied registry mocks, and engine-dependent coverage failure belong under #10024. They should not produce separate tickets per assertion.

The two CITGM policy/configuration failures require different fixes. Allowing Git dependencies does not make --build-from-source a recognized npm CLI flag, and removing that flag does not permit Git dependencies. Both failures occur on supported Node 22, independently of #10024.

#10023 preserves the history from mapbox/node-pre-gyp#58: npm install --build-from-source was a documented and functional way to forward a native installer's configuration through npm. node-pre-gyp still implements source builds; npm 12 now rejects the unknown npm CLI flag before that installer can use it. The migration must verify the actual fixture's installer and preserve any intended source-build coverage, not simply remove the option and assume success.

#10016 and #10020 share the diagnostic weakness of streaming curl -sSL output into tar, but their upstream conditions are distinct. Better HTTP handling does not repair stale repository metadata or guarantee an archive exists. Both nightly jobs selected the same archive, but their logs do not preserve the HTTP response status/body, so a publication race is not a proven fact.

The socket reset in #10022 and the Cygwin setup failure in #10025 are separately owned investigations, not evidence of one shared outage. In particular, the Cygwin downgrade conflict and subsequent installer exit are both observed, but a causal relationship has not been established.

Cross-cutting follow-up owned by this tracker

  • Preserve HTTP status and useful failure diagnostics for source downloads without feeding non-success responses directly into extraction.
  • Correct FINALEXIT=STEPEXIT in the npm integration result aggregation and its template. It currently assigns a literal string and later emits exit: STEPEXIT: numeric argument required. This masks the intended exit status after earlier test failures; it is not another primary cause of the 14 failed jobs.
  • Ensure upstream CITGM fixes reach the published version used by matrix generation. A change merged upstream but not consumed by npm/cli does not close the coverage gap.
  • Apply branch-appropriate workflow/template changes to both release lines and document local equivalents for any new diagnostic or validation commands.
  • Attach post-fix run evidence for both branches and confirm the affected tests actually execute rather than being skipped, suppressed, or bypassed.

Completion criteria

Close this tracker only when every child issue is resolved or has an explicitly justified disposition, the affected coverage is restored on both branches, and any remaining infrastructure uncertainty is documented with an owner and evidence. A rerun that happens to pass is not by itself proof of root cause, and making one error disappear is not proof that the rest of an integration job succeeded.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions