diff --git a/docs/Parallelism.md b/docs/Parallelism.md index c0a27aeb0..915a83ff2 100644 --- a/docs/Parallelism.md +++ b/docs/Parallelism.md @@ -285,8 +285,9 @@ SimpleCov coordinates with parallel test runners through a small pluggable adapt uses its `ParallelTests.first_process?` / `ParallelTests.wait_for_other_processes_to_finish` APIs for precise worker coordination. Activates only when the native `parallel_tests` pid-file contract is present. - **`GenericAdapter`** — catch-all for any runner that follows the `TEST_ENV_NUMBER` / `PARALLEL_TEST_GROUPS` env-var - convention but doesn't ship a Ruby API (parallel_rspec, knapsack-style splitters, custom CI sharding scripts). - Activates when `TEST_ENV_NUMBER` is set and no more-specific adapter is. + convention but doesn't ship a Ruby API (parallel_rspec, [rspec-conductor](https://github.com/markiz/rspec-conductor) + 1.0.7 or later, knapsack-style splitters, custom CI sharding scripts). Activates when `TEST_ENV_NUMBER` is set and no + more-specific adapter is. Adapters are tried in registration order; the first whose `active?` returns `true` is chosen. With both built-ins, this means parallel_tests users get the precise gem-based path and parallel_rspec (or any env-var-only runner) gets the diff --git a/features/parallel_tests.feature b/features/parallel_tests.feature index 309ba7884..0fb4ad82d 100644 --- a/features/parallel_tests.feature +++ b/features/parallel_tests.feature @@ -22,7 +22,7 @@ Feature: SimpleCov.start """ When I open the coverage report generated with `bundle exec parallel_rspec -n 2 spec` - Then I should see the line coverage results for the parallel tests project + Then I should see the line coverage results for the parallel fixture project # Note it's better not to test this in the same scenario as before. # Merging of results might kick in and ruin this. @@ -34,7 +34,7 @@ Feature: SimpleCov.start """ When I open the coverage report generated with `bundle exec rspec spec` - Then I should see the line coverage results for the parallel tests project + Then I should see the line coverage results for the parallel fixture project @branch_coverage Scenario: Running the project with normal rspec and branch coverage @@ -47,7 +47,7 @@ Feature: end """ When I open the coverage report generated with `bundle exec rspec spec` - Then I should see the branch coverage results for the parallel tests project + Then I should see the branch coverage results for the parallel fixture project @branch_coverage Scenario: Running the project with normal rspec and branch coverage @@ -60,7 +60,7 @@ Feature: end """ When I open the coverage report generated with `bundle exec parallel_rspec -n 2 spec` - Then I should see the branch coverage results for the parallel tests project + Then I should see the branch coverage results for the parallel fixture project Scenario: Coverage violations aren't printed until the end Given I install dependencies diff --git a/features/rspec_conductor.feature b/features/rspec_conductor.feature new file mode 100644 index 000000000..1f9c91eb8 --- /dev/null +++ b/features/rspec_conductor.feature @@ -0,0 +1,98 @@ +@rspec @disable-bundler @process_fork + +Feature: + + rspec-conductor is a queue-based parallel spec runner. It forks its workers + from a server process, gives each worker one spec file at a time, and marks + every worker with the parallel_tests environment convention: TEST_ENV_NUMBER + per worker plus PARALLEL_TEST_GROUPS for the worker count (since + rspec-conductor 1.0.7). SimpleCov's generic parallel adapter picks that up + without any configuration, so each worker records under a unique suite name + and the first worker waits for its siblings and builds the merged report. + + # `--workers 2` pins the worker count for a fast, deterministic run. Left to + # default, rspec-conductor spawns one worker per core. Workers that get no + # spec file still load spec_helper at spawn and record an empty resultset, + # so the merged totals come out the same either way. + # + # The scenarios assert on the generated report files rather than on the + # usual "Coverage report generated" line. The conductor server closes each + # worker's output pipes once the worker has sent its run summary, before + # at_exit hooks run, so nothing SimpleCov prints from at_exit can reach + # the output. The report itself is written fine. + + Background: + Given I'm working on the project "parallel_tests" with the Gemfile from "rspec_conductor" + + Scenario: Running it through rspec-conductor produces the same results as a normal rspec run + Given I install dependencies + And SimpleCov for RSpec is configured with: + """ + require 'simplecov' + SimpleCov.start + """ + When I successfully run `bundle exec rspec-conductor --workers 2 spec` + Then the following files should exist: + | coverage/index.html | + | coverage/.resultset.json | + When I open the coverage report + Then I should see the line coverage results for the parallel fixture project + + # In default mode the first worker's TEST_ENV_NUMBER is "" like + # parallel_tests. With --first-is-1 it is "1" instead, and the workers count + # 1..N. Both spellings must resolve to a single reporting worker. + Scenario: Running with --first-is-1 merges just the same + Given I install dependencies + And SimpleCov for RSpec is configured with: + """ + require 'simplecov' + SimpleCov.start + """ + When I successfully run `bundle exec rspec-conductor --workers 2 --first-is-1 spec` + Then the following files should exist: + | coverage/index.html | + | coverage/.resultset.json | + When I open the coverage report + Then I should see the line coverage results for the parallel fixture project + + @branch_coverage + Scenario: Running with branch coverage enabled + Given I install dependencies + And SimpleCov for RSpec is configured with: + """ + require 'simplecov' + SimpleCov.start do + enable_coverage :branch + end + """ + When I successfully run `bundle exec rspec-conductor --workers 2 spec` + Then the following files should exist: + | coverage/index.html | + | coverage/.resultset.json | + When I open the coverage report + Then I should see the branch coverage results for the parallel fixture project + + # Each worker on its own covers only a slice of the project, so a threshold + # check against a single worker's partial result would fail. Only the + # reporting worker, after merging every sibling's slice, may enforce it. + # + # The conductor server swallows a worker's output printed from at_exit and + # ignores its exit status once its run summary is in, so asserting on the + # command's output or exit code proves nothing here. Instead the scenario + # leans on .last_run.json, which SimpleCov writes only when the threshold + # check passes. Finding it filled with the merged 81.48 shows the check ran + # against the merged result, where any single worker's slice would have + # failed and left the file unwritten. + Scenario: Coverage thresholds are enforced against the merged result, not per worker + Given I install dependencies + And SimpleCov for RSpec is configured with: + """ + require 'simplecov' + SimpleCov.start do + minimum_coverage 81.48 + end + """ + When I successfully run `bundle exec rspec-conductor --workers 2 spec` + Then the file "coverage/.last_run.json" should contain "81.48" + When I open the coverage report + Then I should see the line coverage results for the parallel fixture project diff --git a/features/step_definitions/parallel_tests_steps.rb b/features/step_definitions/parallel_fixture_steps.rb similarity index 91% rename from features/step_definitions/parallel_tests_steps.rb rename to features/step_definitions/parallel_fixture_steps.rb index c964a52a9..dcf789af9 100644 --- a/features/step_definitions/parallel_tests_steps.rb +++ b/features/step_definitions/parallel_fixture_steps.rb @@ -1,6 +1,6 @@ # frozen_string_literal: true -Then "I should see the line coverage results for the parallel tests project" do +Then "I should see the line coverage results for the parallel fixture project" do steps %( Then I should see the groups: | name | coverage | files | @@ -15,7 +15,7 @@ ) end -Then "I should see the branch coverage results for the parallel tests project" do +Then "I should see the branch coverage results for the parallel fixture project" do steps %( Then I should see the groups: | name | coverage | files | diff --git a/features/step_definitions/simplecov_steps.rb b/features/step_definitions/simplecov_steps.rb index 4d2cfc596..6b6fa2c38 100644 --- a/features/step_definitions/simplecov_steps.rb +++ b/features/step_definitions/simplecov_steps.rb @@ -159,3 +159,15 @@ def with_cross_worker_install_lock step 'I cd to "project"' end + +# For fixtures that reuse another project's sources wholesale and differ only +# in which gems they pull in (their test_projects directory holds nothing but +# a Gemfile and Gemfile.lock). +Given "I'm working on the project {string} with the Gemfile from {string}" do |project_name, gemfile_project| + step "I'm working on the project \"#{project_name}\"" + + gemfile_source = File.expand_path("../../test_projects/#{gemfile_project}", __dir__) + cd(".") do + FileUtils.cp(Dir.glob("#{gemfile_source}/Gemfile*"), ".") + end +end diff --git a/lib/simplecov/color.rb b/lib/simplecov/color.rb index 6771a2909..6a0ad8879 100644 --- a/lib/simplecov/color.rb +++ b/lib/simplecov/color.rb @@ -45,6 +45,10 @@ def enabled?(stream = $stderr) return true if env_set?("FORCE_COLOR") stream.tty? + rescue IOError + # A parallel runner can close a worker's stdio before its at_exit + # hooks run (rspec-conductor does). A closed stream is not a tty. + false end def for_percent(percent) diff --git a/lib/simplecov/exit_codes.rb b/lib/simplecov/exit_codes.rb index 99f5ff703..5cc25339c 100644 --- a/lib/simplecov/exit_codes.rb +++ b/lib/simplecov/exit_codes.rb @@ -18,6 +18,10 @@ module ExitCodes # warning logs. `print_errors false` remains the intended opt-out. def self.print_error(message) $stderr.puts message # rubocop:disable Style/StderrPuts + rescue IOError + # A parallel runner can close a worker's stderr before its at_exit + # hooks run (rspec-conductor does). The violation still has to set + # the exit status even when its explanation cannot be printed. end end end diff --git a/lib/simplecov/formatter/base.rb b/lib/simplecov/formatter/base.rb index dfb14c74c..5ac52f6ef 100644 --- a/lib/simplecov/formatter/base.rb +++ b/lib/simplecov/formatter/base.rb @@ -32,6 +32,11 @@ def initialize(silent: false, output_dir: nil) # Subclasses call this at the end of their `format`. def emit_status(result) $stderr.puts output_message(result) unless @silent # rubocop:disable Style/StderrPuts + rescue IOError + # A parallel runner can close a worker's stderr before its at_exit + # hooks run (rspec-conductor does). Losing the status line must not + # abort the exit tasks that follow report generation, i.e. the + # threshold checks and the .last_run.json write. end # Subclasses override to prepend a marker (e.g. "JSON ") to the diff --git a/lib/simplecov/parallel_adapters.rb b/lib/simplecov/parallel_adapters.rb index 643722450..98f6d5974 100644 --- a/lib/simplecov/parallel_adapters.rb +++ b/lib/simplecov/parallel_adapters.rb @@ -20,8 +20,10 @@ module SimpleCov # (precise sync + first-process detection via the gem's own API). # - `GenericAdapter` — env-var-only detection for runners that follow # the parallel_tests `TEST_ENV_NUMBER` convention but don't ship a - # Ruby API (parallel_rspec, custom CI sharding, knapsack-style - # splitters). See https://github.com/simplecov-ruby/simplecov/issues/1065. + # Ruby API (parallel_rspec, rspec-conductor 1.0.7+, custom CI + # sharding, knapsack-style splitters). See + # https://github.com/simplecov-ruby/simplecov/issues/1065 and + # https://github.com/simplecov-ruby/simplecov/issues/1156. # # Users can plug in additional adapters: # diff --git a/lib/simplecov/parallel_adapters/generic.rb b/lib/simplecov/parallel_adapters/generic.rb index 4320cf885..0ecd2e314 100644 --- a/lib/simplecov/parallel_adapters/generic.rb +++ b/lib/simplecov/parallel_adapters/generic.rb @@ -7,14 +7,15 @@ module ParallelAdapters # Catch-all adapter for parallel test runners that follow the # `TEST_ENV_NUMBER` / `PARALLEL_TEST_GROUPS` env-var convention but # don't ship a Ruby API for SimpleCov to hook (parallel_rspec, - # knapsack-style splitters, custom CI sharding scripts). Activates - # when `TEST_ENV_NUMBER` is set; doesn't require any specific gem to - # be loaded. + # rspec-conductor since 1.0.7, knapsack-style splitters, custom CI + # sharding scripts). Activates when `TEST_ENV_NUMBER` is set; + # doesn't require any specific gem to be loaded. # # Heuristic for `first_worker?`: the worker whose `TEST_ENV_NUMBER` - # is `""` (parallel_tests/parallel_rspec convention) or `"1"` - # (zero-based runners that start at 1). Any other value is treated - # as a non-first worker. + # is `""` (parallel_tests/parallel_rspec convention, rspec-conductor's + # default) or `"1"` (runners that number from 1, like rspec-conductor + # with `--first-is-1`). Any other value is treated as a non-first + # worker. # # `wait_for_siblings` is inherited from Base as a no-op — without a # runner-provided API the only synchronization available is polling diff --git a/spec/color_spec.rb b/spec/color_spec.rb index 71b95125b..4baced7fc 100644 --- a/spec/color_spec.rb +++ b/spec/color_spec.rb @@ -84,6 +84,13 @@ expect(described_class.enabled?(stdout_not_tty)).to be false end + it "returns false for a closed stream (a parallel runner can close worker stdio before at_exit runs)" do + read, write = IO.pipe + read.close + write.close + expect(described_class.enabled?(write)).to be false + end + it "falls through when SimpleCov has no color config (standalone CLI)" do # In the standalone `exe/simplecov` process, `simplecov/color` is # loaded without `simplecov/configuration`, so `SimpleCov` doesn't diff --git a/spec/exit_codes_spec.rb b/spec/exit_codes_spec.rb new file mode 100644 index 000000000..6959b758e --- /dev/null +++ b/spec/exit_codes_spec.rb @@ -0,0 +1,17 @@ +# frozen_string_literal: true + +require "helper" + +RSpec.describe SimpleCov::ExitCodes do + describe ".print_error" do + it "writes the message to stderr" do + expect(capture_stderr { described_class.print_error("boom") }).to eq("boom\n") + end + + it "stays quiet when stderr was closed before at_exit (e.g. by rspec-conductor)" do + allow($stderr).to receive(:puts).and_raise(IOError, "closed stream") + + expect { described_class.print_error("boom") }.not_to raise_error + end + end +end diff --git a/spec/formatter/base_spec.rb b/spec/formatter/base_spec.rb index e66ec352d..d950c19fc 100644 --- a/spec/formatter/base_spec.rb +++ b/spec/formatter/base_spec.rb @@ -38,6 +38,13 @@ def result_with(coverage_statistics) expect(Warning).not_to have_received(:warn) end + it "loses the status line quietly when stderr was closed before at_exit (e.g. by rspec-conductor)" do + result = result_with(line: statistics(8, 2)) + allow($stderr).to receive(:puts).and_raise(IOError, "closed stream") + + expect { formatter.send(:emit_status, result) }.not_to raise_error + end + it "does not build or emit a status when silent" do silent_formatter = described_class.new(silent: true) result = result_with(line: statistics(8, 2)) diff --git a/test_projects/rspec_conductor/Gemfile b/test_projects/rspec_conductor/Gemfile new file mode 100644 index 000000000..eeca49d69 --- /dev/null +++ b/test_projects/rspec_conductor/Gemfile @@ -0,0 +1,16 @@ +# frozen_string_literal: true + +source "https://rubygems.org" + +gem "rspec" +gem "rspec-conductor" + +# when the tests are executed the project is in tmp/aruba/project +# which is a different nesting from its usual place +if File.exist?("../../simplecov.gemspec") + gem "simplecov", path: "../.." +else + # rubocop:disable Bundler/DuplicatedGem + gem "simplecov", path: "../../.." + # rubocop:enable Bundler/DuplicatedGem +end diff --git a/test_projects/rspec_conductor/Gemfile.lock b/test_projects/rspec_conductor/Gemfile.lock new file mode 100644 index 000000000..850245916 --- /dev/null +++ b/test_projects/rspec_conductor/Gemfile.lock @@ -0,0 +1,46 @@ +PATH + remote: ../.. + specs: + simplecov (1.1.0) + +GEM + remote: https://rubygems.org/ + specs: + diff-lcs (1.6.2) + rspec (3.13.2) + rspec-core (~> 3.13.0) + rspec-expectations (~> 3.13.0) + rspec-mocks (~> 3.13.0) + rspec-conductor (1.0.11) + rspec-core (>= 3.8.0) + rspec-core (3.13.6) + rspec-support (~> 3.13.0) + rspec-expectations (3.13.5) + diff-lcs (>= 1.2.0, < 2.0) + rspec-support (~> 3.13.0) + rspec-mocks (3.13.8) + diff-lcs (>= 1.2.0, < 2.0) + rspec-support (~> 3.13.0) + rspec-support (3.13.7) + +PLATFORMS + arm64-darwin-25 + ruby + +DEPENDENCIES + rspec + rspec-conductor + simplecov! + +CHECKSUMS + diff-lcs (1.6.2) sha256=9ae0d2cba7d4df3075fe8cd8602a8604993efc0dfa934cff568969efb1909962 + rspec (3.13.2) sha256=206284a08ad798e61f86d7ca3e376718d52c0bc944626b2349266f239f820587 + rspec-conductor (1.0.11) sha256=6958c98d4bf07cff6827229160debca81c8f13dd550dd8e1db02e63e3c092618 + rspec-core (3.13.6) sha256=a8823c6411667b60a8bca135364351dda34cd55e44ff94c4be4633b37d828b2d + rspec-expectations (3.13.5) sha256=33a4d3a1d95060aea4c94e9f237030a8f9eae5615e9bd85718fe3a09e4b58836 + rspec-mocks (3.13.8) sha256=086ad3d3d17533f4237643de0b5c42f04b66348c28bf6b9c2d3f4a3b01af1d47 + rspec-support (3.13.7) sha256=0640e5570872aafefd79867901deeeeb40b0c9875a36b983d85f54fb7381c47c + simplecov (1.1.0) + +BUNDLED WITH + 4.0.18