Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions docs/Parallelism.md
Original file line number Diff line number Diff line change
Expand Up @@ -285,8 +285,9 @@ SimpleCov coordinates with parallel test runners through a small pluggable adapt
uses its `ParallelTests.first_process?` / `ParallelTests.wait_for_other_processes_to_finish` APIs for precise worker
coordination. Activates only when the native `parallel_tests` pid-file contract is present.
- **`GenericAdapter`** — catch-all for any runner that follows the `TEST_ENV_NUMBER` / `PARALLEL_TEST_GROUPS` env-var
convention but doesn't ship a Ruby API (parallel_rspec, knapsack-style splitters, custom CI sharding scripts).
Activates when `TEST_ENV_NUMBER` is set and no more-specific adapter is.
convention but doesn't ship a Ruby API (parallel_rspec, [rspec-conductor](https://github.com/markiz/rspec-conductor)
1.0.7 or later, knapsack-style splitters, custom CI sharding scripts). Activates when `TEST_ENV_NUMBER` is set and no
more-specific adapter is.

Adapters are tried in registration order; the first whose `active?` returns `true` is chosen. With both built-ins, this
means parallel_tests users get the precise gem-based path and parallel_rspec (or any env-var-only runner) gets the
Expand Down
8 changes: 4 additions & 4 deletions features/parallel_tests.feature
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Feature:
SimpleCov.start
"""
When I open the coverage report generated with `bundle exec parallel_rspec -n 2 spec`
Then I should see the line coverage results for the parallel tests project
Then I should see the line coverage results for the parallel fixture project

# Note it's better not to test this in the same scenario as before.
# Merging of results might kick in and ruin this.
Expand All @@ -34,7 +34,7 @@ Feature:
SimpleCov.start
"""
When I open the coverage report generated with `bundle exec rspec spec`
Then I should see the line coverage results for the parallel tests project
Then I should see the line coverage results for the parallel fixture project

@branch_coverage
Scenario: Running the project with normal rspec and branch coverage
Expand All @@ -47,7 +47,7 @@ Feature:
end
"""
When I open the coverage report generated with `bundle exec rspec spec`
Then I should see the branch coverage results for the parallel tests project
Then I should see the branch coverage results for the parallel fixture project

@branch_coverage
Scenario: Running the project with normal rspec and branch coverage
Expand All @@ -60,7 +60,7 @@ Feature:
end
"""
When I open the coverage report generated with `bundle exec parallel_rspec -n 2 spec`
Then I should see the branch coverage results for the parallel tests project
Then I should see the branch coverage results for the parallel fixture project

Scenario: Coverage violations aren't printed until the end
Given I install dependencies
Expand Down
98 changes: 98 additions & 0 deletions features/rspec_conductor.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
@rspec @disable-bundler @process_fork

Feature:

rspec-conductor is a queue-based parallel spec runner. It forks its workers
from a server process, gives each worker one spec file at a time, and marks
every worker with the parallel_tests environment convention: TEST_ENV_NUMBER
per worker plus PARALLEL_TEST_GROUPS for the worker count (since
rspec-conductor 1.0.7). SimpleCov's generic parallel adapter picks that up
without any configuration, so each worker records under a unique suite name
and the first worker waits for its siblings and builds the merged report.

# `--workers 2` pins the worker count for a fast, deterministic run. Left to
# default, rspec-conductor spawns one worker per core. Workers that get no
# spec file still load spec_helper at spawn and record an empty resultset,
# so the merged totals come out the same either way.
#
# The scenarios assert on the generated report files rather than on the
# usual "Coverage report generated" line. The conductor server closes each
# worker's output pipes once the worker has sent its run summary, before
# at_exit hooks run, so nothing SimpleCov prints from at_exit can reach
# the output. The report itself is written fine.

Background:
Given I'm working on the project "parallel_tests" with the Gemfile from "rspec_conductor"

Scenario: Running it through rspec-conductor produces the same results as a normal rspec run
Given I install dependencies
And SimpleCov for RSpec is configured with:
"""
require 'simplecov'
SimpleCov.start
"""
When I successfully run `bundle exec rspec-conductor --workers 2 spec`
Then the following files should exist:
| coverage/index.html |
| coverage/.resultset.json |
When I open the coverage report
Then I should see the line coverage results for the parallel fixture project

# In default mode the first worker's TEST_ENV_NUMBER is "" like
# parallel_tests. With --first-is-1 it is "1" instead, and the workers count
# 1..N. Both spellings must resolve to a single reporting worker.
Scenario: Running with --first-is-1 merges just the same
Given I install dependencies
And SimpleCov for RSpec is configured with:
"""
require 'simplecov'
SimpleCov.start
"""
When I successfully run `bundle exec rspec-conductor --workers 2 --first-is-1 spec`
Then the following files should exist:
| coverage/index.html |
| coverage/.resultset.json |
When I open the coverage report
Then I should see the line coverage results for the parallel fixture project

@branch_coverage
Scenario: Running with branch coverage enabled
Given I install dependencies
And SimpleCov for RSpec is configured with:
"""
require 'simplecov'
SimpleCov.start do
enable_coverage :branch
end
"""
When I successfully run `bundle exec rspec-conductor --workers 2 spec`
Then the following files should exist:
| coverage/index.html |
| coverage/.resultset.json |
When I open the coverage report
Then I should see the branch coverage results for the parallel fixture project

# Each worker on its own covers only a slice of the project, so a threshold
# check against a single worker's partial result would fail. Only the
# reporting worker, after merging every sibling's slice, may enforce it.
#
# The conductor server swallows a worker's output printed from at_exit and
# ignores its exit status once its run summary is in, so asserting on the
# command's output or exit code proves nothing here. Instead the scenario
# leans on .last_run.json, which SimpleCov writes only when the threshold
# check passes. Finding it filled with the merged 81.48 shows the check ran
# against the merged result, where any single worker's slice would have
# failed and left the file unwritten.
Scenario: Coverage thresholds are enforced against the merged result, not per worker
Given I install dependencies
And SimpleCov for RSpec is configured with:
"""
require 'simplecov'
SimpleCov.start do
minimum_coverage 81.48
end
"""
When I successfully run `bundle exec rspec-conductor --workers 2 spec`
Then the file "coverage/.last_run.json" should contain "81.48"
When I open the coverage report
Then I should see the line coverage results for the parallel fixture project
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# frozen_string_literal: true

Then "I should see the line coverage results for the parallel tests project" do
Then "I should see the line coverage results for the parallel fixture project" do
steps %(
Then I should see the groups:
| name | coverage | files |
Expand All @@ -15,7 +15,7 @@
)
end

Then "I should see the branch coverage results for the parallel tests project" do
Then "I should see the branch coverage results for the parallel fixture project" do
steps %(
Then I should see the groups:
| name | coverage | files |
Expand Down
12 changes: 12 additions & 0 deletions features/step_definitions/simplecov_steps.rb
Original file line number Diff line number Diff line change
Expand Up @@ -159,3 +159,15 @@ def with_cross_worker_install_lock

step 'I cd to "project"'
end

# For fixtures that reuse another project's sources wholesale and differ only
# in which gems they pull in (their test_projects directory holds nothing but
# a Gemfile and Gemfile.lock).
Given "I'm working on the project {string} with the Gemfile from {string}" do |project_name, gemfile_project|
step "I'm working on the project \"#{project_name}\""

gemfile_source = File.expand_path("../../test_projects/#{gemfile_project}", __dir__)
cd(".") do
FileUtils.cp(Dir.glob("#{gemfile_source}/Gemfile*"), ".")
end
end
4 changes: 4 additions & 0 deletions lib/simplecov/color.rb
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,10 @@ def enabled?(stream = $stderr)
return true if env_set?("FORCE_COLOR")

stream.tty?
rescue IOError
# A parallel runner can close a worker's stdio before its at_exit
# hooks run (rspec-conductor does). A closed stream is not a tty.
false
end

def for_percent(percent)
Expand Down
4 changes: 4 additions & 0 deletions lib/simplecov/exit_codes.rb
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,10 @@ module ExitCodes
# warning logs. `print_errors false` remains the intended opt-out.
def self.print_error(message)
$stderr.puts message # rubocop:disable Style/StderrPuts
rescue IOError
# A parallel runner can close a worker's stderr before its at_exit
# hooks run (rspec-conductor does). The violation still has to set
# the exit status even when its explanation cannot be printed.
end
end
end
Expand Down
5 changes: 5 additions & 0 deletions lib/simplecov/formatter/base.rb
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,11 @@ def initialize(silent: false, output_dir: nil)
# Subclasses call this at the end of their `format`.
def emit_status(result)
$stderr.puts output_message(result) unless @silent # rubocop:disable Style/StderrPuts
rescue IOError
# A parallel runner can close a worker's stderr before its at_exit
# hooks run (rspec-conductor does). Losing the status line must not
# abort the exit tasks that follow report generation, i.e. the
# threshold checks and the .last_run.json write.
end

# Subclasses override to prepend a marker (e.g. "JSON ") to the
Expand Down
6 changes: 4 additions & 2 deletions lib/simplecov/parallel_adapters.rb
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,10 @@ module SimpleCov
# (precise sync + first-process detection via the gem's own API).
# - `GenericAdapter` — env-var-only detection for runners that follow
# the parallel_tests `TEST_ENV_NUMBER` convention but don't ship a
# Ruby API (parallel_rspec, custom CI sharding, knapsack-style
# splitters). See https://github.com/simplecov-ruby/simplecov/issues/1065.
# Ruby API (parallel_rspec, rspec-conductor 1.0.7+, custom CI
# sharding, knapsack-style splitters). See
# https://github.com/simplecov-ruby/simplecov/issues/1065 and
# https://github.com/simplecov-ruby/simplecov/issues/1156.
#
# Users can plug in additional adapters:
#
Expand Down
13 changes: 7 additions & 6 deletions lib/simplecov/parallel_adapters/generic.rb
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,15 @@ module ParallelAdapters
# Catch-all adapter for parallel test runners that follow the
# `TEST_ENV_NUMBER` / `PARALLEL_TEST_GROUPS` env-var convention but
# don't ship a Ruby API for SimpleCov to hook (parallel_rspec,
# knapsack-style splitters, custom CI sharding scripts). Activates
# when `TEST_ENV_NUMBER` is set; doesn't require any specific gem to
# be loaded.
# rspec-conductor since 1.0.7, knapsack-style splitters, custom CI
# sharding scripts). Activates when `TEST_ENV_NUMBER` is set;
# doesn't require any specific gem to be loaded.
#
# Heuristic for `first_worker?`: the worker whose `TEST_ENV_NUMBER`
# is `""` (parallel_tests/parallel_rspec convention) or `"1"`
# (zero-based runners that start at 1). Any other value is treated
# as a non-first worker.
# is `""` (parallel_tests/parallel_rspec convention, rspec-conductor's
# default) or `"1"` (runners that number from 1, like rspec-conductor
# with `--first-is-1`). Any other value is treated as a non-first
# worker.
#
# `wait_for_siblings` is inherited from Base as a no-op — without a
# runner-provided API the only synchronization available is polling
Expand Down
7 changes: 7 additions & 0 deletions spec/color_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,13 @@
expect(described_class.enabled?(stdout_not_tty)).to be false
end

it "returns false for a closed stream (a parallel runner can close worker stdio before at_exit runs)" do
read, write = IO.pipe
read.close
write.close
expect(described_class.enabled?(write)).to be false
end

it "falls through when SimpleCov has no color config (standalone CLI)" do
# In the standalone `exe/simplecov` process, `simplecov/color` is
# loaded without `simplecov/configuration`, so `SimpleCov` doesn't
Expand Down
17 changes: 17 additions & 0 deletions spec/exit_codes_spec.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# frozen_string_literal: true

require "helper"

RSpec.describe SimpleCov::ExitCodes do
describe ".print_error" do
it "writes the message to stderr" do
expect(capture_stderr { described_class.print_error("boom") }).to eq("boom\n")
end

it "stays quiet when stderr was closed before at_exit (e.g. by rspec-conductor)" do
allow($stderr).to receive(:puts).and_raise(IOError, "closed stream")

expect { described_class.print_error("boom") }.not_to raise_error
end
end
end
7 changes: 7 additions & 0 deletions spec/formatter/base_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,13 @@ def result_with(coverage_statistics)
expect(Warning).not_to have_received(:warn)
end

it "loses the status line quietly when stderr was closed before at_exit (e.g. by rspec-conductor)" do
result = result_with(line: statistics(8, 2))
allow($stderr).to receive(:puts).and_raise(IOError, "closed stream")

expect { formatter.send(:emit_status, result) }.not_to raise_error
end

it "does not build or emit a status when silent" do
silent_formatter = described_class.new(silent: true)
result = result_with(line: statistics(8, 2))
Expand Down
16 changes: 16 additions & 0 deletions test_projects/rspec_conductor/Gemfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# frozen_string_literal: true

source "https://rubygems.org"

gem "rspec"
gem "rspec-conductor"

# when the tests are executed the project is in tmp/aruba/project
# which is a different nesting from its usual place
if File.exist?("../../simplecov.gemspec")
gem "simplecov", path: "../.."
else
# rubocop:disable Bundler/DuplicatedGem
gem "simplecov", path: "../../.."
# rubocop:enable Bundler/DuplicatedGem
end
46 changes: 46 additions & 0 deletions test_projects/rspec_conductor/Gemfile.lock
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
PATH
remote: ../..
specs:
simplecov (1.1.0)

GEM
remote: https://rubygems.org/
specs:
diff-lcs (1.6.2)
rspec (3.13.2)
rspec-core (~> 3.13.0)
rspec-expectations (~> 3.13.0)
rspec-mocks (~> 3.13.0)
rspec-conductor (1.0.11)
rspec-core (>= 3.8.0)
rspec-core (3.13.6)
rspec-support (~> 3.13.0)
rspec-expectations (3.13.5)
diff-lcs (>= 1.2.0, < 2.0)
rspec-support (~> 3.13.0)
rspec-mocks (3.13.8)
diff-lcs (>= 1.2.0, < 2.0)
rspec-support (~> 3.13.0)
rspec-support (3.13.7)

PLATFORMS
arm64-darwin-25
ruby

DEPENDENCIES
rspec
rspec-conductor
simplecov!

CHECKSUMS
diff-lcs (1.6.2) sha256=9ae0d2cba7d4df3075fe8cd8602a8604993efc0dfa934cff568969efb1909962
rspec (3.13.2) sha256=206284a08ad798e61f86d7ca3e376718d52c0bc944626b2349266f239f820587
rspec-conductor (1.0.11) sha256=6958c98d4bf07cff6827229160debca81c8f13dd550dd8e1db02e63e3c092618
rspec-core (3.13.6) sha256=a8823c6411667b60a8bca135364351dda34cd55e44ff94c4be4633b37d828b2d
rspec-expectations (3.13.5) sha256=33a4d3a1d95060aea4c94e9f237030a8f9eae5615e9bd85718fe3a09e4b58836
rspec-mocks (3.13.8) sha256=086ad3d3d17533f4237643de0b5c42f04b66348c28bf6b9c2d3f4a3b01af1d47
rspec-support (3.13.7) sha256=0640e5570872aafefd79867901deeeeb40b0c9875a36b983d85f54fb7381c47c
simplecov (1.1.0)

BUNDLED WITH
4.0.18