feat: add --repeat option to run scenarios a fixed number of times - #383
Draft
Willem-Jan Spoel (willemjanspoelphilips) wants to merge 2 commits into
Conversation
Add a --repeat N runner option (with --repeat-tag-filter) that runs each matching scenario N times unconditionally, for sampling/measurement. Unlike --retry it never stops early on a pass and the run fails if any execution fails. Repeats surface as incrementing attempts on the same test case (will_be_retried true->false), so they collapse to a single counted scenario, matching the retry message shape. --repeat and --retry are mutually exclusive in this first cut. Adds a compatibility kit (compatibility/repeat) covering the emitted messages.
Willem-Jan Spoel (willemjanspoelphilips)
requested a review
from a team
as a code owner
August 12, 2026 11:15
Willem-Jan Spoel (willemjanspoelphilips)
requested a review
from a team
as a code owner
August 12, 2026 11:15
Copilot started reviewing on behalf of
Willem-Jan Spoel (willemjanspoelphilips)
August 12, 2026 11:16
View session
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new runner capability to execute each matching scenario a fixed number of times (--repeat N) while reusing the existing “attempt” message shape (similar to --retry) so downstream tooling continues to treat repeated executions as a single test case with multiple attempts.
Changes:
- Introduces
--repeat <N>plus--repeat-tag-filter <expr>and wires both intoRunOptions/ CLI parsing. - Implements repeated execution in the runtime worker by running the same test case multiple times and emitting incrementing
attemptvalues, withwillBeRetriedstayingtrueuntil the final execution. - Adds a compatibility kit validating the emitted message stream for repeated scenarios and documents usage in the README.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Documents --repeat usage and its relationship to hooks and --retry. |
| cucumber_cpp/library/support/Types.hpp | Extends runtime options with repeat and repeatTagExpression. |
| cucumber_cpp/library/runtime/Worker.cpp | Adds repeat-selection logic and loops test-case execution per scenario. |
| cucumber_cpp/library/runtime/TestCaseRunner.hpp | Extends runner API to accept attempt offsets and “will repeat” signaling. |
| cucumber_cpp/library/runtime/TestCaseRunner.cpp | Propagates attempt offsets and sets will_be_retried for repeats. |
| cucumber_cpp/library/Application.hpp | Adds CLI option storage for repeat and repeat tag filter. |
| cucumber_cpp/library/Application.cpp | Adds CLI flags/options and builds runtime options for repeat. |
| compatibility/repeat/repeat.ndjson | New expected message transcript for --repeat behavior. |
| compatibility/repeat/repeat.feature | New compatibility feature defining the repeated scenario. |
| compatibility/repeat/repeat.cpp | Step implementation for the repeat compatibility kit. |
| compatibility/repeat/repeat.arguments.txt | Arguments used to run the repeat compatibility kit. |
| compatibility/BaseCompatibility.cpp | Wires repeat options into the compatibility harness configuration. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+147
to
+151
| const auto repeats = RepeatsForPickle(assembledTestCase.pickle, options); | ||
|
|
||
| bool allPassed = true; | ||
|
|
||
| const auto status = testCaseRunner.Run(); | ||
| for (std::size_t repeat = 0; repeat < repeats; ++repeat) |
| cli.add_option("--retry-tag-filter", options.retryTagFilter, "Only retry scenarios matching this tag expression")->needs(retryOpt); | ||
| auto* repeatOpt = cli.add_option("--repeat", options.repeat, "Number of times to run matching scenarios unconditionally (for sampling/measurement); unlike --retry it does not stop early on pass")->default_val(options.repeat); | ||
| cli.add_option("--repeat-tag-filter", options.repeatTagFilter, "Only repeat scenarios matching this tag expression")->needs(repeatOpt); | ||
| repeatOpt->excludes(retryOpt); |
Copilot started reviewing on behalf of
Willem-Jan Spoel (willemjanspoelphilips)
August 12, 2026 11:34
View session
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.
Suppressed comments (1)
cucumber_cpp/library/runtime/Worker.cpp:151
- When
--fail-fastis triggered (or--dry-runis set), theTestCaseRunneris constructed withskip=trueso the scenario is not executed; however the new--repeatloop still runsrepeatstimes, emitting multiple skipped attempts for the same scenario. This makes fail-fast/dry-run output noisy and is inconsistent with existing retry behavior (skipped cases only produce a single attempt). Consider forcingrepeats=1whenskipis true (and reuse a single computedretries/skip).
const auto repeats = RepeatsForPickle(assembledTestCase.pickle, options);
bool allPassed = true;
for (std::size_t repeat = 0; repeat < repeats; ++repeat)
Willem-Jan Spoel (willemjanspoelphilips)
marked this pull request as draft
August 12, 2026 13:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
--repeat <N>runner option that runs each matching scenario a fixednumber of times unconditionally. It is the counterpart to
--retry: where--retryloops while a scenario is failing (to absorb flakiness and stopsearly on the first pass),
--repeatloops always (to sample or measure ascenario), regardless of the outcome of each execution.
Motivation
--retryonly re-runs failing scenarios and stops as soon as one passes, so itcannot be used to execute a passing scenario a fixed number of times. Collecting
multiple executions of the same scenario — for example to measure timing, warm a
cache, or observe non-deterministic behaviour — currently has no first-class
option.
--repeatfills that gap while reusing the existing attempt machinery.What it does
--repeat <N>: run every matching scenarioNtimes unconditionally.--repeat-tag-filter <expr>: restrict repetition to scenarios matching a tagexpression (mirrors
--retry-tag-filter).Semantics:
--repeatgoverns how many times a scenario runs;it does not change pass/fail behaviour. The run still fails if any execution
fails (no early exit on a pass — that is the one behavioural difference from
--retry).incrementing
attemptvalues on a singletestCase, withwillBeRetriedtransitioning
true → … → falseon the final execution. This matches themessage shape already produced by
--retry, so a repeated scenario collapsesto a single counted scenario in summaries and downstream tooling instead of
inflating the scenario count.
averaging a measurement) is intentionally left to
HOOK_AFTER_FEATURE/HOOK_AFTER_ALL, keeping the runner option focused on execution.--repeatvs--retry--retry--repeatIn this first cut,
--repeatand--retryare mutually exclusive;combining them is rejected with a clear error. Composing them (retry inner,
repeat outer) is a possible follow-up.
Tests
--repeatend-to-end and pins theemitted Cucumber messages (three executions of one scenario surface as
attempt0/1/2 withwillBeRetriedtrue/true/false, collapsing to asingle counted scenario).
--retrybehaviour is unchanged (verified against its compatibilitykit).