Skip to content

feat: add --repeat option to run scenarios a fixed number of times - #383

Draft
Willem-Jan Spoel (willemjanspoelphilips) wants to merge 2 commits into
philips-software:mainfrom
willemjanspoelphilips:feature/repeat-scenario-option
Draft

feat: add --repeat option to run scenarios a fixed number of times#383
Willem-Jan Spoel (willemjanspoelphilips) wants to merge 2 commits into
philips-software:mainfrom
willemjanspoelphilips:feature/repeat-scenario-option

Conversation

@willemjanspoelphilips

Copy link
Copy Markdown

Summary

Adds a --repeat <N> runner option that runs each matching scenario a fixed
number of times unconditionally. It is the counterpart to --retry: where
--retry loops while a scenario is failing (to absorb flakiness and stops
early on the first pass), --repeat loops always (to sample or measure a
scenario), regardless of the outcome of each execution.

Motivation

--retry only re-runs failing scenarios and stops as soon as one passes, so it
cannot be used to execute a passing scenario a fixed number of times. Collecting
multiple executions of the same scenario — for example to measure timing, warm a
cache, or observe non-deterministic behaviour — currently has no first-class
option. --repeat fills that gap while reusing the existing attempt machinery.

What it does

  • --repeat <N>: run every matching scenario N times unconditionally.
  • --repeat-tag-filter <expr>: restrict repetition to scenarios matching a tag
    expression (mirrors --retry-tag-filter).

Semantics:

  • Iteration count only. --repeat governs how many times a scenario runs;
    it does not change pass/fail behaviour. The run still fails if any execution
    fails (no early exit on a pass — that is the one behavioural difference from
    --retry).
  • Reported as attempts on the same test case. Repeated executions surface as
    incrementing attempt values on a single testCase, with willBeRetried
    transitioning true → … → false on the final execution. This matches the
    message shape already produced by --retry, so a repeated scenario collapses
    to a single counted scenario in summaries and downstream tooling instead of
    inflating the scenario count.
  • Aggregation lives in hooks. Combining the repeated executions (e.g.
    averaging a measurement) is intentionally left to HOOK_AFTER_FEATURE /
    HOOK_AFTER_ALL, keeping the runner option focused on execution.

--repeat vs --retry

--retry --repeat
Loops while failing always
Stops early on a pass yes no
Purpose absorb flakiness sampling / measurement
Run status passes once a retry passes fails if any execution fails
Message shape incrementing attempts, one case identical

In this first cut, --repeat and --retry are mutually exclusive;
combining them is rejected with a clear error. Composing them (retry inner,
repeat outer) is a possible follow-up.

Tests

  • Adds a compatibility kit that exercises --repeat end-to-end and pins the
    emitted Cucumber messages (three executions of one scenario surface as
    attempt 0/1/2 with willBeRetried true/true/false, collapsing to a
    single counted scenario).
  • Existing --retry behaviour is unchanged (verified against its compatibility
    kit).

Add a --repeat N runner option (with --repeat-tag-filter) that runs each matching scenario N times unconditionally, for sampling/measurement. Unlike --retry it never stops early on a pass and the run fails if any execution fails.

Repeats surface as incrementing attempts on the same test case (will_be_retried true->false), so they collapse to a single counted scenario, matching the retry message shape. --repeat and --retry are mutually exclusive in this first cut. Adds a compatibility kit (compatibility/repeat) covering the emitted messages.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new runner capability to execute each matching scenario a fixed number of times (--repeat N) while reusing the existing “attempt” message shape (similar to --retry) so downstream tooling continues to treat repeated executions as a single test case with multiple attempts.

Changes:

  • Introduces --repeat <N> plus --repeat-tag-filter <expr> and wires both into RunOptions / CLI parsing.
  • Implements repeated execution in the runtime worker by running the same test case multiple times and emitting incrementing attempt values, with willBeRetried staying true until the final execution.
  • Adds a compatibility kit validating the emitted message stream for repeated scenarios and documents usage in the README.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
README.md Documents --repeat usage and its relationship to hooks and --retry.
cucumber_cpp/library/support/Types.hpp Extends runtime options with repeat and repeatTagExpression.
cucumber_cpp/library/runtime/Worker.cpp Adds repeat-selection logic and loops test-case execution per scenario.
cucumber_cpp/library/runtime/TestCaseRunner.hpp Extends runner API to accept attempt offsets and “will repeat” signaling.
cucumber_cpp/library/runtime/TestCaseRunner.cpp Propagates attempt offsets and sets will_be_retried for repeats.
cucumber_cpp/library/Application.hpp Adds CLI option storage for repeat and repeat tag filter.
cucumber_cpp/library/Application.cpp Adds CLI flags/options and builds runtime options for repeat.
compatibility/repeat/repeat.ndjson New expected message transcript for --repeat behavior.
compatibility/repeat/repeat.feature New compatibility feature defining the repeated scenario.
compatibility/repeat/repeat.cpp Step implementation for the repeat compatibility kit.
compatibility/repeat/repeat.arguments.txt Arguments used to run the repeat compatibility kit.
compatibility/BaseCompatibility.cpp Wires repeat options into the compatibility harness configuration.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +147 to +151
const auto repeats = RepeatsForPickle(assembledTestCase.pickle, options);

bool allPassed = true;

const auto status = testCaseRunner.Run();
for (std::size_t repeat = 0; repeat < repeats; ++repeat)
cli.add_option("--retry-tag-filter", options.retryTagFilter, "Only retry scenarios matching this tag expression")->needs(retryOpt);
auto* repeatOpt = cli.add_option("--repeat", options.repeat, "Number of times to run matching scenarios unconditionally (for sampling/measurement); unlike --retry it does not stop early on pass")->default_val(options.repeat);
cli.add_option("--repeat-tag-filter", options.repeatTagFilter, "Only repeat scenarios matching this tag expression")->needs(repeatOpt);
repeatOpt->excludes(retryOpt);

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Suppressed comments (1)

cucumber_cpp/library/runtime/Worker.cpp:151

  • When --fail-fast is triggered (or --dry-run is set), the TestCaseRunner is constructed with skip=true so the scenario is not executed; however the new --repeat loop still runs repeats times, emitting multiple skipped attempts for the same scenario. This makes fail-fast/dry-run output noisy and is inconsistent with existing retry behavior (skipped cases only produce a single attempt). Consider forcing repeats=1 when skip is true (and reuse a single computed retries/skip).
        const auto repeats = RepeatsForPickle(assembledTestCase.pickle, options);

        bool allPassed = true;

        for (std::size_t repeat = 0; repeat < repeats; ++repeat)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants