Skip to content

feat: add examples tag to scenario tags#771

Open
chapellu wants to merge 7 commits into
pytest-dev:masterfrom
chapellu:master
Open

feat: add examples tag to scenario tags#771
chapellu wants to merge 7 commits into
pytest-dev:masterfrom
chapellu:master

Conversation

@chapellu

@chapellu chapellu commented Apr 2, 2025

Copy link
Copy Markdown

Summary

Scenario Outlines can carry tags on individual Examples: blocks, but those
tags were lost when each example row was rendered into a concrete scenario.
Gherkin parses them; pytest-bdd just never propagated them. This PR carries an
Examples: block's tags onto every scenario generated from that block, so they
behave like any other scenario tag (selection, pytest_bdd_apply_tag, marks).

Motivation

Given a tagged examples table:

Scenario Outline: Login attempts
  Given the user enters "<username>"
  Then the result is "<result>"

  @smoke
  Examples:
    | username | result  |
    | alice    | success |

  @edge-case
  Examples:
    | username | result  |
    | ""       | error   |

Before this PR, neither @smoke nor @Edge-Case reached the generated
scenarios. After it, each generated scenario inherits the tags of the
Examples: block it came from (in addition to the scenario- and feature-level
tags). This makes it possible to filter or mark specific example sets.

What changed

  • ScenarioTemplate.render() now merges the matching Examples block's tags
    into the rendered Scenario.tags.
  • The matching is done by comparing the render context against every row of
    every Examples block (via Examples.as_contexts() — the same generator
    that produces the context), so it works for multi-row blocks and multiple
    blocks, not just the first row.

Tests

  • test_render_scenario_with_example_tags — example tags reach the rendered
    scenario.
  • test_render_scenario_with_example_tags_non_first_row — tags are applied
    regardless of which row the context comes from.
  • Parsing tests for the extended feature fixture.
  • Replaced test_step_outside_scenario_or_background_error with
    test_step_before_first_scenario_is_parsed_as_description: with
    gherkin-official >=29, a step before the first Scenario/Background is no
    longer an error — it is parsed as the feature description. The old test
    asserted a FeatureError that no supported gherkin version raises. See the
    discussion below.

Notes

  • Rebased onto current main.
  • CI matrix: main is currently red independently of this PR (pytest 7.x is
    incompatible with the current code, and gherkin-official 31.0.0 is an
    anomalous release where three unrelated tests fail). I restricted the tox
    matrix to the supported versions so this PR is green; happy to split that
    into its own PR — details in the comment thread.

@jsa34

jsa34 commented Apr 9, 2025

Copy link
Copy Markdown
Collaborator

Thanks for the PR!

Could you add tests for this?

@chapellu

chapellu commented Apr 14, 2025

Copy link
Copy Markdown
Author

@jsa34 I've added some tests but I have issue with the test test_step_outside_scenario_or_background_error that is no longer working with a strange behavior of pytest inception...

@chapellu

Copy link
Copy Markdown
Author

@jsa34 I have the feeling that a step outside a scenario no longer raised an error in get_gherkin_document

@chapellu

Copy link
Copy Markdown
Author

@jsa34 I finally had time to debug a bit and the issue is that the step outside the scenario is interpreted as a comment
image

@chapellu

Copy link
Copy Markdown
Author

@jsa34 I'm removing the failing test as in my point of view the behavior that was tested is no longer relevant.
If you do not agree don't hesitate to comment otherwise the PR will be ready for you to merge

chapellu and others added 6 commits June 17, 2026 09:39
gherkin-official >=29 parses a Given/When/Then line that appears before
the first Scenario or Background as part of the feature description rather
than raising a parse error. The old test_step_outside_scenario_or_background_error
asserted a FeatureError that no supported gherkin version (29..37, latest)
emits anymore, so it failed across the entire tox matrix.

Replace it with a parser-level test that pins the actual current behavior:
the orphan step lands in feature.description and the scenario still parses.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
find_scenario_example_from_context only compared the first row of each
Examples block, so any parametrization context coming from a later row
failed to match and lost its block's tags. Reuse Examples.as_contexts()
- the same generator that produces the rendering context - so matching
covers every row of every block.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The suite does not pass on two dependency versions, independently of this
PR (the same jobs are red on the latest main CI run):

- pytest 7.x: ~12 tests in tests/feature and tests/args fail because the
  current code relies on pytest 8 step-error reporting semantics.
- gherkin-official 31.0.0: an anomalous release; tests pass on 29, 30 and
  32..latest but test_outline_with_escaped_pipes, test_scenario_comments
  and test_parser fail only on 31.

Restrict the matrix to the versions the suite actually supports. Verified
the full py3.12 matrix (pytest 8.0..latest x gherkin 29,30,32..latest)
passes locally.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@chapellu

Copy link
Copy Markdown
Author

@jsa34 @youtux — reviving this one, sorry for the long gap. I've rebased onto current main and cleaned everything up. Summary of where it stands:

What this PR does

Propagates tags declared on an Examples: block onto the scenarios generated from that block. Gherkin already parses Examples-level tags, but they were dropped during rendering — so you
couldn't select/mark scenario-outline rows by their Examples tag. Example:

Scenario Outline: ...
Given
@smoke
Examples:
| x |
| 1 |

@smoke now ends up on the generated scenario's tags.

On the test you asked about

Back in April I'd removed test_step_outside_scenario_or_background_error because it failed and I couldn't explain it. I've now traced it properly: since gherkin-official >=29, a step before
the first Scenario/Background is no longer a parse error — gherkin folds it into the feature description (I'd earlier mis-described it as a comment). That FeatureError is never raised on
any supported gherkin version, so the test asserted behaviour that no longer exists. Rather than delete it silently, I've replaced it with
test_step_before_first_scenario_is_parsed_as_description (in tests/parser/test_parser.py) that pins the actual current behaviour.

About CI

While getting this green I found main's own CI is red independently of this PR — the same five jobs fail on the latest main run. Two pre-existing causes in the tox matrix:

  • pytest 7.x: ~12 tests in tests/feature/tests/args fail because the current code relies on pytest 8 step-error reporting semantics.
  • gherkin-official 31.0.0: an anomalous release — the suite passes on 29, 30 and 32→latest, but test_outline_with_escaped_pipes, test_scenario_comments and test_parser fail only on 31.

I've restricted the matrix to the versions the suite actually supports (pytest>=8.0, gherkin 29,30,32..latest); the full matrix is green now on my fork. Happy to pull that matrix change
into its own PR if you'd rather keep this one scoped to the feature — it fixes main's red CI on its own, so it might be worth merging first regardless.

Could you approve the workflow run on this PR and take another look when you have a moment? Glad to adjust anything.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants