Skip to content

fix(test-forks): report tests dropped by validity markers and by generation - #3468

Open
mkzung wants to merge 2 commits into
ethereum:forks/amsterdamfrom
mkzung:fix/report-validity-marker-deselections
Open

fix(test-forks): report tests dropped by validity markers and by generation#3468
mkzung wants to merge 2 commits into
ethereum:forks/amsterdamfrom
mkzung:fix/report-validity-marker-deselections

Conversation

@mkzung

@mkzung mkzung commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Description

A filter reports what it deselects and a validity marker does not, so two ways of dropping a test look different in the same run. pytest_collection_modifyitems builds filter_stats and prints a line per function for --fill-until and friends, while a param dropped because its fork is outside the marker's validity range leaves nothing behind.

Generation has the same gap one step earlier. When test_fork_set or the intersection with the selected forks comes out empty, pytest_generate_tests drops the test without parametrizing it at all, so it never reaches collection and no count anywhere accounts for it. Only -vv shows it, as a skip.

Two commits, one for each gap. One records validity-marker deselections beside the existing filter stats. Its pair stashes the node id and the reason whenever generation is about to drop a test unparametrized, then reports the total, so a run says how many tests it never generated rather than silently omitting them.

Three tests cover it: a param-level deselection where every case of a function goes, an unparametrized count, and that reporting line itself. Reverting forks.py alone fails all three.

Related Issues or PRs

Follows #3461, which fixed the config that could not collect these directories at all.

Checklist

  • Ran fast static checks to avoid CI fails, see Code Standards & Verifying Changes: just static
  • PR title has the form <type>(<area>): <title>, where <type> and <area> come from an appropriate C-<type>, respectively A-<area>, label. The title should match the target squash commit message.

Cute Animal Picture

   ,__,
   (oo)
   (__)

mkzung added 2 commits August 29, 2026 14:22
`filter_combinations` hands its removals to `pytest_deselected` and prints a
summary. The param-level validity path only deleted them, so they appeared in
no count. On `tests/` that hides 674 removals while the terminal reports 74.

Emptying a test stays non-fatal here: a narrow `--from`/`--until` range
legitimately excludes every param of a test written for later forks.
A test whose validity markers meet no selected fork is never parametrized, so
it becomes no item and no deselection hook can carry it. Against a development
fork this is the whole of ethereum#3260: `valid_at` names forks literally, a subclass
fork is never named, and thirteen tests leave no trace at all.

Collection now ends with a count of them, and names each at `-vv`. On `tests/`
that is one line for 771; `tests/osaka --until=Berlin` reports 67.
@codecov

codecov Bot commented Aug 31, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.90%. Comparing base (c4deda5) to head (4daa2ec).
⚠️ Report is 1 commits behind head on forks/amsterdam.

Additional details and impacted files
@@               Coverage Diff                @@
##           forks/amsterdam    #3468   +/-   ##
================================================
  Coverage            93.90%   93.90%           
================================================
  Files                  624      624           
  Lines                36905    36905           
  Branches              3326     3326           
================================================
  Hits                 34655    34655           
  Misses                1569     1569           
  Partials               681      681           
Flag Coverage Δ
unittests 93.90% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant