Skip to content

feat(ci): test execute, consume and check_eip_versions via collect-only - #3451

Open
danceratopz wants to merge 4 commits into
ethereum:forks/amsterdamfrom
danceratopz:ci-execute-collect
Open

feat(ci): test execute, consume and check_eip_versions via collect-only#3451
danceratopz wants to merge 4 commits into
ethereum:forks/amsterdamfrom
danceratopz:ci-execute-collect

Conversation

@danceratopz

@danceratopz danceratopz commented Aug 26, 2026

Copy link
Copy Markdown
Member

Description

Add cheap collect-only CI guards for our pytest commands. This would have caught the collection crash behind #3438 (fixed in #3450).

  • pytest_hive: This also allows --collect-only with hive simulators w/o actually having to connect to a hive dev backend.
  • New just execute-collect: collect the full tests/ tree via both the execute remote and execute hive plugin stacks (this reproduces the eels/execute, hive mode internal error #3438 crash without the fix in fix(test-execute): ignore tests/spec_tools during collection #3450). Asserts a collected-count floor so a drastic silent shrink also fails.
  • New just consume-collect: fill a mini fixture set (test_dup.py, 16 cases, all formats), then collect it via consume engine/rlp/enginex/direct with exact-count assertions: an empty parametrization (e.g. a missing fixture format) collects a single placeholder item and exits 0, so counts must be checked. Notably, consume enginex currently has no CI coverage at all: its hive-consume.yaml matrix rows are commented out.
  • New just check-eip-versions-collect: collect the EIP version checks over the full tests/ tree with the same collected-count floor. This is the guard for the failure mode fixed in fix(test-cli): ignore tests/json_loader and tests/spec_tools during collection #3461 (verified: it exits non-zero with fix(test-cli): ignore tests/json_loader and tests/spec_tools during collection #3461's parent commit, passes with the fix).
  • Run the three recipes as jobs in test.yaml on every PR.
  • hive-execute.yaml: also trigger on changes to the shared plugins and pytest-execute.ini, both of which the E2E consumes.

Related Issues or PRs

Related to #3438; requires #3450 and #3461 (both merged).

Checklist

  • Ran fast static checks to avoid CI fails, see Code Standards & Verifying Changes: just static
  • PR title has the form <type>(<area>): <title>, where <type> and <area> come from an appropriate C-<type>, respectively A-<area>, label. The title should match the target squash commit message.

Cute Animal Picture

 /\_/\
( o.o )
 > ^ <

@codecov

codecov Bot commented Aug 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.01%. Comparing base (947ef52) to head (ad31c64).
⚠️ Report is 2 commits behind head on forks/amsterdam.

Additional details and impacted files
@@               Coverage Diff                @@
##           forks/amsterdam    #3451   +/-   ##
================================================
  Coverage            94.01%   94.01%           
================================================
  Files                  624      624           
  Lines                36905    36905           
  Branches              3326     3326           
================================================
  Hits                 34695    34695           
  Misses                1533     1533           
  Partials               677      677           
Flag Coverage Δ
unittests 94.01% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@danceratopz danceratopz added C-feat Category: an improvement or new feature A-test-consume Area: execution_testing.cli.pytest_commands.plugins.consume A-test-execute Area: execution_testing.cli.pytest_commands.plugins.execute A-ci Area: Continuous Integration labels Aug 26, 2026
@danceratopz
danceratopz marked this pull request as draft August 26, 2026 08:33
@danceratopz
danceratopz marked this pull request as ready for review August 26, 2026 08:57
Comment thread Justfile Outdated
@danceratopz
danceratopz force-pushed the ci-execute-collect branch 2 times, most recently from c3daad7 to 21b481a Compare August 28, 2026 07:37
@danceratopz danceratopz changed the title feat(ci): collect execute and consume tests on every PR feat(ci): collect execute, consume and check_eip_versions tests on every PR Aug 28, 2026
@danceratopz
danceratopz marked this pull request as draft August 31, 2026 08:10
@danceratopz danceratopz changed the title feat(ci): collect execute, consume and check_eip_versions tests on every PR feat(ci): test execute, consume and check_eip_versions via collect-only Sep 3, 2026
@danceratopz
danceratopz marked this pull request as ready for review September 3, 2026 09:22

@spencer-tb spencer-tb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! One nit/idea :)

Comment thread Justfile Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-ci Area: Continuous Integration A-test-consume Area: execution_testing.cli.pytest_commands.plugins.consume A-test-execute Area: execution_testing.cli.pytest_commands.plugins.execute C-feat Category: an improvement or new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants