Skip to content

test(cluster): unit tests for cluster_getter - #3568

Merged
mkoura merged 2 commits into
masterfrom
cluster_getter_tests
Jul 29, 2026
Merged

test(cluster): unit tests for cluster_getter#3568
mkoura merged 2 commits into
masterfrom
cluster_getter_tests

Conversation

@mkoura

@mkoura mkoura commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

What this does

Adds unit tests for the previously untested ClusterGetter
functionality, building on the test_cluster_getter.py foundation
from #3567. 40 new tests covering:

  • _make_instances_order - light tests iterate the two tail instances
    first (fixed order), heavy tests last, head order randomization,
    single-instance edge case
  • _init_use_resources - CLUSTER always added, locked resources
    filtered out of "use", filter objects passed through, one-shot
    iterator input
  • _test_needs_respin - custom scripts and the non-initial marked
    test exception
  • _update_marked_tests - the staleness matrix: stale mark cleaned up
    with "respin after mark" conversion, marks kept while the instance
    is respun, fresh idle mark kept, running mark refreshed (not
    refreshed while fresh so polling stays write-free, and never cleaned
    no matter how old), the newest record deciding staleness for the
    whole mark, mixed staleness of multiple marks on one instance, and
    no database writes when no marks exist
  • _resolve_resources_availability - lock/use conflicts in all
    directions, locked-implies-in-use dedup, OneOf filter resolution,
    resolved resources recorded only on success
  • _init_prio / _wait_for_prio - waiting on a priority test and all
    three no-wait exceptions
  • _respun_by_other_worker, _is_already_running, _finish_respin
    two-phase contract, _create_test_status_records (plain test,
    marked cleanup, unmarked cleanup, (setup) suffix stripping),
    _check_dead_fraction thresholds and the _fail_on_dead_clusters
    strict check window

Not covered (needs a live cluster): _respin, _is_healthy,
_cluster_needs_respin and the get_cluster_instance loop itself.

Latent contract violation caught while writing the tests

test_one_shot_iterator exposed that _init_use_resources violates
its own Iterable contract: it iterates use_resources twice, so a
one-shot iterator is exhausted after the first pass and every resource
filter (e.g. OneOf) is silently dropped. Latent on the production
path - the caller materialized the argument - but reachable by any
direct call with an iterator. The first commit moves the
materialization into the method, drops the caller's now-redundant
conversion and re-wrap, and makes the return type an explicit list.

Verification

  • All 90 framework tests pass, make lint clean, each commit tested
    individually.
  • Reviewed in a review-fix loop until convergence; the test suite was
    mutation-tested by the reviewers (e.g. max -> min in the
    staleness fold, dropped light-test condition, unconditional mark
    refresh - all caught by the tests).

@mkoura
mkoura requested a review from saratomaz as a code owner July 29, 2026 18:05
mkoura added 2 commits July 29, 2026 20:17
`_init_use_resources` iterates `use_resources` twice, but its
`ResourcesType` contract permits any iterable - a one-shot iterator
would be exhausted after the first pass and every resource filter
(e.g. `OneOf`) silently dropped. The production caller materialized
the iterable itself, so the defect was latent, reachable only by
calling the method with an iterator directly.

Move the materialization into `_init_use_resources`, where the double
iteration lives, so the method honors its own signature. The caller
no longer converts `use_resources` (nor re-wraps the returned list),
and the return type now says `list` explicitly.
Cover the previously untested `ClusterGetter` functionality:

* `_make_instances_order` - light tests iterate the tail instances
  first, heavy tests last, single instance edge case, head order
  randomization.
* `_init_use_resources` - `CLUSTER` always added, locked resources
  filtered out, filter objects preserved, one-shot iterator input.
* `_test_needs_respin` - custom scripts and the non-initial marked
  test exception.
* `_update_marked_tests` - the whole staleness matrix: stale mark
  cleaned up with respin conversion, marks kept during respin, fresh
  idle mark kept, running mark refreshed (or not, when fresh, and
  never cleaned no matter how old), the newest record deciding for
  the whole mark, mixed staleness of multiple marks on one instance,
  no writes when no marks exist.
* `_resolve_resources_availability` - lock/use conflicts, lock implied
  in-use, `OneOf` filter resolution, resolved resources recorded only
  on success.
* `_init_prio` and `_wait_for_prio` - all the no-wait exceptions.
* `_respun_by_other_worker`, `_is_already_running`, `_finish_respin`
  two-phase contract, `_create_test_status_records` (plain, marked
  cleanup, unmarked cleanup), `_check_dead_fraction` thresholds and
  the `_fail_on_dead_clusters` strict check window.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Ready to approve

The functional change is small and clearly correct, and it is backed by substantial, targeted unit test coverage for the affected and related behaviors.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

This PR expands unit-test coverage for the cluster scheduler’s ClusterGetter internals, adding broad behavioral checks around instance selection, resource initialization/resolution, mark staleness handling, respin/priority flows, and dead-cluster thresholds. It also fixes a latent contract violation in _init_use_resources by materializing use_resources internally so one-shot iterators aren’t accidentally exhausted across multiple passes.

Changes:

  • Add extensive unit tests for key ClusterGetter helpers (ordering, resource init/availability resolution, marked-test staleness, respin/priority helpers, status record creation, dead-cluster checks).
  • Fix _init_use_resources to eagerly materialize use_resources, preventing loss of filter objects (e.g., OneOf) when use_resources is a one-shot iterator.
  • Simplify get_cluster_instance by removing redundant use_resources materialization and using the new _init_use_resources return value directly.
File summaries
File Description
framework_tests/test_cluster_getter.py Adds a large suite of focused unit tests covering ClusterGetter internal behaviors and edge cases.
cardano_node_tests/cluster_management/cluster_getter.py Fixes _init_use_resources iterator contract and removes redundant list conversion in get_cluster_instance.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0
  • Review effort level: Low

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

@mkoura
mkoura merged commit 3d749e5 into master Jul 29, 2026
4 checks passed
@mkoura
mkoura deleted the cluster_getter_tests branch July 29, 2026 19:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants