test(cluster): unit tests for cluster_getter - #3568
Conversation
`_init_use_resources` iterates `use_resources` twice, but its `ResourcesType` contract permits any iterable - a one-shot iterator would be exhausted after the first pass and every resource filter (e.g. `OneOf`) silently dropped. The production caller materialized the iterable itself, so the defect was latent, reachable only by calling the method with an iterator directly. Move the materialization into `_init_use_resources`, where the double iteration lives, so the method honors its own signature. The caller no longer converts `use_resources` (nor re-wraps the returned list), and the return type now says `list` explicitly.
Cover the previously untested `ClusterGetter` functionality: * `_make_instances_order` - light tests iterate the tail instances first, heavy tests last, single instance edge case, head order randomization. * `_init_use_resources` - `CLUSTER` always added, locked resources filtered out, filter objects preserved, one-shot iterator input. * `_test_needs_respin` - custom scripts and the non-initial marked test exception. * `_update_marked_tests` - the whole staleness matrix: stale mark cleaned up with respin conversion, marks kept during respin, fresh idle mark kept, running mark refreshed (or not, when fresh, and never cleaned no matter how old), the newest record deciding for the whole mark, mixed staleness of multiple marks on one instance, no writes when no marks exist. * `_resolve_resources_availability` - lock/use conflicts, lock implied in-use, `OneOf` filter resolution, resolved resources recorded only on success. * `_init_prio` and `_wait_for_prio` - all the no-wait exceptions. * `_respun_by_other_worker`, `_is_already_running`, `_finish_respin` two-phase contract, `_create_test_status_records` (plain, marked cleanup, unmarked cleanup), `_check_dead_fraction` thresholds and the `_fail_on_dead_clusters` strict check window.
b0e4e7c to
2275a08
Compare
There was a problem hiding this comment.
🟢 Ready to approve
The functional change is small and clearly correct, and it is backed by substantial, targeted unit test coverage for the affected and related behaviors.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Pull request overview
This PR expands unit-test coverage for the cluster scheduler’s ClusterGetter internals, adding broad behavioral checks around instance selection, resource initialization/resolution, mark staleness handling, respin/priority flows, and dead-cluster thresholds. It also fixes a latent contract violation in _init_use_resources by materializing use_resources internally so one-shot iterators aren’t accidentally exhausted across multiple passes.
Changes:
- Add extensive unit tests for key
ClusterGetterhelpers (ordering, resource init/availability resolution, marked-test staleness, respin/priority helpers, status record creation, dead-cluster checks). - Fix
_init_use_resourcesto eagerly materializeuse_resources, preventing loss of filter objects (e.g.,OneOf) whenuse_resourcesis a one-shot iterator. - Simplify
get_cluster_instanceby removing redundantuse_resourcesmaterialization and using the new_init_use_resourcesreturn value directly.
File summaries
| File | Description |
|---|---|
| framework_tests/test_cluster_getter.py | Adds a large suite of focused unit tests covering ClusterGetter internal behaviors and edge cases. |
| cardano_node_tests/cluster_management/cluster_getter.py | Fixes _init_use_resources iterator contract and removes redundant list conversion in get_cluster_instance. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0
- Review effort level: Low
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
What this does
Adds unit tests for the previously untested
ClusterGetterfunctionality, building on the
test_cluster_getter.pyfoundationfrom #3567. 40 new tests covering:
_make_instances_order- light tests iterate the two tail instancesfirst (fixed order), heavy tests last, head order randomization,
single-instance edge case
_init_use_resources-CLUSTERalways added, locked resourcesfiltered out of "use", filter objects passed through, one-shot
iterator input
_test_needs_respin- custom scripts and the non-initial markedtest exception
_update_marked_tests- the staleness matrix: stale mark cleaned upwith "respin after mark" conversion, marks kept while the instance
is respun, fresh idle mark kept, running mark refreshed (not
refreshed while fresh so polling stays write-free, and never cleaned
no matter how old), the newest record deciding staleness for the
whole mark, mixed staleness of multiple marks on one instance, and
no database writes when no marks exist
_resolve_resources_availability- lock/use conflicts in alldirections, locked-implies-in-use dedup,
OneOffilter resolution,resolved resources recorded only on success
_init_prio/_wait_for_prio- waiting on a priority test and allthree no-wait exceptions
_respun_by_other_worker,_is_already_running,_finish_respintwo-phase contract,
_create_test_status_records(plain test,marked cleanup, unmarked cleanup,
(setup)suffix stripping),_check_dead_fractionthresholds and the_fail_on_dead_clustersstrict check window
Not covered (needs a live cluster):
_respin,_is_healthy,_cluster_needs_respinand theget_cluster_instanceloop itself.Latent contract violation caught while writing the tests
test_one_shot_iteratorexposed that_init_use_resourcesviolatesits own
Iterablecontract: it iteratesuse_resourcestwice, so aone-shot iterator is exhausted after the first pass and every resource
filter (e.g.
OneOf) is silently dropped. Latent on the productionpath - the caller materialized the argument - but reachable by any
direct call with an iterator. The first commit moves the
materialization into the method, drops the caller's now-redundant
conversion and re-wrap, and makes the return type an explicit
list.Verification
make lintclean, each commit testedindividually.
mutation-tested by the reviewers (e.g.
max->minin thestaleness fold, dropped light-test condition, unconditional mark
refresh - all caught by the tests).