refactor(cluster): wall-clock mark staleness and xdist_group pairing

[mkoura]

Motivation

Follow-up to #3563. An abandoned test mark (a group of marked tests
that stopped running) used to be detected by counting scheduler loop
iterations - a heuristic whose expiry time depended on a single
worker's sleep backoff (roughly 25 seconds to 2 minutes) and where
each newly arriving worker started counting from zero. It also
required a cache dict threaded through get_cluster_instance.

What changed

Wall-clock mark staleness

The "current mark" records are now timestamped and the mark is cleaned
up when no marked test was running for MARK_STALENESS_SEC
(60 seconds) - a deterministic wall-clock rule shared by all workers
that sits inside the old heuristic's expiry range.

The records are refreshed via the new refresh_curr_mark from two
places:

• by workers that observe a running marked test while polling the
  cluster instance, and
• in on_test_stop whenever a marked test finishes.

The refresh on test stop is essential: pollers are not guaranteed to
observe a long-running marked test (e.g. a hypothesis test running for
minutes while all other workers are busy), and without it the mark
could expire right between two back-to-back marked tests, causing a
respin of the exact setup the mark exists to protect.

A mark can have one record per worker; the records are refreshed
together and the newest one decides staleness for the whole mark, so
the cleanup fires once per mark. Pollers skip the refresh while the
records are younger than MARK_REFRESH_SEC (15 seconds), so the
polling hot path under the global cluster lock stays write-free.

The stale-mark cleanup now runs before the scheduler reads the mark
state for the current test. Previously the mark records were read
first and the cleanup could delete them right after, letting a test
act on dangling data - skip resource resolution as a "non-initial"
marked test and run with no resource records at all.

The iteration counters, the cache dict and its threading through the
scheduler are gone. StatusRow now exposes the created_at column.

Documentation

How to share expensive setup across multiple tests: prefer
pytest-subtests; for cases where subtests don't work (hypothesis
property based tests) use the cluster manager mark paired with
@pytest.mark.xdist_group of the same name. The custom xdist
scheduler then schedules the marked tests as one work unit on a single
pytest worker, so they run back-to-back and the assigned "marked"
cluster instance is reused instead of expiring as stale between tests.
Documented in agent_docs/subtests.md (with resource_management.md
pointing to it) and in the mark argument docstring of
get_cluster_instance.

Apply to TestPoolCost

TestPoolCost is the only current user of mark=..., but nothing
kept its tests scheduled together - the ORDER5_BYRON priority
applies only to testnet variants that start from Byron. The class now
carries @pytest.mark.xdist_group("minPoolCost").

Verification

• New test_refresh_curr_mark in framework_tests/test_status_db.py;
  all 37 framework tests pass.
• make lint clean, each commit tested individually.
• Recommend a full regression run including TestPoolCost (its
  scheduling changes with the new xdist group).

[Copilot]

Pull request overview

This PR refactors the cluster manager’s “marked tests” lifecycle handling by replacing the previous loop-iteration heuristic with deterministic wall-clock based staleness/refresh rules shared across workers, and documents the recommended pairing of mark=... with @pytest.mark.xdist_group(...) to keep marked tests scheduled back-to-back.

Changes:

• Add timestamp-based staleness/refresh for “current mark” records (including a new refresh_curr_mark API and created_at exposure via StatusRow / snapshot reads).
• Refresh mark timestamps both from scheduler polling and on test stop to prevent inadvertent mark expiry between back-to-back marked tests.
• Document and apply xdist_group pairing (including updating TestPoolCost) and add a framework test for mark refresh behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
framework_tests/test_status_db.py	Adds coverage for refresh_curr_mark updating only the targeted mark+instance records.
cardano_node_tests/tests/test_pools.py	Adds @pytest.mark.xdist_group("minPoolCost") to keep marked tests scheduled together.
cardano_node_tests/cluster_management/status_db.py	Adds created_at to StatusRow and implements refresh_curr_mark for wall-clock refresh semantics.
cardano_node_tests/cluster_management/manager.py	Refreshes mark timestamps on marked test completion during on_test_stop.
cardano_node_tests/cluster_management/cluster_getter.py	Replaces iteration-count heuristic with wall-clock staleness cleanup + periodic refresh in the scheduler loop.
agent_docs/subtests.md	Documents when to use marked tests and the need to pair mark with xdist_group.
agent_docs/resource_management.md	Adds guidance pointing readers to subtests.md for expensive-setup sharing approaches.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.