Skip to content

Add test for off-chain vote data size limit#2946

Open
ArturWieczorek wants to merge 2 commits into
masterfrom
artur/increase-gov-action-file-size
Open

Add test for off-chain vote data size limit#2946
ArturWieczorek wants to merge 2 commits into
masterfrom
artur/increase-gov-action-file-size

Conversation

@ArturWieczorek

@ArturWieczorek ArturWieczorek commented Mar 19, 2025

Copy link
Copy Markdown
Collaborator

Summary

Tests cardano-db-sync's off-chain vote data size limit. db-sync fetches gov-action and DRep anchors up to a fixed size (httpGetOffChainVoteDataSingle in OffChain/Http.hs, currently 3,000,000 bytes); content up to that size is downloaded and stored in off_chain_vote_data, larger content is rejected with a Size error recorded in off_chain_vote_fetch_error and no data row.

New module tests_conway/test_offchain_vote_size.py with TestOffChainVoteSize:

  • test_within_size_limit — an info action with a conformant anchor of exactly OFFCHAIN_VOTE_MAX_BYTES (3,000,000); asserts db-sync stores off_chain_vote_data with is_valid = TRUE and the raw bytes match.
  • test_over_size_limit — an anchor one byte over the limit; asserts db-sync records a Size error in off_chain_vote_fetch_error and stores no off_chain_vote_data row.

The anchor payloads are generated at runtime (a conformant CIP-108 document padded via body.motivation, made unique per run) and served from the internal web server, so no multi-MB fixtures are committed to the repo. A small additive helper dbsync_queries.query_voting_anchor_id(url=...) looks up the anchor id by url (used to find the fetch-error row without going through get_gov_action_proposals).

Verification

Verified end-to-end on a local dev cluster — 2 passed on db-sync 13.7.2.1 (rev db8cdf6, off-chain vote limit = 3,000,000 bytes):

2 passed in 581.65s (0:09:41)

Note

Like the other web.publish-based off-chain tests (e.g. test_drep.py::TestDReps::test_register_wrong_metadata), this test needs db-sync started with --allow-private-offchain-urls so it can fetch the anchor from the cluster's local web server; otherwise db-sync rejects the localhost url and no off_chain_vote_data / off_chain_vote_fetch_error row is produced.

Note on history

This replaces the original approach (padding the shared governance_action_anchor.json, since renamed to ga_anchor.json, to ~1 MB). That would have bloated the shared conformant anchor used by many unrelated tests and, on its own, exercised no db-sync behavior. The branch was reset to master and rebuilt as the boundary test above.

@ArturWieczorek
ArturWieczorek force-pushed the artur/increase-gov-action-file-size branch from ad1ae41 to a2f0285 Compare March 19, 2025 22:14
@ArturWieczorek
ArturWieczorek force-pushed the artur/increase-gov-action-file-size branch from a2f0285 to a2c302a Compare October 31, 2025 18:49
@ArturWieczorek
ArturWieczorek force-pushed the artur/increase-gov-action-file-size branch from a2c302a to 5594757 Compare July 16, 2026 17:41
@ArturWieczorek ArturWieczorek changed the title Increase governance action file size test(db-sync): off-chain vote data size limit Jul 16, 2026
Verify db-sync's off-chain vote data size boundary: an anchor of
3,000,000 bytes is stored in off_chain_vote_data, while one byte over
is rejected with a "Size error" in off_chain_vote_fetch_error and no
data row. Both cases go through an info action whose anchor is
generated at the target size and served from the internal web server.

Add a `query_voting_anchor_id` db-sync query helper to look up an
anchor by url.
@ArturWieczorek
ArturWieczorek force-pushed the artur/increase-gov-action-file-size branch from 5594757 to d36b8f4 Compare July 16, 2026 17:49
@ArturWieczorek

ArturWieczorek commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator Author

CI tests: 💚

https://github.com/IntersectMBO/cardano-node-tests/actions/runs/29530385569/job/87729023887

Local tests: 💚

 WARNING: Using `cardano-cli` from custom path!
 WARNING: Using `cardano-node` from custom path!
============================= test session starts ==============================
platform linux -- Python 3.13.9, pytest-9.0.3, pluggy-1.5.0 -- /home/artur/Projects/cardano-node-tests/.venv/bin/python
hypothesis profile 'default'
metadata: {'Python': '3.13.9', 'Platform': 'Linux-6.8.0-134-generic-x86_64-with-glibc2.39', 'Packages': {'pytest': '9.0.3', 'pluggy': '1.5.0'}, 'Plugins': {'hypothesis': '6.148.7', 'html': '4.1.1', 'metadata': '3.1.1', 'xdist': '3.8.0', 'select': '0.1.2', 'allure-pytest': '2.15.2', 'order': '1.3.0'}, 'github actions url': '', 'cardano-node': '11.0.1', 'cardano-node rev': '97036a66bcf8c89f687ae57a048eecc0389977ef', 'cardano-node ghc': 'ghc-9.6', 'cardano-cli': '11.0.0.0', 'cardano-cli rev': '97036a66bcf8c89f687ae57a048eecc0389977ef', 'cardano-cli ghc': 'ghc-9.6', 'COMMAND_ERA': '', 'TESTNET_VARIANT': 'local_fast', 'NUM_POOLS': '3', 'UTXO_BACKEND': '', 'MIXED_UTXO_BACKENDS': '', 'MAX_TESTS_PER_CLUSTER': 8, 'CLUSTERS_COUNT': 1, 'HAS_CC': 'True', 'cardano-node-tests rev': '4b42695fb48ab6dd4bfd9ee5479793eb5bdb65c4', 'cardano-node-tests url': 'https://github.com/IntersectMBO/cardano-node-tests/tree/4b42695fb48ab6dd4bfd9ee5479793eb5bdb65c4', 'CARDANO_NODE_SOCKET_PATH': '/var/tmp/cardonnay-of-artur/state-cluster0/bft1.socket', 'cardano-cli exe': '/home/artur/.local/bin/cardano-cli', 'cardano-node exe': '/home/artur/.local/bin/cardano-node', 'cardano-submit-api exe': '/home/artur/.local/bin/cardano-submit-api', 'network magic': 42, 'HAS_DBSYNC': 'True', 'db-sync': '13.7.2.1', 'db-sync rev': 'db8cdf668cda632e9a09c30e1159fcb9fb34e952', 'db-sync ghc': 'ghc-9.6', 'db-sync exe': '/home/artur/.local/bin/cardano-db-sync', 'HAS_SMASH': 'True', 'smash': '13.7.2.1', 'smash rev': 'db8cdf668cda632e9a09c30e1159fcb9fb34e952', 'smash ghc': 'ghc-9.6', 'smash exe': '/home/artur/.local/bin/cardano-smash-server'}
rootdir: /home/artur/Projects/cardano-node-tests
configfile: pyproject.toml
plugins: hypothesis-6.148.7, html-4.1.1, metadata-3.1.1, xdist-3.8.0, select-0.1.2, allure-pytest-2.15.2, order-1.3.0
collecting ... collected 2 items

cardano_node_tests/tests/tests_conway/test_offchain_vote_size.py::TestOffChainVoteSize::test_within_size_limit@long 
-------------------------------- live log setup --------------------------------
INFO     cardano_node_tests.tests.conftest:conftest.py:225 Changed CWD to '/home/artur/Projects/cardano-node-tests/tmp/pytest-of-artur/pytest-43'.
-------------------------------- live log call ---------------------------------
INFO     cardano_clusterlib.clusterlib_klass:clusterlib_klass.py:378 Waiting for 23.20 sec for slot no 9005.
WARNING  cardano_node_tests.utils.dbsync_utils:dbsync_utils.py:510 Sleeping 3s before repeating query for the 1 time.
WARNING  cardano_node_tests.utils.dbsync_utils:dbsync_utils.py:510 Sleeping 6s before repeating query for the 2 time.
WARNING  cardano_node_tests.utils.dbsync_utils:dbsync_utils.py:510 Sleeping 11s before repeating query for the 3 time.
WARNING  cardano_node_tests.utils.dbsync_utils:dbsync_utils.py:510 Sleeping 18s before repeating query for the 4 time.
WARNING  cardano_node_tests.utils.dbsync_utils:dbsync_utils.py:510 Sleeping 27s before repeating query for the 5 time.
WARNING  cardano_node_tests.utils.dbsync_utils:dbsync_utils.py:510 Sleeping 38s before repeating query for the 6 time.
WARNING  cardano_node_tests.utils.dbsync_utils:dbsync_utils.py:510 Sleeping 51s before repeating query for the 7 time.
WARNING  cardano_node_tests.utils.dbsync_utils:dbsync_utils.py:510 Sleeping 66s before repeating query for the 8 time.
WARNING  cardano_node_tests.utils.dbsync_utils:dbsync_utils.py:510 Sleeping 83s before repeating query for the 9 time.
PASSED                                                                   [ 50%]
cardano_node_tests/tests/tests_conway/test_offchain_vote_size.py::TestOffChainVoteSize::test_over_size_limit@long 
-------------------------------- live log call ---------------------------------
WARNING  cardano_node_tests.utils.dbsync_utils:dbsync_utils.py:510 Sleeping 3s before repeating query for the 1 time.
WARNING  cardano_node_tests.utils.dbsync_utils:dbsync_utils.py:510 Sleeping 6s before repeating query for the 2 time.
WARNING  cardano_node_tests.utils.dbsync_utils:dbsync_utils.py:510 Sleeping 11s before repeating query for the 3 time.
WARNING  cardano_node_tests.utils.dbsync_utils:dbsync_utils.py:510 Sleeping 18s before repeating query for the 4 time.
WARNING  cardano_node_tests.utils.dbsync_utils:dbsync_utils.py:510 Sleeping 27s before repeating query for the 5 time.
WARNING  cardano_node_tests.utils.dbsync_utils:dbsync_utils.py:510 Sleeping 38s before repeating query for the 6 time.
WARNING  cardano_node_tests.utils.dbsync_utils:dbsync_utils.py:510 Sleeping 51s before repeating query for the 7 time.
WARNING  cardano_node_tests.utils.dbsync_utils:dbsync_utils.py:510 Sleeping 66s before repeating query for the 8 time.
PASSED                                                                   [100%]
------------------------------ live log teardown -------------------------------
INFO     cardano_node_tests.utils.dbsync_conn:dbsync_conn.py:35 Closing connection to db-sync database dbsync0.

@ArturWieczorek
ArturWieczorek marked this pull request as ready for review July 16, 2026 17:53
@ArturWieczorek ArturWieczorek changed the title test(db-sync): off-chain vote data size limit Add test for off-chain vote data size limit Jul 16, 2026
The off-chain vote size tests serve their anchor from the internal
(localhost) web server. Since db-sync 13.7.x the off-chain fetcher
rejects localhost/private hosts at URL-parse time ("Access to localhost
is not allowed") unless launched with `--allow-private-offchain-urls`,
so both tests failed in CI (wrong error / no data row).

The flag cannot be enabled globally: test_smash.py's
`test_fetch_pool_metadata_localhost_rejected` asserts that db-sync
rejects localhost URLs. So the flag is toggled per-test instead:

* Add `DBSyncManager.restart_db_sync` (restart without recreating the
  DB) and `set_allow_private_offchain_urls`, which forces the flag by
  editing the `run-cardano-dbsync` launcher and restarting db-sync.
* Run the size tests on a locked singleton cluster (with governance) so
  the cluster-wide restart is isolated from tests running in parallel,
  and restore the default (flag off) on teardown.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant