fix(logfiles): track inode to apply seek offset to correct file - #3572
Merged
Conversation
The seek offset was applied to the oldest known version of a log file. When the file the offset was recorded for was rotated and not modified since the last search, it was filtered out and the offset was applied to a different file (typically the fresh live log file), silently skipping unsearched content. Track the inode of the log file the seek offset belongs to: * The offset file stores the inode on a second line and is written atomically. An offset file without a valid inode record degrades to the old oldest-file behavior. * _get_rotated_logs applies the seek offset to the file with the matching inode and discards it when no file matches (the file was already fully searched or no longer exists). * expect_errors / expect_messages and find_msgs_in_logs pass the inode captured together with the end-of-file offset (single stat call, so the pair is consistent). Add unit tests covering the offset file round trip and fallbacks, inode-based seek assignment and end-to-end searches across rotation.
The last consumers now read the end-of-file offset and the inode with a single stat call.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes a log-search regression where a persisted seek offset could be incorrectly applied to the wrong log file after rotation, silently skipping the beginning of a fresh live log and missing errors. The solution makes the persisted search state inode-aware so offsets are only applied to the exact file they were recorded for, with safe fallbacks and unit test coverage.
Changes:
- Persist
(seek_offset, inode)in the offset state file and read it back with backward-compatible fallbacks. - Apply seek offsets to the rotated/live log that matches the recorded inode; discard offsets when no inode match exists.
- Update call sites to pass inode alongside offsets and add regression/unit tests covering inode-based behavior and end-to-end scenarios.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| framework_tests/test_logfiles.py | Adds unit/regression tests for inode-aware offset persistence and correct behavior across rotation scenarios. |
| cardano_node_tests/utils/logfiles.py | Implements inode-aware offset storage/loading and inode-based seek assignment for rotated logs; updates related APIs and search flows. |
| cardano_node_tests/utils/helpers.py | Removes get_eof_offset (and now-unused io import) after migrating callers to use stat().st_size. |
| cardano_node_tests/tests/test_blocks.py | Updates log scanning to capture size+inode from a single stat() and passes inode into find_msgs_in_logs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The log search records a seek offset so that each search continues where the
previous one ended. The offset was applied to the oldest known version of the
log file (by modification time). When the file the offset was recorded for was
rotated and not modified since the last search, the timestamp filter dropped it
from the list and the offset was applied to a different file - typically the
fresh live log file. The beginning of that file was then silently skipped, so
errors in it were never reported.
Fix
Track the inode of the log file the seek offset belongs to:
(temp file + rename). An offset file without a valid inode record degrades to
the old oldest-file behavior.
_get_rotated_logsapplies the seek offset to the file with the matchinginode. When no listed file matches, the offset is discarded (the file was
already fully searched or no longer exists) instead of being applied to the
wrong file.
expect_errors/expect_messagesandfind_msgs_in_logspass the inodecaptured together with the end-of-file offset. Both come from a single
statcall, so the pair is consistent even when rotation happens in between.
rotated file keeps its inode.
Tests
New unit tests in
framework_tests/test_logfiles.py(13 new, 37 total):discard when the matching file was filtered out, oldest-file fallback
without inode.
the beginning of a fresh log file are found after rotation, rotated file
with unsearched content is searched from the recorded offset while the
fresh live file is searched from the beginning.
The regression tests fail when the inode logic is reverted (mutation checked).
get_eof_offsetinhelpers.pylost its last consumer and is removed in afollow-up commit.