Skip to content

fix(logfiles): track inode to apply seek offset to correct file - #3572

Merged
mkoura merged 2 commits into
masterfrom
logfiles_rotation_seek
Jul 30, 2026
Merged

fix(logfiles): track inode to apply seek offset to correct file#3572
mkoura merged 2 commits into
masterfrom
logfiles_rotation_seek

Conversation

@mkoura

@mkoura mkoura commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Problem

The log search records a seek offset so that each search continues where the
previous one ended. The offset was applied to the oldest known version of the
log file (by modification time). When the file the offset was recorded for was
rotated and not modified since the last search, the timestamp filter dropped it
from the list and the offset was applied to a different file - typically the
fresh live log file. The beginning of that file was then silently skipped, so
errors in it were never reported.

Fix

Track the inode of the log file the seek offset belongs to:

  • The offset file stores the inode on a second line and is written atomically
    (temp file + rename). An offset file without a valid inode record degrades to
    the old oldest-file behavior.
  • _get_rotated_logs applies the seek offset to the file with the matching
    inode. When no listed file matches, the offset is discarded (the file was
    already fully searched or no longer exists) instead of being applied to the
    wrong file.
  • expect_errors / expect_messages and find_msgs_in_logs pass the inode
    captured together with the end-of-file offset. Both come from a single stat
    call, so the pair is consistent even when rotation happens in between.
  • Rotation is expected to rename the log file (as supervisord does), so the
    rotated file keeps its inode.

Tests

New unit tests in framework_tests/test_logfiles.py (13 new, 37 total):

  • Offset file round trip, missing/garbage/partially valid content fallbacks.
  • Inode-based seek assignment: match on rotated file, match on live file,
    discard when the matching file was filtered out, oldest-file fallback
    without inode.
  • End-to-end searches: repeated searches report each error once, errors at
    the beginning of a fresh log file are found after rotation, rotated file
    with unsearched content is searched from the recorded offset while the
    fresh live file is searched from the beginning.

The regression tests fail when the inode logic is reverted (mutation checked).

get_eof_offset in helpers.py lost its last consumer and is removed in a
follow-up commit.

mkoura added 2 commits July 30, 2026 15:31
The seek offset was applied to the oldest known version of a log file.
When the file the offset was recorded for was rotated and not modified
since the last search, it was filtered out and the offset was applied
to a different file (typically the fresh live log file), silently
skipping unsearched content.

Track the inode of the log file the seek offset belongs to:

* The offset file stores the inode on a second line and is written
  atomically. An offset file without a valid inode record degrades to
  the old oldest-file behavior.
* _get_rotated_logs applies the seek offset to the file with the
  matching inode and discards it when no file matches (the file was
  already fully searched or no longer exists).
* expect_errors / expect_messages and find_msgs_in_logs pass the inode
  captured together with the end-of-file offset (single stat call, so
  the pair is consistent).

Add unit tests covering the offset file round trip and fallbacks,
inode-based seek assignment and end-to-end searches across rotation.
The last consumers now read the end-of-file offset and the inode with
a single stat call.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a log-search regression where a persisted seek offset could be incorrectly applied to the wrong log file after rotation, silently skipping the beginning of a fresh live log and missing errors. The solution makes the persisted search state inode-aware so offsets are only applied to the exact file they were recorded for, with safe fallbacks and unit test coverage.

Changes:

  • Persist (seek_offset, inode) in the offset state file and read it back with backward-compatible fallbacks.
  • Apply seek offsets to the rotated/live log that matches the recorded inode; discard offsets when no inode match exists.
  • Update call sites to pass inode alongside offsets and add regression/unit tests covering inode-based behavior and end-to-end scenarios.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
framework_tests/test_logfiles.py Adds unit/regression tests for inode-aware offset persistence and correct behavior across rotation scenarios.
cardano_node_tests/utils/logfiles.py Implements inode-aware offset storage/loading and inode-based seek assignment for rotated logs; updates related APIs and search flows.
cardano_node_tests/utils/helpers.py Removes get_eof_offset (and now-unused io import) after migrating callers to use stat().st_size.
cardano_node_tests/tests/test_blocks.py Updates log scanning to capture size+inode from a single stat() and passes inode into find_msgs_in_logs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@mkoura
mkoura merged commit 49ef701 into master Jul 30, 2026
4 checks passed
@mkoura
mkoura deleted the logfiles_rotation_seek branch July 30, 2026 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants