fix(logfiles): search unterminated final log lines - #3575
Merged
Conversation
A final line without a trailing newline was never searched. An expected message in such line was reported as missing, and errors in the unterminated final line of a rotated log file were never reported. * In the error search, the unterminated final line of a rotated log file is searched right away - the file will never be appended to, so the line is its final content. The unterminated final line of the live log file stays unsearched until it is complete (it is excluded from the recorded seek offset, so a later search picks it up). This avoids reporting an error whose ignore-marker was not flushed yet and repeated reports of the same partial line. * Presence checks (expect_errors / expect_messages) search the unterminated final line as well, unless the regex constrains what follows the match (end anchors, word boundaries, lookaheads) - for such regexes a match in an incomplete line doesn't imply a match in the complete line. * find_msgs_in_logs searches the unterminated final line only in rotated log files. A truncated line from the live log file must not be returned to callers that parse the line content. Add unit tests for all the above, including ignore rules and look-back map applied to unterminated lines.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes log searching so that unterminated final log lines are handled correctly (searched or deferred) across live vs rotated log files, preventing missed expected messages and missed error detections.
Changes:
- Update error scanning to search an unterminated final line immediately for rotated logs, but defer it for the live log until the line is complete/rotated.
- Update message-finding and message-presence checks to optionally include unterminated final lines with safeguards for regexes that depend on match-end context.
- Add unit tests covering unterminated-line behavior for live vs rotated logs, including ignore rules and look-back suppression.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
cardano_node_tests/utils/logfiles.py |
Implements unterminated final-line handling for _search_log_lines, find_msgs_in_logs, and presence checks, plus helper logic to decide when incomplete lines are safe to search. |
framework_tests/test_logfiles.py |
Adds regression tests for unterminated final-line handling across error search, message finding, presence checks, and ignore/look-back behaviors. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The detection of regexes that constrain what follows the match used a plain substring check, so escaped literals (e.g. "\\$") and characters inside character classes were misclassified as anchors and prevented searching an unterminated final line. Scan the pattern with escape and character class awareness instead.
A "[" inside a "(?#...)" comment (or a verbose mode comment) made the anchor scan think it is inside a character class that never closes, so anchors later in the pattern were not detected and an incomplete line was searched with an end-constrained regex. A compiled regex can never have an unterminated character class. When the scan ends in that state, it desynced, so conservatively report the regex as constraining - that merely degrades to not searching an incomplete line.
The hand-written scan of the regex pattern had false negatives: a "[" inside a "(?#...)" comment opened a phantom character class, and when a later "]" closed it, end anchors in between were not detected. An incomplete line was then searched with an end-constrained regex, which can make a missing message check pass when the complete line would not match. Parse the pattern with the stdlib regex parser instead and walk the parsed tree for end anchors, word boundaries and lookaheads. Escaped literals, character classes, comments and verbose mode are then interpreted exactly as by the re module itself. Differential fuzzing against the parser-based ground truth shows no misclassification.
Cover also the parse tree walk by the conservative fallback, so an unexpected parse tree shape in a future Python version degrades to not searching an incomplete line instead of aborting the whole check. Move the unterminated line note in check_msgs_presence_in_logs docstring before the Args section.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
cardano_node_tests/utils/logfiles.py:667
- The
_subpattern_constrainsdocstring says it checks only for an end anchor or lookahead, but the implementation also treats word-boundary anchors (AT_BOUNDARY/AT_NON_BOUNDARY) as constraining. Updating the docstring to reflect the actual behavior will prevent confusion for future maintainers.
def _subpattern_constrains(parsed: tp.Any) -> bool:
"""Return True when the parsed (sub)pattern contains an end anchor or a lookahead."""
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A final line without a trailing newline was never searched. An expected message in such line was reported as missing, and errors in the unterminated final line of a rotated log file were never reported.
Add unit tests for all the above, including ignore rules and look-back map applied to unterminated lines.