tail: avoid copying each chunk buffer - #14358
Open
haydonryan wants to merge 1 commit into
Open
Conversation
|
GNU testsuite comparison: |
Contributor
xtqqczze
reviewed
Sep 3, 2026
| self.bytes -= chunk.bytes as u64; | ||
| } else { | ||
| *chunk = BytesChunk::new(); | ||
| chunk = Box::new(BytesChunk::new()); |
Contributor
There was a problem hiding this comment.
note: the change in this line is necessary to avoid:
error[E0382]: use of moved value: `chunk`
--> src/uu/tail/src/chunks.rs:292:35
Contributor
Author
There was a problem hiding this comment.
Thank you - checking what went wrong locally with tests, and why it missed this.
Contributor
Author
There was a problem hiding this comment.
Oh wait i miss-read what you typed. My bad. Yes, the line 299 change is required.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Using deepseek v4 flash found another improvement. Validated tests pass and validated with codex as well.
The example below is pretty silly tbh - not really likely that you'd pipe a 400mb file, but shows the logic is sound.
LLM below here:
BytesChunkBuffer::fillandLinesChunkBuffer::filldeep-copied every 8 KiBbacking buffer with
push_back(chunk.clone()), then reused the original as thenext scratch. This moves the chunk into the deque instead and takes the next
working chunk from
pop_front()or a fresh allocation.In the steady state (piped input larger than
num_print), this removes one8 KiB heap allocation and one 8 KiB memcpy per chunk read. The
Box::newonthe fill path is bounded to the initial fill and replaces — not adds to — the
clone's allocation, so per-chunk allocations are unchanged or fewer. No
observable behavior change.
Allocation accounting
Steady state (large piped input — exactly what
BytesChunkBuffer/LinesChunkBufferserve, file ≫num_print, so every iterationis the
ifbranch):clone()= 1 alloc + 8 KiB memcpypop_front()reuse, 0move= 0pop_front()reuse, 0For a 400 MB piped input that's ~49k heap allocs + ~400 MB of memcpy eliminated.
Small-file case (file <
num_print, else branch every time — the only place the concern applies):clone()= 1 alloc + 8 KiB memcpy*chunk = new()in place, 0move= 0Box::new()= 1 alloc + zero-initRelease multicall binary: 14,000,712 -> 14,000,456 bytes (-256 B).