Skip to content

tail: avoid copying each chunk buffer - #14358

Open
haydonryan wants to merge 1 commit into
uutils:mainfrom
haydonryan:tail-avoid-copy-chunk-buffer
Open

tail: avoid copying each chunk buffer#14358
haydonryan wants to merge 1 commit into
uutils:mainfrom
haydonryan:tail-avoid-copy-chunk-buffer

Conversation

@haydonryan

Copy link
Copy Markdown
Contributor

Using deepseek v4 flash found another improvement. Validated tests pass and validated with codex as well.

The example below is pretty silly tbh - not really likely that you'd pipe a 400mb file, but shows the logic is sound.

LLM below here:

BytesChunkBuffer::fill and LinesChunkBuffer::fill deep-copied every 8 KiB
backing buffer with push_back(chunk.clone()), then reused the original as the
next scratch. This moves the chunk into the deque instead and takes the next
working chunk from pop_front() or a fresh allocation.

In the steady state (piped input larger than num_print), this removes one
8 KiB heap allocation and one 8 KiB memcpy per chunk read. The Box::new on
the fill path is bounded to the initial fill and replaces — not adds to — the
clone's allocation, so per-chunk allocations are unchanged or fewer. No
observable behavior change.

Allocation accounting

Steady state (large piped input — exactly what BytesChunkBuffer/LinesChunkBuffer serve, file ≫ num_print, so every iteration
is the if branch):

push scratch per-chunk
baseline clone() = 1 alloc + 8 KiB memcpy pop_front() reuse, 0 1 alloc + 8 KiB memcpy
proposed move = 0 pop_front() reuse, 0 0 alloc + 0 memcpy

For a 400 MB piped input that's ~49k heap allocs + ~400 MB of memcpy eliminated.

Small-file case (file < num_print, else branch every time — the only place the concern applies):

push scratch per-chunk
baseline clone() = 1 alloc + 8 KiB memcpy *chunk = new() in place, 0 1 alloc + memcpy
proposed move = 0 Box::new() = 1 alloc + zero-init 1 alloc + zero-init

Release multicall binary: 14,000,712 -> 14,000,456 bytes (-256 B).

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

GNU testsuite comparison:

Skip an intermittent issue tests/tail/symlink (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/cut/bounded-memory (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/misc/io-errors (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/tail/retry (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/tail/tail-n0f (passes in this run but fails in the 'main' branch)

@xtqqczze

xtqqczze commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Comment thread src/uu/tail/src/chunks.rs
self.bytes -= chunk.bytes as u64;
} else {
*chunk = BytesChunk::new();
chunk = Box::new(BytesChunk::new());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: the change in this line is necessary to avoid:

error[E0382]: use of moved value: `chunk`
   --> src/uu/tail/src/chunks.rs:292:35

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you - checking what went wrong locally with tests, and why it missed this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh wait i miss-read what you typed. My bad. Yes, the line 299 change is required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants