fix(acquisition/file): don't emit partially-read lines - #4684
nicheosala wants to merge 1 commit into
Conversation
|
@nicheosala: There are no 'kind' label on this PR. You need a 'kind' label to generate the release automatically.
DetailsI am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository. |
|
@nicheosala: There are no area labels on this PR. You can add as many areas as you see fit.
DetailsI am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository. |
|
/kind fix |
|
Duplicate of #4678 |
What and why
The file datasource builds its
tail.ConfigwithoutCompleteLines. In nxadm/tailv1.4.11, when
ReadString('\n')returns data with no terminating newline, the fragment ishanded back to
tailFileSync, which sends it as if it were a whole line and then callsseekEnd(). Everything between the fragment and the new EOF is skipped: the rest of thatline, plus whatever the writer appended in the meantime. Those bytes are never read and
never counted in
reads.This is reachable in normal operation. On ext4 with buffered I/O a
write()spanning apage boundary updates
i_sizeper page, while buffered reads don't take the inode lock, soa reader can legitimately observe a line cut at a 4 KiB boundary.
Observed on Caddy JSON access logs (crowdsec 1.8.1, caddy 2.11.4, ext4, single writer with
O_APPEND). Reconstructing the byte offset of each truncation from the file itself:Both cuts land exactly on a page boundary. Each surfaced as
UnmarshalJSON : unexpected end of JSON inputfromcrowdsecurity/caddy-logs, at ~0.06% of lines (2 unparsed / 3513reads). Only one error per incident, i.e. the remainder was skipped rather than re-read.
The same signature has been reported against the Traefik parser.
CompleteLines: truemakes the library buffer the fragment until the newline arrives.Behaviour change: while tailing, a trailing line never terminated by
\nis now heldinstead of being emitted truncated.
Followis hardcoded true on this path, so one-shot isunaffected — it uses
readFile()/bufio.Scanner, nottail.How it was tested
The 500ms wait after
IsTailing()is deliberate:IsTailing()only reports that the filewas registered, while the tailer opens and seeks to EOF asynchronously. Without the wait the
first write can land before the seek and be skipped, and the test passes for the wrong
reason.
Checklist
cscli -o json|rawoutput.AI assistance used: mostly