Skip to content

fix(acquisition/file): don't emit partially-read lines - #4684

Closed
nicheosala wants to merge 1 commit into
crowdsecurity:masterfrom
nicheosala:fix/file-partial-line
Closed

nicheosala wants to merge 1 commit into
crowdsecurity:masterfrom
nicheosala:fix/file-partial-line

Conversation

@nicheosala

Copy link
Copy Markdown

What and why

The file datasource builds its tail.Config without CompleteLines. In nxadm/tail
v1.4.11, when ReadString('\n') returns data with no terminating newline, the fragment is
handed back to tailFileSync, which sends it as if it were a whole line and then calls
seekEnd(). Everything between the fragment and the new EOF is skipped: the rest of that
line, plus whatever the writer appended in the meantime. Those bytes are never read and
never counted in reads.

This is reachable in normal operation. On ext4 with buffered I/O a write() spanning a
page boundary updates i_size per page, while buffered reads don't take the inode lock, so
a reader can legitimately observe a line cut at a 4 KiB boundary.

Observed on Caddy JSON access logs (crowdsec 1.8.1, caddy 2.11.4, ext4, single writer with
O_APPEND). Reconstructing the byte offset of each truncation from the file itself:

fragment=422B line_start=560730 cut=561152 cut%4096=0 line_len=666B
fragment= 95B line_start=860065 cut=860160 cut%4096=0 line_len=733B

Both cuts land exactly on a page boundary. Each surfaced as UnmarshalJSON : unexpected end of JSON input from crowdsecurity/caddy-logs, at ~0.06% of lines (2 unparsed / 3513
reads). Only one error per incident, i.e. the remainder was skipped rather than re-read.
The same signature has been reported against the Traefik parser.

CompleteLines: true makes the library buffer the fragment until the newline arrives.

Behaviour change: while tailing, a trailing line never terminated by \n is now held
instead of being emitted truncated. Follow is hardcoded true on this path, so one-shot is
unaffected — it uses readFile()/bufio.Scanner, not tail.

How it was tested

$ go test ./pkg/acquisition/modules/file/...
ok      github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/file  17.277s

# TestPartialLine without the run.go change:
--- FAIL: TestPartialLine (3.00s)
    received 3 event(s): ["{\"msg\":\"hello" " world\"}" "{\"msg\":\"second\"}"]

# with it:
--- PASS: TestPartialLine (3.00s)
    received 2 event(s): ["{\"msg\":\"hello world\"}" "{\"msg\":\"second\"}"]

The 500ms wait after IsTailing() is deliberate: IsTailing() only reports that the file
was registered, while the tailer opens and seeks to EOF asynchronously. Without the wait the
first write can land before the seek and be skipped, and the test passes for the wrong
reason.

Checklist

  • One concern only.
  • No break to LAPI/CAPI payloads, database schema, config keys, or cscli -o json|raw output.
  • A human has reviewed this diff line by line.
  • A human has tested this change, not only an agent.

AI assistance used: mostly

@github-actions

Copy link
Copy Markdown

@nicheosala: There are no 'kind' label on this PR. You need a 'kind' label to generate the release automatically.

  • /kind feature
  • /kind enhancement
  • /kind refactoring
  • /kind fix
  • /kind chore
  • /kind dependencies
Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

@github-actions

Copy link
Copy Markdown

@nicheosala: There are no area labels on this PR. You can add as many areas as you see fit.

  • /area agent
  • /area local-api
  • /area cscli
  • /area appsec
  • /area security
  • /area configuration
Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

@nicheosala

Copy link
Copy Markdown
Author

/kind fix
/area agent

@blotus

blotus commented Sep 14, 2026

Copy link
Copy Markdown
Member

Duplicate of #4678

@blotus blotus marked this as a duplicate of #4678 Sep 14, 2026
@blotus blotus closed this Sep 14, 2026
@nicheosala
nicheosala deleted the fix/file-partial-line branch September 14, 2026 11:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants