acquisition/file: don't send half-written lines, or skip the lines after them

[n1koo]

What and why

The file datasource silently drops lines when a log is written in bursts. If the tailer hits EOF mid-line, nxadm/tail sends the partial line, then seeks to EOF. That skips everything written in between (tail.go#L336-L342). On a production Traefik log (~65k lines/day, v1.8.1), this showed up as 37 UnmarshalJSON : unexpected end of JSON input errors in 7 days. The skipped lines log nothing.

This sets CompleteLines: true, nxadm/tail's own option for this. It is off by default because the nxadm fork merged hpcloud/tail#126, and was restored as opt-in in nxadm/tail#26. Trade-offs:

• a line is buffered until its newline arrives, so a final line without one is not sent;
• a partial line pending at rotation or truncation is dropped instead of sent broken;
• nxadm then reports read errors as EOF (tail.go#L254-L267), so a tail that hits one waits silently instead of dying with file reader died.

Not covered: at startup the tail seeks to EOF, which can still land mid-line.

Human review: I read the diff line by line and verified the testing below.

How it was tested

$ go test -race -count=10 -run TestLiveAcquisitionPartialLine ./pkg/acquisition/modules/file/
ok  	github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/file	7.314s
$ go test -race ./pkg/acquisition/modules/file/
ok  	github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/file	15.978s

The new test fails on master: the partial line is sent and the lines after it are lost. It is file-only, because this is the only datasource using nxadm/tail.

I replayed 40,000 production lines in request bursts, one write() per line. v1.8.1 parsed 39,961 of them in one run and 39,998 in another; with this patch it parsed all 40,000 both times.

golangci-lint v2.10: 0 issues on linux, windows and freebsd. Tests were run on Linux only.

Checklist

• One concern only.
• No break to LAPI/CAPI payloads, database schema, config keys, or cscli -o json|raw output.
• A human has reviewed this diff line by line.
• A human has tested this change, not only an agent.

AI assistance used: mostly

[github-actions]

@n1koo: There are no 'kind' label on this PR. You need a 'kind' label to generate the release automatically.

• /kind feature
• /kind enhancement
• /kind refactoring
• /kind fix
• /kind chore
• /kind dependencies

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[github-actions]

@n1koo: There are no area labels on this PR. You can add as many areas as you see fit.

• /area agent
• /area local-api
• /area cscli
• /area appsec
• /area security
• /area configuration

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[n1koo]

/kind fix
/area agent

[k4d1sm0]

Another production data point supporting this fix, on v1.7.8.

Setup: a Traefik v3 JSON access log written to a host file (Docker bind mount) and tailed by the CrowdSec agent through a hostPath mount in Kubernetes. Default file datasource settings. Average ~1.4 lines/s with bursts, mostly HLS segment/playlist polling.

Measured over ~21h, by comparing the lines written to the file in the agent's uptime window with the reads counter in cscli metrics show acquisition:

• 111,373 lines written, 110,984 read, so 389 lines never read (0.35%)
• 168 UnmarshalJSON : unexpected end of JSON input errors, all on fragments of lines that are valid JSON in the file
• the remaining ~220 lost lines produced no log at all, which matches the seek-to-EOF behavior described above

I checked v1.8.1 and master: neither sets CompleteLines. Detection impact is small for repetitive traffic like scans, but the silent part is what makes this hard to notice.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 65.19%. Comparing base (a428ab0) to head (3bf3e78).
⚠️ Report is 6 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4678      +/-   ##
==========================================
+ Coverage   64.73%   65.19%   +0.45%     
==========================================
  Files         522      505      -17     
  Lines       39863    39130     -733     
==========================================
- Hits        25807    25511     -296     
+ Misses      11654    11256     -398     
+ Partials     2402     2363      -39     

Flag	Coverage Δ
bats	41.12% <100.00%> (-0.01%)	⬇️
unit-linux	41.91% <100.00%> (-0.06%)	⬇️
unit-windows	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[blotus]

Hello,

It seems the tests are flaky on windows (https://github.com/crowdsecurity/crowdsec/actions/runs/34595197049/job/104708935711?pr=4678#step:6:14353), which is often the case when relying on small sleep for synchronization (the windows runners are extremely slow)