lapi: optionally limit concurrent decision stream requests

[tommy-ng0]

Adds a new config key: api.server.decisions_stream.max_concurrent_requests, next to the page_size from #4733.

What and why

Every in-flight /v1/decisions/stream request holds a page of decisions in memory. With ~1.2k bouncers pulling a large delta at once, that's what put our LAPI into an OOM loop (#4711). #4733 caps the memory per request; this caps how many requests do it at once.

api:
  server:
    decisions_stream:
      page_size: 10000
      max_concurrent_requests: 200

When set, stream requests beyond the limit wait for a free slot, held until the response is fully written. A request that waited reads the cursor after getting its slot, so it still gets an up-to-date delta. A bouncer that disconnects while queued leaves without taking a slot (503, logged at debug), and since nothing was sent its cursor doesn't move. Unset or 0 keeps the current behaviour (no limit); negative warns and also means no limit. Only GET /v1/decisions/stream is limited.

Things to know when sizing it:

• Queued requests wait with no time limit of their own. go-cs-bouncer has no client timeout, but lua-cs-bouncer gives up after STREAM_REQUEST_TIMEOUT (15s by default), so keep queue waits well under the shortest stream timeout in your fleet.
• Peak stream memory is roughly max_concurrent_requests × the memory of one page_size page. This only bounds /v1/decisions/stream; live-mode GET /v1/decisions is unpaged and unlimited, and out of scope here.
• Time spent queued shows up in the existing response time metric. Happy to add a queued/in-flight gauge here or in a follow-up if you want one.

Known limitation - your call on how to handle it. A slot is held until the response is fully written, with no time limit. A bouncer that keeps its connection open but stops reading (a hung process, a suspended VM, a link that silently drops traffic) keeps its slot, and max_concurrent_requests of them would stall delivery to every other bouncer. This only happens when the limit is on.

I tried a per-write deadline (http.ResponseController.SetWriteDeadline, renewed as data goes out). It works over HTTP/1.1, but doing it properly needs two more things. Over HTTP/2, a stalled connection only fails with the server-wide HTTP2Config.WriteByteTimeout. And the deadline has to be paused around the page queries and the cursor update, or a slow query cuts off a healthy bouncer and can advance its cursor on a truncated response. That's a bigger change to the stream code, so I've left it out. Happy to add it here or as a follow-up, whichever you prefer.

Part of #4711

How it was tested

TestLimitStreamConcurrency covers the limiter against a handler that holds its slot until released: requests beyond the limit wait, freeing one slot lets exactly one queued request in, a request canceled while queued gets a 503 without taking a slot, and 0 and -1 don't limit.

TestStreamDecisionMaxConcurrentRequests runs against the real LAPI: the config reaches the route, the slot is given back after a full response, and while one pull is stuck mid-response a second gets a 503 when the limit is 1 and a 200 when unset or negative.

$ go test ./pkg/apiserver/controllers/v1/ -run TestLimitStreamConcurrency -race -count=3 -v
    --- PASS: TestLimitStreamConcurrency/a_request_canceled_while_queued_leaves_without_taking_a_slot (0.10s)
        --- PASS: TestLimitStreamConcurrency/requests_beyond_the_limit_wait_for_a_slot/limit=1 (0.10s)
        --- PASS: TestLimitStreamConcurrency/requests_beyond_the_limit_wait_for_a_slot/limit=3 (0.10s)
        --- PASS: TestLimitStreamConcurrency/zero_or_negative_does_not_limit/limit=0 (0.00s)
        --- PASS: TestLimitStreamConcurrency/zero_or_negative_does_not_limit/limit=-1 (0.00s)
ok  	github.com/crowdsecurity/crowdsec/pkg/apiserver/controllers/v1	1.951s

$ go test ./pkg/apiserver/ -run 'TestStreamDecisionMaxConcurrentRequests|TestStreamDecisionPageSize' -race -count=1
ok  	github.com/crowdsecurity/crowdsec/pkg/apiserver	2.424s

$ TEST_LOCAL_ONLY=1 go test ./pkg/apiserver/... ./pkg/csconfig/... -count=1
ok  	github.com/crowdsecurity/crowdsec/pkg/apiserver	15.383s
ok  	github.com/crowdsecurity/crowdsec/pkg/apiserver/controllers/v1	0.318s
ok  	github.com/crowdsecurity/crowdsec/pkg/csconfig	0.014s

To check the tests aren't tautological, I broke the slot release, the cancel handling, the config wiring and the route wiring, one at a time. A test failed each time.

golangci-lint 2.13.2, run like make lint for linux, windows and freebsd: no issues in the changed files.

Not yet done: a local run against a real instance. Docs: crowdsecurity/crowdsec-docs#1180.

Checklist

• One concern only.
• No break to LAPI/CAPI payloads, database schema, config keys, or cscli -o json|raw output.
• A human has reviewed this diff line by line.
• A human has tested this change, not only an agent.

AI assistance used: mostly

/kind enhancement
/area local-api
/area configuration

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 71.30%. Comparing base (c86bf40) to head (eb8f376).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4747      +/-   ##
==========================================
+ Coverage   71.28%   71.30%   +0.02%     
==========================================
  Files         542      542              
  Lines       36751    36770      +19     
==========================================
+ Hits        26199    26220      +21     
+ Misses      10551    10549       -2     
  Partials        1        1              

Flag	Coverage Δ
bats	45.50% <45.00%> (+<0.01%)	⬆️
unit-linux	47.19% <100.00%> (+0.02%)	⬆️
unit-windows	35.54% <90.00%> (+0.11%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.