Skip to content

feat: add anti_gro setting to defeat TCP-GRO coalescing#45

Open
tracyhatemice wants to merge 1 commit into
hack3ric:masterfrom
tracyhatemice:anti-gro
Open

feat: add anti_gro setting to defeat TCP-GRO coalescing#45
tracyhatemice wants to merge 1 commit into
hack3ric:masterfrom
tracyhatemice:anti-gro

Conversation

@tracyhatemice

Copy link
Copy Markdown
Contributor

Linux 6.18's PPPoE GRO offload (and any TCP-aware GRO path) coalesces a stream of mimic-produced fake-TCP packets into a single superframe, since mimic emits a constant wire ack_seq across a data burst. The receiver's XDP ingress then demangles the superframe as one packet, producing garbage that WireGuard drops -- collapsing ingress throughput.

New per-filter setting anti_gro (default off) makes egress add a per-packet random jitter to the wire ack_seq, which trips the flush check at net/ipv4/tcp_offload.c:337 (flush |= th->ack_seq ^ th2->ack_seq) and prevents coalescing in both the normal merge and fraglist paths. The padding entropy in conn_padding is adjusted to drop ack_seq when anti_gro is on, keeping receiver-side padding computation symmetric.

Both peers must enable the setting; default off preserves compatibility with prior mimic versions.

Linux 6.18's PPPoE GRO offload (and any TCP-aware GRO path) coalesces a
stream of mimic-produced fake-TCP packets into a single superframe,
since mimic emits a constant wire ack_seq across a data burst. The
receiver's XDP ingress then demangles the superframe as one packet,
producing garbage that WireGuard drops -- collapsing ingress throughput.

New per-filter setting anti_gro (default off) makes egress add a
per-packet random jitter to the wire ack_seq, which trips the flush
check at net/ipv4/tcp_offload.c:337 (flush |= th->ack_seq ^ th2->ack_seq)
and prevents coalescing in both the normal merge and fraglist paths.
The padding entropy in conn_padding is adjusted to drop ack_seq when
anti_gro is on, keeping receiver-side padding computation symmetric.

Both peers must enable the setting; default off preserves bit-for-bit
wire compatibility with prior mimic versions.
@hack3ric

Copy link
Copy Markdown
Owner

Thanks for the PR. But I have some questions:

  • Will middlebox conntrack break because of it? I suspect either bpf_prandom_u32() is either too large as jitter, or ack_seq + some_random_number is not valid at all for a TCP connection. Mimic do have a test for conntrack behaviours; you can modify and test it to see if it works.
  • Can we just disable TCP GRO on affected interfaces using ethtool or others? I think solving the root cause is better than a hack.

@airend

airend commented Jul 10, 2026

Copy link
Copy Markdown
  • Will middlebox conntrack break because of it? I suspect either bpf_prandom_u32() is either too large as jitter, or ack_seq + some_random_number is not valid at all for a TCP connection. Mimic do have a test for conntrack behaviours; you can modify and test it to see if it works.

That's a big concern, that some firewalls (one use case) will be even more confused by these large nonlinear jumps. Also, would doing this increase the overhead a lot?

I can also report that this change breaks the BPF verifier on kernel 7.2-rc2 because it cannot prove that the padding length > 0 at the call site, so the program fails to load.

@tracyhatemice

tracyhatemice commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the PR. But I have some questions:

  • Will middlebox conntrack break because of it? I suspect either bpf_prandom_u32() is either too large as jitter, or ack_seq + some_random_number is not valid at all for a TCP connection. Mimic do have a test for conntrack behaviours; you can modify and test it to see if it works.
  • Can we just disable TCP GRO on affected interfaces using ethtool or others? I think solving the root cause is better than a hack.

Middlebox conntrack is something I had not considered.

My mimic runs on the edge router, and my ISP seems fine with the out of window tcp sequence and ack 😅, so the conntrack issue did not surface.

Need to reconsider whether this feature is necessary and, if so, how should it be implemented.

I also tried the existing conntrack test. The current random jitter breaks conntrack, while a small downward jitter does not. I am still testing that approach.

Disabling GRO on the interface does address the issue, although it may also remove the benefits of GRO for non WireGuard traffic on that interface (a throughput vs latency trade).

That's a big concern, that some firewalls (one use case) will be even more confused by these large nonlinear jumps. Also, would doing this increase the overhead a lot?

  • MTU wise - the ack_seq remains the same four byte field, with no additional padding, so this should not add any MTU overhead;
  • CPU wise - the egress already calls bpf_get_prandom_u32() anyway, also no substential CPU cost.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants