macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 201988 ms

[DrJosh9000]

Describe the bug

I have a toy cluster of Pi 5 devices. Every so often one of them stops responding over the network.

Steps to reproduce the behaviour

Unclear, seems to happen spontaneously.

Device (s)

Raspberry Pi 5

System

raspinfo.txt

Logs

Here's the end of the kernel logs for the previous boot:

Sep 26 19:27:03 pi5c kernel: cni0: port 8(veth21d981ec) entered disabled state
Sep 26 19:27:03 pi5c kernel: veth21d981ec (unregistering): left allmulticast mode
Sep 26 19:27:03 pi5c kernel: veth21d981ec (unregistering): left promiscuous mode
Sep 26 19:27:03 pi5c kernel: cni0: port 8(veth21d981ec) entered disabled state
Sep 26 19:30:00 pi5c kernel: cni0: port 8(veth8d5ceb02) entered blocking state
Sep 26 19:30:00 pi5c kernel: cni0: port 8(veth8d5ceb02) entered disabled state
Sep 26 19:30:00 pi5c kernel: veth8d5ceb02: entered allmulticast mode
Sep 26 19:30:00 pi5c kernel: veth8d5ceb02: entered promiscuous mode
Sep 26 19:30:00 pi5c kernel: cni0: port 8(veth8d5ceb02) entered blocking state
Sep 26 19:30:00 pi5c kernel: cni0: port 8(veth8d5ceb02) entered forwarding state
Sep 26 19:30:03 pi5c kernel: cni0: port 8(veth8d5ceb02) entered disabled state
Sep 26 19:30:03 pi5c kernel: veth8d5ceb02 (unregistering): left allmulticast mode
Sep 26 19:30:03 pi5c kernel: veth8d5ceb02 (unregistering): left promiscuous mode
Sep 26 19:30:03 pi5c kernel: cni0: port 8(veth8d5ceb02) entered disabled state
Sep 27 03:29:13 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 5004 ms
Sep 27 03:36:16 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 5316 ms
Sep 27 04:14:19 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 5724 ms
Sep 27 05:02:23 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 5004 ms
Sep 27 07:44:41 pi5c kernel: sd 10:0:0:1: [sdk] 2129920 512-byte logical blocks: (1.09 GB/1.02 GiB)
Sep 27 07:44:41 pi5c kernel: sdk: detected capacity change from 1081344 to 2129920
Sep 27 09:42:53 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 5004 ms
Sep 27 10:16:20 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 5968 ms
Sep 27 13:37:43 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 5540 ms
Sep 27 17:02:58 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 5008 ms
Sep 27 17:14:46 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 5204 ms
Sep 28 04:39:03 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 5120 ms
Sep 28 04:39:08 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 9984 ms
Sep 28 04:39:13 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 15104 ms
Sep 28 04:39:18 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 19968 ms
Sep 28 04:39:23 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 25088 ms
Sep 28 04:39:28 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 29952 ms
Sep 28 04:39:33 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 35072 ms
Sep 28 04:39:38 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 39936 ms
Sep 28 04:39:43 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 45056 ms
Sep 28 04:39:48 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 50180 ms
Sep 28 04:39:54 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 56068 ms
Sep 28 04:39:59 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 60928 ms

...more of that...

Sep 28 08:03:31 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 12273152 ms
Sep 28 08:03:36 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 12278016 ms
Sep 28 08:03:41 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 12283136 ms
Sep 28 08:03:46 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 12288000 ms
Sep 28 08:03:51 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 12293120 ms
Sep 28 08:03:56 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 12297988 ms
Sep 28 08:04:01 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 12303104 ms
Sep 28 08:04:06 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 12307968 ms
Sep 28 08:04:11 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 12313088 ms
Sep 28 08:04:16 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 12317952 ms
Sep 28 08:04:21 pi5c kernel: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 12323072 ms

Additional context

No response

[nbuchwitz]

I've created a branch with two fixes which might help and where flagged in one of the recent sashiko reviews upstream. Can you please test if #7667 has the same issues?

sudo rpi-update pulls/7667

(make sure you have a backup etc.)

[DrJosh9000]

Thanks, trying it out now.