Skip to content

drivers/rtc: fix rcu unexpected reboot#592

Open
yanmarkman wants to merge 1 commit into
sonic-net:202505from
yanmarkman:yanm-drivers-rtc-fix-rcu-unexp-reboot-202505
Open

drivers/rtc: fix rcu unexpected reboot#592
yanmarkman wants to merge 1 commit into
sonic-net:202505from
yanmarkman:yanm-drivers-rtc-fix-rcu-unexp-reboot-202505

Conversation

@yanmarkman

@yanmarkman yanmarkman commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Why needed (PROBLEM):
Under 'chrony' configuration or/and further hwrtc change the RCU on the rtc_timer_do_work() reported and reproduced.

How fixed:
drivers/rtc/interface.c: rtc_timer_do_work()
When processing expired RTC events and rearming them, use "now"
instead of "expiry" to prevent endless loops.

@yanmarkman
drivers/rtc: fix rcu unexpected reboot
79109e6 
Why needed (PROBLEM):
Under 'chrony' configuration or/and further hwrtc change
the RCU on the rtc_timer_do_work() reported and reproduced.

Easy reproduced on armhf Nokia-AC3X.
Not reproduced on SONiC TRIXIE.

How fixed:
drivers/rtc/interface.c: rtc_timer_do_work()
  When processing expired RTC events and rearming them, use "now"
  instead of "expiry" to prevent endless loops.

Signed-off-by: Elad Nachman <enachman@marvell.com>
Signed-off-by: Yan Markman <ymarkman@marvell.com>
@yanmarkman yanmarkman requested a review from a team as a code owner June 22, 2026 16:00
@mssonicbld

mssonicbld commented Jun 22, 2026

Copy link
Copy Markdown

/azp run

@azure-pipelines

azure-pipelines Bot commented Jun 22, 2026

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

paulmenzel
paulmenzel suggested changes Jun 22, 2026
View reviewed changes

@paulmenzel paulmenzel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also send this upstream for review.

Comment thread patch/0018-drivers-rtc-handle-OTF-clock-changes.patch

When processing expired RTC events and rearming them, use now
instead of expiry to prevent endless loops.

@paulmenzel paulmenzel Jun 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a Fixes: tag, and also list the devices in the commit message.

Comment thread patch/0018-drivers-rtc-handle-OTF-clock-changes.patch

+ if (rtc && rtc->ops && rtc->ops->read_time) {
+ if (!rtc->ops->read_time(rtc->dev.parent, &new_tm)) {
+ pr_err("new rtc tm %d %d %d %d %d %d %d %d %d\n",

@paulmenzel paulmenzel Jun 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please re-write. Currently it’s more like a debug message. (Which is good to have.)

Comment thread patch/0018-drivers-rtc-handle-OTF-clock-changes.patch
/* Re-add/fwd periodic timers */
if (ktime_to_ns(timer->period)) {
- timer->node.expires = ktime_add(timer->node.expires,
+ timer->node.expires = ktime_add(now/*timer->node.expires*/,

@paulmenzel paulmenzel Jun 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have git, so commenting this out is not useful.

@yanmarkman

yanmarkman commented Jun 22, 2026 via email

Copy link
Copy Markdown
Contributor Author

@yanmarkman yanmarkman mentioned this pull request Jun 23, 2026
Merged
yanmarkman added a commit to Marvell-switching/sonic-scripts that referenced this pull request Jun 23, 2026
@yanmarkman
patch: armhf 202505: (592)drivers-rtc fix rcu unexp reboot (#255)
486f22c 
sonic-net/sonic-linux-kernel#592

Signed-off-by: Yan Markman <ymarkman@marvell.com>
@yanmarkman

yanmarkman commented Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

HI @paulmenzel
@eladnachman has posted https://lkml.org/lkml/2026/6/24/888 to the Kernel-upstream

@saiarcot895

saiarcot895 commented Jun 27, 2026

Copy link
Copy Markdown
Contributor

I'm inclined to agree with the comments on the mailing list. What is special about your hardware that common code needs to be changed here?

@eladnachman

eladnachman commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

I'm inclined to agree with the comments on the mailing list. What is special about your hardware that common code needs to be changed here?

The HW is not special, the test added that changes the clock back and forth constantly is special, and apparently no other SONIC HW uses RTC as the main clock.

@saiarcot895

saiarcot895 commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

apparently no other SONIC HW uses RTC as the main clock

No. I have done RTC time jumps back and forth on other hardware running SONiC. None of them leads to an expected reboot.

@eladnachman

eladnachman commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

apparently no other SONIC HW uses RTC as the main clock

No. I have done RTC time jumps back and forth on other hardware running SONiC. None of them leads to an expected reboot.

The question is whether those HW's RTC support PIE/UIE, whether it was enabled and rtc_timer_do_work() was actually called. Can you list these HW it was tested on, plus link to a HW diagram of the board and SOC, and finally a link to the dts or for x86_64 lspci -k ? Thanks...

@saiarcot895

saiarcot895 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

@eladnachman

On amd64, I have two physical systems (a laptop and a desktop) where Chrony is managing the RTC; neither have run into CPU hangs. lspci -k output from a Framework AMD laptop (the RTC hardware doesn't appear to show up here):

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Phoenix Root Complex
        Subsystem: Framework Computer Inc. Device 0006
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Phoenix IOMMU
        Subsystem: Framework Computer Inc. Device 0006
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Phoenix Dummy Host Bridge
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Phoenix Dummy Host Bridge
00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Phoenix GPP Bridge
        Subsystem: Advanced Micro Devices, Inc. [AMD] Device 1453
        Kernel driver in use: pcieport
        Kernel modules: shpchp
00:02.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] Phoenix GPP Bridge
        Subsystem: Advanced Micro Devices, Inc. [AMD] Device 1453
        Kernel driver in use: pcieport
        Kernel modules: shpchp
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Phoenix Dummy Host Bridge
00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 19h USB4/Thunderbolt PCIe tunnel
        Subsystem: Advanced Micro Devices, Inc. [AMD] Device 1453
        Kernel driver in use: pcieport
        Kernel modules: shpchp
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Phoenix Dummy Host Bridge
00:04.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 19h USB4/Thunderbolt PCIe tunnel
        Subsystem: Advanced Micro Devices, Inc. [AMD] Device 1453
        Kernel driver in use: pcieport
        Kernel modules: shpchp
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Phoenix Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Phoenix Internal GPP Bridge to Bus [C:A]
        Subsystem: Device 0006:f111
        Kernel driver in use: pcieport
        Kernel modules: shpchp
00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Phoenix Internal GPP Bridge to Bus [C:A]
        Subsystem: Device 0006:f111
        Kernel driver in use: pcieport
        Kernel modules: shpchp
00:08.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Phoenix Internal GPP Bridge to Bus [C:A]
pcilib: Error reading /sys/bus/pci/devices/0000:00:08.3/label: Operation not permitted
        Subsystem: Device 0006:f111
        Kernel driver in use: pcieport
        Kernel modules: shpchp
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 71)
        Subsystem: Framework Computer Inc. Device 0006
        Kernel driver in use: piix4_smbus
        Kernel modules: i2c_piix4, sp5100_tco
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
        Subsystem: Framework Computer Inc. Device 0006
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Phoenix Data Fabric; Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Phoenix Data Fabric; Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Phoenix Data Fabric; Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Phoenix Data Fabric; Function 3
        Kernel driver in use: k10temp
        Kernel modules: k10temp
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Phoenix Data Fabric; Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Phoenix Data Fabric; Function 5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Phoenix Data Fabric; Function 6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Phoenix Data Fabric; Function 7
01:00.0 Network controller: Qualcomm Technologies, Inc WCN785x Wi-Fi 7(802.11be) 320MHz 2x2 [FastConnect 7800] (rev 01)
        Subsystem: Quectel Wireless Solutions Co., Ltd. Device 8000
        Kernel driver in use: ath12k_wifi7_pci
        Kernel modules: ath12k_wifi7
02:00.0 Non-Volatile memory controller: Sandisk Corp SanDisk Extreme Pro / WD Black SN750 / PC SN730 / Red SN700 NVMe SSD
        Subsystem: Sandisk Corp SanDisk Extreme Pro / WD Black SN750 / PC SN730 / Red SN700 NVMe SSD
        Kernel driver in use: nvme
        Kernel modules: nvme
03:00.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016] (rev 02)
        DeviceName: Realtek
        Subsystem: Device 1b45:1114
        Kernel driver in use: pcieport
        Kernel modules: shpchp
04:00.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016] (rev 02)
        Subsystem: Device 1b45:1114
        Kernel driver in use: pcieport
        Kernel modules: shpchp
04:01.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016] (rev 02)
        Subsystem: Device 1b45:1114
        Kernel driver in use: pcieport
        Kernel modules: shpchp
04:02.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016] (rev 02)
        Subsystem: Device 1b45:1114
        Kernel driver in use: pcieport
        Kernel modules: shpchp
04:04.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016] (rev 02)
        Subsystem: Device 1b45:1114
        Kernel driver in use: pcieport
        Kernel modules: shpchp
05:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev 10)
        Subsystem: Device 1b45:0005
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
06:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev 10)
        Subsystem: Device 1b45:0006
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
07:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
        Subsystem: Device 1b45:0007
        Kernel driver in use: igb
        Kernel modules: igb
c1:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Phoenix1 (rev cb)
        Subsystem: Framework Computer Inc. Device 0006
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu
c1:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Radeon High Definition Audio Controller
        Subsystem: Framework Computer Inc. Device 0006
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
c1:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Phoenix CCP/PSP 3.0 Device
        Subsystem: Framework Computer Inc. Device 0006
        Kernel driver in use: ccp
        Kernel modules: ccp
c1:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15b9
        Subsystem: Framework Computer Inc. Device 0006
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
c1:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15ba
        Subsystem: Framework Computer Inc. Device 0006
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
c1:00.5 Multimedia controller: Advanced Micro Devices, Inc. [AMD] Audio Coprocessor (rev 63)
        Subsystem: Framework Computer Inc. Device 0006
        Kernel driver in use: snd_pci_ps
        Kernel modules: snd_pci_acp3x, snd_rn_pci_acp3x, snd_pci_acp5x, snd_pci_acp6x, snd_acp_pci, snd_pci_ps
c1:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Ryzen HD Audio Controller
        Subsystem: Framework Computer Inc. Device 0006
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
c2:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Phoenix Dummy Function
        Subsystem: Framework Computer Inc. Device 0006
c2:00.1 Signal processing controller: Advanced Micro Devices, Inc. [AMD] AMD IPU Device
        Subsystem: Framework Computer Inc. Device 0006
c3:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Phoenix Dummy Function
        Subsystem: Framework Computer Inc. Device 0006
c3:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15c0
        Subsystem: Framework Computer Inc. Device 0006
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
c3:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15c1
        Subsystem: Framework Computer Inc. Device 0006
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
c3:00.5 USB controller: Advanced Micro Devices, Inc. [AMD] Pink Sardine USB4/Thunderbolt NHI controller #1
        Subsystem: Framework Computer Inc. Device 0006
        Kernel driver in use: thunderbolt
        Kernel modules: thunderbolt
c3:00.6 USB controller: Advanced Micro Devices, Inc. [AMD] Pink Sardine USB4/Thunderbolt NHI controller #2
        Subsystem: Framework Computer Inc. Device 0006
        Kernel driver in use: thunderbolt
        Kernel modules: thunderbolt

For arm64 specifically, there is one hardware that I have access to that has an RTC with UIE enabled:

admin@sonic:~$ sudo hwclock --param-get 0
The RTC parameter 0x0 is set to 0x10.

The patch that adds support for this hardware is here.

@yanmarkman

yanmarkman commented Jul 8, 2026 via email

Copy link
Copy Markdown
Contributor Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@paulmenzel paulmenzel paulmenzel requested changes
Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@yanmarkman @mssonicbld @saiarcot895 @eladnachman @paulmenzel