system crash when removing ipmi_msghandler module
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
In Progress
|
Medium
|
Ioanna Alifieraki | ||
Focal |
Fix Released
|
Medium
|
Ioanna Alifieraki | ||
Hirsute |
Won't Fix
|
Medium
|
Ioanna Alifieraki | ||
Impish |
Fix Released
|
Medium
|
Ioanna Alifieraki | ||
Jammy |
In Progress
|
Medium
|
Ioanna Alifieraki |
Bug Description
[IMPACT]
Commit 3b9a907223d7 (ipmi: fix sleep-in-atomic in free_user at cleanup SRCU user->release_
pushes the removal of an ipmi_user into the system's workqueue.
Whenever an ipmi_user struct is about to be removed it is scheduled as a work on the system's workqueue to guarantee the free operation won't be executed in atomic context. When the work is executed the free_user_work() function is invoked which frees the ipmi_user.
When ipmi_msghandler module is removed in cleanup_ipmi() function, there is no check if there are any pending works to be executed.
Therefore, there is a potential race condition :
An ipmi_user is scheduled for removal and shortly after to remove the ipmi_msghandler module.
If the scheduled work delays execution for any reason and the module is removed first, then when the work is executed the pages of free_user_work() are gone and the system crashes with the following :
BUG: unable to handle page fault for address: ffffffffc05c3450
#PF: supervisor instruction fetch in kernel mode
#PF: error_code(0x0010) - not-present page
PGD 635420e067 P4D 635420e067 PUD 6354210067 PMD 4711e51067 PTE 0
Oops: 0010 [#1] SMP PTI
CPU: 19 PID: 29646 Comm: kworker/19:1 Kdump: loaded Not tainted 5.4.0-77-generic #86~18.04.1-Ubuntu
Hardware name: Ciara Technologies ORION RS610-G4-
Workqueue: events 0xffffffffc05c3450
RIP: 0010:0xffffffff
Code: Bad RIP value.
RSP: 0018:ffffb72133
RAX: ffffffffc05c3450 RBX: ffff92a95f56a740 RCX: ffffb7221cfd14e8
RDX: 0000000000000001 RSI: ffff92616040d4b0 RDI: ffffb7221cf404e0
RBP: ffffb721333c3ec0 R08: 000073746e657665 R09: 8080808080808080
R10: ffffb721333c3de0 R11: fefefefefefefeff R12: ffff92a95f570700
R13: ffff92a0a40ece40 R14: ffffb7221cf404e0 R15: 0ffff92a95f57070
FS: 000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffffffc05c3426 CR3: 00000081e9bfc005 CR4: 00000000007606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
? process_
worker_
kthread+0x121/0x140
? process_
? kthread_
ret_from_
Modules linked in: xt_REDIRECT xt_owner ipt_rpfilter xt_CT xt_multiport xt_set ip_set_hash_ip veth xt_statistic ipt_REJECT
... megaraid_sas ahci libahci wmi [last unloaded: ipmi_msghandler]
CR2: ffffffffc05c3450
[TEST CASE]
The user who reported the issue can reproduce reliably by stopping the ipmi related services and then removing the ipmi modules.
I could reproduce the issue only when turning the normal 'work' to delayed work.
[WHERE PROBLEMS COULD OCCUR]
The fixing patch creates a dedicated workqueue for the remove_work struct of ipmi_user when loading the ipmi_msghandler modules and destroys the workqueue when removing the module. Therefore any potential problems would occur during these two operations or when scheduling works on the dedicated workqueue.
[OTHER]
Upstream patches :
1d49eb91e86e (ipmi: Move remove_work to dedicated workqueue)
5a3ba99b62d8 (ipmi: msghandler: Make symbol 'remove_work_wq' static)
CVE References
Changed in linux (Ubuntu Focal): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu Hirsute): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu Impish): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu Jammy): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu Focal): | |
status: | New → Confirmed |
Changed in linux (Ubuntu Hirsute): | |
status: | New → Confirmed |
Changed in linux (Ubuntu Impish): | |
status: | New → Confirmed |
Changed in linux (Ubuntu Jammy): | |
status: | New → In Progress |
assignee: | nobody → Ioanna Alifieraki (joalif) |
Changed in linux (Ubuntu Impish): | |
assignee: | nobody → Ioanna Alifieraki (joalif) |
Changed in linux (Ubuntu Hirsute): | |
assignee: | nobody → Ioanna Alifieraki (joalif) |
Changed in linux (Ubuntu Focal): | |
assignee: | nobody → Ioanna Alifieraki (joalif) |
description: | updated |
description: | updated |
Changed in linux (Ubuntu Focal): | |
status: | Confirmed → Fix Committed |
Changed in linux (Ubuntu Hirsute): | |
status: | Confirmed → Fix Committed |
Changed in linux (Ubuntu Impish): | |
status: | Confirmed → Fix Committed |
https://<email address hidden>/