SMC stats: Wrong bucket calculation for payload of exactly 4096 bytes
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ubuntu on IBM z Systems |
Fix Released
|
Medium
|
Skipper Bug Screeners | ||
linux (Ubuntu) |
Invalid
|
Undecided
|
Unassigned | ||
Jammy |
Fix Released
|
Medium
|
Canonical Kernel Team | ||
Lunar |
Fix Released
|
Medium
|
Canonical Kernel Team | ||
Mantic |
Fix Released
|
Medium
|
Canonical Kernel Team |
Bug Description
SRU Justification:
[Impact]
* There is a wrong bucket calculation for payload of exactly 4096 bytes
in SMC stats counters.
* SMC_STAT_
* The impact is that a system silently updates an incorrect stats counter
in case the underlying kernel is not UBSAN enabled.
(Difficult to detect.)
* But if a kernel is UBSAN enabled, one will see a UBSAN
'array-
[ 26.335381] UBSAN: array-index-
[ 26.335385] index -1 is out of range for type 'u64 [9]'
[ 26.335388] CPU: 0 PID: 274 Comm: iperf3 Tainted: G E 5.15.0-79-generic #86-Ubuntu
[ 26.335391] Hardware name: IBM 8561 T01 772 (KVM/Linux)
[ 26.335393] Call Trace:
[ 26.335397] [<00000000cd92e
[ 26.335404] [<00000000cd92e
[ 26.335406] [<00000000cd52d
[ 26.335411] [<000003ff8033f
[ 26.335425] [<00000000cd6a7
[ 26.335431] [<00000000cd6a7
[ 26.335433] [<00000000cd1d4
[ 26.335438] [<00000000cd1d4
[ 26.335440] [<00000000cd1d7
[ 26.335442] [<00000000cd932
[ 26.335446] [<00000000cd940
[Fix]
* a950a5921db4 a950a5921db450c
[Test Plan]
* Since this got identified while the livepatch for jammy patches got added,
one could run a simiar (or the same) test like mentioned at LP#1639924
(but only jammy comes with official livepatch support).
* Alternatively a dedicated SMC stats counters test with a payload of
exactly 4096 bytes could be done (which is probably easier):
* Install uperf (https:/
(Wondering if it makes sense to pick this up as Debian/Ubuntu package ?!)
* Reset SMC-D stats on client and server side.
* Start uperf at the server side using: # uperf -vs
* Update profile with remote IP (server IP)
and start uperf at client: # uperf -vai 5 -m rr1c-4kx4k---1.xml
with uperf profile:
# cat rr1c-4kx4k---1.xml
<?xml version="1.0"?>
<profile name="TCP_RR">
<group nprocs="1">
<!--group nthreads="1"-->
<!-- if we want to run processes -->
<!--group nprocs="1"-->
<transaction iterations="1">
<flowop type="connect" options=
</transaction>
<transaction iterations="1">
<flowop type="write" options=
<flowop type="read" options=
</transaction>
<transaction iterations="1">
<flowop type="disconnect" />
</transaction>
</group>
</profile>
* The smcd stats output is:
# smcd -d stats reset
SMC-D Connections Summary
Total connections handled 2
SMC connections 2 (client 2, server 0)
v1 0
v2 2
Handshake errors 0 (client 0, server 0)
Avg requests per SMC conn 14.0
TCP fallback 0 (client 0, server 0)
RX Stats
Data transmitted (Bytes) 5796 (5.796K)
Total requests 9
Buffer full 0 (0.00%)
Buffer downgrades 0
Buffer reuses 0
8KB 16KB 32KB 64KB 128KB 256KB 512KB >512KB
Bufs 0 0 0 2 0 0 0 1
Reqs 8 0 0 0 0 0 0 0
TX Stats
Data transmitted (Bytes) 9960 (9.960K)
Total requests 19
Buffer full 0 (0.00%)
Buffer full (remote) 0 (0.00%)
Buffer too small 0 (0.00%)
Buffer too small (remote) 0 (0.00%)
Buffer downgrades 0
Buffer reuses 0
8KB 16KB 32KB 64KB 128KB 256KB 512KB >512KB
Bufs 0 2 0 0 0 0 0 0
Reqs 18 0 0 0 0 0 0 1
Extras
Special socket calls 0
cork 0
nodelay 0
sendpage 0
splice 0
urgent data 0
* Instead of including the payload in the wrong >512 KB buckets,
the output should be to have 19 reqs in the 8 KB buckets for TX stats
and 9 reqs in the 8 KB bucket for RX stats.
[Where problems could occur]
* The changes are in common code /net/smc/
hence any potential negativ impact is not limited to a specific platform.
but limited to statistics for shared memory communication (SMC)
hardware statistics.
* But limited to the functions SMC_STAT_
SMC_
* The bucket (aka pos) calculation is not that trivial,
issues here could cause other calculation error,
that may lead to slightly different, but still wrong numbers.
* Further issues can happen in case variables in use may overflow.
* Issues can also happen if the ternary conditional operators
are not correctly used, which again may lead to wrong calculations.
[Other Info]
* This patch fixes upstream e0e4b8fa5338
("net/smc: Add SMC statistics support").
* Since the fix got upstream accepted with v6.6-rc6,
not only 22.04, but also 23.04 and 23.10 are affected.
(22.10 already reached it EOS).
* Will not affected the current development release (N-release),
since it will at least be based on a 6.6 kernel.
__________
Bug description by Nils H.:
------------
Overview:
------------
The following line in the SMC stats code (net/smc/
#define SMC_STAT_
do { \
typeof(_smc_stats) stats = (_smc_stats); \
typeof(_tech) t = (_tech); \
typeof(_len) l = (_len); \
int _pos = fls64((l) >> 13); \
typeof(_rc) r = (_rc); \
int m = SMC_BUF_MAX - 1; \
this_cpu_
if (r <= 0) \
break; \
_pos = (_pos < m) ? ((l == 1 << (_pos + 12)) ? _pos - 1 : _pos) : m; \ <---
this_cpu_
this_cpu_
} \
while (0)
With l = 4096, _pos evaluates to -1.
Checking with the following uperf profile:
# cat rr1c-4kx4k---1.xml
<?xml version="1.0"?>
<profile name="TCP_RR">
<group nprocs="1">
<!--group nthreads="1"-->
<!-- if we want to run processes -->
<!--group nprocs="1"-->
<transaction iterations="1">
<flowop type="connect" options=
</transaction>
<transaction iterations="1">
<flowop type="write" options=
<flowop type="read" options=
</transaction>
<transaction iterations="1">
<flowop type="disconnect" />
</transaction>
</group>
</profile>
smcd stats output:
# smcd -d stats reset
SMC-D Connections Summary
Total connections handled 2
SMC connections 2 (client 2, server 0)
v1 0
v2 2
Handshake errors 0 (client 0, server 0)
Avg requests per SMC conn 14.0
TCP fallback 0 (client 0, server 0)
RX Stats
Data transmitted (Bytes) 5796 (5.796K)
Total requests 9
Buffer full 0 (0.00%)
Buffer downgrades 0
Buffer reuses 0
8KB 16KB 32KB 64KB 128KB 256KB 512KB >512KB
Bufs 0 0 0 2 0 0 0 1
Reqs 8 0 0 0 0 0 0 0
TX Stats
Data transmitted (Bytes) 9960 (9.960K)
Total requests 19
Buffer full 0 (0.00%)
Buffer full (remote) 0 (0.00%)
Buffer too small 0 (0.00%)
Buffer too small (remote) 0 (0.00%)
Buffer downgrades 0
Buffer reuses 0
8KB 16KB 32KB 64KB 128KB 256KB 512KB >512KB
Bufs 0 2 0 0 0 0 0 0
Reqs 18 0 0 0 0 0 0 1
Extras
Special socket calls 0
cork 0
nodelay 0
sendpage 0
splice 0
urgent data 0
Instead of including the payload in the wrong >512KB buckets, output should be to have 19 reqs in the 8KB buckets for TX stats and 9 reqs in the 8KB bucket for RX stats.
--------
Repro:
--------
0. Install uperf.
1. Reset SMC-D stats on client and server.
2. Start uperf at server side: "uperf -vs".
3. Update profile with remote IP (server IP) and start uperf at client: "uperf -vai 5 -m rr1c-4kx4k---1.xml" (uperf profile, see above)
tags: | added: architecture-s39064 bugnameltc-203868 severity-medium targetmilestone-inin--- |
Changed in ubuntu: | |
assignee: | nobody → Skipper Bug Screeners (skipper-screen-team) |
affects: | ubuntu → linux (Ubuntu) |
Changed in linux (Ubuntu Mantic): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu Lunar): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu Jammy): | |
importance: | Undecided → Medium |
Changed in ubuntu-z-systems: | |
importance: | Undecided → Medium |
summary: |
- [UBUNTU 22.04] SMC stats: Wrong bucket calculation for payload of - exactly 4096 bytes. + SMC stats: Wrong bucket calculation for payload of exactly 4096 bytes |
Changed in linux (Ubuntu Mantic): | |
status: | In Progress → Fix Committed |
Changed in linux (Ubuntu Lunar): | |
status: | In Progress → Fix Committed |
Changed in linux (Ubuntu Jammy): | |
status: | In Progress → Fix Committed |
description: | updated |
tags: | removed: verification-needed-jammy-linux verification-needed-lunar-linux verification-needed-mantic-linux |
tags: |
added: targetmilestone-inin2204 removed: targetmilestone-inin--- |
tags: | added: verification-done |
tags: |
added: verification-done-jammy-linux verification-done-lunar-linux verification-done-mantic-linux removed: verification-done |
Changed in linux (Ubuntu): | |
status: | New → Invalid |
tags: |
added: verification-done removed: verification-done-focal-linux-gcp-tcpx verification-done-jammy-linux verification-done-lunar-linux verification-done-mantic-linux verification-needed-focal-linux-gcp-5.15 verification-needed-jammy-linux-azure-fips verification-needed-jammy-linux-gkeop verification-needed-jammy-linux-hwe-6.2 verification-needed-jammy-linux-hwe-6.5 verification-needed-jammy-linux-intel-iotg verification-needed-jammy-linux-lowlatency verification-needed-jammy-linux-lowlatency-hwe-6.2 verification-needed-jammy-linux-nvidia-6.5 verification-needed-jammy-linux-nvidia-tegra verification-needed-lunar-linux-aws verification-needed-lunar-linux-azure verification-needed-lunar-linux-gcp verification-needed-lunar-linux-kvm verification-needed-lunar-linux-riscv verification-needed-lunar-linux-starfive verification-needed-mantic-linux-azure verification-needed-mantic-linux-gcp verification-needed-mantic-linux-laptop |
tags: |
added: verification-done removed: verification-needed-focal-linux-intel-iotg-5.15 verification-needed-focal-linux-nvidia-tegra-5.15 verification-needed-jammy-linux-lowlatency-hwe-6.5 verification-needed-jammy-linux-nvidia-tegra-igx |
------- Comment From <email address hidden> 2023-10-17 10:23 EDT-------
--- MATTHEW R. commented ---
I encountered this issue recently while verifying a KVM feature. It's worth noting that if a kernel has UBSAN enabled (https:/ /docs.kernel. org/dev- tools/ubsan. html) then rather than silently updating an incorrect stats counter you will also get a UBSAN array-index- out-of- bounds warning every time this occurs. In my case, I bumped into this because I was using an Ubuntu kernel which came with UBSAN enabled. Example of the warning:
[ 26.335369] ======= ======= ======= ======= ======= ======= ======= ======= ======= ======= ======= === out-of- bounds in /build/ linux-O6Qi7m/ linux-5. 15.0/net/ smc/af_ smc.c:2402: 3 63a>] dump_stack_ lvl+0x62/ 0x80 36c>] ubsan_epilogue+ 0x1c/0x48 3c4>] __ubsan_ handle_ out_of_ bounds+ 0x94/0xa0 9da>] smc_sendmsg+ 0x2aa/0x2d0 [smc] 9a4>] sock_sendmsg+ 0x64/0x80 a32>] sock_write_ iter+0x72/ 0xa0 000>] new_sync_ write+0x100/ 0x190 bb8>] vfs_write+ 0x1e8/0x280 014>] ksys_write+ 0xb4/0x100 c7c>] __do_syscall+ 0x1bc/0x1f0 148>] system_ call+0x78/ 0xa0
[ 26.335381] UBSAN: array-index-
[ 26.335385] index -1 is out of range for type 'u64 [9]'
[ 26.335388] CPU: 0 PID: 274 Comm: iperf3 Tainted: G E 5.15.0-79-generic #86-Ubuntu
[ 26.335391] Hardware name: IBM 8561 T01 772 (KVM/Linux)
[ 26.335393] Call Trace:
[ 26.335397] [<00000000cd92e
[ 26.335404] [<00000000cd92e
[ 26.335406] [<00000000cd52d
[ 26.335411] [<000003ff8033f
[ 26.335425] [<00000000cd6a7
[ 26.335431] [<00000000cd6a7
[ 26.335433] [<00000000cd1d4
[ 26.335438] [<00000000cd1d4
[ 26.335440] [<00000000cd1d7
[ 26.335442] [<00000000cd932
[ 26.335446] [<00000000cd940
This makes the issue much more visible.
Worse, if you have panic_on_warn enabled (like I did) then this warning will subsequently trigger a kernel panic.