2021-07-19 16:51:02 |
Vladimir Grevtsev |
description |
[Expected result]
Given the below environment and config, the DPDK bond will be initialized successfully.
[Actual result]
ovs-vsctl show has errors:
Bridge br-dpdk
fail_mode: standalone
datapath_type: netdev
Port dpdk-bond0
Interface dpdk-769d67d
type: dpdk
options: {dpdk-devargs="0000:3e:00.0"}
error: "Error attaching device '0000:3e:00.0' to DPDK"
Interface dpdk-18f5dde
type: dpdk
options: {dpdk-devargs="0000:40:00.0"}
error: "Error attaching device '0000:40:00.0' to DPDK"
Port br-dpdk
Interface br-dpdk
type: internal
ovs-vswitchd.log:
2021-07-19T16:24:49.359Z|00062|dpdk|ERR|EAL: Driver cannot attach the device (0000:3e:00.0)
2021-07-19T16:24:49.359Z|00063|dpdk|ERR|EAL: Failed to attach device on primary process
2021-07-19T16:24:49.359Z|00064|netdev_dpdk|WARN|Error attaching device '0000:3e:00.0' to DPDK
2021-07-19T16:24:49.359Z|00065|netdev|WARN|dpdk-769d67d: could not set configuration (Invalid argument)
2021-07-19T16:24:49.359Z|00066|dpdk|ERR|Invalid port_id=32
2021-07-19T16:24:49.379Z|00067|dpdk|ERR|EAL: Driver cannot attach the device (0000:40:00.0)
2021-07-19T16:24:49.379Z|00068|dpdk|ERR|EAL: Failed to attach device on primary process
2021-07-19T16:24:49.379Z|00069|netdev_dpdk|WARN|Error attaching device '0000:40:00.0' to DPDK
2021-07-19T16:24:49.379Z|00070|netdev|WARN|dpdk-18f5dde: could not set configuration (Invalid argument)
2021-07-19T16:24:49.379Z|00071|dpdk|ERR|Invalid port_id=32
2021-07-19T16:24:49.379Z|00072|bridge|INFO|bridge br-dpdk: added interface br-dpdk on port 65534
2021-07-19T16:24:49.379Z|00073|bridge|INFO|bridge br-int: using datapath ID 00002ea26c9ff64d
== Environment
focal/ussuri, hardware: Huawei CH121 V5 blade servers
cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.4.0-77-generic root=/dev/mapper/vg0-sda--os ro console=tty0 console=ttyS0,115200n8 intel_iommu=on iommu=pt hugepagesz=1G hugepages=1480 default_hugepagesz=1G transparent_hugepage=never isolcpus=0-21,24-45,48-69,72-93
# grep HugePages_ /proc/meminfo
HugePages_Total: 1480
HugePages_Free: 1472
HugePages_Rsvd: 0
HugePages_Surp: 0
# lspci | grep Ethernet
1b:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE backplane (rev 09)
1b:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE backplane (rev 09)
3d:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (4*25GE) (rev 45)
3e:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (4*25GE) (rev 45)
3f:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (4*25GE) (rev 45)
40:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (4*25GE) (rev 45)
# sudo dmesg | grep -e IOMMU
[ 5.744179] DMAR: IOMMU enabled
[ 12.605696] DMAR-IR: IOAPIC id 12 under DRHD base 0xc5ffc000 IOMMU 6
[ 12.612122] DMAR-IR: IOAPIC id 11 under DRHD base 0xb87fc000 IOMMU 5
[ 12.618549] DMAR-IR: IOAPIC id 10 under DRHD base 0xaaffc000 IOMMU 4
[ 12.624977] DMAR-IR: IOAPIC id 18 under DRHD base 0xfbffc000 IOMMU 3
[ 12.631404] DMAR-IR: IOAPIC id 17 under DRHD base 0xee7fc000 IOMMU 2
[ 12.637830] DMAR-IR: IOAPIC id 16 under DRHD base 0xe0ffc000 IOMMU 1
[ 12.644259] DMAR-IR: IOAPIC id 15 under DRHD base 0xd37fc000 IOMMU 0
[ 12.650687] DMAR-IR: IOAPIC id 8 under DRHD base 0x9d7fc000 IOMMU 7
[ 12.657027] DMAR-IR: IOAPIC id 9 under DRHD base 0x9d7fc000 IOMMU 7
"dpdk-devbind.py -s" : https://paste.ubuntu.com/p/g8TqRHkyHJ/
ovs-vsctl show: https://paste.ubuntu.com/p/fYWZ8hzvyq/
ovs-vsctl list Open_Vswitch: https://paste.ubuntu.com/p/gHKxdfxgv3/
overlay used for ovn-chassis deployment: https://pastebin.canonical.com/p/BwTKr3HWvd/
ovs-vswitchd.log: https://paste.ubuntu.com/p/nb9nTgwgK8/
For the record, we've tried changing dpdk-driver from "vfio-pci" to "uio_pci_generic", but that didn't help.
Please, advise, what could be our next steps in troubleshooting this? |
[Expected result]
Given the below environment and config, the DPDK bond will be initialized successfully.
[Actual result]
ovs-vsctl show has errors:
Bridge br-dpdk
fail_mode: standalone
datapath_type: netdev
Port dpdk-bond0
Interface dpdk-769d67d
type: dpdk
options: {dpdk-devargs="0000:3e:00.0"}
error: "Error attaching device '0000:3e:00.0' to DPDK"
Interface dpdk-18f5dde
type: dpdk
options: {dpdk-devargs="0000:40:00.0"}
error: "Error attaching device '0000:40:00.0' to DPDK"
Port br-dpdk
Interface br-dpdk
type: internal
ovs-vswitchd.log:
2021-07-19T16:24:49.359Z|00062|dpdk|ERR|EAL: Driver cannot attach the device (0000:3e:00.0)
2021-07-19T16:24:49.359Z|00063|dpdk|ERR|EAL: Failed to attach device on primary process
2021-07-19T16:24:49.359Z|00064|netdev_dpdk|WARN|Error attaching device '0000:3e:00.0' to DPDK
2021-07-19T16:24:49.359Z|00065|netdev|WARN|dpdk-769d67d: could not set configuration (Invalid argument)
2021-07-19T16:24:49.359Z|00066|dpdk|ERR|Invalid port_id=32
2021-07-19T16:24:49.379Z|00067|dpdk|ERR|EAL: Driver cannot attach the device (0000:40:00.0)
2021-07-19T16:24:49.379Z|00068|dpdk|ERR|EAL: Failed to attach device on primary process
2021-07-19T16:24:49.379Z|00069|netdev_dpdk|WARN|Error attaching device '0000:40:00.0' to DPDK
2021-07-19T16:24:49.379Z|00070|netdev|WARN|dpdk-18f5dde: could not set configuration (Invalid argument)
2021-07-19T16:24:49.379Z|00071|dpdk|ERR|Invalid port_id=32
2021-07-19T16:24:49.379Z|00072|bridge|INFO|bridge br-dpdk: added interface br-dpdk on port 65534
2021-07-19T16:24:49.379Z|00073|bridge|INFO|bridge br-int: using datapath ID 00002ea26c9ff64d
== Environment
focal/ussuri, hardware: Huawei CH121 V5 blade servers
cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.4.0-77-generic root=/dev/mapper/vg0-sda--os ro console=tty0 console=ttyS0,115200n8 intel_iommu=on iommu=pt hugepagesz=1G hugepages=1480 default_hugepagesz=1G transparent_hugepage=never isolcpus=0-21,24-45,48-69,72-93
# grep HugePages_ /proc/meminfo
HugePages_Total: 1480
HugePages_Free: 1472
HugePages_Rsvd: 0
HugePages_Surp: 0
# lspci | grep Ethernet
1b:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE backplane (rev 09)
1b:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE backplane (rev 09)
3d:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (4*25GE) (rev 45)
3e:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (4*25GE) (rev 45)
3f:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (4*25GE) (rev 45)
40:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (4*25GE) (rev 45)
# sudo dmesg | grep -e IOMMU
[ 5.744179] DMAR: IOMMU enabled
[ 12.605696] DMAR-IR: IOAPIC id 12 under DRHD base 0xc5ffc000 IOMMU 6
[ 12.612122] DMAR-IR: IOAPIC id 11 under DRHD base 0xb87fc000 IOMMU 5
[ 12.618549] DMAR-IR: IOAPIC id 10 under DRHD base 0xaaffc000 IOMMU 4
[ 12.624977] DMAR-IR: IOAPIC id 18 under DRHD base 0xfbffc000 IOMMU 3
[ 12.631404] DMAR-IR: IOAPIC id 17 under DRHD base 0xee7fc000 IOMMU 2
[ 12.637830] DMAR-IR: IOAPIC id 16 under DRHD base 0xe0ffc000 IOMMU 1
[ 12.644259] DMAR-IR: IOAPIC id 15 under DRHD base 0xd37fc000 IOMMU 0
[ 12.650687] DMAR-IR: IOAPIC id 8 under DRHD base 0x9d7fc000 IOMMU 7
[ 12.657027] DMAR-IR: IOAPIC id 9 under DRHD base 0x9d7fc000 IOMMU 7
"dpdk-devbind.py -s" : https://paste.ubuntu.com/p/g8TqRHkyHJ/
ovs-vsctl show: https://paste.ubuntu.com/p/fYWZ8hzvyq/
ovs-vsctl list Open_Vswitch: https://paste.ubuntu.com/p/gHKxdfxgv3/
overlay used for ovn-chassis deployment: https://pastebin.canonical.com/p/BwTKr3HWvd/
ovs-vswitchd.log: https://paste.ubuntu.com/p/nb9nTgwgK8/
extract from lspci -nnv: https://paste.ubuntu.com/p/NxKytM4m3M/
For the record, we've tried changing dpdk-driver from "vfio-pci" to "uio_pci_generic", but that didn't help.
Please, advise, what could be our next steps in troubleshooting this? |
|