Given the below environment and config, the DPDK bond will be initialized successfully.
[Actual result]
ovs-vsctl show has errors:
Bridge br-dpdk
fail_mode: standalone datapath_type: netdev
Port dpdk-bond0 Interface dpdk-769d67d type: dpdk options: {dpdk-devargs="0000:3e:00.0"} error: "Error attaching device '0000:3e:00.0' to DPDK" Interface dpdk-18f5dde type: dpdk options: {dpdk-devargs="0000:40:00.0"} error: "Error attaching device '0000:40:00.0' to DPDK"
Port br-dpdk Interface br-dpdk type: internal
ovs-vswitchd.log:
2021-07-19T16:24:49.359Z|00062|dpdk|ERR|EAL: Driver cannot attach the device (0000:3e:00.0)
2021-07-19T16:24:49.359Z|00063|dpdk|ERR|EAL: Failed to attach device on primary process
2021-07-19T16:24:49.359Z|00064|netdev_dpdk|WARN|Error attaching device '0000:3e:00.0' to DPDK
2021-07-19T16:24:49.359Z|00065|netdev|WARN|dpdk-769d67d: could not set configuration (Invalid argument)
2021-07-19T16:24:49.359Z|00066|dpdk|ERR|Invalid port_id=32
2021-07-19T16:24:49.379Z|00067|dpdk|ERR|EAL: Driver cannot attach the device (0000:40:00.0)
2021-07-19T16:24:49.379Z|00068|dpdk|ERR|EAL: Failed to attach device on primary process
2021-07-19T16:24:49.379Z|00069|netdev_dpdk|WARN|Error attaching device '0000:40:00.0' to DPDK
2021-07-19T16:24:49.379Z|00070|netdev|WARN|dpdk-18f5dde: could not set configuration (Invalid argument)
2021-07-19T16:24:49.379Z|00071|dpdk|ERR|Invalid port_id=32
2021-07-19T16:24:49.379Z|00072|bridge|INFO|bridge br-dpdk: added interface br-dpdk on port 65534
2021-07-19T16:24:49.379Z|00073|bridge|INFO|bridge br-int: using datapath ID 00002ea26c9ff64d
# sudo dmesg | grep -e IOMMU
[ 5.744179] DMAR: IOMMU enabled
[ 12.605696] DMAR-IR: IOAPIC id 12 under DRHD base 0xc5ffc000 IOMMU 6
[ 12.612122] DMAR-IR: IOAPIC id 11 under DRHD base 0xb87fc000 IOMMU 5
[ 12.618549] DMAR-IR: IOAPIC id 10 under DRHD base 0xaaffc000 IOMMU 4
[ 12.624977] DMAR-IR: IOAPIC id 18 under DRHD base 0xfbffc000 IOMMU 3
[ 12.631404] DMAR-IR: IOAPIC id 17 under DRHD base 0xee7fc000 IOMMU 2
[ 12.637830] DMAR-IR: IOAPIC id 16 under DRHD base 0xe0ffc000 IOMMU 1
[ 12.644259] DMAR-IR: IOAPIC id 15 under DRHD base 0xd37fc000 IOMMU 0
[ 12.650687] DMAR-IR: IOAPIC id 8 under DRHD base 0x9d7fc000 IOMMU 7
[ 12.657027] DMAR-IR: IOAPIC id 9 under DRHD base 0x9d7fc000 IOMMU 7
[Expected result]
Given the below environment and config, the DPDK bond will be initialized successfully.
[Actual result]
ovs-vsctl show has errors:
Bridge br-dpdk
datapath_ type: netdev
Interface dpdk-769d67d
type: dpdk
options: {dpdk-devargs= "0000:3e: 00.0"}
error: "Error attaching device '0000:3e:00.0' to DPDK"
Interface dpdk-18f5dde
type: dpdk
options: {dpdk-devargs= "0000:40: 00.0"}
error: "Error attaching device '0000:40:00.0' to DPDK"
Interface br-dpdk
type: internal
fail_mode: standalone
Port dpdk-bond0
Port br-dpdk
ovs-vswitchd.log:
2021-07- 19T16:24: 49.359Z| 00062|dpdk| ERR|EAL: Driver cannot attach the device (0000:3e:00.0) 19T16:24: 49.359Z| 00063|dpdk| ERR|EAL: Failed to attach device on primary process 19T16:24: 49.359Z| 00064|netdev_ dpdk|WARN| Error attaching device '0000:3e:00.0' to DPDK 19T16:24: 49.359Z| 00065|netdev| WARN|dpdk- 769d67d: could not set configuration (Invalid argument) 19T16:24: 49.359Z| 00066|dpdk| ERR|Invalid port_id=32 19T16:24: 49.379Z| 00067|dpdk| ERR|EAL: Driver cannot attach the device (0000:40:00.0) 19T16:24: 49.379Z| 00068|dpdk| ERR|EAL: Failed to attach device on primary process 19T16:24: 49.379Z| 00069|netdev_ dpdk|WARN| Error attaching device '0000:40:00.0' to DPDK 19T16:24: 49.379Z| 00070|netdev| WARN|dpdk- 18f5dde: could not set configuration (Invalid argument) 19T16:24: 49.379Z| 00071|dpdk| ERR|Invalid port_id=32 19T16:24: 49.379Z| 00072|bridge| INFO|bridge br-dpdk: added interface br-dpdk on port 65534 19T16:24: 49.379Z| 00073|bridge| INFO|bridge br-int: using datapath ID 00002ea26c9ff64d
2021-07-
2021-07-
2021-07-
2021-07-
2021-07-
2021-07-
2021-07-
2021-07-
2021-07-
2021-07-
2021-07-
== Environment
focal/ussuri, hardware: Huawei CH121 V5 blade servers
cat /proc/cmdline /vmlinuz- 5.4.0-77- generic root=/dev/ mapper/ vg0-sda- -os ro console=tty0 console= ttyS0,115200n8 intel_iommu=on iommu=pt hugepagesz=1G hugepages=1480 default_ hugepagesz= 1G transparent_ hugepage= never isolcpus= 0-21,24- 45,48-69, 72-93
BOOT_IMAGE=
# grep HugePages_ /proc/meminfo
HugePages_Total: 1480
HugePages_Free: 1472
HugePages_Rsvd: 0
HugePages_Surp: 0
# lspci | grep Ethernet
1b:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE backplane (rev 09)
1b:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE backplane (rev 09)
3d:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (4*25GE) (rev 45)
3e:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (4*25GE) (rev 45)
3f:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (4*25GE) (rev 45)
40:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (4*25GE) (rev 45)
# sudo dmesg | grep -e IOMMU
[ 5.744179] DMAR: IOMMU enabled
[ 12.605696] DMAR-IR: IOAPIC id 12 under DRHD base 0xc5ffc000 IOMMU 6
[ 12.612122] DMAR-IR: IOAPIC id 11 under DRHD base 0xb87fc000 IOMMU 5
[ 12.618549] DMAR-IR: IOAPIC id 10 under DRHD base 0xaaffc000 IOMMU 4
[ 12.624977] DMAR-IR: IOAPIC id 18 under DRHD base 0xfbffc000 IOMMU 3
[ 12.631404] DMAR-IR: IOAPIC id 17 under DRHD base 0xee7fc000 IOMMU 2
[ 12.637830] DMAR-IR: IOAPIC id 16 under DRHD base 0xe0ffc000 IOMMU 1
[ 12.644259] DMAR-IR: IOAPIC id 15 under DRHD base 0xd37fc000 IOMMU 0
[ 12.650687] DMAR-IR: IOAPIC id 8 under DRHD base 0x9d7fc000 IOMMU 7
[ 12.657027] DMAR-IR: IOAPIC id 9 under DRHD base 0x9d7fc000 IOMMU 7
"dpdk-devbind.py -s" : https:/ /paste. ubuntu. com/p/g8TqRHkyH J/ /paste. ubuntu. com/p/fYWZ8hzvy q/ /paste. ubuntu. com/p/gHKxdfxgv 3/ /pastebin. canonical. com/p/BwTKr3HWv d/ /paste. ubuntu. com/p/nb9nTgwgK 8/
ovs-vsctl show: https:/
ovs-vsctl list Open_Vswitch: https:/
overlay used for ovn-chassis deployment: https:/
ovs-vswitchd.log: https:/
For the record, we've tried changing dpdk-driver from "vfio-pci" to "uio_pci_generic", but that didn't help.
Please, advise, what could be our next steps in troubleshooting this?