Activity log for bug #1936850

Date Who What changed Old value New value Message
2021-07-19 16:34:04 Vladimir Grevtsev bug added bug
2021-07-19 16:36:05 Vladimir Grevtsev bug added subscriber Canonical Field Critical
2021-07-19 16:51:02 Vladimir Grevtsev description [Expected result] Given the below environment and config, the DPDK bond will be initialized successfully. [Actual result] ovs-vsctl show has errors: Bridge br-dpdk fail_mode: standalone datapath_type: netdev Port dpdk-bond0 Interface dpdk-769d67d type: dpdk options: {dpdk-devargs="0000:3e:00.0"} error: "Error attaching device '0000:3e:00.0' to DPDK" Interface dpdk-18f5dde type: dpdk options: {dpdk-devargs="0000:40:00.0"} error: "Error attaching device '0000:40:00.0' to DPDK" Port br-dpdk Interface br-dpdk type: internal ovs-vswitchd.log: 2021-07-19T16:24:49.359Z|00062|dpdk|ERR|EAL: Driver cannot attach the device (0000:3e:00.0) 2021-07-19T16:24:49.359Z|00063|dpdk|ERR|EAL: Failed to attach device on primary process 2021-07-19T16:24:49.359Z|00064|netdev_dpdk|WARN|Error attaching device '0000:3e:00.0' to DPDK 2021-07-19T16:24:49.359Z|00065|netdev|WARN|dpdk-769d67d: could not set configuration (Invalid argument) 2021-07-19T16:24:49.359Z|00066|dpdk|ERR|Invalid port_id=32 2021-07-19T16:24:49.379Z|00067|dpdk|ERR|EAL: Driver cannot attach the device (0000:40:00.0) 2021-07-19T16:24:49.379Z|00068|dpdk|ERR|EAL: Failed to attach device on primary process 2021-07-19T16:24:49.379Z|00069|netdev_dpdk|WARN|Error attaching device '0000:40:00.0' to DPDK 2021-07-19T16:24:49.379Z|00070|netdev|WARN|dpdk-18f5dde: could not set configuration (Invalid argument) 2021-07-19T16:24:49.379Z|00071|dpdk|ERR|Invalid port_id=32 2021-07-19T16:24:49.379Z|00072|bridge|INFO|bridge br-dpdk: added interface br-dpdk on port 65534 2021-07-19T16:24:49.379Z|00073|bridge|INFO|bridge br-int: using datapath ID 00002ea26c9ff64d == Environment focal/ussuri, hardware: Huawei CH121 V5 blade servers cat /proc/cmdline BOOT_IMAGE=/vmlinuz-5.4.0-77-generic root=/dev/mapper/vg0-sda--os ro console=tty0 console=ttyS0,115200n8 intel_iommu=on iommu=pt hugepagesz=1G hugepages=1480 default_hugepagesz=1G transparent_hugepage=never isolcpus=0-21,24-45,48-69,72-93 # grep HugePages_ /proc/meminfo HugePages_Total: 1480 HugePages_Free: 1472 HugePages_Rsvd: 0 HugePages_Surp: 0 # lspci | grep Ethernet 1b:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE backplane (rev 09) 1b:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE backplane (rev 09) 3d:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (4*25GE) (rev 45) 3e:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (4*25GE) (rev 45) 3f:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (4*25GE) (rev 45) 40:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (4*25GE) (rev 45) # sudo dmesg | grep -e IOMMU [ 5.744179] DMAR: IOMMU enabled [ 12.605696] DMAR-IR: IOAPIC id 12 under DRHD base 0xc5ffc000 IOMMU 6 [ 12.612122] DMAR-IR: IOAPIC id 11 under DRHD base 0xb87fc000 IOMMU 5 [ 12.618549] DMAR-IR: IOAPIC id 10 under DRHD base 0xaaffc000 IOMMU 4 [ 12.624977] DMAR-IR: IOAPIC id 18 under DRHD base 0xfbffc000 IOMMU 3 [ 12.631404] DMAR-IR: IOAPIC id 17 under DRHD base 0xee7fc000 IOMMU 2 [ 12.637830] DMAR-IR: IOAPIC id 16 under DRHD base 0xe0ffc000 IOMMU 1 [ 12.644259] DMAR-IR: IOAPIC id 15 under DRHD base 0xd37fc000 IOMMU 0 [ 12.650687] DMAR-IR: IOAPIC id 8 under DRHD base 0x9d7fc000 IOMMU 7 [ 12.657027] DMAR-IR: IOAPIC id 9 under DRHD base 0x9d7fc000 IOMMU 7 "dpdk-devbind.py -s" : https://paste.ubuntu.com/p/g8TqRHkyHJ/ ovs-vsctl show: https://paste.ubuntu.com/p/fYWZ8hzvyq/ ovs-vsctl list Open_Vswitch: https://paste.ubuntu.com/p/gHKxdfxgv3/ overlay used for ovn-chassis deployment: https://pastebin.canonical.com/p/BwTKr3HWvd/ ovs-vswitchd.log: https://paste.ubuntu.com/p/nb9nTgwgK8/ For the record, we've tried changing dpdk-driver from "vfio-pci" to "uio_pci_generic", but that didn't help. Please, advise, what could be our next steps in troubleshooting this? [Expected result] Given the below environment and config, the DPDK bond will be initialized successfully. [Actual result] ovs-vsctl show has errors:     Bridge br-dpdk         fail_mode: standalone         datapath_type: netdev         Port dpdk-bond0             Interface dpdk-769d67d                 type: dpdk                 options: {dpdk-devargs="0000:3e:00.0"}                 error: "Error attaching device '0000:3e:00.0' to DPDK"             Interface dpdk-18f5dde                 type: dpdk                 options: {dpdk-devargs="0000:40:00.0"}                 error: "Error attaching device '0000:40:00.0' to DPDK"         Port br-dpdk             Interface br-dpdk                 type: internal ovs-vswitchd.log: 2021-07-19T16:24:49.359Z|00062|dpdk|ERR|EAL: Driver cannot attach the device (0000:3e:00.0) 2021-07-19T16:24:49.359Z|00063|dpdk|ERR|EAL: Failed to attach device on primary process 2021-07-19T16:24:49.359Z|00064|netdev_dpdk|WARN|Error attaching device '0000:3e:00.0' to DPDK 2021-07-19T16:24:49.359Z|00065|netdev|WARN|dpdk-769d67d: could not set configuration (Invalid argument) 2021-07-19T16:24:49.359Z|00066|dpdk|ERR|Invalid port_id=32 2021-07-19T16:24:49.379Z|00067|dpdk|ERR|EAL: Driver cannot attach the device (0000:40:00.0) 2021-07-19T16:24:49.379Z|00068|dpdk|ERR|EAL: Failed to attach device on primary process 2021-07-19T16:24:49.379Z|00069|netdev_dpdk|WARN|Error attaching device '0000:40:00.0' to DPDK 2021-07-19T16:24:49.379Z|00070|netdev|WARN|dpdk-18f5dde: could not set configuration (Invalid argument) 2021-07-19T16:24:49.379Z|00071|dpdk|ERR|Invalid port_id=32 2021-07-19T16:24:49.379Z|00072|bridge|INFO|bridge br-dpdk: added interface br-dpdk on port 65534 2021-07-19T16:24:49.379Z|00073|bridge|INFO|bridge br-int: using datapath ID 00002ea26c9ff64d == Environment focal/ussuri, hardware: Huawei CH121 V5 blade servers cat /proc/cmdline BOOT_IMAGE=/vmlinuz-5.4.0-77-generic root=/dev/mapper/vg0-sda--os ro console=tty0 console=ttyS0,115200n8 intel_iommu=on iommu=pt hugepagesz=1G hugepages=1480 default_hugepagesz=1G transparent_hugepage=never isolcpus=0-21,24-45,48-69,72-93 # grep HugePages_ /proc/meminfo HugePages_Total: 1480 HugePages_Free: 1472 HugePages_Rsvd: 0 HugePages_Surp: 0 # lspci | grep Ethernet 1b:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE backplane (rev 09) 1b:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE backplane (rev 09) 3d:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (4*25GE) (rev 45) 3e:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (4*25GE) (rev 45) 3f:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (4*25GE) (rev 45) 40:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (4*25GE) (rev 45) # sudo dmesg | grep -e IOMMU [ 5.744179] DMAR: IOMMU enabled [ 12.605696] DMAR-IR: IOAPIC id 12 under DRHD base 0xc5ffc000 IOMMU 6 [ 12.612122] DMAR-IR: IOAPIC id 11 under DRHD base 0xb87fc000 IOMMU 5 [ 12.618549] DMAR-IR: IOAPIC id 10 under DRHD base 0xaaffc000 IOMMU 4 [ 12.624977] DMAR-IR: IOAPIC id 18 under DRHD base 0xfbffc000 IOMMU 3 [ 12.631404] DMAR-IR: IOAPIC id 17 under DRHD base 0xee7fc000 IOMMU 2 [ 12.637830] DMAR-IR: IOAPIC id 16 under DRHD base 0xe0ffc000 IOMMU 1 [ 12.644259] DMAR-IR: IOAPIC id 15 under DRHD base 0xd37fc000 IOMMU 0 [ 12.650687] DMAR-IR: IOAPIC id 8 under DRHD base 0x9d7fc000 IOMMU 7 [ 12.657027] DMAR-IR: IOAPIC id 9 under DRHD base 0x9d7fc000 IOMMU 7 "dpdk-devbind.py -s" : https://paste.ubuntu.com/p/g8TqRHkyHJ/ ovs-vsctl show: https://paste.ubuntu.com/p/fYWZ8hzvyq/ ovs-vsctl list Open_Vswitch: https://paste.ubuntu.com/p/gHKxdfxgv3/ overlay used for ovn-chassis deployment: https://pastebin.canonical.com/p/BwTKr3HWvd/ ovs-vswitchd.log: https://paste.ubuntu.com/p/nb9nTgwgK8/ extract from lspci -nnv: https://paste.ubuntu.com/p/NxKytM4m3M/ For the record, we've tried changing dpdk-driver from "vfio-pci" to "uio_pci_generic", but that didn't help. Please, advise, what could be our next steps in troubleshooting this?
2021-07-19 16:58:43 Vladimir Grevtsev tags field-critical
2021-07-19 17:11:59 Nobuto Murata bug added subscriber Nobuto Murata
2021-07-19 17:36:43 Vladimir Grevtsev summary Port cannot be added to DPDK-enabled bridge: "Error attaching device to DPDK": Invalid port_id=32 Charm doesn't install necessary DPDK drivers for hinic (Huawei) NICs
2021-07-19 17:36:47 Vladimir Grevtsev removed subscriber Canonical Field Critical
2021-07-19 17:36:53 Vladimir Grevtsev tags field-critical field-high
2021-07-19 17:40:14 Vladimir Grevtsev bug added subscriber Canonical Field High
2021-07-21 15:09:24 Corey Bryant charm-ovn-chassis: status New Triaged
2021-07-21 15:09:31 Corey Bryant charm-ovn-chassis: importance Undecided Wishlist
2021-10-06 03:39:34 Billy Olsen bug task added charm-layer-ovn
2021-10-06 03:40:23 Billy Olsen charm-layer-ovn: status New Triaged
2021-10-06 03:40:25 Billy Olsen charm-layer-ovn: importance Undecided Wishlist
2021-10-08 23:31:47 Billy Olsen charm-layer-ovn: status Triaged In Progress
2022-02-15 11:24:38 James Page charm-layer-ovn: status In Progress Fix Released
2022-02-16 14:57:27 OpenStack Infra charm-ovn-chassis: status Triaged In Progress
2022-02-16 14:59:15 James Page bug task added charm-ovn-dedicated-chassis
2022-02-16 14:59:21 James Page charm-ovn-dedicated-chassis: status New In Progress
2022-02-16 14:59:23 James Page charm-ovn-dedicated-chassis: importance Undecided Wishlist
2022-05-03 03:03:56 OpenStack Infra charm-ovn-dedicated-chassis: status In Progress Fix Committed
2023-06-26 21:28:46 Billy Olsen charm-ovn-chassis: status In Progress Fix Released
2023-06-26 21:28:47 Billy Olsen charm-ovn-dedicated-chassis: status Fix Committed Fix Released