[enhacement] Disable Intel firmware embedded LLDP agent before collecting LLDP data

Bug #1750688 reported by Nobuto Murata
22
This bug affects 4 people
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Medium
Andre Ruiz

Bug Description

I hit a hardware specific case. I see LLDP info was collected on one NIC, but not on the other NIC with MAAS commissioning process.

In the end, some Intel cards including X710 have an embedded LLDP agent in their hardware, which was blocking forwarding LLDP packet to Linux kernel. It would be nice if MAAS disables the firmware embedded agent before running LLDP phase. Otherwise, LLDP info won't be collected.

Workaround would be to run the command below as a custom commissioning script.

i40e_path=/sys/kernel/debug/i40e
for dev in $i40e_path/*; do
    [ -e "$dev" ] || break
    echo lldp stop > "${dev}/command"
done

See:
https://communities.intel.com/message/343041#343041
https://bugs.launchpad.net/ironic-python-agent/+bug/1623659
https://jujucharms.com/u/ivoks/lldpd/#charm-config-i40e-lldp-stop
https://api.jujucharms.com/charmstore/v5/~ivoks/lldpd/archive/hooks/hooks.py

Tags: cpe-onsite

Related branches

Nobuto Murata (nobuto)
tags: added: cpe-onsite
Revision history for this message
Nobuto Murata (nobuto) wrote :

           *-network:0
                description: Ethernet interface
                product: Ethernet Controller X710 for 10GbE SFP+
                vendor: Intel Corporation
                physical id: 0
                bus info: pci@0000:01:00.0
                logical name: eno1
                version: 01
                serial: 24:6e:96:XX:XX:XX
                size: 10Gbit/s
                width: 64 bits
                clock: 33MHz
                capabilities: pm msi msix pciexpress vpd bus_master cap_list rom ethernet physical fibre
                configuration: autonegotiation=off broadcast=yes driver=i40e driverversion=1.4.25-k duplex=full firmware=6.00 0x800034e6 18.3.6 latency=0 link=yes multicast=yes slave=yes speed=10Gbit/s

Nobuto Murata (nobuto)
description: updated
Nobuto Murata (nobuto)
description: updated
Revision history for this message
Andres Rodriguez (andreserl) wrote :

Hi Nobuto,

What's the best way to discover this interface automatically? Is there a file we can check somewhere under sys/kernel to allow us to identify this interface?

Changed in maas:
milestone: none → 2.5.0alpha2
importance: Undecided → Medium
status: New → Incomplete
summary: - Disable Intel firmware embedded LLDP agent before collecting LLDP data
+ [enhacement] Disable Intel firmware embedded LLDP agent before
+ collecting LLDP data
Revision history for this message
Nobuto Murata (nobuto) wrote :

> What's the best way to discover this interface automatically? Is there a file we can check somewhere under sys/kernel to allow us to identify this interface?

The path is /sys/kernel/debug/i40e/<PCI_ID>/command like:
/sys/kernel/debug/i40e/0000:05:00.0/command

I suppose MAAS can write "lldp stop" whenever the paths exist. Because it's not persistent across reboots so it safely can be modified in the commissioning phase.

Changed in maas:
status: Incomplete → New
Changed in maas:
milestone: 2.5.0alpha2 → 2.5.0beta1
Changed in maas:
milestone: 2.5.0beta1 → 2.5.0beta2
status: New → Triaged
Changed in maas:
milestone: 2.5.0beta2 → 2.5.0rc1
Changed in maas:
milestone: 2.5.0rc1 → 2.5.x
Changed in maas:
milestone: 2.5.x → next
Revision history for this message
Adam Collard (adam-collard) wrote :

This bug has not seen any activity in the last 6 months, so it is being automatically closed.

If you are still experiencing this issue, please feel free to re-open.

MAAS Team

Changed in maas:
status: Triaged → Invalid
Changed in maas:
status: Invalid → New
Alberto Donato (ack)
Changed in maas:
milestone: next → none
status: New → Triaged
Revision history for this message
Andre Ruiz (andre-ruiz) wrote :

I'm developing a tool that will rely on lldp data collected by maas to draw a network map to catch errors and miscabling during HAP phase of cloud install in customers. It is very important that this gets fixed.

I know I can upload scripts to fix that (have it working right now), but this severely compromises the workflow because we normally use the "fce" tool (cpe-foundation) that does the maas installation and configuration for us and it would require us to:

1) patch fce for a workaroung (idea that the FCE team does not like, unless we clearly declare this is a bug that is going to get fixed in the future)

2) make the user go out of automatic process and inject an extra script in maas in the middle of the installation.

My goal is to have this fixed upstream as soon as possible and to get fce to workaround it until new version is generally available.

The fix seems simple enough. I think the best place to fix it is to send commands to the card on the same "lldp install" script, right before enabling the deamon.

I understand that we probably lack the hardware in our lab to test this, but I have access to clients hardware and have tested this, it works. And it is simple enough to have no consequences for any other case.

Shoud I just go ahead and submit an MP?

Revision history for this message
Andre Ruiz (andre-ruiz) wrote :
Changed in maas:
assignee: nobody → Andre Ruiz (andre-ruiz)
Changed in maas:
milestone: none → next
status: Triaged → Fix Committed
Changed in maas:
milestone: next → 2.9.0b4
Lee Trager (ltrager)
Changed in maas:
status: Fix Committed → Fix Released
Revision history for this message
Dan Streetman (ddstreet) wrote :

Technically the 'standard' way to enable/disable i40e fw lldp is to use ethtool, e.g.:

root@dwalin:~# ethtool --show-priv-flags enp4s0f0 | grep fw-lldp
disable-fw-lldp : off

root@dwalin:~# ethtool --set-priv-flags enp4s0f0 disable-fw-lldp on

root@dwalin:~# ethtool --show-priv-flags enp4s0f0 | grep fw-lldp
disable-fw-lldp : on

The debugfs api to i40e is probably not going to change though, it should be ok to use.

Revision history for this message
Andre Ruiz (andre-ruiz) wrote :

Thanks Dan, I did not know about that way. I agree it would be more interesting, and I could update the patch if needed, but since the debug interface is stable I'll leave it like that for now. We can revisit this in the future if needed.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.