Broadcom NXE controller Cumulus devices ID's are missing in 16.04.03 'kernel-fixed-upstream'

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1715368

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.13 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.13

We installed the latest upstream kernel and we do see the Broadcom devices listed.
Below is the mail I have copied from the Engineer who verified this.

Hi Anand,

Installed 4.13.0-041300 upstream kernel on top on Ubuntu 16.04.03 base kernel. It has bnxt_en 1.7.0 driver and both CuW A0 and CuW+ B1 cards are getting detected with this driver. Below is the modinfo bnxt_en driver output.

root@ubuntu160403-maaa:~# modinfo bnxt_en
filename:
/lib/modules/4.13.0-041300-generic/kernel/drivers/net/ethernet/broadcom/bnxt/bnxt_en.ko
version: 1.7.0
description: Broadcom BCM573xx network driver
license: GPL
srcversion: C20CD2026202947F8BF535A
alias: pci:v000014E4d000016E5sv*sd*bc*sc*i*
alias: pci:v000014E4d000016E1sv*sd*bc*sc*i*
alias: pci:v000014E4d000016DCsv*sd*bc*sc*i*
alias: pci:v000014E4d000016D3sv*sd*bc*sc*i*
alias: pci:v000014E4d000016CBsv*sd*bc*sc*i*
alias: pci:v000014E4d000016C1sv*sd*bc*sc*i*
alias: pci:v000014E4d00001609sv*sd*bc*sc*i*
alias: pci:v000014E4d00001606sv*sd*bc*sc*i*
alias: pci:v000014E4d00001614sv*sd*bc*sc*i*
alias: pci:v000014E4d000016F1sv*sd*bc*sc*i*
alias: pci:v000014E4d000016EFsv*sd*bc*sc*i*
alias: pci:v000014E4d000016EEsv*sd*bc*sc*i*
alias: pci:v000014E4d000016EDsv*sd*bc*sc*i*
alias: pci:v000014E4d000016ECsv*sd*bc*sc*i*
alias: pci:v000014E4d000016EBsv*sd*bc*sc*i*
alias: pci:v000014E4d000016EAsv*sd*bc*sc*i*
alias: pci:v000014E4d000016E9sv*sd*bc*sc*i*
alias: pci:v000014E4d000016E8sv*sd*bc*sc*i*
alias: pci:v000014E4d000016E7sv*sd*bc*sc*i*
alias: pci:v000014E4d000016E3sv*sd*bc*sc*i*
alias: pci:v000014E4d000016E2sv*sd*bc*sc*i*
alias: pci:v000014E4d000016DFsv*sd*bc*sc*i*
alias: pci:v000014E4d000016DEsv*sd*bc*sc*i*
alias: pci:v000014E4d000016D9sv*sd*bc*sc*i*
alias: pci:v000014E4d000016D8sv*sd*bc*sc*i*
alias: pci:v000014E4d000016D7sv*sd*bc*sc*i*
alias: pci:v000014E4d000016D6sv*sd*bc*sc*i*
alias: pci:v000014E4d000016D5sv*sd*bc*sc*i*
alias: pci:v000014E4d000016D4sv*sd*bc*sc*i*
alias: pci:v000014E4d000016D2sv*sd*bc*sc*i*
alias: pci:v000014E4d000016D1sv*sd*bc*sc*i*
alias: pci:v000014E4d000016D0sv*sd*bc*sc*i*
alias: pci:v000014E4d000016CFsv*sd*bc*sc*i*
alias: pci:v000014E4d000016CEsv*sd*bc*sc*i*
alias: pci:v000014E4d000016CDsv*sd*bc*sc*i*
alias: pci:v000014E4d000016CCsv*sd*bc*sc*i*
alias: pci:v000014E4d000016CAsv*sd*bc*sc*i*
alias: pci:v000014E4d000016C9sv*sd*bc*sc*i*
alias: pci:v000014E4d000016C8sv*sd*bc*sc*i*
alias: pci:v000014E4d000016C0sv*sd*bc*sc*i*
depends:
intree: Y
name: bnxt_en
vermagic: 4.13.0-041300-generic SMP mod_unload

I'd like to perform a reverse bisect to figure out which commit upstream fixes this regression. We need to figure out the last kernel that had this issue and the first kernel that did not.

Can you test the following kernels and report back? We are looking for the last kernel version that has the bug and the first that does not:

v4.11 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.11
v4.12 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.12
v4.13-rc1: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.13-rc1

You don't have to test every kernel, just up until the first kernel that does not have this bug.

Thanks in advance!

Hi Joseph,

we don't see this issue with all of the below kernels:

v4.11 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.11
v4.12 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.12
v4.13-rc1: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.13-rc1

all these kernels have same bnxt_en 1.7.0 driver and Broadcom NetXtreme-E controllers are getting detected with this driver.

This issue is seen on the Ubuntu 16.04.03 base kernel(4.4.0-87). Below are the Ubuntu 16.04.03 base OS/Kernel details:

root@ubuntu160403-maaa:~# cat /etc/issue
Ubuntu 16.04.3 LTS \n \l

root@ubuntu160403-maaa:~# uname -a
Linux ubuntu160403-maaa 4.4.0-87-generic #110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

root@ubuntu160403-maaa:~# modinfo bnxt_en
filename: /lib/modules/4.4.0-87-generic/kernel/drivers/net/ethernet/broadcom/bnxt/bnxt_en.ko
version: 0.1.24
description: Broadcom BCM573xx network driver
license: GPL
srcversion: 0110469E9118262B301CA46
alias: pci:v000014E4d000016D3sv*sd*bc*sc*i*
alias: pci:v000014E4d000016CBsv*sd*bc*sc*i*
alias: pci:v000014E4d000016D2sv*sd*bc*sc*i*
alias: pci:v000014E4d000016D1sv*sd*bc*sc*i*
alias: pci:v000014E4d000016CAsv*sd*bc*sc*i*
alias: pci:v000014E4d000016C9sv*sd*bc*sc*i*
depends: vxlan
intree: Y
vermagic: 4.4.0-87-generic SMP mod_unload modversions

Can you also test the follow kernels:

4.5 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.5-wily/
4.8 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.8/
4.10 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.10/

Again, you don't have to test every kernel, just up until the first kernel that does not have this bug.

Thanks in advance!

Joseph,
Murali has tried with 4.4 kernel as well and even that kernel has Cu device
ID's. Can we not just pull in the latest kernel patch which is 4.13.

Thanks,
Anand

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of
Joseph Salisbury
Sent: Wednesday, September 13, 2017 8:45 PM
To: <email address hidden>
Subject: [Bug 1715368] Re: Broadcom NXE controller Cumulus devices ID's are
missing in 16.04.03 'kernel-fixed-upstream'

Can you also test the follow kernels:

4.5 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.5-wily/
4.8 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.8/
4.10 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.10/

Again, you don't have to test every kernel, just up until the first kernel
that does not have this bug.

Thanks in advance!

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1715368

Title:
  Broadcom NXE controller Cumulus devices ID's are missing in 16.04.03
  'kernel-fixed-upstream'

Status in linux package in Ubuntu:
  Triaged
Status in linux source package in Xenial:
  Triaged
Status in linux source package in Zesty:
  Triaged

Bug description:
  Broadcom NXE controller Cumulus devices ID's are missing in 16.04.03.
  CuW B1 device IDs are missing with Ubuntu 16.04.03 inbox bnxt_en
driver(0.1.24)

  Below is the modinfo output of Ubuntu 16.04.03 inbox bnxt_en driver.

  root@ubuntu160403:~# modinfo bnxt_en
  filename:
/lib/modules/4.4.0-87-generic/kernel/drivers/net/ethernet/broadcom/bnxt/bnxt_en.ko
  version: 0.1.24
  description: Broadcom BCM573xx network driver
  license: GPL
  srcversion: 0110469E9118262B301CA46
  alias: pci:v000014E4d000016D3sv*sd*bc*sc*i*
  alias: pci:v000014E4d000016CBsv*sd*bc*sc*i*
  alias: pci:v000014E4d000016D2sv*sd*bc*sc*i*
  alias: pci:v000014E4d000016D1sv*sd*bc*sc*i*
  alias: pci:v000014E4d000016CAsv*sd*bc*sc*i*
  alias: pci:v000014E4d000016C9sv*sd*bc*sc*i*
  depends: vxlan
  intree: Y
  vermagic: 4.4.0-87-generic SMP mod_unload modversions

  OS Details:
  root@ubuntu160403:~# cat /etc/issue
  Ubuntu 16.04.3 LTS \n \l

  root@ubuntu160403:~# uname -a
  Linux ubuntu160403 4.4.0-87-generic #110-Ubuntu SMP Tue Jul 18 12:55:35
UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1715368/+subscriptions

The purpose of going through a "Reverse" bisect is to identify the commit that fixes this bug and adds the proper CuW B1 device IDs. To do that, we would want to figure out the last kernel that did not have the ids and the first kernel that did.

However, it sounds like all the upstream kernels you tested so far do not exhibit this bug (4.4 -> 4.13), but the 16.04.3 kernel does have the bug?

Also, the 16.04.3 kernel should be 4.10 based, but you see version 4.4.0-87? Is that correct?

Hi Joseph,

Yes, Ubuntu 16.04.03 has 4.4.0-87-generic kernel and we see the issue on "4.4.0-87-generic" kernel which has "bnxt_en 0.1.24" driver.

Hi Joseph,

Yes Ubuntu 16.04.03 has 4.4.0-87-generic kernel

i tried installing all the below kernels on top of Ubuntu 16.04.03 base Kernel(4.4.0-87-generic)
4.5 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.5-wily/
4.8 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.8/
4.10 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.10/

Kernel 4.5 has bnxt_en 1.0.0 driver:
====================================
 root@ubuntu160403-maaa:~# uname -a
 Linux ubuntu160403-maaa 4.5.0-040500-generic #201603140130 SMP Mon Mar 14 05:32:22 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
 root@ubuntu160403-maaa:~#
 root@ubuntu160403-maaa:~#
 root@ubuntu160403-maaa:~# modinfo bnxt_en
 filename: /lib/modules/4.5.0-040500-generic/kernel/drivers/net/ethernet/broadcom/bnxt/bnxt_en.ko
 version: 1.0.0
 description: Broadcom BCM573xx network driver
 license: GPL
 srcversion: 7211E701779D5B8510C6CC8
 alias: pci:v000014E4d000016D3sv*sd*bc*sc*i*
 alias: pci:v000014E4d000016CBsv*sd*bc*sc*i*
 alias: pci:v000014E4d000016D2sv*sd*bc*sc*i*
 alias: pci:v000014E4d000016D1sv*sd*bc*sc*i*
 alias: pci:v000014E4d000016D0sv*sd*bc*sc*i*
 alias: pci:v000014E4d000016CAsv*sd*bc*sc*i*
 alias: pci:v000014E4d000016C9sv*sd*bc*sc*i*
 alias: pci:v000014E4d000016C8sv*sd*bc*sc*i*
 depends: vxlan
 intree: Y
 vermagic: 4.5.0-040500-generic SMP mod_unload modversions

Kernel 4.8 has bnxt_en 1.3.0 driver:
====================================
root@ubuntu160403-maaa:~# uname -a
Linux ubuntu160403-maaa 4.8.0-040800-generic #201610022031 SMP Mon Oct 3 00:32:57 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
root@ubuntu160403-maaa:~#
root@ubuntu160403-maaa:~# modinfo bnxt_en
filename: /lib/modules/4.8.0-040800-generic/kernel/drivers/net/ethernet/broadcom/bnxt/bnxt_en.ko
version: 1.3.0
description: Broadcom BCM573xx network driver
license: GPL
srcversion: 6BBCAEB93DCB1C67F3F3479
alias: pci:v000014E4d000016E1sv*sd*bc*sc*i*
alias: pci:v000014E4d000016DCsv*sd*bc*sc*i*
alias: pci:v000014E4d000016D3sv*sd*bc*sc*i*
alias: pci:v000014E4d000016CBsv*sd*bc*sc*i*
alias: pci:v000014E4d000016EEsv*sd*bc*sc*i*
alias: pci:v000014E4d000016ECsv*sd*bc*sc*i*
alias: pci:v000014E4d000016E9sv*sd*bc*sc*i*
alias: pci:v000014E4d000016E8sv*sd*bc*sc*i*
alias: pci:v000014E4d000016E7sv*sd*bc*sc*i*
alias: pci:v000014E4d000016E3sv*sd*bc*sc*i*
alias: pci:v000014E4d000016E2sv*sd*bc*sc*i*
alias: pci:v000014E4d000016DFsv*sd*bc*sc*i*
alias: pci:v000014E4d000016DEsv*sd*bc*sc*i*
alias: pci:v000014E4d000016D9sv*sd*bc*sc*i*
alias: pci:v000014E4d000016D8sv*sd*bc*sc*i*
alias: pci:v000014E4d000016D7sv*sd*bc*sc*i*
alias: pci:v000014E4d000016D6sv*sd*bc*sc*i*
alias: pci:v000014E4d000016D5sv*sd*bc*sc*i*
alias: pci:v000014E4d000016D4sv*sd*bc*sc*i*
alias: pci:v000014E4d000016D2sv*sd*bc*...

Hi Joseph,
Did we zero down on the kernel that we see this issue with.
Can we pull in the latest bnxt driver for 16.04.03 updates ?

Thanks,
Anand

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of
Joseph Salisbury
Sent: Thursday, September 21, 2017 12:54 AM
To: <email address hidden>
Subject: [Bug 1715368] Re: Broadcom NXE controller Cumulus devices ID's are
missing in 16.04.03 'kernel-fixed-upstream'

The purpose of going through a "Reverse" bisect is to identify the commit
that fixes this bug and adds the proper CuW B1 device IDs. To do that, we
would want to figure out the last kernel that did not have the ids and the
first kernel that did.

However, it sounds like all the upstream kernels you tested so far do not
exhibit this bug (4.4 -> 4.13), but the 16.04.3 kernel does have the bug?

Also, the 16.04.3 kernel should be 4.10 based, but you see version 4.4.0-87?
Is that correct?

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1715368

Title:
  Broadcom NXE controller Cumulus devices ID's are missing in 16.04.03
  'kernel-fixed-upstream'

Status in linux package in Ubuntu:
  Triaged
Status in linux source package in Xenial:
  Triaged
Status in linux source package in Zesty:
  Triaged

Bug description:
  Broadcom NXE controller Cumulus devices ID's are missing in 16.04.03.
  CuW B1 device IDs are missing with Ubuntu 16.04.03 inbox bnxt_en
driver(0.1.24)

  Below is the modinfo output of Ubuntu 16.04.03 inbox bnxt_en driver.

  root@ubuntu160403:~# modinfo bnxt_en
  filename:
/lib/modules/4.4.0-87-generic/kernel/drivers/net/ethernet/broadcom/bnxt/bnxt_en.ko
  version: 0.1.24
  description: Broadcom BCM573xx network driver
  license: GPL
  srcversion: 0110469E9118262B301CA46
  alias: pci:v000014E4d000016D3sv*sd*bc*sc*i*
  alias: pci:v000014E4d000016CBsv*sd*bc*sc*i*
  alias: pci:v000014E4d000016D2sv*sd*bc*sc*i*
  alias: pci:v000014E4d000016D1sv*sd*bc*sc*i*
  alias: pci:v000014E4d000016CAsv*sd*bc*sc*i*
  alias: pci:v000014E4d000016C9sv*sd*bc*sc*i*
  depends: vxlan
  intree: Y
  vermagic: 4.4.0-87-generic SMP mod_unload modversions

  OS Details:
  root@ubuntu160403:~# cat /etc/issue
  Ubuntu 16.04.3 LTS \n \l

  root@ubuntu160403:~# uname -a
  Linux ubuntu160403 4.4.0-87-generic #110-Ubuntu SMP Tue Jul 18 12:55:35
UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1715368/+subscriptions

We don't generally pull in a new version of a driver to fix a bug. There is a document that describes the SRU process[0].

For this bug, what we would want to do is find the specific commit in the newer driver that fixes the bug, then SRU it to the 16.04.03 kernel.

To identify the commit that fixes this, we would need to perform a "Reverse" kernel bisect[1], which is why I asked for the testing of various kernel versions in comment #6.

It looks like you tested those kernels per comment #9. Is it correct that the 4.5 kernel with version: 1.0.0 does not have the bug? If that is the case, we can bisect between 4.4 and 4.5.

[0] https://wiki.ubuntu.com/StableReleaseUpdates

[1] https://wiki.ubuntu.com/Kernel/KernelBisection

You don't need to test multiple kernels or perform a bisect. You know the latest kernel has the IDs, so just run a git blame on that module and see what commit added those new IDs to the list.

Can you run: 'lspci -vvvnn' on a kernel that is working properly and attach the output? That will give us the exact pci id's that need to be added.

Thanks!

Hi Joseph,

Attached is the output of lspci -vvvnn command. This is taken on 4.13.0-041300-generic which had bnxt_en 1.7.0 driver.