[Dell PowerEdge R510] Regression: Kernel 3.2.0-64 fails to boot with USB3 controller card

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1328984

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

I am unable to run the apport-collect command for two reasons:

1. The bug in question renders the machine unbootable. To boot the machine and run apport-collect, it is necessary to change either the software or hardware configuration. This would create an environment in which bug is not reproducible.

2. Two affected machines are servers located behind a corporate proxy. Apport-collect does not allow transmission through a proxy.

As instructed, I am changing the status to 'Confirmed'.

Maciej Puzio:

1) The purpose of the apport-collect is to gather the necessary debug information about your hardware, not provide logs about the crash.

2) One may utilize https://help.ubuntu.com/community/ReportingBugs#Filing_bugs_when_off-line to attach the file to this report.

Thank you for the information. I collected apport data, but I am currently unable to submit it, as the procedure outlined on the linked page still requires usage of tools that do not work with a proxy. Unless I run into more apport-related problems, I will be able to submit it later today or tomorrow.

In the meantime, I managed to replicate the regression on a different hardware (still using HighPoint RocketU 1144C and Areca ARC-5040). Should I include my observations, error logs and apport-collected data from that machine in this bug report, or should I create a new bug?

I am unable to submit apport-collected data due to what appears to be numerous bugs in apport tools:

1. Submitting data directly from the affected machine is not possible due to apport not being able to connect through a proxy.

2. Following instructions on referenced page to submit previously created .apport file using ubuntu-bug does not work, because ubuntu-bug ignores file specified in -c or --crash-file parameter, and proceeds to gather system information from the machine on which it is running. As this is a wrong machine, I had no choice but to cancel the submission.

3. In addition, ubuntu-bug generates the following error:
No packages found matching linux.
ERROR: hook /usr/share/apport/general-hooks/cloud_archive.py crashed

At this point debugging apport is not my priority. I would be grateful if you indicate an alternative way to proceed.

Maciej Puzio, could you please boot into 14.04 Server via http://releases.ubuntu.com/trusty/ (or 10.04/other release if that doesn't work), gather the apport-collect file, use a USB stick to move the data to another computer, and attach the file manually to this report?

I'm attaching the file generated by command:
apport-cli -f -p linux --save bug1328984.apport
This was done with the machine running 12.04 Server with kernel 3.2.0-63.

May I ask for the reply to my question about the results of testing the problem on a different hardware? (The regression has been reproduced, but symptoms are somewhat different, and I worry about confusion which may result from mixing discussion about two hardware configurations. On the other hand, I am reluctant to start a new bug report, as this appears to be a single bug.)

Maciej Puzio, it's strongly preferred one reports on a per hardware basis, as it's cheap to mark something a duplicate of another if determined as such.

Maciej Puzio, could you please test the latest upstream kernel available from the first line at the top page (not the daily folder) following https://wiki.ubuntu.com/KernelMainlineBuilds ? It will allow additional upstream developers to examine the issue. Once you've tested the upstream kernel, please comment on which kernel version specifically you tested. If this bug is fixed in the mainline kernel, please add the following tags:
kernel-fixed-upstream
kernel-fixed-upstream-VERSION-NUMBER

where VERSION-NUMBER is the version number of the kernel you tested. For example:
kernel-fixed-upstream-3.15

This can be done by clicking on the yellow circle with a black pencil icon next to the word Tags located at the bottom of the bug description. As well, please remove the tag:
needs-upstream-testing

If the mainline kernel does not fix this bug, please add the following tags:
kernel-bug-exists-upstream
kernel-bug-exists-upstream-VERSION-NUMBER

As well, please remove the tag:
needs-upstream-testing

Once testing of the upstream kernel is complete, please mark this bug's Status as Confirmed. Please let us know your results. Thank you for your understanding.

I had the same issue on a server that has been running Ubuntu 12.04 LTS without a single problem for over two years.

After upgrading to linux-image-3.2.0-64-generic on the system with a Supermicro X9SCM mainboard, two HighPoint RocketU 1144A USB 3.0 controllers and at least one USB 3.0 disk attached, I observed the same infinite udevd loop (although the message details were different). When I disconnected all USB 3.0 disks, the system booted.

My workaround was to revert to the previously installed kernel, linux-image-3.2.0-61-generic.

I observed the udevd loop with only one Western Digital My Passport 0748 2TB USB 3.0 disk connected.

Bard Hemmer, thank you for your comment. So your hardware and problem may be tracked, could you please file a new report with Ubuntu by executing the following in a terminal while booted into the default Ubuntu kernel (not a mainline one) via:
ubuntu-bug linux

For more on this, please read the official Ubuntu documentation:
Ubuntu Bug Control and Ubuntu Bug Squad: https://wiki.ubuntu.com/Bugs/BestPractices#X.2BAC8-Reporting.Focus_on_One_Issue
Ubuntu Kernel Team: https://wiki.ubuntu.com/KernelTeam/KernelTeamBugPolicies#Filing_Kernel_Bug_reports
Ubuntu Community: https://help.ubuntu.com/community/ReportingBugs#Bug_reporting_etiquette

When opening up the new report, please feel free to subscribe me to it.

Thank you for your understanding.

Helpful bug reporting tips:
https://wiki.ubuntu.com/ReportingBugs

I have created a new bug report describing this problem replicated on another hardware: bug 1330530
As that is a test machine entirely devoted to this issue, I will test the upstream kernel on it and post the results in bug 1330530.

Regarding testing of the upstream kernel on PowerEdge machines, these are production servers, and I need to schedule a maintenance window in order to take one of them offline. I am required to give an advance notification to users, so this is not something that can be done on a very short notice, or very often (once per week is max I can do). For this reason, may I ask if there are any other conceivable tests that I could run? It would speed things up considerably if I could use the maintenance window to do as many tests as possible, rather than do one at a time.

I have tested kernels 3.16.0-031600rc1-generic and 3.2.60-030260-generic. On the former, the problem does not appear, on the latter, the bug is replicated with similar symptoms as on 3.2.0-64. I used a flash drive with a vanilla Ubuntu 12.04 desktop install for all tests. To summarize kernels tested so far:
Good kernels: 3.2.0-63, 3.16.0-031600rc1
Bad kernels: 3.2.0-64, 3.2.60-030260

I also tested this issue on three additional machines, and the results were the same. So I have now five different hardware configurations (including one from bug 1330530) that are affected by this problem and show very similar symptoms. In fact, I was not able to find a computer that would not replicate this regression. If we also take into account Bard Hemmer's hardware, we can reasonably conclude that the issue is not related to motherboard/chipset/CPU/BIOS. It is however related to HighPoint RocketU 1144C add-in adapter that I used in all my tests.

I would like to note that symptoms are similar on various hardware, but not identical. The errors are generally similar (xhci, udev, modprobe), but it appears that timing differences cause the issue to occur at different parts of the boot process, depending on the hardware. So far I have seen:

1. Dropping to initramfs shell in the middle of the boot ("Gave up waiting for root device." ... ALERT! [boot drive] does not exist! Dropping to shell!")

2. An error loop preventing system to boot (as described in this report). In this case I am not sure whether this is an infinite loop, or if the system would boot after a long delay.

3. Boot is delayed by 18 minutes, during which time numerous errors are thrown. After 18 minutes, OS boots fine.

4. System boots to text console, rather than the graphical login screen. It is possible to log on to the console. Within seconds, xhci and/or udev errors start appearing in the syslog. After two minutes, screen goes blank, and the console seems unresponsive for another 16 minutes. Following that, the graphical login screen appears, and from this point system behaves fine.

5. As in 4, but after two minutes in the text console, incomplete graphical login screen appears. Password box is missing and the background is not fully loaded. After another 16 minutes, login screen loads missing parts, and system behaves OK. In this case it is possible to switch between text and graphical consoles during these 16 minutes, but the graphical console becomes a purple empty screen after the switch.

It is also worth noting that symptoms are highly dependent on the external device(s) attached to RocketU's ports. Here is a summary:

1. No device connected to RocletU adapter - no problems during boot
2. USB3 flash drives (tested two models) - no problems during boot
3. Areca ARC-5040 enclosure - bug is triggered
4. WD MyPassport 2TB US 3.0 drive - bug is triggered
5. Transcend USB 3.0 SD card reader (TS-RDF5K) - bug is triggered with different symptoms: only a small delay (~15 seconds) and small number of xhci errors occur during boot, but the device does not work when OS is fully booted.

All the above devices work fine with "good" kernels. Note that I tested three RocketU controllers and five Ar...

Maciej Puzio, the next step is to fully reverse commit bisect the kernel in order to identify the offending commit. Could you please do this following https://wiki.ubuntu.com/Kernel/KernelBisection#How_do_I_reverse_bisect_the_upstream_kernel.3F ?

tl;dr: This bug report is a duplicate of bug 1330530.

I am going to focus my efforts on bug 1330530, and I do not intend to do any work on this bug report until bug 1330530 is resolved. This is because doing the same work twice is not a good use of time and effort of anybody involved. Please do not change the status to incomplete, and do not request same steps or actions as for bug 1330530. Instead, please mark this bug as a duplicate of 1330530. This will focus everybody's attention on the problem and minimize confusion. Thank you.