Quanta D52B-1U unable to PXE-boot in EFI mode

Jeff lane has noted that in the logs, the system PXE-boots off a8:1e:84:f2:96:c5 (which MAAS identifies as enp59s0f0), whereas in EFI mode, it uses a8:1e:84:f2:96:c6 (enp59s0f1).

Hi Rod,

Is this with Secure boot enabled? Could you also please attach:

Tcpdump of the pxe process for the specific MAC address.

That said, if there are other servers that boot fine in EFI mode, this, to me, would normally indicate that there is a bug in grub itself or in the firmware. I’m opening a task against grub.

@Rod,

MAAS doesn’t make any differentiation on what PXE interface the machine boots on provided that we identify the MAC address and send the configuration for it. The raw tcpdump should give us more info.

Please also attach the full console log.

Rod said this might be critical, so I'll add it for reference...

From what I could tell in rackd.log, every EFI boot always comes from a8:1e:84:f2:96:c6 and every BIOS PXE boot comes from a8:1e:84:f2:96:c5.

rackd.log:2018-03-01 12:32:26 provisioningserver.rackdservices.tftp: [info] bootx64.efi requested by a8:1e:84:f2:96:c6
rackd.log:2018-03-01 12:32:26 provisioningserver.rackdservices.tftp: [info] grubx64.efi requested by a8:1e:84:f2:96:c6
rackd.log:2018-03-01 12:36:42 provisioningserver.rackdservices.tftp: [info] pxelinux.0 requested by a8:1e:84:f2:96:c5
rackd.log:2018-03-01 12:36:42 provisioningserver.rackdservices.tftp: [info] pxelinux.0 requested by a8:1e:84:f2:96:c5
rackd.log:2018-03-01 12:36:42 provisioningserver.rackdservices.tftp: [info] ldlinux.c32 requested by a8:1e:84:f2:96:c5
rackd.log:2018-03-01 12:36:42 provisioningserver.rackdservices.tftp: [info] pxelinux.cfg/01-a8-1e-84-f2-96-c5 requested by a8:1e:84:f2:96:c5
rackd.log:2018-03-01 12:36:42 provisioningserver.rackdservices.tftp: [info] pxelinux.cfg/0A010AA4 requested by a8:1e:84:f2:96:c5
rackd.log:2018-03-01 12:36:42 provisioningserver.rackdservices.tftp: [info] pxelinux.cfg/0A010AA requested by a8:1e:84:f2:96:c5
rackd.log:2018-03-01 12:36:42 provisioningserver.rackdservices.tftp: [info] pxelinux.cfg/0A010A requested by a8:1e:84:f2:96:c5
rackd.log:2018-03-01 12:36:42 provisioningserver.rackdservices.tftp: [info] pxelinux.cfg/0A010 requested by a8:1e:84:f2:96:c5
rackd.log:2018-03-01 12:36:42 provisioningserver.rackdservices.tftp: [info] pxelinux.cfg/0A01 requested by a8:1e:84:f2:96:c5
rackd.log:2018-03-01 12:36:42 provisioningserver.rackdservices.tftp: [info] pxelinux.cfg/0A0 requested by a8:1e:84:f2:96:c5
rackd.log:2018-03-01 12:36:42 provisioningserver.rackdservices.tftp: [info] pxelinux.cfg/0A requested by a8:1e:84:f2:96:c5
rackd.log:2018-03-01 12:36:42 provisioningserver.rackdservices.tftp: [info] pxelinux.cfg/0 requested by a8:1e:84:f2:96:c5
rackd.log:2018-03-01 12:36:42 provisioningserver.rackdservices.tftp: [info] pxelinux.cfg/default requested by a8:1e:84:f2:96:c5
rackd.log:2018-03-01 12:36:42 provisioningserver.rackdservices.tftp: [info] ifcpu64.c32 requested by a8:1e:84:f2:96:c5
rackd.log:2018-03-01 12:36:42 provisioningserver.rackdservices.tftp: [info] libcom32.c32 requested by a8:1e:84:f2:96:c5
rackd.log:2018-03-01 12:36:42 provisioningserver.rackdservices.tftp: [info] ubuntu/amd64/hwe-16.04/xenial/daily/boot-kernel requested by a8:1e:84:f2:96:c5
rackd.log:2018-03-01 12:36:43 provisioningserver.rackdservices.tftp: [info] ubuntu/amd64/hwe-16.04/xenial/daily/boot-initrd requested by a8:1e:84:f2:96:c5

Here's a tcpdump capture of a commissioning attempt. I don't know what you mean by "full console log." The original bug report includes a full capture of the IPMI SOL console, and there is no "logs" tab in the MAAS web UI for the node.

Oh, this was all with Secure Boot disabled, as far as I can tell. (I see no Secure Boot options in the server's setup screens.)

I got into the system for a bit and tried forcing it to PXE off of each NIC, both cases resulted in the same thing... PXE request for bootx64.efi is made (and apparently succeeds), and then request for grubx64.efi happens and we eventually get dumped to a grub prompt.

I did notice this, though, in both cases the request for bootx64.efi happens twice... is this expected:

2018-03-01 15:33:32 provisioningserver.rackdservices.tftp: [info] bootx64.efi requested by a8:1e:84:f2:96:c6
2018-03-01 15:33:32 provisioningserver.rackdservices.tftp: [info] bootx64.efi requested by a8:1e:84:f2:96:c6
2018-03-01 15:33:32 provisioningserver.rackdservices.tftp: [info] grubx64.efi requested by a8:1e:84:f2:96:c6
2018-03-01 15:36:19 provisioningserver.rackdservices.tftp: [info] bootx64.efi requested by a8:1e:84:f2:96:c5
2018-03-01 15:36:19 provisioningserver.rackdservices.tftp: [info] bootx64.efi requested by a8:1e:84:f2:96:c5
2018-03-01 15:36:19 provisioningserver.rackdservices.tftp: [info] grubx64.efi requested by a8:1e:84:f2:96:c5

I've perused the tcpdump output. It looks like jehan is requesting TFTP data (GRUB) from weavile (the MAAS server), and that goes OK; then there's a LONG string of exchanges like this (copied from the wireshark summary screen):

6399 109.865912 QuantaCo_f2:96:c5 Broadcast ARP 60 Who has 10.1.10.2? Tell 10.1.10.164
6400 109.865935 HewlettP_f5:69:f1 QuantaCo_f2:96:c5 ARP 42 10.1.10.2 is at 48:0f:cf:f5:69:f1

This goes on for about 31 seconds, according to the time stamps. The mapping of IP addresses and MAC addresses looks correct (weavile is 10.1.10.2 and 48:0f:cf:f5:69:f1; jehan is a8:1e:84:f2:96:c5 and a8:1e:84:f2:96:c6, using whatever IP address weavile gives it, which does seem to be 10.1.10.164 on this run.

So in that pcap when I look at it with vim, I see things like:
¤Û^V^Fj^B^L*Å^@^C^C<9a>ons^@feature_200_final^@feature_nativedisk_cmd^@feature_timeout_style^@^@Possible commands are:^@Possible devices are:^@Possible files are:^@Possible partitions are:^@Possible arguments are:^@Possible things are:^@ %s^@ ^@
^@%s/x86_64-efi/command.lst^@module isn't loaded^@superusers^@user '%s' not found^@Enter username: ^@^H^@%c^@
^@Enter password: ^@access denied^@^@Check whether user is in USERLIST.^@[USERLIST]^@authenticate^@%s/x86_64-efi/fs.lst^@Warning: syntax error (missing slash) in `%s'
^@Warning: invalid foreground color `%s'

Which eventually:

¤Û^V^Fj^B^L*Å^@^C^C<9b>arning: invalid background color `%s'
^@black^@blue^@green^@cyan^@red^@magenta^@brown^@light-gray^@dark-gray^@light-blue^@light-green^@light-cyan^@light-red^@light-magenta^@yellow^@white^@)^@^@%s,%s^@" ^@' ^@ ^@.^@..^@%s/^@set^@-u^@--help^@--usage^@--%s^@/^@Sunday^@Monday^@Tuesday^@Wednesday^@Thursday^@Friday^@Saturday^@
   ^@Falling back to `%s'^@

^@
^@Press any key to continue...^@
  ^@Failed to boot both default and fallback entries.
^@timeout^@%d^@default^@timeout_style^@%d ^@theme^@gfxmenu^@module `%s' isn't loaded^@gfxterm^@fallback^@chosen^@0^@boot^@menu^M]<98>Zyñ^N^@<^@^@^@<^@^@^@H^OÏõiñ¨^^<84>ò<96>Å^H^@E^@^@ d6^@^@@^Qíï

Could this be that the firmware is the issue here?

Looking at the logs, I also see this:

2018-03-01 12:31:13 provisioningserver.rackdservices.tftp: [info] bootx64.efi requested by a8:1e:84:f2:96:c5
2018-03-01 12:31:13 provisioningserver.rackdservices.tftp: [info] bootx64.efi requested by a8:1e:84:f2:96:c5
2018-03-01 12:31:13 provisioningserver.rackdservices.tftp: [info] grubx64.efi requested by a8:1e:84:f2:96:c5
2018-03-01 12:32:25 provisioningserver.rackdservices.tftp: [info] bootx64.efi requested by a8:1e:84:f2:96:c6
2018-03-01 12:32:26 provisioningserver.rackdservices.tftp: [info] bootx64.efi requested by a8:1e:84:f2:96:c6
2018-03-01 12:32:26 provisioningserver.rackdservices.tftp: [info] grubx64.efi requested by a8:1e:84:f2:96:c6

Idk if that's the firmware doing it automatically (e.g. trying to boot from one interface, and then another) or someone manually is doing it, but judging for the logs i would imagine one of two things:

1. The firmware is taking a long time to download the requested file and timesout after 30 seconds
2. The firmware just doesn't really do anything once it downloads the file.

I'm assuming that other EFI machines work just fine, I would say this could even be a firmware related issue. Can you confirm other EFI machines work?

Andres, the fragments you're seeing in the pcap file look like GRUB code to me. Keep in mind that's just a raw capture of EVERYTHING that passed over the MAAS server's local interface for the time during which I was booting the node, so that includes the GRUB binary that was passed from the MAAS server to the node, and even a few interactions with unrelated machines. You'd need to use wireshark or something similar to properly decode it.

As noted in my comment #9, above, it looks to me like ARP responses from the MAAS server are getting lost by the node AFTER GRUB has been (presumably successfully, although I've not tried to analyze the tcpdump results to verify this) delivered to the node. This could be a GRUB bug, a Quanta firmware bug, or a bug in how the two interact with each other. Then too, my knowledge of how they all interact is limited, so my conclusion may be in error.

As to the rest, both Jeff and I tried completely disabling each of the machine's two network ports (they're 10Gbps fiber connections, FWIW). In some of my own tests, I tried typing "exit" at the "grub>" prompt. When both ports were enabled, this resulted in a second boot attempt, presumably one from each device. The TFTP request log snippet you've quoted looks consistent with one of those runs.

Other EFI machines do work fine; I deployed two yesterday, for instance. (They were both Quanta machines, but a different model.)

Just for grins, Rod, do we have any 1Gb PCIe cards we could throw in Jehan (when you're in Lex next time, or maybe sfeole can) to see if we can get it to PXE from SOMETHING in efi mode?

Also, you flashed the system firmware, how about the firmware for the NICs?

This may be a duplicate of bug #1437353.