MAAS fails to detect SuperMicro-based server's power type

Hi Roderick,

Could you please provide with cloud-init logs from commissioning/enlistment?

CHeers.

Andres, Where would I find those logs? I included the whole /var/log/maas tree from the server.

Rod,

You'll need to gather the logs from either the commissioning or enlistment environment.

CAn you have in-band access to the BMC though?

Sorry, Andres, but I don't know how to gather those logs. Can you point me to documentation?

I do have in-band access to the BMC.

I've added the requested log files from the enlistment phase.

Here's the failure:

"subprocess.CalledProcessError: Command '('bmc-config', '--commit', '--key-pair=User4:Lan_Enable_IPMI_Msgs=Yes')' returned non-zero exit status 1"

Jason, do you need something else from me to help pin this down further?

If you could provide access to the system I can poke around. I need to
research that command a bit more. I'm not sure why that command isn't
working - I need to research a bit and see if it's a mandatory command in
IPMI 1.5. Maybe we shouldn't always be calling it - not sure.

On Tue, Jun 3, 2014 at 10:51 AM, Roderick Smith <email address hidden>
wrote:

> Jason, do you need something else from me to help pin this down further?
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1312863
>
> Title:
> MAAS fails to detect SuperMicro-based server's power type
>
> Status in MAAS:
> Triaged
>
> Bug description:
> I have a computer with a SuperMicro X7SPA-HF-O motherboard. MAAS does
> not detect its power type correctly during enlistment; the power type
> comes up as unknown. Setting the power type manually and entering the
> correct username and password then enables MAAS to control the server.
>
> Interestingly, ipmitool shows the following in its user summary after
> an enlistment:
>
> $ ipmitool -H 172.24.21.51 -U ADMIN -P pass user list
> ID Name Callin Link Auth IPMI Msg Channel Priv Limit
> 2 ADMIN false false true ADMINISTRATOR
> 3 root true false false ADMINISTRATOR
> 4 maas true false false Unknown (0x00)
>
> This is my personal system, but I can provide enablement access if
> necessary.
>
> I'm attaching my MAAS log files from immediately after an enlistment
> and commissioning (with the manually-entered credentials). Here's my
> MAAS version information:
>
> $ dpkg -l '*maas*'|cat
> Desired=Unknown/Install/Remove/Purge/Hold
> |
> Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
> ||/ Name Version
> Architecture Description
>
> +++-=====================================================-===================================================-============-===============================================================================
> ii maas
> 1.5+bzr2252-0ubuntu1 all MAAS
> server all-in-one metapackage
> ii maas-cli
> 1.5+bzr2252-0ubuntu1 all MAAS
> command line API tool
> ii maas-cluster-controller
> 1.5+bzr2252-0ubuntu1 all MAAS
> server cluster controller
> ii maas-common
> 1.5+bzr2252-0ubuntu1 all MAAS
> server common files
> ii maas-dhcp
> 1.5+bzr2252-0ubuntu1 all MAAS DHCP
> server
> ii maas-dns
> 1.5+bzr2252-0ubuntu1 all MAAS DNS
> server
> ii maas-region-controller
> 1.5+bzr2252-0ubuntu1 all MAAS
> server complete region controller
> ii maas-region-controller-min
> 1.5+bzr2252-0ubuntu1 all MAAS
> Server minimum region ...

OK. The system is at home now and I'm in Lexington, so it'll have to wait until tomorrow, but I can certainly bring it in for a few days and give you access through the lab machines. I could have sworn it was using IPMI 2.0, but I have no way of verifying that at the moment.

The supermicro BMC is requiring Lan_Privilege_Limit to be set for a user before setting Lan_Enable_IPMI_Msgs = Yes for a user. If I swap the order of the two in maas_ipmi_autodetect.py, things work.

So the question is if swapping the order of those two will work on other systems too.

I linked a branch that fixes the issue for supermicro. I haven't tested it on any other BMCs. The IPMI spec is ambiguous about the order of setting these parameters. This is a tricky area to change and will require testing across the widest range we can get before we can know if this will be a safe change.

The code this was fixed in doesn't run on the MAAS server - it runs on the server being enlisted/commissioned, so you can test the fix by running the script on a server that has Ubuntu installed.

To test it on a server
1) Install Trusty on the server
2) sudo apt-get install freeipmi-tools
3) Download the patched version of the script: http://bazaar.launchpad.net/~jason-hobbs/maas/fix-supermicro-ipmi-detection/view/head:/etc/maas/templates/commissioning-user-data/snippets/maas_ipmi_autodetect.py
4) There are two cases that need to be tested - an existing 'maas' IPMI user, and no existing 'maas' IPMI user. If you have an existing 'maas' IPMI user, you should be able to disable it with: 'sudo bmc-config --commit --key-pair=User5:Username=""' - change User5 to whatever your current MAAS user is. You can list all users by doing "sudo bmc-config --checkout"
5) After clearing out the MAAS user, run the script: sudo python maas_ipmi_autodetect.py
6) That should result in a MAAS username, password, and IP being spit out to the console.
7) To test the case of an existing MAAS user, just run the script again - you should see a different password spit out that time.

I've tested (as in post #14) several systems:

1) The SuperMicro board that prompted this bug report
2) An OCP Roadrunner
3) An Intel Decathlete
4) A Dell PowerEdge T110
5) A Cisco C220

All reacted to the new maas_ipmi_autodetect.py script as expected. The Cisco had problems disabling the maas user, though:

~$ sudo bmc-config --commit --key-pair=User2:Username=""
ERROR: Failed to commit `User2:Username': Invalid/Unsupported Config

This seems to be unrelated to the fix, and AFAIK it's not a MAAS bug. I renamed the original maas user via the machine's HTTP BMC interface instead.

This bug also affects Softlayer bare metal servers (at least the ones I've had provisioned in their Amsterdam DC). The maas_ipmi_autodetect.py from Jason's branch worked. Could we please get this fix into mainline?

root@beggs:~# python /tmp/mainline_maas_ipmi_autodetect.py
Traceback (most recent call last):
  File "/tmp/mainline_maas_ipmi_autodetect.py", line 310, in <module>
    main()
  File "/tmp/mainline_maas_ipmi_autodetect.py", line 276, in main
    configure_ipmi_user(IPMI_MAAS_USER, IPMI_MAAS_PASSWORD)
  File "/tmp/mainline_maas_ipmi_autodetect.py", line 216, in configure_ipmi_user
    apply_ipmi_user_settings(user_settings)
  File "/tmp/mainline_maas_ipmi_autodetect.py", line 193, in apply_ipmi_user_settings
    verify_ipmi_user_settings(ipmi_user_number, user_settings)
  File "/tmp/mainline_maas_ipmi_autodetect.py", line 169, in verify_ipmi_user_settings
    value = bmc_user_get(user_number, key)
  File "/tmp/mainline_maas_ipmi_autodetect.py", line 73, in bmc_user_get
    raw = bmc_get(key)
  File "/tmp/mainline_maas_ipmi_autodetect.py", line 55, in bmc_get
    output = run_command(command)
  File "/tmp/mainline_maas_ipmi_autodetect.py", line 48, in run_command
    output = subprocess.check_output(command_args, stderr=subprocess.STDOUT)
  File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '('bmc-config', '--checkout', '--key-pair=User4:Lan_Privilege_Limit')' returned non-zero exit status 1
root@beggs:~# python /tmp/LP_1312863_maas_ipmi_autodetect.py
maas,wY1Zkm9x,10.104.159.149,LAN_2_0
root@beggs:~#

I've tested it on an HP DL360e G8 too, which has ILO4. I think this has coverage from enough of the major vendors to go ahead and land it now.

The hardware is:

Motherboard: SuperMicro X9DRI-LN4F+_R1.2A
Remote Mgmt Card: SuperMicro Winbond WPCM450 - Onboard

Full lshw output: https://pastebin.canonical.com/112186/

I've just fired off an e-mail to the tps-reports mailing list telling them of the imminent change; some of the folks there have access to more hardware for testing, if you want to wait another couple of days.

Rod, I've landed the change but more testing prior to the next release will still be good.