Commissioning changes password in MAAS but not on node

Bug #2032849 reported by Rod Smith
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
MAAS
New
Medium
Unassigned

Bug Description

When commissioning hoggus, a Cisco UCS-S3260-M5SRB in the Server Certification lab, MAAS changes the password that's already set for the "maas" account for that machine, but the password on the server itself is not changed. The result is a successful commissioning, but a "power error" in the MAAS web UI and an inability to do deploy the machine until the passwords are brought back into sync. Using the "skip configuring supported BMC controllers with a MAAS generated username and password" works around this bug, but of course the user has no idea to use this option beforehand; and even if they do, it's easy to forget to click the button. This node is configured to use Redfish because IPMI has been unreliable.

Ideally, the relevant MAAS commissioning script should detect that the password change has failed and then not change the MAAS password (or change it back to its old value).

I'm attaching recent MAAS log files to this bug report.

Tags: bug-council
Revision history for this message
Rod Smith (rodsmith) wrote :
Revision history for this message
Alexsander de Souza (alexsander-souza) wrote :

Hi,

can you get us the output of the following command:

maas $ADMIN node-script-result download $SYSTEM_ID current-commissioning

Changed in maas:
status: New → Incomplete
Revision history for this message
Rod Smith (rodsmith) wrote :

Here you go....

Alberto Donato (ack)
Changed in maas:
status: Incomplete → Triaged
milestone: none → 3.5.0
importance: Undecided → High
Alberto Donato (ack)
Changed in maas:
status: Triaged → New
importance: High → Undecided
milestone: 3.5.0 → none
Revision history for this message
Alberto Donato (ack) wrote :

Could you please confirm that calling `bmc-config --commit --key-pair=User3:Password=<somepassword>` actually sets the password for the BMC?
I.e. it doesn't just succeed, but the password is actually updated.

Changed in maas:
status: New → Incomplete
Revision history for this message
Rod Smith (rodsmith) wrote :

Alberto, I've tried that on the machine in question (hoggus), and the `bmc-config --commit --key-pair=User3:Password=<somepassword>` command SEEMS to succeed (it produces no error message and has a return value of 0), but the password is NOT changed; the old password remains valid.

Changed in maas:
status: Incomplete → New
tags: added: bug-council
Revision history for this message
Alberto Donato (ack) wrote :

This looks like a possible bug in bmc-config (from freeipmi-tools), or an issue with the specific hardware (which might be require a specific workaround or extra config option).

Revision history for this message
Jeff Lane  (bladernr) wrote :

@ack the machine is configured for Redfish, should MAAS even be trying to change a user account for machines configured by Redfish? (I thought that password change was only for a MAAS IPMI user account, and if it's not configured to use IPMI why is it even running that code?)

Revision history for this message
Alexsander de Souza (alexsander-souza) wrote :

MAAS is trying to configure IPMI because it's failing to detect the Redfish interface:

  INFO: Checking for HP Moonshot...
  INFO: Checking for Redfish...
  ERROR: Redfish configuration failed. Missing SMBIOS data
  INFO: Checking for IPMI...
  INFO: IPMI detected!
  INFO: Reading current IPMI BMC values...

If we look at the output of 'dmidecode -t 42', it's in fact missing the info we expect:

  Handle 0x0024, DMI type 42, 12 bytes
  Management Controller Host Interface

We expect something like:

  Handle 0x00D0, DMI type 42, 169 bytes
  Management Controller Host Interface
    Host Interface Type: Network
    (...)
    idVendor: 0x04b3
    idProduct: 0x4010
    Protocol ID: 04 (Redfish over IP)
        Service UUID: a4b13f22-d0f3-11ea-aca9-e5b8d327f39f
        (...)

I'm not sure that we can work-around this. Commissioning scripts run in an untrusted environment, so they cannot fetch any host-specific data from MAAS to make up for the missing data, and without it the Redfish setup should fail and the script should attempt to configure IPMI (which kind of succeeded).

Revision history for this message
Alexsander de Souza (alexsander-souza) wrote :

As an improvement we can explore the viability of MAAS telling the commissioning scripts about the expected power method, so the script can fail immediately instead of using any fallback.

Changed in maas:
status: New → Triaged
importance: Undecided → Medium
milestone: none → 3.5.0
Revision history for this message
Rod Smith (rodsmith) wrote :

What about having MAAS do this:

chage_password(old_pw, new_pw)
if !test_password(new_pw)
  if test_password(old_pw)
    revert_to_old_password(old_pw)

This should cause MAAS to ignore any password change that doesn't succeed, no matter what the cause of the failure.

Revision history for this message
Jerzy Husakowski (jhusakowski) wrote :

MAAS fails to detect Redfish, and proceeds to configure IPMI. We see that the command to set the password succeeds without an error, however: which password is then left unchanged - the Redfish one (expected) or the IPMI one (unexpected)? Could you please clarify how you conclude that the password is unchanged?

Changed in maas:
status: Triaged → Incomplete
Revision history for this message
Rod Smith (rodsmith) wrote :

MAAS changes the password it believes the node to have, but the change on the node itself was unsuccessful. For instance:

1) Node is already commissioned, with a known password for both web & Redfish access. (IPMI access has never worked on this node.)
2) Re-commission the node in MAAS. This seems to succeed; but....
3) MAAS shows "power error" and cannot control the node.
4) It's possible to log into the BMC's web UI using the old password.
5) Changing the password to the old password in the MAAS web UI clears the "power error" and enables MAAS to control the node once again.

It's been long enough since we enlisted this node that I don't recall what happened during enlistment.

Changed in maas:
status: Incomplete → New
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.