MAAS

IPMI access lost (not just maas user) after commissioning

Bug #1944605 reported by dann frazier
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Undecided
Christian Grabowski
MAAS 3.2.0-beta1

Bug Description

MAAS 1:3.0.0-10029-g.986ea3e45-0ubuntu1~20.04.1

I manually enlisted a system (an Ampere Mt. Jade), filling in my existing IPMI credentials for the factory default ADMIN user. Once I did so, the system began to commission as expected. However, after running 30-maas-01-bmc-config, MAAS lost access to the system and was unable to check the power status.

"Failed to query node's BMC - Access denied while performing power action: cipher suite unavailable. Check BMC configuration and try again."

So far this sounds like bug 1940319, albeit on a different model system (that was an Nvidia DGX-2). However, unlike bug 1940319, choosing a different cipher suite option in the MAAS pull down does not fix it. And worse, this doesn't just impact the newly created "maas" user. The factory default ADMIN user is also impacted, causing me to be unable to manage power or get serial-over-lan access to the console. I didn't see anything in the BMC Web UI that I could configure to correct the situation - everything looks the same on an identically configured system w/o this problem. And of course, I can't send IPMI commands to try and change cipher config because I can't authenticate. I had to resort to resetting the BMC to factory defaults.

Related branches

~cgrabowski/maas:remove_disabling_of_ipmi_ciphers
MAAS Lander: Approve
Alberto Donato (community): Approve
Diff: 523 lines (+70/-315)
7 files modified
src/maasserver/forms/parameters.py (+1/-0)
src/maasserver/forms/settings.py (+24/-1)
src/maasserver/models/config.py (+1/-0)
src/maasserver/models/tests/test_config.py (+4/-0)
src/metadataserver/builtin_scripts/commissioning_scripts/bmc_config.py (+8/-131)
src/metadataserver/builtin_scripts/commissioning_scripts/tests/test_bmc_config.py (+30/-183)
src/provisioningserver/drivers/power/ipmi.py (+2/-0)
Revision history for this message
Björn Tillenius (bjornt) wrote :

Could you try to commission the machine with "Allow SSH access and prevent machine powering off" checked?

Then you can ssh to the machine and check what MAAS did to the BMC.

Changed in maas:
status: New → Incomplete
Revision history for this message
dann frazier (dannf) wrote :

Sure, it is in that state now. What would you like for me to collect?

Revision history for this message
dann frazier (dannf) wrote :

Noting bug 1916860 here in case it maybe related/duplicate.

Jeff Lane  (bladernr)
tags: added: blocks-hwcert-server
Revision history for this message
Björn Tillenius (bjornt) wrote :

If you can poke around to see what could be wrong, then that would be great.

I guess to start, the output for 'bmc-config -o' and 'ipmitool lan print' would be good to see.

If you have time, it would also be good to reset the BMC, then commission the machine with 'Skip configuring supported BMC controllers...' and provide the output of the commands above. That way we can compare what works, and what doesn't work.

Changed in maas:
status: Incomplete → New
status: New → Incomplete
Revision history for this message
dann frazier (dannf) wrote :

We have 2 identical machines, one that was commissioned with MAAS 2.9.2 and therefore did not hit this problem, and the other that did. Here's a diff of the suggested commands (w/ basic IP setting differences removed):

diff -urpN /tmp/good/bmc-config-o.out /tmp/bad/bmc-config-o.out
--- /tmp/good/bmc-config-o.out 2021-10-01 08:52:20.720961235 -0600
+++ /tmp/bad/bmc-config-o.out 2021-10-01 08:52:33.944949672 -0600
@@ -86,7 +86,7 @@ Section User3
  ## Possible values: 0-17, 0 is unlimited; May be reset to 0 if not specified
  ## Lan_Session_Limit
  ## Possible values: Yes/No
- SOL_Payload_Access Yes
+ SOL_Payload_Access No
 EndSection
 Section User4
  ## Give Username
@@ -399,7 +399,7 @@ Section Rmcpplus_Conf_Privilege
  ## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
  Maximum_Privilege_Cipher_Suite_Id_1 Unused
  ## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
- Maximum_Privilege_Cipher_Suite_Id_2 Administrator
+ Maximum_Privilege_Cipher_Suite_Id_2 Unused
  ## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
  Maximum_Privilege_Cipher_Suite_Id_3 Unused
  ## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
diff -urpN /tmp/good/ipmitool-lan-print.out /tmp/bad/ipmitool-lan-print.out
--- /tmp/good/ipmitool-lan-print.out 2021-10-01 08:56:13.884755091 -0600
+++ /tmp/bad/ipmitool-lan-print.out 2021-10-01 08:56:21.100748638 -0600
@@ -16,7 +16,7 @@ Backup Gateway MAC : 00:00:00:00:00
 802.1q VLAN ID : Disabled
 802.1q VLAN Priority : 0
 RMCP+ Cipher Suites : 1,2,3,6,7,8,11,12,15,16,17
-Cipher Suite Priv Max : XXaXXXXXXXaXXXX
+Cipher Suite Priv Max : XXXXXXXXXXXXXXX
                         : X=Cipher Suite Unused
                         : c=CALLBACK
                         : u=USER

Revision history for this message
Björn Tillenius (bjornt) wrote :

Hmm. So it looks like 3.0 disables all the ciphers for some reason.

Could you arrange so that I can log in to the working machine? It's probably easier, than having you run a bunch of commands.

Changed in maas:
status: Incomplete → New
status: New → Incomplete
Revision history for this message
dann frazier (dannf) wrote : Re: [Bug 1944605] Re: IPMI access lost (not just maas user) after commissioning

On Mon, Nov 8, 2021 at 3:06 AM Björn Tillenius
<email address hidden> wrote:
>
> Hmm. So it looks like 3.0 disables all the ciphers for some reason.
>
> Could you arrange so that I can log in to the working machine? It's
> probably easier, than having you run a bunch of commands.

Sure, will do so off-bug.

  -dann

Revision history for this message
Björn Tillenius (bjornt) wrote :

I did some investigation on one of the affected machines (bizzy), and two other machines. The results are here:

  https://pastebin.ubuntu.com/p/bvBJ6pHrjp/

It seems that all three machines work a bit differently when it comes to bmc-config and ipmitool.

From that, it seems that we can't really the output from bmc-config and ipmitools to decide which ciphers are enabled, unless I'm missing something?

Given that, I'd propose that we shouldn't disable any ciphers on the BMC. Disabling ciphers doesn't add much value anyway, since MAAS will use a secure cipher to communicate anyway.

It probably makes sense to always assume 3 as the cipher, since that's what ipmitool uses by default, and let the admin change it if they want something more secure. Eventually we could not set it at all and let MAAS try out different ciphers the first time it connects to the BMC.

Anyone else has a better idea?

Changed in maas:
status: Incomplete → New
status: New → Incomplete
Revision history for this message
dann frazier (dannf) wrote :

I've no better idea fwiw. You may also want to take a look at https://bugs.launchpad.net/maas/+bug/1916860/comments/25 if you haven't already which I believe is also recommending MAAS stop disabling ciphers.

Christian Grabowski (cgrabowski)
Changed in maas:
status: Incomplete → Fix Committed
assignee: nobody → Christian Grabowski (cgrabowski)
milestone: none → next
Revision history for this message
dann frazier (dannf) wrote :

Are there plans to backport this to 3.1/stable?

Christian Grabowski (cgrabowski)
Changed in maas:
milestone: next → 3.2.0-beta1
Christian Grabowski (cgrabowski)
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.