MAAS nodes that fail commissioning continue to use stale commissioning scripts

Bug #2083076 reported by slines
26
This bug affects 2 people
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
High
Unassigned
3.5
Invalid
High
Unassigned
3.6
Fix Released
High
Unassigned
3.7
Fix Released
High
Unassigned

Bug Description

Version: maas/jammy,now 1:3.5.1-16317-g.409891638-0ubuntu1~22.04.1

Interface: UI

What happened: Nodes that are in the "Failed commissioning" state will continue to use old/cached commissioning scripts and ignore modifications to those scripts made through the UI. If you delete the failed node, re-enlist the node, and then commission the node, it will pick up the new version of the commissioning script.

Steps to reproduce: Upload a custom commissioning script like 42-my-custom-comissioning-script.py which contains a syntax error. Commission a new node and wait until it enters the failed to commission state when it executes the custom script. Deleted the 42-my-custom-comissioning-script.py from maas, correct the sytax error in 42-my-custom-comissioning-script.py, and re-uploaded it. Then do action->Commission on the node with failed commissioning, and it will re-execute the old script with the syntax error and not the new one that you uploaded. Delete the failed node, re-enlist the node, and then commission the node, and it will pick up the new version of the commissioning script.

Revision history for this message
slines (slines) wrote :
Revision history for this message
Nick De Villiers (nickdv99) wrote :

Hey slines, thanks for the report!

I tried reproducing the issue with the steps you gave, but MAAS would always use the new copy of the script for me - could you provide me with the steps/commands you used to build and set up your MAAS environment?

Changed in maas:
status: New → Incomplete
Revision history for this message
Andrew Lamzed-Short (andyls) wrote :

slines, does this bug still occur for you? If so, please provide us with the build steps you used to setup and configure your MAAS environment so that we may reproduce the issue and help get it resolved for you.

Changed in maas:
status: Incomplete → New
status: New → Incomplete
Revision history for this message
Jacopo Rota (r00ta) wrote :

Multiple people are facing this https://bugs.launchpad.net/maas/+bug/2091554 but we still need a reproducer

Changed in maas:
status: Incomplete → Triaged
importance: Undecided → High
milestone: none → 3.6.x
Revision history for this message
Anton Troyanov (troyanov) wrote :

Here is another bug reported with a similar symptom https://bugs.launchpad.net/maas/+bug/2101225

Revision history for this message
Dmytro Kazantsev (dkazants) wrote :
Download full text (3.5 KiB)

Similar issue happens when I try to commission IBM Mainframe s390x LPARs. I get HTTP 500 error and in the log I see:

Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: maasserver: [error] ################################ Exception: Status for scriptresult 17448 is not running or pending (9) ################################
Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: maasserver: [error] Traceback (most recent call last):
Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: File "/snap/maas/39401/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 181, in get_response
Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: response = wrapped_callback(request, callback_args, **callback_kwargs)
Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: File "/snap/maas/39401/lib/python3.10/site-packages/maasserver/utils/views.py", line 298, in view_atomic_with_post_commit_savepoint
Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: return view_atomic(args, kwargs)
Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: File "/usr/lib/python3.10/contextlib.py", line 79, in inner
Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: return func(*args, kwds)
Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: File "/snap/maas/39401/lib/python3.10/site-packages/maasserver/api/support.py", line 62, in _call
Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: response = super().call(request, args, **kwargs)
Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: File "/snap/maas/39401/usr/lib/python3/dist-packages/django/views/decorators/vary.py", line 20, in inner_func
Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: response = func(args, kwargs)
Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: File "/snap/maas/39401/usr/lib/python3/dist-packages/piston3/resource.py", line 196, in call
Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: result = self.error_handler(e, request, meth, em_format)
Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: File "/snap/maas/39401/usr/lib/python3/dist-packages/piston3/resource.py", line 194, in call
Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: result = meth(request, *args, kwargs)
Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: File "/snap/maas/39401/lib/python3.10/site-packages/maasserver/api/support.py", line 371, in dispatch
Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: return function(self, request, args, *kwargs)
Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: File "/snap/maas/39401/lib/python3.10/site-packages/metadataserver/api.py", line 880, in signal
Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: target_status = process(node, request, status)
Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: File "/snap/maas/39401/lib/python3.10/site-packages/metadataserver/api.py", line 683, in _process_commissioning
Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: self._store_results(
Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: File "/snap/maas/39401/lib/python3.10/site-packages/metadataserver/api.py", line 565, in _store_results
Jun 07 08:34:29 juju-83a692-0 maas-regiond[1833721]: scri...

Read more...

tags: added: bug-council
tags: removed: bug-council
Revision history for this message
Stamatis Katsaounis (skatsaounis) wrote :

The behavior that is present in Dmytro's logs is not related with scripts but script results. In addition, it is not related to this bug. Instead, it is a symptom of this bug: https://bugs.launchpad.net/maas/+bug/2112637

Revision history for this message
slines (slines) wrote :

I no longer see this behavior in v3.6.1. You may close the bug. Thanks!

Revision history for this message
Thorsten Merten (thorsten-merten) wrote :

Changing to fix released for mainline due to above comment. Thanks for the update.

Changed in maas:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.