CentOS 6 deployment fails to deploy with Bionic

Bug #1775424 reported by Anthony Yarbrough
30
This bug affects 5 people
Affects Status Importance Assigned to Milestone
MAAS
Triaged
Medium
Unassigned

Bug Description

* Operating system: Ubuntu 18.04
* MAAS version: 2.4.0 (6981-g011e51b7a-0ubuntu1~18.04.1)

root@maas1:~# ls -lah /var/log/maas/
total 16M
drwxr-xr-x 4 maas maas 4.0K May 23 06:25 .
drwxrwxr-x 18 root syslog 4.0K Jun 6 06:25 ..
-rw-r--r-- 1 syslog syslog 932K Jun 6 15:41 maas.log
drwxr-x--- 2 proxy proxy 4.0K Jun 6 06:25 proxy
-rw-r--r-- 1 maas maas 5.0M Jun 6 15:42 rackd.log
-rw-r--r-- 1 maas maas 9.3M Jun 6 15:49 regiond.log
-rw-r--r-- 1 maas maas 391K May 23 06:24 regiond.log.1.gz
drwxr-xr-x 6 syslog syslog 4.0K May 31 13:56 rsyslog

# dpkg -l '*maas*'|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===============================-======================================-============-=================================================
ii maas 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all "Metal as a Service" is a physical cloud and IPAM
ii maas-cli 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS client and command-line interface
un maas-cluster-controller <none> <none> (no description available)
ii maas-common 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS server common files
ii maas-dhcp 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS DHCP server
un maas-dns <none> <none> (no description available)
ii maas-proxy 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS Caching Proxy
ii maas-rack-controller 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all Rack Controller for MAAS
ii maas-region-api 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all Region controller API service for MAAS
ii maas-region-controller 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all Region Controller for MAAS
un maas-region-controller-min <none> <none> (no description available)
un python-django-maas <none> <none> (no description available)
un python-maas-client <none> <none> (no description available)
un python-maas-provisioningserver <none> <none> (no description available)
ii python3-django-maas 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS server Django web framework (Python 3)
ii python3-maas-client 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS python API client (Python 3)
ii python3-maas-provisioningserver 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS server provisioning libraries (Python 3)

I'm running into an issue related to CentOS 6 deployments failing with multiple MAAS installations using different network configurations (one simple and one complex) for the machines being deployed. I thought it might have been related to the complex network config however I've confirmed the same problem exists with what I would consider a simple network config. Below is the error output found in the installation log once the CentOS 6 deployment fails:

----------------------------------------------------------------------------------

Traceback (most recent call last):

  File "/tmp/tmpzkdjc8hq/target/curtin/curtin-hooks.py", line 393, in <module>

    main()

  File "/tmp/tmpzkdjc8hq/target/curtin/curtin-hooks.py", line 376, in main

    grub_root = get_grub_root(target)

  File "/tmp/tmpzkdjc8hq/target/curtin/curtin-hooks.py", line 208, in get_grub_root

    return regex.groups()[0]

AttributeError: 'NoneType' object has no attribute 'groups'

Unexpected error while running command.

Command: ['/tmp/tmpzkdjc8hq/target/curtin/curtin-hooks']

Exit code: 1

Reason: -

Stdout: ''

Stderr: ''

curtin: Installation failed with exception: Unexpected error while running command.

Command: ['curtin', 'curthooks']

Exit code: 3

Reason: -

Stdout: Traceback (most recent call last):

          File "/tmp/tmpzkdjc8hq/target/curtin/curtin-hooks.py", line 393, in <module>

            main()

          File "/tmp/tmpzkdjc8hq/target/curtin/curtin-hooks.py", line 376, in main

            grub_root = get_grub_root(target)

          File "/tmp/tmpzkdjc8hq/target/curtin/curtin-hooks.py", line 208, in get_grub_root

            return regex.groups()[0]

        AttributeError: 'NoneType' object has no attribute 'groups'

        Unexpected error while running command.

        Command: ['/tmp/tmpzkdjc8hq/target/curtin/curtin-hooks']

        Exit code: 1

        Reason: -

        Stdout: ''

        Stderr: ''

Stderr: ''

----------------------------------------------------------------------------------

Here are the steps to recreate the problem:

1.) Enlist the server into MAAS and rename it to the appropriate server name
2.) Mark the machine as broken and configure the network interfaces as follows:
    - eth0 - private subnet used for MAAS deployments
    - eth1 - public /29 subnet dedicated to specific server (gateway to internet here)
3.) Mark the machine as fixed and then commission the machine ensuring to select the commissioning option to retain the current network config
4.) Once the server has been successfully commissioned, acquire the server (admin user) and deploy with CentOS 6.
5.) The server indicates the installation has begun and CentOS 6 cloud installer image is downloaded for install/extraction however the error listed above is produced immediately after the image download then the status changes to 'Failed Deployment'

I have no problems successfully installing via MAAS operating systems Ubuntu 14.04, Ubuntu 16.04, Ubuntu 18.04 and CentOS 7 using the same exact steps listed above.

If you need any additional information or have any questions please let me know and I'll be happy to oblige.

Thanks!

Related branches

Revision history for this message
Andres Rodriguez (andreserl) wrote :

Hi there,

I've deployed centos6 and i cannot reproduce your issue https://pastebin.ubuntu.com/p/t7H9hbQMKZ/

Could you confirm a few things:

1. Which image have you used?
2. Is there any customization done, or just straight forward deployment?

Changed in maas:
status: New → Incomplete
Revision history for this message
Anthony Yarbrough (virento) wrote :

I am using the stock image provided by maas.io and no customizations have been done, everything was left at default.

Revision history for this message
Andres Rodriguez (andreserl) wrote :

Can you provide the output of your machine:

maas 'user' machine read <system-id>

and:

maas 'user' machine get-curtin-config <system_id>

Revision history for this message
Lee Trager (ltrager) wrote :

I was able to reproduce this when I had my commissioning operating system set to Bionic, could you please try 16.04 Xenial?

summary: - CentOS 6 deployment fails with AttributeError: 'NoneType' object has no
- attribute 'groups'
+ CentOS 6 deployment fails to deploy with Bionic
Changed in maas:
status: Incomplete → Triaged
Revision history for this message
Daniel Souza (danielsouzasp) wrote :

Hello,

I'm facing this same error with MaaS 2.3.5 to deploy CentOS 6, any other OS works fine.

The node has 1x NVMe 1 TB + 2x SSD 500 GB, I'm trying this config:
Name |Model | Serial|BootDevice typeUsed forHealth
nvme0n1 |Physical | MBR |partitioned with 1 partition Ok
nvme0n1-part1 |Partition| ext4 |formatted filesystem mounted at /

Error:
Traceback (most recent call last):
  File "/tmp/tmph5qpnwg3/target/curtin/curtin-hooks.py", line 393, in <module>
    main()
  File "/tmp/tmph5qpnwg3/target/curtin/curtin-hooks.py", line 376, in main
    grub_root = get_grub_root(target)
  File "/tmp/tmph5qpnwg3/target/curtin/curtin-hooks.py", line 208, in get_grub_root
    return regex.groups()[0]
AttributeError: 'NoneType' object has no attribute 'groups'
Unexpected error while running command.
Command: ['/tmp/tmph5qpnwg3/target/curtin/curtin-hooks']
-

If I ran "maas myadmin machine get-curtin-config" there is no custom storage config.

Does someone know it is known bug?

Revision history for this message
Andres Rodriguez (andreserl) wrote :

Hi Daniel,

Could you please confirm where do these images come from ? Note that MAAS 2.3.5 doesn't support storage configuration for CentOS, this support is only available in 2.5+.

Also, please make sure you have xenial as the commissioning image.

Changed in maas:
status: Triaged → Invalid
status: Invalid → Incomplete
Revision history for this message
Daniel Souza (danielsouzasp) wrote :

Hi Andres,

I'm using default MaaS repository and I also set xenial as commissioning image, so I'll think of update to 2.5.

Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for MAAS because there has been no activity for 60 days.]

Changed in maas:
status: Incomplete → Expired
Revision history for this message
Fabio Augusto Miranda Martins (fabio.martins) wrote :

I can also reproduce this problem with MAAS 2.4.2 + Ubuntu 18.04.2 + kernel 4.15.0-45. I'm using the CentOS 6 image synced from maas.io through the regular MAAS Web GUI (no curtin customizations):

https://pastebin.ubuntu.com/p/n235yRgvdk/

Revision history for this message
Fabio Augusto Miranda Martins (fabio.martins) wrote :

Log for the failure

Changed in maas:
status: Expired → Confirmed
Revision history for this message
Ryan Harper (raharper) wrote :

The source of the issue is around the ext4 filesystem and grub 0.97[1]. In Bionic, ext4 has 64bit and metadata_csum features which grub 0.97 in Centos6 cannot handle read. There are couple of workarounds here:

A) Use Xenial to deploy Centos6
B) If you use bionic, then you'll need custom storage where you create a /boot partition smaller than 512Mb and put either ext2 or ext3 as the filesystem format.

1. https://bugzilla.redhat.com/show_bug.cgi?id=1099237

Revision history for this message
Ryan Harper (raharper) wrote :

Scratch separate /boot, the curtin-hooks in centos6 do not support separate /boot. So, switching the filesystem format to ext3 should work fine.

Revision history for this message
Stefan Miktov Ivanov (tension183) wrote :

I have this issue only wen i use software RAID here it is the messeges.

Revision history for this message
Björn Tillenius (bjornt) wrote :

So, as I understand it, the workaround is to set Xenial to be the commissioning environment, but we need to make it so that Xenial is used when deploying Centos6.

Changed in maas:
status: Confirmed → Triaged
importance: Undecided → Medium
Revision history for this message
Jose Delarosa (jdelaros1) wrote :

I am seeing the exact same problem, even when commissioning with Xenial (both ga and hwe kernels). Was there ever a more permanent solution?

Revision history for this message
Adam Collard (adam-collard) wrote (last edit ):

We plan to address this as a feature that explicitly models the dependencies between target deployment image and the ephemeral image that MAAS uses to run curtin in (and install that image).

This will, for example, give MAAS the knowledge that CentOS 6 requires Xenial, CentOS 7 requires Focal etc.

Internally, see PF-3237

Changed in maas:
milestone: none → 3.4.0
Alberto Donato (ack)
Changed in maas:
milestone: 3.4.0 → 3.4.x
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.