MAAS 3.1 no longer supports arbitrary custom images

Bug #1960278 reported by Vern Hart
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
High
Christian Grabowski

Bug Description

I have a customer who has used a custom image on MAAS 3.0 that can no longer be deployed on 3.1.

The specific image is based on a vyos-equuleus iso which in turn is based on debian/buster. It took some curtin preseed magic to get the image to deploy properly but it worked.

After upgrading to 3.1, attempts to deploy with the custom image uploaded with 3.0 were met with this error:

    not enough values to unpack (expected 2, got 1)

We deleted the image and tried to re-upload it:

    $ maas root boot-resources create name=custom/vyos-equuleus title="Vyos Equuleus 20200220z" architecture=amd64/generic content@=$HOME/vyos-equuleus-20200220z-maas-amd64.tgz
    {"base_image": ["custom images require a valid base image name"]}

We tried adding a base_image parameter (debian/buster as this is what it is based on) and this provided a new error.

    $ maas root boot-resources create name=custom/vyos-equuleus title="Vyos Equuleus 20200220z" architecture=amd64/generic content@=$HOME/vyos-equuleus-20200220z-maas-amd64.tgz base_image=debian/buster
    {"base_image": ["custom images require a valid non-custom OS type base image"]}

In an attempt to "trick" maas into accepting the image, we tried ubuntu/focal as the base_image. The upload was successful but when trying to deploy the new image it failed "'curtin' executing late commands". Also in the logs:

    Marking node failed - Missing boot image custom/amd64/ga-20.04/vyos-equuleus.

Related branches

Revision history for this message
Björn Tillenius (bjornt) wrote :

Would it be possible to provide the image (or a similar image that has the same problems), together with the curtin preseed changes you did?

Changed in maas:
status: New → Incomplete
Revision history for this message
Vern Hart (vern) wrote :
Download full text (3.4 KiB)

I started with a live iso provided by the customer called vyos-equuleus-20200220z-amd64.iso. Here are my notes for converting this to a maas deployable image:

  # mount the iso
  mkdir /tmp/vyos
  sudo mount -o loop vyos-equuleus-20200220z-amd64.iso /tmp/vyos
  # extract the squashfs
  sudo unsquashfs -d /tmp/vyos-root /tmp/vyos/live/filesystem.squashfs
  # enter vyos-root
  sudo LC_ALL=C chroot /tmp/vyos-root
  # install required packages and enable fastboot
  apt update
  apt install cloud-init -y
  sed -i 's/^\(GRUB_CMDLINE_LINUX="\?\)/\1fastboot /' /etc/default/grub
  # start ssh
  cat <<EOF> /etc/cloud/cloud.cfg.d/99-ssh.cfg
  runcmd:
    - systemctl enable ssh
    - systemctl start ssh
  EOF
  # make sure the interfaces.d directory is supported
  grep ^source /etc/network/interfaces ||
    echo "source /etc/network/interfaces.d/*" >> /etc/network/interfaces
  # create a tar
  cd /tmp/vyos-root/
  sudo tar -C /tmp/vyos-root -czf ~/vyos-equuleus-20200220z-maas-amd64.tgz *
  sudo chown ubuntu.ubuntu ~/vyos-equuleus-20200220z-maas-amd64.tgz
  cd -
  # upload to maas
  maas root boot-resources create name=custom/vyos-equuleus title="Vyos Equuleus 20200220z" architecture=amd64/generic content@=$HOME/vyos-equuleus-20200220z-maas-amd64.tgz

The resulting 378M image is here: https://drive.google.com/file/d/1yjXrvkUHIlkN8_Rv3PT_6KbiaHe6VcyK/view

After adding a curtin_userdata_custom file I was able to boot the image on a kvm machine (though ssh keys are not properly installed -- default user pw is vyos vyos). The last of my notes:

# create the following file in /var/snap/maas/current/preseeds/curtin_userdata_custom
# and copy it to all three maas servers
# (including the cloud-config line that looks like a comment)

#cloud-config
kernel:
  fallback-package: linux-image-5.4.99-amd64-vyos
  package: linux-image-5.4.99-amd64-vyos
  # default debian buster kernel is 4.19.0
  #fallback-package: linux-image-amd64
  #package: linux-image-amd64

apt:
  preserve_sources_list: false
  sources_list: | # written by curtin custom preseed
    deb http://deb.debian.org/debian buster main contrib non-free
    deb-src http://deb.debian.org/debian buster main contrib non-free
    deb http://deb.debian.org/debian-security buster/updates main contrib non-free
    deb-src http://deb.debian.org/debian-security buster/updates main contrib non-free
    deb http://deb.debian.org/debian buster-updates main contrib non-free
    deb-src http://deb.debian.org/debian buster-updates main contrib non-free
    deb http://deb.debian.org/debian buster-backports main contrib non-free
    deb-src http://deb.debian.org/debian buster-backports main contrib non-free

early_commands:
  000_update_repositories: apt update
  010_install_apt_https: apt install -y apt-transport-https ca-certificates efibootmgr xfsprogs

debconf_selections:
 maas: |
  {{for line in str(curtin_preseed).splitlines()}}
  {{line}}
  {{endfor}}

late_commands:
  maas: [wget, '--no-proxy', '{{node_disable_pxe_url}}', '--post-data', '{{node_disable_pxe_data}}', '-O', '/dev/null']
  00_target: mount --bind $TARGET_MOUNT_POINT /mnt
  90_datasource: host=$(grep metadata_url /etc/cloud/cloud.cfg.d/91_kernel_cmdline_...

Read more...

Vern Hart (vern)
Changed in maas:
status: Incomplete → New
Bill Wear (billwear)
Changed in maas:
status: New → Triaged
importance: Undecided → High
Changed in maas:
assignee: nobody → Christian Grabowski (cgrabowski)
milestone: none → next
status: Triaged → In Progress
Changed in maas:
status: In Progress → Fix Committed
Changed in maas:
milestone: next → 3.2.0-beta2
status: Fix Committed → Fix Released
Revision history for this message
Srdjan Macak (srdjanmack5) wrote :

Hello team,

Is this released to 3.2.0 ?

I'm having the same issues with 3.2.0-11989-g.84a255c14,

could you please double check ?

Thank you.

Revision history for this message
Semyon Kolesnikov (lotarcsombra) wrote :

Hello team,

I'm too having this issue with 3.2.0-11989-g.84a255c14,

Please, help to resolve it.

Revision history for this message
Deepak Sharma (sadysnaat) wrote :

We are also facing the same issue

maas version 2.9
where we have custom image
https://maasgoldenimage.s3.amazonaws.com/u-1804-0-k-12014-0.tar.gz

imported

we are able to deploy machine using the same

upgrade maas to 3.1 and we start seeing the following error

root@maas:/var/snap/maas/common/log/http# maas sc_eng machine deploy bba763 osystem=custom distro_series=u-1804-0-k-12014-0 -d
500 Internal Server Error

   Content-Length: 47
     Content-Type: text/plain; charset=utf-8
             Date: Thu, 14 Jul 2022 12:04:49 GMT
           Server: TwistedWeb/18.9.0
           Status: 500
             Vary: Cookie
  X-Frame-Options: SAMEORIGIN

not enough values to unpack (expected 2, got 1)
root@maas:/var/snap/maas/common/log/http#

Revision history for this message
Christian Grabowski (cgrabowski) wrote :

Those who are still seeing this in 3.2, are you seeing this on upload of an arbitrary custom image? If so, what command are you using to upload? Or is this occurring on an existing image uploaded from a previous maas version?

Revision history for this message
Semyon Kolesnikov (lotarcsombra) wrote :

I'm using this command maas $PROFILE boot-resources create name="custom/focal-cno-0.2" title='Ubuntu 20.04 Focal CNO 0.3' architechture=amd64/generic filetype='tgz' content@=ubuntu-cno.tar.gz @cgrabowski

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.