no way to tell when boot-source info has synced and boot-source-selections can be made

Bug #1806763 reported by Jason Hobbs
18
This bug affects 2 people
Affects Status Importance Assigned to Milestone
MAAS
Triaged
High
Unassigned

Bug Description

After adding a boot source, there is nothing to indicate that the boot source's information has been synced into maas's db and maas is ready to allow boot source selections to be made.

In this log, I try to make a boot source selection immediately after adding a boot source, and it fails. If I way another 20 seconds and try again, it works. But, there is nothing to read from the API that says "ok, you can make a boot source selection." So, I just have to add a 'sleep', which is racy.

ubuntu@staging-cpe-84e78778-1c0d-4f7b-b73b-3933b6b23033:~/cpe/foundation$ maas root boot-sources create url=http://images.maas.io/ephemeral-v3/daily/ keyring_filename=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg; maas root boot-source-selections create 4 os="ubuntu" release="xenial" arches="amd64" subarches="*" labels="*"
Success.
Machine-readable output follows:
{
    "created": "2018-12-04T21:05:37.160",
    "url": "http://images.maas.io/ephemeral-v3/daily/",
    "keyring_data": "",
    "resource_uri": "/MAAS/api/2.0/boot-sources/4/",
    "updated": "2018-12-04T21:05:37.160",
    "keyring_filename": "/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg",
    "id": 4
}
{"os": ["OS ubuntu with release xenial has no available images for download"]}
ubuntu@staging-cpe-84e78778-1c0d-4f7b-b73b-3933b6b23033:~/cpe/foundation$ maas root boot-source-selections create 4 os="ubuntu" release="xenial" arches="amd64" subarches="*" labels="*"
Success.
Machine-readable output follows:
{
    "release": "xenial",
    "labels": [
        "*"
    ],
    "arches": [
        "amd64"
    ],
    "id": 4,
    "boot_source_id": 4,
    "os": "ubuntu",
    "resource_uri": "/MAAS/api/2.0/boot-sources/4/selections/4/",
    "subarches": [
        "*"
    ]
}

This is with maas 2.3.5.

description: updated
Revision history for this message
Jason Hobbs (jason-hobbs) wrote :

subscribed to field-high as this has been hit in the field.

tags: added: cpe-onsite
Revision history for this message
Lee Trager (ltrager) wrote :

When a new boot-source has been added the sources list is updated in a post-commit hook. Because of this there currently isn't a way to check if the boot-sources list is being updated over the API or UI. There are a couple ways we may be able to fix this I'll have to discuss with the team what the best option is.

Revision history for this message
Andres Rodriguez (andreserl) wrote : Re: [Bug 1806763] Re: no way to tell when boot-source info has synced and boot-source-selections can be made

This seems to fall under completely new functionality given that MAAS has
never been able to provide status about accessing/processing the sources.

The error seems accurate though because MAAS has not finished processing
sources and doesn’t have such image in its knowledge.

On Mon, Mar 25, 2019 at 10:50 PM Lee Trager <email address hidden>
wrote:

> When a new boot-source has been added the sources list is updated in a
> post-commit hook. Because of this there currently isn't a way to check
> if the boot-sources list is being updated over the API or UI. There are
> a couple ways we may be able to fix this I'll have to discuss with the
> team what the best option is.
>
> --
> You received this bug notification because you are subscribed to MAAS.
> https://bugs.launchpad.net/bugs/1806763
>
> Title:
> no way to tell when boot-source info has synced and boot-source-
> selections can be made
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1806763/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: product=maas; status=New; importance=Undecided;
> assignee=None;
> Launchpad-Bug-Tags: cdo-qa cpe-onsite foundations-engine
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: jason-hobbs ltrager
> Launchpad-Bug-Reporter: Jason Hobbs (jason-hobbs)
> Launchpad-Bug-Modifier: Lee Trager (ltrager)
> Launchpad-Message-Rationale: Subscriber (MAAS)
> Launchpad-Message-For: andreserl
>
--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer

Revision history for this message
Jason Hobbs (jason-hobbs) wrote :

It is a design bug. Adding boot sources and making selections isn't new functionality, and right now, due to an incomplete API, is inherently racy.

The problem is we don't have a way to add boot sources and make boot source selections in a non-racy manner, because the API for this is incomplete and doesn't include a way to tell when boot sources have been synced.

This leads to failures in the field, where we add a boot source, try to make boot source selections, and get errors. This has occurred in multiple field deployments, and we do not have a workaround for it.

We've tried using sleep() between adding boot source and making selections, but that is inherently racy and we haven't been able to figure out any specific amount of time that guarantees the boot sources are available to make selections from.

Do you have some other method in mind of solving this problem that doesn't involve an API change? We need to be able to tell when boot sources are ready and that no error or timeout has occurred that will prevent the boot source from becoming ready.

Changed in maas:
status: New → Incomplete
status: Incomplete → Opinion
status: Opinion → Invalid
status: Invalid → Triaged
importance: Undecided → Medium
Changed in maas:
importance: Medium → High
milestone: none → 3.4.0
Revision history for this message
Jerzy Husakowski (jhusakowski) wrote :

Image storage will change in MAAS 3.4 and this issue needs to be reevaluated after that change.

Changed in maas:
milestone: 3.4.0 → 3.5.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.