Schema Missing kernel_id, ramdisk_id causes #1447193

Fix proposed to branch: master
Review: https://review.openstack.org/176379

I used slightly different reproduction steps.

# glance --os-image-api-version 1 image-list
+----+------+-------------+------------------+------+--------+
| ID | Name | Disk Format | Container Format | Size | Status |
+----+------+-------------+------------------+------+--------+
+----+------+-------------+------------------+------+--------+

# glance --os-image-api-version 1 image-create --name test --disk-format raw --container-format bare --property ramdisk_id= --property kernel_id= --file ./junkfile
+-----------------------+--------------------------------------+
| Property | Value |
+-----------------------+--------------------------------------+
| Property 'kernel_id' | None |
| Property 'ramdisk_id' | None |
| checksum | 0632d04c8ec91b3c8c4ea5ae1e38908a |
| container_format | bare |
| created_at | 2015-06-05T20:03:54.817722 |
| deleted | False |
| deleted_at | None |
| disk_format | raw |
| id | 1438711d-2e4d-47f3-a092-d7da541c2451 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | test |
| owner | 963177847ec149c9b84f488140c737cd |
| protected | False |
| size | 5 |
| status | active |
| updated_at | 2015-06-05T20:03:55.411470 |
| virtual_size | None |
+-----------------------+--------------------------------------+

# glance --os-image-api-version 2 image-list
None is not of type u'string'

Failed validating u'type' in schema[u'properties'][u'ramdisk_id']:
    {u'description': u'ID of image stored in Glance that should be used as the ramdisk when booting an AMI-style image.',
     u'is_base': False,
     u'pattern': u'^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$',
     u'type': u'string'}

On instance[u'ramdisk_id']:
    None

The proposed fix https://review.openstack.org/#/c/176379/2 fixes the problem.

All VM snapshot images created by libvirt KVM driver in Nova that did not have ramdisk or kernel_id values will exhibit this behavior and they will fail to boot/spawn/deploy when Nova moves to use Glance v2.

Image using the rbd ephemeral snapshot code, https://review.openstack.org/#/c/125963/, are already exhibiting the failure behavior because this spawn path uses a Glance v2 show call to get image locations.

Status changed to 'Confirmed' because the bug affects multiple users.

I'm going to mention this bug, as I did on #1447193 that I'm using a new glanceclient:
# glance --version
0.17.1

While I'm adding yet another comment I thought I'd drop in the nova stack for how this fails spawn in code that uses clone and image locations.

  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2388, in spawn
    admin_pass=admin_password)
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2786, in _create_image
    instance, size, fallback_from_host)
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5916, in _try_fetch_image_cache
    size=size)
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 231, in cache
    *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 737, in create_image
    prepare_template(target=base, max_size=size, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 221, in fetch_func_sync
    fetch_func(target=target, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2778, in clone_fallback_to_fetch
    backend.clone(context, disk_images['image_id'])
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 762, in clone
    include_locations=True)
  File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 93, in get
    show_deleted=show_deleted)
  File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 301, in show
    _reraise_translated_image_exception(image_id)
  File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 299, in show
    image = self._client.call(context, version, 'get', image_id)
  File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 219, in call
    return getattr(client.images, method)(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 177, in get
    return self.model(**body)
  File "/usr/lib/python2.7/site-packages/warlock/core.py", line 32, in __init__
    base_class.__init__(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/warlock/model.py", line 34, in __init__
    raise ValueError(str(exc))
ValueError: None is not of type u'string'

Failed validating u'type' in schema[u'additionalProperties']:
    {u'type': u'string'}

On instance[u'ramdisk_id']:
    None

This issue may be linked to this https://bugs.launchpad.net/nova/+bug/1467570 . Basically, I have just been confirmed that Nova provisions instances from rbd by querying glanceV2. While it affects the code for ephemeral rbd snapshots, as said earlier, this also affect the code for provisioning instances in Kilo and prevents VMs from booting from snapshot when using a rbd backend.

Hence, a backport of this bug fix for kilo would be strongly appreciated.

This is purely a Glance bug and can be recreated without Nova in the picture at all, therefore I do not believe that the bug fix mentioned in comment 5 will fix this.

See the recreation steps in comment 2. This can be completely abstracted from Nova. The bug is that Glance v1 allows you to set properties with no value, while Glance v2 uses schema validation and validates that those same 2 properties MUST have a string value.

After that point, once you bring Nova into the picture, snapshot images created with libvirt in Nova will have an issue once Nova moves to use Glance v2 for all image access. This is because kernel_id and ramdisk_id can be set to no value on images created in earlier releases using Glance v1 and the Glance image list/ show APIs will fail on those images.

I installed a stable kilo devstack, but still wasn't able to get the image-list reproducer to work:

$ glance --version
0.17.2

$ glance --os-image-api-version 1 image-create --name test --disk-format raw --container-format bare --property ramdisk_id= --property kernel_id= --file /etc/fstab
+-----------------------+--------------------------------------+
| Property | Value |
+-----------------------+--------------------------------------+
| Property 'kernel_id' | None |
| Property 'ramdisk_id' | None |
| checksum | 9cb02fe7fcac26f8a25d6db3109063ae |
| container_format | bare |
| created_at | 2015-07-23T15:55:41.000000 |
| deleted | False |
| deleted_at | None |
| disk_format | raw |
| id | a139ed67-e254-44d7-9499-78b13b498bb4 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | test |
| owner | 3447cea05d6947658d73791ed9e0ed9f |
| protected | False |
| size | 145 |
| status | active |
| updated_at | 2015-07-23T15:55:42.000000 |
| virtual_size | None |
+-----------------------+--------------------------------------+

$ glance --os-image-api-version 2 image-list
+--------------------------------------+---------------------------------+
| ID | Name |
+--------------------------------------+---------------------------------+
| 75fb76bc-cb1c-49b0-a821-704a3525dfcc | cirros-0.3.2-x86_64-uec |
| 9325073e-fc51-494f-b80b-f9de075dc00f | cirros-0.3.2-x86_64-uec-kernel |
| 9cf53d72-8b1b-4f94-9ecc-6c2061c74034 | cirros-0.3.2-x86_64-uec-ramdisk |
| e156d634-3087-4361-a6ad-89fbf544ae2e | cirros-0.3.4-x86_64-uec |
| f0134c5f-1563-4700-8f33-880334a5f834 | cirros-0.3.4-x86_64-uec-kernel |
| d22d2060-3b6c-44fa-823b-e7d108b43d9d | cirros-0.3.4-x86_64-uec-ramdisk |
| a139ed67-e254-44d7-9499-78b13b498bb4 | test |
+--------------------------------------+---------------------------------+

I'm struggling to understand why I'm not able to reproduce using image-list. I do suspect that the image-list issue has gone away with an updated client, but can't be 100% sure.

The nova stack trace in comment #4 is from an image-show rather than an image list.

I am able to reproduce using image-show (as suggested by Mike Fedosin) ie

$ glance --os-image-api-version 1 image-create --name test --disk-format raw --container-format bare --property ramdisk_id= --property kernel_id=

$ glance --os-image-api-version 2 image-show b51e04cc...

Reviewed: https://review.openstack.org/176379
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=82c63b1df4399db22dfd71e3326a6bfc1b372a9c
Submitter: Jenkins
Branch: master

commit 82c63b1df4399db22dfd71e3326a6bfc1b372a9c
Author: Jorge Niedbalski <email address hidden>
Date: Wed Apr 22 12:54:29 2015 -0300

    Allow ramdisk_id, kernel_id to be null on schema

    Since is possible to create images with no
    ramdisk_id, kernel_id parameters on glance v1,
    then the schema-image.json needs to be modified to
    allow null, string values for both attributes
    and prevent LP #1447193 to happen with a glance v2
    client.

    APIImpact

    Closes-Bug: #1447215
    Related-Bug: #1447193

    Change-Id: I645c797553ccaa84062f9ad28f2349ae738a0c96
    Signed-off-by: Jorge Niedbalski <email address hidden>

Reviewed: https://review.openstack.org/205432
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=4b269775964f75300d3af56a4c57234d54a32e94
Submitter: Jenkins
Branch: stable/kilo

commit 4b269775964f75300d3af56a4c57234d54a32e94
Author: Jorge Niedbalski <email address hidden>
Date: Wed Apr 22 12:54:29 2015 -0300

    Allow ramdisk_id, kernel_id to be null on schema

    Since is possible to create images with no
    ramdisk_id, kernel_id parameters on glance v1,
    then the schema-image.json needs to be modified to
    allow null, string values for both attributes
    and prevent LP #1447193 to happen with a glance v2
    client.

    APIImpact

    Closes-Bug: #1447215
    Related-Bug: #1447193

    Change-Id: I645c797553ccaa84062f9ad28f2349ae738a0c96
    Signed-off-by: Jorge Niedbalski <email address hidden>
    (cherry picked from commit 82c63b1df4399db22dfd71e3326a6bfc1b372a9c)

The attachment "Vivid Patch" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

This bug was fixed in the package glance - 1:2015.1.1-0ubuntu2

---------------
glance (1:2015.1.1-0ubuntu2) vivid; urgency=medium

  * Additional support for stable/kilo (LP: #1481008):
    - d/p/fix-test-artifacts-plugin-loader.patch: Added to fix failing tests.
    - d/p/skip-glance-search-tests.patch: Added to fix failing tests.
    - d/p/skip-online-tests.patch: Added to skip tests that require online access
      during Ubuntu builds.

 -- Corey Bryant <email address hidden> Wed, 05 Aug 2015 17:53:11 -0400