OpenStack Compute (nova)

Encryption key is not properly formatted before being passed to dmcrypt

  1. Series ocata
  2. Bug #1688342
Bug #1688342 reported by Jackie Truong
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Jackie Truong
Newton
Fix Committed
Medium
Lee Yarwood
Ocata
Fix Committed
Medium
Lee Yarwood

Bug Description

Description
===========
A TypeError occurs when using Nova to boot an instance with ephemeral storage encryption enabled.

When key management was moved from Nova to Castellan in the Newton release, the key retrieval return value was changed from being formatted as a list of unsigned ints (in the case of an octet stream) [1] to not being formatted at all [2]. Nova's dmcrypt still expects the retrieved key [3][4] to be formatted as an array of unsigned bytes [5].

References:
[1] https://github.com/openstack/nova/blob/9702e5d2e9433c67067847c02fb100cd808a3596/nova/keymgr/barbican.py#L297-L323
[2] https://github.com/openstack/castellan/blob/8d3f1cf375d28047cfd5f251afbb3f6d3ea7cda5/castellan/key_manager/barbican_key_manager.py#L527-L549
[3] https://github.com/openstack/nova/blob/d219a3dcdc9c51bff3ebf2df086ea61a840ea3e9/nova/virt/libvirt/imagebackend.py#L714-L715
[4] https://github.com/openstack/nova/blob/d219a3dcdc9c51bff3ebf2df086ea61a840ea3e9/nova/virt/libvirt/imagebackend.py#L674-L680
[5] https://github.com/openstack/nova/blob/d219a3dcdc9c51bff3ebf2df086ea61a840ea3e9/nova/virt/libvirt/storage/dmcrypt.py#L48-L70

Steps to reproduce
==================
1. Set up an LVM device:
Create a backing file:
  $ truncate nova-lvm -s 2G

Mount the backing file on a loop device:
  $ sudo losetup /dev/loop1 nova-lvm

Prepare the device for LVM:
  $ sudo pvcreate /dev/loop1

Create the LVM group on the loop device:
  $ sudo vgcreate nova-lvm /dev/loop1

2. Set up a devstack environment with ephemeral storage encryption enabled by adding the following lines to `lib/nova`:
  iniset $NOVA_CONF ephemeral_storage_encryption enabled "True"
  iniset $NOVA_CONF ephemeral_storage_encryption cipher "aes-xts-plain64"
  iniset $NOVA_CONF ephemeral_storage_encryption key_size "256"
  iniset $NOVA_CONF libvirt images_type "lvm"
  iniset $NOVA_CONF libvirt images_volume_group "nova-lvm"

3. Stack:
  $ ./stack

4. Use Nova to boot an instance:
  $ nova boot --flavor 1 --image {image_id}

Expected result
===============
Ephemeral storage encryption succeeds and Nova successfully boots the instance.

Actual result
=============
Ephemeral storage encryption fails with a TypeError (similar results can be seen from Barbican Tempest gate failures [1]):

2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] Traceback (most recent call last):
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/compute/manager.py", line 2122, in _build_resources
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] yield resources
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/compute/manager.py", line 1927, in _build_and_run_instance
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] block_device_info=block_device_info)
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2688, in spawn
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] block_device_info=block_device_info)
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3095, in _create_image
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] fallback_from_host)
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3211, in _create_and_inject_local_root
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] instance, size, fallback_from_host)
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6780, in _try_fetch_image_cache
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] size=size)
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 227, in cache
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] *args, **kwargs)
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 735, in create_image
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] create_lvm_image(base, size)
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] return f(*args, **kwargs)
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 693, in create_lvm_image
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] encrypt_lvm_image()
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 680, in encrypt_lvm_image
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] key)
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/storage/dmcrypt.py", line 64, in create_volume
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] key = ''.join(map(lambda byte: "%02x" % byte, key))
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/storage/dmcrypt.py", line 64, in <lambda>
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] key = ''.join(map(lambda byte: "%02x" % byte, key))
2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] TypeError: %x format: a number is required, not str

Nova fails to boot the instance.

[1] http://logs.openstack.org/59/455459/16/check/gate-barbican-simple-crypto-dsvm-tempest-ubuntu-xenial-nv/9c0fa48/logs/screen-n-cpu.txt.gz#_2017-04-24_21_01_37_841

Environment
===========

Latest `master` branch for all projects, except for Nova, which has patch [1] applied to get around the error addressed by the patch. However, remnants of the TypeError can still be seen without this patch [2].

[1] https://review.openstack.org/#/c/462348/
[2] http://logs.openstack.org/59/455459/17/check/gate-barbican-simple-crypto-dsvm-tempest-ubuntu-xenial-nv/26e8659/logs/screen-n-cpu.txt.gz#_2017-04-30_16_16_43_532

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/462674

Changed in nova:
assignee: nobody → Jackie Truong (jackie-truong)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/462674
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=53a71c1241aac70018c16d174032427a172378ed
Submitter: Jenkins
Branch: master

commit 53a71c1241aac70018c16d174032427a172378ed
Author: Jackie Truong <email address hidden>
Date: Thu May 4 12:51:22 2017 -0400

    Fix decoding of encryption key passed to dmcrypt

    This patch fixes the decoding of the encryption key passed to dmcrypt.
    During the key management move from Nova to Castellan, in the Newton
    release, conversion of the encryption key (from a string to list of
    unsigned ints) was removed from the key retrieval method. This patch
    updates dmcrypt to decode an encryption key string, rather than a list
    of unsigned ints. See the linked bug for more information.

    The method used to decode the encryption key has been updated to use
    binascii, as done in os-brick [1], to maintain consistency. The key
    generation and decoding portions of test_dmcrypt have been updated to
    reflect this change and ensure compatibility with both, Python 2 and
    Python 3.

    [1] https://github.com/openstack/os-brick/blob/6cf9b1cd689f70a2c50c0fa83a9a9f7c502712a1/os_brick/encryptors/cryptsetup.py#L100-L102

    Depends-On: I5fe3e5d5e5a9694d0dbe5b59248e5eaf89858c62
    Closes-Bug: #1688342
    Change-Id: I050585ecb55742a972038cf72b0650321ded2856

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/466480

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/466481

Matt Riedemann (mriedem)
Changed in nova:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/ocata)

Reviewed: https://review.openstack.org/466480
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=bc8d0723e6e43d9b95ff9a5c4cbf7b0bec503a2f
Submitter: Jenkins
Branch: stable/ocata

commit bc8d0723e6e43d9b95ff9a5c4cbf7b0bec503a2f
Author: Jackie Truong <email address hidden>
Date: Thu May 4 12:51:22 2017 -0400

    Fix decoding of encryption key passed to dmcrypt

    This patch fixes the decoding of the encryption key passed to dmcrypt.
    During the key management move from Nova to Castellan, in the Newton
    release, conversion of the encryption key (from a string to list of
    unsigned ints) was removed from the key retrieval method. This patch
    updates dmcrypt to decode an encryption key string, rather than a list
    of unsigned ints. See the linked bug for more information.

    The method used to decode the encryption key has been updated to use
    binascii, as done in os-brick [1], to maintain consistency. The key
    generation and decoding portions of test_dmcrypt have been updated to
    reflect this change and ensure compatibility with both, Python 2 and
    Python 3.

    [1] https://github.com/openstack/os-brick/blob/6cf9b1cd689f70a2c50c0fa83a9a9f7c502712a1/os_brick/encryptors/cryptsetup.py#L100-L102

    Closes-Bug: #1688342
    Change-Id: I050585ecb55742a972038cf72b0650321ded2856
    (cherry picked from commit 53a71c1241aac70018c16d174032427a172378ed)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 15.0.5

This issue was fixed in the openstack/nova 15.0.5 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.0.0.0b2

This issue was fixed in the openstack/nova 16.0.0.0b2 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/newton)

Reviewed: https://review.openstack.org/466481
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=10a62ec99e2839f0fb163d38f37fa09e9f7b6c1c
Submitter: Jenkins
Branch: stable/newton

commit 10a62ec99e2839f0fb163d38f37fa09e9f7b6c1c
Author: Jackie Truong <email address hidden>
Date: Thu May 4 12:51:22 2017 -0400

    Fix decoding of encryption key passed to dmcrypt

    This patch fixes the decoding of the encryption key passed to dmcrypt.
    During the key management move from Nova to Castellan, in the Newton
    release, conversion of the encryption key (from a string to list of
    unsigned ints) was removed from the key retrieval method. This patch
    updates dmcrypt to decode an encryption key string, rather than a list
    of unsigned ints. See the linked bug for more information.

    The method used to decode the encryption key has been updated to use
    binascii, as done in os-brick [1], to maintain consistency. The key
    generation and decoding portions of test_dmcrypt have been updated to
    reflect this change and ensure compatibility with both, Python 2 and
    Python 3.

    [1] https://github.com/openstack/os-brick/blob/6cf9b1cd689f70a2c50c0fa83a9a9f7c502712a1/os_brick/encryptors/cryptsetup.py#L100-L102

    Closes-Bug: #1688342
    Change-Id: I050585ecb55742a972038cf72b0650321ded2856
    (cherry picked from commit 53a71c1241aac70018c16d174032427a172378ed)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 14.0.8

This issue was fixed in the openstack/nova 14.0.8 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.