Cinder

Volume retype fails from non-encrypted to encrypted volume type

Bug #1687880 reported by Dinesh Gavand
32
This bug affects 5 people
Affects Status Importance Assigned to Milestone
Cinder
Confirmed
High
Unassigned

Bug Description

Env:
One Devstack with stable/ocata build which has lvm as backend.

Test steps:
1. Create a volume type "lvmdriver-1" with the only driver as backend in extra_spec.

2. Create another volume type "LUKS" with encription using below command:
   cinder encryption-type-create --cipher aes-xts-plain64 --key_size 256 --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor

3. Both the volume type must be configured with the same volume_backend_name in extra specs.

3. Create a volume <volume1> with volume type lvmdriver-1.

4. Retype above volume from lvmdriver-1 to LUKS.

Expect result:
The retype should succeed.

Actual result:
Retype failed. Observed below traces in c-vol.log

2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server Traceback (most recent call last):
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server result = func(ctxt, **new_args)
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 2396, in retype
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server new_reservations, status_update)
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server self.force_reraise()
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 2392, in retype
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server new_type_id=new_type_id)
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 2034, in migrate_volume
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server volume.save()
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server self.force_reraise()
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 2027, in migrate_volume
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server self._migrate_volume_generic(ctxt, volume, host, new_type_id)
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 1846, in _migrate_volume_generic
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server new_volume)
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server self.force_reraise()
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 1824, in _migrate_volume_generic
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server self._copy_volume_data(ctxt, volume, new_volume, remote='dest')
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 1741, in _copy_volume_data
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server {'src': src_vol['id'], 'dest': dest_vol['id']})
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server self.force_reraise()
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 1736, in _copy_volume_data
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server sparse=sparse_copy_volume)
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/utils.py", line 583, in copy_volume
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server sparse=sparse)
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/utils.py", line 467, in _copy_volume_with_path
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server execute(*cmd, run_as_root=True)
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/utils.py", line 123, in execute
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server return processutils.execute(*cmd, **kwargs)
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 400, in execute
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server cmd=sanitized_cmd)
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command.
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/disk/by-path/ip-10.50.128.121:3260-iscsi-iqn.2010-10.org.openstack:volume-b29bf38d-7bb7-485f-b316-8ca22f0cdd11-lun-1 of=/dev/disk/by-path/ip-10.50.128.121:3260-iscsi-iqn.2010-10.org.openstack:volume-7ff511b8-ee05-4077-9da7-ae30a864ce45-lun-1 count=1073741824 bs=1M iflag=count_bytes,direct oflag=direct
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server Exit code: 1
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server Stdout: u''
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server Stderr: u"/bin/dd: error writing '/dev/disk/by-path/ip-10.50.128.121:3260-iscsi-iqn.2010-10.org.openstack:volume-7ff511b8-ee05-4077-9da7-ae30a864ce45-lun-1': No space left on device\n1023+0 records in\n1022+0 records out\n1071644672 bytes (1.1 GB, 1022 MiB) copied, 10.0875 s, 106 MB/s\n"
2017-05-03 01:29:45.429 TRACE oslo_messaging.rpc.server

Eric Harney (eharney)
Changed in cinder:
assignee: nobody → Eric Harney (eharney)
Revision history for this message
Sean McGinnis (sean-mcginnis) wrote : Bug Assignee Expired

Unassigning due to no activity for > 6 months.

Changed in cinder:
assignee: Eric Harney (eharney) → nobody
Sofia Enriquez (lsofia-enriquez)
Changed in cinder:
status: New → Confirmed
assignee: nobody → Sofia Enriquez (lsofia-enriquez)
Revision history for this message
Sofia Enriquez (lsofia-enriquez) wrote :

Looks like this happens because the source volume is not the same size (larger) than the target. We're writing off the end of the device. In addition, that's related to cinder not being smart enough about managing the space loss due to encryption headers on encrypted volumes.

Revision history for this message
Sofia Enriquez (lsofia-enriquez) wrote :

This topic was discussed at Victoria PTG:

Firstly, we should change the API to allow a new size to be specified during a volume migration. For retype, Gorka suggested that we could allow a flag similar to the current migration_policy in the retype request, which gives the user the option to say it's OK to perform a migration if it's required. We could have a flag indicating that it's OK for the volume to be increased in size if it's required for the retype to succeed. This way we don't go behind the user's back -- we fail if the volume won't fit, and allow them to retype to a new size if they want the retype to succeed.

Finally, I should add a note to the docs explaining that retyping an unencrypted volume to the same size encrypted volume will most likely fail (so that users know not to try this in older releases)

https://wiki.openstack.org/wiki/CinderVictoriaPTGSummary#Sizing_encrypted_volumes

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/743869

Changed in cinder:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.opendev.org/732988
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=662b8210aab8e08bdddbf30e2b21ae357f3be32e
Submitter: Zuul
Branch: master

commit 662b8210aab8e08bdddbf30e2b21ae357f3be32e
Author: Sofia Enriquez <email address hidden>
Date: Tue Jun 2 19:31:34 2020 +0000

    Doc note warning about retyping an unencrypted/encrypted volume.

    As we discussed at Victoria PTG[1] it would be nice to have a note
    warning users to don’t try this.

    [1] https://wiki.openstack.org/wiki/CinderVictoriaPTGSummary#Sizing_encrypted_volumes

    Partial-Bug: #1687880
    Change-Id: Ifa896c5f7336d3ce2a382ded9dcb4b3d595220db

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/745199

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.opendev.org/745199
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=09ad89b7eecd6ea654a17690996c2ec4a63dd31e
Submitter: Zuul
Branch: master

commit 09ad89b7eecd6ea654a17690996c2ec4a63dd31e
Author: Sofia Enriquez <email address hidden>
Date: Thu Aug 6 18:13:24 2020 +0000

    Doc note warning about retyping unencrypted/encrypted volume

    As suggested in 662b8210aab8e08bdddbf30e2b21ae357f3be32e and
    discussed at Victoria PTG[1] it would be nice to have a notes
    warning users to don’t try this.

    [1] https://wiki.openstack.org/wiki/CinderVictoriaPTGSummary#Sizing_encrypted_volumes

    Change-Id: I3fd514126dbdf56d4d4d8e423e98e462238c683f
    Partial-Bug: #1687880

Sofia Enriquez (lsofia-enriquez)
Changed in cinder:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (master)

Change abandoned by "Sofia Enriquez <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/779114
Reason: This isn't the right way to do this

Sofia Enriquez (lsofia-enriquez)
Changed in cinder:
assignee: Sofia Enriquez (lsofia-enriquez) → nobody
status: In Progress → Confirmed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.