Mitaka Resize VMware Ephemeral Root Disk -> Internal Server Error 500

Bug #1609413 reported by Fabian Wiesel
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Confirmed
Low
Unassigned

Bug Description

Description
===========
I am observing an internal server error when trying to resize the root disk.

The error is: 'NoneType' object has no attribute 'backing'
"nova/virt/vmwareapi/vmops.py", line 1314, in _resize_create_ephemerals_and_swap ds_ref = vmdk.device.backing.datastore

As I see it, it is caused by the assumption, that the root disk is named after the UUID here:
https://github.com/openstack/nova/blob/master/nova/virt/vmwareapi/vm_util.py#L634
To my knowledge, that is not generally the case, and VSphere quite happily renames it (e.g after a Storage VMotion, which may happen automatically, when the ephemeral storage is in a SDRS cluster).
In my case, the root disk is actually named <uuid>-00001.vmdk, resulting (presumably) in the following:

  root_disk= '<uuid>.vmdk'
There is no backing of the file,
  root_device = None
And given those two variables (https://github.com/openstack/nova/blob/master/nova/virt/vmwareapi/vm_util.py#L657-L658)

Resulting in vmdk.device = None (the same as root_device) here
  https://github.com/openstack/nova/blob/stable/mitaka/nova/virt/vmwareapi/vmops.py#L1314

Steps to reproduce
==================
* Create Instance
* Trigger Storage VMotion to a different datastore (there are possibly different ways to cause the rename)
* Resize the ephemeral root device

Expected result
===============
* The VM is running with a resized root disk

Actual result
=============
The following error message:

Message:'NoneType' object has no attribute 'backing'
Code: 500
Details: File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 375, in decorated_function return function(self, context, *args, **kwargs) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 4054, in finish_resize self._set_instance_obj_error_state(context, instance) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 4042, in finish_resize disk_info, image_meta) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 4007, in _finish_resize old_instance_type) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 4002, in _finish_resize block_device_info, power_on) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/virt/vmwareapi/driver.py", line 306, in finish_migration block_device_info, power_on) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/virt/vmwareapi/vmops.py", line 1443, in finish_migration block_device_info) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/virt/vmwareapi/vmops.py", line 1314, in _resize_create_ephemerals_and_swap ds_ref = vmdk.device.backing.datastore

Environment
===========

VSphere 6.0
Two ephemeral datastores on VMFS 5 (FcoE), not in a SDRS Cluster
Own Kolla Build from mitaka/stable (be033eef7296d132c8f7eb5e42203f5a3a947de6)

Revision history for this message
Fabian Wiesel (fabian-wiesel) wrote :

If I am not mistaken, I can take care of fixing the bug, and submit a patch proposal.

Is there any reason, why not the first disk on the first controller is taken?

tags: added: vmware
Revision history for this message
Chinmaya Bharadwaj (acbharadwaj) wrote :

Check if this bug related, there is a fix merged in master.
https://bugs.launchpad.net/nova/+bug/1444446

Revision history for this message
Chinmaya Bharadwaj (acbharadwaj) wrote :

The back ported patch for stable/mitaka is still in review.

https://review.openstack.org/#/c/347726/

Revision history for this message
Fabian Wiesel (fabian-wiesel) wrote :

Hi,

the patch is not really related. In my case, the root disk is an ephemeral disk, not a cinder volume.

It simply fails to recognise the ephemeral root disk, because the assumption "root_disk= '<uuid>.vmdk'" does not hold true.

From what I read, the proposed patch would simply silently skip resizing the ephemeral root disk.

Revision history for this message
Chinmaya Bharadwaj (acbharadwaj) wrote :

May i know how did you create the VM with ephemeral disk as root disk? if you create a VM with a flavor mentioning an ephemeral disk size, an ephemeral disk will be attached to the VM, it wont be the root disk.

Revision history for this message
YuYang (yuyangwang1985) wrote :

i change the get root disk code logic.is this can fix you problem?
https://review.openstack.org/#/c/402256/

Revision history for this message
Fabian Wiesel (fabian-wiesel) wrote :

Unfortunately, that won't work. After a vmotion, all disks will be named "<vm name>-<number>.vmdk"
So, any disk would match the pattern you probably wanted to test for.

Sean Dague (sdague)
Changed in nova:
status: New → Confirmed
importance: Undecided → Low
Revision history for this message
Sean Dague (sdague) wrote :

Automatically discovered version mitaka in description. If this is incorrect, please update the description to include 'nova version: ...'

tags: added: openstack-version.mitaka
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.