glusterfs: Instance is not using the correct volume snapshot file after reboot

This analysis does not sound quite right to me, but I could be missing something. connect_volume() uses data['name'] to determine the filename. This is populated with the info from the Cinder GlusterFS driver's initialize_connection(), which sets it correctly.

The responsibility for calculating which filename should be attached to the VM is purely on the Cinder side, Nova doesn't do that calculation.

You are correct. I was only looking at where the file name came from in nova, but not the actual root source of the file name, which is in initialize_connection.

I debugged this problem a bit more. It looks like the nova volume manager is using a stale copy of the block_device_mapping. The block_device_mapping needs to be refreshed in order for the updated volume snapshot to be used.

On power on, the nova manager (nova/compute/manager.py ) does:
1. start_instance
2. _power_on
3. _get_instance_volume_block_device_info

The structure for this method is:
def _get_instance_volume_block_device_info(self, context, instance, refresh_conn_info=False, bdms=None):
    if not bdms:
        bdms = (block_device_obj.BlockDeviceMappingList.
                    get_by_instance_uuid(context, instance['uuid']))
        block_device_mapping = (
            driver_block_device.convert_volumes(bdms) +
            driver_block_device.convert_snapshots(bdms) +
            driver_block_device.convert_images(bdms))
    ....
block_device_obj.BlockDeviceMappingList.get_by_instance_uuid() goes and queries the database to construct the bdms, which will contain stale data.

In order for this to work properly, refresh_conn_info must be set to True. However, since _get_instance_volume_block_device_info is used by everyone, I am not sure if setting refresh_conn_info = True is such a good solution, since it will affect other cinder drivers also.

Looking at it closer, I'd say that we just need to refresh the connection info once we are done with the snapshot.

We should be careful to avoid any race conditions tho, but basically what we need to do in both volume_snapshot_create and _delete methods of the libvirt driver is:

  from nova.virt import block_device as driver_block_device
  bdm = block_device_obj.BlockDeviceMapping.get_by_volume_id(
                    context, volume_id)
  driver_bdm = driver_block_device.DriverVolumeBlockDevice(bdm)
  driver_bdm.refresh_connection_info(context, instance, self._volume_api, self)

and you should be good to go.

The only caveat here is that refresh_connection_info will call out to Cinder's initialize_connection which may be problematic since the snapshot is in "creating" state. But nothing that a dumb polling loop won't solve (sigh). In reality we should probably rething these interactions.

Nikola: Thank you for the pointer :) I will try out your advice above.

Fix proposed to branch: master
Review: https://review.openstack.org/87432

Reviewed: https://review.openstack.org/87432
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=329b594436aad814e43740ea26841342a8772aff
Submitter: Jenkins
Branch: master

commit 329b594436aad814e43740ea26841342a8772aff
Author: Thang Pham <email address hidden>
Date: Mon Apr 14 22:47:06 2014 -0400

    libvirt: Refresh volume connection_info after volume snapshot

    The following patch is related to the guest assisted snapshot
    functionality. When you take a snapshot of a volume
    (e.g. GlusterFS) attached to a running instance, a new snapshot
    file is created, i.e. volume-<uuid>.<snapshot-uuid>. The
    instance uses this file as the active volume. If you shutdown
    and restart the instance, nova will reattach the base volume
    (volume-<uuid>) to the instance instead of the snapshot volume
    (volume-<uuid>.<snapshot-uuid>). The expected behavior is to
    have the snapshot volume reattach to the instance. This is
    caused by stale data being returned from the database when
    _get_instance_volume_block_device_info is called during
    _power_on. To fix this bug, this patch calls
    refresh_connection_info to update the database in both
    volume_snapshot_create and _volume_snapshot_delete methods of the
    libvirt driver.

    Change-Id: I0f340a3f879580e7981d97863bc299e33d71aa84
    Closes-Bug: #1304695

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/98255

Reviewed: https://review.openstack.org/98255
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=96212b194d390c4de9298376aefe84e3a70008de
Submitter: Jenkins
Branch: stable/icehouse

commit 96212b194d390c4de9298376aefe84e3a70008de
Author: Thang Pham <email address hidden>
Date: Mon Apr 14 22:47:06 2014 -0400

    libvirt: Refresh volume connection_info after volume snapshot

    The following patch is related to the guest assisted snapshot
    functionality. When you take a snapshot of a volume
    (e.g. GlusterFS) attached to a running instance, a new snapshot
    file is created, i.e. volume-<uuid>.<snapshot-uuid>. The
    instance uses this file as the active volume. If you shutdown
    and restart the instance, nova will reattach the base volume
    (volume-<uuid>) to the instance instead of the snapshot volume
    (volume-<uuid>.<snapshot-uuid>). The expected behavior is to
    have the snapshot volume reattach to the instance. This is
    caused by stale data being returned from the database when
    _get_instance_volume_block_device_info is called during
    _power_on. To fix this bug, this patch calls
    refresh_connection_info to update the database in both
    volume_snapshot_create and _volume_snapshot_delete methods of the
    libvirt driver.

    Conflicts:
     nova/tests/virt/libvirt/test_libvirt.py

    Change-Id: I0f340a3f879580e7981d97863bc299e33d71aa84
    Closes-Bug: #1304695