Migration: LVM volume migration may end up with the additional volume in destination VG of on the same host

Bug #1489335 reported by Vincent Hou
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
High
Vincent Hou

Bug Description

This issue happens to available volumes migrating from one LVM VG to another VG on the same host. The method copy_volume using dd command has no exception handling provided, so if the dd fails by accident, the destination volume created in dest VG still remains.

Propose to fix:
Add a try and catch block to copy_volume, and remove the dest volume in the exception handling part.

Background: I identified some critical issues and unreasonable code behaviors during my test of volume migration. My initial plan was to fix them in my blueprint of volume migration improvement. However, there is only one week left for the feature acceptance and some of my major patches may not make into master in L release, so I decide to log these issues on by one and get them fixed as soon as possible in L release.
@mtanimo, @jbernard, and @every migration contributor, some of your undergoing patches may cover these issues, but as I claimed above, major feature patches may miss the deadline of L release. Feel free to reach out to me for any migration issues. Thank you for your support.
@Every Cinder contributor, thank you for you understanding.

Tags: migration
Vincent Hou (houshengbo)
Changed in cinder:
assignee: nobody → Vincent Hou (houshengbo)
Revision history for this message
Jon Bernard (jbernard) wrote :

The code in https://review.openstack.org/#/c/187270/ should resolve this.

Changed in cinder:
assignee: Vincent Hou (houshengbo) → Jon Bernard (jbernard)
status: New → In Progress
Jay Bryant (jsbryant)
tags: added: migration
Changed in cinder:
importance: Undecided → High
Revision history for this message
Vincent Hou (houshengbo) wrote :

@Jon, I do not think you get my point exactly.
This issue needs to be resolved in the method migrate_volume from the LVM driver.
I will give the fix.

Changed in cinder:
assignee: Jon Bernard (jbernard) → Vincent Hou (houshengbo)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/218750

Changed in cinder:
assignee: Vincent Hou (houshengbo) → Sean Dague (sdague)
Changed in cinder:
assignee: Sean Dague (sdague) → Vincent Hou (houshengbo)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/187270
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=f586043fa969b9d1dcf4933aacbf615f53691093
Submitter: Jenkins
Branch: master

commit f586043fa969b9d1dcf4933aacbf615f53691093
Author: Jon Bernard <email address hidden>
Date: Mon Jun 1 14:09:30 2015 -0400

    Add support for file I/O volume migration

    This patch changes the generic volume migration logic to support
    non-attachable volumes. Non-attachable refers to volume drivers that do
    not support attachment via the typical iSCSI or similar protocols where
    a block device is made available on the host machine. Device drivers
    such as RBD make volumes available to cinder via a file handle that
    proxies read() and write() calls to the Ceph cluster.

    This patch improves the generic migration logic to determine whether a
    migration operation can proceed with dd using block device paths or file
    operations on handles returned from the os-brick connectors.

    Changes to the RBD driver are included to correctly rename the target
    volume during the completion phase of a successful migration.

    It appears there is still some work to be done for attached in-use
    volume migration for certain configurations. Successful tests were seen
    for:

      LVM to LVM (available and in-use)
      LVM to/from NFS (available and in-use)
      LVM to/from Ceph (available)
      Ceph to LVM (in-use)
      NFS to/from Ceph (available)
      Ceph to NFS (in-use)

    Failures were seen (due to Nova) for the following:

      LVM to Ceph (in-use)
      NFS to Ceph (in-use)

    (Pulled from gate, cinder can no longer pass unit tests)

    Blueprint: generic-volume-migration
    Closes-Bug: #1489335
    Closes-Bug: #1489337
    Change-Id: Iece2776fa751152f97b389ddab426e50c6f79bea

Changed in cinder:
status: In Progress → Fix Committed
Revision history for this message
Vincent Hou (houshengbo) wrote :

This one is not fixed. I will give the fix

Changed in cinder:
status: Fix Committed → New
Changed in cinder:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/218750
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=224688ce914444531181af9dff1ce794d7108490
Submitter: Jenkins
Branch: master

commit 224688ce914444531181af9dff1ce794d7108490
Author: Vincent Hou <email address hidden>
Date: Mon Aug 31 15:14:37 2015 +0800

    LVM: add the exception handling to volume copy

    If the volume copy fails from one vg to another one on
    the same host, the destination volume on the destination
    pool should be removed.

    Change-Id: I41922e197e09827582e660a6a00b86aa7044093e
    Closes-Bug: #1489335

Changed in cinder:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in cinder:
milestone: none → liberty-rc1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in cinder:
milestone: liberty-rc1 → 7.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.