Creating volume from snapshot on real/production/multicluster installation of OpenStack is broken

I'm now working on some solution and will paste patch when it is ready.

I prepared solution of this issue in my OpenSack Nova fork on GitHUB at https://github.com/pulchart/nova/commit/76e59b9628153b6c71150797f7fe35804502624d

How it works:
- each new snapshot is exported via iSCSI (like volumes)
- nova-volume, on which we creating the new volume from snapshot, register this iSCSI export into local system (if iscsi_ip_preffix filter pass)
- nova-volume can now use local copy
- after that the iSCSI snapshot is unregistered from local system

Unfortunately "host" and "provider_location" are missing in DB schema so I created some hacks:
- "host" is used from snapshot's volume (LVM snapshot is on same host as volume), there is a new API call "snapshot_get_host"
- iSCSI discovery is used instead of using location from database

Fixes:
- "_run_iscsiadm" from nova/volume/driver.py is synchronized with same function from libvirt

I forget to "init_host" function, so snapshot is not registered after server reboot. I will fix it and send updated patch.

Fixed in 1008866_v2.patch
- new: snapshot iSCSI exports are initialized by nova-volume start
(https://github.com/pulchart/nova/commit/a2df1c340fe874f2d2100e3e59b89140974ac51c)

1008866_v3.patch:
- "create_volume_from_snapshot" function need to be synchronized

....
+ @utils.synchronized('create_volume_from_snapshot')
     def create_volume_from_snapshot(self, volume, snapshot):
....

Rebased git source link: https://github.com/pulchart/nova/commit/e13c308e11b306109c04223c8f5d282f612d34e0

daily updates :)

1008866_v4.patch: https://github.com/pulchart/nova/commit/a0c858c22305772f8acd4c3194392f60bd78464b

@Jaroslav: would you consider pushing the change to Gerrit ?
See http://wiki.openstack.org/HowToContribute#If_you.27re_a_developer.2C_start_here:

I have to handle one latest issue of this patch before I push it into "world". I try to find solution with any DB table/entry update, unfortunately it still not correctly works without updating "iscsi_targets" table.

So current solution expect this:
1/ all volume.(id)s on nova-volume on one hostX != snapshot.(id)s on the same hostX
2/ created snapshot.id < max(volume.id)
That is caused due to using/mixing "iscsi_targets.volume_id" with both "volume.id"/"snapshot.id". This workaround woks for my use case but can cause unxepected problems in some other situations.

I have seen some blueprints about changing volume_id, snapshot_id to UUID which is uniq across whole stack, so I thing this migration to UUID will solve all problems of this issue.

I wonder if a simpler solution might just be to force volumes created from snapshots onto the same host as the snapshot. In other words, skip the scheduler altogether and just send the message directly to the right host if snapshot_id is set. Clearly this could lead to an overload of some hosts in the system, but it seems much more managable in the short term. Then we have more time to work on a robust solution for migrating/creating volumes on different hosts.

That is simple solution for implementation but not usable for cloud use-case(s). We have to be able create volume from snapshot in every place of In the Private Cloud (= on any volume node in cloud). Attached patch is now used and it solve many problems that we had.

Question: Do we use (or create) "Cloud solution" or "a simple tool for managing local LVM stuff"?
My answer is: We do NOT (!) need simple LVM tool manager for local volume node. We need cloud solution.

I will have some time this week for updating attached patch and fix the last problem of proposed solution. Please keep tuned :).

I understand that we want a better soution, but I think your patch is too large to backport into stable/essex and therefore should be done against the new cinder project.

The simple solution is appropriate for an essex backport. The more complex solution can be done in cinder, as it still requires some cleanup.

For example, the attach to snapshot code is essentially the same as attach to volume code and should be put into a little library of some sort. Also, I couldn't tell from the patch if it is using discovery by default, but if so it should be storing provider_location in the snapshots as well and connecting via an initialize_connection call in the same way it does for volumes.

In other words, this is a large set of changes that should be planned coherently in the cinder project.

> I understand that we want a better soution, but I think your patch is too large to
> backport into stable/essex and therefore should be
> done against the new cinder project.

Yes I understand too. Unfortunately, proposed simple solution will not fix this regression of this feature for production environment with many nova-volume nodes. But, yes you can workaround "the error/exception in log" by this way (until volume space is not exhausted). In our use-case we will rollback the simple fix and apply our fix in stable/essex release in our build of OpenStack.

> The simple solution is appropriate for an essex backport.
> The more complex solution can be done in cinder, as it still requires some cleanup.

Yes completely agree, cleanups are needed for future release. This patch was created in hurry without thinking about best practices.

> For example, the attach to snapshot code is essentially the same as
> attach to volume code and should be put into a little library of some sort.
> Also, I couldn't tell from the patch if it is using discovery by default,
> but if so it should be storing provider_location in the snapshots as
> well and connecting via an initialize_connection call in the same
> way it does for volumes.

I completely agree.

> In other words, this is a large set of changes that should be planned coherently in the cinder project.

Yes, again I completely agree without comments because review comments was expected ;). Please fell free to re-factor this patch/code for cinder project. I will be happy man whit the best solution/code :). This patch was for essex = do it without any DB modifications, without rewriting current functions if it is possible, do it immediately because this regression was blocking issue for our Private Cloud based on Essex.

Fix proposed to branch: master
Review: https://review.openstack.org/10649

Reviewed: https://review.openstack.org/9761
Committed: http://github.com/openstack/cinder/commit/99456bd690445443ae05c0d4fe1ec43ba6090f6f
Submitter: Jenkins
Branch: master

commit 99456bd690445443ae05c0d4fe1ec43ba6090f6f
Author: ZhuRongze <email address hidden>
Date: Fri Jul 13 12:07:13 2012 +0000

    Send 'create volume from snapshot' to the proper host

    A simple solution for bug 1008866. When creating volume from snapshot on
    multicluster, in volume it will check if snapshot_id is set. If snapshot_id
    is set, make the call create volume directly to the volume host where the
    snapshot resides instead of passing it through the scheduler. So snapshot can
    be copy to new volume.

    Change-Id: Ie9c1a77f62abc40e294b1d0c604cf885652728da

Reviewed: https://review.openstack.org/10649
Committed: http://github.com/openstack/nova/commit/6795de644b8a8a1879543101d85ba90674219c8b
Submitter: Jenkins
Branch: master

commit 6795de644b8a8a1879543101d85ba90674219c8b
Author: ZhuRongze <email address hidden>
Date: Wed Aug 1 13:23:13 2012 +0000

    Send 'create volume from snapshot' to the proper host

    A simple solution for bug 1008866. When creating volume from snapshot on
    multicluster, in volume it will check if snapshot_id is set. If snapshot_id
    is set and FLAGS.snapshot_same_host is true, make the call create volume
    directly to the volume host where the snapshot resides instead of passing it
    through the scheduler. So snapshot can be copy to new volume. The same as
    review 9761.

    Change-Id: Ic182eb4563b9462704c5969d5116629442df316a

Fix proposed to branch: master
Review: https://review.openstack.org/12675

Reviewed: https://review.openstack.org/12675
Committed: http://github.com/openstack/cinder/commit/3eaf43a9f5a26a51a89347cffe39bfae2b12c2d6
Submitter: Jenkins
Branch: master

commit 3eaf43a9f5a26a51a89347cffe39bfae2b12c2d6
Author: Rongze Zhu <email address hidden>
Date: Sun Sep 9 15:35:26 2012 +0800

    Prevent from bug #1008866 is reverted

    Fixes bug #1047841.

    Commit 2f5360753308eb8b10581fc3c026c1b66f42ebdc (Adds new volume API
    extensions) reverted a part of commit
    99456bd690445443ae05c0d4fe1ec43ba6090f6f (Send 'create volume from
    snapshot' to the proper host), so bug #1008866 is reproduced. I make
    API.create_volume to call _cast_create_volume in cinder/volume/api.py,
    it Prevent from bug #1008866 is reverted.

    Change-Id: I1bf0b7c5fc47da756bce95128f8fd770d14399b0

Fix proposed to branch: master
Review: https://review.openstack.org/12854