Share Replication: File-based locks don't provide concurrency control in multi-node/multi-AZ deployments

After much deliberation and multiple design summit discussions, we have the impression that this is not to be treated as a bug. We will introduce a mechanism to provide distributed locking management underneath manila. However, deployers today have an option of using file locks living on a shared file system across distributed services.

The tooz abstraction is proposed here: https://review.openstack.org/#/c/318336

It will not be back ported to the releases where this "bug" exists;

Please note, that tooz can also be used with no further configuration, i.e, default to using file locks as is the current behavior for the share replication feature to work as intended.

Reviewed: https://review.openstack.org/318336
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=02ab18c5dfa8d49631052d9951b27d57a5340968
Submitter: Jenkins
Branch: master

commit 02ab18c5dfa8d49631052d9951b27d57a5340968
Author: Goutham Pacha Ravi <email address hidden>
Date: Tue May 17 10:55:20 2016 -0400

    Tooz integration

    Manila currently uses file locks from oslo_concurrency to
    coordinate operations racing with each other to perform a
    particular action. In many situations, deployers may need a
    distributed lock to a local file lock (or even a file lock living on
    a shared file system). This need is accentuated if they were running
    Manila services in HA or if they were using Share Replication across
    AZs where manila-share services were running off different controllers
    that would not be able to share a common oslo_concurrency
    file lock or be protected against service/lock management failures.

    Integrate Tooz library with helper methods to create a locking
    coordinator and allow deployers to make the choice between file
    and distributed locks.

    Start the manila share service with Tooz backed coordination.

    Replace the locks used for Share Replication work-flows in the
    share manager to use Tooz based locks.

    Co-Authored-By: Goutham Pacha Ravi <email address hidden>
    Co-Authored-By: Szymon Wroblewski <email address hidden>
    Co-Authored-By: Tom Barron <email address hidden>

    Related-Bug: #1585241
    Partially-implements: bp distributed-locking-with-tooz
    Change-Id: I710e86bd42034fa3b93b87ff77fa48ada8661168

Goutham any update on this defect?

Thanks for checking Jason. These file locks have still not been replaced by tooz based locks. There is going to be a general effort to replace all file locks used by the core share manager service from oslo_concurrency to tooz based locks that can be back-ended by a distributed lock management system (such as tooz+etcd or tooz+zookeeper). We're going to discuss this at the Denver PTG in April 2019 [1]. I'll take an AI to come back and update this bug report.

[1] https://etherpad.openstack.org/p/manila-denver-train-ptg-planning

This bug has been added to work items in Train. Please see this wiki for progress on these items, and if it will be acked/completed in the Train release: https://wiki.openstack.org/wiki/Manila/TrainCycle#Active-Active_Share_Service

Goutham is this pushed out to Ussuri?

Yes, Thanks for checking, Jason!

Additional comments http://eavesdrop.openstack.org/meetings/manila/2020/manila.2020-09-03-15.00.log.html

Thanks for sharing https://wiki.openstack.org/wiki/Manila/TrainCycle#Active-Active_Share_Service <a href="https://kepalabergetar.today/">,</a>