PowerMax driver may deadlock moving volumes between SGs

Bug #1980870 reported by Gorka Eguileor
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
In Progress
Low
Gorka Eguileor

Bug Description

There's a potential deadlock scenario in PowerMax's masking.py move_volume_between_storage_groups" method.

The method uses 2 locks, one for the source Storage Group and another for the destination Storage Group, and it could happen that if 2 requests going in opposite directions are received simultaneously their first lock acquisition interleaves resulting in a deadlock situation.

    @coordination.synchronized(
        "emc-sg-{source_storagegroup_name}-{serial_number}")
    @coordination.synchronized(
        "emc-sg-{target_storagegroup_name}-{serial_number}")
    def move_volume_between_storage_groups(
            self, serial_number, device_id, source_storagegroup_name,
            target_storagegroup_name, extra_specs, force=False,
            parent_sg=None):

The scenario would be like this:

- User requests an instance migration from A to B
- User requests an instance migration from B to A
- Driver acquires the first lock for A-to-B which is something like cinder-emc-sg-SGA-###
- Driver acquires the first lock for B-to-A which is something like cinder-emc-sgSGB-###

The deadlock happens because A-to-B waits forever for the lock held by the B-to-A operation, which in turn cannot proceed because it’s waiting for lock help by A-to-B.

Changed in cinder:
status: New → In Progress
Revision history for this message
Sofia Enriquez (lsofia-enriquez) wrote :
Changed in cinder:
importance: Undecided → Low
tags: added: deadlock powermax
Eric Harney (eharney)
tags: added: drivers
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.