HNAS iSCSI Driver errors in concurrent scenarios

Bug #1475007 reported by Erlon R. Cruz
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Undecided
Erlon R. Cruz

Bug Description

It might happen that multiple cinder nodes are trying to add volumes to the same target. In this situation, the driver can get an error trying to add a volume to a hlun just taken by other instance. In this case we backoff and retry ISCSI_ATTACH_RETRIES times.

How to reproduce:
1 - Create 10 or more instances at the same time (boot from image, creates a new volume)

The following error will be shown:

=================================================================

1b57d45-f97b-4dae-9c17-b02c24f5b92b to volume: 9a3dc42a-5a3a-4e9c-889a-1383d412c803
2015-07-15 13:56:11.223 TRACE cinder.volume.flows.manager.create_volume Traceback (most recent call last):
2015-07-15 13:56:11.223 TRACE cinder.volume.flows.manager.create_volume File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 501, in _copy_image_to_volume
2015-07-15 13:56:11.223 TRACE cinder.volume.flows.manager.create_volume copy_image_to_volume(context, volume_ref, image_service, image_id)
2015-07-15 13:56:11.223 TRACE cinder.volume.flows.manager.create_volume File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2015-07-15 13:56:11.223 TRACE cinder.volume.flows.manager.create_volume return f(*args, **kwargs)
2015-07-15 13:56:11.223 TRACE cinder.volume.flows.manager.create_volume File "/opt/stack/cinder/cinder/volume/driver.py", line 558, in copy_image_to_volume
2015-07-15 13:56:11.223 TRACE cinder.volume.flows.manager.create_volume attach_info, volume = self._attach_volume(context, volume, properties)
2015-07-15 13:56:11.223 TRACE cinder.volume.flows.manager.create_volume File "/opt/stack/cinder/cinder/volume/driver.py", line 701, in _attach_volume
2015-07-15 13:56:11.223 TRACE cinder.volume.flows.manager.create_volume raise exception.VolumeBackendAPIException(data=err_msg)
2015-07-15 13:56:11.223 TRACE cinder.volume.flows.manager.create_volume VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: Unexpected error while running command.
2015-07-15 13:56:11.223 TRACE cinder.volume.flows.manager.create_volume Command: ssc -u supervisor -p supervisor 172.24.44.15 console-context --evs 2 iscsi-target addlu cinder-default volume-9a3dc42a-5a3a-4e9c-889a-1383d412c803 4
2015-07-15 13:56:11.223 TRACE cinder.volume.flows.manager.create_volume Exit code: 1
2015-07-15 13:56:11.223 TRACE cinder.volume.flows.manager.create_volume Stdout: u"Couldn't add logical unit to target, error: Logical unit in use\n"
2015-07-15 13:56:11.223 TRACE cinder.volume.flows.manager.create_volume

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/203780

Changed in cinder:
assignee: nobody → Erlon R. Cruz (sombrafam)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/203780
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=a95fef70b407a400c27195bd1cfbf09858511e9f
Submitter: Jenkins
Branch: master

commit a95fef70b407a400c27195bd1cfbf09858511e9f
Author: Erlon R. Cruz <email address hidden>
Date: Fri Jul 10 15:36:22 2015 -0300

    Fix concurrent attaches on HNAS iSCSI driver

    It might happen that multiple cinder nodes/backends are trying to add
    volumes to the same target. In this situation, the driver can get an
    error trying to add a volume to a hlun just taken by other instance.
    This patch fix this avoiding concurrency using mutual exclusion and adds
    retries to the failed attempts to cover scenarios where multiple cinder
    hosts are used.

    Closes-bug: #1475007
    Change-Id: Ie2d3b286eecbf0299ac0c0e32d3e098bb5d11e4f

Changed in cinder:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in cinder:
milestone: none → liberty-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in cinder:
milestone: liberty-3 → 7.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.