| 2015-06-01 16:35:39 |
Gorka Eguileor |
description |
Since rtslib has no synchronized access to configfs [1] it can happen that rtstool or rtslib try to access something that is no longer there because other Cinder request has removed the target.
These errors can happen on creating, attaching, detaching and deleting volumes, and resulting log messages are weird/misleading because the exception refers to a different volume-id from the one we were working on.
I have located different errors that are originated from this same issue:
1)
TRACE oslo.messaging.rpc.dispatcher Stderr: u'Traceback (most recent call last):
File "/bin/cinder-rtstool", line 280, in <module>
sys.exit(main())
File "/bin/cinder-rtstool", line 251, in main
get_targets()
File "/bin/cinder-rtstool", line 162, in get_targets
for t, dump in _get_targets(rtsroot):
File "/bin/cinder-rtstool", line 51, in _get_targets
return [(target, target.dump()) for target in rtsroot.targets]
File "/usr/lib/python2.7/site-packages/rtslib/target.py", line 136, in dump
d[\'tpgs\'] = [tpg.dump() for tpg in self.tpgs]
File "/usr/lib/python2.7/site-packages/rtslib/target.py", line 430, in dump
d[\'luns\'] = [lun.dump() for lun in self.luns]
File "/usr/lib/python2.7/site-packages/rtslib/target.py", line 640, in dump
(self.storage_object.plugin, self.storage_object.name)
File "/usr/lib/python2.7/site-packages/rtslib/target.py", line 557, in _get_storage_object
return tcm.StorageObject.so_from_path(alias_path)
File "/usr/lib/python2.7/site-packages/rtslib/tcm.py", line 113, in so_from_path
return so_mapping[so_type](so_name)
KeyError: \'lun\'
2)
Stderr: u'Traceback (most recent call last):
File "/bin/cinder-rtstool", line 258, in <module>
sys.exit(main())
File "/bin/cinder-rtstool", line 229, in main
get_targets()
File "/bin/cinder-rtstool", line 141, in get_targets
print(x.dump()[\'wwn\'])
File "/usr/lib/python2.7/site-packages/rtslib/target.py", line 136, in dump
d[\'tpgs\'] = [tpg.dump() for tpg in self.tpgs]
File "/usr/lib/python2.7/site-packages/rtslib/target.py", line 427, in dump
d = super(TPG, self).dump()
File "/usr/lib/python2.7/site-packages/rtslib/node.py", line 218, in dump
params[item] = self.get_parameter(item)
File "/usr/lib/python2.7/site-packages/rtslib/node.py", line 190, in get_parameter
return fread(path)
File "/usr/lib/python2.7/site-packages/rtslib/utils.py", line 89, in fread
with open(path, \'r\') as file_fd:
IOError: [Errno 2] No such file or directory: \'/sys/kernel/config/target/iscsi/iqn.2010-10.org.openstack:volume-1468da95-22a3-470c-8ac9-4b27cac8acb3/tpgt_1/param/IFMarkInt\'\n'
3)
2015-05-17 20:51:49.829 62162 ERROR cinder.brick.iscsi.iscsi [req-6981f2b5-2259-49d8-a8ce-6d41de344944 50a742bdfe4e4a8ca0ddc150fc648904 648ffe15b9fe4a1bba71e4643d4dbbf2 - - -] Unexpected error while running command. Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf cinder-rtstool delete iqn.2010-10.org.openstack:volume-ad12c1e0-6746-4bce-9b94-5df1cae4ee2a Exit code: 1 Stdout: '' Stderr: 'Traceback (most recent call last):
File "/bin/cinder-rtstool", line 238, in <module>
sys.exit(main())
File "/bin/cinder-rtstool", line 224, in main
delete(iqn)
File "/bin/cinder-rtstool", line 151, in delete
for x in rtsroot.storage_objects:
File "/usr/lib/python2.7/site-packages/rtslib/root.py", line 86, in _list_storage_objects
for so in StorageObject.all():
File "/usr/lib/python2.7/site-packages/rtslib/tcm.py", line 102, in all
yield cls.so_from_path(so_dir)
File "/usr/lib/python2.7/site-packages/rtslib/tcm.py", line 117, in so_from_path
return mapping[so_type](so_name)
File "/usr/lib/python2.7/site-packages/rtslib/tcm.py", line 678, in __init__
super(BlockStorageObject, self).__init__(name, \'lookup\')
File "/usr/lib/python2.7/site-packages/rtslib/tcm.py", line 48, in __init__
self._backstore = _Backstore(name, type(self), mode)
File "/usr/lib/python2.7/site-packages/rtslib/tcm.py", line 785, in __init__
(self._plugin, name))
rtslib.utils.RTSLibError: Storage object block/iqn.2010-10.org.openstack:volume-fb12df40-d1e3-4a4e-b47d-0162ce9573cc not found
The cleanest solution is synchronizing access in Cinder.
[1]: https://github.com/agrover/rtslib-fb/issues/36 |
Since rtslib has no synchronized access to configfs [1] it can happen that rtstool or rtslib try to access something that is no longer there because other Cinder request has removed the target.
These errors can happen on creating, attaching, detaching and deleting volumes, and resulting log messages are weird/misleading because the exception refers to a different volume-id from the one we were working on.
The reason why the exception is from another volume is because library was looping through all elements, and when it tried to access one to get its info to do a dump (for example) the info no longer exists and that's unexpected and raises an Exception.
I have located different errors that are originated from this same issue:
1)
TRACE oslo.messaging.rpc.dispatcher Stderr: u'Traceback (most recent call last):
File "/bin/cinder-rtstool", line 280, in <module>
sys.exit(main())
File "/bin/cinder-rtstool", line 251, in main
get_targets()
File "/bin/cinder-rtstool", line 162, in get_targets
for t, dump in _get_targets(rtsroot):
File "/bin/cinder-rtstool", line 51, in _get_targets
return [(target, target.dump()) for target in rtsroot.targets]
File "/usr/lib/python2.7/site-packages/rtslib/target.py", line 136, in dump
d[\'tpgs\'] = [tpg.dump() for tpg in self.tpgs]
File "/usr/lib/python2.7/site-packages/rtslib/target.py", line 430, in dump
d[\'luns\'] = [lun.dump() for lun in self.luns]
File "/usr/lib/python2.7/site-packages/rtslib/target.py", line 640, in dump
(self.storage_object.plugin, self.storage_object.name)
File "/usr/lib/python2.7/site-packages/rtslib/target.py", line 557, in _get_storage_object
return tcm.StorageObject.so_from_path(alias_path)
File "/usr/lib/python2.7/site-packages/rtslib/tcm.py", line 113, in so_from_path
return so_mapping[so_type](so_name)
KeyError: \'lun\'
2)
Stderr: u'Traceback (most recent call last):
File "/bin/cinder-rtstool", line 258, in <module>
sys.exit(main())
File "/bin/cinder-rtstool", line 229, in main
get_targets()
File "/bin/cinder-rtstool", line 141, in get_targets
print(x.dump()[\'wwn\'])
File "/usr/lib/python2.7/site-packages/rtslib/target.py", line 136, in dump
d[\'tpgs\'] = [tpg.dump() for tpg in self.tpgs]
File "/usr/lib/python2.7/site-packages/rtslib/target.py", line 427, in dump
d = super(TPG, self).dump()
File "/usr/lib/python2.7/site-packages/rtslib/node.py", line 218, in dump
params[item] = self.get_parameter(item)
File "/usr/lib/python2.7/site-packages/rtslib/node.py", line 190, in get_parameter
return fread(path)
File "/usr/lib/python2.7/site-packages/rtslib/utils.py", line 89, in fread
with open(path, \'r\') as file_fd:
IOError: [Errno 2] No such file or directory: \'/sys/kernel/config/target/iscsi/iqn.2010-10.org.openstack:volume-1468da95-22a3-470c-8ac9-4b27cac8acb3/tpgt_1/param/IFMarkInt\'\n'
3)
2015-05-17 20:51:49.829 62162 ERROR cinder.brick.iscsi.iscsi [req-6981f2b5-2259-49d8-a8ce-6d41de344944 50a742bdfe4e4a8ca0ddc150fc648904 648ffe15b9fe4a1bba71e4643d4dbbf2 - - -] Unexpected error while running command. Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf cinder-rtstool delete iqn.2010-10.org.openstack:volume-ad12c1e0-6746-4bce-9b94-5df1cae4ee2a Exit code: 1 Stdout: '' Stderr: 'Traceback (most recent call last):
File "/bin/cinder-rtstool", line 238, in <module>
sys.exit(main())
File "/bin/cinder-rtstool", line 224, in main
delete(iqn)
File "/bin/cinder-rtstool", line 151, in delete
for x in rtsroot.storage_objects:
File "/usr/lib/python2.7/site-packages/rtslib/root.py", line 86, in _list_storage_objects
for so in StorageObject.all():
File "/usr/lib/python2.7/site-packages/rtslib/tcm.py", line 102, in all
yield cls.so_from_path(so_dir)
File "/usr/lib/python2.7/site-packages/rtslib/tcm.py", line 117, in so_from_path
return mapping[so_type](so_name)
File "/usr/lib/python2.7/site-packages/rtslib/tcm.py", line 678, in __init__
super(BlockStorageObject, self).__init__(name, \'lookup\')
File "/usr/lib/python2.7/site-packages/rtslib/tcm.py", line 48, in __init__
self._backstore = _Backstore(name, type(self), mode)
File "/usr/lib/python2.7/site-packages/rtslib/tcm.py", line 785, in __init__
(self._plugin, name))
rtslib.utils.RTSLibError: Storage object block/iqn.2010-10.org.openstack:volume-fb12df40-d1e3-4a4e-b47d-0162ce9573cc not found
The cleanest solution is synchronizing access in Cinder.
[1]: https://github.com/agrover/rtslib-fb/issues/36 |
|
