| 2011-10-09 12:56:11 |
Eric Dodemont |
bug |
|
|
added bug |
| 2011-10-09 12:57:56 |
Eric Dodemont |
description |
I cannot attach a volume to an instance if both are on different hosts.
This a problem which is specific to tgt which is used on Oneiric. I file that bug in Nova (Ubuntu) and not in Nova (Openstack), even though it is not an Ubuntu packaging problem.
Example:
SW: OpenStack 2011.3 (Diablo) -- Package: 2011.3-0ubuntu6
OS: Ubuntu 11.10 (Oneiric -- Beta2)
node1 (192.168.1.201): vol-00000004
node2 (192.168.1.202): i-00000004
root@node1:~# euca-attach-volume -i i-00000004 -d /dev/vdc vol-00000004
Error seen in nova-compute.log:
Error: iSCSI device not found at /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0
Explanation: nova-compute is waiting for a device to be configured as
/dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-1
but the device is configued as
/dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0
nova-compute waits for xxx-lun-0, but xxx-lun-1 is configured:
root@node2:/dev/disk/by-path# ls -l
lrwxrwxrwx 1 root root 9 2011-10-08 22:02 ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-1 -> ../../sdd
I guess this is a difference between the iet and tgt softwares used to configure iSCSI targets.
On Oneiric, tgt is used by default, and iet cannot be used because the kernel module is no more available.
In nova.conf:
--iscsi_helper=tgtadm
If I adapt the code from lun-0 to lun-1, everything works well:
diff /usr/share/pyshared/nova/volume/driver.py-patched /usr/share/pyshared/nova/volume/driver.py-unpatched
536c536
< mount_device = ("/dev/disk/by-path/ip-%s-iscsi-%s-lun-1" %
---
> mount_device = ("/dev/disk/by-path/ip-%s-iscsi-%s-lun-0" %
Complete nova-compute.log:
root@node2:~# tail -50 /var/log/nova/nova-compute.log
2011-10-08 22:02:31,179 DEBUG nova.rpc [-] received {u'_context_roles': [u'projectmanager'], u'_context_request_id': u'8911d1d6-66eb-48eb-ab9a-f7a387d858fa', u'_context_read_deleted': False, u'args':
{u'instance_id': 4, u'mountpoint': u'/dev/vdc', u'volume_id': 4}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'project-one',
u'_context_timestamp': u'2011-10-08T20:02:31.145628', u'_context_user_id': u'admin', u'method': u'attach_volume', u'_context_remote_address': u'192.168.1.201'} from (pid=1585) __call__
/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600
2011-10-08 22:02:31,180 DEBUG nova.rpc [-] unpacked context: {'user_id': u'admin', 'roles': [u'projectmanager'], 'timestamp': u'2011-10-08T20:02:31.145628', 'auth_token': None, 'msg_id': None,
'remote_address': u'192.168.1.201', 'strategy': u'noauth', 'is_admin': True, 'request_id': u'8911d1d6-66eb-48eb-ab9a-f7a387d858fa', 'project_id': u'project-one', 'read_deleted': False} from (pid=1585)
_unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646
2011-10-08 22:02:31,181 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: decorating: |<function attach_volume at 0x2946230>|
2011-10-08 22:02:31,181 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: arguments: |<nova.compute.manager.ComputeManager object at 0x21e5790>| |
<nova.rpc.impl_kombu.RpcContext object at 0x43fdf10>| |4|
2011-10-08 22:02:31,181 DEBUG nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] instance 4: getting locked state from (pid=1585) get_lock /usr/lib/python2.7/dist-
packages/nova/compute/manager.py:1165
2011-10-08 22:02:31,333 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: locked: |False|
2011-10-08 22:02:31,334 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: admin: |True|
2011-10-08 22:02:31,334 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: executing: |<function attach_volume at 0x2946230>|
2011-10-08 22:02:31,410 AUDIT nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] instance 4: attaching volume 4 to /dev/vdc
2011-10-08 22:02:31,447 WARNING nova.volume.driver [-] ISCSI provider_location not stored, using discovery
2011-10-08 22:02:31,447 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m discovery -t sendtargets -p node1 from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:31,489 DEBUG nova.volume.driver [-] ISCSI Discovery: Found 192.168.1.201:3260,1 iqn.2010-10.org.openstack:volume-00000004 from (pid=1585) _get_iscsi_properties /usr/lib/python2.7/dist-
packages/nova/volume/driver.py:479
2011-10-08 22:02:31,490 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --login from (pid=1585) execute
/usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,015 DEBUG nova.volume.driver [-] iscsiadm ('--login',): stdout=Logging in to [iface: default, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
Login to [iface: default, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]: successful
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:32,015 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --op update -n node.startup -v automatic from
(pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,033 DEBUG nova.volume.driver [-] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-
packages/nova/volume/driver.py:506
2011-10-08 22:02:32,033 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry. Try
number: 0
2011-10-08 22:02:32,034 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute
/usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,052 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:33,054 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry. Try
number: 1
2011-10-08 22:02:33,054 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute
/usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:33,073 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:37,077 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry. Try
number: 2
2011-10-08 22:02:37,077 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute
/usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:37,095 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:46,100 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 117, in decorated_function
(nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1242, in attach_volume
(nova.rpc): TRACE: volume_id)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/volume/manager.py", line 245, in setup_compute_volume
(nova.rpc): TRACE: path = self.driver.discover_volume(context, volume_ref)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/volume/driver.py", line 546, in discover_volume
(nova.rpc): TRACE: (mount_device))
(nova.rpc): TRACE: Error: iSCSI device not found at /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0
(nova.rpc): TRACE:
2011-10-08 22:03:52,889 INFO nova.compute.manager [-] Updating host status
2011-10-08 22:05:55,440 INFO nova.compute.manager [-] Updating host status
root@node2:/dev/disk/by-path# ls -l
total 0
lrwxrwxrwx 1 root root 9 2011-10-08 22:02 ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-1 -> ../../sdd
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0 -> ../../sda
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part5 -> ../../sda5
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:1:0 -> ../../sdb
lrwxrwxrwx 1 root root 10 2011-10-08 21:38 pci-0000:00:1f.2-scsi-0:0:1:0-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:0:0 -> ../../sdc
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:0:0-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:1:0 -> ../../sr0
root@node1:~# tgtadm --lld iscsi --op show --mode target
Target 1: iqn.2010-10.org.openstack:volume-00000004
System information:
Driver: iscsi
State: ready
I_T nexus information:
I_T nexus: 1
Initiator: iqn.1993-08.org.debian:01:63503b146b7
Connection: 0
IP Address: 192.168.1.202 <========= initiator (i-00000004) on node2 is connected to target (vol-00000004) on node1
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1 <==================================== LUN = 1 (and not 0)
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 1074 MB, Block size: 512
Online: Yes
Removable media: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/nova-volumes/volume-00000004
Backing store flags:
Account information:
ACL information:
ALL |
I cannot attach a volume to an instance if both are on different hosts.
This is a problem which is specific to tgt which is used on Oneiric. I file that bug in Nova (Ubuntu) and not in Nova (Openstack), even though it is not an Ubuntu packaging problem.
Example:
SW: OpenStack 2011.3 (Diablo) -- Package: 2011.3-0ubuntu6
OS: Ubuntu 11.10 (Oneiric -- Beta2)
node1 (192.168.1.201): vol-00000004
node2 (192.168.1.202): i-00000004
root@node1:~# euca-attach-volume -i i-00000004 -d /dev/vdc vol-00000004
Error seen in nova-compute.log:
Error: iSCSI device not found at /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0
Explanation: nova-compute is waiting for a device to be configured as
/dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-1
but the device is configued as
/dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0
nova-compute waits for xxx-lun-0, but xxx-lun-1 is configured:
root@node2:/dev/disk/by-path# ls -l
lrwxrwxrwx 1 root root 9 2011-10-08 22:02 ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-1 -> ../../sdd
I guess this is a difference between the iet and tgt softwares used to configure iSCSI targets.
On Oneiric, tgt is used by default, and iet cannot be used because the kernel module is no more available.
In nova.conf:
--iscsi_helper=tgtadm
If I adapt the code from lun-0 to lun-1, everything works well:
diff /usr/share/pyshared/nova/volume/driver.py-patched /usr/share/pyshared/nova/volume/driver.py-unpatched
536c536
< mount_device = ("/dev/disk/by-path/ip-%s-iscsi-%s-lun-1" %
---
> mount_device = ("/dev/disk/by-path/ip-%s-iscsi-%s-lun-0" %
Complete nova-compute.log:
root@node2:~# tail -50 /var/log/nova/nova-compute.log
2011-10-08 22:02:31,179 DEBUG nova.rpc [-] received {u'_context_roles': [u'projectmanager'], u'_context_request_id': u'8911d1d6-66eb-48eb-ab9a-f7a387d858fa', u'_context_read_deleted': False, u'args': {u'instance_id': 4, u'mountpoint': u'/dev/vdc', u'volume_id': 4}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'project-one', u'_context_timestamp': u'2011-10-08T20:02:31.145628', u'_context_user_id': u'admin', u'method': u'attach_volume', u'_context_remote_address': u'192.168.1.201'} from (pid=1585) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600
2011-10-08 22:02:31,180 DEBUG nova.rpc [-] unpacked context: {'user_id': u'admin', 'roles': [u'projectmanager'], 'timestamp': u'2011-10-08T20:02:31.145628', 'auth_token': None, 'msg_id': None, 'remote_address': u'192.168.1.201', 'strategy': u'noauth', 'is_admin': True, 'request_id': u'8911d1d6-66eb-48eb-ab9a-f7a387d858fa', 'project_id': u'project-one', 'read_deleted': False} from (pid=1585) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646
2011-10-08 22:02:31,181 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: decorating: |<function attach_volume at 0x2946230>|
2011-10-08 22:02:31,181 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: arguments: |<nova.compute.manager.ComputeManager object at 0x21e5790>| |<nova.rpc.impl_kombu.RpcContext object at 0x43fdf10>| |4|
2011-10-08 22:02:31,181 DEBUG nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] instance 4: getting locked state from (pid=1585) get_lock /usr/lib/python2.7/dist-packages/nova/compute/manager.py:1165
2011-10-08 22:02:31,333 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: locked: |False|
2011-10-08 22:02:31,334 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: admin: |True|
2011-10-08 22:02:31,334 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: executing: |<function attach_volume at 0x2946230>|
2011-10-08 22:02:31,410 AUDIT nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] instance 4: attaching volume 4 to /dev/vdc
2011-10-08 22:02:31,447 WARNING nova.volume.driver [-] ISCSI provider_location not stored, using discovery
2011-10-08 22:02:31,447 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m discovery -t sendtargets -p node1 from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:31,489 DEBUG nova.volume.driver [-] ISCSI Discovery: Found 192.168.1.201:3260,1 iqn.2010-10.org.openstack:volume-00000004 from (pid=1585) _get_iscsi_properties /usr/lib/python2.7/dist-packages/nova/volume/driver.py:479
2011-10-08 22:02:31,490 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --login from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,015 DEBUG nova.volume.driver [-] iscsiadm ('--login',): stdout=Logging in to [iface: default, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
Login to [iface: default, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]: successful
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:32,015 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --op update -n node.startup -v automatic from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,033 DEBUG nova.volume.driver [-] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:32,033 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry. Try number: 0
2011-10-08 22:02:32,034 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,052 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:33,054 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry. Try number: 1
2011-10-08 22:02:33,054 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:33,073 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:37,077 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry. Try number: 2
2011-10-08 22:02:37,077 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:37,095 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:46,100 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 117, in decorated_function
(nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1242, in attach_volume
(nova.rpc): TRACE: volume_id)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/volume/manager.py", line 245, in setup_compute_volume
(nova.rpc): TRACE: path = self.driver.discover_volume(context, volume_ref)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/volume/driver.py", line 546, in discover_volume
(nova.rpc): TRACE: (mount_device))
(nova.rpc): TRACE: Error: iSCSI device not found at /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0
(nova.rpc): TRACE:
2011-10-08 22:03:52,889 INFO nova.compute.manager [-] Updating host status
2011-10-08 22:05:55,440 INFO nova.compute.manager [-] Updating host status
root@node2:/dev/disk/by-path# ls -l
total 0
lrwxrwxrwx 1 root root 9 2011-10-08 22:02 ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-1 -> ../../sdd
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0 -> ../../sda
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part5 -> ../../sda5
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:1:0 -> ../../sdb
lrwxrwxrwx 1 root root 10 2011-10-08 21:38 pci-0000:00:1f.2-scsi-0:0:1:0-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:0:0 -> ../../sdc
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:0:0-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:1:0 -> ../../sr0
root@node1:~# tgtadm --lld iscsi --op show --mode target
Target 1: iqn.2010-10.org.openstack:volume-00000004
System information:
Driver: iscsi
State: ready
I_T nexus information:
I_T nexus: 1
Initiator: iqn.1993-08.org.debian:01:63503b146b7
Connection: 0
IP Address: 192.168.1.202 <========= initiator (i-00000004) on node2 is connected to target (vol-00000004) on node1
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1 <==================================== LUN = 1 (and not 0)
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 1074 MB, Block size: 512
Online: Yes
Removable media: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/nova-volumes/volume-00000004
Backing store flags:
Account information:
ACL information:
ALL |
|
| 2011-10-09 13:00:12 |
Eric Dodemont |
description |
I cannot attach a volume to an instance if both are on different hosts.
This is a problem which is specific to tgt which is used on Oneiric. I file that bug in Nova (Ubuntu) and not in Nova (Openstack), even though it is not an Ubuntu packaging problem.
Example:
SW: OpenStack 2011.3 (Diablo) -- Package: 2011.3-0ubuntu6
OS: Ubuntu 11.10 (Oneiric -- Beta2)
node1 (192.168.1.201): vol-00000004
node2 (192.168.1.202): i-00000004
root@node1:~# euca-attach-volume -i i-00000004 -d /dev/vdc vol-00000004
Error seen in nova-compute.log:
Error: iSCSI device not found at /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0
Explanation: nova-compute is waiting for a device to be configured as
/dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-1
but the device is configued as
/dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0
nova-compute waits for xxx-lun-0, but xxx-lun-1 is configured:
root@node2:/dev/disk/by-path# ls -l
lrwxrwxrwx 1 root root 9 2011-10-08 22:02 ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-1 -> ../../sdd
I guess this is a difference between the iet and tgt softwares used to configure iSCSI targets.
On Oneiric, tgt is used by default, and iet cannot be used because the kernel module is no more available.
In nova.conf:
--iscsi_helper=tgtadm
If I adapt the code from lun-0 to lun-1, everything works well:
diff /usr/share/pyshared/nova/volume/driver.py-patched /usr/share/pyshared/nova/volume/driver.py-unpatched
536c536
< mount_device = ("/dev/disk/by-path/ip-%s-iscsi-%s-lun-1" %
---
> mount_device = ("/dev/disk/by-path/ip-%s-iscsi-%s-lun-0" %
Complete nova-compute.log:
root@node2:~# tail -50 /var/log/nova/nova-compute.log
2011-10-08 22:02:31,179 DEBUG nova.rpc [-] received {u'_context_roles': [u'projectmanager'], u'_context_request_id': u'8911d1d6-66eb-48eb-ab9a-f7a387d858fa', u'_context_read_deleted': False, u'args': {u'instance_id': 4, u'mountpoint': u'/dev/vdc', u'volume_id': 4}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'project-one', u'_context_timestamp': u'2011-10-08T20:02:31.145628', u'_context_user_id': u'admin', u'method': u'attach_volume', u'_context_remote_address': u'192.168.1.201'} from (pid=1585) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600
2011-10-08 22:02:31,180 DEBUG nova.rpc [-] unpacked context: {'user_id': u'admin', 'roles': [u'projectmanager'], 'timestamp': u'2011-10-08T20:02:31.145628', 'auth_token': None, 'msg_id': None, 'remote_address': u'192.168.1.201', 'strategy': u'noauth', 'is_admin': True, 'request_id': u'8911d1d6-66eb-48eb-ab9a-f7a387d858fa', 'project_id': u'project-one', 'read_deleted': False} from (pid=1585) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646
2011-10-08 22:02:31,181 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: decorating: |<function attach_volume at 0x2946230>|
2011-10-08 22:02:31,181 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: arguments: |<nova.compute.manager.ComputeManager object at 0x21e5790>| |<nova.rpc.impl_kombu.RpcContext object at 0x43fdf10>| |4|
2011-10-08 22:02:31,181 DEBUG nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] instance 4: getting locked state from (pid=1585) get_lock /usr/lib/python2.7/dist-packages/nova/compute/manager.py:1165
2011-10-08 22:02:31,333 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: locked: |False|
2011-10-08 22:02:31,334 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: admin: |True|
2011-10-08 22:02:31,334 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: executing: |<function attach_volume at 0x2946230>|
2011-10-08 22:02:31,410 AUDIT nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] instance 4: attaching volume 4 to /dev/vdc
2011-10-08 22:02:31,447 WARNING nova.volume.driver [-] ISCSI provider_location not stored, using discovery
2011-10-08 22:02:31,447 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m discovery -t sendtargets -p node1 from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:31,489 DEBUG nova.volume.driver [-] ISCSI Discovery: Found 192.168.1.201:3260,1 iqn.2010-10.org.openstack:volume-00000004 from (pid=1585) _get_iscsi_properties /usr/lib/python2.7/dist-packages/nova/volume/driver.py:479
2011-10-08 22:02:31,490 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --login from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,015 DEBUG nova.volume.driver [-] iscsiadm ('--login',): stdout=Logging in to [iface: default, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
Login to [iface: default, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]: successful
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:32,015 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --op update -n node.startup -v automatic from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,033 DEBUG nova.volume.driver [-] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:32,033 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry. Try number: 0
2011-10-08 22:02:32,034 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,052 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:33,054 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry. Try number: 1
2011-10-08 22:02:33,054 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:33,073 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:37,077 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry. Try number: 2
2011-10-08 22:02:37,077 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:37,095 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:46,100 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 117, in decorated_function
(nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1242, in attach_volume
(nova.rpc): TRACE: volume_id)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/volume/manager.py", line 245, in setup_compute_volume
(nova.rpc): TRACE: path = self.driver.discover_volume(context, volume_ref)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/volume/driver.py", line 546, in discover_volume
(nova.rpc): TRACE: (mount_device))
(nova.rpc): TRACE: Error: iSCSI device not found at /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0
(nova.rpc): TRACE:
2011-10-08 22:03:52,889 INFO nova.compute.manager [-] Updating host status
2011-10-08 22:05:55,440 INFO nova.compute.manager [-] Updating host status
root@node2:/dev/disk/by-path# ls -l
total 0
lrwxrwxrwx 1 root root 9 2011-10-08 22:02 ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-1 -> ../../sdd
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0 -> ../../sda
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part5 -> ../../sda5
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:1:0 -> ../../sdb
lrwxrwxrwx 1 root root 10 2011-10-08 21:38 pci-0000:00:1f.2-scsi-0:0:1:0-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:0:0 -> ../../sdc
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:0:0-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:1:0 -> ../../sr0
root@node1:~# tgtadm --lld iscsi --op show --mode target
Target 1: iqn.2010-10.org.openstack:volume-00000004
System information:
Driver: iscsi
State: ready
I_T nexus information:
I_T nexus: 1
Initiator: iqn.1993-08.org.debian:01:63503b146b7
Connection: 0
IP Address: 192.168.1.202 <========= initiator (i-00000004) on node2 is connected to target (vol-00000004) on node1
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1 <==================================== LUN = 1 (and not 0)
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 1074 MB, Block size: 512
Online: Yes
Removable media: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/nova-volumes/volume-00000004
Backing store flags:
Account information:
ACL information:
ALL |
I cannot attach a volume to an instance if both are on different hosts.
This is a problem which is specific to tgt which is used on Oneiric. I file that bug in Nova (Ubuntu) and not in Nova (Openstack), even though it is not an Ubuntu packaging problem.
Example:
SW: OpenStack 2011.3 (Diablo) -- Package: 2011.3-0ubuntu6
OS: Ubuntu 11.10 (Oneiric -- Beta2)
node1 (192.168.1.201): vol-00000004
node2 (192.168.1.202): i-00000004
root@node1:~# euca-attach-volume -i i-00000004 -d /dev/vdc vol-00000004
Error seen in nova-compute.log:
Error: iSCSI device not found at /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0
Explanation:
nova-compute is waiting for a device to be configured as:
/dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-1
but the device is configured as:
/dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0
nova-compute waits for xxx-lun-0, but xxx-lun-1 is configured:
root@node2:/dev/disk/by-path# ls -l
lrwxrwxrwx 1 root root 9 2011-10-08 22:02 ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-1 -> ../../sdd
I guess this is a difference between the iet and tgt softwares used to configure iSCSI targets.
On Oneiric, tgt is used by default, and iet cannot be used because the kernel module is no more available.
In nova.conf:
--iscsi_helper=tgtadm
If I adapt the code from lun-0 to lun-1, everything works well:
diff /usr/share/pyshared/nova/volume/driver.py-patched /usr/share/pyshared/nova/volume/driver.py-unpatched
536c536
< mount_device = ("/dev/disk/by-path/ip-%s-iscsi-%s-lun-1" %
---
> mount_device = ("/dev/disk/by-path/ip-%s-iscsi-%s-lun-0" %
Complete nova-compute.log:
root@node2:~# tail -50 /var/log/nova/nova-compute.log
2011-10-08 22:02:31,179 DEBUG nova.rpc [-] received {u'_context_roles': [u'projectmanager'], u'_context_request_id': u'8911d1d6-66eb-48eb-ab9a-f7a387d858fa', u'_context_read_deleted': False, u'args': {u'instance_id': 4, u'mountpoint': u'/dev/vdc', u'volume_id': 4}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'project-one', u'_context_timestamp': u'2011-10-08T20:02:31.145628', u'_context_user_id': u'admin', u'method': u'attach_volume', u'_context_remote_address': u'192.168.1.201'} from (pid=1585) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600
2011-10-08 22:02:31,180 DEBUG nova.rpc [-] unpacked context: {'user_id': u'admin', 'roles': [u'projectmanager'], 'timestamp': u'2011-10-08T20:02:31.145628', 'auth_token': None, 'msg_id': None, 'remote_address': u'192.168.1.201', 'strategy': u'noauth', 'is_admin': True, 'request_id': u'8911d1d6-66eb-48eb-ab9a-f7a387d858fa', 'project_id': u'project-one', 'read_deleted': False} from (pid=1585) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646
2011-10-08 22:02:31,181 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: decorating: |<function attach_volume at 0x2946230>|
2011-10-08 22:02:31,181 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: arguments: |<nova.compute.manager.ComputeManager object at 0x21e5790>| |<nova.rpc.impl_kombu.RpcContext object at 0x43fdf10>| |4|
2011-10-08 22:02:31,181 DEBUG nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] instance 4: getting locked state from (pid=1585) get_lock /usr/lib/python2.7/dist-packages/nova/compute/manager.py:1165
2011-10-08 22:02:31,333 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: locked: |False|
2011-10-08 22:02:31,334 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: admin: |True|
2011-10-08 22:02:31,334 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: executing: |<function attach_volume at 0x2946230>|
2011-10-08 22:02:31,410 AUDIT nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] instance 4: attaching volume 4 to /dev/vdc
2011-10-08 22:02:31,447 WARNING nova.volume.driver [-] ISCSI provider_location not stored, using discovery
2011-10-08 22:02:31,447 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m discovery -t sendtargets -p node1 from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:31,489 DEBUG nova.volume.driver [-] ISCSI Discovery: Found 192.168.1.201:3260,1 iqn.2010-10.org.openstack:volume-00000004 from (pid=1585) _get_iscsi_properties /usr/lib/python2.7/dist-packages/nova/volume/driver.py:479
2011-10-08 22:02:31,490 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --login from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,015 DEBUG nova.volume.driver [-] iscsiadm ('--login',): stdout=Logging in to [iface: default, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
Login to [iface: default, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]: successful
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:32,015 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --op update -n node.startup -v automatic from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,033 DEBUG nova.volume.driver [-] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:32,033 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry. Try number: 0
2011-10-08 22:02:32,034 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,052 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:33,054 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry. Try number: 1
2011-10-08 22:02:33,054 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:33,073 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:37,077 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry. Try number: 2
2011-10-08 22:02:37,077 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:37,095 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:46,100 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 117, in decorated_function
(nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1242, in attach_volume
(nova.rpc): TRACE: volume_id)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/volume/manager.py", line 245, in setup_compute_volume
(nova.rpc): TRACE: path = self.driver.discover_volume(context, volume_ref)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/volume/driver.py", line 546, in discover_volume
(nova.rpc): TRACE: (mount_device))
(nova.rpc): TRACE: Error: iSCSI device not found at /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0
(nova.rpc): TRACE:
2011-10-08 22:03:52,889 INFO nova.compute.manager [-] Updating host status
2011-10-08 22:05:55,440 INFO nova.compute.manager [-] Updating host status
root@node2:/dev/disk/by-path# ls -l
total 0
lrwxrwxrwx 1 root root 9 2011-10-08 22:02 ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-1 -> ../../sdd
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0 -> ../../sda
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part5 -> ../../sda5
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:1:0 -> ../../sdb
lrwxrwxrwx 1 root root 10 2011-10-08 21:38 pci-0000:00:1f.2-scsi-0:0:1:0-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:0:0 -> ../../sdc
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:0:0-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:1:0 -> ../../sr0
root@node1:~# tgtadm --lld iscsi --op show --mode target
Target 1: iqn.2010-10.org.openstack:volume-00000004
System information:
Driver: iscsi
State: ready
I_T nexus information:
I_T nexus: 1
Initiator: iqn.1993-08.org.debian:01:63503b146b7
Connection: 0
IP Address: 192.168.1.202 <========= initiator (i-00000004) on node2 is connected to target (vol-00000004) on node1
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1 <==================================== LUN = 1 (and not 0)
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 1074 MB, Block size: 512
Online: Yes
Removable media: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/nova-volumes/volume-00000004
Backing store flags:
Account information:
ACL information:
ALL |
|
| 2011-10-09 16:55:08 |
Dave Walker |
nova (Ubuntu): milestone |
|
oneiric-updates |
|
| 2011-10-09 16:55:18 |
Dave Walker |
nova (Ubuntu): assignee |
|
Chuck Short (zulcss) |
|
| 2011-10-09 16:55:31 |
Dave Walker |
nova (Ubuntu): importance |
Undecided |
High |
|
| 2011-10-09 16:55:34 |
Dave Walker |
nova (Ubuntu): status |
New |
Triaged |
|
| 2011-10-09 16:55:45 |
Dave Walker |
tags |
iscsi tgt volume |
iscsi server-o-rs tgt volume |
|
| 2011-10-09 19:35:39 |
Eric Dodemont |
description |
I cannot attach a volume to an instance if both are on different hosts.
This is a problem which is specific to tgt which is used on Oneiric. I file that bug in Nova (Ubuntu) and not in Nova (Openstack), even though it is not an Ubuntu packaging problem.
Example:
SW: OpenStack 2011.3 (Diablo) -- Package: 2011.3-0ubuntu6
OS: Ubuntu 11.10 (Oneiric -- Beta2)
node1 (192.168.1.201): vol-00000004
node2 (192.168.1.202): i-00000004
root@node1:~# euca-attach-volume -i i-00000004 -d /dev/vdc vol-00000004
Error seen in nova-compute.log:
Error: iSCSI device not found at /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0
Explanation:
nova-compute is waiting for a device to be configured as:
/dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-1
but the device is configured as:
/dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0
nova-compute waits for xxx-lun-0, but xxx-lun-1 is configured:
root@node2:/dev/disk/by-path# ls -l
lrwxrwxrwx 1 root root 9 2011-10-08 22:02 ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-1 -> ../../sdd
I guess this is a difference between the iet and tgt softwares used to configure iSCSI targets.
On Oneiric, tgt is used by default, and iet cannot be used because the kernel module is no more available.
In nova.conf:
--iscsi_helper=tgtadm
If I adapt the code from lun-0 to lun-1, everything works well:
diff /usr/share/pyshared/nova/volume/driver.py-patched /usr/share/pyshared/nova/volume/driver.py-unpatched
536c536
< mount_device = ("/dev/disk/by-path/ip-%s-iscsi-%s-lun-1" %
---
> mount_device = ("/dev/disk/by-path/ip-%s-iscsi-%s-lun-0" %
Complete nova-compute.log:
root@node2:~# tail -50 /var/log/nova/nova-compute.log
2011-10-08 22:02:31,179 DEBUG nova.rpc [-] received {u'_context_roles': [u'projectmanager'], u'_context_request_id': u'8911d1d6-66eb-48eb-ab9a-f7a387d858fa', u'_context_read_deleted': False, u'args': {u'instance_id': 4, u'mountpoint': u'/dev/vdc', u'volume_id': 4}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'project-one', u'_context_timestamp': u'2011-10-08T20:02:31.145628', u'_context_user_id': u'admin', u'method': u'attach_volume', u'_context_remote_address': u'192.168.1.201'} from (pid=1585) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600
2011-10-08 22:02:31,180 DEBUG nova.rpc [-] unpacked context: {'user_id': u'admin', 'roles': [u'projectmanager'], 'timestamp': u'2011-10-08T20:02:31.145628', 'auth_token': None, 'msg_id': None, 'remote_address': u'192.168.1.201', 'strategy': u'noauth', 'is_admin': True, 'request_id': u'8911d1d6-66eb-48eb-ab9a-f7a387d858fa', 'project_id': u'project-one', 'read_deleted': False} from (pid=1585) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646
2011-10-08 22:02:31,181 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: decorating: |<function attach_volume at 0x2946230>|
2011-10-08 22:02:31,181 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: arguments: |<nova.compute.manager.ComputeManager object at 0x21e5790>| |<nova.rpc.impl_kombu.RpcContext object at 0x43fdf10>| |4|
2011-10-08 22:02:31,181 DEBUG nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] instance 4: getting locked state from (pid=1585) get_lock /usr/lib/python2.7/dist-packages/nova/compute/manager.py:1165
2011-10-08 22:02:31,333 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: locked: |False|
2011-10-08 22:02:31,334 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: admin: |True|
2011-10-08 22:02:31,334 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: executing: |<function attach_volume at 0x2946230>|
2011-10-08 22:02:31,410 AUDIT nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] instance 4: attaching volume 4 to /dev/vdc
2011-10-08 22:02:31,447 WARNING nova.volume.driver [-] ISCSI provider_location not stored, using discovery
2011-10-08 22:02:31,447 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m discovery -t sendtargets -p node1 from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:31,489 DEBUG nova.volume.driver [-] ISCSI Discovery: Found 192.168.1.201:3260,1 iqn.2010-10.org.openstack:volume-00000004 from (pid=1585) _get_iscsi_properties /usr/lib/python2.7/dist-packages/nova/volume/driver.py:479
2011-10-08 22:02:31,490 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --login from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,015 DEBUG nova.volume.driver [-] iscsiadm ('--login',): stdout=Logging in to [iface: default, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
Login to [iface: default, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]: successful
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:32,015 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --op update -n node.startup -v automatic from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,033 DEBUG nova.volume.driver [-] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:32,033 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry. Try number: 0
2011-10-08 22:02:32,034 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,052 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:33,054 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry. Try number: 1
2011-10-08 22:02:33,054 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:33,073 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:37,077 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry. Try number: 2
2011-10-08 22:02:37,077 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:37,095 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:46,100 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 117, in decorated_function
(nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1242, in attach_volume
(nova.rpc): TRACE: volume_id)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/volume/manager.py", line 245, in setup_compute_volume
(nova.rpc): TRACE: path = self.driver.discover_volume(context, volume_ref)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/volume/driver.py", line 546, in discover_volume
(nova.rpc): TRACE: (mount_device))
(nova.rpc): TRACE: Error: iSCSI device not found at /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0
(nova.rpc): TRACE:
2011-10-08 22:03:52,889 INFO nova.compute.manager [-] Updating host status
2011-10-08 22:05:55,440 INFO nova.compute.manager [-] Updating host status
root@node2:/dev/disk/by-path# ls -l
total 0
lrwxrwxrwx 1 root root 9 2011-10-08 22:02 ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-1 -> ../../sdd
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0 -> ../../sda
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part5 -> ../../sda5
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:1:0 -> ../../sdb
lrwxrwxrwx 1 root root 10 2011-10-08 21:38 pci-0000:00:1f.2-scsi-0:0:1:0-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:0:0 -> ../../sdc
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:0:0-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:1:0 -> ../../sr0
root@node1:~# tgtadm --lld iscsi --op show --mode target
Target 1: iqn.2010-10.org.openstack:volume-00000004
System information:
Driver: iscsi
State: ready
I_T nexus information:
I_T nexus: 1
Initiator: iqn.1993-08.org.debian:01:63503b146b7
Connection: 0
IP Address: 192.168.1.202 <========= initiator (i-00000004) on node2 is connected to target (vol-00000004) on node1
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1 <==================================== LUN = 1 (and not 0)
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 1074 MB, Block size: 512
Online: Yes
Removable media: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/nova-volumes/volume-00000004
Backing store flags:
Account information:
ACL information:
ALL |
I cannot attach a volume to an instance if both are on different hosts.
This is a problem which is specific to tgt which is used on Oneiric. I file that bug in Nova (Ubuntu) and not in Nova (Openstack), even though it is not an Ubuntu packaging problem.
Example:
SW: OpenStack 2011.3 (Diablo) -- Package: 2011.3-0ubuntu6
OS: Ubuntu 11.10 (Oneiric -- Beta2)
node1 (192.168.1.201): vol-00000004
node2 (192.168.1.202): i-00000004
root@node1:~# euca-attach-volume -i i-00000004 -d /dev/vdc vol-00000004
Error seen in nova-compute.log:
Error: iSCSI device not found at /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0
Explanation:
nova-compute is waiting for a device to be configured as:
/dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0
but the device is configured as:
/dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-1
nova-compute waits for xxx-lun-0, but xxx-lun-1 is configured:
root@node2:/dev/disk/by-path# ls -l
lrwxrwxrwx 1 root root 9 2011-10-08 22:02 ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-1 -> ../../sdd
I guess this is a difference between the iet and tgt softwares used to configure iSCSI targets.
On Oneiric, tgt is used by default, and iet cannot be used because the kernel module is no more available.
In nova.conf:
--iscsi_helper=tgtadm
If I adapt the code from lun-0 to lun-1, everything works well:
diff /usr/share/pyshared/nova/volume/driver.py-patched /usr/share/pyshared/nova/volume/driver.py-unpatched
536c536
< mount_device = ("/dev/disk/by-path/ip-%s-iscsi-%s-lun-1" %
---
> mount_device = ("/dev/disk/by-path/ip-%s-iscsi-%s-lun-0" %
Complete nova-compute.log:
root@node2:~# tail -50 /var/log/nova/nova-compute.log
2011-10-08 22:02:31,179 DEBUG nova.rpc [-] received {u'_context_roles': [u'projectmanager'], u'_context_request_id': u'8911d1d6-66eb-48eb-ab9a-f7a387d858fa', u'_context_read_deleted': False, u'args': {u'instance_id': 4, u'mountpoint': u'/dev/vdc', u'volume_id': 4}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'project-one', u'_context_timestamp': u'2011-10-08T20:02:31.145628', u'_context_user_id': u'admin', u'method': u'attach_volume', u'_context_remote_address': u'192.168.1.201'} from (pid=1585) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600
2011-10-08 22:02:31,180 DEBUG nova.rpc [-] unpacked context: {'user_id': u'admin', 'roles': [u'projectmanager'], 'timestamp': u'2011-10-08T20:02:31.145628', 'auth_token': None, 'msg_id': None, 'remote_address': u'192.168.1.201', 'strategy': u'noauth', 'is_admin': True, 'request_id': u'8911d1d6-66eb-48eb-ab9a-f7a387d858fa', 'project_id': u'project-one', 'read_deleted': False} from (pid=1585) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646
2011-10-08 22:02:31,181 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: decorating: |<function attach_volume at 0x2946230>|
2011-10-08 22:02:31,181 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: arguments: |<nova.compute.manager.ComputeManager object at 0x21e5790>| |<nova.rpc.impl_kombu.RpcContext object at 0x43fdf10>| |4|
2011-10-08 22:02:31,181 DEBUG nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] instance 4: getting locked state from (pid=1585) get_lock /usr/lib/python2.7/dist-packages/nova/compute/manager.py:1165
2011-10-08 22:02:31,333 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: locked: |False|
2011-10-08 22:02:31,334 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: admin: |True|
2011-10-08 22:02:31,334 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: executing: |<function attach_volume at 0x2946230>|
2011-10-08 22:02:31,410 AUDIT nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] instance 4: attaching volume 4 to /dev/vdc
2011-10-08 22:02:31,447 WARNING nova.volume.driver [-] ISCSI provider_location not stored, using discovery
2011-10-08 22:02:31,447 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m discovery -t sendtargets -p node1 from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:31,489 DEBUG nova.volume.driver [-] ISCSI Discovery: Found 192.168.1.201:3260,1 iqn.2010-10.org.openstack:volume-00000004 from (pid=1585) _get_iscsi_properties /usr/lib/python2.7/dist-packages/nova/volume/driver.py:479
2011-10-08 22:02:31,490 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --login from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,015 DEBUG nova.volume.driver [-] iscsiadm ('--login',): stdout=Logging in to [iface: default, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
Login to [iface: default, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]: successful
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:32,015 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --op update -n node.startup -v automatic from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,033 DEBUG nova.volume.driver [-] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:32,033 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry. Try number: 0
2011-10-08 22:02:32,034 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,052 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:33,054 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry. Try number: 1
2011-10-08 22:02:33,054 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:33,073 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:37,077 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry. Try number: 2
2011-10-08 22:02:37,077 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:37,095 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:46,100 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 117, in decorated_function
(nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1242, in attach_volume
(nova.rpc): TRACE: volume_id)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/volume/manager.py", line 245, in setup_compute_volume
(nova.rpc): TRACE: path = self.driver.discover_volume(context, volume_ref)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/volume/driver.py", line 546, in discover_volume
(nova.rpc): TRACE: (mount_device))
(nova.rpc): TRACE: Error: iSCSI device not found at /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0
(nova.rpc): TRACE:
2011-10-08 22:03:52,889 INFO nova.compute.manager [-] Updating host status
2011-10-08 22:05:55,440 INFO nova.compute.manager [-] Updating host status
root@node2:/dev/disk/by-path# ls -l
total 0
lrwxrwxrwx 1 root root 9 2011-10-08 22:02 ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-1 -> ../../sdd
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0 -> ../../sda
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part5 -> ../../sda5
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:1:0 -> ../../sdb
lrwxrwxrwx 1 root root 10 2011-10-08 21:38 pci-0000:00:1f.2-scsi-0:0:1:0-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:0:0 -> ../../sdc
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:0:0-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:1:0 -> ../../sr0
root@node1:~# tgtadm --lld iscsi --op show --mode target
Target 1: iqn.2010-10.org.openstack:volume-00000004
System information:
Driver: iscsi
State: ready
I_T nexus information:
I_T nexus: 1
Initiator: iqn.1993-08.org.debian:01:63503b146b7
Connection: 0
IP Address: 192.168.1.202 <========= initiator (i-00000004) on node2 is connected to target (vol-00000004) on node1
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1 <==================================== LUN = 1 (and not 0)
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 1074 MB, Block size: 512
Online: Yes
Removable media: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/nova-volumes/volume-00000004
Backing store flags:
Account information:
ACL information:
ALL |
|
| 2011-10-12 17:58:24 |
Launchpad Janitor |
branch linked |
|
lp:~ubuntu-server-dev/nova/diablo |
|
| 2011-10-14 08:16:37 |
Martin Pitt |
nominated for series |
|
Ubuntu Oneiric |
|
| 2011-10-14 08:16:37 |
Martin Pitt |
bug task added |
|
nova (Ubuntu Oneiric) |
|
| 2011-10-14 08:17:52 |
Martin Pitt |
nova (Ubuntu Oneiric): status |
New |
Fix Committed |
|
| 2011-10-14 08:17:53 |
Martin Pitt |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2011-10-14 08:17:54 |
Martin Pitt |
bug |
|
|
added subscriber SRU Verification |
| 2011-10-14 08:17:58 |
Martin Pitt |
tags |
iscsi server-o-rs tgt volume |
iscsi server-o-rs tgt verification-needed volume |
|
| 2011-10-16 19:59:36 |
Dave Walker |
nominated for series |
|
Ubuntu Precise |
|
| 2011-10-16 19:59:36 |
Dave Walker |
bug task added |
|
nova (Ubuntu Precise) |
|
| 2011-10-16 19:59:49 |
Dave Walker |
nova (Ubuntu Oneiric): milestone |
|
oneiric-updates |
|
| 2011-10-16 19:59:52 |
Dave Walker |
nova (Ubuntu Precise): milestone |
oneiric-updates |
precise-alpha-1 |
|
| 2011-10-16 20:00:02 |
Dave Walker |
bug |
|
|
added subscriber ubuntu-server-release-tracking |
| 2011-10-16 22:05:11 |
Launchpad Janitor |
nova (Ubuntu Precise): status |
Triaged |
Fix Released |
|
| 2011-10-18 15:53:08 |
Martin Pitt |
tags |
iscsi server-o-rs tgt verification-needed volume |
iscsi server-o-rs tgt verification-failed volume |
|
| 2011-10-18 20:56:25 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/nova |
|
| 2011-10-18 20:56:31 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/oneiric-proposed/nova |
|
| 2011-10-25 03:37:42 |
livemoon |
bug |
|
|
added subscriber deadsun |
| 2011-10-25 05:00:33 |
livemoon |
nova (Ubuntu Oneiric): status |
Fix Committed |
Incomplete |
|
| 2011-10-26 17:22:27 |
Chuck Short |
bug task added |
|
nova |
|
| 2011-10-26 17:22:39 |
Chuck Short |
nova (Ubuntu Precise): status |
Fix Released |
In Progress |
|
| 2011-11-03 18:12:32 |
Nathan Koterba |
bug |
|
|
added subscriber Nathan Koterba |
| 2011-11-05 15:10:58 |
Albert Damen |
bug |
|
|
added subscriber Albert Damen |
| 2011-12-01 22:23:26 |
Dave Walker |
nova (Ubuntu): milestone |
precise-alpha-1 |
precise-alpha-2 |
|
| 2011-12-02 14:02:02 |
Thierry Carrez |
nova: importance |
Undecided |
Medium |
|
| 2011-12-02 14:02:02 |
Thierry Carrez |
nova: status |
New |
Confirmed |
|
| 2011-12-02 14:03:16 |
Thierry Carrez |
nova: status |
Confirmed |
In Progress |
|
| 2011-12-02 14:03:16 |
Thierry Carrez |
nova: assignee |
|
Chuck Short (zulcss) |
|
| 2011-12-16 18:38:24 |
Launchpad Janitor |
branch linked |
|
lp:~ubuntu-server-dev/nova/essex |
|
| 2011-12-16 18:40:14 |
Launchpad Janitor |
nova (Ubuntu Precise): status |
In Progress |
Fix Released |
|
| 2012-01-13 10:12:58 |
Mark McLoughlin |
nova: status |
In Progress |
Fix Committed |
|
| 2012-01-25 09:54:03 |
Thierry Carrez |
nova: status |
Fix Committed |
Fix Released |
|
| 2012-01-25 09:54:03 |
Thierry Carrez |
nova: milestone |
|
essex-3 |
|
| 2012-03-09 14:27:09 |
Tom Ellis |
bug |
|
|
added subscriber Tom Ellis |
| 2012-04-05 09:59:58 |
Thierry Carrez |
nova: milestone |
essex-3 |
2012.1 |
|
| 2013-01-22 16:06:46 |
Launchpad Janitor |
branch linked |
|
lp:~openstack-ubuntu-testing/nova/raring-grizzly |
|
| 2014-12-03 09:34:55 |
Rolf Leggewie |
nova (Ubuntu Oneiric): status |
Incomplete |
Won't Fix |
|
