Cannot attach volumes to instances if tgt is used
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Medium
|
Chuck Short | ||
nova (Ubuntu) |
Fix Released
|
High
|
Chuck Short | ||
Oneiric |
Won't Fix
|
Undecided
|
Unassigned | ||
Precise |
Fix Released
|
High
|
Chuck Short |
Bug Description
I cannot attach a volume to an instance if both are on different hosts.
This is a problem which is specific to tgt which is used on Oneiric. I file that bug in Nova (Ubuntu) and not in Nova (Openstack), even though it is not an Ubuntu packaging problem.
Example:
SW: OpenStack 2011.3 (Diablo) -- Package: 2011.3-0ubuntu6
OS: Ubuntu 11.10 (Oneiric -- Beta2)
node1 (192.168.1.201): vol-00000004
node2 (192.168.1.202): i-00000004
root@node1:~# euca-attach-volume -i i-00000004 -d /dev/vdc vol-00000004
Error seen in nova-compute.log:
Error: iSCSI device not found at /dev/disk/
Explanation:
nova-compute is waiting for a device to be configured as:
/dev/disk/
but the device is configured as:
/dev/disk/
nova-compute waits for xxx-lun-0, but xxx-lun-1 is configured:
root@node2:
lrwxrwxrwx 1 root root 9 2011-10-08 22:02 ip-192.
I guess this is a difference between the iet and tgt softwares used to configure iSCSI targets.
On Oneiric, tgt is used by default, and iet cannot be used because the kernel module is no more available.
In nova.conf:
--iscsi_
If I adapt the code from lun-0 to lun-1, everything works well:
diff /usr/share/
536c536
< mount_device = ("/dev/
---
> mount_device = ("/dev/
Complete nova-compute.log:
root@node2:~# tail -50 /var/log/
2011-10-08 22:02:31,179 DEBUG nova.rpc [-] received {u'_context_roles': [u'projectmanag
2011-10-08 22:02:31,180 DEBUG nova.rpc [-] unpacked context: {'user_id': u'admin', 'roles': [u'projectmanag
2011-10-08 22:02:31,181 INFO nova.compute.
2011-10-08 22:02:31,181 INFO nova.compute.
2011-10-08 22:02:31,181 DEBUG nova.compute.
2011-10-08 22:02:31,333 INFO nova.compute.
2011-10-08 22:02:31,334 INFO nova.compute.
2011-10-08 22:02:31,334 INFO nova.compute.
2011-10-08 22:02:31,410 AUDIT nova.compute.
2011-10-08 22:02:31,447 WARNING nova.volume.driver [-] ISCSI provider_location not stored, using discovery
2011-10-08 22:02:31,447 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m discovery -t sendtargets -p node1 from (pid=1585) execute /usr/lib/
2011-10-08 22:02:31,489 DEBUG nova.volume.driver [-] ISCSI Discovery: Found 192.168.
2011-10-08 22:02:31,490 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-
2011-10-08 22:02:32,015 DEBUG nova.volume.driver [-] iscsiadm ('--login',): stdout=Logging in to [iface: default, target: iqn.2010-
Login to [iface: default, target: iqn.2010-
stderr= from (pid=1585) _run_iscsiadm /usr/lib/
2011-10-08 22:02:32,015 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-
2011-10-08 22:02:32,033 DEBUG nova.volume.driver [-] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= from (pid=1585) _run_iscsiadm /usr/lib/
2011-10-08 22:02:32,033 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/
2011-10-08 22:02:32,034 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-
2011-10-08 22:02:32,052 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-
stderr= from (pid=1585) _run_iscsiadm /usr/lib/
2011-10-08 22:02:33,054 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/
2011-10-08 22:02:33,054 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-
2011-10-08 22:02:33,073 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-
stderr= from (pid=1585) _run_iscsiadm /usr/lib/
2011-10-08 22:02:37,077 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/
2011-10-08 22:02:37,077 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-
2011-10-08 22:02:37,095 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-
stderr= from (pid=1585) _run_iscsiadm /usr/lib/
2011-10-08 22:02:46,100 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: rval = node_func(
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: volume_id)
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: path = self.driver.
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: (mount_device))
(nova.rpc): TRACE: Error: iSCSI device not found at /dev/disk/
(nova.rpc): TRACE:
2011-10-08 22:03:52,889 INFO nova.compute.
2011-10-08 22:05:55,440 INFO nova.compute.
root@node2:
total 0
lrwxrwxrwx 1 root root 9 2011-10-08 22:02 ip-192.
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:
lrwxrwxrwx 1 root root 10 2011-10-08 21:38 pci-0000:
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:
root@node1:~# tgtadm --lld iscsi --op show --mode target
Target 1: iqn.2010-
System information:
Driver: iscsi
State: ready
I_T nexus information:
I_T nexus: 1
IP Address: 192.168.1.202 <========= initiator (i-00000004) on node2 is connected to target (vol-00000004) on node1
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1 <======
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 1074 MB, Block size: 512
Online: Yes
Backing store type: rdwr
Backing store path: /dev/nova-
Backing store flags:
Account information:
ACL information:
ALL
description: | updated |
description: | updated |
description: | updated |
Changed in nova (Ubuntu Oneiric): | |
milestone: | none → oneiric-updates |
Changed in nova (Ubuntu Precise): | |
milestone: | oneiric-updates → precise-alpha-1 |
tags: |
added: verification-failed removed: verification-needed |
Changed in nova (Ubuntu Oneiric): | |
status: | Fix Committed → Incomplete |
Changed in nova (Ubuntu Precise): | |
status: | Fix Released → In Progress |
Changed in nova (Ubuntu): | |
milestone: | precise-alpha-1 → precise-alpha-2 |
Changed in nova: | |
importance: | Undecided → Medium |
status: | New → Confirmed |
assignee: | nobody → Chuck Short (zulcss) |
status: | Confirmed → In Progress |
Changed in nova: | |
milestone: | none → essex-3 |
status: | Fix Committed → Fix Released |
Changed in nova: | |
milestone: | essex-3 → 2012.1 |
Thank you for the wonderfully detailed bug report.
I am intrigued why they differ by 1.. "/
This might need some smarter logic for us to be able to push a conditional mount_device value upstream, to support both formats.