OpenStack Compute (nova)

[Nova-volume][tgtd] targets are not persistent

Bug #1011159 reported by David Naori
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
High
Chuck Short
Cinder 2012.2 "folsom"
OpenStack Compute (nova)
Fix Released
Undecided
Chuck Short
OpenStack Compute (nova) 2012.2 "folsom"

Bug Description

[root@camel-nova nova]# nova volume-create 10

[root@camel-nova nova]# tgt-admin --show ALL
Target 1: iqn.2010-10.org.openstack:volume-0000000e
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET 00010000
            SCSI SN: beaf10
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            Backing store type: null
            Backing store path: None
            Backing store flags:
        LUN: 1
            Type: disk
            SCSI ID: IET 00010001
            SCSI SN: beaf11
            Size: 10737 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            Backing store type: rdwr
            Backing store path: /dev/nova-volumes/volume-0000000e
            Backing store flags:
    Account information:
    ACL information:
        ALL
Target 3: iqn.2010-10.org.openstack:volume-0000000b
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET 00030000
            SCSI SN: beaf30
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            Backing store type: null
            Backing store path: None
            Backing store flags:
        LUN: 1
            Type: disk
            SCSI ID: IET 00030001
            SCSI SN: beaf31
            Size: 2147 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            Backing store type: rdwr
            Backing store path: /dev/nova-volumes/volume-0000000b
            Backing store flags:
    Account information:
    ACL information:
        ALL
[root@camel-nova nova]# /etc/init.d/tgtd restart
Stopping SCSI target daemon: [ OK ]
Starting SCSI target daemon: [ OK ]
[root@camel-nova nova]# tgt-admin --show ALL
[root@camel-nova nova]#

[root@camel-nova nova]# nova volume-attach toattach 14 /dev/vdc

2012-06-10 16:18:08 ERROR nova.rpc.amqp [req-9e485674-61b2-4d6b-9a28-119636879512 2f4edfa99cab42de92eacda360043116 7eed65dd55474b9e94cd412d9f66b406] Exception during message handling
2012-06-10 16:18:08 TRACE nova.rpc.amqp Traceback (most recent call last):
2012-06-10 16:18:08 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 252, in _process_data
2012-06-10 16:18:08 TRACE nova.rpc.amqp rval = node_func(context=ctxt, **node_args)
2012-06-10 16:18:08 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 114, in wrapped
2012-06-10 16:18:08 TRACE nova.rpc.amqp return f(*args, **kw)
2012-06-10 16:18:08 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 153, in decorated_function
2012-06-10 16:18:08 TRACE nova.rpc.amqp function(self, context, instance_uuid, *args, **kwargs)
2012-06-10 16:18:08 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 177, in decorated_function
2012-06-10 16:18:08 TRACE nova.rpc.amqp sys.exc_info())
2012-06-10 16:18:08 TRACE nova.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2012-06-10 16:18:08 TRACE nova.rpc.amqp self.gen.next()
2012-06-10 16:18:08 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 171, in decorated_function
2012-06-10 16:18:08 TRACE nova.rpc.amqp return function(self, context, instance_uuid, *args, **kwargs)
2012-06-10 16:18:08 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1729, in attach_volume
2012-06-10 16:18:08 TRACE nova.rpc.amqp connector)
2012-06-10 16:18:08 TRACE nova.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2012-06-10 16:18:08 TRACE nova.rpc.amqp self.gen.next()
2012-06-10 16:18:08 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1721, in attach_volume
2012-06-10 16:18:08 TRACE nova.rpc.amqp mountpoint)
2012-06-10 16:18:08 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 114, in wrapped
2012-06-10 16:18:08 TRACE nova.rpc.amqp return f(*args, **kw)
2012-06-10 16:18:08 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py", line 532, in attach_volume
2012-06-10 16:18:08 TRACE nova.rpc.amqp mount_device)
2012-06-10 16:18:08 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py", line 524, in volume_driver_method
2012-06-10 16:18:08 TRACE nova.rpc.amqp return method(connection_info, *args, **kwargs)
2012-06-10 16:18:08 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/utils.py", line 946, in inner
2012-06-10 16:18:08 TRACE nova.rpc.amqp retval = f(*args, **kwargs)
2012-06-10 16:18:08 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py", line 153, in connect_volume
2012-06-10 16:18:08 TRACE nova.rpc.amqp check_exit_code=[0, 255])
2012-06-10 16:18:08 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py", line 110, in _run_iscsiadm
2012-06-10 16:18:08 TRACE nova.rpc.amqp check_exit_code=check_exit_code)
2012-06-10 16:18:08 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/utils.py", line 243, in execute
2012-06-10 16:18:08 TRACE nova.rpc.amqp cmd=' '.join(cmd))
2012-06-10 16:18:08 TRACE nova.rpc.amqp ProcessExecutionError: Unexpected error while running command.
2012-06-10 16:18:08 TRACE nova.rpc.amqp Command: sudo nova-rootwrap iscsiadm -m node -T iqn.2010-10.org.openstack:volume-0000000e -p 10.35.97.20:3260 --login
2012-06-10 16:18:08 TRACE nova.rpc.amqp Exit code: 19
2012-06-10 16:18:08 TRACE nova.rpc.amqp Stdout: 'Logging in to [iface: default, target: iqn.2010-10.org.openstack:volume-0000000e, portal: 10.35.97.20,3260] (multiple)\n'
2012-06-10 16:18:08 TRACE nova.rpc.amqp Stderr: 'iscsiadm: Could not login to [iface: default, target: iqn.2010-10.org.openstack:volume-0000000e, portal: 10.35.97.20,3260].\niscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)\niscsiadm: Could not log into all portals\n'
2012-06-10 16:18:08 TRACE nova.rpc.amqp

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/10414

Changed in nova:
assignee: nobody → Chuck Short (zulcss)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/10414
Committed: http://github.com/openstack/nova/commit/1ba3dfec786c60e53ceb3b682bcf927224102a0b
Submitter: Jenkins
Branch: master

commit 1ba3dfec786c60e53ceb3b682bcf927224102a0b
Author: Chuck Short <email address hidden>
Date: Fri Jul 27 10:25:51 2012 -0500

    Add persistent volumes for tgtd.

    Currently if you restart the server running nova-volume
    or restart tgt, you will loose your iscsi targets that
    have been created. This is not good.

    In order for iscsi targets to be persistent across
    reboots or restarts, one has to have the target's configuration
    information in /etc/tgt/targets.conf or /etc/tgt/conf.d.
    So when tgtd is restarted then the iscsi targets will be there
    as expected.

    This patch will add a configuration file to $state_path/volumes
    when the volume is created. The configuration file is identified by
    the volume uuid. It creates a logicalunit when the volume is created
    as well. The iscsi target and configuration file
    will be removed once the volume has been removed as well.

    In order to use this, you have to include the following in
    your /etc/tgt/targets.conf

    include $state_path/volumes/*

    For upgrades, it will just re-create the volumes
    already in the volumes table.

    Fixes LP: #1011159

    Change-Id: I38fc096ab881ccb52cb688ae46d9d36b0a7b3a45
    Signed-off-by: Chuck Short <email address hidden>

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
John Griffith (john-griffith) wrote :

Chuck Short has submitted https://review.openstack.org/#/c/10417/ to address this on Cinder

We're currently waiting for the needed devstack change to land:
https://review.openstack.org/#/c/10467/

After this jenkins/gate jobs should pass and fix can be merged.

Changed in cinder:
status: New → In Progress
assignee: nobody → Chuck Short (zulcss)
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/10417
Committed: http://github.com/openstack/cinder/commit/c401895f8f2b6f1693601f7996da88d7b3d45fa0
Submitter: Jenkins
Branch: master

commit c401895f8f2b6f1693601f7996da88d7b3d45fa0
Author: Chuck Short <email address hidden>
Date: Fri Jul 27 12:05:55 2012 -0500

    Add persistent volumes for tgtd.

    Currently if you restart the server running nova-volume
    or restart tgt, you will loose your iscsi targets that
    have been created. This is not good.

    In order for iscsi targets to be persistent across
    reboots or restarts, one has to have the target's configuration
    information in /etc/tgt/targets.conf or /etc/tgt/conf.d.
    So when tgtd is restarted then the iscsi targets will be there
    as expected.

    This patch will add a configuration file to $state_path/volumes
    when the volume is created. The configuration file is identified by
    the volume uuid. It creates a logicalunit when the volume is created
    as well. The iscsi target and configuration file
    will be removed once the volume has been removed as well.

    In order to use this, you have to include the following in
    your /etc/tgt/targets.conf

    include $state_path/volumes/*

    For upgrades, it will just re-create the volumes
    already in the volumes table.

    Fixes LP: #1011159

    Change-Id: Ib74dcc1efa0332842041b4c045ca5fa85d4a32ca
    Signed-off-by: Chuck Short <email address hidden>

Changed in cinder:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → folsom-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in cinder:
milestone: none → folsom-3
status: Fix Committed → Fix Released
Revision history for this message
Akira Yoshiyama (yosshy) wrote :

This patch looks broken with ISCSIDriver/TgtAdm for us. (d2babbfd4b59bde09fe17f7b69c86d857a82d104)
Anyone else?

Thierry Carrez (ttx)
Changed in cinder:
milestone: folsom-3 → 2012.2
Thierry Carrez (ttx)
Changed in nova:
milestone: folsom-3 → 2012.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.