iscsid does not start automatically having iscsi root

Bug #575240 reported by Xavi Serrano
38
This bug affects 6 people
Affects Status Importance Assigned to Milestone
open-iscsi (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

Binary package hint: open-iscsi

Affected package: open-iscsi

   # apt-cache policy open-iscsi
   open-iscsi:
     Installed: 2.0.871-0ubuntu4
     Candidate: 2.0.871-0ubuntu4
     Version table:
    *** 2.0.871-0ubuntu4 0
           500 http://es.archive.ubuntu.com/ubuntu/ lucid/main Packages
           100 /var/lib/dpkg/status

Fresh install of Ubuntu Server 10.04 (official release, no RC).
root filesystem in iSCSI.
/boot in a separate filesystem.
swap in a dedicated partition (in iSCSI as well).

Tested on both real and virtual machines.

Boot process seems to be OK.
root filesystem is correctly mounted.
"iscsiadm -m session" shows connection to the target:

   # iscsiadm -m session
   tcp: [1] 192.168.21.128:3260,2 iqn.1986-03.com.sun:02:55ea9bbb-38e6-4161-ae96-80b1b0aa08fa

But increasing debug level, we can see we are getting an 111 error when contacting iscsid (111 = connection refused):

   # iscsiadm -m session -P1
   Target: iqn.1986-03.com.sun:02:55ea9bbb-38e6-4161-ae96-80b1b0aa08fa
           Current Portal: 192.168.21.128:3260,2
           Persistent Portal: 192.168.21.128:3260,2
                   **********
                   Interface:
                   **********
                   Iface Name: default
                   Iface Transport: tcp
                   Iface Initiatorname: iqn.1993-08.org.debian:01:e0f7a4361e16
                   Iface IPaddress: 192.168.21.38
                   Iface HWaddress: <empty>
                   Iface Netdev: <empty>
                   SID: 1
   iscsiadm: can not connect to iSCSI daemon (111)!
                   iSCSI Connection State: Unknown
                   iSCSI Session State: LOGGED_IN
                   Internal iscsid Session State: Unknown

The problem is that iscsid is not running:

   # /etc/init.d/open-iscsi status
    * iscsid is not running

   # ps -efa | grep -i iscsi
   root 232 2 0 17:15 ? 00:00:00 [iscsi_eh]
   root 238 2 0 17:15 ? 00:00:00 [iscsi_q_2]
   root 851 782 0 17:32 pts/0 00:00:00 grep --color=auto -i iscsi

open-iscsi SysV init scripts are installed (according to /var/lib/dpkg/info/open-iscsi.postinst):

   # find /etc/rc*.d -iname "*iscsi*"
   /etc/rc0.d/S41open-iscsi
   /etc/rc1.d/S41open-iscsi
   /etc/rc6.d/S41open-iscsi
   /etc/rcS.d/S25open-iscsi

iscsid must be running in order to handle connection errors to the iSCSI target, among many other things.

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: open-iscsi 2.0.871-0ubuntu4
ProcVersionSignature: Ubuntu 2.6.32-21.32-server 2.6.32.11+drm33.2
Uname: Linux 2.6.32-21-server x86_64
Architecture: amd64
Date: Tue May 4 17:16:25 2010
InstallationMedia: Ubuntu-Server 10.04 LTS "Lucid Lynx" - Release amd64 (20100427)
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: open-iscsi

Revision history for this message
Xavi Serrano (xserrano) wrote :
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in open-iscsi (Ubuntu):
status: New → Confirmed
Revision history for this message
Will Rouesnel (w-rouesnel) wrote :

This appears to be an upstart bug and some type of regression between Jaunty and Natty.

The solution for me was to edit rc-sysinit.conf -

in Natty it's start condition is:
start on filesystem and net-device-up IFACE=lo

Whereas in Jaunty its:
start on filesystem

When I edited the script back to the way Jaunty was, it started working correctly. It's possible this is also affecting other rcS scripts, since my logs indicated that open-iscsi was never being invoked at all.

Revision history for this message
Will Rouesnel (w-rouesnel) wrote :

It's also worth noting that as is iscsid in new Ubuntus (Natty has it, probably others past Jaunty as well) will fail and disconnect targets when it's invoked on an iscsi root because of a kernel change (it treats "(null)" as real credentials, rather then indicating no credentials and will fail out of discovery sessions and node logins ).

To fix this I had to build open-iscsi against the newer 2.0.872 version. A straight rebuild with the new sources and the Ubuntu debian folder worked fine. It's a 3 line patch to fix the issue, and the new version should probably be rolled out as an upgrade since without using an ISCSI root is badly broken.

Revision history for this message
Mark Tomich (mstomich) wrote :

recompiling the "current stable" open-iscsi requires a lot of work on Oneiric because it hasn't been updated to recognize the change to kernel v3.0; additionally, once I got it to compile, it gave me oopses in the init sequence.

Furthermore, a simpler and "easier to maintain" solution for me was to simply roll iscsid into the initramfs by extracting the attached tarball over "/" and then running "update-initramfs -ck all"

Revision history for this message
Mark Tomich (mstomich) wrote :

Okay, forget what I just said in comment 5... installing open-iscsi v2.0-872 userspace *does* seem to be the best option for Oneiric users, *however*, the packaged version on the open-iscsi website doesn't play nice with the 3.0 kernel. Instead, I used the git master, and skipped building the modules.

Revision history for this message
Jonathan McCune (jonmccune-j) wrote :

I have setup a client system running Ubuntu 12.04 using an iscsi root filesystem. There is no problem with the root filesystem; everything boots up just fine. However, I experience the exact same problems as those reported in the initial bug description. I.e., iscsid is not running, so connecting to other iscsi targets doesn't work. Starting iscsid causes the mounted root filesystem to misbehave. The above description suggests that it remounts read-only. I have seen that some of the time, and I have also seen a complete hang.

I tried adding the "initramfs patch to roll in iscsid" from comment #5; this appears to have made no difference.

I'm going to try installing open-iscsi from git as proposed in comment #6. I'll report back on my luck.

Strange that I'm having these problems, though. Everything is currently working just fine with some Oneiric (Ubuntu 11.10) clients. I'm tempted to let the installer upgrade a copy of the Oneiric image to see if the problem recurs.

Regards,
-Jon

Revision history for this message
Jonathan McCune (jonmccune-j) wrote :

Okay, I have success building the open-iscsi userspace from source.

$ git clone git://github.com/mikechristie/open-iscsi.git
# HEAD is at commit dd73b7d12b6bc5f4f4d08c2ac7dcfc5f00f6fd29
$ cd open-iscsi
$ make
$ make install
$ update-initramfs -c -k all
# manually copy /boot/init* to the appropriate iscsi target via scp to the fileserver itself

Now, after booting, 'ps -ef | grep iscsi' shows two instances of iscsid running. 'iscsiadm -m session -P 1' returns the expected output, and I can mount my /boot just fine.

I have not tried backing out Mark's "initramfs patch to roll in iscsid".

One strange thing is that my client (ahem.. initiator) connected to all of the available targets on the iscsi server, not just the one of interest to me. So far no harm done, so I may not spend the time to track it down.

Thanks Mark, for the tips.

Regards,
-Jon

Revision history for this message
Jonathan McCune (jonmccune-j) wrote :

In attempting to debug this earlier, I had set 'node.startup = automatic' in /etc/iscsi/iscsid.conf, so that during discovery /etc/iscsi/nodes/* were all set to automatically start. This is completely unrelated to the main problem identified by this bug; I just wanted to make that explicit, since I mentioned the issue in comment #8.

Thanks,
-Jon

Revision history for this message
martins (martins256) wrote :

I had the same problem as described above. I am running ubuntu 12.04.2 which is booted via iSCSI (no internal hard drives). iscsid is not running and when I start it and try to attach a drive the root filesystem will change to read-only. The fix was as Jonathan McCune described to recompile open-iscsi. I chose version 2.0.873 which is used in ubuntu 12.10.

apt-get install git make gcc
git://github.com/mikechristie/open-iscsi.git
cd open-iscsi
git checkout 2.0.873
make
make install
update-initramfs -c -k all
reboot

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.