iscsiadm hangs in recvmsg()
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
open-iscsi (Fedora) |
Fix Released
|
High
|
|||
open-iscsi (Ubuntu) |
Fix Released
|
High
|
Adam Stokes | ||
Precise |
Fix Released
|
High
|
Adam Stokes | ||
Quantal |
Fix Released
|
High
|
Adam Stokes |
Bug Description
[Impact]
iscsid can hang or crash
[Test Case]
1) This issue is exacerbated by increased logging output. To configure open-iscsi to emit the maximum amount of messages, edit /etc/init.
start-stop-daemon --start --quiet --exec $DAEMON
to:
start-stop-daemon --start --quiet --exec $DAEMON -- -d 8
On your storage server, create multiple LUN targets and expose them to your Ubuntu system. I recommend 3 or more.
Edit the attached repro.sh script to set the TARGET variable to point to the correct IP of your storage server. Execute this script on your Ubuntu system. When an error occurs, this script should hang or begin emitting errors.
[Regression Potential]
I would expect regressions here to have the same symptoms of the original bug - e.g. hangs, crashes.
Original report follows.
We occasionally see iscsiadm hang indefinitely on a system. It is difficult to reproduce, there maybe months between occurrences. This happened again today, and we collected the following information.
We captured an strace of a failing iscsiadm discover process, I'll attach it here. All iscsiadm processes that are hung are also in recvmsg() when we attach strace.
It was observed that there was one iscsid process - where normally there is 2. The pids are normally consecutive, so I presume one forks the other before daemonizing.
syslog contains the following messages from iscsid:
Sep 20 15:55:57 cn5 iscsid: semop up failed 4
Sep 20 15:56:05 cn5 iscsid: semop down failed
Sep 20 15:56:23 cn5 iscsid: semop up failed 4
Sep 20 16:02:23 cn5 iscsid: semop up failed 4
Sep 20 16:02:26 cn5 iscsid: semop up failed 4
Sep 20 16:02:26 cn5 iscsid: semop up failed 4
Sep 20 16:02:26 cn5 iscsid: semop up failed 4
Sep 20 16:02:32 cn5 iscsid: semop up failed 4
Sep 20 18:01:49 cn5 iscsid: semop up failed 4
Sep 20 18:03:56 cn5 iscsid: semop up failed 4
Sep 20 18:05:03 cn5 iscsid: semop up failed 4
This sounds similar to the issue described in https:/
Related branches
- Marc Deslauriers: Approve
- Ubuntu branches: Pending requested
-
Diff: 70 lines (+25/-8)2 files modifieddebian/changelog (+7/-0)
usr/log.c (+18/-8)
tags: | added: precise |
Changed in open-iscsi (Ubuntu): | |
importance: | Undecided → High |
Changed in open-iscsi (Ubuntu Quantal): | |
status: | In Progress → Fix Released |
Changed in open-iscsi (Ubuntu Precise): | |
status: | New → Confirmed |
importance: | Undecided → High |
Changed in open-iscsi (Ubuntu Quantal): | |
milestone: | ubuntu-12.04.2 → none |
Changed in open-iscsi (Ubuntu Precise): | |
milestone: | none → ubuntu-12.04.2 |
assignee: | nobody → Adam Stokes (adam-stokes) |
tags: |
added: verification-done removed: verification-needed |
Changed in open-iscsi (Fedora): | |
importance: | Unknown → High |
status: | Unknown → Fix Released |
Description of problem:
Each process must have its own semarg structure - or they step on each
others' toes - which could cause either deadlocks or smearing of the shared
memory protected by the semaphore.
Version-Release number of selected component (if applicable):
iscsi-initiator -utils- 6.2.0.871- 0.18.fc12. src.rpm (Anything based on 871 series).
How reproducible:
Unknown.
Steps to Reproduce:
1. Configure the iscsi initator util with at least two target devices.
2. Configure it to the maximum amount of logging
3. ?????
Actual results:
Logging can possibly cause a deadlock in waiting for semaphore access.
Expected results:
No deadlock to occur.
Additional info:
This is a backport of the patch from : http:// git.kernel. org/?p= linux/kernel/ git/mnc/ open-iscsi. git;a=commitdif f;h=b95a7ca80b2 55f0c5b34b26693 67b629c8489552; hp=fb4f2d3072be e96606d01e3535c 100dc99b8d331 to the 871 series.
Mike Christie posted Guy Kerens patch (I think). This patch is already in the EL6 build.