during sys_exit, ip BUGs when dropping an ext3 superblock during namespace exit
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Confirmed
|
High
|
Unassigned |
Bug Description
I've hit the same BUG twice in as many working days. I'll give a brief description of what I'm doing, paste the two BUG reports, then paste the requested information.
I've been doing some performance testing on OpenStack. The tests just involve booting and deleting a bunch of virtual machine instances. A couple of things happen during VM creation that seem to cause the bug on. First, tap devices are created in various network namespaces. Second, nbd devices, served locally by qemu-nbd, with ext3 data are connected to /dev/nbd* and mounted and fiddled with.
The ip program is used to operate in the network namespaces (ip netns exec ...). Occasionally, when ip exits, it hits a BUG in some block layer code. Apparently the BUGging ip task had the last reference to a mount.
This BUG is fairly nasty because it happens when the superblock's s_umount semaphore is held, so other tasks that try to do stuff with the mount deadlock.
Here are the processes involved in today's BUG:
15488 sudo ip netns exec qdhcp-1ca153f0-
15492 \_ [ip]
27607 /bin/mount /dev/nbd13 /tmp/openstack-
27592 /usr/bin/qemu-nbd -c /dev/nbd13 /opt/stack/
% sudo cat /proc/27607/stack
[<ffffffff811df
[<ffffffff811df
[<ffffffff811df
[<ffffffff811a4
[<ffffffff811a4
[<ffffffff811b4
[<ffffffff811b5
[<ffffffff811b6
[<ffffffff811a5
[<ffffffff811a6
[<ffffffff816f5
[<fffffffffffff
% sudo cat /proc/27592/
[<ffffffff81369
[<ffffffff811a9
[<ffffffff811de
[<ffffffff81340
[<ffffffff81343
[<ffffffff811df
[<ffffffff811df
[<ffffffff811df
[<ffffffff811a4
[<ffffffff811a4
[<ffffffff811b4
[<ffffffff811b5
[<ffffffff811b6
[<ffffffff811a5
[<ffffffff811a6
[<ffffffff816f5
[<fffffffffffff
I'm running on the stock ubuntu kernel; no custom modules have been inserted.
This is the first BUG cut from a few days ago. I don't have the entire dmesg log from this run:
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.131201] ------------[ cut here ]------------
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.131469] Kernel BUG at ffffffff811d77d4 [verbose debug info unavailable]
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.131881] invalid opcode: 0000 [#1] SMP
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.132148] Modules linked in: xt_LOG(F) vhost_net vhost macvtap(F) macvlan(F) nf_conntrack_
xt_mac(F) xt_physdev(F) veth(F) xt_nat(F) xt_REDIRECT(F) ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_
) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) stp(F) llc(F) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables
(F) openvswitch(OF) dm_multipath(F) scsi_dh(F) nbd(F) vesafb(F) x86_pkg_
l(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) gpio_ich microcode(F) joydev(F) sb_edac edac_core mei_me mei lpc_ich ioatdma mac_hid lp(F) p
arport(F) dm_snapshot(F) hid_generic igb isci ixgbe i2c_algo_bit usbhid dca hid libsas(F) ahci(F) ptp(F) libahci(
Nov 15 14:52:44 node-0025904feb5c kernel: F) pps_core(F) mdio scsi_transport_
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.138029] CPU: 15 PID: 27942 Comm: ip Tainted: GF O 3.11.0-12-generic #19-Ubuntu
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.131881] invalid opcode: 0000 [#1] SMP
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.132148] Modules linked in: xt_LOG(F) vhost_net vhost macvtap(F) macvlan(F) nf_conntrack_
xt_mac(F) xt_physdev(F) veth(F) xt_nat(F) xt_REDIRECT(F) ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_
) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) stp(F) llc(F) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables
) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) stp(F) llc(F) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables
(F) openvswitch(OF) dm_multipath(F) scsi_dh(F) nbd(F) vesafb(F) x86_pkg_
l(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) gpio_ich microcode(F) joydev(F) sb_edac edac_core mei_me mei lpc_ich ioatdma mac_hid lp(F) p
arport(F) dm_snapshot(F) hid_generic igb isci ixgbe i2c_algo_bit usbhid dca hid libsas(F) ahci(F) ptp(F) libahci(
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.131881] invalid opcode: 0000 [#1] SMP
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.132148] Modules linked in: xt_LOG(F) vhost_net vhost macvtap(F) macvlan(F) nf_conntrack_
xt_mac(F) xt_physdev(F) veth(F) xt_nat(F) xt_REDIRECT(F) ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_
) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) stp(F) llc(F) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables
(F) openvswitch(OF) dm_multipath(F) scsi_dh(F) nbd(F) vesafb(F) x86_pkg_
l(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) gpio_ich microcode(F) joydev(F) sb_edac edac_core mei_me mei lpc_ich ioatdma mac_hid lp(F) p
arport(F) dm_snapshot(F) hid_generic igb isci ixgbe i2c_algo_bit usbhid dca hid libsas(F) ahci(F) ptp(F) libahci(
Nov 15 14:52:44 node-0025904feb5c kernel: F) pps_core(F) mdio scsi_transport_
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.138029] CPU: 15 PID: 27942 Comm: ip Tainted: GF O 3.11.0-12-generic #19-Ubuntu
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.131881] invalid opcode: 0000 [#1] SMP
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.132148] Modules linked in: xt_LOG(F) vhost_net vhost macvtap(F) macvlan(F) nf_conntrack_
xt_mac(F) xt_physdev(F) veth(F) xt_nat(F) xt_REDIRECT(F) ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_
) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) stp(F) llc(F) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables
(F) openvswitch(OF) dm_multipath(F) scsi_dh(F) nbd(F) vesafb(F) x86_pkg_
l(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) gpio_ich microcode(F) joydev(F) sb_edac edac_core mei_me mei lpc_ich ioatdma mac_hid lp(F) p
arport(F) dm_snapshot(F) hid_generic igb isci ixgbe i2c_algo_bit usbhid dca hid libsas(F) ahci(F) ptp(F) libahci(
Nov 15 14:52:44 node-0025904feb5c kernel: F) pps_core(F) mdio scsi_transport_
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.138029] CPU: 15 PID: 27942 Comm: ip Tainted: GF O 3.11.0-12-generic #19-Ubuntu
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.157795] Hardware name: Supermicro X9DRFR/X9DRFR, BIOS 1.0a 09/11/2012
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.167959] task: ffff88053c631770 ti: ffff88060d788000 task.ti: ffff88060d788000
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.189055] RIP: 0010:[<
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.209728] RSP: 0018:ffff88060d
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.220248] RAX: 0000000000000005 RBX: ffff880772e019c0 RCX: ffff88060d789fd8
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.241884] RDX: 0000000000000000 RSI: ffff880772e019c0 RDI: 0000000000000411
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.263641] RBP: ffff88060d789cb8 R08: 0000000000000000 R09: 0000000000000002
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.286483] R10: ffff880772e01068 R11: ffffffff8128badd R12: 0000000000000411
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.309833] R13: ffff8807a6499400 R14: ffff88076b528000 R15: 00007fff6b62dc00
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.333067] FS: 00007fb0e72c274
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.356632] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.368780] CR2: 00007fb1a1025f50 CR3: 00000007be808000 CR4: 00000000000427e0
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.384379] nbd9: unknown partition table
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.403462] Stack:
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.414237] ffff880772e019c0 0000000000000411 ffff8807a6499400 ffff88076b528000
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.436155] ffff88060d789cd8 ffffffff811d84a5 ffff880772e019c0 ffff88060155e400
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.457077] kjournald starting. Commit interval 5 seconds
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.480247] EXT3-fs (nbd9): using internal journal
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.480251] EXT3-fs (nbd9): mounted filesystem with ordered data mode
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.469329] ffff88060d789ce8 ffffffff811d8543 ffff88060d789d10 ffffffff812310d0
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.513658] Call Trace:
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.524287] [<ffffffff811d8
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.534920] [<ffffffff811d8
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.545623] [<ffffffff81231
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.556287] [<ffffffff81231
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.566461] [<ffffffff811a8
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.576653] [<ffffffff811a9
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.586668] [<ffffffff811a9
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.596335] [<ffffffff811a9
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.606253] [<ffffffff811c5
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.615477] [<ffffffff811c5
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.624622] [<ffffffff811c5
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.634156] [<ffffffff811c7
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.643059] [<ffffffff81089
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.651554] [<ffffffff81089
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.659768] [<ffffffff81089
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.667855] [<ffffffff81064
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.676050] [<ffffffff81064
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.685242] [<ffffffff81064
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.693422] [<ffffffff816f5
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.701737] Code: 52 00 00 89 d8 5b 41 5c 41 5d 41 5e 5d c3 66 0f 1f 84 00 00 00 00 00 40 f6 c7 01 0f 84 e7 fe ff ff f0 80 66 01 f7 e9 dd fe ff ff <0f> 0b 49 8b 54 24 70 c1 e0 09 41 f6 c5 01 41 89 44 24 30 89 42
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.725473] RIP [<ffffffff811d7
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.733137] RSP <ffff88060d789c98>
Nov 15 14:52:44 node-0025904feb5c kernel: [1902861.753260] ---[ end trace 79cbd7bd94cca2ea ]---
This is the second BUG cut from today. The full dmesg is attached.
[Mon Nov 18 17:23:02 2013] ------------[ cut here ]------------
[Mon Nov 18 17:23:02 2013] Kernel BUG at ffffffff811d77d4 [verbose debug info unavailable]
[Mon Nov 18 17:23:02 2013] invalid opcode: 0000 [#1] SMP
[Mon Nov 18 17:23:02 2013] Modules linked in: nf_conntrack_
[Mon Nov 18 17:23:02 2013] CPU: 19 PID: 15492 Comm: ip Tainted: GF O 3.11.0-12-generic #19-Ubuntu
[Mon Nov 18 17:23:02 2013] Hardware name: Supermicro X9DRFR/X9DRFR, BIOS 1.0a 09/11/2012
[Mon Nov 18 17:23:02 2013] task: ffff8810516f9770 ti: ffff88103e7ca000 task.ti: ffff88103e7ca000
[Mon Nov 18 17:23:02 2013] RIP: 0010:[<
[Mon Nov 18 17:23:02 2013] RSP: 0018:ffff88103e
[Mon Nov 18 17:23:02 2013] RAX: 0000000000000005 RBX: ffff880701165820 RCX: ffff88103e7cbfd8
[Mon Nov 18 17:23:02 2013] RDX: 0000000000000000 RSI: ffff880701165820 RDI: 0000000000000411
[Mon Nov 18 17:23:02 2013] nbd15: unknown partition table
[Mon Nov 18 17:23:02 2013] RBP: ffff88103e7cbcb8 R08: 0000000000000000 R09: 0000000000000002
[Mon Nov 18 17:23:02 2013] R10: ffff880701165888 R11: ffffffff8128badd R12: 0000000000000411
[Mon Nov 18 17:23:02 2013] kjournald starting. Commit interval 5 seconds
[Mon Nov 18 17:23:02 2013] R13: ffff880802fef400 R14: ffff880eff6b4180 R15: 00007fff345eb930
[Mon Nov 18 17:23:02 2013] EXT3-fs (nbd15): using internal journal
[Mon Nov 18 17:23:02 2013] EXT3-fs (nbd15): mounted filesystem with ordered data mode
[Mon Nov 18 17:23:02 2013] FS: 00007f3a9ddf174
[Mon Nov 18 17:23:02 2013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Mon Nov 18 17:23:02 2013] CR2: 00007fd38a321be0 CR3: 0000000001c0e000 CR4: 00000000000407f0
[Mon Nov 18 17:23:02 2013] Stack:
[Mon Nov 18 17:23:02 2013] ffff880701165820 0000000000000411 ffff880802fef400 ffff880eff6b4180
[Mon Nov 18 17:23:02 2013] ffff88103e7cbcd8 ffffffff811d84a5 ffff880701165820 ffff88085422c000
[Mon Nov 18 17:23:02 2013] ffff88103e7cbce8 ffffffff811d8543 ffff88103e7cbd10 ffffffff812310d0
[Mon Nov 18 17:23:02 2013] Call Trace:
[Mon Nov 18 17:23:02 2013] [<ffffffff811d8
[Mon Nov 18 17:23:02 2013] [<ffffffff811d8
[Mon Nov 18 17:23:02 2013] [<ffffffff81231
[Mon Nov 18 17:23:02 2013] [<ffffffff81231
[Mon Nov 18 17:23:02 2013] [<ffffffff811a8
[Mon Nov 18 17:23:02 2013] [<ffffffff811a9
[Mon Nov 18 17:23:02 2013] [<ffffffff811a9
[Mon Nov 18 17:23:02 2013] [<ffffffff811a9
[Mon Nov 18 17:23:02 2013] [<ffffffff811c5
[Mon Nov 18 17:23:02 2013] [<ffffffff811c5
[Mon Nov 18 17:23:02 2013] [<ffffffff811c5
[Mon Nov 18 17:23:02 2013] [<ffffffff811c7
[Mon Nov 18 17:23:02 2013] [<ffffffff81089
[Mon Nov 18 17:23:02 2013] [<ffffffff81089
[Mon Nov 18 17:23:02 2013] [<ffffffff81089
[Mon Nov 18 17:23:02 2013] [<ffffffff81064
[Mon Nov 18 17:23:02 2013] [<ffffffff81064
[Mon Nov 18 17:23:02 2013] [<ffffffff81064
[Mon Nov 18 17:23:02 2013] [<ffffffff816f5
[Mon Nov 18 17:23:02 2013] Code: 52 00 00 89 d8 5b 41 5c 41 5d 41 5e 5d c3 66 0f 1f 84 00 00 00 00 00 40 f6 c7 01 0f 84 e7 fe ff ff f0 80 66 01 f7 e9 dd fe ff ff <0f> 0b 49 8b 54 24 70 c1 e0 09 41 f6 c5 01 41 89 44 24 30 89 42
[Mon Nov 18 17:23:02 2013] RIP [<ffffffff811d7
[Mon Nov 18 17:23:02 2013] RSP <ffff88103e7cbc98>
[Mon Nov 18 17:23:02 2013] ---[ end trace 5fa4e7d1e416ec13 ]---
% uname -a
Linux node-0025904feb5c 3.11.0-12-generic #19-Ubuntu SMP Wed Oct 9 16:20:46 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
% cat /proc/version_
Ubuntu 3.11.0-
---
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Nov 15 15:56 seq
crw-rw---- 1 root audio 116, 33 Nov 15 15:56 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.12.5-0ubuntu2.1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 13.10
HibernationDevice: RESUME=
IwConfig: Error: [Errno 2] No such file or directory
Lsusb:
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 0557:2221 ATEN International Co., Ltd Winbond Hermon
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
MachineType: Supermicro X9DRFR
MarkForUpload: True
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
LANGUAGE=en_CA:en
TERM=xterm-
PATH=(custom, no user)
LANG=en_CA.UTF-8
SHELL=/usr/bin/zsh
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=
ProcVersionSign
RelatedPackageV
linux-
linux-
linux-firmware 1.116
RfKill: Error: [Errno 2] No such file or directory
Tags: saucy
Uname: Linux 3.11.0-12-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:
dmi.bios.date: 09/11/2012
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1.0a
dmi.board.
dmi.board.name: X9DRFR
dmi.board.vendor: Supermicro
dmi.board.version: 0123456789
dmi.chassis.
dmi.chassis.type: 3
dmi.chassis.vendor: Supermicro
dmi.chassis.
dmi.modalias: dmi:bvnAmerican
dmi.product.name: X9DRFR
dmi.product.
dmi.sys.vendor: Supermicro
affects: | linux-meta (Ubuntu) → linux (Ubuntu) |
This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:
apport-collect 1252508
and then change the status of the bug to 'Confirmed'.
If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.
This change has been made by an automated script, maintained by the Ubuntu Kernel Team.