| 2017-07-24 17:36:46 |
Rafael David Tinoco |
bug |
|
|
added bug |
| 2017-07-24 17:37:39 |
Rafael David Tinoco |
linux (Ubuntu): assignee |
|
Rafael David Tinoco (inaddy) |
|
| 2017-07-24 17:37:42 |
Rafael David Tinoco |
linux (Ubuntu): importance |
Undecided |
Medium |
|
| 2017-07-24 17:37:45 |
Rafael David Tinoco |
linux (Ubuntu): status |
New |
Confirmed |
|
| 2017-07-24 17:41:12 |
Rafael David Tinoco |
linux (Ubuntu): milestone |
|
ubuntu-16.04.3 |
|
| 2017-07-24 17:41:24 |
Rafael David Tinoco |
nominated for series |
|
Ubuntu Xenial |
|
| 2017-07-24 19:22:22 |
Rafael David Tinoco |
description |
#### This scenario is testing [iscsi <-> scsi <-> disk <-> xfs]
[ 551.125604] sd 2:0:0:1: rejecting I/O to offline device
[ 551.125615] sd 2:0:0:1: rejecting I/O to offline device
[ 551.125627] sd 2:0:0:1: rejecting I/O to offline device
[ 551.125639] sd 2:0:0:1: rejecting I/O to offline device
[ 551.135216] XFS (sda1): metadata I/O error: block 0xeffe01 ("xfs_trans_read_buf_map") error 5 numblks 1
[ 551.135274] XFS (sda1): page discard on page ffffea0002a89cc0, inode 0x83, offset 6442385408.
# when XFS shuts down because of an error (or offline disk, example):
[ 551.850498] XFS (sda1): xfs_do_force_shutdown(0x2) called from line 1197 of file /build/linux-lts-xenial-roXrYH/linux-lts-xenial-4.4.0/fs/xfs/xfs_log.c. Return address = 0xffffffffc0300388
[ 551.850568] XFS (sda1): Log I/O Error Detected. Shutting down filesystem
[ 551.850618] XFS (sda1): xfs_log_force: error -5 returned.
[ 551.850630] XFS (sda1): Failing async write on buffer block 0x77ff08. Retrying async write.
[ 551.850634] XFS (sda1): Failing async write on buffer block 0x77ff10. Retrying async write.
[ 551.850638] XFS (sda1): Failing async write on buffer block 0x77ff01. Retrying async write.
[ 551.853171] XFS (sda1): Please umount the filesystem and rectify the problem(s)
[ 551.874131] XFS (sda1): metadata I/O error: block 0x1dffc49 ("xlog_iodone") error 5 numblks 64
[ 551.877993] XFS (sda1): xfs_do_force_shutdown(0x2) called from line 1197 of file /build/linux-lts-xenial-roXrYH/linux-lts-xenial-4.4.0/fs/xfs/xfs_log.c. Return address = 0xffffffffc0300388
[ 551.899036] XFS (sda1): xfs_log_force: error -5 returned.
[ 569.323074] XFS (sda1): xfs_log_force: error -5 returned.
[ 599.403085] XFS (sda1): xfs_log_force: error -5 returned.
[ 629.483111] XFS (sda1): xfs_log_force: error -5 returned.
[ 659.563115] XFS (sda1): xfs_log_force: error -5 returned.
[ 689.643014] XFS (sda1): xfs_log_force: error -5 returned.
# when I execute:
# sudo umount /dev/sda1:
[81634.923043] XFS (sda1): xfs_log_force: error -5 returned.
[81640.739097] XFS (sda1): xfs_log_force: error -5 returned.
[81640.739137] XFS (sda1): Unmounting Filesystem
[81640.739463] XFS (sda1): xfs_log_force: error -5 returned.
[81640.739508] XFS (sda1): xfs_log_force: error -5 returned.
[81640.742741] sd 2:0:0:1: rejecting I/O to offline device
[81640.745576] blk_update_request: 25 callbacks suppressed
[81640.745601] blk_update_request: I/O error, dev sda, sector 0
# i was able to umount and then to remove iscsi disk.
# but if i try to unload the xfs module:
inaddy@(trustyiscsicli):~$ sudo rmmod xfs
[82211.059301] =============================================================================
[82211.063764] BUG xfs_log_ticket (Tainted: G OE ): Objects remaining in xfs_log_ticket on kmem_cache_close()
[82211.067450] -----------------------------------------------------------------------------
[82211.067450]
[82211.070580] INFO: Slab 0xffffea0002eb7640 objects=22 used=1 fp=0xffff8800badd9f18 flags=0xffffc000000080
[82211.074430] INFO: Object 0xffff8800badd9228 @offset=552
[82211.076133] kmem_cache_destroy xfs_log_ticket: Slab cache still has objects
AND
[82211.059301] =============================================================================
[82211.063764] BUG xfs_log_ticket (Tainted: G OE ): Objects remaining in xfs_log_ticket on kmem_cache_close()
[82211.067450] -----------------------------------------------------------------------------
[82211.067450]
[82211.070570] Disabling lock debugging due to kernel taint
[82211.070580] INFO: Slab 0xffffea0002eb7640 objects=22 used=1 fp=0xffff8800badd9f18 flags=0xffffc000000080
[82211.073964] CPU: 3 PID: 32230 Comm: rmmod Tainted: G B OE 4.4.0-74-generic #95~14.04.1-Ubuntu
[82211.073970] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
[82211.073975] 0000000000000000 ffff8800baf53d28 ffffffff813dce3c ffffea0002eb7640
[82211.073984] ffff880036614300 ffff8800baf53e00 ffffffff811dc4b4 ffff880000000020
[82211.073991] ffff8800baf53e10 ffff8800baf53dc0 656a624f02e294c0 616d657220737463
[82211.074013] Call Trace:
[82211.074039] [<ffffffff813dce3c>] dump_stack+0x63/0x87
[82211.074066] [<ffffffff811dc4b4>] slab_err+0xb4/0xe0
[82211.074081] [<ffffffff8118cfd2>] ? __free_kmem_pages+0x22/0x40
[82211.074089] [<ffffffff811dcc6c>] ? __free_slab+0xcc/0x1a0
[82211.074097] [<ffffffff811df799>] ? __kmalloc+0x229/0x270
[82211.074106] [<ffffffff811e14b1>] ? __kmem_cache_shutdown+0x111/0x2b0
[82211.074127] [<ffffffff811e14d1>] __kmem_cache_shutdown+0x131/0x2b0
[82211.074157] [<ffffffff811a85ef>] kmem_cache_destroy+0x1bf/0x220
[82211.074289] [<ffffffffc031a368>] xfs_destroy_zones+0xeb/0x10a [xfs]
[82211.074382] [<ffffffffc031a3fc>] exit_xfs_fs+0x6f/0xc73 [xfs]
[82211.074395] [<ffffffff81101f3d>] SyS_delete_module+0x18d/0x220
[82211.074404] [<ffffffff8107986b>] ? exit_to_usermode_loop+0x78/0xa2
[82211.074416] [<ffffffff818066b6>] entry_SYSCALL_64_fastpath+0x16/0x75
[82211.074430] INFO: Object 0xffff8800badd9228 @offset=552
[82211.076133] kmem_cache_destroy xfs_log_ticket: Slab cache still has objects
[82211.078221] CPU: 3 PID: 32230 Comm: rmmod Tainted: G B OE 4.4.0-74-generic #95~14.04.1-Ubuntu
[82211.078226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
[82211.078230] 0000000000000000 ffff8800baf53e60 ffffffff813dce3c ffff880036614300
[82211.078238] ffff880036614300 ffff8800baf53ec8 ffffffff811a8648 0000000000000000
[82211.078245] 00ff8800baf53eb8 ffff8800baf53e80 ffff8800baf53e80 ffff8800baf53e90
[82211.078253] Call Trace:
[82211.078262] [<ffffffff813dce3c>] dump_stack+0x63/0x87
[82211.078271] [<ffffffff811a8648>] kmem_cache_destroy+0x218/0x220
[82211.078334] [<ffffffffc031a368>] xfs_destroy_zones+0xeb/0x10a [xfs]
[82211.078382] [<ffffffffc031a3fc>] exit_xfs_fs+0x6f/0xc73 [xfs]
[82211.078389] [<ffffffff81101f3d>] SyS_delete_module+0x18d/0x220
[82211.078395] [<ffffffff8107986b>] ? exit_to_usermode_loop+0x78/0xa2
[82211.078401] [<ffffffff818066b6>] entry_SYSCALL_64_fastpath+0x16/0x75
[82269.173849] SGI XFS with ACLs, security attributes, realtime, no debug enabled |
[Impact]
* xfs kernel memory leak in case of xfs shutdown due to i/o errors
* if xfs on iscsi, iscsi disconnection and module unload will case mem leak
[Test Case]
* configure tgtd with 1 lun and make it available through tcp/ip
* configure open-iscsi to map this lun with a timeout >= 1
* in tgtd server, drop iscsi packets and watch client to have i/o errors
* after sometime (depending on timeout) xfs will call shutdown
* after shutdown you try to remove xfs module and it will leak
[Regression Potential]
* based on upstream fix
* tested in the same environment
* potential damage to xfs
[Other Info]
Original Description:
#### This scenario is testing [iscsi <-> scsi <-> disk <-> xfs]
[ 551.125604] sd 2:0:0:1: rejecting I/O to offline device
[ 551.125615] sd 2:0:0:1: rejecting I/O to offline device
[ 551.125627] sd 2:0:0:1: rejecting I/O to offline device
[ 551.125639] sd 2:0:0:1: rejecting I/O to offline device
[ 551.135216] XFS (sda1): metadata I/O error: block 0xeffe01 ("xfs_trans_read_buf_map") error 5 numblks 1
[ 551.135274] XFS (sda1): page discard on page ffffea0002a89cc0, inode 0x83, offset 6442385408.
# when XFS shuts down because of an error (or offline disk, example):
[ 551.850498] XFS (sda1): xfs_do_force_shutdown(0x2) called from line 1197 of file /build/linux-lts-xenial-roXrYH/linux-lts-xenial-4.4.0/fs/xfs/xfs_log.c. Return address = 0xffffffffc0300388
[ 551.850568] XFS (sda1): Log I/O Error Detected. Shutting down filesystem
[ 551.850618] XFS (sda1): xfs_log_force: error -5 returned.
[ 551.850630] XFS (sda1): Failing async write on buffer block 0x77ff08. Retrying async write.
[ 551.850634] XFS (sda1): Failing async write on buffer block 0x77ff10. Retrying async write.
[ 551.850638] XFS (sda1): Failing async write on buffer block 0x77ff01. Retrying async write.
[ 551.853171] XFS (sda1): Please umount the filesystem and rectify the problem(s)
[ 551.874131] XFS (sda1): metadata I/O error: block 0x1dffc49 ("xlog_iodone") error 5 numblks 64
[ 551.877993] XFS (sda1): xfs_do_force_shutdown(0x2) called from line 1197 of file /build/linux-lts-xenial-roXrYH/linux-lts-xenial-4.4.0/fs/xfs/xfs_log.c. Return address = 0xffffffffc0300388
[ 551.899036] XFS (sda1): xfs_log_force: error -5 returned.
[ 569.323074] XFS (sda1): xfs_log_force: error -5 returned.
[ 599.403085] XFS (sda1): xfs_log_force: error -5 returned.
[ 629.483111] XFS (sda1): xfs_log_force: error -5 returned.
[ 659.563115] XFS (sda1): xfs_log_force: error -5 returned.
[ 689.643014] XFS (sda1): xfs_log_force: error -5 returned.
# when I execute:
# sudo umount /dev/sda1:
[81634.923043] XFS (sda1): xfs_log_force: error -5 returned.
[81640.739097] XFS (sda1): xfs_log_force: error -5 returned.
[81640.739137] XFS (sda1): Unmounting Filesystem
[81640.739463] XFS (sda1): xfs_log_force: error -5 returned.
[81640.739508] XFS (sda1): xfs_log_force: error -5 returned.
[81640.742741] sd 2:0:0:1: rejecting I/O to offline device
[81640.745576] blk_update_request: 25 callbacks suppressed
[81640.745601] blk_update_request: I/O error, dev sda, sector 0
# i was able to umount and then to remove iscsi disk.
# but if i try to unload the xfs module:
inaddy@(trustyiscsicli):~$ sudo rmmod xfs
[82211.059301] =============================================================================
[82211.063764] BUG xfs_log_ticket (Tainted: G OE ): Objects remaining in xfs_log_ticket on kmem_cache_close()
[82211.067450] -----------------------------------------------------------------------------
[82211.067450]
[82211.070580] INFO: Slab 0xffffea0002eb7640 objects=22 used=1 fp=0xffff8800badd9f18 flags=0xffffc000000080
[82211.074430] INFO: Object 0xffff8800badd9228 @offset=552
[82211.076133] kmem_cache_destroy xfs_log_ticket: Slab cache still has objects
AND
[82211.059301] =============================================================================
[82211.063764] BUG xfs_log_ticket (Tainted: G OE ): Objects remaining in xfs_log_ticket on kmem_cache_close()
[82211.067450] -----------------------------------------------------------------------------
[82211.067450]
[82211.070570] Disabling lock debugging due to kernel taint
[82211.070580] INFO: Slab 0xffffea0002eb7640 objects=22 used=1 fp=0xffff8800badd9f18 flags=0xffffc000000080
[82211.073964] CPU: 3 PID: 32230 Comm: rmmod Tainted: G B OE 4.4.0-74-generic #95~14.04.1-Ubuntu
[82211.073970] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
[82211.073975] 0000000000000000 ffff8800baf53d28 ffffffff813dce3c ffffea0002eb7640
[82211.073984] ffff880036614300 ffff8800baf53e00 ffffffff811dc4b4 ffff880000000020
[82211.073991] ffff8800baf53e10 ffff8800baf53dc0 656a624f02e294c0 616d657220737463
[82211.074013] Call Trace:
[82211.074039] [<ffffffff813dce3c>] dump_stack+0x63/0x87
[82211.074066] [<ffffffff811dc4b4>] slab_err+0xb4/0xe0
[82211.074081] [<ffffffff8118cfd2>] ? __free_kmem_pages+0x22/0x40
[82211.074089] [<ffffffff811dcc6c>] ? __free_slab+0xcc/0x1a0
[82211.074097] [<ffffffff811df799>] ? __kmalloc+0x229/0x270
[82211.074106] [<ffffffff811e14b1>] ? __kmem_cache_shutdown+0x111/0x2b0
[82211.074127] [<ffffffff811e14d1>] __kmem_cache_shutdown+0x131/0x2b0
[82211.074157] [<ffffffff811a85ef>] kmem_cache_destroy+0x1bf/0x220
[82211.074289] [<ffffffffc031a368>] xfs_destroy_zones+0xeb/0x10a [xfs]
[82211.074382] [<ffffffffc031a3fc>] exit_xfs_fs+0x6f/0xc73 [xfs]
[82211.074395] [<ffffffff81101f3d>] SyS_delete_module+0x18d/0x220
[82211.074404] [<ffffffff8107986b>] ? exit_to_usermode_loop+0x78/0xa2
[82211.074416] [<ffffffff818066b6>] entry_SYSCALL_64_fastpath+0x16/0x75
[82211.074430] INFO: Object 0xffff8800badd9228 @offset=552
[82211.076133] kmem_cache_destroy xfs_log_ticket: Slab cache still has objects
[82211.078221] CPU: 3 PID: 32230 Comm: rmmod Tainted: G B OE 4.4.0-74-generic #95~14.04.1-Ubuntu
[82211.078226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
[82211.078230] 0000000000000000 ffff8800baf53e60 ffffffff813dce3c ffff880036614300
[82211.078238] ffff880036614300 ffff8800baf53ec8 ffffffff811a8648 0000000000000000
[82211.078245] 00ff8800baf53eb8 ffff8800baf53e80 ffff8800baf53e80 ffff8800baf53e90
[82211.078253] Call Trace:
[82211.078262] [<ffffffff813dce3c>] dump_stack+0x63/0x87
[82211.078271] [<ffffffff811a8648>] kmem_cache_destroy+0x218/0x220
[82211.078334] [<ffffffffc031a368>] xfs_destroy_zones+0xeb/0x10a [xfs]
[82211.078382] [<ffffffffc031a3fc>] exit_xfs_fs+0x6f/0xc73 [xfs]
[82211.078389] [<ffffffff81101f3d>] SyS_delete_module+0x18d/0x220
[82211.078395] [<ffffffff8107986b>] ? exit_to_usermode_loop+0x78/0xa2
[82211.078401] [<ffffffff818066b6>] entry_SYSCALL_64_fastpath+0x16/0x75
[82269.173849] SGI XFS with ACLs, security attributes, realtime, no debug enabled |
|
| 2017-07-25 13:30:31 |
Rafael David Tinoco |
linux (Ubuntu): importance |
Medium |
Low |
|
| 2017-07-25 20:48:39 |
Rafael David Tinoco |
description |
[Impact]
* xfs kernel memory leak in case of xfs shutdown due to i/o errors
* if xfs on iscsi, iscsi disconnection and module unload will case mem leak
[Test Case]
* configure tgtd with 1 lun and make it available through tcp/ip
* configure open-iscsi to map this lun with a timeout >= 1
* in tgtd server, drop iscsi packets and watch client to have i/o errors
* after sometime (depending on timeout) xfs will call shutdown
* after shutdown you try to remove xfs module and it will leak
[Regression Potential]
* based on upstream fix
* tested in the same environment
* potential damage to xfs
[Other Info]
Original Description:
#### This scenario is testing [iscsi <-> scsi <-> disk <-> xfs]
[ 551.125604] sd 2:0:0:1: rejecting I/O to offline device
[ 551.125615] sd 2:0:0:1: rejecting I/O to offline device
[ 551.125627] sd 2:0:0:1: rejecting I/O to offline device
[ 551.125639] sd 2:0:0:1: rejecting I/O to offline device
[ 551.135216] XFS (sda1): metadata I/O error: block 0xeffe01 ("xfs_trans_read_buf_map") error 5 numblks 1
[ 551.135274] XFS (sda1): page discard on page ffffea0002a89cc0, inode 0x83, offset 6442385408.
# when XFS shuts down because of an error (or offline disk, example):
[ 551.850498] XFS (sda1): xfs_do_force_shutdown(0x2) called from line 1197 of file /build/linux-lts-xenial-roXrYH/linux-lts-xenial-4.4.0/fs/xfs/xfs_log.c. Return address = 0xffffffffc0300388
[ 551.850568] XFS (sda1): Log I/O Error Detected. Shutting down filesystem
[ 551.850618] XFS (sda1): xfs_log_force: error -5 returned.
[ 551.850630] XFS (sda1): Failing async write on buffer block 0x77ff08. Retrying async write.
[ 551.850634] XFS (sda1): Failing async write on buffer block 0x77ff10. Retrying async write.
[ 551.850638] XFS (sda1): Failing async write on buffer block 0x77ff01. Retrying async write.
[ 551.853171] XFS (sda1): Please umount the filesystem and rectify the problem(s)
[ 551.874131] XFS (sda1): metadata I/O error: block 0x1dffc49 ("xlog_iodone") error 5 numblks 64
[ 551.877993] XFS (sda1): xfs_do_force_shutdown(0x2) called from line 1197 of file /build/linux-lts-xenial-roXrYH/linux-lts-xenial-4.4.0/fs/xfs/xfs_log.c. Return address = 0xffffffffc0300388
[ 551.899036] XFS (sda1): xfs_log_force: error -5 returned.
[ 569.323074] XFS (sda1): xfs_log_force: error -5 returned.
[ 599.403085] XFS (sda1): xfs_log_force: error -5 returned.
[ 629.483111] XFS (sda1): xfs_log_force: error -5 returned.
[ 659.563115] XFS (sda1): xfs_log_force: error -5 returned.
[ 689.643014] XFS (sda1): xfs_log_force: error -5 returned.
# when I execute:
# sudo umount /dev/sda1:
[81634.923043] XFS (sda1): xfs_log_force: error -5 returned.
[81640.739097] XFS (sda1): xfs_log_force: error -5 returned.
[81640.739137] XFS (sda1): Unmounting Filesystem
[81640.739463] XFS (sda1): xfs_log_force: error -5 returned.
[81640.739508] XFS (sda1): xfs_log_force: error -5 returned.
[81640.742741] sd 2:0:0:1: rejecting I/O to offline device
[81640.745576] blk_update_request: 25 callbacks suppressed
[81640.745601] blk_update_request: I/O error, dev sda, sector 0
# i was able to umount and then to remove iscsi disk.
# but if i try to unload the xfs module:
inaddy@(trustyiscsicli):~$ sudo rmmod xfs
[82211.059301] =============================================================================
[82211.063764] BUG xfs_log_ticket (Tainted: G OE ): Objects remaining in xfs_log_ticket on kmem_cache_close()
[82211.067450] -----------------------------------------------------------------------------
[82211.067450]
[82211.070580] INFO: Slab 0xffffea0002eb7640 objects=22 used=1 fp=0xffff8800badd9f18 flags=0xffffc000000080
[82211.074430] INFO: Object 0xffff8800badd9228 @offset=552
[82211.076133] kmem_cache_destroy xfs_log_ticket: Slab cache still has objects
AND
[82211.059301] =============================================================================
[82211.063764] BUG xfs_log_ticket (Tainted: G OE ): Objects remaining in xfs_log_ticket on kmem_cache_close()
[82211.067450] -----------------------------------------------------------------------------
[82211.067450]
[82211.070570] Disabling lock debugging due to kernel taint
[82211.070580] INFO: Slab 0xffffea0002eb7640 objects=22 used=1 fp=0xffff8800badd9f18 flags=0xffffc000000080
[82211.073964] CPU: 3 PID: 32230 Comm: rmmod Tainted: G B OE 4.4.0-74-generic #95~14.04.1-Ubuntu
[82211.073970] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
[82211.073975] 0000000000000000 ffff8800baf53d28 ffffffff813dce3c ffffea0002eb7640
[82211.073984] ffff880036614300 ffff8800baf53e00 ffffffff811dc4b4 ffff880000000020
[82211.073991] ffff8800baf53e10 ffff8800baf53dc0 656a624f02e294c0 616d657220737463
[82211.074013] Call Trace:
[82211.074039] [<ffffffff813dce3c>] dump_stack+0x63/0x87
[82211.074066] [<ffffffff811dc4b4>] slab_err+0xb4/0xe0
[82211.074081] [<ffffffff8118cfd2>] ? __free_kmem_pages+0x22/0x40
[82211.074089] [<ffffffff811dcc6c>] ? __free_slab+0xcc/0x1a0
[82211.074097] [<ffffffff811df799>] ? __kmalloc+0x229/0x270
[82211.074106] [<ffffffff811e14b1>] ? __kmem_cache_shutdown+0x111/0x2b0
[82211.074127] [<ffffffff811e14d1>] __kmem_cache_shutdown+0x131/0x2b0
[82211.074157] [<ffffffff811a85ef>] kmem_cache_destroy+0x1bf/0x220
[82211.074289] [<ffffffffc031a368>] xfs_destroy_zones+0xeb/0x10a [xfs]
[82211.074382] [<ffffffffc031a3fc>] exit_xfs_fs+0x6f/0xc73 [xfs]
[82211.074395] [<ffffffff81101f3d>] SyS_delete_module+0x18d/0x220
[82211.074404] [<ffffffff8107986b>] ? exit_to_usermode_loop+0x78/0xa2
[82211.074416] [<ffffffff818066b6>] entry_SYSCALL_64_fastpath+0x16/0x75
[82211.074430] INFO: Object 0xffff8800badd9228 @offset=552
[82211.076133] kmem_cache_destroy xfs_log_ticket: Slab cache still has objects
[82211.078221] CPU: 3 PID: 32230 Comm: rmmod Tainted: G B OE 4.4.0-74-generic #95~14.04.1-Ubuntu
[82211.078226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
[82211.078230] 0000000000000000 ffff8800baf53e60 ffffffff813dce3c ffff880036614300
[82211.078238] ffff880036614300 ffff8800baf53ec8 ffffffff811a8648 0000000000000000
[82211.078245] 00ff8800baf53eb8 ffff8800baf53e80 ffff8800baf53e80 ffff8800baf53e90
[82211.078253] Call Trace:
[82211.078262] [<ffffffff813dce3c>] dump_stack+0x63/0x87
[82211.078271] [<ffffffff811a8648>] kmem_cache_destroy+0x218/0x220
[82211.078334] [<ffffffffc031a368>] xfs_destroy_zones+0xeb/0x10a [xfs]
[82211.078382] [<ffffffffc031a3fc>] exit_xfs_fs+0x6f/0xc73 [xfs]
[82211.078389] [<ffffffff81101f3d>] SyS_delete_module+0x18d/0x220
[82211.078395] [<ffffffff8107986b>] ? exit_to_usermode_loop+0x78/0xa2
[82211.078401] [<ffffffff818066b6>] entry_SYSCALL_64_fastpath+0x16/0x75
[82269.173849] SGI XFS with ACLs, security attributes, realtime, no debug enabled |
[Impact]
* xfs kernel memory leak in case of xfs shutdown due to i/o errors
* if xfs on iscsi, iscsi disconnection and module unload will case mem leak
[Test Case]
* configure tgtd with 1 lun and make it available through tcp/ip
* configure open-iscsi to map this lun
* make sure node.session.timeo.replacement_timeout = 0 in iscsid.conf
* mount a xfs volume using the lun from tgtd host, run bonnie -d /xfsdir
* in tgtd server, drop iscsi packets and watch client to have i/o errors
* after sometime (depending on timeout) xfs will call shutdown
* make sure the i/o errors led to xfs shutdown (comment #3)
* after shutdown you try to remove xfs module and it will leak
[Regression Potential]
* based on upstream fix
* tested in the same environment
* potential damage to xfs
[Other Info]
Original Description:
#### This scenario is testing [iscsi <-> scsi <-> disk <-> xfs]
[ 551.125604] sd 2:0:0:1: rejecting I/O to offline device
[ 551.125615] sd 2:0:0:1: rejecting I/O to offline device
[ 551.125627] sd 2:0:0:1: rejecting I/O to offline device
[ 551.125639] sd 2:0:0:1: rejecting I/O to offline device
[ 551.135216] XFS (sda1): metadata I/O error: block 0xeffe01 ("xfs_trans_read_buf_map") error 5 numblks 1
[ 551.135274] XFS (sda1): page discard on page ffffea0002a89cc0, inode 0x83, offset 6442385408.
# when XFS shuts down because of an error (or offline disk, example):
[ 551.850498] XFS (sda1): xfs_do_force_shutdown(0x2) called from line 1197 of file /build/linux-lts-xenial-roXrYH/linux-lts-xenial-4.4.0/fs/xfs/xfs_log.c. Return address = 0xffffffffc0300388
[ 551.850568] XFS (sda1): Log I/O Error Detected. Shutting down filesystem
[ 551.850618] XFS (sda1): xfs_log_force: error -5 returned.
[ 551.850630] XFS (sda1): Failing async write on buffer block 0x77ff08. Retrying async write.
[ 551.850634] XFS (sda1): Failing async write on buffer block 0x77ff10. Retrying async write.
[ 551.850638] XFS (sda1): Failing async write on buffer block 0x77ff01. Retrying async write.
[ 551.853171] XFS (sda1): Please umount the filesystem and rectify the problem(s)
[ 551.874131] XFS (sda1): metadata I/O error: block 0x1dffc49 ("xlog_iodone") error 5 numblks 64
[ 551.877993] XFS (sda1): xfs_do_force_shutdown(0x2) called from line 1197 of file /build/linux-lts-xenial-roXrYH/linux-lts-xenial-4.4.0/fs/xfs/xfs_log.c. Return address = 0xffffffffc0300388
[ 551.899036] XFS (sda1): xfs_log_force: error -5 returned.
[ 569.323074] XFS (sda1): xfs_log_force: error -5 returned.
[ 599.403085] XFS (sda1): xfs_log_force: error -5 returned.
[ 629.483111] XFS (sda1): xfs_log_force: error -5 returned.
[ 659.563115] XFS (sda1): xfs_log_force: error -5 returned.
[ 689.643014] XFS (sda1): xfs_log_force: error -5 returned.
# when I execute:
# sudo umount /dev/sda1:
[81634.923043] XFS (sda1): xfs_log_force: error -5 returned.
[81640.739097] XFS (sda1): xfs_log_force: error -5 returned.
[81640.739137] XFS (sda1): Unmounting Filesystem
[81640.739463] XFS (sda1): xfs_log_force: error -5 returned.
[81640.739508] XFS (sda1): xfs_log_force: error -5 returned.
[81640.742741] sd 2:0:0:1: rejecting I/O to offline device
[81640.745576] blk_update_request: 25 callbacks suppressed
[81640.745601] blk_update_request: I/O error, dev sda, sector 0
# i was able to umount and then to remove iscsi disk.
# but if i try to unload the xfs module:
inaddy@(trustyiscsicli):~$ sudo rmmod xfs
[82211.059301] =============================================================================
[82211.063764] BUG xfs_log_ticket (Tainted: G OE ): Objects remaining in xfs_log_ticket on kmem_cache_close()
[82211.067450] -----------------------------------------------------------------------------
[82211.067450]
[82211.070580] INFO: Slab 0xffffea0002eb7640 objects=22 used=1 fp=0xffff8800badd9f18 flags=0xffffc000000080
[82211.074430] INFO: Object 0xffff8800badd9228 @offset=552
[82211.076133] kmem_cache_destroy xfs_log_ticket: Slab cache still has objects
AND
[82211.059301] =============================================================================
[82211.063764] BUG xfs_log_ticket (Tainted: G OE ): Objects remaining in xfs_log_ticket on kmem_cache_close()
[82211.067450] -----------------------------------------------------------------------------
[82211.067450]
[82211.070570] Disabling lock debugging due to kernel taint
[82211.070580] INFO: Slab 0xffffea0002eb7640 objects=22 used=1 fp=0xffff8800badd9f18 flags=0xffffc000000080
[82211.073964] CPU: 3 PID: 32230 Comm: rmmod Tainted: G B OE 4.4.0-74-generic #95~14.04.1-Ubuntu
[82211.073970] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
[82211.073975] 0000000000000000 ffff8800baf53d28 ffffffff813dce3c ffffea0002eb7640
[82211.073984] ffff880036614300 ffff8800baf53e00 ffffffff811dc4b4 ffff880000000020
[82211.073991] ffff8800baf53e10 ffff8800baf53dc0 656a624f02e294c0 616d657220737463
[82211.074013] Call Trace:
[82211.074039] [<ffffffff813dce3c>] dump_stack+0x63/0x87
[82211.074066] [<ffffffff811dc4b4>] slab_err+0xb4/0xe0
[82211.074081] [<ffffffff8118cfd2>] ? __free_kmem_pages+0x22/0x40
[82211.074089] [<ffffffff811dcc6c>] ? __free_slab+0xcc/0x1a0
[82211.074097] [<ffffffff811df799>] ? __kmalloc+0x229/0x270
[82211.074106] [<ffffffff811e14b1>] ? __kmem_cache_shutdown+0x111/0x2b0
[82211.074127] [<ffffffff811e14d1>] __kmem_cache_shutdown+0x131/0x2b0
[82211.074157] [<ffffffff811a85ef>] kmem_cache_destroy+0x1bf/0x220
[82211.074289] [<ffffffffc031a368>] xfs_destroy_zones+0xeb/0x10a [xfs]
[82211.074382] [<ffffffffc031a3fc>] exit_xfs_fs+0x6f/0xc73 [xfs]
[82211.074395] [<ffffffff81101f3d>] SyS_delete_module+0x18d/0x220
[82211.074404] [<ffffffff8107986b>] ? exit_to_usermode_loop+0x78/0xa2
[82211.074416] [<ffffffff818066b6>] entry_SYSCALL_64_fastpath+0x16/0x75
[82211.074430] INFO: Object 0xffff8800badd9228 @offset=552
[82211.076133] kmem_cache_destroy xfs_log_ticket: Slab cache still has objects
[82211.078221] CPU: 3 PID: 32230 Comm: rmmod Tainted: G B OE 4.4.0-74-generic #95~14.04.1-Ubuntu
[82211.078226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
[82211.078230] 0000000000000000 ffff8800baf53e60 ffffffff813dce3c ffff880036614300
[82211.078238] ffff880036614300 ffff8800baf53ec8 ffffffff811a8648 0000000000000000
[82211.078245] 00ff8800baf53eb8 ffff8800baf53e80 ffff8800baf53e80 ffff8800baf53e90
[82211.078253] Call Trace:
[82211.078262] [<ffffffff813dce3c>] dump_stack+0x63/0x87
[82211.078271] [<ffffffff811a8648>] kmem_cache_destroy+0x218/0x220
[82211.078334] [<ffffffffc031a368>] xfs_destroy_zones+0xeb/0x10a [xfs]
[82211.078382] [<ffffffffc031a3fc>] exit_xfs_fs+0x6f/0xc73 [xfs]
[82211.078389] [<ffffffff81101f3d>] SyS_delete_module+0x18d/0x220
[82211.078395] [<ffffffff8107986b>] ? exit_to_usermode_loop+0x78/0xa2
[82211.078401] [<ffffffff818066b6>] entry_SYSCALL_64_fastpath+0x16/0x75
[82269.173849] SGI XFS with ACLs, security attributes, realtime, no debug enabled |
|
| 2017-07-25 20:51:07 |
Rafael David Tinoco |
bug |
|
|
added subscriber Szilard Cserey |
| 2017-07-27 15:19:19 |
Stefan Bader |
bug task added |
|
linux (Ubuntu Xenial) |
|
| 2017-08-07 03:04:18 |
Rafael David Tinoco |
linux (Ubuntu Xenial): status |
New |
In Progress |
|
| 2017-08-07 03:04:20 |
Rafael David Tinoco |
linux (Ubuntu): status |
Confirmed |
In Progress |
|
| 2017-08-07 03:04:25 |
Rafael David Tinoco |
linux (Ubuntu Xenial): importance |
Undecided |
Low |
|
| 2017-08-07 03:04:27 |
Rafael David Tinoco |
linux (Ubuntu Xenial): assignee |
|
Rafael David Tinoco (inaddy) |
|
| 2017-08-07 03:04:37 |
Rafael David Tinoco |
linux (Ubuntu Xenial): milestone |
|
ubuntu-16.04.3 |
|
| 2017-08-08 08:12:31 |
Kleber Sacilotto de Souza |
linux (Ubuntu Xenial): status |
In Progress |
Fix Committed |
|
| 2017-08-08 12:54:06 |
Rafael David Tinoco |
tags |
|
sts |
|
| 2017-08-16 16:33:22 |
Kleber Sacilotto de Souza |
tags |
sts |
sts verification-needed-xenial |
|
| 2017-08-23 11:40:30 |
Rafael David Tinoco |
tags |
sts verification-needed-xenial |
sts verification-done |
|
| 2017-08-28 10:14:02 |
Launchpad Janitor |
linux (Ubuntu Xenial): status |
Fix Committed |
Fix Released |
|
| 2017-08-28 10:14:02 |
Launchpad Janitor |
cve linked |
|
2015-7837 |
|
| 2017-08-28 10:14:02 |
Launchpad Janitor |
cve linked |
|
2017-1000111 |
|
| 2017-08-28 10:14:02 |
Launchpad Janitor |
cve linked |
|
2017-1000112 |
|
| 2017-08-28 10:14:02 |
Launchpad Janitor |
cve linked |
|
2017-7495 |
|
| 2018-05-19 01:37:46 |
Rafael David Tinoco |
linux (Ubuntu): status |
In Progress |
Fix Released |
|
| 2018-05-19 01:37:50 |
Rafael David Tinoco |
linux (Ubuntu): assignee |
Rafael David Tinoco (inaddy) |
|
|
| 2018-05-19 01:37:53 |
Rafael David Tinoco |
linux (Ubuntu Xenial): assignee |
Rafael David Tinoco (inaddy) |
|
|
