We are trying to debug the kernel using KASAN and we found that when a VM is booting in our cloud, on the virtualised kernel, there is a use-after-free access that should not be there.
The failing VM was running on a host with kernel 3.13.0-66-generic (trusty). Hosts' qemu version: 1:2.2+dfsg-5expubuntu9.3~cloud0. Hosts' seabios: 1.7.5-1ubuntu1~cloud0
The flavour of this VM is 4 CPUs, 8G RAM, 80G of root disk, 0 G swap and 0 G ephemeral disk.
Here is the trace from KASAN (from the VM):
The error message can be observed in the dmesg when the guest VM booted with v3.13.0-65 with KASAN enabled and "slub_debug=PU,kmalloc-32" in kernel command line.
==================================================================
BUG: KASan: out of bounds access in blk_mq_register_disk+0x193/0x260 at addr ffff8801f43f4d90
Read of size 8 by task swapper/0/1
=============================================================================
BUG kmalloc-32 (Not tainted): kasan: bad access detected
-----------------------------------------------------------------------------
Disabling lock debugging due to kernel taint
INFO: Allocated in blk_mq_init_hw_queues+0x778/0x920 age=5 cpu=1 pid=1
__slab_alloc+0x4f8/0x560
__kmalloc_node+0xad/0x310
blk_mq_init_hw_queues+0x778/0x920
blk_mq_init_queue+0x5f7/0x6c0
virtblk_probe+0x207/0x980
virtio_dev_probe+0x1be/0x280
driver_probe_device+0xe2/0x5c0
__driver_attach+0xc3/0xd0
bus_for_each_dev+0x95/0xe0
driver_attach+0x2b/0x30
bus_add_driver+0x268/0x360
driver_register+0xd3/0x1a0
register_virtio_driver+0x3c/0x60
init+0x53/0x80
do_one_initcall+0xda/0x1a0
kernel_init_freeable+0x1eb/0x27e
INFO: Freed in kzfree+0x2d/0x40 age=13 cpu=0 pid=8
__slab_free+0x2ab/0x3f0
kfree+0x161/0x170
kzfree+0x2d/0x40
aa_free_task_context+0x5d/0xa0
apparmor_cred_free+0x24/0x40
security_cred_free+0x2b/0x30
put_cred_rcu+0x38/0x140
rcu_nocb_kthread+0x25a/0x410
kthread+0x101/0x120
ret_from_fork+0x58/0x90
INFO: Slab 0xffffea0007d0fd00 objects=23 used=21 fp=0xffff8801f43f52d0 flags=0x2ffff0000004080
INFO: Object 0xffff8801f43f4d70 @offset=3440 fp=0xffff8801f43f5830
Bytes b4 ffff8801f43f4d60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object ffff8801f43f4d70: 00 ac 61 f7 01 88 ff ff 00 ac 69 f7 01 88 ff ff ..a.......i.....
Object ffff8801f43f4d80: 00 ac 71 f7 01 88 ff ff 00 ac 79 f7 01 88 ff ff ..q.......y.....
CPU: 0 PID: 1 Comm: swapper/0 Tainted: G B 3.13.0-65-generic #105
Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.7.5-20150310_111955-batsu 04/01/2014
ffffea0007d0fd00 ffff8801f40cf9a8 ffffffff81a6ce35 ffff8801f7001c00
ffff8801f40cf9d8 ffffffff81244aed ffff8801f7001c00 ffffea0007d0fd00
ffff8801f43f4d70 ffff8801f779ac98 ffff8801f40cfa00 ffffffff8124ac36
Call Trace:
[<ffffffff81a6ce35>] dump_stack+0x45/0x56
[<ffffffff81244aed>] print_trailer+0xfd/0x170
[<ffffffff8124ac36>] object_err+0x36/0x40
[<ffffffff8124cbf9>] kasan_report_error+0x1e9/0x3a0
[<ffffffff81319427>] ? sysfs_get+0x17/0x50
[<ffffffff814dee6b>] ? kobject_add_internal+0x29b/0x4a0
[<ffffffff8124d260>] kasan_report+0x40/0x50
[<ffffffff81696f00>] ? dev_printk_emit+0x20/0x40
[<ffffffff814ae7c3>] ? blk_mq_register_disk+0x193/0x260
[<ffffffff8124bee9>] __asan_load8+0x69/0xa0
[<ffffffff814ae7c3>] blk_mq_register_disk+0x193/0x260
[<ffffffff814a1572>] blk_register_queue+0xd2/0x170
[<ffffffff814b24cf>] add_disk+0x31f/0x720
[<ffffffff816ced9a>] virtblk_probe+0x58a/0x980
[<ffffffff816cd4c0>] ? virtblk_restore+0x100/0x100
[<ffffffff81601b8e>] virtio_dev_probe+0x1be/0x280
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d0d2>] driver_probe_device+0xe2/0x5c0
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d6e3>] __driver_attach+0xc3/0xd0
[<ffffffff8169a355>] bus_for_each_dev+0x95/0xe0
[<ffffffff8169c89b>] driver_attach+0x2b/0x30
[<ffffffff8169c298>] bus_add_driver+0x268/0x360
[<ffffffff8169dfe3>] driver_register+0xd3/0x1a0
[<ffffffff8218e4b9>] ? loop_init+0x14b/0x14b
[<ffffffff8160213c>] register_virtio_driver+0x3c/0x60
[<ffffffff8218e50c>] init+0x53/0x80
[<ffffffff8100212a>] do_one_initcall+0xda/0x1a0
[<ffffffff8213816b>] kernel_init_freeable+0x1eb/0x27e
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
[<ffffffff81a5bcde>] kernel_init+0xe/0x130
[<ffffffff81a83028>] ret_from_fork+0x58/0x90
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
Memory state around the buggy address:
ffff8801f43f4c80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff8801f43f4d00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc 00 00
>ffff8801f43f4d80: 00 00 fc fc fc fc fc fc fc fc fc fc fc fc fc fc
^
ffff8801f43f4e00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff8801f43f4e80: fc fc fc fc fc fc fc fc fc 00 00 00 00 fc fc fc
==================================================================
This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:
apport-collect 1534054
and then change the status of the bug to 'Confirmed'.
If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.
This change has been made by an automated script, maintained by the Ubuntu Kernel Team.