Ubuntu
linux package

running 'rr' make check causes kernel "hard LOCKUP"

Bug #1535307 reported by Peter Maydell
264
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Triaged
High
Unassigned

Bug Description

Running the 'rr' make check on Ubuntu Trusty causes the machine to become unusable because the kernel crashes. (rr build-and-make-check instructions: https://github.com/mozilla/rr/wiki/Building-And-Installing)

This is 3.13.0-74-generic #118-Ubuntu for x86_64.

This is rr bug https://github.com/mozilla/rr/issues/1614 (which includes information about a Debian Jessie kernel crash initially and also the Ubuntu crash later on in the same report), but since crashing the kernel is a kernel bug I'm reporting that part here.

kern.log reports:

Jan 18 12:33:12 e104462 kernel: [ 367.805326] ------------[ cut here ]------------
Jan 18 12:33:12 e104462 kernel: [ 367.805333] WARNING: CPU: 2 PID: 4294 at /build/linux-_xRakU/linux-3.13.0/kernel/watchdog.c:245 watchdog_overflow_callback+0x
9c/0xd0()
Jan 18 12:33:12 e104462 kernel: [ 367.805334] Watchdog detected hard LOCKUP on cpu 2
Jan 18 12:33:12 e104462 kernel: [ 367.805335] Modules linked in: pci_stub vboxpci(OX) vboxnetadp(OX) vboxnetflt(OX) vboxdrv(OX) bnep rfcomm bluetooth nfsd auth_rpcgss nfs_acl binfmt_misc nfs lockd sunrpc fscache dm_crypt snd_hda_codec_hdmi intel_rapl snd_hda_codec_realtek x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel snd_hda_intel snd_hda_codec kvm pl2303 usbserial snd_hwdep snd_pcm crct10dif_pclmul crc32_pclmul snd_page_alloc snd_seq_midi snd_seq_midi_event snd_rawmidi aesni_intel aes_x86_64 lrw snd_seq gf128mul glue_helper ablk_helper cryptd snd_seq_device serio_raw snd_timer lpc_ich snd soundcore mei_me mei shpchp mac_hid parport_pc ppdev lp parport hid_generic usbhid hid usb_storage nouveau mxm_wmi i2c_algo_bit ttm drm_kms_helper e1000e psmouse ahci drm ptp libahci pps_core video wmi
Jan 18 12:33:12 e104462 kernel: [ 367.805365] CPU: 2 PID: 4294 Comm: rr:async_signal Tainted: G OX 3.13.0-74-generic #118-Ubuntu
Jan 18 12:33:12 e104462 kernel: [ 367.805366] Hardware name: LENOVO 32281S1/MAHOBAY, BIOS 9SKT71AUS 07/02/2013
Jan 18 12:33:12 e104462 kernel: [ 367.805367] 0000000000000009 ffff8804033abc78 ffffffff81724b70 ffff8804033abcc0
Jan 18 12:33:12 e104462 kernel: [ 367.805369] ffff8804033abcb0 ffffffff810678bd ffff880407798000 0000000000000000
Jan 18 12:33:12 e104462 kernel: [ 367.805370] ffff8804033abde0 0000000000000000 ffff8804033abf58 ffff8804033abd10
Jan 18 12:33:12 e104462 kernel: [ 367.805372] Call Trace:
Jan 18 12:33:12 e104462 kernel: [ 367.805376] [<ffffffff81724b70>] dump_stack+0x45/0x56
Jan 18 12:33:12 e104462 kernel: [ 367.805379] [<ffffffff810678bd>] warn_slowpath_common+0x7d/0xa0
Jan 18 12:33:12 e104462 kernel: [ 367.805380] [<ffffffff8106792c>] warn_slowpath_fmt+0x4c/0x50
Jan 18 12:33:12 e104462 kernel: [ 367.805382] [<ffffffff8110e3b0>] ? restart_watchdog_hrtimer+0x50/0x50
Jan 18 12:33:12 e104462 kernel: [ 367.805384] [<ffffffff8110e44c>] watchdog_overflow_callback+0x9c/0xd0
Jan 18 12:33:12 e104462 kernel: [ 367.805386] [<ffffffff811461ee>] __perf_event_overflow+0x8e/0x250
Jan 18 12:33:12 e104462 kernel: [ 367.805389] [<ffffffff810293e8>] ? x86_perf_event_set_period+0xe8/0x150
Jan 18 12:33:12 e104462 kernel: [ 367.805390] [<ffffffff81146d14>] perf_event_overflow+0x14/0x20
Jan 18 12:33:12 e104462 kernel: [ 367.805392] [<ffffffff8103084d>] intel_pmu_handle_irq+0x1ed/0x3f0
Jan 18 12:33:12 e104462 kernel: [ 367.805395] [<ffffffff8172ebdb>] perf_event_nmi_handler+0x2b/0x50
Jan 18 12:33:12 e104462 kernel: [ 367.805397] [<ffffffff8172e358>] nmi_handle.isra.2+0x88/0x180
Jan 18 12:33:12 e104462 kernel: [ 367.805399] [<ffffffff8172e619>] do_nmi+0x1c9/0x3e0
Jan 18 12:33:12 e104462 kernel: [ 367.805400] [<ffffffff8172d6ea>] nmi+0x5a/0xbf
Jan 18 12:33:12 e104462 kernel: [ 367.805402] ---[ end trace 1511edbd209fafac ]---

followed a little later by

Jan 18 12:33:33 e104462 kernel: [ 392.175012] BUG: soft lockup - CPU#1 stuck for 23s! [condvar_stress-:3380]
Jan 18 12:33:33 e104462 kernel: [ 392.175014] Modules linked in: pci_stub vboxpci(OX) vboxnetadp(OX) vboxnetflt(OX) vboxdrv(OX) bnep rfcomm bluetooth nfsd auth_rpcgss nfs_acl binfmt_misc nfs lockd sunrpc fscache dm_crypt snd_hda_codec_hdmi intel_rapl snd_hda_codec_realtek x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel snd_hda_intel snd_hda_codec kvm pl2303 usbserial snd_hwdep snd_pcm crct10dif_pclmul crc32_pclmul snd_page_alloc snd_seq_midi snd_seq_midi_event snd_rawmidi aesni_intel aes_x86_64 lrw snd_seq gf128mul glue_helper ablk_helper cryptd snd_seq_device serio_raw snd_timer lpc_ich snd soundcore mei_me mei shpchp mac_hid parport_pc ppdev lp parport hid_generic usbhid hid usb_storage nouveau mxm_wmi i2c_algo_bit ttm drm_kms_helper e1000e psmouse ahci drm ptp libahci pps_core video wmi
Jan 18 12:33:33 e104462 kernel: [ 392.175054] CPU: 1 PID: 3380 Comm: condvar_stress- Tainted: G W OX 3.13.0-74-generic #118-Ubuntu
Jan 18 12:33:33 e104462 kernel: [ 392.175055] Hardware name: LENOVO 32281S1/MAHOBAY, BIOS 9SKT71AUS 07/02/2013
Jan 18 12:33:33 e104462 kernel: [ 392.175056] task: ffff880401c4b000 ti: ffff8804067ca000 task.ti: ffff8804067ca000
Jan 18 12:33:33 e104462 kernel: [ 392.175057] RIP: 0010:[<ffffffff810dc9ae>] [<ffffffff810dc9ae>] smp_call_function_many+0x26e/0x2d0
Jan 18 12:33:33 e104462 kernel: [ 392.175061] RSP: 0018:ffff8804067cbbd8 EFLAGS: 00000202
Jan 18 12:33:33 e104462 kernel: [ 392.175062] RAX: 0000000000000002 RBX: ffff8804067cbb88 RCX: ffff88041dc96ff0
Jan 18 12:33:33 e104462 kernel: [ 392.175063] RDX: 0000000000000002 RSI: 0000000000000100 RDI: 0000000000000000
Jan 18 12:33:33 e104462 kernel: [ 392.175063] RBP: ffff8804067cbc28 R08: ffff88041dc53c88 R09: 0000000000000004
Jan 18 12:33:33 e104462 kernel: [ 392.175064] R10: ffff88041dc53c88 R11: ffff880402114510 R12: ffff88041dc56a58
Jan 18 12:33:33 e104462 kernel: [ 392.175065] R13: 000000000096716a R14: ffff880396d23000 R15: ffffffff00000000
Jan 18 12:33:33 e104462 kernel: [ 392.175066] FS: 0000000000000000(0000) GS:ffff88041dc40000(0000) knlGS:0000000000000000
Jan 18 12:33:33 e104462 kernel: [ 392.175067] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 18 12:33:33 e104462 kernel: [ 392.175067] CR2: 00002b6017fb2000 CR3: 0000000001c0e000 CR4: 00000000001407e4
Jan 18 12:33:33 e104462 kernel: [ 392.175068] DR0: 000000005589a248 DR1: 0000000000000000 DR2: 0000000000000000
Jan 18 12:33:33 e104462 kernel: [ 392.175069] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
Jan 18 12:33:33 e104462 kernel: [ 392.175070] Stack:
Jan 18 12:33:33 e104462 kernel: [ 392.175070] ffff88041dc53ca8 0000000000013c40 0000000000000000 ffffffff8101a620
Jan 18 12:33:33 e104462 kernel: [ 392.175072] 0000010000000001 ffffffff81096b94 ffffffff8101a620 0000000000000000
Jan 18 12:33:33 e104462 kernel: [ 392.175074] ffffffff81096b95 0000000000000000 ffff8804067cbc50 ffffffff810dcb6d
Jan 18 12:33:33 e104462 kernel: [ 392.175075] Call Trace:
Jan 18 12:33:33 e104462 kernel: [ 392.175079] [<ffffffff8101a620>] ? arch_unregister_cpu+0x30/0x30
Jan 18 12:33:33 e104462 kernel: [ 392.175082] [<ffffffff81096b94>] ? perf_event_task_sched_out+0x14/0xa0
Jan 18 12:33:33 e104462 kernel: [ 392.175083] [<ffffffff8101a620>] ? arch_unregister_cpu+0x30/0x30
Jan 18 12:33:33 e104462 kernel: [ 392.175085] [<ffffffff81096b95>] ? perf_event_task_sched_out+0x15/0xa0
Jan 18 12:33:33 e104462 kernel: [ 392.175086] [<ffffffff810dcb6d>] on_each_cpu+0x2d/0x60
Jan 18 12:33:33 e104462 kernel: [ 392.175088] [<ffffffff81096b94>] ? perf_event_task_sched_out+0x14/0xa0
Jan 18 12:33:33 e104462 kernel: [ 392.175089] [<ffffffff8101adeb>] text_poke_bp+0x5b/0xc0
Jan 18 12:33:33 e104462 kernel: [ 392.175090] [<ffffffff81096b94>] ? perf_event_task_sched_out+0x14/0xa0
Jan 18 12:33:33 e104462 kernel: [ 392.175092] [<ffffffff81018053>] arch_jump_label_transform+0xa3/0x120
Jan 18 12:33:33 e104462 kernel: [ 392.175095] [<ffffffff8114e09f>] __jump_label_update+0x5f/0x80
Jan 18 12:33:33 e104462 kernel: [ 392.175096] [<ffffffff8114e15d>] jump_label_update+0x9d/0xb0
Jan 18 12:33:33 e104462 kernel: [ 392.175098] [<ffffffff8114e3bd>] __static_key_slow_dec+0x4d/0xb0
Jan 18 12:33:33 e104462 kernel: [ 392.175100] [<ffffffff8114e442>] static_key_slow_dec+0x22/0x50
Jan 18 12:33:33 e104462 kernel: [ 392.175102] [<ffffffff811403be>] sw_perf_event_destroy+0x2e/0x80
Jan 18 12:33:33 e104462 kernel: [ 392.175104] [<ffffffff81141233>] __free_event+0x23/0x60
Jan 18 12:33:33 e104462 kernel: [ 392.175106] [<ffffffff811415b6>] free_event+0x96/0x170
Jan 18 12:33:33 e104462 kernel: [ 392.175108] [<ffffffff811429a8>] perf_event_release_kernel+0x68/0xc0
Jan 18 12:33:33 e104462 kernel: [ 392.175109] [<ffffffff81142a93>] put_event+0x93/0xc0
Jan 18 12:33:33 e104462 kernel: [ 392.175111] [<ffffffff81142ad0>] perf_release+0x10/0x20
Jan 18 12:33:33 e104462 kernel: [ 392.175114] [<ffffffff811c0084>] __fput+0xe4/0x260
Jan 18 12:33:33 e104462 kernel: [ 392.175116] [<ffffffff811c024e>] ____fput+0xe/0x10
Jan 18 12:33:33 e104462 kernel: [ 392.175119] [<ffffffff8108863c>] task_work_run+0xac/0xd0
Jan 18 12:33:33 e104462 kernel: [ 392.175121] [<ffffffff81069e28>] do_exit+0x2b8/0xa50
Jan 18 12:33:33 e104462 kernel: [ 392.175123] [<ffffffff8107a1ef>] ? ptrace_notify+0x5f/0xc0
Jan 18 12:33:33 e104462 kernel: [ 392.175124] [<ffffffff8106a63f>] do_group_exit+0x3f/0xa0
Jan 18 12:33:33 e104462 kernel: [ 392.175126] [<ffffffff8106a6b4>] SyS_exit_group+0x14/0x20
Jan 18 12:33:33 e104462 kernel: [ 392.175128] [<ffffffff8173596f>] tracesys+0xe1/0xe6
Jan 18 12:33:33 e104462 kernel: [ 392.175129] Code: 3b 05 5f 8e c3 00 89 c2 0f 8d 25 fe ff ff 48 98 49 8b 4d 00 48 03 0c c5 e0 39 d1 81 f6 41 20 01 74 cb 0f 1f 00 f3 90 f6 41 20 01 <75> f8 eb be 0f b6 4d d0 48 8b 55 c0 44 89 ef 48 8b 75 c8 e8 ea

Jan 18 12:33:53 e104462 kernel: [ 412.308582] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 7, t=15002 jiffies, g=5986, c=5985, q=0)
Jan 18 12:33:53 e104462 kernel: [ 412.308587] sending NMI to all CPUs:
Jan 18 12:33:53 e104462 kernel: [ 412.308590] NMI backtrace for cpu 3
Jan 18 12:33:53 e104462 kernel: [ 412.308594] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G W OX 3.13.0-74-generic #118-Ubuntu
Jan 18 12:33:53 e104462 kernel: [ 412.308595] Hardware name: LENOVO 32281S1/MAHOBAY, BIOS 9SKT71AUS 07/02/2013
Jan 18 12:33:53 e104462 kernel: [ 412.308597] task: ffff880407654800 ti: ffff880407658000 task.ti: ffff880407658000
Jan 18 12:33:53 e104462 kernel: [ 412.308598] RIP: 0010:[<ffffffff813ea2c8>] [<ffffffff813ea2c8>] intel_idle+0xd8/0x140
Jan 18 12:33:53 e104462 kernel: [ 412.308603] RSP: 0018:ffff880407659e28 EFLAGS: 00000046
Jan 18 12:33:53 e104462 kernel: [ 412.308604] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000001
Jan 18 12:33:53 e104462 kernel: [ 412.308606] RDX: 0000000000000000 RSI: ffffffff81c93ec0 RDI: 0000000000000003
Jan 18 12:33:53 e104462 kernel: [ 412.308607] RBP: ffff880407659e50 R08: ffff88041dcd0310 R09: 0000000000000020
Jan 18 12:33:53 e104462 kernel: [ 412.308608] R10: 0000000000000052 R11: 000000000000002e R12: 0000000000000001
Jan 18 12:33:53 e104462 kernel: [ 412.308609] R13: 0000000000000000 R14: 0000000000000001 R15: ffffffff81c93f30
Jan 18 12:33:53 e104462 kernel: [ 412.308611] FS: 0000000000000000(0000) GS:ffff88041dcc0000(0000) knlGS:0000000000000000
Jan 18 12:33:53 e104462 kernel: [ 412.308612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 18 12:33:53 e104462 kernel: [ 412.308613] CR2: 00002b8ad8346000 CR3: 000000038617e000 CR4: 00000000001407e0
Jan 18 12:33:53 e104462 kernel: [ 412.308614] Stack:
Jan 18 12:33:53 e104462 kernel: [ 412.308615] 0000000307659e50 ffff88041dcda300 ffffffff81c93ec0 0000005ff92a0f32
Jan 18 12:33:53 e104462 kernel: [ 412.308618] 0000000000000001 ffff880407659e88 ffffffff815d65a0 ffff880407659f38
Jan 18 12:33:53 e104462 kernel: [ 412.308620] ffff88041dcda300 0000000000000001 0000000000000003 ffffffff81c93ec0
Jan 18 12:33:53 e104462 kernel: [ 412.308623] Call Trace:
Jan 18 12:33:53 e104462 kernel: [ 412.308628] [<ffffffff815d65a0>] cpuidle_enter_state+0x40/0xc0
Jan 18 12:33:53 e104462 kernel: [ 412.308630] [<ffffffff815d66d9>] cpuidle_idle_call+0xb9/0x1f0
Jan 18 12:33:53 e104462 kernel: [ 412.308634] [<ffffffff8101d3ee>] arch_cpu_idle+0xe/0x30
Jan 18 12:33:53 e104462 kernel: [ 412.308637] [<ffffffff810bf475>] cpu_startup_entry+0xc5/0x290
Jan 18 12:33:53 e104462 kernel: [ 412.308640] [<ffffffff810415ed>] start_secondary+0x21d/0x2d0
Jan 18 12:33:53 e104462 kernel: [ 412.308642] Code: 48 89 d1 48 2d c8 1f 00 00 0f 01 c8 0f ae f0 65 48 8b 04 25 30 b8 00 00 48 8b 80 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ea 9b 8a 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 62 9f ce
Jan 18 12:33:53 e104462 kernel: [ 412.308662] NMI backtrace for cpu 7
Jan 18 12:33:53 e104462 kernel: [ 412.308664] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G W OX 3.13.0-74-generic #118-Ubuntu
Jan 18 12:33:53 e104462 kernel: [ 412.308665] Hardware name: LENOVO 32281S1/MAHOBAY, BIOS 9SKT71AUS 07/02/2013
Jan 18 12:33:53 e104462 kernel: [ 412.308666] task: ffff880407663000 ti: ffff880407678000 task.ti: ffff880407678000
Jan 18 12:33:53 e104462 kernel: [ 412.308667] RIP: 0010:[<ffffffff81372352>] [<ffffffff81372352>] __const_udelay+0x12/0x30
Jan 18 12:33:53 e104462 kernel: [ 412.308670] RSP: 0018:ffff88041ddc3df0 EFLAGS: 00000046
Jan 18 12:33:53 e104462 kernel: [ 412.308672] RAX: 0000000001062560 RBX: 0000000000002710 RCX: 0000000000000004
Jan 18 12:33:53 e104462 kernel: [ 412.308676] RDX: 0000000000cf0a24 RSI: 0000000000000100 RDI: 0000000000418958
Jan 18 12:33:53 e104462 kernel: [ 412.308680] RBP: ffff88041ddc3e08 R08: 0000000000000082 R09: 000000000000045f
Jan 18 12:33:53 e104462 kernel: [ 412.308683] R10: 0000000000000000 R11: ffff88041ddc3b2e R12: ffffffff81c4e200
Jan 18 12:33:53 e104462 kernel: [ 412.308687] R13: ffffffff81d14660 R14: ffffffff81c4e200 R15: 0000000000000007
Jan 18 12:33:53 e104462 kernel: [ 412.308691] FS: 0000000000000000(0000) GS:ffff88041ddc0000(0000) knlGS:0000000000000000
Jan 18 12:33:53 e104462 kernel: [ 412.308695] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 18 12:33:53 e104462 kernel: [ 412.308699] CR2: 000000000223be18 CR3: 00000000b74e6000 CR4: 00000000001407e0
Jan 18 12:33:53 e104462 kernel: [ 412.308703] Stack:
Jan 18 12:33:53 e104462 kernel: [ 412.308707] ffff88041ddc3e08 ffffffff8104533f ffff88041ddcd840 ffff88041ddc3e60
Jan 18 12:33:53 e104462 kernel: [ 412.308709] ffffffff810cb5b1 ffffffff81c4e200 ffffffff00000002 0000000000000000
Jan 18 12:33:53 e104462 kernel: [ 412.308711] 0000000000000001 ffff880407663000 0000000000000000 0000000000000007
Jan 18 12:33:53 e104462 kernel: [ 412.308712] Call Trace:
Jan 18 12:33:53 e104462 kernel: [ 412.308713] <IRQ>
Jan 18 12:33:53 e104462 kernel: [ 412.308714] [<ffffffff8104533f>] ? arch_trigger_all_cpu_backtrace+0x8f/0xb0
Jan 18 12:33:53 e104462 kernel: [ 412.308718] [<ffffffff810cb5b1>] rcu_check_callbacks+0x631/0x650
Jan 18 12:33:53 e104462 kernel: [ 412.308721] [<ffffffff810765c7>] update_process_times+0x47/0x70
Jan 18 12:33:53 e104462 kernel: [ 412.308725] [<ffffffff810d6795>] tick_sched_handle.isra.17+0x25/0x60
Jan 18 12:33:53 e104462 kernel: [ 412.308727] [<ffffffff810d6811>] tick_sched_timer+0x41/0x60
Jan 18 12:33:53 e104462 kernel: [ 412.308730] [<ffffffff8108eab7>] __run_hrtimer+0x77/0x1d0
Jan 18 12:33:53 e104462 kernel: [ 412.308733] [<ffffffff810d67d0>] ? tick_sched_handle.isra.17+0x60/0x60
Jan 18 12:33:53 e104462 kernel: [ 412.308736] [<ffffffff8108f27f>] hrtimer_interrupt+0xef/0x230
Jan 18 12:33:53 e104462 kernel: [ 412.308739] [<ffffffff81043737>] local_apic_timer_interrupt+0x37/0x60
Jan 18 12:33:53 e104462 kernel: [ 412.308742] [<ffffffff81737b0f>] smp_apic_timer_interrupt+0x3f/0x60
Jan 18 12:33:53 e104462 kernel: [ 412.308745] [<ffffffff8173649d>] apic_timer_interrupt+0x6d/0x80
Jan 18 12:33:53 e104462 kernel: [ 412.308746] <EOI>
Jan 18 12:33:53 e104462 kernel: [ 412.308747] [<ffffffff815d65b2>] ? cpuidle_enter_state+0x52/0xc0
Jan 18 12:33:53 e104462 kernel: [ 412.308751] [<ffffffff815d66d9>] cpuidle_idle_call+0xb9/0x1f0
Jan 18 12:33:53 e104462 kernel: [ 412.308758] [<ffffffff8101d3ee>] arch_cpu_idle+0xe/0x30
Jan 18 12:33:53 e104462 kernel: [ 412.308764] [<ffffffff810bf475>] cpu_startup_entry+0xc5/0x290
Jan 18 12:33:53 e104462 kernel: [ 412.308770] [<ffffffff810415ed>] start_secondary+0x21d/0x2d0
Jan 18 12:33:53 e104462 kernel: [ 412.308771] Code: 89 e5 ff 15 69 be 91 00 5d c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 8d 04 bd 00 00 00 00 65 48 8b 14 25 a0 29 01 00 <48> 8d 0c 12 48 c1 e2 06 48 89 e5 48 29 ca f7 e2 48 8d 7a 01 ff
Jan 18 12:33:53 e104462 kernel: [ 412.308791] NMI backtrace for cpu 4
Jan 18 12:33:53 e104462 kernel: [ 412.308794] CPU: 4 PID: 7670 Comm: rr Tainted: G W OX 3.13.0-74-generic #118-Ubuntu
Jan 18 12:33:53 e104462 kernel: [ 412.308795] Hardware name: LENOVO 32281S1/MAHOBAY, BIOS 9SKT71AUS 07/02/2013
Jan 18 12:33:53 e104462 kernel: [ 412.308797] task: ffff8803853f8000 ti: ffff880386154000 task.ti: ffff880386154000
Jan 18 12:33:53 e104462 kernel: [ 412.308798] RIP: 0033:[<00002b11ba2fd65f>] [<00002b11ba2fd65f>] 0x2b11ba2fd65f
Jan 18 12:33:53 e104462 kernel: [ 412.308810] RSP: 002b:00007fff827726c0 EFLAGS: 00000246
Jan 18 12:33:53 e104462 kernel: [ 412.308811] RAX: 0000000000000008 RBX: 0000000000000000 RCX: 00007fff827726c2
Jan 18 12:33:53 e104462 kernel: [ 412.308812] RDX: fffffffffffffff8 RSI: 00007fff827726c0 RDI: 0000000000000000
Jan 18 12:33:53 e104462 kernel: [ 412.308813] RBP: 00007fff82772ab0 R08: 00002b11ba665060 R09: 00007fff827726c2
Jan 18 12:33:53 e104462 kernel: [ 412.308815] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000022
Jan 18 12:33:53 e104462 kernel: [ 412.308816] R13: 0000000000000000 R14: 000000000000003a R15: 00007fff82772ac0
Jan 18 12:33:53 e104462 kernel: [ 412.308817] FS: 00002b11b9475000(0000) GS:ffff88041dd00000(0000) knlGS:0000000000000000
Jan 18 12:33:53 e104462 kernel: [ 412.308819] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 18 12:33:53 e104462 kernel: [ 412.308820] CR2: 00002b11b944f000 CR3: 00000003899d1000 CR4: 00000000001407e0
Jan 18 12:33:53 e104462 kernel: [ 412.308821]
Jan 18 12:33:53 e104462 kernel: [ 412.308822] NMI backtrace for cpu 1
Jan 18 12:33:53 e104462 kernel: [ 412.308826] CPU: 1 PID: 3380 Comm: condvar_stress- Tainted: G W OX 3.13.0-74-generic #118-Ubuntu
Jan 18 12:33:53 e104462 kernel: [ 412.308827] Hardware name: LENOVO 32281S1/MAHOBAY, BIOS 9SKT71AUS 07/02/2013
Jan 18 12:33:53 e104462 kernel: [ 412.308828] task: ffff880401c4b000 ti: ffff8804067ca000 task.ti: ffff8804067ca000
Jan 18 12:33:53 e104462 kernel: [ 412.308830] RIP: 0010:[<ffffffff810dc9ae>] [<ffffffff810dc9ae>] smp_call_function_many+0x26e/0x2d0
Jan 18 12:33:53 e104462 kernel: [ 412.308834] RSP: 0018:ffff8804067cbbd8 EFLAGS: 00000202
Jan 18 12:33:53 e104462 kernel: [ 412.308835] RAX: 0000000000000002 RBX: ffff88041dc53c88 RCX: ffff88041dc96ff0
Jan 18 12:33:53 e104462 kernel: [ 412.308836] RDX: 0000000000000002 RSI: 0000000000000100 RDI: 0000000000000000
Jan 18 12:33:53 e104462 kernel: [ 412.308837] RBP: ffff8804067cbc28 R08: ffff88041dc53c88 R09: 0000000000000004
Jan 18 12:33:53 e104462 kernel: [ 412.308838] R10: ffff88041dc53c88 R11: ffff880402114510 R12: 0000000000000008
Jan 18 12:33:53 e104462 kernel: [ 412.308839] R13: ffff88041dc53c80 R14: ffff88041ddd3c50 R15: ffff88041ddd6ff0
Jan 18 12:33:53 e104462 kernel: [ 412.308841] FS: 0000000000000000(0000) GS:ffff88041dc40000(0000) knlGS:0000000000000000
Jan 18 12:33:53 e104462 kernel: [ 412.308842] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 18 12:33:53 e104462 kernel: [ 412.308844] CR2: 00002b6017fb2000 CR3: 0000000001c0e000 CR4: 00000000001407e4
Jan 18 12:33:53 e104462 kernel: [ 412.308845] DR0: 000000005589a248 DR1: 0000000000000000 DR2: 0000000000000000
Jan 18 12:33:53 e104462 kernel: [ 412.308846] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
Jan 18 12:33:53 e104462 kernel: [ 412.308847] Stack:
Jan 18 12:33:53 e104462 kernel: [ 412.308848] ffff88041dc53ca8 0000000000013c40 0000000000000000 ffffffff8101a620
Jan 18 12:33:53 e104462 kernel: [ 412.308850] 0000010000000001 ffffffff81096b94 ffffffff8101a620 0000000000000000
Jan 18 12:33:53 e104462 kernel: [ 412.308853] ffffffff81096b95 0000000000000000 ffff8804067cbc50 ffffffff810dcb6d
Jan 18 12:33:53 e104462 kernel: [ 412.308855] Call Trace:
Jan 18 12:33:53 e104462 kernel: [ 412.308858] [<ffffffff8101a620>] ? arch_unregister_cpu+0x30/0x30
Jan 18 12:33:53 e104462 kernel: [ 412.308861] [<ffffffff81096b94>] ? perf_event_task_sched_out+0x14/0xa0
Jan 18 12:33:53 e104462 kernel: [ 412.308863] [<ffffffff8101a620>] ? arch_unregister_cpu+0x30/0x30
Jan 18 12:33:53 e104462 kernel: [ 412.308866] [<ffffffff81096b95>] ? perf_event_task_sched_out+0x15/0xa0
Jan 18 12:33:53 e104462 kernel: [ 412.308868] [<ffffffff810dcb6d>] on_each_cpu+0x2d/0x60
Jan 18 12:33:53 e104462 kernel: [ 412.308870] [<ffffffff81096b94>] ? perf_event_task_sched_out+0x14/0xa0
Jan 18 12:33:53 e104462 kernel: [ 412.308872] [<ffffffff8101adeb>] text_poke_bp+0x5b/0xc0
Jan 18 12:33:53 e104462 kernel: [ 412.308875] [<ffffffff81096b94>] ? perf_event_task_sched_out+0x14/0xa0
Jan 18 12:33:53 e104462 kernel: [ 412.308877] [<ffffffff81018053>] arch_jump_label_transform+0xa3/0x120
Jan 18 12:33:53 e104462 kernel: [ 412.308880] [<ffffffff8114e09f>] __jump_label_update+0x5f/0x80
Jan 18 12:33:53 e104462 kernel: [ 412.308883] [<ffffffff8114e15d>] jump_label_update+0x9d/0xb0
Jan 18 12:33:53 e104462 kernel: [ 412.308885] [<ffffffff8114e3bd>] __static_key_slow_dec+0x4d/0xb0
Jan 18 12:33:53 e104462 kernel: [ 412.308888] [<ffffffff8114e442>] static_key_slow_dec+0x22/0x50
Jan 18 12:33:53 e104462 kernel: [ 412.308891] [<ffffffff811403be>] sw_perf_event_destroy+0x2e/0x80
Jan 18 12:33:53 e104462 kernel: [ 412.308893] [<ffffffff81141233>] __free_event+0x23/0x60
Jan 18 12:33:53 e104462 kernel: [ 412.308896] [<ffffffff811415b6>] free_event+0x96/0x170
Jan 18 12:33:53 e104462 kernel: [ 412.308898] [<ffffffff811429a8>] perf_event_release_kernel+0x68/0xc0
Jan 18 12:33:53 e104462 kernel: [ 412.308901] [<ffffffff81142a93>] put_event+0x93/0xc0
Jan 18 12:33:53 e104462 kernel: [ 412.308903] [<ffffffff81142ad0>] perf_release+0x10/0x20
Jan 18 12:33:53 e104462 kernel: [ 412.308907] [<ffffffff811c0084>] __fput+0xe4/0x260
Jan 18 12:33:53 e104462 kernel: [ 412.308910] [<ffffffff811c024e>] ____fput+0xe/0x10
Jan 18 12:33:53 e104462 kernel: [ 412.308913] [<ffffffff8108863c>] task_work_run+0xac/0xd0
Jan 18 12:33:53 e104462 kernel: [ 412.308915] [<ffffffff81069e28>] do_exit+0x2b8/0xa50
Jan 18 12:33:53 e104462 kernel: [ 412.308918] [<ffffffff8107a1ef>] ? ptrace_notify+0x5f/0xc0
Jan 18 12:33:53 e104462 kernel: [ 412.308920] [<ffffffff8106a63f>] do_group_exit+0x3f/0xa0
Jan 18 12:33:53 e104462 kernel: [ 412.308923] [<ffffffff8106a6b4>] SyS_exit_group+0x14/0x20
Jan 18 12:33:53 e104462 kernel: [ 412.308925] [<ffffffff8173596f>] tracesys+0xe1/0xe6
Jan 18 12:33:53 e104462 kernel: [ 412.308926] Code: 3b 05 5f 8e c3 00 89 c2 0f 8d 25 fe ff ff 48 98 49 8b 4d 00 48 03 0c c5 e0 39 d1 81 f6 41 20 01 74 cb 0f 1f 00 f3 90 f6 41 20 01 <75> f8 eb be 0f b6 4d d0 48 8b 55 c0 44 89 ef 48 8b 75 c8 e8 ea
Jan 18 12:33:53 e104462 kernel: [ 412.308947] NMI backtrace for cpu 5
Jan 18 12:33:53 e104462 kernel: [ 412.308950] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G W OX 3.13.0-74-generic #118-Ubuntu
Jan 18 12:33:53 e104462 kernel: [ 412.308951] Hardware name: LENOVO 32281S1/MAHOBAY, BIOS 9SKT71AUS 07/02/2013
Jan 18 12:33:53 e104462 kernel: [ 412.308952] task: ffff880407660000 ti: ffff88040765c000 task.ti: ffff88040765c000
Jan 18 12:33:53 e104462 kernel: [ 412.308953] RIP: 0010:[<ffffffff813ea2c8>] [<ffffffff813ea2c8>] intel_idle+0xd8/0x140
Jan 18 12:33:53 e104462 kernel: [ 412.308956] RSP: 0018:ffff88040765de28 EFLAGS: 00000046
Jan 18 12:33:53 e104462 kernel: [ 412.308957] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
Jan 18 12:33:53 e104462 kernel: [ 412.308958] RDX: 0000000000000000 RSI: ffffffff81c93ec0 RDI: 0000000001c0e000
Jan 18 12:33:53 e104462 kernel: [ 412.308959] RBP: ffff88040765de50 R08: ffff88041dd50314 R09: 0000000000000018
Jan 18 12:33:53 e104462 kernel: [ 412.308960] R10: 000000000000996a R11: 0000000000014554 R12: 0000000000000004
Jan 18 12:33:53 e104462 kernel: [ 412.308962] R13: 0000000000000020 R14: 0000000000000003 R15: ffffffff81c94038
Jan 18 12:33:53 e104462 kernel: [ 412.308963] FS: 0000000000000000(0000) GS:ffff88041dd40000(0000) knlGS:0000000000000000
Jan 18 12:33:53 e104462 kernel: [ 412.308964] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 18 12:33:53 e104462 kernel: [ 412.308965] CR2: 00007fbf41458000 CR3: 0000000001c0e000 CR4: 00000000001407e0
Jan 18 12:33:53 e104462 kernel: [ 412.308966] Stack:
Jan 18 12:33:53 e104462 kernel: [ 412.308967] 000000050765de50 ffff88041dd5a300 ffffffff81c93ec0 0000005ff8ef3747
Jan 18 12:33:53 e104462 kernel: [ 412.308969] 0000000000000004 ffff88040765de88 ffffffff815d65a0 ffff88040765df38
Jan 18 12:33:53 e104462 kernel: [ 412.308972] ffff88041dd5a300 0000000000000004 0000000000000005 ffffffff81c93ec0
Jan 18 12:33:53 e104462 kernel: [ 412.308974] Call Trace:
Jan 18 12:33:53 e104462 kernel: [ 412.308977] [<ffffffff815d65a0>] cpuidle_enter_state+0x40/0xc0
Jan 18 12:33:53 e104462 kernel: [ 412.308979] [<ffffffff815d66d9>] cpuidle_idle_call+0xb9/0x1f0
Jan 18 12:33:53 e104462 kernel: [ 412.308981] [<ffffffff8101d3ee>] arch_cpu_idle+0xe/0x30
Jan 18 12:33:53 e104462 kernel: [ 412.308983] [<ffffffff810bf475>] cpu_startup_entry+0xc5/0x290
Jan 18 12:33:53 e104462 kernel: [ 412.308986] [<ffffffff810415ed>] start_secondary+0x21d/0x2d0
Jan 18 12:33:53 e104462 kernel: [ 412.308987] Code: 48 89 d1 48 2d c8 1f 00 00 0f 01 c8 0f ae f0 65 48 8b 04 25 30 b8 00 00 48 8b 80 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ea 9b 8a 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 62 9f ce
Jan 18 12:33:53 e104462 kernel: [ 412.309007] NMI backtrace for cpu 0
Jan 18 12:33:53 e104462 kernel: [ 412.309010] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W OX 3.13.0-74-generic #118-Ubuntu
Jan 18 12:33:53 e104462 kernel: [ 412.309012] Hardware name: LENOVO 32281S1/MAHOBAY, BIOS 9SKT71AUS 07/02/2013
Jan 18 12:33:53 e104462 kernel: [ 412.309013] task: ffffffff81c15480 ti: ffffffff81c00000 task.ti: ffffffff81c00000
Jan 18 12:33:53 e104462 kernel: [ 412.309018] RIP: 0010:[<ffffffff813ea2c8>] [<ffffffff813ea2c8>] intel_idle+0xd8/0x140
Jan 18 12:33:53 e104462 kernel: [ 412.309027] RSP: 0018:ffffffff81c01e38 EFLAGS: 00000046
Jan 18 12:33:53 e104462 kernel: [ 412.309031] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
Jan 18 12:33:53 e104462 kernel: [ 412.309036] RDX: 0000000000000000 RSI: ffffffff81c93ec0 RDI: 0000000001c0e000
Jan 18 12:33:53 e104462 kernel: [ 412.309041] RBP: ffffffff81c01e60 R08: ffff88041dc10308 R09: 0000000000000018
Jan 18 12:33:53 e104462 kernel: [ 412.309046] R10: 0000000000001676 R11: 0000000000000fe7 R12: 0000000000000004
Jan 18 12:33:53 e104462 kernel: [ 412.309051] R13: 0000000000000020 R14: 0000000000000003 R15: ffffffff81c94038
Jan 18 12:33:53 e104462 kernel: [ 412.309055] FS: 0000000000000000(0000) GS:ffff88041dc00000(0000) knlGS:0000000000000000
Jan 18 12:33:53 e104462 kernel: [ 412.309060] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 18 12:33:53 e104462 kernel: [ 412.309065] CR2: 00002b04404fc018 CR3: 0000000001c0e000 CR4: 00000000001407f0
Jan 18 12:33:53 e104462 kernel: [ 412.309069] Stack:
Jan 18 12:33:53 e104462 kernel: [ 412.309074] 0000000081c01e60 ffff88041dc1a300 ffffffff81c93ec0 0000005ff9303bca
Jan 18 12:33:53 e104462 kernel: [ 412.309090] 0000000000000004 ffffffff81c01e98 ffffffff815d65a0 ffffffffffffffff
Jan 18 12:33:53 e104462 kernel: [ 412.309106] ffff88041dc1a300 0000000000000004 0000000000000000 ffffffff81c93ec0
Jan 18 12:33:53 e104462 kernel: [ 412.309109] Call Trace:
Jan 18 12:33:53 e104462 kernel: [ 412.309113] [<ffffffff815d65a0>] cpuidle_enter_state+0x40/0xc0
Jan 18 12:33:53 e104462 kernel: [ 412.309115] [<ffffffff815d66d9>] cpuidle_idle_call+0xb9/0x1f0
Jan 18 12:33:53 e104462 kernel: [ 412.309118] [<ffffffff8101d3ee>] arch_cpu_idle+0xe/0x30
Jan 18 12:33:53 e104462 kernel: [ 412.309120] [<ffffffff810bf475>] cpu_startup_entry+0xc5/0x290
Jan 18 12:33:53 e104462 kernel: [ 412.309123] [<ffffffff81712d77>] rest_init+0x77/0x80
Jan 18 12:33:53 e104462 kernel: [ 412.309127] [<ffffffff81d34f70>] start_kernel+0x438/0x443
Jan 18 12:33:53 e104462 kernel: [ 412.309129] [<ffffffff81d34941>] ? repair_env_string+0x5c/0x5c
Jan 18 12:33:53 e104462 kernel: [ 412.309132] [<ffffffff81d34120>] ? early_idt_handler_array+0x120/0x120
Jan 18 12:33:53 e104462 kernel: [ 412.309135] [<ffffffff81d345ee>] x86_64_start_reservations+0x2a/0x2c
Jan 18 12:33:53 e104462 kernel: [ 412.309137] [<ffffffff81d34733>] x86_64_start_kernel+0x143/0x152
Jan 18 12:33:53 e104462 kernel: [ 412.309138] Code: 48 89 d1 48 2d c8 1f 00 00 0f 01 c8 0f ae f0 65 48 8b 04 25 30 b8 00 00 48 8b 80 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ea 9b 8a 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 62 9f ce
Jan 18 12:33:53 e104462 kernel: [ 412.309159] NMI backtrace for cpu 2
Jan 18 12:33:53 e104462 kernel: [ 412.309163] CPU: 2 PID: 4294 Comm: rr:async_signal Tainted: G W OX 3.13.0-74-generic #118-Ubuntu
Jan 18 12:33:53 e104462 kernel: [ 412.309164] Hardware name: LENOVO 32281S1/MAHOBAY, BIOS 9SKT71AUS 07/02/2013
Jan 18 12:33:53 e104462 kernel: [ 412.309166] task: ffff880402be1800 ti: ffff8804033aa000 task.ti: ffff8804033aa000
Jan 18 12:33:53 e104462 kernel: [ 412.309167] RIP: 0033:[<0000000070000000>] [<0000000070000000>] 0x70000000
Jan 18 12:33:53 e104462 kernel: [ 412.309175] RSP: 002b:00007fff67b293f0 EFLAGS: 00000046
Jan 18 12:33:53 e104462 kernel: [ 412.309180] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffffffffffff
Jan 18 12:33:53 e104462 kernel: [ 412.309185] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Jan 18 12:33:53 e104462 kernel: [ 412.309189] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Jan 18 12:33:53 e104462 kernel: [ 412.309194] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Jan 18 12:33:53 e104462 kernel: [ 412.309198] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Jan 18 12:33:53 e104462 kernel: [ 412.309200] FS: 0000000000000000(0000) GS:ffff88041dc80000(0000) knlGS:0000000000000000
Jan 18 12:33:53 e104462 kernel: [ 412.309201] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 18 12:33:53 e104462 kernel: [ 412.309202] CR2: 00002b0420247000 CR3: 0000000405874000 CR4: 00000000001407e4
Jan 18 12:33:53 e104462 kernel: [ 412.309203]
Jan 18 12:33:53 e104462 kernel: [ 412.309205] NMI backtrace for cpu 6
Jan 18 12:33:53 e104462 kernel: [ 412.309207] CPU: 6 PID: 0 Comm: swapper/6 Tainted: G W OX 3.13.0-74-generic #118-Ubuntu
Jan 18 12:33:53 e104462 kernel: [ 412.309208] Hardware name: LENOVO 32281S1/MAHOBAY, BIOS 9SKT71AUS 07/02/2013
Jan 18 12:33:53 e104462 kernel: [ 412.309209] task: ffff880407661800 ti: ffff88040765e000 task.ti: ffff88040765e000
Jan 18 12:33:53 e104462 kernel: [ 412.309210] RIP: 0010:[<ffffffff813ea2c8>] [<ffffffff813ea2c8>] intel_idle+0xd8/0x140
Jan 18 12:33:53 e104462 kernel: [ 412.309212] RSP: 0018:ffff88040765fe28 EFLAGS: 00000046
Jan 18 12:33:53 e104462 kernel: [ 412.309217] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
Jan 18 12:33:53 e104462 kernel: [ 412.309221] RDX: 0000000000000000 RSI: ffffffff81c93ec0 RDI: 0000000000000006
Jan 18 12:33:53 e104462 kernel: [ 412.309225] RBP: ffff88040765fe50 R08: ffff88041dd90304 R09: 0000000000000018
Jan 18 12:33:53 e104462 kernel: [ 412.309226] R10: 00000000000270e8 R11: 0000000000002e29 R12: 0000000000000004
Jan 18 12:33:53 e104462 kernel: [ 412.309227] R13: 0000000000000020 R14: 0000000000000003 R15: ffffffff81c94038
Jan 18 12:33:53 e104462 kernel: [ 412.309228] FS: 0000000000000000(0000) GS:ffff88041dd80000(0000) knlGS:0000000000000000
Jan 18 12:33:53 e104462 kernel: [ 412.309229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 18 12:33:53 e104462 kernel: [ 412.309230] CR2: 000000000341d3e8 CR3: 0000000001c0e000 CR4: 00000000001407e0
Jan 18 12:33:53 e104462 kernel: [ 412.309231] Stack:
Jan 18 12:33:53 e104462 kernel: [ 412.309235] 000000060765fe50 ffff88041dd9a300 ffffffff81c93ec0 0000005ff5b2830a
Jan 18 12:33:53 e104462 kernel: [ 412.309239] 0000000000000004 ffff88040765fe88 ffffffff815d65a0 ffff88040765ff38
Jan 18 12:33:53 e104462 kernel: [ 412.309243] ffff88041dd9a300 0000000000000004 0000000000000006 ffffffff81c93ec0
Jan 18 12:33:53 e104462 kernel: [ 412.309245] Call Trace:
Jan 18 12:33:53 e104462 kernel: [ 412.309248] [<ffffffff815d65a0>] cpuidle_enter_state+0x40/0xc0
Jan 18 12:33:53 e104462 kernel: [ 412.309250] [<ffffffff815d66d9>] cpuidle_idle_call+0xb9/0x1f0
Jan 18 12:33:53 e104462 kernel: [ 412.309252] [<ffffffff8101d3ee>] arch_cpu_idle+0xe/0x30
Jan 18 12:33:53 e104462 kernel: [ 412.309254] [<ffffffff810bf475>] cpu_startup_entry+0xc5/0x290
Jan 18 12:33:53 e104462 kernel: [ 412.309257] [<ffffffff810415ed>] start_secondary+0x21d/0x2d0
Jan 18 12:33:53 e104462 kernel: [ 412.309258] Code: 48 89 d1 48 2d c8 1f 00 00 0f 01 c8 0f ae f0 65 48 8b 04 25 30 b8 00 00 48 8b 80 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ea 9b 8a 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 62 9f ce
Jan 18 12:34:21 e104462 kernel: [ 440.197840] BUG: soft lockup - CPU#1 stuck for 22s! [condvar_stress-:3380]
Jan 18 12:34:21 e104462 kernel: [ 440.197843] Modules linked in: pci_stub vboxpci(OX) vboxnetadp(OX) vboxnetflt(OX) vboxdrv(OX) bnep rfcomm bluetooth nfsd auth_rpcgss nfs_acl binfmt_misc nfs lockd sunrpc fscache dm_crypt snd_hda_codec_hdmi intel_rapl snd_hda_codec_realtek x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel snd_hda_intel snd_hda_codec kvm pl2303 usbserial snd_hwdep snd_pcm crct10dif_pclmul crc32_pclmul snd_page_alloc snd_seq_midi snd_seq_midi_event snd_rawmidi aesni_intel aes_x86_64 lrw snd_seq gf128mul glue_helper ablk_helper cryptd snd_seq_device serio_raw snd_timer lpc_ich snd soundcore mei_me mei shpchp mac_hid parport_pc ppdev lp parport hid_generic usbhid hid usb_storage nouveau mxm_wmi i2c_algo_bit ttm drm_kms_helper e1000e psmouse ahci drm ptp libahci pps_core video wmi
Jan 18 12:34:21 e104462 kernel: [ 440.197874] CPU: 1 PID: 3380 Comm: condvar_stress- Tainted: G W OX 3.13.0-74-generic #118-Ubuntu
Jan 18 12:34:21 e104462 kernel: [ 440.197875] Hardware name: LENOVO 32281S1/MAHOBAY, BIOS 9SKT71AUS 07/02/2013
Jan 18 12:34:21 e104462 kernel: [ 440.197876] task: ffff880401c4b000 ti: ffff8804067ca000 task.ti: ffff8804067ca000
Jan 18 12:34:21 e104462 kernel: [ 440.197877] RIP: 0010:[<ffffffff810dc9ae>] [<ffffffff810dc9ae>] smp_call_function_many+0x26e/0x2d0
Jan 18 12:34:21 e104462 kernel: [ 440.197882] RSP: 0018:ffff8804067cbbd8 EFLAGS: 00000202
Jan 18 12:34:21 e104462 kernel: [ 440.197882] RAX: 0000000000000002 RBX: ffff8804067cbb88 RCX: ffff88041dc96ff0
Jan 18 12:34:21 e104462 kernel: [ 440.197883] RDX: 0000000000000002 RSI: 0000000000000100 RDI: 0000000000000000
Jan 18 12:34:21 e104462 kernel: [ 440.197884] RBP: ffff8804067cbc28 R08: ffff88041dc53c88 R09: 0000000000000004
Jan 18 12:34:21 e104462 kernel: [ 440.197884] R10: ffff88041dc53c88 R11: ffff880402114510 R12: ffff88041dc56a58
Jan 18 12:34:21 e104462 kernel: [ 440.197885] R13: 000000000096716a R14: ffff880396d23000 R15: ffffffff00000000
Jan 18 12:34:21 e104462 kernel: [ 440.197886] FS: 0000000000000000(0000) GS:ffff88041dc40000(0000) knlGS:0000000000000000
Jan 18 12:34:21 e104462 kernel: [ 440.197887] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 18 12:34:21 e104462 kernel: [ 440.197888] CR2: 00002b6017fb2000 CR3: 0000000001c0e000 CR4: 00000000001407e4
Jan 18 12:34:21 e104462 kernel: [ 440.197889] DR0: 000000005589a248 DR1: 0000000000000000 DR2: 0000000000000000
Jan 18 12:34:21 e104462 kernel: [ 440.197890] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
Jan 18 12:34:21 e104462 kernel: [ 440.197890] Stack:
Jan 18 12:34:21 e104462 kernel: [ 440.197891] ffff88041dc53ca8 0000000000013c40 0000000000000000 ffffffff8101a620
Jan 18 12:34:21 e104462 kernel: [ 440.197893] 0000010000000001 ffffffff81096b94 ffffffff8101a620 0000000000000000
Jan 18 12:34:21 e104462 kernel: [ 440.197894] ffffffff81096b95 0000000000000000 ffff8804067cbc50 ffffffff810dcb6d
Jan 18 12:34:21 e104462 kernel: [ 440.197896] Call Trace:
Jan 18 12:34:21 e104462 kernel: [ 440.197899] [<ffffffff8101a620>] ? arch_unregister_cpu+0x30/0x30
Jan 18 12:34:21 e104462 kernel: [ 440.197902] [<ffffffff81096b94>] ? perf_event_task_sched_out+0x14/0xa0
Jan 18 12:34:21 e104462 kernel: [ 440.197903] [<ffffffff8101a620>] ? arch_unregister_cpu+0x30/0x30
Jan 18 12:34:21 e104462 kernel: [ 440.197905] [<ffffffff81096b95>] ? perf_event_task_sched_out+0x15/0xa0
Jan 18 12:34:21 e104462 kernel: [ 440.197906] [<ffffffff810dcb6d>] on_each_cpu+0x2d/0x60
Jan 18 12:34:21 e104462 kernel: [ 440.197908] [<ffffffff81096b94>] ? perf_event_task_sched_out+0x14/0xa0
Jan 18 12:34:21 e104462 kernel: [ 440.197909] [<ffffffff8101adeb>] text_poke_bp+0x5b/0xc0
Jan 18 12:34:21 e104462 kernel: [ 440.197911] [<ffffffff81096b94>] ? perf_event_task_sched_out+0x14/0xa0
Jan 18 12:34:21 e104462 kernel: [ 440.197912] [<ffffffff81018053>] arch_jump_label_transform+0xa3/0x120
Jan 18 12:34:21 e104462 kernel: [ 440.197915] [<ffffffff8114e09f>] __jump_label_update+0x5f/0x80
Jan 18 12:34:21 e104462 kernel: [ 440.197917] [<ffffffff8114e15d>] jump_label_update+0x9d/0xb0
Jan 18 12:34:21 e104462 kernel: [ 440.197918] [<ffffffff8114e3bd>] __static_key_slow_dec+0x4d/0xb0
Jan 18 12:34:21 e104462 kernel: [ 440.197920] [<ffffffff8114e442>] static_key_slow_dec+0x22/0x50
Jan 18 12:34:21 e104462 kernel: [ 440.197922] [<ffffffff811403be>] sw_perf_event_destroy+0x2e/0x80
Jan 18 12:34:21 e104462 kernel: [ 440.197924] [<ffffffff81141233>] __free_event+0x23/0x60
Jan 18 12:34:21 e104462 kernel: [ 440.197926] [<ffffffff811415b6>] free_event+0x96/0x170
Jan 18 12:34:21 e104462 kernel: [ 440.197928] [<ffffffff811429a8>] perf_event_release_kernel+0x68/0xc0
Jan 18 12:34:21 e104462 kernel: [ 440.197929] [<ffffffff81142a93>] put_event+0x93/0xc0
Jan 18 12:34:21 e104462 kernel: [ 440.197931] [<ffffffff81142ad0>] perf_release+0x10/0x20
Jan 18 12:34:21 e104462 kernel: [ 440.197934] [<ffffffff811c0084>] __fput+0xe4/0x260
Jan 18 12:34:21 e104462 kernel: [ 440.197936] [<ffffffff811c024e>] ____fput+0xe/0x10
Jan 18 12:34:21 e104462 kernel: [ 440.197939] [<ffffffff8108863c>] task_work_run+0xac/0xd0
Jan 18 12:34:21 e104462 kernel: [ 440.197941] [<ffffffff81069e28>] do_exit+0x2b8/0xa50
Jan 18 12:34:21 e104462 kernel: [ 440.197943] [<ffffffff8107a1ef>] ? ptrace_notify+0x5f/0xc0
Jan 18 12:34:21 e104462 kernel: [ 440.197944] [<ffffffff8106a63f>] do_group_exit+0x3f/0xa0
Jan 18 12:34:21 e104462 kernel: [ 440.197946] [<ffffffff8106a6b4>] SyS_exit_group+0x14/0x20
Jan 18 12:34:21 e104462 kernel: [ 440.197948] [<ffffffff8173596f>] tracesys+0xe1/0xe6
Jan 18 12:34:21 e104462 kernel: [ 440.197949] Code: 3b 05 5f 8e c3 00 89 c2 0f 8d 25 fe ff ff 48 98 49 8b 4d 00 48 03 0c c5 e0 39 d1 81 f6 41 20 01 74 cb 0f 1f 00 f3 90 f6 41 20 01 <75> f8 eb be 0f b6 4d d0 48 8b 55 c0 44 89 ef 48 8b 75 c8 e8 ea

(and more of the same, presumably triggered by further tests in the test suite.)

At this point some bits of the system were unresponsive (like the X mouse pointer). I tried a clean shutdown via ssh, but this hung and I had to hard-reboot the machine.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-3.13.0-74-generic 3.13.0-74.118
ProcVersionSignature: Ubuntu 3.13.0-74.118-generic 3.13.11-ckt30
Uname: Linux 3.13.0-74-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.19
Architecture: amd64
Date: Mon Jan 18 13:17:39 2016
HibernationDevice: RESUME=UUID=81c6feee-5e1e-4949-be9b-fba2bcaed754
InstallationDate: Installed on 2014-05-22 (606 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
IwConfig:
 eth0 no wireless extensions.

 lo no wireless extensions.
MachineType: LENOVO 32281S1
ProcFB: 0 nouveaufb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-74-generic root=UUID=30201b50-8929-40b1-822f-7dfbf8a5a134 ro quiet splash vt.handoff=7
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-74-generic N/A
 linux-backports-modules-3.13.0-74-generic N/A
 linux-firmware 1.127.19
RfKill:

SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 07/02/2013
dmi.bios.vendor: LENOVO
dmi.bios.version: 9SKT71AUS
dmi.board.name: MAHOBAY
dmi.board.vendor: LENOVO
dmi.board.version: NOK
dmi.chassis.type: 3
dmi.chassis.vendor: To Be Filled By O.E.M.
dmi.chassis.version: To Be Filled By O.E.M.
dmi.modalias: dmi:bvnLENOVO:bvr9SKT71AUS:bd07/02/2013:svnLENOVO:pn32281S1:pvrThinkCentreM92p:rvnLENOVO:rnMAHOBAY:rvrNOK:cvnToBeFilledByO.E.M.:ct3:cvrToBeFilledByO.E.M.:
dmi.product.name: 32281S1
dmi.product.version: ThinkCentre M92p
dmi.sys.vendor: LENOVO

Revision history for this message
Peter Maydell (pmaydell) wrote :
Revision history for this message
Brad Figg (brad-figg) wrote : Status changed to Confirmed

This change was made by a bot.

Changed in linux (Ubuntu):
status: New → Confirmed
Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.4 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4-wily

Changed in linux (Ubuntu):
importance: Undecided → Medium
status: Confirmed → Incomplete
tags: added: kernel-da-key
Revision history for this message
Peter Maydell (pmaydell) wrote :

No, I don't think this was associated with a kernel upgrade, though it's hard to say for certain as I hadn't run the test suite in some months. I suspect it's more that rr's test suite got more complex and included some stress tests that reveal pre-existing kernel bugs.

I will test the upstream kernel.

Revision history for this message
Peter Maydell (pmaydell) wrote :
Download full text (3.9 KiB)

I tested with linux-headers-4.4.0-040400 / linux-headers-4.4.0-040400-generic / linux-image-4.4.0-040400-generic 4.4.0-040400.201601101930. The kernel lockups did *not* reproduce.

Two caveats:

(1) I did get this kernel warning in the log:
Jan 21 18:00:08 e104462 kernel: [ 171.577000] ------------[ cut here ]------------
Jan 21 18:00:08 e104462 kernel: [ 171.577005] WARNING: CPU: 6 PID: 14136 at /home/kernel/COD/linux/kernel/signal.c:2072 get_signal+0x597/0x600()
Jan 21 18:00:08 e104462 kernel: [ 171.577006] Modules linked in: rfcomm bnep bluetooth nfsd auth_rpcgss nfs_acl nfs binfmt_misc lockd grace sunrpc fscache drbg ansi_cprng snd_hda_codec_hdmi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm dm_crypt irqbypass crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw snd_hda_codec_realtek snd_hda_codec_generic gf128mul glue_helper ablk_helper snd_hda_intel cryptd snd_hda_codec snd_hda_core snd_hwdep input_leds pl2303 snd_pcm usbserial snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd 8250_fintek soundcore mei_me mei shpchp ie31200_edac lpc_ich ppdev edac_core serio_raw mac_hid parport_pc lp parport hid_generic usbhid hid uas usb_storage nouveau mxm_wmi i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops e1000e drm ahci psmouse libahci ptp pps_core wmi fjes video
Jan 21 18:00:08 e104462 kernel: [ 171.577037] CPU: 6 PID: 14136 Comm: block-z8iSdZiRK Not tainted 4.4.0-040400-generic #201601101930
Jan 21 18:00:08 e104462 kernel: [ 171.577038] Hardware name: LENOVO 32281S1/MAHOBAY, BIOS 9SKT71AUS 07/02/2013
Jan 21 18:00:08 e104462 kernel: [ 171.577039] 0000000000000000 00000000a495bf0f ffff88040afdbd20 ffffffff813c8d94
Jan 21 18:00:08 e104462 kernel: [ 171.577040] 0000000000000000 ffff88040afdbd58 ffffffff8107dba2 0000000000000000
Jan 21 18:00:08 e104462 kernel: [ 171.577042] ffff88040afdbe38 0000000000008000 ffff88040afdbf58 ffff880405d58000
Jan 21 18:00:08 e104462 kernel: [ 171.577043] Call Trace:
Jan 21 18:00:08 e104462 kernel: [ 171.577046] [<ffffffff813c8d94>] dump_stack+0x44/0x60
Jan 21 18:00:08 e104462 kernel: [ 171.577049] [<ffffffff8107dba2>] warn_slowpath_common+0x82/0xc0
Jan 21 18:00:08 e104462 kernel: [ 171.577050] [<ffffffff8107dcea>] warn_slowpath_null+0x1a/0x20
Jan 21 18:00:08 e104462 kernel: [ 171.577051] [<ffffffff8108d0e7>] get_signal+0x597/0x600
Jan 21 18:00:08 e104462 kernel: [ 171.577053] [<ffffffff81016517>] do_signal+0x37/0x6f0
Jan 21 18:00:08 e104462 kernel: [ 171.577055] [<ffffffff817f9925>] ? schedule+0x35/0x80
Jan 21 18:00:08 e104462 kernel: [ 171.577057] [<ffffffff81080b96>] ? do_exit+0x736/0xab0
Jan 21 18:00:08 e104462 kernel: [ 171.577058] [<ffffffff81021272>] ? __switch_to_xtra+0xe2/0x1b0
Jan 21 18:00:08 e104462 kernel: [ 171.577061] [<ffffffff8100320c>] exit_to_usermode_loop+0x8c/0xd0
Jan 21 18:00:08 e104462 kernel: [ 171.577062] [<ffffffff81003c2e>] syscall_return_slowpath+0x4e/0x60
Jan 21 18:00:08 e104462 kernel: [ 171.577064] [<ffffffff817fdb18>] int_ret_from_sys_call+0x25/0x8f
Jan 21 18:00:08 e104462 kernel: [ 171.577065] ---[ end trace 1c1af43c0c2249f3 ]---

but it seems to be a completely dif...

Read more...

tags: added: kernel-fixed-upstream
Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Peter Maydell (pmaydell) wrote :

I retried with a workaround for the rr bug I described in "caveat (2)" and the kernel still does not lockup, so I am now confident that this bug is not present in the upstream kernel.

Revision history for this message
penalvch (penalvch) wrote :

Peter Maydell, the next step is to fully reverse commit bisect from kernel 3.13 to 4.4 in order to identify the last bad commit, followed immediately by the first good one. Once this good commit has been identified, it may be reviewed for backporting. Could you please do this following https://wiki.ubuntu.com/Kernel/KernelBisection#How_do_I_reverse_bisect_the_upstream_kernel.3F ?

Please note, finding adjacent kernel versions is not fully commit bisecting.

After the fix commit (not kernel version) has been identified, then please mark this report Status Confirmed.

Thank you for your understanding.

Helpful bug reporting tips:
https://wiki.ubuntu.com/ReportingBugs

tags: added: bios-outdated-9skt91a kernel-fixed-upstream-4.5-rc1 needs-reverse-bisect
Changed in linux (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Peter Maydell (pmaydell) wrote :

I've now completed the kernel git bisect. git bisect says the commit which fixed this issue is 29722cd4ef666705b2eda1c3ba44435488e509eb ("x86/asm/entry/64: Save R11 into pt_regs->flags on SYSCALL64 fastpath").

This fits in with the discovery on the rr side that the rr commit which started causing the test suite to deadlock the kernel is https://github.com/mozilla/rr/commit/36ac389649e3e84caa46d64487813c018857f831 which is an R11-related change.

rr upstream have now committed https://github.com/mozilla/rr/commit/11e67a66a2218effb9381b9ac86a2085b0f5d5d1 which is essentially working around this kernel bug. However, since "unprivileged user can deadlock the kernel" is a local-user DoS exploit I think the fix is still worth backporting.

I haven't attempted to analyze the kernel change to determine why the kernel was crashing before or if you could do something more interesting than just crash it by setting r11 to something other than 0 in the userspace code.

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Peter Maydell (pmaydell) wrote :

PS: it's possible that that commit doesn't actually fix the underlying kernel crash, it just means that rr isn't triggering it any more, and that if you modified EFLAGS via the ptrace interface rather than r11 you'd get the crash back again.

Seth Arnold (seth-arnold)
information type: Public → Public Security
penalvch (penalvch)
tags: added: cherry-pick reverse-bisect-done
removed: needs-reverse-bisect
Changed in linux (Ubuntu):
importance: Medium → High
status: Confirmed → Triaged
Revision history for this message
Peter Maydell (pmaydell) wrote :

I wrote: "PS: it's possible that that commit doesn't actually fix the underlying kernel crash, it just means that rr isn't triggering it any more, and that if you modified EFLAGS via the ptrace interface rather than r11 you'd get the crash back again."
but looking at the kernel I think that is impossible, since the set_flags() function in arch/x86/kernel/ptrace.c sanitizes eflags if set via ptrace. The issue fixed by 29722cd4ef66 is that setting r11 in some situations would bypass the sanitizing.

To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.