------- Comment From <email address hidden> 2017-08-23 09:29 EDT------- (In reply to comment #28) > Retested kdump today (23rd Aug 2017) on Ubuntu1610 and kdump hangs still: > ------------------------- > root@thymelp3:~# echo c> /proc/sysrq-trigger > [ 1314.534126] sysrq: SysRq : Trigger a crash > [ 1314.534139] Unable to handle kernel paging request for data at address > 0x00000000 > [ 1314.534143] Faulting instruction address: 0xc0000000006a2428 > [ 1314.534147] Oops: Kernel access of bad area, sig: 11 [#1] > [ 1314.534150] SMP NR_CPUS=2048 NUMA pSeries > [ 1314.534154] Modules linked in: nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss > nfsv4 nfs lockd grace fscache binfmt_misc pseries_rng vmx_crypto sunrpc > ip_tables x_tables autofs4 dm_round_robin btrfs xor raid6_pq lpfc > crc32c_vpmsum be2net scsi_transport_fc scsi_dh_emc scsi_dh_rdac scsi_dh_alua > dm_multipath > [ 1314.534177] CPU: 9 PID: 3421 Comm: bash Not tainted 4.8.0-59-generic > #64-Ubuntu > [ 1314.534181] task: c0000003efc25200 task.stack: c0000000fb970000 > [ 1314.534184] NIP: c0000000006a2428 LR: c0000000006a3478 CTR: > c0000000006a2400 > [ 1314.534187] REGS: c0000000fb9739f0 TRAP: 0300 Not tainted > (4.8.0-59-generic) > [ 1314.534190] MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE> CR: 28222222 > XER: 00000001 > [ 1314.534198] CFAR: c000000000008750 DAR: 0000000000000000 DSISR: 42000000 > SOFTE: 1 > GPR00: c0000000006a3478 c0000000fb973c70 c000000001467500 0000000000000063 > GPR04: c0000003ff64aca0 c0000003ff65fb40 c0000003ff380000 00000000000080a0 > GPR08: 0000000000000007 0000000000000001 0000000000000000 0000000000000001 > GPR12: c0000000006a2400 c000000007b35100 0000000000000000 0000000022000000 > GPR16: 0000000010170dc8 000001000bbf0258 0000000010140528 00000000100c6f60 > GPR20: 0000000000000000 000000001017dd58 0000000010152bf0 000000001017b608 > GPR24: 00003fffd72098a4 00003fffd72098a0 c00000000137e6e0 0000000000000004 > GPR28: c00000000137eaa0 0000000000000063 c000000001332590 0000000000000000 > [ 1314.534242] NIP [c0000000006a2428] sysrq_handle_crash+0x28/0x30 > [ 1314.534246] LR [c0000000006a3478] __handle_sysrq+0xe8/0x280 > [ 1314.534248] Call Trace: > [ 1314.534250] [c0000000fb973c70] [c0000000006a3458] > __handle_sysrq+0xc8/0x280 (unreliable) > [ 1314.534255] [c0000000fb973d10] [c0000000006a3bcc] > write_sysrq_trigger+0x6c/0x90 > [ 1314.534260] [c0000000fb973d40] [c0000000003adb48] proc_reg_write+0x88/0xd0 > [ 1314.534265] [c0000000fb973d70] [c0000000003105ac] __vfs_write+0x3c/0x70 > [ 1314.534268] [c0000000fb973d90] [c000000000311814] vfs_write+0xd4/0x240 > [ 1314.534272] [c0000000fb973de0] [c000000000313368] SyS_write+0x68/0x110 > [ 1314.534276] [c0000000fb973e30] [c000000000009584] system_call+0x38/0xec > [ 1314.534279] Instruction dump: > [ 1314.534281] 60000000 60000000 3c4c00dc 38425100 7c0802a6 60000000 > 3d22001a 3949bc60 > [ 1314.534288] 39200001 912a0000 7c0004ac 39400000 <992a0000> 4e800020 > 3c4c00dc 384250d0 > [ 1314.534296] ---[ end trace efc32115f1d43c62 ]--- > [ 1314.537099] > [ 1314.537123] Sending IPI to other CPUs > [ 1314.538149] IPI complete > I'm in purgatory > -> smp_release_cpus() > spinning_secondaries = 8 > <- smp_release_cpus() > [ 0.172393] pci 001b:50:00.0: of_irq_parse_pci() failed with rc=-22 > [ 0.425077] Kernel panic - not syncing: Out of memory and no killable > processes... > [ 0.425077] > [ 0.425100] CPU: 2 PID: 1 Comm: swapper/1 Not tainted 4.8.0-59-generic > #64-Ubuntu > [ 0.425102] Call Trace: > [ 0.425105] [c00000000d10b220] [c000000008b0fe4c] dump_stack+0xb0/0xf0 > (unreliable) > [ 0.425110] [c00000000d10b260] [c000000008b0bf58] panic+0x144/0x308 > [ 0.425114] [c00000000d10b2f0] [c000000008249c2c] > out_of_memory+0x48c/0x570 > [ 0.425117] [c00000000d10b3a0] [c000000008250ad8] > __alloc_pages_nodemask+0xdf8/0xe20 > [ 0.425122] [c00000000d10b560] [c0000000082c6da8] > alloc_page_interleave+0x58/0xc0 > [ 0.425126] [c00000000d10b5a0] [c0000000082c7678] > alloc_pages_current+0x168/0x1d0 > [ 0.425130] [c00000000d10b600] [c0000000082435e8] > __page_cache_alloc+0x118/0x160 > [ 0.425134] [c00000000d10b640] [c0000000082437b4] > pagecache_get_page+0x184/0x3c0 > [ 0.425138] [c00000000d10b6b0] [c000000008243a34] > grab_cache_page_write_begin+0x44/0x70 > [ 0.425142] [c00000000d10b6e0] [c00000000834bf6c] > simple_write_begin+0x4c/0x1b0 > [ 0.425146] [c00000000d10b730] [c000000008243264] > generic_perform_write+0x104/0x280 > [ 0.425150] [c00000000d10b7d0] [c000000008245540] > __generic_file_write_iter+0x1e0/0x230 > [ 0.425154] [c00000000d10b830] [c00000000824567c] > generic_file_write_iter+0xec/0x250 > [ 0.425158] [c00000000d10b870] [c00000000831050c] > new_sync_write+0xec/0x150 > [ 0.425162] [c00000000d10b900] [c000000008311814] vfs_write+0xd4/0x240 > [ 0.425165] [c00000000d10b950] [c000000008313368] SyS_write+0x68/0x110 > [ 0.425169] [c00000000d10b9a0] [c000000008ea5d0c] xwrite+0x4c/0xb0 > [ 0.425173] [c00000000d10b9e0] [c000000008ea5e60] do_copy+0xf0/0x170 > [ 0.425176] [c00000000d10ba10] [c000000008ea59c4] write_buffer+0x5c/0x88 > [ 0.425180] [c00000000d10ba40] [c000000008ea5a50] flush_buffer+0x60/0xec > [ 0.425183] [c00000000d10ba90] [c000000008eec4c8] __gunzip+0x378/0x47c > [ 0.425187] [c00000000d10bb10] [c000000008ea650c] > unpack_to_rootfs+0x1c8/0x338 > [ 0.425191] [c00000000d10bbc0] [c000000008ea688c] > populate_rootfs+0x94/0x17c > [ 0.425195] [c00000000d10bc40] [c00000000800b948] > do_one_initcall+0x68/0x1d0 > [ 0.425198] [c00000000d10bd00] [c000000008ea42e8] > kernel_init_freeable+0x278/0x360 > [ 0.425202] [c00000000d10bdc0] [c00000000800c1b4] kernel_init+0x24/0x170 > [ 0.425206] [c00000000d10be30] [c0000000080098f0] > ret_from_kernel_thread+0x5c/0x6c > [ 0.429518] ---[ end Kernel panic - not syncing: Out of memory and no > killable processes...
Looks like not enough memory was reserved for KDump. Set crashkernel value based on below recommendations and try again:
https://wiki.ubuntu.com/ppc64el/Recommendations#Crash_Kernel_recommendations
Thanks Hari
------- Comment From <email address hidden> 2017-08-23 09:29 EDT------- ------- ------- ---- ME,IR,DR, RI,LE> CR: 28222222 crash+0x28/ 0x30 sysrq+0xe8/ 0x280 sysrq+0xc8/ 0x280 (unreliable) trigger+ 0x6c/0x90 write+0x88/ 0xd0 0x3c/0x70 0xd4/0x240 0x68/0x110 call+0x38/ 0xec secondaries = 8 0xb0/0xf0 memory+ 0x48c/0x570 pages_nodemask+ 0xdf8/0xe20 interleave+ 0x58/0xc0 current+ 0x168/0x1d0 cache_alloc+ 0x118/0x160 get_page+ 0x184/0x3c0 page_write_ begin+0x44/ 0x70 write_begin+ 0x4c/0x1b0 perform_ write+0x104/ 0x280 file_write_ iter+0x1e0/ 0x230 file_write_ iter+0xec/ 0x250 write+0xec/ 0x150 0xd4/0x240 0x68/0x110 0x5c/0x88 0x60/0xec 0x378/0x47c to_rootfs+ 0x1c8/0x338 rootfs+ 0x94/0x17c initcall+ 0x68/0x1d0 init_freeable+ 0x278/0x360 init+0x24/ 0x170 kernel_ thread+ 0x5c/0x6c
(In reply to comment #28)
> Retested kdump today (23rd Aug 2017) on Ubuntu1610 and kdump hangs still:
> -------
> root@thymelp3:~# echo c> /proc/sysrq-trigger
> [ 1314.534126] sysrq: SysRq : Trigger a crash
> [ 1314.534139] Unable to handle kernel paging request for data at address
> 0x00000000
> [ 1314.534143] Faulting instruction address: 0xc0000000006a2428
> [ 1314.534147] Oops: Kernel access of bad area, sig: 11 [#1]
> [ 1314.534150] SMP NR_CPUS=2048 NUMA pSeries
> [ 1314.534154] Modules linked in: nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss
> nfsv4 nfs lockd grace fscache binfmt_misc pseries_rng vmx_crypto sunrpc
> ip_tables x_tables autofs4 dm_round_robin btrfs xor raid6_pq lpfc
> crc32c_vpmsum be2net scsi_transport_fc scsi_dh_emc scsi_dh_rdac scsi_dh_alua
> dm_multipath
> [ 1314.534177] CPU: 9 PID: 3421 Comm: bash Not tainted 4.8.0-59-generic
> #64-Ubuntu
> [ 1314.534181] task: c0000003efc25200 task.stack: c0000000fb970000
> [ 1314.534184] NIP: c0000000006a2428 LR: c0000000006a3478 CTR:
> c0000000006a2400
> [ 1314.534187] REGS: c0000000fb9739f0 TRAP: 0300 Not tainted
> (4.8.0-59-generic)
> [ 1314.534190] MSR: 8000000000009033 <SF,EE,
> XER: 00000001
> [ 1314.534198] CFAR: c000000000008750 DAR: 0000000000000000 DSISR: 42000000
> SOFTE: 1
> GPR00: c0000000006a3478 c0000000fb973c70 c000000001467500 0000000000000063
> GPR04: c0000003ff64aca0 c0000003ff65fb40 c0000003ff380000 00000000000080a0
> GPR08: 0000000000000007 0000000000000001 0000000000000000 0000000000000001
> GPR12: c0000000006a2400 c000000007b35100 0000000000000000 0000000022000000
> GPR16: 0000000010170dc8 000001000bbf0258 0000000010140528 00000000100c6f60
> GPR20: 0000000000000000 000000001017dd58 0000000010152bf0 000000001017b608
> GPR24: 00003fffd72098a4 00003fffd72098a0 c00000000137e6e0 0000000000000004
> GPR28: c00000000137eaa0 0000000000000063 c000000001332590 0000000000000000
> [ 1314.534242] NIP [c0000000006a2428] sysrq_handle_
> [ 1314.534246] LR [c0000000006a3478] __handle_
> [ 1314.534248] Call Trace:
> [ 1314.534250] [c0000000fb973c70] [c0000000006a3458]
> __handle_
> [ 1314.534255] [c0000000fb973d10] [c0000000006a3bcc]
> write_sysrq_
> [ 1314.534260] [c0000000fb973d40] [c0000000003adb48] proc_reg_
> [ 1314.534265] [c0000000fb973d70] [c0000000003105ac] __vfs_write+
> [ 1314.534268] [c0000000fb973d90] [c000000000311814] vfs_write+
> [ 1314.534272] [c0000000fb973de0] [c000000000313368] SyS_write+
> [ 1314.534276] [c0000000fb973e30] [c000000000009584] system_
> [ 1314.534279] Instruction dump:
> [ 1314.534281] 60000000 60000000 3c4c00dc 38425100 7c0802a6 60000000
> 3d22001a 3949bc60
> [ 1314.534288] 39200001 912a0000 7c0004ac 39400000 <992a0000> 4e800020
> 3c4c00dc 384250d0
> [ 1314.534296] ---[ end trace efc32115f1d43c62 ]---
> [ 1314.537099]
> [ 1314.537123] Sending IPI to other CPUs
> [ 1314.538149] IPI complete
> I'm in purgatory
> -> smp_release_cpus()
> spinning_
> <- smp_release_cpus()
> [ 0.172393] pci 001b:50:00.0: of_irq_parse_pci() failed with rc=-22
> [ 0.425077] Kernel panic - not syncing: Out of memory and no killable
> processes...
> [ 0.425077]
> [ 0.425100] CPU: 2 PID: 1 Comm: swapper/1 Not tainted 4.8.0-59-generic
> #64-Ubuntu
> [ 0.425102] Call Trace:
> [ 0.425105] [c00000000d10b220] [c000000008b0fe4c] dump_stack+
> (unreliable)
> [ 0.425110] [c00000000d10b260] [c000000008b0bf58] panic+0x144/0x308
> [ 0.425114] [c00000000d10b2f0] [c000000008249c2c]
> out_of_
> [ 0.425117] [c00000000d10b3a0] [c000000008250ad8]
> __alloc_
> [ 0.425122] [c00000000d10b560] [c0000000082c6da8]
> alloc_page_
> [ 0.425126] [c00000000d10b5a0] [c0000000082c7678]
> alloc_pages_
> [ 0.425130] [c00000000d10b600] [c0000000082435e8]
> __page_
> [ 0.425134] [c00000000d10b640] [c0000000082437b4]
> pagecache_
> [ 0.425138] [c00000000d10b6b0] [c000000008243a34]
> grab_cache_
> [ 0.425142] [c00000000d10b6e0] [c00000000834bf6c]
> simple_
> [ 0.425146] [c00000000d10b730] [c000000008243264]
> generic_
> [ 0.425150] [c00000000d10b7d0] [c000000008245540]
> __generic_
> [ 0.425154] [c00000000d10b830] [c00000000824567c]
> generic_
> [ 0.425158] [c00000000d10b870] [c00000000831050c]
> new_sync_
> [ 0.425162] [c00000000d10b900] [c000000008311814] vfs_write+
> [ 0.425165] [c00000000d10b950] [c000000008313368] SyS_write+
> [ 0.425169] [c00000000d10b9a0] [c000000008ea5d0c] xwrite+0x4c/0xb0
> [ 0.425173] [c00000000d10b9e0] [c000000008ea5e60] do_copy+0xf0/0x170
> [ 0.425176] [c00000000d10ba10] [c000000008ea59c4] write_buffer+
> [ 0.425180] [c00000000d10ba40] [c000000008ea5a50] flush_buffer+
> [ 0.425183] [c00000000d10ba90] [c000000008eec4c8] __gunzip+
> [ 0.425187] [c00000000d10bb10] [c000000008ea650c]
> unpack_
> [ 0.425191] [c00000000d10bbc0] [c000000008ea688c]
> populate_
> [ 0.425195] [c00000000d10bc40] [c00000000800b948]
> do_one_
> [ 0.425198] [c00000000d10bd00] [c000000008ea42e8]
> kernel_
> [ 0.425202] [c00000000d10bdc0] [c00000000800c1b4] kernel_
> [ 0.425206] [c00000000d10be30] [c0000000080098f0]
> ret_from_
> [ 0.429518] ---[ end Kernel panic - not syncing: Out of memory and no
> killable processes...
Looks like not enough memory was reserved for KDump.
Set crashkernel value based on below recommendations and try again:
https:/ /wiki.ubuntu. com/ppc64el/ Recommendations #Crash_ Kernel_ recommendations
Thanks
Hari