kswapd takes 100% CPU although there is no swap

Bug #1476211 reported by Andreas E.
This bug report is a duplicate of:  Bug #1518457: kswapd0 100% CPU usage. Edit Remove
56
This bug affects 10 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Confirmed
Medium
Unassigned

Bug Description

On a notebook with neither swap partition nor swap file (because of SSD and surplus of RAM), kswapd0 suddenly runs amok and takes 100% of CPU activity. When this happens, usually at most 5.5 of 8 GB RAM were in use, so there is no reason to swap.

This happens so suddenly, that no user input is possible anymore, eg. I can barely switch to tty to kill processes that take most RAM. The processor becomes extremely hot and I can only turn the computer off and reboot. I consider this as critical and harmful to my hardware.

I noticed that Ubuntu installs by default with swappiness 60. Now I changed that to 10 because I would never encounter the case that 8GB are in use, but it didn't have an effect.

Expected behavior:
- on systems without RAM, kswapd should never attempt to swap
- if there is not enough RAM, kernel should kill processes to free up RAM
- no process should be able to claim so much CPU activity, that the computer becomes unresponsive (it should always be possible to control/stop/kill processes over tty)

Actual behavior:
- kswapd tries to swap although there is no swap space

ProblemType: Bug
DistroRelease: Ubuntu 15.04
Package: linux-image-3.19.0-22-generic 3.19.0-22.22
ProcVersionSignature: Ubuntu 3.19.0-22.22-generic 3.19.8-ckt1
Uname: Linux 3.19.0-22-generic x86_64
ApportVersion: 2.17.2-0ubuntu1.1
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: andreas 2368 F.... pulseaudio
 /dev/snd/controlC0: andreas 2368 F.... pulseaudio
CurrentDesktop: Unity
Date: Mon Jul 20 12:49:24 2015
EcryptfsInUse: Yes
InstallationDate: Installed on 2015-06-06 (43 days ago)
InstallationMedia: Ubuntu 15.04 "Vivid Vervet" - Release amd64 (20150422)
MachineType: LENOVO 20266
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-22-generic root=UUID=8a0dc2f3-fcf3-4e5b-8e23-1441bd36e3dc ro noprompt persistent quiet splash acpi_backlight=vendor vt.handoff=7 init=/sbin/upstart
RelatedPackageVersions:
 linux-restricted-modules-3.19.0-22-generic N/A
 linux-backports-modules-3.19.0-22-generic N/A
 linux-firmware 1.143.1
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 10/31/2013
dmi.bios.vendor: LENOVO
dmi.bios.version: 76CN31WW
dmi.board.asset.tag: No Asset Tag
dmi.board.name: Yoga2
dmi.board.vendor: LENOVO
dmi.board.version: 31900058STD
dmi.chassis.asset.tag: No Asset Tag
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Lenovo Yoga 2 Pro
dmi.modalias: dmi:bvnLENOVO:bvr76CN31WW:bd10/31/2013:svnLENOVO:pn20266:pvrLenovoYoga2Pro:rvnLENOVO:rnYoga2:rvr31900058STD:cvnLENOVO:ct10:cvrLenovoYoga2Pro:
dmi.product.name: 20266
dmi.product.version: Lenovo Yoga 2 Pro
dmi.sys.vendor: LENOVO

Revision history for this message
Andreas E. (andreas-e) wrote :
description: updated
Revision history for this message
Brad Figg (brad-figg) wrote : Status changed to Confirmed

This change was made by a bot.

Changed in linux (Ubuntu):
status: New → Confirmed
Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.2 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.2-rc3-unstable/

Changed in linux (Ubuntu):
importance: Undecided → Medium
status: Confirmed → Incomplete
Revision history for this message
Andreas E. (andreas-e) wrote :

I put 4.2.0-040200rc3-generic under RAM stress. I still can not get to over 5GB (of 8GB) before kswapd turns the system unusable.

tags: added: kernel-bug-exists-upstream
Changed in linux (Ubuntu):
status: Incomplete → Confirmed
penalvch (penalvch)
tags: added: bios-outdated-76cn43ww
Changed in linux (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Andreas E. (andreas-e) wrote :

[[Ubuntu does not provide an update path for (these) Lenovo BIOS. Maybe it would be nice to have a solution for this as well, to avoid users being bound to buggy and outdated BIOS. To add, the given instructions to the wiki seem very abandonned and do not (not all) apply to nowadays ROMs of >10MB. Figuring that out cost me a day. ]]

After installing the Win8/BIOS, the problem still occurs.
  sudo dmidecode -s bios-version && sudo dmidecode -s bios-release-date
  76CN43WW
  03/30/2015

When I intentionally launch so many applications to trigger the freezes, I noticed that `top` displays actually different numbers:
  KiB Mem: 8085948 total, 8023384 used, 62564 free, 3564 buffers
  KiB Swap: 0 total, 0 used, 0 free. 2976336 cached Mem
The before mentioned numbers were from Gnome system monitor (oh!), which always displays only 5GB / 65% of RAM. This time had also top opened before the freeze.
Do I interprete it right that under these conditions the memory is indeed full and that system monitor displays (used‒cached) ?

The laptop was still frozen about 10min until the Kernel decided (or was able to) kill processes and became repsonsive again. On a second test, it didn't become responsive even after 20min.

So concluding,
• there could be a bug in system monitor to display (used ‒ cached) RAM
• I still don't know what it is doing within these 10 – ∞ minutes. I still think the Kernel should be able to start killing processes (earlier), or there should be warnings that can take effect before the computer becomes unresponsive.

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
penalvch (penalvch)
tags: added: latest-bios-76cn43ww
removed: bios-outdated-76cn43ww
Revision history for this message
Sam Soldatenko (sam-ngs-ru) wrote :
Download full text (10.6 KiB)

I have similar behaviour on my Ubuntu 15.04, Intel Atom D2500 Dual LAN, Dual COM Fanless Mini-ITX PC with 4 GB RAM and 128 GB SSD.

In few days after reboot kswapd0 starts to take 100% CPU (see highlighted row here https://gist.github.com/SamSoldatenko/83eaf196721bc51d7d55#file-gistfile1-txt-L29 ). No load. 3Gb free memory (out of 4), 10Gb swap space.

$ uname -a
Linux minisrv1 3.19.0-26-generic #28-Ubuntu SMP Tue Aug 11 14:16:45 UTC 2015 i686 i686 i686 GNU/Linux

$ sudo dmidecode -s bios-version && sudo dmidecode -s bios-release-date
CCCDT10N.86A.0032.2012.0323.1510
03/23/2012

$ uptime
 22:39:01 up 8 days, 5:43, 1 user, load average: 1.34, 1.23, 1.23

$ sudo ps auxf
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2 0.0 0.0 0 0 ? S Sep26 0:15 [kthreadd]
root 3 0.0 0.0 0 0 ? S Sep26 4:39 \_ [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [kworker/0:0H]
root 7 0.6 0.0 0 0 ? S Sep26 76:29 \_ [rcu_sched]
root 8 0.0 0.0 0 0 ? S Sep26 0:00 \_ [rcu_bh]
root 9 0.5 0.0 0 0 ? S Sep26 64:30 \_ [migration/0]
root 10 0.2 0.0 0 0 ? S Sep26 23:57 \_ [watchdog/0]
root 11 0.2 0.0 0 0 ? S Sep26 26:02 \_ [watchdog/1]
root 12 0.6 0.0 0 0 ? S Sep26 75:46 \_ [migration/1]
root 13 0.0 0.0 0 0 ? S Sep26 4:38 \_ [ksoftirqd/1]
root 15 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [kworker/1:0H]
root 16 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [khelper]
root 17 0.0 0.0 0 0 ? S Sep26 0:00 \_ [kdevtmpfs]
root 18 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [netns]
root 19 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [perf]
root 20 0.0 0.0 0 0 ? S Sep26 0:05 \_ [khungtaskd]
root 21 0.0 0.0 0 0 ? S< Sep26 0:01 \_ [writeback]
root 22 0.0 0.0 0 0 ? SN Sep26 0:00 \_ [ksmd]
root 23 0.0 0.0 0 0 ? SN Sep26 0:40 \_ [khugepaged]
root 24 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [crypto]
root 25 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [kintegrityd]
root 26 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [bioset]
root 27 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [kblockd]
root 28 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [ata_sff]
root 29 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [md]
root 30 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [devfreq_wq]
root 34 31.5 0.0 0 0 ? R Sep26 3748:10 \_ [kswapd0]
root 35 0.0 0.0 0 0 ? S Sep26 0:00 \_ [fsnotify_mark]
root 36 0.0 0.0 0 0 ? S Sep26 0:00 \_ [ecryptfs-kthrea]
root 48 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [kthrotld]
root 49 0.0 0.0 ...

Revision history for this message
Andreas E. (andreas-e) wrote :

This is still the case in Ubuntu 15.10.
It would really be cool if this could be fixed, because it causes data loss and regularly interrupts my work. I cannot imagine this same software runs on servers, IoT and systems that need reliability.

Revision history for this message
Ski (kacoroski) wrote :

I am on xbuntu 15.10 and this affected me after I upgraded. I have 16GB ram and kswapd kicks in, locks it up for 10 - 30 seconds multiple times an hour. vmstat 5 shows:

1 0 1208112 569056 69376 7019032 0 0 84 347 1634 2951 25 1 73 0 0
 2 0 1208072 164192 2212 6619100 14 6 163 10 2177 5029 26 3 70 0 0
 0 20 2366584 204456 316 6367792 48 234926 25103 235206 6928 2748 2 8 5 86 0
 1 5 3294060 151216 548 6413108 27001 193908 88306 193970 5196 6590 2 10 13 76 0
 2 1 4065264 172748 3172 6488812 72246 163202 106635 163531 8388 11018 2 22 43 33 0
 5 0 4806852 136160 4444 6469352 76391 153371 98650 154074 8352 9759 2 26 37 35 0
 1 1 5573372 145112 816 6440936 73558 157316 87340 157662 6802 8592 2 21 44 33 0
 0 4 6383744 168976 1800 6463780 74014 166143 88715 166679 6388 8669 2 27 41 31 0
 1 1 7024704 146832 1924 6466112 68409 135974 78778 136382 6356 8203 1 19 42 38 0
 2 0 7736084 181116 1928 6476672 56372 144775 64527 145122 8093 6853 1 16 41 43 0
 0 7 8381188 197892 432 6456988 74326 159450 87829 160040 8327 8744 4 23 45 28 0
 1 0 3944136 3240020 1400 6488888 9170 67 14278 74 2112 4084 22 3 69 6 0
 1 0 3926636 2819400 2060 6521132 3596 0 7028 0 1848 3374 25 3 71 1 0

So you can see when it kicks in. I have made the following modifications and still have the problem:

/etc/sysctl.conf:
vm.swappiness=10

/etc/rc.local:
# Fix khugepaged issues (20150706)
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/defrag

# Another fix for khugepaged issues (20151210)
echo madvise > /sys/kernel/mm/transparent_hugepage/enabled

Revision history for this message
mm (mtl-0) wrote :

After upgrading to Xubuntu 15.10 on 2 home server boxes I am experiencing a similar problem: kswapd0 uses 99,9% of CPU. The only way to make it stop is to issue the swapoff command. When I try to issue swapon again it automatically produces 99,9% of CPU load again. (Even if the swap isn’t getting filled)

Any clue about that? It’s really annoying.

Revision history for this message
mm (mtl-0) wrote :

Update: Even with no swap (partition/file) attached the kswapd0 is generating 100% cpu load.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.