memory threshold is too high

Lets try with 100MB and get some mileage on proposed

I’ve changed the threshold to 100MB and tested on my M10.
This is too low, the system OOM killer kicks in before the browser gets a chance to unload background tabs:

May 2 16:25:39 ubuntu-phablet kernel: [ 3173.400686] (0)[54:kswapd0]lowmemorykiller: Killing 'webbrowser-app' (7341), adj 1, score_adj 110,
May 2 16:25:39 ubuntu-phablet kernel: [ 3173.400686] to free 550600kB on behalf of 'kswapd0' (54) because
May 2 16:25:39 ubuntu-phablet kernel: [ 3173.400686] cache 65196kB is below limit 65536kB for oom_score_adj 12
May 2 16:25:39 ubuntu-phablet kernel: [ 3173.400686] Free memory is 4344kB above reserved

Note that in my tests I had several apps open (both native and webapps, 4 of them in total) plus the browser with a dozen tabs including a live hangout, a google doc being edited, google maps, google plus, facebook and a 1080p movie trailer playing in youtube.

The last "free memory" value reported by the browser before it was killed was 237980KB, i.e. 232MB.
Note that this value is not just the free RAM, it’s the sum of the following entries from /proc/meminfo:
  MemFree
  Buffers
  Cached

(https://bazaar.launchpad.net/~phablet-team/webbrowser-app/trunk/view/head:/src/app/meminfo.cpp#L83)

I raised the threshold to 300MB, but this time the system OOM kicked in while the browser was reporting 345512KB free memory (337MB), which roughly translates to 17% free memory on that device.

Hi

I'm not the best person to comment on this issue except for the fact that this but also affects me, but shouldn't the kernel start swapping memory?

The system OOM killer typically kicks in when it’s running out of memory (including swap).

First of all: I agree with the original bug report, relying on % of total system memory is likely to over- or under-estimate the actual memory situation. A short-term solution would be to select a fixed threshold in accordance with the OOM setup configured on our kernels, thus avoiding the issues presented in #2. Mid-term, I would propose two adjustments:

(1.) The system should let applications know if a low-on-memory situation is encountered, enabling the app to free up memory *before* the kernel OOM handler kicks in.
(2.) For the browser specifically: We should fine-tune how memory is freed up/tabs are torn down. "Least recently used" is an obvious choice, but we should also factor in if a tab is playing audio/video or actively accessing the camera/microphone and adjust likelihood of being killed accordingly.

(1.) and (2.) together should solve the issue in a way that handles memory pressure situations as gracefully as possible.

An approach similar to what we currently do in the browser has been proposed for WebKit, as an alternative on configurations where memory cgroups are not available, see https://bugs.webkit.org/show_bug.cgi?id=155255.

However only MemFree is considered, and there are two (arbitrary) thresholds:
 - 300MB: non critical memory pressure
 - 100MB: critical memory pressure

I am going to experiment with this approach and will report back here.

I did some extensive stress-testing on an M10, an MX4 and a E4.5, all running the latest rc-proposed image. I launched a number of applications (some native, some webapps) in the background, and then opened a number of tabs in the browser app (with the mechanism to unload tabs temporarily disabled).

I observed that even when under quite some pressure (with more than 10 tabs open and active, including memory-hungry apps like google plus, hangouts, google docs, google maps, twitter, facebook), the free memory (MemFree+Buffers+Cached) rarely seems to go below 200MB (more often on the E4.5 than on the other two devices, which is expected because the E4.5 has 1GB RAM whereas the other two have 2GB RAM).

I had to push the limits quite a bit to get the system OOM killer to kick in, and it never killed the browser app, always a background app.

Those numbers and observations seem to indicate that an arbitrary threshold of 200MB should work reasonably well, instead of a percentage of the available memory. I’ll experiment further with that value.

And to follow up on previous comments: with the current approach we’ll never be able to fully prevent the OOM killer from kicking in (nor would it be desirable), what we need is an acceptable threshold that gives good results (prevents the browser from being killed while in the foreground most of the time) while being conservative enough to not close too many background tabs.

Testing the linked branch on M10, and I got several occurrences of the OOM killer killing the browser app while it was the foreground app, before the browser got a chance to unload any background tab.

That happens more often when there are fewer apps in the background (because the OOM killer has fewer apps to select from).

That’s with a threshold of 200MB. I’ve bumped that value by increments of 50MB until it reached 400MB, where it seems to be stable enough (i.e. it anticipates correctly the OOM killer by killing background tabs).

Obviously a hardcoded value of 400MB isn’t realistic on the E4.5 that only has 1GB RAM, so I’m thinking of going back to a percentage. 400MB on the M10 is 20% (keep in mind this is not the amount of free RAM at any given time, it’s the sum of MemFree+Buffers+Cached).

I’ll update the branch and test it on all my devices.

400MB works much better as a threshold on my M10 indeed. However I’m seeing something fishy: whenever the browser unloads a background tab to free up some memory, a small amount of memory is indeed reclaimed (usually less than 100MB), but the browser process continues to use up a lot of memory. When browsing pictures on 500px.com, eventually all other tabs get unloaded, and the only renderer process left active is consuming around 10% memory, but the browser process is consuming up to 30% memory. After a while left idle, it goes down to < 10%, which indicates that a lot of memory has been reclaimed at once (late garbage collection maybe?), which alleviates significantly the memory pressure.

My wild guess is that the memory used by the unloaded tabs is not garbage-collected by the QML engine right away when the tabs are unloaded. This seems to be confirmed by adding an explicit call to gc() in the unload() function of BrowserTab: unloading a tab now seems to reclaim a much larger chunk of memory, and the memory used by the browser process remains under control.

And 200MB seems to work well on the E4.5, so 20% seems to be a reasonable threshold overall.

However I’m still seeing an issue with the memory pressure mechanism: on a tablet (wide layout) the least recently viewed tabs are unloaded first as expected, but on a phone (narrow layout), the most recently viewed tabs are unloaded first, which is the opposite of what should happen. Looking into that.

That was an easy one: an incorrect type for the timestamp meant that the value was capped at MAX_INT and never changed, so the first tab in the list was always selected for unloading. This is now fixed in the linked branch.

What if all unloaded tabs are saved as a binary file on the hard disk and loaded from the hard disk if the user switches back to that tab. The advantage would be that there is no need to reload the tab from the net.

Is that a way to solve it?

@JOnathanJOnes: this is already the case: unloaded tabs are being serialized on disk. There are several other factors at play that might trigger some network activity when reloading a tab though, depending on the cache policy requested by the server, and more generally the dynamic nature of the page. Static content will probably not issue any requests to the network when reloaded, whereas a highly dynamic web app will do.

This bug was fixed in the package webbrowser-app - 0.23+16.04.20160509.3-0ubuntu1

---------------
webbrowser-app (0.23+16.04.20160509.3-0ubuntu1) xenial; urgency=medium

  [ CI Train Bot ]
  * Resync trunk.

  [ Olivier Tilloy ]
  * Fine-tune the custom memory-pressure handler, from data gathered on
    several devices. (LP: #1576639)
  * Update translation template.

 -- Olivier Tilloy <email address hidden> Mon, 09 May 2016 17:56:03 +0000

Hello Victor, or anyone else affected,

Accepted webbrowser-app into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/webbrowser-app/0.23+16.04.20161028-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Tested and verified in an up-to-date xenial amd64 VM, with webbrowser-app 0.23+16.04.20161028-0ubuntu1.

The VM has 784MB of memory, and after opening a number of memory-hungry tabs, I can see that the oldest ones are being unloaded to free up some memory, instead of the system taking down the entire application.

Hello Victor, or anyone else affected,

Accepted webbrowser-app into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/webbrowser-app/0.23+16.04.20161028-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Tested and verified in an up-to-date xenial amd64 VM, with webbrowser-app 0.23+16.04.20161028-0ubuntu2.

The VM has 768MB of memory, and after opening a number of memory-hungry tabs, I can see that the oldest ones are being unloaded to free up some memory, instead of the system taking down the entire application.

This bug was fixed in the package webbrowser-app - 0.23+16.04.20161028-0ubuntu2

---------------
webbrowser-app (0.23+16.04.20161028-0ubuntu2) xenial; urgency=medium

  * SRU for selected bug fixes:
    - LP: #1565055: support for google hangouts
    - LP: #1573017: SAML detection logic broken in webapp container
    - LP: #1572673: invalid variable access error in webapp container
    - LP: #1466427: dynamic Ubuntu version in default UA string
    - LP: #1576639: fine-tune the custom memory-pressure handler
    - LP: #1581025: secure connection icon not showing
    - LP: #1580290: only load the current tab when the session is restored
    - LP: #1577806: browser and container open under the same instance
    - LP: #1565063: dynamic Chromium version in default UA string

 -- Olivier Tilloy <email address hidden> Fri, 28 Oct 2016 09:17:33 +0000

The verification of the Stable Release Update for webbrowser-app has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.