cannot allocate memory errors

Bug #1671120 reported by Junien Fridrick
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
New
High
Unassigned

Bug Description

Hi,

We have an instance where a controller has free memory, but juju keeps logging this kind of errors :

2017-03-08 14:13:37 ERROR juju.worker.dependency engine.go:547 "disk-manager" manifold worker returned unexpected error: cannot list block devices: lsblk failed: fork/exec /bin/lsblk: cannot allocate memory

$ free -m
              total used free shared buff/cache available
Mem: 7982 6934 218 83 829 669
Swap: 0 0 0

I think that's because juju will use exec.Command() to run external commands, and this uses a simple clone() as show below from an strace -e process :

[pid 13429] clone(child_stack=0, flags=SIGCHLD) = -1 ENOMEM (Cannot allocate memory)

Using clone() this way means that the address space of the calling process (jujud) will be copied, and jujud is using a hefty amount of memory :

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 13429 360 28.2 4031016 2311676 ? Sl 09:48 974:17 /var/lib/juju/tools/machine-1/jujud machine --data-dir /var/lib/juju --machine-id 1 --debug

This was already reported and discussed in https://bugs.launchpad.net/juju-core/+bug/1382556, but for juju 1.25, and apparently exclusively for "juju run", which is different from what I'm reporting, since this is 2.1.1 and not caused by a "juju run".

With the new juju2 way of having controllers host multiple models, increasing the resources used by jujud, this bug is, presumably, more likely to appear.

Thank you

Tags: canonical-is
Junien Fridrick (axino)
tags: added: canonical-is
Changed in juju:
importance: Undecided → High
Revision history for this message
Paul Collins (pjdc) wrote :

This sounds like LP:1516676, which also refers to LP:1382556.

Revision history for this message
Joel Sing (jsing) wrote :

It is worth noting that this may possibly be resolved by Go 1.9:

https://github.com/golang/go/issues/5838
https://go-review.googlesource.com/c/37439/

Revision history for this message
Anastasia (anastasia-macmood) wrote :

Marking as duplicate as per comment # 2.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.