Launchpad itself

recipe builds can use too much memory

Bug #676657 reported by Aaron Bentley
28
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
High
Aaron Bentley
Launchpad itself 10.12

Bug Description

It is believed that excess memory use was a factor in https://wiki.canonical.com/IncidentReports/2010-10-28-LP-build-manager-not-dispatching and https://wiki.canonical.com/IncidentReports/2010-11-17-buildd-manager-disabling-builders

We believe that excessive memory use drove the builders so far into swap that they became unresponsive.

Related branches

lp://qastaging/~abentley/launchpad/builder-limits
Данило Шеган (community): Approve
Diff: 19 lines (+2/-0)
1 file modified
lib/canonical/buildd/buildrecipe (+2/-0)
Tim Penhey (thumper)
Changed in launchpad-code:
status: Triaged → In Progress
assignee: nobody → Aaron Bentley (abentley)
Revision history for this message
Launchpad QA Bot (lpqabot) wrote : Bug fixed by a commit

Fixed in stable r11943 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/11943>.

Changed in launchpad-code:
milestone: none → 10.12
tags: added: qa-needstesting
Changed in launchpad-code:
status: In Progress → Fix Committed
Aaron Bentley (abentley)
tags: added: qa-ok
removed: qa-needstesting
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Is "bzr: out of memory" during recipe build is a new feature introduced by this fix?

E.g. https://code.launchpad.net/~mingw-w64/+recipe/gcc4.4+trunk/+build/8623

Revision history for this message
Aaron Bentley (abentley) wrote : Re: [Bug 676657] Re: recipe builds can use too much memory

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/24/2010 01:27 AM, Dmitrijs Ledkovs wrote:
> Is "bzr: out of memory" during recipe build is a new feature introduced
> by this fix?

Yes, more or less. We are in the unfortunate position that bzr uses
excessive memory with large branches and the build farm (not just a
single builder) appears to become unstable when excessive memory is used.

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkzstDwACgkQ0F+nu1YWqI0uOQCdELFMtio4W8S1a6w3tZ0t2889
lTYAnRSYU8vHoXcDgvQGV3FU8R1VXB8P
=rpGW
-----END PGP SIGNATURE-----

Revision history for this message
Martin Pool (mbp) wrote :

On 24 November 2010 17:44, Aaron Bentley <email address hidden> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 11/24/2010 01:27 AM, Dmitrijs Ledkovs wrote:
>> Is "bzr: out of memory" during recipe build is a new feature introduced
>> by this fix?

We're working on reducing bzr memory usage through other bugs,
separate from this. One way to help is to attach a traceback showing
where bzr stopped when it ran out of memory.

@aaron maybe bzr should be run with -Derror in the builders?

--
Martin

Revision history for this message
Aaron Bentley (abentley) wrote :

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/24/2010 01:52 AM, Martin Pool wrote:
> @aaron maybe bzr should be run with -Derror in the builders?

We're currently running an old version of bzr, so the results wouldn't
necessarily be relevant. But also, these memory issues with large
branches are very easy to reproduce locally just by branching, so I
don't think bzr needs help finding them. My gut says we should stick to
the normal behaviour that people are used to, for now.

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkztHJoACgkQ0F+nu1YWqI2JhQCfa0pyvEupl5JFGQ1jkhZZOOSW
XvsAnidbQtg/3RWgnAWllhSP1z6E21WQ
=Eq52
-----END PGP SIGNATURE-----

Curtis Hovey (sinzui)
Changed in launchpad-code:
status: Fix Committed → Fix Released
Revision history for this message
era (era) wrote :

In the meantime see also bug #109114

Revision history for this message
Rohan Garg (rohangarg) wrote :

Not quite fixed yet : https://code.launchpad.net/~neon/+recipe/project-neon-calligra

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.