MemoryError in groupcompress to_bytes during commit of large tree

Bug #408526 reported by Jan Danielsson
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Bazaar
Confirmed
Medium
Unassigned

Bug Description

$ export CVS_RSH=ssh
$ export <email address hidden>:/cvsroot
$ cd ~/bazaar
$ cvs export -r netbsd-5-0-RELEASE -d netbsd-5.0 src
(Now go drink 10 cups of tea)
$ cd netbsd-5.0
$ bzr init --2a
Created a standalone tree (format: 2a)
$ bzr add
(Wait. There was one ignored file on my system)
$ bzr ignored
usr.bin/nvi/docs/USD.doc/vi.ref/ref.so *.so
$ bzr add usr.bin/nvi/docs/USD.doc/vi.ref/ref.so
adding usr.bin/nvi/docs/USD.doc/vi.ref/ref.so
$ ulimit -d
262144
$ bzr commit -m "Initial commit."
[---]
added x11/tools/xkbcomp/xkbfile-xkbmisc.c
aborting commit write group: MemoryError("Can't allocate memory to compress data",)
bzr: ERROR: exceptions.MemoryError: Can't allocate memory to compress data

Traceback (most recent call last):
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/commands.py", line 729, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/commands.py", line 924, in run_bzr
    ret = run(*run_argv)
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/commands.py", line 560, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/builtins.py", line 2937, in run
    exclude=safe_relpath_files(tree, exclude))
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/workingtree_4.py", line 226, in commit
    result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/mutabletree.py", line 228, in commit
    *args, **kwargs)
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/commit.py", line 363, in commit
    self._update_builder_with_changes()
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/commit.py", line 644, in _update_builder_with_changes
    self.work_tree, self.basis_revid, iter_changes):
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/repository.py", line 762, in record_iter_changes
    file_id, lines, heads, nostore_sha)
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/repository.py", line 831, in _add_text_to_weave
    check_content=False)[0:2]
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/groupcompress.py", line 1008, in add_lines
    nostore_sha=nostore_sha))[0]
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/groupcompress.py", line 1508, in _insert_record_stream
    flush()
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/groupcompress.py", line 1404, in flush
    bytes = self._compressor.flush().to_bytes()
  File "/usr/pkg/lib/python2.5/site-packages/bzrlib/groupcompress.py", line 290, in to_bytes
    self._z_content = compress(self._content)
MemoryError: Can't allocate memory to compress data

bzr 1.16.1 on python 2.5.4 (netbsd4)
arguments: ['/usr/pkg/bin/bzr', 'commit', '-m', 'Initial commit.']
encoding: '646', fsenc: '646', lang: None
plugins:
  bzrtools /usr/pkg/lib/python2.5/site-packages/bzrlib/plugins/bzrtools [1.16]
  launchpad /usr/pkg/lib/python2.5/site-packages/bzrlib/plugins/launchpad [1.16.1]
  netrc_credential_store /usr/pkg/lib/python2.5/site-packages/bzrlib/plugins/netrc_credential_store [1.16.1]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.
$ rm -Rf .bzr/
$ bzr init --2a
Created a standalone tree (format: 2a)
$ bzr add
(Wait. There was one ignored file on my system)
$ bzr ignored
usr.bin/nvi/docs/USD.doc/vi.ref/ref.so *.so
$ bzr add usr.bin/nvi/docs/USD.doc/vi.ref/ref.so
adding usr.bin/nvi/docs/USD.doc/vi.ref/ref.so
$ ulimit -d 524288
$ ulimit -d
524288
$ bzr commit -m "Initial commit."
[---]
Committed revision 1.
(Runs ok!)

So, bzr uses a very large amount of memory during the commit phase. I ran top during the commit, and the python process appears to be using something just over 256MB, so 512 is a little bit excessive. But, IMHO, 256MB should have been more than enough.

Running NetBSD/amd64 4.0 with:
$ bzr --version
Bazaar (bzr) 1.16.1
  Python interpreter: /usr/pkg/bin/python2.5 2.5.4
  Python standard library: /usr/pkg/lib/python2.5
  bzrlib: /usr/pkg/lib/python2.5/site-packages/bzrlib
  Bazaar configuration: /home/jan/.bazaar
  Bazaar log file: /home/jan/.bzr.log

Revision history for this message
Andrew Bennetts (spiv) wrote :

I'd be curious to know if this is any better for you in 2.1? We did land some memory use improvements that may have helped this case.

Changed in bzr:
importance: Undecided → Medium
status: New → Confirmed
Revision history for this message
Andrew Bennetts (spiv) wrote :

p.s. Thanks for pointing out that we'd wrongly marked this as a dupe of bug 109114.

Revision history for this message
Martin Pool (mbp) wrote :

So the complaint here is that the tree has many files, but no large individual files?

tags: added: 2a
summary: - bzr commit on large projects require large amounts of memory
+ MemoryError in groupcompress to_bytes during commit of large tree
Revision history for this message
Jan Danielsson (jan.m.danielsson) wrote :

Replying to Martin Pool,

Yes, that's it exactly. However, as one can see from the log I posted, I fixed it by bumping up the memory limit for the process. But I wrote another bug report, which I think also was marked as a dupe. After I managed to do the initial commit, I tried branching. That failed with a memory error too. But in that case I didn't get it to finish, even with the memory limit bumped up. So it would seem that it consumes even more memory when branching.

Replying to Andrew Bennetts,

I will, and I'll get back with the results.

Jelmer Vernooij (jelmer)
tags: added: check-for-breezy
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.