apt's lists/partial fills disk

This is the next day, and this time the offending file is: it.archive.ubuntu.com_ubuntu_dists_vivid_restricted_binary-amd64_Packages.bz2

The repeating data is still 0x10000 bytes, but the contents are different (another copy attached).

I changed the server to US and the problem doesn't seem to occur any more. Perhaps this is an issue with the Italy server, but nevertheless it is strange that apt would continuously write to the disk due to an error on the server.

Status changed to 'Confirmed' because the bug affects multiple users.

i was on italian server too with same issue. Just migrated to principal server.....let's see what happens

Please:
  1. Report to <https://www.debian.org/Bugs/>.
  2. Paste the new report URL here.
  3. Set this bug status back to "confirmed".

Thank you.

I've been unable to recreate this on a Vivid amd64 chroot after adding i386 as an architecture.

What IP address are you connecting to? There may be multiple servers for it.archive.ubuntu.com and only one may be a problem.

same here:

$ ll /var/lib/apt/lists/partial/it.archive.ubuntu.com_ubuntu_dists_vivid-updates_universe_binary-amd64_Packages.bz2
-rw-r--r-- 1 root root 95G apr 25 14:37 /var/lib/apt/lists/partial/it.archive.ubuntu.com_ubuntu_dists_vivid-updates_universe_binary-amd64_Packages.bz2

I have the same problems, using the Swedish update servers. I booted my computer and leaved it for lunch, and when I came back the CPU was running at 100% (http by apt). I kept it running for a while, and then I starting investigating this issue. I checked my /var/lib/apt/lists/partial/ and noticed that I had a file there over 100G big and growing all the time, so I rebooted, deleted the file and changed to the principal servers.

For people experiencing this bug it'd be useful to know specifically what mirror (domain name) they are connecting to and which IP address for that domain name.

I've seen this a few times. It appears to be an issue with recovering partially downloaded files, and my workaround was to rm /var/lib/apt/lists/partial/*

Looking through some notes from a few weeks ago, the backtrace was:

(gdb) backtrace
#0 0x00003fffb5b5d6ec in SHA1Transform (state=state@entry=0x3fffecb9b5e0,
    buffer=buffer@entry=0x3fffecb97043 "f\350'\315\060\372C343\345\030\301b\212'\371\376\271\202\237uiF\247\325\361\245\017o\277\344\031\245\275\370\361\256=C\033\277Y\b\242\021\257I\233\245\321FU\220\216{(\327J\nlB\021\364\360\373\311\066$\304\303\001\031\\\352Þ‚\024\271\\\321\f\023\255\342tK?b\f\361\306j\037\325\323\nH\036\061\026J\313A0d\306Bo\006\031\246q8\361W\247b@\243\373\322\316a\240y\226tC\213\271#'\"\301N\030pDb\203D\234gEz\237\031\341\062\020ÄŠÙªK\276\232\201!\332T\227\\\026\242\034\270\031ÛŒ\273\272\036\033\006\257q\255u\350\243*\332\373\256\256\205m1 \213\266*\\eN\033\272a"...)
    at /build/buildd/apt-1.0.9.7ubuntu4/apt-pkg/contrib/sha1.cc:108
#1 0x00003fffb5b5ec9c in SHA1Summation::Add (this=0x3fffecb9b598,
    data=0x3fffecb87c10 "n\202\341\067\231\267g\251N\316\324[0\361\070\230`sxtй\300b\200Y\347\252V\323 \310ƃ\273I'\236", len=65536)
    at /build/buildd/apt-1.0.9.7ubuntu4/apt-pkg/contrib/sha1.cc:262
#2 0x00003fffb5c7a6e0 in Add (Size=65536,
    Data=0x3fffecb87c10 "n\202\341\067\231\267g\251N\316\324[0\361\070\230`sxtй\300b\200Y\347\252V\323 \310ƃ\273I'\236", this=0x3fffecb9b530)
    at ../build/include/apt-pkg/hashes.h:167
#3 CircleBuf::Write (this=0x3fffecb83b90, Fd=4)
    at /build/buildd/apt-1.0.9.7ubuntu4/methods/http.cc:222
#4 0x00003fffb5c7b518 in HttpServerState::Go (this=0x3fffecb83960, File=
    0x3fffecb81980, ToFile=true)
    at /build/buildd/apt-1.0.9.7ubuntu4/methods/http.cc:652

We were stuck in the circular buffer code, the input and output pointers got out of whack such that we would write the same buffer continually.

I also experiencing this issue 15.04 64 bit even with picking one mirror. Until this bug has been fixed, I had to setup an hourly cron job, to remove everything in partial directory.

I'm randomly experiencing this problem using the German servers for 15.04. It does not affect a specific package, each time this happens some package keeps on growing until the disk is full.

I can make this issue come and go by disabling the gnome3.16 PPAs.

Specifically this occurs when a 404 for a particularly repo (i.e. vivid not available in a PPA). Then a second apt-get update causes the /partial dir to fill up until my root is full.

I took another look at this. The problem begins in ServerState::HeaderLine() Where we parse a Content-Range header:

Content-Range: bytes 587-600/601

So we set StartPos to 587. Then in HttpServerState::RunData() we do

In.Limit(Size - StartPos);

where Size is 14. We pass a negative number into Limit() which means MaxGet is set to something less than OutP, and we loop forever.

I think I see the issue. We parse both Content-Length: and Content-Range: headers.

When parsing the Content-Range: header, we set Size to the total size of the file.

If the Content-Length header comes last, we reset Size to the size of the data we are receiving, not the total size of the file.

The attached patch fixes it for me, although we do get the infamous Hash sum mismatch which clears after rerunning apt-get:

Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/vivid-security/universe/i18n/Translation-en Hash Sum mismatch

Another issue, but I'd love to fix that one day.

Michael, can you have a look at this one?

i already get rid of this issue. i tried many things including auto remove
apt.....now this /var/lib/apt/lists/partial file receives some file while i
make some update, but after update the new files are removed by itself. I
dont know which app is doing this activity.. Please make a query to an
expert.

On Wed, May 13, 2015 at 3:21 AM, Anton Blanchard <email address hidden> wrote:

> I took another look at this. The problem begins in
> ServerState::HeaderLine() Where we parse a Content-Range header:
>
> Content-Range: bytes 587-600/601
>
> So we set StartPos to 587. Then in HttpServerState::RunData() we do
>
> In.Limit(Size - StartPos);
>
> where Size is 14. We pass a negative number into Limit() which means
> MaxGet is set to something less than OutP, and we loop forever.
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1448684).
> https://bugs.launchpad.net/bugs/1445239
>
> Title:
> apt's lists/partial fills disk
>
> Status in APT:
> New
> Status in apt package in Ubuntu:
> Confirmed
> Status in apt source package in Vivid:
> Confirmed
>
> Bug description:
> Since I upgraded to vivid, apt fills the disk space at
> /var/lib/apt/lists/partial. This was triggered by the automatic update
> every day, and as a solution I killed the processes named `http`
> spawned by apt, and removed the offending file from lists/partial
> manually. Until now, `apt-get update` and `apt-get upgrade` had been
> working. However, today, when I tried `apt-get update`, it got stuck
> in some file, continuously increasing its size when at 100%.
>
> I tried inspecting the file to see if there is a repetition of data,
> and there seems to be, as I have suspected.
>
> First of all, the file is it.archive.ubuntu
> .com_ubuntu_dists_vivid_main_binary-i386_Packages.bz2 this time, but
> I'm not sure if the previous time it was also this file.
>
> I used `tail` and `hexdump` to figure out whether a certain pattern
> shows up in the file. Here are some examples:
>
> $ tail -c 200000
> it.archive.ubuntu.com_ubuntu_dists_vivid_main_binary-i386_Packages.bz2 |
> hexdump | grep '\<bbdf\>'
> 0000c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a
> 0010c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a
> 0020c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a
> 0030c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a
>
> $ tail -c 200000
> it.archive.ubuntu.com_ubuntu_dists_vivid_main_binary-i386_Packages.bz2 |
> hexdump | grep '\<9b95\>'
> 0000c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a
> 0010c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a
> 0020c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a
> 0030c80 8d9e 63ad bbdf 40a7 3f3e 2b4a 9b95 aa5a
>
> $ tail -c 200000
> it.archive.ubuntu.com_ubuntu_dists_vivid_main_binary-i386_Packages.bz2 |
> hexdump | grep '\<1234\>'
> 0002630 fd50 b08d 7082 1234 85b3 a61e 2921 e0cd
> 0012630 fd50 b08d 7082 1234 85b3 a61e 2921 e0cd
> 0022630 fd50 b08d 7082 1234 85b3 a61e 2921 e0cd
>
> $ tail -c 200000
> it.archive.ubuntu.com_ubuntu_dists_vivid_main_binary-i386_Packages.bz2 |
> hexdump | grep '\<76a8\>'
> 000fa90 747d 76a8 078a 67b6 60d9 83f5 5ff3 787c
> 001fa90 747d 76a8 078a 6...

The attachment "Don't parse Content-Length if Content-Range has been parsed" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

Issue occurred again for me today when I interrupted apt-get update and then reran a few minutes later.

This is fixed in git now and I will prepare a wily and vivid-proposed upload.

Thanks a lot to Anton for the patch and the excellent analysis of the issue!

This bug was fixed in the package apt - 1.0.9.10ubuntu1

---------------
apt (1.0.9.10ubuntu1) wily; urgency=low

  * merged from debian/sid

apt (1.0.9.10) unstable; urgency=medium

  [ Michael Vogt ]
  * Fix crash in pkgDPkgPM::WriteApportReport(() (LP: #1436626)
  * Move sysconf(_SC_OPEN_MAX); out of the for() loop to avoid unneeded
    syscalls
  * Fix endless loop in apt-get update that can cause disk fillup
    (LP: #1445239)

  [ Helmut Grohne ]
  * parse arch-qualified Provides correctly (Closes: 777071)

 -- Michael Vogt <email address hidden> Fri, 22 May 2015 18:08:10 +0200

I have a fully updated 15.04 machine and I'm still having issues with apt cache eating up my disk. I get into a situation where `http` is my most active process and my apt partial cache is growing quickly. See the logs below, which show the top processes, the large file sizes in the partial cache, and the file size growth in the partial cache. When the disk fills completely, the system is then no longer able to boot into unity and I am required to manually delete files with a command line login. ..

top:
 2581 root 20 0 40260 5012 4660 R 97.0 0.0 11:04.46 http
 2582 root 20 0 40260 4992 4636 R 97.0 0.0 10:40.69 http

:~$ ls /var/lib/apt/lists/partial/ -lS|head -n 5
total 93832500
-rw-r--r-- 1 root root 49198669906 May 29 10:41 security.ubuntu.com_ubuntu_dists_vivid-security_universe_binary-i386_Packages.bz2
-rw-r--r-- 1 root root 46855568434 May 29 10:41 us.archive.ubuntu.com_ubuntu_dists_vivid-updates_main_binary-i386_Packages.bz2
-rw-r--r-- 1 root root 7017495 Apr 24 14:45 us.archive.ubuntu.com_ubuntu_dists_vivid_universe_source_Sources.bz2
-rw-r--r-- 1 root root 6486447 Apr 24 14:40 us.archive.ubuntu.com_ubuntu_dists_vivid_universe_binary-i386_Packages.bz2
:~$ ls /var/lib/apt/lists/partial/ -lS|head -n 5
total 94526900
-rw-r--r-- 1 root root 49566396416 May 29 10:41 security.ubuntu.com_ubuntu_dists_vivid-security_universe_binary-i386_Packages.bz2
-rw-r--r-- 1 root root 47198911538 May 29 10:41 us.archive.ubuntu.com_ubuntu_dists_vivid-updates_main_binary-i386_Packages.bz2
-rw-r--r-- 1 root root 7017495 Apr 24 14:45 us.archive.ubuntu.com_ubuntu_dists_vivid_universe_source_Sources.bz2
-rw-r--r-- 1 root root 6486447 Apr 24 14:40 us.archive.ubuntu.com_ubuntu_dists_vivid_universe_binary-i386_Packages.bz2

This has only been fixed in the development release of Ubuntu, Wily Werewolf, and we are awaiting a fix from the developer for Vivid (Ubuntu 15.04).

This is uploaded as a SRU and in https://launchpad.net/ubuntu/vivid/+queue?queue_state=1&queue_text=

Hello Shahbaz, or anyone else affected,

Accepted apt into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.0.9.7ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hi Brian,

Sorry for my lack of responsiveness, but I'd essentially been moving countries and I'm no longer in Italy to begin with. On another computer where I'm typing this response, this issue doesn't happen to begin with (Canada servers). Hopefully others would be able to test this out.

Brian,

I've run across the same problem and have installed your package from -proposed.

I will let you know after 2-3d if the problem resurfaces. If you don't hear from me by the end of the week, consider the problem fixed for me.

Brian,

I was using sg.archive.u.c

It's still affecting me. I'm using the CICA server (in Spain)

@Xavi could you test the proposed package as suggested in comment #29?

I had been observing a disk full a few times due to this issue, including this morning, after which I found this report. I just installed apt 1.0.9.7ubuntu4.1 from vivid-proposed using "apt-get install apt/vivid-proposed".

I now have:
$ apt-cache policy apt
apt:
  Installed: 1.0.9.7ubuntu4.1
  Candidate: 1.0.9.7ubuntu4.1
  Version table:
 *** 1.0.9.7ubuntu4.1 0
        400 http://us.archive.ubuntu.com/ubuntu/ vivid-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1.0.9.7ubuntu4 0
        500 http://us.archive.ubuntu.com/ubuntu/ vivid/main amd64 Packages

I ran "apt-get update" manually several times and did not see the disk-filling-up problem again. I will continue to keep my eye on it.

Marking verification-done based on comments 31 and 35.

This bug was fixed in the package apt - 1.0.9.7ubuntu4.1

---------------
apt (1.0.9.7ubuntu4.1) vivid-proposed; urgency=low

  * test/integration/test-apt-download-progress:
    - disable as its too unreliable
  * Fix endless loop in apt-get update that can cause disk fillup
    LP: #1445239

 -- Michael Vogt <email address hidden> Wed, 10 Jun 2015 16:09:44 +0200

The verification of the Stable Release Update for apt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.