popularity-contest will stat() every file installed file or ~80k files on my system which grinds things to a halt

The script is simply too resource intensive. If there is a way to make it run over a longer period of time, and require less I/O resources at any given point in time, this would be much preferable to the current situation. I am considering uninstalling it (for now), since providing statistics isn't worth the performance penalty it brings, in my view.

As of version 1.52ubuntu2 popularity-contest started as a regular process. I suppose changing IO and CPU scheduling priorities Idle should place less load to the system, but I cannot notice the change in my environment. Both cases works on my P4-3GHz 3GiB or RAM with two 7200 RPM disks equally well.

Anyway, I agree that system with less RAM or slower disk may suffer from popcon activity. But we have to test things to decide if nice/ionice is enough or we should limit IO with some other means. Please try to do some job while popularity contest is running to check how much it slows down your system.

The following test should simulate current situation:

sudo sh -c "echo 3 > /proc/sys/vm/drop_caches; sync"
time popularity-contest

The next test should run popularity-contest as a low-priority background task

sudo sh -c "echo 3 > /proc/sys/vm/drop_caches; sync"
time ionice -c 3 nice popularity-contest

In case, changing priority is enough, patch it really trivial:

=== modified file 'debian/cron.daily'
--- debian/cron.daily 2011-05-23 15:38:47 +0000
+++ debian/cron.daily 2011-06-10 15:35:17 +0000
@@ -50,7 +50,7 @@

 run_popcon()
 {
- su -s /bin/sh -c "/usr/sbin/popularity-contest" nobody
+ su -s /bin/sh -c "/usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/popularity-contest" nobody
 }

 do_sendmail()

The problem won't go away if you ionice it. It'll just run with ionice priority, but it'll still being doing an ostensibly completely redundant stat() of files on the filesystem, instead of just reading from `dpkg -l`.

So even if that could alleviate the case where it's racing other system processes for I/O, it won't solve the case of your disk being spun up for a long time (since it would otherwise be idle) due to popularity contest.

Why does it need to stat files that packages install in the first place? Can't it just trust that dpkg(1) reports installed packages correctly?

No it can't. The goal of popularity-contest not only collect what packages are installed, but also determine what packages were recently used. In order to do so, It checks what files are currently open and scans last access time for files belonging to installed packages.

Probably, you may ask an option to disable collection of usage statistics, but it probably makes popcon results less usable. Such request should be forwarded to upstream developers in debain BTS, I think.

Status changed to 'Confirmed' because the bug affects multiple users.

I've now opted out of popularity-contest. It's not worth it to me to have my system grind to a halt every week, and I generally just get pissed off and kill the process anyway. Microsoft collects application usage information in Windows as well, and while the information isn't always 100% accurate, Programs and Features loads in only a few seconds. C'mon, Ubuntu... you're not allowed to lose to Windows! :P