manage.py update_rules uses a lot of memory

Today we were alerted for a performance problem (nagios checks timing out, load average > 300) on one of the rec.ubuntu.com appservers, saci.

This is the host that runs "python manage.py update_rules" every morning at 02:00 UTC (as per RT#56699).

At its peak the process was consuming 6.4GB of memory, which is over 50% of the available physical memory in the system.

Here's the log from this run:

0.03757: Loading carts...
192.63163: Filtering carts...
192.69437: Growing candidate set...
192.69440: Found 0 candidates.
192.69443: Pruned down to 0 large sets.
192.69446: Generated 0 rules
192.83213: Loading carts...
577.71705: Filtering carts...
693.45866: Growing candidate set...
693.60202: Found 11628 candidates.
1128.97660: Pruned down to 4519 large sets.
1129.02459: Generated 3808 rules
1152.77195: Updating Rule table...

For comparison the log from the original QA run in October 2012 was:

0.03763: Loading carts...
12.15095: Filtering carts...
12.15115: Growing candidate set...
12.15118: Found 0 candidates.
12.15120: Pruned down to 0 large sets.
12.15123: Generated 0 rules
12.20738: Loading carts...
21.94895: Filtering carts...
26.98057: Growing candidate set...
27.02475: Found 11325 candidates.
111.22743: Pruned down to 4054 large sets.
111.24513: Generated 3026 rules
111.47725: Updating Rule table...