no report/log of successful packages

There's not, at the moment, any obvious table of "successful import", in the way there is one for failures, so we could possibly infer this from other tables, or perhaps just add it as a new table. When the job succeeds (in `finish_job`) we delete the job and the failure record, and we don't obviously create anything else.

So I think we should either add a 'success' table, or refactor the current `failure` into being something like an `outcomes` record that could be either success or failure.

Also, in there, this code seems strange:

    def finish_job(self, package, job_id, success, output):
        c = self.conn.cursor()
        try:
            now = datetime.datetime.utcnow()
            if job_id > 0:
                row = c.execute(self.JOBS_TABLE_SELECT_JOB,
                                (job_id,)).fetchone()
                if row is not None:
                    c.execute(self.JOBS_TABLE_UPDATE,
                            (row[1], 0, row[3], row[4], row[5], now, row[0]))
            if success:
                row = c.execute(self.FAILURES_TABLE_FIND, (package,)).fetchone()
                if row is not None:
                    c.execute('delete from %s where package=?'
                            % self.JOBS_TABLE, (package,))
                c.execute(self.FAILURES_TABLE_DELETE, (package,))
                c.execute(self.OLD_FAILURES_TABLE_DELETE, (package,))
            else:
                self._set_failure(c, package, output, now)
            self.conn.commit()

I wonder what the logic is behind deleting the job if there was previously a failure record.

Just using:

   for f in progress_log-20110[789]* progress_log-201110* ; do echo -n "$f " ; ~/src/udd/trunk/analyze_log.py $f | grep ^Imported | wc -l ; done

on a set of file I rsync regurlarly from jubany gives the results below.

Roughly, 200 successful imports every day with some spikes up to 1000.

So 6000 imports/month.

progress_log-20110729 267
progress_log-20110730 208
progress_log-20110731 221
progress_log-20110801 176
progress_log-20110802 171
progress_log-20110803 161
progress_log-20110804 181
progress_log-20110805 175
progress_log-20110806 209
progress_log-20110807 190
progress_log-20110808 155
progress_log-20110809 1021
progress_log-20110810 310
progress_log-20110811 301
progress_log-20110812 251
progress_log-20110813 187
progress_log-20110814 157
progress_log-20110815 115
progress_log-20110816 184
progress_log-20110817 264
progress_log-20110818 237
progress_log-20110819 222
progress_log-20110820 286
progress_log-20110821 190
progress_log-20110822 115
progress_log-20110823 250
progress_log-20110824 664
progress_log-20110825 261
progress_log-20110826 328
progress_log-20110827 1056
progress_log-20110828 186
progress_log-20110829 148
progress_log-20110830 221
progress_log-20110831 199
progress_log-20110901 564
progress_log-20110902 337
progress_log-20110903 304
progress_log-20110904 198
progress_log-20110905 229
progress_log-20110906 177
progress_log-20110907 472
progress_log-20110908 402
progress_log-20110909 211
progress_log-20110910 274
progress_log-20110911 178
progress_log-20110912 258
progress_log-20110913 242
progress_log-20110914 160
progress_log-20110915 151
progress_log-20110916 139
progress_log-20110917 185
progress_log-20110918 104
progress_log-20110919 27
progress_log-20110920 86
progress_log-20110921 163
progress_log-20110922 390
progress_log-20110923 352
progress_log-20110924 260
progress_log-20110925 145
progress_log-20110926 144
progress_log-20110927 259
progress_log-20110928 288
progress_log-20110929 280
progress_log-20110930 277
progress_log-20111001 228
progress_log-20111002 150
progress_log-20111003 186
progress_log-20111004 155
progress_log-20111005 502
progress_log-20111006 164
progress_log-20111007 200
progress_log-20111008 978
progress_log-20111009 754
progress_log-20111010 273

So, the general kind of thing I'd like to see to start with is in the
web view, a list of successful imports mixed in with the failed
imports, so that we can see if it's failing entirely, making progress,
or just having some failures.

Later on it would be nice to hold onto this data so that we can graph
success vs failure, and perhaps find packages that used to work but
now failed, or vice versa.