Launchpad itself

log parser should not read entire file into memory

Bug #588288 reported by Michael Nelson
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
Medium
Michael Nelson
Launchpad itself 10.07

Bug Description

This is related to bug 139855. Currently when parsing a log file, the entire contents to be parsed is read into memory via readlines() rather than processing each line individually.

Additionally, it would be good to allow a config option to limit the number of lines processed at a time.

Related branches

lp://qastaging/~michael.nelson/launchpad/588288-log-parser-dont-read-entire-file
Graham Binns (community): Approve (code)
Diff: 167 lines (+67/-13)
4 files modified
configs/testrunner/launchpad-lazr.conf (+1/-0)
lib/canonical/config/schema-lazr.conf (+4/-0)
lib/lp/services/apachelogparser/base.py (+30/-11)
lib/lp/services/apachelogparser/tests/test_apachelogparser.py (+32/-2)
Revision history for this message
Ursula Junque (ursinha) wrote : Bug fixed by a commit

Fixed in stable r10942 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/10942>

Changed in launchpad-foundations:
milestone: 10.06 → 10.05
status: In Progress → Fix Committed
tags: added: qa-needstesting
Ursula Junque (ursinha)
Changed in launchpad-foundations:
milestone: 10.05 → 10.06
Revision history for this message
Julian Edwards (julian-edwards) wrote :

Got this during QA testing, dunno if it's related:

2010-06-07 12:23:58 ERROR Error (Unable to parse: 0] "GET /ubuntu-langpack/ubuntu/dists/edgy/main/binary-i386/Packages.gz HTTP/1.1" 200 20268 "-" "Debian APT-HTTP/1.3" with the ^(\S*) (\S*) (\S*) (\[[^\]]+\]) \"([^"\\]*(?:\\.[^"\\]*)*)\" (\S*) (\S*) \"([^"\\]*(?:\\.[^"\\]*)*)\" \"([^"\\]*(?:\\.[^"\\]*)*)\"$ regular expression) while parsing "0] "GET /ubuntu-langpack/ubuntu/dists/edgy/main/binary-i386/Packages.gz HTTP/1.1" 200 20268 "-" "Debian APT-HTTP/1.3"

Revision history for this message
Julian Edwards (julian-edwards) wrote :

And that seems to prevent it scanning the file further.

Revision history for this message
Julian Edwards (julian-edwards) wrote :

see bug 590766 for this issue

Revision history for this message
Michael Nelson (michael.nelson) wrote :

I've fixed the issue (liked to bug 590766) and qa'd on df, so we can mark this qa-ok once the branch lands. I'll land a separate config change to limit the number of lines processed initially for ppa parsing.

Michael Nelson (michael.nelson)
tags: added: qa-ok
removed: qa-needstesting
Michael Nelson (michael.nelson)
Changed in launchpad-foundations:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.