Launchpad itself

buildd-manager should be more rsync-friendly with log syncing

Bug #589604 reported by Tom Haddon
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
Medium
Henning Eggers
Launchpad itself 10.08

Bug Description

The buildd-manager use twisted's default logging, which rotates every 977K and uses a numbered extentsion. This means all files change contents on every sync, which is very rsync unfriendly. Ideally we'd be able to use some external method of rotating logs so that we could use daily rotation with dateext.

Related branches

lp://qastaging/~henninge/twisted/ticket-3363
lp://qastaging/~henninge/launchpad/buildd-manager-logging
Māris Fogels (community): Approve (code)
Diff: 325 lines (+163/-16)
5 files modified
daemons/buildd-manager.tac (+7/-0)
lib/canonical/launchpad/daemons/tachandler.py (+27/-12)
lib/lp/buildmaster/tests/harness.py (+13/-0)
lib/lp/buildmaster/tests/test_manager.py (+81/-2)
lib/lp/services/twistedsupport/loggingsupport.py (+35/-2)
Tom Haddon (mthaddon)
tags: added: canonincal-losas-lp
Julian Edwards (julian-edwards)
Changed in soyuz:
status: New → Triaged
importance: Undecided → Medium
tags: added: buildd-manager
Revision history for this message
Julian Edwards (julian-edwards) wrote :

This is how the Landscape team does it:

https://pastebin.canonical.com/33150/

Revision history for this message
Robert Collins (lifeless) wrote :

This is a subtle buh - and that fragment is interesting - its basically not safe (no callFromThread) and otherwise the same as the built in twistd setup. (Thanks spiv for analysing this)

twistd -l foo.log
will behave identically except that it will use SIGUSR1 not SIGUSR2 to rotate.

Both twistd and the fragment you pasted use LogFile.fromFullPath, so you get the default rotation behaviour.

The constructor parameter rotateLength when set to None will disable rotation.

See pydoc twisted.python.logfile.LogFile for details.

I think that a small patch to twistd would let us say --rotate=None and disable rotation altogether, which is what we want - that lets us manually rotate at our schedule rather than being forced to stay under 1 MB.

I'm going to request that in the upstream now.

Revision history for this message
Robert Collins (lifeless) wrote :

Since I can't find a dialog, I'll just link - http://twistedmatrix.com/trac/ticket/4557 - filed about this.

tags: added: canonical-losas-lp
removed: canonincal-losas-lp
Revision history for this message
Henning Eggers (henninge) wrote :

So, I have a branch here that disables twistd's own logging (every 1000000 bytes) and makes it accept SIGUSR1 to re-open its log file. That should satisfy the needs for proper logging, shouldn't it?

Changed in soyuz:
assignee: nobody → Henning Eggers (henninge)
status: Triaged → In Progress
milestone: none → 10.08
Revision history for this message
Henning Eggers (henninge) wrote :

Oh, my patch for twisted was rejected, btw.

Revision history for this message
Launchpad QA Bot (lpqabot) wrote : Bug fixed by a commit

Fixed in stable r11244 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/11244>.

tags: added: qa-needstesting
Changed in soyuz:
status: In Progress → Fix Committed
Julian Edwards (julian-edwards)
tags: added: qa-ok
removed: qa-needstesting
Julian Edwards (julian-edwards)
Changed in soyuz:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.