support a profile decorator for use in staging and development environments

+1, I want to schedule this.

Things that can help move it along:
 - help me land https://bugs.edge.launchpad.net/zope.app.publication/+bug/598816 and
  https://code.edge.launchpad.net/~lifeless/zope.publisher/start-request-event/+merge/28558 - they seem stalled even though Tres was happy with them.

 - We either need to move to a newer zope, or apply a small patch I have to our zope.app.publication (a backport of the two above patches) : I think its a matter of 'make a new egg in the sourcedeps, edit the versions list to use it', but I'm not sure of the QA process used on changes at that level.

 - I have a patch to profiling which then uses this event and makes it easier to reason about.

 - my uniquefileallocator patch passes all tests and will land once PQM opens again; that provides the basis for writing the file to disk Just-In-Time

 - bzrlib's lsprof wrapper is ready to provide mutex support for this - but we can just ignore that in the short term, things will be tolerable ;)

 - I, or someone else, needs to move profiling from per-thread to per-request using uniquefileallocator

 - And finally convert to using callgrind files.

All in all only a couple more hours work, but longer than that due to various latencies. I'll keep this bug apprised of my progress.

https://code.edge.launchpad.net/~lifeless/launchpad/lsprof/+merge/28583 is the profiling branch that probably needs a rereview as we just added cachegrind/lsprof support.

zope.publisher 3.12.4 has Robert's patch for that package. I merged his patch to zope.app.publication but did not make a release because it didn't seem like we were in a rush. In any case, upstream now has both passes.

Ok, the lsprof branch has landed. That branch enables us to turn profiling on and off for all requests at once.

I have started a new branch to add on-demand profiling support for the ++profile++ URL:

  lp:~mars/launchpad/lsprof-on-demand

My branch adds the config switch to enable on-demand profiling, and some code that runs the on-demand profile test. However, getting Zope to skip the ++profile++ part of the URL is a problem. We do this for ++oops++, but Gary says that it is a hack, and not something we should copy. So the next step is to ask Gary about a better way to do traversal skips like this.

We might also want ++profile++ to become a real view - one that not only generates the profile, but shows you a helpful page telling you where to get it (or that lets you download it directly).

I have to put this work on hold for now. If someone else wants to tackle the on-demand bit, please feel free to do so.

It is now available for development environments. I do not believe that it is safe for staging environments yet. It needs to wait on feature flag work to be completed, or farther along, at which point we will limit the functionality on staging to launchpad developers.

What is missing from feature flags?

Fixed in stable r11465 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/11465>.

> What is missing from feature flags?

Perhaps nothing. The docs I found seemed incomplete when I looked at this, but I may have been reading the wrong docs, or too fast.

AIUI, this effort will have two parts: defining a proper scope for Canonical Launchpad developers (or something similar), and then hooking up the feature flag to the ++profile++ code.

It may be a bit tricky because the profile may be started before the request's principal is authenticated. If that's the case--and I think it is--then the only downside is that arbitrary users will be able to start a profile, even though they will not actually receive the data. For staging, hopefully that's not too bad. For production, that seems unacceptable to me.

I'm fairly sure authentication happens before traversal starts in our
system, so its unprofiled. I agree it would be unacceptable - because
profiled locks other stuff out!