internal service calls cannot handle interrupted requests

I believe branches have a field for showing connection errors already;
there's no particular need to log this if the data is put into that
field.

This appears to be a problem communicating with our own XMLRPC server, not the remote branch. It seems OOPS-worthy, because we cannot complete a pull correctly if we cannot write the results to LP.

This is a common problem for tellurium, which is a staging server. It is rare for crowberry, the production version; the most recent instance was OOPS-1948SMP25, and OOPS-1908SMP33 before that.

According to the log, this is happening when trying to record the outcome of a mirror operation.

I agree that a problem connecting to the internal API should be an OOPS. Its a bit worrying that some of the OOPS are coming from [qa]staging with a production prefix; perhaps thats historic and fixed, but lets check for it anyhow.

Of the actual production examples, OOPS-1948SMP25 appears to be due to the banana meltdown: https://wiki.canonical.com/IncidentReports/2011-05-02-LP-Frontend-server-banana-died

There are two sorts of failure modes here - cannot talk to haproxy, and server-melted down mid-request. For the former IS are going to implement LVS around the middle of June. That leaves dealing with requests that were in-progress at the time the backend blows up; for those we may want to do a retry mechanism, but meltdowns are so rare that the number of in-flight requests affected will be tiny - and its appropriate that we find out that happened. Accordingly I think this is as fixed as it sensibly can be short of re-requesting those requests. I'm going to leave it open, narrowed in focus to that situation.