codehosting1 still had connections 37 minutes after shutdown signle given to it

Also, it took longer than 30 minutes when run against codehost1 today. I manually killed it after 37 minutes. I manually killed codehost2 after 15 minutes, as agreed with Gary.

We expect long bzr operations - some clones are more than 30 minutes.

So the 30 minutes window is deliberate, and -perhaps- will need to be larger.

Once bug 819604 is fixed we can lower the timeout back to a small value (e.g. 60 seconds) (long enough for the server to respond, but not so long that idle connections hang around.

We need the deploy scripts to deal with this, as we will for all the services that go through a quiescing mode, and have long uninterruptible operations happen on them.

So, next time this happens - next time it goes beyond 30 minutes, we need to get the connection information (using ps) of the connections from the one that hasn't shutdown, and identify what operations are being done over them so we can determine if its a slow operation, or someone doing extensive work over bzrlib.

If its the former, we need a longer wait period; if its the latter, we need to drop the connection at an idle point in between two commands - and we may need a little bzrlib cooperation to make this happen (e.g. signal the bzrlib services once the master is told to shutdown, so they can shutdown at the first opportunity they get).