I've noticed some ha jobs timing out with no apparent errors, and looking at the graphite data this is likely because our ha deployments are just taking close to the 1 hour 20 minute timeout, so any slowdown in a job is going to cause it to hit the timeout.
Performance seems to have regressed pretty significantly over the past month or so. You can see this in the graph at https://66.187.229.172/S/F (give it time, it's processing a month of CI metrics). The undercloud install time (which did increase some due to additional services) is also included to demonstrate that it's not just an environmental problem.
Unfortunately, until a few days ago we didn't split things out by release, so the older metrics probably look somewhat better than they should because stable branch jobs are also included, and those are consistently faster to deploy than master (see below for details on this).
I do have some suspicion that part of the regression may be related to moving around the nova and keystone initialization in https://github.com/openstack/puppet-tripleo/commit/3b00ffc728b47e132b3ed8bc460f8697ddb32047 and https://github.com/openstack/puppet-tripleo/commit/bb63f514d22ea82d17947a5972b4da16e66b5a36. It's hard to say for sure from the numbers available though because of the branch issue in the older data.
Note that we don't collect metrics on failed jobs, so any jobs that exceed 4800 seconds for the deploy are not going to show up here, so the average might still be skewed a little low.
Further supporting the possibility that the step changes are involved, here are the deployment times for the longest four steps as of a week ago:
2017-02-07 06:50:21.000 | ControllerDeployment_Step5 961.0
2017-02-07 06:50:21.000 | ControllerDeployment_Step4 683.0
2017-02-07 06:50:21.000 | ControllerDeployment_Step3 509.0
2017-02-07 06:50:21.000 | ControllerDeployment_Step2 469.0
And a recent job:
2017-02-13 21:10:27.000 | ControllerDeployment_Step3 1109.0
2017-02-13 21:10:27.000 | ControllerDeployment_Step4 1070.0
2017-02-13 21:10:27.000 | ControllerDeployment_Step5 950.0
2017-02-13 21:10:27.000 | ControllerDeployment_Step2 535.0
Steps 3 and 4 have nearly doubled in length. Step 2 is a little longer, which suggests the testenv for this job may have been somewhat slower too, but even if you subtract a corresponding amount of time from each of the other three steps you still end up much longer than before.
So, while the immediate concern is ci jobs timing out, this is also a pretty significant regression in performance in a release that was already running noticeably slower than newton (which, in turn, was noticeably slower than mitaka - sensing a trend here?). For reference, here's a graph including the newton and mitaka-specific ha deploy times: https://66.187.229.172/S/G Mitaka jobs on average deploy over 25 minutes faster, and even Newton, which is more similar feature-wise, is around 18 minutes faster right now.
I'm not sure if we have time to do anything in Ocata so late in the cycle, but it's something we definitely need to look into.
I think oooq jobs experiencing the same problem, half of them finishes with timeout. I track it here: https:/