Android builds are failing due to android.git.linaro.org timeouts

IS tells me we hit peak loads of 30 at around 0300 and 0500 and of 50 at around 0800-0900 UTC times. I am trying to get someone to watch over potential culprits tomorrow.

Zach also started https://android-build.linaro.org/builds/~linaro-android/galaxynexus-jb-gcc47-aosp-blob/#build=64 now that we don't have as much load to ensure it is the slowness/load that's biting us.

Recording the jobs that I came across which are failing due to android.git.linaro.org time outs that I felt would be helpful for testing once the problem is fixed.

linaro-android_panda-jb-gcc47-tilt-tracking-blob #58 (not built)
linaro-android_panda-jb-gcc47-tilt-stable-blob
linaro-android_panda-ics-gcc47-omapzoom-stable-blob
linaro-android_panda-ics-gcc47-tilt-stable-blob
linaro-android_galaxynexus-jb-gcc47-aosp-blob
linaro-android_snowball-ics-gcc47-igloo-stable-blob
linaro-android_snowball-ics-gcc47-igloo-tracking-blob
linaro-android_panda-jb-gcc47-tilt-tracking-blob-tiny
linaro-android_snowball-ics-gcc46-igloo-tracking-blob
linaro-android_vexpress-jb-gcc47-armlt-tracking-open
liuyq0307_jb-panda-stable
linaro-android_panda-jb-gcctrunk-tilt-tracking-blob

Last build https://android-build.linaro.org/builds/~linaro-android/galaxynexus-jb-gcc47-aosp-blob/#build=69 failed with the same reason but the peak load of android.git.linaro.org was 9.3 at 5:21am right before build failed.

linaro-android project should consider moving their manifests to use http:// which is a much better scalable hosting solution we provide.

please try applying this manifest to all builds.

seems there is no real solution; we could do a manual update of the seed tarball to include more. or we can manually reschedule them in a way that they don't interfere

As mentioned in the email thread, we don't have "non-smart" HTTP for android.git.linaro.org yet. I've asked for it to be set-up (see https://rt.linaro.org/Ticket/Display.html?id=608). For "non-smart" HTTP git access on git.linaro.org, one should use http://git.linaro.org/git-ro/ as the base URL (https://rt.linaro.org/Ticket/Display.html?id=607 to update links on gitweb).

During today's investigation, following issues were found/adjustments made:

1. Some builds were found to use very high values of SYNC_JOBS. So, SYNC_JOBS is made non-overridable from build configs, such option was useful during initial setup of the service to use by build engineers, but no longer needed and causes issues in production run.

2. Default SYNC_JOBS values was lower to 5 from 8.

3. Build timeouts were increased to 3.5hrs.

So far, the results of these changes are optimistic (except that we hit lp:1055546, which broke initial run of builds with these new settings).

Even with 3.5hrs build timeout, we still have few builds fail due to it:

https://android-build.linaro.org/jenkins/view/All/job/berolinux_nexus7-jb-gcc47-aosp-blob/9/

That build spent 23min in seed download + 14min in git checkout, total 37min, which is not that *much* higher than baseline of 20min. So it seems, JB builds are what require more time.

Still, we have https://android-build.linaro.org/jenkins/view/All/job/linaro-android_origen-jb-gcc47-samsunglt-tracking-blob/18/ which build in whopping 2h12m.

So, dispersion of build times is too big for some reason.

Ok, comparing 2 builds

https://android-build.linaro.org/jenkins/view/All/job/linaro-android_origen-jb-gcc47-samsunglt-tracking-blob/18/
jenkins time 2h12m (132m)
real 7m36.007s
real 7m2.529s
real 6039.54 (101m)
rest 16.5m

https://android-build.linaro.org/jenkins/view/All/job/linaro-android_origen-jb-gcc47-samsunglt-tracking-blob/22/
jenkins time 3h17m (197m)
real 26m2.595s
real 10m43.538s
real 6740.36 (112m)
rest 48m

So, we have varying time on all 4 stages of build: seed download, git checkout, compilation, publishing. Of these, publishing provides the biggest absolute variation - 30mins!

Decision: bump timeout 15mins more to account for cases like https://android-build.linaro.org/jenkins/view/All/job/berolinux_nexus7-jb-gcc47-aosp-blob/9/

Considering fixed.

Status update: there're still cases of build timeouts. It appears that seed has gone stale/sideways (i.e. simply missed few big projects), so I'm currently regenerating it from scratch and testing.

From mail:

Further status update on this matter:

Even after adjustments described above, we had few occasions of builds
failing due to git errors on repo sync. So, my idea was that the seed
grew too stale and "sidetracked" compared to what latest manifests we're
working with contain. So, I generated a new seed over last week, made
some test builds and in made it default just before the weekend. As
some stats, the new seed is 12Gb, while old one was 16Gb. So, some
"cruft" was dropped, but new seed may be not ideal in case old releases
may need to be rebuilt, so old seed is available as
http://android-build.linaro.org/seed/uniseed1.tar.gz just in case.

Otherwise, I'm closing
https://bugs.launchpad.net/linaro-android-infrastructure/+bug/1052502