Connectivity issues to pypi mirrors cause easy_install to fail.

Related fix proposed to branch: master
Review: https://review.openstack.org/98112

Reviewed: https://review.openstack.org/98112
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=192ebc86d1e6542f220c1ba4961b0a5c98e2e4b2
Submitter: Jenkins
Branch: master

commit 192ebc86d1e6542f220c1ba4961b0a5c98e2e4b2
Author: Sean Dague <email address hidden>
Date: Thu Jun 5 09:58:58 2014 -0400

    tracking bug for connection issues between HP and RAX

    Change-Id: I66ea650896e57f852f15390509cee7902c45ee2c
    Related-Bug: #1326813

We would possibly benefit from per-provider/region/az mirrors.

Related fix proposed to branch: master
Review: https://review.openstack.org/169156

I've update the bug title and description because hpcloud doesn't exist anymore and this bug is being used to track easy_install connection failures.

As noted in https://bugs.launchpad.net/openstack-gate/+bug/1449136 pip will retry to download packages on these errors (default is 5 attempts) this seems to make installation via pip much more reliable. We likely want to avoid using easy_install as much as possible so when we find these failures it is probably worthwhile to see if we can get pip to install the package rather than easy_install.

Related fix proposed to branch: master
Review: https://review.openstack.org/490954

Reviewed: https://review.openstack.org/490954
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=fd8389bfa993d4d32de8f525ffee64758e1df5b5
Submitter: Jenkins
Branch: master

commit fd8389bfa993d4d32de8f525ffee64758e1df5b5
Author: Clark Boylan <email address hidden>
Date: Fri Aug 4 08:52:30 2017 -0700

    Use clearer query for 1326813

    This bug, 1326813, is an old one that was about rackspace and hpcloud.
    We don't use hpcloud anymore (it doesn't exist) and yet we were getting
    hits on this query. Digging in the bug is actually an error of
    easy_install (distutils) talking to its pypi mirror. I have updated the
    bug to better reflect this and this updates the query to query on
    distutils specific errors to keep them tightly coupled together.

    This should make it much easier to understand what is going on when we
    see hits for this bug.

    Change-Id: Id1c5211702f9de889a87facba9694c7a8c52e1e1
    Related-bug: 1326813

While this bug describes an old issue it still happens these days, even after PYPI migrated to a CDN for serving its wheels.

Behaviour changed a little bit since original times because now pip supports retries which works really nicesly, unles... the CDN randomly gives a 404 response for stuff that should never get a 404, like core python packages, ones that installed on most virtual environments like pip, pbr,,..

We all know that this is lie, as these packages never vanished from the pypi server, still one response like this would ruin even a 4 hour build job on CI.

Using proxies, transparent or not for accelerating access pypi does not help because they will respect the 404 response.

By HTTP specification 404 is not retriable so what we need is to add an optional feature in pip that would enable PIP_RETRY_ON_404, just like a timeout error.

This would address the issue of flakiness. I know this for sure because I did something similar for python-jira library where Atlassian Cloud was randomly giving invalid HTTP codes. In that case I was able to subclass python requests session class to add retries, and it worked very well.

I thik we need to raise a bug on pip for this to track it there because that is the only place wheere this can really be fixed. We cannot really fix CDNs or people mirrors, but we probably can make pip more resilient.