kolla-ansible

DockerHub limits causing failure despite having a local proxy

  1. Series victoria
  2. Bug #1942134
Bug #1942134 reported by Boris Lukashev
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
kolla-ansible
Fix Released
High
Radosław Piliszek
Ussuri
Triaged
Medium
Radosław Piliszek
Victoria
Triaged
Medium
Radosław Piliszek
Wallaby
Triaged
Medium
Radosław Piliszek
Xena
Fix Released
High
Radosław Piliszek

Bug Description

Large deployments fail consistently with dockerhub rate limit errors (sorta defeats the point of dockerhub imo), despite having a local proxy set up per https://docs.openstack.org/kolla-ansible/latest/user/multinode.html using
```
docker_custom_config:
  registry-mirrors:
    - http://<ip_address>:<mapped_port from container>
```
The error occurs with the proxy running or not running, so i'm guessing that its being ignored at runtime.

Is there a better way to avoid these limits, or instructions somewhere on just building the images in-house to the local registry?

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

Hmm, it would confirm my earlier observation that, while the mirror caches images, it still asks DockerHub about them each time, causing the quota to deplete. Sad.

Anyhow, yes; it's generally recommended for production to entirely control your images, i.e., build them yourself using Kolla (kolla-build command) and publish to a locally-controlled registry.

For the registry I use Harbor: https://goharbor.io/
It's quite popular and works well.

As for building the images: https://docs.openstack.org/kolla/latest/admin/image-building.html

Changed in kolla-ansible:
status: New → Triaged
Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

I have added this issue for discussion during our internal meeting: https://wiki.openstack.org/wiki/Meetings/Kolla

Revision history for this message
Boris Lukashev (rageltman) wrote :

Thank you sir.
Now that Harbor is not a VMWare product, is it feasible to include it in the kolla-ansible stack as a dedicated proxy-cache using their quicksetup appprach sans TLS?

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

We are open to code contributions. It could be done with TLS as well but without will probably be simpler.

Revision history for this message
Piotr Parczewski (parczewski) wrote :

Classic on-premise Docker registry is already included if you don't need fancy Harbor features. https://docs.openstack.org/kolla-ansible/latest/user/multinode.html#option-1-local-registry

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

"Included" is a big word. It's merely simpler to use. If one does not need fancy features, often useful in production, then the barebones version is fine as it's enough to run for Kolla Ansible.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/808486

Changed in kolla-ansible:
status: Triaged → In Progress
Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

ml notified http://lists.openstack.org/pipermail/openstack-discuss/2021-September/024834.html

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

We will default to Quay.io in Xena+. And inform users about the choice on the stable branches.

Changed in kolla-ansible:
importance: Undecided → High
assignee: nobody → Radosław Piliszek (yoctozepto)
Revision history for this message
Boris Lukashev (rageltman) wrote :

Thank you

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/808486
Committed: https://opendev.org/openstack/kolla-ansible/commit/0d9477de3899886165eae2b3a06ef316bebf59d8
Submitter: "Zuul (22348)"
Branch: master

commit 0d9477de3899886165eae2b3a06ef316bebf59d8
Author: Radosław Piliszek <email address hidden>
Date: Fri Sep 10 18:30:25 2021 +0000

    Switch default images source to quay.io

    Docs adapted to match.
    Removed the unsupported-for-quay option to set up
    a pull-through cache.

    Closes-Bug: #1942134
    Change-Id: If5a26b1ba4bf35bc29306c24f608396dbf5e3371

Changed in kolla-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 13.0.0.0rc1

This issue was fixed in the openstack/kolla-ansible 13.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.