Exception: Error inspecting image: docker://docker.io/ server misbehaving

Bug #1795718 reported by wes hayutin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Invalid
Critical
Unassigned

Bug Description

2018-10-02 16:52:36.977 13286 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] rendering j2 template to file: /home/zuul/tripleo-heat-installer-templates/./network/config/multiple-nics/compute-dvr.yaml
2018-10-02 16:52:36.977 13286 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] jinja2 rendering normal template controller-v6.j2.yaml
2018-10-02 16:52:36.977 13286 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] rendering j2 template to file: /home/zuul/tripleo-heat-installer-templates/./network/config/multiple-nics/controller-v6.yaml
2018-10-02 16:53:02.480 13286 ERROR tripleoclient.v1.tripleo_deploy.Deploy [ ] Exception: Error inspecting image: docker://docker.io/tripleorocky/centos-binary-barbican-api:current-tripleo
time="2018-10-02T16:53:02Z" level=fatal msg="pinging docker registry returned: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 127.0.0.1:53: server misbehaving"

2018-10-02 16:53:02.490 13286 ERROR tripleoclient.v1.tripleo_deploy.Deploy [ ] None
2018-10-02 16:53:03.657 13286 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Not cleaning working directory /home/zuul/tripleo-heat-installer-templates
2018-10-02 16:53:03.692 13286 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Not cleaning ansible directory /home/zuul/undercloud-ansible-vpyQEW
2018-10-02 16:53:03.718 13286 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Install artifact is located at /home/zuul/undercloud-install-20181002165302.tar.bzip2

http://logs.openstack.org/73/607173/1/gate/tripleo-ci-centos-7-containerized-undercloud-upgrades/703ce71/logs/undercloud/home/zuul/install-undercloud.log.txt.gz#_2018-10-02_16_53_02_480

Source:
http://git.openstack.org/cgit/openstack/tripleo-common/tree/tripleo_common/image/image_uploader.py#n298

Maybe the msg needs to be updated?

    if process.returncode != 0:
            not_found_msgs = (
                u'manifest unknown',
                # returned by docker.io
                u'requested access to the resource is denied'
            )

Revision history for this message
Alex Schultz (alex-schultz) wrote :

So i think the issue here might be that unbound is not properly caching the registry-1.docker.io dns lookup due to the config.

http://logs.openstack.org/25/602425/5/check/tripleo-ci-centos-7-containers-multinode/e2bb2ec/logs/undercloud/etc/unbound/conf.d/ttl.conf.txt.gz
# Ansible managed

server:
    cache-min-ttl: 900
    cache-max-ttl: 86400

Unfortunately the TTL for docker.io is 900 so it's skipping the cache.

Oct 02 16:52:43 unbound[1731:1] info: incoming scrubbed packet: ;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 0
;; flags: qr rd ra ; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
;; QUESTION SECTION:
docker.io. IN DS

;; ANSWER SECTION:

;; AUTHORITY SECTION:
io. 900 IN SOA a0.nic.io. noc.afilias-nst.info. 1497465179 10800 3600 2764800 900

;; ADDITIONAL SECTION:
;; MSG SIZE rcvd: 90
http://logs.openstack.org/73/607173/1/gate/tripleo-ci-centos-7-containerized-undercloud-upgrades/703ce71/logs/undercloud/var/lib/unbound/unbound.log.txt.gz

wes hayutin (weshayutin)
tags: removed: alert
Changed in tripleo:
milestone: stein-1 → stein-2
Changed in tripleo:
milestone: stein-2 → stein-3
Changed in tripleo:
status: Triaged → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.