Overcloud deploy fails in scenario jobs - "no id" for containers

Bug #1802265 reported by Sagi (Sergey) Shnaidman
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned

Bug Description

In all scenario jobs overcloud deployment fails, messages are different, but mostly because "no id" for some container:

http://logs.openstack.org/98/604298/74/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/3a844f7/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-11-08_02_59_44
2018-11-08 02:59:43 | fatal: [centos-7-ovh-bhs1-0000412455]: FAILED! => {"changed": true, "cmd": "docker tag 192.168.24.1:8787/tripleomaster/centos-binary-cinder-backup:current-tripleo-updated-20181108001130 192.168.24.1:8787/tripleomaster/centos-binary-cinder-backup:pcmklatest", "delta": "0:00:00.031695", "end": "2018-11-08 02:59:25.283962", "msg": "non-zero return code", "rc": 1, "start"Exception occured while running the command
2018-11-08 02:59:43 | Traceback (most recent call last):
2018-11-08 02:59:43 | File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 28, in run
2018-11-08 02:59:43 | super(Command, self).run(parsed_args)
2018-11-08 02:59:43 | File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
2018-11-08 02:59:43 | return super(Command, self).run(parsed_args)
2018-11-08 02:59:43 | File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
2018-11-08 02:59:43 | return_code = self.take_action(parsed_args) or 0
2018-11-08 02:59:43 | File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 928, in take_action
2018-11-08 02:59:43 | verbosity=self.app_args.verbose_level)
2018-11-08 02:59:43 | File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/deployment.py", line 306, in config_download
2018-11-08 02:59:43 | raise exceptions.DeploymentError("Overcloud configuration failed.")
2018-11-08 02:59:43 | DeploymentError: Overcloud configuration failed.

http://logs.openstack.org/98/604298/74/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/ecb6a02/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-11-08_02_04_27
2018-11-08 02:04:27 | fatal: [centos-7-rax-iad-0000412459]: FAILED! => {"changed": true, "cmd": "docker tag 192.168.24.1:8787/tripleomaster/centos-binary-cinder-backup:current-tripleo-updated-20181108001837 192.168.24.1:8787/tripleomaster/centos-binary-cinder-backup:pcmklatest", "delta": "0:00:00.039849", "end": "2018-11-08 02:04:20.631821", "msg": "non-zero return code", "rc": 1, "start": "2018-11-08 02:04:20.591972", "stderr": "Error response from daemon: no such id: 192.168.24.1:8787/tripleomaster/centos-binary-cinder-backup:current-tripleo-updated-20181108001837", "stderr_lines": ["Error response from daemon: no such id: 192.168.24.1:8787/tripleomaster/centos-binary-cinder-backup:current-tripleo-updated-20181108001837"], "stdout": "", "stdout_lines": []}

http://logs.openstack.org/98/604298/74/check/tripleo-ci-centos-7-scenario003-multinode-oooq-container/89a6466/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
2018-11-08 01:45:27 | "Error: Evaluation Error: Error while evaluating a Resource Statement,

Duplicate declaration: Class[Designate::Db] is already declared; cannot redeclare at /etc/puppet/modules/tripleo/manifests/profile/base/designate/central.pp:58 at /etc/puppet/modules/tripleo/manifests/profile/base/designate/central.pp:58:5 on node centos-7-vexxhost-sjc1-0000412498.localdomain",

2018-11-08 01:45:34 | =============================================================================== Exception occured while running the command
2018-11-08 01:45:34 | Traceback (most recent call last):
2018-11-08 01:45:34 | File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 28, in run
2018-11-08 01:45:34 | super(Command, self).run(parsed_args)
2018-11-08 01:45:34 | File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
2018-11-08 01:45:34 | return super(Command, self).run(parsed_args)
2018-11-08 01:45:34 | File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
2018-11-08 01:45:34 | return_code = self.take_action(parsed_args) or 0
2018-11-08 01:45:34 | File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 928, in take_action
2018-11-08 01:45:34 | verbosity=self.app_args.verbose_level)
2018-11-08 01:45:34 | File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/deployment.py", line 306, in config_download
2018-11-08 01:45:34 | raise exceptions.DeploymentError("Overcloud configuration failed.")
2018-11-08 01:45:34 | DeploymentError: Overcloud configuration failed.
2018-11-08 01:45:34 | Overcloud configuration failed.

http://logs.openstack.org/98/604298/74/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/7f4b42b/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-11-08_01_50_55

2018-11-08 01:50:55 | fatal: [centos-7-inap-mtl01-0000412462]: FAILED! => {"changed": true, "cmd": "docker tag 192.168.24.1:8787/tripleomaster/centos-binary-manila-share:current-tripleo-updated-20181108000959 192.168.24.1:8787/tripleomaster/centos-binary-manila-share:pcmklatest", "delta": "0:00:00.030324", "end": "2018-11-08 01:50:47.136684", "msg": "non-zero return code", "rc": 1, "start": "2018-11-08 01:50:47.106360", "stderr": "Error response from daemon: no such id: 192.168.24.1:8787/tripleomaster/centos-binary-manila-share:current-tripleo-updated-20181108000959", "stderr_lines": ["Error response fromException occured while running the command
2018-11-08 01:50:55 | Traceback (most recent call last):
2018-11-08 01:50:55 | File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 28, in run
2018-11-08 01:50:55 | super(Command, self).run(parsed_args)
2018-11-08 01:50:55 | File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
2018-11-08 01:50:55 | return super(Command, self).run(parsed_args)
2018-11-08 01:50:55 | File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
2018-11-08 01:50:55 | return_code = self.take_action(parsed_args) or 0
2018-11-08 01:50:55 | File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 928, in take_action
2018-11-08 01:50:55 | verbosity=self.app_args.verbose_level)
2018-11-08 01:50:55 | File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/deployment.py", line 306, in config_download
2018-11-08 01:50:55 | raise exceptions.DeploymentError("Overcloud configuration failed.")
2018-11-08 01:50:55 | DeploymentError: Overcloud configuration failed.
2018-11-08 01:50:55 | Overcloud configuration failed.
2018-11-08 01:50:55 | END return value: 1
2018-11-08 01:50:55 | daemon: no such id: 192.168.24.1:8787/tripleomaster/centos-binary-manila-share:current-tripleo-updated-20181108000959"], "stdout": "", "stdout_lines": []}

http://logs.openstack.org/98/604298/74/check/tripleo-ci-centos-7-scenario007-multinode-oooq-container/967272c/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-11-08_02_07_54

2018-11-08 02:07:54 | fatal: [centos-7-rax-ord-0000412463]: FAILED! => {"changed": true, "cmd": "docker tag 192.168.24.1:8787/tripleomaster/centos-binary-ovn-northd:current-tripleo-updated-20181108002031 192.168.24.1:8787/tripleomaster/centos-binary-ovn-northd:pcmklatest", "delta": "0:00:00.037287", "end": "2018-11-08 02:07:42.579543", "msg": "non-zero return code", "rc": 1, "start": "2018-11-08 02:07:42.542256", "stderr": "Error response from daemon: no such id: 192.168.24.1:8787/tripleomaster/centos-binary-ovn-northd:current-tripleo-updated-20181108002031", "stderr_lines": ["Error response from daemon: no such id: 192.168.24.1:8787/tripleomaster/centos-binary-ovn-northd:current-tripleo-updated-20181108002031"], "stdout": "", "stdout_lines": []}
2018-11-08 02:07:54 |

From first job:
The image exists:

http://logs.openstack.org/98/604298/74/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/3a844f7/logs/undercloud/var/log/extra/docker/docker_allinfo.log.txt.gz
192.168.24.1:8787/tripleomaster/centos-binary-cinder-backup current-tripleo-updated-20181108001130 b6c1a3208d6d 26 minutes ago 1.21 GB

but container doesn't exist

Tags: ci
summary: - Overcloud deploy fails in scenario jobs
+ Overcloud deploy fails in scenario jobs - "no id" for containers
Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/616565

Revision history for this message
Alex Schultz (alex-schultz) wrote :

https://review.openstack.org/#/c/616022/ is not the breaking fix. The revert did not fix the scenario jobs. Something else broke it.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-common (master)

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.openstack.org/616565
Reason: this didn't fix the scenario jobs

Changed in tripleo:
milestone: stein-2 → stein-3
Revision history for this message
wes hayutin (weshayutin) wrote :
Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.