The WebSocket timed out before the Workflow completed in rocky/stain jobs

Bug #1875833 reported by Amol Kahat
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
Incomplete
Wishlist
Amol Kahat

Bug Description

Description
===========

Queens and rocky jobs are failing with error The WebSocket timed out before the Workflow completed.

Affected jobs:
==============
- periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-queens
- periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens
- periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-queens
- periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens

Pipeline link:
=============
- https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-wednesday-weekend

Actual Error:
=============
Timed out waiting for messages from Execution (ID: c49f2b5f-5b36-4ebc-8765-fedc40ba9dcd, State: RUNNING). The WebSocket timed out before the Workflow completed.

same error reflecting in different jobs.

Logs:
=====
2020-04-28 23:52:46 | TASK [Write docker-puppet-tasks json files] ************************************
2020-04-28 23:52:46 | task path: /var/lib/mistral/c49f2b5f-5b36-4ebc-8765-fedc40ba9dcd/common_deploy_steps_tasks.yaml:248
2020-04-28 23:52:46 | Tuesday 28 April 2020 23:52:43 +0000 (0:00:00.502) 0:04:48.756 *********
2020-04-29 00:53:36 | changed: [overcloud-controller-0] => (item=step_3) => {"changed": true, "checksum": "de9ae0fa287ee107bf45fe11d5b6fc7bfbbf6ebc", "dest": "/var/lib/docker-puppet/docker-puppet-tasks3.json", "gid": 0, "group": "root", "item": ["step_3", [{"config_image": "192.168.24.1:8787/tripleoqueens/centos-binary-keystone:c9e0fef7c387836d4d3ad46d073f10c312857a4f_5fce6f98", "config_volume": "keystone_init_tasks", "puppet_tags": "keystone_config,keystone_domain_config,keystone_endpoint,keystone_identity_provider,keystone_paste_ini,keystone_role,keystone_service,keystone_tenant,keystone_user,keystone_user_role,keystone_domain", "step_config": "include ::tripleo::profile::base::keystone"}]], "md5sum": "7e5f0188d1467da028173d810e033929", "mode": "0600", "owner": "root", "sTimed out waiting for messages from Execution (ID: c49f2b5f-5b36-4ebc-8765-fedc40ba9dcd, State: RUNNING). The WebSocket timed out before the Workflow completed.
2020-04-29 00:53:36 |
2020-04-29 00:53:36 | END return value: 1
2020-04-29 00:53:36 | econtext": "system_u:object_r:var_lib_t:s0", "size": 444, "src": "/home/tripleo-admin/.ansible/tmp/ansible-tmp-1588117964.24-35593494992396/source", "state": "file", "uid": 0}
2020-04-29 00:53:36 |
2020-04-29 00:53:36 | TASK [Set host puppet debugging fact string] ***********************************
2020-04-29 00:53:36 | task path: /var/lib/mistral/c49f2b5f-5b36-4ebc-8765-fedc40ba9dcd/common_deploy_steps_tasks.yaml:261
2020-04-29 00:53:36 | Tuesday 28 April 2020 23:52:45 +0000 (0:00:01.089) 0:04:49.845 *********
2020-04-29 00:53:36 |
2020-04-29 00:53:36 | TASK [Write the config_step hieradata] *****************************************
2020-04-29 00:53:36 | task path: /var/lib/mistral/c49f2b5f-5b36-4ebc-8765-fedc40ba9dcd/common_deploy_steps_tasks.yaml:268
2020-04-29 00:53:36 | Tuesday 28 April 2020 23:52:45 +0000 (0:00:00.240) 0:04:50.086 *********
2020-04-29 00:53:36 | changed: [overcloud-controller-2] => {"changed": true, "checksum": "dfdcc7695edd230e7a2c06fc7b739bfa56506d8f", "dest": "/etc/puppet/hieradata/config_step.json", "gid": 0, "group": "root", "md5sum": "f0ef53dcc6eb8440334b1ebaa90bfd63", "mode": "0600", "owner": "root", "secontext": "system_u:object_r:puppet_etc_t:s0", "size": 11, "src": "/home/tripleo-admin/.ansible/tmp/ansible-tmp-1588117965.36-137127088675725/source", "state": "file", "uid": 0}
2020-04-29 00:53:36 |
2020-04-29 00:53:36 | changed: [overcloud-controller-1] => {"changed": true, "checksum": "dfdcc7695edd230e7a2c06fc7b739bfa56506d8f", "dest": "/etc/puppet/hieradata/config_step.json", "gid": 0, "group": "root", "md5sum": "f0ef53dcc6eb8440334b1ebaa90bfd63", "mode": "0600", "owner": "root", "secontext": "system_u:object_r:puppet_etc_t:s0", "size": 11, "src": "/home/tripleo-admin/.ansible/tmp/ansible-tmp-1588117965.43-89045811942861/source", "state": "file", "uid": 0}
2020-04-29 00:53:36 | changed: [overcloud-controller-0] => {"changed": true, "checksum": "dfdcc7695edd230e7a2c06fc7b739bfa56506d8f", "dest": "/etc/puppet/hieradata/config_step.json", "gid": 0, "group": "root", "md5sum": "f0ef53dcc6eb8440334b1ebaa90bfd63", "mode": "0600", "owner": "root", "secontext": "system_u:object_r:puppet_etc_t:s0", "size": 11, "src": "/home/tripleo-admin/.ansible/tmp/ansible-tmp-1588117965.5-26684484695720/source", "state": "file", "uid": 0}
2020-04-29 00:53:36 | changed: [overcloud-novacompute-0] => {"changed": true, "checksum": "dfdcc7695edd230e7a2c06fc7b739bfa56506d8f", "dest": "/etc/puppet/hieradata/config_step.json", "gid": 0, "group": "root", "md5sum": "f0ef53dcc6eb8440334b1ebaa90bfd63", "mode": "0600", "owner": "root", "secontext": "system_u:object_r:puppet_etc_t:s0", "size": 11, "src": "/home/tripleo-admin/.ansible/tmp/ansible-tmp-1588117965.62-210910173425716/source", "state": "file", "uid": 0}
2020-04-29 00:53:36 |
2020-04-29 00:53:36 | TASK [Run puppet host configuration for step 1] ********************************
2020-04-29 00:53:36 | task path: /var/lib/mistral/c49f2b5f-5b36-4ebc-8765-fedc40ba9dcd/common_deploy_steps_tasks.yaml:274
2020-04-29 00:53:36 | Tuesday 28 April 2020 23:52:46 +0000 (0:00:01.243) 0:04:51.330 *********
2020-04-29 00:53:36 | changed: [overcloud-novacompute-0] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true}
2020-04-29 00:53:36 |
2020-04-29 00:53:37 | + status_code=1
2020-04-29 00:53:37 | + openstack stack list
2020-04-29 00:53:37 | + grep -q overcloud
2020-04-29 00:53:41 | + openstack stack list
2020-04-29 00:53:41 | + grep -Eq '(CREATE|UPDATE)_COMPLETE'
2020-04-29 00:53:45 | + exit 1

Amol Kahat (amolkahat)
tags: added: alert promotion-blocker
removed: aler
tags: added: in-stable-stein
Amol Kahat (amolkahat)
Changed in tripleo:
importance: High → Critical
assignee: nobody → amolkahat (amolkahat)
milestone: none → ussuri-rc2
Revision history for this message
Amol Kahat (amolkahat) wrote :
Revision history for this message
wes hayutin (weshayutin) wrote :

This needs more details before proceeding...

wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-rc2 → ussuri-rc3
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-rc3 → victoria-1
Changed in tripleo:
milestone: victoria-1 → victoria-3
wes hayutin (weshayutin)
tags: removed: ci promotion-blocker
Changed in tripleo:
importance: Critical → Wishlist
status: New → Invalid
status: Invalid → Incomplete
Changed in tripleo:
milestone: victoria-3 → wallaby-1
Changed in tripleo:
milestone: wallaby-1 → wallaby-2
Changed in tripleo:
milestone: wallaby-2 → wallaby-3
Revision history for this message
Marios Andreou (marios-b) wrote :

Bug status has been set to 'Incomplete' and target milestone has been removed due to inactivity. If you disagree please re-set these values and reach out to us on freenode #tripleo

Changed in tripleo:
milestone: wallaby-3 → none
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.