victoria upgrade jobs aren't using content provider passed dlrn hash tag

Bug #1908246 reported by Marios Andreou
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Marios Andreou

Bug Description

At [1] the undercloud-upgrade [2] the minor update and [3] the standalone-upgrade jobs are failing during the upgrade trying to pull containers for image prepare:

        * 2020-12-14 22:00:20 | 2020-12-14 22:00:20.851286 | fa163e7f-9be0-c5bc-bae3-000000000c0a | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Not found image: http://199.204.45.249:5001/v2/tripleovictoria/openstack-rabbitmq/manifests/73035ce83cf7ac1c83c997421ed746c6", "msg": "Error running container image prepare: Not found image: http://199.204.45.249:5001/v2/tripleovictoria/openstack-rabbitmq/manifests/73035ce83cf7ac1c83c997421ed746c6", "params": {}, "success": false}

The above is from [1]. Using the same logs as an example the hash is passed in the content provider vars with [4]

        * registry_ip_address_branch:
       ussuri: 199.204.45.238
       victoria: 199.204.45.249
     provider_dlrn_hash_branch:
       ussuri: 33e5f8f0b4d9897e80872782519b2589
       victoria: b6ac01208810425bd22a88947cc043d1
     provider_dlrn_hash_tag_branch:
       ussuri: 33e5f8f0b4d9897e80872782519b2589
       victoria: b6ac01208810425bd22a88947cc043d1

but it is not consumed by the job, looking at the container image prepare params [5]

        * tag: 73035ce83cf7ac1c83c997421ed746c6
           namespace: 199.204.45.249:5001/tripleovictoria

Looks like a nit in [6] - we saw this today because there was a victoria promotion so the value of current-tripleo was updated.

[1] https://d32eb2db9c879b955fb3-cf05ec96aa1131dfad9173080b1236b7.ssl.cf5.rackcdn.com/766992/1/check/tripleo-ci-centos-8-undercloud-upgrade-victoria/297a324/logs/undercloud/home/zuul/undercloud_upgrade.log
[2] https://13dfe0d64677b4197a7a-b241a3720f6d18d48bd2c3bcf23977c9.ssl.cf2.rackcdn.com/766992/1/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-victoria/46c505f/logs/undercloud/home/zuul/undercloud_install.log
[3] https://ef542942f55810504313-258e10c2898eead8733af072f71435b9.ssl.cf5.rackcdn.com/764090/1/gate/tripleo-ci-centos-8-standalone-upgrade-victoria/b0a4910/logs/undercloud/home/zuul/standalone_upgrade.log
[4] https://d32eb2db9c879b955fb3-cf05ec96aa1131dfad9173080b1236b7.ssl.cf5.rackcdn.com/766992/1/check/tripleo-ci-centos-8-undercloud-upgrade-victoria/297a324/zuul-info/inventory.yaml
[5] https://d32eb2db9c879b955fb3-cf05ec96aa1131dfad9173080b1236b7.ssl.cf5.rackcdn.com/766992/1/check/tripleo-ci-centos-8-undercloud-upgrade-victoria/297a324/logs/undercloud/home/zuul/containers-prepare-parameter.yaml
[6] https://opendev.org/openstack/tripleo-quickstart/src/commit/8717fac68cc82c5ae0d92ea6017c63eb4050be78/config/release/tripleo-ci/CentOS-8/victoria.yml#L7

Tags: alert ci
Revision history for this message
Marios Andreou (marios-b) wrote :

fix at https://review.opendev.org/c/openstack/tripleo-quickstart/+/767112 tripleo-quickstart master: Make sure the victoria consumers are using the right dlrn tag

description: updated
description: updated
Revision history for this message
Marios Andreou (marios-b) wrote :

OK looks like the release file fix at [1] makes the job green again

seen at [2] (which Depends-On:[1] ) :

tripleo-ci-centos-8-undercloud-upgrade-victoria https://zuul.opendev.org/t/openstack/build/196698e4950a4bd2aa4bb980d5f6a716 : SUCCESS in 1h 40m 26s

tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-victoria https://zuul.opendev.org/t/openstack/build/e8065bfbd9e94c2fa08b3065f6a0da86 : SUCCESS in 2h 20m 50s

I will move the bug fix released once [1] lands (currently in the gate)

[1] https://review.opendev.org/c/openstack/tripleo-quickstart/+/767112
[2] https://review.opendev.org/c/openstack/tripleo-upgrade/+/766992

Changed in tripleo:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.