Multiple node_failure on rdo jobs blocking rdo check jobs

blocks promotions too https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-main

periodic-tripleo-centos-8-master-promote-promoted-components-to-tripleo-ci-testing openstack/tripleo-ci master openstack-periodic-integration-main master 0 secs 2020-12-07 08:10:32 NODE_FAILURE

so nothing else ran since that failed

and it affects all the integration pipelines in fact master/victoria/ussuri

NODE_FAILURE on all the 'promote-consistent-to-tripleo-ci-testing' job

https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-main
https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable1
https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable2
https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable3-centos7
https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable3
https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable4-5
https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable6

Discussion with dpwalik
<dpawlik|EoD> chandankumar: hey. We have some issue with Vexxhost glance service. Images has been recreated by nodepool and set as "ready" when on vexxhost side image is in uploading state
<dpawlik|EoD> chandankumar: and now in the nodepool I see HttpException: 500: i n nova
<dpawlik|EoD> I will do a high prio ticket
<chandankumar> dpawlik|EoD, Thanks for the info :-)

seems there was _some_ improvement since the master pipeline has some green [1] but the rest are still hitting some errors see [2][3] eg

periodic-tripleo-centos-8-ussuri-promote-promoted-components-to-tripleo-ci-testing openstack/tripleo-ci master openstack-periodic-integration-stable2 master 3 mins 27 secs 2020-12-08 04:15:42 SUCCESS

periodic-tripleo-centos-8-buildimage-overcloud-hardened-full-ussuri openstack/tripleo-ci master openstack-periodic-integration-stable2 master 9 mins 1 sec 2020-12-08 04:41:40 RETRY_LIMIT

periodic-tripleo-centos-8-buildimage-ironic-python-agent-ussuri openstack/tripleo-ci master openstack-periodic-integration-stable2 master 10 mins 48 secs 2020-12-08 04:41:13 RETRY_LIMIT

periodic-tripleo-centos-8-buildimage-overcloud-full-ussuri openstack/tripleo-ci master openstack-periodic-integration-stable2 master 8 mins 33 secs 2020-12-08 04:46:20 RETRY_LIMIT

periodic-tripleo-ci-build-containers-ubi-8-push-ussuri openstack/tripleo-ci master openstack-periodic-integration-stable2 master 10 mins 40 secs 2020-12-08 04:48:24 RETRY_LIMIT

periodic-tripleo-ci-centos-8-standalone-upgrade-ussuri openstack/tripleo-ci master openstack-periodic-integration-stable2 master 0 secs Invalid date SKIPPED

(and then the rest are skipped).
Don't know if this is the same ongoing issue or something new.

[1] https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-main
[2] https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable1
[3] https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable2

OK so after digging some more it seems comment #4 was premature.

The issues in the victoria pipeline [1] are a new bug reported there https://bugs.launchpad.net/tripleo/+bug/1907245

The issues in the ussuri pipeline [2] are actually a different bug there https://bugs.launchpad.net/tripleo/+bug/1907193 e.g. [3]

So I *think* we can call the original issue in this bug fixed now. Possibly leave it around for another day to verify.

[1] https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable1
[2] https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable2
[3] https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-hardened-full-ussuri/2d56775/job-output.txt