Master Periodic jobs: Sc001/002 failing with Error: \", \" Problem: package rdma-core-26.0-8.el8.x86_64 requires dracut, but none of the providers can be installed

Master periodic jobs Sc001/002 are failing with below error:-

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-master/dc4bd03/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/7be4f48/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

~~~
Error: \", \" Problem: package rdma-core-26.0-8.el8.x86_64 requires dracut, but none of the providers can be installed\", \" - package dracut-049-70.git20200228.el8.x86_64 requires systemd-udev >= 219, but none of the providers can be installed\", \" - package libibverbs-26.0-8.el8.x86_64 requires rdma-core(x86-64) = 26.0-8.el8, but none of the providers can be installed\", \" - package systemd-udev-239-30.el8_2.x86_64 requires systemd(x86-64) = 239-30.el8_2, but none of the providers can be installed\", \" - package librados2-2:14.2.9-1.el8.x86_64 requires libibverbs.so.1()(64bit), but none of the providers can be installed\", \" - package librados2-2:14.2.9-1.el8.x86_64 requires libibverbs.so.1(IBVERBS_1.0)(64bit), but none of the providers can be installed\", \" - package librados2-2:14.2.9-1.el8.x86_64 requires libibverbs.so.1(IBVERBS_1.1)(64bit), but none of the providers can be installed\", \" - cannot install both systemd-239-30.el8_2.x86_64 and systemd-239-31.el8_2.2.x86_64\", \" - cannot install the best update candidate for package librados2-1:12.2.7-9.el8.x86_64\", \" - problem with installed package systemd-239-31.el8_2.2.x86_64\", \"level=debug msg=\\\\\"error running [bash -x /tmp/yum_update.sh delorean-current,quickstart-centos-ceph-nautilus] in container \\\\\\\\\\\\\"centos-binary-gnocchi-statsd-working-container\\\\\\\\\\\\\": error while running runtime: exit status 1\\\\\"\"], \"stdout\": \"Updating Subscription Management repositories.\\\\nUnable to read consumer identity\\\\nSubscription Manager is operating in container mode.\\\\nThis system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.\\\\nLast metadata expiration check: 0:00:27 ago on Mon 27 Jul 2020 09:25:49 AM UTC.\\\\n(try to add \\'--allowerasing\\' to command line to replace conflicting packages or \\'--skip-broken\\' to skip uninstallable packages or \\'--nobest\\' to use not only best candidate packages)\", \"stdout_lines\": [\"Updating Subscription Management repositories.\", \"Unable to read consumer identity\", \"Subscription Manager is operating in container mode.\", \"This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.\", \"Last metadata expiration check: 0:00:27 ago on Mon 27 Jul 2020 09:25:49 AM UTC.\", \"(try to add \\'--allowerasing\\' to command line to replace conflicting packages or \\'--skip-broken\\' to skip uninstallable packages or \\'--nobest\\' to use not only best candidate packages)\"]}\\n\\x1b[Ke30=\\x1b[4D\\x1b[K\\x1b[KeyJ1dWlkIjogImQzYTViNWQ1LTA4OGItNGVmNS05MzBiLWJjNzQ5NGVkZGFhYiJ9\\x1b[64D\\x1b[K\\nPLAY RECAP
~~~

This issue is also discuss in patch: https://review.rdoproject.org/r/#/c/28663/

Snippet:-
~~~
For fixing librados2 issue, librados2 can be added explicitly to the package installation list https://github.com/openstack/tripleo-common/blob/master/container-images/tcib/base/os/gnocchi-base/gnocchi-base.yaml#L4-L5, which will pull librados and trigger upgrade/downgrade.

The issue happened after new ubi8 image released recently which have new systemd packages installed then centos8 base repos, So when there is attempt to install python3-cradox, due to dnf behavior mentioned above it avoids downgrade to systemd from centos8-base repos and picks old librados2-12 from appstream repo.

Mixing centos8 and ubi8 can cause similar issue in future too, so should keep enabled ubi8 repos instead of removing them(happening currently as we bind mount /etc/yum.repos.d:/etc/yum.repos.d) during container build.
~~~

A work in progress patch[1] is already proposed and test run[2] is green using it.

[1] https://review.opendev.org/#/c/743263/
[2] https://review.rdoproject.org/r/#/c/28723/