manila-tempest-*-centos-7 jobs are broken

There are three centos-7 jobs in the check queue in manila (manila-tempest-minimal-dsvm-cephfs-native-centos-7, manila-tempest-minimal-dsvm-cephfs-nfs-centos-7 and manila-tempest-minimal-dsvm-lvm-centos-7). All three of these jobs fail during DevStack setup:

Sample log snip:

2018-05-17 16:07:04.405 | ++ functions-common:sudo_with_proxies:2212 : sudo http_proxy= https_proxy= no_proxy= yum install -y bc bridge-utils curl dbus gcc gcc-c++ gettext git-core graphviz httpd httpd-devel libffi-devel libjpeg-turbo-devel libxml2-devel libxslt-devel libyaml-devel mariadb-devel net-tools openssh-server openssl openssl-devel pcre-devel pkgconfig postgresql-devel psmisc pyOpenSSL python-devel redhat-rpm-config systemd-devel tar tcpdump unzip util-linux wget which zlib-devel memcached mod_ssl sqlite conntrack-tools curl dnsmasq dnsmasq-utils ebtables gawk genisoimage iptables iputils kpartx libxml2-python m2crypto mysql-devel numpy parted polkit sqlite sudo cryptsetup genisoimage iscsi-initiator-utils libosinfo lvm2 sg3_utils sysfsutils acl dnsmasq dnsmasq-utils ebtables haproxy iptables iputils mysql-devel sqlite sudo conntrack-tools keepalived ipset lvm2
2018-05-17 16:07:04.594 | Loaded plugins: fastestmirror
2018-05-17 16:07:04.670 | Loading mirror speeds from cached hostfile
2018-05-17 16:07:04.769 | http://mirror.centos.org/%24contentdir/7/virt/x86_64/kvm-common/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
2018-05-17 16:07:04.770 | Trying other mirror.
2018-05-17 16:07:04.770 | To address this issue please refer to the below wiki article
2018-05-17 16:07:04.770 |
2018-05-17 16:07:04.770 | https://wiki.centos.org/yum-errors
2018-05-17 16:07:04.770 |
2018-05-17 16:07:04.770 | If above article doesn't help to resolve this issue please use https://bugs.centos.org/.
2018-05-17 16:07:04.770 |
2018-05-17 16:07:04.775 |
2018-05-17 16:07:04.775 |
2018-05-17 16:07:04.775 | One of the configured repositories failed (CentOS-7 - QEMU EV),
2018-05-17 16:07:04.775 | and yum doesn't have enough cached data to continue. At this point the only
2018-05-17 16:07:04.775 | safe thing yum can do is fail. There are a few ways to work "fix" this:
2018-05-17 16:07:04.775 |
2018-05-17 16:07:04.775 | 1. Contact the upstream for the repository and get them to fix the problem.
2018-05-17 16:07:04.775 |
2018-05-17 16:07:04.775 | 2. Reconfigure the baseurl/etc. for the repository, to point to a working
2018-05-17 16:07:04.775 | upstream. This is most often useful if you are using a newer
2018-05-17 16:07:04.775 | distribution release than is supported by the repository (and the
2018-05-17 16:07:04.775 | packages for the previous distribution release still work).
2018-05-17 16:07:04.775 |
2018-05-17 16:07:04.775 | 3. Run the command with the repository temporarily disabled
2018-05-17 16:07:04.775 | yum --disablerepo=centos-qemu-ev ...
2018-05-17 16:07:04.776 |
2018-05-17 16:07:04.776 | 4. Disable the repository permanently, so yum won't use it by default. Yum
2018-05-17 16:07:04.776 | will then just ignore the repository until you permanently enable it
2018-05-17 16:07:04.776 | again or use --enablerepo for temporary usage:
2018-05-17 16:07:04.776 |
2018-05-17 16:07:04.776 | yum-config-manager --disable centos-qemu-ev
2018-05-17 16:07:04.776 | or
2018-05-17 16:07:04.776 | subscription-manager repos --disable=centos-qemu-ev
2018-05-17 16:07:04.776 |
2018-05-17 16:07:04.776 | 5. Configure the failing repository to be skipped, if it is unavailable.
2018-05-17 16:07:04.776 | Note that yum will try to contact the repo. when it runs most commands,
2018-05-17 16:07:04.776 | so will have to try and fail each time (and thus. yum will be be much
2018-05-17 16:07:04.776 | slower). If it is a very temporary problem though, this is often a nice
2018-05-17 16:07:04.776 | compromise:
2018-05-17 16:07:04.776 |
2018-05-17 16:07:04.776 | yum-config-manager --save --setopt=centos-qemu-ev.skip_if_unavailable=true
2018-05-17 16:07:04.776 |
2018-05-17 16:07:04.776 | failure: repodata/repomd.xml from centos-qemu-ev: [Errno 256] No more mirrors to try.
2018-05-17 16:07:04.776 | http://mirror.centos.org/$contentdir/7/virt/x86_64/kvm-common/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
2018-05-17 16:07:04.804 | ++ functions-common:yum_install:1351 : echo YUM_FAILED 1
2018-05-17 16:07:04.804 | YUM_FAILED 1
2018-05-17 16:07:04.809 | ++ functions-common:yum_install:1352 : result=1
2018-05-17 16:07:04.813 | ++ functions-common:yum_install:1354 : time_stop yum_install
2018-05-17 16:07:04.817 | ++ functions-common:time_stop:2254 : local name
2018-05-17 16:07:04.821 | ++ functions-common:time_stop:2255 : local end_time
2018-05-17 16:07:04.825 | ++ functions-common:time_stop:2256 : local elapsed_time
2018-05-17 16:07:04.828 | ++ functions-common:time_stop:2257 : local total
2018-05-17 16:07:04.832 | ++ functions-common:time_stop:2258 : local start_time
2018-05-17 16:07:04.836 | ++ functions-common:time_stop:2260 : name=yum_install
2018-05-17 16:07:04.840 | ++ functions-common:time_stop:2261 : start_time=1526573224366
2018-05-17 16:07:04.843 | ++ functions-common:time_stop:2263 : [[ -z 1526573224366 ]]
2018-05-17 16:07:04.847 | +++ functions-common:time_stop:2266 : date +%s%3N
2018-05-17 16:07:04.854 | ++ functions-common:time_stop:2266 : end_time=1526573224850
2018-05-17 16:07:04.858 | ++ functions-common:time_stop:2267 : elapsed_time=484
2018-05-17 16:07:04.861 | ++ functions-common:time_stop:2268 : total=2398
2018-05-17 16:07:04.865 | ++ functions-common:time_stop:2270 : _TIME_START[$name]=
2018-05-17 16:07:04.869 | ++ functions-common:time_stop:2271 : _TIME_TOTAL[$name]=2882
2018-05-17 16:07:04.873 | ++ functions-common:yum_install:1363 : '[' 1 == 2 ']'
2018-05-17 16:07:04.877 | ++ functions-common:yum_install:1367 : return 1
2018-05-17 16:07:04.882 | + functions-common:real_install_package:1 : exit_trap
2018-05-17 16:07:04.886 | + ./stack.sh:exit_trap:515 : local r=1
2018-05-17 16:07:04.891 | ++ ./stack.sh:exit_trap:516 : jobs -p
2018-05-17 16:07:04.896 | + ./stack.sh:exit_trap:516 : jobs=
2018-05-17 16:07:04.900 | + ./stack.sh:exit_trap:519 : [[ -n '' ]]
2018-05-17 16:07:04.904 | + ./stack.sh:exit_trap:525 : '[' -f '' ']'
2018-05-17 16:07:04.908 | + ./stack.sh:exit_trap:530 : kill_spinner
2018-05-17 16:07:04.913 | + ./stack.sh:kill_spinner:425 : '[' '!' -z '' ']'
2018-05-17 16:07:04.917 | + ./stack.sh:exit_trap:532 : [[ 1 -ne 0 ]]
2018-05-17 16:07:04.921 | + ./stack.sh:exit_trap:533 : echo 'Error on exit'
2018-05-17 16:07:04.922 | Error on exit
2018-05-17 16:07:04.926 | + ./stack.sh:exit_trap:535 : type -p generate-subunit
2018-05-17 16:07:04.930 | + ./stack.sh:exit_trap:538 : [[ -z /opt/stack/new ]]
2018-05-17 16:07:04.934 | + ./stack.sh:exit_trap:541 : /opt/stack/new/devstack/tools/worlddump.py -d /opt/stack/new
2018-05-17 16:07:04.963 | World dumping... see /opt/stack/new/worlddump-2018-05-17-160704.txt for details
2018-05-17 16:07:05.024 | /bin/sh: brctl: command not found
2018-05-17 16:07:05.772 | + ./stack.sh:exit_trap:550 : exit 1

Detailed devstacklog attached with this bug report.

You may also be able to peruse the log files as long as they are available on the log server:
Failed on testing:https://review.openstack.org/#/c/556197/4
Log links:

http://logs.openstack.org/97/556197/4/check/manila-tempest-minimal-dsvm-cephfs-native-centos-7/b3b77e9/
http://logs.openstack.org/97/556197/4/check/manila-tempest-minimal-dsvm-cephfs-nfs-centos-7/f04133b/
http://logs.openstack.org/97/556197/4/check/manila-tempest-minimal-dsvm-lvm-centos-7/857d0ff/

RCA
====

There was a recent change announced on the CentOS mailing lists (https://lists.centos.org/pipermail/centos-devel/2018-March/016542.html) regarding housing centos-release-SIG repos for CPU architectures other than x86_64. We rely on centos-release-pike within our test jobs and the repo centos-qemu-ev is found under http://mirror.centos.org/centos/7/virt/x86_64/kvm-common.

To accommodate "altarch"s, there is now a YUM variable called $contentdir which determines where the repos may be:

They'd be in either of the following two folders:
http://mirror.centos.org/centos/7/virt
http://mirror.centos.org/altarch/7/virt

With $contentdir set, the URL resolves correctly: http://mirror.centos.org/$contentdir/7/virt