tripleo-ci-centos-9-content-provider is failing to build the base container - No public keys imported

Marking this as a promotion-blocker as more tests are failing now:

https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-content-provider&skip=0

also hitting wallaby

https://f58720b81b6e6994f37f-a6e7a7719a505df489be2c28a60c0cf4.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/tripleo-heat-templates/stable/wallaby/tripleo-ci-centos-9-content-provider-wallaby/5a30f1e/logs/undercloud/home/zuul/container_image_build.log

problem might be with the latest ubi9 image?

Looking at [1] for example in the base-buid.log:

STEP 1/32: FROM registry.access.redhat.com/ubi9:latest
Trying to pull registry.access.redhat.com/ubi9:latest...

and nothing else

[1] https://f58720b81b6e6994f37f-a6e7a7719a505df489be2c28a60c0cf4.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/tripleo-heat-templates/stable/wallaby/tripleo-ci-centos-9-content-provider-wallaby/5a30f1e/logs/undercloud/home/zuul/workspace/logs/container-builds/430a3a31-2b6e-42c1-8ca9-3acc1951f099/base/base-build.log

With respect to comment #3, I missed the comment in the description about the update date for the image.

Noting though that per [1] it was updated 28/02/2023 (not 28/03) so it is unlikely to be the ubi9 image itself as we've been using that without issue for a month

I actually sanity checked with an older image at [2] but same result

So something changed that affects how we verify the image signatures

Interesting to note this is in both upstream/opendev and RDO jobs

[1] https://catalog.redhat.com/software/containers/ubi9/ubi-init/615bdc22075b022acc111bf6?container-tabs=overview
[2] https://review.opendev.org/c/openstack/tripleo-ci/+/879529/1/zuul.d/content-provider.yaml

comparing a green run at [1] with a failing log at [2] I found the attached diff (attached to this comment)

that one is interesting:

$ diff good bad

183c183
< gnupg2.x86_64 2.3.3-2.el9 @anaconda
---
> gnupg2.x86_64 2.3.3-3.el9 @quickstart-centos-base

Apparently that was released in the last few days [3]

Trying to pin it with [4] and testing at [5]

[1] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-9-quay-push-wallaby/3f7a8b3/logs/system/installed_pkgs.txt
[2] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-9-quay-push-wallaby/60b5331/logs/system/installed_pkgs.txt
[3] https://kojihub.stream.centos.org/koji/buildinfo?buildID=31391
[4] https://review.opendev.org/c/openstack/tripleo-quickstart/+/879535
[5] https://review.rdoproject.org/r/c/testproject/+/48205

got a green result at [1] with the exclude for gnupg2.x86_64 2.3.3-3.el9

we can go with [2] as a workaround to unblock us for now

[1] https://review.rdoproject.org/r/c/testproject/+/48205/1#message-3b4f3331ad61bef64d76554fda8803f64bdcae75

[2] https://review.opendev.org/c/openstack/tripleo-quickstart/+/879535

Noting the release notes at [1] - I wonder if the problem is the SHA-1 digest. In any case we will likely need some followup here the package exclude at [2] is only a workaround and we will be hit by this again when a newer version appears

* Thu Mar 30 2023 Jakub Jelen <email address hidden> - 2.3.3-3
- Mark SHA-1 digest as weak to follow SHA-1 disablement in RHEL9 (#2070722)
- Fix interaction with SSH by not requiring the MD5 digest (#2073567)
- Fix creation of AEAD packets (#2128058)

[1] https://kojihub.stream.centos.org/koji/buildinfo?buildID=31391
[2] https://review.opendev.org/c/openstack/tripleo-quickstart/+/879535

Pretty sure we can point to the sha-1 seeing this:

[root@e40833e9c636 yum.repos.d]# gpg --dry-run --import /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
gpg: keyblock resource '/root/.gnupg/pubring.kbx': No such file or directory
gpg: Note: signatures using the SHA1 algorithm are rejected
gpg: key 05B555B38483C65D: 1 bad signature
gpg: Total number processed: 1

I've also reported this against centos itself here:
https://bugzilla.redhat.com/show_bug.cgi?id=2184640

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/879535
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/7467dd032d3873c1e0c1153713298c6b1f56793b
Submitter: "Zuul (22348)"
Branch: master

commit 7467dd032d3873c1e0c1153713298c6b1f56793b
Author: Marios Andreou <email address hidden>
Date: Wed Apr 5 11:07:53 2023 +0300

    Exclude latest gnupg2 & Pin to last good version

    This adds exclude for gnupg2-2.3.3-3.el9 and downgrade to last good
    version gnupg2.x86_64 2.3.3-2.el9 in centos9 branches for the related
    bug hitting check/gate and periodics. This is a workaround to unblock
    the gates.

    Related-Bug: 2015309

    Change-Id: Ifb1c06291b070159b43e4364a527fe8b81157704

tl;dr problem is with latest gnupg2-2.3.3-3.el9 which removed SHA-1
and registry pubkeys were still using SHA-1
For CS9, updated containers-common is needed: https://bugzilla.redhat.com/show_bug.cgi?id=2185380

From CS8 repos, https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bcd/periodic/opendev.org/openstack/neutron/stable/zed/tripleo-ci-centos-9-content-provider/bcd6705/logs/undercloud/var/log/extra/rpm-list.txt
```
containers-common-1-50.el9.x86_64
```
 https://gitlab.com/redhat/centos-stream/rpms/containers-common/-/commit/6cc0714194238d59a47d93b7730b3317eebde172 - containers-common-1-51.el9 got tagged 2 weeks ago.

@Alan thank you for adding the bug link.
 https://gitlab.com/redhat/centos-stream/rpms/containers-common/-/commit/6cc0714194238d59a47d93b7730b3317eebde172 - containers-common-1-51.el9 got tagged 2 weeks ago and does contains the updated RPM-GPG-KEY-redhat keys. We need to wait a newer containers-common package.

For podified control plane container build line, we need these two reviews:
- https://github.com/openstack-k8s-operators/repo-setup/pull/11
- https://review.rdoproject.org/r/c/rdo-jobs/+/48261

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/880055

https://mirror.stream.centos.org/9-stream/BaseOS/source/tree/Packages/gnupg2-2.3.3-3.el9.src.rpm infected rpm is removed from the mirror. Better to revert all the patches.

https://review.rdoproject.org/r/c/testproject/+/48260/5#message-2f8347cfbd56626465fd516b0ad36d3dba93bbc3

RDO nodepool image is still using the affected rpm. We have to wait for the new images to be built
and then merge the reverts.

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/880055
Committed: https://opendev.org/openstack/tripleo-ansible/commit/9ea5b8874a0c6cbaff1f30539fbc671afd013252
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 9ea5b8874a0c6cbaff1f30539fbc671afd013252
Author: Chandan Kumar <email address hidden>
Date: Tue Apr 11 19:38:43 2023 +0530

    Exclude latest gnupg2 & Pin to last good version

    This adds exclude for gnupg2-2.3.3-3.el9 and downgrade to last good
    version gnupg2.x86_64 2.3.3-2.el9 in centos9 branches for the related
    bug hitting check/gate and periodics. This is a workaround to unblock
    the gates.

    Related-Bug: 2015309

    Change-Id: I6dd684e65aeaa7d61127f51e97846bc2c65ff62a
    Signed-off-by: Chandan Kumar <email address hidden>

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/879863
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/5b587a7f65a3537157dd8b0df2415176aec447cd
Submitter: "Zuul (22348)"
Branch: master

commit 5b587a7f65a3537157dd8b0df2415176aec447cd
Author: chandan kumar <email address hidden>
Date: Tue Apr 11 06:35:00 2023 +0000

    Revert "Exclude latest gnupg2 & Pin to last good version"

    This reverts commit 7467dd032d3873c1e0c1153713298c6b1f56793b.

    Reason for revert: https://mirror.stream.centos.org/9-stream/BaseOS/source/tree/Packages/gnupg2-2.3.3-3.el9.src.rpm infected rpm is removed from the mirror.

    Closes-Bug: #2015309

    Change-Id: I6f36ab0a68a1824f3e0b317a9f859c06d73cd7af