Kubernetes Control Plane Charm

Ceph allow_insecure_global_id_reclaim false on bionic causes ceph-fs service not to deploy

Bug #1952812 reported by Chris Johnston
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Kubernetes Control Plane Charm
Triaged
High
Unassigned

Bug Description

I have deployed 1.22.4 on bionic with Ceph nautilus for ceph-fs storage. As per CVE-2021-20288 [1] I have set `auth_allow_insecure_global_id_reclaim` to `false`. Upon setting this to false the `csi-cephfsplugin-provisioner` and `csi-metrics-cephfsplugin` services are removed. Looking at `juju debug-log` I see:

unit-kubernetes-master-1: 17:13:59 ERROR unit.kubernetes-master/1.juju-log Timeout attempting to determine if CephFS is enabled

It looks like this error comes from `query_cephfs_enabled` [2]. Upon further investigation I found that even though Ceph is deployed as Nautilus, the kubernetes-master nodes are installing the Ceph packages from the bionic repos, thus a mismatch of 12.2.13 on the masters and 14.2.22 on the ceph-* nodes.

If I set `auth_allow_insecure_global_id_reclaim` back to `true` the ceph-fs services are redeployed.

Since there is not a way to define which repository the ceph packages come from for kubernetes-master we are unable to resolve the CVE without manual intervention on the k-m nodes.

[1] https://docs.ceph.com/en/latest/security/CVE-2021-20288/
[2] https://github.com/charmed-kubernetes/charm-kubernetes-master/blob/a5d8758c19fe4718887291b35656e55992adf879/lib/charms/layer/kubernetes_master.py#L203-L215

See original description
Tags: sts
Chris Johnston (cjohnston)
description: updated
Revision history for this message
dongdong tao (taodd) wrote :

This does look like there is a general requirement for Kubernetes charms being able to specify the cloud-archive ceph packages that will be used in order to match the ceph cluster version

Kubernetes charm may need to add a "source" option to specify the ceph package that should be installed.

George Kraft (cynerva)
Changed in charm-kubernetes-master:
importance: Undecided → Medium
status: New → Triaged
Ponnuvel Palaniyappan (pponnuvel)
tags: added: sts
George Kraft (cynerva)
Changed in charm-kubernetes-master:
importance: Medium → High
Revision history for this message
Ponnuvel Palaniyappan (pponnuvel) wrote :

Previously there was a question of why `apt upgrade` doesn't install from
UCA sources. Bug [0] was raised with Ceph O7K Engineering to see if this
could be fixed in the packaging.

However, it was rejected with the reasoning [1]:
"Stepping between UCA pockets is really akin to a distribution upgrade so
I think its appropriate that only a dist-upgrade action would complete the
transition correctly (or an install as detailed in #8).

We could provide transitional packages but this would need to be done as
patches ontop of the ceph packages from the associated source Ubuntu
repository and as there is a clear path to complete an upgrade successfully
I'm reticent to follow this path."

This is in-line with the OpenStack upgrade guidelines [1].

kubernetes-master does install ceph-common [2]. Perhaps the charm needs
have a some way to call that function as simply changing `install_sources`
doesn't seem to do that.

[0] https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1960065
[1] https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1960065/comments/10
[2] https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/upgrade-openstack.html#ensure-cloud-node-software-is-up-to-date
[2] https://github.com/charmed-kubernetes/charm-kubernetes-control-plane/blob/master/lib/charms/layer/kubernetes_master.py#L171-L200

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.