Investigate 'profile rbd' for rbd client ceph auth caps

Bug #1892781 reported by Dan Hill
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceph Monitor Charm
New
Undecided
Unassigned

Bug Description

[Impact]
Ceph librbd clients make use of exclusive locks [0] to co-ordinate access to RBD. In the event that an exclusive lock is mismanaged (hung task, crash, network partition, etc.) a new client can negotiate a blacklist with the ceph mon. This requires that the client have the correct mon auth caps configured to allow the 'osd blacklist' command.

Without blacklist caps, a mismanaged exclusive lock will prevent nova-compute instance from starting.

The ceph-mon charm should configure nova/cinder and glance rbd clients with this capability.

[Workaround]
Pre-Luminous, the osd blacklist command must be explicitly allowed:
mon 'allow r, allow command "osd blacklist"'

Post-luminous, upstream recommends using 'profile rbd' when specifying auth caps for rbd clients:
> V12.0.0 Luminous: Specifying user authorization capabilities for RBD clients has been simplified. The general syntax for using RBD capability profiles is “mon ‘profile rbd’ osd ‘profile rbd[-read-only][ pool={pool-name}[, …]]’”. [1]

[Other Info]
There is an upstream guide for rbd OpenStack integration that details how to setup ceph client authentication for nova/cinder and glance [2].

An important note! Luminous and mimic do not support mgr profiles. This was introduced in nautilus 14.2.8 [3]:
> The MGR now accepts profile rbd and profile rbd-read-only user caps. These caps can be used to provide users access to MGR-based RBD functionality such as `rbd perf image iostat` and `rbd perf image iotop`.

[0] https://docs.ceph.com/docs/master/rbd/rbd-exclusive-locks/#rbd-exclusive-locks
[1] https://docs.ceph.com/docs/master/releases/luminous/#major-changes-from-kraken
[2] https://docs.ceph.com/docs/master/rbd/rbd-openstack/#setup-ceph-client-authentication
[3] https://docs.ceph.com/docs/master/releases/nautilus/#id4

Tags: sts
Dan Hill (hillpd)
tags: added: sts
Revision history for this message
James Page (james-page) wrote :
Revision history for this message
James Page (james-page) wrote :

This rang a bell - I'm fairly sure we give everything the blacklist permission based on the fixes for bug 1773449

Revision history for this message
James Page (james-page) wrote :

Recent deployment:

client.cinder-ceph
        key: AQDvKS1fm/CpJBAAfVPcC7U8xdUiECRVNifGjA==
        caps: [mon] allow r; allow command "osd blacklist"
        caps: [osd] allow rwx
client.glance
        key: AQCCKS1fcTjVNhAAbMNII51knVtDyoueZRsLnA==
        caps: [mon] allow r; allow command "osd blacklist"
        caps: [osd] allow rwx
client.nova-compute
        key: AQCBKS1f7P8SFRAAtayvH3RHZH/c5PeAVHv36A==
        caps: [mon] allow r; allow command "osd blacklist"
        caps: [osd] allow rwx

Changed in charm-ceph-mon:
status: New → Incomplete
Revision history for this message
Dan Hill (hillpd) wrote :

It is good to see the workaround has already been implemented.

We should still consider two things:
1. Using the rbd profile instead of the explicit "osd blacklist". The profile was created to address future bugs where a change in caps is required to fix a critical issue.
2. Adding the rbd profile to the [mgr] caps. This enables the commands: `rbd perf image iostat` and `rbd perf image iotop` on 14.2.8+

Changed in charm-ceph-mon:
status: Incomplete → New
Dan Hill (hillpd)
summary: - Failure to start ceph-backed instance after crash
+ Investigate 'profile rbd' for rbd client ceph auth caps
Revision history for this message
Dan Hill (hillpd) wrote :

Pacific renamed "blacklist" to "blocklist" [0].

The blocklist command name change broke Nova's ability to recover rbd exclusive locks from an ungraceful shutdown.

Using 'rbd profile' caps for both the mon/osd will cover this change and future changes.

[0] https://docs.ceph.com/en/latest/releases/pacific/#id30

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.