Ceph Monitor Charm

When removing ceph-mon unit, monmap is not updated

Bug #1833252 reported by Giuseppe Petralia
32
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Ceph Monitor Charm
In Progress
Wishlist
Luciano Lo Giudice

Bug Description

charm-ceph-mon revision 37.

I have a ceph cluster with 3 monitors.

monitor-count is 3 in ceph-mon configuration.

I removed one ceph-mon unit but monmap was not updated causing
all the ceph commands to hang on any ceph monitor unit,
hitting connection timeout when trying to talk with the already removed units.

To fix this I had to manually remove the monitor following the instruction in:
http://docs.ceph.com/docs/luminous/rados/operations/add-or-rm-mons/#removing-monitors-from-an-unhealthy-cluster

Chris MacNaughton (chris.macnaughton)
tags: added: scaleback
Changed in charm-ceph-mon:
status: New → Incomplete
status: Incomplete → Confirmed
James Hebden (ec0)
tags: added: canonical-bootstack
Revision history for this message
Ryan Beisner (1chb1n) wrote :

There are known gaps in scaling back / scaling down OpenStack Charms.

The relation-departed Juju primitive is essentially NotImplemented across all of OpenStack Charms. That needs to be addressed on the higher level, as a concerted effort across the lot of charms.

We are tracking this type of issue with LP tag: "scaleback" in order to get a hit list of items to start with, and so that we can identify any common themes or areas where we should focus first.

FYI, https://bugs.launchpad.net/bugs/+bugs?field.tag=scaleback

Changed in charm-ceph-mon:
importance: Undecided → Wishlist
Revision history for this message
Andrea Ieri (aieri) wrote :

For a simpler workaround, delete the mons you plan to remove before removing the unit.

On the unit you want to remove:
service ceph-mon stop
ceph mon remove `hostname`

The charms will take care of cleaning up ceph.conf and removing mgrs.

Revision history for this message
Andrea Ieri (aieri) wrote :

Subscribed field-high.
This bug is impacting scale-back operations, but has a clear workaround that could be integrated into the charm without having to touch relation-departed hooks.

James Page (james-page)
Changed in charm-ceph-mon:
status: Confirmed → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-ceph-mon (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ceph-mon/+/812623

Changed in charm-ceph-mon:
status: Triaged → In Progress
Luciano Lo Giudice (lmlogiudice)
Changed in charm-ceph-mon:
assignee: nobody → Luciano Lo Giudice (lmlogiudice)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on charm-ceph-mon (master)

Change abandoned by "James Page <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/charm-ceph-mon/+/812623
Reason: This review is > 12 weeks without comment, and failed testing the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.