unable to create pools before OSD's are up and running

I think this is less about gnocchi, and more about ceph-mon (which does the actual pool creation). In later ceph releases, pool creation fails if no OSD's are present at the point in time the pool is created. Its possible to work around this to some extent by setting the expected-osd-count configuration option on ceph-mon, but even this has some chance of racing.

I think we need to evolve the ceph-mon/ceph-osd relation to allow ceph-mon to determine when OSD's are up and usable; at this point pools can be created.

FWIW this behaviour appears to have been introduced in newer Ceph releases.

I'm still digging into the bug - my original 'can't create pools without OSD's' appears to be untrue to trying to figure out why the broker request actually fails.

Reproduced:

2018-07-11 11:18:54 INFO juju-log client:5: Creating pool 'glance' (replicas=3)
2018-07-11 11:18:54 DEBUG client-relation-changed Error ERANGE: pg_num 200 size 3 would mean 1200 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-07-11 11:18:55 ERROR juju-log client:5: Command '['ceph', '--id', 'admin', 'osd', 'pool', 'create', 'glance', '200']' returned non-zero exit status 34.
2018-07-11 11:18:55 ERROR juju-log client:5: Unexpected error occurred while processing requests: {'api-version': 1, 'request-id': '2a919b75-84fc-11e8-ba20-fa163e7e0b1a', 'ops': [{'group': 'images', 'name': 'glance', 'weight': 5, 'replicas': 3, 'pg_num': None, 'group-namespace': None, 'op': 'create-pool'}]}

Feature: https://ceph.com/community/new-luminous-pg-overdose-protection/

One possible workaround is to set mon_max_pg_per_osd to something stupidly high; this will allow the pool to be created sans any OSD's actually being in. The OSD pool config will still be right sized based on the expected OSD count (if used).

For example:

juju config ceph-mon config-flags="{'global': {'mon max pg per osd': 100000}}"

This works around the issue; we need a more complete solution but this at least gets automated deployments with vault based dmcrypt keys running again.

Ok thanks, we will try this workaround in one of our deployment to see if we can reproduce the behavior.

Raising Ubuntu task; it would be nice if we could tell the ceph-mon cluster how many OSD's there *will* be, rather than it defaulting to '3' if there are no OSD's currently checked in.

This will require a new configuration option and adoption by the ceph project.

For a charm based deployment, the value of 'in' osds is unreliable until the cloud is fully deployed; having this trip hazard is non-ideal.

Change abandoned by Seyeong Kim (<email address hidden>) on branch: master
Review: https://review.openstack.org/581392
Reason: Need discussion about fundamental behaviour.