bluestore-block-db-size default size should be at least 4%

The 4% recommendation is generally accurate for rdb/cephfs. For object storage, this can trend higher when there are huge numbers of small objects.

Please be aware of Rocksdb limitations here. RocksDB will only place the next level of the dB on flash if there is enough room for the whole level. With ceph's default rocksdb settings, this means that sizes lockstep between roughly 3GB, 30GB, and 300GB [0]. Anything in-between results in wasted space.

[0] https://github.com/facebook/rocksdb/wiki/Leveled-Compaction

The Octopus documentation on the recommended `db` sizes notes [0]:

"The general recommendation is to have block.db size in between 1% to 4% of block size. For RGW workloads, it is recommended that the block.db size isn’t smaller than 4% of block, because RGW heavily uses it to store its metadata. For example, if the block size is 1TB, then block.db shouldn’t be less than 40GB. For RBD workloads, 1% to 2% of block size is usually enough."

Should we have different default size based whether it's RGW or RBD worklaod?

[0] https://docs.ceph.com/docs/octopus/rados/configuration/bluestore-config-ref/#sizing

Fix proposed to branch: master
Review: https://review.opendev.org/733613

Change abandoned by "Chris MacNaughton <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/charm-ceph-osd/+/733613
Reason: Closing this review as it has updates requested for over a year and no updates submitted.

ceph-volume will calculate this for you. If we just change the charm to set no default, it will just use the ceph-volume 4% instead. Then if they want/need to customise to a smaller size, the user can set it explicitly.

I am not sure if ceph-volume calculates 4% wal size. Can't find any reference in the docs.

If it indeed does calculate 4% or simply divides the total space available of the DB device(s) equally across all the OSD devices, then either is probably OK and certainly better than the hardcoded values we have in the charms.

Patch to remove defaults: https://review.opendev.org/c/openstack/charms.ceph/+/851650