Comment 10 for bug 1903221

I think I found the issue. In the log for failed OSD initialization we can see "_aio_start io_setup(2) failed with EAGAIN; try increasing /proc/sys/fs/aio-max-nr". So I checked both aio-max-nr and aio-nr and realized the system is close to the limit. When I increased the limit and manually run ceph-volume, it succeeded.

# Read the limit
root@storage-1:~# cat /proc/sys/fs/aio-max-nr
65536

# Read the current value
root@storage-1:~# cat /proc/sys/fs/aio-nr
63490

# Then I increased /proc/sys/fs/aio-max-nr to 1048576
root@storage-1:~# echo 1048576 > /proc/sys/fs/aio-max-nr
root@storage-1:~# cat /proc/sys/fs/aio-max-nr
1048576

After this I was able to successfully initialize new OSD.

I'm wondering if this is something that charm could calculate and adjust?