We thought that this problem is caused by high OSD fragmentation, until today. For now even OSDs with fragmentation rating < .3 are affected. We don't use separate DB/WAL partition in this setup and strings like this before failing:
2020-07-25 11:08:22.961 7f6f489d5700 1 bluefs _allocate failed to allocate 0x33dd4c5 on bdev 1, free 0x2bc0000; fallback to bdev 2
2020-07-25 11:08:22.961 7f6f489d5700 1 bluefs _allocate unable to allocate 0x33dd4c5 on bdev 2, free 0xffffffffffffffff; fallback to slow device expander
look suspicious for us.
We use 4KiB bluefs and bluestore block sizes as well as store the objects ~1KiB size and it looks like this makes the issue to be reproduced much more frequently. But, as I can see on the tracker / telegram channels, different people face with it from time to time, for example:
https://paste.ubuntu.com/p/GDCXDrnrtX/ (telegram link
https://t.me/ceph_users/376)
Did anyone able to identify the root cause and/or find a workaround for it?