Hi Stephan,  it looks like you've faced the following bug: https://tracker.ceph.com/issues/47883 To workaround it you might want to switch both bluestore and bluefs allocators back to bitmap for now. The fixes for Octopus/Nautilus are on their ways: https://github.com/ceph/ceph/pull/38474 https://github.com/ceph/ceph/pull/38475 Hope this helps, Igor On 12/16/2020 5:51 PM, Stephan Austermühle wrote:

Hi all,

in various Rook operated Ceph clusters I have seen OSDs going into a CrashLoop due to

debug 2020-12-16T13:19:25.500+0000 7fc4c3f13f40  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1608124765507105, "job": 1, "event": "recovery_started", "log_files": [1400, 1402]} debug 2020-12-16T13:19:25.500+0000 7fc4c3f13f40  4 rocksdb: [db/db_impl_open.cc:583] Recovering log #1400 mode 0 debug 2020-12-16T13:19:27.724+0000 7fc4c3f13f40  1 bluefs _allocate failed to allocate 0x43ce43d on bdev 1, free 0x2e50000; fallback to bdev 2 debug 2020-12-16T13:19:27.724+0000 7fc4c3f13f40  1 bluefs _allocate unable to allocate 0x43ce43d on bdev 2, free 0xffffffffffffffff; fallback to slow device expander debug 2020-12-16T13:19:27.724+0000 7fc4c3f13f40 -1 bluestore(/var/lib/ceph/osd/ceph-1) allocate_bluefs_freespace failed to allocate on 0x3d1b0000 min_size 0x43d0000 > allocated total 0x300000 bluefs_shared_alloc_size 0x10000 allocated 0x300000 available 0x b019c8000 debug 2020-12-16T13:19:27.724+0000 7fc4c3f13f40 -1 bluefs _allocate failed to expand slow device to fit +0x43ce43d debug 2020-12-16T13:19:27.724+0000 7fc4c3f13f40 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x43ce43d /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.6/rpm/el8/BUILD/ceph-15.2.6/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7fc4c3f13f40 time 2020-12-16T13:19:27.731533+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.6/rpm/el8/BUILD/ceph-15.2.6/src/os/bluestore/BlueFS.cc: 2721: ceph_abort_msg("bluefs enospc")

The OSD is not really full:

# ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-1 inferring bluefs devices from bluestore path 1 : device size 0x18ffc00000 : own 0x[bffe10000~fffe0000] = 0xfffe0000 : using 0xfd190000(4.0 GiB) : bluestore has 0x82b360000(33 GiB) available Expanding DB/WAL...

Expanding the underlying block device by just 1 gig followed by "ceph-bluestore-tool bluefs-bdev-expand" and "ceph-bluestore-tool repair" resolves the situation. In general, larger OSDs seem to reduce the likeliness for this issue.

Ceph version is v15.2.6.

Is this a known bug?

Ceph report and logs are attached.

Thanks for your help

Stephan

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io