I found some strange behavior in BlueFS! It seems on each heartbeat timed out on my OSD nodes ceph_bluefs_read_prefetch_bytes will raise to 400MB/s and it's too huge because in normal it has just 20MB sometimes 1KB/s on it also client throughput is just 20MB on its disk! Is there any way to find out what are these reads for? I don't have any backfilling and there is just regular scrub and deep scrubs on my server! Thanks. On Thu, Dec 24, 2020 at 12:47 AM Seena Fallah <seenafallah@gmail.com> wrote:

I have enabled bluefs_buffered_io on some of my OSD nodes and disable some others based on the server node situation and I'm experiencing this issue on both of them!

How can manual RocksDB compaction help?

Can you please share with me the topic names for this issue on the mailing list?

On Wed, Dec 23, 2020 at 7:04 PM Igor Fedotov <ifedotov@suse.de> wrote:

...

Hi Seena,

one of the frequent cause for such a timeout is slow RocksDB operationing. Which in turn might be caused by bluefs_buffered_io set to false and/or DB "fragmentation" after massive data removal.

Hence the potential workarounds are adjusting bluefs_buffered_io and manual RocksDB compaction.

This topic has been discussed in this mailing list and relevant tickets multiple times.

Thanks,

Igor

On 12/23/2020 3:24 PM, Seena Fallah wrote:

...

Hi,

All my OSD nodes in the SSD tier are getting heartbeat_map timed out randomly and I don't find why!

7ff2ed3f2700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7ff2c8943700' had timed out after 15

It occurs many times in a day and causes my cluster to be down.

Is there any way to find why the OSDs get time out? I don't think it's because of heartbeat and there is an issue with OSD that came to the heartbeat to be timeout because ODSs don't suicide and OSDs get too slow and cause downtime on RBD and S3 gateway because the queue is full!

Thanks. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io