I'm not sure if your OSDs have their rocksDB on faster devices, if not it sounds a lot like rocksdb fragmentation [1] leading to a very high load on the OSDs and occasionally crashing OSDs. If you don't plan to delete so much data at once on a regular basis you could sit this one out, but one solution is to re-create the OSDs with rocksDB/WAL on faster devices. [1] https://www.mail-archive.com/ceph-users@ceph.io/msg03160.html Zitat von Francois Legrand <fleg@lpnhe.in2p3.fr>:
Thanks for the hint. I tryed but it doesn't seems to change anything... Moreover, as the osds seems quite loaded I had regularly some osd marked down which triggered some new peering and thus more load !!! I set the osd no down flag, but I still have some osd reported (wrongly) as down (and back up in the minute) which generate peering and remapping. I don't really understand the action of no down parameter ! Is there a way to tell ceph not to peer immediately after an osd is reported down (let say wait for 60s) ? I am thinking about restarting all osd (or maybe the whole cluster) to get osd_op_queue_cut_off changed to high and osd_op_thread_timeout to something higher than 15 (but I don't think it will really improve the situation). F.
Le 25/06/2020 à 14:26, Wout van Heeswijk a écrit :
Hi Francois,
Have you already looked at the option "osd_delete_sleep"? It will not speed up the process but I will give you some control over your cluster performance.
Something like:
ceph tell osd.\* injectargs '--osd_delete_sleep1' kind regards,
Wout 42on On 25-06-2020 09:57, Francois Legrand wrote:
Does someone have an idea ? F. _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io