Hi Boris, This sounds a bit like https://tracker.ceph.com/issues/53729. https://tracker.ceph.com/issues/53729#note-65 might help you diagnose whether this is the case. Josh On Tue, Feb 21, 2023 at 9:29 AM Boris Behrens <bb@kervyn.de> wrote:
Hi, today I wanted to increase the PGs from 2k -> 4k and random OSDs went offline in the cluster. After some investigation we saw, that the OSDs got OOM killed (I've seen a host that went from 90GB used memory to 190GB before OOM kills happen).
We have around 24 SSD OSDs per host and 128GB/190GB/265GB memory in these hosts. All of them experienced OOM kills. All hosts are octopus / ubuntu 20.04.
And on every step new OSDs crashed with OOM. (We now set the pg_num/pgp_num to 2516 to stop the process). The OSD logs do not show anything why this might happen. Some OSDs also segfault.
I now started to stop all OSDs on a host, and do a "ceph-bluestore-tool repair" and a "ceph-kvstore-tool bluestore-kv compact" on all OSDs. This takes for the 8GB OSDs around 30 minutes. When I start the OSDs I instantly get a lot of slow OPS from all the other OSDs when the OSD come up (the 8TB OSDs take around 10 minutes with "load_pgs".
I am unsure what I can do to restore normal cluster performance. Any ideas or suggestions or maybe even known bugs? Maybe a line for what I can search in the logs.
Cheers Boris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io