OSD down when deleting CephFS files/leveldb compaction
I have one or two more stability issues I'm trying to solve in a cluster that I inherited that I just can't seem to figure out. One issue may be the cause for the other. This is a Jewel 10.2.11 cluster with ~760 - 10TB HDDs and 5GB journals on SSD. When a large number of files are deleted from CephFS (and possibly when leveldb compacts), the OSD will stop responding to heartbeats and get marked down, then come back and start recovery and then other OSDs will have the same issue until client load on the cluster eases up then it settles down. Is there a way to have leveldb compact more frequently or cause it to come up for air more frequently and respond to heartbeats and process some IO? I thought splitting PGs would help, but we are still seeing the problem (previously ~20 PGs per OSD to now ~150). I still have some space on the SSDs that I can double, almost triple the journal, but not sure if that will help in this situation. The other issue I'm seeing is that some IO just gets stuck when the OSDs are getting marked down and coming back through the cluster. Thanks, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Wed, Oct 2, 2019 at 10:56 PM Robert LeBlanc <robert@leblancnet.us> wrote:
Is there a way to have leveldb compact more frequently or cause it to come up for air more frequently and respond to heartbeats and process some IO?
you can manually trigger a compaction via the admin socket (or was it via ceph tell?) with the compact command, but I don't think that this helps with your workload.
I thought splitting PGs would help, but we are still seeing the problem (previously ~20 PGs per OSD to now ~150). I still have some space on the SSDs that I can double, almost triple the journal, but not sure if that will help in this situation.
no, a larger journal will not help for leveldb workloads. Big difference between FileStore journals and BlueStore DB devices: BlueStore actually puts all the metadata onto the SSD permanently, a FileStore journal is just a journal (and 5 GB is large enough, it won't use that much space for small operations like deletions). (The answer that you don't want to hear is probably the best way forward: upgrade to BlueStore) Paul
The other issue I'm seeing is that some IO just gets stuck when the OSDs are getting marked down and coming back through the cluster.
Thanks, Robert LeBlanc
---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Wed, Oct 2, 2019 at 3:41 PM Paul Emmerich <paul.emmerich@croit.io> wrote:
On Wed, Oct 2, 2019 at 10:56 PM Robert LeBlanc <robert@leblancnet.us> wrote:
Is there a way to have leveldb compact more frequently or cause it to come up for air more frequently and respond to heartbeats and process some IO?
you can manually trigger a compaction via the admin socket (or was it via ceph tell?) with the compact command, but I don't think that this helps with your workload.
This would be really helpful, I've just been restarting OSD processes. After some digging, it looks like it was added to Luminous [1]. :(
I thought splitting PGs would help, but we are still seeing the problem (previously ~20 PGs per OSD to now ~150). I still have some space on the SSDs that I can double, almost triple the journal, but not sure if that will help in this situation.
no, a larger journal will not help for leveldb workloads.
Big difference between FileStore journals and BlueStore DB devices: BlueStore actually puts all the metadata onto the SSD permanently, a FileStore journal is just a journal (and 5 GB is large enough, it won't use that much space for small operations like deletions).
(The answer that you don't want to hear is probably the best way forward: upgrade to BlueStore)
We are planning to replace this cluster in ~6 months, so we don't want to upgrade it. I'm just looking for things that I may have missed that will keep the pager quieter until then. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1441693 ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
participants (2)
-
Paul Emmerich
-
Robert LeBlanc