Thanks, Igor. I mentioned earlier that according to the OSD logs compaction wasn't an issue. I did run `ceph-kvstore-tool` offline though, it completed rather quickly without any warnings or errors, but the OSD kept showing excessive latency. I did something rather radical: rebooted the node and redeployed all its OSDs, now the "slow" OSD is showing latency more in line with that of other OSDs. /Z On Thu, 27 Apr 2023 at 23:10, Igor Fedotov <igor.fedotov@croit.io> wrote:
Hi Zakhar,
you might want to try offline DB compaction using ceph-kvstore-tool for this specific OSD.
Periodically we observe OSD perf drop due to degraded RocksDB performance, particularly after bulk data removal/migration.. Compaction is quite helpful in this case.
Thanks,
Igor
On 26/04/2023 20:22, Zakhar Kirpichenko wrote:
Hi,
I have a Ceph 16.2.12 cluster with uniform hardware, same drive make/model, etc. A particular OSD is showing higher latency than usual in `ceph osd perf`, usually mid to high tens of milliseconds while other OSDs show low single digits, although its drive's I/O stats don't look different from those of other drives. The workload is mainly random 4K reads and writes, the cluster is being used as Openstack VM storage.
Is there a way to trace, which particular PG, pool and disk image or object cause this OSD's excessive latency? Is there a way to tell Ceph to
I would appreciate any advice or pointers.
Best regards, Zakhar _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io