Hi Vladimir, no it is the same on all monitors. Actually I got triggered because I got slow responses on my rados gateway with the radosgw-admin command and narrowed it down to slow respons for rados commands anywhere in the cluster. The cluster is not that busy and all osds and monitors use very little resources compared to what they have onboard Kind Regards Marcel Kuiper

Hi Marcel, If this issue related to only one monitor? If yes, check the overall node status: average load, disk I/O, RAM consumption, swap size, etc. Could be not a ceph-related issue.

Regards, Vladimir.

On Wed, Aug 26, 2020 at 9:07 AM Marcel Kuiper <ceph@mknet.nl> wrote:

...

Hi

One of my clusters running nautilus 14.2.8 is very slow (13 seconds or so where my other clusters are returning almost instantanious) when doing a 'rados --pool rc3-se.rgw.buckets.index ls' from one of the monitors.

I checked - ceph status => OK - routing to/from osds ok (I see a lot of established connections to osds due to the command, nothing in syn_sent indicating incomplete handshake) - ping times are OK - no interface errors - no packet drops - no increasing send queus - and as far as I can see nothing out of the ordinary in mon and osd logs

I have no clue how to debug the issue. If someone has pointers it would be much appreciated

Kind Regards

Marcel _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

On 26/08/2020 15:59, Stefan Kooman wrote:

On 2020-08-26 15:20, Marcel Kuiper wrote:

...

Hi Vladimir,

no it is the same on all monitors. Actually I got triggered because I got slow responses on my rados gateway with the radosgw-admin command and narrowed it down to slow respons for rados commands anywhere in the cluster.

Do you have a very large amount of objects. And / or a lot of OMAP data and thus large rocksdb databases? We have seen slowness (and slow ops) from having very large rocksdb databases due to a lot of OMAP data concentrated on only a few nodes (cephfs metadata only). You might suffer from the same thing.

Manual rocksdb compaction on the OSDs might help.

In addition: Keep in mind that RADOS was never designed to list objects fast. The more Placement Groups you have the slower a listing will be. Wido

Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Sorry that had to be Wido/Stefan Another question is: hoe to use this ceph-kvstore-tool tool to compact the rocksdb? (can't find a lot of examples) The WAL and DB are on a separate NVMe. The directoy structure for an osd looks like: root@se-rc3-st8vfr2t2:/var/lib/ceph/osd# ls -l ceph-174 total 24 lrwxrwxrwx 1 ceph ceph 93 Aug 27 10:12 block -> /dev/ceph-97d39775-65ef-41a6-a9fe-94a108c0816d/osd-block-7f83916e-7250-4935-89af-d678a9bb9f29 lrwxrwxrwx 1 ceph ceph 27 Aug 27 10:12 block.db -> /dev/ceph-db-nvme0n1/db-sdd -rw------- 1 ceph ceph 37 Aug 27 10:12 ceph_fsid -rw------- 1 ceph ceph 37 Aug 27 10:12 fsid -rw------- 1 ceph ceph 57 Aug 27 10:12 keyring -rw------- 1 ceph ceph 6 Aug 27 10:12 ready -rw------- 1 ceph ceph 10 Aug 27 10:12 type -rw------- 1 ceph ceph 4 Aug 27 10:12 whoami Kind Regards Marcel Kuiper

Hi Wido/Joost

pg_num is 64. It is not that we use 'rados ls' for operations. We just noticed as a difference that on this cluster it takes about 15 seconds to return on pool .rgw.root or rc3-se.rgw.buckets.index and our other clusters return almost instantaniously

Is there a way that I can determine from statistics that manual compaction might help (besides doing the compaction and notice the difference in behaviour). Any pointers in investigating this further would be much appreciated

Is there operational impact to be expected when compacting manually?

Kind Regards

Marcel Kuiper

...

On 26/08/2020 15:59, Stefan Kooman wrote:

Thank you Stefan, I'm going to give that a try Kind Regards Marcel Kuiper

On 2020-08-27 13:29, Marcel Kuiper wrote:

...

Sorry that had to be Wido/Stefan

What does "ceph osd df" give you? There is a column with "OMAP" and "META". OMAP is ~ 13 B, META 26 GB in our setup. Quite a few files in cephfs (main reason we have large OMAP).

...

Another question is: hoe to use this ceph-kvstore-tool tool to compact the rocksdb? (can't find a lot of examples)

If you want to do a whole host a the time:

systemctl stop ceph-osd.target

wait a few seconds till all processes are closed.

for osd in `ls /var/lib/ceph/osd/`; do (ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/$osd compact &);done

This works for us (no seperate WAL/DB). Check the help of the ceph-kvstore-tool if you have to do anything special with separate DB / WAL devices.

Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

On 2020-08-31 14:16, Marcel Kuiper wrote:

The compaction of the bluestore-kv's helped indeed. The repons is back to acceptable levels

Just curious. Did you do any cluster expansion and or PG expansion before the slowness occurred? Gr. Stefan

On 2020-09-01 10:51, Marcel Kuiper wrote:

As a matter of fact we did. We doubled the storage nodes from 25 to 50. Total osds now 460.

You want to share your thoughts on that?

Yes. We observed the same thing with expansions. The OSDs will be very busy (with multiple threads per OSD) on housekeeping after the OMAP data has been moved to another OSD (and eating up all CPU power it can get). But even after that there is a lot of garbage left behind not gets cleaned up. At least not with regular housekeeping / online compaction. Manual compaction for clusters with a lot of OMAP data feels like a necessity (and ideally shouln't be). Gr. Stefan

On 2020-09-02 23:50, Wido den Hollander wrote:

Indeed, it shouldn't be.

This config option should make it easier in a future release: https://github.com/ceph/ceph/commit/93e4c56ecc13560e0dad69aaa67afc3ca053fb4c

[osd] osd_compact_on_start = true

Then just restart the OSDs and they will compact on boot. No need for external scripts. Just put this into the ceph.conf.

The mon config store won't work as there is no connection with the Monitors at that point in the code.

Yay! Thanks Wido Gr. Stefan

Hi Stefan I can't recall that that was the case and unfortunately we do not have enough history for our performance measurements to look back We are on nautilus. Please let me know your findings when you do your pg expansion on nautilus Grtz Marcel

OK, I'm really curious if you observed the following behaviour:

During, or shortly after the rebalance, did you see high CPU usage of the OSDs? In particular the ones that hosted the PGs before they were moved to the new nodes? As in ~ 300 % CPU per OSD (increasing from a few percent to 300% non stop)? RocksDB is doing housekeeping, And we observed before, and today again, on Mimic 13.2.8, that with a lot of OMAP/META data the OSDs that have to clean up consume a ridiculous amount of CPU (for hours on end). Triggering loads of slow ops and latency spikes in the somtimes (tens) of seconds.

Are you running nautilus? If you haven't seen this behaviour this might have been fixed in Nautlilus. Or you cluster is different from ours. We will do PG expansion after we have upgraded to Nautilus, so we'll definitely know by then.

Thanks,

Stefan