26 Aug
2020
26 Aug
'20
5:36 p.m.
Hi
One of my clusters running nautilus 14.2.8 is very slow (13 seconds or so
where my other clusters are returning almost instantanious) when doing a
'rados --pool rc3-se.rgw.buckets.index ls' from one of the monitors.
I checked
- ceph status => OK
- routing to/from osds ok (I see a lot of established connections to osds
due to the command, nothing in syn_sent indicating incomplete handshake)
- ping times are OK
- no interface errors
- no packet drops
- no increasing send queus
- and as far as I can see nothing out of the ordinary in mon and osd logs
I have no clue how to debug the issue. If someone has pointers it would be
much appreciated
Kind Regards
Marcel
26 Aug
26 Aug
5:42 p.m.
Hi Marcel,
If this issue related to only one monitor?
If yes, check the overall node status: average load, disk I/O, RAM
consumption, swap size, etc. Could be not a ceph-related issue.
Regards,
Vladimir.
On Wed, Aug 26, 2020 at 9:07 AM Marcel Kuiper <ceph(a)mknet.nl> wrote:
Hi
One of my clusters running nautilus 14.2.8 is very slow (13 seconds or so
where my other clusters are returning almost instantanious) when doing a
'rados --pool rc3-se.rgw.buckets.index ls' from one of the monitors.
I checked
- ceph status => OK
- routing to/from osds ok (I see a lot of established connections to osds
due to the command, nothing in syn_sent indicating incomplete handshake)
- ping times are OK
- no interface errors
- no packet drops
- no increasing send queus
- and as far as I can see nothing out of the ordinary in mon and osd logs
I have no clue how to debug the issue. If someone has pointers it would be
much appreciated
Kind Regards
Marcel
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:50 p.m.
Hi Vladimir,
no it is the same on all monitors. Actually I got triggered because I got
slow responses on my rados gateway with the radosgw-admin command and
narrowed it down to slow respons for rados commands anywhere in the
cluster.
The cluster is not that busy and all osds and monitors use very little
resources compared to what they have onboard
Kind Regards
Marcel Kuiper
Hi Marcel,
If this issue related to only one monitor?
If yes, check the overall node status: average load, disk I/O, RAM
consumption, swap size, etc. Could be not a ceph-related issue.
Regards,
Vladimir.
On Wed, Aug 26, 2020 at 9:07 AM Marcel Kuiper <ceph(a)mknet.nl> wrote:
Hi
One of my clusters running nautilus 14.2.8 is very slow (13 seconds or
so
where my other clusters are returning almost instantanious) when doing a
'rados --pool rc3-se.rgw.buckets.index ls' from one of the monitors.
I checked
- ceph status => OK
- routing to/from osds ok (I see a lot of established connections to
osds
due to the command, nothing in syn_sent indicating incomplete handshake)
- ping times are OK
- no interface errors
- no packet drops
- no increasing send queus
- and as far as I can see nothing out of the ordinary in mon and osd
logs
I have no clue how to debug the issue. If someone has pointers it would
be
much appreciated
Kind Regards
Marcel
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:29 p.m.
On 2020-08-26 15:20, Marcel Kuiper wrote:
Hi Vladimir,
no it is the same on all monitors. Actually I got triggered because I got
slow responses on my rados gateway with the radosgw-admin command and
narrowed it down to slow respons for rados commands anywhere in the
cluster.
Do you have a very large amount of objects. And / or a lot of OMAP data
and thus large rocksdb databases? We have seen slowness (and slow ops)
from having very large rocksdb databases due to a lot of OMAP data
concentrated on only a few nodes (cephfs metadata only). You might
suffer from the same thing.
Manual rocksdb compaction on the OSDs might help.
Gr. Stefan
6:58 p.m.
On 26/08/2020 15:59, Stefan Kooman wrote:
On 2020-08-26 15:20, Marcel Kuiper wrote:
In addition: Keep in mind that RADOS was never designed to list objects
fast. The more Placement Groups you have the slower a listing will be.
Wido
>
> Gr. Stefan
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
Hi Vladimir,
no it is the same on all monitors. Actually I got triggered because I got
slow responses on my rados gateway with the radosgw-admin command and
narrowed it down to slow respons for rados commands anywhere in the
cluster.
Do you have a very large amount of objects. And / or a lot of OMAP data
and thus large rocksdb databases? We have seen slowness (and slow ops)
from having very large rocksdb databases due to a lot of OMAP data
concentrated on only a few nodes (cephfs metadata only). You might
suffer from the same thing.
Manual rocksdb compaction on the OSDs might help.
27 Aug
27 Aug
2:37 p.m.
Hi Wido/Joost
pg_num is 64. It is not that we use 'rados ls' for operations. We just
noticed as a difference that on this cluster it takes about 15 seconds to
return on pool .rgw.root or rc3-se.rgw.buckets.index and our other
clusters return almost instantaniously
Is there a way that I can determine from statistics that manual compaction
might help (besides doing the compaction and notice the difference in
behaviour). Any pointers in investigating this further would be much
appreciated
Is there operational impact to be expected when compacting manually?
Kind Regards
Marcel Kuiper
On 26/08/2020 15:59, Stefan Kooman wrote:
On 2020-08-26 15:20, Marcel Kuiper wrote:
In addition: Keep in mind that RADOS was never designed to list objects
fast. The more Placement Groups you have the slower a listing will be.
Wido
Hi Vladimir,
no it is the same on all monitors. Actually I got triggered because I
got
slow responses on my rados gateway with the radosgw-admin command and
narrowed it down to slow respons for rados commands anywhere in the
cluster.
Do you have a very large amount of objects. And / or a lot of OMAP data
and thus large rocksdb databases? We have seen slowness (and slow ops)
from having very large rocksdb databases due to a lot of OMAP data
concentrated on only a few nodes (cephfs metadata only). You might
suffer from the same thing.
Manual rocksdb compaction on the OSDs might help.
Gr. Stefan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:59 p.m.
Sorry that had to be Wido/Stefan
Another question is: hoe to use this ceph-kvstore-tool tool to compact the
rocksdb? (can't find a lot of examples)
The WAL and DB are on a separate NVMe. The directoy structure for an osd
looks like:
root@se-rc3-st8vfr2t2:/var/lib/ceph/osd# ls -l ceph-174
total 24
lrwxrwxrwx 1 ceph ceph 93 Aug 27 10:12 block ->
/dev/ceph-97d39775-65ef-41a6-a9fe-94a108c0816d/osd-block-7f83916e-7250-4935-89af-d678a9bb9f29
lrwxrwxrwx 1 ceph ceph 27 Aug 27 10:12 block.db ->
/dev/ceph-db-nvme0n1/db-sdd
-rw------- 1 ceph ceph 37 Aug 27 10:12 ceph_fsid
-rw------- 1 ceph ceph 37 Aug 27 10:12 fsid
-rw------- 1 ceph ceph 57 Aug 27 10:12 keyring
-rw------- 1 ceph ceph 6 Aug 27 10:12 ready
-rw------- 1 ceph ceph 10 Aug 27 10:12 type
-rw------- 1 ceph ceph 4 Aug 27 10:12 whoami
Kind Regards
Marcel Kuiper
Hi Wido/Joost
pg_num is 64. It is not that we use 'rados ls' for operations. We just
noticed as a difference that on this cluster it takes about 15 seconds to
return on pool .rgw.root or rc3-se.rgw.buckets.index and our other
clusters return almost instantaniously
Is there a way that I can determine from statistics that manual compaction
might help (besides doing the compaction and notice the difference in
behaviour). Any pointers in investigating this further would be much
appreciated
Is there operational impact to be expected when compacting manually?
Kind Regards
Marcel Kuiper
>
>
> On 26/08/2020 15:59, Stefan Kooman wrote:
28 Aug
28 Aug
2:32 p.m.
On 2020-08-27 13:29, Marcel Kuiper wrote:
Sorry that had to be Wido/Stefan
What does "ceph osd df" give you? There is a column with "OMAP" and
"META". OMAP is ~ 13 B, META 26 GB in our setup. Quite a few files in
cephfs (main reason we have large OMAP).
Another question is: hoe to use this ceph-kvstore-tool tool to compact the
rocksdb? (can't find a lot of examples)
If you want to do a whole host a the time:
systemctl stop ceph-osd.target
wait a few seconds till all processes are closed.
for osd in `ls /var/lib/ceph/osd/`; do (ceph-kvstore-tool bluestore-kv
/var/lib/ceph/osd/$osd compact &);done
This works for us (no seperate WAL/DB). Check the help of the
ceph-kvstore-tool if you have to do anything special with separate DB /
WAL devices.
Gr. Stefan
4:30 p.m.
Thank you Stefan, I'm going to give that a try
Kind Regards
Marcel Kuiper
On 2020-08-27 13:29, Marcel Kuiper wrote:
Sorry that had to be Wido/Stefan
What does "ceph osd df" give you? There is a column with "OMAP" and
"META". OMAP is ~ 13 B, META 26 GB in our setup. Quite a few files in
cephfs (main reason we have large OMAP).
Another question is: hoe to use this ceph-kvstore-tool tool to compact
the
rocksdb? (can't find a lot of examples)
If you want to do a whole host a the time:
systemctl stop ceph-osd.target
wait a few seconds till all processes are closed.
for osd in `ls /var/lib/ceph/osd/`; do (ceph-kvstore-tool bluestore-kv
/var/lib/ceph/osd/$osd compact &);done
This works for us (no seperate WAL/DB). Check the help of the
ceph-kvstore-tool if you have to do anything special with separate DB /
WAL devices.
Gr. Stefan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
31 Aug
31 Aug
4:46 p.m.
The compaction of the bluestore-kv's helped indeed. The repons is back to
acceptable levels
Thanks for the help
Thank you Stefan, I'm going to give that a try
Kind Regards
Marcel Kuiper
On 2020-08-27 13:29, Marcel Kuiper wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Sorry that had to be Wido/Stefan
What does "ceph osd df" give you? There is a column with "OMAP" and
"META". OMAP is ~ 13 B, META 26 GB in our setup. Quite a few files in
cephfs (main reason we have large OMAP).
Another question is: hoe to use this ceph-kvstore-tool tool to compact
the
rocksdb? (can't find a lot of examples)
If you want to do a whole host a the time:
systemctl stop ceph-osd.target
wait a few seconds till all processes are closed.
for osd in `ls /var/lib/ceph/osd/`; do (ceph-kvstore-tool bluestore-kv
/var/lib/ceph/osd/$osd compact &);done
This works for us (no seperate WAL/DB). Check the help of the
ceph-kvstore-tool if you have to do anything special with separate DB /
WAL devices.
Gr. Stefan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1:21 p.m.
As a matter of fact we did. We doubled the storage nodes from 25 to 50.
Total osds now 460.
You want to share your thoughts on that?
Regards
Marcel
On 2020-08-31 14:16, Marcel Kuiper wrote:
The compaction of the bluestore-kv's helped
indeed. The repons is back
to
acceptable levels
Just curious. Did you do any cluster expansion and or PG expansion
before the slowness occurred?
Gr. Stefan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
2 Sep
2 Sep
2:37 p.m.
On 2020-09-01 10:51, Marcel Kuiper wrote:
As a matter of fact we did. We doubled the storage
nodes from 25 to 50.
Total osds now 460.
You want to share your thoughts on that?
Yes. We observed the same thing with expansions. The OSDs will be very
busy (with multiple threads per OSD) on housekeeping after the OMAP data
has been moved to another OSD (and eating up all CPU power it can get).
But even after that there is a lot of garbage left behind not gets
cleaned up. At least not with regular housekeeping / online compaction.
Manual compaction for clusters with a lot of OMAP data feels like a
necessity (and ideally shouln't be).
Gr. Stefan
3 Sep
3 Sep
2:20 a.m.
On 02/09/2020 12:07, Stefan Kooman wrote:
On 2020-09-01 10:51, Marcel Kuiper wrote:
Indeed, it shouldn't be.
This config option should make it easier in a future release:
https://github.com/ceph/ceph/commit/93e4c56ecc13560e0dad69aaa67afc3ca053fb4c
[osd]
osd_compact_on_start = true
Then just restart the OSDs and they will compact on boot. No need for
external scripts. Just put this into the ceph.conf.
The mon config store won't work as there is no connection with the
Monitors at that point in the code.
Wido
>
> Gr. Stefan
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
As a matter of fact we did. We doubled the
storage nodes from 25 to 50.
Total osds now 460.
You want to share your thoughts on that?
Yes. We observed the same thing with expansions. The OSDs will be very
busy (with multiple threads per OSD) on housekeeping after the OMAP data
has been moved to another OSD (and eating up all CPU power it can get).
But even after that there is a lot of garbage left behind not gets
cleaned up. At least not with regular housekeeping / online compaction.
Manual compaction for clusters with a lot of OMAP data feels like a
necessity (and ideally shouln't be).
11 a.m.
On 2020-09-02 23:50, Wido den Hollander wrote:
Indeed, it shouldn't be.
This config option should make it easier in a future release:
https://github.com/ceph/ceph/commit/93e4c56ecc13560e0dad69aaa67afc3ca053fb4c
[osd]
osd_compact_on_start = true
Then just restart the OSDs and they will compact on boot. No need for
external scripts. Just put this into the ceph.conf.
The mon config store won't work as there is no connection with the
Monitors at that point in the code.
Yay!
Thanks Wido
Gr. Stefan
10 Sep
10 Sep
7:45 p.m.
On 2020-09-01 10:51, Marcel Kuiper wrote:
As a matter of fact we did. We doubled the storage
nodes from 25 to 50.
Total osds now 460.
You want to share your thoughts on that?
OK, I'm really curious if you observed the following behaviour:
During, or shortly after the rebalance, did you see high CPU usage of
the OSDs? In particular the ones that hosted the PGs before they were
moved to the new nodes? As in ~ 300 % CPU per OSD (increasing from a few
percent to 300% non stop)? RocksDB is doing housekeeping, And we
observed before, and today again, on Mimic 13.2.8, that with a lot of
OMAP/META data the OSDs that have to clean up consume a ridiculous
amount of CPU (for hours on end). Triggering loads of slow ops and
latency spikes in the somtimes (tens) of seconds.
Are you running nautilus? If you haven't seen this behaviour this might
have been fixed in Nautlilus. Or you cluster is different from ours. We
will do PG expansion after we have upgraded to Nautilus, so we'll
definitely know by then.
Thanks,
Stefan
11 Sep
11 Sep
12:13 p.m.
Hi Stefan
I can't recall that that was the case and unfortunately we do not have
enough history for our performance measurements to look back
We are on nautilus. Please let me know your findings when you do your pg
expansion on nautilus
Grtz
Marcel
OK, I'm really curious if you observed the
following behaviour:
During, or shortly after the rebalance, did you see high CPU usage of
the OSDs? In particular the ones that hosted the PGs before they were
moved to the new nodes? As in ~ 300 % CPU per OSD (increasing from a few
percent to 300% non stop)? RocksDB is doing housekeeping, And we
observed before, and today again, on Mimic 13.2.8, that with a lot of
OMAP/META data the OSDs that have to clean up consume a ridiculous
amount of CPU (for hours on end). Triggering loads of slow ops and
latency spikes in the somtimes (tens) of seconds.
Are you running nautilus? If you haven't seen this behaviour this might
have been fixed in Nautlilus. Or you cluster is different from ours. We
will do PG expansion after we have upgraded to Nautilus, so we'll
definitely know by then.
Thanks,
Stefan
1317
days inactive
1333
days old
16 comments
4 participants
participants (4)
-
Marcel Kuiper -
Stefan Kooman -
Vladimir Sigunov -
Wido den Hollander