Hey guys, I'm trying to figure out what's happening to my backup cluster that often grinds to a halt when cephfs automatically removes snapshots. Almost all OSD's go to 100% CPU, ceph complains about slow ops, and CephFS stops doing client i/o. I'm graphing the cumulative value of the snaptrimq_len value, and that slowly decreases over time. One night it takes an hour, but other days, like today, my cluster has been down for almost 20 hours, and I think we're half way. Funny thing is that in both cases, the snaptrimq_len value initially goes to the same value, around 3000, and then slowly decreases, but my guess is that the number of objects that need to be trimmed varies hugely every day. Is there a way to show the size of cephfs snapshots, or get the number of objects or bytes that need snaptrimming? Perhaps I can graph that and see where the differences are. That won't explain why my cluster bogs down, but at least it gives some visibility. Running 17.2.6 everywhere by the way. Angelo.
On Fri, Aug 4, 2023 at 5:41 PM Angelo Höngens <angelo@hongens.nl> wrote:
Hey guys,
I'm trying to figure out what's happening to my backup cluster that often grinds to a halt when cephfs automatically removes snapshots.
CephFS does not "automatically" remove snapshots. Do you mean the snap_schedule mgr module?
Almost all OSD's go to 100% CPU, ceph complains about slow ops, and CephFS stops doing client i/o.
What health warnings do you see? You can try configuring snap trim: https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/#confval-...
I'm graphing the cumulative value of the snaptrimq_len value, and that slowly decreases over time. One night it takes an hour, but other days, like today, my cluster has been down for almost 20 hours, and I think we're half way. Funny thing is that in both cases, the snaptrimq_len value initially goes to the same value, around 3000, and then slowly decreases, but my guess is that the number of objects that need to be trimmed varies hugely every day.
Is there a way to show the size of cephfs snapshots, or get the number of objects or bytes that need snaptrimming?
Unfortunately, no.
Perhaps I can graph that and see where the differences are.
That won't explain why my cluster bogs down, but at least it gives some visibility. Running 17.2.6 everywhere by the way.
Please let us know how configuring snaptrim helps or not. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
On 07/08/2023 18:04, Patrick Donnelly wrote:
I'm trying to figure out what's happening to my backup cluster that often grinds to a halt when cephfs automatically removes snapshots.
CephFS does not "automatically" remove snapshots. Do you mean the snap_schedule mgr module?
Yup.
Almost all OSD's go to 100% CPU, ceph complains about slow ops, and CephFS stops doing client i/o.
What health warnings do you see? You can try configuring snap trim:
https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/#confval-...
Mostly a looot of SLOW_OPS. And I guess as a result of that MDS_CLIENT_LATE_RELEASE, MDS_CLIENT_OLDEST_TID, MDS_SLOW_METADATA_IO, MDS_TRIM warnings.
That won't explain why my cluster bogs down, but at least it gives some visibility. Running 17.2.6 everywhere by the way.
Please let us know how configuring snaptrim helps or not.
When I set nosnaptrim, all I/O immediately restores. When I unset nosnaptrim, i/o stops again. One of the symptoms is that OSD's go to about 350% cpu per daemon. I got the feeling for a while that setting osd_snap_trim_sleep_ssd to 1 helped. I have 120 HDD osd's with wal/journal on ssd, does it even use this value? Everything seemed stable, but eventually another few days passed, and suddenly removing a snapshot brought the cluster down again. So I guess that wasn't the cause. Now what I'm trying to do is set osd_max_trimming_pgs to 0 for all disks, and slowly setting it to 1 for a few osd's. This seems to work for a while, but still it brings the cluster down every now and then, and if not, the cluster is so slow it's almost unusable. This whole troubleshooting process is taking weeks. I just noticed that when 'the problem occurs', a lot of OSD's on a host (15 osd's per host) start using a lot of CPU, even though for example only 3 OSD's on this machine have their osd_max_trimming_pgs set to 1, the rest to 0. Disk doesn't seem to be the bottleneck. Restarting the daemons seems to solve the problem for a while, although the high cpu usage pops up on a different osd node every time. I am at a loss here. I'm almost thinking it's some kind of bug in the osd daemons, but I have no idea how to troubleshoot this. Angelo.
Hi Angelo, was this cluster upgraded (major version upgrade) before these issues started? We observed that with certain paths of a major version upgrade and the only way to fix that was to re-deploy all OSDs step by step. You can try a rocks-DB compaction first. If that doesn't help, rebuilding the OSDs might be the only way out. You should also confirm that all ceph-daemons are on the same version and that require-osd-release is reporting the same major version as well: ceph report | jq '.osdmap.require_osd_release' Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Angelo Hongens <angelo@hongens.nl> Sent: Saturday, August 19, 2023 9:58 AM To: Patrick Donnelly Cc: ceph-users@ceph.io Subject: [ceph-users] Re: snaptrim number of objects On 07/08/2023 18:04, Patrick Donnelly wrote:
I'm trying to figure out what's happening to my backup cluster that often grinds to a halt when cephfs automatically removes snapshots.
CephFS does not "automatically" remove snapshots. Do you mean the snap_schedule mgr module?
Yup.
Almost all OSD's go to 100% CPU, ceph complains about slow ops, and CephFS stops doing client i/o.
What health warnings do you see? You can try configuring snap trim:
https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/#confval-...
Mostly a looot of SLOW_OPS. And I guess as a result of that MDS_CLIENT_LATE_RELEASE, MDS_CLIENT_OLDEST_TID, MDS_SLOW_METADATA_IO, MDS_TRIM warnings.
That won't explain why my cluster bogs down, but at least it gives some visibility. Running 17.2.6 everywhere by the way.
Please let us know how configuring snaptrim helps or not.
When I set nosnaptrim, all I/O immediately restores. When I unset nosnaptrim, i/o stops again. One of the symptoms is that OSD's go to about 350% cpu per daemon. I got the feeling for a while that setting osd_snap_trim_sleep_ssd to 1 helped. I have 120 HDD osd's with wal/journal on ssd, does it even use this value? Everything seemed stable, but eventually another few days passed, and suddenly removing a snapshot brought the cluster down again. So I guess that wasn't the cause. Now what I'm trying to do is set osd_max_trimming_pgs to 0 for all disks, and slowly setting it to 1 for a few osd's. This seems to work for a while, but still it brings the cluster down every now and then, and if not, the cluster is so slow it's almost unusable. This whole troubleshooting process is taking weeks. I just noticed that when 'the problem occurs', a lot of OSD's on a host (15 osd's per host) start using a lot of CPU, even though for example only 3 OSD's on this machine have their osd_max_trimming_pgs set to 1, the rest to 0. Disk doesn't seem to be the bottleneck. Restarting the daemons seems to solve the problem for a while, although the high cpu usage pops up on a different osd node every time. I am at a loss here. I'm almost thinking it's some kind of bug in the osd daemons, but I have no idea how to troubleshoot this. Angelo. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 21/08/2023 12:38, Frank Schilder wrote:
Hi Angelo,
was this cluster upgraded (major version upgrade) before these issues started? We observed that with certain paths of a major version upgrade and the only way to fix that was to re-deploy all OSDs step by step.
You can try a rocks-DB compaction first. If that doesn't help, rebuilding the OSDs might be the only way out.
You should also confirm that all ceph-daemons are on the same version and that require-osd-release is reporting the same major version as well:
ceph report | jq '.osdmap.require_osd_release'
Hey Frank, No, this cluster was clean installed with 17.2.6! All quincy. Angelo.
Hello, on my testcluster I played a bit with ceph quincy (17.2.6). I also see slow ops while deleting snapshots. With the previous major (pacific) this wasn't a issue. In my case this is related to the new mclock scheduler which is defaulted with quincy. With "ceph config set global osd_op_queue wpq". Thie issue is gone.(after restarting the OSDs of course). wpq was the previous default scheduler. Maybe this will help you. On the other hand, mclock shouldn't break down the cluster in this way. At least not with "high_client_ops" which I used. Maybe someone should have a look at this. Manuel On Fri, 4 Aug 2023 17:40:42 -0400 Angelo Höngens <angelo@hongens.nl> wrote:
Hey guys,
I'm trying to figure out what's happening to my backup cluster that often grinds to a halt when cephfs automatically removes snapshots. Almost all OSD's go to 100% CPU, ceph complains about slow ops, and CephFS stops doing client i/o.
I'm graphing the cumulative value of the snaptrimq_len value, and that slowly decreases over time. One night it takes an hour, but other days, like today, my cluster has been down for almost 20 hours, and I think we're half way. Funny thing is that in both cases, the snaptrimq_len value initially goes to the same value, around 3000, and then slowly decreases, but my guess is that the number of objects that need to be trimmed varies hugely every day.
Is there a way to show the size of cephfs snapshots, or get the number of objects or bytes that need snaptrimming? Perhaps I can graph that and see where the differences are.
That won't explain why my cluster bogs down, but at least it gives some visibility. Running 17.2.6 everywhere by the way.
Angelo. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 21/08/2023 16:47, Manuel Lausch wrote:
Hello,
on my testcluster I played a bit with ceph quincy (17.2.6). I also see slow ops while deleting snapshots. With the previous major (pacific) this wasn't a issue. In my case this is related to the new mclock scheduler which is defaulted with quincy. With "ceph config set global osd_op_queue wpq". Thie issue is gone.(after restarting the OSDs of course). wpq was the previous default scheduler.
Maybe this will help you.
On the other hand, mclock shouldn't break down the cluster in this way. At least not with "high_client_ops" which I used. Maybe someone should have a look at this.
Manuel
Hey Manuel, You made me a happy man (for now!) In short: wpq indeed seems to do waaaaay better in my setup. We did a lot of tuning with the mclock scheduler, tuned osd_mclock_max_capacity_iops_hdd, tried a lot of different settings for osd_snap_trim_sleep_hdd/ssd, etc, but it did not yet have the desired effect. The only thing that prevented my cluster from going down, was setting osd_max_trimming_pgs to 0 on all disks, and set it to 1 or 2 for a few OSD's at a time. As soon as I enable too many OSD's, everything would bog down, slow ops everywhere, hanging cephfs clients, etc. I think I could do a max of 100 objects/sec snaptrimming. I also played around with the different mclock profiles to speed up recovery. I think with high_client_ops, we got 400-600MB/s client io, and about 50MB/s recovery io (we had a few degraded objects and some rebalancing). With the high_recovery_ops profile I think I was able to get around 400-500MB/s client write, and 300MB/s recovery. As soon as I enabled snaptrimming, stuff would get quite a bit slower. At your suggestion I just changed the osd_op_queue to wpq, removed almost all other osd config variables and restarted all osd's. Now I see 400-600MB/s client i/o (normal) AND I see recovery at 1500MB/s(!) AND it's also snaptrimmimng at 250 objects/sec. And I haven't seen the first slow op warning yet! I'm still cautious, but for now, this looks very positive! This also leads me to agree with you there's 'something wrong' with the mclock scheduler. I was almost starting to suspect hardware issues or something like that, I was at my wit's end. Angelo. On Mon, Aug 21, 2023 at 4:49 PM Manuel Lausch <manuel.lausch@1und1.de> wrote:
Hello,
on my testcluster I played a bit with ceph quincy (17.2.6). I also see slow ops while deleting snapshots. With the previous major (pacific) this wasn't a issue. In my case this is related to the new mclock scheduler which is defaulted with quincy. With "ceph config set global osd_op_queue wpq". Thie issue is gone.(after restarting the OSDs of course). wpq was the previous default scheduler.
Maybe this will help you.
On the other hand, mclock shouldn't break down the cluster in this way. At least not with "high_client_ops" which I used. Maybe someone should have a look at this.
Manuel
On Fri, 4 Aug 2023 17:40:42 -0400 Angelo Höngens <angelo@hongens.nl> wrote:
Hey guys,
I'm trying to figure out what's happening to my backup cluster that often grinds to a halt when cephfs automatically removes snapshots. Almost all OSD's go to 100% CPU, ceph complains about slow ops, and CephFS stops doing client i/o.
I'm graphing the cumulative value of the snaptrimq_len value, and that slowly decreases over time. One night it takes an hour, but other days, like today, my cluster has been down for almost 20 hours, and I think we're half way. Funny thing is that in both cases, the snaptrimq_len value initially goes to the same value, around 3000, and then slowly decreases, but my guess is that the number of objects that need to be trimmed varies hugely every day.
Is there a way to show the size of cephfs snapshots, or get the number of objects or bytes that need snaptrimming? Perhaps I can graph that and see where the differences are.
That won't explain why my cluster bogs down, but at least it gives some visibility. Running 17.2.6 everywhere by the way.
Angelo. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 8/21/23 17:38, Angelo Höngens wrote:
On 21/08/2023 16:47, Manuel Lausch wrote:
Hello,
on my testcluster I played a bit with ceph quincy (17.2.6). I also see slow ops while deleting snapshots. With the previous major (pacific) this wasn't a issue. In my case this is related to the new mclock scheduler which is defaulted with quincy. With "ceph config set global osd_op_queue wpq". Thie issue is gone.(after restarting the OSDs of course). wpq was the previous default scheduler.
Maybe this will help you.
On the other hand, mclock shouldn't break down the cluster in this way. At least not with "high_client_ops" which I used. Maybe someone should have a look at this.
Manuel Hey Manuel,
You made me a happy man (for now!)
In short: wpq indeed seems to do waaaaay better in my setup.
We did a lot of tuning with the mclock scheduler, tuned osd_mclock_max_capacity_iops_hdd, tried a lot of different settings for osd_snap_trim_sleep_hdd/ssd, etc, but it did not yet have the desired effect. The only thing that prevented my cluster from going down, was setting osd_max_trimming_pgs to 0 on all disks, and set it to 1 or 2 for a few OSD's at a time. As soon as I enable too many OSD's, everything would bog down, slow ops everywhere, hanging cephfs clients, etc. I think I could do a max of 100 objects/sec snaptrimming.
I also played around with the different mclock profiles to speed up recovery. I think with high_client_ops, we got 400-600MB/s client io, and about 50MB/s recovery io (we had a few degraded objects and some rebalancing). With the high_recovery_ops profile I think I was able to get around 400-500MB/s client write, and 300MB/s recovery. As soon as I enabled snaptrimming, stuff would get quite a bit slower.
At your suggestion I just changed the osd_op_queue to wpq, removed almost all other osd config variables and restarted all osd's.
Now I see 400-600MB/s client i/o (normal) AND I see recovery at 1500MB/s(!) AND it's also snaptrimmimng at 250 objects/sec. And I haven't seen the first slow op warning yet!
I'm still cautious, but for now, this looks very positive!
This also leads me to agree with you there's 'something wrong' with the mclock scheduler. I was almost starting to suspect hardware issues or something like that, I was at my wit's end.
Angelo.
If you have the inclination, I would also be very curious if enabling this helps: "rocksdb_cf_compact_on_deletion" This is a new feature we added in reef and backported to quincy/pacific in a disabled state that issues compaction if too many tombstones are encountered during iteration in RocksDB. You can control how quickly to issue compactions using: "rocksdb_cf_compact_on_deletion_trigger" (default is 16384, shrink to increase compaction frequency) "rocksdb_cf_compact_on_deletion_sliding_window" (default is 32768, grow to increase compaction frequency) The combination of these two parameters dictates how many X tombstones you must encounter over Y keys before triggering a compaction. The default is pretty conservative so you may need to play with it if you are hitting too many tombstones. If compactions are trigger too frequently you can increase the number of X allowed tombstones per Y keys. Mark
On Mon, Aug 21, 2023 at 4:49 PM Manuel Lausch <manuel.lausch@1und1.de> wrote:
Hello,
on my testcluster I played a bit with ceph quincy (17.2.6). I also see slow ops while deleting snapshots. With the previous major (pacific) this wasn't a issue. In my case this is related to the new mclock scheduler which is defaulted with quincy. With "ceph config set global osd_op_queue wpq". Thie issue is gone.(after restarting the OSDs of course). wpq was the previous default scheduler.
Maybe this will help you.
On the other hand, mclock shouldn't break down the cluster in this way. At least not with "high_client_ops" which I used. Maybe someone should have a look at this.
Manuel
On Fri, 4 Aug 2023 17:40:42 -0400 Angelo Höngens <angelo@hongens.nl> wrote:
Hey guys,
I'm trying to figure out what's happening to my backup cluster that often grinds to a halt when cephfs automatically removes snapshots. Almost all OSD's go to 100% CPU, ceph complains about slow ops, and CephFS stops doing client i/o.
I'm graphing the cumulative value of the snaptrimq_len value, and that slowly decreases over time. One night it takes an hour, but other days, like today, my cluster has been down for almost 20 hours, and I think we're half way. Funny thing is that in both cases, the snaptrimq_len value initially goes to the same value, around 3000, and then slowly decreases, but my guess is that the number of objects that need to be trimmed varies hugely every day.
Is there a way to show the size of cephfs snapshots, or get the number of objects or bytes that need snaptrimming? Perhaps I can graph that and see where the differences are.
That won't explain why my cluster bogs down, but at least it gives some visibility. Running 17.2.6 everywhere by the way.
Angelo. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Best Regards, Mark Nelson Head of R&D (USA) Clyso GmbH p: +49 89 21552391 12 a: Loristraße 8 | 80335 München | Germany w: https://clyso.com | e: mark.nelson@clyso.com We are hiring: https://www.clyso.com/jobs/
This also leads me to agree with you there's 'something wrong' with the mclock scheduler. I was almost starting to suspect hardware issues or something like that, I was at my wit's end.
Could you update this thread with the exact quincy version by running: $ ceph versions and $ ceph config show-with-defaults osd.N | grep osd_mclock Please replace N with any valid OSD id. I suspect that the quincy version you are running on doesn't have the latest changes we made to the Reef upstream release. Recent changes introduced significant improvements to the mClock profiles and address slow recovery/backfill rates. The improvements to the mClock profiles should also help throttle snaptrim operations. Snaptrim operation with mClock currently uses a static cost as defined by osd_snap_trim_cost. There are improvements planned around this soon. For e.g., the cost must be dynamic and reflect the size of the object being trimmed. -Sridhar
On 23/08/2023 08:27, Sridhar Seshasayee wrote:
This also leads me to agree with you there's 'something wrong' with the mclock scheduler. I was almost starting to suspect hardware issues or something like that, I was at my wit's end.
Could you update this thread with the exact quincy version by running:
$ ceph versions
and
$ ceph config show-with-defaults osd.N | grep osd_mclock
Please replace N with any valid OSD id.
I suspect that the quincy version you are running on doesn't have the latest changes we made to the Reef upstream release. Recent changes introduced significant improvements to the mClock profiles and address slow recovery/backfill rates. The improvements to the mClock profiles should also help throttle snaptrim operations.
Snaptrim operation with mClock currently uses a static cost as defined by osd_snap_trim_cost. There are improvements planned around this soon. For e.g., the cost must be dynamic and reflect the size of the object being trimmed. -Sridhar
Here's the requested info, even though I'm going to stay on wpq for a while. # ceph versions { "mon": { "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 3 }, "mgr": { "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 3 }, "osd": { "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 117 }, "mds": { "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 3 }, "overall": { "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 126 } } I'm using the docker image registry.<company>.local/quay-proxy/ceph/ceph@sha256:673b48521fd53e1b4bc7dda96335505c4d4b2e13d7bb92bf2e7782e2083094c9. # ceph config show-with-defaults osd.0 | grep osd_mclock osd_mclock_cost_per_byte_usec 0.000000 default osd_mclock_cost_per_byte_usec_hdd 2.600000 default osd_mclock_cost_per_byte_usec_ssd 0.011000 default osd_mclock_cost_per_io_usec 0.000000 default osd_mclock_cost_per_io_usec_hdd 11400.000000 default osd_mclock_cost_per_io_usec_ssd 50.000000 default osd_mclock_force_run_benchmark_on_init false default osd_mclock_iops_capacity_threshold_hdd 500.000000 default osd_mclock_iops_capacity_threshold_ssd 80000.000000 default osd_mclock_max_capacity_iops_hdd 250.000000 mon osd_mclock_max_capacity_iops_ssd 21500.000000 default osd_mclock_override_recovery_settings true mon osd_mclock_profile high_client_ops mon osd_mclock_scheduler_anticipation_timeout 0.000000 default osd_mclock_scheduler_background_best_effort_lim 999999 default osd_mclock_scheduler_background_best_effort_res 1 default osd_mclock_scheduler_background_best_effort_wgt 1 default osd_mclock_scheduler_background_recovery_lim 999999 default osd_mclock_scheduler_background_recovery_res 1 default osd_mclock_scheduler_background_recovery_wgt 1 default osd_mclock_scheduler_client_lim 999999 default osd_mclock_scheduler_client_res 1 default osd_mclock_scheduler_client_wgt 1 default osd_mclock_skip_benchmark false default # ceph config show-with-defaults osd.0 | grep trim_cost osd_snap_trim_cost 1048576 default Angelo.
participants (7)
-
Angelo Hongens
-
Angelo Höngens
-
Frank Schilder
-
Manuel Lausch
-
Mark Nelson
-
Patrick Donnelly
-
Sridhar Seshasayee