Dear all, In the previous year, we were forced to migrate all data from one cluster due to a critical issue with snaptrimq length, which caused the cluster to become unstable. Unfortunately, we are now experiencing the same problem again. We've been flooded with a massive queue of snaptrims despite a massive lack of snapshot deletion. Within seconds, every PG received over 2 million snaps to delete. It looks like some kind of database corruption. 1.1 SNAPTRIMQ_LEN 2264067 Last Year it was Octopus cluster (15.2.17). Now it is cluster in Quincy (17.2.7). Originally cluster was created in version 14.2.16 and upgraded -> 15.2.17 -> 17.2.7. Below I attached background/history in chronological order from earliest. I have also attach logs from snaptrimming osd in text file. ---------- BACKGROUND: All was good:
2025-10-16T21:30:09.776167+0000 mgr.mgr-01-<cluster-fsid> (mgr.8232603159) 47659 : cluster [DBG] pgmap v22191: 8193 pgs: 2 active+clean+scrubbing+deep, 8191 active+clean; 183 TiB data, 542 TiB used, 419 TiB / 960 TiB avail; 491 MiB/s rd, 359 MiB/s wr, 20.99k op/s
First laggy pg:
2025-10-16T21:30:11.789813+0000 mgr.mgr-01-<cluster-fsid> (mgr.8232603159) 47661 : cluster [DBG] pgmap v22192: 8193 pgs: 4 active+clean+laggy, 2 active+clean+scrubbing+deep, 8187 active+clean; 183 TiB data, 542 TiB used, 419 TiB / 960 TiB avail; 328 MiB/s rd, 247 MiB/s wr, 14.01k op/s
First hearbeat problem:
2025-10-16T21:30:21.893+0000 7f578efd7700 -1 osd.yyy 602287 heartbeat_check: no reply from xyx.xyz.xyz.xyz:6802 osd.xxx since back 2025-10-16T21:29:57.827954+0000 front 2025-10-16T21:29:57.828552+0000 (oldest deadline 2025-10-16T21:30:21.328590+0000)
First report of failed osd:
2025-10-16T21:30:21.841714+0000 mon.mon-01-<cluster-fsid> (mon.0) 5235394 : cluster [DBG] osd.16 reported failed by osd.182
First failed osd:
2025-10-16T21:30:21.841681+0000 mon.mon-01-<cluster-fsid> (mon.0) 5235393 : cluster [INF] osd.15 failed (root=default,rack=rack123,host=host-123) (2 reporters from different host after 23.000420 >= grace 20.000000)
Host down:
2025-10-16T21:30:22.914804+0000 mon.mon-01-<cluster-fsid> (mon.0) 5235451 : cluster [WRN] Health check failed: 1 host (24 osds) down (OSD_HOST_DOWN)
Some osd up again:
2025-10-16T21:30:22.892201+0000 mon.mon-01-<cluster-fsid> (mon.0) 5235450 : cluster [DBG] osd.22 failure report canceled by osd.64 2025-10-16T21:30:21.904+0000 7f4c358ca700 1 mon.mon-04-<cluster-fsid>@1(peon).osd e602288 e602288: 283 total, 266 up, 283 in
Cluster during recovering:
2025-10-16T21:30:23.865481+0000 mgr.mgr-01-<cluster-fsid> (mgr.8232603159) 47668 : cluster [DBG] pgmap v22200: 8193 pgs: 34 peering, 648 stale+active+clean, 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 676 active+clean+laggy, 60 active+undersized+degraded, 6773 active+clean; 183 TiB data, 542 TiB used, 419 TiB / 960 TiB avail; 4.7 MiB/s rd, 45 MiB/s wr, 1.28k op/s; 372351/152450688 objects degraded (0.244%)
Again all down:
2025-10-16T21:30:23.936587+0000 mon.mon-01-<cluster-fsid> (mon.0) 5235453 : cluster [DBG] osdmap e602290: 283 total, 259 up, 283 in
... few osds were flapping up and down x few times (about 5) (259->266->259...->259) ... Finally problem was born:
2025-10-16T21:30:34.405+0000 7fe5613fc700 0 log_channel(cluster) log [WRN] : Health check failed: snap trim queue for 4133 pg(s) >= 1024 (mon_osd_snap_trim_queue_warn_on) (PG_SLOW_SNAP_TRIMMING)
During problem our monitors with joy were compacting rocksdb:
2025-10-16T21:30:26.224+0000 7f4c310c1700 4 rocksdb: [db/db_impl/db_impl_compaction_flush.cc:1615] [default] Manual compaction starting (mon-04-<cluster-fsid>.caas.ovh.net) 2025-10-16T21:30:26.820+0000 7f0fefa75700 4 rocksdb: [db/db_impl/db_impl_compaction_flush.cc:1615] [default] Manual compaction starting (mon-05-<cluster-fsid>.caas.ovh.net)
Of course we ended with 2M+ snaps to remove on every pg. Additionally after few minutes we saw such operation on mon:
2025-10-16T21:37:11.434+0000 7fe5613fc700 -1 mon.mon-01-<cluster-fsid>@0(leader) e10 get_health_metrics reporting 1 slow ops, oldest is pool_op(delete unmanaged snap pool 1 tid 7113677 name v602484)
Do you have any idea what might be the reason for this issue and how to address it? Have you encountered a similar problem elsewhere? Currently, we are running snaptrim, but it may take several weeks to complete, and during this time, the cluster is severely impacted. Maybe do You have any idea what to check next, where can we found a clue. Thank You for any help, Best Regards, Lukasz Lucki Gomulka
Hi, are you using mclock scheduler (default in Quincy)? Until Reef 18.2.4 there was a default value set for osd_snap_trim_cost (1M bytes) which blocked snaptrims [0]. This was fixed in [1] and backported to Reef. But it's unlikely that this was your issue in Octopus as mclock became the default in Pacific, IIRC. Since Quincy is also EOL, I'd recommed to update further, if possible. Were you able to avoid OSD flapping with the nodown flag (ceph osd set nodown)? This can help to keep the cluster more stable in such situations. Can you add some more details about your setup like: ceph -s ceph osd df tree ceph osd pool ls detail ceph df Are you using HDD OSDs or HDDs with dedicated DB/WAL? How many snapshots are you generating? Regards, Eugen [0] https://tracker.ceph.com/issues/67702 [1] https://tracker.ceph.com/issues/63604
Hello! Thank You for Your answer, Right now we have problems with cluster on NVMe so DB naturally is also on NVMe. Last time we have issue on HDD+NVMe(for DB/WAL). Now we have mClock scheduler. Generally we are able to snaptrim but it cost CPU resources. We can snaptrim thousands of snapshots per day. But the main problem and question still remain. Why we got without any reason 2.5M of snaps to remove. What should we do. How to prevent. It is a chance that if 2.5M of snaps came in seconds then it is possible to remove it in few seconds. Some db corruption maybe easy to clean. Current cluster: version - 17.2.7 mclock scheduler I attached also outputs from: ceph -s ceph osd df tree ceph osd pool ls detail ceph df I have to add that in json there is no entry in removed_snaps but in normal output in removed_snaps_queue we can see:
pool 1 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8192 pgp_num 8192 autoscale_mode off last_change 667587 lfor 0/0/1211 > flags hashpspool,selfmanaged_snaps stripe_width 0 target_size_ratio 0.5 application rbd removed_snaps_queue [58ae2~3,58af4~3,58cd5~5,58cdd~3 ... million+ ... c59ae~3,4c59b6~3,4c59bc~3]
Similar malformed output is in attached `ceph df`. Pool of course have still available space but commands shows `0%`. Best Regards, Lukasz Lucki Gomulka ________________________________ From: Eugen Block <eblock@nde.ag> Sent: 03 November 2025 11:53:06 To: ceph-users@ceph.io Subject: [ceph-users] Re: Snaptrim flood Hi, are you using mclock scheduler (default in Quincy)? Until Reef 18.2.4 there was a default value set for osd_snap_trim_cost (1M bytes) which blocked snaptrims [0]. This was fixed in [1] and backported to Reef. But it's unlikely that this was your issue in Octopus as mclock became the default in Pacific, IIRC. Since Quincy is also EOL, I'd recommed to update further, if possible. Were you able to avoid OSD flapping with the nodown flag (ceph osd set nodown)? This can help to keep the cluster more stable in such situations. Can you add some more details about your setup like: ceph -s ceph osd df tree ceph osd pool ls detail ceph df Are you using HDD OSDs or HDDs with dedicated DB/WAL? How many snapshots are you generating? Regards, Eugen [0] https://tracker.ceph.com/issues/67702 [1] https://tracker.ceph.com/issues/63604 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I don’t have much time right now to look deeper, but I agree, the 100% used column is something to look into. That might either be the root cause for snaptrims not happening or it might play some role in it. Zitat von Lukasz Gomulka <lukasz.gomulka@ovhcloud.com>:
Hello! Thank You for Your answer, Right now we have problems with cluster on NVMe so DB naturally is also on NVMe. Last time we have issue on HDD+NVMe(for DB/WAL). Now we have mClock scheduler. Generally we are able to snaptrim but it cost CPU resources. We can snaptrim thousands of snapshots per day. But the main problem and question still remain. Why we got without any reason 2.5M of snaps to remove. What should we do. How to prevent. It is a chance that if 2.5M of snaps came in seconds then it is possible to remove it in few seconds. Some db corruption maybe easy to clean. Current cluster:
version - 17.2.7 mclock scheduler
I attached also outputs from:
ceph -s ceph osd df tree ceph osd pool ls detail ceph df
I have to add that in json there is no entry in removed_snaps but in normal output in removed_snaps_queue we can see:
pool 1 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8192 pgp_num 8192 autoscale_mode off last_change 667587 lfor 0/0/1211 > flags hashpspool,selfmanaged_snaps stripe_width 0 target_size_ratio 0.5 application rbd removed_snaps_queue [58ae2~3,58af4~3,58cd5~5,58cdd~3 ... million+ ... c59ae~3,4c59b6~3,4c59bc~3]
Similar malformed output is in attached `ceph df`. Pool of course have still available space but commands shows `0%`.
Best Regards, Lukasz Lucki Gomulka
________________________________ From: Eugen Block <eblock@nde.ag> Sent: 03 November 2025 11:53:06 To: ceph-users@ceph.io Subject: [ceph-users] Re: Snaptrim flood
Hi,
are you using mclock scheduler (default in Quincy)? Until Reef 18.2.4 there was a default value set for osd_snap_trim_cost (1M bytes) which blocked snaptrims [0]. This was fixed in [1] and backported to Reef. But it's unlikely that this was your issue in Octopus as mclock became the default in Pacific, IIRC. Since Quincy is also EOL, I'd recommed to update further, if possible. Were you able to avoid OSD flapping with the nodown flag (ceph osd set nodown)? This can help to keep the cluster more stable in such situations. Can you add some more details about your setup like:
ceph -s ceph osd df tree ceph osd pool ls detail ceph df
Are you using HDD OSDs or HDDs with dedicated DB/WAL? How many snapshots are you generating?
Regards, Eugen
[0] https://tracker.ceph.com/issues/67702 [1] https://tracker.ceph.com/issues/63604 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I think there's some more investigation required to get to the bottom of this. Do you by any chance have a snap schedule enabled which would create snapshots automatically? Do you see that many snapshots in 'rbd -p pool ls --long' output? By the way, having millions of purged snapshots can have quite a heavy impact (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YRY2CGWSFHTE...). I'm still not sure what to think about the 100% used output, though. Zitat von Eugen Block <eblock@nde.ag>:
I don’t have much time right now to look deeper, but I agree, the 100% used column is something to look into. That might either be the root cause for snaptrims not happening or it might play some role in it.
Zitat von Lukasz Gomulka <lukasz.gomulka@ovhcloud.com>:
Hello! Thank You for Your answer, Right now we have problems with cluster on NVMe so DB naturally is also on NVMe. Last time we have issue on HDD+NVMe(for DB/WAL). Now we have mClock scheduler. Generally we are able to snaptrim but it cost CPU resources. We can snaptrim thousands of snapshots per day. But the main problem and question still remain. Why we got without any reason 2.5M of snaps to remove. What should we do. How to prevent. It is a chance that if 2.5M of snaps came in seconds then it is possible to remove it in few seconds. Some db corruption maybe easy to clean. Current cluster:
version - 17.2.7 mclock scheduler
I attached also outputs from:
ceph -s ceph osd df tree ceph osd pool ls detail ceph df
I have to add that in json there is no entry in removed_snaps but in normal output in removed_snaps_queue we can see:
pool 1 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8192 pgp_num 8192 autoscale_mode off last_change 667587 lfor 0/0/1211 > flags hashpspool,selfmanaged_snaps stripe_width 0 target_size_ratio 0.5 application rbd removed_snaps_queue [58ae2~3,58af4~3,58cd5~5,58cdd~3 ... million+ ... c59ae~3,4c59b6~3,4c59bc~3]
Similar malformed output is in attached `ceph df`. Pool of course have still available space but commands shows `0%`.
Best Regards, Lukasz Lucki Gomulka
________________________________ From: Eugen Block <eblock@nde.ag> Sent: 03 November 2025 11:53:06 To: ceph-users@ceph.io Subject: [ceph-users] Re: Snaptrim flood
Hi,
are you using mclock scheduler (default in Quincy)? Until Reef 18.2.4 there was a default value set for osd_snap_trim_cost (1M bytes) which blocked snaptrims [0]. This was fixed in [1] and backported to Reef. But it's unlikely that this was your issue in Octopus as mclock became the default in Pacific, IIRC. Since Quincy is also EOL, I'd recommed to update further, if possible. Were you able to avoid OSD flapping with the nodown flag (ceph osd set nodown)? This can help to keep the cluster more stable in such situations. Can you add some more details about your setup like:
ceph -s ceph osd df tree ceph osd pool ls detail ceph df
Are you using HDD OSDs or HDDs with dedicated DB/WAL? How many snapshots are you generating?
Regards, Eugen
[0] https://tracker.ceph.com/issues/67702 [1] https://tracker.ceph.com/issues/63604 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Eugen Block
-
Lukasz Gomulka