Accumulation of removed_snaps_queue After Deleting Snapshots in Ceph RBD
Hello, I'm encountering an issue with Ceph when using it as the backend storage for OpenStack Cinder. Specifically, after deleting RBD snapshots through Cinder, I've noticed a significant increase in the removed_snaps_queue entries within the corresponding Ceph pool. It seems to affect the pool's performance and space efficiency. I understand that snapshot deletion in Cinder is an asynchronous operation, and Ceph itself uses a lazy deletion mechanism to handle snapshot removal. However, even after allowing sufficient time, the entries in removed_snaps_queue do not decrease as expected. I have several questions for the community: Are there recommended methods or best practices for managing or reducing entries in removed_snaps_queue? Is there any tool or command that can safely clear these residual snapshot entries without affecting the integrity of active snapshots and data? Is this issue known, and are there any bug reports or plans for fixes related to it? Thank you very much for your assistance!
Hello, Which version of Ceph are you using? Are all of your OSDs currently up+in? If you're HEALTH_OK and all OSDs are up, snaptrim should work through the removed_snaps_queue and clear it over time, but I have seen cases where this seems to get stuck and restarting OSDs can help. Josh On Wed, Feb 7, 2024 at 12:01 PM localhost Liam <imluyuan@gmail.com> wrote:
Hello,
I'm encountering an issue with Ceph when using it as the backend storage for OpenStack Cinder. Specifically, after deleting RBD snapshots through Cinder, I've noticed a significant increase in the removed_snaps_queue entries within the corresponding Ceph pool. It seems to affect the pool's performance and space efficiency.
I understand that snapshot deletion in Cinder is an asynchronous operation, and Ceph itself uses a lazy deletion mechanism to handle snapshot removal. However, even after allowing sufficient time, the entries in removed_snaps_queue do not decrease as expected.
I have several questions for the community:
Are there recommended methods or best practices for managing or reducing entries in removed_snaps_queue? Is there any tool or command that can safely clear these residual snapshot entries without affecting the integrity of active snapshots and data? Is this issue known, and are there any bug reports or plans for fixes related to it? Thank you very much for your assistance! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks, we are storing a lot less stress. 0. I rebooted 30 OSDs on one machine and the queue was not reduced, but the storage space was released in large amounts. 1. why did the reboot OSD release so much space? Here are Ceph details.. ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable) cluster: id: 9acc3734-b27b-4bc3-84b8-c7762f2294c6 health: HEALTH_OK services: mon: 3 daemons, quorum onf-akl-stor001,onf-akl-stor002,onf-akl-stor003 (age 11d) mgr: onf-akl-stor001(active, since 3M), standbys: onf-akl-stor002 osd: 101 osds: 98 up (since 41s), 98 in (since 11d) rgw: 2 daemons active (2 hosts, 1 zones) data: pools: 7 pools, 2209 pgs objects: 25.47M objects, 58 TiB usage: 115 TiB used, 184 TiB / 299 TiB avail pgs: 2183 active+clean 24 active+clean+snaptrim 2 active+clean+scrubbing+deep io: client: 38 MiB/s rd, 226 MiB/s wr, 1.32k op/s rd, 2.27k op/s wr
24 active+clean+snaptrim
I see snaptrimming happening in your status output - do you know if that was happening before restarting those OSDs? This is the mechanism by which OSDs clean up deleted snapshots, and once all OSDs have completed snaptrim for a given snapshot it should be removed from the removed_snaps_queue.
ceph version 16.2.7
You may want to consider upgrading. 16.2.8 has a fix for https://tracker.ceph.com/issues/52026, which is an issue that can cause snaptrim to not happen in some circumstances. Josh On Mon, Feb 12, 2024 at 6:00 PM localhost Liam <imluyuan@gmail.com> wrote:
Thanks, we are storing a lot less stress. 0. I rebooted 30 OSDs on one machine and the queue was not reduced, but the storage space was released in large amounts. 1. why did the reboot OSD release so much space?
Here are Ceph details..
ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)
cluster: id: 9acc3734-b27b-4bc3-84b8-c7762f2294c6 health: HEALTH_OK
services: mon: 3 daemons, quorum onf-akl-stor001,onf-akl-stor002,onf-akl-stor003 (age 11d) mgr: onf-akl-stor001(active, since 3M), standbys: onf-akl-stor002 osd: 101 osds: 98 up (since 41s), 98 in (since 11d) rgw: 2 daemons active (2 hosts, 1 zones)
data: pools: 7 pools, 2209 pgs objects: 25.47M objects, 58 TiB usage: 115 TiB used, 184 TiB / 299 TiB avail pgs: 2183 active+clean 24 active+clean+snaptrim 2 active+clean+scrubbing+deep
io: client: 38 MiB/s rd, 226 MiB/s wr, 1.32k op/s rd, 2.27k op/s wr _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Josh Baergen
-
localhost Liam