24 active+clean+snaptrim
I see snaptrimming happening in your status output - do you know if that was happening before restarting those OSDs? This is the mechanism by which OSDs clean up deleted snapshots, and once all OSDs have completed snaptrim for a given snapshot it should be removed from the removed_snaps_queue.
ceph version 16.2.7
You may want to consider upgrading. 16.2.8 has a fix for https://tracker.ceph.com/issues/52026, which is an issue that can cause snaptrim to not happen in some circumstances. Josh On Mon, Feb 12, 2024 at 6:00 PM localhost Liam <imluyuan@gmail.com> wrote:
Thanks, we are storing a lot less stress. 0. I rebooted 30 OSDs on one machine and the queue was not reduced, but the storage space was released in large amounts. 1. why did the reboot OSD release so much space?
Here are Ceph details..
ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)
cluster: id: 9acc3734-b27b-4bc3-84b8-c7762f2294c6 health: HEALTH_OK
services: mon: 3 daemons, quorum onf-akl-stor001,onf-akl-stor002,onf-akl-stor003 (age 11d) mgr: onf-akl-stor001(active, since 3M), standbys: onf-akl-stor002 osd: 101 osds: 98 up (since 41s), 98 in (since 11d) rgw: 2 daemons active (2 hosts, 1 zones)
data: pools: 7 pools, 2209 pgs objects: 25.47M objects, 58 TiB usage: 115 TiB used, 184 TiB / 299 TiB avail pgs: 2183 active+clean 24 active+clean+snaptrim 2 active+clean+scrubbing+deep
io: client: 38 MiB/s rd, 226 MiB/s wr, 1.32k op/s rd, 2.27k op/s wr _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io