Good morning, I would be grateful if anybody could shed some light on this, I can't reproduce it in my lab clusters so I was hoping for the community. A customer has 2 clusters with rbd mirroring (snapshots) enabled, it seems to work fine, they have regular checks and the images on the remote site are synced correctly. While investigating a different issue (could be related though) we noticed that there are quite a lot of snapshots in the trash namespace but for unkown reasons they are not purged. This is some example output from the remote site (the primary site looks similar): ---snip--- # rbd snap ls --all <pool>/<image> SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 23962608 42c2e4c3-e1ab-480f-9d29-aab5555c751b 20 GiB Tue Sep 20 20:33:18 2022 trash (.mirror.non_primary.61ef3a1e-6e5b-4147-ac62-552e4776dd70.ef345a74-2585-48ac-8e25-09b15c20c877) 96796437 24cc96d4-51c6-447e-870d-1545d8ec9308 20 GiB Tue Apr 18 12:15:28 2023 trash (.mirror.non_primary.61ef3a1e-6e5b-4147-ac62-552e4776dd70.a6ae46e4-cbb5-4264-b2e4-6b565820aa16) 110025019 .mirror.non_primary.61ef3a1e-6e5b-4147-ac62-552e4776dd70.0fc73e88-6671-4ba8-a118-8fcbcd422c22 20 GiB Mon May 15 16:27:39 2023 mirror (non-primary peer_uuids:[] 18ac37b9-a6c6-4f64-ba43-a1f0d02b3f96:115986586 copied) ---snip--- They're on latest Octopus and use all config defaults. There are more than 500 images in that pool with a snapshot schedule of 3 minutes for each images. A couple of questions arised and hopefully at least some of them can be answered: - Why are there snapshots in the trash namespace from September 2022, how can they be cleaned up and why aren't they cleaned up automatically? Not all images have trash entries though. - Could disabling mirroring for those images help getting rid of the trash, then enable mirroring again? - Is the default osd_max_trimming_pgs = 2 a bottleneck here? They have a week old sst files in the store.db, leading to long startup duration for MONs after reboot/restart. Would increasing that value help getting rid of the trash entries and maybe also trim the mon store? - Regarding the general snapshot mirroring procedure, the default of rbd_mirroring_max_mirroring_snapshots is 5, but I assume that the number of active snapshots would only grow in case of a disruption between those sites, correct? If the sync works there's no need to keep 5 snapshots, right? I'm still looking into these things myself but I'd appreciate anyone chiming in here. Thanks! Eugen