On Sun, Jul 30, 2023 at 7:09 PM Ilya Dryomov <idryomov@gmail.com> wrote:
On Sat, Jul 29, 2023 at 2:11 PM Mykola Golub <to.my.trociny@gmail.com> wrote:
2) How can we fix the current situation in the cluster? Would it be safe enough to just run `ceph-kvstore-tool rocksdb store.db rm-prefix osd_snap` to remove all osd_snap keys (including purged_epoch keys)? Due to large db size I don't think it would be possible to selectively remove keys with `ceph-kvstore-tool rocksdb store.db rm {prefix} {key}` command and we may use only the `rm-prefix` command. Looking at the code and actually trying it in a test environment it seems like it could work, but I may be missing something dangerous here?
Adding Radek.
Trying to resurrect this thread. Actually this is the question we are most interested in, i.e how the osd_snap prefixed keys can be properly trimmed. Note, it is easy to write a script that would list all existing snapshots in the pool with `rbd snap ls --all {image}` command and then issue `remove_self_managed_snap` for all IDs that are not in this list. One thing is that it would run for an unrealistically long time. But another thing is that removing a snapshot may merge purged_snap ranges (reducing the number of purged_snap keys) but it will add purged_epoch key (which contains a list of snapshots removed for this epoch). And I don't see how the purged_epoch keys are supposed to be normally trimmed. It looks like we never trim these keys. -- Mykola Golub