Insane number of "osd_snap purged_snap" keys in monstore db due to rbd-mirror
Hi, We have a customer with an abnormally large number of "osd_snap / purged_snap_{pool}_{snapid}" keys in monstore db: almost 40 million. Among other problems it causes a very long mon synchronization on startup. Our understanding is that the cause is that a mirroring snapshot creation is very frequently interrupted in their environment, most likely due to connectivity issues between the sites. The assumption is based on the fact that they have a lot of rbd "trash" snapshots, which may happen when an rbd snapshot removal is interrupted. (A mirroring snapshot creation usually includes removal of some older snapshot to keep the total number of the image mirroring snapshots under the limit). We removed all "trash" snapshots manually, so currently they have a limited number of "expected" snapshots but the number of purged_snap keys is still the same large. So, our understanding is that if an rbd snapshot creation is frequently interrupted there is a chance it will be interrupted in or just after SnapshotCreateRequest::send_allocate_snap_id [1], when it requests a new snap id from the mon. As a result this id is not tracked by rbd and never removed, and snap id holes like this make "purged_snap_{pool}_{snapid}" ranges never merge. To confirm that this scenario is likely I ran the following simple test that interrupted rbd mirror snapshot creation at random time: for i in `seq 500`;do rbd mirror image snapshot test& PID=$! sleep $((RANDOM % 5)).$((RANDOM % 10)) kill $PID && sleep 30 done Running this with debug_rbd=30, from the rbd client logs I see that it was interrupted in send_allocate_snap_id 74 times, which is (surprisingly) very high. And after the experiment, and after removing the rbd image with all tracked snapshots (i.e having the pool with no known rbd snapshots), I see "purged_snap_{pool}_{snapid}" keys for ranges that I believe will never be merged. So the questions are: 1) Is there a way we could improve this to avoid monstore growing large? 2) How can we fix the current situation in the cluster? Would it be safe enough to just run `ceph-kvstore-tool rocksdb store.db rm-prefix osd_snap` to remove all osd_snap keys (including purged_epoch keys)? Due to large db size I don't think it would be possible to selectively remove keys with `ceph-kvstore-tool rocksdb store.db rm {prefix} {key}` command and we may use only the `rm-prefix` command. Looking at the code and actually trying it in a test environment it seems like it could work, but I may be missing something dangerous here? If (1) is not possible, then maybe we could provide a tool/command for users to clean the keys if they observe this issue? [1] https://github.com/ceph/ceph/blob/e45272df047af71825445aeb6503073ba06123b0/s... Thanks, -- Mykola Golub
On Sat, Jul 29, 2023 at 2:11 PM Mykola Golub <to.my.trociny@gmail.com> wrote:
Hi,
We have a customer with an abnormally large number of "osd_snap / purged_snap_{pool}_{snapid}" keys in monstore db: almost 40 million. Among other problems it causes a very long mon synchronization on startup.
Our understanding is that the cause is that a mirroring snapshot creation is very frequently interrupted in their environment, most likely due to connectivity issues between the sites. The assumption is
Hi Mykola, I'm missing how connectivity issues between the sites can lead to mirror snapshot creation being interrupted on the primary cluster. Isn't that operation local to the cluster? Also, do you know who/what is actually interrupting it? Even if mirror snapshot creation ends up taking a while for some reason, I don't think anything in RBD would interrupt it.
based on the fact that they have a lot of rbd "trash" snapshots, which may happen when an rbd snapshot removal is interrupted. (A mirroring snapshot creation usually includes removal of some older snapshot to keep the total number of the image mirroring snapshots under the limit).
We removed all "trash" snapshots manually, so currently they have a limited number of "expected" snapshots but the number of purged_snap keys is still the same large.
So, our understanding is that if an rbd snapshot creation is frequently interrupted there is a chance it will be interrupted in or just after SnapshotCreateRequest::send_allocate_snap_id [1], when it requests a new snap id from the mon. As a result this id is not tracked by rbd and never removed, and snap id holes like this make "purged_snap_{pool}_{snapid}" ranges never merge.
To confirm that this scenario is likely I ran the following simple test that interrupted rbd mirror snapshot creation at random time:
for i in `seq 500`;do rbd mirror image snapshot test& PID=$! sleep $((RANDOM % 5)).$((RANDOM % 10)) kill $PID && sleep 30 done
Running this with debug_rbd=30, from the rbd client logs I see that it was interrupted in send_allocate_snap_id 74 times, which is (surprisingly) very high.
This applies to regular user (i.e. non-mirror) snapshots too.
And after the experiment, and after removing the rbd image with all tracked snapshots (i.e having the pool with no known rbd snapshots), I see "purged_snap_{pool}_{snapid}" keys for ranges that I believe will never be merged.
So the questions are:
1) Is there a way we could improve this to avoid monstore growing large?
Nothing simple comes to mind. The issue is that getting a snap ID on the monitor and registering a snapshot with the image on the OSDs are fundamentally separate steps, with the latter requiring a snap ID from the former. Unless the process of allocating a snap ID itself becomes two-step, where a freshly allocated snap ID is initially marked inactive and later, after it gets persisted, it's switched to active with a separate request to the monitor, one could always generate "forgotten" snap IDs if they try hard enough. (I'm assuming that in such a two-step process, monitors would clean up inactive snap IDs after a timeout.) In general, I don't think we are resilient to these scenarios. I suspect there are many similar "some piece of metadata is left behind if the command is killed at the wrong moment" issues lurking there.
2) How can we fix the current situation in the cluster? Would it be safe enough to just run `ceph-kvstore-tool rocksdb store.db rm-prefix osd_snap` to remove all osd_snap keys (including purged_epoch keys)? Due to large db size I don't think it would be possible to selectively remove keys with `ceph-kvstore-tool rocksdb store.db rm {prefix} {key}` command and we may use only the `rm-prefix` command. Looking at the code and actually trying it in a test environment it seems like it could work, but I may be missing something dangerous here?
Adding Radek. Thanks, Ilya
On Sun, Jul 30, 2023 at 7:09 PM Ilya Dryomov <idryomov@gmail.com> wrote:
I'm missing how connectivity issues between the sites can lead to mirror snapshot creation being interrupted on the primary cluster. Isn't that operation local to the cluster?
Also, do you know who/what is actually interrupting it? Even if mirror snapshot creation ends up taking a while for some reason, I don't think anything in RBD would interrupt it.
Indeed, just thinking more about it, it does not look like the connectivity issue that was interrupting it. The first thing noticed was a lot of "purge" (former mirroring) snapshots, so my logical assumption was that snapshot removal was interrupted and we also knew from the customer about "connectivity issues" between sites, so it was just my assumption that those interruptions were due to network issues. At first I was thinking that "snap id leak" might have happened on snapshot removal. And I used the test with creating primary snapshots because it was also removing snapshots. But reviewing the code I have not found any suspicious place where we could leak the snap id on snapshot removal, while the testing showed that it is rather possible to leak it on snapshot creation. So currently I think it happens on snapshot creation, just forgot to revisit my initial assumption what could cause the interruption. Ok, then another suspect could be the rbd_support mgr module. They are still running octopus (latest), there are more than 500 mirroring images, and the snapshot schedule was configured for 3 minutes for each image. I expect it could cause a considerable load and could trigger this interruption somehow. Could it be due to blacklisting? (Recently we added to rbd_support module the ability to restart rados when it is blacklisted). Now after our recommendation I believe they have the schedule changed to 30 minutes interval. Unfortunately the communication with the customer is troublesome, I get only limited secondhand information and there are a lot of assumptions here. Currently we are more interested in actually how to fix the large number of purged_snap keys in the monstore, still I thought it would be useful to report some details how it could happen.
This applies to regular user (i.e. non-mirror) snapshots too.
Sure, mirroring is just a case when you may hit it due to frequent use. Thanks, -- Mykola Golub
On Mon, Jul 31, 2023 at 9:50 AM Mykola Golub <to.my.trociny@gmail.com> wrote:
So currently I think it happens on snapshot creation, just forgot to revisit my initial assumption what could cause the interruption. Ok, then another suspect could be the rbd_support mgr module.
On the other hand, they have large number of purged_snap keys on both the primary and the secondary clusters. If it had been due the rbd_support mgr module I would have expected to see it on the primary cluster only (they do not do "two way" mirroring as far as I know). -- Mykola Golub
On Mon, Jul 31, 2023 at 9:01 AM Mykola Golub <to.my.trociny@gmail.com> wrote:
On Mon, Jul 31, 2023 at 9:50 AM Mykola Golub <to.my.trociny@gmail.com> wrote:
So currently I think it happens on snapshot creation, just forgot to revisit my initial assumption what could cause the interruption. Ok, then another suspect could be the rbd_support mgr module.
On the other hand, they have large number of purged_snap keys on both the primary and the secondary clusters. If it had been due the rbd_support mgr module I would have expected to see it on the primary cluster only (they do not do "two way" mirroring as far as I know).
Hi Mykola, Yeah, rbd_support module is just a fancy way of calling "rbd mirror image snapshot" command (rbd_mirror_image_create_snapshot API). There is nothing in the Python code itself that deals with snap IDs. Answering your other question: recovering from blocklisting in rbd_support module is very recent and hasn't been backported even to pacific [1], let alone octopus which went EOL a year ago. [1] https://github.com/ceph/ceph/pull/51464 Thanks, Ilya
On Sun, Jul 30, 2023 at 7:09 PM Ilya Dryomov <idryomov@gmail.com> wrote:
On Sat, Jul 29, 2023 at 2:11 PM Mykola Golub <to.my.trociny@gmail.com> wrote:
2) How can we fix the current situation in the cluster? Would it be safe enough to just run `ceph-kvstore-tool rocksdb store.db rm-prefix osd_snap` to remove all osd_snap keys (including purged_epoch keys)? Due to large db size I don't think it would be possible to selectively remove keys with `ceph-kvstore-tool rocksdb store.db rm {prefix} {key}` command and we may use only the `rm-prefix` command. Looking at the code and actually trying it in a test environment it seems like it could work, but I may be missing something dangerous here?
Adding Radek.
Trying to resurrect this thread. Actually this is the question we are most interested in, i.e how the osd_snap prefixed keys can be properly trimmed. Note, it is easy to write a script that would list all existing snapshots in the pool with `rbd snap ls --all {image}` command and then issue `remove_self_managed_snap` for all IDs that are not in this list. One thing is that it would run for an unrealistically long time. But another thing is that removing a snapshot may merge purged_snap ranges (reducing the number of purged_snap keys) but it will add purged_epoch key (which contains a list of snapshots removed for this epoch). And I don't see how the purged_epoch keys are supposed to be normally trimmed. It looks like we never trim these keys. -- Mykola Golub
participants (2)
-
Ilya Dryomov
-
Mykola Golub