29 Jul
2023
29 Jul
'23
5:10 a.m.
Hi,
We have a customer with an abnormally large number of "osd_snap /
purged_snap_{pool}_{snapid}" keys in monstore db: almost 40
million. Among other problems it causes a very long mon
synchronization on startup.
Our understanding is that the cause is that a mirroring snapshot
creation is very frequently interrupted in their environment, most
likely due to connectivity issues between the sites. The assumption is
based on the fact that they have a lot of rbd "trash" snapshots, which
may happen when an rbd snapshot removal is interrupted. (A mirroring
snapshot creation usually includes removal of some older snapshot to keep
the total number of the image mirroring snapshots under the limit).
We removed all "trash" snapshots manually, so currently they have a
limited number of "expected" snapshots but the number of purged_snap
keys is still the same large.
So, our understanding is that if an rbd snapshot creation is
frequently interrupted there is a chance it will be interrupted in or
just after SnapshotCreateRequest::send_allocate_snap_id [1], when
it requests a new snap id from the mon. As a result this id is not
tracked by rbd and never removed, and snap id holes like this make
"purged_snap_{pool}_{snapid}" ranges never merge.
To confirm that this scenario is likely I ran the following simple test
that interrupted rbd mirror snapshot creation at random time:
for i in `seq 500`;do
rbd mirror image snapshot test&
PID=$!
sleep $((RANDOM % 5)).$((RANDOM % 10))
kill $PID && sleep 30
done
Running this with debug_rbd=30, from the rbd client logs I see that it
was interrupted in send_allocate_snap_id 74 times, which is (surprisingly)
very high.
And after the experiment, and after removing the rbd image with all
tracked snapshots (i.e having the pool with no known rbd snapshots),
I see "purged_snap_{pool}_{snapid}" keys for ranges that I believe will
never be merged.
So the questions are:
1) Is there a way we could improve this to avoid monstore growing large?
2) How can we fix the current situation in the cluster? Would it be safe
enough to just run `ceph-kvstore-tool rocksdb store.db rm-prefix osd_snap`
to remove all osd_snap keys (including purged_epoch keys)? Due to
large db size I don't think it would be possible to selectively remove
keys with `ceph-kvstore-tool rocksdb store.db rm {prefix} {key}`
command and we may use only the `rm-prefix` command. Looking at the
code and actually trying it in a test environment it seems like it could
work, but I may be missing something dangerous here?
If (1) is not possible, then maybe we could provide a tool/command
for users to clean the keys if they observe this issue?
[1]
https://github.com/ceph/ceph/blob/e45272df047af71825445aeb6503073ba06123b0/…
Thanks,
--
Mykola Golub
30 Jul
30 Jul
9:09 a.m.
On Sat, Jul 29, 2023 at 2:11 PM Mykola Golub <to.my.trociny(a)gmail.com> wrote:
Hi,
We have a customer with an abnormally large number of "osd_snap /
purged_snap_{pool}_{snapid}" keys in monstore db: almost 40
million. Among other problems it causes a very long mon
synchronization on startup.
Our understanding is that the cause is that a mirroring snapshot
creation is very frequently interrupted in their environment, most
likely due to connectivity issues between the sites. The assumption is
Hi Mykola,
I'm missing how connectivity issues between the sites can lead to
mirror snapshot creation being interrupted on the primary cluster.
Isn't that operation local to the cluster?
Also, do you know who/what is actually interrupting it? Even if mirror
snapshot creation ends up taking a while for some reason, I don't think
anything in RBD would interrupt it.
based on the fact that they have a lot of rbd
"trash" snapshots, which
may happen when an rbd snapshot removal is interrupted. (A mirroring
snapshot creation usually includes removal of some older snapshot to keep
the total number of the image mirroring snapshots under the limit).
We removed all "trash" snapshots manually, so currently they have a
limited number of "expected" snapshots but the number of purged_snap
keys is still the same large.
So, our understanding is that if an rbd snapshot creation is
frequently interrupted there is a chance it will be interrupted in or
just after SnapshotCreateRequest::send_allocate_snap_id [1], when
it requests a new snap id from the mon. As a result this id is not
tracked by rbd and never removed, and snap id holes like this make
"purged_snap_{pool}_{snapid}" ranges never merge.
To confirm that this scenario is likely I ran the following simple test
that interrupted rbd mirror snapshot creation at random time:
for i in `seq 500`;do
rbd mirror image snapshot test&
PID=$!
sleep $((RANDOM % 5)).$((RANDOM % 10))
kill $PID && sleep 30
done
Running this with debug_rbd=30, from the rbd client logs I see that it
was interrupted in send_allocate_snap_id 74 times, which is (surprisingly)
very high.
This applies to regular user (i.e. non-mirror) snapshots too.
And after the experiment, and after removing the rbd image with all
tracked snapshots (i.e having the pool with no known rbd snapshots),
I see "purged_snap_{pool}_{snapid}" keys for ranges that I believe will
never be merged.
So the questions are:
1) Is there a way we could improve this to avoid monstore growing large?
Nothing simple comes to mind. The issue is that getting a snap ID on
the monitor and registering a snapshot with the image on the OSDs are
fundamentally separate steps, with the latter requiring a snap ID from
the former. Unless the process of allocating a snap ID itself becomes
two-step, where a freshly allocated snap ID is initially marked
inactive and later, after it gets persisted, it's switched to active
with a separate request to the monitor, one could always generate
"forgotten" snap IDs if they try hard enough. (I'm assuming that in
such a two-step process, monitors would clean up inactive snap IDs
after a timeout.)
In general, I don't think we are resilient to these scenarios.
I suspect there are many similar "some piece of metadata is left behind
if the command is killed at the wrong moment" issues lurking there.
2) How can we fix the current situation in the cluster? Would it be safe
enough to just run `ceph-kvstore-tool rocksdb store.db rm-prefix osd_snap`
to remove all osd_snap keys (including purged_epoch keys)? Due to
large db size I don't think it would be possible to selectively remove
keys with `ceph-kvstore-tool rocksdb store.db rm {prefix} {key}`
command and we may use only the `rm-prefix` command. Looking at the
code and actually trying it in a test environment it seems like it could
work, but I may be missing something dangerous here?
Adding Radek.
Thanks,
Ilya
11:50 p.m.
On Sun, Jul 30, 2023 at 7:09 PM Ilya Dryomov <idryomov(a)gmail.com> wrote:
I'm missing how connectivity issues between the
sites can lead to
mirror snapshot creation being interrupted on the primary cluster.
Isn't that operation local to the cluster?
Also, do you know who/what is actually interrupting it? Even if mirror
snapshot creation ends up taking a while for some reason, I don't think
anything in RBD would interrupt it.
Indeed, just thinking more about it, it does not look like the
connectivity issue
that was interrupting it.
The first thing noticed was a lot of "purge" (former mirroring) snapshots,
so my logical assumption was that snapshot removal was interrupted and we
also knew from the customer about "connectivity issues" between sites, so it
was just my assumption that those interruptions were due to network issues.
At first I was thinking that "snap id leak" might have happened on snapshot
removal. And I used the test with creating primary snapshots because it was
also removing snapshots. But reviewing the code I have not found any suspicious
place where we could leak the snap id on snapshot removal, while the testing
showed that it is rather possible to leak it on snapshot creation.
So currently I think it happens on snapshot creation, just forgot to revisit my
initial assumption what could cause the interruption. Ok, then another suspect
could be the rbd_support mgr module.
They are still running octopus (latest), there are more than 500
mirroring images,
and the snapshot schedule was configured for 3 minutes for each image. I expect
it could cause a considerable load and could trigger this interruption somehow.
Could it be due to blacklisting? (Recently we added to rbd_support module the
ability to restart rados when it is blacklisted).
Now after our recommendation I believe they have the schedule changed to
30 minutes interval.
Unfortunately the communication with the customer is troublesome, I get only
limited secondhand information and there are a lot of assumptions here.
Currently we are more interested in actually how to fix the large number of
purged_snap keys in the monstore, still I thought it would be useful to report
some details how it could happen.
This applies to regular user (i.e. non-mirror)
snapshots too.
Sure, mirroring is just a case when you may hit it due to frequent use.
Thanks,
--
Mykola Golub
31 Jul
31 Jul
12:01 a.m.
On Mon, Jul 31, 2023 at 9:50 AM Mykola Golub <to.my.trociny(a)gmail.com> wrote:
So currently I think it happens on snapshot creation,
just forgot to revisit my
initial assumption what could cause the interruption. Ok, then another suspect
could be the rbd_support mgr module.
On the other hand, they have large number of purged_snap keys on both
the primary
and the secondary clusters. If it had been due the rbd_support mgr
module I would
have expected to see it on the primary cluster only (they do not do
"two way" mirroring
as far as I know).
--
Mykola Golub
8 Aug
8 Aug
6:01 a.m.
On Mon, Jul 31, 2023 at 9:01 AM Mykola Golub <to.my.trociny(a)gmail.com> wrote:
On Mon, Jul 31, 2023 at 9:50 AM Mykola Golub <to.my.trociny(a)gmail.com> wrote:
Hi Mykola,
Yeah, rbd_support module is just a fancy way of calling "rbd mirror
image snapshot" command (rbd_mirror_image_create_snapshot API). There
is nothing in the Python code itself that deals with snap IDs.
Answering your other question: recovering from blocklisting in
rbd_support module is very recent and hasn't been backported even to
pacific [1], let alone octopus which went EOL a year ago.
[1] https://github.com/ceph/ceph/pull/51464
Thanks,
Ilya
So currently I think it happens on snapshot
creation, just forgot to revisit my
initial assumption what could cause the interruption. Ok, then another suspect
could be the rbd_support mgr module.
On the other hand, they have large number of purged_snap keys on both
the primary
and the secondary clusters. If it had been due the rbd_support mgr
module I would
have expected to see it on the primary cluster only (they do not do
"two way" mirroring
as far as I know).
5:07 a.m.
On Sun, Jul 30, 2023 at 7:09 PM Ilya Dryomov <idryomov(a)gmail.com> wrote:
Trying to resurrect this thread. Actually this is the question we are
most interested
in, i.e how the osd_snap prefixed keys can be properly trimmed.
Note, it is easy to write a script that would list all existing
snapshots in the pool
with `rbd snap ls --all {image}` command and then issue
`remove_self_managed_snap`
for all IDs that are not in this list. One thing is that it would run
for an unrealistically
long time. But another thing is that removing a snapshot may merge purged_snap
ranges (reducing the number of purged_snap keys) but it will add purged_epoch
key (which contains a list of snapshots removed for this epoch). And I
don't see how
the purged_epoch keys are supposed to be normally trimmed. It looks
like we never
trim these keys.
--
Mykola Golub
On Sat, Jul 29, 2023 at 2:11 PM Mykola Golub <to.my.trociny(a)gmail.com> wrote:
2) How can we
fix the current situation in the cluster? Would it be safe
enough to just run `ceph-kvstore-tool rocksdb store.db rm-prefix osd_snap`
to remove all osd_snap keys (including purged_epoch keys)? Due to
large db size I don't think it would be possible to selectively remove
keys with `ceph-kvstore-tool rocksdb store.db rm {prefix} {key}`
command and we may use only the `rm-prefix` command. Looking at the
code and actually trying it in a test environment it seems like it could
work, but I may be missing something dangerous here?
Adding Radek.
275
days inactive
285
days old
5 comments
2 participants
participants (2)
-
Ilya Dryomov -
Mykola Golub