27 Apr
2023
27 Apr
'23
10:07 a.m.
Hello,
we are running a 3-node ceph cluster with version 17.2.6.
For CephFS snapshots we have configured the following snap schedule with
retention:
/PATH 2h 72h15d6m
But we observed that max 50 snapshot are preserved. If a new snapshot is
created the oldest 51st is deleted.
Is there a limit for maximum cephfs snapshots or maybe this is a bug?
I have found the setting "mds_max_snaps_per_dir" which is 100 by default
but I think this is not related to my problem?
Thanks,
Tobias
27 Apr
27 Apr
11:33 a.m.
On Thu, 27 Apr 2023 09:07:10 +0200
Tobias Hachmer <t.hachmer(a)s-v.de> wrote:
But we observed that max 50 snapshot are preserved. If
a new snapshot is
created the oldest 51st is deleted.
Is there a limit for maximum cephfs snapshots or maybe this is a bug?
I've been wondering the same thing for about 6 months now and found the
reason just yesterday.
The snap-schedule mgr module has a hard limit on how many snapshots it
preserves, see [1]. It's even documented at [2] in section
"Limitations" near the end of the page.
The commit[3] implementing this does not only not explain the reason
for the number at all, it doesn't even mention the fact it implements
this.
Given the limitation is per directory, I'm currently trying this:
/ 1d 30d
/foo 1h 48h
/bar 1h 48h
I forgot to activate the new schedules yesterday so I can't say whether
it works as expected yet.
Cheers,
sur5r
[1]
https://github.com/ceph/ceph/blob/3d7761bd59b8e5ebac1d9a136d020f0f8d2eaf32/…
[2] https://docs.ceph.com/en/quincy/cephfs/snap-schedule/
[3] https://github.com/ceph/ceph/commit/a48efa43dbe4c623ae88b84ef538ee306fc1eee8
--
ceterum censeo microsoftem esse delendam.
12:10 p.m.
Hi sur5r,
Am 4/27/23 um 10:33 schrieb Jakob Haufe:
On Thu, 27 Apr 2023 09:07:10 +0200
Tobias Hachmer <t.hachmer(a)s-v.de> wrote:
Thanks. I've red the documentation, but it's not clear enough. I thought
"the retention list will be shortened to the newest 50 snapshots" will
just truncate the list and not delete the snapshots, effectively.
So as you stated the max. number of snapshots is currently a hard limit.
Can anyone clarify the reasons for this? If there's a big reason to hard
limit this it would be great to schedule snapshots more granular e.g.
mo-fr every two hours between 8am-6pm.
But we observed that max 50 snapshot are
preserved. If a new snapshot is
created the oldest 51st is deleted.
Is there a limit for maximum cephfs snapshots or maybe this is a bug?
I've been wondering the same thing for about 6 months now and found the
reason just yesterday.
The snap-schedule mgr module has a hard limit on how many snapshots it
preserves, see [1]. It's even documented at [2] in section
"Limitations" near the end of the page.
The commit[3] implementing this does not only not explain the reason
for the number at all, it doesn't even mention the fact it implements
this. Given the limitation is per directory, I'm
currently trying this:
/ 1d 30d
/foo 1h 48h
/bar 1h 48h
I forgot to activate the new schedules yesterday so I can't say whether
it works as expected yet.
Please let me know if this works.
Thanks,
Tobias
28 Apr
28 Apr
9:28 a.m.
Hi Tobias,
On Thu, Apr 27, 2023 at 2:42 PM Tobias Hachmer <t.hachmer(a)s-v.de> wrote:
--
Cheers,
Venky
Hi sur5r,
Am 4/27/23 um 10:33 schrieb Jakob Haufe:
This was done so that a particular directory does not eat up all the
snapshots - there is a per directory limit on the number of snapshots
controlled by mds_max_snaps_per_dir which defaults to 100 and
therefore MAX_SNAPS_PER_PATH was chosen to be much lower than that.
Also, at one point the kclient wasn't able to handle more than 400
snapshots (per file system), but we have come a long way from that and
that is not a constraint right now.
On Thu, 27 Apr 2023 09:07:10 +0200
Tobias Hachmer <t.hachmer(a)s-v.de> wrote:
Thanks. I've red the documentation, but it's not clear enough. I thought
"the retention list will be shortened to the newest 50 snapshots" will
just truncate the list and not delete the snapshots, effectively.
So as you stated the max. number of snapshots is currently a hard limit.
Can anyone clarify the reasons for this? If there's a big reason to hard
limit this it would be great to schedule snapshots more granular e.g.
mo-fr every two hours between 8am-6pm. But we observed that max 50 snapshot are
preserved. If a new snapshot is
created the oldest 51st is deleted.
Is there a limit for maximum cephfs snapshots or maybe this is a bug?
I've been wondering the same thing for about 6 months now and found the
reason just yesterday.
The snap-schedule mgr module has a hard limit on how many snapshots it
preserves, see [1]. It's even documented at [2] in section
"Limitations" near the end of the page.
The commit[3] implementing this does not only not explain the reason
for the number at all, it doesn't even mention the fact it implements
this. Given the limitation is per directory, I'm
currently trying this:
/ 1d 30d
/foo 1h 48h
/bar 1h 48h
I forgot to activate the new schedules yesterday so I can't say whether
it works as expected yet.
Please let me know if this works.
Thanks,
Tobias
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:02 p.m.
On Thu, 27 Apr 2023 11:10:07 +0200
Tobias Hachmer <t.hachmer-YT5KBw9Z7BM(a)public.gmane.org> wrote:
It doesn't.
I haven't re-visited the code yet, but for some reason the lower level
schedules get deactivated again, seemingly each time they are supposed
to create a snapshot.
Cheers,
sur5r
--
ceterum censeo microsoftem esse delendam.
Given the
limitation is per directory, I'm currently trying this:
/ 1d 30d
/foo 1h 48h
/bar 1h 48h
I forgot to activate the new schedules yesterday so I can't say whether
it works as expected yet.
Please let me know if this works. 
8:16 p.m.
If a dir doesn't exist at the moment of snapshot creation, then the
schedule is deactivated for that dir.
On Fri, Apr 28, 2023 at 8:39 PM Jakob Haufe <sur5r(a)sur5r.net> wrote:
On Thu, 27 Apr 2023 11:10:07 +0200
Tobias Hachmer <t.hachmer-YT5KBw9Z7BM(a)public.gmane.org> wrote:
It doesn't.
I haven't re-visited the code yet, but for some reason the lower level
schedules get deactivated again, seemingly each time they are supposed
to create a snapshot.
Cheers,
sur5r
--
ceterum censeo microsoftem esse delendam.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Milind
> Given the limitation is per directory,
I'm currently trying this:
>
> / 1d 30d
> /foo 1h 48h
> /bar 1h 48h
>
> I forgot to activate the new schedules yesterday so I can't say
whether
it works
as expected yet.
Please let me know if this works.
29 Apr
29 Apr
3:39 p.m.
On Fri, 28 Apr 2023 22:46:32 +0530 Milind Changire wrote:
If a dir doesn't exist at the moment of snapshot
creation, then the
schedule is deactivated for that dir.
Ha! Good catch! As often, I completely forgot about the /volumes prefix...
--
ceterum censeo microsoftem esse delendam.
28 Apr
28 Apr
6:19 a.m.
There's a default/hard limit of 50 snaps that's maintained for any dir via
the definition MAX_SNAPS_PER_PATH = 50 in the source file
src/pybind/mgr/snap_schedule/fs/schedule_client.py.
Every time the snapshot names are read for pruning, the last thing done is
to check the length of the list and keep only MAX_SNAPS_PER_PATH and the
rest are pruned.
Jakob Haufe has pointed it out correctly.
On Thu, Apr 27, 2023 at 12:38 PM Tobias Hachmer <t.hachmer(a)s-v.de> wrote:
Hello,
we are running a 3-node ceph cluster with version 17.2.6.
For CephFS snapshots we have configured the following snap schedule with
retention:
/PATH 2h 72h15d6m
But we observed that max 50 snapshot are preserved. If a new snapshot is
created the oldest 51st is deleted.
Is there a limit for maximum cephfs snapshots or maybe this is a bug?
I have found the setting "mds_max_snaps_per_dir" which is 100 by default
but I think this is not related to my problem?
Thanks,
Tobias
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Milind
11:23 a.m.
FYI, PR - https://github.com/ceph/ceph/pull/51278
On Fri, Apr 28, 2023 at 8:49 AM Milind Changire <mchangir(a)redhat.com> wrote:
There's a default/hard limit of 50 snaps
that's maintained for any dir via
the definition MAX_SNAPS_PER_PATH = 50 in the source file
src/pybind/mgr/snap_schedule/fs/schedule_client.py.
Every time the snapshot names are read for pruning, the last thing done is
to check the length of the list and keep only MAX_SNAPS_PER_PATH and the
rest are pruned.
Jakob Haufe has pointed it out correctly.
On Thu, Apr 27, 2023 at 12:38 PM Tobias Hachmer <t.hachmer(a)s-v.de> wrote:
--
Milind
Hello,
we are running a 3-node ceph cluster with version 17.2.6.
For CephFS snapshots we have configured the following snap schedule with
retention:
/PATH 2h 72h15d6m
But we observed that max 50 snapshot are preserved. If a new snapshot is
created the oldest 51st is deleted.
Is there a limit for maximum cephfs snapshots or maybe this is a bug?
I have found the setting "mds_max_snaps_per_dir" which is 100 by default
but I think this is not related to my problem?
Thanks,
Tobias
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Milind
7:30 p.m.
FYI, PR - https://github.com/ceph/ceph/pull/51278
Thanks!
I just applied this to my cluster and will report back. Looks simple
enough, tbh.
Cheers,
sur5r
--
ceterum censeo microsoftem esse delendam.
386
days inactive
388
days old
9 comments
4 participants
participants (4)
-
Jakob Haufe -
Milind Changire -
Tobias Hachmer -
Venky Shankar