I don't see sensible output for the commands: # ls -ld /<mount-path>/volumes/subvolgrp/test # ls -l /<mount-path>/volumes/subvolgrp/test/.snap please remember to replace /<mount-path> with the path to the mount point on your system I'm presuming /<mount-path> is the path where you have mounted the root dir of your cephfs filesystem On Thu, Oct 5, 2023 at 1:44 PM Kushagr Gupta <kushagrguptasps.mun@gmail.com> wrote:
Hi Milind,
Thank you for your response. Please find the logs attached, as instructed.
Thanks and Regards, Kushagra Gupta
On Thu, Oct 5, 2023 at 12:09 PM Milind Changire <mchangir@redhat.com> wrote:
this is really odd
Please run following commands and send over their outputs: # ceph status # ceph fs status # ceph report # ls -ld /<mount-path>/volumes/subvolgrp/test # ls -l /<mount-path>/volumes/subvolgrp/test/.snap
On Thu, Oct 5, 2023 at 11:17 AM Kushagr Gupta <kushagrguptasps.mun@gmail.com> wrote:
Hi Milind,Team
Thank you for your response @Milind Changire
The only thing I can think of is a stale mgr that wasn't restarted after an upgrade. Was an upgrade performed lately ?
Yes an upgrade was performed after which we faced this. But we were facing this issue previously as well. Another interesting thing which we observed was that even after the upgrade, the schedules that we created before upgrade were still running.
But to eliminate this, I installed a fresh cluster after purging the old one. Commands used for as follows: ``` ansible-playbook -i hosts infrastructure-playbooks/purge-cluster.yml ansible-playbook -i hosts site.yml ```
After this kindly note the commands which we followed: ``` [root@storagenode-1 ~]# ceph mgr module enable snap_schedule [root@storagenode-1 ~]# ceph config set mgr mgr/snap_schedule/log_level debug [root@storagenode-1 ~]# sudo ceph fs subvolumegroup create cephfs subvolgrp [root@storagenode-1 ~]# ceph fs subvolume create cephfs test subvolgrp [root@storagenode-1 ~]# date Thu Oct 5 04:23:09 UTC 2023 [root@storagenode-1 ~]# ceph fs snap-schedule add /volumes/subvolgrp/test 1h 2023-10-05T04:30:00 Schedule set for path /volumes/subvolgrp/test [root@storagenode-1 ~]# ceph fs snap-schedule list / --recursive=true /volumes/subvolgrp/test 1h [root@storagenode-1 ~]# ceph fs snap-schedule status /volumes/subvolgrp/test {"fs": "cephfs", "subvol": null, "path": "/volumes/subvolgrp/test", "rel_path": "/volumes/subvolgrp/test", "schedule": "1h", "retention": {}, "start": "2023-10-05T04:30:00", "created": "2023-10-05T04:23:39", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} [root@storagenode-1 ~]# ceph fs subvolume info cephfs test subvolgrp { "atime": "2023-10-05 04:20:18", "bytes_pcent": "undefined", "bytes_quota": "infinite", "bytes_used": 0, "created_at": "2023-10-05 04:20:18", "ctime": "2023-10-05 04:20:18", "data_pool": "cephfs_data", "features": [ "snapshot-clone", "snapshot-autoprotect", "snapshot-retention" ], "gid": 0, "mode": 16877, "mon_addrs": [ "[abcd:abcd:abcd::34]:6789", "[abcd:abcd:abcd::35]:6789", "[abcd:abcd:abcd::36]:6789" ], "mtime": "2023-10-05 04:20:18", "path": "/volumes/subvolgrp/test/73d82b1a-6fb1-4160-a388-66b898967a85", "pool_namespace": "", "state": "complete", "type": "subvolume", "uid": 0 } [root@storagenode-1 ~]# [root@storagenode-1 ~]# ceph fs snap-schedule status /volumes/subvolgrp/test {"fs": "cephfs", "subvol": null, "path": "/volumes/subvolgrp/test", "rel_path": "/volumes/subvolgrp/test", "schedule": "1h", "retention": {"h": 4}, "start": "2023-10-05T04:30:00", "created": "2023-10-05T04:23:39", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} [root@storagenode-1 ~]# date Thu Oct 5 05:31:20 UTC 2023 [root@storagenode-1 ~]# ```
Could you please help us. Are we doing something wrong? Because still the schedules are not getting created.
Thanks and Regards, Kushagra Gupta
On Wed, Oct 4, 2023 at 9:33 PM Milind Changire <mchangir@redhat.com> wrote:
On Wed, Oct 4, 2023 at 7:19 PM Kushagr Gupta <kushagrguptasps.mun@gmail.com> wrote:
Hi Milind,
Thank you for your swift response.
>How many hours did you wait after the "start time" and decide to restart mgr ? We waited for ~3 days before restarting the mgr-service.
The only thing I can think of is a stale mgr that wasn't restarted after an upgrade. Was an upgrade performed lately ?
Did the dir exist at the time the snapshot was scheduled to take place. If it didn't then the schedule gets disabled until explicitly enabled.
There was one more instance where we waited for 2 hours and then re-started and in the third hour the schedule started working.
Could you please guide us if we are doing anything wrong. Kindly let us know if any logs are required.
Thanks and Regards, Kushagra Gupta
On Wed, Oct 4, 2023 at 5:39 PM Milind Changire <mchangir@redhat.com> wrote:
On Wed, Oct 4, 2023 at 3:40 PM Kushagr Gupta <kushagrguptasps.mun@gmail.com> wrote: > > Hi Team,Milind > > Ceph-version: Quincy, Reef > OS: Almalinux 8 > > Issue: snap_schedule works after 1 hour of schedule > > Description: > > We are currently working in a 3-node ceph cluster. > We are currently exploring the scheduled snapshot capability of the ceph-mgr module. > To enable/configure scheduled snapshots, we followed the following link: > > > > https://docs.ceph.com/en/quincy/cephfs/snap-schedule/ > > > > We were able to create snap schedules for the subvolumes as suggested. > But we have observed a two very strange behaviour: > 1. The snap_schedules only work when we restart the ceph-mgr service on the mgr node: > We then restarted the mgr-service on the active mgr node, and after 1 hour it started getting created. I am attaching the log file for the same after restart. Thre behaviour looks abnormal.
A mgr restart is not required for the schedule to get triggered. How many hours did you wait after the "start time" and decide to restart mgr ?
> > So, for eg consider the below output: > ``` > [root@storagenode-1 ~]# ceph fs snap-schedule status /volumes/subvolgrp/test3 > {"fs": "cephfs", "subvol": null, "path": "/volumes/subvolgrp/test3", "rel_path": "/volumes/subvolgrp/test3", "schedule": "1h", "retention": {}, "start": "2023-10-04T07:20:00", "created": "2023-10-04T07:18:41", "first": "2023-10-04T08:20:00", "last": "2023-10-04T09:20:00", "last_pruned": null, "created_count": 2, "pruned_count": 0, "active": true} > [root@storagenode-1 ~]# > ``` > As we can see in the above o/p, we created the schedule at 2023-10-04T07:18:41. The schedule was suppose to start at 2023-10-04T07:20:00 but it started at 2023-10-04T08:20:00
seems normal behavior to me the schedule starts countdown for 1h from 2023-10-04T07:20:00 and created first snapshot at 2023-10-04T08:20:00
> > Any input w.r.t the same will be of great help. > > Thanks and Regards > Kushagra Gupta
-- Milind
-- Milind
-- Milind
-- Milind