Hi Folks I'm writing a backup/sync process for our ceph cluster. The process takes a snapshot of the system that's being backed up and rsync's from that to another ceph cluster I was hoping to use the snapshot xattrs to verify a successful backup by comparing ceph.dir.rbytes and ceph.dir.rentries between the snapshot and backup target. However, while files added to the source directory after the snapshot is taken don't appear in the snapshot, ceph.dir.rbytes and ceph.dir.rctime are updated in the xattrs of both. $ fallocate -l 1K test1 $ getfattr -n ceph.dir.rbytes . # file: . ceph.dir.rbytes="1024" $ mkdir .snap/testsnap $ fallocate -l 2K test2 $ getfattr -n ceph.dir.rbytes . .snap/testsnap # file: . ceph.dir.rbytes="3072" # file: .snap/testsnap ceph.dir.rbytes="3072" $ getfattr -n ceph.dir.rctime . .snap/testsnap # file: . ceph.dir.rctime="1573558448.09736131125" # file: .snap/testsnap ceph.dir.rctime="1573558448.09736131125" $ tree -a --si . .snap . |-- [1.0k] test1 `-- [2.0k] test2 .snap `-- [3.1k] testsnap `-- [1.0k] test1 1 directory, 3 files Have I misunderstood? Is this expected behaviour? Cheers Toby -- Toby Darling, Scientific Computing (2N249) MRC Laboratory of Molecular Biology Francis Crick Avenue Cambridge Biomedical Campus Cambridge CB2 0QH Phone 01223 267070
On Tue, Nov 12, 2019 at 6:39 AM Toby Darling <toby@mrc-lmb.cam.ac.uk> wrote:
Hi Folks
I'm writing a backup/sync process for our ceph cluster. The process takes a snapshot of the system that's being backed up and rsync's from that to another ceph cluster
I was hoping to use the snapshot xattrs to verify a successful backup by comparing ceph.dir.rbytes and ceph.dir.rentries between the snapshot and backup target.
However, while files added to the source directory after the snapshot is taken don't appear in the snapshot, ceph.dir.rbytes and ceph.dir.rctime are updated in the xattrs of both.
I don't remember for sure, but I think you're falling victim to rstat propagation being lazy here, and metadata in snapshots being somewhat delayed. Can you try doing a sync before you take the snapshot and see if that changes the behavior? -Greg
$ fallocate -l 1K test1
$ getfattr -n ceph.dir.rbytes . # file: . ceph.dir.rbytes="1024"
$ mkdir .snap/testsnap
$ fallocate -l 2K test2
$ getfattr -n ceph.dir.rbytes . .snap/testsnap # file: . ceph.dir.rbytes="3072"
# file: .snap/testsnap ceph.dir.rbytes="3072"
$ getfattr -n ceph.dir.rctime . .snap/testsnap # file: . ceph.dir.rctime="1573558448.09736131125"
# file: .snap/testsnap ceph.dir.rctime="1573558448.09736131125"
$ tree -a --si . .snap . |-- [1.0k] test1 `-- [2.0k] test2 .snap `-- [3.1k] testsnap `-- [1.0k] test1
1 directory, 3 files
Have I misunderstood? Is this expected behaviour?
Cheers Toby -- Toby Darling, Scientific Computing (2N249) MRC Laboratory of Molecular Biology Francis Crick Avenue Cambridge Biomedical Campus Cambridge CB2 0QH Phone 01223 267070 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 11/13/19 2:42 AM, Gregory Farnum wrote:
On Tue, Nov 12, 2019 at 6:39 AM Toby Darling <toby@mrc-lmb.cam.ac.uk> wrote:
Hi Folks
I'm writing a backup/sync process for our ceph cluster. The process takes a snapshot of the system that's being backed up and rsync's from that to another ceph cluster
I was hoping to use the snapshot xattrs to verify a successful backup by comparing ceph.dir.rbytes and ceph.dir.rentries between the snapshot and backup target.
However, while files added to the source directory after the snapshot is taken don't appear in the snapshot, ceph.dir.rbytes and ceph.dir.rctime are updated in the xattrs of both.
I don't remember for sure, but I think you're falling victim to rstat propagation being lazy here, and metadata in snapshots being somewhat delayed. Can you try doing a sync before you take the snapshot and see if that changes the behavior? -Greg
yes, rstats in snapshot are not reliable. you'd better to another method to verify your backup. Yan, Zheng
$ fallocate -l 1K test1
$ getfattr -n ceph.dir.rbytes . # file: . ceph.dir.rbytes="1024"
$ mkdir .snap/testsnap
$ fallocate -l 2K test2
$ getfattr -n ceph.dir.rbytes . .snap/testsnap # file: . ceph.dir.rbytes="3072"
# file: .snap/testsnap ceph.dir.rbytes="3072"
$ getfattr -n ceph.dir.rctime . .snap/testsnap # file: . ceph.dir.rctime="1573558448.09736131125"
# file: .snap/testsnap ceph.dir.rctime="1573558448.09736131125"
$ tree -a --si . .snap . |-- [1.0k] test1 `-- [2.0k] test2 .snap `-- [3.1k] testsnap `-- [1.0k] test1
1 directory, 3 files
Have I misunderstood? Is this expected behaviour?
Cheers Toby -- Toby Darling, Scientific Computing (2N249) MRC Laboratory of Molecular Biology Francis Crick Avenue Cambridge Biomedical Campus Cambridge CB2 0QH Phone 01223 267070 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 13/11/2019 01:49, Yan, Zheng wrote:
On 11/13/19 2:42 AM, Gregory Farnum wrote:
On Tue, Nov 12, 2019 at 6:39 AM Toby Darling <toby@mrc-lmb.cam.ac.uk> wrote:
Hi Folks
I'm writing a backup/sync process for our ceph cluster. The process takes a snapshot of the system that's being backed up and rsync's from that to another ceph cluster
I was hoping to use the snapshot xattrs to verify a successful backup by comparing ceph.dir.rbytes and ceph.dir.rentries between the snapshot and backup target.
However, while files added to the source directory after the snapshot is taken don't appear in the snapshot, ceph.dir.rbytes and ceph.dir.rctime are updated in the xattrs of both.
I don't remember for sure, but I think you're falling victim to rstat propagation being lazy here, and metadata in snapshots being somewhat delayed. Can you try doing a sync before you take the snapshot and see if that changes the behavior? -Greg
yes, rstats in snapshot are not reliable. you'd better to another method to verify your backup.
OK, thanks for clarifying. Greg, just to clarify - I wasn't worried that the update of the snapshot's rstat was lazy, it was that it was being updated at all. I thought it'd be static - an indication of what was in the snapshot. Cheers Toby
Yan, Zheng
$ fallocate -l 1K test1
$ getfattr -n ceph.dir.rbytes . # file: . ceph.dir.rbytes="1024"
$ mkdir .snap/testsnap
$ fallocate -l 2K test2
$ getfattr -n ceph.dir.rbytes . .snap/testsnap # file: . ceph.dir.rbytes="3072"
# file: .snap/testsnap ceph.dir.rbytes="3072"
$ getfattr -n ceph.dir.rctime . .snap/testsnap # file: . ceph.dir.rctime="1573558448.09736131125"
# file: .snap/testsnap ceph.dir.rctime="1573558448.09736131125"
$ tree -a --si . .snap . |-- [1.0k] test1 `-- [2.0k] test2 .snap `-- [3.1k] testsnap `-- [1.0k] test1
1 directory, 3 files
Have I misunderstood? Is this expected behaviour?
Cheers Toby -- Toby Darling, Scientific Computing (2N249) MRC Laboratory of Molecular Biology Francis Crick Avenue Cambridge Biomedical Campus Cambridge CB2 0QH Phone 01223 267070 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Cheers Toby -- Toby Darling, Scientific Computing (2N249) MRC Laboratory of Molecular Biology Francis Crick Avenue Cambridge Biomedical Campus Cambridge CB2 0QH Phone 01223 267070
participants (3)
-
Gregory Farnum
-
Toby Darling
-
Yan, Zheng