Hi, we run two clusters with Squid 19.2.2 and would like to use CephFS mirroring. Unfortunately, the feature is still not working correctly and cannot be used. So we are looking for an alternative for the synchronization. One option would be to synchronize the snapshots to the second cluster with rsync. What alternative do you use for CephFS mirroring? What tools and processes do you use? Should CephFS mirroring be fixed? Do you know anything new about this? Thanks and best regards, Ralf
Hi, I use rsync to copy data (~10TB) to backup storage. To speed things up I use the ceph.dir.rctime extended attribute to instantly ignore sub-trees that haven't changed without iterating through their contents. I have to maintain the ceph.dir.rctime value between backups: I just keep it in a file per top-level directory on the target storage. It works great because most directories don't change between backups. On a more volatile dataset it wouldn't work as well... Hope this helps, Best, Le 12/06/2025 à 15:34, Ralf Schulze a écrit :
[Vous ne recevez pas souvent de courriers de ralf.schulze@hrz.tu-chemnitz.de. Découvrez pourquoi ceci est important à https://aka.ms/LearnAboutSenderIdentification ]
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi,
we run two clusters with Squid 19.2.2 and would like to use CephFS mirroring. Unfortunately, the feature is still not working correctly and cannot be used.
So we are looking for an alternative for the synchronization. One option would be to synchronize the snapshots to the second cluster with rsync.
What alternative do you use for CephFS mirroring? What tools and processes do you use?
Should CephFS mirroring be fixed? Do you know anything new about this?
Thanks and best regards, Ralf
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eric, On 6/12/25 17:33, Eric Le Lay wrote:
I use rsync to copy data (~10TB) to backup storage.
To speed things up I use the ceph.dir.rctime extended attribute to instantly ignore sub-trees that haven't changed without iterating through their contents. I have to maintain the ceph.dir.rctime value between backups: I just keep it in a file per top-level directory on the target storage. This sounds interesting. Can you give some advice how to set this up with rsync?
-- Best regards / Mit freundlichen Grüßen Daniel Vogelbacher
Hi, On 12.06.25 21:58, Daniel Vogelbacher wrote:
Hi Eric,
On 6/12/25 17:33, Eric Le Lay wrote:
I use rsync to copy data (~10TB) to backup storage.
To speed things up I use the ceph.dir.rctime extended attribute to instantly ignore sub-trees that haven't changed without iterating through their contents. I have to maintain the ceph.dir.rctime value between backups: I just keep it in a file per top-level directory on the target storage. This sounds interesting. Can you give some advice how to set this up with rsync?
the rctime attribute is useful, but I wouldn't rely on it. As far as I know it is stored in a directory inode, so each operation on a file or directory will update the rctime on all path elements (not sure whether this happens synchronously or asynchronously). The problem is the fact that it is just a single value. Imagine one rogue user or rogue host that touches a file in a subdirectory, set the ctime to 01/01/2300, and then removes the file. Although the removal is the last operation, setting the ctime will also update the rctime of all path elements. And the removal if the file cannot revert this. So your backup check will detect a last change in 01/01/2300 for the subtree and probably performs a complete rsync. Even if all files are still the same. rctime is fine for controlled environments without rogue elements (== no users... ;-) ). And it can definitely be used to skip subtrees. But the check can easily be rendered useless. Best regards, Burkhard
Hi, I agree with Daniel, but I have been successfully using this type of backup strategy for two years between two CEPH clusters, with 1.6 PB and over 1 billion files/folders. A few comments: 1) Using the rctime attribute is better than nothing. 2) The rctime=01/01/2300 issue is not really a problem. I run non-recursive rsyncs on existing folders that have an rctime > previous backup date. In the case of rctime=01/01/2300, this results in unnecessary rsyncs, but in reality, they are very fast. So far, I have not observed any degradation in backup times. 3) I take a snapshot before each backup, and the date of the previous snapshot gives me the rctime reference value of the folders/files to be backed up (so no need to keep the rctime values in a file like Eric does). 4) fpart/fpsync is my best friend for transferring, with parallel rsyncs, new folders that have more than 1000 entries (ceph.dir.rentries attribute). Every evening, I generate a hundred backup jobs (1 job for each project directory with 20 CPUs allocated to each job for parallel rsyncs) that are executed on an HPC cluster in a specific queue with only 3 nodes. There is quite a lot of user activity during the day and the average backup time is around 2 hours. Obviously, this depends on the volume to be transferred and/or deleted on the target cluster. For now, everything is working well, but the main limitation of this strategy remains the management of directory moves/renames (creation of the new directory and deletion of the old one). Kind regards, Arnaud Le 13/06/2025 08:58, « Burkhard Linke » <Burkhard.Linke@computational.bio.uni-giessen.de <mailto:Burkhard.Linke@computational.bio.uni-giessen.de>> a écrit : Hi, On 12.06.25 21:58, Daniel Vogelbacher wrote:
Hi Eric,
On 6/12/25 17:33, Eric Le Lay wrote:
I use rsync to copy data (~10TB) to backup storage.
To speed things up I use the ceph.dir.rctime extended attribute to instantly ignore sub-trees that haven't changed without iterating through their contents. I have to maintain the ceph.dir.rctime value between backups: I just keep it in a file per top-level directory on the target storage. This sounds interesting. Can you give some advice how to set this up with rsync?
the rctime attribute is useful, but I wouldn't rely on it. As far as I know it is stored in a directory inode, so each operation on a file or directory will update the rctime on all path elements (not sure whether this happens synchronously or asynchronously). The problem is the fact that it is just a single value. Imagine one rogue user or rogue host that touches a file in a subdirectory, set the ctime to 01/01/2300, and then removes the file. Although the removal is the last operation, setting the ctime will also update the rctime of all path elements. And the removal if the file cannot revert this. So your backup check will detect a last change in 01/01/2300 for the subtree and probably performs a complete rsync. Even if all files are still the same. rctime is fine for controlled environments without rogue elements (== no users... ;-) ). And it can definitely be used to skip subtrees. But the check can easily be rendered useless. Best regards, Burkhard
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
Le 13/06/2025 à 08:57, Burkhard Linke a écrit :
Hi,
On 12.06.25 21:58, Daniel Vogelbacher wrote:
Hi Eric,
On 6/12/25 17:33, Eric Le Lay wrote:
I use rsync to copy data (~10TB) to backup storage.
To speed things up I use the ceph.dir.rctime extended attribute to instantly ignore sub-trees that haven't changed without iterating through their contents. I have to maintain the ceph.dir.rctime value between backups: I just keep it in a file per top-level directory on the target storage. This sounds interesting. Can you give some advice how to set this up with rsync?
the rctime attribute is useful, but I wouldn't rely on it. As far as I know it is stored in a directory inode, so each operation on a file or directory will update the rctime on all path elements (not sure whether this happens synchronously or asynchronously).
The problem is the fact that it is just a single value. Imagine one rogue user or rogue host that touches a file in a subdirectory, set the ctime to 01/01/2300, and then removes the file. Although the removal is the last operation, setting the ctime will also update the rctime of all path elements. And the removal if the file cannot revert this. So your backup check will detect a last change in 01/01/2300 for the subtree and probably performs a complete rsync. Even if all files are still the same.
rctime is fine for controlled environments without rogue elements (== no users... ;-) ). And it can definitely be used to skip subtrees. But the check can easily be rendered useless.
Best regards,
Burkhard
Thanks Burkhard for the warning. There has been at least one case reported [1] Would checking for rctime for a date in the future make sense?
Le 13/06/2025 à 08:57, Burkhard Linke a écrit :
Hi,
On 12.06.25 21:58, Daniel Vogelbacher wrote:
Hi Eric,
On 6/12/25 17:33, Eric Le Lay wrote:
I use rsync to copy data (~10TB) to backup storage.
To speed things up I use the ceph.dir.rctime extended attribute to instantly ignore sub-trees that haven't changed without iterating through their contents. I have to maintain the ceph.dir.rctime value between backups: I just keep it in a file per top-level directory on the target storage. This sounds interesting. Can you give some advice how to set this up with rsync?
the rctime attribute is useful, but I wouldn't rely on it. As far as I know it is stored in a directory inode, so each operation on a file or directory will update the rctime on all path elements (not sure whether this happens synchronously or asynchronously).
The problem is the fact that it is just a single value. Imagine one rogue user or rogue host that touches a file in a subdirectory, set the ctime to 01/01/2300, and then removes the file. Although the removal is the last operation, setting the ctime will also update the rctime of all path elements. And the removal if the file cannot revert this. So your backup check will detect a last change in 01/01/2300 for the subtree and probably performs a complete rsync. Even if all files are still the same.
rctime is fine for controlled environments without rogue elements (== no users... ;-) ). And it can definitely be used to skip subtrees. But the check can easily be rendered useless.
Best regards,
Burkhard
Thanks Burkhard for the warning. Indeed there has been at least one case reported [1]. Checking for an rctime in the future during backup and raising an error would make sense. Otherwise, the script: 1. grabs $SRCDIR and first-level subdirectory rctimes via `find $SRCDIR -type d -maxdepth 1 -exec getfattr -n ceph.dir.rctime '{}' ';'` 2. loads cached metadata from the target filesystem (one file per dir, in a special $DSTDIR/.rctimes dir) 3. if $SRCDIR has the same rctime as in cached metadata you can exit early 4. for each directory in $SRCDIR if its rctime is bigger or it is not in cached metadata, run rsync on it, save rctime (don't re-read it but use the rctime from before the rsync in case of file changes during rsync) 5. for each directory in target filesystem if it not in srcdir delete it and the cached metadata 6. rsync all files directly in $SRCDIR, remove all files directly in $DSTDIR but not in $SRCDIR 7. update cached metadata for $SRCDIR Cheers, [1] https://www.spinics.net/lists/ceph-users/msg75172.html
Hi Ralf, On Thu, Jun 12, 2025 at 7:49 PM Ralf Schulze <ralf.schulze@hrz.tu-chemnitz.de> wrote:
Hi,
we run two clusters with Squid 19.2.2 and would like to use CephFS mirroring. Unfortunately, the feature is still not working correctly and cannot be used.
So we are looking for an alternative for the synchronization. One option would be to synchronize the snapshots to the second cluster with rsync.
What alternative do you use for CephFS mirroring? What tools and processes do you use?
Should CephFS mirroring be fixed? Do you know anything new about this?
A set of important backports are making it to the supported release which would greatly improve the user experience. Also, with the upcoming squid release (19.2.3), the cephfs mirror will be able to transfer only changed blocks in a file, thereby improving sync speed.
Thanks and best regards, Ralf _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Cheers, Venky
participants (6)
-
Burkhard Linke
-
Daniel Vogelbacher
-
Eric Le Lay
-
MARTEL Arnaud
-
Ralf Schulze
-
Venky Shankar