I don't think so. It is exactly the same location in all tests and it is reproducible. Why would a move be a copy on some MDSs/OSDs but not others? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Marc <Marc@f1-outsourcing.eu> Sent: 22 June 2021 13:53:25 To: Frank Schilder; 'ceph-users@ceph.io' Subject: RE: ceph fs mv does copy, not move Could this not be related to the mds and different osd's being used?

-----Original Message----- From: Frank Schilder <frans@dtu.dk> Sent: Tuesday, 22 June 2021 13:25 To: ceph-users@ceph.io Subject: [ceph-users] Re: ceph fs mv does copy, not move

I get really strange timings depending on kernel version; see below. Did the patch of the kernel client get lost? The only difference between gnosis and smb01 is that gnosis is physical and smb01 is a KVM. Both have direct access to the client network and use the respective kernel clients.

Timings of moving a 3.2G file around, crossing quota realms:

[root@gnosis ~]# uname -r 5.9.9-1.el7.elrepo.x86_64

[root@gnosis ~]# time mv /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/ real 0m0.013s user 0m0.000s sys 0m0.001s

[root@gnosis ~]# time mv /mnt/cephfs/B/testfile.bin /mnt/cephfs/A/ real 0m0.008s user 0m0.000s sys 0m0.001s

[root@gnosis ~]# time cp /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/ real 0m16.807s user 0m0.000s sys 0m2.150s

[root@gnosis ~]# time \rm /mnt/cephfs/B/testfile.bin real 0m0.004s user 0m0.000s sys 0m0.001s

[root@smb01 ~]# uname -r 5.12.12-1.el7.elrepo.x86_64

[root@smb01 ~]# time mv /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/ real 0m8.545s user 0m0.004s sys 0m1.999s

[root@smb01 ~]# time mv /mnt/cephfs/B/testfile.bin /mnt/cephfs/A/ real 0m2.986s user 0m0.003s sys 0m1.644s

[root@smb01 ~]# time cp /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/ real 0m3.564s user 0m0.003s sys 0m1.671s

[root@smb01 ~]# time \rm /mnt/cephfs/B/testfile.bin real 0m0.022s user 0m0.001s sys 0m0.000s

[root@smb01 ~]# uname -r 5.4.127-1.el7.elrepo.x86_64

[root@smb01 ~]# time mv /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/ real 0m17.268s user 0m0.000s sys 0m2.010s

[root@smb01 ~]# time mv /mnt/cephfs/B/testfile.bin /mnt/cephfs/A/ real 0m3.404s user 0m0.002s sys 0m1.635s

[root@smb01 ~]# time cp /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/ real 0m3.569s user 0m0.002s sys 0m1.825s

[root@smb01 ~]# time \rm /mnt/cephfs/B/testfile.bin real 0m0.031s user 0m0.001s sys 0m0.001s

[root@smb01 ~]# uname -r 3.10.0-1160.31.1.el7.x86_64

[root@smb01 ~]# time mv /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/ real 0m25.695s user 0m0.003s sys 0m1.936s

[root@smb01 ~]# time mv /mnt/cephfs/B/testfile.bin /mnt/cephfs/A/ real 0m3.173s user 0m0.001s sys 0m2.302s

[root@smb01 ~]# time cp /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/ real 0m2.674s user 0m0.004s sys 0m2.057s

[root@smb01 ~]# time \rm /mnt/cephfs/B/testfile.bin real 0m0.016s user 0m0.000s sys 0m0.001s

================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 22 June 2021 11:15:06 To: ceph-users@ceph.io Subject: [ceph-users] ceph fs mv does copy, not move

Dear all,

some time ago I reported that the kernel client resorts to a copy instead of move when moving a file across quota domains. I was told that the fuse client does not have this problem. If enough space is available, a move should be a move, not a copy.

Today, I tried to move a large file across quota domains testing botn, the kernel- and the fuse client. Both still resort to a copy even though this issue was addressed quite a while ago (https://lists.ceph.io/hyperkitty/list/ceph- users@ceph.io/thread/44AEIHNEGKV4VGCARRTARGFZ264CR4T7/#XY7ZCE3KWHI4QSUNZ HDWL3QZQFOHXRQW). The versions I'm using are (CentOS 7)

# yum list installed | grep ceph-fuse ceph-fuse.x86_64 2:13.2.10-0.el7 @ceph

# uname -r 3.10.0-1160.31.1.el7.x86_64

Any suggestions how to get this to work? I have to move directories containing 100+ TB.

Many thanks, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Dear Patrick, thanks for letting me know. Could you please consider to make this a ceph client mount option, for example, '-o fast_move', that enables a code path that enforces an mv to be a proper atomic mv with the risk that in some corner cases the target quota is overrun? With this option enabled, a move should either be a move or fail outright with "out of disk quota" (no partial move, no cp+rm at all). The fail should only occur if it is absolutely obvious that the target quota will be exceeded. Any corner cases are the responsibility of the operator. Application crashes due to incorrect error handling are acceptable. Reasoning: From a user's/operator's side, the preferred functionality is that in cases where a definite quota overrun can securely be detected in advance, the move should actually fail with "out of disk quota" instead of resorting to cp+rm, potentially leading to partial moves and a total mess for users/operators to clean up. In any other case, the quota should simply be ignored and the move should be a complete atomic move with the risk of exceeding the target quota and IO to stall. A temporary stall or fail of IO until the operator increases the quota again is, in my opinion and use case, highly preferable over the alternative of cp+rm. A quota or a crashed job is fast to fix, a partial move is not. Some background: We use ceph fs as an HPC home file system and as a back-end store. Being able to move data quickly across the entire file system is essential, because users re-factor their directory structure containing huge amounts of data quite often for various reasons. On our system, we set file system quotas mainly for psychological reasons. We run a cron job that adjusts the quotas every day to show between 20% and 30% free capacity on the mount points. The psychological side here is to give an incentive to users to clean up temporary data. It is not intended to limit usage seriously, only to limit what can be done in between cron job runs as a safe-guard. The pool quotas set the real hard limits. I'm in the process of migrating 100+TB right now and am really happy that I still have a client where I can do an O(1) move. It would be a disaster if I had now to use rsync or similar, which would take weeks. Please, in such situations where developers seem to have to make a definite choice, consider the possibility of offering operators to choose the alternative that suits their use case best. Adding further options seems far better than limiting functionality in a way that becomes a terrible burden in certain, if not many use cases. In ceph fs there have been many such decisions that allow for different answers from a user/operator perspective. For example, I would prefer if I could get rid of the attempted higher POSIX compliance level of ceph fs compared with Lustre, just disable all the client-caps and cache-coherence management and turn it into an awesome scale-out parallel file system. The attempt of POSIX compliant handling of simultaneous writes to files offers nothing to us, but costs huge in performance and forces users to move away from perfectly reasonable HPC work flows. Also, that it takes a TTL to expire before changes on one client become visible on another (unless direct_io is used for all IO) is perfectly acceptable for us given the potential performance gain due to simpler client-MDS communication. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Patrick Donnelly <pdonnell@redhat.com> Sent: 24 June 2021 05:29:45 To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] ceph fs mv does copy, not move Hello Frank, On Tue, Jun 22, 2021 at 2:16 AM Frank Schilder <frans@dtu.dk> wrote:

Dear all,

some time ago I reported that the kernel client resorts to a copy instead of move when moving a file across quota domains. I was told that the fuse client does not have this problem. If enough space is available, a move should be a move, not a copy.

Today, I tried to move a large file across quota domains testing botn, the kernel- and the fuse client. Both still resort to a copy even though this issue was addressed quite a while ago (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/44AEIHNEGKV4...). The versions I'm using are (CentOS 7)

# yum list installed | grep ceph-fuse ceph-fuse.x86_64 2:13.2.10-0.el7 @ceph

# uname -r 3.10.0-1160.31.1.el7.x86_64

Any suggestions how to get this to work? I have to move directories containing 100+ TB.

ceph-fuse reverted this behavior in: https://tracker.ceph.com/issues/48203 The kernel had a patch around that time too. In summary, it was not possible to accurately account for the quota usage prior to doing the rename. Rather than allow a quota to potentially be massively overrun, we fell back to the old behavior of not allowing it. -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

Hi Stefan,

Isn't that where LazyIO is for? See ...

Yes, it is, to some extend. However, there are many large HPC applications that will not start using exotic libraries for IO. A parallel file system offers everything that is needed with standard OS library calls. This is better solved on the FS than the client side. We put the link to lazy IO on our cluster documentation over a year ago, but I cannot imagine any of our users starting to invest porting massive applications even though we have ceph. So far, nobody did. Its also that HPC uses MPI, which comes with IO libraries users don't have influence on. I don't see this becoming a relevant alternative to a parallel file system any-time soon. Sorry. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Stefan Kooman <stefan@bit.nl> Sent: 24 June 2021 20:01:16 To: Frank Schilder; Patrick Donnelly Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: ceph fs mv does copy, not move On 6/24/21 5:34 PM, Frank Schilder wrote:

Please, in such situations where developers seem to have to make a definite choice, consider the possibility of offering operators to choose the alternative that suits their use case best. Adding further options seems far better than limiting functionality in a way that becomes a terrible burden in certain, if not many use cases.

Yeah, I agree.

In ceph fs there have been many such decisions that allow for different answers from a user/operator perspective. For example, I would prefer if I could get rid of the attempted higher POSIX compliance level of ceph fs compared with Lustre, just disable all the client-caps and cache-coherence management and turn it into an awesome scale-out parallel file system. The attempt of POSIX compliant handling of simultaneous writes to files offers nothing to us, but costs huge in performance and forces users to move away from perfectly reasonable HPC work flows. Also, that it takes a TTL to expire before changes on one client become visible on another (unless direct_io is used for all IO) is perfectly acceptable for us given the potential performance gain due to simpler client-MDS communication.

Isn't that where LazyIO is for? See https://docs.ceph.com/en/latest/cephfs/lazyio/ Gr. Stefan

Dear Marc

Adding to this. I can remember that I was surprised that a mv on cephfs between directories linked to different pools

This is documented behaviour and should not be surprising. Placement is assigned on file creation time. Hence, placement changes only affect newly created files, existing files retain their placement. To perform a migration, a full copy must be executed. This is, in fact, what is expected and also wished for (only unavoidable data movement should happen automatically, optional data movement on explicit request). A move is not a file creation and, therefore, cannot request a change of data placement. It is a request for re-linking a file/directory to another root (a move on an fs is a pointer-like operation, not a data operation), which is the reason why a move should be and usually is atomic and O(1). Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Marc <Marc@f1-outsourcing.eu> Sent: 25 June 2021 10:21:16 To: Frank Schilder; Patrick Donnelly Cc: ceph-users@ceph.io Subject: RE: [ceph-users] Re: ceph fs mv does copy, not move Adding to this. I can remember that I was surprised that a mv on cephfs between directories linked to different pools, only some meta(?) data was moved/changed and some data stayed still in the old pool. I am not sure if this is still the same in newer ceph versions, but I rather see data being moved completely. That is what everyone expects, regardless if this would take more time in this case between different pools.

-----Original Message----- From: Frank Schilder <frans@dtu.dk> Sent: Thursday, 24 June 2021 17:34 To: Patrick Donnelly <pdonnell@redhat.com> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: ceph fs mv does copy, not move

Dear Patrick,

thanks for letting me know.

Could you please consider to make this a ceph client mount option, for example, '-o fast_move', that enables a code path that enforces an mv to be a proper atomic mv with the risk that in some corner cases the target quota is overrun? With this option enabled, a move should either be a move or fail outright with "out of disk quota" (no partial move, no cp+rm at all). The fail should only occur if it is absolutely obvious that the target quota will be exceeded. Any corner cases are the responsibility of the operator. Application crashes due to incorrect error handling are acceptable.

Reasoning:

From a user's/operator's side, the preferred functionality is that in cases where a definite quota overrun can securely be detected in advance, the move should actually fail with "out of disk quota" instead of resorting to cp+rm, potentially leading to partial moves and a total mess for users/operators to clean up. In any other case, the quota should simply be ignored and the move should be a complete atomic move with the risk of exceeding the target quota and IO to stall. A temporary stall or fail of IO until the operator increases the quota again is, in my opinion and use case, highly preferable over the alternative of cp+rm. A quota or a crashed job is fast to fix, a partial move is not.

Some background:

We use ceph fs as an HPC home file system and as a back-end store. Being able to move data quickly across the entire file system is essential, because users re-factor their directory structure containing huge amounts of data quite often for various reasons.

On our system, we set file system quotas mainly for psychological reasons. We run a cron job that adjusts the quotas every day to show between 20% and 30% free capacity on the mount points. The psychological side here is to give an incentive to users to clean up temporary data. It is not intended to limit usage seriously, only to limit what can be done in between cron job runs as a safe-guard. The pool quotas set the real hard limits.

I'm in the process of migrating 100+TB right now and am really happy that I still have a client where I can do an O(1) move. It would be a disaster if I had now to use rsync or similar, which would take weeks.

Please, in such situations where developers seem to have to make a definite choice, consider the possibility of offering operators to choose the alternative that suits their use case best. Adding further options seems far better than limiting functionality in a way that becomes a terrible burden in certain, if not many use cases.

In ceph fs there have been many such decisions that allow for different answers from a user/operator perspective. For example, I would prefer if I could get rid of the attempted higher POSIX compliance level of ceph fs compared with Lustre, just disable all the client-caps and cache- coherence management and turn it into an awesome scale-out parallel file system. The attempt of POSIX compliant handling of simultaneous writes to files offers nothing to us, but costs huge in performance and forces users to move away from perfectly reasonable HPC work flows. Also, that it takes a TTL to expire before changes on one client become visible on another (unless direct_io is used for all IO) is perfectly acceptable for us given the potential performance gain due to simpler client-MDS communication.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Patrick Donnelly <pdonnell@redhat.com> Sent: 24 June 2021 05:29:45 To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] ceph fs mv does copy, not move

Hello Frank,

On Tue, Jun 22, 2021 at 2:16 AM Frank Schilder <frans@dtu.dk> wrote:

...

Dear all,

some time ago I reported that the kernel client resorts to a copy

instead of move when moving a file across quota domains. I was told that the fuse client does not have this problem. If enough space is available, a move should be a move, not a copy.

...

Today, I tried to move a large file across quota domains testing botn,

the kernel- and the fuse client. Both still resort to a copy even though this issue was addressed quite a while ago (https://lists.ceph.io/hyperkitty/list/ceph- users@ceph.io/thread/44AEIHNEGKV4VGCARRTARGFZ264CR4T7/#XY7ZCE3KWHI4QSUNZ HDWL3QZQFOHXRQW). The versions I'm using are (CentOS 7)

...

# yum list installed | grep ceph-fuse ceph-fuse.x86_64 2:13.2.10-0.el7

@ceph

...

# uname -r 3.10.0-1160.31.1.el7.x86_64

Any suggestions how to get this to work? I have to move directories

containing 100+ TB.

ceph-fuse reverted this behavior in: https://tracker.ceph.com/issues/48203 The kernel had a patch around that time too.

In summary, it was not possible to accurately account for the quota usage prior to doing the rename. Rather than allow a quota to potentially be massively overrun, we fell back to the old behavior of not allowing it.

-- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io