Some time ago I made a surprising observation. I reorganised a directory structure and needed to move a folder one level up with a command like mv A/B/ B B contained something like 9TB in very large files. To my surprise, this command didn't return for a couple of minutes and I started to look what was going on. What I discovered was, that the mv command actually performed a full copy with a subsequent remove. I had to wait for several hours for the move to complete. I tried to reproduce this today to collect further information. However, this behaviour seems not reproducible. No matter what I try, mv completes almost instantly. I was running the original mv on mimic 13.2.2 and retried now with mimic 13.2.8. In addition, there was an OS upgrade from Centos 7.6 to 7.7. I'm using the kernel-ml versions (5.xxx). Only one cephfs mount was present at all times. My questions are: 1) Was there a change from 13.2.2 to 13.2.8 explaining this? 2) Are there (rare) conditions under which an mv on cephfs becomes a cp+rm? 3) Am I seeing ghosts? Thanks for clues and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
On Thu, Mar 26, 2020 at 5:49 AM Frank Schilder <frans@dtu.dk> wrote:
Some time ago I made a surprising observation. I reorganised a directory structure and needed to move a folder one level up with a command like
mv A/B/ B
B contained something like 9TB in very large files. To my surprise, this command didn't return for a couple of minutes and I started to look what was going on. What I discovered was, that the mv command actually performed a full copy with a subsequent remove. I had to wait for several hours for the move to complete.
I tried to reproduce this today to collect further information. However, this behaviour seems not reproducible. No matter what I try, mv completes almost instantly.
I was running the original mv on mimic 13.2.2 and retried now with mimic 13.2.8. In addition, there was an OS upgrade from Centos 7.6 to 7.7. I'm using the kernel-ml versions (5.xxx). Only one cephfs mount was present at all times.
My questions are:
1) Was there a change from 13.2.2 to 13.2.8 explaining this?
No.
2) Are there (rare) conditions under which an mv on cephfs becomes a cp+rm?
Not as part of CephFS.
3) Am I seeing ghosts?
The only way I can imagine this happening is if you had separate CephFS mounts for cephfs/A/B and cephfs/ and you did the move between them, instead of within the cephfs/ mount. :/ -Greg
Thanks for clues and best regards,
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Is there is a possibility that there was a quota involved? I've seen moves between quota zones to cause a copy then delete. -- Adam On Thu, Mar 26, 2020 at 9:14 AM Gregory Farnum <gfarnum@redhat.com> wrote:
On Thu, Mar 26, 2020 at 5:49 AM Frank Schilder <frans@dtu.dk> wrote:
Some time ago I made a surprising observation. I reorganised a directory structure and needed to move a folder one level up with a command like
mv A/B/ B
B contained something like 9TB in very large files. To my surprise, this command didn't return for a couple of minutes and I started to look what was going on. What I discovered was, that the mv command actually performed a full copy with a subsequent remove. I had to wait for several hours for the move to complete.
I tried to reproduce this today to collect further information. However, this behaviour seems not reproducible. No matter what I try, mv completes almost instantly.
I was running the original mv on mimic 13.2.2 and retried now with mimic 13.2.8. In addition, there was an OS upgrade from Centos 7.6 to 7.7. I'm using the kernel-ml versions (5.xxx). Only one cephfs mount was present at all times.
My questions are:
1) Was there a change from 13.2.2 to 13.2.8 explaining this?
No.
2) Are there (rare) conditions under which an mv on cephfs becomes a cp+rm?
Not as part of CephFS.
3) Am I seeing ghosts?
The only way I can imagine this happening is if you had separate CephFS mounts for cephfs/A/B and cephfs/ and you did the move between them, instead of within the cephfs/ mount. :/ -Greg
Thanks for clues and best regards,
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I was wondering that but at least in the userspace client the quotas will just throw EXDEV or EDQUOT if it will exceed quotas... ...and EXDEV might trigger the kernel to do a copy-and-delete, I guess? Not sure. On Thu, Mar 26, 2020 at 7:18 AM Adam Tygart <mozes@ksu.edu> wrote:
Is there is a possibility that there was a quota involved? I've seen moves between quota zones to cause a copy then delete.
-- Adam
On Thu, Mar 26, 2020 at 9:14 AM Gregory Farnum <gfarnum@redhat.com> wrote:
On Thu, Mar 26, 2020 at 5:49 AM Frank Schilder <frans@dtu.dk> wrote:
Some time ago I made a surprising observation. I reorganised a directory structure and needed to move a folder one level up with a command like
mv A/B/ B
B contained something like 9TB in very large files. To my surprise, this command didn't return for a couple of minutes and I started to look what was going on. What I discovered was, that the mv command actually performed a full copy with a subsequent remove. I had to wait for several hours for the move to complete.
I tried to reproduce this today to collect further information. However, this behaviour seems not reproducible. No matter what I try, mv completes almost instantly.
I was running the original mv on mimic 13.2.2 and retried now with mimic 13.2.8. In addition, there was an OS upgrade from Centos 7.6 to 7.7. I'm using the kernel-ml versions (5.xxx). Only one cephfs mount was present at all times.
My questions are:
1) Was there a change from 13.2.2 to 13.2.8 explaining this?
No.
2) Are there (rare) conditions under which an mv on cephfs becomes a cp+rm?
Not as part of CephFS.
3) Am I seeing ghosts?
The only way I can imagine this happening is if you had separate CephFS mounts for cephfs/A/B and cephfs/ and you did the move between them, instead of within the cephfs/ mount. :/ -Greg
Thanks for clues and best regards,
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Dear all, thanks for the hints. I was afraid I see ghosts. Yes, we have quotas on many directories. I'm not entirely sure if there were different quotas involved. I believe it was set only on the highest level, but now that you mention it ... I will try with quotas again, didn't include this in my scenario. This information is really important, because it will affect how I as an admin can move data between folders with quotas on them, which are typically used for sub-directory mounts. For a user this would look like a magical move between file systems. I will get back. Thanks! Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Gregory Farnum <gfarnum@redhat.com> Sent: 26 March 2020 15:20:51 To: Beocat KSU Cc: ceph-users; Frank Schilder Subject: Re: [ceph-users] Re: Move on cephfs not O(1)? I was wondering that but at least in the userspace client the quotas will just throw EXDEV or EDQUOT if it will exceed quotas... ...and EXDEV might trigger the kernel to do a copy-and-delete, I guess? Not sure. On Thu, Mar 26, 2020 at 7:18 AM Adam Tygart <mozes@ksu.edu> wrote:
Is there is a possibility that there was a quota involved? I've seen moves between quota zones to cause a copy then delete.
-- Adam
On Thu, Mar 26, 2020 at 9:14 AM Gregory Farnum <gfarnum@redhat.com> wrote:
On Thu, Mar 26, 2020 at 5:49 AM Frank Schilder <frans@dtu.dk> wrote:
Some time ago I made a surprising observation. I reorganised a directory structure and needed to move a folder one level up with a command like
mv A/B/ B
B contained something like 9TB in very large files. To my surprise, this command didn't return for a couple of minutes and I started to look what was going on. What I discovered was, that the mv command actually performed a full copy with a subsequent remove. I had to wait for several hours for the move to complete.
I tried to reproduce this today to collect further information. However, this behaviour seems not reproducible. No matter what I try, mv completes almost instantly.
I was running the original mv on mimic 13.2.2 and retried now with mimic 13.2.8. In addition, there was an OS upgrade from Centos 7.6 to 7.7. I'm using the kernel-ml versions (5.xxx). Only one cephfs mount was present at all times.
My questions are:
1) Was there a change from 13.2.2 to 13.2.8 explaining this?
No.
2) Are there (rare) conditions under which an mv on cephfs becomes a cp+rm?
Not as part of CephFS.
3) Am I seeing ghosts?
The only way I can imagine this happening is if you had separate CephFS mounts for cephfs/A/B and cephfs/ and you did the move between them, instead of within the cephfs/ mount. :/ -Greg
Thanks for clues and best regards,
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Dear all, yes, this is it, quotas. In the structure A/B/ there was a quota set on A. Hence, B was moved out of this zone and this does indeed change mv to be a cp+rm. The obvious follow-up. What is the procedure for properly moving data as an administrator? Do I really need to unset quotas, do the move and set quotas back again? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 26 March 2020 16:35:07 To: Gregory Farnum; Beocat KSU Cc: ceph-users Subject: [ceph-users] Re: Move on cephfs not O(1)? Dear all, thanks for the hints. I was afraid I see ghosts. Yes, we have quotas on many directories. I'm not entirely sure if there were different quotas involved. I believe it was set only on the highest level, but now that you mention it ... I will try with quotas again, didn't include this in my scenario. This information is really important, because it will affect how I as an admin can move data between folders with quotas on them, which are typically used for sub-directory mounts. For a user this would look like a magical move between file systems. I will get back. Thanks! Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Gregory Farnum <gfarnum@redhat.com> Sent: 26 March 2020 15:20:51 To: Beocat KSU Cc: ceph-users; Frank Schilder Subject: Re: [ceph-users] Re: Move on cephfs not O(1)? I was wondering that but at least in the userspace client the quotas will just throw EXDEV or EDQUOT if it will exceed quotas... ...and EXDEV might trigger the kernel to do a copy-and-delete, I guess? Not sure. On Thu, Mar 26, 2020 at 7:18 AM Adam Tygart <mozes@ksu.edu> wrote:
Is there is a possibility that there was a quota involved? I've seen moves between quota zones to cause a copy then delete.
-- Adam
On Thu, Mar 26, 2020 at 9:14 AM Gregory Farnum <gfarnum@redhat.com> wrote:
On Thu, Mar 26, 2020 at 5:49 AM Frank Schilder <frans@dtu.dk> wrote:
Some time ago I made a surprising observation. I reorganised a directory structure and needed to move a folder one level up with a command like
mv A/B/ B
B contained something like 9TB in very large files. To my surprise, this command didn't return for a couple of minutes and I started to look what was going on. What I discovered was, that the mv command actually performed a full copy with a subsequent remove. I had to wait for several hours for the move to complete.
I tried to reproduce this today to collect further information. However, this behaviour seems not reproducible. No matter what I try, mv completes almost instantly.
I was running the original mv on mimic 13.2.2 and retried now with mimic 13.2.8. In addition, there was an OS upgrade from Centos 7.6 to 7.7. I'm using the kernel-ml versions (5.xxx). Only one cephfs mount was present at all times.
My questions are:
1) Was there a change from 13.2.2 to 13.2.8 explaining this?
No.
2) Are there (rare) conditions under which an mv on cephfs becomes a cp+rm?
Not as part of CephFS.
3) Am I seeing ghosts?
The only way I can imagine this happening is if you had separate CephFS mounts for cephfs/A/B and cephfs/ and you did the move between them, instead of within the cephfs/ mount. :/ -Greg
Thanks for clues and best regards,
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Thu, Mar 26, 2020 at 9:13 AM Frank Schilder <frans@dtu.dk> wrote:
Dear all,
yes, this is it, quotas. In the structure A/B/ there was a quota set on A. Hence, B was moved out of this zone and this does indeed change mv to be a cp+rm.
The obvious follow-up. What is the procedure for properly moving data as an administrator? Do I really need to unset quotas, do the move and set quotas back again?
Ah-ha, this is a difference between the userspace and kernel clients. :( The kernel client always returns EXDEV if crossing "quota realms" (different quota roots). I'm not sure why it behaves that way as userspace is different: * If fhere is a quota in the target directory, and * If the target directory's existing data, plus the file(s) being moved, exceed the target directory's quota, then userspace returns EXDEV/EDQUOT. This seems like the right kernel behavior as well... Jeff, is that a known issue for some reason? Should we make a new bug? :) -Greg
Best regards,
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 26 March 2020 16:35:07 To: Gregory Farnum; Beocat KSU Cc: ceph-users Subject: [ceph-users] Re: Move on cephfs not O(1)?
Dear all, thanks for the hints. I was afraid I see ghosts.
Yes, we have quotas on many directories. I'm not entirely sure if there were different quotas involved. I believe it was set only on the highest level, but now that you mention it ...
I will try with quotas again, didn't include this in my scenario.
This information is really important, because it will affect how I as an admin can move data between folders with quotas on them, which are typically used for sub-directory mounts. For a user this would look like a magical move between file systems.
I will get back. Thanks!
Best regards,
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Gregory Farnum <gfarnum@redhat.com> Sent: 26 March 2020 15:20:51 To: Beocat KSU Cc: ceph-users; Frank Schilder Subject: Re: [ceph-users] Re: Move on cephfs not O(1)?
I was wondering that but at least in the userspace client the quotas will just throw EXDEV or EDQUOT if it will exceed quotas... ...and EXDEV might trigger the kernel to do a copy-and-delete, I guess? Not sure.
On Thu, Mar 26, 2020 at 7:18 AM Adam Tygart <mozes@ksu.edu> wrote:
Is there is a possibility that there was a quota involved? I've seen moves between quota zones to cause a copy then delete.
-- Adam
On Thu, Mar 26, 2020 at 9:14 AM Gregory Farnum <gfarnum@redhat.com> wrote:
On Thu, Mar 26, 2020 at 5:49 AM Frank Schilder <frans@dtu.dk> wrote:
Some time ago I made a surprising observation. I reorganised a directory structure and needed to move a folder one level up with a command like
mv A/B/ B
B contained something like 9TB in very large files. To my surprise, this command didn't return for a couple of minutes and I started to look what was going on. What I discovered was, that the mv command actually performed a full copy with a subsequent remove. I had to wait for several hours for the move to complete.
I tried to reproduce this today to collect further information. However, this behaviour seems not reproducible. No matter what I try, mv completes almost instantly.
I was running the original mv on mimic 13.2.2 and retried now with mimic 13.2.8. In addition, there was an OS upgrade from Centos 7.6 to 7.7. I'm using the kernel-ml versions (5.xxx). Only one cephfs mount was present at all times.
My questions are:
1) Was there a change from 13.2.2 to 13.2.8 explaining this?
No.
2) Are there (rare) conditions under which an mv on cephfs becomes a cp+rm?
Not as part of CephFS.
3) Am I seeing ghosts?
The only way I can imagine this happening is if you had separate CephFS mounts for cephfs/A/B and cephfs/ and you did the move between them, instead of within the cephfs/ mount. :/ -Greg
Thanks for clues and best regards,
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Thu, 2020-03-26 at 10:32 -0700, Gregory Farnum wrote:
On Thu, Mar 26, 2020 at 9:13 AM Frank Schilder <frans@dtu.dk> wrote:
Dear all,
yes, this is it, quotas. In the structure A/B/ there was a quota set on A. Hence, B was moved out of this zone and this does indeed change mv to be a cp+rm.
The obvious follow-up. What is the procedure for properly moving data as an administrator? Do I really need to unset quotas, do the move and set quotas back again?
Ah-ha, this is a difference between the userspace and kernel clients. :( The kernel client always returns EXDEV if crossing "quota realms" (different quota roots). I'm not sure why it behaves that way as userspace is different: * If fhere is a quota in the target directory, and * If the target directory's existing data, plus the file(s) being moved, exceed the target directory's quota, then userspace returns EXDEV/EDQUOT. This seems like the right kernel behavior as well...
Jeff, is that a known issue for some reason? Should we make a new bug? :)
(cc'ing Luis since he wrote cafe21a4fb3075, which added this check) In ceph_rename, we have this, which is probably what you're hitting: /* don't allow cross-quota renames */ if ((old_dir != new_dir) && (!ceph_quota_is_same_realm(old_dir, new_dir))) return -EXDEV; That does seem to just check whether the realms are the same and doesn't actually check the space in them. I haven't studied this in detail, but it may be hard to ensure that we won't exceed the quota in a race-free way. How do you ensure that another thread doesn't do something that would exceed the quota just as you issue the rename request? This is fairly trivial in the userland client since it does everything under the BCM (Big Client Mutex), but won't be in the kernel client. Opening a bug for this won't hurt, but it may not be simple to implement. -- Jeff Layton <jlayton@redhat.com>
On Fri, 2020-03-27 at 07:36 -0400, Jeff Layton wrote:
On Thu, 2020-03-26 at 10:32 -0700, Gregory Farnum wrote:
On Thu, Mar 26, 2020 at 9:13 AM Frank Schilder <frans@dtu.dk> wrote:
Dear all,
yes, this is it, quotas. In the structure A/B/ there was a quota set on A. Hence, B was moved out of this zone and this does indeed change mv to be a cp+rm.
The obvious follow-up. What is the procedure for properly moving data as an administrator? Do I really need to unset quotas, do the move and set quotas back again?
Ah-ha, this is a difference between the userspace and kernel clients. :( The kernel client always returns EXDEV if crossing "quota realms" (different quota roots). I'm not sure why it behaves that way as userspace is different: * If fhere is a quota in the target directory, and * If the target directory's existing data, plus the file(s) being moved, exceed the target directory's quota, then userspace returns EXDEV/EDQUOT. This seems like the right kernel behavior as well...
Jeff, is that a known issue for some reason? Should we make a new bug? :)
(cc'ing Luis since he wrote cafe21a4fb3075, which added this check)
In ceph_rename, we have this, which is probably what you're hitting:
/* don't allow cross-quota renames */ if ((old_dir != new_dir) && (!ceph_quota_is_same_realm(old_dir, new_dir))) return -EXDEV;
That does seem to just check whether the realms are the same and doesn't actually check the space in them.
I haven't studied this in detail, but it may be hard to ensure that we won't exceed the quota in a race-free way. How do you ensure that another thread doesn't do something that would exceed the quota just as you issue the rename request?
This is fairly trivial in the userland client since it does everything under the BCM (Big Client Mutex), but won't be in the kernel client.
Opening a bug for this won't hurt, but it may not be simple to implement.
Ok, I gave it a harder look and it doesn't look too bad, given that quotas are sort of "sloppy" anyway. We could just do a check that we won't exceed quota on the destination directory. The catch is that there may be other details to deal with too in moving an existing inode under a different snaprealm. You're welcome to open a tracker bug for this, but I'm not sure who will take on that project and when. Thanks, -- Jeff Layton <jlayton@redhat.com>
Made a ticket in the Linux Kernel Client tracker: https://tracker.ceph.com/issues/44791 I naively don't think this should be very complicated at all, except that I recall hearing something about locking issues with quota realms in the kclient? But the userspace client definitely doesn't have to do anything tricky with moving things, it's pretty much a normal move with some glue to point at a different quota realm. -Greg On Fri, Mar 27, 2020 at 7:34 AM Jeff Layton <jlayton@redhat.com> wrote:
On Fri, 2020-03-27 at 07:36 -0400, Jeff Layton wrote:
On Thu, 2020-03-26 at 10:32 -0700, Gregory Farnum wrote:
On Thu, Mar 26, 2020 at 9:13 AM Frank Schilder <frans@dtu.dk> wrote:
Dear all,
yes, this is it, quotas. In the structure A/B/ there was a quota set on A. Hence, B was moved out of this zone and this does indeed change mv to be a cp+rm.
The obvious follow-up. What is the procedure for properly moving data as an administrator? Do I really need to unset quotas, do the move and set quotas back again?
Ah-ha, this is a difference between the userspace and kernel clients. :( The kernel client always returns EXDEV if crossing "quota realms" (different quota roots). I'm not sure why it behaves that way as userspace is different: * If fhere is a quota in the target directory, and * If the target directory's existing data, plus the file(s) being moved, exceed the target directory's quota, then userspace returns EXDEV/EDQUOT. This seems like the right kernel behavior as well...
Jeff, is that a known issue for some reason? Should we make a new bug? :)
(cc'ing Luis since he wrote cafe21a4fb3075, which added this check)
In ceph_rename, we have this, which is probably what you're hitting:
/* don't allow cross-quota renames */ if ((old_dir != new_dir) && (!ceph_quota_is_same_realm(old_dir, new_dir))) return -EXDEV;
That does seem to just check whether the realms are the same and doesn't actually check the space in them.
I haven't studied this in detail, but it may be hard to ensure that we won't exceed the quota in a race-free way. How do you ensure that another thread doesn't do something that would exceed the quota just as you issue the rename request?
This is fairly trivial in the userland client since it does everything under the BCM (Big Client Mutex), but won't be in the kernel client.
Opening a bug for this won't hurt, but it may not be simple to implement.
Ok, I gave it a harder look and it doesn't look too bad, given that quotas are sort of "sloppy" anyway. We could just do a check that we won't exceed quota on the destination directory. The catch is that there may be other details to deal with too in moving an existing inode under a different snaprealm.
You're welcome to open a tracker bug for this, but I'm not sure who will take on that project and when.
Thanks, -- Jeff Layton <jlayton@redhat.com>
Thanks a lot! Have a good weekend. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Gregory Farnum <gfarnum@redhat.com> Sent: 27 March 2020 21:18:37 To: Jeff Layton Cc: Frank Schilder; Zheng Yan; ceph-users; Luis Henriques Subject: Re: [ceph-users] Re: Move on cephfs not O(1)? Made a ticket in the Linux Kernel Client tracker: https://tracker.ceph.com/issues/44791 I naively don't think this should be very complicated at all, except that I recall hearing something about locking issues with quota realms in the kclient? But the userspace client definitely doesn't have to do anything tricky with moving things, it's pretty much a normal move with some glue to point at a different quota realm. -Greg On Fri, Mar 27, 2020 at 7:34 AM Jeff Layton <jlayton@redhat.com> wrote:
On Fri, 2020-03-27 at 07:36 -0400, Jeff Layton wrote:
On Thu, 2020-03-26 at 10:32 -0700, Gregory Farnum wrote:
On Thu, Mar 26, 2020 at 9:13 AM Frank Schilder <frans@dtu.dk> wrote:
Dear all,
yes, this is it, quotas. In the structure A/B/ there was a quota set on A. Hence, B was moved out of this zone and this does indeed change mv to be a cp+rm.
The obvious follow-up. What is the procedure for properly moving data as an administrator? Do I really need to unset quotas, do the move and set quotas back again?
Ah-ha, this is a difference between the userspace and kernel clients. :( The kernel client always returns EXDEV if crossing "quota realms" (different quota roots). I'm not sure why it behaves that way as userspace is different: * If fhere is a quota in the target directory, and * If the target directory's existing data, plus the file(s) being moved, exceed the target directory's quota, then userspace returns EXDEV/EDQUOT. This seems like the right kernel behavior as well...
Jeff, is that a known issue for some reason? Should we make a new bug? :)
(cc'ing Luis since he wrote cafe21a4fb3075, which added this check)
In ceph_rename, we have this, which is probably what you're hitting:
/* don't allow cross-quota renames */ if ((old_dir != new_dir) && (!ceph_quota_is_same_realm(old_dir, new_dir))) return -EXDEV;
That does seem to just check whether the realms are the same and doesn't actually check the space in them.
I haven't studied this in detail, but it may be hard to ensure that we won't exceed the quota in a race-free way. How do you ensure that another thread doesn't do something that would exceed the quota just as you issue the rename request?
This is fairly trivial in the userland client since it does everything under the BCM (Big Client Mutex), but won't be in the kernel client.
Opening a bug for this won't hurt, but it may not be simple to implement.
Ok, I gave it a harder look and it doesn't look too bad, given that quotas are sort of "sloppy" anyway. We could just do a check that we won't exceed quota on the destination directory. The catch is that there may be other details to deal with too in moving an existing inode under a different snaprealm.
You're welcome to open a tracker bug for this, but I'm not sure who will take on that project and when.
Thanks, -- Jeff Layton <jlayton@redhat.com>
On Fri, Mar 27, 2020 at 07:36:15AM -0400, Jeff Layton wrote:
On Thu, 2020-03-26 at 10:32 -0700, Gregory Farnum wrote:
On Thu, Mar 26, 2020 at 9:13 AM Frank Schilder <frans@dtu.dk> wrote:
Dear all,
yes, this is it, quotas. In the structure A/B/ there was a quota set on A. Hence, B was moved out of this zone and this does indeed change mv to be a cp+rm.
The obvious follow-up. What is the procedure for properly moving data as an administrator? Do I really need to unset quotas, do the move and set quotas back again?
Ah-ha, this is a difference between the userspace and kernel clients. :( The kernel client always returns EXDEV if crossing "quota realms" (different quota roots). I'm not sure why it behaves that way as userspace is different: * If fhere is a quota in the target directory, and * If the target directory's existing data, plus the file(s) being moved, exceed the target directory's quota, then userspace returns EXDEV/EDQUOT. This seems like the right kernel behavior as well...
Jeff, is that a known issue for some reason? Should we make a new bug? :)
(cc'ing Luis since he wrote cafe21a4fb3075, which added this check)
In ceph_rename, we have this, which is probably what you're hitting:
/* don't allow cross-quota renames */ if ((old_dir != new_dir) && (!ceph_quota_is_same_realm(old_dir, new_dir))) return -EXDEV;
That does seem to just check whether the realms are the same and doesn't actually check the space in them.
(Sorry for the delay replying.) So, I did a bit of digging (it has been a while since last time I looked at this code) and the reason for this -EXDEV was to actually mimic the code in the userspace client code (see bbfeaaea53f1 ("client: Forbid moving files across quota trees")). In the meantime, it seems like this restriction has been removed from the userspace client in commit b8954e5734b3 ("client: optimize rename operation under different quota root"). I'll try to have a look at this in the next few days and see how this could be included in the kernel client as well. Cheers, -- Luis
I haven't studied this in detail, but it may be hard to ensure that we won't exceed the quota in a race-free way. How do you ensure that another thread doesn't do something that would exceed the quota just as you issue the rename request?
This is fairly trivial in the userland client since it does everything under the BCM (Big Client Mutex), but won't be in the kernel client.
Opening a bug for this won't hurt, but it may not be simple to implement. -- Jeff Layton <jlayton@redhat.com> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Adam Tygart
-
Frank Schilder
-
Gregory Farnum
-
Jeff Layton
-
Luis Henriques