Unintuitive (buggy?) CephFS behaviour when dealing with pool_namespace layout attribute
Hello everyone, I'm seeing some behaviour in CephFS that strikes me as unexpected, and I wonder if others have thoughts about it. Consider this scenario: * Ceph Reef (18.2.4) deployed with Cephadm running on Ubuntu Jammy, CephFS client is running kernel 5.15.0-133-generic. * CephFS is mounted to /mnt, using a CephX identity that has rwp permissions. * CephFS is using a single data pool named cephfs_data. * In the CephFS root directory, there is a subdirectory named "restricted". That subdirectory currently has a single file, "test2.bin", of 8 MiB. There is no other data in the data pool. I can verify this with the following rados command: rados -n client.cephfs -p cephfs_data ls 10000000003.00000001 10000000003.00000000 Now I run this, from the root of my mounted CephFS: cd /mnt setfattr -n ceph.dir.layout.pool_namespace -v restricted /mnt/restricted I realise this is naughty, because there is currently a file named /mnt/restricted/test2.bin, and I'm not supposed to set layout attributes on a non-empty directory. However, the command does succeed, and I am able to read back the ceph.dir.layout xattr: getfattr -n ceph.dir.layout restricted # file: restricted ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data pool_namespace=restricted" If at this stage I try to read RADOS objects in the "restricted" namespace, I get an empty list: rados -p cephfs_data -N restricted ls (empty output) Now I move my test file out of the restricted subdirectory and back in: mv /mnt/restricted/test2.bin /mnt/ sync mv /mnt/test2.bin /mnt/restricted/ sync No change: rados -p cephfs_data -N restricted ls (empty output) Next, I try moving the file out, and *copying* it back in: mv /mnt/restricted/test2.bin /mnt/ sync cp /mnt/test2.bin /mnt/restricted/ sync Now, I do see objects in the "restricted" namespace: rados -p cephfs_data -N restricted ls 100000001fd.00000001 100000001fd.00000000 Also, if I create a *new* file in the "restricted" subdirectory, then its RADOS objects do end up in the correct namespace: dd if=/dev/urandom of=/mnt/restricted/test3.bin count=3 bs=4M rados -p cephfs_data -N restricted ls 100000001fe.00000000 100000001fe.00000002 100000001fe.00000001 100000001fd.00000001 100000001fd.00000000 By contrast, here's what happens to a non-empty *file,* when I try to set ceph.file.layout: dd if=/dev/urandom of=/mnt/test4.bin count=4 bs=4M setfattr -n ceph.file.layout.pool_namespace -v restricted test4.bin setfattr: test4.bin: Directory not empty So that fails, somewhat expectedly (although the error message is odd). In summary, there are two things that confuse me here: 1. Why does setting ceph.dir.layout.pool_namespace on a non-empty directory succeed, when setting ceph.file.layout.pool_namespace on a non-empty file fails (and even confusingly with a "Directory not empty" message)? 2. Considering that setting ceph.dir.layout.pool_namespace on a non-empty directory does succeed, why does mv'ing a file to a directory with a different pool_namespace, and then mv'ing it back, not result in its RADOS objects moving to the other namespace? So I'm curious: is this a bug (or two), or have I been misunderstanding what's actually the expected behaviour? Cheers, Florian
Hi, just some assumptions based on my experience with cephfs: - you cannot change existing objects; setting a different pool will not automagically move data. the data pool seems to be stored in the inode information - this also explains why changing the pool of a file does not work. ceph is unable to move the data on its own - so after changing the pool for a directory, all existing files will remain in their original pool - only newly created files will inherit the changed pool assignment - moving a file does not create a new inode, so its pool assignment cannot be changed by moving The last time we want to perform a large scale pool change we had to copy each file / directory to actually move the data. Best regards, Burkhard Linke
Hi Florian, Point 1 is certainly a bug regarding the choice of terms in the response (confusion between file and directory). Point 2 is known (cf. https://ewal.dev/cephfs-migrating-files-between-pools) and described in the documentation: only new files are written to the new pool after setting the new layout. The 'mv' command does not move the Rados objects associated with the file from one pool to another. Only reading and rewriting the file does. What reflections/impediments led to this? I do not know. But what you observed is expected. Hope this helps, Cheers, Frédéric. ________________________________ De : Florian Haas <florian.haas@cleura.com> Envoyé : mercredi 5 mars 2025 14:07 À : ceph-users Objet : [ceph-users] Unintuitive (buggy?) CephFS behaviour when dealing with pool_namespace layout attribute Hello everyone, I'm seeing some behaviour in CephFS that strikes me as unexpected, and I wonder if others have thoughts about it. Consider this scenario: * Ceph Reef (18.2.4) deployed with Cephadm running on Ubuntu Jammy, CephFS client is running kernel 5.15.0-133-generic. * CephFS is mounted to /mnt, using a CephX identity that has rwp permissions. * CephFS is using a single data pool named cephfs_data. * In the CephFS root directory, there is a subdirectory named "restricted". That subdirectory currently has a single file, "test2.bin", of 8 MiB. There is no other data in the data pool. I can verify this with the following rados command: rados -n client.cephfs -p cephfs_data ls 10000000003.00000001 10000000003.00000000 Now I run this, from the root of my mounted CephFS: cd /mnt setfattr -n ceph.dir.layout.pool_namespace -v restricted /mnt/restricted I realise this is naughty, because there is currently a file named /mnt/restricted/test2.bin, and I'm not supposed to set layout attributes on a non-empty directory. However, the command does succeed, and I am able to read back the ceph.dir.layout xattr: getfattr -n ceph.dir.layout restricted # file: restricted ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data pool_namespace=restricted" If at this stage I try to read RADOS objects in the "restricted" namespace, I get an empty list: rados -p cephfs_data -N restricted ls (empty output) Now I move my test file out of the restricted subdirectory and back in: mv /mnt/restricted/test2.bin /mnt/ sync mv /mnt/test2.bin /mnt/restricted/ sync No change: rados -p cephfs_data -N restricted ls (empty output) Next, I try moving the file out, and *copying* it back in: mv /mnt/restricted/test2.bin /mnt/ sync cp /mnt/test2.bin /mnt/restricted/ sync Now, I do see objects in the "restricted" namespace: rados -p cephfs_data -N restricted ls 100000001fd.00000001 100000001fd.00000000 Also, if I create a *new* file in the "restricted" subdirectory, then its RADOS objects do end up in the correct namespace: dd if=/dev/urandom of=/mnt/restricted/test3.bin count=3 bs=4M rados -p cephfs_data -N restricted ls 100000001fe.00000000 100000001fe.00000002 100000001fe.00000001 100000001fd.00000001 100000001fd.00000000 By contrast, here's what happens to a non-empty *file,* when I try to set ceph.file.layout: dd if=/dev/urandom of=/mnt/test4.bin count=4 bs=4M setfattr -n ceph.file.layout.pool_namespace -v restricted test4.bin setfattr: test4.bin: Directory not empty So that fails, somewhat expectedly (although the error message is odd). In summary, there are two things that confuse me here: 1. Why does setting ceph.dir.layout.pool_namespace on a non-empty directory succeed, when setting ceph.file.layout.pool_namespace on a non-empty file fails (and even confusingly with a "Directory not empty" message)? 2. Considering that setting ceph.dir.layout.pool_namespace on a non-empty directory does succeed, why does mv'ing a file to a directory with a different pool_namespace, and then mv'ing it back, not result in its RADOS objects moving to the other namespace? So I'm curious: is this a bug (or two), or have I been misunderstanding what's actually the expected behaviour? Cheers, Florian
On 05/03/2025 20:45, Frédéric Nass wrote:
Hi Florian,
Point 1 is certainly a bug regarding the choice of terms in the response (confusion between file and directory).
Well... no, I don't think so. Rather, I'd guess it's simply a result of setfattr returning ENOTEMPTY (errno 39), which the shell then renders as "Directory not empty". Perhaps setting the ceph.file.layout attribute on an empty file should really result in EPERM (errno 1) though. Or maybe even EFBIG (errno 27, "File too large), because the operation is only supported on a file that is 0 bytes in length, and fails on any file larger than that. But, like I said, to avoid confusion I really do think that a setfattr command manipulating ceph.dir.layout on a non-empty *directory* ought to return ENOTEMPTY, which it currently does not. Cheers, Florian
On 06/03/2025 09:46, Florian Haas wrote:
On 05/03/2025 20:45, Frédéric Nass wrote:
Hi Florian,
Point 1 is certainly a bug regarding the choice of terms in the response (confusion between file and directory).
Well... no, I don't think so. Rather, I'd guess it's simply a result of setfattr returning ENOTEMPTY (errno 39), which the shell then renders as "Directory not empty".
Perhaps setting the ceph.file.layout attribute on an empty file should really result in EPERM (errno 1) though. Or maybe even EFBIG (errno 27, "File too large), because the operation is only supported on a file that is 0 bytes in length, and fails on any file larger than that.
Sorry, confusing error. I meant to posit, of course, that setting the ceph.file.layout attribute on a *non*-empty file should result in EPERM or EFBIG, rather than ENOTEMPTY. Apologies.
This is certainly intended behavior. If you checked the layout on the particular file, you would see it hasn’t changed. Directory layouts are the default for new files, not a control mechanism for existing files. It might be confusing, so we can talk about different presentations if there’s a better option (though this has been the behavior as long as we’ve had this functionality), but auto-migrating data is a significant challenge we haven’t sorted through and will have distinct controls if we ever manage to make it happen. -Greg On Wed, Mar 5, 2025 at 5:06 AM Florian Haas <florian.haas@cleura.com> wrote:
Hello everyone,
I'm seeing some behaviour in CephFS that strikes me as unexpected, and I wonder if others have thoughts about it.
Consider this scenario:
* Ceph Reef (18.2.4) deployed with Cephadm running on Ubuntu Jammy, CephFS client is running kernel 5.15.0-133-generic. * CephFS is mounted to /mnt, using a CephX identity that has rwp permissions. * CephFS is using a single data pool named cephfs_data. * In the CephFS root directory, there is a subdirectory named "restricted". That subdirectory currently has a single file, "test2.bin", of 8 MiB.
There is no other data in the data pool. I can verify this with the following rados command:
rados -n client.cephfs -p cephfs_data ls 10000000003.00000001 10000000003.00000000
Now I run this, from the root of my mounted CephFS:
cd /mnt setfattr -n ceph.dir.layout.pool_namespace -v restricted /mnt/restricted
I realise this is naughty, because there is currently a file named /mnt/restricted/test2.bin, and I'm not supposed to set layout attributes on a non-empty directory. However, the command does succeed, and I am able to read back the ceph.dir.layout xattr:
getfattr -n ceph.dir.layout restricted # file: restricted ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data pool_namespace=restricted"
If at this stage I try to read RADOS objects in the "restricted" namespace, I get an empty list:
rados -p cephfs_data -N restricted ls (empty output)
Now I move my test file out of the restricted subdirectory and back in:
mv /mnt/restricted/test2.bin /mnt/ sync mv /mnt/test2.bin /mnt/restricted/ sync
No change:
rados -p cephfs_data -N restricted ls (empty output)
Next, I try moving the file out, and *copying* it back in:
mv /mnt/restricted/test2.bin /mnt/ sync cp /mnt/test2.bin /mnt/restricted/ sync
Now, I do see objects in the "restricted" namespace:
rados -p cephfs_data -N restricted ls 100000001fd.00000001 100000001fd.00000000
Also, if I create a *new* file in the "restricted" subdirectory, then its RADOS objects do end up in the correct namespace:
dd if=/dev/urandom of=/mnt/restricted/test3.bin count=3 bs=4M
rados -p cephfs_data -N restricted ls 100000001fe.00000000 100000001fe.00000002 100000001fe.00000001 100000001fd.00000001 100000001fd.00000000
By contrast, here's what happens to a non-empty *file,* when I try to set ceph.file.layout:
dd if=/dev/urandom of=/mnt/test4.bin count=4 bs=4M
setfattr -n ceph.file.layout.pool_namespace -v restricted test4.bin setfattr: test4.bin: Directory not empty
So that fails, somewhat expectedly (although the error message is odd).
In summary, there are two things that confuse me here:
1. Why does setting ceph.dir.layout.pool_namespace on a non-empty directory succeed, when setting ceph.file.layout.pool_namespace on a non-empty file fails (and even confusingly with a "Directory not empty" message)?
2. Considering that setting ceph.dir.layout.pool_namespace on a non-empty directory does succeed, why does mv'ing a file to a directory with a different pool_namespace, and then mv'ing it back, not result in its RADOS objects moving to the other namespace?
So I'm curious: is this a bug (or two), or have I been misunderstanding what's actually the expected behaviour?
Cheers, Florian _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Burkhard Linke
-
Florian Haas
-
Frédéric Nass
-
Gregory Farnum