Unexpected behavior of directory mtime after being set explicitly
Hi Team, I'm writing to bring to your attention an issue we have encountered with the "mtime" (modification time) behavior for directories in the Ceph filesystem. Upon observation, we have noticed that when the mtime of a directory (let's say: dir1) is explicitly changed in CephFS, subsequent additions of files or directories within 'dir1' fail to update the directory's mtime as expected. This behavior appears to be specific to CephFS - we have reproduced this issue on both Quincy and Pacific. Similar steps work as expected in the ext4 filesystem amongst others. Reproduction steps: 1. Create a directory - mkdir dir1 2. Modify mtime using the touch command - touch dir1 3. Create a file or directory inside of 'dir1' - mkdir dir1/dir2 Expected result: mtime for dir1 should change to the time the file or directory was created in step 3 Actual result: there was no change to the mtime for 'dir1' Note : For more detail, kindly find the attached logs. Our queries are : 1. Is this expected behavior for CephFS? 2. If so, can you explain why the directory behavior is inconsistent depending on whether the mtime for the directory has previously been manually updated. Best Regards, Sandip Divekar Component QA Lead SDET.
Sandip, What type of client are you using ? kernel client or fuse client ? If it's the kernel client, then it's a bug. FYI - Pacific and Quincy fuse clients do the right thing On Wed, May 24, 2023 at 9:24 PM Sandip Divekar < sandip.divekar@hitachivantara.com> wrote:
Hi Team,
I'm writing to bring to your attention an issue we have encountered with the "mtime" (modification time) behavior for directories in the Ceph filesystem.
Upon observation, we have noticed that when the mtime of a directory (let's say: dir1) is explicitly changed in CephFS, subsequent additions of files or directories within 'dir1' fail to update the directory's mtime as expected.
This behavior appears to be specific to CephFS - we have reproduced this issue on both Quincy and Pacific. Similar steps work as expected in the ext4 filesystem amongst others.
Reproduction steps: 1. Create a directory - mkdir dir1 2. Modify mtime using the touch command - touch dir1 3. Create a file or directory inside of 'dir1' - mkdir dir1/dir2 Expected result: mtime for dir1 should change to the time the file or directory was created in step 3 Actual result: there was no change to the mtime for 'dir1'
Note : For more detail, kindly find the attached logs.
Our queries are : 1. Is this expected behavior for CephFS? 2. If so, can you explain why the directory behavior is inconsistent depending on whether the mtime for the directory has previously been manually updated.
Best Regards, Sandip Divekar Component QA Lead SDET.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Milind
Hello Milind, We are using Ceph Kernel Client. But we found this same behavior while using Libcephfs library. Should we treat this as a bug? Or Is there any existing bug for similar issue ? Thanks and Regards, Sandip Divekar From: Milind Changire <mchangir@redhat.com> Sent: Thursday, May 25, 2023 4:24 PM To: Sandip Divekar <sandip.divekar@hitachivantara.com> Cc: ceph-users@ceph.io; dev@ceph.io Subject: Re: [ceph-users] Unexpected behavior of directory mtime after being set explicitly ***** EXTERNAL EMAIL ***** Sandip, What type of client are you using ? kernel client or fuse client ? If it's the kernel client, then it's a bug. FYI - Pacific and Quincy fuse clients do the right thing On Wed, May 24, 2023 at 9:24 PM Sandip Divekar <sandip.divekar@hitachivantara.com<mailto:sandip.divekar@hitachivantara.com>> wrote: Hi Team, I'm writing to bring to your attention an issue we have encountered with the "mtime" (modification time) behavior for directories in the Ceph filesystem. Upon observation, we have noticed that when the mtime of a directory (let's say: dir1) is explicitly changed in CephFS, subsequent additions of files or directories within 'dir1' fail to update the directory's mtime as expected. This behavior appears to be specific to CephFS - we have reproduced this issue on both Quincy and Pacific. Similar steps work as expected in the ext4 filesystem amongst others. Reproduction steps: 1. Create a directory - mkdir dir1 2. Modify mtime using the touch command - touch dir1 3. Create a file or directory inside of 'dir1' - mkdir dir1/dir2 Expected result: mtime for dir1 should change to the time the file or directory was created in step 3 Actual result: there was no change to the mtime for 'dir1' Note : For more detail, kindly find the attached logs. Our queries are : 1. Is this expected behavior for CephFS? 2. If so, can you explain why the directory behavior is inconsistent depending on whether the mtime for the directory has previously been manually updated. Best Regards, Sandip Divekar Component QA Lead SDET. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> -- Milind
Hi Milind I just tried this using the ceph kernel client and ceph-common 17.2.6 package in the latest Fedora kernel, against Ceph 17.2.6 and it worked perfectly... There must be some other factor in play. Chris On 25/05/2023 13:04, Sandip Divekar wrote:
Hello Milind,
We are using Ceph Kernel Client. But we found this same behavior while using Libcephfs library.
Should we treat this as a bug? Or Is there any existing bug for similar issue ?
Thanks and Regards, Sandip Divekar
From: Milind Changire <mchangir@redhat.com> Sent: Thursday, May 25, 2023 4:24 PM To: Sandip Divekar <sandip.divekar@hitachivantara.com> Cc: ceph-users@ceph.io; dev@ceph.io Subject: Re: [ceph-users] Unexpected behavior of directory mtime after being set explicitly
***** EXTERNAL EMAIL ***** Sandip, What type of client are you using ? kernel client or fuse client ?
If it's the kernel client, then it's a bug.
FYI - Pacific and Quincy fuse clients do the right thing
On Wed, May 24, 2023 at 9:24 PM Sandip Divekar <sandip.divekar@hitachivantara.com<mailto:sandip.divekar@hitachivantara.com>> wrote: Hi Team,
I'm writing to bring to your attention an issue we have encountered with the "mtime" (modification time) behavior for directories in the Ceph filesystem.
Upon observation, we have noticed that when the mtime of a directory (let's say: dir1) is explicitly changed in CephFS, subsequent additions of files or directories within 'dir1' fail to update the directory's mtime as expected.
This behavior appears to be specific to CephFS - we have reproduced this issue on both Quincy and Pacific. Similar steps work as expected in the ext4 filesystem amongst others.
Reproduction steps: 1. Create a directory - mkdir dir1 2. Modify mtime using the touch command - touch dir1 3. Create a file or directory inside of 'dir1' - mkdir dir1/dir2 Expected result: mtime for dir1 should change to the time the file or directory was created in step 3 Actual result: there was no change to the mtime for 'dir1'
Note : For more detail, kindly find the attached logs.
Our queries are : 1. Is this expected behavior for CephFS? 2. If so, can you explain why the directory behavior is inconsistent depending on whether the mtime for the directory has previously been manually updated.
Best Regards, Sandip Divekar Component QA Lead SDET.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
-- Milind _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Chris, Kindly request you that follow steps given in previous mail and paste the output here. The reason behind this request is that we have encountered an issue which is easily reproducible on Latest version of both quincy and pacific, also we have thoroughly investigated the matter and we are certain that No other factors are at play in this scenario. Note : We have used Debian 11 for testing. sdsadmin@ceph-pacific-1:~$ uname -a Linux ceph-pacific-1 5.10.0-10-amd64 #1 SMP Debian 5.10.84-1 (2021-12-08) x86_64 GNU/Linux sdsadmin@ceph-pacific-1:~$ sudo ceph -v ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable) Thanks for your prompt reply. Regards Sandip Divekar -----Original Message----- From: Chris Palmer <chris.palmer@idnet.com> Sent: Thursday, May 25, 2023 7:25 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: Unexpected behavior of directory mtime after being set explicitly ***** EXTERNAL EMAIL ***** Hi Milind I just tried this using the ceph kernel client and ceph-common 17.2.6 package in the latest Fedora kernel, against Ceph 17.2.6 and it worked perfectly... There must be some other factor in play. Chris On 25/05/2023 13:04, Sandip Divekar wrote:
Hello Milind,
We are using Ceph Kernel Client. But we found this same behavior while using Libcephfs library.
Should we treat this as a bug? Or Is there any existing bug for similar issue ?
Thanks and Regards, Sandip Divekar
From: Milind Changire <mchangir@redhat.com> Sent: Thursday, May 25, 2023 4:24 PM To: Sandip Divekar <sandip.divekar@hitachivantara.com> Cc: ceph-users@ceph.io; dev@ceph.io Subject: Re: [ceph-users] Unexpected behavior of directory mtime after being set explicitly
***** EXTERNAL EMAIL ***** Sandip, What type of client are you using ? kernel client or fuse client ?
If it's the kernel client, then it's a bug.
FYI - Pacific and Quincy fuse clients do the right thing
On Wed, May 24, 2023 at 9:24 PM Sandip Divekar <sandip.divekar@hitachivantara.com<mailto:sandip.divekar@hitachivantara.com>> wrote: Hi Team,
I'm writing to bring to your attention an issue we have encountered with the "mtime" (modification time) behavior for directories in the Ceph filesystem.
Upon observation, we have noticed that when the mtime of a directory (let's say: dir1) is explicitly changed in CephFS, subsequent additions of files or directories within 'dir1' fail to update the directory's mtime as expected.
This behavior appears to be specific to CephFS - we have reproduced this issue on both Quincy and Pacific. Similar steps work as expected in the ext4 filesystem amongst others.
Reproduction steps: 1. Create a directory - mkdir dir1 2. Modify mtime using the touch command - touch dir1 3. Create a file or directory inside of 'dir1' - mkdir dir1/dir2 Expected result: mtime for dir1 should change to the time the file or directory was created in step 3 Actual result: there was no change to the mtime for 'dir1'
Note : For more detail, kindly find the attached logs.
Our queries are : 1. Is this expected behavior for CephFS? 2. If so, can you explain why the directory behavior is inconsistent depending on whether the mtime for the directory has previously been manually updated.
Best Regards, Sandip Divekar Component QA Lead SDET.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
-- Milind _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I haven’t checked the logs, but the most obvious way this happens is if the mtime set on the directory is in the future compared to the time on the client or server making changes — CephFS does not move times backwards. (This causes some problems but prevents many, many others when times are not synchronized well across the clients and servers.) -Greg On Thu, May 25, 2023 at 7:58 AM Sandip Divekar < sandip.divekar@hitachivantara.com> wrote:
Hi Chris,
Kindly request you that follow steps given in previous mail and paste the output here.
The reason behind this request is that we have encountered an issue which is easily reproducible on Latest version of both quincy and pacific, also we have thoroughly investigated the matter and we are certain that No other factors are at play in this scenario.
Note : We have used Debian 11 for testing. sdsadmin@ceph-pacific-1:~$ uname -a Linux ceph-pacific-1 5.10.0-10-amd64 #1 SMP Debian 5.10.84-1 (2021-12-08) x86_64 GNU/Linux sdsadmin@ceph-pacific-1:~$ sudo ceph -v ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)
Thanks for your prompt reply.
Regards Sandip Divekar
-----Original Message----- From: Chris Palmer <chris.palmer@idnet.com> Sent: Thursday, May 25, 2023 7:25 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: Unexpected behavior of directory mtime after being set explicitly
***** EXTERNAL EMAIL *****
Hi Milind I just tried this using the ceph kernel client and ceph-common 17.2.6 package in the latest Fedora kernel, against Ceph 17.2.6 and it worked perfectly... There must be some other factor in play. Chris
Hello Milind,
We are using Ceph Kernel Client. But we found this same behavior while using Libcephfs library.
Should we treat this as a bug? Or Is there any existing bug for similar issue ?
Thanks and Regards, Sandip Divekar
From: Milind Changire <mchangir@redhat.com> Sent: Thursday, May 25, 2023 4:24 PM To: Sandip Divekar <sandip.divekar@hitachivantara.com> Cc: ceph-users@ceph.io; dev@ceph.io Subject: Re: [ceph-users] Unexpected behavior of directory mtime after being set explicitly
***** EXTERNAL EMAIL ***** Sandip, What type of client are you using ? kernel client or fuse client ?
If it's the kernel client, then it's a bug.
FYI - Pacific and Quincy fuse clients do the right thing
On Wed, May 24, 2023 at 9:24 PM Sandip Divekar < sandip.divekar@hitachivantara.com<mailto:sandip.divekar@hitachivantara.com>> wrote: Hi Team,
I'm writing to bring to your attention an issue we have encountered with
On 25/05/2023 13:04, Sandip Divekar wrote: the "mtime" (modification time) behavior for directories in the Ceph filesystem.
Upon observation, we have noticed that when the mtime of a directory (let's say: dir1) is explicitly changed in CephFS, subsequent additions
of files or directories within 'dir1' fail to update the directory's mtime as expected.
This behavior appears to be specific to CephFS - we have reproduced this
issue on both Quincy and Pacific. Similar steps work as expected in the ext4 filesystem amongst others.
Reproduction steps: 1. Create a directory - mkdir dir1 2. Modify mtime using the touch command - touch dir1 3. Create a file or directory inside of 'dir1' - mkdir dir1/dir2 Expected result: mtime for dir1 should change to the time the file or directory was created in step 3 Actual result: there was no change to the mtime for 'dir1'
Note : For more detail, kindly find the attached logs.
Our queries are : 1. Is this expected behavior for CephFS? 2. If so, can you explain why the directory behavior is inconsistent
depending on whether the mtime for the directory has previously been manually updated.
Best Regards, Sandip Divekar Component QA Lead SDET.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
-- Milind _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Greogry, We are setting the mtime to 01 Jan 1970 00:00 1. Create a directory "dir1" 2. set mtime of the "dir1 to 0 -> i.e 1 jan 1970 3. Create child directory in "dir1" i.e mkdir dir1/dir2 OR Create a file in "dir1" i.e "touch dir1/file1 4. stat "dir1" Linux FS : updates the mtime of "dir1" Ceph FS: DOESNOT update the mtime of "dir1" Linux command output : CEPHFS =========================================================================== root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# mkdir dir1 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# stat dir1 File: dir1 Size: 0 Blocks: 0 IO Block: 65536 directory Device: 28h/40d Inode: 1099511714911 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2023-05-24 11:09:25.260851345 +0530 Modify: 2023-05-24 11:09:25.260851345 +0530 Change: 2023-05-24 11:09:25.260851345 +0530 Birth: 2023-05-24 11:09:25.260851345 +0530 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# touch -m -d '26 Aug 1982 22:00' dir1 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# stat dir1/ File: dir1/ Size: 0 Blocks: 0 IO Block: 65536 directory Device: 28h/40d Inode: 1099511714911 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2023-05-24 11:09:25.260851345 +0530 Modify: 1982-08-26 22:00:00.000000000 +0530 Change: 2023-05-24 11:10:04.881454967 +0530 Birth: 2023-05-24 11:09:25.260851345 +0530 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# mkdir dir1/dir2 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# stat dir1/ File: dir1/ Size: 1 Blocks: 0 IO Block: 65536 directory Device: 28h/40d Inode: 1099511714911 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2023-05-24 11:09:25.260851345 +0530 Modify: 1982-08-26 22:00:00.000000000 +0530 Change: 2023-05-24 11:10:19.141672220 +0530 Birth: 2023-05-24 11:09:25.260851345 +0530 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# -Joe From: Gregory Farnum <gfarnum@redhat.com> Sent: Thursday, May 25, 2023 8:43 PM To: Sandip Divekar <sandip.divekar@hitachivantara.com> Cc: Chris Palmer <chris.palmer@idnet.com>; Gavin Lucas <gavin.lucas@hitachivantara.com>; Joseph Fernandes <joseph.fernandes@hitachivantara.com>; Simon Crosland <Simon.Crosland@hitachivantara.com>; ceph-users@ceph.io; dev@ceph.io Subject: Re: [ceph-users] Re: Unexpected behavior of directory mtime after being set explicitly ***** EXTERNAL EMAIL ***** I haven’t checked the logs, but the most obvious way this happens is if the mtime set on the directory is in the future compared to the time on the client or server making changes — CephFS does not move times backwards. (This causes some problems but prevents many, many others when times are not synchronized well across the clients and servers.) -Greg On Thu, May 25, 2023 at 7:58 AM Sandip Divekar <sandip.divekar@hitachivantara.com<mailto:sandip.divekar@hitachivantara.com>> wrote: Hi Chris, Kindly request you that follow steps given in previous mail and paste the output here. The reason behind this request is that we have encountered an issue which is easily reproducible on Latest version of both quincy and pacific, also we have thoroughly investigated the matter and we are certain that No other factors are at play in this scenario. Note : We have used Debian 11 for testing. sdsadmin@ceph-pacific-1:~$ uname -a Linux ceph-pacific-1 5.10.0-10-amd64 #1 SMP Debian 5.10.84-1 (2021-12-08) x86_64 GNU/Linux sdsadmin@ceph-pacific-1:~$ sudo ceph -v ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable) Thanks for your prompt reply. Regards Sandip Divekar -----Original Message----- From: Chris Palmer <chris.palmer@idnet.com<mailto:chris.palmer@idnet.com>> Sent: Thursday, May 25, 2023 7:25 PM To: ceph-users@ceph.io<mailto:ceph-users@ceph.io> Subject: [ceph-users] Re: Unexpected behavior of directory mtime after being set explicitly ***** EXTERNAL EMAIL ***** Hi Milind I just tried this using the ceph kernel client and ceph-common 17.2.6 package in the latest Fedora kernel, against Ceph 17.2.6 and it worked perfectly... There must be some other factor in play. Chris On 25/05/2023 13:04, Sandip Divekar wrote:
Hello Milind,
We are using Ceph Kernel Client. But we found this same behavior while using Libcephfs library.
Should we treat this as a bug? Or Is there any existing bug for similar issue ?
Thanks and Regards, Sandip Divekar
From: Milind Changire <mchangir@redhat.com<mailto:mchangir@redhat.com>> Sent: Thursday, May 25, 2023 4:24 PM To: Sandip Divekar <sandip.divekar@hitachivantara.com<mailto:sandip.divekar@hitachivantara.com>> Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io>; dev@ceph.io<mailto:dev@ceph.io> Subject: Re: [ceph-users] Unexpected behavior of directory mtime after being set explicitly
***** EXTERNAL EMAIL ***** Sandip, What type of client are you using ? kernel client or fuse client ?
If it's the kernel client, then it's a bug.
FYI - Pacific and Quincy fuse clients do the right thing
On Wed, May 24, 2023 at 9:24 PM Sandip Divekar <sandip.divekar@hitachivantara.com<mailto:sandip.divekar@hitachivantara.com><mailto:sandip.divekar@hitachivantara.com<mailto:sandip.divekar@hitachivantara.com>>> wrote: Hi Team,
I'm writing to bring to your attention an issue we have encountered with the "mtime" (modification time) behavior for directories in the Ceph filesystem.
Upon observation, we have noticed that when the mtime of a directory (let's say: dir1) is explicitly changed in CephFS, subsequent additions of files or directories within 'dir1' fail to update the directory's mtime as expected.
This behavior appears to be specific to CephFS - we have reproduced this issue on both Quincy and Pacific. Similar steps work as expected in the ext4 filesystem amongst others.
Reproduction steps: 1. Create a directory - mkdir dir1 2. Modify mtime using the touch command - touch dir1 3. Create a file or directory inside of 'dir1' - mkdir dir1/dir2 Expected result: mtime for dir1 should change to the time the file or directory was created in step 3 Actual result: there was no change to the mtime for 'dir1'
Note : For more detail, kindly find the attached logs.
Our queries are : 1. Is this expected behavior for CephFS? 2. If so, can you explain why the directory behavior is inconsistent depending on whether the mtime for the directory has previously been manually updated.
Best Regards, Sandip Divekar Component QA Lead SDET.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io<mailto:ceph-users@ceph.io>> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>>
-- Milind _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
Copy-pasting reply from Joseph. ============================================================================================= Hello Greogry, We are setting the mtime to 01 Jan 1970 00:00 1. Create a directory "dir1" 2. set mtime of the "dir1 to 0 -> i.e 1 jan 1970 3. Create child directory in "dir1" i.e mkdir dir1/dir2 OR Create a file in "dir1" i.e "touch dir1/file1 4. stat "dir1" Linux FS : updates the mtime of "dir1" Ceph FS: DOESNOT update the mtime of "dir1" Linux command output : CEPHFS =========================================================================== root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# mkdir dir1 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# stat dir1 File: dir1 Size: 0 Blocks: 0 IO Block: 65536 directory Device: 28h/40d Inode: 1099511714911 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2023-05-24 11:09:25.260851345 +0530 Modify: 2023-05-24 11:09:25.260851345 +0530 Change: 2023-05-24 11:09:25.260851345 +0530 Birth: 2023-05-24 11:09:25.260851345 +0530 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# touch -m -d '26 Aug 1982 22:00' dir1 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# stat dir1/ File: dir1/ Size: 0 Blocks: 0 IO Block: 65536 directory Device: 28h/40d Inode: 1099511714911 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2023-05-24 11:09:25.260851345 +0530 Modify: 1982-08-26 22:00:00.000000000 +0530 Change: 2023-05-24 11:10:04.881454967 +0530 Birth: 2023-05-24 11:09:25.260851345 +0530 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# mkdir dir1/dir2 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# stat dir1/ File: dir1/ Size: 1 Blocks: 0 IO Block: 65536 directory Device: 28h/40d Inode: 1099511714911 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2023-05-24 11:09:25.260851345 +0530 Modify: 1982-08-26 22:00:00.000000000 +0530 Change: 2023-05-24 11:10:19.141672220 +0530 Birth: 2023-05-24 11:09:25.260851345 +0530 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# -Joe ============================================================================================= Thanks and Regards Sandip Divekar From: Gregory Farnum <gfarnum@redhat.com> Sent: Thursday, May 25, 2023 8:43 PM To: Sandip Divekar <sandip.divekar@hitachivantara.com> Cc: Chris Palmer <chris.palmer@idnet.com>; Gavin Lucas <gavin.lucas@hitachivantara.com>; Joseph Fernandes <joseph.fernandes@hitachivantara.com>; Simon Crosland <Simon.Crosland@hitachivantara.com>; ceph-users@ceph.io; dev@ceph.io Subject: Re: [ceph-users] Re: Unexpected behavior of directory mtime after being set explicitly ***** EXTERNAL EMAIL ***** I haven’t checked the logs, but the most obvious way this happens is if the mtime set on the directory is in the future compared to the time on the client or server making changes — CephFS does not move times backwards. (This causes some problems but prevents many, many others when times are not synchronized well across the clients and servers.) -Greg On Thu, May 25, 2023 at 7:58 AM Sandip Divekar <sandip.divekar@hitachivantara.com<mailto:sandip.divekar@hitachivantara.com>> wrote: Hi Chris, Kindly request you that follow steps given in previous mail and paste the output here. The reason behind this request is that we have encountered an issue which is easily reproducible on Latest version of both quincy and pacific, also we have thoroughly investigated the matter and we are certain that No other factors are at play in this scenario. Note : We have used Debian 11 for testing. sdsadmin@ceph-pacific-1:~$ uname -a Linux ceph-pacific-1 5.10.0-10-amd64 #1 SMP Debian 5.10.84-1 (2021-12-08) x86_64 GNU/Linux sdsadmin@ceph-pacific-1:~$ sudo ceph -v ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable) Thanks for your prompt reply. Regards Sandip Divekar -----Original Message----- From: Chris Palmer <chris.palmer@idnet.com<mailto:chris.palmer@idnet.com>> Sent: Thursday, May 25, 2023 7:25 PM To: ceph-users@ceph.io<mailto:ceph-users@ceph.io> Subject: [ceph-users] Re: Unexpected behavior of directory mtime after being set explicitly ***** EXTERNAL EMAIL ***** Hi Milind I just tried this using the ceph kernel client and ceph-common 17.2.6 package in the latest Fedora kernel, against Ceph 17.2.6 and it worked perfectly... There must be some other factor in play. Chris On 25/05/2023 13:04, Sandip Divekar wrote:
Hello Milind,
We are using Ceph Kernel Client. But we found this same behavior while using Libcephfs library.
Should we treat this as a bug? Or Is there any existing bug for similar issue ?
Thanks and Regards, Sandip Divekar
From: Milind Changire <mchangir@redhat.com<mailto:mchangir@redhat.com>> Sent: Thursday, May 25, 2023 4:24 PM To: Sandip Divekar <sandip.divekar@hitachivantara.com<mailto:sandip.divekar@hitachivantara.com>> Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io>; dev@ceph.io<mailto:dev@ceph.io> Subject: Re: [ceph-users] Unexpected behavior of directory mtime after being set explicitly
***** EXTERNAL EMAIL ***** Sandip, What type of client are you using ? kernel client or fuse client ?
If it's the kernel client, then it's a bug.
FYI - Pacific and Quincy fuse clients do the right thing
On Wed, May 24, 2023 at 9:24 PM Sandip Divekar <sandip.divekar@hitachivantara.com<mailto:sandip.divekar@hitachivantara.com><mailto:sandip.divekar@hitachivantara.com<mailto:sandip.divekar@hitachivantara.com>>> wrote: Hi Team,
I'm writing to bring to your attention an issue we have encountered with the "mtime" (modification time) behavior for directories in the Ceph filesystem.
Upon observation, we have noticed that when the mtime of a directory (let's say: dir1) is explicitly changed in CephFS, subsequent additions of files or directories within 'dir1' fail to update the directory's mtime as expected.
This behavior appears to be specific to CephFS - we have reproduced this issue on both Quincy and Pacific. Similar steps work as expected in the ext4 filesystem amongst others.
Reproduction steps: 1. Create a directory - mkdir dir1 2. Modify mtime using the touch command - touch dir1 3. Create a file or directory inside of 'dir1' - mkdir dir1/dir2 Expected result: mtime for dir1 should change to the time the file or directory was created in step 3 Actual result: there was no change to the mtime for 'dir1'
Note : For more detail, kindly find the attached logs.
Our queries are : 1. Is this expected behavior for CephFS? 2. If so, can you explain why the directory behavior is inconsistent depending on whether the mtime for the directory has previously been manually updated.
Best Regards, Sandip Divekar Component QA Lead SDET.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io<mailto:ceph-users@ceph.io>> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>>
-- Milind _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
Hi Sandip Ceph servers (debian11/ceph base with Proxmox installed on top - NOT the ceph that comes with Proxmox!): ceph@pve1:~$ uname -a Linux pve1 5.15.107-2-pve #1 SMP PVE 5.15.107-2 (2023-05-10T09:10Z) x86_64 GNU/Linux ceph@pve1:~$ ceph version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Fedora workstation. I waited until the minute had clicked over before doing each step: [chris@rex mtime]$ uname -a Linux rex.palmer 6.2.15-300.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Thu May 11 17:37:39 UTC 2023 x86_64 GNU/Linux [chris@rex mtime]$ rpm -q ceph-common ceph-common-17.2.6-2.fc38.x86_64 [chris@rex mtime]$ df . Filesystem 1K-blocks Used Available Use% Mounted on 192.168.80.121,192.168.80.122,192.168.80.123:/data2 8589930496 4944801792 3645128704 58% /mnt/data2 [chris@rex mtime]$ mount|grep data2 systemd-1 on /mnt/data2 type autofs (rw,relatime,fd=61,pgrp=1,timeout=600,minproto=5,maxproto=5,direct,pipe_ino=22804) 192.168.80.121,192.168.80.122,192.168.80.123:/data2 on /mnt/data2 type ceph (rw,noatime,nodiratime,name=data2-rex,secret=<hidden>,fsid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx,acl,_netdev,x-systemd.mount-timeout=30,x-systemd.automount,x-systemd.idle-timeout=600) [chris@rex mtime]$ date; mkdir one; ls -ld one Thu 25 May 16:57:28 BST 2023 drwxrwxr-x 2 chris groupname 0 May 25 16:57 one [chris@rex mtime]$ date; touch one; ls -ld one Thu 25 May 16:58:14 BST 2023 drwxrwxr-x 2 chris groupname 0 May 25 16:58 one [chris@rex mtime]$ date; mkdir one/two; ls -ld one Thu 25 May 16:59:26 BST 2023 drwxrwxr-x 3 chris groupname 1 May 25 16:59 one I also repeated it with the test run on the ceph debian11 server, having mounted the cephfs filesystem on the ceph server - exactly the same result. I then repeated it again on a pure debian11 ceph 17.2.6 cluster, using a debian11 client, and it also worked as expected. All systems have latest patches applied. Hope that helps Chris On 25/05/2023 15:57, Sandip Divekar wrote:
Hi Chris,
Kindly request you that follow steps given in previous mail and paste the output here.
The reason behind this request is that we have encountered an issue which is easily reproducible on Latest version of both quincy and pacific, also we have thoroughly investigated the matter and we are certain that No other factors are at play in this scenario.
Note : We have used Debian 11 for testing. sdsadmin@ceph-pacific-1:~$ uname -a Linux ceph-pacific-1 5.10.0-10-amd64 #1 SMP Debian 5.10.84-1 (2021-12-08) x86_64 GNU/Linux sdsadmin@ceph-pacific-1:~$ sudo ceph -v ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)
Thanks for your prompt reply.
Regards Sandip Divekar
-----Original Message----- From: Chris Palmer<chris.palmer@idnet.com> Sent: Thursday, May 25, 2023 7:25 PM To:ceph-users@ceph.io Subject: [ceph-users] Re: Unexpected behavior of directory mtime after being set explicitly
***** EXTERNAL EMAIL *****
Hi Milind I just tried this using the ceph kernel client and ceph-common 17.2.6 package in the latest Fedora kernel, against Ceph 17.2.6 and it worked perfectly... There must be some other factor in play. Chris
On 25/05/2023 13:04, Sandip Divekar wrote:
Hello Milind,
We are using Ceph Kernel Client. But we found this same behavior while using Libcephfs library.
Should we treat this as a bug? Or Is there any existing bug for similar issue ?
Thanks and Regards, Sandip Divekar
From: Milind Changire<mchangir@redhat.com> Sent: Thursday, May 25, 2023 4:24 PM To: Sandip Divekar<sandip.divekar@hitachivantara.com> Cc:ceph-users@ceph.io;dev@ceph.io Subject: Re: [ceph-users] Unexpected behavior of directory mtime after being set explicitly
***** EXTERNAL EMAIL ***** Sandip, What type of client are you using ? kernel client or fuse client ?
If it's the kernel client, then it's a bug.
FYI - Pacific and Quincy fuse clients do the right thing
On Wed, May 24, 2023 at 9:24 PM Sandip Divekar <sandip.divekar@hitachivantara.com<mailto:sandip.divekar@hitachivantara.com>> wrote: Hi Team,
I'm writing to bring to your attention an issue we have encountered with the "mtime" (modification time) behavior for directories in the Ceph filesystem.
Upon observation, we have noticed that when the mtime of a directory (let's say: dir1) is explicitly changed in CephFS, subsequent additions of files or directories within 'dir1' fail to update the directory's mtime as expected.
This behavior appears to be specific to CephFS - we have reproduced this issue on both Quincy and Pacific. Similar steps work as expected in the ext4 filesystem amongst others.
Reproduction steps: 1. Create a directory - mkdir dir1 2. Modify mtime using the touch command - touch dir1 3. Create a file or directory inside of 'dir1' - mkdir dir1/dir2 Expected result: mtime for dir1 should change to the time the file or directory was created in step 3 Actual result: there was no change to the mtime for 'dir1'
Note : For more detail, kindly find the attached logs.
Our queries are : 1. Is this expected behavior for CephFS? 2. If so, can you explain why the directory behavior is inconsistent depending on whether the mtime for the directory has previously been manually updated.
Best Regards, Sandip Divekar Component QA Lead SDET.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
-- Milind _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
Hi Chris, I think, you have missed one steps and that is to change mtime for directory explicitly. Please have a look at highlighted steps. CEPHFS =========================================================================== root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# mkdir dir1 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# stat dir1 File: dir1 Size: 0 Blocks: 0 IO Block: 65536 directory Device: 28h/40d Inode: 1099511714911 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2023-05-24 11:09:25.260851345 +0530 Modify: 2023-05-24 11:09:25.260851345 +0530 Change: 2023-05-24 11:09:25.260851345 +0530 Birth: 2023-05-24 11:09:25.260851345 +0530 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# touch -m -d '26 Aug 1982 22:00' dir1 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# stat dir1/ File: dir1/ Size: 0 Blocks: 0 IO Block: 65536 directory Device: 28h/40d Inode: 1099511714911 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2023-05-24 11:09:25.260851345 +0530 Modify: 1982-08-26 22:00:00.000000000 +0530 Change: 2023-05-24 11:10:04.881454967 +0530 Birth: 2023-05-24 11:09:25.260851345 +0530 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# mkdir dir1/dir2 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# stat dir1/ File: dir1/ Size: 1 Blocks: 0 IO Block: 65536 directory Device: 28h/40d Inode: 1099511714911 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2023-05-24 11:09:25.260851345 +0530 Modify: 1982-08-26 22:00:00.000000000 +0530 Change: 2023-05-24 11:10:19.141672220 +0530 Birth: 2023-05-24 11:09:25.260851345 +0530 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# Note : In a last step, it is expected that “Modify” time should change. Thanks and Regards Sandip Divekar From: Chris Palmer <chris.palmer@idnet.com> Sent: Thursday, May 25, 2023 9:46 PM To: Sandip Divekar <sandip.divekar@hitachivantara.com>; ceph-users@ceph.io Cc: dev@ceph.io; Gavin Lucas <gavin.lucas@hitachivantara.com>; Joseph Fernandes <joseph.fernandes@hitachivantara.com>; Simon Crosland <Simon.Crosland@hitachivantara.com> Subject: Re: [ceph-users] Re: Unexpected behavior of directory mtime after being set explicitly ***** EXTERNAL EMAIL ***** Hi Sandip Ceph servers (debian11/ceph base with Proxmox installed on top - NOT the ceph that comes with Proxmox!): ceph@pve1:~$ uname -a Linux pve1 5.15.107-2-pve #1 SMP PVE 5.15.107-2 (2023-05-10T09:10Z) x86_64 GNU/Linux ceph@pve1:~$ ceph version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Fedora workstation. I waited until the minute had clicked over before doing each step: [chris@rex mtime]$ uname -a Linux rex.palmer 6.2.15-300.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Thu May 11 17:37:39 UTC 2023 x86_64 GNU/Linux [chris@rex mtime]$ rpm -q ceph-common ceph-common-17.2.6-2.fc38.x86_64 [chris@rex mtime]$ df . Filesystem 1K-blocks Used Available Use% Mounted on 192.168.80.121,192.168.80.122,192.168.80.123:/data2 8589930496 4944801792 3645128704 58% /mnt/data2 [chris@rex mtime]$ mount|grep data2 systemd-1 on /mnt/data2 type autofs (rw,relatime,fd=61,pgrp=1,timeout=600,minproto=5,maxproto=5,direct,pipe_ino=22804) 192.168.80.121,192.168.80.122,192.168.80.123:/data2 on /mnt/data2 type ceph (rw,noatime,nodiratime,name=data2-rex,secret=<hidden>,fsid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx,acl,_netdev,x-systemd.mount-timeout=30,x-systemd.automount,x-systemd.idle-timeout=600) [chris@rex mtime]$ date; mkdir one; ls -ld one Thu 25 May 16:57:28 BST 2023 drwxrwxr-x 2 chris groupname 0 May 25 16:57 one [chris@rex mtime]$ date; touch one; ls -ld one Thu 25 May 16:58:14 BST 2023 drwxrwxr-x 2 chris groupname 0 May 25 16:58 one [chris@rex mtime]$ date; mkdir one/two; ls -ld one Thu 25 May 16:59:26 BST 2023 drwxrwxr-x 3 chris groupname 1 May 25 16:59 one I also repeated it with the test run on the ceph debian11 server, having mounted the cephfs filesystem on the ceph server - exactly the same result. I then repeated it again on a pure debian11 ceph 17.2.6 cluster, using a debian11 client, and it also worked as expected. All systems have latest patches applied. Hope that helps Chris On 25/05/2023 15:57, Sandip Divekar wrote: Hi Chris, Kindly request you that follow steps given in previous mail and paste the output here. The reason behind this request is that we have encountered an issue which is easily reproducible on Latest version of both quincy and pacific, also we have thoroughly investigated the matter and we are certain that No other factors are at play in this scenario. Note : We have used Debian 11 for testing. sdsadmin@ceph-pacific-1:~$ uname -a Linux ceph-pacific-1 5.10.0-10-amd64 #1 SMP Debian 5.10.84-1 (2021-12-08) x86_64 GNU/Linux sdsadmin@ceph-pacific-1:~$ sudo ceph -v ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable) Thanks for your prompt reply. Regards Sandip Divekar -----Original Message----- From: Chris Palmer <chris.palmer@idnet.com><mailto:chris.palmer@idnet.com> Sent: Thursday, May 25, 2023 7:25 PM To: ceph-users@ceph.io<mailto:ceph-users@ceph.io> Subject: [ceph-users] Re: Unexpected behavior of directory mtime after being set explicitly ***** EXTERNAL EMAIL ***** Hi Milind I just tried this using the ceph kernel client and ceph-common 17.2.6 package in the latest Fedora kernel, against Ceph 17.2.6 and it worked perfectly... There must be some other factor in play. Chris On 25/05/2023 13:04, Sandip Divekar wrote: Hello Milind, We are using Ceph Kernel Client. But we found this same behavior while using Libcephfs library. Should we treat this as a bug? Or Is there any existing bug for similar issue ? Thanks and Regards, Sandip Divekar From: Milind Changire <mchangir@redhat.com><mailto:mchangir@redhat.com> Sent: Thursday, May 25, 2023 4:24 PM To: Sandip Divekar <sandip.divekar@hitachivantara.com><mailto:sandip.divekar@hitachivantara.com> Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io>; dev@ceph.io<mailto:dev@ceph.io> Subject: Re: [ceph-users] Unexpected behavior of directory mtime after being set explicitly ***** EXTERNAL EMAIL ***** Sandip, What type of client are you using ? kernel client or fuse client ? If it's the kernel client, then it's a bug. FYI - Pacific and Quincy fuse clients do the right thing On Wed, May 24, 2023 at 9:24 PM Sandip Divekar <sandip.divekar@hitachivantara.com<mailto:sandip.divekar@hitachivantara.com><mailto:sandip.divekar@hitachivantara.com><mailto:sandip.divekar@hitachivantara.com>> wrote: Hi Team, I'm writing to bring to your attention an issue we have encountered with the "mtime" (modification time) behavior for directories in the Ceph filesystem. Upon observation, we have noticed that when the mtime of a directory (let's say: dir1) is explicitly changed in CephFS, subsequent additions of files or directories within 'dir1' fail to update the directory's mtime as expected. This behavior appears to be specific to CephFS - we have reproduced this issue on both Quincy and Pacific. Similar steps work as expected in the ext4 filesystem amongst others. Reproduction steps: 1. Create a directory - mkdir dir1 2. Modify mtime using the touch command - touch dir1 3. Create a file or directory inside of 'dir1' - mkdir dir1/dir2 Expected result: mtime for dir1 should change to the time the file or directory was created in step 3 Actual result: there was no change to the mtime for 'dir1' Note : For more detail, kindly find the attached logs. Our queries are : 1. Is this expected behavior for CephFS? 2. If so, can you explain why the directory behavior is inconsistent depending on whether the mtime for the directory has previously been manually updated. Best Regards, Sandip Divekar Component QA Lead SDET. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io> -- Milind _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
Hi Chris / Gregory, Did you get a chance to investigate this issue ? Thanks and Regards Sandip Divekar From: Sandip Divekar Sent: Thursday, May 25, 2023 11:16 PM To: Chris Palmer <chris.palmer@idnet.com>; ceph-users@ceph.io Cc: dev@ceph.io; Gavin Lucas <gavin.lucas@hitachivantara.com>; Joseph Fernandes <joseph.fernandes@hitachivantara.com>; Simon Crosland <Simon.Crosland@hitachivantara.com> Subject: RE: [ceph-users] Re: Unexpected behavior of directory mtime after being set explicitly Hi Chris, I think, you have missed one steps and that is to change mtime for directory explicitly. Please have a look at highlighted steps. CEPHFS =========================================================================== root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# mkdir dir1 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# stat dir1 File: dir1 Size: 0 Blocks: 0 IO Block: 65536 directory Device: 28h/40d Inode: 1099511714911 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2023-05-24 11:09:25.260851345 +0530 Modify: 2023-05-24 11:09:25.260851345 +0530 Change: 2023-05-24 11:09:25.260851345 +0530 Birth: 2023-05-24 11:09:25.260851345 +0530 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# touch -m -d '26 Aug 1982 22:00' dir1 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# stat dir1/ File: dir1/ Size: 0 Blocks: 0 IO Block: 65536 directory Device: 28h/40d Inode: 1099511714911 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2023-05-24 11:09:25.260851345 +0530 Modify: 1982-08-26 22:00:00.000000000 +0530 Change: 2023-05-24 11:10:04.881454967 +0530 Birth: 2023-05-24 11:09:25.260851345 +0530 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# mkdir dir1/dir2 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# stat dir1/ File: dir1/ Size: 1 Blocks: 0 IO Block: 65536 directory Device: 28h/40d Inode: 1099511714911 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2023-05-24 11:09:25.260851345 +0530 Modify: 1982-08-26 22:00:00.000000000 +0530 Change: 2023-05-24 11:10:19.141672220 +0530 Birth: 2023-05-24 11:09:25.260851345 +0530 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# Note : In a last step, it is expected that “Modify” time should change. Thanks and Regards Sandip Divekar From: Chris Palmer <chris.palmer@idnet.com<mailto:chris.palmer@idnet.com>> Sent: Thursday, May 25, 2023 9:46 PM To: Sandip Divekar <sandip.divekar@hitachivantara.com<mailto:sandip.divekar@hitachivantara.com>>; ceph-users@ceph.io<mailto:ceph-users@ceph.io> Cc: dev@ceph.io<mailto:dev@ceph.io>; Gavin Lucas <gavin.lucas@hitachivantara.com<mailto:gavin.lucas@hitachivantara.com>>; Joseph Fernandes <joseph.fernandes@hitachivantara.com<mailto:joseph.fernandes@hitachivantara.com>>; Simon Crosland <Simon.Crosland@hitachivantara.com<mailto:Simon.Crosland@hitachivantara.com>> Subject: Re: [ceph-users] Re: Unexpected behavior of directory mtime after being set explicitly ***** EXTERNAL EMAIL ***** Hi Sandip Ceph servers (debian11/ceph base with Proxmox installed on top - NOT the ceph that comes with Proxmox!): ceph@pve1:~$ uname -a Linux pve1 5.15.107-2-pve #1 SMP PVE 5.15.107-2 (2023-05-10T09:10Z) x86_64 GNU/Linux ceph@pve1:~$ ceph version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Fedora workstation. I waited until the minute had clicked over before doing each step: [chris@rex mtime]$ uname -a Linux rex.palmer 6.2.15-300.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Thu May 11 17:37:39 UTC 2023 x86_64 GNU/Linux [chris@rex mtime]$ rpm -q ceph-common ceph-common-17.2.6-2.fc38.x86_64 [chris@rex mtime]$ df . Filesystem 1K-blocks Used Available Use% Mounted on 192.168.80.121,192.168.80.122,192.168.80.123:/data2 8589930496 4944801792 3645128704 58% /mnt/data2 [chris@rex mtime]$ mount|grep data2 systemd-1 on /mnt/data2 type autofs (rw,relatime,fd=61,pgrp=1,timeout=600,minproto=5,maxproto=5,direct,pipe_ino=22804) 192.168.80.121,192.168.80.122,192.168.80.123:/data2 on /mnt/data2 type ceph (rw,noatime,nodiratime,name=data2-rex,secret=<hidden>,fsid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx,acl,_netdev,x-systemd.mount-timeout=30,x-systemd.automount,x-systemd.idle-timeout=600) [chris@rex mtime]$ date; mkdir one; ls -ld one Thu 25 May 16:57:28 BST 2023 drwxrwxr-x 2 chris groupname 0 May 25 16:57 one [chris@rex mtime]$ date; touch one; ls -ld one Thu 25 May 16:58:14 BST 2023 drwxrwxr-x 2 chris groupname 0 May 25 16:58 one [chris@rex mtime]$ date; mkdir one/two; ls -ld one Thu 25 May 16:59:26 BST 2023 drwxrwxr-x 3 chris groupname 1 May 25 16:59 one I also repeated it with the test run on the ceph debian11 server, having mounted the cephfs filesystem on the ceph server - exactly the same result. I then repeated it again on a pure debian11 ceph 17.2.6 cluster, using a debian11 client, and it also worked as expected. All systems have latest patches applied. Hope that helps Chris On 25/05/2023 15:57, Sandip Divekar wrote: Hi Chris, Kindly request you that follow steps given in previous mail and paste the output here. The reason behind this request is that we have encountered an issue which is easily reproducible on Latest version of both quincy and pacific, also we have thoroughly investigated the matter and we are certain that No other factors are at play in this scenario. Note : We have used Debian 11 for testing. sdsadmin@ceph-pacific-1:~$ uname -a Linux ceph-pacific-1 5.10.0-10-amd64 #1 SMP Debian 5.10.84-1 (2021-12-08) x86_64 GNU/Linux sdsadmin@ceph-pacific-1:~$ sudo ceph -v ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable) Thanks for your prompt reply. Regards Sandip Divekar -----Original Message----- From: Chris Palmer <chris.palmer@idnet.com><mailto:chris.palmer@idnet.com> Sent: Thursday, May 25, 2023 7:25 PM To: ceph-users@ceph.io<mailto:ceph-users@ceph.io> Subject: [ceph-users] Re: Unexpected behavior of directory mtime after being set explicitly ***** EXTERNAL EMAIL ***** Hi Milind I just tried this using the ceph kernel client and ceph-common 17.2.6 package in the latest Fedora kernel, against Ceph 17.2.6 and it worked perfectly... There must be some other factor in play. Chris On 25/05/2023 13:04, Sandip Divekar wrote: Hello Milind, We are using Ceph Kernel Client. But we found this same behavior while using Libcephfs library. Should we treat this as a bug? Or Is there any existing bug for similar issue ? Thanks and Regards, Sandip Divekar From: Milind Changire <mchangir@redhat.com><mailto:mchangir@redhat.com> Sent: Thursday, May 25, 2023 4:24 PM To: Sandip Divekar <sandip.divekar@hitachivantara.com><mailto:sandip.divekar@hitachivantara.com> Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io>; dev@ceph.io<mailto:dev@ceph.io> Subject: Re: [ceph-users] Unexpected behavior of directory mtime after being set explicitly ***** EXTERNAL EMAIL ***** Sandip, What type of client are you using ? kernel client or fuse client ? If it's the kernel client, then it's a bug. FYI - Pacific and Quincy fuse clients do the right thing On Wed, May 24, 2023 at 9:24 PM Sandip Divekar <sandip.divekar@hitachivantara.com<mailto:sandip.divekar@hitachivantara.com><mailto:sandip.divekar@hitachivantara.com><mailto:sandip.divekar@hitachivantara.com>> wrote: Hi Team, I'm writing to bring to your attention an issue we have encountered with the "mtime" (modification time) behavior for directories in the Ceph filesystem. Upon observation, we have noticed that when the mtime of a directory (let's say: dir1) is explicitly changed in CephFS, subsequent additions of files or directories within 'dir1' fail to update the directory's mtime as expected. This behavior appears to be specific to CephFS - we have reproduced this issue on both Quincy and Pacific. Similar steps work as expected in the ext4 filesystem amongst others. Reproduction steps: 1. Create a directory - mkdir dir1 2. Modify mtime using the touch command - touch dir1 3. Create a file or directory inside of 'dir1' - mkdir dir1/dir2 Expected result: mtime for dir1 should change to the time the file or directory was created in step 3 Actual result: there was no change to the mtime for 'dir1' Note : For more detail, kindly find the attached logs. Our queries are : 1. Is this expected behavior for CephFS? 2. If so, can you explain why the directory behavior is inconsistent depending on whether the mtime for the directory has previously been manually updated. Best Regards, Sandip Divekar Component QA Lead SDET. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io> -- Milind _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
Hi Team, I kindly request your input on whether we should consider the “mtime” issue as bug or not. Thanks Sandip From: Sandip Divekar Sent: Monday, May 29, 2023 2:19 PM To: Chris Palmer <chris.palmer@idnet.com>; ceph-users@ceph.io; Gregory Farnum <gfarnum@redhat.com> Cc: dev@ceph.io; Gavin Lucas <gavin.lucas@hitachivantara.com>; Joseph Fernandes <joseph.fernandes@hitachivantara.com>; Simon Crosland <Simon.Crosland@hitachivantara.com> Subject: RE: [ceph-users] Re: Unexpected behavior of directory mtime after being set explicitly Hi Chris / Gregory, Did you get a chance to investigate this issue ? Thanks and Regards Sandip Divekar From: Sandip Divekar Sent: Thursday, May 25, 2023 11:16 PM To: Chris Palmer <chris.palmer@idnet.com<mailto:chris.palmer@idnet.com>>; ceph-users@ceph.io<mailto:ceph-users@ceph.io> Cc: dev@ceph.io<mailto:dev@ceph.io>; Gavin Lucas <gavin.lucas@hitachivantara.com<mailto:gavin.lucas@hitachivantara.com>>; Joseph Fernandes <joseph.fernandes@hitachivantara.com<mailto:joseph.fernandes@hitachivantara.com>>; Simon Crosland <Simon.Crosland@hitachivantara.com<mailto:Simon.Crosland@hitachivantara.com>> Subject: RE: [ceph-users] Re: Unexpected behavior of directory mtime after being set explicitly Hi Chris, I think, you have missed one steps and that is to change mtime for directory explicitly. Please have a look at highlighted steps. CEPHFS =========================================================================== root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# mkdir dir1 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# stat dir1 File: dir1 Size: 0 Blocks: 0 IO Block: 65536 directory Device: 28h/40d Inode: 1099511714911 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2023-05-24 11:09:25.260851345 +0530 Modify: 2023-05-24 11:09:25.260851345 +0530 Change: 2023-05-24 11:09:25.260851345 +0530 Birth: 2023-05-24 11:09:25.260851345 +0530 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# touch -m -d '26 Aug 1982 22:00' dir1 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# stat dir1/ File: dir1/ Size: 0 Blocks: 0 IO Block: 65536 directory Device: 28h/40d Inode: 1099511714911 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2023-05-24 11:09:25.260851345 +0530 Modify: 1982-08-26 22:00:00.000000000 +0530 Change: 2023-05-24 11:10:04.881454967 +0530 Birth: 2023-05-24 11:09:25.260851345 +0530 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# mkdir dir1/dir2 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# stat dir1/ File: dir1/ Size: 1 Blocks: 0 IO Block: 65536 directory Device: 28h/40d Inode: 1099511714911 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2023-05-24 11:09:25.260851345 +0530 Modify: 1982-08-26 22:00:00.000000000 +0530 Change: 2023-05-24 11:10:19.141672220 +0530 Birth: 2023-05-24 11:09:25.260851345 +0530 root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# Note : In a last step, it is expected that “Modify” time should change. Thanks and Regards Sandip Divekar From: Chris Palmer <chris.palmer@idnet.com<mailto:chris.palmer@idnet.com>> Sent: Thursday, May 25, 2023 9:46 PM To: Sandip Divekar <sandip.divekar@hitachivantara.com<mailto:sandip.divekar@hitachivantara.com>>; ceph-users@ceph.io<mailto:ceph-users@ceph.io> Cc: dev@ceph.io<mailto:dev@ceph.io>; Gavin Lucas <gavin.lucas@hitachivantara.com<mailto:gavin.lucas@hitachivantara.com>>; Joseph Fernandes <joseph.fernandes@hitachivantara.com<mailto:joseph.fernandes@hitachivantara.com>>; Simon Crosland <Simon.Crosland@hitachivantara.com<mailto:Simon.Crosland@hitachivantara.com>> Subject: Re: [ceph-users] Re: Unexpected behavior of directory mtime after being set explicitly ***** EXTERNAL EMAIL ***** Hi Sandip Ceph servers (debian11/ceph base with Proxmox installed on top - NOT the ceph that comes with Proxmox!): ceph@pve1:~$ uname -a Linux pve1 5.15.107-2-pve #1 SMP PVE 5.15.107-2 (2023-05-10T09:10Z) x86_64 GNU/Linux ceph@pve1:~$ ceph version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Fedora workstation. I waited until the minute had clicked over before doing each step: [chris@rex mtime]$ uname -a Linux rex.palmer 6.2.15-300.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Thu May 11 17:37:39 UTC 2023 x86_64 GNU/Linux [chris@rex mtime]$ rpm -q ceph-common ceph-common-17.2.6-2.fc38.x86_64 [chris@rex mtime]$ df . Filesystem 1K-blocks Used Available Use% Mounted on 192.168.80.121,192.168.80.122,192.168.80.123:/data2 8589930496 4944801792 3645128704 58% /mnt/data2 [chris@rex mtime]$ mount|grep data2 systemd-1 on /mnt/data2 type autofs (rw,relatime,fd=61,pgrp=1,timeout=600,minproto=5,maxproto=5,direct,pipe_ino=22804) 192.168.80.121,192.168.80.122,192.168.80.123:/data2 on /mnt/data2 type ceph (rw,noatime,nodiratime,name=data2-rex,secret=<hidden>,fsid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx,acl,_netdev,x-systemd.mount-timeout=30,x-systemd.automount,x-systemd.idle-timeout=600) [chris@rex mtime]$ date; mkdir one; ls -ld one Thu 25 May 16:57:28 BST 2023 drwxrwxr-x 2 chris groupname 0 May 25 16:57 one [chris@rex mtime]$ date; touch one; ls -ld one Thu 25 May 16:58:14 BST 2023 drwxrwxr-x 2 chris groupname 0 May 25 16:58 one [chris@rex mtime]$ date; mkdir one/two; ls -ld one Thu 25 May 16:59:26 BST 2023 drwxrwxr-x 3 chris groupname 1 May 25 16:59 one I also repeated it with the test run on the ceph debian11 server, having mounted the cephfs filesystem on the ceph server - exactly the same result. I then repeated it again on a pure debian11 ceph 17.2.6 cluster, using a debian11 client, and it also worked as expected. All systems have latest patches applied. Hope that helps Chris On 25/05/2023 15:57, Sandip Divekar wrote: Hi Chris, Kindly request you that follow steps given in previous mail and paste the output here. The reason behind this request is that we have encountered an issue which is easily reproducible on Latest version of both quincy and pacific, also we have thoroughly investigated the matter and we are certain that No other factors are at play in this scenario. Note : We have used Debian 11 for testing. sdsadmin@ceph-pacific-1:~$ uname -a Linux ceph-pacific-1 5.10.0-10-amd64 #1 SMP Debian 5.10.84-1 (2021-12-08) x86_64 GNU/Linux sdsadmin@ceph-pacific-1:~$ sudo ceph -v ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable) Thanks for your prompt reply. Regards Sandip Divekar -----Original Message----- From: Chris Palmer <chris.palmer@idnet.com><mailto:chris.palmer@idnet.com> Sent: Thursday, May 25, 2023 7:25 PM To: ceph-users@ceph.io<mailto:ceph-users@ceph.io> Subject: [ceph-users] Re: Unexpected behavior of directory mtime after being set explicitly ***** EXTERNAL EMAIL ***** Hi Milind I just tried this using the ceph kernel client and ceph-common 17.2.6 package in the latest Fedora kernel, against Ceph 17.2.6 and it worked perfectly... There must be some other factor in play. Chris On 25/05/2023 13:04, Sandip Divekar wrote: Hello Milind, We are using Ceph Kernel Client. But we found this same behavior while using Libcephfs library. Should we treat this as a bug? Or Is there any existing bug for similar issue ? Thanks and Regards, Sandip Divekar From: Milind Changire <mchangir@redhat.com><mailto:mchangir@redhat.com> Sent: Thursday, May 25, 2023 4:24 PM To: Sandip Divekar <sandip.divekar@hitachivantara.com><mailto:sandip.divekar@hitachivantara.com> Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io>; dev@ceph.io<mailto:dev@ceph.io> Subject: Re: [ceph-users] Unexpected behavior of directory mtime after being set explicitly ***** EXTERNAL EMAIL ***** Sandip, What type of client are you using ? kernel client or fuse client ? If it's the kernel client, then it's a bug. FYI - Pacific and Quincy fuse clients do the right thing On Wed, May 24, 2023 at 9:24 PM Sandip Divekar <sandip.divekar@hitachivantara.com<mailto:sandip.divekar@hitachivantara.com><mailto:sandip.divekar@hitachivantara.com><mailto:sandip.divekar@hitachivantara.com>> wrote: Hi Team, I'm writing to bring to your attention an issue we have encountered with the "mtime" (modification time) behavior for directories in the Ceph filesystem. Upon observation, we have noticed that when the mtime of a directory (let's say: dir1) is explicitly changed in CephFS, subsequent additions of files or directories within 'dir1' fail to update the directory's mtime as expected. This behavior appears to be specific to CephFS - we have reproduced this issue on both Quincy and Pacific. Similar steps work as expected in the ext4 filesystem amongst others. Reproduction steps: 1. Create a directory - mkdir dir1 2. Modify mtime using the touch command - touch dir1 3. Create a file or directory inside of 'dir1' - mkdir dir1/dir2 Expected result: mtime for dir1 should change to the time the file or directory was created in step 3 Actual result: there was no change to the mtime for 'dir1' Note : For more detail, kindly find the attached logs. Our queries are : 1. Is this expected behavior for CephFS? 2. If so, can you explain why the directory behavior is inconsistent depending on whether the mtime for the directory has previously been manually updated. Best Regards, Sandip Divekar Component QA Lead SDET. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io> -- Milind _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
Yeah, it's a bug. I have raised on ceph tracker to follow this: https://tracker.ceph.com/issues/61584 And I have found the root cause, more detail please see my comments on the above tracker. I am still going through the code to find one way to fix it. Thanks - Xiubo On 6/5/23 13:42, Sandip Divekar wrote:
Hi Team,
I kindly request your input on whether we should consider the “mtime” issue as bug or not.
Thanks Sandip
From: Sandip Divekar Sent: Monday, May 29, 2023 2:19 PM To: Chris Palmer <chris.palmer@idnet.com>; ceph-users@ceph.io; Gregory Farnum <gfarnum@redhat.com> Cc: dev@ceph.io; Gavin Lucas <gavin.lucas@hitachivantara.com>; Joseph Fernandes <joseph.fernandes@hitachivantara.com>; Simon Crosland <Simon.Crosland@hitachivantara.com> Subject: RE: [ceph-users] Re: Unexpected behavior of directory mtime after being set explicitly
Hi Chris / Gregory,
Did you get a chance to investigate this issue ?
Thanks and Regards Sandip Divekar
From: Sandip Divekar Sent: Thursday, May 25, 2023 11:16 PM To: Chris Palmer <chris.palmer@idnet.com<mailto:chris.palmer@idnet.com>>; ceph-users@ceph.io<mailto:ceph-users@ceph.io> Cc: dev@ceph.io<mailto:dev@ceph.io>; Gavin Lucas <gavin.lucas@hitachivantara.com<mailto:gavin.lucas@hitachivantara.com>>; Joseph Fernandes <joseph.fernandes@hitachivantara.com<mailto:joseph.fernandes@hitachivantara.com>>; Simon Crosland <Simon.Crosland@hitachivantara.com<mailto:Simon.Crosland@hitachivantara.com>> Subject: RE: [ceph-users] Re: Unexpected behavior of directory mtime after being set explicitly
Hi Chris,
I think, you have missed one steps and that is to change mtime for directory explicitly. Please have a look at highlighted steps.
CEPHFS
===========================================================================
root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# mkdir dir1
root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# stat dir1
File: dir1
Size: 0 Blocks: 0 IO Block: 65536 directory
Device: 28h/40d Inode: 1099511714911 Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2023-05-24 11:09:25.260851345 +0530
Modify: 2023-05-24 11:09:25.260851345 +0530
Change: 2023-05-24 11:09:25.260851345 +0530
Birth: 2023-05-24 11:09:25.260851345 +0530
root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# touch -m -d '26 Aug 1982 22:00' dir1
root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# stat dir1/
File: dir1/
Size: 0 Blocks: 0 IO Block: 65536 directory
Device: 28h/40d Inode: 1099511714911 Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2023-05-24 11:09:25.260851345 +0530
Modify: 1982-08-26 22:00:00.000000000 +0530
Change: 2023-05-24 11:10:04.881454967 +0530
Birth: 2023-05-24 11:09:25.260851345 +0530
root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# mkdir dir1/dir2
root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538# stat dir1/
File: dir1/
Size: 1 Blocks: 0 IO Block: 65536 directory
Device: 28h/40d Inode: 1099511714911 Links: 3
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2023-05-24 11:09:25.260851345 +0530
Modify: 1982-08-26 22:00:00.000000000 +0530
Change: 2023-05-24 11:10:19.141672220 +0530
Birth: 2023-05-24 11:09:25.260851345 +0530
root@sds-ceph:/mnt/cephfs/volumes/_nogroup/test1/d5052b71-39ec-4d0a-9b0b-2091e1723538#
Note : In a last step, it is expected that “Modify” time should change.
Thanks and Regards Sandip Divekar
From: Chris Palmer <chris.palmer@idnet.com<mailto:chris.palmer@idnet.com>> Sent: Thursday, May 25, 2023 9:46 PM To: Sandip Divekar <sandip.divekar@hitachivantara.com<mailto:sandip.divekar@hitachivantara.com>>; ceph-users@ceph.io<mailto:ceph-users@ceph.io> Cc: dev@ceph.io<mailto:dev@ceph.io>; Gavin Lucas <gavin.lucas@hitachivantara.com<mailto:gavin.lucas@hitachivantara.com>>; Joseph Fernandes <joseph.fernandes@hitachivantara.com<mailto:joseph.fernandes@hitachivantara.com>>; Simon Crosland <Simon.Crosland@hitachivantara.com<mailto:Simon.Crosland@hitachivantara.com>> Subject: Re: [ceph-users] Re: Unexpected behavior of directory mtime after being set explicitly
***** EXTERNAL EMAIL ***** Hi Sandip
Ceph servers (debian11/ceph base with Proxmox installed on top - NOT the ceph that comes with Proxmox!):
ceph@pve1:~$ uname -a
Linux pve1 5.15.107-2-pve #1 SMP PVE 5.15.107-2 (2023-05-10T09:10Z) x86_64 GNU/Linux
ceph@pve1:~$ ceph version
ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
Fedora workstation. I waited until the minute had clicked over before doing each step:
[chris@rex mtime]$ uname -a
Linux rex.palmer 6.2.15-300.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Thu May 11 17:37:39 UTC 2023 x86_64 GNU/Linux
[chris@rex mtime]$ rpm -q ceph-common
ceph-common-17.2.6-2.fc38.x86_64
[chris@rex mtime]$ df .
Filesystem 1K-blocks Used Available Use% Mounted on
192.168.80.121,192.168.80.122,192.168.80.123:/data2 8589930496 4944801792 3645128704 58% /mnt/data2
[chris@rex mtime]$ mount|grep data2
systemd-1 on /mnt/data2 type autofs (rw,relatime,fd=61,pgrp=1,timeout=600,minproto=5,maxproto=5,direct,pipe_ino=22804)
192.168.80.121,192.168.80.122,192.168.80.123:/data2 on /mnt/data2 type ceph (rw,noatime,nodiratime,name=data2-rex,secret=<hidden>,fsid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx,acl,_netdev,x-systemd.mount-timeout=30,x-systemd.automount,x-systemd.idle-timeout=600)
[chris@rex mtime]$ date; mkdir one; ls -ld one
Thu 25 May 16:57:28 BST 2023
drwxrwxr-x 2 chris groupname 0 May 25 16:57 one
[chris@rex mtime]$ date; touch one; ls -ld one
Thu 25 May 16:58:14 BST 2023
drwxrwxr-x 2 chris groupname 0 May 25 16:58 one
[chris@rex mtime]$ date; mkdir one/two; ls -ld one
Thu 25 May 16:59:26 BST 2023
drwxrwxr-x 3 chris groupname 1 May 25 16:59 one
I also repeated it with the test run on the ceph debian11 server, having mounted the cephfs filesystem on the ceph server - exactly the same result.
I then repeated it again on a pure debian11 ceph 17.2.6 cluster, using a debian11 client, and it also worked as expected.
All systems have latest patches applied.
Hope that helps Chris On 25/05/2023 15:57, Sandip Divekar wrote:
Hi Chris,
Kindly request you that follow steps given in previous mail and paste the output here.
The reason behind this request is that we have encountered an issue which is easily reproducible on
Latest version of both quincy and pacific, also we have thoroughly investigated the matter and we are certain that
No other factors are at play in this scenario.
Note : We have used Debian 11 for testing.
sdsadmin@ceph-pacific-1:~$ uname -a
Linux ceph-pacific-1 5.10.0-10-amd64 #1 SMP Debian 5.10.84-1 (2021-12-08) x86_64 GNU/Linux
sdsadmin@ceph-pacific-1:~$ sudo ceph -v
ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)
Thanks for your prompt reply.
Regards
Sandip Divekar
-----Original Message-----
From: Chris Palmer <chris.palmer@idnet.com><mailto:chris.palmer@idnet.com>
Sent: Thursday, May 25, 2023 7:25 PM
To: ceph-users@ceph.io<mailto:ceph-users@ceph.io>
Subject: [ceph-users] Re: Unexpected behavior of directory mtime after being set explicitly
***** EXTERNAL EMAIL *****
Hi Milind
I just tried this using the ceph kernel client and ceph-common 17.2.6 package in the latest Fedora kernel, against Ceph 17.2.6 and it worked perfectly...
There must be some other factor in play.
Chris
On 25/05/2023 13:04, Sandip Divekar wrote:
Hello Milind,
We are using Ceph Kernel Client.
But we found this same behavior while using Libcephfs library.
Should we treat this as a bug? Or
Is there any existing bug for similar issue ?
Thanks and Regards,
Sandip Divekar
From: Milind Changire <mchangir@redhat.com><mailto:mchangir@redhat.com>
Sent: Thursday, May 25, 2023 4:24 PM
To: Sandip Divekar <sandip.divekar@hitachivantara.com><mailto:sandip.divekar@hitachivantara.com>
Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io>; dev@ceph.io<mailto:dev@ceph.io>
Subject: Re: [ceph-users] Unexpected behavior of directory mtime after
being set explicitly
***** EXTERNAL EMAIL *****
Sandip,
What type of client are you using ?
kernel client or fuse client ?
If it's the kernel client, then it's a bug.
FYI - Pacific and Quincy fuse clients do the right thing
On Wed, May 24, 2023 at 9:24 PM Sandip Divekar <sandip.divekar@hitachivantara.com<mailto:sandip.divekar@hitachivantara.com><mailto:sandip.divekar@hitachivantara.com><mailto:sandip.divekar@hitachivantara.com>> wrote:
Hi Team,
I'm writing to bring to your attention an issue we have encountered with the "mtime" (modification time) behavior for directories in the Ceph filesystem.
Upon observation, we have noticed that when the mtime of a directory
(let's say: dir1) is explicitly changed in CephFS, subsequent additions of files or directories within 'dir1' fail to update the directory's mtime as expected.
This behavior appears to be specific to CephFS - we have reproduced this issue on both Quincy and Pacific. Similar steps work as expected in the ext4 filesystem amongst others.
Reproduction steps:
1. Create a directory - mkdir dir1
2. Modify mtime using the touch command - touch dir1 3. Create a file
or directory inside of 'dir1' - mkdir dir1/dir2 Expected result:
mtime for dir1 should change to the time the file or directory was
created in step 3 Actual result:
there was no change to the mtime for 'dir1'
Note : For more detail, kindly find the attached logs.
Our queries are :
1. Is this expected behavior for CephFS?
2. If so, can you explain why the directory behavior is inconsistent depending on whether the mtime for the directory has previously been manually updated.
Best Regards,
Sandip Divekar
Component QA Lead SDET.
_______________________________________________
ceph-users mailing list --
ceph-users@ceph.io<mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io>
--
Milind
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an
email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Raised one PR to fix this, please see https://github.com/ceph/ceph/pull/51931. Thanks - Xiubo On 5/24/23 23:52, Sandip Divekar wrote:
Hi Team,
I'm writing to bring to your attention an issue we have encountered with the "mtime" (modification time) behavior for directories in the Ceph filesystem.
Upon observation, we have noticed that when the mtime of a directory (let's say: dir1) is explicitly changed in CephFS, subsequent additions of files or directories within 'dir1' fail to update the directory's mtime as expected.
This behavior appears to be specific to CephFS - we have reproduced this issue on both Quincy and Pacific. Similar steps work as expected in the ext4 filesystem amongst others.
Reproduction steps: 1. Create a directory - mkdir dir1 2. Modify mtime using the touch command - touch dir1 3. Create a file or directory inside of 'dir1' - mkdir dir1/dir2 Expected result: mtime for dir1 should change to the time the file or directory was created in step 3 Actual result: there was no change to the mtime for 'dir1'
Note : For more detail, kindly find the attached logs.
Our queries are : 1. Is this expected behavior for CephFS? 2. If so, can you explain why the directory behavior is inconsistent depending on whether the mtime for the directory has previously been manually updated.
Best Regards, Sandip Divekar Component QA Lead SDET.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (7)
-
Chris Palmer
-
Gregory Farnum
-
Joseph Fernandes
-
Milind Changire
-
Sandip Divekar
-
sandip.divekar＠hitachivantara.com
-
Xiubo Li