[Cephfs] Can't get snapshot under a subvolume
Hello all, We're getting a "Operation not permitted" error while trying to create a snapshot on the client. It is somehow related to previously-asked Pacific issue mentioned here: https://www.spinics.net/lists/ceph-users/msg67908.html We are on squid (19.2.1) and the given workaround seems to be only temporary. Anyway, here are some details about the issue: Cephx side: ``` client.mount-update-production key: *** caps: [mds] allow rws fsname=production-cephfs path=/volumes/_nogroup/update-production caps: [mon] allow r fsname=production-cephfs caps: [osd] allow rw tag cephfs data=production-cephfs ``` FS flags: ``` ~> sudo ceph fs lsflags production-cephfs joinable allow_snaps allow_multimds_snaps ``` I got the path via following command: ``` ~> sudo ceph fs subvolume getpath production-cephfs update-production /volumes/_nogroup/update-production/e155ba30-d201-4653-be3f-86533324ee5f ``` And mounted to client with mentioned key: ``` ~> mount | grep ceph mount-update-production@00000000-0000-0000-0000-000000000000.production-cephfs=/volumes/_nogroup/update-production/e155ba30-d201-4653-be3f-86533324ee5f on /opt/aptly/.aptly type ceph (rw,noatime,name=mount-update-production,secret=<hidden>,acl,mon_addr=[multiple mon addresses here]) ``` Everything works flawlessly. Yet, when we want to get a snapshot: ``` /opt/aptly/.aptly/pool/9c/0b ~> sudo mkdir .snap/test mkdir: cannot create directory ‘.snap/test’: Operation not permitted ``` If I mount the whole FS with admin rights and run `setfattr -n ceph.dir.subvolume -v 0` for _all_ directories of following path one by one: /volumes/_nogroup/update-production/e155ba30-d201-4653-be3f-86533324ee5f, then it is solving the issue. But it does not last; not sure what is triggering it again, but if I try snapshotting anything next day, getting the same "Operation not permitted" error again. Is there something I can try, or am I missing something obvious? Thanks, Gürkan
Hello Gürkan, Indeed, by design, snapshots are only possible at the top of a subvolume (which is invisible in a pod). This is deliberate, as otherwise they would become a mechanism for quota evasion. You can try CSI-level snapshots, which use cephfs-level snapshots under the hood. Or, you can even try giving your pod a permission to talk to the CSI driver, which would then make snapshots on its behalf, but I don't know offhand how to do it. In any case, please also try asking in Kubernetes forums. On Ceph side, unfortunately, everything works as intended. On Sat, Mar 1, 2025 at 2:58 AM <ceph@gurkan.in> wrote:
Hello all,
We're getting a "Operation not permitted" error while trying to create a snapshot on the client. It is somehow related to previously-asked Pacific issue mentioned here: https://www.spinics.net/lists/ceph-users/msg67908.html
We are on squid (19.2.1) and the given workaround seems to be only temporary. Anyway, here are some details about the issue:
Cephx side:
``` client.mount-update-production key: *** caps: [mds] allow rws fsname=production-cephfs path=/volumes/_nogroup/update-production caps: [mon] allow r fsname=production-cephfs caps: [osd] allow rw tag cephfs data=production-cephfs ```
FS flags: ``` ~> sudo ceph fs lsflags production-cephfs joinable allow_snaps allow_multimds_snaps ```
I got the path via following command: ``` ~> sudo ceph fs subvolume getpath production-cephfs update-production /volumes/_nogroup/update-production/e155ba30-d201-4653-be3f-86533324ee5f ```
And mounted to client with mentioned key: ``` ~> mount | grep ceph mount-update-production@00000000-0000-0000-0000-000000000000.production-cephfs=/volumes/_nogroup/update-production/e155ba30-d201-4653-be3f-86533324ee5f on /opt/aptly/.aptly type ceph (rw,noatime,name=mount-update-production,secret=<hidden>,acl,mon_addr=[multiple mon addresses here]) ```
Everything works flawlessly. Yet, when we want to get a snapshot: ``` /opt/aptly/.aptly/pool/9c/0b ~> sudo mkdir .snap/test mkdir: cannot create directory ‘.snap/test’: Operation not permitted ```
If I mount the whole FS with admin rights and run `setfattr -n ceph.dir.subvolume -v 0` for _all_ directories of following path one by one: /volumes/_nogroup/update-production/e155ba30-d201-4653-be3f-86533324ee5f, then it is solving the issue. But it does not last; not sure what is triggering it again, but if I try snapshotting anything next day, getting the same "Operation not permitted" error again.
Is there something I can try, or am I missing something obvious?
Thanks, Gürkan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
Hi,
This is deliberate, as otherwise they would become a mechanism for quota evasion.
This.. does not make much sense. If I give the setfattr command, everything works fine. Plus the documentation says following: Arbitrary subtrees. Snapshots are created within any directory you choose, and cover all data in the file system under that directory. Ref: https://docs.ceph.com/en/squid/dev/cephfs-snapshots/
In any case, please also try asking in Kubernetes forums. On Ceph side, unfortunately, everything works as intended.
I am also not using Kubernetes. This is a deployment over Debian bookworm VMs. Never mentioned a pod, the client is another Debian VM.
Hello Gürkan, Let me clarify and correct my answer. I incorrectly assumed that you use Kubernetes, because its CSI driver is, by far, the main consumer of subvolumes. Still, let me explain this use case, as the limitations you observe naturally follow from it. Subvolumes exist to implement a notion of managed mountable directories with a given maximum size, as required, e.g., by Kubernetes RWX Persistent Volumes. In Kubernetes, the CSI driver, when it needs to, creates a subvolume, sets a quota on it, creates a dummy subdirectory, and mounts it in the pod that needs the Persistent Volume. As such, the pod has no access to the top directory of the persistent volume and thus cannot increase the quota by changing the xattr. However, if Ceph permitted snapshots at arbitrary points within the volume, a malicious pod could have created a snapshot, deleted everything (not for real, "thanks" to the snapshot), written new files, and thus evaded the quota. Thus, the only point where snapshots are allowed is the top directory of the subvolume, where the CSI driver can do it. Therefore, the answer to your original question is: if you want client-managed snapshots, do not use subvolumes, they are the wrong abstraction for you. Just create plain old directories outside of the /volumes path and mount them on the client. And yes, the documentation you mention does need to be corrected. On Sun, Mar 2, 2025 at 5:44 AM Gürkan G <ceph@gurkan.in> wrote:
Hi,
This is deliberate, as otherwise they would become a mechanism for quota evasion.
This.. does not make much sense. If I give the setfattr command, everything works fine. Plus the documentation says following:
Arbitrary subtrees. Snapshots are created within any directory you choose, and cover all data in the file system under that directory. Ref: https://docs.ceph.com/en/squid/dev/cephfs-snapshots/
In any case, please also try asking in Kubernetes forums. On Ceph side, unfortunately, everything works as intended.
I am also not using Kubernetes. This is a deployment over Debian bookworm VMs. Never mentioned a pod, the client is another Debian VM. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
Subvolumes exist to implement a notion of managed mountable
However, if Ceph permitted snapshots at arbitrary points within
Hi again Alexander, Thanks for taking the time. directories with a given maximum size, as required, e.g., by Kubernetes RWX Persistent Volumes. I highly doubt that the main reason was this, since (afaik) snapshots feature predate Kubernetes. the volume, a malicious pod could have created a snapshot, deleted everything (not for real, "thanks" to the snapshot), written new files, and thus evaded the quota. Then the admin can opt to not use allow_new_snaps in that FS, or provide key without "s" flag so client would be unable to create snapshots. I am not proficient with cpp, but looks like even MDS code has special handling of snapshots+subvolumes, even within quota-restricted ones. But again, this might be an oversight and conflict of features on MDS side.
And yes, the documentation you mention does need to be corrected.
Could you point for the place to report this to the documentation? I would've designed my implementation completely different if this was not implied. Thanks again, Gürkan On 02/03/2025 02.10, Alexander Patrakov wrote:
Hello Gürkan,
Let me clarify and correct my answer.
I incorrectly assumed that you use Kubernetes, because its CSI driver is, by far, the main consumer of subvolumes. Still, let me explain this use case, as the limitations you observe naturally follow from it.
Subvolumes exist to implement a notion of managed mountable directories with a given maximum size, as required, e.g., by Kubernetes RWX Persistent Volumes. In Kubernetes, the CSI driver, when it needs to, creates a subvolume, sets a quota on it, creates a dummy subdirectory, and mounts it in the pod that needs the Persistent Volume. As such, the pod has no access to the top directory of the persistent volume and thus cannot increase the quota by changing the xattr. However, if Ceph permitted snapshots at arbitrary points within the volume, a malicious pod could have created a snapshot, deleted everything (not for real, "thanks" to the snapshot), written new files, and thus evaded the quota. Thus, the only point where snapshots are allowed is the top directory of the subvolume, where the CSI driver can do it.
Therefore, the answer to your original question is: if you want client-managed snapshots, do not use subvolumes, they are the wrong abstraction for you. Just create plain old directories outside of the /volumes path and mount them on the client.
And yes, the documentation you mention does need to be corrected.
On Sun, Mar 2, 2025 at 5:44 AM Gürkan G <ceph@gurkan.in> wrote:
Hi,
This is deliberate, as otherwise they would become a mechanism for quota evasion. This.. does not make much sense. If I give the setfattr command, everything works fine. Plus the documentation says following:
Arbitrary subtrees. Snapshots are created within any directory you choose, and cover all data in the file system under that directory. Ref: https://docs.ceph.com/en/squid/dev/cephfs-snapshots/
In any case, please also try asking in Kubernetes forums. On Ceph side, unfortunately, everything works as intended. I am also not using Kubernetes. This is a deployment over Debian bookworm VMs. Never mentioned a pod, the client is another Debian VM.
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Gürkan, On Sun, Mar 2, 2025 at 4:20 PM Gürkan G <ceph@gurkan.in> wrote:
Hi again Alexander,
Thanks for taking the time.
Subvolumes exist to implement a notion of managed mountable directories with a given maximum size, as required, e.g., by Kubernetes RWX Persistent Volumes.
I highly doubt that the main reason was this, since (afaik) snapshots feature predate Kubernetes.
However, if Ceph permitted snapshots at arbitrary points within the volume, a malicious pod could have created a snapshot, deleted everything (not for real, "thanks" to the snapshot), written new files, and thus evaded the quota.
Then the admin can opt to not use allow_new_snaps in that FS, or provide key without "s" flag so client would be unable to create snapshots.
I am not proficient with cpp, but looks like even MDS code has special handling of snapshots+subvolumes, even within quota-restricted ones.
But again, this might be an oversight and conflict of features on MDS side.
And yes, the documentation you mention does need to be corrected.
Could you point for the place to report this to the documentation? I would've designed my implementation completely different if this was not implied.
The proper place to report documentation bugs is https://pad.ceph.com/p/Report_Documentation_Bugs, linked from the top of every Ceph documentation page.
Thanks again,
Gürkan
On 02/03/2025 02.10, Alexander Patrakov wrote:
Hello Gürkan,
Let me clarify and correct my answer.
I incorrectly assumed that you use Kubernetes, because its CSI driver is, by far, the main consumer of subvolumes. Still, let me explain this use case, as the limitations you observe naturally follow from it.
Subvolumes exist to implement a notion of managed mountable directories with a given maximum size, as required, e.g., by Kubernetes RWX Persistent Volumes. In Kubernetes, the CSI driver, when it needs to, creates a subvolume, sets a quota on it, creates a dummy subdirectory, and mounts it in the pod that needs the Persistent Volume. As such, the pod has no access to the top directory of the persistent volume and thus cannot increase the quota by changing the xattr. However, if Ceph permitted snapshots at arbitrary points within the volume, a malicious pod could have created a snapshot, deleted everything (not for real, "thanks" to the snapshot), written new files, and thus evaded the quota. Thus, the only point where snapshots are allowed is the top directory of the subvolume, where the CSI driver can do it.
Therefore, the answer to your original question is: if you want client-managed snapshots, do not use subvolumes, they are the wrong abstraction for you. Just create plain old directories outside of the /volumes path and mount them on the client.
And yes, the documentation you mention does need to be corrected.
On Sun, Mar 2, 2025 at 5:44 AM Gürkan G <ceph@gurkan.in> wrote:
Hi,
This is deliberate, as otherwise they would become a mechanism for quota evasion. This.. does not make much sense. If I give the setfattr command, everything works fine. Plus the documentation says following:
Arbitrary subtrees. Snapshots are created within any directory you choose, and cover all data in the file system under that directory. Ref: https://docs.ceph.com/en/squid/dev/cephfs-snapshots/
In any case, please also try asking in Kubernetes forums. On Ceph side, unfortunately, everything works as intended. I am also not using Kubernetes. This is a deployment over Debian bookworm VMs. Never mentioned a pod, the client is another Debian VM.
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
participants (3)
-
Alexander Patrakov
-
ceph＠gurkan.in
-
Gürkan G