Re: CephFS with cache-tier kernel-mount client unable to write (Nautilus)
On Thu, Jan 23, 2020 at 2:36 PM Ilya Dryomov <idryomov@gmail.com> wrote:
On Wed, Jan 22, 2020 at 6:18 PM Hayashida, Mami <mami.hayashida@uky.edu> wrote:
Thanks, Ilya.
I just tried modifying the osd cap for client.testuser by getting rid of "tag cephfs data=cephfs_test" part and confirmed this key does work (i.e. lets the CephFS client read/write). It now reads:
[client.testuser] key = XXXYYYYZZZ caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rw" // previously "allow rw tag cephfs data=cephfs_test"
I tried removing either "tag cephfs" or "data=cephfs_test" (and leaving the other), but neither worked.
Now, here is my question: will not having the "allow rw tag cephfs data=<file system name>" (under osd caps) result in a security/privacy loophole in a production cluster? (I am still trying to assess whether having a Cache Tier behind CephFS is worth all the headaches...)
It's probably not worth it. Unless you have a specific tiered workload in mind and your cache pool is large enough for it, I'd recommend staying away from cache tiering.
"allow rw" for osd is only marginally more restrictive than client.admin's "allow *", allowing the user to read/write every object in the cluster. Scratch my reply about doing it by hand -- try the following:
$ ceph osd pool application enable cephfs-data-cache cephfs $ ceph osd pool application set cephfs-data-cache cephfs data cephfs_test $ ceph fs authorize cephfs_test ... (as before)
You will see the same "allow rw tag cephfs data=cephfs_test" cap in "ceph auth list" output, but it should allow accessing cephfs-data-cache.
Dropping ceph-users@lists.ceph.com and resending to ceph-users@ceph.io. Thanks, Ilya
Thanks, Ilya. First, I was not sure whether to post my question on @ceph.io or @ lists.ceph.com (I subscribe to both) -- should I use @ceph.io in the future? Second, thanks for your advice on cache-tiering -- I was starting to feel that way but always good to know what Ceph "experts" would say. Third, I tried enabling (and setting) the pool application commands you outlined but got errors (Ceph is not allowing me to enable/set application on the cache tier) $ ceph osd pool application enable cephfs-data-cache cephfs Error EINVAL: application must be enabled on base tier $ ceph osd pool application set cephfs-data-cache cephfs data cephfs_test Error EINVAL: application metadata must be set on base tier Since at this point, it is highly unlikely that we will be utilizing cache-tier on our production clusters, and there is a work around it (by manually creating a CephFS client key), this is nothing serious or urgent; but I thought I should let you guys know. Again, thanks for your help! Mami On Thu, Jan 23, 2020 at 8:40 AM Ilya Dryomov <idryomov@gmail.com> wrote:
On Thu, Jan 23, 2020 at 2:36 PM Ilya Dryomov <idryomov@gmail.com> wrote:
On Wed, Jan 22, 2020 at 6:18 PM Hayashida, Mami <mami.hayashida@uky.edu>
wrote:
Thanks, Ilya.
I just tried modifying the osd cap for client.testuser by getting rid
of "tag cephfs data=cephfs_test" part and confirmed this key does work (i.e. lets the CephFS client read/write). It now reads:
[client.testuser] key = XXXYYYYZZZ caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rw" // previously "allow rw tag cephfs
data=cephfs_test"
I tried removing either "tag cephfs" or "data=cephfs_test" (and
leaving the other), but neither worked.
Now, here is my question: will not having the "allow rw tag cephfs
data=<file system name>" (under osd caps) result in a security/privacy loophole in a production cluster? (I am still trying to assess whether having a Cache Tier behind CephFS is worth all the headaches...)
It's probably not worth it. Unless you have a specific tiered workload in mind and your cache pool is large enough for it, I'd recommend staying away from cache tiering.
"allow rw" for osd is only marginally more restrictive than client.admin's "allow *", allowing the user to read/write every object in the cluster. Scratch my reply about doing it by hand -- try the following:
$ ceph osd pool application enable cephfs-data-cache cephfs $ ceph osd pool application set cephfs-data-cache cephfs data cephfs_test $ ceph fs authorize cephfs_test ... (as before)
You will see the same "allow rw tag cephfs data=cephfs_test" cap in "ceph auth list" output, but it should allow accessing cephfs-data-cache.
Dropping ceph-users@lists.ceph.com and resending to ceph-users@ceph.io.
Thanks,
Ilya
On Thu, Jan 23, 2020 at 3:31 PM Hayashida, Mami <mami.hayashida@uky.edu> wrote:
Thanks, Ilya.
First, I was not sure whether to post my question on @ceph.io or @lists.ceph.com (I subscribe to both) -- should I use @ceph.io in the future?
Yes. I got the following when I replied to your previous email: As you may or may not be aware, most of the Ceph mailing lists have migrated to a new self-hosted instance of Mailman 3. For the past few months, both this list and ceph-users@ceph.io have been enabled. As of January 22, 2020, mail sent to ceph-users@lists.ceph.com will no longer be delivered. This domain will remain so that permalinks to archives are preserved. Please use the new address ceph-users@ceph.io instead.
Second, thanks for your advice on cache-tiering -- I was starting to feel that way but always good to know what Ceph "experts" would say.
Third, I tried enabling (and setting) the pool application commands you outlined but got errors (Ceph is not allowing me to enable/set application on the cache tier)
$ ceph osd pool application enable cephfs-data-cache cephfs Error EINVAL: application must be enabled on base tier $ ceph osd pool application set cephfs-data-cache cephfs data cephfs_test Error EINVAL: application metadata must be set on base tier
Since at this point, it is highly unlikely that we will be utilizing cache-tier on our production clusters, and there is a work around it (by manually creating a CephFS client key), this is nothing serious or urgent; but I thought I should let you guys know.
I haven't actually tried it. There is probably a way to make it work (recreating the pools and doing the application stuff before tiering or something along those lines), but yeah, cache tiering is not without sharp edges... Thanks, Ilya
participants (2)
-
Hayashida, Mami
-
Ilya Dryomov