samba ceph-vfs and scrubbing interval
Hi all, i‘m running a 3 node ceph cluster setup with collocated mons and mds for actually 3 filesystems at home since mimic. I’m planning to downgrade to one FS and use RBD in the future, but this is another story. I’m using the cluster as cold storage on spindles with EC-pools for archive purposes. The cluster usually does not run 24/7. I actually managed to upgrade to octopus without problems yesterday. So first of all: great job with the release. Now I have a little problem and a general question to address. I have tried to share the CephFS via samba and the ceph-vfs module but I could not manage to get write access (read access is not a problem) to the share (even with the admin key). When I share the mounted path (kernel module or fuser mount) instead as usual there are no problems at all. Is ceph-vfs generally read only and I missed this point? Furthermore I suppose, that there is no possibility to choose between the different mds namespaces, right? Now the general question. Since the cluster does not run 24/7 as stated and is turned on perhaps once a week for a couple of hours on demand, what are reasonable settings for the scrubbing intervals? As I said, the storage is cold and there is mostly read i/o. The archiving process adds approximately 0.5 % of new data of the cluster’s total storage capacity. Stay healthy and regards, Marco Savoca
On Fri, 2020-03-27 at 12:00 +0100, Marco Savoca wrote:
Hi all,
i‘m running a 3 node ceph cluster setup with collocated mons and mds for actually 3 filesystems at home since mimic. I’m planning to downgrade to one FS and use RBD in the future, but this is another story. I’m using the cluster as cold storage on spindles with EC-pools for archive purposes. The cluster usually does not run 24/7. I actually managed to upgrade to octopus without problems yesterday. So first of all: great job with the release.
Now I have a little problem and a general question to address.
I have tried to share the CephFS via samba and the ceph-vfs module but I could not manage to get write access (read access is not a problem) to the share (even with the admin key). When I share the mounted path (kernel module or fuser mount) instead as usual there are no problems at all. Is ceph-vfs generally read only and I missed this point?
No. I haven't tested it in some time, but it does allow clients to write. When you say you can't get write access, what are you doing to test this, and what error are you getting back?
Furthermore I suppose, that there is no possibility to choose between the different mds namespaces, right?
Yeah, doesn't look like anyone has added that. That would probably be pretty easy to add, though it would take a little while to trickle out to the distros.
Now the general question. Since the cluster does not run 24/7 as stated and is turned on perhaps once a week for a couple of hours on demand, what are reasonable settings for the scrubbing intervals? As I said, the storage is cold and there is mostly read i/o. The archiving process adds approximately 0.5 % of new data of the cluster’s total storage capacity.
-- Jeff Layton <jlayton@redhat.com>
Hi Marco and Jeff, On Fri, 27 Mar 2020 08:04:56 -0400, Jeff Layton wrote:
i‘m running a 3 node ceph cluster setup with collocated mons and mds for actually 3 filesystems at home since mimic. I’m planning to downgrade to one FS and use RBD in the future, but this is another story. I’m using the cluster as cold storage on spindles with EC-pools for archive purposes. The cluster usually does not run 24/7. I actually managed to upgrade to octopus without problems yesterday. So first of all: great job with the release.
Now I have a little problem and a general question to address.
I have tried to share the CephFS via samba and the ceph-vfs module but I could not manage to get write access (read access is not a problem) to the share (even with the admin key). When I share the mounted path (kernel module or fuser mount) instead as usual there are no problems at all. Is ceph-vfs generally read only and I missed this point?
No. I haven't tested it in some time, but it does allow clients to write. When you say you can't get write access, what are you doing to test this, and what error are you getting back?
Is write access granted via a supplementary group ID? If so, this might be https://bugzilla.samba.org/show_bug.cgi?id=14053 . Fixing libcephfs supplementary group ID fallback behaviour was discussed earlier via https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/PCIOZRE5FJCQ2LZXLZC... Cheers, David
Hi Davod, Hi Jeff, Indeed! The files and directories group permissions are set to the “users” group. If the the primary group of the samba user is set to users group it works as expected, elsewise not. I’m using Ubuntu 18.04 on the samba server with samba from the district repo. Cheers, Marco
Am 31.03.2020 um 01:30 schrieb David Disseldorp <ddiss@suse.de>:
Hi Marco and Jeff,
On Fri, 27 Mar 2020 08:04:56 -0400, Jeff Layton wrote:
i‘m running a 3 node ceph cluster setup with collocated mons and mds for actually 3 filesystems at home since mimic. I’m planning to downgrade to one FS and use RBD in the future, but this is another story. I’m using the cluster as cold storage on spindles with EC-pools for archive purposes. The cluster usually does not run 24/7. I actually managed to upgrade to octopus without problems yesterday. So first of all: great job with the release.
Now I have a little problem and a general question to address.
I have tried to share the CephFS via samba and the ceph-vfs module but I could not manage to get write access (read access is not a problem) to the share (even with the admin key). When I share the mounted path (kernel module or fuser mount) instead as usual there are no problems at all. Is ceph-vfs generally read only and I missed this point?
No. I haven't tested it in some time, but it does allow clients to write. When you say you can't get write access, what are you doing to test this, and what error are you getting back?
Is write access granted via a supplementary group ID? If so, this might be https://bugzilla.samba.org/show_bug.cgi?id=14053 . Fixing libcephfs supplementary group ID fallback behaviour was discussed earlier via https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/PCIOZRE5FJCQ2LZXLZC...
Cheers, David
participants (3)
-
David Disseldorp
-
Jeff Layton
-
Marco Savoca