21 Mar
2024
21 Mar
'24
7:12 p.m.
Hello Ceph List,
I'd like to formally let the wider community know of some work I've been
involved with for a while now: adding Managed SMB Protocol Support to Ceph.
SMB being the well known network file protocol native to Windows systems and
supported by MacOS (and Linux). The other key word "managed" meaning
integrating with Ceph management tooling - in this particular case cephadm for
orchestration and eventually a new MGR module for managing SMB shares.
The effort is still in it's very early stages. We have a PR adding initial
support for Samba Containers to cephadm [1] and a prototype for an smb MGR
module [2]. We plan on using container images based on the samba-container
project [3] - a team I am already part of. What we're aiming for is a feature
set similar to the current NFS integration in Ceph, but with a focus on
bridging non-Linux/Unix clients to CephFS using a protocol built into those
systems.
A few major features we have planned include:
* Standalone servers (internally defined users/groups)
* Active Directory Domain Member Servers
* Clustered Samba support
* Exporting Samba stats via Prometheus metrics
* A `ceph` cli workflow loosely based on the nfs mgr module
I wanted to share this information in case there's wider community interest in
this effort. I'm happy to take your questions / thoughts / suggestions in this
email thread, via Ceph slack (or IRC), or feel free to attend a Ceph
Orchestration weekly meeting! I try regularly attend and we sometimes discuss
design aspects of the smb effort there. It's on the Ceph Community Calendar.
Thanks!
[1] - https://github.com/ceph/ceph/pull/55068
[2] - https://github.com/ceph/ceph/pull/56350
[3] - https://github.com/samba-in-kubernetes/samba-container/
Thanks for reading,
--John Mulligan
21 Mar
21 Mar
7:25 p.m.
I think this is fantastic. Looking forward to the sambaxp talk too!
CephFS + SMB is something we make use of very much of, and have had a lot of
success working with. It is nice to see it getting some more integration.
Regards,
Bailey
-----Original Message-----
From: John Mulligan <phlogistonjohn(a)asynchrono.us>
Sent: March 21, 2024 4:13 PM
To: ceph-users(a)ceph.io; dev(a)ceph.io
Subject: [ceph-users] Call for Interest: Managed SMB Protocol Support
Hello Ceph List,
I'd like to formally let the wider community know of some work I've been
involved with for a while now: adding Managed SMB Protocol Support to
Ceph.
SMB being the well known network file protocol native to Windows systems
and supported by MacOS (and Linux). The other key word "managed"
meaning integrating with Ceph management tooling - in this particular case
cephadm for orchestration and eventually a new MGR module for managing
SMB shares.
The effort is still in it's very early stages. We have a PR adding initial
support
for Samba Containers to cephadm [1] and a prototype
for an smb MGR
module [2]. We plan on using container images based on the samba-
container project [3] - a team I am already part of. What we're aiming for
is
a
feature set similar to the current NFS integration in
Ceph, but with a
focus on
bridging non-Linux/Unix clients to CephFS using a
protocol built into
those
systems.
A few major features we have planned include:
* Standalone servers (internally defined users/groups)
* Active Directory Domain Member Servers
* Clustered Samba support
* Exporting Samba stats via Prometheus metrics
* A `ceph` cli workflow loosely based on the nfs mgr module
I wanted to share this information in case there's wider community
interest
in this effort. I'm happy to take your questions /
thoughts / suggestions
in
this email thread, via Ceph slack (or IRC), or feel
free to attend a Ceph
Orchestration weekly meeting! I try regularly attend and we sometimes
discuss design aspects of the smb effort there. It's on the Ceph Community
Calendar.
Thanks!
[1] - https://github.com/ceph/ceph/pull/55068
[2] - https://github.com/ceph/ceph/pull/56350
[3] - https://github.com/samba-in-kubernetes/samba-container/
Thanks for reading,
--John Mulligan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email
to ceph-users-leave(a)ceph.io
22 Mar
22 Mar
6:56 p.m.
Hi John,
A few major features we have planned include:
* Standalone servers (internally defined users/groups)
No concerns here
* Active Directory Domain Member Servers
In the second case, what is the plan regarding UID mapping? Is NFS
coexistence planned, or a concurrent mount of the same directory using
CephFS directly?
In fact, I am quite skeptical, because, at least in my experience,
every customer's SAMBA configuration as a domain member is a unique
snowflake, and cephadm would need an ability to specify arbitrary UID
mapping configuration to match what the customer uses elsewhere - and
the match must be precise.
Here is what I have seen or was told about:
1. We don't care about interoperability with NFS or CephFS, so we just
let SAMBA invent whatever UIDs and GIDs it needs using the "tdb2"
idmap backend. It's completely OK that workstations get different UIDs
and GIDs, as only SIDs traverse the wire.
2. [not seen in the wild, the customer did not actually implement it,
it's a product of internal miscommunication, and I am not sure if it
is valid at all] We don't care about interoperability with CephFS,
and, while we have NFS, security guys would not allow running NFS
non-kerberized. Therefore, no UIDs or GIDs traverse the wire, only
SIDs and names. Therefore, all we need is to allow both SAMBA and NFS
to use shared UID mapping allocated on as-needed basis using the
"tdb2" idmap module, and it doesn't matter that these UIDs and GIDs
are inconsistent with what clients choose.
3. We don't care about ACLs at all, and don't care about CephFS
interoperability. We set ownership of all new files to root:root 0666
using whatever options are available [well, I would rather use a
dedicated nobody-style uid/gid here]. All we care about is that only
authorized workstations or authorized users can connect to each NFS or
SMB share, and we absolutely don't want them to be able to set custom
ownership or ACLs.
4. We care about NFS and CephFS file ownership being consistent with
what Windows clients see. We store all UIDs and GIDs in Active
Directory using the rfc2307 schema, and it's mandatory that all
servers (especially SAMBA - thanks to the "ad" idmap backend) respect
that and don't try to invent anything [well, they do - BUILTIN/Users
gets its GID through tdb2]. Oh, and by the way, we have this strangely
low-numbered group that everybody gets wrong unless they set "idmap
config CORP : range = 500-999999".
5. We use a few static ranges for algorithmic ID translation using the
idmap rid backend. Everything works.
6. We use SSSD, which provides consistent IDs everywhere, and for a
few devices which can't use it, we configured compatible idmap rid
ranges for use with winbindd. The only problem is that we like
user-private groups, and only SSSD has support for them (although we
admit it's our fault that we enabled this non-default option).
7. We store ID mappings in non-AD LDAP and use winbindd with the
"ldap" idmap backend.
I am sure other weird but valid setups exist - please extend the list
if you can.
Which of the above scenarios would be supportable without resorting to
the old way of installing SAMBA manually alongside the cluster?
--
Alexander E. Patrakov
25 Mar
25 Mar
7:57 a.m.
Hi,
On 3/22/24 19:56, Alexander E. Patrakov wrote:
In fact, I am quite skeptical, because, at least in my
experience,
every customer's SAMBA configuration as a domain member is a unique
snowflake, and cephadm would need an ability to specify arbitrary UID
mapping configuration to match what the customer uses elsewhere - and
the match must be precise.
Yes, there has to be a great flexibility possible in the configuration
of the SMB service.
BTW: It would be great of the orchestrator could configure Ganesha to
export NFs shares with Kerberos security, but this is off-topic in this
thread.
Oh, and by the way, we have this strangely
low-numbered group that everybody gets wrong unless they set "idmap
config CORP : range = 500-999999".
This is because Debian changed the standard minimum uid/gid somewhere in
the 2000s. And if you have an "old" company running Debian since before
then you have user IDs and group IDs in the range 500 - 1000.
Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43
Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B
Geschäftsführer: Peer Heinlein - Sitz: Berlin
3:01 p.m.
On Friday, March 22, 2024 2:56:22 PM EDT Alexander E. Patrakov wrote:
Hi John,
In the immediate future the plan is to have a very simple, fairly
"opinionated" idmapping scheme based on the autorid backend.
Sharing the same directories over both NFS and SMB at the same time, also
known as "multi-protocol", is not planned for now, however we're all aware
that there's often a demand for this feature and we're aware of the complexity
it brings. I expect we'll work on that at some point but not initially.
Similarly, sharing the same directories over a SMB share and directly on a
cephfs mount won't be blocked but we won't recommend it.
A few major features we have planned include:
* Standalone servers (internally defined users/groups)
No concerns here
* Active Directory Domain Member Servers
In the second case, what is the plan regarding UID mapping? Is NFS
coexistence planned, or a concurrent mount of the same directory using
CephFS directly?
In fact, I am quite skeptical, because, at least in my experience,
every customer's SAMBA configuration as a domain member is a unique
snowflake, and cephadm would need an ability to specify arbitrary UID
mapping configuration to match what the customer uses elsewhere - and
the match must be precise.
I agree - our initial use case is something along the lines:
Users of a Ceph Cluster that have Windows systems, Mac systems, or appliances
that are joined to an existing AD
but are not currently interoperating with the Ceph cluster.
I expect to add some idpapping configuration and agility down the line,
especially supporting some form of rfc2307 idmapping (where unix IDs are
stored in AD).
But those who already have idmapping schemes and samba accessing ceph will
probably need to just continue using the existing setups as we don't have an
immediate plan for migrating those users.
Here is what I have seen or was told about:
1. We don't care about interoperability with NFS or CephFS, so we just
let SAMBA invent whatever UIDs and GIDs it needs using the "tdb2"
idmap backend. It's completely OK that workstations get different UIDs
and GIDs, as only SIDs traverse the wire.
This is pretty close to our initial plan but I'm not clear why you'd think
that "workstations get different UIDs and GIDs". For all systems acessing the
(same) ceph cluster the id mapping should be consistent.
You did make me consider multi-cluster use cases with something like cephfs
volume mirroring - that's something that I hadn't thought of before *but*
using an algorithmic mapping backend like autorid (and testing) I think we're
mostly OK there.
2. [not seen in the wild, the customer did not
actually implement it,
it's a product of internal miscommunication, and I am not sure if it
is valid at all] We don't care about interoperability with CephFS,
and, while we have NFS, security guys would not allow running NFS
non-kerberized. Therefore, no UIDs or GIDs traverse the wire, only
SIDs and names. Therefore, all we need is to allow both SAMBA and NFS
to use shared UID mapping allocated on as-needed basis using the
"tdb2" idmap module, and it doesn't matter that these UIDs and GIDs
are inconsistent with what clients choose.
Unfortunately, I don't really understand this item. Fortunately, you say it
was only considered not implemented. :-)
3. We don't care about ACLs at all, and don't
care about CephFS
interoperability. We set ownership of all new files to root:root 0666
using whatever options are available [well, I would rather use a
dedicated nobody-style uid/gid here]. All we care about is that only
authorized workstations or authorized users can connect to each NFS or
SMB share, and we absolutely don't want them to be able to set custom
ownership or ACLs.
Some times known as the "drop-box" use case I think (not to be confused with
the cloud app of a similar name).
We could probably implement something like that as an option but I had not
considered it before.
4. We care about NFS and CephFS file ownership being
consistent with
what Windows clients see. We store all UIDs and GIDs in Active
Directory using the rfc2307 schema, and it's mandatory that all
servers (especially SAMBA - thanks to the "ad" idmap backend) respect
that and don't try to invent anything [well, they do - BUILTIN/Users
gets its GID through tdb2]. Oh, and by the way, we have this strangely
low-numbered group that everybody gets wrong unless they set "idmap
config CORP : range = 500-999999".
This is oh so similar to a project I worked on prior to working with Ceph.
I think we'll need to do this one eventually but maybe not this year.
One nice side-effect of running in containers is that the low-id number is less
of an issue because the ids only matter within the container context (and only
then if using the kernel file system access methods). We have much more
flexibility with IDs in a container.
5. We use a few static ranges for algorithmic ID
translation using the
idmap rid backend. Everything works.
See above.
6. We use SSSD, which provides consistent IDs
everywhere, and for a
few devices which can't use it, we configured compatible idmap rid
ranges for use with winbindd. The only problem is that we like
user-private groups, and only SSSD has support for them (although we
admit it's our fault that we enabled this non-default option).
7. We store ID mappings in non-AD LDAP and use winbindd with the
"ldap" idmap backend.
For now, we're only planning to do idmapping with winbind and AD. We'd
probably only consider non-AD ldap and/or ssd if there was strong and loud
demand for it.
I am sure other weird but valid setups exist - please
extend the list
if you can.
Which of the above scenarios would be supportable without resorting to
the old way of installing SAMBA manually alongside the cluster?
I hope I covered the above with some inline replies. This was great food for
thought and at just the right level of technical detail. So thank you very
much for replying, this is exactly the kind of discussion I want to have now
where the design is still young and flexible.
One other cool thing I plan on doing is supporting multiple samba containers
running on the same cluster (even the same node if I can wrangle the network
properly). So one could in fact have completely different domain joins and/or
configurations. While I wouldn't suggest anyone run a whole lot of different
configurations on the same cluster - this idea already allows for some level of
agility between schemes. Later on we might be able to use that as a building
block for migration tools, either from an existing samba setup or between
configurations.
Also, I plan on adding `global_custom_options` and `share_custom_options` for
special overrides for development, qa, and experimentation but those are
strongly within the "you break it, you bought it" realm. But these could be
used for experimenting with idmapping schemes without having them all baked
into the smb mgr module code.
Thanks for the discussion!
--John M.
7:22 p.m.
On Mon, Mar 25, 2024 at 11:01 PM John Mulligan
<phlogistonjohn(a)asynchrono.us> wrote:
Yes, for whatever reason, people do this, even though it is cumbersome
to manage.
So - are you going to use the kernel-based mount or the ceph vfs
module? My tests indicate that, in situations where there are
frequently accessed files, allowing the kernel to cache them in RAM
(which the vfs module does not do) can create a big boost in
performance. Also, SUSE considers the ceph vfs module a
non-recommended solution apparently for the same performance-related
reason, see https://documentation.suse.com/ses/7/html/ses-all/cha-ses-cifs.html
See above.
However, as I said, providing a way to use the "rid" backend with
statically defined domains and ranges in addition to the default
"autorid" backend would be, for me, a good-enough substitute for SSSD.
Multiple SAMBA containers are also good for high availability (with
ctdb) or scale-out (with round-robin DNS).
On Friday, March 22, 2024 2:56:22 PM EDT Alexander E. Patrakov wrote:
OK, the docs for clustered SAMBA do mention the autorid backend in
examples. It's a shame that the manual page does not explicitly list
it as compatible with clustered setups.
However, please consider that the majority of Linux distributions
(tested: CentOS, Fedora, Alt Linux, Ubuntu, OpenSUSE) use "realmd" to
join AD domains by default (where "default" means a pointy-clicky way
in a workstation setup), which uses SSSD, and therefore, by this
opinionated choice of the autorid backend, you create mappings that
disagree with the supposed majority and the default. This will create
problems in the future when you do consider NFS coexistence.
Well, it's a different topic that most organizations that I have seen
seem to ignore this default. Maybe those that don't have any problems
don't have any reason to talk to me? I think that more research is
needed here on whether RedHat's and GNOME's push of SSSD is something
not-ready or indeed the de-facto standard setup.
Even if you don't want to use SSSD, providing an option to provision a
few domains with idmap rid backend with statically configured ranges
(as an override to autorid) would be a good step forward, as this can
be made compatible with the default RedHat setup.
Hi John,
In the immediate future the plan is to have a very simple, fairly
"opinionated" idmapping scheme based on the autorid backend. A few major features we have planned include:
* Standalone servers (internally defined users/groups)
No concerns here
* Active Directory Domain Member Servers
In the second case, what is the plan regarding UID mapping? Is NFS
coexistence planned, or a concurrent mount of the same directory using
CephFS directly? Sharing the same directories over both NFS and SMB at
the same time, also
known as "multi-protocol", is not planned for now, however we're all aware
that there's often a demand for this feature and we're aware of the complexity
it brings. I expect we'll work on that at some point but not initially.
Similarly, sharing the same directories over a SMB share and directly on a
cephfs mount won't be blocked but we won't recommend it.
OK. Feature request: in the case if there are several CephFS
filesystems, support configuration of which one to serve.
In fact, I am quite skeptical, because, at least in my experience,
every customer's SAMBA configuration as a domain member is a unique
snowflake, and cephadm would need an ability to specify arbitrary UID
mapping configuration to match what the customer uses elsewhere - and
the match must be precise.
I agree - our initial use case is something along the lines:
Users of a Ceph Cluster that have Windows systems, Mac systems, or appliances
that are joined to an existing AD
but are not currently interoperating with the Ceph cluster.
I expect to add some idpapping configuration and agility down the line,
especially supporting some form of rfc2307 idmapping (where unix IDs are
stored in AD).
But those who already have idmapping schemes and samba accessing ceph will
probably need to just continue using the existing setups as we don't have an
immediate plan for migrating those users.
The tdb2 backend (used in my example) is not algorithmic, it is
allocating. That is, it sequentially allocates IDs on the
first-seen-first-allocated basis. Yet this is what this customer uses,
presumably because it is the only backend that explicitly specifies
clustering operation in its manual page.
And the "autorid" backend is also not fully algorithmic, it allocates
ranges to domains on the same sequential basis (see
https://github.com/samba-team/samba/blob/6fb98f70c6274e172787c8d5f73aa93920…),
and therefore can create mismatching mappings if two workstations or
servers have seen the users DOMA\usera and DOMB\userb in a different
order. It is even mentioned in the manual page. SSSD largely avoids
this problem by hashing the domain portion of the SID instead of
allocating the subranges on a sequential basis.
Here is what I have seen or was told about:
1. We don't care about interoperability with NFS or CephFS, so we just
let SAMBA invent whatever UIDs and GIDs it needs using the "tdb2"
idmap backend. It's completely OK that workstations get different UIDs
and GIDs, as only SIDs traverse the wire.
This is pretty close to our initial plan but I'm not clear why you'd think
that "workstations get different UIDs and GIDs". For all systems acessing the
(same) ceph cluster the id mapping should be consistent.
You did make me consider multi-cluster use cases with something like cephfs
volume mirroring - that's something that I hadn't thought of before *but*
using an algorithmic mapping backend like autorid (and testing) I think we're
mostly OK there. 2. [not seen in the wild, the customer did not
actually implement it,
it's a product of internal miscommunication, and I am not sure if it
is valid at all] We don't care about interoperability with CephFS,
and, while we have NFS, security guys would not allow running NFS
non-kerberized. Therefore, no UIDs or GIDs traverse the wire, only
SIDs and names. Therefore, all we need is to allow both SAMBA and NFS
to use shared UID mapping allocated on as-needed basis using the
"tdb2" idmap module, and it doesn't matter that these UIDs and GIDs
are inconsistent with what clients choose.
Unfortunately, I don't really understand this item. Fortunately, you say it
was only considered not implemented. :-)
3. We don't care about ACLs at all, and
don't care about CephFS
interoperability. We set ownership of all new files to root:root 0666
using whatever options are available [well, I would rather use a
dedicated nobody-style uid/gid here]. All we care about is that only
authorized workstations or authorized users can connect to each NFS or
SMB share, and we absolutely don't want them to be able to set custom
ownership or ACLs.
Some times known as the "drop-box" use case I think (not to be confused with
the cloud app of a similar name).
We could probably implement something like that as an option but I had not
considered it before.
4. We care about NFS and CephFS file ownership
being consistent with
what Windows clients see. We store all UIDs and GIDs in Active
Directory using the rfc2307 schema, and it's mandatory that all
servers (especially SAMBA - thanks to the "ad" idmap backend) respect
that and don't try to invent anything [well, they do - BUILTIN/Users
gets its GID through tdb2]. Oh, and by the way, we have this strangely
low-numbered group that everybody gets wrong unless they set "idmap
config CORP : range = 500-999999".
This is oh so similar to a project I worked on prior to working with Ceph.
I think we'll need to do this one eventually but maybe not this year.
One nice side-effect of running in containers is that the low-id number is less
of an issue because the ids only matter within the container context (and only
then if using the kernel file system access methods). We have much more
flexibility with IDs in a container. 5. We use a few static ranges for algorithmic ID
translation using the
idmap rid backend. Everything works.
See above.
6. We use SSSD, which provides consistent IDs
everywhere, and for a
few devices which can't use it, we configured compatible idmap rid
ranges for use with winbindd. The only problem is that we like
user-private groups, and only SSSD has support for them (although we
admit it's our fault that we enabled this non-default option).
7. We store ID mappings in non-AD LDAP and use winbindd with the
"ldap" idmap backend.
For now, we're only planning to do idmapping with winbind and AD. We'd
probably only consider non-AD ldap and/or ssd if there was strong and loud
demand for it. I am sure other weird but valid setups exist -
please extend the list
if you can.
Which of the above scenarios would be supportable without resorting to
the old way of installing SAMBA manually alongside the cluster?
I hope I covered the above with some inline replies. This was great food for
thought and at just the right level of technical detail. So thank you very
much for replying, this is exactly the kind of discussion I want to have now
where the design is still young and flexible.
One other cool thing I plan on doing is supporting multiple samba containers
running on the same cluster (even the same node if I can wrangle the network
properly). So one could in fact have completely different domain joins and/or
configurations. While I wouldn't suggest anyone run a whole lot of different
configurations on the same cluster - this idea already allows for some level of
agility between schemes. Later on we might be able to use that as a building
block for migration tools, either from an existing samba setup or between
configurations. Also, I plan on adding `global_custom_options` and
`share_custom_options` for
special overrides for development, qa, and experimentation but those are
strongly within the "you break it, you bought it" realm. But these could be
used for experimenting with idmapping schemes without having them all baked
into the smb mgr module code.
Great, thanks!
--
Alexander E. Patrakov
7:40 p.m.
On Monday, March 25, 2024 3:22:26 PM EDT Alexander E. Patrakov wrote:
Putting it on the list.
Yes, for whatever reason, people do this, even though it is cumbersome
to manage.
Agreed. Thanks for the reminder. This will certainly need to go on the test
plan.
So - are you going to use the kernel-based mount or the ceph vfs
module? My tests indicate that, in situations where there are
frequently accessed files, allowing the kernel to cache them in RAM
(which the vfs module does not do) can create a big boost in
performance. Also, SUSE considers the ceph vfs module a
non-recommended solution apparently for the same performance-related
reason, see
https://documentation.suse.com/ses/7/html/ses-all/cha-ses-cifs.html
The prototype module only uses the vfs module due to the extreme simplicity of
setting it up in containers. Otherwise, we're trying to keep our options open
and are investigating multiple approaches currently.
See above.
However, as I said, providing a way to use the "rid" backend with
statically defined domains and ranges in addition to the default
"autorid" backend would be, for me, a good-enough substitute for SSSD.
Sounds reasonable. I've done it that way in a prior role too, so it's somewhat
familiar. Thanks!
Multiple SAMBA containers are also good for high availability (with
ctdb) or scale-out (with round-robin DNS).
Once again, thanks for the feedback. This discussion is very welcome!
On Mon, Mar 25, 2024 at 11:01 PM John Mulligan
<phlogistonjohn(a)asynchrono.us> wrote:
Thanks, I'll keep that in mind.
On Friday, March 22, 2024 2:56:22 PM EDT
Alexander E. Patrakov wrote:
OK, the docs for clustered SAMBA do mention the autorid backend in
examples. It's a shame that the manual page does not explicitly list
it as compatible with clustered setups.
However, please consider that the majority of Linux distributions
(tested: CentOS, Fedora, Alt Linux, Ubuntu, OpenSUSE) use "realmd" to
join AD domains by default (where "default" means a pointy-clicky way
in a workstation setup), which uses SSSD, and therefore, by this
opinionated choice of the autorid backend, you create mappings that
disagree with the supposed majority and the default. This will create
problems in the future when you do consider NFS coexistence.
Hi John,
In the immediate future the plan is to have a very simple, fairly
"opinionated" idmapping scheme based on the autorid backend. A few major features we have planned include:
* Standalone servers (internally defined users/groups)
No concerns here
* Active Directory Domain Member Servers
In the second case, what is the plan regarding UID mapping? Is NFS
coexistence planned, or a concurrent mount of the same directory using
CephFS directly? Well, it's a different topic that most
organizations that I have seen
seem to ignore this default. Maybe those that don't have any problems
don't have any reason to talk to me? I think that more research is
needed here on whether RedHat's and GNOME's push of SSSD is something
not-ready or indeed the de-facto standard setup.
I think it's a bit of a mix, but am not sure either.
Even if you don't want to use SSSD, providing an
option to provision a
few domains with idmap rid backend with statically configured ranges
(as an override to autorid) would be a good step forward, as this can
be made compatible with the default RedHat setup.
That's reasonable. Thanks for the suggestion.
Sharing the same directories over both NFS and
SMB at the same time, also
known as "multi-protocol", is not planned for now, however we're all aware
that there's often a demand for this feature and we're aware of the
complexity it brings. I expect we'll work on that at some point but not
initially. Similarly, sharing the same directories over a SMB share and
directly on a cephfs mount won't be blocked but we won't recommend it.
OK. Feature request: in the case if there are several CephFS
filesystems, support configuration of which one to serve.
In fact, I am quite skeptical, because, at least in my
experience,
every customer's SAMBA configuration as a domain member is a unique
snowflake, and cephadm would need an ability to specify arbitrary UID
mapping configuration to match what the customer uses elsewhere - and
the match must be precise.
I agree - our initial use case is something along the lines:
Users of a Ceph Cluster that have Windows systems, Mac systems, or
appliances that are joined to an existing AD
but are not currently interoperating with the Ceph cluster.
I expect to add some idpapping configuration and agility down the line,
especially supporting some form of rfc2307 idmapping (where unix IDs are
stored in AD). But those who already have idmapping schemes and
samba accessing ceph will
probably need to just continue using the existing setups as we don't have
an immediate plan for migrating those users.
The tdb2 backend (used in my example) is not algorithmic, it is
allocating. That is, it sequentially allocates IDs on the
first-seen-first-allocated basis. Yet this is what this customer uses,
presumably because it is the only backend that explicitly specifies
clustering operation in its manual page.
And the "autorid" backend is also not fully algorithmic, it allocates
ranges to domains on the same sequential basis (see
https://github.com/samba-team/samba/blob/6fb98f70c6274e172787c8d5f73aa939201
71e7c/source3/winbindd/idmap_autorid_tdb.c#L82), and therefore can create
mismatching mappings if two workstations or servers have seen the users
DOMA\usera and DOMB\userb in a different order. It is even mentioned in the
manual page. SSSD largely avoids this problem by hashing the domain portion
of the SID instead of
allocating the subranges on a sequential basis.
Here is what I have seen or was told about:
1. We don't care about interoperability with NFS or CephFS, so we just
let SAMBA invent whatever UIDs and GIDs it needs using the "tdb2"
idmap backend. It's completely OK that workstations get different UIDs
and GIDs, as only SIDs traverse the wire.
This is pretty close to our initial plan but I'm not clear why you'd think
that "workstations get different UIDs and GIDs". For all systems acessing
the (same) ceph cluster the id mapping should be consistent.
You did make me consider multi-cluster use cases with something like
cephfs
volume mirroring - that's something that I hadn't thought of before *but*
using an algorithmic mapping backend like autorid (and testing) I think
we're mostly OK there. 2. [not seen in the wild, the customer did not
actually implement it,
it's a product of internal miscommunication, and I am not sure if it
is valid at all] We don't care about interoperability with CephFS,
and, while we have NFS, security guys would not allow running NFS
non-kerberized. Therefore, no UIDs or GIDs traverse the wire, only
SIDs and names. Therefore, all we need is to allow both SAMBA and NFS
to use shared UID mapping allocated on as-needed basis using the
"tdb2" idmap module, and it doesn't matter that these UIDs and GIDs
are inconsistent with what clients choose.
Unfortunately, I don't really understand this item. Fortunately, you say
it
was only considered not implemented. :-)
3. We don't care about ACLs at all, and
don't care about CephFS
interoperability. We set ownership of all new files to root:root 0666
using whatever options are available [well, I would rather use a
dedicated nobody-style uid/gid here]. All we care about is that only
authorized workstations or authorized users can connect to each NFS or
SMB share, and we absolutely don't want them to be able to set custom
ownership or ACLs.
Some times known as the "drop-box" use case I think (not to be confused
with the cloud app of a similar name).
We could probably implement something like that as an option but I had not
considered it before.
4. We care about NFS and CephFS file ownership
being consistent with
what Windows clients see. We store all UIDs and GIDs in Active
Directory using the rfc2307 schema, and it's mandatory that all
servers (especially SAMBA - thanks to the "ad" idmap backend) respect
that and don't try to invent anything [well, they do - BUILTIN/Users
gets its GID through tdb2]. Oh, and by the way, we have this strangely
low-numbered group that everybody gets wrong unless they set "idmap
config CORP : range = 500-999999".
This is oh so similar to a project I worked on prior to working with Ceph.
I think we'll need to do this one eventually but maybe not this year.
One nice side-effect of running in containers is that the low-id number is
less of an issue because the ids only matter within the container context
(and only then if using the kernel file system access methods). We have
much more flexibility with IDs in a container. 5. We use a few static ranges for algorithmic ID
translation using the
idmap rid backend. Everything works.
See above.
6. We use SSSD, which provides consistent IDs
everywhere, and for a
few devices which can't use it, we configured compatible idmap rid
ranges for use with winbindd. The only problem is that we like
user-private groups, and only SSSD has support for them (although we
admit it's our fault that we enabled this non-default option).
7. We store ID mappings in non-AD LDAP and use winbindd with the
"ldap" idmap backend.
For now, we're only planning to do idmapping with winbind and AD. We'd
probably only consider non-AD ldap and/or ssd if there was strong and loud
demand for it. I am sure other weird but valid setups exist - please
extend the list
if you can.
Which of the above scenarios would be supportable without resorting to
the old way of installing SAMBA manually alongside the cluster?
I hope I covered the above with some inline replies. This was great food
for thought and at just the right level of technical detail. So thank you
very much for replying, this is exactly the kind of discussion I want to
have now where the design is still young and flexible.
One other cool thing I plan on doing is supporting multiple samba
containers running on the same cluster (even the same node if I can
wrangle the network properly). So one could in fact have completely
different domain joins and/or configurations. While I wouldn't suggest
anyone run a whole lot of different configurations on the same cluster -
this idea already allows for some level of agility between schemes. Later
on we might be able to use that as a building block for migration tools,
either from an existing samba setup or between configurations. Also, I plan on adding `global_custom_options`
and `share_custom_options`
for special overrides for development, qa, and experimentation but those
are strongly within the "you break it, you bought it" realm. But these
could be used for experimenting with idmapping schemes without having
them all baked into the smb mgr module code.
Great, thanks! 
27 Mar
27 Mar
2:53 a.m.
This is great, we are currently using the smb protocol heavily to
export kernel-mounted cephfs.
But I encountered a problem. When there are many smb clients
enumerating or listing the same directory, the smb server will
experience high load, and the smb process will become D state.
This problem has been going on for some time and no suitable solution
has been found yet.
John Mulligan <phlogistonjohn(a)asynchrono.us> 于2024年3月26日周二 03:43写道：
>
> On Monday, March 25, 2024 3:22:26 PM EDT Alexander E. Patrakov wrote:
> > On Mon, Mar 25, 2024 at 11:01 PM John Mulligan
> >
> > <phlogistonjohn(a)asynchrono.us> wrote:
> > > On Friday, March 22, 2024 2:56:22 PM EDT Alexander E. Patrakov wrote:
> > > > Hi John,
> > > >
> > > > > A few major features we have planned include:
> > > > > * Standalone servers (internally defined users/groups)
> > > >
> > > > No concerns here
> > > >
> > > > > * Active Directory Domain Member Servers
> > > >
> > > > In the second case, what is the plan regarding UID mapping? Is NFS
> > > > coexistence planned, or a concurrent mount of the same directory
using
> > > > CephFS directly?
> > >
> > > In the immediate future the plan is to have a very simple, fairly
> > > "opinionated" idmapping scheme based on the autorid backend.
> >
> > OK, the docs for clustered SAMBA do mention the autorid backend in
> > examples. It's a shame that the manual page does not explicitly list
> > it as compatible with clustered setups.
> >
> > However, please consider that the majority of Linux distributions
> > (tested: CentOS, Fedora, Alt Linux, Ubuntu, OpenSUSE) use "realmd" to
> > join AD domains by default (where "default" means a pointy-clicky way
> > in a workstation setup), which uses SSSD, and therefore, by this
> > opinionated choice of the autorid backend, you create mappings that
> > disagree with the supposed majority and the default. This will create
> > problems in the future when you do consider NFS coexistence.
> >
>
> Thanks, I'll keep that in mind.
>
> > Well, it's a different topic that most organizations that I have seen
> > seem to ignore this default. Maybe those that don't have any problems
> > don't have any reason to talk to me? I think that more research is
> > needed here on whether RedHat's and GNOME's push of SSSD is something
> > not-ready or indeed the de-facto standard setup.
> >
>
> I think it's a bit of a mix, but am not sure either.
>
>
> > Even if you don't want to use SSSD, providing an option to provision a
> > few domains with idmap rid backend with statically configured ranges
> > (as an override to autorid) would be a good step forward, as this can
> > be made compatible with the default RedHat setup.
>
> That's reasonable. Thanks for the suggestion.
>
>
> >
> > > Sharing the same directories over both NFS and SMB at the same time, also
> > > known as "multi-protocol", is not planned for now, however
we're all aware
> > > that there's often a demand for this feature and we're aware of
the
> > > complexity it brings. I expect we'll work on that at some point but
not
> > > initially. Similarly, sharing the same directories over a SMB share and
> > > directly on a cephfs mount won't be blocked but we won't recommend
it.
> >
> > OK. Feature request: in the case if there are several CephFS
> > filesystems, support configuration of which one to serve.
> >
>
> Putting it on the list.
>
> > > > In fact, I am quite skeptical, because, at least in my experience,
> > > > every customer's SAMBA configuration as a domain member is a
unique
> > > > snowflake, and cephadm would need an ability to specify arbitrary UID
> > > > mapping configuration to match what the customer uses elsewhere - and
> > > > the match must be precise.
> > >
> > > I agree - our initial use case is something along the lines:
> > > Users of a Ceph Cluster that have Windows systems, Mac systems, or
> > > appliances that are joined to an existing AD
> > > but are not currently interoperating with the Ceph cluster.
> > >
> > > I expect to add some idpapping configuration and agility down the line,
> > > especially supporting some form of rfc2307 idmapping (where unix IDs are
> > > stored in AD).
> >
> > Yes, for whatever reason, people do this, even though it is cumbersome
> > to manage.
> >
> > > But those who already have idmapping schemes and samba accessing ceph will
> > > probably need to just continue using the existing setups as we don't
have
> > > an immediate plan for migrating those users.
> > >
> > > > Here is what I have seen or was told about:
> > > >
> > > > 1. We don't care about interoperability with NFS or CephFS, so we
just
> > > > let SAMBA invent whatever UIDs and GIDs it needs using the
"tdb2"
> > > > idmap backend. It's completely OK that workstations get different
UIDs
> > > > and GIDs, as only SIDs traverse the wire.
> > >
> > > This is pretty close to our initial plan but I'm not clear why
you'd think
> > > that "workstations get different UIDs and GIDs". For all systems
acessing
> > > the (same) ceph cluster the id mapping should be consistent.
> > > You did make me consider multi-cluster use cases with something like
> > > cephfs
> > > volume mirroring - that's something that I hadn't thought of before
*but*
> > > using an algorithmic mapping backend like autorid (and testing) I think
> > > we're mostly OK there.
> >
> > The tdb2 backend (used in my example) is not algorithmic, it is
> > allocating. That is, it sequentially allocates IDs on the
> > first-seen-first-allocated basis. Yet this is what this customer uses,
> > presumably because it is the only backend that explicitly specifies
> > clustering operation in its manual page.
> >
> > And the "autorid" backend is also not fully algorithmic, it allocates
> > ranges to domains on the same sequential basis (see
> > https://github.com/samba-team/samba/blob/6fb98f70c6274e172787c8d5f73aa939201
> > 71e7c/source3/winbindd/idmap_autorid_tdb.c#L82), and therefore can create
> > mismatching mappings if two workstations or servers have seen the users
> > DOMA\usera and DOMB\userb in a different order. It is even mentioned in the
> > manual page. SSSD largely avoids this problem by hashing the domain portion
> > of the SID instead of
> > allocating the subranges on a sequential basis.
> >
>
> Agreed. Thanks for the reminder. This will certainly need to go on the test
> plan.
>
> > > > 2. [not seen in the wild, the customer did not actually implement it,
> > > > it's a product of internal miscommunication, and I am not sure if
it
> > > > is valid at all] We don't care about interoperability with
CephFS,
> > > > and, while we have NFS, security guys would not allow running NFS
> > > > non-kerberized. Therefore, no UIDs or GIDs traverse the wire, only
> > > > SIDs and names. Therefore, all we need is to allow both SAMBA and NFS
> > > > to use shared UID mapping allocated on as-needed basis using the
> > > > "tdb2" idmap module, and it doesn't matter that these
UIDs and GIDs
> > > > are inconsistent with what clients choose.
> > >
> > > Unfortunately, I don't really understand this item. Fortunately, you
say
> > > it
> > > was only considered not implemented. :-)
> > >
> > > > 3. We don't care about ACLs at all, and don't care about
CephFS
> > > > interoperability. We set ownership of all new files to root:root 0666
> > > > using whatever options are available [well, I would rather use a
> > > > dedicated nobody-style uid/gid here]. All we care about is that only
> > > > authorized workstations or authorized users can connect to each NFS
or
> > > > SMB share, and we absolutely don't want them to be able to set
custom
> > > > ownership or ACLs.
> > >
> > > Some times known as the "drop-box" use case I think (not to be
confused
> > > with the cloud app of a similar name).
> > > We could probably implement something like that as an option but I had not
> > > considered it before.
> > >
> > > > 4. We care about NFS and CephFS file ownership being consistent with
> > > > what Windows clients see. We store all UIDs and GIDs in Active
> > > > Directory using the rfc2307 schema, and it's mandatory that all
> > > > servers (especially SAMBA - thanks to the "ad" idmap
backend) respect
> > > > that and don't try to invent anything [well, they do -
BUILTIN/Users
> > > > gets its GID through tdb2]. Oh, and by the way, we have this
strangely
> > > > low-numbered group that everybody gets wrong unless they set
"idmap
> > > > config CORP : range = 500-999999".
> > >
> > > This is oh so similar to a project I worked on prior to working with Ceph.
> > > I think we'll need to do this one eventually but maybe not this year.
> > > One nice side-effect of running in containers is that the low-id number is
> > > less of an issue because the ids only matter within the container context
> > > (and only then if using the kernel file system access methods). We have
> > > much more flexibility with IDs in a container.
> >
> > So - are you going to use the kernel-based mount or the ceph vfs
> > module? My tests indicate that, in situations where there are
> > frequently accessed files, allowing the kernel to cache them in RAM
> > (which the vfs module does not do) can create a big boost in
> > performance. Also, SUSE considers the ceph vfs module a
> > non-recommended solution apparently for the same performance-related
> > reason, see
> > https://documentation.suse.com/ses/7/html/ses-all/cha-ses-cifs.html
>
>
> The prototype module only uses the vfs module due to the extreme simplicity of
> setting it up in containers. Otherwise, we're trying to keep our options open
> and are investigating multiple approaches currently.
>
> > > > 5. We use a few static ranges for algorithmic ID translation using
the
> > > > idmap rid backend. Everything works.
> > >
> > > See above.
> > >
> > > > 6. We use SSSD, which provides consistent IDs everywhere, and for a
> > > > few devices which can't use it, we configured compatible idmap
rid
> > > > ranges for use with winbindd. The only problem is that we like
> > > > user-private groups, and only SSSD has support for them (although we
> > > > admit it's our fault that we enabled this non-default option).
> > > > 7. We store ID mappings in non-AD LDAP and use winbindd with the
> > > > "ldap" idmap backend.
> > >
> > > For now, we're only planning to do idmapping with winbind and AD.
We'd
> > > probably only consider non-AD ldap and/or ssd if there was strong and loud
> > > demand for it.
> >
> > See above.
> >
> > However, as I said, providing a way to use the "rid" backend with
> > statically defined domains and ranges in addition to the default
> > "autorid" backend would be, for me, a good-enough substitute for
SSSD.
> >
>
> Sounds reasonable. I've done it that way in a prior role too, so it's
somewhat
> familiar. Thanks!
>
> > > > I am sure other weird but valid setups exist - please extend the list
> > > > if you can.
> > > >
> > > > Which of the above scenarios would be supportable without resorting
to
> > > > the old way of installing SAMBA manually alongside the cluster?
> > >
> > > I hope I covered the above with some inline replies. This was great food
> > > for thought and at just the right level of technical detail. So thank you
> > > very much for replying, this is exactly the kind of discussion I want to
> > > have now where the design is still young and flexible.
> > >
> > > One other cool thing I plan on doing is supporting multiple samba
> > > containers running on the same cluster (even the same node if I can
> > > wrangle the network properly). So one could in fact have completely
> > > different domain joins and/or configurations. While I wouldn't suggest
> > > anyone run a whole lot of different configurations on the same cluster -
> > > this idea already allows for some level of agility between schemes. Later
> > > on we might be able to use that as a building block for migration tools,
> > > either from an existing samba setup or between configurations.
> >
> > Multiple SAMBA containers are also good for high availability (with
> > ctdb) or scale-out (with round-robin DNS).
> >
> > > Also, I plan on adding `global_custom_options` and `share_custom_options`
> > > for special overrides for development, qa, and experimentation but those
> > > are strongly within the "you break it, you bought it" realm. But
these
> > > could be used for experimenting with idmapping schemes without having
> > > them all baked into the smb mgr module code.
> >
> > Great, thanks!
>
> Once again, thanks for the feedback. This discussion is very welcome!
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:02 p.m.
On Tuesday, March 26, 2024 10:53:29 PM EDT David Yang wrote:
Once again, thanks for the feedback. This discussion is very welcome!
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
This is great, we are currently using the smb protocol
heavily to
export kernel-mounted cephfs.
But I encountered a problem. When there are many smb clients
enumerating or listing the same directory, the smb server will
experience high load, and the smb process will become D state.
This problem has been going on for some time and no suitable solution
has been found yet.
Thanks for the heads up. I'll make sure concurrent dir access is part of the
test plan.
John Mulligan <phlogistonjohn(a)asynchrono.us>
于2024年3月26日周二 03:43写道：
>
>
> On Monday, March 25, 2024 3:22:26 PM EDT Alexander E. Patrakov wrote:
>
> > On Mon, Mar 25, 2024 at 11:01 PM John Mulligan
> >
> >
> >
> > <phlogistonjohn(a)asynchrono.us> wrote:
> >
> > > On Friday, March 22, 2024 2:56:22 PM EDT Alexander E. Patrakov wrote:
> > >
> > > > Hi John,
> > > >
> > > >
> > > >
> > > > > A few major features we have planned include:
> > > > > * Standalone servers (internally defined users/groups)
> > > >
> > > >
> > > >
> > > > No concerns here
> > > >
> > > >
> > > >
> > > > > * Active Directory Domain Member Servers
> > > >
> > > >
> > > >
> > > > In the second case, what is the plan regarding UID mapping? Is NFS
> > > > coexistence planned, or a concurrent mount of the same directory
> > > > using
> > > > CephFS directly?
> > >
> > >
> > >
> > > In the immediate future the plan is to have a very simple, fairly
> > > "opinionated" idmapping scheme based on the autorid backend.
> >
> >
> >
> > OK, the docs for clustered SAMBA do mention the autorid backend in
> > examples. It's a shame that the manual page does not explicitly list
> > it as compatible with clustered setups.
> >
> >
> >
> > However, please consider that the majority of Linux distributions
> > (tested: CentOS, Fedora, Alt Linux, Ubuntu, OpenSUSE) use "realmd"
to
> > join AD domains by default (where "default" means a pointy-clicky
way
> > in a workstation setup), which uses SSSD, and therefore, by this
> > opinionated choice of the autorid backend, you create mappings that
> > disagree with the supposed majority and the default. This will create
> > problems in the future when you do consider NFS coexistence.
> >
> >
>
>
>
> Thanks, I'll keep that in mind.
>
>
>
> > Well, it's a different topic that most organizations that I have seen
> > seem to ignore this default. Maybe those that don't have any problems
> > don't have any reason to talk to me? I think that more research is
> > needed here on whether RedHat's and GNOME's push of SSSD is something
> > not-ready or indeed the de-facto standard setup.
> >
> >
>
>
>
> I think it's a bit of a mix, but am not sure either.
>
>
>
>
> > Even if you don't want to use SSSD, providing an option to provision a
> > few domains with idmap rid backend with statically configured ranges
> > (as an override to autorid) would be a good step forward, as this can
> > be made compatible with the default RedHat setup.
>
>
>
> That's reasonable. Thanks for the suggestion.
>
>
>
>
> >
> >
> > > Sharing the same directories over both NFS and SMB at the same time,
> > > also
> > > known as "multi-protocol", is not planned for now, however
we're all
> > > aware
> > > that there's often a demand for this feature and we're aware of
the
> > > complexity it brings. I expect we'll work on that at some point but
> > > not
> > > initially. Similarly, sharing the same directories over a SMB share
> > > and
> > > directly on a cephfs mount won't be blocked but we won't
recommend
> > > it.
> >
> >
> >
> > OK. Feature request: in the case if there are several CephFS
> > filesystems, support configuration of which one to serve.
> >
> >
>
>
>
> Putting it on the list.
>
>
>
> > > > In fact, I am quite skeptical, because, at least in my experience,
> > > > every customer's SAMBA configuration as a domain member is a
unique
> > > > snowflake, and cephadm would need an ability to specify arbitrary
> > > > UID
> > > > mapping configuration to match what the customer uses elsewhere -
> > > > and
> > > > the match must be precise.
> > >
> > >
> > >
> > > I agree - our initial use case is something along the lines:
> > > Users of a Ceph Cluster that have Windows systems, Mac systems, or
> > > appliances that are joined to an existing AD
> > > but are not currently interoperating with the Ceph cluster.
> > >
> > >
> > >
> > > I expect to add some idpapping configuration and agility down the
> > > line,
> > > especially supporting some form of rfc2307 idmapping (where unix IDs
> > > are
> > > stored in AD).
> >
> >
> >
> > Yes, for whatever reason, people do this, even though it is cumbersome
> > to manage.
> >
> >
> >
> > > But those who already have idmapping schemes and samba accessing ceph
> > > will
> > > probably need to just continue using the existing setups as we don't
> > > have
> > > an immediate plan for migrating those users.
> > >
> > >
> > >
> > > > Here is what I have seen or was told about:
> > > >
> > > >
> > > >
> > > > 1. We don't care about interoperability with NFS or CephFS, so
we
> > > > just
> > > > let SAMBA invent whatever UIDs and GIDs it needs using the
"tdb2"
> > > > idmap backend. It's completely OK that workstations get
different
> > > > UIDs
> > > > and GIDs, as only SIDs traverse the wire.
> > >
> > >
> > >
> > > This is pretty close to our initial plan but I'm not clear why
you'd
> > > think
> > > that "workstations get different UIDs and GIDs". For all
systems
> > > acessing
> > > the (same) ceph cluster the id mapping should be consistent.
> > > You did make me consider multi-cluster use cases with something like
> > > cephfs
> > > volume mirroring - that's something that I hadn't thought of
before
> > > *but*
> > > using an algorithmic mapping backend like autorid (and testing) I
> > > think
> > > we're mostly OK there.
> >
> >
> >
> > The tdb2 backend (used in my example) is not algorithmic, it is
> > allocating. That is, it sequentially allocates IDs on the
> > first-seen-first-allocated basis. Yet this is what this customer uses,
> > presumably because it is the only backend that explicitly specifies
> > clustering operation in its manual page.
> >
> >
> >
> > And the "autorid" backend is also not fully algorithmic, it
allocates
> > ranges to domains on the same sequential basis (see
> > https://github.com/samba-team/samba/blob/6fb98f70c6274e172787c8d5f73aa93
> > 9201
71e7c/source3/winbindd/idmap_autorid_tdb.c#L82), and therefore can
> > create mismatching mappings if two
workstations or servers have seen
> > the users DOMA\usera and DOMB\userb in a different order. It is even
> > mentioned in the manual page. SSSD largely avoids this problem by
> > hashing the domain portion of the SID instead of
> > allocating the subranges on a sequential basis.
> >
> >
>
>
>
> Agreed. Thanks for the reminder. This will certainly need to go on the
> test
plan.
>
>
>
> > > > 2. [not seen in the wild, the customer did not actually implement
> > > > it,
> > > > it's a product of internal miscommunication, and I am not sure
if
> > > > it
> > > > is valid at all] We don't care about interoperability with
CephFS,
> > > > and, while we have NFS, security guys would not allow running NFS
> > > > non-kerberized. Therefore, no UIDs or GIDs traverse the wire, only
> > > > SIDs and names. Therefore, all we need is to allow both SAMBA and
> > > > NFS
> > > > to use shared UID mapping allocated on as-needed basis using the
> > > > "tdb2" idmap module, and it doesn't matter that these
UIDs and GIDs
> > > > are inconsistent with what clients choose.
> > >
> > >
> > >
> > > Unfortunately, I don't really understand this item. Fortunately, you
> > > say
> > > it
> > > was only considered not implemented. :-)
> > >
> > >
> > >
> > > > 3. We don't care about ACLs at all, and don't care about
CephFS
> > > > interoperability. We set ownership of all new files to root:root
> > > > 0666
> > > > using whatever options are available [well, I would rather use a
> > > > dedicated nobody-style uid/gid here]. All we care about is that
> > > > only
> > > > authorized workstations or authorized users can connect to each NFS
> > > > or
> > > > SMB share, and we absolutely don't want them to be able to set
> > > > custom
> > > > ownership or ACLs.
> > >
> > >
> > >
> > > Some times known as the "drop-box" use case I think (not to be
> > > confused
> > > with the cloud app of a similar name).
> > > We could probably implement something like that as an option but I had
> > > not
> > > considered it before.
> > >
> > >
> > >
> > > > 4. We care about NFS and CephFS file ownership being consistent
> > > > with
> > > > what Windows clients see. We store all UIDs and GIDs in Active
> > > > Directory using the rfc2307 schema, and it's mandatory that all
> > > > servers (especially SAMBA - thanks to the "ad" idmap
backend)
> > > > respect
> > > > that and don't try to invent anything [well, they do -
> > > > BUILTIN/Users
> > > > gets its GID through tdb2]. Oh, and by the way, we have this
> > > > strangely
> > > > low-numbered group that everybody gets wrong unless they set
"idmap
> > > > config CORP : range = 500-999999".
> > >
> > >
> > >
> > > This is oh so similar to a project I worked on prior to working with
> > > Ceph.
> > > I think we'll need to do this one eventually but maybe not this year.
> > > One nice side-effect of running in containers is that the low-id
> > > number is
> > > less of an issue because the ids only matter within the container
> > > context
> > > (and only then if using the kernel file system access methods). We
> > > have
> > > much more flexibility with IDs in a container.
> >
> >
> >
> > So - are you going to use the kernel-based mount or the ceph vfs
> > module? My tests indicate that, in situations where there are
> > frequently accessed files, allowing the kernel to cache them in RAM
> > (which the vfs module does not do) can create a big boost in
> > performance. Also, SUSE considers the ceph vfs module a
> > non-recommended solution apparently for the same performance-related
> > reason, see
> > https://documentation.suse.com/ses/7/html/ses-all/cha-ses-cifs.html
>
>
>
>
> The prototype module only uses the vfs module due to the extreme
> simplicity of
setting it up in containers. Otherwise, we're trying to
> keep our options open and are investigating
multiple approaches
> currently.
>
>
>
> > > > 5. We use a few static ranges for algorithmic ID translation using
> > > > the
> > > > idmap rid backend. Everything works.
> > >
> > >
> > >
> > > See above.
> > >
> > >
> > >
> > > > 6. We use SSSD, which provides consistent IDs everywhere, and for a
> > > > few devices which can't use it, we configured compatible idmap
rid
> > > > ranges for use with winbindd. The only problem is that we like
> > > > user-private groups, and only SSSD has support for them (although
> > > > we
> > > > admit it's our fault that we enabled this non-default option).
> > > > 7. We store ID mappings in non-AD LDAP and use winbindd with the
> > > > "ldap" idmap backend.
> > >
> > >
> > >
> > > For now, we're only planning to do idmapping with winbind and AD.
> > > We'd
> > > probably only consider non-AD ldap and/or ssd if there was strong and
> > > loud
> > > demand for it.
> >
> >
> >
> > See above.
> >
> >
> >
> > However, as I said, providing a way to use the "rid" backend with
> > statically defined domains and ranges in addition to the default
> > "autorid" backend would be, for me, a good-enough substitute for
SSSD.
> >
> >
>
>
>
> Sounds reasonable. I've done it that way in a prior role too, so it's
> somewhat
familiar. Thanks!
> I am
sure other weird but valid setups exist - please extend the
> list
> if you can.
>
>
>
> Which of the above scenarios would be supportable without resorting
> to
> the old way of installing SAMBA manually alongside the cluster?
I hope I covered the above with some inline replies. This was great
food
for thought and at just the right level of technical detail. So thank
you
very much for replying, this is exactly the kind of discussion I want
to
have now where the design is still young and flexible.
One other cool thing I plan on doing is supporting multiple samba
containers running on the same cluster (even the same node if I can
wrangle the network properly). So one could in fact have completely
different domain joins and/or configurations. While I wouldn't
suggest
anyone run a whole lot of different configurations on the same cluster
-
this idea already allows for some level of agility between schemes.
Later
on we might be able to use that as a building block for migration
tools,
either from an existing samba setup or between configurations.
Multiple SAMBA containers are also good for high availability (with
ctdb) or scale-out (with round-robin DNS).
Also, I plan on adding `global_custom_options`
and
`share_custom_options`
for special overrides for development, qa, and experimentation but
those
are strongly within the "you break it, you bought it" realm. But
these
could be used for experimenting with idmapping schemes without
having
them all baked into the smb mgr module code.
Great, thanks! 
25 Mar
25 Mar
5:46 p.m.
Hi John,
On 3/21/24 20:12, John Mulligan wrote:
I'd like to formally let the wider community know
of some work I've been
involved with for a while now: adding Managed SMB Protocol Support to Ceph.
SMB being the well known network file protocol native to Windows systems and
supported by MacOS (and Linux). The other key word "managed" meaning
integrating with Ceph management tooling - in this particular case cephadm for
orchestration and eventually a new MGR module for managing SMB shares.
The effort is still in it's very early stages. We have a PR adding initial
support for Samba Containers to cephadm [1] and a prototype for an smb MGR
module [2]. We plan on using container images based on the samba-container
project [3] - a team I am already part of. What we're aiming for is a feature
set similar to the current NFS integration in Ceph, but with a focus on
bridging non-Linux/Unix clients to CephFS using a protocol built into those
systems.
A few major features we have planned include:
* Standalone servers (internally defined users/groups)
* Active Directory Domain Member Servers
* Clustered Samba support
* Exporting Samba stats via Prometheus metrics
* A `ceph` cli workflow loosely based on the nfs mgr module
I wanted to share this information in case there's wider community interest in
this effort.
certainly! :)
If it makes sense, you may want to pull in samba-technical where it
makes sense. If there's a need, you can also pull me in directly into
meetings or other channels to discuss things.
Looking forward to seeing you at SambaXP, at least virtually. Any plans
to attend SDC from you or others from your team?
Cheers!
-slow
5:56 p.m.
On Monday, March 25, 2024 1:46:26 PM EDT Ralph Boehme wrote:
Hi John,
On 3/21/24 20:12, John Mulligan wrote:
> I'd like to formally let the wider community know of some work I've been
> involved with for a while now: adding Managed SMB Protocol Support to
> Ceph.
SMB being the well known network file protocol native to Windows
> systems and supported by MacOS (and Linux). The
other key word "managed"
> meaning integrating with Ceph management tooling - in this particular
> case cephadm for orchestration and eventually a new MGR module for
> managing SMB shares.
> The effort is still in it's very early stages. We have a PR adding
> initial
> support for Samba Containers to cephadm [1] and a prototype for an smb
> MGR
> module [2]. We plan on using container images based on the
> samba-container
> project [3] - a team I am already part of. What we're aiming for is a
> feature
set similar to the current NFS integration in Ceph, but with a
> focus on bridging non-Linux/Unix clients to
CephFS using a protocol built
> into those systems.
>
> A few major features we have planned include:
> * Standalone servers (internally defined users/groups)
> * Active Directory Domain Member Servers
> * Clustered Samba support
> * Exporting Samba stats via Prometheus metrics
> * A `ceph` cli workflow loosely based on the nfs mgr module
>
> I wanted to share this information in case there's wider community
> interest in
this effort.
certainly! :)
If it makes sense, you may want to pull in samba-technical where it
makes sense.
Absolutely. I'm currently focusing on the basics and those are mostly good-
to-go for our needs in current samba releases. In the future, I'm sure we'll
run into times where technical help or changes will be needed.
If there's a need, you can also pull me in
directly into
meetings or other channels to discuss things.
Thanks! I appreciate it!
Looking forward to seeing you at SambaXP, at least
virtually.
You too. :-)
Any plans to attend SDC from you or others from your
team?
I'm unsure. I'll ask around.
11 Apr
11 Apr
1:35 p.m.
Hi John,
I've finally came around to finish the database client driver for Samba
to talk to Ceph via Python librados and implement some changes that do
improve performance compared to the one from Samuel I used last year:
<https://git.samba.org/?p=slow/samba.git;a=shortlog;h=refs/heads/dbwrap_py>
I'm doing my testing and benchmarking on a very basic Ceph cluster and I
would *love* to run benchmarks against a more beefy cluster and not a
single server partitioned into VMs also lacking decent storage...
By chance, do you have such a thing accessible and would be able to run
a few benchmark? Anyone else interested? :)
It's a bit more involved as it also requires a few more VMs used as SMB
client and clustered Samba...
Cheers!
-slow
On 3/25/24 18:56, John Mulligan wrote:
On Monday, March 25, 2024 1:46:26 PM EDT Ralph Boehme
wrote:
Hi John,
On 3/21/24 20:12, John Mulligan wrote:
> I'd like to formally let the wider community know of some work I've been
> involved with for a while now: adding Managed SMB Protocol Support to
> Ceph.
SMB being the well known network file protocol native to Windows
> systems and supported by MacOS (and Linux).
The other key word "managed"
> meaning integrating with Ceph management tooling - in this particular
> case cephadm for orchestration and eventually a new MGR module for
> managing SMB shares.
> The effort is still in it's very early stages. We have a PR adding
> initial
> support for Samba Containers to cephadm [1] and a prototype for an smb
> MGR
> module [2]. We plan on using container images based on the
> samba-container
> project [3] - a team I am already part of. What we're aiming for is a
> feature
set similar to the current NFS integration in Ceph, but with a
> focus on bridging non-Linux/Unix clients to
CephFS using a protocol built
> into those systems.
>
> A few major features we have planned include:
> * Standalone servers (internally defined users/groups)
> * Active Directory Domain Member Servers
> * Clustered Samba support
> * Exporting Samba stats via Prometheus metrics
> * A `ceph` cli workflow loosely based on the nfs mgr module
>
> I wanted to share this information in case there's wider community
> interest in
this effort.
certainly! :)
If it makes sense, you may want to pull in samba-technical where it
makes sense.
Absolutely. I'm currently focusing on the basics and those are mostly good-
to-go for our needs in current samba releases. In the future, I'm sure we'll
run into times where technical help or changes will be needed.
If there's a need, you can also pull me in
directly into
meetings or other channels to discuss things.
Thanks! I appreciate it!
Looking forward to seeing you at SambaXP, at
least virtually.
You too. :-)
Any plans to attend SDC from you or others from
your team?
I'm unsure. I'll ask around.
1:55 p.m.
On Thursday, April 11, 2024 9:35:28 AM EDT Ralph Boehme wrote:
Hi John,
I've finally came around to finish the database client driver for Samba
to talk to Ceph via Python librados and implement some changes that do
improve performance compared to the one from Samuel I used last year:
<https://git.samba.org/?p=slow/samba.git;a=shortlog;h=refs/heads/dbwrap_py>
I'm doing my testing and benchmarking on a very basic Ceph cluster and I
would *love* to run benchmarks against a more beefy cluster and not a
single server partitioned into VMs also lacking decent storage...
By chance, do you have such a thing accessible and would be able to run
a few benchmark? Anyone else interested? :)
It's a bit more involved as it also requires a few more VMs used as SMB
client and clustered Samba...
Cheers!
-slow
I haven't done much perf testing myself, as I usually get by with a "few vms
on a laptop" approach. There may be some opportunities to run things in the
ceph sepia lab but I would have to ask around first as I typically only use it
to run teuthology tests on smithi nodes.
Do you have a deadline?
Rather than route this discussion through me, it may make sense for you to
join the `#sepia` channel on ceph's slack instance. (go to https://ceph.io/en/
community/connect/ and scroll all the way to the bottom for a link to join the
slack instance) and discuss what opportunities / restrictions there are with
the real experts :-)
2:11 p.m.
Hi John
On 4/11/24 15:55, John Mulligan wrote:
I haven't done much perf testing myself, as I
usually get by with a "few vms
on a laptop" approach. There may be some opportunities to run things in the
ceph sepia lab but I would have to ask around first as I typically only use it
to run teuthology tests on smithi nodes.
oh, that looks helpful. I'll check.
Do you have a deadline?
You bet so: next Wednesday when I'm talking about all of this at
SambaXP... :)
Rather than route this discussion through me, it may
make sense for you to
join the `#sepia` channel on ceph's slack instance. (go to https://ceph.io/en/
community/connect/ and scroll all the way to the bottom for a link to join the
slack instance) and discuss what opportunities / restrictions there are with
the real experts :-)
ok, thanks!
Cheers!
-slow
28 Mar
28 Mar
1:15 a.m.
Yes, I'd love this!
A lot of companies want samba for simple file access from windows/mac
clients. I know quite some companies that buy netapp as 'easy smb storage'.
Having ceph do built-in (or bolt-on) samba instead of having to manage
external samba clusters would be nice, and would make it more accessible
to replace above storage.
And the result would be better integration between samba and ceph.
Perhaps in code, but also in documentation and example configs.
I set up my own 2 physical node samba cluster, with gluster to host the
CTDB lock file. (with a third machine, a vm, to act as the third node in
the gluster cluster). According to 45drives, saving the CTDB lock file
in CephFS is a bad idea, and doing some rados-mutex thingy was too
complex for me. And this whole solution feels a bit hackish, although it
works wonders. Having a unified tried and tested solution where everyone
is doing the same thing, sounds great!
Angelo.
On 21/03/2024 15:12, John Mulligan wrote:
> Hello Ceph List,
>
> I'd like to formally let the wider community know of some work I've been
> involved with for a while now: adding Managed SMB Protocol Support to Ceph.
> SMB being the well known network file protocol native to Windows systems and
> supported by MacOS (and Linux). The other key word "managed" meaning
> integrating with Ceph management tooling - in this particular case cephadm for
> orchestration and eventually a new MGR module for managing SMB shares.
>
> The effort is still in it's very early stages. We have a PR adding initial
> support for Samba Containers to cephadm [1] and a prototype for an smb MGR
> module [2]. We plan on using container images based on the samba-container
> project [3] - a team I am already part of. What we're aiming for is a feature
> set similar to the current NFS integration in Ceph, but with a focus on
> bridging non-Linux/Unix clients to CephFS using a protocol built into those
> systems.
>
> A few major features we have planned include:
> * Standalone servers (internally defined users/groups)
> * Active Directory Domain Member Servers
> * Clustered Samba support
> * Exporting Samba stats via Prometheus metrics
> * A `ceph` cli workflow loosely based on the nfs mgr module
>
> I wanted to share this information in case there's wider community interest in
> this effort. I'm happy to take your questions / thoughts / suggestions in this
> email thread, via Ceph slack (or IRC), or feel free to attend a Ceph
> Orchestration weekly meeting! I try regularly attend and we sometimes discuss
> design aspects of the smb effort there. It's on the Ceph Community Calendar.
> Thanks!
>
>
> [1] - https://github.com/ceph/ceph/pull/55068
> [2] - https://github.com/ceph/ceph/pull/56350
> [3] - https://github.com/samba-in-kubernetes/samba-container/
>
>
> Thanks for reading,
> --John Mulligan
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:28 p.m.
Hey,
We make use of the ctdb_mutex_ceph_rados_helper so the lock file just gets stored within
CephFS metadata pool rather than on a shared CephFS mount as a file.
We don't recommend storing directly on CephFS as if the mount hosting the lock file is
to go down we have seen the mds mark as stale, and then another client can't reach the
lockfile causing problems with CTDB.
I believe the source Angelo is referring to is within a video we did about cephfs+samba a
couple years ago, Brett does a much better job of explaining it than me hahahaha:
https://www.youtube.com/watch?v=Gel9elLSEsQ
It's at 2:44
Regards,
Bailey
-----Original Message-----
From: Alexander E. Patrakov <patrakov(a)gmail.com>
Sent: March 28, 2024 3:13 AM
To: Angelo Hongens <angelo(a)hongens.nl>
Cc: ceph-users(a)ceph.io
Subject: [ceph-users] Re: Call for Interest: Managed SMB Protocol Support
On Thu, Mar 28, 2024 at 9:17 AM Angelo Hongens <angelo(a)hongens.nl>
wrote:
According to 45drives, saving the CTDB lock file
in CephFS is a bad
idea
Could you please share a link to their page that says this?
--
Alexander E. Patrakov
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email
to ceph-users-leave(a)ceph.io
28
days inactive
49
days old
16 comments
7 participants
participants (7)
-
Alexander E. Patrakov -
Angelo Hongens -
Bailey Allison -
David Yang -
John Mulligan -
Ralph Boehme -
Robert Sander