- Dev - lists.ceph.io

Re: [ceph-users] Re: Call for Interest: Managed SMB Protocol Support

by John Mulligan

On Tuesday, March 26, 2024 10:53:29 PM EDT David Yang wrote: > This is great, we are currently using the smb protocol heavily to > export kernel-mounted cephfs. > But I encountered a problem. When there are many smb clients > enumerating or listing the same directory, the smb server will > experience high load, and the smb process will become D state. > This problem has been going on for some time and no suitable solution > has been found yet. > Thanks for the heads up. I'll make sure concurrent dir access is part of the test plan. > John Mulligan <phlogistonjohn(a)asynchrono.us> 于2024年3月26日周二 03:43写道： > > > > > > > On Monday, March 25, 2024 3:22:26 PM EDT Alexander E. Patrakov wrote: > > > > > On Mon, Mar 25, 2024 at 11:01 PM John Mulligan > > > > > > > > > > > > <phlogistonjohn(a)asynchrono.us> wrote: > > > > > > > On Friday, March 22, 2024 2:56:22 PM EDT Alexander E. Patrakov wrote: > > > > > > > > > Hi John, > > > > > > > > > > > > > > > > > > > > > A few major features we have planned include: > > > > > > * Standalone servers (internally defined users/groups) > > > > > > > > > > > > > > > > > > > > No concerns here > > > > > > > > > > > > > > > > > > > > > * Active Directory Domain Member Servers > > > > > > > > > > > > > > > > > > > > In the second case, what is the plan regarding UID mapping? Is NFS > > > > > coexistence planned, or a concurrent mount of the same directory > > > > > using > > > > > CephFS directly? > > > > > > > > > > > > > > > > In the immediate future the plan is to have a very simple, fairly > > > > "opinionated" idmapping scheme based on the autorid backend. > > > > > > > > > > > > OK, the docs for clustered SAMBA do mention the autorid backend in > > > examples. It's a shame that the manual page does not explicitly list > > > it as compatible with clustered setups. > > > > > > > > > > > > However, please consider that the majority of Linux distributions > > > (tested: CentOS, Fedora, Alt Linux, Ubuntu, OpenSUSE) use "realmd" to > > > join AD domains by default (where "default" means a pointy-clicky way > > > in a workstation setup), which uses SSSD, and therefore, by this > > > opinionated choice of the autorid backend, you create mappings that > > > disagree with the supposed majority and the default. This will create > > > problems in the future when you do consider NFS coexistence. > > > > > > > > > > > > > > Thanks, I'll keep that in mind. > > > > > > > > > Well, it's a different topic that most organizations that I have seen > > > seem to ignore this default. Maybe those that don't have any problems > > > don't have any reason to talk to me? I think that more research is > > > needed here on whether RedHat's and GNOME's push of SSSD is something > > > not-ready or indeed the de-facto standard setup. > > > > > > > > > > > > > > I think it's a bit of a mix, but am not sure either. > > > > > > > > > > > Even if you don't want to use SSSD, providing an option to provision a > > > few domains with idmap rid backend with statically configured ranges > > > (as an override to autorid) would be a good step forward, as this can > > > be made compatible with the default RedHat setup. > > > > > > > > That's reasonable. Thanks for the suggestion. > > > > > > > > > > > > > > > > > > Sharing the same directories over both NFS and SMB at the same time, > > > > also > > > > known as "multi-protocol", is not planned for now, however we're all > > > > aware > > > > that there's often a demand for this feature and we're aware of the > > > > complexity it brings. I expect we'll work on that at some point but > > > > not > > > > initially. Similarly, sharing the same directories over a SMB share > > > > and > > > > directly on a cephfs mount won't be blocked but we won't recommend > > > > it. > > > > > > > > > > > > OK. Feature request: in the case if there are several CephFS > > > filesystems, support configuration of which one to serve. > > > > > > > > > > > > > > Putting it on the list. > > > > > > > > > > > In fact, I am quite skeptical, because, at least in my experience, > > > > > every customer's SAMBA configuration as a domain member is a unique > > > > > snowflake, and cephadm would need an ability to specify arbitrary > > > > > UID > > > > > mapping configuration to match what the customer uses elsewhere - > > > > > and > > > > > the match must be precise. > > > > > > > > > > > > > > > > I agree - our initial use case is something along the lines: > > > > Users of a Ceph Cluster that have Windows systems, Mac systems, or > > > > appliances that are joined to an existing AD > > > > but are not currently interoperating with the Ceph cluster. > > > > > > > > > > > > > > > > I expect to add some idpapping configuration and agility down the > > > > line, > > > > especially supporting some form of rfc2307 idmapping (where unix IDs > > > > are > > > > stored in AD). > > > > > > > > > > > > Yes, for whatever reason, people do this, even though it is cumbersome > > > to manage. > > > > > > > > > > > > > But those who already have idmapping schemes and samba accessing ceph > > > > will > > > > probably need to just continue using the existing setups as we don't > > > > have > > > > an immediate plan for migrating those users. > > > > > > > > > > > > > > > > > Here is what I have seen or was told about: > > > > > > > > > > > > > > > > > > > > 1. We don't care about interoperability with NFS or CephFS, so we > > > > > just > > > > > let SAMBA invent whatever UIDs and GIDs it needs using the "tdb2" > > > > > idmap backend. It's completely OK that workstations get different > > > > > UIDs > > > > > and GIDs, as only SIDs traverse the wire. > > > > > > > > > > > > > > > > This is pretty close to our initial plan but I'm not clear why you'd > > > > think > > > > that "workstations get different UIDs and GIDs". For all systems > > > > acessing > > > > the (same) ceph cluster the id mapping should be consistent. > > > > You did make me consider multi-cluster use cases with something like > > > > cephfs > > > > volume mirroring - that's something that I hadn't thought of before > > > > *but* > > > > using an algorithmic mapping backend like autorid (and testing) I > > > > think > > > > we're mostly OK there. > > > > > > > > > > > > The tdb2 backend (used in my example) is not algorithmic, it is > > > allocating. That is, it sequentially allocates IDs on the > > > first-seen-first-allocated basis. Yet this is what this customer uses, > > > presumably because it is the only backend that explicitly specifies > > > clustering operation in its manual page. > > > > > > > > > > > > And the "autorid" backend is also not fully algorithmic, it allocates > > > ranges to domains on the same sequential basis (see > > > https://github.com/samba-team/samba/blob/6fb98f70c6274e172787c8d5f73aa93 > > > 9201 71e7c/source3/winbindd/idmap_autorid_tdb.c#L82), and therefore can > > > create mismatching mappings if two workstations or servers have seen > > > the users DOMA\usera and DOMB\userb in a different order. It is even > > > mentioned in the manual page. SSSD largely avoids this problem by > > > hashing the domain portion of the SID instead of > > > allocating the subranges on a sequential basis. > > > > > > > > > > > > > > Agreed. Thanks for the reminder. This will certainly need to go on the > > test plan. > > > > > > > > > > > 2. [not seen in the wild, the customer did not actually implement > > > > > it, > > > > > it's a product of internal miscommunication, and I am not sure if > > > > > it > > > > > is valid at all] We don't care about interoperability with CephFS, > > > > > and, while we have NFS, security guys would not allow running NFS > > > > > non-kerberized. Therefore, no UIDs or GIDs traverse the wire, only > > > > > SIDs and names. Therefore, all we need is to allow both SAMBA and > > > > > NFS > > > > > to use shared UID mapping allocated on as-needed basis using the > > > > > "tdb2" idmap module, and it doesn't matter that these UIDs and GIDs > > > > > are inconsistent with what clients choose. > > > > > > > > > > > > > > > > Unfortunately, I don't really understand this item. Fortunately, you > > > > say > > > > it > > > > was only considered not implemented. :-) > > > > > > > > > > > > > > > > > 3. We don't care about ACLs at all, and don't care about CephFS > > > > > interoperability. We set ownership of all new files to root:root > > > > > 0666 > > > > > using whatever options are available [well, I would rather use a > > > > > dedicated nobody-style uid/gid here]. All we care about is that > > > > > only > > > > > authorized workstations or authorized users can connect to each NFS > > > > > or > > > > > SMB share, and we absolutely don't want them to be able to set > > > > > custom > > > > > ownership or ACLs. > > > > > > > > > > > > > > > > Some times known as the "drop-box" use case I think (not to be > > > > confused > > > > with the cloud app of a similar name). > > > > We could probably implement something like that as an option but I had > > > > not > > > > considered it before. > > > > > > > > > > > > > > > > > 4. We care about NFS and CephFS file ownership being consistent > > > > > with > > > > > what Windows clients see. We store all UIDs and GIDs in Active > > > > > Directory using the rfc2307 schema, and it's mandatory that all > > > > > servers (especially SAMBA - thanks to the "ad" idmap backend) > > > > > respect > > > > > that and don't try to invent anything [well, they do - > > > > > BUILTIN/Users > > > > > gets its GID through tdb2]. Oh, and by the way, we have this > > > > > strangely > > > > > low-numbered group that everybody gets wrong unless they set "idmap > > > > > config CORP : range = 500-999999". > > > > > > > > > > > > > > > > This is oh so similar to a project I worked on prior to working with > > > > Ceph. > > > > I think we'll need to do this one eventually but maybe not this year. > > > > One nice side-effect of running in containers is that the low-id > > > > number is > > > > less of an issue because the ids only matter within the container > > > > context > > > > (and only then if using the kernel file system access methods). We > > > > have > > > > much more flexibility with IDs in a container. > > > > > > > > > > > > So - are you going to use the kernel-based mount or the ceph vfs > > > module? My tests indicate that, in situations where there are > > > frequently accessed files, allowing the kernel to cache them in RAM > > > (which the vfs module does not do) can create a big boost in > > > performance. Also, SUSE considers the ceph vfs module a > > > non-recommended solution apparently for the same performance-related > > > reason, see > > > https://documentation.suse.com/ses/7/html/ses-all/cha-ses-cifs.html > > > > > > > > > > The prototype module only uses the vfs module due to the extreme > > simplicity of setting it up in containers. Otherwise, we're trying to > > keep our options open and are investigating multiple approaches > > currently. > > > > > > > > > > > 5. We use a few static ranges for algorithmic ID translation using > > > > > the > > > > > idmap rid backend. Everything works. > > > > > > > > > > > > > > > > See above. > > > > > > > > > > > > > > > > > 6. We use SSSD, which provides consistent IDs everywhere, and for a > > > > > few devices which can't use it, we configured compatible idmap rid > > > > > ranges for use with winbindd. The only problem is that we like > > > > > user-private groups, and only SSSD has support for them (although > > > > > we > > > > > admit it's our fault that we enabled this non-default option). > > > > > 7. We store ID mappings in non-AD LDAP and use winbindd with the > > > > > "ldap" idmap backend. > > > > > > > > > > > > > > > > For now, we're only planning to do idmapping with winbind and AD. > > > > We'd > > > > probably only consider non-AD ldap and/or ssd if there was strong and > > > > loud > > > > demand for it. > > > > > > > > > > > > See above. > > > > > > > > > > > > However, as I said, providing a way to use the "rid" backend with > > > statically defined domains and ranges in addition to the default > > > "autorid" backend would be, for me, a good-enough substitute for SSSD. > > > > > > > > > > > > > > Sounds reasonable. I've done it that way in a prior role too, so it's > > somewhat familiar. Thanks! > > > > > > > > > > > I am sure other weird but valid setups exist - please extend the > > > > > list > > > > > if you can. > > > > > > > > > > > > > > > > > > > > Which of the above scenarios would be supportable without resorting > > > > > to > > > > > the old way of installing SAMBA manually alongside the cluster? > > > > > > > > > > > > > > > > I hope I covered the above with some inline replies. This was great > > > > food > > > > for thought and at just the right level of technical detail. So thank > > > > you > > > > very much for replying, this is exactly the kind of discussion I want > > > > to > > > > have now where the design is still young and flexible. > > > > > > > > > > > > > > > > One other cool thing I plan on doing is supporting multiple samba > > > > containers running on the same cluster (even the same node if I can > > > > wrangle the network properly). So one could in fact have completely > > > > different domain joins and/or configurations. While I wouldn't > > > > suggest > > > > anyone run a whole lot of different configurations on the same cluster > > > > - > > > > this idea already allows for some level of agility between schemes. > > > > Later > > > > on we might be able to use that as a building block for migration > > > > tools, > > > > either from an existing samba setup or between configurations. > > > > > > > > > > > > Multiple SAMBA containers are also good for high availability (with > > > ctdb) or scale-out (with round-robin DNS). > > > > > > > > > > > > > Also, I plan on adding `global_custom_options` and > > > > `share_custom_options` > > > > for special overrides for development, qa, and experimentation but > > > > those > > > > are strongly within the "you break it, you bought it" realm. But > > > > these > > > > could be used for experimenting with idmapping schemes without > > > > having > > > > them all baked into the smb mgr module code. > > > > > > > > > > > > Great, thanks! > > > > > > > > Once again, thanks for the feedback. This discussion is very welcome! > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 weeks, 5 days

1
0
0 0

Interest in Participating in Ceph Project for Google Summer of Code

by Xu, Jiachen

Dear Team, My name is Jiachen Xu, I'm a junior Computer Science student at the University of Florida, interested in joining the Ceph project for Google Summer of Code. Although new to Ceph, I have experience with AWS and a strong foundation in data structures and operating systems. Beyond that, I am willing to grow and learn with the team, especially learning knowledge that were outside of classroom. Excited to learn and contribute to the Future of Storage with Ceph. Looking forward to the opportunity. Sincerely, Jiachen Xu

2 weeks, 5 days

2
1
0 0

How can I set osd fast shutdown = true

by Suyash Dongre

Hello, I want to set osd fast shutdown = true, how should I achieve this? Regards, Suyash

2 weeks, 6 days

1
0
0 0

Query regarding sts token refresh error code on expiry

by Ajinkya Deshpande

Dear Team, I have a query regarding STS token refresh. In the case of CEPH, when a token expires and we attempt any operation, we receive an "access denied" error. However, with other S3 providers like Amazon, we receive the correct error code "ExpiredToken." Is this discrepancy a known issue with CEPH , and if so, do we have plans to address it? Having "access denied" as the error code in the event of token expiry seems incorrect. Given the above scenario, please let us know how token refresh can be handled with CEPH ? Ajinkya Deshpande Sr Software Engineer NetBackup Engineering Veritas Technologies LLC Mobile: (7776835515) Ajinkya.Deshpande(a)veritas.com

3 weeks

2
1
0 0

librbd error type

by Anatoly Finch

Currently, the error type of librbd is int, which contains an error code of system_error category. Each function uses their own meaning for different error codes, also a code can be used for different situations, e.g. PreRemoveRequest returns EBUSY when an image has watchers, on acquiring a lock, or when the image is being migrated. Librados uses the same int as an error type, however neorados uses boost::system::error_code. Latest version of boost::system::error_code is {std::error_code, std::source_location*}. And std::error_code is {int, error_category*} (in stdlibc++ and libcxx). It is possible to use custom error types, internally, however user-facing API have to use a popular type such as std or boost error_code, otherwise users would have to convert error types. As std::error_code is the standard error type, and boost::system::error_code is convertible to it Various alternative approaches to errors, such as Boost.LEAF, Boost.Outcome and the corresponding std::error p1028 proposal, all of them are compatible with std(boost)::error_code. As std::error_code is a pair of an int and a pointer, it fits into two registers (eax:rdx), so changing result type from int to std::error_code basically adds one instruction: std::error_code f() { return SomeErrEnum::Code1; } with properly defined std::is_error_code_enum<SomeErrEnum> trait, and a SomeErrCategory type with a corresponding error category global value, compiles to f(): mov edx, OFFSET some_category_var mov eax, 1 ret Full code - https://godbolt.org/z/K7vv5h8oo Although there are corner cases such as surprisingly ineffective error_category& cat() { static SomeErrCategory c; return c; } where "static" requires branching for a guard variable, it is possible to keep overhead to adding single instruction for the error category. Back to the librbd. It is indeed not possible to simply change it's C++ API from returning int to error_code. Also, we cannot add overloads with different return types, we'd have to also change arguments. Passing error_code as an output parameter works int flatten(); // current API void flatten(std::error_code& ec); // alternative API but it doesn't look great, ideally we'd rather use "std::expected<T, error_code> f()" and not "void f(T& out, error_code& ec)". We can also add asynchronous API like in neorados, with CompletionTokens and callbacks that receive error_code. However, much of librbd code is synchronous (Operations), so rewriting it to an async code would needlessly complicate things. Is there a way to make librbd return std::error_code, so that we could use different error categories, enums, for more distinct error codes?

3 weeks, 5 days

1
0
0 0

Ceph Leadership Team meeting 2024-03-18

by Nizamudeen A

hi, below are the topics discussed: - Re-evaluate 19.1.0 * cephfs waiting for 2 feature PR, to be finished by this week * rgw: https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/CT5MJ44ELP5OYLUVXV… * blockers: centos9 containers: https://github.com/ceph/ceph-container/pull/2183 ceph-object-corpus: https://github.com/ceph/ceph-object-corpus/pull/17 (testing in https://github.com/ceph/ceph/pull/54735) jammy packages for quincy to support upgrades: https://github.com/ceph/ceph-build/pull/2206 * target date is March 27. Will be re-evaluating again on next monday (3/25) - Replacement for Trello * Patrick proposed redmine agile board: https://tracker.ceph.com/projects/ceph-qa/agile/board It basically works the same. There are new issue statuses only used by the "QA Run" tracker (like "Bug" or "Backport") which correspond to what's used in Trello. * Yuri is going to give redmine a try: https://tracker.ceph.com/projects/ceph-qa/issues?query_id=309 - Redmine upgrade * We are on 3.3.1 (Oct 2016): https://www.redmine.org/versions/122 * Current stable is: redmine-5.1.2 * talk with Dan/Adam to see the possibilities for upgrading! - Patrick will follow-up * https://www.redmine.org/projects/redmine/wiki/RedmineUpgrade - Retrospective on github team permissions * write access is needed for almost everything * need to rely on branch protection hooks instead of permissions * re-iterate on the criteria for merging PRs (more discussion needed, will be followed up in the coming week CLTs) The full meeting minutes can be found here: https://pad.ceph.com/p/clt-weekly-minutes Regards -- Nizamudeen A Software Engineer Red Hat <https://www.redhat.com/> <https://www.redhat.com/>

3 weeks, 6 days

1
0
0 0

PSA: CephFS/MDS config defer_client_eviction_on_laggy_osds

by Venky Shankar

If you are using CephFS on Pacific v16.2.14(+), the MDS config `defer_client_eviction_on_laggy_osds' is enabled by default. This config is used to not evict cephfs clients if OSDs are laggy[1]. However, this can result in a single client holding up the MDS in servicing other clients. To avoid this, please disable the config by executing ceph config set mds defer_client_eviction_on_laggy_osds false This issue doesn't affect any Quincy or Reef releases. Furthermore, the config is being disabled by default[2] till the fix is being worked on. [1]: https://tracker.ceph.com/issues/58023 [2]: https://tracker.ceph.com/issues/64685 -- Cheers, Venky

1 month

1
0
0 0

[PSA] push/admin access to ceph/ceph.io

by Patrick Donnelly

All, At the CLT today we discussed the proliferation of write/admin access on the ceph repository. One of the consequences of this has been that Ceph's code guidelines have not been followed in merges [1]. Additionally, having too many folks -- many of whom have retired from active development -- with write access to the repository presents security concerns. With the CLT's support, I have addressed this by pruning write/admin access to the Ceph repository to only these Github teams: - https://github.com/orgs/ceph/teams/ceph-maintainers - https://github.com/orgs/ceph/teams/ceph-release-team - https://github.com/orgs/ceph/teams/admins - https://github.com/orgs/ceph/people?query=role%3Aowner "ceph-maintainers" is a new team that includes component team leads and senior Ceph engineers. If you feel you should be in this list and were missed (sorry!), please reply to this mail. "ceph-release-team" is a new team that includes folks working on Ceph releases, right now Yuri. "admins" is an extant team that includes members who help administrate the Ceph project. The members of the Ceph org who are owners have write/admin privileges regardless of team organization. I've included that for completeness. Anyone not in these aforementioned teams will (should) be unable to push to ceph.git [2]. Please coordinate with your component team lead for merging your changes. [1] https://github.com/ceph/ceph/blob/main/SubmittingPatches.rst [2] https://github.com/ceph/ceph/settings/access -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 month

2
2
0 0

Ceph Leadership Team meeting 2024-03-13

by Ilya Dryomov

Hello, Topics discussed: - a noticeable backlog of "make check" jobs and shaman builds (>6 hours) - mostly self-inflicted because folks have been retriggering "make check" a lot recently in the hopes of working around a number of transient failures - even more so if the pool of jenkins workers used for "make check" and shaman builds is the same - Laura will confirm in the infra meeting - Patrick will downgrade github org owners that aren't active to regular members - proposal to prune down the list of individuals with write access to ceph.git repo (Patrick) - component leads and long-time senior contributors only - the goal is to enforce our SubmittingPatches.rst rules better - also some additional security - concerns over fractured issue tracking - the most recent example is ceph-nvmeof using github issues, but there is also nvmeof subproject on tracker.ceph.com - other notable examples are ceph-csi and go-ceph, although there hasn't been anything on tracker.ceph.com for these - conclusion: github issues or any other issue tracking system is fine as long as there is no tight coupling to ceph.git - question: does a repo being brought in as a submodule as is the case with ceph-nvmeof in https://github.com/ceph/ceph/pull/54671 constitute tight coupling? - in this case the submodule is intended to bring in two .proto files for use by NVMeofGwMonitorClient daemon, everything else in ceph-nvmeof repo shouldn't be looked at - is this the best way to do that -- could these files just be copied? - dashboard already carries a copy of one of them (src/pybind/mgr/dashboard/services/proto/gateway.proto) - creating a pypy package is another option - QA nightlies - now that they are back, need to ensure they are looked at! - poll among component leads: is status quo where all results go to ceph-qa list OK or do we need to have teuthology email people/teams directly? - let's tally up next week Topics moved to next week: - 19.1.0 status - trello to limit free workspaces to 10 collaborators - need a replacement for Yuri's board at the very least Thanks, Ilya

1 month

1
0
0 0

References to common bad code / practices

by Suyash Dongre

Hi there. I am creating a static analysis tool using clang-tidy for the Ceph codebase to quickly find out potential bugs. I wanted references to or examples of bad code or code which is frequently used which when written badly can result in failures. With this I hope to create the tool to check for bad code. Any help would be appreciated. Thank you. Regards, Suyash

1 month

1
0
0 0

2024

2023

2022

2021

2020

2019

Dev