May 2020 - Dev - lists.ceph.io

test, please ignore (unless you are dev-owner)

by Tim Serong

Hi All, Sorry for the noise, I'm still not receiving mail from this list (or from ceph-users for that matter) and needed to do another test. David (or whoever else is dev-owner), if it's not too much trouble can you please send me a snippet from the mailman/postfix log for when *this* message is delivered back to me? Thanks a lot in advance, and again, sorry for the noise. Regards, Tim -- Tim Serong Senior Clustering Engineer SUSE tserong(a)suse.com

2 years, 10 months

1
0
0 0

RGW STS Support in Nautilus ?

by Wyllys Ingersoll

Running Nautilus 14.2.9 and trying to follow the STS example given here: https://docs.ceph.com/docs/master/radosgw/STS/ to setup a policy for AssumeRoleWithWebIdentity using KeyCloak (8.0.1) as the OIDC provider. I am able to see in the rgw debug logs that the token being passed from the client is passing the introspection check, but it always ends up failing the final authorization to access the requested bucket resource and is rejected with a 403 status "AccessDenied". I configured my policy as described in the 2nd example on the STS page above. I suspect the problem is with the "StringEquals" condition statement in the AssumeRolePolicy document (I could be wrong though). The example shows using the keycloak URI followed by ":app_id" matching with the name of the keycloak client application ("customer-portal" in the example). My keycloak setup does not have any such field in the introspection result and I can't seem to figure out how to make this all work. I cranked up the logging to 20/20 and still did not see any hints as to what part of the policy is causing the access to be denied. Any suggestions? -Wyllys Ingersoll

2 years, 10 months

3
6
0 0

osd_memory_target in Rook, set automatically via k8s Resource Requests, and hardware recommendations

by Blaine Gardner

(Please use reply-all so that people I've explicitly tagged and I can continue to get direct email replies) I have a conundrum around developing best practices for Rook/Ceph clusters around OSD memory targets and hardware recommendations for OSDs. I want to lay down some bulleted notes first. - 'osd_memory_target' defaults to 4GB - OSDs attempt to keep memory allocation to 'osd_memory_target' - BUT... this is only best-effort - AND... there is no guarantee that the kernel will actually reclaim memory that OSDs release/unmap - Therefore, we (SUSE) have developed a recommendation that ... Total OSD node RAM required = (num OSDs) x (1 GB + osd_memory_target) + 16 GB - In Rook/Kubernetes, Ceph OSDs will read the POD_MEMORY_REQUEST and POD_MEMORY_LIMIT env vars to infer a new default value for 'osd_memory_target' - POD_MEMORY_REQUEST translates directly to 'osd_memory_target' 1:1 - POD_MEMORY_LIMIT (if REQUEST is unset) will set 'osd_memory_target' using the formula ( LIMIT x osd_memory_target_cgroup_limit_ratio ) - the default 'osd_memory_target' will be = min(REQUEST, LIMIT*ratio) - Lars has suggested that setting limits is not a best practice for Ceph; when limits are encountered, Ceph is likely in a failure state, and killing daemons could result in a "thundering herds" distributed systems problem As you can see, there is a self-referential problem here. The OSD hardware recommendation should inform us how to set k8s resource limits for OSDs; however, doing so will affect osd_memory_target, which alters the recommendation, which further alters our k8s resource limit circularly forever. We can address this issue with a semi-workaround currently: set osd_memory_target explicitly in Ceph's config, and set an appropriate k8s resource request matching (osd_memory_target + 1GB + some extra) to meet the hardware recommendation. However, means that the Ceph feature of setting osd_memory_target based on resource requests isn't really used because it doesn't behave to actual best practices. And setting a realistic k8s resource request is useful for kubernetes so that it won't schedule more daemons onto a node than the node can realistically support. Long-term, I wonder if it is good to add into Ceph a computation that [[ osd_memory_target = REQUEST + osd_memory_request_overhead ]] where the osd_memory_requests overhead defaults to 1GB or somewhat higher. Please discuss, and let me know if anything here seems like I've gotten it wrong or if there are other options I haven't seen. Cheers, and happy Tuesday! Blaine

2 years, 10 months

3
3
0 0

DocuBetter Meeting -- EMEA 13 May 2020

by John Zachary Dover

There is a general documentation meeting called the "DocuBetter Meeting", and it is held every two weeks. The next DocuBetter Meeting will be on 13 May 2020 at 0830 PST, and will run for thirty minutes. Everyone with a documentation-related request or complaint is invited. The meeting will be held here: https://bluejeans.com/908675367 Send documentation-related requests and complaints to me by replying to this email and CCing me at zac.dover(a)gmail.com. The next DocuBetter meeting is scheduled for: 13 May 2020 0830 PST 13 May 2020 1630 UTC 14 May 2020 0230 AEST Etherpad: https://pad.ceph.com/p/Ceph_Documentation Meeting: https://bluejeans.com/908675367 Thanks, everyone. Zac Dover

2 years, 10 months

1
0
0 0

Tracing RocksDB Compaction

by Yiming Zhang

Hi All, Is there a way to get the exact time and size of each level compaction in RocksDB with BlueStore? My goal is to know when does compact triggered and which Transaction is compacted. I can use the rocksdb perf data, but it lacks the details info about each compaction. Below is an example of rocksdb perf data: "rocksdb_compaction_statistics": "", "": "", "": "** Compaction Stats [default] **", "": "Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop", "": "----------------------------------------------------------------------------------------------------------------------------------------------------------------------------", "": " L0 1/0 35.73 MB 0.2 0.0 0.0 0.0 0.2 0.2 0.0 1.0 0.0 80.0 3.02 2.55 8 0.377 0 0", "": " L1 2/0 83.15 MB 0.3 0.3 0.2 0.1 0.1 0.1 0.0 0.7 181.7 96.8 1.45 0.78 2 0.724 1429K 858K", "": " Sum 3/0 118.88 MB 0.0 0.3 0.2 0.1 0.4 0.3 0.0 1.6 58.9 85.5 4.47 3.33 10 0.447 1429K 858K", "": " Int 0/0 0.00 KB 0.0 0.3 0.2 0.1 0.4 0.3 0.0 1.6 58.9 85.5 4.47 3.33 9 0.496 1429K 858K", "": "", "": "** Compaction Stats [default] **", "": "Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop", "": "-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------", "": " Low 0/0 0.00 KB 0.0 0.3 0.2 0.1 0.1 0.1 0.0 0.0 181.7 96.8 1.45 0.78 2 0.724 1429K 858K", "": "High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.2 0.2 0.0 0.0 0.0 80.1 3.02 2.55 7 0.431 0 0", "": "User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.00 0.00 1 0.001 0 0”, Thanks, Yiming

2 years, 10 months

2
1
0 0

rfc: Accounts in RGW

by Abhishek Lekshmanan

The basic premise is for an account to be a container for users, and also related functionality like roles & groups. This would converge similar to the AWS concept of an account, where the AWS account can further create iam users/roles or groups. Every account can have a root user or user(s) with permissions to administer creation of users and allot quotas within an account. These can be implemented with a new account cap. IAM set of apis already have a huge subset of functionality to summarize accounts and inspect/create users/roles or groups. Every account would also store the membership of its users/groups and roles, (similar to user's buckets) though we'd ideally limit to < 10k users/roles or groups per account. In order to deal with the currently used tenants which namespace buckets, but also currently stand in for the account id in the policy language & ARNs, we'd have a tenant_id attribute in the account, which if set will prevent cross tenant users being added. Though this is not enforced when the tenant id isn't set, accounts without this field set can potentially add users across tenants, so this is one of the cases where we expect the account owner to know what they are doing. We'd transition away from <tenant>:<user> in the Policy principal to <account-id>:<user>, so if users with different tenants are in the same account we'd expect the user to change their policies to reuse the account ids. In terms of regular operations IO costs, the user info would have an account id attribute, and if non empty we'd have to read the Account root user policies and /or public access configuration, though other attributes like list of users/roles and groups would only be read for necessary IAM/admin apis. Quotas ~~~~~~ For quotas we can implement one of the following ways - a user_bytes/buckets quota, which would be alloted to every user - a total account quota, in which case it is the responsibility of the account user to allot a quota upon user creation Though for operations themselves it is th user quota that comes into play. APIs ~~~~ - creating an account itself should be available via the admin tooling/apis - Ideally creation of a root user under an account would still have to be explicitly, though we could consider adding this to the account creation process itself to simplify things. - For further user creation and management, we could start implementing to the iam set of apis in the future, though currently we already have admin apis for user creation and the like, and we could allow the user with account caps to do these operations Deviations ~~~~~~~~~~ Some apis like list buckets in AWS list all the buckets in the user account and not the specific iam user, we'd probably still list only the user buckets, though we could consider this for the account root user. Wrt to the openstack swift apis, we'd still keep the current user_id -> swift account id mapping, so no breakage is expected wrt end user apis, so the account stats and related apis would be similar to the older version where it is still user's summary that is displayed Comments on if this is the right direction? -- Abhishek

2 years, 10 months

1
0
0 0

RadosGW

by Abhinav Singh

Hello everyone, I have a small doubt, GET /{api version}/{account} HTTP/1.1 Host: {fqdn} X-Auth-Token: {auth-token} this is a swift request to get bucket from cluster what is {api version} and {auth-token} in this request, account I have as "test". And should I use curl to make these requests?

2 years, 10 months

2
2
0 0

octopus repo lock

by Yuri Weinstein

We'd like to lock the repo and start QE validation on v15.2.2 early next week. PRs still in progress: https://github.com/ceph/ceph/pulls?q=is%3Aopen+label%3Aoctopus-batch-1+labe… cephfs - still re-running and needs to be approved/merged - https://trello.com/c/36K5aNYm rados/rgw - https://trello.com/c/3Tp1Bjzu (ready for review) https://trello.com/c/p07LqZvi (almost ready) Current plan is to merge all PRs this week, lock the repo, upgrade/test the sepia lab to the latest octopus release and and start QE validation in parallel. Thx YuriW

2 years, 10 months

1
2
0 0

Error getting bucket stats

by Seena Fallah

Hi all. I'm getting lots of error getting bucket stats from my rgw instances and I can't figure out what's going wrong! Can we add bucket name to this error to see whats happening in the cluster? https://github.com/ceph/ceph/pull/34653 Thanks.

2 years, 10 months

2
8
0 0

State of cephadm

by Sebastian Wagner

Hi list, let me quickly summarize the current state of the cephadm development and give some insights in the current list of topics for the near future and for the pacific release. For an introduction to cephadm, see [0]. With the initial Octopus release, cephadm was already able to deploy the usual Ceph services. With the upcoming 15.2.2, cephadm will gain some UX improvements, which make cephadm actually usable now. In addition, cephadm can also now deploy NFS ganesha. For the upcoming development, we have four themes where cephadm needs to enhance: 1. Streamline the user experience This consists of topics like making the current state of cephadm visible to the user, improve the responsiveness of the CLI, improve the day 1 experience, improve the official documentation, an improved integration with the Ceph Dashboard and improvements to particular services (RGW, MDS, MGR, etc.) 2. Teuthology improvements This theme contains things like making the dashboard use the cephadm.py Teuthology task, enabling the RGW test again, making the CephFS test runner compatible to cephadm, and allowing other Ceph components to base on the cephadm.py task. 3. High availability. We'd like to add the capability to cephadm to deploy high available services. This might contain new services like haproxy, keepalived or maybe some internal DNS. 4. Resilience This contains items like adding an MGR thrasher to Teuthology and making all commands of cephadm idempotent. In addition to those four themes, there are also other topics as well: * the current scheduler essentially randomizes the placement of daemons across the cluster. There are certainly many improvements possible here. * It would be great to enhance cstart (similar to vstart, but with containers) and thus improve the developer experience * Enhance the compatibility to Rook (daemon placements, upgrades of Ceph and OSDs management) For more info, please have a look at the CDS a few weeks back: [1] And if you'd like to get involved, feel free to join our weekly orchestrator sync meeting on Monday or have a look at our tracker [2] Best, Sebastian [0]: https://ceph.io/ceph-management/introducing-cephadm/ [1]: https://pad.ceph.com/p/orchestrator-pacific [2]: https://tracker.ceph.com/projects/orchestrator/issues -- SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg). Geschäftsführer: Felix Imendörffer

2 years, 10 months

1
0
0 0

2023

2022

2021

2020

2019

Dev May 2020