- ceph-users - lists.ceph.io

Repurposing some Dell R750s for Ceph
by Drew Weaver 12 Jul '24

12 Jul '24

Hello, We would like to repurpose some Dell PowerEdge R750s for a Ceph cluster. Currently the servers have one H755N RAID controller for each 8 drives. (2 total) I have been asking their technical support what needs to happen in order for us to just rip out those raid controllers and cable the backplane directly to the motherboard/PCIe lanes and they haven't been super enthusiastic about helping me. I get it just buy another 50 servers, right? No big deal. I have the diagrams that show how each configuration should be connected, I think I just need the right cable(s), my question is has anyone done this work before and was it worth it? Also bonus if anyone has an R750 that has the drives directly connected to the backplane and can find the part number of the cable that connects the backplane to the motherboard I would greatly appreciate that part number. My sales guys are "having a hard time locating it". Thanks, -Drew

7 13

Help with Mirroring
by Dave Hall 12 Jul '24

12 Jul '24

Hello. I would like to use mirroring to facilitate migrating from an existing Nautilus cluster to a new cluster running Reef. RIght now I'm looking at RBD mirroring. I have studied the RBD Mirroring section of the documentation, but it is unclear to me which commands need to be issued on each cluster and, for commands that have both clusters as arguments, when to specify site-a where vs. site-b. Another concern: Both the old and new cluster internally have the default name 'Ceph' - when I set up the second cluster I saw no obvious reason to change from the default. If these will cause a problem with mirroring, is there a workaround? In the long run I will also be migrating a bunch of RGW data. If there are advantages to using mirroring for this I'd be glad to know. (BTW, the plan is to gradually decommission the systems from the old cluster and add them to the new cluster. In this context, I am looking to enable and disable mirroring on specific RBD images and RGW buckets as the client workload is migrated from accessing the old cluster to accessing the new. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

5 4

cephadm for Ubuntu 24.04
by Stefan Kooman 12 Jul '24

12 Jul '24

Hi, Is it possible to only build "cephadm", so not the other ceph packages / daemons? Or can we think about a way to have cephadm packages build for all supported mainstream linux releases during the supported lifetime of a Ceph release: i.e. debian, Ubuntu LTS, CentOS Stream? I went ahead and upgraded one of our (pre-prod) ceph nodes to Ubuntu 24.04 LTS. As it is managed by cephadm and running containers this should be no problem, right? Right. Except that there are is no "cephadm" package (or any other ceph package) for Ubuntu 24.04 from Ceph repository. So, then just build those packages yourself, right? Right. After some python pyyaml 6.0.0 / cython issues in monitoring/ceph-mixin/requirements-alerts.txt and requirements-lint.txt, fixed by bumping pyyaml to 6.0.1) and a hard python 3.10 requirement (I bumped that to 3.12) I ran into an issue with "arrow_ext" breaking the build process which I'm currently trying to figure out ... I know supporting Ceph packages for a bunch of distros is a lot of work, but having _just_ cephadm available on a wider range of platforms, would really help here. It helps avoid upgrading both Ceph and the OS at the same time. This allows the use of the latest (kernel) OS improvements, all while not touching Ceph. Thanks, Gr. Stefan

7 11

reef 18.2.3 QE validation status
by Yuri Weinstein 12 Jul '24

12 Jul '24

Details of this release are summarized here: https://tracker.ceph.com/issues/65393#note-1 Release Notes - TBD LRC upgrade - TBD Seeking approvals/reviews for: smoke - infra issues, still trying, Laura PTL rados - Radek, Laura approved? Travis? Nizamudeen? rgw - Casey approved? fs - Venky approved? orch - Adam King approved? krbd - Ilya approved powercycle - seems fs related, Venky, Brad PTL ceph-volume - will require https://github.com/ceph/ceph/pull/56857/commits/63fe3921638f1fb7fc065907a9e… Guillaume is fixing it. TIA

10 29

AssumeRoleWithWebIdentity in RGW with Azure AD
by Ryan Rempel 12 Jul '24

12 Jul '24

I'm trying to setup the OIDC provider for RGW so that I can have roles that can be assumed by people logging into their regular Azure AD identities. The client I'm planning to use is Cyberduck – it seems like one of the few GUI S3 clients that manages the OIDC login process in a way that could work for relatively naive users. I've gotten a fair ways down the road. I've been able to configure Cyberduck so that it performs the login with Azure AD, gets an identity token, and then sends it to Ceph to engage with the AssumeRoleWithWebIdentity process. However, I then get an error, which shows up in the Ceph rgw logs like this: 2024-07-08T17:18:09.749+0000 7fb2d7845700 0 req 15967124976712370684 1.284013867s sts:assume_role_web_identity Signature validation failed: evp verify final failed: 0 error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding I turned the logging for rgw up to 20 to see if I could follow along to see how much of the process succeeds and learn more about what fails. I can then see logging messages from this file in the source code: https://github.com/ceph/ceph/blob/08d7ff952d78d1bbda04d5ff7e3db1e733301072/… We get to WebTokenEngine::get_from_jwt, and it logs the JWT payload in a way that seems to be as expected. The logs then indicate that a request is sent to the /.well-known/openid-configuration endpoint that appears to be appropriate for the issuer of the JWT. The logs eventually indicate what looks like a successful and appropriate response to that. The logs then show that a request is sent to the jwks_uri that is indicated in the openid-configuration document. The response to that is logged, and it appears to be appropriate. We then get some logging starting with "Certificate is", so it looks like we're getting as far as WebTokenEngine::validate_signature. So, several things appear to have happened successfully – we've loading the OIDC provider that corresponds to the iss, and we've found a client ID that corresponds to what I registered when I configured things. (This is why I say we appear to be a fair ways down the road – a lot of this is working). It looks as though what's happening in the code now is that it's iterating through the certificates given in the jwks_uri content. There are 6 certificates listed, but the code only gets as far as the first one. Looking at the code, what appears to be happening is that, among the various certificates in the jwks_uri, it's finding the first one which matches a thumbprint registered with Ceph (that is, which I registered with Ceph). This must be succeeding (for the first certificate), because the "Signature validation failed" logging comes later. So, the code does verify that the thumbprint of the first certificate matches one of the thumbprints I registered with Ceph for this OIDC provider. We then get to a part of the code where it tries to verify the JWT using the certificate, with jwt::verify. Given what gets logged ("Signature validateion failed: ", this must be throwing an exception. The thing I find surprising about this is that there really isn't any reason to think that the first certificate listed in the jwks_uri content is going to be the certificate used to sign the JWT. If I understand JWT correctly, it's appropriate to sign the JWT with any of the certificates listed in the jwks_uri content. Furthermore, the JWT header includes a reference to the kid, so it's possible for Ceph to know exactly which certificate the JWT purports to be signed by. And, Ceph knows that there might be multiple thumbprints, because we can register 5. So, the logic of trying the first valid certificate in x5c and then stopping if it fails seems broken, actually. I suppose what I could do as a workaround is try to figure out whether Azure AD is consistently using the same kid to sign the JWTs for me, and then only register that thumbprint with Ceph. Then, Ceph would actually choose the correct certificate (as the others wouldn't match a thumbprint I registered). I may try this – in part, just to verify what I think is happening. But it would be awfully fragile – I don't believe there is any requirement in JWT to just use one of the certificates listed in x5c. An alternative would be to try rewriting the code to apply a different kind of logic. The way it ought to work (it seems to me) is something like this: * Get the openid_configuration, and get the jwks stuff from the jwks_uri (which Ceph does already). * Look at the header of the JWT to see which kid it purports to be signed by. * Find the certificate that corresponds to that kid (from the jwks_uri content) * Validate the JWT with that certificate. That ought to work, at least given what I'm seeing. (But, I'm not a JWT expert, so I don't know whether there is something unusual in how Azure AD generates JWT's and handles the jwks_uri content). Anyway, I'm curious whether anyone else has been trying to get this to work with Azure AD, and whether they have run into similar problems. And, of course, whether I appear to be misunderstanding anything about how this is supposed to work. Ryan Rempel Director of Information Technology Canadian Mennonite University

2 3

[RFC][UADK integration][Acceleration of zlib compressor]
by Rongqi Sun 11 Jul '24

11 Jul '24

Hi Ceph community, UADK is an open source accelerator framework, the kernel support part is UACCE <https://docs.kernel.org/misc-devices/uacce.html>, which has been merged in Kernel for several years, targeting to provide Shared Virtual Addressing (SVA) between accelerators and processes. UADK provider users with a unified programming interface to efficiently harness the capabilities of hardware accelerators. It furnishes users with fundamental library and driver support. Now UADK can offer the hardware accelerator engine(e.g: Kunpeng KAE), Arm SVE and Crypto Extension CPU. UADK <https://github.com/Linaro/uadk> is hosted by Linaro. UADK now has already supports different communities for compressor and encryption, such as OpenSSL, DPDK and SPDK, so now, we would like to bring it to Ceph for Acceleration of zlib compressor. As first step, we can see that, 1. save almost 50% cpu usage compared with no-isal compression in RBD 4M workload 2. save almost 40% cpu usage compared with no-isal compression in RGW put op (4M) workload The PR <https://github.com/ceph/ceph/pull/58336> is under review, welcome any comments or reviews. Have a nice day~ Rongqi Sun

2 1

July's User + Developer Monthly Meeting
by Noah Lehman 11 Jul '24

11 Jul '24

Hi Ceph users, July's User + Developer Monthly meeting will be happening Wednesday, July 24th at 10AM EDT. We look forward to seeing you there! Event details: https://hubs.la/Q02GgVNb0 Best, Noah

1 0

Changing ip addr
by Albert Shih 11 Jul '24

11 Jul '24

Hi everyone I just change the subnet of my cluster. The cephfs part seem to working well. But I got many error with Jul 11 10:08:35 hostname ceph-*************** ts=2024-07-11T08:08:35.364Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://OLD_IP:8765/sd/prometheus/sd-config?service=alertmanager\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" I didn't find which service does the discovery and was unable to change the URL. I try a ceph config-key dump |grep OLD_IP and didn't find it. So where this information are store ? Regards JAS -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: jeu. 11 juil. 2024 10:22:58 CEST

2 5

Multi site sync details
by Huseyin Cotuk 11 Jul '24

11 Jul '24

Hello Cephers, I am wondering whether it is possible to get the number of objects that are not synced yet in a multi site radosgw configuration. "radosgw-admin sync status” gives the number of shards that are behind. Similarly “radosgw-admin bucket sync status —bucket {bucket_name}” also gives the status of the shards belong to the given bucket. I just want to get the number of objects that are not synced yet. The only way I found is to get bucket stats from each zone and compare the number of objects. While one of the zone the following output (only related part), "usage": { "rgw.main": { "size": 20896350288253, "size_actual": 22604324229120, "size_utilized": 20896350288253, "size_kb": 20406592079, "size_kb_actual": 22074535380, "size_kb_utilized": 20406592079, "num_objects": 749099512 } }, the other zone gives the output below. "usage": { "rgw.main": { "size": 20896350967086, "size_actual": 22604325040128, "size_utilized": 20896350967086, "size_kb": 20406592742, "size_kb_actual": 22074536172, "size_kb_utilized": 20406592742, "num_objects": 749099571 } }, I look for the difference of number of objects between two zones ( 749099571 - 749099512 = 59 objects). But I am not sure about these stats that they provide the number of objects that are already synced and related to its own zone. The metadata of the multisite radosgw is hold on the master zone. Does anybody clarify that the number of objects on the bucket stats query belongs to every zone’s own bucket or multi site shared metadata? I have doubts that the difference between the bucket stats output from each zone may be a result of command’s execution time. Any clue will be appreciated. Best regards, Huseyin Cotuk hcotuk(a)gmail.com

1 0

use of db_slots in DriveGroup specification?
by Robert Sander 11 Jul '24

11 Jul '24

Hi, what is the purpose of the db_slots attribute in a DriveGroup specification? My interpretation of the documentation is that I can define how many OSDs use one db device. https://docs.ceph.com/en/reef/cephadm/services/osd/#additional-options "db_slots - How many OSDs per DB device" The default for the cephadm orchestrator is to create as many DB volumes on a DB device as needed for the number of OSDs. In a scenario where there are empty slots for HDDs and an existing DB device should not be used fully "db_slots" could be used. But even if db_slots is in an OSD service spec the orchestrator will only create as many DB volumes as there are HDDs currently available. There is a discussion from 2021 where the use of "block_db_size" and "limit" is suggested: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/6EVOYOHS3BT… Shouldn't db_slots make that easier? Is this a bug in the orchestrator? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 2