- ceph-users - lists.ceph.io

Re: Stuck OSD service specification - can't remove
by Eugen Block 16 Mar '23

16 Mar '23

Hi, I tried to respond directly in the web ui of the mailing list but my message is queued for moderation. I just wanted to update a solution that worked for me when a service spec is stuck in a pending state, maybe this will help others in the same situation. While playing around with a test cluster I ended up with a "deleting" osd service spec. The SUSE team has an article [1] for this case, the following helped me resolve this issue. I had three different osd specs in place for the same three nodes: ---snip--- osd 3 <deleting> 3w nautilus2;nautilus3 osd.osd-hdd-ssd 3 2m ago 2w nautilus;nautilus2;nautilus3 osd.osd-hdd-ssd-mix 3 2m ago - <unmanaged> ---snip--- I replaced the "service_name" with the more suiting value ("osd.osd-hdd-ssd") in the unit.meta file of each OSD containing the invalid spec, then restarted each affected OSD. It probably wouldn't have been necessary but I wanted to see the effect immediately, so I failed over the mgr (ceph mgr fail), now I only have one valid osd spec. ---snip--- # before nautilus3:~ # grep service_name /var/lib/ceph/201a2fbc-ce7b-44a3-9ed7-39427972083b/osd.3/unit.meta "service_name": "osd", # after nautilus3:~ # grep service_name /var/lib/ceph/201a2fbc-ce7b-44a3-9ed7-39427972083b/osd.3/unit.meta "service_name": "osd.osd-hdd-ssd", nautilus3:~ # ceph orch ls osd NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd.osd-hdd-ssd 9 10m ago 2w nautilus;nautilus2;nautilus3 ---snip--- Regards, Eugen [1] https://www.suse.com/support/kb/doc/?id=000020667

1 0

External Auth (AssumeRoleWithWebIdentity) , STS by default, generic policies and isolation by ownership
by Christian Rohmann 16 Mar '23

16 Mar '23

Hello ceph-users, unhappy with the capabilities in regards to bucket access policies when using the Keystone authentication module I posted to this ML a while back - https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/S2TV7GVFJT… In general I'd still like to hear how others are making use of external authentication and STS and what your experiences are in replacing e.g. Keystone authentication In the meantime we looked into OIDC authentication (via Keycloak) and the potentials there. While this works in general, AssumeRoleWithWebIdentity comes back with an STS token and that can be used to access S3 buckets, I am wondering about a few things: 1) How to enable STS for everyone (without user-individual policy to AssumeRole) In the documentation on STS (https://docs.ceph.com/en/quincy/radosgw/STS/#sts-in-ceph) and also STS-Lite (https://docs.ceph.com/en/quincy/radosgw/STSLite/#sts-lite) it's implied at one has to attach an dedicated policy to allow for STS to each user individually. This does not scale well with thousands of users. Also when using a federated / external authentication, there is no explicit user creation "A shadow user is created corresponding to every federated user. The user id is derived from the ‘sub’ field of the incoming web token." Is there a way to automatically have a role corresponding to each user that can be assumed via a OIDC token? So an implicit role that would allow for an externally authenticated user to have full access to S3 and all buckets owned? Looking at STS Lite documentation, it seems all the more natural to be able to allow keystone users to make use of STS. Is there any way to apply such an AssumeRole policy "globally" or for a whole set of users at the same time? I just found PR https://github.com/ceph/ceph/pull/44434 aiming to add policy variables such as ${aws:username} to allow for generic policies. But this is more about restricting bucket names or granting access to certain pattern of names. 2) Isolation in S3 Multi-Tenancy with external IdP (AssumeRoleWithWebIdentity), how does bucket ownership come into play? Following the question about generic policies for STS I am wondering about the role (no pun intended) that the bucket ownership or tenant play here? If one creates a role policy of e.g. {"Version":"2012-10-17","Statement":{"Effect":"Allow","Action":"s3:*","Resource":"arn:aws:s3:::*"}} Would this allow someone assuming this role access to all, "*", buckets, or just those owned by the user that created this role policy? In case of Keystone auth the owner of a bucket is the project, not the individual (human) user. So this creates somewhat of a tenant which I'd want to isolate. 3) Allowing users to create their own roles and policies by default Is there a way to allow users to create their own roles and policies to use them by default? All the examples talk about the requirement for admin caps and individual setting of '--caps="user-policy=*'. If there was a default role + policy (question #1) that could be applied to externally authenticated users, I'd like for them to be able to create new roles and policies to grant access to their buckets to other users. Regards Christian

2 1

How to repair the OSDs while WAL/DB device breaks down
by Norman 16 Mar '23

16 Mar '23

hi, everyone, I have a question about repairing the broken WAL/DB device. I have a cluster with 8 OSDs, and 4 WAL/DB devices(1 OSD per WAL/DB device), and hwo can I repair the OSDs quickly if one WAL/DB device breaks down without rebuilding the them? Thanks.

2 4

Expression of Interest in Participating in GSoC 2023 with Your Team
by Arush Sharma 16 Mar '23

16 Mar '23

Dear Ceph Team, I hope this email finds you well. I am writing to express my keen interest in participating in the Google Summer of Code (GSoC) program 2023 with your team. I am a 3rd year B.tech student in Computer Science Engineering, with a strong passion for [specific area of interest related to the team's project(s)]. I have experience working in C++, and I believe that I can contribute significantly to your project by bringing my expertise, enthusiasm, and commitment. I have been following the GSoC program, and I understand the dedication and hard work required to complete a project successfully. Therefore, I am willing to commit my time and effort to meet the expectations and requirements of the program. I am open to learning new technologies and programming languages, and I believe that this opportunity will help me grow both personally and professionally. I have reviewed the list of your team's project ideas, and I am particularly interested in Disk Fragmentation Simulator. I would appreciate it if you could provide me with any additional information or resources that may be helpful to better understand the project requirements and goals. Thank you for taking the time to read my email, and I look forward to hearing back from you soon. Best regards, Arush Sharma

2 1

Upgrade 16.2.11 -> 17.2.0 failed
by bbk 16 Mar '23

16 Mar '23

Dear List, Today i was sucessfully upgrading with cephadm from 16.2.8 -> 16.2.9 -> 16.2.10 -> 16.2.11 Now i wanted to upgrade to 17.2.0 but after starting the upgrade with ``` # ceph orch upgrade start --ceph-version 17.2.0 ``` The orch manager module seems to be gone now and the upgrade don't seem to run. ``` # ceph orch upgrade status Error ENOENT: No orchestrator configured (try `ceph orch set backend`) # ceph orch set backend cephadm Error ENOENT: Module not found ``` During the failed upgrade all nodes had the 16.2.11 cephadm installed. Fortunately the cluster is still running... somehow. I installed the latest 17.2.X cephadm on all nodes and rebooted them nodes, but this didn't help. Does someone have a hint? Yours, bbk

4 7

Bluestore RocksDB Compression how to set
by Feng, Hualong 15 Mar '23

15 Mar '23

Hi all I want to study the effect of bluestore rocksdb compression on ceph and whether it is necessary to optimize it. But currently, bluestore rocksdb compression is disabled by default in ceph. I simply replaced the rocksdb compression algorithm, and then performed a 4KB rand read fio test, but the test results are not different. So I would like to ask you, is there any scenario where rocksdb needs to be compressed in ceph, and if so, what type of test is used to better demonstrate the impact of rocksdb compression on the upper layer of ceph. // create 10GB image ceph osd pool create rbd 128 rbd pool init rbd rbd create --size 10240 --image rbd/image --image-format 2 --thick-provision //fio [global] ioengine=rbd iodepth=128 rw=randwrite bs=4KB time_based=1 ramp_time=60s runtime=300s clientname=admin pool=rbd group_reporting buffer_compress_percentage=80 refill_buffers buffer_pattern=0xdeadbeef [volumes] rbdname=image numjobs=1 Thanks -Hualong

1 0

Ganesha NFS: Files disappearing
by Patrick Schlangen 15 Mar '23

15 Mar '23

Hi, today I saw a strange situation where files which were copied to a cephfs via Ganesha NFS (deployed via cephadm) disappeared from the NFS directory and then did not show up anymore until I restarted the ganesha instance. This could be observed on different NFS client hosts. While the files were not showing, they could still be seen via cephfs-shell, so I assume the issue is somewhere on the NFS side. Interestingly, the folder modification time (shown in ls) matched the time the file was copied to that folder. Any idea what could be the issue here? Unfortunately there were no interesting logs and I couldn't find a way yet to reproduce this. The ceph version in use is 17.2.5. Thanks, Patrick

2 2

10x more used space than expected
by Gaël THEROND 15 Mar '23

15 Mar '23

Hi everyone, I’ve got a quick question regarding one of our RadosGW bucket. This bucket is used to store docker registries, and the total amount of data we use is supposed to be 4.5Tb BUT it looks like ceph told us we rather use ~53Tb of data. One interesting thing is, this bucket seems to shard for unknown reason as it is supposed to be disabled by default, but even taking that into account we’re not supposed to see such a massive amount of additional data isn’t it? Here is the bucket stats of it: https://paste.opendev.org/show/bdWFRvNFtxyHnbPfXWu9/

4 8

Stuck OSD service specification - can't remove
by David Orman 15 Mar '23

15 Mar '23

Has anybody run into a 'stuck' OSD service specification? I've tried to delete it, but it's stuck in 'deleting' state, and has been for quite some time (even prior to upgrade, on 15.2.x). This is on 16.2.3: NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd.osd_spec 504/525 <deleting> 12m label:osd root@ceph01:/# ceph orch rm osd.osd_spec Removed service osd.osd_spec From active monitor: debug 2021-05-06T23:14:48.909+0000 7f17d310b700 0 log_channel(cephadm) log [INF] : Remove service osd.osd_spec Yet in ls, it's still there, same as above. --export on it: root@ceph01:/# ceph orch ls osd.osd_spec --export service_type: osd service_id: osd_spec service_name: osd.osd_spec placement: {} unmanaged: true spec: filter_logic: AND objectstore: bluestore We've tried --force, as well, with no luck. To be clear, the --export even prior to delete looks nothing like the actual service specification we're using, even after I re-apply it, so something seems 'bugged'. Here's the OSD specification we're applying: service_type: osd service_id: osd_spec placement: label: "osd" data_devices: rotational: 1 db_devices: rotational: 0 db_slots: 12 I would appreciate any insight into how to clear this up (without removing the actual OSDs, we're just wanting to apply the updated service specification - we used to use host placement rules and are switching to label-based). Thanks, David

3 4

Last day to sponsor Cephalocon Amsterdam 2023
by Mike Perez 14 Mar '23

14 Mar '23

Hi everyone, Today is the last day to sponsor Cephalocon Amsterdam 2023! I want to thank our current sponsors: Platinum: IBM Silver: 42on, Canonical Ubuntu, Clyso Startup: Koor Also, thank you to Clyso for their lanyard add-on and 42on's offsite attendee party. We are still short in covering the costs for the event, so I'm asking for contributors and members of the Ceph Foundation to consider applying today. https://events.linuxfoundation.org/cephalocon/sponsor/ Sponsor Prospectus: https://events.linuxfoundation.org/wp-content/uploads/2023/03/sponsor-ceph-… Please get in touch with us at sponsorships(a)ceph.foundation to get started. Thank you! -- Mike Perez

1 0