- ceph-users - lists.ceph.io

Orchestrator not automating services / OSD issue
by Michael Baer 24 Apr '24

24 Apr '24

Hi, This problem started with trying to add a new storage server into a quincy v17.2.6 ceph cluster. Whatever I did, I could not add the drives on the new host as OSDs: via dashboard, via cephadm shell, by setting osd unmanaged to false. But what I started realizing is that orchestrator will also no longer automatically manage services. I.e. if a service is set to manage by labels, removing and adding labels to different hosts for that service has no affect. Same if I set a service to be manage via hostnames. Same if I try to drain a host (the services/podman containers just keep running). Although, I am able to add/rm services via 'cephadm shell ceph orch daemon add/rm'. But Ceph will not manage automatically using labels/hostnames. This apparently includes OSD daemons. I can not create and on the new host either automatically or manually, but I'm hoping the services/OSD issues are related and not two issues. I haven't been able to find any obvious errors in /var/log/ceph, /var/log/syslog, logs <container>, etc. I have been able to get 'slow ops' errors on monitors by trying to add OSDs manually (and having to restart the monitor). I've also gotten cephadm shell to hang. And had to restart managers. I'm not an expert and it could be something obvious, but I haven't been able to figure out a solution. If anyone has any suggestions, I would greatly appreciate them. Thanks, Mike -- Michael Baer ceph(a)mikesoffice.com

2 2

Re: ceph-users Digest, Vol 118, Issue 85
by duluxoz 24 Apr '24

24 Apr '24

Hi Eugen, Thank you for a viable solution to our underlying issue - I'll attempt to implement it shortly. :-) However, with all the respect in world, I believe you are incorrect when you say the doco is correct (but I will be more than happy to be proven wrong). :-) The relevant text (extracted from the document page (the last couple of paragraphs)) says: ~~~ If a client already has a capability for file-system name |a| and path |dir1|, running |fs authorize| again for FS name |a| but path |dir2|, instead of modifying the capabilities client already holds, a new cap for |dir2| will be granted: cephfsauthorizeaclient.x/dir1rw cephauthgetclient.x [client.x] key = AQC1tyVknMt+JxAAp0pVnbZGbSr/nJrmkMNKqA== caps mds = "allow rw fsname=a path=/dir1" caps mon = "allow r fsname=a" caps osd = "allow rw tag cephfs data=a" cephfsauthorizeaclient.x/dir2rw updated caps for client.x cephauthgetclient.x [client.x] key = AQC1tyVknMt+JxAAp0pVnbZGbSr/nJrmkMNKqA== caps mds = "allow rw fsname=a path=dir1, allow rw fsname=a path=dir2" caps mon = "allow r fsname=a" caps osd = "allow rw tag cephfs data=a" ~~~ The above *seems* to me to say (as per the 2nd `cephauthgetclient.x` example) that a 2nd directory (dir2) *will* be added to the `client.x` authorisation. HOWEVER, this does not work in practice - hence my original query. This is what we originally attempted to do (word for word, only substituting our CechFS name for "a") and we got the error in the original post. So if the doco says that something can be done *and* gives a working example, but an end-user (admin) cannot achieve the same results but gets an error instead when following the exact same commands, then either the doco is incorrect *or* there is something else wrong. BUT your statement ("running 'ceph fs authorize' will overwrite the existing caps, it will not add more caps to the client") is in direct contradiction to the documentation ("If a client already has a capability for file-system name |a| and path |dir1|, running |fs authorize| again for FS name |a| but path |dir2|, instead of modifying the capabilities client already holds, a new cap for |dir2| will be granted"). So there's some sort of "disconnect" there. :-) Cheers On 24/04/2024 17:33, ceph-users-request(a)ceph.io wrote: > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. Re: Latest Doco Out Of Date? (Eugen Block) > 2. Re: stretched cluster new pool and second pool with nvme > (Eugen Block) > 3. Re: Latest Doco Out Of Date? (Frank Schilder) > > _______________________________________________ > ceph-users mailing list --ceph-users(a)ceph.io > To unsubscribe send an email toceph-users-leave(a)ceph.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s

1 0

Latest Doco Out Of Date?
by duluxoz 24 Apr '24

24 Apr '24

Hi All, In reference to this page from the Ceph documentation: https://docs.ceph.com/en/latest/cephfs/client-auth/, down the bottom of that page it says that you can run the following commands: ~~~ ceph fs authorize a client.x /dir1 rw ceph fs authorize a client.x /dir2 rw ~~~ This will allow `client.x` to access both `dir1` and `dir2`. So, having a use case where we need to do this, we are, HOWEVER, getting the following error on running the 2nd command on a Reef 18.2.2 cluster: `Error EINVAL: client.x already has fs capabilities that differ from those supplied. To generate a new auth key for client.x, first remove client.x from configuration files, execute 'ceph auth rm client.x', then execute this command again.` Something we're doing wrong, or is the doco "out of date" (mind you, that's from the "latest" version of the doco, and the "reef" version), or is something else going on? Thanks in advance for the help Cheers Dulux-Oz

4 9

List of bridges irc/slack/discord
by Alvaro Soto 23 Apr '24

23 Apr '24

(Last update) - https://github.com/orgs/opensource-latinamerica/discussions/3 ~~~ Adding a few unofficial/unregistered Ceph IRC channels (cephadm, crimson) IRC -> slack.oss.lat OFTC: starlingx -> slack: starlingx OFTC: openstack-latinamerica -> slack: stack-latinamerica OFTC: openstack-freezer -> slack: stack-freezer OFTC: ceph -> slack: ceph OFTC: sepia -> slack: sepia OFTC: cephfs -> slack: cephfs OFTC: ceph-dashboard -> slack: ceph-dashboard OFTC: ceph-devel -> slack: ceph-devel OFTC:cephadm -> slack:cephadm OFTC:crimson -> slack:crimson IRC -> ceph-storage.slack.com OFTC: ceph -> slack: ceph OFTC: sepia -> slack: sepia OFTC: cephfs -> slack: cephfs OFTC: ceph-dashboard -> slack: ceph-dashboard OFTC: ceph-devel -> slack: ceph-devel OFTC:cephadm -> slack:cephadm OFTC:crimson -> slack:crimson IRC -> discord(a)ceph.io OFTC: ceph -> discord(a)ceph.io: ceph OFTC: sepia -> discord(a)ceph.io: sepia OFTC: cephfs -> discord(a)ceph.io: cephfs OFTC: ceph-dashboard -> discord(a)ceph.io: ceph-dashboard OFTC: ceph-devel -> discord(a)ceph.io: ceph-devel OFTC:cephadm -> discord@ceph.io:cephadm OFTC:crimson -> discord@ceph.io:crimson Ceph invite URL: https://discord.gg/vacj9cZSmm ~~~ Cheers! -- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.

1 0

Best practice and expected benefits of using separate WAL and DB devices with Bluestore
by Niklaus Hofer 23 Apr '24

23 Apr '24

Dear all We have an HDD ceph cluster that could do with some more IOPS. One solution we are considering is installing NVMe SSDs into the storage nodes and using them as WAL- and/or DB devices for the Bluestore OSDs. However, we have some questions about this and are looking for some guidance and advice. The first one is about the expected benefits. Before we undergo the efforts involved in the transition, we are wondering if it is even worth it. How much of a performance boost one can expect when adding NVMe SSDs for WAL-devices to an HDD cluster? Plus, how much faster than that does it get with the DB also being on SSD. Are there rule-of-thumb number of that? Or maybe someone has done benchmarks in the past? The second question is of more practical nature. Are there any best-practices on how to implement this? I was thinking we won't do one SSD per HDD - surely an NVMe SSD is plenty fast to handle the traffic from multiple OSDs. But what is a good ratio? Do I have one NVMe SSD per 4 HDDs? Per 6 or even 8? Also, how should I chop-up the SSD, using partitions or using LVM? Last but not least, if I have one SSD handle WAL and DB for multiple OSDs, losing that SSD means losing multiple OSDs. How do people deal with this risk? Is it generally deemed acceptable or is this something people tend to mitigate and if so how? Do I run multiple SSDs in RAID? I do realize that for some of these, there might not be the one perfect answer that fits all use cases. I am looking for best practices and in general just trying to avoid any obvious mistakes. Any advice is much appreciated. Sincerely Niklaus Hofer -- stepping stone AG Wasserwerkgasse 7 CH-3011 Bern Telefon: +41 31 332 53 63 www.stepping-stone.ch niklaus.hofer(a)stepping-stone.ch

7 6

Status of IPv4 / IPv6 dual stack?
by Robert Sander 23 Apr '24

23 Apr '24

Hi, as the documentation sends mixed signals in https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#ipv… "Note Binding to IPv4 is enabled by default, so if you just add the option to bind to IPv6 you’ll actually put yourself into dual stack mode." and https://docs.ceph.com/en/latest/rados/configuration/msgr2/#address-formats "Note The ability to bind to multiple ports has paved the way for dual-stack IPv4 and IPv6 support. That said, dual-stack operation is not yet supported as of Quincy v17.2.0." just the quick questions: Is a dual stacked networking with IPv4 and IPv6 now supported or not? From which version on is it considered stable? Are OSDs now able to register themselves with two IP addresses in the cluster map? MONs too? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

6 6

Why CEPH is better than other storage solutions?
by sebcio_t＠o2.pl 23 Apr '24

23 Apr '24

Hi, I have problem to answer to this question: Why CEPH is better than other storage solutions? I know this high level texts about - scalability, - flexibility, - distributed, - cost-Effectiveness What convince me, but could be received also against, is ceph as a product has everything what I need it mean: block storage (RBD), file storage (CephFS), object storage (S3, Swift) and "plugins" to run NFS, NVMe over Fabric, NFS on object storage. Also many other features which are usually sold as a option (mirroring, geo replication, etc) in paid solutions. I have problem to write it done piece by piece. I want convince my managers we are going in good direction. Why not something from robin.io or purestorage, netapp, dell/EMC. From opensource longhorn or openEBS. If you have ideas please write it. Thanks, S.

8 8

Stuck in replay?
by Erich Weiler 23 Apr '24

23 Apr '24

Hi All, We have a somewhat serious situation where we have a cephfs filesystem (18.2.1), and 2 active MDSs (one standby). ThI tried to restart one of the active daemons to unstick a bunch of blocked requests, and the standby went into 'replay' for a very long time, then RAM on that MDS server filled up, and it just stayed there for a while then eventually appeared to give up and switched to the standby, but the cycle started again. So I restarted that MDS, and now I'm in a situation where I see this: # ceph fs status slugfs - 29 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay slugfs.pr-md-01.xdtppo 3958k 57.1k 12.2k 0 1 resolve slugfs.pr-md-02.sbblqq 0 3 1 0 POOL TYPE USED AVAIL cephfs_metadata metadata 997G 2948G cephfs_md_and_data data 0 87.6T cephfs_data data 773T 175T STANDBY MDS slugfs.pr-md-03.mclckv MDS version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) It just stays there indefinitely. All my clients are hung. I tried restarting all MDS daemons and they just went back to this state after coming back up. Is there any way I can somehow escape this state of indefinite replay/resolve? Thanks so much! I'm kinda nervous since none of my clients have filesystem access at the moment... cheers, erich

5 8

s3 bucket policy subusers - access denied
by sinan＠turka.nl 23 Apr '24

23 Apr '24

I want to achieve the following: - Create an user - Create 2 subusers - Create 2 buckets - Apply a policy for each bucket - A subuser should only have access to its own bucket Problem: Getting a 403 AccessDenied with subuser credentials when uploading files. I did the following: radosgw-admin user create --uid=foo-user --display_name="Foo Test User" radosgw-admin subuser create --uid=foo-user --gen-access-key --gen-secret --key-type=s3 --subuser=foo-user-subuser radosgw-admin subuser create --uid=foo-user --gen-access-key --gen-secret --key-type=s3 --subuser=foo-user-subuser2 Resulting in: { "user_id": "foo-user", "display_name": "Foo Test User", "email": "", "suspended": 0, "max_buckets": 1000, "subusers": [ { "id": "foo-user:foo-user-subuser", "permissions": "<none>" }, { "id": "foo-user:foo-user-subuser2", "permissions": "<none>" } ], "keys": [ { "user": "foo-user:foo-user-subuser", "access_key": "<key>", "secret_key": "<key>" }, { "user": "foo-user:foo-user-subuser2", "access_key": "<key>", "secret_key": "<key>" }, { "user": "foo-user", "access_key": "<key>", "secret_key": "<key>" } ], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "default_storage_class": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw", "mfa_ids": [] } Using the credentials of the main account (user: foo-user) creating buckets and setting policies: s3cmd mb s3://foo-bucket s3cmd mb s3://foo-bucket2 s3cmd setpolicy foo-test-subuser-policy s3://foo-bucket s3cmd setpolicy foo-test-subuser2-policy s3://foo-bucket2 Resulting in (I am showing just foo-bucket, but the same goes for foo-bucket2): # s3cmd info s3://foo-bucket s3://foo-bucket/ (bucket): Payer: BucketOwner Ownership: none Versioning:none Expiration rule: none Block Public Access: none Policy: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam:::user/foo-user:foo-user-subuser" ] }, "Action": [ "s3:AbortMultipartUpload", "s3:DeleteObject", "s3:GetObject", "s3:ListBucketMultipartUploads", "s3:ListBucket", "s3:ListMultipartUploadParts", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::foo-bucket" ] } ] } CORS: none ACL: Foo Test User: FULL_CONTROL When I try to upload files (using the subuser foo-user-subuser credentials) it doesn't work: # s3cmd ls 2024-04-23 06:59 s3://foo-bucket 2024-04-23 10:05 s3://foo-bucket2 # s3cmd put ~/Documents/file_2.txt s3://foo-bucket upload: '/home/foo/Documents/file_2.txt' -> 's3://foo-bucket/file_2.txt' [1 of 1] 10 of 10 100% in 0s 18.96 B/s done ERROR: S3 error: 403 (AccessDenied) What is wrong with my policy? I thought that I did exactly the same earlier and it worked, but I am in doubt now.... Thanks!

1 0

ceph api rgw/role
by farhad kh 23 Apr '24

23 Apr '24

hi , i used ceph api for create rgw/role but ther is not api for delete or edit rgw/role . how can i delete them or edit ?

2 1