- ceph-users - lists.ceph.io

by duluxoz

Hi All, In reference to this page from the Ceph documentation: https://docs.ceph.com/en/latest/cephfs/client-auth/, down the bottom of that page it says that you can run the following commands: ~~~ ceph fs authorize a client.x /dir1 rw ceph fs authorize a client.x /dir2 rw ~~~ This will allow `client.x` to access both `dir1` and `dir2`. So, having a use case where we need to do this, we are, HOWEVER, getting the following error on running the 2nd command on a Reef 18.2.2 cluster: `Error EINVAL: client.x already has fs capabilities that differ from those supplied. To generate a new auth key for client.x, first remove client.x from configuration files, execute 'ceph auth rm client.x', then execute this command again.` Something we're doing wrong, or is the doco "out of date" (mind you, that's from the "latest" version of the doco, and the "reef" version), or is something else going on? Thanks in advance for the help Cheers Dulux-Oz

3 weeks, 2 days

4
9
0 0

List of bridges irc/slack/discord

by Alvaro Soto

(Last update) - https://github.com/orgs/opensource-latinamerica/discussions/3 ~~~ Adding a few unofficial/unregistered Ceph IRC channels (cephadm, crimson) IRC -> slack.oss.lat OFTC: starlingx -> slack: starlingx OFTC: openstack-latinamerica -> slack: stack-latinamerica OFTC: openstack-freezer -> slack: stack-freezer OFTC: ceph -> slack: ceph OFTC: sepia -> slack: sepia OFTC: cephfs -> slack: cephfs OFTC: ceph-dashboard -> slack: ceph-dashboard OFTC: ceph-devel -> slack: ceph-devel OFTC:cephadm -> slack:cephadm OFTC:crimson -> slack:crimson IRC -> ceph-storage.slack.com OFTC: ceph -> slack: ceph OFTC: sepia -> slack: sepia OFTC: cephfs -> slack: cephfs OFTC: ceph-dashboard -> slack: ceph-dashboard OFTC: ceph-devel -> slack: ceph-devel OFTC:cephadm -> slack:cephadm OFTC:crimson -> slack:crimson IRC -> discord(a)ceph.io OFTC: ceph -> discord(a)ceph.io: ceph OFTC: sepia -> discord(a)ceph.io: sepia OFTC: cephfs -> discord(a)ceph.io: cephfs OFTC: ceph-dashboard -> discord(a)ceph.io: ceph-dashboard OFTC: ceph-devel -> discord(a)ceph.io: ceph-devel OFTC:cephadm -> discord@ceph.io:cephadm OFTC:crimson -> discord@ceph.io:crimson Ceph invite URL: https://discord.gg/vacj9cZSmm ~~~ Cheers! -- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.

3 weeks, 2 days

1
0
0 0

Best practice and expected benefits of using separate WAL and DB devices with Bluestore

by Niklaus Hofer

Dear all We have an HDD ceph cluster that could do with some more IOPS. One solution we are considering is installing NVMe SSDs into the storage nodes and using them as WAL- and/or DB devices for the Bluestore OSDs. However, we have some questions about this and are looking for some guidance and advice. The first one is about the expected benefits. Before we undergo the efforts involved in the transition, we are wondering if it is even worth it. How much of a performance boost one can expect when adding NVMe SSDs for WAL-devices to an HDD cluster? Plus, how much faster than that does it get with the DB also being on SSD. Are there rule-of-thumb number of that? Or maybe someone has done benchmarks in the past? The second question is of more practical nature. Are there any best-practices on how to implement this? I was thinking we won't do one SSD per HDD - surely an NVMe SSD is plenty fast to handle the traffic from multiple OSDs. But what is a good ratio? Do I have one NVMe SSD per 4 HDDs? Per 6 or even 8? Also, how should I chop-up the SSD, using partitions or using LVM? Last but not least, if I have one SSD handle WAL and DB for multiple OSDs, losing that SSD means losing multiple OSDs. How do people deal with this risk? Is it generally deemed acceptable or is this something people tend to mitigate and if so how? Do I run multiple SSDs in RAID? I do realize that for some of these, there might not be the one perfect answer that fits all use cases. I am looking for best practices and in general just trying to avoid any obvious mistakes. Any advice is much appreciated. Sincerely Niklaus Hofer -- stepping stone AG Wasserwerkgasse 7 CH-3011 Bern Telefon: +41 31 332 53 63 www.stepping-stone.ch niklaus.hofer(a)stepping-stone.ch

3 weeks, 2 days

7
6
0 0

Status of IPv4 / IPv6 dual stack?

by Robert Sander

Hi, as the documentation sends mixed signals in https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#ipv… "Note Binding to IPv4 is enabled by default, so if you just add the option to bind to IPv6 you’ll actually put yourself into dual stack mode." and https://docs.ceph.com/en/latest/rados/configuration/msgr2/#address-formats "Note The ability to bind to multiple ports has paved the way for dual-stack IPv4 and IPv6 support. That said, dual-stack operation is not yet supported as of Quincy v17.2.0." just the quick questions: Is a dual stacked networking with IPv4 and IPv6 now supported or not? From which version on is it considered stable? Are OSDs now able to register themselves with two IP addresses in the cluster map? MONs too? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

3 weeks, 2 days

6
6
0 0

Why CEPH is better than other storage solutions?

by sebcio_t＠o2.pl

Hi, I have problem to answer to this question: Why CEPH is better than other storage solutions? I know this high level texts about - scalability, - flexibility, - distributed, - cost-Effectiveness What convince me, but could be received also against, is ceph as a product has everything what I need it mean: block storage (RBD), file storage (CephFS), object storage (S3, Swift) and "plugins" to run NFS, NVMe over Fabric, NFS on object storage. Also many other features which are usually sold as a option (mirroring, geo replication, etc) in paid solutions. I have problem to write it done piece by piece. I want convince my managers we are going in good direction. Why not something from robin.io or purestorage, netapp, dell/EMC. From opensource longhorn or openEBS. If you have ideas please write it. Thanks, S.

3 weeks, 2 days

8
8
0 0

Stuck in replay?

by Erich Weiler

Hi All, We have a somewhat serious situation where we have a cephfs filesystem (18.2.1), and 2 active MDSs (one standby). ThI tried to restart one of the active daemons to unstick a bunch of blocked requests, and the standby went into 'replay' for a very long time, then RAM on that MDS server filled up, and it just stayed there for a while then eventually appeared to give up and switched to the standby, but the cycle started again. So I restarted that MDS, and now I'm in a situation where I see this: # ceph fs status slugfs - 29 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay slugfs.pr-md-01.xdtppo 3958k 57.1k 12.2k 0 1 resolve slugfs.pr-md-02.sbblqq 0 3 1 0 POOL TYPE USED AVAIL cephfs_metadata metadata 997G 2948G cephfs_md_and_data data 0 87.6T cephfs_data data 773T 175T STANDBY MDS slugfs.pr-md-03.mclckv MDS version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) It just stays there indefinitely. All my clients are hung. I tried restarting all MDS daemons and they just went back to this state after coming back up. Is there any way I can somehow escape this state of indefinite replay/resolve? Thanks so much! I'm kinda nervous since none of my clients have filesystem access at the moment... cheers, erich

3 weeks, 3 days

5
8
0 0

s3 bucket policy subusers - access denied

by sinan＠turka.nl

I want to achieve the following: - Create an user - Create 2 subusers - Create 2 buckets - Apply a policy for each bucket - A subuser should only have access to its own bucket Problem: Getting a 403 AccessDenied with subuser credentials when uploading files. I did the following: radosgw-admin user create --uid=foo-user --display_name="Foo Test User" radosgw-admin subuser create --uid=foo-user --gen-access-key --gen-secret --key-type=s3 --subuser=foo-user-subuser radosgw-admin subuser create --uid=foo-user --gen-access-key --gen-secret --key-type=s3 --subuser=foo-user-subuser2 Resulting in: { "user_id": "foo-user", "display_name": "Foo Test User", "email": "", "suspended": 0, "max_buckets": 1000, "subusers": [ { "id": "foo-user:foo-user-subuser", "permissions": "<none>" }, { "id": "foo-user:foo-user-subuser2", "permissions": "<none>" } ], "keys": [ { "user": "foo-user:foo-user-subuser", "access_key": "<key>", "secret_key": "<key>" }, { "user": "foo-user:foo-user-subuser2", "access_key": "<key>", "secret_key": "<key>" }, { "user": "foo-user", "access_key": "<key>", "secret_key": "<key>" } ], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "default_storage_class": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw", "mfa_ids": [] } Using the credentials of the main account (user: foo-user) creating buckets and setting policies: s3cmd mb s3://foo-bucket s3cmd mb s3://foo-bucket2 s3cmd setpolicy foo-test-subuser-policy s3://foo-bucket s3cmd setpolicy foo-test-subuser2-policy s3://foo-bucket2 Resulting in (I am showing just foo-bucket, but the same goes for foo-bucket2): # s3cmd info s3://foo-bucket s3://foo-bucket/ (bucket): Payer: BucketOwner Ownership: none Versioning:none Expiration rule: none Block Public Access: none Policy: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam:::user/foo-user:foo-user-subuser" ] }, "Action": [ "s3:AbortMultipartUpload", "s3:DeleteObject", "s3:GetObject", "s3:ListBucketMultipartUploads", "s3:ListBucket", "s3:ListMultipartUploadParts", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::foo-bucket" ] } ] } CORS: none ACL: Foo Test User: FULL_CONTROL When I try to upload files (using the subuser foo-user-subuser credentials) it doesn't work: # s3cmd ls 2024-04-23 06:59 s3://foo-bucket 2024-04-23 10:05 s3://foo-bucket2 # s3cmd put ~/Documents/file_2.txt s3://foo-bucket upload: '/home/foo/Documents/file_2.txt' -> 's3://foo-bucket/file_2.txt' [1 of 1] 10 of 10 100% in 0s 18.96 B/s done ERROR: S3 error: 403 (AccessDenied) What is wrong with my policy? I thought that I did exactly the same earlier and it worked, but I am in doubt now.... Thanks!

3 weeks, 3 days

1
0
0 0

ceph api rgw/role

by farhad kh

hi , i used ceph api for create rgw/role but ther is not api for delete or edit rgw/role . how can i delete them or edit ?

3 weeks, 3 days

2
1
0 0

Question about PR merge

by Erich Weiler

Hello, We are tracking PR #56805: https://github.com/ceph/ceph/pull/56805 And the resolution of this item would potentially fix a pervasive and ongoing issue that needs daily attention in our cephfs cluster. I was wondering if it would be included in 18.2.3 which I *think* should be released soon? Is there any way of knowing if that is true? Thanks again, erich

3 weeks, 3 days

4
6
0 0

RGWs stop processing requests after upgrading to Reef

by Iain Stott

Hi, We have recently upgraded one of our clusters from Quincy 17.2.6 to Reef 18.2.1, since then we have had 3 instances of our RGWs stop processing requests. We have 3 hosts that run a single instance of RGW on each, and all 3 just seem to stop processing requests at the same time causing our storage to become unavailable. A restart or redeploy of the RGW service brings them back ok. The cluster was deployed using ceph ansible, but since we have adopted it to cephadm which is how the upgrade was performed. We have enabled debug logging as there was nothing out of the ordinary in normal logs and are currently sifting through them from the last crash. We are just wondering if it possible to run Quincy RGWs instead of Reef as we didn't have this issue prior to the upgrade? We have 3 clusters in a multisite setup, we are holding off on upgrading the other 2 clusters due to this issue. Thanks Iain Iain Stott OpenStack Engineer Iain.Stott(a)thg.com [THG Ingenuity Logo]<https://www.thg.com> www.thg.com<https://www.thg.com/> [LinkedIn]<https://www.linkedin.com/company/thgplc/?originalSubdomain=uk> [Instagram] <https://www.instagram.com/thg> [X] <https://twitter.com/thgplc?lang=en>

3 weeks, 3 days

2
2
0 0

2024

2023

2022

2021

2020

2019

ceph-users