- ceph-users - lists.ceph.io

Re: Ceph iSCSI GW is too slow when compared with Raw RBD performance
by Work Ceph 04 Jul '23

04 Jul '23

Awesome, thanks for the info! By any chance, do you happen to know what configurations you needed to adjust to make Veeam perform a bit better? On Fri, Jun 23, 2023 at 10:42 AM Anthony D'Atri <aad(a)dreamsnake.net> wrote: > Yes, with someone I did some consulting for. Veeam seems to be one of the > prevalent uses for ceph-iscsi, though I'd try to use the native RBD client > instead if possible. > > Veeam appears by default to store really tiny blocks, so there's a lot of > protocol overhead. I understand that Veeam can be configured to use "large > blocks" that can make a distinct difference. > > > > On Jun 23, 2023, at 09:33, Work Ceph <work.ceph.user.mailing(a)gmail.com> > wrote: > > Great question! > > Yes, one of the slowness was detected in a Veeam setup. Have you > experienced that before? > > On Fri, Jun 23, 2023 at 10:32 AM Anthony D'Atri <aad(a)dreamsnake.net> > wrote: > >> Are you using Veeam by chance? >> >> > On Jun 22, 2023, at 21:18, Work Ceph <work.ceph.user.mailing(a)gmail.com> >> wrote: >> > >> > Hello guys, >> > >> > We have a Ceph cluster that runs just fine with Ceph Octopus; we use RBD >> > for some workloads, RadosGW (via S3) for others, and iSCSI for some >> Windows >> > clients. >> > >> > We started noticing some unexpected performance issues with iSCSI. I >> mean, >> > an SSD pool is reaching 100MB of write speed for an image, when it can >> > reach up to 600MB+ of write speed for the same image when mounted and >> > consumed directly via RBD. >> > >> > Is that performance degradation expected? We would expect some >> degradation, >> > but not as much as this one. >> > >> > Also, we have a question regarding the use of Intel Turbo boost. Should >> we >> > disable it? Is it possible that the root cause of the slowness in the >> iSCSI >> > GW is caused by the use of Intel Turbo boost feature, which reduces the >> > clock of some cores? >> > >> > Any feedback is much appreciated. >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >

2 2

radosgw new zonegroup hammers master with metadata sync
by Boris Behrens 04 Jul '23

04 Jul '23

Hi, yesterday I added a new zonegroup and it looks like it seems to cycle over the same requests over and over again. In the log of the main zone I see these requests: 2023-06-20T09:48:37.979+0000 7f8941fb3700 1 beast: 0x7f8a602f3700: fd00:2380:0:24::136 - - [2023-06-20T09:48:37.979941+0000] "GET /admin/log?type=metadata&id=62&period=e8fc96f1-ae86-4dc1-b432-470b0772fded&max-entries=100&&rgwx-zonegroup=b39392eb-75f8-47f0-b4f3-7d3882930b26 HTTP/1.1" 200 44 - - - Only thing that changes is the &id. We have two other zonegroups that are configured identical (ceph.conf and period) and these don;t seem to spam the main rgw. root@host:~# radosgw-admin sync status realm 5d6f2ea4-b84a-459b-bce2-bccac338b3ef (main) zonegroup b39392eb-75f8-47f0-b4f3-7d3882930b26 (dc3) zone 96f5eca9-425b-4194-a152-86e310e91ddb (dc3) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master root@host:~# radosgw-admin period get { "id": "e8fc96f1-ae86-4dc1-b432-470b0772fded", "epoch": 92, "predecessor_uuid": "5349ac85-3d6d-4088-993f-7a1d4be3835a", "sync_status": [ "", ... "" ], "period_map": { "id": "e8fc96f1-ae86-4dc1-b432-470b0772fded", "zonegroups": [ { "id": "b39392eb-75f8-47f0-b4f3-7d3882930b26", "name": "dc3", "api_name": "dc3", "is_master": "false", "endpoints": [ ], "hostnames": [ ], "hostnames_s3website": [ ], "master_zone": "96f5eca9-425b-4194-a152-86e310e91ddb", "zones": [ { "id": "96f5eca9-425b-4194-a152-86e310e91ddb", "name": "dc3", "endpoints": [ ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "5d6f2ea4-b84a-459b-bce2-bccac338b3ef", "sync_policy": { "groups": [] } }, ... -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

3 5

Delete or move files from lost+found in cephfs
by Thomas Widhalm 04 Jul '23

04 Jul '23

Hi, I had some trouble in the past with my CephFS which I was able to resolve - mostly with your help. Now I have about 150GB of data in lost+found in my CephFS. No matter what I try and how I change permissions, every time when I try to delete or move something from there I only get the reply: "mv: cannot remove 'lost+found/100002b047c': Read-only file system". I searched the web and configuration items but I didn't find a way to get rid of these files. I copyied most of them to another place, identified them and have them back. So in lost+found there are mostly useless copies. Cheers, Thomas -- http://www.widhalm.or.at GnuPG : 6265BAE6 , A84CB603 Threema: H7AV7D33 Telegram, Signal: widhalmt(a)widhalm.or.at

2 2

upgrading from 15.2.17 to 16.2.11 - Health ERROR
by xadhoom76＠gmail.com 03 Jul '23

03 Jul '23

hi , starting upgrade from 15.2.17 i got this error Module 'cephadm' has failed: Expecting value: line 1 column 1 (char 0) Cluster was in health ok before starting.

5 9

db/wal pvmoved ok, but gui show old metadatas
by Christophe BAILLON 03 Jul '23

03 Jul '23

Hello, we have a Ceph 17.2.5 cluster with a total of 26 nodes, where 15 nodes that have faulty NVMe drives, where the db/wal resides (one NVMe for the first 6 OSDs and another for the remaining 6). We replaced them with new drives and pvmoved it to avoid losing the OSDs. So far, there are no issues, and the OSDs are functioning properly. ceph see the correct news disks root@node02:/# ceph daemon osd.26 list_devices [ { "device": "/dev/nvme0n1", "device_id": "INTEL_SSDPEDME016T4S_CVMD516500851P6KGN" }, { "device": "/dev/sdc", "device_id": "SEAGATE_ST18000NM004J_ZR52TT830000C148JFSJ" } ] However, the Cephadm GUI still shows the old NVMe drives and hasn't recognized the device change. How can we make the GUI and Cephadm recognize the new devices? I tried restarting the managers, thinking that it would rescan the OSDs during startup, but it didn't work. If you have any ideas, I would appreciate it. Should I perform something like that: ceph orch daemon reconfig osd.* Thank you for your help.

1 1

Get bucket placement target
by mahnoosh shahidi 03 Jul '23

03 Jul '23

Hi all, Is there any way for clients (without rgw-admin access) to get the placement target of their S3 buckets? The "GetBucketLocation'' api returns "default" for all placement targets and I couldn't find any other S3 api for this purpose. Can anyone help me with this?

3 5

list of rgw instances in ceph status
by Boris Behrens 03 Jul '23

03 Jul '23

Hi, might be a dump question, but is there a way to list the rgw instances that are running in a ceph cluster? Before pacific it showed up in `ceph status` but now it only tells me how many daemons are active, now which daemons are active. ceph orch ls tells me that I need to configure a backend but we are not at the stage that we are going to implement the orchestrator yet. Cheers Boris

2 2

dashboard for rgw NoSuchKey
by farhad kh 03 Jul '23

03 Jul '23

I deploy the rgw service and the default pool is created automatically But I get an error in the dashboard `` Error connecting to Object Gateway: RGW REST API request failed with default 404 status code","HostId":"736528-default-default"}') `` There is a dashboard user but I created the bucket manually # radosgw-admin user info --uid=dashboard { "user_id": "dashboard", "display_name": "Ceph Dashboard", "email": "", "suspended": 0, "max_buckets": 1000, "subusers": [], "keys": [ { "user": "dashboard", "access_key": "C8YG708VBA3M3AAJW2U2", "secret_key": "NpkmIZ5JJVnu3EFa0ytv5vO64NGttK9ks7A3gEQP" } ], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "system": "true", "default_placement": "", "default_storage_class": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw", "mfa_ids": [] } ---------------- # radosgw-admin buckets list [ "dashboard" ] How can I solve the problem?

1 0

Re: Transmit rate metric based per bucket
by Ondřej Kukla 03 Jul '23

03 Jul '23

Well in fact it does. For example in our setup we are parsing the bucket name from the URL. It’s a bit tricky as a client could use both the domain name and path base styles, but that is not an issue for us. Alternatively you can parse and analyse logs directly from RGWs which have the bucket information without URL parsing needed. Ondrej > On 3. 7. 2023, at 9:58, Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> wrote: > > Hi, > > Yeah, I'm using haproxy but that doesn't have the bucket information :/ > > Istvan Szabo > Staff Infrastructure Engineer > --------------------------------------------------- > Agoda Services Co., Ltd. > e: istvan.szabo(a)agoda.com > --------------------------------------------------- > > -----Original Message----- > From: Ondřej Kukla <ondrej(a)kuuk.la> > Sent: Monday, July 3, 2023 2:25 PM > To: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> > Subject: Re: [ceph-users] Transmit rate metric based per bucket > > Email received from the internet. If in doubt, don't click any link nor open any attachment ! > ________________________________ > > Hello, Istvan, > > As far as I’m avare there is no way to do it directly on the RGW. What you can do is using a LB in front of the RGWs something like Nginx or HAProxy. > > Then just collect the LB logs using ELK stack, Loki etc. and sum the body bytes sent per request type and bucket. > > Ondrej > >> On 20. 6. 2023, at 7:01, Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> wrote: >> >> Hello, >> >> I'd like to know is there a way to query some metrics/logs in octopus (or if has newer version I'm interested for the future too) about the bandwidth used in the bucket for put/get operations? >> >> Thank you >> >> ________________________________ >> This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an >> email to ceph-users-leave(a)ceph.io >

1 0

RadosGW strange behavior when using a presigned url generated by SDK PHP
by Huy Nguyen 02 Jul '23

02 Jul '23

Hi, I tried to generate a presigned url using SDK PHP, but it doesn't work. (I also tried to use boto3 with the same configures and the url works normally) Here is my php code: <?php require 'aws-autoloader.php'; use Aws\S3\S3Client; use Aws\Exception\AwsException; $s3Client = new Aws\S3\S3Client([ 'version' => '2006-03-01', 'region' => 'us-east-1', 'signature_version' => 'v4', 'use_path_style_endpoint' => true, 'endpoint' => 'http://hn.ss.bfcplatform.vn', 'credentials' => [ 'key' => 'DNMZAFE6G2PP8H9P05UU', 'secret' => 'XXX', ] ]); $cmd = $s3Client->getCommand('PutObject', [ 'Bucket' => 'huynnp-testbucket1', 'Key' => 'testfile.txt', ]); $request = $s3Client->createPresignedRequest($cmd, '+60 minutes'); // Set the expiration time as desired $presignedUrl = (string)$request->getUri(); echo "$presignedUrl"; ?> and then: curl -X PUT -T testfile.txt `php s3.php` <?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><RequestId>tx00000b7bb3b2deb6a6ef2-00649d5ebd-d1d50041-hn-1</RequestId><HostId>d1d50041-hn-1-hn</HostId></Error> I enable the debug_rgw and what I can see is really strange. the domain has been added :8084, so it make "canonical request hash" and "signature" between client and server unmatched. I can't explain why does this happens 2023-06-29T17:10:46.880+0700 7f26014b0700 10 v4 credential format = DNMZAFE6G2PP8H9P05UU/20230629/us-east-1/s3/aws4_request 2023-06-29T17:10:46.880+0700 7f26014b0700 10 access key id = DNMZAFE6G2PP8H9P05UU 2023-06-29T17:10:46.880+0700 7f26014b0700 10 credential scope = 20230629/us-east-1/s3/aws4_request 2023-06-29T17:10:46.880+0700 7f26014b0700 10 req 15647562574720867919 1000005ns canonical headers format = host:hn.ss.bfcplatform.vn:8084 2023-06-29T17:10:46.880+0700 7f26014b0700 10 payload request hash = UNSIGNED-PAYLOAD 2023-06-29T17:10:46.880+0700 7f26014b0700 10 canonical request = PUT /huynnp-testbucket1/testfile.txt X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=DNMZAFE6G2PP8H9P05UU%2F20230629%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230629T101046Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host host:hn.ss.bfcplatform.vn:8084 host UNSIGNED-PAYLOAD 2023-06-29T17:10:46.880+0700 7f26014b0700 10 canonical request hash = d28e6c3104aff99e9928f902892627d2b284a29d489fbb034ed5c90aa21c566a 2023-06-29T17:10:46.880+0700 7f26014b0700 10 string to sign = AWS4-HMAC-SHA256 20230629T101046Z 20230629/us-east-1/s3/aws4_request d28e6c3104aff99e9928f902892627d2b284a29d489fbb034ed5c90aa21c566a

2 4