July 2023 - ceph-users - lists.ceph.io

Cannot get backfill speed up
by Jesper Krogh 07 Jul '23

07 Jul '23

Hi. Fresh cluster - but despite setting: jskr@dkcphhpcmgt028:/$ sudo ceph config show osd.0 | grep recovery_max_active_ssd osd_recovery_max_active_ssd 50 mon default[20] jskr@dkcphhpcmgt028:/$ sudo ceph config show osd.0 | grep osd_max_backfills osd_max_backfills 100 mon default[10] I still get jskr@dkcphhpcmgt028:/$ sudo ceph status cluster: id: 5c384430-da91-11ed-af9c-c780a5227aff health: HEALTH_OK services: mon: 3 daemons, quorum dkcphhpcmgt031,dkcphhpcmgt029,dkcphhpcmgt028 (age 16h) mgr: dkcphhpcmgt031.afbgjx(active, since 33h), standbys: dkcphhpcmgt029.bnsegi, dkcphhpcmgt028.bxxkqd mds: 2/2 daemons up, 1 standby osd: 40 osds: 40 up (since 45h), 40 in (since 39h); 21 remapped pgs data: volumes: 2/2 healthy pools: 9 pools, 495 pgs objects: 24.85M objects, 60 TiB usage: 117 TiB used, 159 TiB / 276 TiB avail pgs: 10655690/145764002 objects misplaced (7.310%) 474 active+clean 15 active+remapped+backfilling 6 active+remapped+backfill_wait io: client: 0 B/s rd, 1.4 MiB/s wr, 0 op/s rd, 116 op/s wr recovery: 328 MiB/s, 108 objects/s progress: Global Recovery Event (9h) [==========================..] (remaining: 25m) With these numbers for the setting - I would expect to get more than 15 active backfilling... (and based on SSD's and 2x25gbit network, I can also spend more resources on recovery than 328 MiB/s Thanks, . -- Jesper Krogh

4 3

MDSs report slow metadata IOs
by Ben 07 Jul '23

07 Jul '23

Hi, see many of this in cluster log channel. many are blocked with long period of seconds. It should hurt client access performance. Any ideas to get rid of them? Thanks, Ben --------------------------------- 7/7/23 4:48:50 PM [WRN] Health check update: 8 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 7/7/23 4:48:09 PM [WRN] Health check update: 7 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 7/7/23 4:48:09 PM [INF] MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 30 secs, oldest blocked for 559 secs 7/7/23 4:47:47 PM [WRN] Health check update: 8 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 7/7/23 4:47:11 PM [WRN] Health check update: 7 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 7/7/23 4:47:10 PM [INF] MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 30 secs, oldest blocked for 377 secs 7/7/23 4:46:22 PM [WRN] Health check update: 8 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 7/7/23 4:46:12 PM [WRN] Health check update: 7 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 7/7/23 4:45:40 PM [WRN] Health check update: 6 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 7/7/23 4:45:40 PM [INF] MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 30 secs, oldest blocked for 199 secs 7/7/23 4:45:12 PM [WRN] Health check update: 7 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 7/7/23 4:45:07 PM [WRN] Health check update: 6 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 7/7/23 4:44:58 PM [INF] MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 30 secs, oldest blocked for 565 secs 7/7/23 4:44:58 PM [WRN] Health check update: 7 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)

2 1

pg_num != pgp_num - and unable to change.
by Jesper Krogh 06 Jul '23

06 Jul '23

Hi. Fresh cluster - after a dance where the autoscaler did not work (returned black) as described in the doc - I now seemingly have it working. It has bumpted target to something reasonable -- and is slowly incrementing pg_num and pgp_num by 2 over time (hope this is correct?) But . jskr@dkcphhpcmgt028:/$ sudo ceph osd pool ls detail | grep 62 pool 22 'cephfs.archive.ec62data' erasure profile ecprof62 size 8 min_size 7 crush_rule 3 object_hash rjenkins pg_num 150 pgp_num 22 pg_num_target 512 pgp_num_target 512 autoscale_mode on last_change 9159 lfor 0/0/9147 flags hashpspool,ec_overwrites,selfmanaged_snaps,bulk stripe_width 24576 pg_num_min 128 target_size_ratio 0.4 application cephfs pg_num = 150 pgp_num = 22 and setting pgp_num seemingly have zero effect on the system .. not even with autoscaling set to off. jskr@dkcphhpcmgt028:/$ sudo ceph osd pool set cephfs.archive.ec62data pg_autoscale_mode off set pool 22 pg_autoscale_mode to off jskr@dkcphhpcmgt028:/$ sudo ceph osd pool set cephfs.archive.ec62data pgp_num 150 set pool 22 pgp_num to 150 jskr@dkcphhpcmgt028:/$ sudo ceph osd pool set cephfs.archive.ec62data pg_num_min 128 set pool 22 pg_num_min to 128 jskr@dkcphhpcmgt028:/$ sudo ceph osd pool set cephfs.archive.ec62data pg_num 150 set pool 22 pg_num to 150 jskr@dkcphhpcmgt028:/$ sudo ceph osd pool set cephfs.archive.ec62data pg_autoscale_mode on set pool 22 pg_autoscale_mode to on jskr@dkcphhpcmgt028:/$ sudo ceph progress PG autoscaler increasing pool 22 PGs from 150 to 512 (14s) [............................] jskr@dkcphhpcmgt028:/$ sudo ceph osd pool ls detail | grep 62 pool 22 'cephfs.archive.ec62data' erasure profile ecprof62 size 8 min_size 7 crush_rule 3 object_hash rjenkins pg_num 150 pgp_num 22 pg_num_target 512 pgp_num_target 512 autoscale_mode on last_change 9159 lfor 0/0/9147 flags hashpspool,ec_overwrites,selfmanaged_snaps,bulk stripe_width 24576 pg_num_min 128 target_size_ratio 0.4 application cephfs pgp_num != pg_num ? In earlier versions of ceph (without autoscaler) I have only experienced that setting pg_num and pgp_num took immidiate effect? Jesper jskr@dkcphhpcmgt028:/$ sudo ceph version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) jskr@dkcphhpcmgt028:/$ sudo ceph health HEALTH_OK jskr@dkcphhpcmgt028:/$ sudo ceph status cluster: id: 5c384430-da91-11ed-af9c-c780a5227aff health: HEALTH_OK services: mon: 3 daemons, quorum dkcphhpcmgt031,dkcphhpcmgt029,dkcphhpcmgt028 (age 15h) mgr: dkcphhpcmgt031.afbgjx(active, since 32h), standbys: dkcphhpcmgt029.bnsegi, dkcphhpcmgt028.bxxkqd mds: 2/2 daemons up, 1 standby osd: 40 osds: 40 up (since 44h), 40 in (since 39h); 33 remapped pgs data: volumes: 2/2 healthy pools: 9 pools, 495 pgs objects: 24.85M objects, 60 TiB usage: 117 TiB used, 158 TiB / 276 TiB avail pgs: 13494029/145763897 objects misplaced (9.257%) 462 active+clean 23 active+remapped+backfilling 10 active+remapped+backfill_wait io: client: 0 B/s rd, 1.1 MiB/s wr, 0 op/s rd, 94 op/s wr recovery: 705 MiB/s, 208 objects/s progress: -- Jesper Krogh

3 2

CephFS snapshots: impact of moving data
by Kuhring, Mathias 06 Jul '23

06 Jul '23

Dear Ceph community, We want to restructure (i.e. move around) a lot of data (hundreds of terabyte) in our CephFS. And now I was wondering what happens within snapshots when I move data around within a snapshotted folder. I.e. do I need to account for a lot increased storage usage due to older snapshots differing from the new restructured state? In the end it is just metadata changes. Are the snapshots aware of this? Consider the following examples. Copying data: Let's say I have a folder /test, with a file XYZ in sub-folder /test/sub1 and an empty sub-folder /test/sub2. I create snapshot snapA in /test/.snap, copy XYZ to sub-folder /test/sub2, delete it from /test/sub1 and create another snapshot snapB. I would have two snapshots each with distinct copies of XYZ, hence using double the space in the FS: /test/.snap/snapA/sub1/XYZ <-- copy 1 /test/.snap/snapA/sub2/ /test/.snap/snapB/sub1/ /test/.snap/snapB/sub2/XYZ <-- copy 2 Moving data: Let's assume the same structure. But now after creating snapshot snapA, I move XYZ to sub-folder /test/sub2 and then create the other snapshot snapB. The directory tree will look the same. But how is this treated internally? Once I move the data, will there be an actually copy created in snapA to represent the old state? Or will this remain the same data (like a link to the inode or so)? And hence not double the storage used for that file. I couldn't find (or understand) anything related to this in the docs. The closest seems to be the hard-link section here: https://docs.ceph.com/en/quincy/dev/cephfs-snapshots/#hard-links Which unfortunately goes a bit over my head. So I'm not sure if this answers my question. Thank you all for your help. Appreciate it. Best Wishes, Mathias Kuhring

3 2

Ceph Quarterly (CQ) - Issue #1
by Zac Dover 06 Jul '23

06 Jul '23

The first issue of "Ceph Quarterly" is attached to this email. Ceph Quarterly (or "CQ") is an overview of the past three months of upstream Ceph development. We provide CQ in three formats: A4, letter, and plain text wrapped at 80 columns. Zac Dover Upstream Documentation Ceph Foundation

2 1

Rook on bare-metal?
by zssas＠tuta.io 06 Jul '23

06 Jul '23

Hello! I am looking to simplify ceph management on bare-metal by deploying Rook onto kubernetes that has been deployed on bare metal (rke). I have used rook in a cloud environment but I have not used it on bare-metal. I am wondering if anyone here runs rook in bare-metal? Would you recommend it to cephadm or would you steer clear of it?

5 4

RGW accessing real source IP address of a client (e.g. in S3 bucket policies)
by Christian Rohmann 06 Jul '23

06 Jul '23

Hello Ceph-Users, context or motivation of my question is S3 bucket policies and other cases using the source IP address as condition. I was wondering if and how RadosGW is able to access the source IP address of clients if receiving their connections via a loadbalancer / reverse proxy like HAProxy. So naturally that is where the connection originates from in that case, rendering a policy based on IP addresses useless. Depending on whether the connection balanced as HTTP or TCP there are two ways to carry information about the actual source: * In case of HTTP via headers like "X-Forwarded-For". This is apparently supported only for logging the source in the "rgw ops log" ([1])? Or is this info used also when evaluating the source IP condition within a bucket policy? * In case of TCP loadbalancing, there is the proxy protocol v2. This unfortunately seems not even supposed by the BEAST library which RGW uses. I opened feature requests ... ** https://tracker.ceph.com/issues/59422 ** https://github.com/chriskohlhoff/asio/issues/1091 ** https://github.com/boostorg/beast/issues/2484 but there is no outcome yet. Regards Christian [1] https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_remote_addr…

3 5

[multisite] The purpose of zonegroup
by Yixin Jin 05 Jul '23

05 Jul '23

Hi folks, In the multisite environment, we can get one realm that contains multiple zonegroups, each in turn can have multiple zones. However, the purpose of zonegroup isn't clear to me. It seems that when a user is created, its metadata is synced to all zones within the same realm, regardless whether they are in different zonegroups or not. The same happens to buckets. Therefore, what is the purpose of having zonegroups? Wouldn't it be easier to just have realm and zones? Thanks,Yixin

4 6

CLT Meeting minutes 2023-07-05
by Radoslaw Zarzynski 05 Jul '23

05 Jul '23

Hello! Releasing Reef --------------------- * RC2 is out but we still have several PRs to go, including blockers. * RC3 might be worth doing but we Reef shall go before end of the month. Misc ------- * For the sake of unittesting of dencoders interoperatbility we're going to impose some extra work (like registering types within ceph-dencoder) on developers writing encodable structs. This will be discussed further in a CDM. * A lab issue got fixed. Regards Radek

1 0

Slow ACL Changes in Secondary Zone
by Ramin Najjarbashi 05 Jul '23

05 Jul '23

Dear Ceph community, I'm facing an issue with ACL changes in the secondary zone of my Ceph cluster after making modifications to the API name in the master zone of my master zonegroup. I would appreciate any insights or suggestions on how to resolve this problem. Here's the background information on my setup: - I have two clusters that are part of a single realm. - Each cluster has a zone within a single zonegroup. - Initially, all functionality was working perfectly fine. However, the problem arose when I changed the API name in the master zone of my master zonegroup. Since then, all functionalities appear to be functioning as expected, except for ACL changes, which have become extremely slow specifically in the secondary zone. Whenever I attempt to change the ACL in the secondary zone, it takes approximately 10 seconds or more for a response to be received. I would like to understand why this delay is occurring and find a solution to improve the performance of ACL changes in the secondary zone. Any suggestions, explanations, or guidance would be greatly appreciated. Thank you in advance for your help.

1 0

Mishap after disk replacement, db and block split into separate OSD's in ceph-volume
by Mikael Öhman 05 Jul '23

05 Jul '23

We had a fauly disk which was causing many errors, and replacement took a while so we had to try to stop ceph from using the OSD in during this time. However I think we must have done that wrong and after the disk replacement our ceph orch seems to have picked up /dev/sdp and added the a new osd and automatically (588), without a separate DB device (since that was still taken by the old OSD 31 maybe? I'm not sure how to ). This led to issues where osd31 of course wouldn't start, and some actions were attempted to clear this out, which might have just caused more harm. Long story short, we are currently in a odd position where we have still have ceph-volume lvm list osd.31 with only a [db] section: ====== osd.31 ====== [db] /dev/ceph-1b309b1e-a4a6-4861-b16c-7c06ecde1a3d/osd-db-fb09a714-f955-4418-99f2-6bccd8c6220e block device /dev/ceph-48f7dbd8-4a7c-4f7e-8962-104e756ae864/osd-block-33538b36-52b3-421d-bf66-6c729a057707 block uuid bykFYi-z8T6-OWXp-i1OB-H7CE-uLDm-Td6QTI cephx lockbox secret cluster fsid 5406fed0-d52b-11ec-beff-7ed30a54847b cluster name ceph crush device class None db device /dev/ceph-1b309b1e-a4a6-4861-b16c-7c06ecde1a3d/osd-db-fb09a714-f955-4418-99f2-6bccd8c6220e db uuid Vy3aOA-qseQ-RIDT-741e-z7o0-y376-kKTXRE encrypted 0 osd fsid 33538b36-52b3-421d-bf66-6c729a057707 osd id 31 osdspec affinity osd_spec type db vdo 0 devices /dev/nvme0n1 and a seperate extra osd.588 (which is running) which has taken only the [block] device ===== osd.588 ====== [block] /dev/ceph-f63ef837-3b18-47a4-be55-d5c2c0db8927/osd-block-58b33b8f-9623-46b3-a86a-3061602a76b5 block device /dev/ceph-f63ef837-3b18-47a4-be55-d5c2c0db8927/osd-block-58b33b8f-9623-46b3-a86a-3061602a76b5 block uuid KYHzBq-zgJJ-Nw93-j7Jx-Oz5i-BMuU-ndtTCH cephx lockbox secret cluster fsid 5406fed0-d52b-11ec-beff-7ed30a54847b cluster name ceph crush device class encrypted 0 osd fsid 58b33b8f-9623-46b3-a86a-3061602a76b5 osd id 588 osdspec affinity all-available-devices type block vdo 0 devices /dev/sdp I figured the best action was to clear out both of these faulty OSDs via orch "ceph orch osd rm XX" but osd 31 isn't recognized [ceph: root@mimer-osd01 /]# ceph orch osd rm 31 Unable to find OSDs: ['31'] Deleting 588 is recognized. Should I attempt to clear out the osd.31 from ceph-volume manually? I'd really like to get back to a situation where I have osd.31 with the osd fsid that matches the device names, with /dev/sdp and /dev/nmve0n1 but I'm really afraid of just breaking things even more. From what i can see from files laying around, the OSD spec we have is simply: placement: host_pattern: "mimer-osd01" service_id: osd_spec service_type: osd spec: data_devices: rotational: 1 db_devices: rotational: 0 in case this matters. I appreciate any help or guidance. Best regards, Mikael

1 0

What is the best way to use disks with different sizes
by wodel youchi 05 Jul '23

05 Jul '23

Hi, I will be deploying a Proxmox HCI cluster with 3 nodes. Each node has 3 nvme disks of 3.8Tb each and a 4th nvme disk of 7.6Tb. Technically I need one pool. Is it good practice to use all disks to create the one pool I need, or is it better to create two pools, one on each group of disks? If the former is good (use all disks and create one pool), should I take into account the difference in disk size? Regards.

3 4

Erasure coding and backfilling speed
by jesper＠krogh.cc 04 Jul '23

04 Jul '23

Hi. I have a Ceph (NVME) based cluster with 12 hosts and 40 OSD's .. currently it is backfilling pg's but I cannot get it to run more than 20 backfilling (pgs) at the same time (6+2 profile) osd_max_backfills = 100 and osd_recovery_max_active_ssd = 50 (non-sane) but it still stops at 20 with 40+ in backfill_wait Any idea about how to speed it up? Thanks.

1 0

Ceph iSCSI GW not working with VMware VMFS and Windows Clustered Storage Volumes (CSV)
by Work Ceph 04 Jul '23

04 Jul '23

Hello guys, We have a Ceph cluster that runs just fine with Ceph Octopus; we use RBD for some workloads, RadosGW (via S3) for others, and iSCSI for some Windows clients. Recently, we had the need to add some VMWare clusters as clients for the iSCSI GW and also Windows systems with the use of Clustered Storage Volumes (CSV), and we are facing a weird situation. In windows for instance, the iSCSI block can be mounted, formatted and consumed by all nodes, but when we add in the CSV it fails with some generic exception. The same happens in VMWare, when we try to use it with VMFS it fails. We do not seem to find the root cause for these errors. However, the errors seem to be linked to the situation of multiple nodes consuming the same block by shared file systems. Have you guys seen this before? Are we missing some basic configuration in the iSCSI GW?

4 7

Re: Ceph iSCSI GW is too slow when compared with Raw RBD performance
by Work Ceph 04 Jul '23

04 Jul '23

Awesome, thanks for the info! By any chance, do you happen to know what configurations you needed to adjust to make Veeam perform a bit better? On Fri, Jun 23, 2023 at 10:42 AM Anthony D'Atri <aad(a)dreamsnake.net> wrote: > Yes, with someone I did some consulting for. Veeam seems to be one of the > prevalent uses for ceph-iscsi, though I'd try to use the native RBD client > instead if possible. > > Veeam appears by default to store really tiny blocks, so there's a lot of > protocol overhead. I understand that Veeam can be configured to use "large > blocks" that can make a distinct difference. > > > > On Jun 23, 2023, at 09:33, Work Ceph <work.ceph.user.mailing(a)gmail.com> > wrote: > > Great question! > > Yes, one of the slowness was detected in a Veeam setup. Have you > experienced that before? > > On Fri, Jun 23, 2023 at 10:32 AM Anthony D'Atri <aad(a)dreamsnake.net> > wrote: > >> Are you using Veeam by chance? >> >> > On Jun 22, 2023, at 21:18, Work Ceph <work.ceph.user.mailing(a)gmail.com> >> wrote: >> > >> > Hello guys, >> > >> > We have a Ceph cluster that runs just fine with Ceph Octopus; we use RBD >> > for some workloads, RadosGW (via S3) for others, and iSCSI for some >> Windows >> > clients. >> > >> > We started noticing some unexpected performance issues with iSCSI. I >> mean, >> > an SSD pool is reaching 100MB of write speed for an image, when it can >> > reach up to 600MB+ of write speed for the same image when mounted and >> > consumed directly via RBD. >> > >> > Is that performance degradation expected? We would expect some >> degradation, >> > but not as much as this one. >> > >> > Also, we have a question regarding the use of Intel Turbo boost. Should >> we >> > disable it? Is it possible that the root cause of the slowness in the >> iSCSI >> > GW is caused by the use of Intel Turbo boost feature, which reduces the >> > clock of some cores? >> > >> > Any feedback is much appreciated. >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >

2 2

radosgw new zonegroup hammers master with metadata sync
by Boris Behrens 04 Jul '23

04 Jul '23

Hi, yesterday I added a new zonegroup and it looks like it seems to cycle over the same requests over and over again. In the log of the main zone I see these requests: 2023-06-20T09:48:37.979+0000 7f8941fb3700 1 beast: 0x7f8a602f3700: fd00:2380:0:24::136 - - [2023-06-20T09:48:37.979941+0000] "GET /admin/log?type=metadata&id=62&period=e8fc96f1-ae86-4dc1-b432-470b0772fded&max-entries=100&&rgwx-zonegroup=b39392eb-75f8-47f0-b4f3-7d3882930b26 HTTP/1.1" 200 44 - - - Only thing that changes is the &id. We have two other zonegroups that are configured identical (ceph.conf and period) and these don;t seem to spam the main rgw. root@host:~# radosgw-admin sync status realm 5d6f2ea4-b84a-459b-bce2-bccac338b3ef (main) zonegroup b39392eb-75f8-47f0-b4f3-7d3882930b26 (dc3) zone 96f5eca9-425b-4194-a152-86e310e91ddb (dc3) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master root@host:~# radosgw-admin period get { "id": "e8fc96f1-ae86-4dc1-b432-470b0772fded", "epoch": 92, "predecessor_uuid": "5349ac85-3d6d-4088-993f-7a1d4be3835a", "sync_status": [ "", ... "" ], "period_map": { "id": "e8fc96f1-ae86-4dc1-b432-470b0772fded", "zonegroups": [ { "id": "b39392eb-75f8-47f0-b4f3-7d3882930b26", "name": "dc3", "api_name": "dc3", "is_master": "false", "endpoints": [ ], "hostnames": [ ], "hostnames_s3website": [ ], "master_zone": "96f5eca9-425b-4194-a152-86e310e91ddb", "zones": [ { "id": "96f5eca9-425b-4194-a152-86e310e91ddb", "name": "dc3", "endpoints": [ ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "5d6f2ea4-b84a-459b-bce2-bccac338b3ef", "sync_policy": { "groups": [] } }, ... -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

3 5

Delete or move files from lost+found in cephfs
by Thomas Widhalm 04 Jul '23

04 Jul '23

Hi, I had some trouble in the past with my CephFS which I was able to resolve - mostly with your help. Now I have about 150GB of data in lost+found in my CephFS. No matter what I try and how I change permissions, every time when I try to delete or move something from there I only get the reply: "mv: cannot remove 'lost+found/100002b047c': Read-only file system". I searched the web and configuration items but I didn't find a way to get rid of these files. I copyied most of them to another place, identified them and have them back. So in lost+found there are mostly useless copies. Cheers, Thomas -- http://www.widhalm.or.at GnuPG : 6265BAE6 , A84CB603 Threema: H7AV7D33 Telegram, Signal: widhalmt(a)widhalm.or.at

2 2

upgrading from 15.2.17 to 16.2.11 - Health ERROR
by xadhoom76＠gmail.com 04 Jul '23

04 Jul '23

hi , starting upgrade from 15.2.17 i got this error Module 'cephadm' has failed: Expecting value: line 1 column 1 (char 0) Cluster was in health ok before starting.

5 9

db/wal pvmoved ok, but gui show old metadatas
by Christophe BAILLON 03 Jul '23

03 Jul '23

Hello, we have a Ceph 17.2.5 cluster with a total of 26 nodes, where 15 nodes that have faulty NVMe drives, where the db/wal resides (one NVMe for the first 6 OSDs and another for the remaining 6). We replaced them with new drives and pvmoved it to avoid losing the OSDs. So far, there are no issues, and the OSDs are functioning properly. ceph see the correct news disks root@node02:/# ceph daemon osd.26 list_devices [ { "device": "/dev/nvme0n1", "device_id": "INTEL_SSDPEDME016T4S_CVMD516500851P6KGN" }, { "device": "/dev/sdc", "device_id": "SEAGATE_ST18000NM004J_ZR52TT830000C148JFSJ" } ] However, the Cephadm GUI still shows the old NVMe drives and hasn't recognized the device change. How can we make the GUI and Cephadm recognize the new devices? I tried restarting the managers, thinking that it would rescan the OSDs during startup, but it didn't work. If you have any ideas, I would appreciate it. Should I perform something like that: ceph orch daemon reconfig osd.* Thank you for your help.

1 1

Get bucket placement target
by mahnoosh shahidi 03 Jul '23

03 Jul '23

Hi all, Is there any way for clients (without rgw-admin access) to get the placement target of their S3 buckets? The "GetBucketLocation'' api returns "default" for all placement targets and I couldn't find any other S3 api for this purpose. Can anyone help me with this?

3 5

list of rgw instances in ceph status
by Boris Behrens 03 Jul '23

03 Jul '23

Hi, might be a dump question, but is there a way to list the rgw instances that are running in a ceph cluster? Before pacific it showed up in `ceph status` but now it only tells me how many daemons are active, now which daemons are active. ceph orch ls tells me that I need to configure a backend but we are not at the stage that we are going to implement the orchestrator yet. Cheers Boris

2 2

dashboard for rgw NoSuchKey
by farhad kh 03 Jul '23

03 Jul '23

I deploy the rgw service and the default pool is created automatically But I get an error in the dashboard `` Error connecting to Object Gateway: RGW REST API request failed with default 404 status code","HostId":"736528-default-default"}') `` There is a dashboard user but I created the bucket manually # radosgw-admin user info --uid=dashboard { "user_id": "dashboard", "display_name": "Ceph Dashboard", "email": "", "suspended": 0, "max_buckets": 1000, "subusers": [], "keys": [ { "user": "dashboard", "access_key": "C8YG708VBA3M3AAJW2U2", "secret_key": "NpkmIZ5JJVnu3EFa0ytv5vO64NGttK9ks7A3gEQP" } ], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "system": "true", "default_placement": "", "default_storage_class": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw", "mfa_ids": [] } ---------------- # radosgw-admin buckets list [ "dashboard" ] How can I solve the problem?

1 0

Re: Transmit rate metric based per bucket
by Ondřej Kukla 03 Jul '23

03 Jul '23

Well in fact it does. For example in our setup we are parsing the bucket name from the URL. It’s a bit tricky as a client could use both the domain name and path base styles, but that is not an issue for us. Alternatively you can parse and analyse logs directly from RGWs which have the bucket information without URL parsing needed. Ondrej > On 3. 7. 2023, at 9:58, Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> wrote: > > Hi, > > Yeah, I'm using haproxy but that doesn't have the bucket information :/ > > Istvan Szabo > Staff Infrastructure Engineer > --------------------------------------------------- > Agoda Services Co., Ltd. > e: istvan.szabo(a)agoda.com > --------------------------------------------------- > > -----Original Message----- > From: Ondřej Kukla <ondrej(a)kuuk.la> > Sent: Monday, July 3, 2023 2:25 PM > To: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> > Subject: Re: [ceph-users] Transmit rate metric based per bucket > > Email received from the internet. If in doubt, don't click any link nor open any attachment ! > ________________________________ > > Hello, Istvan, > > As far as I’m avare there is no way to do it directly on the RGW. What you can do is using a LB in front of the RGWs something like Nginx or HAProxy. > > Then just collect the LB logs using ELK stack, Loki etc. and sum the body bytes sent per request type and bucket. > > Ondrej > >> On 20. 6. 2023, at 7:01, Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> wrote: >> >> Hello, >> >> I'd like to know is there a way to query some metrics/logs in octopus (or if has newer version I'm interested for the future too) about the bandwidth used in the bucket for put/get operations? >> >> Thank you >> >> ________________________________ >> This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an >> email to ceph-users-leave(a)ceph.io >

1 0

RadosGW strange behavior when using a presigned url generated by SDK PHP
by Huy Nguyen 03 Jul '23

03 Jul '23

Hi, I tried to generate a presigned url using SDK PHP, but it doesn't work. (I also tried to use boto3 with the same configures and the url works normally) Here is my php code: <?php require 'aws-autoloader.php'; use Aws\S3\S3Client; use Aws\Exception\AwsException; $s3Client = new Aws\S3\S3Client([ 'version' => '2006-03-01', 'region' => 'us-east-1', 'signature_version' => 'v4', 'use_path_style_endpoint' => true, 'endpoint' => 'http://hn.ss.bfcplatform.vn', 'credentials' => [ 'key' => 'DNMZAFE6G2PP8H9P05UU', 'secret' => 'XXX', ] ]); $cmd = $s3Client->getCommand('PutObject', [ 'Bucket' => 'huynnp-testbucket1', 'Key' => 'testfile.txt', ]); $request = $s3Client->createPresignedRequest($cmd, '+60 minutes'); // Set the expiration time as desired $presignedUrl = (string)$request->getUri(); echo "$presignedUrl"; ?> and then: curl -X PUT -T testfile.txt `php s3.php` <?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><RequestId>tx00000b7bb3b2deb6a6ef2-00649d5ebd-d1d50041-hn-1</RequestId><HostId>d1d50041-hn-1-hn</HostId></Error> I enable the debug_rgw and what I can see is really strange. the domain has been added :8084, so it make "canonical request hash" and "signature" between client and server unmatched. I can't explain why does this happens 2023-06-29T17:10:46.880+0700 7f26014b0700 10 v4 credential format = DNMZAFE6G2PP8H9P05UU/20230629/us-east-1/s3/aws4_request 2023-06-29T17:10:46.880+0700 7f26014b0700 10 access key id = DNMZAFE6G2PP8H9P05UU 2023-06-29T17:10:46.880+0700 7f26014b0700 10 credential scope = 20230629/us-east-1/s3/aws4_request 2023-06-29T17:10:46.880+0700 7f26014b0700 10 req 15647562574720867919 1000005ns canonical headers format = host:hn.ss.bfcplatform.vn:8084 2023-06-29T17:10:46.880+0700 7f26014b0700 10 payload request hash = UNSIGNED-PAYLOAD 2023-06-29T17:10:46.880+0700 7f26014b0700 10 canonical request = PUT /huynnp-testbucket1/testfile.txt X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=DNMZAFE6G2PP8H9P05UU%2F20230629%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230629T101046Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host host:hn.ss.bfcplatform.vn:8084 host UNSIGNED-PAYLOAD 2023-06-29T17:10:46.880+0700 7f26014b0700 10 canonical request hash = d28e6c3104aff99e9928f902892627d2b284a29d489fbb034ed5c90aa21c566a 2023-06-29T17:10:46.880+0700 7f26014b0700 10 string to sign = AWS4-HMAC-SHA256 20230629T101046Z 20230629/us-east-1/s3/aws4_request d28e6c3104aff99e9928f902892627d2b284a29d489fbb034ed5c90aa21c566a

2 4