- ceph-users - lists.ceph.io

Ceph Quarterly (CQ) - Issue #1
by Zac Dover 06 Jul '23

06 Jul '23

The first issue of "Ceph Quarterly" is attached to this email. Ceph Quarterly (or "CQ") is an overview of the past three months of upstream Ceph development. We provide CQ in three formats: A4, letter, and plain text wrapped at 80 columns. Zac Dover Upstream Documentation Ceph Foundation

2 1

Rook on bare-metal?
by zssas＠tuta.io 06 Jul '23

06 Jul '23

Hello! I am looking to simplify ceph management on bare-metal by deploying Rook onto kubernetes that has been deployed on bare metal (rke). I have used rook in a cloud environment but I have not used it on bare-metal. I am wondering if anyone here runs rook in bare-metal? Would you recommend it to cephadm or would you steer clear of it?

5 4

RGW accessing real source IP address of a client (e.g. in S3 bucket policies)
by Christian Rohmann 06 Jul '23

06 Jul '23

Hello Ceph-Users, context or motivation of my question is S3 bucket policies and other cases using the source IP address as condition. I was wondering if and how RadosGW is able to access the source IP address of clients if receiving their connections via a loadbalancer / reverse proxy like HAProxy. So naturally that is where the connection originates from in that case, rendering a policy based on IP addresses useless. Depending on whether the connection balanced as HTTP or TCP there are two ways to carry information about the actual source: * In case of HTTP via headers like "X-Forwarded-For". This is apparently supported only for logging the source in the "rgw ops log" ([1])? Or is this info used also when evaluating the source IP condition within a bucket policy? * In case of TCP loadbalancing, there is the proxy protocol v2. This unfortunately seems not even supposed by the BEAST library which RGW uses. I opened feature requests ... ** https://tracker.ceph.com/issues/59422 ** https://github.com/chriskohlhoff/asio/issues/1091 ** https://github.com/boostorg/beast/issues/2484 but there is no outcome yet. Regards Christian [1] https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_remote_addr…

3 5

[multisite] The purpose of zonegroup
by Yixin Jin 05 Jul '23

05 Jul '23

Hi folks, In the multisite environment, we can get one realm that contains multiple zonegroups, each in turn can have multiple zones. However, the purpose of zonegroup isn't clear to me. It seems that when a user is created, its metadata is synced to all zones within the same realm, regardless whether they are in different zonegroups or not. The same happens to buckets. Therefore, what is the purpose of having zonegroups? Wouldn't it be easier to just have realm and zones? Thanks,Yixin

4 6

CLT Meeting minutes 2023-07-05
by Radoslaw Zarzynski 05 Jul '23

05 Jul '23

Hello! Releasing Reef --------------------- * RC2 is out but we still have several PRs to go, including blockers. * RC3 might be worth doing but we Reef shall go before end of the month. Misc ------- * For the sake of unittesting of dencoders interoperatbility we're going to impose some extra work (like registering types within ceph-dencoder) on developers writing encodable structs. This will be discussed further in a CDM. * A lab issue got fixed. Regards Radek

1 0

Slow ACL Changes in Secondary Zone
by Ramin Najjarbashi 05 Jul '23

05 Jul '23

Dear Ceph community, I'm facing an issue with ACL changes in the secondary zone of my Ceph cluster after making modifications to the API name in the master zone of my master zonegroup. I would appreciate any insights or suggestions on how to resolve this problem. Here's the background information on my setup: - I have two clusters that are part of a single realm. - Each cluster has a zone within a single zonegroup. - Initially, all functionality was working perfectly fine. However, the problem arose when I changed the API name in the master zone of my master zonegroup. Since then, all functionalities appear to be functioning as expected, except for ACL changes, which have become extremely slow specifically in the secondary zone. Whenever I attempt to change the ACL in the secondary zone, it takes approximately 10 seconds or more for a response to be received. I would like to understand why this delay is occurring and find a solution to improve the performance of ACL changes in the secondary zone. Any suggestions, explanations, or guidance would be greatly appreciated. Thank you in advance for your help.

1 0

Mishap after disk replacement, db and block split into separate OSD's in ceph-volume
by Mikael Öhman 05 Jul '23

05 Jul '23

We had a fauly disk which was causing many errors, and replacement took a while so we had to try to stop ceph from using the OSD in during this time. However I think we must have done that wrong and after the disk replacement our ceph orch seems to have picked up /dev/sdp and added the a new osd and automatically (588), without a separate DB device (since that was still taken by the old OSD 31 maybe? I'm not sure how to ). This led to issues where osd31 of course wouldn't start, and some actions were attempted to clear this out, which might have just caused more harm. Long story short, we are currently in a odd position where we have still have ceph-volume lvm list osd.31 with only a [db] section: ====== osd.31 ====== [db] /dev/ceph-1b309b1e-a4a6-4861-b16c-7c06ecde1a3d/osd-db-fb09a714-f955-4418-99f2-6bccd8c6220e block device /dev/ceph-48f7dbd8-4a7c-4f7e-8962-104e756ae864/osd-block-33538b36-52b3-421d-bf66-6c729a057707 block uuid bykFYi-z8T6-OWXp-i1OB-H7CE-uLDm-Td6QTI cephx lockbox secret cluster fsid 5406fed0-d52b-11ec-beff-7ed30a54847b cluster name ceph crush device class None db device /dev/ceph-1b309b1e-a4a6-4861-b16c-7c06ecde1a3d/osd-db-fb09a714-f955-4418-99f2-6bccd8c6220e db uuid Vy3aOA-qseQ-RIDT-741e-z7o0-y376-kKTXRE encrypted 0 osd fsid 33538b36-52b3-421d-bf66-6c729a057707 osd id 31 osdspec affinity osd_spec type db vdo 0 devices /dev/nvme0n1 and a seperate extra osd.588 (which is running) which has taken only the [block] device ===== osd.588 ====== [block] /dev/ceph-f63ef837-3b18-47a4-be55-d5c2c0db8927/osd-block-58b33b8f-9623-46b3-a86a-3061602a76b5 block device /dev/ceph-f63ef837-3b18-47a4-be55-d5c2c0db8927/osd-block-58b33b8f-9623-46b3-a86a-3061602a76b5 block uuid KYHzBq-zgJJ-Nw93-j7Jx-Oz5i-BMuU-ndtTCH cephx lockbox secret cluster fsid 5406fed0-d52b-11ec-beff-7ed30a54847b cluster name ceph crush device class encrypted 0 osd fsid 58b33b8f-9623-46b3-a86a-3061602a76b5 osd id 588 osdspec affinity all-available-devices type block vdo 0 devices /dev/sdp I figured the best action was to clear out both of these faulty OSDs via orch "ceph orch osd rm XX" but osd 31 isn't recognized [ceph: root@mimer-osd01 /]# ceph orch osd rm 31 Unable to find OSDs: ['31'] Deleting 588 is recognized. Should I attempt to clear out the osd.31 from ceph-volume manually? I'd really like to get back to a situation where I have osd.31 with the osd fsid that matches the device names, with /dev/sdp and /dev/nmve0n1 but I'm really afraid of just breaking things even more. From what i can see from files laying around, the OSD spec we have is simply: placement: host_pattern: "mimer-osd01" service_id: osd_spec service_type: osd spec: data_devices: rotational: 1 db_devices: rotational: 0 in case this matters. I appreciate any help or guidance. Best regards, Mikael

1 0

What is the best way to use disks with different sizes
by wodel youchi 04 Jul '23

04 Jul '23

Hi, I will be deploying a Proxmox HCI cluster with 3 nodes. Each node has 3 nvme disks of 3.8Tb each and a 4th nvme disk of 7.6Tb. Technically I need one pool. Is it good practice to use all disks to create the one pool I need, or is it better to create two pools, one on each group of disks? If the former is good (use all disks and create one pool), should I take into account the difference in disk size? Regards.

3 4

Erasure coding and backfilling speed
by jesper＠krogh.cc 04 Jul '23

04 Jul '23

Hi. I have a Ceph (NVME) based cluster with 12 hosts and 40 OSD's .. currently it is backfilling pg's but I cannot get it to run more than 20 backfilling (pgs) at the same time (6+2 profile) osd_max_backfills = 100 and osd_recovery_max_active_ssd = 50 (non-sane) but it still stops at 20 with 40+ in backfill_wait Any idea about how to speed it up? Thanks.

1 0

Ceph iSCSI GW not working with VMware VMFS and Windows Clustered Storage Volumes (CSV)
by Work Ceph 04 Jul '23

04 Jul '23

Hello guys, We have a Ceph cluster that runs just fine with Ceph Octopus; we use RBD for some workloads, RadosGW (via S3) for others, and iSCSI for some Windows clients. Recently, we had the need to add some VMWare clusters as clients for the iSCSI GW and also Windows systems with the use of Clustered Storage Volumes (CSV), and we are facing a weird situation. In windows for instance, the iSCSI block can be mounted, formatted and consumed by all nodes, but when we add in the CSV it fails with some generic exception. The same happens in VMWare, when we try to use it with VMFS it fails. We do not seem to find the root cause for these errors. However, the errors seem to be linked to the situation of multiple nodes consuming the same block by shared file systems. Have you guys seen this before? Are we missing some basic configuration in the iSCSI GW?

4 7