- ceph-users - lists.ceph.io

Use local reads with rados_replica_read_policy
by Eugen Block 17 Jun '25

17 Jun '25

Hi *, I have a question regarding the upcoming feature to optimize read performance [0] by reading from the nearest OSD, especially in a stretch cluster across two sites (or more). Anthony pointed me to [1], looks like a new config option will be introduced in Tentacle: rados_replica_read_policy Will this config option be limited to the "official" stretch mode? Or will it be possible to utilize it independent of the cluster layout? Thanks! Eugen [0] https://ceph.io/en/news/blog/2025/stretch-cluuuuuuuuusters-part2/ [1] https://github.com/ceph/ceph/blob/d28e5fe890016235e302122f955fc910c96f2d43/…

3 6

19.2.2: large number of scrubs suddenly not completed in time
by Michel Jouvin 17 Jun '25

17 Jun '25

Hi, We upgraded one of our production cluster (480 OSDs, 13,5 K PGs, most pools EC 9+6) from 18.2.7 to 19.2.2 on May 26. It was healthy when we upgraded it and remained so until last Thrusday (June 5) where, doing a `ceph -s`, I saw that there were 1 deep scrub and ~1000 scrubs late: 1 pgs not deep-scrubbed in time 1168 pgs not scrubbed in time Looking again Friday morning (~16 hours later), this number increased a lot: 27 pgs not deep-scrubbed in time 3252 pgs not scrubbed in time Checking our configuration on Friday, we found that osd_max_scrubs was set to 1 instead of using the new default since Reef which is 3 (probably a leftover of a config change after a problem 18 monts ago). We unset the specific value and this led to a reduction of these numbers in the next 24 hours (~2700 scrubs late) but since then (2 days) it remained stable initially and is now increasing slowly. This morning the situation is: 294 pgs not deep-scrubbed in time 3013 pgs not scrubbed in time My guess is that the real issue is the late scrubs that results in many OSD reaching the limit of 3 concurrent scrubs and that it has the consequences that some deep scrubs cannot run too (I've in mind that the limit applies both to shallow scubs and deep scrubs, am I right?). I checked the main cluster logs and didn't find any error or warning, related to OSDs like slow ops, slow requests...The only things we have spotted through our monitoring system is a dramatic decrease (~75%) of IOPS on each OSD server right after the 19.2.2 upgrade but it is not necessarily the sign of a problem. I guess it may in particular be a consequence of the new deep scrubs parameters, osd_shallow_scrub_chunk_min/max, which a probably intended to reduced the deep scrub IOPS load. The release notes for Squid also mention a change in osd_op_num_shards_hdd and osd_op_num_threads_per_shard_hdd, I don't know if they may also have impact. Up to now, no users has reported any issue so it seems to be a problem only with scrubs. I'm wondering where to start looking for an issue or anything related to 19.2.2 is already known. We increased the deep scrub interval from 10 to 14 days a few days before the upgrade (we saw that there was permanently 1 deep scrub late, a different PG all the time) and kept the standard 7 day interval for scrubs. Looking at the number of scrubs and deep scrubs per day, it doesn't look weird (see below). I guess that if we retart all OSDs we'll clear the problem but we'd like to understand what happened and be sure it is not something related to Squid, before upgrade our other production cluster. Any hint/advice will be highly appreciated. I took a snapshot of `ceph pg dump pgs_brief` regularly and I'll try to identify if there are some stucked scrubs and what are the OSDs involved but with 500 OSDs and 18 OSD servers, it may not be obvious... Best regards, Michel _Distribution of scrubs_ (first number is the number of scrubs during the day) (number increases on June 6 after setting osd_max_scrubs=3) 295 "2025-05-25 1578 "2025-05-26 300 "2025-05-27 392 "2025-05-28 578 "2025-05-29 707 "2025-05-30 611 "2025-05-31 819 "2025-06-01 679 "2025-06-02 724 "2025-06-03 698 "2025-06-04 726 "2025-06-05 1577 "2025-06-06 1393 "2025-06-07 1962 "2025-06-08 645 "2025-06-09 _Distribution of deep scrubs per day_ (number increases on June 6 after setting osd_max_scrubs=3 and starts to decrease again on June 8, when the number of late subs increases again, probably because we hit the limit of 3 scrubs per OSD) 22 "2025-05-12 63 "2025-05-13 101 "2025-05-14 127 "2025-05-15 173 "2025-05-16 238 "2025-05-17 305 "2025-05-18 387 "2025-05-19 450 "2025-05-20 564 "2025-05-21 675 "2025-05-22 716 "2025-05-23 871 "2025-05-24 1071 "2025-05-25 801 "2025-05-26 188 "2025-05-27 292 "2025-05-28 335 "2025-05-29 409 "2025-05-30 371 "2025-05-31 514 "2025-06-01 440 "2025-06-02 504 "2025-06-03 478 "2025-06-04 546 "2025-06-05 1132 "2025-06-06 1022 "2025-06-07 662 "2025-06-08 227 "2025-06-09

2 4

18.2.7 - IOPS issue
by Devender Singh 17 Jun '25

17 Jun '25

Dear All I upgrade my clusters from 18.2.4 & 18.2.6 to 18.2.7. 1. I started facing multiple disks went down.. 2. Fio test showing 5-35 IOPS on cephfs? $ fio --name=latency-test --ioengine=libaio --rw=randread --bs=4k --size=512M --numjobs=1 --iodepth=1 --direct=1 --runtime=60 --time_based --group_reporting latency-test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.28 Starting 1 process Jobs: 1 (f=1): [r(1)][100.0%][r=100KiB/s][r=25 IOPS][eta 00m:00s] latency-test: (groupid=0, jobs=1): err= 0: pid=3684386: Mon Jun 16 23:16:51 2025 read: IOPS=142, BW=572KiB/s (586kB/s)(33.5MiB/60001msec) slat (usec): min=46, max=250, avg=75.77, stdev=19.79 clat (usec): min=571, max=716047, avg=6913.78, stdev=33009.42 lat (usec): min=630, max=716107, avg=6989.89, stdev=33009.90 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 857], 10.00th=[ 914], 20.00th=[ 979], | 30.00th=[ 1037], 40.00th=[ 1074], 50.00th=[ 1123], 60.00th=[ 1172], | 70.00th=[ 1254], 80.00th=[ 1696], 90.00th=[ 4424], 95.00th=[ 22938], | 99.00th=[160433], 99.50th=[233833], 99.90th=[446694], 99.95th=[492831], | 99.99th=[717226] bw ( KiB/s): min= 8, max= 1824, per=100.00%, avg=581.15, stdev=438.28, samples=118 iops : min= 2, max= 456, avg=145.29, stdev=109.57, samples=118 lat (usec) : 750=0.99%, 1000=22.51% lat (msec) : 2=62.04%, 4=3.73%, 10=4.09%, 20=1.40%, 50=2.11% lat (msec) : 100=1.47%, 250=1.24%, 500=0.38%, 750=0.05% cpu : usr=0.12%, sys=1.18%, ctx=8922, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=8580,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=572KiB/s (586kB/s), 572KiB/s-572KiB/s (586kB/s-586kB/s), io=33.5MiB (35.1MB), run=60001-60001msec 3. RBD vis are even showing same number of IOS? Did anyone also face similar issue? 4. I have another version 19.2.2 where it shows almost 9000 IOPS… 5. On version 18.2.4 also facing same less numbers of IOPS. Any solution if someone can help.. Regards Dev

1 0

CephFS mirroring alternative?
by Ralf Schulze 16 Jun '25

16 Jun '25

Hi, we run two clusters with Squid 19.2.2 and would like to use CephFS mirroring. Unfortunately, the feature is still not working correctly and cannot be used. So we are looking for an alternative for the synchronization. One option would be to synchronize the snapshots to the second cluster with rsync. What alternative do you use for CephFS mirroring? What tools and processes do you use? Should CephFS mirroring be fixed? Do you know anything new about this? Thanks and best regards, Ralf

6 7

Re: v18.2.7 Reef released
by Erwin Bogaard 16 Jun '25

16 Jun '25

It looks like there is an issue with the package-based 18.2.7 release, when upgrading from 18.2.6 on el9. There seems to be a new (unfulfilled) dependency that prevents the packages from installing: Problem 1: cannot install the best update candidate for package librgw2-2:18.2.6-0.el9.x86_64 - problem with installed package librgw2-2:18.2.6-0.el9.x86_64 - nothing provides libcrypto.so.3(OPENSSL_3.4.0)(64bit) needed by librgw2-2:18.2.7-0.el9.x86_64 from ceph-reef This issue prevents almost all ceph packages from installing. 18.2.6 is installed and fully functional on this machine, so it seems something happened between 18.2.6 and 18.2.6. Hope someone can confirm this is happening at their severs as well and/or confirm or deny this is a packaging problem. Regards, Erwin

11 17

partitioning OSD Storage drive
by filip Mutterer 16 Jun '25

16 Jun '25

Is it possible to partition an OSD storage drive? As I only have 2 S-ATA ports it sounds like a better choice to use a small part of the two SSDs for the operating system and leave the rest to Ceph. Can I do this? I already have a setup like this with ZFS and it can have significant downsides. but we can live with slow and save data due to this combination.

3 2

radosgw enforce bucket encryption with sse-s3
by Boris 13 Jun '25

13 Jun '25

This is a follow up question to the sse-kms thread, because the KMS team is now working on the transit engine and we will POC with openbao. Is there a way to enforce the bucket encryption for every bucket? As how I understand it, I need to enable the bucket encryption after creating. If we could enforce that, I could a header check in the haproxy to make it transparent for the user. cheers Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 2

Infiniband (RDMA) connectivity problems
by Jan Marek 12 Jun '25

12 Jun '25

Hello, we had sometimes problem with OSD in our all-flash NVMe CEPH cluster. Our cluster is made by cephadm, using Debian bookworm as a "basic" system and podman as a container solution. We are using two-ports Mellanox 100Gbps adapters: # lspci | grep Mellanox 43:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] 43:00.1 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] We are using TCP/IP for front-end communication and RDMA for internal communication. Sometimes (after cca 2-3 month of normal working) one OSD container from whole machine have a problem to connect to other OSD processes, but it is still connected to MON process, which goes to the situation, when it is "half-dead" - it cannot connects to other OSD, other OSD cannot connect to it, but it reports to the MON process, that it is still alive. We have experience, that this situation can resolve by restart container of this OSD. We have script, which is running every 5 minute and which searching in logs for this pattern: start_waiting_for_healthy If is this pattern found, we restart of container with appropriate OSD daemon. After restart this container, CEPH cluster goes quickly to HEALTHY state. I'm adding log, where this situation occured, up to restarting container. It was an osd.12 container with PID 3920171. In my opinion there is some problem to open infiniband connection - some problems in infiniband stack... :-( I'm open to you questions, although I cannot do some special operations, because it is our production cluster... Many thanks for this software :-) Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html

3 4

When is osd_mclock_capacity_iops_hdd/ssd updated?
by Michel Jouvin 11 Jun '25

11 Jun '25

Hi, We have a production cluster homogeneous in terms of HDD devices and I'm a little bit surprised that the osd_mclock_capacity_iops_hdd computed for each OSD is so different (basically a x2 between the lowest and highest values). I understand that sometimes the OSD bench is perturbated by external factors leading to an inappropriate value and if it is too low, it may impact in particular deep scrub duration. The documentation [1] explains that the OSD bench is run at OSD initialization, which is, I guess, startup time. It also suggests that it can be run manually but I guess it makes no sense if the OSD is up because of the concurrent activity. Am I right? How is the bench command supposed to be run as I guess it doesn't work if the OSD is not up? Also, the documentation explains that you can define a value that you measured and seems to suggest that once defined, it will not be updated. Am I right? If yes, does it mean that once the automatic bench has determined a value the only way to update it is to delete it from the config and restart the OSD (if you want the automatic bench to update/redefine it)? Best regards, Michel [1] https://docs.ceph.com/en/squid/rados/configuration/mclock-config-ref/

4 4

pool create via cli ignores size parameter
by Eugen Block 11 Jun '25

11 Jun '25

Hi, without having checked the tracker, does anyone have an explanation why the size parameter is not applied when creating a pool via CLI? According to the help output for 'ceph osd pool create -h' you can specify expected_num_objects (btw. I don't understand what impact that has, all I could find was related to filestore OSDs and PG splitting) and then the size: profile>] [<rule>] [<expected_num_objects:int>] [<size:int>] [<pg_num_min:int>] [<pg_num_max: If I specify both parameters, the pool gets its default size according to osd_pool_default_size (this is a single node cluster, hence default size 2): soc9-ceph:~ # ceph osd pool create temp5 4 replicated replicated_rule 0 4 pool 4 'temp5' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 4 pgp_num 4 autoscale_mode off last_change 260 flags hashpspool stripe_width 0 read_balance_score 1.00 I get the same result on a larger test cluster with multiple hosts. Of course I know that I could just set the size afterwards, but is the command broken? Has anyone looked into this? Thanks! Eugen

1 3