- ceph-users - lists.ceph.io

After upgrading Ceph cluster from Octopus (v15.2.13) to Quincy (v17.2.8), read latency on HDD OSDs increased from 15ms to around 100ms during peak I/O periods.
by Best Regards 09 Sep '25

09 Sep '25

Hi,  We used Ceph's RBD feature in triple-replica mode. Every day at midnight, the virtual machine guest_os has a logrotate task to cut and compress various logs. Before the upgrade, at midnight, IO latency increased from 7ms to around 15ms, which had a minor impact. However, after upgrading to version Q, latency jumped from 7ms to 100ms, with some OSDs even experiencing latency reaching 200ms, significantly impacting guest_os operations. We upgraded from O to P, and then from P to Q the next day. Version P ran for a day with normal latency, but after upgrading to version Q, latency increased dramatically (this refers to the midnight period; there was no noticeable difference at other times). I checked the monitoring data before and after the upgrade, from the pool level to the OSD level and then to the driver level, and found that the ceph_pool_rd indicator and the ceph_osd_op_r (all OSDs associated with the pool) indicator did not show any significant increase before and after the upgrade, but the node_disk_reads_completed_total indicator (associated with the OSD) increased significantly after the upgrade. Then I randomly selected several related OSDs and checked the monitoring data before and after the upgrade, and found that the number of read-type IOs did increase, almost doubling from the original 150 ops to 300+ ops, which seems to exceed the capacity of the HDD and the latency is very large. I searched a lot of blogs, and they said that it might be related to the bluestore_min_alloc_size_hdd parameter being reduced from 65536 to 4096, but many articles do not recommend modifying this value multiple times and should be handled with caution, so I ask for help from the experts to confirm this problem for me, and whether there is a safer solution. Thanks. Best Regards wu_chulin(a)qq.com Best Regards

2 2

CSC Meeting Notes September 8th 2025
by Adam King 08 Sep '25

08 Sep '25

- Tentacle RC0 finishing touches, open items - Radek and Laura will update release announcement for core today - Requiring teuthology runs for PRs that modify src/ dir - RFE: https://tracker.ceph.com/issues/72743 - would also be nice to have some process to make sure PRs don't wait around too long without a run - especially those spanning multiple components or areas without a clear owner - Our organization on Github needs some attention, we have banners mentioning we need to accept the ToS, one about sales tax, and it looks like our number of members is greater than our number of seats - banners have been around for some time - appears to only be visible for those with the admin role in the ceph org - US sales tax and exemption: Please update your billing information for verification and add a sales tax exemption certificate, if applicable. Learn more about this change. - Terms of Service review required. Organizations on the enterprise plan must now accept the GitHub Customer Agreement. Not doing so may put your organization at risk of being downgraded. - Tentacle updates - Will Tentacle release GA be based on RC1 (current RC), or will there be an RC2 before GA? - dependent on results of ongoing testing - topic to be moved to next week - If RC2 is planned, when will it be built? - What is the expected timeline for RC2 and Tentacle GA? - no responses on https://github.com/ceph/ceph/pull/63912 to add ubuntu 24 to qa suites - Adam King to add to orch run

1 0

Moving to Zoom for today's CDS session
by Anthony Middleton 08 Sep '25

08 Sep '25

Jitsi seems to be down. Here's the link for the CDS session - https://us06web.zoom.us/j/84591547745?pwd=nc9T96xfWKbeO5J5GxEBgxUKsCIdxc.1

1 0

RadosGW read/list-all user across all tenants, users, buckets
by Jacques Hoffmann 08 Sep '25

08 Sep '25

Hello Ceph:) We are wondering if it is possible to create some RadosGW credentials/user with readonly (and allowed to list everything) on all buckets. We are working on a backup strategy, and it would be nice if there was a user that can list all buckets and read all object from all users, with just one access-key/secret-key pair. Is that possible? Ceph Squid does support the IAM API, but that just raises more questions than it answers. Can we get this readonly access across all tenants, for all users and buckets, globally? I would assume that tenants are now somewhat isolated, so maybe this is not possible. The rules of this are fairly unclear to me currently, especially because so far I've assumed that bucket names are globally unique, which i am no longer sure of with multitenancy. Thank you, Jacques Hoffmann -- Jacques Hoffmann <jacques.hoffmann(a)vis.ethz.ch> VIS - Verein der Informatikstudierenden an der ETH Zürich CAB E 31, Universitätstr. 6, ETH Zentrum, CH-8092 Zürich https://www.vis.ethz.ch

2 1

Deploy haproxy.nfs
by Thomas 07 Sep '25

07 Sep '25

Hi, I'm experimenting with Ceph+NFS and I would like to set up a NFS-HA solution. I tried to deploy ingress.nfs service via Dashboard. It worked well on amd64, but it failed on arm64: Failed while placing haproxy.nfs.tp-ceph.wbCeph-2.yvjyvl on wbCeph-2.example.com: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-145d1d9a-698a-11f0-ac29-2ccf67085bc4-haproxy-nfs-tp-ceph-wbCeph-2-yvjyvl /usr/bin/podman: stderr Error: no such container ceph-145d1d9a-698a-11f0-ac29-2ccf67085bc4-haproxy-nfs-tp-ceph-wbCeph-2-yvjyvl Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-145d1d9a-698a-11f0-ac29-2ccf67085bc4-haproxy.nfs.tp-ceph.wbCeph-2.yvjyvl /usr/bin/podman: stderr Error: no such container ceph-145d1d9a-698a-11f0-ac29-2ccf67085bc4-haproxy.nfs.tp-ceph.wbCeph-2.yvjyvl Deploy daemon haproxy.nfs.tp-ceph.wbCeph-2.yvjyvl ... Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/haproxy:2.3 -e NODE_NAME=wbCeph-2.example.com quay.io/ceph/haproxy:2.3 -c %u %g /var/lib stat: stderr WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64) stat: stderr ERROR (catatonit:2): failed to exec pid1: Exec format error ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/haproxy:2.3 -e NODE_NAME=wbCeph-2.example.com quay.io/ceph/haproxy:2.3 -c %u %g /var/lib: WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64) ERROR (catatonit:2): failed to exec pid1: Exec format error How can I tell ceph to pull the image for the arm64 platform? Thank's in advance. Thomas

1 0

nfs-ganesha, subvolumes and subtree checking?
by Davíð Steinn Geirsson 06 Sep '25

06 Sep '25

Hey all, When I moved my file server shares to CephFS, I set each share on its own CephFS. The reason was this comment in the nfs-ganesha example config: # Note that FSAL_CEPH does not support subtree checking, so there is # no way to validate that a filehandle presented by a client is # reachable via an exported subtree. # # For that reason, we just export "/" here. Now, this is fine for low numbers of shares, but as they have grown it feels a bit overkill to be creating two or more new pools (metadata + data + sometimes another EC data) for each share. Tuning the PG numbers for those pools is also kind of a pain. I'm wondering, would using a subvolume for the share provide the needed security isolation? Best, Davíð

2 3

Question about shard placement in erasure code pools
by Soeren Malchow 06 Sep '25

06 Sep '25

Dears, maybe someon can explain/answer my question. We are working on setting up a new ceph cluster (underneath a proxmox cluster), the setup is as follows 3 datacenters 3 hosts per datacenter (all in one rack) 8 physical disks per host This is the profile configutation ceph osd erasure-code-profile set ec_profile plugin=isa technique=reed_sol_van k=4 m=2 crush-root=default crush-failure-domain=datacenter crush-device-class=nvme crush-osds-per-failure-domain=2 crush-num-failure-domains=3 We have a hierarchy - created with a custom_location_hook that contains * datacenter 01 * rack 01 * host 01 * host 02 * host 03 * datacenter 02 * rack 01 * host 01 * host 02 * host 03 * datacenter 03 * rack 01 * host 01 * host 02 * host 03 So this means (my understanding) that we are placing 2 shards of data in each datacenter. We want to make sure that we only have max one share of data per host, and in future if possible also distributed across racks once we should ahve more per DC. Is the bucket hierarchy automatically taken into account when choosing the OSD ? Or if not, where do i start, with CRUSH rules ? (I am hesitant to manually modify CRUSH rules. Thanks in advance Soeren

2 1

Is there a preferred EC plugin?
by Boris 05 Sep '25

05 Sep '25

Hi, we currently use the jerasure plugin, but maybe there is a new better default? We use EC mostly for large rgw data with files >1GB. Cheers Boris

3 3

Append to EC pool object (docs vs. reality)
by Jan Kasprzak 05 Sep '25

05 Sep '25

Hello, Ceph users, my erasure-coded pool is inconsistent with the documentation, apparently existing objects cannot be appended to. The documentation states the following: ================ https://docs.ceph.com/en/latest/rados/operations/erasure-code/ By default, erasure-coded pools work only with operations that perform full object writes and appends (for example, RGW). ================ Appends do not work for me: $ ceph osd pool create test-ec erasure k5m2 pool 'test-ec' created $ echo -n 12345678 | rados --pool test-ec put testobj - $ rados --pool test-ec stat testobj test-ec/testobj mtime 2025-09-05T12:51:20.000000+0200, size 8 $ echo -n 12345678 | rados --pool test-ec put testobj - --offset 8 error putting test-ec/testobj: (95) Operation not supported $ echo -n 12345678 | rados --pool test-ec append testobj - error appending test-ec/testobj: (95) Operation not supported When I enable the allow_ec_overwrites flag, everything works as expected: $ ceph osd pool set test-ec allow_ec_overwrites true set pool 73 allow_ec_overwrites to true $ echo -n 12345678 | rados --pool test-ec put testobj - --offset 8 $ echo -n 12345678 | rados --pool test-ec append testobj - $ rados --pool test-ec stat testobj test-ec/testobj mtime 2025-09-05T12:54:30.000000+0200, size 24 Should the documentation be fixed somehow (maybe appending works only at certain block sizes or whatever), or am I doing something incorrect? My cluster is Ceph 19.2.2. Thanks, -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

2 2

Re: snap-schedule not running
by Sophonet 04 Sep '25

04 Sep '25

Quick update / follow-up question: In the mgr logs I have seen that the schedule is indeed running, but snapshots cannot be created due to an error when cephfs.LibCephFS.mkdir ist called (Operation not permitted). Apparently, the scheduler is not using client.admin as ID (for the keyring when connecting to the target filesystem), since "ceph auth get client.admin" shows "caps mds = allow *". I have not found any documentation about setting the client.ID for a schedule. Any hints how the permission error can be avoided? Thanks, sophonet (I am posting this update for the third time now, hoping that the two earlier attempts that did not show up in the list are not just delayed...)

2 3