- ceph-users - lists.ceph.io

Re: Happy holidays to all the Cephalopods out there!
by Anthony D'Atri 23 Dec '25

23 Dec '25

May the tentacles be with you > On Dec 23, 2025, at 4:43 AM, Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: > > And a happy new year, too! > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Happy holidays to all the Cephalopods out there!
by Wannes Smet 23 Dec '25

23 Dec '25

Thank you 🙂 To all of you too!! Kr, Wannes ________________________________ From: Eugen Block via ceph-users <ceph-users(a)ceph.io> Sent: Tuesday, December 23, 2025 10:44 To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Happy holidays to all the Cephalopods out there! And a happy new year, too! _______________________________________________ ceph-users mailing list -- ceph-users@ ceph. io To unsubscribe send an email to ceph-users-leave@ ceph. io ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd And a happy new year, too! _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Happy holidays to all the Cephalopods out there!
by Eugen Block 23 Dec '25

23 Dec '25

And a happy new year, too!

1 0

Re: Low number of inflight compare to op_wip
by Anthony D'Atri 22 Dec '25

22 Dec '25

> On Dec 22, 2025, at 6:47 AM, George Shuklin via ceph-users <ceph-users(a)ceph.io> wrote: > > On 12/16/25 3:37 PM, Anthony D'Atri via ceph-users wrote: >>> I'm tryting to debug low performance on nvme-based cluster. >> Which specific drives? > > INTEL SSDPF2KE032T1 P5520, excellent drive. We see people on the list using random client-class drives, so the drives in use are a first question to ask, because so many people don't understand the differences. I work with a community member who has his CephFS metadata pool on *Sabrent* client drives. Poster child for "flirting with disaster". > > Ceph accepts about 20 requests, proccess them slowly, and then write to devices (super fast), so device sits idle most of the time. OSD daemons are not CPU/memory constrained (~40 cores idle, plenty of memory), os it's just disparity between Ceph OSD speed and backend speed. For sure there are serializations in the PG and OSD code, which is one reason why a healthy PG ratio matters. Remember that NVMe SSDs can process lots of operations in parallel, too. Have you discussed this with Mark Nelson? > I remember, few years ago (on lesser hardware) I done benchmark for ceph using brd (block ram disk) and ceph was able to churn up to 10k IOPS per daemon. Nowdays, it can up to 20k, I think. Mark would know for sure. When doing such tests, be sure that your pool is size=1 so that you aren't constrained by replication ops. In practice, network latency and replication are often significant factors in speed. Also, is the subject system free from swap? Do you have a static osd_memory_target, or do you have the autotuner enabled? > Actually, it's a good question: what is the maxium iops a single OSD daemon can deliver with perfectly fast underlaying storage and negligible network? Mark territory. > >>> but I've noticed, that in-flight for drives is, basically, around 1. That means, that at a given time only one request is processed. This is match match for OSD count/3 /latency formula, and with one in-flight nvme is showing about 10% of it's specs (Intel, DC grade). >> Have you updated to the latest firmware with SST? > > Yep. Certified and the most up to date firmware. Good, with the latest SST? Or if you got the drives from Dell, DSU? I worked for the Solidigm product team for a year, some of the firmware updates are very important. > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Low number of inflight compare to op_wip
by Janne Johansson 22 Dec '25

22 Dec '25

> Actually, it's a good question: what is the maxium iops a single OSD > daemon can deliver with perfectly fast underlaying storage and > negligible network? If it's possible to run an OSD over some ramdisk device, this should be easy to test. Don't know if rocksdb+lvm+bluestore makes this harder, but on filestore it was certainly possible. -- May the most significant bit of your life be positive.

2 1

Re: Low number of inflight compare to op_wip
by George Shuklin 22 Dec '25

22 Dec '25

On 12/16/25 3:37 PM, Anthony D'Atri via ceph-users wrote: >> I'm tryting to debug low performance on nvme-based cluster. > Which specific drives? INTEL SSDPF2KE032T1 Per specs it should give about 300k IOPS. >> I have 24 NVME in 4 servers, plenty of CPU, cluster perfectly balanced, no scrubbing or replication atm, 1024 pg for 24 OSD/17TB. >> >> I expect to see some performance (~250k IOPS total, 50% r/w, few hundred volumes with capped by IOPS, pre-warmed). I see about half of it. >> >> I looked at drive utilization, it's about 70% (per atop) > That's meaningless for SSDs. Yes, I looked deeper, and I found that most of the time there is 0 or 1, with very rarely 2 inflight operations, and all of them are done (from nvme point of view) within 50-70µs. At the same time (under constant load from fio) I can see about 10-20 op_wip, so my current theory goes like this: Ceph accepts about 20 requests, proccess them slowly, and then write to devices (super fast), so device sits idle most of the time. OSD daemons are not CPU/memory constrained (~40 cores idle, plenty of memory), os it's just disparity between Ceph OSD speed and backend speed. I remember, few years ago (on lesser hardware) I done benchmark for ceph using brd (block ram disk) and ceph was able to churn up to 10k IOPS per daemon. Nowdays, it can up to 20k, I think. Actually, it's a good question: what is the maxium iops a single OSD daemon can deliver with perfectly fast underlaying storage and negligible network? >> but I've noticed, that in-flight for drives is, basically, around 1. That means, that at a given time only one request is processed. This is match match for OSD count/3 /latency formula, and with one in-flight nvme is showing about 10% of it's specs (Intel, DC grade). > Have you updated to the latest firmware with SST? Yep. Certified and the most up to date firmware.

1 0

Re: Upgrade Advice: Ceph Reef (18.2.7) to Squid (19.2.3) on Proxmox VMs
by Văn Trần Quốc 22 Dec '25

22 Dec '25

Hi everyone, I just want to say a big thank you to Eugen, Wannes, and Frédéric for sharing your experiences and logs. I really appreciate the quick and detailed help from the community. Best regards, Van Tran

1 0

multi site divergence?
by Christopher Durham 22 Dec '25

22 Dec '25

Hi, I have a multi site configuration with two sites that works just find for S3 bucket replication in both directions for the sites. - Rocky 9 and Ceph 19.2.2. The customer now wishes to add some additional higher speed storage on one side that does not get replicated for the buckets on that storage. My thought is that this should be a separate pool. and use bucket sync policy to prevent buckets created on that pool from replicating. But as I understand it, the bucket itself must exist on both sides. What do I do to properly create that bucket on the side that will not have the data? Do I do something clever with placement groups to have the bucket created on existing storage on the one side, but have it created on the new storage pool on the side where the new storage will be? Thanks for any information you all may have -Chris

1 0

S3 API support for cross-tenant in Tentacle
by Pierre-Olivier Quirion 19 Dec '25

19 Dec '25

Hello all, The Tentacle release document states that "S3 API support for cross-tenant names such as Bucket='tenant:bucketname' [will be removed]". Does it mean that there will be no way to share data directly with users if they are not in the tenant or the account, or does it mean that there will be another way to do it but with another syntax, and what about cross tenancy sharing with the swift api? Cheers, P_O

1 0

Re: Is there an efficient way to list all namespaces in a pool?
by Casey Bodley 19 Dec '25

19 Dec '25

On Fri, Dec 19, 2025 at 6:32 AM Florian Haas via ceph-users <ceph-users(a)ceph.io> wrote: > > Hello everyone, > > I've been wondering if there is a way of listing all namespaces in a > RADOS pool in a way that is both human- and machine-readable, and is > more efficient than this: > > rados -p <pool> ls --all -f json \ > | jq -r "[.[] | .namespace] | unique" > > There are of course several variations of this approach, but they all > seem to boil down to enumerating *all objects* and then picking out > their namespaces — which is, shall we say, impractical if a pool > contains millions of objects. > > Does a fundamentally different option exist? the osds don't maintain an index of namespace names, so i'm afraid you're stuck scanning all objects > > Cheers, > Florian > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0