- ceph-users - lists.ceph.io

Re: Temporay credentials in STS
by Pritha Srivastava 16 Dec '25

16 Dec '25

Hi, The temporary credentials are present in the assume role call response. Thanks, Pritha On Tue, Dec 16, 2025 at 1:09 AM Rok Jaklič via ceph-users < ceph-users(a)ceph.io> wrote: > Hi, > > I try to follow https://docs.ceph.com/en/reef/radosgw/STS/ > > If I print role_response, there are no credentials in the response object? > Where can I get credentials for TESTER1 after creating a role and setting > policy for it? > > Kind regards, > Rok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

2 1

Re: Low number of inflight compare to op_wip
by Anthony D'Atri 16 Dec '25

16 Dec '25

> I'm tryting to debug low performance on nvme-based cluster. Which specific drives? > I have 24 NVME in 4 servers, plenty of CPU, cluster perfectly balanced, no scrubbing or replication atm, 1024 pg for 24 OSD/17TB. > > I expect to see some performance (~250k IOPS total, 50% r/w, few hundred volumes with capped by IOPS, pre-warmed). I see about half of it. > > I looked at drive utilization, it's about 70% (per atop) That's meaningless for SSDs. > but I've noticed, that in-flight for drives is, basically, around 1. That means, that at a given time only one request is processed. This is match match for OSD count/3 /latency formula, and with one in-flight nvme is showing about 10% of it's specs (Intel, DC grade). Have you updated to the latest firmware with SST? > > I looked at osd (they loaded uniformely, so any of them shows the same results). > > I see "op_wip": 10-27, but in-flight value is about 0-2, mostly around 1. > > I can't get away from the feeling that somehow osd is doing operations (almost) sequential. I already played with osd_op_num_threads_per_shard (4), osd_op_num_shards (8), set mclock profile to high_client_ops, ms_async_op_threads 24, and it can't get more inflight ios. > > I feel I miss something. How to make Ceph to send more requests to underlaying NVME? > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Achieve no loss-of-write with 2 node failure?
by Anthony D'Atri 16 Dec '25

16 Dec '25

> >> If I set min_size=k+1, does that mean I'd need a minimum of m=3 to avoid loss-of-write with a 2-node failure? > > Yes. Having at least k+m+1 failure domains (nodes in your case) also, subtly, means that your failure domains can be of different aggregate CRUSH weights without the capacity delta being unusable. > >> And to avoid loss-of-write with a 2-node failure, would I need: >> a) a minimum of k+m nodes, or > > This is the bare minimum for EC without any recovery location. > >> b) a minimum of k+m+2 nodes? > > This is recommended because with this number of nodes the cluster still has "spare" nodes to recover to in case of a node failure. Indeed, Ceph is all about strong consistency. It would rather interdict writes than have you make a risky write. If up-front CapEx is your concern, remember that you don't have to fully populate nodes with drives, at least not initially. I sometimes recommend a minimum of 7 nodes so that 4+2 or 3+3 EC can be done safely. Seven nodes half-full is better in multiple ways than 4 nodes fully populated. As for nodes, used Dell R640 can be had with 8 or 10 NVMe bays and lots of cores very inexpensively these days. Will they give another (let's be honest) 10 years of service? Hard to say. But today's SSDs can be transplanted into tomorrow's chassis, and when the latter is cheap, one can easily afford to lay in a couple of spares. It's like with full-frame digital photography: a body from 10 years ago is meh by today's standards, but that $10,000 lens for shooting sports ball is still golden.

1 0

Re: Problems with orchestrator
by Robert Sander 16 Dec '25

16 Dec '25

Hi, Am 16.12.25 um 2:03 PM schrieb Carsten Götze via ceph-users: > Is there a way to force the orchestrator to sync its state information with the nodes? Try to restart the active MGR daemon with systemctl on its node. > Where do I find meaningful logs for the orchestrator? The journal of the active MGR contains cephadm output Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Problems with orchestrator
by Carsten Götze 16 Dec '25

16 Dec '25

Hi! I'm running a ceph cluster using cephadm and recently upgraded from squid to tentacle 20.2.0. Until recently everything worked fine until I started the nfs module. The nfs daemons were reported to be running but after some 10 minutes or so all of them were reported to be dead except one. nfs service on port 2049 was never provided on any of the nodes even while the daemons were supposed to be running. As i found out later, the nfs daemons were never started at all, because the setup process required a systemd-firewalld to be installed on the system which of course wasn't. After some headaches with the newly installed firewalld I decided to roll back, delete the firewalld and postpone the nfs deployment. I then tried to stop the nfs daemons with 'ceph orch daemon stop', which did nothing, even after waiting some 10 minutes. I had to reissue the command several times to make the reportedly dead nfs daemons vanish from the 'ceph orch ps' list. The one daemon that was reported to be still running however would only die after 'ceph orch daemon stop --force' and was in an 'error' state thereafter and could not be removed from the 'ceph orch ps' list by no means. So I decided to delete the managing nfs service from the 'ceph orch ls' list, in hope that it would also tear down the remaining nfs daemon. This obviously was a bad idea since, the service is now in the state of deleting. However it cannot be deleted, because there is still the one daemon in error state, which cannot be deleted because it was never running at all. As a last measure I forcefully removed the docker container on the node with the cephadm command, but even though there are no traces left of that nfs daemon, it is still listed when running 'ceph orch ps'. I also noticed that the 'ceph orch device ls' is out of sync with reality and 'ceph orch ps' is still listing osds that I've already shutdown and deleted. I therefore suspect, that the orchestrator has stopped collecting state information from the nodes. Is there a way to force the orchestrator to sync its state information with the nodes? Where do I find meaningful logs for the orchestrator? With best regards, Carsten Götze

1 0

Re: Achieve no loss-of-write with 2 node failure?
by Robert Sander 16 Dec '25

16 Dec '25

Hi, Am 16.12.25 um 1:05 PM schrieb Andrew Klaassen via ceph-users: > If I set min_size=k+1, does that mean I'd need a minimum of m=3 to avoid loss-of-write with a 2-node failure? Yes. > And to avoid loss-of-write with a 2-node failure, would I need: > > a) a minimum of k+m nodes, or This is the bare minimum for EC without any recovery location. > b) a minimum of k+m+2 nodes? This is recommended because with this number of nodes the cluster still has "spare" nodes to recover to in case of a node failure. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Achieve no loss-of-write with 2 node failure?
by Andrew Klaassen 16 Dec '25

16 Dec '25

I've been trying to wrap my head around avoiding loss-of-write with a 2-node failure with erasure coding. If I set min_size=k+1, does that mean I'd need a minimum of m=3 to avoid loss-of-write with a 2-node failure? And to avoid loss-of-write with a 2-node failure, would I need: a) a minimum of k+m nodes, or b) a minimum of k+m+2 nodes? If the answer is k+m nodes, would I want to go with k+m+2 nodes anyway for other reasons? Thanks. Andrew

1 0

Low number of inflight compare to op_wip
by George Shuklin 16 Dec '25

16 Dec '25

I'm tryting to debug low performance on nvme-based cluster. I have 24 NVME in 4 servers, plenty of CPU, cluster perfectly balanced, no scrubbing or replication atm, 1024 pg for 24 OSD/17TB. I expect to see some performance (~250k IOPS total, 50% r/w, few hundred volumes with capped by IOPS, pre-warmed). I see about half of it. I looked at drive utilization, it's about 70% (per atop), but I've noticed, that in-flight for drives is, basically, around 1. That means, that at a given time only one request is processed. This is match match for OSD count/3 /latency formula, and with one in-flight nvme is showing about 10% of it's specs (Intel, DC grade). I looked at osd (they loaded uniformely, so any of them shows the same results). I see "op_wip": 10-27, but in-flight value is about 0-2, mostly around 1. I can't get away from the feeling that somehow osd is doing operations (almost) sequential. I already played with osd_op_num_threads_per_shard (4), osd_op_num_shards (8), set mclock profile to high_client_ops, ms_async_op_threads 24, and it can't get more inflight ios. I feel I miss something. How to make Ceph to send more requests to underlaying NVME?

1 0

Temporay credentials in STS
by Rok Jaklič 15 Dec '25

15 Dec '25

Hi, I try to follow https://docs.ceph.com/en/reef/radosgw/STS/ If I print role_response, there are no credentials in the response object? Where can I get credentials for TESTER1 after creating a role and setting policy for it? Kind regards, Rok

1 0

Re: Fwd: On your post: "[Tentacle 20.2.0]: Inconsistent pg's after enabling ec optimisation flag"
by David Walter 15 Dec '25

15 Dec '25

Hi Jeff, What migration did you do that you say the mismatches went away? Did the crashing of OSDs also stop? Migration to a new pool on the same or different OSDs? With or without EC optimization enabled? Or re-writing the data in the same pool by purging one OSD at a time? Best, David On 12/5/25 11:02, Jeff Bailey via ceph-users wrote: > I've seen OSD's crash when running an fstrim on an RBD with an EC data > pool with the EC optimizations enabled. Always two OSD's. It's a 8+2 > pool but I don't know if the "2"s are a coincidence or not. Other than > that, I haven't seen any OSD crashes with EC optimizations. We had > the same size mismatches that went away after migrating the data. > > > On 12/5/2025 10:34 AM, Reto Gysi via ceph-users wrote: >> Hi Bill >> >> Am Fr., 5. Dez. 2025 um 10:02 Uhr schrieb Bill Scales < >> bill_scales(a)uk.ibm.com>: >> >>> Hi Reto, >>> >>> Sorry to hear about your problems with turning on ec optimizations. >>> I've >>> led the team that developed this feature, so we are keen to help >>> understand >>> what happened to your system. >>> >>> Your configuration looks fine as far as being supported with ec >>> optimizations. The daemons (mons, osds and mgr) need to be running >>> tentacle >>> code to use this feature, there is no requirement to update any of your >>> clients. >>> >>> Do you have any more examples of the inconsistent objects logs that you >>> can share with me? >>> >>> There was a bug that we fixed late in the development cycle which was >>> scrubbing incorrectly reporting a size mismatch for objects written >>> prior >>> to turning on optimizations. This is because prior to turning on >>> optimizations objects are padded to a multiple of the stripe_width, >>> afterwards objects are no longer padded. The scrub code was >>> occasionally >>> getting confused and incorrectly reporting an inconsistency. In this >>> case >>> the scrub was reporting false positive errors - there was nothing wrong >>> with the data and no problems accessing the data. I notice the log you >>> shared in the email was for a size mismatch, I'm interested whether >>> all the >>> logs were for mismatched sizes. We will do some further work to confirm >>> that the fix is in the 20.2.0 tentacle release and that there are no >>> problems with the back port. >>> >> As far I could see the errors where all about the mismatched sizes. >> >> >>> You also mention that you saw some OSD crashes. Do you have any further >>> information about these? >>> >> I just could cause another crash with starting up a windows VM, and >> doing a >> WIndows file history backup to a drive that is on an RBD Image on pool >> rbd_ecpool where I had the allow_ec_optimization flag enabled. >> >> <disk type="network" device="disk"> >> <driver name="qemu" type="raw" cache="writethrough" io="threads" >> discard="unmap" detect_zeroes="unmap"/> >> <auth username="admin"> >> <secret type="ceph" uuid="878b0bc5-c471-4ec6-a92a-f65282ffbdf6"/> >> </auth> >> <source protocol="rbd" name="rbd/game_windows_backup_drive" >> index="3"> >> <host name="zephir" port="6789"/> >> </source> >> <target dev="vdd" bus="virtio"/> >> <alias name="virtio-disk3"/> >> <address type="pci" domain="0x0000" bus="0x00" slot="0x02" >> function="0x0"/> >> </disk> >> >> root@zephir:~# ceph -s >> cluster: >> id: 27923302-87a5-11ec-ac5b-976d21a49941 >> health: HEALTH_WARN >> 2 osds down >> Reduced data availability: 49 pgs inactive >> Degraded data redundancy: 15722192/79848340 objects degraded >> (19.690%), 246 pgs degraded, 247 pgs undersized >> >> services: >> mon: 3 daemons, quorum zephir,debian,raspi (age 20h) >> [leader: >> zephir] >> mgr: zephir.enywvy(active, since 21h), standbys: >> debian.nusuye >> mds: 3/3 daemons up, 3 standby >> osd: 18 osds: 16 up (since 5m), 18 in (since 3d); 8 >> remapped >> pgs >> cephfs-mirror: 2 daemons active (2 hosts) >> rbd-mirror: 2 daemons active (2 hosts) >> rgw: 1 daemon active (1 hosts, 1 zones) >> tcmu-runner: 5 portals active (2 hosts) >> >> data: >> volumes: 3/3 healthy >> pools: 25 pools, 450 pgs >> objects: 17.52M objects, 62 TiB >> usage: 116 TiB used, 63 TiB / 179 TiB avail >> pgs: 10.889% pgs not active >> 15722192/79848340 objects degraded (19.690%) >> 197 active+undersized+degraded >> 183 active+clean >> 49 undersized+degraded+peered >> 15 active+clean+scrubbing >> 5 active+clean+scrubbing+deep >> 1 active+undersized >> >> io: >> client: 1.7 KiB/s rd, 1023 B/s wr, 1 op/s rd, 0 op/s wr >> >> root@zephir:~# ceph osd status >> ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE >> 0 debian 11.2T 5289G 0 0 0 0 exists,up >> 1 zephir 11.6T 4846G 0 0 0 0 exists >> 2 zephir 10.7T 5827G 0 0 2 4 exists,up >> 3 zephir 11.0T 5497G 0 0 0 0 exists >> 4 zephir 159G 1376G 0 0 2 0 exists,up >> 5 zephir 120G 1415G 0 0 0 0 exists,up >> 6 debian 11.3T 5200G 0 0 1 0 exists,up >> 7 zephir 305G 1230G 0 0 0 0 exists,up >> 8 zephir 10.9T 5594G 0 0 0 0 exists,up >> 9 zephir 11.9T 4616G 0 0 0 0 exists,up >> 10 zephir 235G 1300G 0 0 0 0 exists,up >> 11 zephir 223G 1312G 0 0 0 0 exists,up >> 12 zephir 11.0T 5479G 0 0 0 0 exists,up >> 13 debian 118G 631G 0 0 3 8 exists,up >> 14 debian 345G 1190G 0 0 0 0 exists,up >> 15 debian 11.4T 5079G 0 0 0 0 exists,up >> 16 debian 554G 3029G 0 0 3 8 exists,up >> 17 debian 12.5T 5874G 0 0 1 0 exists,up >> root@zephir:~# >> >> I attached the log of osd.1 to >> https://filebin.net/jwvs6kuqrc7hx8id >> >> Best Regards, >> >> Reto >> >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1