- ceph-users - lists.ceph.io

Re: Problems with orchestrator
by Carsten Götze 17 Dec '25

17 Dec '25

Hi! Thank you for the pointers. Unfortunately restarting the active mgr daemon didn't help. However looking at cephadm.log gave me at least a clue what was going on. It seems that installing and deinstalling systemd-firewalld has completely messed up the firewall rules. And restarting one of the nodes has made things even worse. The osd's on that node died after a few minutes. I had to restart the osd's systemd services, only to see them tumbling down a few minutes later. Only after I finally flushed all the firewall rules from every single node things went back to "normal". However, orchestrator is still out of sync with reality. So a few more questions: Where does orchestrator store its state information? Is firewalld required for tentacle? I didn't need it for squid, but It sure is now required for the nfs module.Do the other modules also use it to control the firewall? With firewalld installed, would restarting the nodes help to restore the firewall rules? What ports have to be open in the firewall? So far I have: 3000, 3300, 5666, 6789, 6800:7568, 8443, 9093, 9094, 9095, 9100. Anything else? Would it be safe to downgrade to squid again? With best regards Carsten Götze > Am 16.12.2025 um 14:10 schrieb Robert Sander via ceph-users <ceph-users(a)ceph.io>: > > Hi, > > Am 16.12.25 um 2:03 PM schrieb Carsten Götze via ceph-users: > >> Is there a way to force the orchestrator to sync its state information with the nodes? > > Try to restart the active MGR daemon with systemctl on its node. > >> Where do I find meaningful logs for the orchestrator? > > The journal of the active MGR contains cephadm output > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Temporay credentials in STS
by Rok Jaklič 17 Dec '25

17 Dec '25

To answer to myself and for those interested. "You" just need one user in this case TESTER and policy_document has to be: policy_document = '''{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam::TENANT:user/USER"]},"Action":["sts:AssumeRole"]}]}''' If TENANT is TENANT1 and USER is USER1, then policy_document should be: policy_document = '''{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam::TENANT1:user/USER1"]},"Action":["sts:AssumeRole"]}]}''' Kind regards, Rok On Tue, Dec 16, 2025 at 3:55 PM Rok Jaklič via ceph-users < ceph-users(a)ceph.io> wrote: > I get an error like, when trying to assume role: > response = sts_client.assume_role( > File > "/home/user/um-test-venv/lib64/python3.9/site-packages/botocore/client.py", > line 602, in _api_call > return self._make_api_call(operation_name, kwargs) > File > > "/home/user/um-test-venv/lib64/python3.9/site-packages/botocore/context.py", > line 123, in wrapper > return func(*args, **kwargs) > File > "/home/user/um-test-venv/lib64/python3.9/site-packages/botocore/client.py", > line 1078, in _make_api_call > raise error_class(parsed_response, operation_name) > botocore.exceptions.ClientError: An error occurred (Unknown) when calling > the AssumeRole operation: Unknown > (um-test-venv) [user@server ~]$ client_loop: send disconnect: Broken pipe > > Documentation is a little bit ambiguous. There is user TESTER and user > TESTER1. Do I need to create both users? > > --- > > What is the right way to set policy and role for it if I have just USER1 > and TENANT1 and ACCESS_KEY1 and SECRET_KEY1 and want to have temporary > credentials? > > On Tue, Dec 16, 2025 at 6:31 AM Pritha Srivastava <prsrivas(a)redhat.com> > wrote: > > > Hi, > > > > The temporary credentials are present in the assume role call response. > > > > Thanks, > > Pritha > > > > > > On Tue, Dec 16, 2025 at 1:09 AM Rok Jaklič via ceph-users < > > ceph-users(a)ceph.io> wrote: > > > >> Hi, > >> > >> I try to follow https://docs.ceph.com/en/reef/radosgw/STS/ > >> > >> If I print role_response, there are no credentials in the response > object? > >> Where can I get credentials for TESTER1 after creating a role and > setting > >> policy for it? > >> > >> Kind regards, > >> Rok > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Achieve no loss-of-write with 2 node failure?
by Anthony D'Atri 16 Dec '25

16 Dec '25

> >> Note: larger HDDs have really low IOPS-per-TB; SSDs avoid that >> issue but cheap SSDs do not have PLP so write IOPS are much >> lower than read IOPS. > > That is something I've seen mentioned a lot, so we've only got PLP drives on the shopping list. The tentative current shopping list is 24x 7.68TB Samsung PM893 or Kingston DC600M drives. PLP is crucial, but there are other factors. If you have a choice, NVMe SSDs give a lot more bang for the buck than SATA. The Kingstons don't have IIRC stellar performance, but maybe you don't need it to be. > >> Whether the drive is SSD or HDD larger >> ones also usually mean large PGs which is not so good. With SSDs >> at least it is possible (and in some cases advisable) to split >> them into multiple OSDs though. > > Could we just increase the number of PGs to avoid this? With Quincy and later releases, the former advice to split became a wash. Splitting HDDs was never widely practiced or advised. With modern SSDs > 30 TB splitting might make sense. There are serializations both in OSD and PG code. > >> That is indeed a good suggestion: the fewer the drives per >> server the better. Ideally just one drive per server :-). > > This might just be possible, since we've got a couple of racks of render nodes that I can probably make the case to retire from render duties. Would I actually see a major advantage going from 6 nodes to 8, from 8 to 12, or from 12 to 24? (Given 24 disks in each case.) Those chassis can even only have their server trays partly populated too. Were you to have 8 chassis, you could make `chassis` your CRUSH failure domain, with however many drives per chassis/host. > > Andrew > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Re: Achieve no loss-of-write with 2 node failure?
by Andrew Klaassen 16 Dec '25

16 Dec '25

On Tuesday, December 16, 2025 at 02:33:33 p.m. EST, Anthony D'Atri via ceph-users <ceph-users(a)ceph.io> wrote: > Indeed. I suggest, if Proxmox lets you, encoding the chassis into your > CRUSH topology, even if you don't use it today. Maybe in the future > you can add chassis, which would expand your CRUSH options. > > root default > chassis chassis1 > host host1 > host host2 > host host3 > host host4 > chassis chassis2 > host host5 > ... > > With `host` as the CRUSH failure domain, you do run the risk of a PG > being placed on 3 or even 4 nodes within a single chassis, which > would not be good. A custom CRUSH rule might let you constrain > so that PG placements always span chassis; those are beyond my ken. > Someone else on the list could likely help with that rarified task. > > Alternately there are the shiny new MSR CRUSH rules: > https://docs.ceph.com/en/latest/dev/crush-msr/ > https://www.youtube.com/watch?v=JpkLPkizUt4 Thanks, I will look into that. Andrew

1 0

Re: Achieve no loss-of-write with 2 node failure?
by Andrew Klaassen 16 Dec '25

16 Dec '25

On Tuesday, December 16, 2025 at 03:28:01 p.m. EST, Peter Grandi via ceph-users <ceph-users(a)ceph.io> wrote: > * Ceph is designed around the idea of many servers and many > small drives with a few small drives per server as it needs to > have lots of IOPS-per-TB and low impact per system failure. > * Minimizing up-front costs means fewer larger drives and higher > density servers. > * The common result is extreme congestion during balancing and > recovery times and high latencies during parallel accesses. > This mailing list is full of "cost-optimized" horror stories. Good to know, and thanks for the further explanations why. > Note: larger HDDs have really low IOPS-per-TB; SSDs avoid that > issue but cheap SSDs do not have PLP so write IOPS are much > lower than read IOPS. That is something I've seen mentioned a lot, so we've only got PLP drives on the shopping list. The tentative current shopping list is 24x 7.68TB Samsung PM893 or Kingston DC600M drives. > Whether the drive is SSD or HDD larger > ones also usually mean large PGs which is not so good. With SSDs > at least it is possible (and in some cases advisable) to split > them into multiple OSDs though. Could we just increase the number of PGs to avoid this? > That is indeed a good suggestion: the fewer the drives per > server the better. Ideally just one drive per server :-). This might just be possible, since we've got a couple of racks of render nodes that I can probably make the case to retire from render duties. Would I actually see a major advantage going from 6 nodes to 8, from 8 to 12, or from 12 to 24? (Given 24 disks in each case.) Andrew

1 0

Proposal to end support for some erasure code plugins
by Jamie Pryde 16 Dec '25

16 Dec '25

Hi all, I want to make you aware of the Fast EC team's proposal to end support for some erasure code plugins and techniques. We want to do this for a few reasons: 1. Continuing to support a large list of plugins and techniques requires development and test effort. We don't think this effort is valuable because 2. Telemetry data shows that most Ceph clusters that are using EC are using Jerasure reed_sol_van or ISA-L reed_sol_van, and 3. Benchmarking shows that reed_sol_van has very good performance on modern CPUs. We are proposing that the following list of plugins and techniques should be deprecated in the Umbrella release, and support be removed in the V release: * Jerasure (reed_sol_r6_op, cauchy_orig, cauchy_good, liberation, blaum_roth, liber8tion) * SHEC * CLAY The following list of plugins and techniques would continue to be supported in the V release: * Jerasure (reed_sol_van) * ISA-L (reed_sol_van, cauchy) * LRC (Although Fast EC doesn’t currently support LRC and we wouldn't recommend using LRC yet, we think we will be able to use it in future to improve support for erasure coded pools in stretched clusters.) I have written a post for the ceph.io blog which contains more details and shows some of the data behind the proposal. It also talks about the pool migration feature we are planning to add in Umbrella. This can be used to migrate data from a pool to a new EC pool that uses a supported plugin/technique. You can find the blog PR here: https://github.com/ceph/ceph.io/pull/988 We are interested to hear if anyone has any objections to this plan. Please reply to this mail or leave a comment on the blog PR if you do have any objections or concerns. We plan to publish the blog post sometime mid-January. Thanks, -- Jamie Pryde Software Engineer IBM UK Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN

1 0

Re: Achieve no loss-of-write with 2 node failure?
by pg＠ceph.list.sabi.co.UK 16 Dec '25

16 Dec '25

>>> b) a minimum of k+m+2 nodes? > [...] If up-front CapEx is your concern The perhaps Ceph is not such a great idea: * Ceph is designed around the idea of many servers and many small drives with a few small drives per server as it needs to have lots of IOPS-per-TB and low impact per system failure. * Minimizing up-front costs means fewer larger drives and higher density servers. * The common result is extreme congestion during balancing and recovery times and high latencies during parallel accesses. This mailing list is full of "cost-optimized" horror stories. Note: larger HDDs have really low IOPS-per-TB; SSDs avoid that issue but cheap SSDs do not have PLP so write IOPS are much lower than read IOPS. Whether the drive is SSD or HDD larger ones also usually mean large PGs which is not so good. With SSDs at least it is possible (and in some cases advisable) to split them into multiple OSDs though. > remember that you don't have to fully populate nodes with > drives, at least not initially. That is indeed a good suggestion: the fewer the drives per server the better. Ideally just one drive per server :-). > I sometimes recommend a minimum of 7 nodes so that 4+2 or 3+3 > EC can be done safely. For me 7 buckets and 6-wide EC stripes is less desirable as that means that *any* bucket failure will cause *nearly all* objects to be degraded, and if one uses CephFS which chunks files across objects, too bad. I have seen long, long, long recovery storms in configurations like that. > Seven nodes half-full is better in multiple ways than 4 nodes > fully populated. Indeed. But with 6-7 buckets I would only use 3-ways replication.

1 0

Re: Achieve no loss-of-write with 2 node failure?
by Anthony D'Atri 16 Dec '25

16 Dec '25

> We're in a situation where we've got nodes that we can use, so it's mostly about the drive cost and configuration. Ack. > The nodes are in chassis of 4 nodes each. Would there be any problem with putting a 4+2 or 3+3 config into 8 nodes? With `host` as the CRUSH failure domain, that works. > > And any problem with having 7 monitors running on Proxmox nodes that are part of the Ceph cluster but aren't running any OSDs? Don't deploy 7. 5 is plenty. No reason to couple mons with OSDs. It's a good idea, though, to ensure that you spread mons across failure domains, so that losing one host or chassis doesn't lead to a loss of quorum. > > (Obviously we wouldn't be able to survive the loss of a chassis in this config.) Indeed. I suggest, if Proxmox lets you, encoding the chassis into your CRUSH topology, even if you don't use it today. Maybe in the future you can add chassis, which would expand your CRUSH options. root default chassis chassis1 host host1 host host2 host host3 host host4 chassis chassis2 host host5 ... With `host` as the CRUSH failure domain, you do run the risk of a PG being placed on 3 or even 4 nodes within a single chassis, which would not be good. A custom CRUSH rule might let you constrain so that PG placements always span chassis; those are beyond my ken. Someone else on the list could likely help with that rarified task. Alternately there are the shiny new MSR CRUSH rules: https://docs.ceph.com/en/latest/dev/crush-msr/ https://www.youtube.com/watch?v=JpkLPkizUt4 > > Andrew > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Achieve no loss-of-write with 2 node failure?
by Andrew Klaassen 16 Dec '25

16 Dec '25

On Tuesday, December 16, 2025 at 07:16:28 a.m. EST, Robert Sander via ceph-users <ceph-users(a)ceph.io> wrote: > This is recommended because with this number of nodes the cluster still > has "spare" nodes to recover to in case of a node failure. Thanks. Andrew

1 0

Re: Achieve no loss-of-write with 2 node failure?
by Andrew Klaassen 16 Dec '25

16 Dec '25

On Tuesday, December 16, 2025 at 08:21:43 a.m. EST, Anthony D'Atri via ceph-users <ceph-users(a)ceph.io> wrote: > If up-front CapEx is your concern, remember that you don't have to fully > populate nodes with drives, at least not initially. I sometimes recommend > a minimum of 7 nodes so that 4+2 or 3+3 EC can be done safely. Seven nodes > half-full is better in multiple ways than 4 nodes fully populated. > > As for nodes, used Dell R640 can be had with 8 or 10 NVMe bays and lots > of cores very inexpensively these days. Will they give another (let's be honest) > 10 years of service? Hard to say. But today's SSDs can be transplanted into > tomorrow's chassis, and when the latter is cheap, one can easily afford to lay > in a couple of spares. It's like with full-frame digital photography: a body > from 10 years ago is meh by today's standards, but that $10,000 lens for > shooting sports ball is still golden. We're in a situation where we've got nodes that we can use, so it's mostly about the drive cost and configuration. The nodes are in chassis of 4 nodes each. Would there be any problem with putting a 4+2 or 3+3 config into 8 nodes? And any problem with having 7 monitors running on Proxmox nodes that are part of the Ceph cluster but aren't running any OSDs? (Obviously we wouldn't be able to survive the loss of a chassis in this config.) Andrew

1 0