- ceph-users - lists.ceph.io

Re: Achieve no loss-of-write with 2 node failure?
by pg＠ceph.list.sabi.co.UK 30 Dec '25

30 Dec '25

>> For Ceph they are really necessary for log/database drives >> and ideally also for Ceph and CephFS metadata but non-PLP >> SSDs can be used if they are many for data if the workload >> does not require lots of small data writes. > It’s not about IOPS, it's about not losing data in flight when > the host/rack/DC loses power. * The filesystem (BlueStore or whatever) or DBMS (RocksDB or whatever) is required to issue "write buffer commit" commands to the device whenever necessary to ensure metadata or data is not lost by cache power loss and remains consistent. * The main effect of PLP is to allow the device to ignore "write buffer commit" commands and do commits less often, and that is what improves IOPS (and write amplification in the case of SSDs). * _Some_ loss is inevitable: there can be data still in the middle of being written if the loss happen during a "write to device" operation and efficient filesystems can do pretty large ones. Note: as to the latter point some non-PLP SSDs still have minimal capacitor backup to be able to complete writes from the RAM write buffer to the flash chip. Those that do not are less desirable :-). > [...] 4 hosts is the prudent minimum for R3 pools. For EC, > k+m+1 at a minimum. [...] But the prudent minimum number of buckets is usually not a great idea because of the big percent of object replicas damaged by a bucket failure. I would prefer as a minimum twice the width of replication/stripes for buckets, especially for EC pools where recovery is particularly expensive. > I usually recommend k + m < 12. Above, say, 6+2 the space > savings by going wider diminish, and the write performance and > other factors often aren’t worth the increasingly thin delta. > [...] I’ve seen a cluster with dense toploaders full of > spinners require that weighting up a single 8TB SSD had to be > done over 4 weeks so as to not tank the S3 API. Good anecdote (I have seen several similar cases). So I think that 4+2 (only in desperate cases 6+2) is the widest that is cost-effective; wider may have cost that is lower up-front but the lifetime cost in terms of huge recovery times and the impact on client workload can be quite bad. > At least one commercial vendor considers a cluster that > requires > 8 hours to recover from the loss of a single OSD to > be unsupported. That is a good argument from that vendor. > If DC space is free, sure. Until it isn’t. [...] There are false economies and a major one is to accept a very high cost of "maintenance" operations like recovery and scrubbing as in the anecdote above. That said fiber infrastructure makes distance rather less important than in the past so data centres can be located in cheaper areas (also I am a fan of "colocation" data centres). Still High Frequency Trading people like to put their servers as near as possible to the physical location of exchange data centres to really minimize latency and space there is usually tight and expensive so high density servers and racks make more sense. > Just don’t waste money on 3 DWPD SKUs or RAID HBAs. Usually but I have seen some cases where the DB/WAL SSDs really should have been 3 DWPD, especially if the SSD is shared among data-only OSDs, which is almost inevitable in case of servers with many HDDs. In those cases better to be safe than sorry.

1 0

Re: Achieve no loss-of-write with 2 node failure?
by Anthony D'Atri 29 Dec '25

29 Dec '25

> For Ceph they are really necessary for log/database drives and > ideally also for Ceph and CephFS metadata but non-PLP SSDs can > be used if they are many for data if the workload does not > require lots of small data writes. After all one can use for > that also HDDs (as long as they are small so IOPS-per-TB are not > too low). It’s not about IOPS, it's about not losing data in flight when the host/rack/DC loses power. >> The tentative current shopping list is 24x 7.68TB Samsung >> PM893 or Kingston DC600M drives. > > I have got several dozen DC600M (in some Lustre setups) and they > seem very reliable with good speed, and the PM893 have a good > reputation even if I have never tried them. But it will be hard > to source them now. I have recently tried 7.68TB and 30.72TB > SSDs from a relatively new chinese brand "Memblaze" I’m wary of firmware availability. ymmv. > >> Could we just increase the number of PGs to avoid this? > > Yes, rather more than "usual" guidelines but that has downsides > too but I think less severe than having too large PGs. In 2025, I would agree. More memory use and more peering. > However too large PGs are less of a problem on SSDs as transfer rates > are much higher and they have lots of IOPS. Very large PGs are still a problem for the balancer. > The usual Ceph guidelines say something like 100-150 PGs per OSD To be clear, PG *replicas*, not PGs proper. So inn an R3 pool that means 3 * pg_num. For an EC 4+2 pool, it means 6 * pg_num. When you have a mix of multiple pools, the manual calculations become tricky, though one can use the venerable pgcalc to help with the math. > but I have unfortunately seen 18TB HDDs with around that much > and it was pretty bad. My guess is that the guideline was > applicable to HDDs in the 1TB-2TB range. The guidance was ~200 at one point, then it was retconned to 100 to help keep people from OOMing themselves. In 2025 and with BlueStore, not Filestore OSDs, my sense is that a target of 100 is too low, especially when interpreted by the PG autoscaler, with which that default is actually a max, and you’ll generally see an even lower number. For modern clusters (BlueStore, > 2TB OSDs) I suggest global advanced mon_max_pg_per_osd 600 global advanced mon_target_pg_per_osd 300 mgr advanced mgr/balancer/upmap_max_deviation 1 mgr basic target_max_misplaced_ratio 0.020000 with the PG autoscaler enabled for all pools. RGW index, RBD/CephFS metadata pools should have a BIAS of 4, and no OSDS should have a legacy reweight value other than 1.000000 With these non-default values, the PG autoscaler usually does okay. I’ve seen a cluster with defaults with like a dozen RBD pools. Each pool usually has a minimum pg_num of 32, so in a modestly-sized cluster, a dozen of those takes a significant bite out of the autoscaler’s budget, and its figurative hands are tied. A cluster with only a single, RBD pool (and yes the trivial .mgr pool) is easier for the autoscaler to deal with, though you’ll still usually end up with an inadequate PGS value (right column of `ceph osd df` aka the PG ratio is what we’re targeting). I’ve seen more than one cluster as low as ~20 PG replicas per OSD due to this, and one (/me puckers and shakes) with single digits. > The basic issue that as to aggregating Ceph can only do one level so one has to balance > too few PGs resulting in huge numbers of objects per PG and > amplified recovery traffic This is not wrong, you do effectively more in some cases data that doesn’t strictly need to be moved, it comes along for the ride. Very large PGs also complicate scrubs. cf. https://www.evernote.com/shard/s189/sh/3a5bde91-e94b-4dbb-adbe-9d8a1379f3d5… With a wildly inadequate PG count, each PG represents a significant fraction of the storage on each PG, and the balancer can’t do a very good job. Another, rough analogy is consider how smooth a wood board gets with 40 grit sandpaper vs 400 grit. Rough analogy, see what I did there? ;) As an aside, for posterity I must note that PGs do not have a fixed size, they represent a *fraction* of the total data stored in a pool. > while more PGs means larger per-pool > and per-OSD PG tables. Several thousand PGs per pool is rather > common nowadays and I suspect it can be more. There’s a default max of 64K as a safeguard. Who among us has never fat-fingered an extra zero or two? With very large clusters, this can AFAIK be raised. I’ve worked on clusters with ~3100 OSDs, and there are definite ones larger in the wild. The orchestrator has issues larger than ~3500 OSDs currently but that’s being worked on. >>> That is indeed a good suggestion: the fewer the drives per >>> server the better. Ideally just one drive per server :-). > > To explain a bit more: in an ideal world Ceph was designed to > run on something like a lot of NUCs each fronting a small HDD or > an SSD. Well, I wouldn’t necessarily say that, https://ceph.io/en/news/blog/2016/500-osd-ceph-cluster/ notwithstanding. For sure there are certain aspects and defaults that are holdovers from the days of 1TB spinners, 1GE networking, and less-evolved CRUSH code and tunables. Like the optional replication network, but I digress... I often recommend a minimum of 7 chassis so that 4+2 EC can be done without gymnastics, even if not fully populated with drives. Plan for enough CPU cores, though. RAM can be upgrade, so long as you plan for empty slots. If you fill all the slots with 16 GiB modules and need to expand, you’re stuck. Back in the day the Sun 4/110 could be ordered with a minimum of 8 MiB. They shipped 32x 256KB SIMMs, filling all the slots, and requiring tossing those useless modules in order to add more. So for SSDs, a 1U server with 10-15 capacity drives (depending on U.2, E1.S, E3.S) is a starting point, with M.2 or rear boot. For spinners, I personally suggest a max of 24 SATA slots. Sure you can get a 100 slot toploader, but beware the pitfalls of a cluster comprising only 3 of those, the weight your rack/floor can support, how far that monster sticks out into the aisle, potentially preventing doors from closing, if you can afford fancy-pants doors. If you can, that’s money better spent on getting SSDs instead ;) Modest drive counts also help avoid the need for a replication network or higher link speeds (100 GE+). > In theory one could put the OSD daemon code in firmware and have > OSD-protocol instead of SCSI-protocol HDDs or SSDs with an > Ethernet socket instead of a SAS/SATA/USB/NVME socket. See the above experiment. > The reason why many setups have many-OSD servers is because the > capacity/price curve has a (shallow) sweet spot at around a > couple dozen OSDs per server but often "penny wise is pound > foolish". Yep. Consider what happens when 100 OSDs are offline at once during a failure, and that’s fully 1/3 of your cluster. >> Would I actually see a major advantage going from 6 nodes to >> 8, from 8 to 12, or from 12 to 24? (Given 24 disks in each >> case.) Diminishing returns, but 8 gives you the ability to run EC with k+m <= 7, 12 gives you the ability to run 8+3. > I guess it is not too bad to have up to half a dozen OSDs per > server (depending on network bandwidth, RAM, CPU usage, size, > ...) but in your case 6 per server would mean just 4 servers > which seems fragile to me. 4 hosts is the prudent minimum for R3 pools. For EC, k+m+1 at a minimum. Now if you’re converging with compute, the tradeoffs are different. > The issue is particularly severe with EC "pools" (especially of > the EC profile is wide which is a bad idea anyhow in most cases) I usually recommend k + m < 12. Above, say, 6+2 the space savings by going wider diminish, and the write performance and other factors often aren’t worth the increasingly thin delta. https://docs.ceph.com/en/reef/rados/operations/erasure-code/#id2 > and less bad with 3-way replication as then having just 6 > buckets means that on average the loss of a bucket will trigger > recovery on "only" half of the objects/PGs and so on. Recovery > eats *a lot* of write IOPS (and secondarily network banwidth) so > it is less costly with SSDs (those with PLP in particular). I’ve seen a cluster with dense toploaders full of spinners require that weighting up a single 8TB SSD had to be done over 4 weeks so as to not tank the S3 API. Consider how long it takes to recover from the loss of a 30+ TB spinner, or a node with a hundred of them. During which time you have reduced redundancy and thus increased risk. For large spinners strongly consider m = 3 for this reason. At least one commercial vendor considers a cluster that requires > 8 hours to recover from the loss of a single OSD to be unsupported. https://www.youtube.com/watch?v=Wy3EetoIqng >> I hate to see new spend on SATA/SAS SSDs, since those >> interfaces are progressively disappearing from product >> roadmaps. I would take 4-6 NVMe nodes any day. > > Unfortunately many of us still have old SATA-only or SATA+SAS > servers and it is often better to have more low-density old > servers than fewer higher-density new servers (even if one can > afford the latter). If DC space is free, sure. Until it isn’t. You can also mix and match with Ceph, within reason. You can get a used R640 with a bunch of cores really inexpensively these days. The chassis might not outlive the SSDs, but the SSDs are the bulk of the cost. Just don’t waste money on 3 DWPD SKUs or RAID HBAs. > For the future the ideal for smaller organizations would be U.3 > servers so one can buy new U.3 NVME devices and recycle existing > SATA or SAS devices along them but given the current constraint > on RAM availability I guess getting more servers at all will be > rather hard in 2026. Universal slots, sure - but note that a U.2 / U.3 chassis slot doesn’t necessarily accept SATA, and SAS usually means paying for an annoying HBA. Since HDDs these days are mostly LFF, a universal slot also means suboptimal RU density. When you skip the RAID HBA expense, the cost of an enhanced BMC license you don’t really need, procure wisely, an all-NVMe server including drives can actually cost less. I’ve seen it with my own eyes.

1 0

Re: Achieve no loss-of-write with 2 node failure?
by pg＠ceph.list.sabi.co.UK 29 Dec '25

29 Dec '25

>>> On Tue, 16 Dec 2025 21:46:36 +0000, Anthony D'Atri via >>> ceph-users <ceph-users(a)ceph.io> said: > I hate to see new spend on SATA/SAS SSDs, since those > interfaces are progressively disappearing from product > roadmaps. I would take 4-6 NVMe nodes any day. Unfortunately many of us still have old SATA-only or SATA+SAS servers and it is often better to have more low-density old servers than fewer higher-density new servers (even if one can afford the latter). For the future the ideal for smaller organizations would be U.3 servers so one can buy new U.3 NVME devices and recycle existing SATA or SAS devices along them but given the current constraint on RAM availability I guess getting more servers at all will be rather hard in 2026. Besides there used to be the strange situation where (I guess because of the huge installed base of U.2 servers) U.2 models were somewhat more available (and cheaper) than U.3 ones or E3.S or E1.S ones but all this is moot again because of the RAM (etc.) availability issues.

1 0

Re: Achieve no loss-of-write with 2 node failure?
by pg＠ceph.list.sabi.co.UK 29 Dec '25

29 Dec '25

>> Note: larger HDDs have really low IOPS-per-TB; SSDs avoid that >> issue but cheap SSDs do not have PLP so write IOPS are much >> lower than read IOPS. > That is something I've seen mentioned a lot, so we've only got > PLP drives on the shopping list. For Ceph they are really necessary for log/database drives and ideally also for Ceph and CephFS metadata but non-PLP SSDs can be used if they are many for data if the workload does not require lots of small data writes. After all one can use for that also HDDs (as long as they are small so IOPS-per-TB are not too low). > The tentative current shopping list is 24x 7.68TB Samsung > PM893 or Kingston DC600M drives. I have got several dozen DC600M (in some Lustre setups) and they seem very reliable with good speed, and the PM893 have a good reputation even if I have never tried them. But it will be hard to source them now. I have recently tried 7.68TB and 30.72TB SSDs from a relatively new chinese brand "Memblaze" and they were pretty good and there seems to be still some availability of those outside the USA, they have just setup an european subsidiary. >> Whether the drive is SSD or HDD larger ones also usually mean >> large PGs which is not so good. With SSDs at least it is >> possible (and in some cases advisable) to split them into >> multiple OSDs though. > Could we just increase the number of PGs to avoid this? Yes, rather more than "usual" guidelines but that has downsides too but I think less severe than having too large PGs. However too large PGs are less of a problem on SSDs as transfer rates are much higher and they have lots of IOPS. The usual Ceph guidelines say something like 100-150 PGs per OSD but I have unfortunately seen 18TB HDDs with around that much and it was pretty bad. My guess is that the guideline was applicable to HDDs in the 1TB-2TB range. The basic issue that as to aggregating Ceph can only do one level so one has to balance too few PGs resulting in huge numbers of objects per PG and amplified recovery traffic while more PGs means larger per-pool and per-OSD PG tables. Several thousand PGs per pool is rather common nowadays and I suspect it can be more. >> That is indeed a good suggestion: the fewer the drives per >> server the better. Ideally just one drive per server :-). To explain a bit more: in an ideal world Ceph was designed to run on something like a lot of NUCs each fronting a small HDD or an SSD. The basic idea of Ceph was/should be to turn each HDD/SSD from a block storage device into an object storage device by fronting it with an OSD daemon and accessing it over ethernet. In theory one could put the OSD daemon code in firmware and have OSD-protocol instead of SCSI-protocol HDDs or SSDs with an Ethernet socket instead of a SAS/SATA/USB/NVME socket. The reason why many setups have many-OSD servers is because the capacity/price curve has a (shallow) sweet spot at around a couple dozen OSDs per server but often "penny wise is pound foolish". > Would I actually see a major advantage going from 6 nodes to > 8, from 8 to 12, or from 12 to 24? (Given 24 disks in each > case.) I guess it is not too bad to have up to half a dozen OSDs per server (depending on network bandwidth, RAM, CPU usage, size, ...) but in your case 6 per server would mean just 4 servers which seems fragile to me. More smaller OSDs and more servers increase parallelism and available banwidth which may or may not be required, but even if the parallelism and banwidth requirements are not high the main issue with too many OSDs per server (apart from resource usage) is that then there are too few "buckets" (if server == bucket) and almost any bucket loss triggers recovery/rebalance of almost every PG/object. The issue is particularly severe with EC "pools" (especially of the EC profile is wide which is a bad idea anyhow in most cases) and less bad with 3-way replication as then having just 6 buckets means that on average the loss of a bucket will trigger recovery on "only" half of the objects/PGs and so on. Recovery eats *a lot* of write IOPS (and secondarily network banwidth) so it is less costly with SSDs (those with PLP in particular).

1 0

Re: Cephadm role an its relation with Ceph relases
by John Mulligan 29 Dec '25

29 Dec '25

On Monday, December 29, 2025 10:22:13 AM Eastern Standard Time Iztok Gregori via ceph-users wrote: > On 29/12/25 14:57, Anthony D'Atri wrote: > > > > > > >> > >> > >> In my case it will be used to "adopt" the services, but after that what > >> is its intended usage? The "orchestrator" should be used to add/remove > >> new hosts and to upgrade the cluster. > > > > > Cephadm *is* the orchestrator. The nomenclature may seem misleading. > > There’s part that runs on each node as well as a Manager module, and CLI > > commands. > > Understand. My question was around the "cephadm" command, the one > documentation says that has to be installed on all the nodes to start > the "adoption" process. I'm not sure if this command is needed only for > that process or is an integral part of the cephadm/orchestrator. The cephadm system is split into two parts - a manager module and the cephadm command. The command is often called "the binary" or a little more verbosely "the cephadm binary". A system may often have two or more copies of the cephadm binary on a node. There's the copy that is installed by an administrator (typically) that is run for bootstrapping or adoption and for some administrative commands. Then there's a copy that is pushed to nodes from the mgr module. This copy of the binary is coupled to the mgr module (you can find cached copies under /var/lib/ ceph). I hope this short architectural description is helpful. :-) > > > >> I'm asking this question because my upgrade path consists in multiple > >> consecutive upgrades and I see that for the latest version of Quincy > >> (17.2.9) and for newer Ceph releases there is no cephadm "package" for > >> el8 > > > > > Check the compatibility matrix in the docs. > > > Mhmm... I suspect that I'm entering in some edge case here (end exiting > the subject of this thread). I have a cluster running Ceph Octopus > 15.2.17 on EL8 (almost all the nodes, some nodes are EL7 but can be > reinstalled/removed). My goal is to upgrade it AND to embrace cephadm as > orchestrator. My plan is/was: > > 1. Adopt cephadm (with octopus containers) > 2. Upgrade to Quincy. > 3. Upgrade to Squid. > > But now looking to the various compatibility matrices I'm start to worry > that this is not possible: > > - Podman on EL8 is version 4, but from the table > https://docs.ceph.com/en/squid/cephadm/compatibility/#cephadm-compatibility-> with-podman podman will not run with Octopus. > - There is no mention of EL8 in > https://docs.ceph.com/en/squid/start/os-recommendations/ > > I could skip the adoption with Octopus containers and go directly with > adoption+upgrade to Quincy. But I'll end up with in any case with most > hosts running EL8 which I cannot find in any matrices. > > So the best solution will be to upgrade all the machines to EL9 and > concurrently upgrade Ceph to Quincy. For the MON and MGR shouldn't be > too much of a problem (it should be possible to add a new monitor with > newer ceph release to an existing cluster), but how to reinstall the OSD > servers without data movement? > > In theory it should suffice to put the cluster in 'noout', reinstall the > host without touching the OSD disks and then recreate the OSDs. Is this > possible? > > Thank you for your time! > Iztok > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Cephadm role an its relation with Ceph relases
by Anthony D'Atri 29 Dec '25

29 Dec '25

>>> >>> In my case it will be used to "adopt" the services, but after that what is its intended usage? The "orchestrator" should be used to add/remove new hosts and to upgrade the cluster. >> Cephadm *is* the orchestrator. The nomenclature may seem misleading. There’s part that runs on each node as well as a Manager module, and CLI commands. > > Understand. My question was around the "cephadm" command, the one documentation says that has to be installed on all the nodes to start the "adoption" process. I'm not sure if this command is needed only for that process or is an integral part of the cephadm/orchestrator. Both. >>> I'm asking this question because my upgrade path consists in multiple consecutive upgrades and I see that for the latest version of Quincy (17.2.9) and for newer Ceph releases there is no cephadm "package" for el8 >> Check the compatibility matrix in the docs. > > Mhmm... I suspect that I'm entering in some edge case here (end exiting the subject of this thread). Edge cases keep this list spicy. > I have a cluster running Ceph Octopus 15.2.17 on EL8 (almost all the nodes, some nodes are EL7 but can be reinstalled/removed). Yep, you’ll want to bump those to EL8 as a first step. Depending on your setup you may be able to do that without having to recreate the OSDs. Any OSDs that are still Filestore or with bluestore_min_alloc_size > 4KiB will want to be redeployed at some point though, which given what I’ve been reading lately should probably be done on Pacific, Reef, or Quincy. Unless you have coarse-IU QLC. > My goal is to upgrade it AND to embrace cephadm as orchestrator. As well it should — it will take multiple steps to get there. > My plan is/was: > > 1. Adopt cephadm (with octopus containers) > 2. Upgrade to Quincy. > 3. Upgrade to Squid. > > But now looking to the various compatibility matrices I'm start to worry that this is not possible: > > - Podman on EL8 is version 4, but from the table https://docs.ceph.com/en/squid/cephadm/compatibility/#cephadm-compatibility… podman will not run with Octopus. I think updating to Pacific or Quincy before messing with containerization is the way to go. Pacific and Quincy are expected, vis-a-vis that table, to work with Podman 3+. Note this section in the Quincy release notes: Users should expect to see the el8 rpm subdirectory empty and the “dnf” commands are expected to fail with 17.2.9. They can choose to use 17.2.9 RPM packages for centos 8/el8 provided by CERN as a community member or stay at 17.2.7 following instructions from https://docs.ceph.com/en/latest/install/get-packages/#rhel, the ceph.repo file should point to https://download.ceph.com/rpm-17.2.7/el8 instead of https://download.ceph.com/rpm-quincy/el8 > - There is no mention of EL8 in https://docs.ceph.com/en/squid/start/os-recommendations/ You’ll want to be on EL9 before going to Squid. > I could skip the adoption with Octopus containers and go directly with adoption+upgrade to Quincy. I do recommend minimizing the changes you make in one step. Maybe something like: * Update EL7 hosts to EL8 * Update Ceph packages to Quincy * Update hosts to EL9 * Adopt OSDs * Update Ceph to Squid > But I'll end up with in any case with most hosts running EL8 which I cannot find in any matrices. EL8 is an ex-parrot. > So the best solution will be to upgrade all the machines to EL9 and concurrently upgrade Ceph to Quincy. See above re one thing at a time. Trust me on this. > For the MON and MGR shouldn't be too much of a problem (it should be possible to add a new monitor with newer ceph release to an existing cluster), but how to reinstall the OSD servers without data movement? This varies depending on your deployment, so I hesitate to make an authoritative statement. Depending on whether one has RHEL, CentOS*, and/or Rocky, the options for in-situ OS upgrades vary. Are you using LUKS-encrypted OSDs aka dmcrypt? If you can update to 8 then 9 in-situ, then the OSDs should just work. Worst case you manually activate them after the OS update. One node at a time. If you can’t update in-situ, or want to change your partitioning (get rid of swap, stop balkanizing the boot volume, etc), then you can have your OS provisioning system protect the OSD devices. You don’t have boot and OSDs on the same devices, I hope? Kickstart: ignoredisk, clearpart —drives, etc. Don’t assume that the drive enumeration and usage is the same on every system. Set the `noout` flag, reimage the OS, install the Ceph packages, reboot. The OSDs may come right up. If they don’t, try activating them. Lather, rinse, repeat. Unset noout. Worst case, wipe one system entirely and deploy new OSDs, unset noout, once recovery completes, move to the next. > In theory it should suffice to put the cluster in 'noout', reinstall the host without touching the OSD disks and then recreate the OSDs. Is this possible? I’ve done it, when circumstances required moving from Ubuntu Precise to RHEL 7 in-situ. It’s easier if you’re using ceph-volume than with ceph-disk, but possible either way. > > Thank you for your time! > Iztok > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Cephadm role an its relation with Ceph relases
by Anthony D'Atri 29 Dec '25

29 Dec '25

> > In my case it will be used to "adopt" the services, but after that what is its intended usage? The "orchestrator" should be used to add/remove new hosts and to upgrade the cluster. Cephadm *is* the orchestrator. The nomenclature may seem misleading. There’s part that runs on each node as well as a Manager module, and CLI commands. > > I'm asking this question because my upgrade path consists in multiple consecutive upgrades and I see that for the latest version of Quincy (17.2.9) and for newer Ceph releases there is no cephadm "package" for el8 Check the compatibility matrix in the docs. > (my cluster is 95% Rocky Linux 8). EL8 in general has been deprecated for a couple of years. Toolchain and infrastructure limitations limit the ability to support older OSes indefinitely. That package and container strategies are still both supported complicates this. > I think that for the adoption and the (concurrent?) update to Quincy the latest provided cephadm package for el8 (cephadm-17.2.7.el8) should be adequate (right?), I’ll have to defer to others on that. I’ve BCCd the question to the upgrade lead. > but what happens when I'll upgrade the cluster to squid/tentacle? There are no cephadm packages for that releases for EL8... All roads lead to EL9. A typical strategy is to land on a Ceph release supported by both the current and next major OS releases, update the OS, then proceed to the desired Ceph release. Ceph upgrades themselves mostly become trivial once you’re containerized.

2 1

Re: Best upgrade path?
by Eugen Block 29 Dec '25

29 Dec '25

Hi, the easiest way would be to convert the cluster to cephadm [0] and then upgrade it to Reef first, then to Squid or Tentacle. I hope you mean 19.2.3, the x.2.z release is the stable version. If you haven't done a conversion yet, I recommend to try it on a test cluster first, just to get familiar with the tools. Although I can't recall having any serious issues during conversion, I'd still advise to test it first. And make sure to read the upgrade notes for each version you're going to upgrade to. Regards, Eugen [0] https://docs.ceph.com/en/latest/cephadm/adoption/ Zitat von Jan Marek via ceph-users <ceph-users(a)ceph.io>: > Hello, > > I have a ceph cluster in version 16.1.10 on Ubuntu not managed by > cephadm. > > I would like to upgrade it to 19.1.3 and to cephadm mode. > > Please, can you advise me the best upgrade path? > > Thanks a lot. > > Sincerely > Jan Marek > -- > Ing. Jan Marek > správce IT > > Jihočeská univerzita v Českých Budějovicích > Centum informačních technologií > Branišovská 1645/31a, 370 05 České Budějovice > T/ +420 38 903 2080 > M/ +420 604 660 121 > E/ jmarek(a)jcu.cz > www.jcu.cz

1 0

Cephadm role an its relation with Ceph relases
by Iztok Gregori 29 Dec '25

29 Dec '25

Hi to all! I'm in the process to upgrading my Ceph cluster (from Octopus to Quincy, then to squid/tentacle) and, at the same time, to adopt the cluster with cephadm. I'm not sure to understand the role of cephadm and, more importantly, the relation between cephadm and Ceph versions. I understand that cephadm is used to: - bootstrap a new cluster - "prepare" a host (cheking and installing packages, as docker/podman, lvm2, ntp...) - adopt a "legacy" cluster In my case it will be used to "adopt" the services, but after that what is its intended usage? The "orchestrator" should be used to add/remove new hosts and to upgrade the cluster. I'm asking this question because my upgrade path consists in multiple consecutive upgrades and I see that for the latest version of Quincy (17.2.9) and for newer Ceph releases there is no cephadm "package" for el8 (my cluster is 95% Rocky Linux 8). I think that for the adoption and the (concurrent?) update to Quincy the latest provided cephadm package for el8 (cephadm-17.2.7.el8) should be adequate (right?), but what happens when I'll upgrade the cluster to squid/tentacle? There are no cephadm packages for that releases for EL8... My gut feeling is that it doesn't matter, I'll use cephadm to "adopt" the cluster then "orchestrator" to manage it. The new nodes will be (at least) EL9 and the older nodes will be reinstalled/phased out when needed. But I'm not sure about it, so I ask for your wisdom. Thank you for your attention! Iztok -- Iztok Gregori ICT Systems and Services Elettra - Sincrotrone Trieste S.C.p.A. Telephone: +39 040 3758948 http://www.elettra.eu

1 0

Best upgrade path?
by Jan Marek 29 Dec '25

29 Dec '25

Hello, I have a ceph cluster in version 16.1.10 on Ubuntu not managed by cephadm. I would like to upgrade it to 19.1.3 and to cephadm mode. Please, can you advise me the best upgrade path? Thanks a lot. Sincerely Jan Marek -- Ing. Jan Marek správce IT Jihočeská univerzita v Českých Budějovicích Centum informačních technologií Branišovská 1645/31a, 370 05 České Budějovice T/ +420 38 903 2080 M/ +420 604 660 121 E/ jmarek(a)jcu.cz www.jcu.cz

1 0