- ceph-users - lists.ceph.io

Re: Expanding cluster
by Anthony D'Atri 28 Dec '25

28 Dec '25

>> >> >>>> I strongly recommend running 5 mons. Given how you phrase the above it >>>> seems as though you might mean 3 dedicated mon nodes? If so I’d still >> run >>>> 5 mons, add the mon cephadm host label to two of the OSD nodes. >>>> >>> >>> Right now. only one mon is "active" and used all of the time. You would >>> still run 5 mons anyway? >> >> Are you conflating mons and mgrs? Mons form a quorum, you need more than >> half of them able to reach each other. >> > > No. So you’re running a total of 1 mon daemon? > >>> I checked bluestore_min_alloc_size and indeed there is a 4K value set for >>> all of them. Even on new ones. >> >> Groovy. You checked with “ceph osd metadata”? >> > > Yes. Should it be set to different size/value? No. The default used to be 64 KiB; in the Octopus / Pacific era code improvements were made that allowed the defaults to change to 4 KiB. This is baked into each OSD, which is why I asked how I did: you can inject or set in central config or ceph.conf a value different from how a given OSD was built, but it doesn’t actually take effect. This is a subtlety often missed, in fact just last week I worked with a vendor doc team to correct their docs.

2 1

Re: Expanding cluster
by Rok Jaklič 28 Dec '25

28 Dec '25

On Sun, Dec 28, 2025 at 5:01 PM Anthony D'Atri via ceph-users < ceph-users(a)ceph.io> wrote: > > > >> I strongly recommend running 5 mons. Given how you phrase the above it > >> seems as though you might mean 3 dedicated mon nodes? If so I’d still > run > >> 5 mons, add the mon cephadm host label to two of the OSD nodes. > >> > > > > Right now. only one mon is "active" and used all of the time. You would > > still run 5 mons anyway? > > Are you conflating mons and mgrs? Mons form a quorum, you need more than > half of them able to reach each other. > No. > > > >> Do you have your index pool on SSDs? How old are those existing OSDs? > If > >> they were deployed in the 6 TB spinner era, they may have the old > >> bluestore_min_alloc_size value baked in. Check `ceph osd metadata`, > see if > >> any are not 4KiB. If you have or will have a significant population of > >> small RGW objects, serially redeploying the old OSDs — especially if > >> they’re Filestore — would have advantages. > >> > > > > Index pool is on HDD. HDDs (mostly Seagate) are actually quite old > > At 6 TB that was kinda assumed ;) > > > > from 2016 and lately we are seeing a bigger failure rate, > > That’s very much to be expected. > However they were deployed around 2020, so they have little less working hours. > > > I checked bluestore_min_alloc_size and indeed there is a 4K value set for > > all of them. Even on new ones. > > Groovy. You checked with “ceph osd metadata”? > Yes. Should it be set to different size/value?

1 0

Re: Expanding cluster
by Anthony D'Atri 28 Dec '25

28 Dec '25

>> I strongly recommend running 5 mons. Given how you phrase the above it >> seems as though you might mean 3 dedicated mon nodes? If so I’d still run >> 5 mons, add the mon cephadm host label to two of the OSD nodes. >> > > Right now. only one mon is "active" and used all of the time. You would > still run 5 mons anyway? Are you conflating mons and mgrs? Mons form a quorum, you need more than half of them able to reach each other. Yes I would run 5 mons, to survive a double failure. At at least 3 mgrs. > > NVME is used only for OS. I was considering offloading WAL+DB to NVME, but > that would mean 3-5NVME per OSD/JBOD node and we do not have configuration > like that right now. Nothing stops you from adding those, though for many applications the benefit isn’t worth it. > >> Do you have your index pool on SSDs? How old are those existing OSDs? If >> they were deployed in the 6 TB spinner era, they may have the old >> bluestore_min_alloc_size value baked in. Check `ceph osd metadata`, see if >> any are not 4KiB. If you have or will have a significant population of >> small RGW objects, serially redeploying the old OSDs — especially if >> they’re Filestore — would have advantages. >> > > Index pool is on HDD. HDDs (mostly Seagate) are actually quite old At 6 TB that was kinda assumed ;) > from 2016 and lately we are seeing a bigger failure rate, That’s very much to be expected. > I checked bluestore_min_alloc_size and indeed there is a 4K value set for > all of them. Even on new ones. Groovy. You checked with “ceph osd metadata”? > > We are replacing failed disks on "same OSD id" with something like: > > # destroy failed osd > systemctl stop ceph-osd@{id} > ceph osd down osd.{id} > ceph osd destroy {id} --yes-i-really-mean-it > > # prepare new one on same osd id > ceph-volume lvm prepare --osd-id {id} --data /dev/sdX > ceph-volume lvm list > ceph-volume lvm activate {id} {osd fsid} > > Is this "a problem"? Not at all.

1 0

Re: Expanding cluster
by Anthony D'Atri 28 Dec '25

28 Dec '25

> we are planning to expand our ceph cluster (reef) from 8 OSD nodes and 3 > monitors to 12 OSD nodes and 3 monitors. I strongly recommend running 5 mons. Given how you phrase the above it seems as though you might mean 3 dedicated mon nodes? If so I’d still run 5 mons, add the mon cephadm host label to two of the OSD nodes. > Currently each OSD node has a JBOD > with 28 6TB HDD disks and we are using EC 3+2, since at first there were > just 5 OSD nodes. Situations like that are why I often suggest starting with at least 7 nodes, even if they’re half populated, that lets you do 4+2 EC without CRUSH gymnastics. > No NVMEs are used since we do not have enough of them on > one particular node. ? Do you mean as OSDs, or to offload WAL+DB? Either way isn’t an all or nothing proposition. > Bare metal, no containers I suggest considering migrating to cephadm. The orchestrator works pretty well these days. > we build binaries for Alma > Linux ourselves. We are using RGW S3 frontend and have on average around > 50MB of upload/write during the day with spikes to up to 700MB. > > It works well. Do you have your index pool on SSDs? How old are those existing OSDs? If they were deployed in the 6 TB spinner era, they may have the old bluestore_min_alloc_size value baked in. Check `ceph osd metadata`, see if any are not 4KiB. If you have or will have a significant population of small RGW objects, serially redeploying the old OSDs — especially if they’re Filestore — would have advantages. > New osd nodes will have JBOD with 30 12TB disks. Also no NVMEs for > OSDs, just for OS. > > We will add each new node to the cluster then adding new OSD one by one > (until one rebalances), then draining old OSDs one by one (until one > rebalances). Will be a slow process, but we do not want to overload > anything. That will work, but will be inefficient. I suggest instead the process described here: https://ceph.io/assets/pdfs/events/2024/ceph-days-nyc/Mastering%20Ceph%20Op… Basically, disable rebalancing, add all the OSDs. No data moves. upmap PGs to where they are currently, and let the balancer incrementally do the work. You can modulate that process in the usual fashion. > I am thinking of creating a new EC pool with 8+3 (leaving one node for > "backup") and migrating user per user to the new EC pool. Glad to see that you’re planning for >= k+m+1 failure domains, if you only had 11 hosts total, you would effectively strand a lot of the raw capacity on the new systems. Through Tentacle you can add a second RGW storage class on the new data pool and use LC policies to migrate the data transparently, and set the default storageclass for given users to the new one. In Tentacle, turn on EC optimizations on the EC pools before moving data, so that new writes will save you the prior zero-padding. With Umbrella, due out … sometime next year, it is planned to be able to do such migrations transparently at the RADOS level. AIUI the old pool will need to remain permanently, but it’ll be worth it.

2 1

Expanding cluster
by Rok Jaklič 27 Dec '25

27 Dec '25

Hi, we are planning to expand our ceph cluster (reef) from 8 OSD nodes and 3 monitors to 12 OSD nodes and 3 monitors. Currently each OSD node has a JBOD with 28 6TB HDD disks and we are using EC 3+2, since at first there were just 5 OSD nodes. No NVMEs are used since we do not have enough of them on one particular node. Bare metal, no containers, we build binaries for Alma Linux ourselves. We are using RGW S3 frontend and have on average around 50MB of upload/write during the day with spikes to up to 700MB. It works well. New osd nodes will have JBOD with 30 12TB disks. Also no NVMEs for OSDs, just for OS. We will add each new node to the cluster then adding new OSD one by one (until one rebalances), then draining old OSDs one by one (until one rebalances). Will be a slow process, but we do not want to overload anything. I am thinking of creating a new EC pool with 8+3 (leaving one node for "backup") and migrating user per user to the new EC pool. Has anyone had a similar experience? Any thoughts and advice appreciated. Kind regards, Rok

1 0

Re: Ceph system disk on non-raid drive
by Anthony D'Atri 24 Dec '25

24 Dec '25

> Just my 2 cents, please wait for a second opinion. I wouldn't want to give incorrect/incomplete advice. You must be new to the interwebs ;) > It's best practice to present Ceph the raw block devices on which you're going to deploy your OSDs. I think the OP was asking about the boot / OS volume, not OSDs > No (active) RAID controller in between, just the raw device. You can use a RAID controller See my history of posts about those things. Nothing but trouble and expense. Sometimes a pure-NVMe system without any HBA costs *less* than one with a RAID HBA. Almost nobody monitors RAID HBAs properly, or keeps the firmware updated. > but configure it so that the OS sees the RAW device, not a block device the raid controller itself presents to the OS. I think i't called IT-mode or hbamode. Integrated Target HBAs are all passthrough all the time. IR Integrated RAID are capable of virtual drives. Sometimes one sees people flashing IT firmware on IR HBAs for better performance, though this practice is fraught. These days IR HBAs can usually pass-through non-VD drives so the OS sees them directly, which makes for improved observability and the ability to do wacky things like update firmware on them. In the past one would have to wrap every drive in degenerate VD just for the OS to see it. > Please someone correct me if I'm wrong but AFAIK, it doesn't really matter on which block devices you're running your OS, including where the Ceph binaries are located and/or /var/lib/ceph. Just make sure it's reliable storage and sufficiently fast like you would spec your regular servers. Sharing boot with data is in general not a great strategy. Boot drives are not designed for an OSD duty cycle, and I’ve seen them cause Ceph daemon impact when too much other crap is presented to them. Separate boot drives also make it a lot easier to reimage the OS without losing OSDs. > Also make sure to choose suitable block devices for OSDs! HDDs are definitely possible and used in Ceph clusters but result in a relatively slow cluster unless you've got a massive amount of them. We lose money with every sale but make up for it in volume! > Eg, if you want to use it for archival purposes, and performance doesn't really matter, it might work. At scale, including things like DC racks, coarse-IU QLC can actually cost less than HDDs, and less risky because it doesn’t take a month or more to recover from a failure. > For SSDs, definitely go enterprise class with PLP! Don't bargain on PLP and don't go for consumer class SSDs. Its a big big no-no you definitely want to avoid! Absolutely. I know of a cluster with the CephFS metadata pool on Sabrents. Ticking bomb. > > And in case you skipped the paragraph above: Don't compromise on PLP and go for Enterprise class SSDs, pretty please 😉🙂. > > And while thinking about SSDs, our Ceph support partner once told me there are certain SSDs that are known to cause problems. I don't know the specifics of it, but once you're speccing your cluster, I think it's a good idea to post your proposal here or in the Slack channel for review. Who knows, someone might possibly chime in with an actual experience with your hardware. We have to be careful when discussing specific companies, but most SSD failures are firmware and can be addressed in-situ with an upgrade. I’ve been through this myself. > > Tentacle has just been released, while it's a stable release you might not want to go Tentacle just yet until there's at least one point release. > > There's Squid, but you might want to be careful and disable bluestore_elastic_shared_blobs before you start deploying OSDs especially if you're looking to use Erasure Coded pools rather than replicated pools. … or deploy on the latest Reef then upgrade. > > I discussed about some practices for ceph node hardware configuration and > specifically about system disk on non-RAID drive (e.g. internal m2). It’s a strategy. > > Failure domain is host or rack, nvme cluster. > > Is it OK or is it one of possible and "adopted" way to deploy ceph nodes > with system disk without RAID1 (hw or sw)? Do you have some experience with > such nodes? Sure, if you like. I would shy away from doing that on smaller clusters, where the loss of a whole failure domain means, say, 5% or more of cluster capacity. If you have 250 nodes in your cluster, then losing one isn’t nearly so big a deal. HBA RAID is IMHO mostly an artifact of the days of a graphical desktop OS mistaken for a server OS. I would much rather use MD; ymmv.

1 0

Re: Ceph system disk on non-raid drive
by Wannes Smet 24 Dec '25

24 Dec '25

Hi, Just my 2 cents, please wait for a second opinion. I wouldn't want to give incorrect/incomplete advice. It's best practice to present Ceph the raw block devices on which you're going to deploy your OSDs. No (active) RAID controller in between, just the raw device. You can use a RAID controller but configure it so that the OS sees the RAW device, not a block device the raid controller itself presents to the OS. I think i't called IT-mode or hbamode. Please someone correct me if I'm wrong but AFAIK, it doesn't really matter on which block devices you're running your OS, including where the Ceph binaries are located and/or /var/lib/ceph. Just make sure it's reliable storage and sufficiently fast like you would spec your regular servers. Also make sure to choose suitable block devices for OSDs! HDDs are definitely possible and used in Ceph clusters but result in a relatively slow cluster unless you've got a massive amount of them. Eg, if you want to use it for archival purposes, and performance doesn't really matter, it might work. For SSDs, definitely go enterprise class with PLP! Don't bargain on PLP and don't go for consumer class SSDs. Its a big big no-no you definitely want to avoid! And in case you skipped the paragraph above: Don't compromise on PLP and go for Enterprise class SSDs, pretty please 😉🙂. And while thinking about SSDs, our Ceph support partner once told me there are certain SSDs that are known to cause problems. I don't know the specifics of it, but once you're speccing your cluster, I think it's a good idea to post your proposal here or in the Slack channel for review. Who knows, someone might possibly chime in with an actual experience with your hardware. Tentacle has just been released, while it's a stable release you might not want to go Tentacle just yet until there's at least one point release. There's Squid, but you might want to be careful and disable bluestore_elastic_shared_blobs before you start deploying OSDs especially if you're looking to use Erasure Coded pools rather than replicated pools. I wasn't aware while deploying my cluster of the bluestore_elastic_shared_blobs bug. I did redeploy all my OSDs just to be on the safe side which took around a week to complete. And last but not least: Here is the official documentation which provides a good overview of what you might want for hardware: https://docs.ceph.com/en/tentacle/start/hardware-recommendations/ You'll probably pick up on some more dos and don'ts. Cheers and happy holidays! Wannes ________________________________ From: Nekopilot via ceph-users <ceph-users(a)ceph.io> Sent: Wednesday, December 24, 2025 11:12 To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Ceph system disk on non-raid drive Dear ceph users I discussed about some practices for ceph node hardware configuration and specifically about system disk on non-RAID drive (e. g. internal m2). Failure domain is host or rack, nvme cluster. Is it OK or is it one of possible and ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Dear ceph users I discussed about some practices for ceph node hardware configuration and specifically about system disk on non-RAID drive (e.g. internal m2). Failure domain is host or rack, nvme cluster. Is it OK or is it one of possible and "adopted" way to deploy ceph nodes with system disk without RAID1 (hw or sw)? Do you have some experience with such nodes? _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Ceph system disk on non-raid drive
by Eugen Block 24 Dec '25

24 Dec '25

Hi, we switched to such a setup to have one more slot for OSDs per server when we needed to expand our cluster. We have autoinstallation profiles for all kinds of servers, if a node fails (os-wise) we’re able to deploy it within half an hour or so. So for us it works. Regards, Eugen Zitat von Nekopilot via ceph-users <ceph-users(a)ceph.io>: > Dear ceph users > > I discussed about some practices for ceph node hardware configuration and > specifically about system disk on non-RAID drive (e.g. internal m2). > > Failure domain is host or rack, nvme cluster. > > Is it OK or is it one of possible and "adopted" way to deploy ceph nodes > with system disk without RAID1 (hw or sw)? Do you have some experience with > such nodes? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Ceph system disk on non-raid drive
by Nekopilot 24 Dec '25

24 Dec '25

Dear ceph users I discussed about some practices for ceph node hardware configuration and specifically about system disk on non-RAID drive (e.g. internal m2). Failure domain is host or rack, nvme cluster. Is it OK or is it one of possible and "adopted" way to deploy ceph nodes with system disk without RAID1 (hw or sw)? Do you have some experience with such nodes?

1 0

Re: Happy holidays to all the Cephalopods out there!
by Alex Gorbachev 23 Dec '25

23 Dec '25

Wishing you a Tentacool New Year! May all your problems be Luminous in hindsight. -- Alex Gorbachev ISS On Tue, Dec 23, 2025 at 4:44 AM Eugen Block via ceph-users < ceph-users(a)ceph.io> wrote: > And a happy new year, too! > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0