nodes with high density of OSDs
Hello everybody! I have a 4 nodes with 112 OSDs each and 18.2.4. OSD consist of db on SSD and data on HDD For some reason, when I reboot node, not all OSDs get up because some VG or LV are not active. To make it alive again I manually do vgchange -ay $VG_NAME or lvchange -ay $LV_NAME. I suspect it is linked to high amount of vg/lv but cannot find an answer. Maybe you can gimme a hint how to struggle it over?
Hi Alex, Which OS? I had the same problem regarding not automatic activation of LVM's on an older version of Ubuntu. I never found a workaround except by upgrading to a newer release.
-----Oorspronkelijk bericht----- Van: Alex from North <service.plant@ya.ru> Verzonden: donderdag 10 april 2025 13:17 Aan: ceph-users@ceph.io Onderwerp: [ceph-users] nodes with high density of OSDs
Hello everybody! I have a 4 nodes with 112 OSDs each and 18.2.4. OSD consist of db on SSD and data on HDD For some reason, when I reboot node, not all OSDs get up because some VG or LV are not active. To make it alive again I manually do vgchange -ay $VG_NAME or lvchange -ay $LV_NAME.
I suspect it is linked to high amount of vg/lv but cannot find an answer.
Maybe you can gimme a hint how to struggle it over? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
That's quite a large number of storage units per machine. My suspicion is that since you have apparently an unusually high number of LVs coming online at boot, the time it takes to linearly activate them is long enough to overlap with the point in time that ceph starts bringing up its storage-dependent components. Likely not only OSDs, but other resources that might keep internal databases and the like. The cure for that under systemd would be to make Ceph - or at least its storage-dependent services - wait on LV availability. The fun part is figuring out how to do that. Offhand, I don't know what in systemd controls the activation of LVM resources and it's almost certainly being done asynchronously, so you'd need to provide a detector service that could determine when things were available. Then you'd have to tweak Ceph not to start until the safe time has arrived. You might be able to edit the master ceph target to add such a dependency using an /etc/systemd/system override, but admittedly that doesn't cover allowing everything to come up as soon as possible but no sooner. In particular, it would be hard to edit the individual OSDs to wait on their LVs, as the systemd components for OSDs on an administered system are constructed dynamically and do not persist when the system reboots, so it would likely require a worst-case delay. Regards, Tim On 4/10/25 07:45, Alex from North wrote:
Hello Dominique! Os is quite new - Ubuntu 22.04 with all the latest upgrades. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I have a 4 nodes with 112 OSDs each [...]
As an aside I rekon that is not such a good idea as Ceph was designed for one-small-OSD per small-server and lots of them, but lots of people of course know better.
Maybe you can gimme a hint how to struggle it over?
That is not so much a Ceph question but a distribution question anyhow there are two possible hints that occur to me: * In most distributions the automatic activation of block devices is done by the kernel plus 'udevd' rules and/or 'systemd' units. * There are timeouts for activation of storage devices and on a system with many, depending on type etc., there may be a default setting to activate them serially instead of in parallel to prevent sudden power consumption and other surges, so some devices may not activate because of timeouts. You can start by asking the sysadmin for those machines to look at system logs (distribution dependent) for storage device activation reports to confirm whether the guesses above apply to your situation and if confirmed you can ask them to change the relevant settings for the distribution used.
Peter, I don't think udev factors in based on the original question. Firstly, because I'm not sure udev deals with permanently-attached devices (it's more for hot-swap items). Secondly, because the original complaint mentioned LVM specifically. I agree that the hosts seem overloaded, by the way. It sounds like large disks are being subdivided into many smaller disks, which would be bad for Ceph to do on HDDs, and while SSDs don't have the seek and rotational liabilities of HDDs, it's still questionable as to how many connections you really should be making to one physical unit that way. Ceph, for reasons I never discovered prefers that you create OSDs that either own an entire physical disk or an LVM Logical Volume, but NOT a disk partition. I find it curious, since LVs aren't necessarily contiguous space (again, more of a liability for HDDs than SSDs). unlike traditional partitions, but there you are. Incidentally, LVs are contained in Volume Groups, and the whole can end up with parts scattered over multiple Physical Volumes (PVs). When an LVM-supporting OS boots, part of the process is to run an lvscan (lvscan -ay) to locate and activate Logical Volumes, and from the information given, it's assumed that the lvscan process hasn't completed before Ceph starts up and begins trying to use them. The boot lvscan is normally pretty quick, since it would be rare to have more than a dozen or so LVs in the system. But in this case, more than 100 LVs are being configured at boot time and the systemd boot process doesn't currently account for the extra time needed to do that. If I haven't got my facts too badly scrambled, LVs end up being mapped to dm devices, but that's something I normally only pay attention to when hardware isn't behaving so I'm not really expert on that. Hope that helps, Tim On 4/10/25 16:43, Peter Grandi wrote:
I have a 4 nodes with 112 OSDs each [...] As an aside I rekon that is not such a good idea as Ceph was designed for one-small-OSD per small-server and lots of them, but lots of people of course know better.
Maybe you can gimme a hint how to struggle it over? That is not so much a Ceph question but a distribution question anyhow there are two possible hints that occur to me:
* In most distributions the automatic activation of block devices is done by the kernel plus 'udevd' rules and/or 'systemd' units.
* There are timeouts for activation of storage devices and on a system with many, depending on type etc., there may be a default setting to activate them serially instead of in parallel to prevent sudden power consumption and other surges, so some devices may not activate because of timeouts.
You can start by asking the sysadmin for those machines to look at system logs (distribution dependent) for storage device activation reports to confirm whether the guesses above apply to your situation and if confirmed you can ask them to change the relevant settings for the distribution used. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Filestore IIRC used partitions, with cute hex GPT types for various states and roles. Udev activation was sometimes problematic, and LVM tags are more flexible and reliable than the prior approach. There no doubt is more to it but that’s what I recall.
On Apr 10, 2025, at 9:11 PM, Tim Holloway <timh@mousetech.com> wrote:
Peter,
I don't think udev factors in based on the original question. Firstly, because I'm not sure udev deals with permanently-attached devices (it's more for hot-swap items). Secondly, because the original complaint mentioned LVM specifically.
I agree that the hosts seem overloaded, by the way. It sounds like large disks are being subdivided into many smaller disks, which would be bad for Ceph to do on HDDs, and while SSDs don't have the seek and rotational liabilities of HDDs, it's still questionable as to how many connections you really should be making to one physical unit that way.
Ceph, for reasons I never discovered prefers that you create OSDs that either own an entire physical disk or an LVM Logical Volume, but NOT a disk partition. I find it curious, since LVs aren't necessarily contiguous space (again, more of a liability for HDDs than SSDs). unlike traditional partitions, but there you are. Incidentally, LVs are contained in Volume Groups, and the whole can end up with parts scattered over multiple Physical Volumes (PVs).
When an LVM-supporting OS boots, part of the process is to run an lvscan (lvscan -ay) to locate and activate Logical Volumes, and from the information given, it's assumed that the lvscan process hasn't completed before Ceph starts up and begins trying to use them. The boot lvscan is normally pretty quick, since it would be rare to have more than a dozen or so LVs in the system.
But in this case, more than 100 LVs are being configured at boot time and the systemd boot process doesn't currently account for the extra time needed to do that.
If I haven't got my facts too badly scrambled, LVs end up being mapped to dm devices, but that's something I normally only pay attention to when hardware isn't behaving so I'm not really expert on that.
Hope that helps,
Tim
On 4/10/25 16:43, Peter Grandi wrote:
I have a 4 nodes with 112 OSDs each [...] As an aside I rekon that is not such a good idea as Ceph was designed for one-small-OSD per small-server and lots of them, but lots of people of course know better.
Maybe you can gimme a hint how to struggle it over? That is not so much a Ceph question but a distribution question anyhow there are two possible hints that occur to me:
* In most distributions the automatic activation of block devices is done by the kernel plus 'udevd' rules and/or 'systemd' units.
* There are timeouts for activation of storage devices and on a system with many, depending on type etc., there may be a default setting to activate them serially instead of in parallel to prevent sudden power consumption and other surges, so some devices may not activate because of timeouts.
You can start by asking the sysadmin for those machines to look at system logs (distribution dependent) for storage device activation reports to confirm whether the guesses above apply to your situation and if confirmed you can ask them to change the relevant settings for the distribution used. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Den fre 11 apr. 2025 kl 09:59 skrev Anthony D'Atri <anthony.datri@gmail.com>:
Filestore IIRC used partitions, with cute hex GPT types for various states and roles. Udev activation was sometimes problematic, and LVM tags are more flexible and reliable than the prior approach. There no doubt is more to it but that’s what I recall.
Filestore used to have softlinks towards the journal device (if used) which pointed to sdX where that X of course would jump around if you changed the number of drives on the box, or the kernel disk detection order changed, breaking the OSD. -- May the most significant bit of your life be positive.
I thought those links were to the by-uuid paths for that reason?
On Apr 11, 2025, at 6:39 AM, Janne Johansson <icepic.dz@gmail.com> wrote:
Den fre 11 apr. 2025 kl 09:59 skrev Anthony D'Atri <anthony.datri@gmail.com>:
Filestore IIRC used partitions, with cute hex GPT types for various states and roles. Udev activation was sometimes problematic, and LVM tags are more flexible and reliable than the prior approach. There no doubt is more to it but that’s what I recall.
Filestore used to have softlinks towards the journal device (if used) which pointed to sdX where that X of course would jump around if you changed the number of drives on the box, or the kernel disk detection order changed, breaking the OSD.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I just checked an OSD and the "block" entry is indeed linked to storage using a /dev/mapper uuid LV, not a /dev/device. When ceph builds an LV-based OSD, it creates a VG whose name is "ceph-uuuuu", where "uuuu" is a UUID, and an LV named "osd-block-vvvv", where "vvvv" is also a uuid. So although you'd map the osd to something like /dev/vdb in a VM, the actual name ceph uses is uuid-based (and lvm-based) and thus not subject to change with alterations in the hardware as the uuids are part of the metadata in VGs and LVs created by ceph. Since I got that from a VM, I can't vouch for all cases, but I thought it especially interesting that a ceph was creating LVM counterparts even for devices that were not themselves LVM-based. And yeah, I understand that it's the amount of OSD replicate data that counts more than the number of hosts, but when an entire host goes down and there are few hosts, that can take a large bite out of the replicas. Tim On 4/11/25 10:36, Anthony D'Atri wrote:
I thought those links were to the by-uuid paths for that reason?
On Apr 11, 2025, at 6:39 AM, Janne Johansson <icepic.dz@gmail.com> wrote:
Den fre 11 apr. 2025 kl 09:59 skrev Anthony D'Atri <anthony.datri@gmail.com>:
Filestore IIRC used partitions, with cute hex GPT types for various states and roles. Udev activation was sometimes problematic, and LVM tags are more flexible and reliable than the prior approach. There no doubt is more to it but that’s what I recall. Filestore used to have softlinks towards the journal device (if used) which pointed to sdX where that X of course would jump around if you changed the number of drives on the box, or the kernel disk detection order changed, breaking the OSD.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Filestore, pre-ceph-volume may have been entirely different. IIRC LVM is used these days to exploit persistent metadata tags.
On Apr 11, 2025, at 4:03 PM, Tim Holloway <timh@mousetech.com> wrote:
I just checked an OSD and the "block" entry is indeed linked to storage using a /dev/mapper uuid LV, not a /dev/device. When ceph builds an LV-based OSD, it creates a VG whose name is "ceph-uuuuu", where "uuuu" is a UUID, and an LV named "osd-block-vvvv", where "vvvv" is also a uuid. So although you'd map the osd to something like /dev/vdb in a VM, the actual name ceph uses is uuid-based (and lvm-based) and thus not subject to change with alterations in the hardware as the uuids are part of the metadata in VGs and LVs created by ceph.
Since I got that from a VM, I can't vouch for all cases, but I thought it especially interesting that a ceph was creating LVM counterparts even for devices that were not themselves LVM-based.
And yeah, I understand that it's the amount of OSD replicate data that counts more than the number of hosts, but when an entire host goes down and there are few hosts, that can take a large bite out of the replicas.
Tim
On 4/11/25 10:36, Anthony D'Atri wrote:
I thought those links were to the by-uuid paths for that reason?
On Apr 11, 2025, at 6:39 AM, Janne Johansson <icepic.dz@gmail.com> wrote:
Den fre 11 apr. 2025 kl 09:59 skrev Anthony D'Atri <anthony.datri@gmail.com>:
Filestore IIRC used partitions, with cute hex GPT types for various states and roles. Udev activation was sometimes problematic, and LVM tags are more flexible and reliable than the prior approach. There no doubt is more to it but that’s what I recall. Filestore used to have softlinks towards the journal device (if used) which pointed to sdX where that X of course would jump around if you changed the number of drives on the box, or the kernel disk detection order changed, breaking the OSD.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Apparently those UUIDs aren't as reliable as I thought. I've had problems with a server box that hosts a ceph VM. Looks like the mobo disk controller is unreliable AND one of the disks passes SMART but has interface problems. So I moved the disks to an alternate box. Between relocation and dropping the one disk, neither of the 2 OSDs for that host will come up. If everything was running solely on static UUIDs, the good disk should have been findable even if its physical disk device name shifted. But it wasn't. Which brings up something I've wondered about for some time. Shouldn't it be possible for OSDs to be portable? That is, if a box goes bad, in theory I should be able to remove the drive and jack it into a hot-swap bay on another server and have that server able to import the relocated OSD. True, the metadata for an OSD is currently located on its host, but it seems like it should be possible to carry a copy on the actual device. Tim On 4/11/25 16:23, Anthony D'Atri wrote:
Filestore, pre-ceph-volume may have been entirely different. IIRC LVM is used these days to exploit persistent metadata tags.
On Apr 11, 2025, at 4:03 PM, Tim Holloway <timh@mousetech.com> wrote:
I just checked an OSD and the "block" entry is indeed linked to storage using a /dev/mapper uuid LV, not a /dev/device. When ceph builds an LV-based OSD, it creates a VG whose name is "ceph-uuuuu", where "uuuu" is a UUID, and an LV named "osd-block-vvvv", where "vvvv" is also a uuid. So although you'd map the osd to something like /dev/vdb in a VM, the actual name ceph uses is uuid-based (and lvm-based) and thus not subject to change with alterations in the hardware as the uuids are part of the metadata in VGs and LVs created by ceph.
Since I got that from a VM, I can't vouch for all cases, but I thought it especially interesting that a ceph was creating LVM counterparts even for devices that were not themselves LVM-based.
And yeah, I understand that it's the amount of OSD replicate data that counts more than the number of hosts, but when an entire host goes down and there are few hosts, that can take a large bite out of the replicas.
Tim
On 4/11/25 10:36, Anthony D'Atri wrote:
I thought those links were to the by-uuid paths for that reason?
On Apr 11, 2025, at 6:39 AM, Janne Johansson <icepic.dz@gmail.com> wrote:
Den fre 11 apr. 2025 kl 09:59 skrev Anthony D'Atri <anthony.datri@gmail.com>:
Filestore IIRC used partitions, with cute hex GPT types for various states and roles. Udev activation was sometimes problematic, and LVM tags are more flexible and reliable than the prior approach. There no doubt is more to it but that’s what I recall. Filestore used to have softlinks towards the journal device (if used) which pointed to sdX where that X of course would jump around if you changed the number of drives on the box, or the kernel disk detection order changed, breaking the OSD.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 12/4/25 20:56, Tim Holloway wrote:
Which brings up something I've wondered about for some time. Shouldn't it be possible for OSDs to be portable? That is, if a box goes bad, in theory I should be able to remove the drive and jack it into a hot-swap bay on another server and have that server able to import the relocated OSD.
True, the metadata for an OSD is currently located on its host, but it seems like it should be possible to carry a copy on the actual device.
It seems to me the theoretical way to do this would be to `ceph-volume lvm migrate` it to the HDD, then move the HDD to a new machine. That would require the box not be fatally bad though, so I'm not sure how much that helps. Are there lower level tools which could be used instead of the above? Greg.
One possibility would be so have ceph simply set aside space on the OSD and echo the metadata there automatically. Then a mechanism could scan for un-adopted drives and import as needed. So even a dead host would be OK as long as the device/LV was still usable. I've migrated non-ceph LVs, after all. Tim On 4/12/25 10:25, Gregory Orange wrote:
On 12/4/25 20:56, Tim Holloway wrote:
Which brings up something I've wondered about for some time. Shouldn't it be possible for OSDs to be portable? That is, if a box goes bad, in theory I should be able to remove the drive and jack it into a hot-swap bay on another server and have that server able to import the relocated OSD.
True, the metadata for an OSD is currently located on its host, but it seems like it should be possible to carry a copy on the actual device. It seems to me the theoretical way to do this would be to `ceph-volume lvm migrate` it to the HDD, then move the HDD to a new machine.
That would require the box not be fatally bad though, so I'm not sure how much that helps. Are there lower level tools which could be used instead of the above?
Greg. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Apparently those UUIDs aren't as reliable as I thought.
I've had problems with a server box that hosts a ceph VM.
VM?
Looks like the mobo disk controller is unreliable
Lemme guess, it is an IR / RoC / RAID type? As opposed to JBOB / IT? If the former and it’s an LSI SKU as most are, I’d love if you could send me privately the output of storcli64 /c0 show termlog >/tmp/termlog.txt Sometimes flakiness is actually with the drive backplane, especially when it has an embedded expander. In either case, updating HBA firmware sometimes makes a real difference. And drive firmware.
AND one of the disks passes SMART
I’m curious if it shows SATA downshifts.
but has interface problems. So I moved the disks to an alternate box.
Between relocation and dropping the one disk, neither of the 2 OSDs for that host will come up. If everything was running solely on static UUIDs, the good disk should have been findable even if its physical disk device name shifted. But it wasn't.
Did you try ceph-volume lvm activate —all ?
Which brings up something I've wondered about for some time. Shouldn't it be possible for OSDs to be portable?
I haven’t tried it much, but that *should* be true, modulo CRUSH location.
That is, if a box goes bad, in theory I should be able to remove the drive and jack it into a hot-swap bay on another server and have that server able to import the relocated OSD.
I’ve effectively done a chassis swap, moving all the drives including the boot volume, but that admittedly was in the ceph-disk days.
True, the metadata for an OSD is currently located on its host, but it seems like it should be possible to carry a copy on the actual device.
My limited understanding is that *is* the case with LVM.
Tim
On 4/11/25 16:23, Anthony D'Atri wrote:
Filestore, pre-ceph-volume may have been entirely different. IIRC LVM is used these days to exploit persistent metadata tags.
On Apr 11, 2025, at 4:03 PM, Tim Holloway <timh@mousetech.com> wrote:
I just checked an OSD and the "block" entry is indeed linked to storage using a /dev/mapper uuid LV, not a /dev/device. When ceph builds an LV-based OSD, it creates a VG whose name is "ceph-uuuuu", where "uuuu" is a UUID, and an LV named "osd-block-vvvv", where "vvvv" is also a uuid. So although you'd map the osd to something like /dev/vdb in a VM, the actual name ceph uses is uuid-based (and lvm-based) and thus not subject to change with alterations in the hardware as the uuids are part of the metadata in VGs and LVs created by ceph.
Since I got that from a VM, I can't vouch for all cases, but I thought it especially interesting that a ceph was creating LVM counterparts even for devices that were not themselves LVM-based.
And yeah, I understand that it's the amount of OSD replicate data that counts more than the number of hosts, but when an entire host goes down and there are few hosts, that can take a large bite out of the replicas.
Tim
On 4/11/25 10:36, Anthony D'Atri wrote:
I thought those links were to the by-uuid paths for that reason?
On Apr 11, 2025, at 6:39 AM, Janne Johansson <icepic.dz@gmail.com> wrote:
Den fre 11 apr. 2025 kl 09:59 skrev Anthony D'Atri <anthony.datri@gmail.com>:
Filestore IIRC used partitions, with cute hex GPT types for various states and roles. Udev activation was sometimes problematic, and LVM tags are more flexible and reliable than the prior approach. There no doubt is more to it but that’s what I recall. Filestore used to have softlinks towards the journal device (if used) which pointed to sdX where that X of course would jump around if you changed the number of drives on the box, or the kernel disk detection order changed, breaking the OSD.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
When I first migrated to Ceph, my servers were all running CentOS 7, which I (wrongly) thought could not handle anything above Octopus, and on top of that, I initially did legacy installs. So in order to run Pacific and to keep the overall clutter in the physical box configuration down, I made my Ceph hosts VMs. With cephadm, it's easier to run off the direct physical layer, but I'm likely to keep the VMs. I have added 1 or 2 hosts since then that don't virtualize ceph, but since my farm isn't big enough to justify a complete set of storage-only boxes, I'll likely continue with VMs for the foreseeable future. This mobo is circa 2011, but the model has worked so well for my needs that I've made it the backbone for all the big boxes. There are 6 onboard SATA ports, capable of being set up as RAID in the BIOS, but I run them in basic mode. I finally got my ceph health totally clean this week, but I'd been seeing 1-2 PGs get corrupted overnight several times and this morning came in and the entire box had powered itself off. Since I'd just pulled the CPU fan for its annual cat hair removal, there was no logical excuse for that, and so I pulled the box and swapped the functioning drives to a new box. I'm going to test the RAM, and then probably swap the mobo on the retired box. One disk was definitely faulty, SMART or not, as it gave the same errors in the replacement box. The other OSD disk and the OS drive were also throwing occasional errors, but that went away on the replacement box. The I/O errors were being reported by the base OS, so I don't consider it a fault in the VM or in ceph. SMART has never been very good about giving me useful warnings before a disk blew out. On metadata, yes, LV, VG, and PV metadata are stored in their respective storage definitions. The ceph metadata is in filesystem form on the host (/var/lib/ceph/...), but I've no doubt that ceph could find a way to replicate it into the OSD itself. Tim On 4/12/25 11:13, Anthony D'Atri wrote:
Apparently those UUIDs aren't as reliable as I thought.
I've had problems with a server box that hosts a ceph VM. VM?
Looks like the mobo disk controller is unreliable Lemme guess, it is an IR / RoC / RAID type? As opposed to JBOB / IT?
If the former and it’s an LSI SKU as most are, I’d love if you could send me privately the output of
storcli64 /c0 show termlog >/tmp/termlog.txt
Sometimes flakiness is actually with the drive backplane, especially when it has an embedded expander. In either case, updating HBA firmware sometimes makes a real difference.
And drive firmware.
AND one of the disks passes SMART I’m curious if it shows SATA downshifts.
but has interface problems. So I moved the disks to an alternate box.
Between relocation and dropping the one disk, neither of the 2 OSDs for that host will come up. If everything was running solely on static UUIDs, the good disk should have been findable even if its physical disk device name shifted. But it wasn't. Did you try
ceph-volume lvm activate —all
?
Which brings up something I've wondered about for some time. Shouldn't it be possible for OSDs to be portable? I haven’t tried it much, but that *should* be true, modulo CRUSH location.
That is, if a box goes bad, in theory I should be able to remove the drive and jack it into a hot-swap bay on another server and have that server able to import the relocated OSD. I’ve effectively done a chassis swap, moving all the drives including the boot volume, but that admittedly was in the ceph-disk days.
True, the metadata for an OSD is currently located on its host, but it seems like it should be possible to carry a copy on the actual device. My limited understanding is that *is* the case with LVM.
Tim
On 4/11/25 16:23, Anthony D'Atri wrote:
Filestore, pre-ceph-volume may have been entirely different. IIRC LVM is used these days to exploit persistent metadata tags.
On Apr 11, 2025, at 4:03 PM, Tim Holloway <timh@mousetech.com> wrote:
I just checked an OSD and the "block" entry is indeed linked to storage using a /dev/mapper uuid LV, not a /dev/device. When ceph builds an LV-based OSD, it creates a VG whose name is "ceph-uuuuu", where "uuuu" is a UUID, and an LV named "osd-block-vvvv", where "vvvv" is also a uuid. So although you'd map the osd to something like /dev/vdb in a VM, the actual name ceph uses is uuid-based (and lvm-based) and thus not subject to change with alterations in the hardware as the uuids are part of the metadata in VGs and LVs created by ceph.
Since I got that from a VM, I can't vouch for all cases, but I thought it especially interesting that a ceph was creating LVM counterparts even for devices that were not themselves LVM-based.
And yeah, I understand that it's the amount of OSD replicate data that counts more than the number of hosts, but when an entire host goes down and there are few hosts, that can take a large bite out of the replicas.
Tim
On 4/11/25 10:36, Anthony D'Atri wrote:
I thought those links were to the by-uuid paths for that reason?
On Apr 11, 2025, at 6:39 AM, Janne Johansson <icepic.dz@gmail.com> wrote:
Den fre 11 apr. 2025 kl 09:59 skrev Anthony D'Atri <anthony.datri@gmail.com>: > Filestore IIRC used partitions, with cute hex GPT types for various states and roles. Udev activation was sometimes problematic, and LVM tags are more flexible and reliable than the prior approach. There no doubt is more to it but that’s what I recall. Filestore used to have softlinks towards the journal device (if used) which pointed to sdX where that X of course would jump around if you changed the number of drives on the box, or the kernel disk detection order changed, breaking the OSD.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
OSDs are absolutely portable. I've moved them around by simply migrating the journal back into the spinner, moving the drive, pulling the journal back out and then doing ceph-volume lvm activate all. /var/lib/ceph/ are all tmpfs mounts generated on boot. This is for "physical" setups and not containers. Ymmv -- Paul Mezzanini Platform Engineer III Research Computing Rochester Institute of Technology ________________________________ From: Tim Holloway <timh@mousetech.com> Sent: Saturday, April 12, 2025 1:13:05 PM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: nodes with high density of OSDs When I first migrated to Ceph, my servers were all running CentOS 7, which I (wrongly) thought could not handle anything above Octopus, and on top of that, I initially did legacy installs. So in order to run Pacific and to keep the overall clutter in the physical box configuration down, I made my Ceph hosts VMs. With cephadm, it's easier to run off the direct physical layer, but I'm likely to keep the VMs. I have added 1 or 2 hosts since then that don't virtualize ceph, but since my farm isn't big enough to justify a complete set of storage-only boxes, I'll likely continue with VMs for the foreseeable future. This mobo is circa 2011, but the model has worked so well for my needs that I've made it the backbone for all the big boxes. There are 6 onboard SATA ports, capable of being set up as RAID in the BIOS, but I run them in basic mode. I finally got my ceph health totally clean this week, but I'd been seeing 1-2 PGs get corrupted overnight several times and this morning came in and the entire box had powered itself off. Since I'd just pulled the CPU fan for its annual cat hair removal, there was no logical excuse for that, and so I pulled the box and swapped the functioning drives to a new box. I'm going to test the RAM, and then probably swap the mobo on the retired box. One disk was definitely faulty, SMART or not, as it gave the same errors in the replacement box. The other OSD disk and the OS drive were also throwing occasional errors, but that went away on the replacement box. The I/O errors were being reported by the base OS, so I don't consider it a fault in the VM or in ceph. SMART has never been very good about giving me useful warnings before a disk blew out. On metadata, yes, LV, VG, and PV metadata are stored in their respective storage definitions. The ceph metadata is in filesystem form on the host (/var/lib/ceph/...), but I've no doubt that ceph could find a way to replicate it into the OSD itself. Tim On 4/12/25 11:13, Anthony D'Atri wrote:
Apparently those UUIDs aren't as reliable as I thought.
I've had problems with a server box that hosts a ceph VM. VM?
Looks like the mobo disk controller is unreliable Lemme guess, it is an IR / RoC / RAID type? As opposed to JBOB / IT?
If the former and it’s an LSI SKU as most are, I’d love if you could send me privately the output of
storcli64 /c0 show termlog >/tmp/termlog.txt
Sometimes flakiness is actually with the drive backplane, especially when it has an embedded expander. In either case, updating HBA firmware sometimes makes a real difference.
And drive firmware.
AND one of the disks passes SMART I’m curious if it shows SATA downshifts.
but has interface problems. So I moved the disks to an alternate box.
Between relocation and dropping the one disk, neither of the 2 OSDs for that host will come up. If everything was running solely on static UUIDs, the good disk should have been findable even if its physical disk device name shifted. But it wasn't. Did you try
ceph-volume lvm activate —all
?
Which brings up something I've wondered about for some time. Shouldn't it be possible for OSDs to be portable? I haven’t tried it much, but that *should* be true, modulo CRUSH location.
That is, if a box goes bad, in theory I should be able to remove the drive and jack it into a hot-swap bay on another server and have that server able to import the relocated OSD. I’ve effectively done a chassis swap, moving all the drives including the boot volume, but that admittedly was in the ceph-disk days.
True, the metadata for an OSD is currently located on its host, but it seems like it should be possible to carry a copy on the actual device. My limited understanding is that *is* the case with LVM.
Tim
On 4/11/25 16:23, Anthony D'Atri wrote:
Filestore, pre-ceph-volume may have been entirely different. IIRC LVM is used these days to exploit persistent metadata tags.
On Apr 11, 2025, at 4:03 PM, Tim Holloway <timh@mousetech.com> wrote:
I just checked an OSD and the "block" entry is indeed linked to storage using a /dev/mapper uuid LV, not a /dev/device. When ceph builds an LV-based OSD, it creates a VG whose name is "ceph-uuuuu", where "uuuu" is a UUID, and an LV named "osd-block-vvvv", where "vvvv" is also a uuid. So although you'd map the osd to something like /dev/vdb in a VM, the actual name ceph uses is uuid-based (and lvm-based) and thus not subject to change with alterations in the hardware as the uuids are part of the metadata in VGs and LVs created by ceph.
Since I got that from a VM, I can't vouch for all cases, but I thought it especially interesting that a ceph was creating LVM counterparts even for devices that were not themselves LVM-based.
And yeah, I understand that it's the amount of OSD replicate data that counts more than the number of hosts, but when an entire host goes down and there are few hosts, that can take a large bite out of the replicas.
Tim
On 4/11/25 10:36, Anthony D'Atri wrote:
I thought those links were to the by-uuid paths for that reason?
On Apr 11, 2025, at 6:39 AM, Janne Johansson <icepic.dz@gmail.com> wrote:
Den fre 11 apr. 2025 kl 09:59 skrev Anthony D'Atri <anthony.datri@gmail.com>: > Filestore IIRC used partitions, with cute hex GPT types for various states and roles. Udev activation was sometimes problematic, and LVM tags are more flexible and reliable than the prior approach. There no doubt is more to it but that’s what I recall. Filestore used to have softlinks towards the journal device (if used) which pointed to sdX where that X of course would jump around if you changed the number of drives on the box, or the kernel disk detection order changed, breaking the OSD.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
For administered (container) OSDs, the setup would likely be similar. If my experience is indicative, the mere presence of an OSD's metadata directory under /var/lib/ceph/<fsid> should be enough to cause ceph to generate the container. So all that's necessary is to move the OSD metadata over there and probably restart ceph on that node. Although a support utility would undoubtedly be useful to make the process more normal to ordinary ceph operations. The downside here is that since the metadata has to be pulled off the host before migrating, if the host - or at least the /var/lib/ceph drive - is dead, it's too late to capture that data, which is why it would be nice to replicate it on the actual USD store. On 4/12/25 17:45, Paul Mezzanini wrote:
OSDs are absolutely portable. I've moved them around by simply migrating the journal back into the spinner, moving the drive, pulling the journal back out and then doing ceph-volume lvm activate all.
/var/lib/ceph/ are all tmpfs mounts generated on boot.
This is for "physical" setups and not containers. Ymmv
--
Paul Mezzanini Platform Engineer III Research Computing Rochester Institute of Technology ________________________________ From: Tim Holloway <timh@mousetech.com> Sent: Saturday, April 12, 2025 1:13:05 PM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: nodes with high density of OSDs
When I first migrated to Ceph, my servers were all running CentOS 7, which I (wrongly) thought could not handle anything above Octopus, and on top of that, I initially did legacy installs. So in order to run Pacific and to keep the overall clutter in the physical box configuration down, I made my Ceph hosts VMs. With cephadm, it's easier to run off the direct physical layer, but I'm likely to keep the VMs. I have added 1 or 2 hosts since then that don't virtualize ceph, but since my farm isn't big enough to justify a complete set of storage-only boxes, I'll likely continue with VMs for the foreseeable future.
This mobo is circa 2011, but the model has worked so well for my needs that I've made it the backbone for all the big boxes. There are 6 onboard SATA ports, capable of being set up as RAID in the BIOS, but I run them in basic mode. I finally got my ceph health totally clean this week, but I'd been seeing 1-2 PGs get corrupted overnight several times and this morning came in and the entire box had powered itself off. Since I'd just pulled the CPU fan for its annual cat hair removal, there was no logical excuse for that, and so I pulled the box and swapped the functioning drives to a new box. I'm going to test the RAM, and then probably swap the mobo on the retired box.
One disk was definitely faulty, SMART or not, as it gave the same errors in the replacement box. The other OSD disk and the OS drive were also throwing occasional errors, but that went away on the replacement box. The I/O errors were being reported by the base OS, so I don't consider it a fault in the VM or in ceph. SMART has never been very good about giving me useful warnings before a disk blew out.
On metadata, yes, LV, VG, and PV metadata are stored in their respective storage definitions. The ceph metadata is in filesystem form on the host (/var/lib/ceph/...), but I've no doubt that ceph could find a way to replicate it into the OSD itself.
Tim
On 4/12/25 11:13, Anthony D'Atri wrote:
Apparently those UUIDs aren't as reliable as I thought.
I've had problems with a server box that hosts a ceph VM. VM?
Looks like the mobo disk controller is unreliable Lemme guess, it is an IR / RoC / RAID type? As opposed to JBOB / IT?
If the former and it’s an LSI SKU as most are, I’d love if you could send me privately the output of
storcli64 /c0 show termlog >/tmp/termlog.txt
Sometimes flakiness is actually with the drive backplane, especially when it has an embedded expander. In either case, updating HBA firmware sometimes makes a real difference.
And drive firmware.
AND one of the disks passes SMART I’m curious if it shows SATA downshifts.
but has interface problems. So I moved the disks to an alternate box.
Between relocation and dropping the one disk, neither of the 2 OSDs for that host will come up. If everything was running solely on static UUIDs, the good disk should have been findable even if its physical disk device name shifted. But it wasn't. Did you try
ceph-volume lvm activate —all
?
Which brings up something I've wondered about for some time. Shouldn't it be possible for OSDs to be portable? I haven’t tried it much, but that *should* be true, modulo CRUSH location.
That is, if a box goes bad, in theory I should be able to remove the drive and jack it into a hot-swap bay on another server and have that server able to import the relocated OSD. I’ve effectively done a chassis swap, moving all the drives including the boot volume, but that admittedly was in the ceph-disk days.
True, the metadata for an OSD is currently located on its host, but it seems like it should be possible to carry a copy on the actual device. My limited understanding is that *is* the case with LVM.
Tim
On 4/11/25 16:23, Anthony D'Atri wrote:
Filestore, pre-ceph-volume may have been entirely different. IIRC LVM is used these days to exploit persistent metadata tags.
On Apr 11, 2025, at 4:03 PM, Tim Holloway <timh@mousetech.com> wrote:
I just checked an OSD and the "block" entry is indeed linked to storage using a /dev/mapper uuid LV, not a /dev/device. When ceph builds an LV-based OSD, it creates a VG whose name is "ceph-uuuuu", where "uuuu" is a UUID, and an LV named "osd-block-vvvv", where "vvvv" is also a uuid. So although you'd map the osd to something like /dev/vdb in a VM, the actual name ceph uses is uuid-based (and lvm-based) and thus not subject to change with alterations in the hardware as the uuids are part of the metadata in VGs and LVs created by ceph.
Since I got that from a VM, I can't vouch for all cases, but I thought it especially interesting that a ceph was creating LVM counterparts even for devices that were not themselves LVM-based.
And yeah, I understand that it's the amount of OSD replicate data that counts more than the number of hosts, but when an entire host goes down and there are few hosts, that can take a large bite out of the replicas.
Tim
On 4/11/25 10:36, Anthony D'Atri wrote:
I thought those links were to the by-uuid paths for that reason?
> On Apr 11, 2025, at 6:39 AM, Janne Johansson <icepic.dz@gmail.com> wrote: > > Den fre 11 apr. 2025 kl 09:59 skrev Anthony D'Atri <anthony.datri@gmail.com>: >> Filestore IIRC used partitions, with cute hex GPT types for various states and roles. Udev activation was sometimes problematic, and LVM tags are more flexible and reliable than the prior approach. There no doubt is more to it but that’s what I recall. > Filestore used to have softlinks towards the journal device (if used) > which pointed to sdX where that X of course would jump around if you > changed the number of drives on the box, or the kernel disk detection > order changed, breaking the OSD. > > -- > May the most significant bit of your life be positive. > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
When I first migrated to Ceph, my servers were all running CentOS 7, which I (wrongly) thought could not handle anything above Octopus,
Containerized deployments do have the advantage of less coupling to the underlying OS for dependencies, though the very latest CentOS 9 containers may have issues on the old CentOS kernel.
and on top of that, I initially did legacy installs. So in order to run Pacific and to keep the overall clutter in the physical box configuration down, I made my Ceph hosts VMs. With cephadm, it's easier to run off the direct physical layer, but I'm likely to keep the VMs.
VMs conventionally present more overhead than containers, fwiw.
I have added 1 or 2 hosts since then that don't virtualize ceph, but since my farm isn't big enough to justify a complete set of storage-only boxes, I'll likely continue with VMs for the foreseeable future.
Whatever floats your boat.
This mobo is circa 2011, but the model has worked so well for my needs that I've made it the backbone for all the big boxes.
PCIe …. gen 2?
There are 6 onboard SATA ports, capable of being set up as RAID in the BIOS, but I run them in basic mode. I finally got my ceph health totally clean this week, but I'd been seeing 1-2 PGs get corrupted overnight several times
As a function of scrubs? Scrubs are often when latent issues are surfaced.
and this morning came in and the entire box had powered itself off. Since I'd just pulled the CPU fan for its annual cat hair removal, there was no logical excuse for that
Don’t get me started about cats ;)
and so I pulled the box and swapped the functioning drives to a new box. I'm going to test the RAM, and then probably swap the mobo on the retired box.
One disk was definitely faulty, SMART or not, as it gave the same errors in the replacement box. The other OSD disk and the OS drive were also throwing occasional errors, but that went away on the replacement box. The I/O errors were being reported by the base OS, so I don't consider it a fault in the VM or in ceph. SMART has never been very good about giving me useful warnings before a disk blew out.
The overall pass/fail self-reported status isn’t worth much, but watching SATA downshifts UDMA/CRC errors grown defects increased rate of LBA reallocation can help predict some issues before the drive becomes a real problem.
On metadata, yes, LV, VG, and PV metadata are stored in their respective storage definitions. The ceph metadata is in filesystem form on the host (/var/lib/ceph/...), but I've no doubt that ceph could find a way to replicate it into the OSD itself.
Filestore or BlueStore?
Tim
On 4/12/25 11:13, Anthony D'Atri wrote:
Apparently those UUIDs aren't as reliable as I thought.
I've had problems with a server box that hosts a ceph VM. VM?
Looks like the mobo disk controller is unreliable Lemme guess, it is an IR / RoC / RAID type? As opposed to JBOB / IT?
If the former and it’s an LSI SKU as most are, I’d love if you could send me privately the output of
storcli64 /c0 show termlog >/tmp/termlog.txt
Sometimes flakiness is actually with the drive backplane, especially when it has an embedded expander. In either case, updating HBA firmware sometimes makes a real difference.
And drive firmware.
AND one of the disks passes SMART I’m curious if it shows SATA downshifts.
but has interface problems. So I moved the disks to an alternate box.
Between relocation and dropping the one disk, neither of the 2 OSDs for that host will come up. If everything was running solely on static UUIDs, the good disk should have been findable even if its physical disk device name shifted. But it wasn't. Did you try
ceph-volume lvm activate —all
?
Which brings up something I've wondered about for some time. Shouldn't it be possible for OSDs to be portable? I haven’t tried it much, but that *should* be true, modulo CRUSH location.
That is, if a box goes bad, in theory I should be able to remove the drive and jack it into a hot-swap bay on another server and have that server able to import the relocated OSD. I’ve effectively done a chassis swap, moving all the drives including the boot volume, but that admittedly was in the ceph-disk days.
True, the metadata for an OSD is currently located on its host, but it seems like it should be possible to carry a copy on the actual device. My limited understanding is that *is* the case with LVM.
Tim
On 4/11/25 16:23, Anthony D'Atri wrote:
Filestore, pre-ceph-volume may have been entirely different. IIRC LVM is used these days to exploit persistent metadata tags.
On Apr 11, 2025, at 4:03 PM, Tim Holloway <timh@mousetech.com> wrote:
I just checked an OSD and the "block" entry is indeed linked to storage using a /dev/mapper uuid LV, not a /dev/device. When ceph builds an LV-based OSD, it creates a VG whose name is "ceph-uuuuu", where "uuuu" is a UUID, and an LV named "osd-block-vvvv", where "vvvv" is also a uuid. So although you'd map the osd to something like /dev/vdb in a VM, the actual name ceph uses is uuid-based (and lvm-based) and thus not subject to change with alterations in the hardware as the uuids are part of the metadata in VGs and LVs created by ceph.
Since I got that from a VM, I can't vouch for all cases, but I thought it especially interesting that a ceph was creating LVM counterparts even for devices that were not themselves LVM-based.
And yeah, I understand that it's the amount of OSD replicate data that counts more than the number of hosts, but when an entire host goes down and there are few hosts, that can take a large bite out of the replicas.
Tim
On 4/11/25 10:36, Anthony D'Atri wrote:
I thought those links were to the by-uuid paths for that reason?
> On Apr 11, 2025, at 6:39 AM, Janne Johansson <icepic.dz@gmail.com> wrote: > > Den fre 11 apr. 2025 kl 09:59 skrev Anthony D'Atri <anthony.datri@gmail.com>: >> Filestore IIRC used partitions, with cute hex GPT types for various states and roles. Udev activation was sometimes problematic, and LVM tags are more flexible and reliable than the prior approach. There no doubt is more to it but that’s what I recall. > Filestore used to have softlinks towards the journal device (if used) > which pointed to sdX where that X of course would jump around if you > changed the number of drives on the box, or the kernel disk detection > order changed, breaking the OSD. > > -- > May the most significant bit of your life be positive. > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Tim! First of all, thanks for the detailed answer! Yes, probably in set up of 4 nodes by 116 OSD it looks a bit overloaded, but what if I have 10 nodes? Yes, nodes itself are still heavy but in a row it seems to be not that dramatic, no? However, in a docu I see that it is quite common for systemd to fail on boot and even showed a way to escape. ``` It is common to have failures when a system is coming up online. The devices are sometimes not fully available and this unpredictable behavior may cause an OSD to not be ready to be used. There are two configurable environment variables used to set the retry behavior: CEPH_VOLUME_SYSTEMD_TRIES: Defaults to 30 CEPH_VOLUME_SYSTEMD_INTERVAL: Defaults to 5 ``` But if where should I set these vars? If I set it as ENV vars in bashrc of root it doesnt seem to work as ceph starts at the boot time when root env vars are not active yet...
Hi Alex, I think one of the scariest things about your setup is that there are only 4 nodes (I'm assuming that means Ceph hosts carrying OSDs). I've been bouncing around different configurations lately between some of my deployment issues and cranky old hardware and I presently am down to 4 hosts with 1-2 OSDs per host. If even one of those hosts goes down, Ceph gets unhappy. If 2 are offline at once, Ceph goes into self-defense mode. I'd hate to think of 116 OSDs at risk on a single host. I got curious about when LVM comes online, and I believe that the vgchange command that activates the LVs is actually in the initrd file before systemd comes up if a system was configured for LVM support. That's necessary, in fact, since the live root partition can be and often is an LV itself. As for for systemd dependencies, that's something I've been doing a lot of tuning on myself, as things like my backup system won't work if certain volumes aren't mounted, so I've had to add "RequiresVolume" dependencies, plus some daemons require other daemons. So it's an interesting dance. At this point I think that the best way to ensure that all LVs are online would be to add overrides under /etc/systemd/system/ceph.service (probably needs the fsid in the service name, too). Include a beforeStartup command that scans the proc ps list and loops until the vgscan process no longer show up (command completed). But I really would reconsider both your host and OSD count. Larger OSDs and more hosts would give better reliability and performance. Tim On 4/11/25 03:53, Alex from North wrote:
Hello Tim! First of all, thanks for the detailed answer! Yes, probably in set up of 4 nodes by 116 OSD it looks a bit overloaded, but what if I have 10 nodes? Yes, nodes itself are still heavy but in a row it seems to be not that dramatic, no?
However, in a docu I see that it is quite common for systemd to fail on boot and even showed a way to escape.
``` It is common to have failures when a system is coming up online. The devices are sometimes not fully available and this unpredictable behavior may cause an OSD to not be ready to be used.
There are two configurable environment variables used to set the retry behavior:
CEPH_VOLUME_SYSTEMD_TRIES: Defaults to 30
CEPH_VOLUME_SYSTEMD_INTERVAL: Defaults to 5 ```
But if where should I set these vars? If I set it as ENV vars in bashrc of root it doesnt seem to work as ceph starts at the boot time when root env vars are not active yet... _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I think one of the scariest things about your setup is that there are only 4 nodes (I'm assuming that means Ceph hosts carrying OSDs). I've been bouncing around different configurations lately between some of my deployment issues and cranky old hardware and I presently am down to 4 hosts with 1-2 OSDs per host. If even one of those hosts goes down, Ceph gets unhappy. If 2 are offline at once, Ceph goes into self-defense mode. I'd hate to think of 116 OSDs at risk on a single host.
My sense is that from a cluster perspective it’s not so much a function of the absolute number of OSDs that go down as the percentage of the cluster that a host represents. If a cluster comprises 20x hosts each with 116 OSDs, one going down is only 5% of the whole. One of the concerns is maintaining enough space to recover that many OSDs’ worth of data, if mon_osd_down_out_subtree_limit is not used to forestall most whole-host recovery.
I got curious about when LVM comes online, and I believe that the vgchange command that activates the LVs is actually in the initrd file before systemd comes up if a system was configured for LVM support. That's necessary, in fact, since the live root partition can be and often is an LV itself.
As for for systemd dependencies, that's something I've been doing a lot of tuning on myself, as things like my backup system won't work if certain volumes aren't mounted, so I've had to add "RequiresVolume" dependencies, plus some daemons require other daemons. So it's an interesting dance.
At this point I think that the best way to ensure that all LVs are online would be to add overrides under /etc/systemd/system/ceph.service (probably needs the fsid in the service name, too). Include a beforeStartup command that scans the proc ps list and loops until the vgscan process no longer show up (command completed).
I was thinking ExecStartPre=/bin/sleep 60 or so as an override to keep it simple, but feel free to get surgical. With of course Ansible or other automation to persist the override for new/updated/changed hosts.
But I really would reconsider both your host and OSD count. Larger OSDs and more hosts would give better reliability and performance.
Indeed. If such a chassis is picked due to perceived cost savings over all else, there is the cost of not doing the job, but moreover having only 4 prevents the use of a reasonably wide EC profile, which probably costs more in Capex than having a larger number of more conventional chassis. I think we haven’t seen the OP’s drive size, but I’ll bet that they’re already at least 20TB HDDs, with the usual SATA bottleneck. Ultradense toploaders can also exhibit HBA and backplane saturation. I think you may have meant “Fewer OSDs per host and more hosts”, Sir Enchanter.
Tim
On 4/11/25 03:53, Alex from North wrote:
Hello Tim! First of all, thanks for the detailed answer! Yes, probably in set up of 4 nodes by 116 OSD it looks a bit overloaded, but what if I have 10 nodes? Yes, nodes itself are still heavy but in a row it seems to be not that dramatic, no?
However, in a docu I see that it is quite common for systemd to fail on boot and even showed a way to escape.
``` It is common to have failures when a system is coming up online. The devices are sometimes not fully available and this unpredictable behavior may cause an OSD to not be ready to be used.
There are two configurable environment variables used to set the retry behavior:
CEPH_VOLUME_SYSTEMD_TRIES: Defaults to 30
CEPH_VOLUME_SYSTEMD_INTERVAL: Defaults to 5 ```
But if where should I set these vars? If I set it as ENV vars in bashrc of root it doesnt seem to work as ceph starts at the boot time when root env vars are not active yet... _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi,
On 11 Apr 2025, at 10:53, Alex from North <service.plant@ya.ru> wrote:
Hello Tim! First of all, thanks for the detailed answer! Yes, probably in set up of 4 nodes by 116 OSD it looks a bit overloaded, but what if I have 10 nodes? Yes, nodes itself are still heavy but in a row it seems to be not that dramatic, no?
However, in a docu I see that it is quite common for systemd to fail on boot and even showed a way to escape.
Currently we don't have 116 OSD chassis, but operate the 60 OSD chassis without any issues with LVM activation. We use Ceph Pacific (16.2.15) Welcome to CentOS Stream 8 4.18.0-553.6.1.el8.x86_64 Platform: Supermicro Super Server (X12DPi-N6) BIOS: 1.9 (American Megatrends International, LLC. 02/07/2024) CPU: Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz [2P/32C] RAM: 382Gi free / 628Gi total LoadAvg: 2.94 2.87 3.27 up 10 weeks, 2 hours, 36 minutes [operator@host:/]$ scsis Device /dev/sda [TOSHIBA MG10ACA2: <cut>]: 0:0:0:0 Device /dev/sdb [TOSHIBA MG10ACA2: <cut>]: 0:0:1:0 Device /dev/sdc [TOSHIBA MG10ACA2: <cut>]: 0:0:2:0 Device /dev/sdd [TOSHIBA MG10ACA2: <cut>]: 0:0:3:0 Device /dev/sde [TOSHIBA MG10ACA2: <cut>]: 0:0:4:0 Device /dev/sdf [TOSHIBA MG10ACA2: <cut>]: 0:0:5:0 Device /dev/sdg [TOSHIBA MG10ACA2: <cut>]: 0:0:6:0 Device /dev/sdh [TOSHIBA MG10ACA2: <cut>]: 0:0:7:0 Device /dev/sdi [TOSHIBA MG10ACA2: <cut>]: 0:0:8:0 Device /dev/sdj [TOSHIBA MG10ACA2: <cut>]: 0:0:9:0 Device /dev/sdk [TOSHIBA MG10ACA2: <cut>]: 0:0:10:0 Device /dev/sdl [TOSHIBA MG10ACA2: <cut>]: 0:0:11:0 Device /dev/sdm [TOSHIBA MG10ACA2: <cut>]: 0:0:12:0 Device /dev/sdn [TOSHIBA MG10ACA2: <cut>]: 0:0:13:0 Device /dev/sdo [TOSHIBA MG10ACA2: <cut>]: 0:0:14:0 Device /dev/sdp [TOSHIBA MG10ACA2: <cut>]: 0:0:15:0 Device /dev/sdq [TOSHIBA MG10ACA2: <cut>]: 0:0:16:0 Device /dev/sdr [TOSHIBA MG10ACA2: <cut>]: 0:0:17:0 Device /dev/sds [TOSHIBA MG10ACA2: <cut>]: 0:0:18:0 Device /dev/sdt [TOSHIBA MG10ACA2: <cut>]: 0:0:19:0 Device /dev/sdu [TOSHIBA MG10ACA2: <cut>]: 0:0:21:0 Device /dev/sdv [TOSHIBA MG10ACA2: <cut>]: 0:0:22:0 Device /dev/sdw [TOSHIBA MG10ACA2: <cut>]: 0:0:23:0 Device /dev/sdx [TOSHIBA MG10ACA2: <cut>]: 0:0:24:0 Device /dev/sdy [TOSHIBA MG10ACA2: <cut>]: 0:0:25:0 Device /dev/sdz [TOSHIBA MG10ACA2: <cut>]: 0:0:26:0 Device /dev/sdaa [TOSHIBA MG10ACA2: <cut>]: 0:0:27:0 Device /dev/sdab [TOSHIBA MG10ACA2: <cut>]: 0:0:28:0 Device /dev/sdac [TOSHIBA MG10ACA2: <cut>]: 0:0:29:0 Device /dev/sdad [TOSHIBA MG10ACA2: <cut>]: 0:0:30:0 Device /dev/sdae [TOSHIBA MG10ACA2: <cut>]: 0:0:31:0 Device /dev/sdaf [TOSHIBA MG10ACA2: <cut>]: 0:0:32:0 Device /dev/sdag [TOSHIBA MG10ACA2: <cut>]: 0:0:33:0 Device /dev/sdah [TOSHIBA MG10ACA2: <cut>]: 0:0:34:0 Device /dev/sdai [TOSHIBA MG10ACA2: <cut>]: 0:0:35:0 Device /dev/sdaj [TOSHIBA MG10ACA2: <cut>]: 0:0:36:0 Device /dev/sdak [TOSHIBA MG10ACA2: <cut>]: 0:0:37:0 Device /dev/sdal [TOSHIBA MG10ACA2: <cut>]: 0:0:38:0 Device /dev/sdam [TOSHIBA MG10ACA2: <cut>]: 0:0:39:0 Device /dev/sdan [TOSHIBA MG10ACA2: <cut>]: 0:0:40:0 Device /dev/sdao [TOSHIBA MG10ACA2: <cut>]: 0:0:42:0 Device /dev/sdap [TOSHIBA MG10ACA2: <cut>]: 0:0:43:0 Device /dev/sdaq [TOSHIBA MG10ACA2: <cut>]: 0:0:44:0 Device /dev/sdar [TOSHIBA MG10ACA2: <cut>]: 0:0:45:0 Device /dev/sdas [TOSHIBA MG10ACA2: <cut>]: 0:0:46:0 Device /dev/sdat [TOSHIBA MG10ACA2: <cut>]: 0:0:47:0 Device /dev/sdau [TOSHIBA MG10ACA2: <cut>]: 0:0:48:0 Device /dev/sdav [TOSHIBA MG10ACA2: <cut>]: 0:0:49:0 Device /dev/sdaw [TOSHIBA MG10ACA2: <cut>]: 0:0:50:0 Device /dev/sdax [TOSHIBA MG10ACA2: <cut>]: 0:0:51:0 Device /dev/sday [TOSHIBA MG10ACA2: <cut>]: 0:0:52:0 Device /dev/sdaz [TOSHIBA MG10ACA2: <cut>]: 0:0:53:0 Device /dev/sdba [TOSHIBA MG10ACA2: <cut>]: 0:0:54:0 Device /dev/sdbb [TOSHIBA MG10ACA2: <cut>]: 0:0:55:0 Device /dev/sdbc [TOSHIBA MG10ACA2: <cut>]: 0:0:56:0 Device /dev/sdbd [TOSHIBA MG10ACA2: <cut>]: 0:0:57:0 Device /dev/sdbe [TOSHIBA MG10ACA2: <cut>]: 0:0:58:0 Device /dev/sdbf [TOSHIBA MG10ACA2: <cut>]: 0:0:59:0 Device /dev/sdbg [TOSHIBA MG10ACA2: <cut>]: 0:0:60:0 Device /dev/sdbh [TOSHIBA MG10ACA2: <cut>]: 0:0:61:0 Inspected 60 SCSI devices Good luck, k
participants (9)
-
Alex from North
-
Anthony D'Atri
-
Dominique Ramaekers
-
Gregory Orange
-
Janne Johansson
-
Konstantin Shalygin
-
Paul Mezzanini
-
pg＠ceph.list.sabi.co.UK
-
Tim Holloway