When I first migrated to Ceph, my servers were all running CentOS 7, which I (wrongly) thought could not handle anything above Octopus, and on top of that, I initially did legacy installs. So in order to run Pacific and to keep the overall clutter in the physical box configuration down, I made my Ceph hosts VMs. With cephadm, it's easier to run off the direct physical layer, but I'm likely to keep the VMs. I have added 1 or 2 hosts since then that don't virtualize ceph, but since my farm isn't big enough to justify a complete set of storage-only boxes, I'll likely continue with VMs for the foreseeable future. This mobo is circa 2011, but the model has worked so well for my needs that I've made it the backbone for all the big boxes. There are 6 onboard SATA ports, capable of being set up as RAID in the BIOS, but I run them in basic mode. I finally got my ceph health totally clean this week, but I'd been seeing 1-2 PGs get corrupted overnight several times and this morning came in and the entire box had powered itself off. Since I'd just pulled the CPU fan for its annual cat hair removal, there was no logical excuse for that, and so I pulled the box and swapped the functioning drives to a new box. I'm going to test the RAM, and then probably swap the mobo on the retired box. One disk was definitely faulty, SMART or not, as it gave the same errors in the replacement box. The other OSD disk and the OS drive were also throwing occasional errors, but that went away on the replacement box. The I/O errors were being reported by the base OS, so I don't consider it a fault in the VM or in ceph. SMART has never been very good about giving me useful warnings before a disk blew out. On metadata, yes, LV, VG, and PV metadata are stored in their respective storage definitions. The ceph metadata is in filesystem form on the host (/var/lib/ceph/...), but I've no doubt that ceph could find a way to replicate it into the OSD itself. Tim On 4/12/25 11:13, Anthony D'Atri wrote:
Apparently those UUIDs aren't as reliable as I thought.
I've had problems with a server box that hosts a ceph VM. VM?
Looks like the mobo disk controller is unreliable Lemme guess, it is an IR / RoC / RAID type? As opposed to JBOB / IT?
If the former and it’s an LSI SKU as most are, I’d love if you could send me privately the output of
storcli64 /c0 show termlog >/tmp/termlog.txt
Sometimes flakiness is actually with the drive backplane, especially when it has an embedded expander. In either case, updating HBA firmware sometimes makes a real difference.
And drive firmware.
AND one of the disks passes SMART I’m curious if it shows SATA downshifts.
but has interface problems. So I moved the disks to an alternate box.
Between relocation and dropping the one disk, neither of the 2 OSDs for that host will come up. If everything was running solely on static UUIDs, the good disk should have been findable even if its physical disk device name shifted. But it wasn't. Did you try
ceph-volume lvm activate —all
?
Which brings up something I've wondered about for some time. Shouldn't it be possible for OSDs to be portable? I haven’t tried it much, but that *should* be true, modulo CRUSH location.
That is, if a box goes bad, in theory I should be able to remove the drive and jack it into a hot-swap bay on another server and have that server able to import the relocated OSD. I’ve effectively done a chassis swap, moving all the drives including the boot volume, but that admittedly was in the ceph-disk days.
True, the metadata for an OSD is currently located on its host, but it seems like it should be possible to carry a copy on the actual device. My limited understanding is that *is* the case with LVM.
Tim
On 4/11/25 16:23, Anthony D'Atri wrote:
Filestore, pre-ceph-volume may have been entirely different. IIRC LVM is used these days to exploit persistent metadata tags.
On Apr 11, 2025, at 4:03 PM, Tim Holloway <timh@mousetech.com> wrote:
I just checked an OSD and the "block" entry is indeed linked to storage using a /dev/mapper uuid LV, not a /dev/device. When ceph builds an LV-based OSD, it creates a VG whose name is "ceph-uuuuu", where "uuuu" is a UUID, and an LV named "osd-block-vvvv", where "vvvv" is also a uuid. So although you'd map the osd to something like /dev/vdb in a VM, the actual name ceph uses is uuid-based (and lvm-based) and thus not subject to change with alterations in the hardware as the uuids are part of the metadata in VGs and LVs created by ceph.
Since I got that from a VM, I can't vouch for all cases, but I thought it especially interesting that a ceph was creating LVM counterparts even for devices that were not themselves LVM-based.
And yeah, I understand that it's the amount of OSD replicate data that counts more than the number of hosts, but when an entire host goes down and there are few hosts, that can take a large bite out of the replicas.
Tim
On 4/11/25 10:36, Anthony D'Atri wrote:
I thought those links were to the by-uuid paths for that reason?
On Apr 11, 2025, at 6:39 AM, Janne Johansson <icepic.dz@gmail.com> wrote:
Den fre 11 apr. 2025 kl 09:59 skrev Anthony D'Atri <anthony.datri@gmail.com>: > Filestore IIRC used partitions, with cute hex GPT types for various states and roles. Udev activation was sometimes problematic, and LVM tags are more flexible and reliable than the prior approach. There no doubt is more to it but that’s what I recall. Filestore used to have softlinks towards the journal device (if used) which pointed to sdX where that X of course would jump around if you changed the number of drives on the box, or the kernel disk detection order changed, breaking the OSD.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io