Tim, Thank you for your guidance. Your points are completely understood. It was more that I couldn't figure out why the Dashboard was telling me that the destroyed OSD was still using /dev/sdi when the physical disk with that serial number was at /dev/sdc, and when another OSD was also reporting /dev/sdi. I figured that there must be some information buried somewhere. I don't know where this metadata comes from or how it gets updated when things like 'drive letters' change, but the metadata matched what the dashboard showed, so now I know something new. Regarding the process for bringing the OSD back online with a new HDD, I am still having some difficulties. I used the steps in the Adding/Removing OSDs document under Removing the OSD, and the OSD mostly appears to be gone. However, attempts to use 'ceph-volume lvm prepare' to build the remplacement OSD are failing, Same thing with 'ceph orch daemon add osd'. I think the problem might be that the NVMe LV that was the WAL/DB for the failed OSD did not get cleaned up, but on my systems 4 OSDs use the same NVMe drive for WAL/DB, so I'm not sure how to proceed. Any suggestions would be welcome. Thanks. -Dave -- Dave Hall Binghamton University kdhall@binghamton.edu On Tue, Oct 29, 2024 at 3:13 PM Tim Holloway <timh@mousetech.com> wrote:
Take care when reading the output of "ceph osd metadata". When you are running the OSD as an administered service, it's running in a container, and a container is a miniature VM. So, for example, it may report your OS as "CentOS Stream 8" even if your actual machine is running Ubuntu.
The biggest pitfall is in paths, because in certain cases - definitely for OSDs - internally the path for the OSD metadata and data store will be /var/lib/ceph/osd, but the actual path in the machine's OS will be /var/lib/ceph/<fsid>/osd, where the container simply mounts that for its internal path.
In other words, "ceph osd metadata" formulates its reports by having the containers assemble the report data and the output is thus the OSD's internal view, not your server's view.
Tim
On 10/28/24 14:01, Dave Hall wrote:
Hello.
Thanks to Rober's reply to 'Influencing the osd.id <http://osd.id>' I've learned two new commands today. I can now see that 'ceph osd metadata' confirms that I have two OSDs pointing to the same physical disk name:
root@ceph09:/# ceph osd metadata 12 | grep sdi "bluestore_bdev_devices": "sdi", "device_ids":
"nvme0n1=SAMSUNG_MZPLL1T6HEHP-00003_S3HBNA0KA03264,sdi=SEAGATE_ST12000NM0027_*ZJV5TX47*0000C9470ZWA",
"device_paths":
"nvme0n1=/dev/disk/by-path/pci-0000:83:00.0-nvme-1,sdi=/dev/disk/by-path/pci-0000:41:00.0-sas-phy18-lun-0",
"devices": "nvme0n1,sdi", "objectstore_numa_unknown_devices": "nvme0n1,sdi", root@ceph09:/# ceph osd metadata 9 | grep sdi "bluestore_bdev_devices": "sdi", "device_ids":
"nvme1n1=Samsung_SSD_983_DCT_M.2_1.92TB_S48DNC0N701016D,sdi=SEAGATE_ST12000NM0027_*ZJV5SMTQ*0000C9128FE0",
"device_paths":
"nvme1n1=/dev/disk/by-path/pci-0000:01:00.0-nvme-1,sdi=/dev/disk/by-path/pci-0000:41:00.0-sas-phy6-lun-0",
"devices": "nvme1n1,sdi", "objectstore_numa_unknown_devices": "sdi",
However, even though OSD 12 is saying sdi, at least it is pointing to the serial number of the failed disk. However, the disk with that serial number is currently residing at /dev/sdc.
Is there a way to force the record for the destroyed OSD to point to /dev/sdc?
Thanks.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu
On Mon, Oct 28, 2024 at 11:47 AM Dave Hall <kdhall@binghamton.edu>
wrote:
Hello.
The following is on a Reef Podman installation:
In attempting to deal over the weekend with a failed OSD disk, I have somehow managed to have two OSDs pointing to the same HDD, as shown below.
image.png
To be sure, the failure occurred on OSD.12, which was pointing to /dev/sdi.
I disabled the systemd unit for OSD.12 because it kept restarting. I then destroyed it.
When I physically removed the failed disk and rebooted the system, the disk enumeration changed. So, before the reboot, OSD.12 was using /dev/sdi. After the reboot, OSD.9 moved to /dev/sdi.
I didn't know that I had an issue until 'ceph-volume lvm prepare' failed. It was in the process of investigating this that I found the above. Right now I have reinserted the failed disk and rebooted, hoping that OSD.12 would find its old disk by some other means, but no joy.
My concern is that if I run 'ceph osd rm' I could take out OSD.9. I could take the precaution of marking OSD.9 out and let it drain, but I'd rather not. I am, perhaps, more inclined to manually clear the lingering configuration associated with OSD.12 if someone could send me the list of commands. Otherwise, I'm open to suggestions.
Thanks.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io