Many thanks for your response, Eugen! I tried to fail mgr twice, unfortunately that had no effect on the issue. Neither `cephadm ceph-volume inventory` nor `ceph device ls-by-host ceph03` have the failed drive on the list. Though your assumption is correct, the spec appears to explicitly include the failed drive: --- service_type: osd service_id: ceph03_combined_osd service_name: osd.ceph03_combined_osd placement: hosts: - ceph03 spec: data_devices: paths: ... - /dev/sde ... db_devices: paths: - /dev/nvme0n1 - /dev/nvme1n1 filter_logic: AND objectstore: bluestore --- Do you know the best way to remove the device from the spec? /Z On Fri, 16 Feb 2024 at 14:10, Eugen Block <eblock@nde.ag> wrote:
Hi,
sometimes the easiest fix is to failover the mgr, have you tried that? If that didn't work, can you share the drivegroup spec?
ceph orch ls <your_osd_spec> --export
Does it contain specific device paths or something? Does 'cephadm ls' on that node show any traces of the previous OSD? I'd probably try to check some things like
cephadm ceph-volume inventory ceph device ls-by-host <host>
Regards, Eugen
Zitat von Zakhar Kirpichenko <zakhar@gmail.com>:
Hi,
We had a physical drive malfunction in one of our Ceph OSD hosts managed by cephadm (Ceph 16.2.14). I have removed the drive from the system, and the kernel no longer sees it:
ceph03 ~]# ls -al /dev/sde ls: cannot access '/dev/sde': No such file or directory
I have removed the corresponding OSD from cephadm, crush map, etc. For all intents and purposes that OSD and its block device no longer exist:
root@ceph01:/# ceph orch ps | grep osd.26 root@ceph01:/# ceph osd tree| grep 26 root@ceph01:/# ceph orch device ls | grep -E "ceph03.*sde"
None of the above commands return anything. Cephadm correctly sees 8 remaining OSDs on the host:
root@ceph01:/# ceph orch ls | grep ceph03_c osd.ceph03_combined_osd 8 33s ago 2y ceph03
Unfortunately, cephadm appears to be trying to apply a spec to host ceph03 including the disk that is now missing:
RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=
quay.io/ceph/ceph@sha256:843f112990e6489362c625229c3ea3d90b8734bd5e14e0aeaf89942fbb980a8b
-e NODE_NAME=ceph03 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=ceph03_combined_osd -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86:/var/run/ceph:z -v /var/log/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86:/var/log/ceph:z -v
/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash:/var/lib/ceph/crash:z
-v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpc7b33pf0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpq45nkmd6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z
quay.io/ceph/ceph@sha256:843f112990e6489362c625229c3ea3d90b8734bd5e14e0aeaf89942fbb980a8b
lvm batch --no-auto /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi --db-devices /dev/nvme0n1 /dev/nvme1n1 --yes --no-systemd
Note that `lvm batch` includes the missing drive, /dev/sde. This fails because the drive no longer exists. Other than this cephadm ceph-volume thingy, the cluster is healthy.How can I tell cephadm that it should stop trying to use /dev/sde, which no longer exists, without affecting other OSDs on the host?
I would very much appreciate any advice or pointers.
Best regards, Zakhar _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io