Dear Ceph users, after a host reboot one of the OSDs is now stuck down (and out). I tried several times to restart it and even to reboot the host, but it still remains down. # ceph -s cluster: id: b1029256-7bb3-11ec-a8ce-ac1f6b627b45 health: HEALTH_WARN 4 OSD(s) have spurious read errors (muted: OSD_SLOW_PING_TIME_BACK OSD_SLOW_PING_TIME_FRONT) services: mon: 5 daemons, quorum bofur,balin,aka,romolo,dwalin (age 16h) mgr: bofur.tklnrn(active, since 16h), standbys: aka.wzystq, balin.hvunfe mds: 2/2 daemons up, 1 standby osd: 104 osds: 103 up (since 16h), 103 in (since 13h); 4 remapped pgs data: volumes: 1/1 healthy pools: 3 pools, 529 pgs objects: 18.85M objects, 41 TiB usage: 56 TiB used, 139 TiB / 195 TiB avail pgs: 68130/150150628 objects misplaced (0.045%) 522 active+clean 4 active+remapped+backfilling 3 active+clean+scrubbing+deep io: recovery: 46 MiB/s, 21 objects/s The host is reachable (its other OSDs are in) and from the systemd logs of the OSD I don't see anything wrong: $ sudo systemctl status ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45@osd.34 ● ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45@osd.34.service - Ceph osd.34 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 Loaded: loaded (/etc/systemd/system/ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45@.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2023-06-12 17:00:25 CEST; 15h ago Main PID: 36286 (bash) Tasks: 11 (limit: 152154) Memory: 20.0M CGroup: /system.slice/system-ceph\x2db1029256\x2d7bb3\x2d11ec\x2da8ce\x2dac1f6b627b45.slice/ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45@osd.34.service ├─36286 /bin/bash /var/lib/ceph/b1029256-7bb3-11ec-a8ce-ac1f6b627b45/osd.34/unit.run └─36657 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45-osd-34 --pids-limit=0 -e CONTAINER_IMAGE=snack14/ceph-wizard@sha> Jun 12 17:00:25 balin systemd[1]: Started Ceph osd.34 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45. Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-34 Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-34 --no-mon-config --dev /dev/mapper/ceph--9a4c3927--d3da--4b49--80fe--6cdc00c7897c-osd--block--36d2f793--e5c7--4247--a314--bcc40389d50d Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/ceph--9a4c3927--d3da--4b49--80fe--6cdc00c7897c-osd--block--36d2f793--e5c7--4247--a314--bcc40389d50d Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-6 Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/ln -s /dev/mapper/ceph--9a4c3927--d3da--4b49--80fe--6cdc00c7897c-osd--block--36d2f793--e5c7--4247--a314--bcc40389d50d /var/lib/ceph/osd/ceph-34/block Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-34 Jun 12 17:00:27 balin bash[36306]: --> ceph-volume raw activate successful for osd ID: 34 Jun 12 17:00:29 balin bash[36657]: debug 2023-06-12T15:00:29.066+0000 7f818e356540 -1 Falling back to public interface I'd need some help to understand how to fix this. Thank you, Nicola
Hi, did you check the MON logs? They should contain some information about the reason why the OSD is marked down and out. You could also just try to mark it in yourself, does it change anything? $ ceph osd in 34 I would also take another look into the OSD logs: cephadm logs --name osd.34 Zitat von Nicola Mori <mori@fi.infn.it>:
Dear Ceph users,
after a host reboot one of the OSDs is now stuck down (and out). I tried several times to restart it and even to reboot the host, but it still remains down.
# ceph -s cluster: id: b1029256-7bb3-11ec-a8ce-ac1f6b627b45 health: HEALTH_WARN 4 OSD(s) have spurious read errors (muted: OSD_SLOW_PING_TIME_BACK OSD_SLOW_PING_TIME_FRONT)
services: mon: 5 daemons, quorum bofur,balin,aka,romolo,dwalin (age 16h) mgr: bofur.tklnrn(active, since 16h), standbys: aka.wzystq, balin.hvunfe mds: 2/2 daemons up, 1 standby osd: 104 osds: 103 up (since 16h), 103 in (since 13h); 4 remapped pgs
data: volumes: 1/1 healthy pools: 3 pools, 529 pgs objects: 18.85M objects, 41 TiB usage: 56 TiB used, 139 TiB / 195 TiB avail pgs: 68130/150150628 objects misplaced (0.045%) 522 active+clean 4 active+remapped+backfilling 3 active+clean+scrubbing+deep
io: recovery: 46 MiB/s, 21 objects/s
The host is reachable (its other OSDs are in) and from the systemd logs of the OSD I don't see anything wrong:
$ sudo systemctl status ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45@osd.34 ● ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45@osd.34.service - Ceph osd.34 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 Loaded: loaded (/etc/systemd/system/ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45@.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2023-06-12 17:00:25 CEST; 15h ago Main PID: 36286 (bash) Tasks: 11 (limit: 152154) Memory: 20.0M CGroup: /system.slice/system-ceph\x2db1029256\x2d7bb3\x2d11ec\x2da8ce\x2dac1f6b627b45.slice/ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45@osd.34.service ├─36286 /bin/bash /var/lib/ceph/b1029256-7bb3-11ec-a8ce-ac1f6b627b45/osd.34/unit.run └─36657 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45-osd-34 --pids-limit=0 -e CONTAINER_IMAGE=snack14/ceph-wizard@sha>
Jun 12 17:00:25 balin systemd[1]: Started Ceph osd.34 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45. Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-34 Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-34 --no-mon-config --dev /dev/mapper/ceph--9a4c3927--d3da--4b49--80fe--6cdc00c7897c-osd--block--36d2f793--e5c7--4247--a314--bcc40389d50d Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/ceph--9a4c3927--d3da--4b49--80fe--6cdc00c7897c-osd--block--36d2f793--e5c7--4247--a314--bcc40389d50d Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-6 Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/ln -s /dev/mapper/ceph--9a4c3927--d3da--4b49--80fe--6cdc00c7897c-osd--block--36d2f793--e5c7--4247--a314--bcc40389d50d /var/lib/ceph/osd/ceph-34/block Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-34 Jun 12 17:00:27 balin bash[36306]: --> ceph-volume raw activate successful for osd ID: 34 Jun 12 17:00:29 balin bash[36657]: debug 2023-06-12T15:00:29.066+0000 7f818e356540 -1 Falling back to public interface
I'd need some help to understand how to fix this. Thank you,
Nicola
I have restarted all the monitors and managers, but still the osd remains down. But I found that cephadm actually sees -it running: # ceph orch ps | grep osd.34 osd.34 balin running (14m) 108s ago 8M 75.3M 793M 17.2.6 b1a23658afad 5b9dbea262c7 # ceph osd tree | grep 34 34 hdd 1.81940 osd.34 down 0 1.00000 I really need help with this since I don't know what more to look at. Thanks in advance, Nicola On 13/06/23 08:35, Nicola Mori wrote:
Dear Ceph users,
after a host reboot one of the OSDs is now stuck down (and out). I tried several times to restart it and even to reboot the host, but it still remains down.
# ceph -s cluster: id: b1029256-7bb3-11ec-a8ce-ac1f6b627b45 health: HEALTH_WARN 4 OSD(s) have spurious read errors (muted: OSD_SLOW_PING_TIME_BACK OSD_SLOW_PING_TIME_FRONT)
services: mon: 5 daemons, quorum bofur,balin,aka,romolo,dwalin (age 16h) mgr: bofur.tklnrn(active, since 16h), standbys: aka.wzystq, balin.hvunfe mds: 2/2 daemons up, 1 standby osd: 104 osds: 103 up (since 16h), 103 in (since 13h); 4 remapped pgs
data: volumes: 1/1 healthy pools: 3 pools, 529 pgs objects: 18.85M objects, 41 TiB usage: 56 TiB used, 139 TiB / 195 TiB avail pgs: 68130/150150628 objects misplaced (0.045%) 522 active+clean 4 active+remapped+backfilling 3 active+clean+scrubbing+deep
io: recovery: 46 MiB/s, 21 objects/s
The host is reachable (its other OSDs are in) and from the systemd logs of the OSD I don't see anything wrong:
$ sudo systemctl status ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45@osd.34 ● ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45@osd.34.service - Ceph osd.34 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 Loaded: loaded (/etc/systemd/system/ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45@.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2023-06-12 17:00:25 CEST; 15h ago Main PID: 36286 (bash) Tasks: 11 (limit: 152154) Memory: 20.0M CGroup: /system.slice/system-ceph\x2db1029256\x2d7bb3\x2d11ec\x2da8ce\x2dac1f6b627b45.slice/ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45@osd.34.service ├─36286 /bin/bash /var/lib/ceph/b1029256-7bb3-11ec-a8ce-ac1f6b627b45/osd.34/unit.run └─36657 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45-osd-34 --pids-limit=0 -e CONTAINER_IMAGE=snack14/ceph-wizard@sha>
Jun 12 17:00:25 balin systemd[1]: Started Ceph osd.34 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45. Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-34 Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-34 --no-mon-config --dev /dev/mapper/ceph--9a4c3927--d3da--4b49--80fe--6cdc00c7897c-osd--block--36d2f793--e5c7--4247--a314--bcc40389d50d Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/ceph--9a4c3927--d3da--4b49--80fe--6cdc00c7897c-osd--block--36d2f793--e5c7--4247--a314--bcc40389d50d Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-6 Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/ln -s /dev/mapper/ceph--9a4c3927--d3da--4b49--80fe--6cdc00c7897c-osd--block--36d2f793--e5c7--4247--a314--bcc40389d50d /var/lib/ceph/osd/ceph-34/block Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-34 Jun 12 17:00:27 balin bash[36306]: --> ceph-volume raw activate successful for osd ID: 34 Jun 12 17:00:29 balin bash[36657]: debug 2023-06-12T15:00:29.066+0000 7f818e356540 -1 Falling back to public interface
I'd need some help to understand how to fix this. Thank you,
Nicola
-- Nicola Mori, Ph.D. INFN sezione di Firenze Via Bruno Rossi 1, 50019 Sesto F.no (Italy) +390554572660 mori@fi.infn.it
Hi, I have seen this behaviour when the OSD host cluster interface was down but the public interface was up. I suggest checking the network interfaces and the connectivity. Regards! On Thu, Jun 15, 2023 at 11:08 AM Nicola Mori <mori@fi.infn.it> wrote:
I have restarted all the monitors and managers, but still the osd remains down. But I found that cephadm actually sees -it running:
# ceph orch ps | grep osd.34 osd.34 balin running (14m) 108s ago 8M 75.3M 793M 17.2.6 b1a23658afad 5b9dbea262c7
# ceph osd tree | grep 34 34 hdd 1.81940 osd.34 down 0 1.00000
I really need help with this since I don't know what more to look at. Thanks in advance,
Nicola
Dear Ceph users,
after a host reboot one of the OSDs is now stuck down (and out). I tried several times to restart it and even to reboot the host, but it still remains down.
# ceph -s cluster: id: b1029256-7bb3-11ec-a8ce-ac1f6b627b45 health: HEALTH_WARN 4 OSD(s) have spurious read errors (muted: OSD_SLOW_PING_TIME_BACK OSD_SLOW_PING_TIME_FRONT)
services: mon: 5 daemons, quorum bofur,balin,aka,romolo,dwalin (age 16h) mgr: bofur.tklnrn(active, since 16h), standbys: aka.wzystq, balin.hvunfe mds: 2/2 daemons up, 1 standby osd: 104 osds: 103 up (since 16h), 103 in (since 13h); 4 remapped
On 13/06/23 08:35, Nicola Mori wrote: pgs
data: volumes: 1/1 healthy pools: 3 pools, 529 pgs objects: 18.85M objects, 41 TiB usage: 56 TiB used, 139 TiB / 195 TiB avail pgs: 68130/150150628 objects misplaced (0.045%) 522 active+clean 4 active+remapped+backfilling 3 active+clean+scrubbing+deep
io: recovery: 46 MiB/s, 21 objects/s
The host is reachable (its other OSDs are in) and from the systemd logs of the OSD I don't see anything wrong:
$ sudo systemctl status ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45@osd.34 ● ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45@osd.34.service - Ceph
osd.34
for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 Loaded: loaded (/etc/systemd/system/ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45@.service;
enabled; vendor preset: disabled) Active: active (running) since Mon 2023-06-12 17:00:25 CEST; 15h ago Main PID: 36286 (bash) Tasks: 11 (limit: 152154) Memory: 20.0M CGroup:
/system.slice/system-ceph\x2db1029256\x2d7bb3\x2d11ec\x2da8ce\x2dac1f6b627b45.slice/ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45@osd.34.service
├─36286 /bin/bash /var/lib/ceph/b1029256-7bb3-11ec-a8ce-ac1f6b627b45/osd.34/unit.run └─36657 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45-osd-34 --pids-limit=0 -e CONTAINER_IMAGE=snack14/ceph-wizard@sha>
Jun 12 17:00:25 balin systemd[1]: Started Ceph osd.34 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45. Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-34 Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-34 --no-mon-config --dev
/dev/mapper/ceph--9a4c3927--d3da--4b49--80fe--6cdc00c7897c-osd--block--36d2f793--e5c7--4247--a314--bcc40389d50d
Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -h ceph:ceph
/dev/mapper/ceph--9a4c3927--d3da--4b49--80fe--6cdc00c7897c-osd--block--36d2f793--e5c7--4247--a314--bcc40389d50d
Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-6 Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/ln -s
/dev/mapper/ceph--9a4c3927--d3da--4b49--80fe--6cdc00c7897c-osd--block--36d2f793--e5c7--4247--a314--bcc40389d50d /var/lib/ceph/osd/ceph-34/block
Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-34 Jun 12 17:00:27 balin bash[36306]: --> ceph-volume raw activate successful for osd ID: 34 Jun 12 17:00:29 balin bash[36657]: debug 2023-06-12T15:00:29.066+0000 7f818e356540 -1 Falling back to public interface
I'd need some help to understand how to fix this. Thank you,
Nicola
-- Nicola Mori, Ph.D. INFN sezione di Firenze Via Bruno Rossi 1, 50019 Sesto F.no (Italy) +390554572660 mori@fi.infn.it _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Dario, I think the connectivity is ok. My cluster has just a public interface, and all of the other services on the same machine (osds and mgr) work flawlessly so I guess the connectivity is ok. Or in other words, I don't know what to look for in the network since all the other services work, do you have any suggestion? Nicola
Hello, Have you increased the osd debug level to get more output? Does dmesg on the host machine report anything? Are there any smart errors on the drive? Regards, Curt On Thu, Jun 15, 2023, 13:30 Nicola Mori <mori@fi.infn.it> wrote:
Hi Dario,
I think the connectivity is ok. My cluster has just a public interface, and all of the other services on the same machine (osds and mgr) work flawlessly so I guess the connectivity is ok. Or in other words, I don't know what to look for in the network since all the other services work, do you have any suggestion?
Nicola _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Curt, I increased the debug level but still the OSD daemon doesn't log anything more than I already posted. dmesg does not report anything suspect (the osd disk has the very same messages as other disks for working osds), and smart is not very helpful: # smartctl -a /dev/sdf smartctl 7.1 2020-04-05 r5049 [x86_64-linux-4.18.0-477.13.1.el8_8.x86_64] (local build) Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Vendor: WDC Product: WD2002FYPS-02W3B Revision: R001 Compliance: SPC-3 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Logical block size: 512 bytes Rotation Rate: 10000 rpm Logical Unit id: 0x0004d927fffff850 Serial number: WD-WCAVY7349539 Device type: disk Transport protocol: Fibre channel (FCP-2) Local Time is: Thu Jun 15 11:52:57 2023 CEST SMART support is: Available - device has SMART capability. SMART support is: Enabled Temperature Warning: Disabled or Not Supported === START OF READ SMART DATA SECTION === SMART Health Status: OK Current Drive Temperature: 0 C Drive Trip Temperature: 0 C Elements in grown defect list: 0 Error Counter logging not supported Device does not support Self Test logging The only suspect thing I found is that the systemd log for the culprit OSD is somehow missing a part w.r.t. to the other OSDs. Here is the problematic one: Jun 15 11:59:43 balin systemd[1]: Started Ceph osd.34 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45. Jun 15 12:00:06 balin bash[2776]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-34 Jun 15 12:00:06 balin bash[2776]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-34 --no-mon-config --dev /dev/mapper/ceph--9a4c3927--d3da--4b49--80fe--6cdc00c7897c-osd--block--36d2f793--e5c7--4247--a314--bcc40389d50d Jun 15 12:00:06 balin bash[2776]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/ceph--9a4c3927--d3da--4b49--80fe--6cdc00c7897c-osd--block--36d2f793--e5c7--4247--a314--bcc40389d50d Jun 15 12:00:06 balin bash[2776]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-6 Jun 15 12:00:06 balin bash[2776]: Running command: /usr/bin/ln -s /dev/mapper/ceph--9a4c3927--d3da--4b49--80fe--6cdc00c7897c-osd--block--36d2f793--e5c7--4247--a314--bcc40389d50d /var/lib/ceph/osd/ceph-34/block Jun 15 12:00:06 balin bash[2776]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-34 Jun 15 12:00:06 balin bash[2776]: --> ceph-volume raw activate successful for osd ID: 34 Jun 15 12:00:12 balin bash[5536]: debug 2023-06-15T10:00:12.977+0000 7f8e1c57b540 -1 Falling back to public interface while for all the other OSDs it looks like: Jun 15 11:59:43 balin systemd[1]: Started Ceph osd.29 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45. Jun 15 12:00:06 balin bash[2793]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-29 Jun 15 12:00:06 balin bash[2793]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-29 --no-mon-config --dev /dev/mapper/ceph--06d03e18--2c8b--48a1--9bf6--7de5ff16af83-osd--block--5be3d54e--1fc8--400f--8664--b7d0d509f9b5 Jun 15 12:00:06 balin bash[2793]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/ceph--06d03e18--2c8b--48a1--9bf6--7de5ff16af83-osd--block--5be3d54e--1fc8--400f--8664--b7d0d509f9b5 Jun 15 12:00:06 balin bash[2793]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-5 Jun 15 12:00:06 balin bash[2793]: Running command: /usr/bin/ln -s /dev/mapper/ceph--06d03e18--2c8b--48a1--9bf6--7de5ff16af83-osd--block--5be3d54e--1fc8--400f--8664--b7d0d509f9b5 /var/lib/ceph/osd/ceph-29/block Jun 15 12:00:06 balin bash[2793]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-29 Jun 15 12:00:06 balin bash[2793]: --> ceph-volume raw activate successful for osd ID: 29 Jun 15 12:00:15 balin bash[5687]: debug 2023-06-15T10:00:15.093+0000 7fdbc133a540 -1 Falling back to public interface Jun 15 12:00:36 balin bash[5687]: debug 2023-06-15T10:00:36.528+0000 7fdbc133a540 -1 bdev(0x55bc3ba7dc00 /var/lib/ceph/osd/ceph-29/block) read_random stalled read 0x101c28398f~1149 (buffered) since 357.9605221s, timeout is 5.0000000s Jun 15 12:00:42 balin bash[5687]: debug 2023-06-15T10:00:42.915+0000 7fdbc133a540 -1 bdev(0x55bc3ba7dc00 /var/lib/ceph/osd/ceph-29/block) read_random stalled read 0x1236eeff30~f4f (buffered) since 364.5580473s, timeout is 5.0000000s Jun 15 12:01:10 balin bash[5687]: debug 2023-06-15T10:01:10.767+0000 7fdbc133a540 -1 bdev(0x55bc3ba7dc00 /var/lib/ceph/osd/ceph-29/block) read stalled read 0x4544a98000~8000 (buffered) since 392.2135163s, timeout is 5.0000000s Jun 15 12:01:12 balin bash[5687]: debug 2023-06-15T10:01:12.414+0000 7fdbc133a540 -1 osd.29 158243 log_to_monitors true Jun 15 12:01:17 balin bash[5687]: debug 2023-06-15T10:01:17.454+0000 7fdbb211c700 -1 osd.29 158243 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory So it looks like the osd stops after "Falling back to public interface". Actually I see one OSD process that uses no CPU and much less memory than the others. So probably the osd is partially stuck, maybe waiting for the disk but correctly communicating with the cluster... I'll replace the disk and see if that helps. Nicola
I have been able to (sort-of) fix the problem by removing the problematic OSD, zapping the disk and starting a new OSD. The new OSD is backfilling, but now the problem is that some parts of Ceph are still waiting for the OSD removal, and the OSD (despite not running anymore on the host) is seen as a stray daemon: # ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.34 on host balin not managed by cephadm # ceph osd tree | grep 34 34 hdd 1.81940 osd.34 down 0 1.00000 # ceph orch osd rm status OSD HOST STATE PGS REPLACE FORCE ZAP DRAIN STARTED AT 34 balin done, waiting for purge 0 False True False # ceph orch osd rm stop osd.34 Unable to find OSD in the queue: osd.34 I restarted the mgrs but it didn't help. Any suggestion? Nicola
The osd daemon finally disappeared without further intervention. I guess I should have had more patience and wait the purge process to finish. Thanks to everybody who helped. Nicola Il 15 giugno 2023 15:02:16 CEST, Nicola Mori <mori@fi.infn.it> ha scritto:
I have been able to (sort-of) fix the problem by removing the problematic OSD, zapping the disk and starting a new OSD. The new OSD is backfilling, but now the problem is that some parts of Ceph are still waiting for the OSD removal, and the OSD (despite not running anymore on the host) is seen as a stray daemon:
# ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.34 on host balin not managed by cephadm
# ceph osd tree | grep 34 34 hdd 1.81940 osd.34 down 0 1.00000
# ceph orch osd rm status OSD HOST STATE PGS REPLACE FORCE ZAP DRAIN STARTED AT 34 balin done, waiting for purge 0 False True False
# ceph orch osd rm stop osd.34 Unable to find OSD in the queue: osd.34
I restarted the mgrs but it didn't help. Any suggestion?
Nicola
-- Nicola Mori, Ph.D. INFN sezione di Firenze Via Bruno Rossi 1, 50019 Sesto F.no (Italy) +390554572660 mori@fi.infn.it
After cluster enters healthy state mgr should re-check stray daemons, a lot of activities are on hold while cluster is in warning state. In the event it does not disappear after cluster is healthy than mgr restart should help. Kind regards, Nino On Fri, Jun 16, 2023 at 10:24 PM Nicola Mori <mori@fi.infn.it> wrote:
The osd daemon finally disappeared without further intervention. I guess I should have had more patience and wait the purge process to finish. Thanks to everybody who helped.
Nicola
Il 15 giugno 2023 15:02:16 CEST, Nicola Mori <mori@fi.infn.it> ha scritto:
I have been able to (sort-of) fix the problem by removing the problematic
OSD, zapping the disk and starting a new OSD. The new OSD is backfilling, but now the problem is that some parts of Ceph are still waiting for the OSD removal, and the OSD (despite not running anymore on the host) is seen as a stray daemon:
# ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.34 on host balin not managed by cephadm
# ceph osd tree | grep 34 34 hdd 1.81940 osd.34 down 0 1.00000
# ceph orch osd rm status OSD HOST STATE PGS REPLACE FORCE ZAP DRAIN
STARTED AT
34 balin done, waiting for purge 0 False True False
# ceph orch osd rm stop osd.34 Unable to find OSD in the queue: osd.34
I restarted the mgrs but it didn't help. Any suggestion?
Nicola
-- Nicola Mori, Ph.D. INFN sezione di Firenze Via Bruno Rossi 1, 50019 Sesto F.no (Italy) +390554572660 mori@fi.infn.it _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Curt
-
Dario Graña
-
Eugen Block
-
Nicola Mori
-
Nino Kotur