Created no osd(s) on host, already created?
Hello list, We have a Ceph cluster with two management nodes and six data nodes. Each data node has 28 HDD disks. One disk recently failed in one of the nodes, corresponding to osd.2. To replace the disk, we took the osd.2 out, stopped it, and after a few days removed it, basically: ceph osd out osd.2 ceph osd ok-to-stop osd.2 Once Ok to stop, then: ceph orch daemon stop osd.2 Once stopped: ceph osd crush remove osd.2 ceph auth del osd.2 ceph osd rm osd.2 Then checked `ceph osd tree`, `ceph orch ps`, `ceph -s`, and all confirmed that the OSD was gone. We then proceeded to physically replace the failed drive on the node. It was /dev/sdac before, and after replacing it (hot swap) the system identified it as /dev/sdah. I zapped it on the node with `sgdisk --zap-all /dev/sdah` and, back on the management node, I could see that the disk was now showing up and marked as available with `ceph orch device ls`. Then I proceeded to add a new OSD to that disk with: `ceph orch daemon add osd node-osd1:/dev/sdah`, which then failed with: Created no osd(s) on host server-osd1; already created? Which was rather strange, because total OSDs was still showing as 1 less, no new OSD was showing up in `ceph osd tree`. What was even odder is that the disk that was showing as available in `ceph orch device ls` now shows as *not* available, and looking at lsblk's output in the node it seems that it was populated by Ceph: # lsblk /dev/sdah NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdah 66:16 0 16.4T 0 disk └─ceph--157c7c44--b519--4f6d--a54b--ecd466cf81d0-osd--block--10ada574--b99c--466d--8ad6--2c97d17d1f66 253:59 0 16.4T 0 lvm Any hints on how to proceed to get the OSD added back with this disk? Thank you for any suggestions! - Gustavo
Alright, now `ceph orch device ls` show the disk as locked: HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS node-osd1 /dev/sdah hdd LENOVO_ST18000NM004J_ZR5F5TH00000W413B814 18.0T No 10m ago locked I also noticed that in `ceph osd crush dump` where osd.2 was listed before in the device list now there's a mysterious "device2": { "devices": [ { "id": 0, "name": "osd.0", "class": "hdd" }, { "id": 1, "name": "osd.1", "class": "hdd" }, { "id": 2, "name": "device2" }, { "id": 3, "name": "osd.3", "class": "hdd" }, ... Even though lsblk shows a Ceph LVM volume created on the disk, it seems that it was not complete: # lsblk /dev/sdah NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdah 66:16 0 16.4T 0 disk └─ceph--157c7c44--b519--4f6d--a54b--ecd466cf81d0-osd--block--10ada574--b99c--466d--8ad6--2c97d17d1f66 253:59 0 16.4T 0 lvm # blkid /dev/sdah # lvs | grep 157c7c44 # vgs | grep 157c7c44 # pvs | grep sdah # It seems that OSD creation was in progress and somehow only got halfway there. Could it have to do with the fact that our cluster is not in the best shape, though it is recovery well? ceph -s below: cluster: id: 26315dca-383a-11ee-9d49-00620b4c2392 health: HEALTH_ERR 987 scrub errors Possible data damage: 17 pgs inconsistent 2192 pgs not deep-scrubbed in time 2192 pgs not scrubbed in time services: mon: 5 daemons, quorum node-admin1,node-admin2,node-osd1,node-osd2,node-osd3 (age 17M) mgr: node-admin2.sipadf(active, since 17M), standbys: node-admin1.nwaovh mds: 2/2 daemons up, 2 standby osd: 167 osds: 167 up (since 3d), 167 in (since 2h); 237 remapped pgs data: volumes: 2/2 healthy pools: 9 pools, 2273 pgs objects: 475.75M objects, 1.1 PiB usage: 1.6 PiB used, 1.1 PiB / 2.7 PiB avail pgs: 73449562/2835243480 objects misplaced (2.591%) 2024 active+clean 228 active+remapped+backfilling 12 active+clean+inconsistent 5 active+remapped+inconsistent+backfilling 4 active+remapped+backfill_wait io: client: 56 MiB/s wr, 0 op/s rd, 97 op/s wr recovery: 1.0 GiB/s, 440 objects/s progress: Global Recovery Event (7w) [=========================...] (remaining: 6d) ________________________________ From: Gustavo Garcia Rondina <grondina@uchicago.edu> Sent: Thursday, March 6, 2025 4:04 PM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Created no osd(s) on host, already created? Hello list, We have a Ceph cluster with two management nodes and six data nodes. Each data node has 28 HDD disks. One disk recently failed in one of the nodes, corresponding to osd.2. To replace the disk, we took the osd.2 out, stopped it, and after a few days removed it, basically: ceph osd out osd.2 ceph osd ok-to-stop osd.2 Once Ok to stop, then: ceph orch daemon stop osd.2 Once stopped: ceph osd crush remove osd.2 ceph auth del osd.2 ceph osd rm osd.2 Then checked `ceph osd tree`, `ceph orch ps`, `ceph -s`, and all confirmed that the OSD was gone. We then proceeded to physically replace the failed drive on the node. It was /dev/sdac before, and after replacing it (hot swap) the system identified it as /dev/sdah. I zapped it on the node with `sgdisk --zap-all /dev/sdah` and, back on the management node, I could see that the disk was now showing up and marked as available with `ceph orch device ls`. Then I proceeded to add a new OSD to that disk with: `ceph orch daemon add osd node-osd1:/dev/sdah`, which then failed with: Created no osd(s) on host server-osd1; already created? Which was rather strange, because total OSDs was still showing as 1 less, no new OSD was showing up in `ceph osd tree`. What was even odder is that the disk that was showing as available in `ceph orch device ls` now shows as *not* available, and looking at lsblk's output in the node it seems that it was populated by Ceph: # lsblk /dev/sdah NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdah 66:16 0 16.4T 0 disk └─ceph--157c7c44--b519--4f6d--a54b--ecd466cf81d0-osd--block--10ada574--b99c--466d--8ad6--2c97d17d1f66 253:59 0 16.4T 0 lvm Any hints on how to proceed to get the OSD added back with this disk? Thank you for any suggestions! - Gustavo _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, can you show the output of 'ceph orch ls osd --export'? I would look in the cephadm.log and ceph-volume.log on that node as well as in the active mgr log. If you already have an osd service that would pick up this osd, and you zapped it, its creation might have been interrupted. If you create all your OSDs manually with 'ceph orch daemon add osd...' this theory doesn't make much sense. The replaced OSD would not necessarily get the same ID as before (2), just to mention it. Zitat von Gustavo Garcia Rondina <grondina@uchicago.edu>:
Alright, now `ceph orch device ls` show the disk as locked:
HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS node-osd1 /dev/sdah hdd LENOVO_ST18000NM004J_ZR5F5TH00000W413B814 18.0T No 10m ago locked
I also noticed that in `ceph osd crush dump` where osd.2 was listed before in the device list now there's a mysterious "device2":
{ "devices": [ { "id": 0, "name": "osd.0", "class": "hdd" }, { "id": 1, "name": "osd.1", "class": "hdd" }, { "id": 2, "name": "device2" }, { "id": 3, "name": "osd.3", "class": "hdd" }, ...
Even though lsblk shows a Ceph LVM volume created on the disk, it seems that it was not complete:
# lsblk /dev/sdah NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdah 66:16 0 16.4T 0 disk └─ceph--157c7c44--b519--4f6d--a54b--ecd466cf81d0-osd--block--10ada574--b99c--466d--8ad6--2c97d17d1f66 253:59 0 16.4T 0 lvm # blkid /dev/sdah # lvs | grep 157c7c44 # vgs | grep 157c7c44 # pvs | grep sdah #
It seems that OSD creation was in progress and somehow only got halfway there.
Could it have to do with the fact that our cluster is not in the best shape, though it is recovery well? ceph -s below:
cluster: id: 26315dca-383a-11ee-9d49-00620b4c2392 health: HEALTH_ERR 987 scrub errors Possible data damage: 17 pgs inconsistent 2192 pgs not deep-scrubbed in time 2192 pgs not scrubbed in time
services: mon: 5 daemons, quorum node-admin1,node-admin2,node-osd1,node-osd2,node-osd3 (age 17M) mgr: node-admin2.sipadf(active, since 17M), standbys: node-admin1.nwaovh mds: 2/2 daemons up, 2 standby osd: 167 osds: 167 up (since 3d), 167 in (since 2h); 237 remapped pgs
data: volumes: 2/2 healthy pools: 9 pools, 2273 pgs objects: 475.75M objects, 1.1 PiB usage: 1.6 PiB used, 1.1 PiB / 2.7 PiB avail pgs: 73449562/2835243480 objects misplaced (2.591%) 2024 active+clean 228 active+remapped+backfilling 12 active+clean+inconsistent 5 active+remapped+inconsistent+backfilling 4 active+remapped+backfill_wait
io: client: 56 MiB/s wr, 0 op/s rd, 97 op/s wr recovery: 1.0 GiB/s, 440 objects/s
progress: Global Recovery Event (7w) [=========================...] (remaining: 6d)
________________________________ From: Gustavo Garcia Rondina <grondina@uchicago.edu> Sent: Thursday, March 6, 2025 4:04 PM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Created no osd(s) on host, already created?
Hello list,
We have a Ceph cluster with two management nodes and six data nodes. Each data node has 28 HDD disks. One disk recently failed in one of the nodes, corresponding to osd.2. To replace the disk, we took the osd.2 out, stopped it, and after a few days removed it, basically:
ceph osd out osd.2 ceph osd ok-to-stop osd.2
Once Ok to stop, then:
ceph orch daemon stop osd.2
Once stopped:
ceph osd crush remove osd.2 ceph auth del osd.2 ceph osd rm osd.2
Then checked `ceph osd tree`, `ceph orch ps`, `ceph -s`, and all confirmed that the OSD was gone. We then proceeded to physically replace the failed drive on the node. It was /dev/sdac before, and after replacing it (hot swap) the system identified it as /dev/sdah.
I zapped it on the node with `sgdisk --zap-all /dev/sdah` and, back on the management node, I could see that the disk was now showing up and marked as available with `ceph orch device ls`.
Then I proceeded to add a new OSD to that disk with: `ceph orch daemon add osd node-osd1:/dev/sdah`, which then failed with:
Created no osd(s) on host server-osd1; already created?
Which was rather strange, because total OSDs was still showing as 1 less, no new OSD was showing up in `ceph osd tree`. What was even odder is that the disk that was showing as available in `ceph orch device ls` now shows as *not* available, and looking at lsblk's output in the node it seems that it was populated by Ceph:
# lsblk /dev/sdah NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdah 66:16 0 16.4T 0 disk └─ceph--157c7c44--b519--4f6d--a54b--ecd466cf81d0-osd--block--10ada574--b99c--466d--8ad6--2c97d17d1f66 253:59 0 16.4T 0 lvm
Any hints on how to proceed to get the OSD added back with this disk?
Thank you for any suggestions!
- Gustavo
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eugen,
From: Eugen Block <eblock@nde.ag> Sent: Friday, March 7, 2025 1:21 AM
can you show the output of 'ceph orch ls osd --export'?
# ceph orch ls osd --export service_type: osd service_id: osd_spec service_name: osd.osd_spec placement: host_pattern: node-osd5 spec: data_devices: rotational: 1 db_devices: rotational: 0 filter_logic: AND objectstore: bluestore
I would look in the cephadm.log and ceph-volume.log on that node as well as in the active mgr log.
The cephadm.log indicates a race condition while zapping: 2025-03-06 16:34:31,728 7f033324db80 INFO /usr/bin/podman: stderr --> Zapping: /dev/sdah 2025-03-06 16:34:31,728 7f033324db80 INFO /usr/bin/podman: stderr stderr: wipefs: error: /dev/sdah: probing initialization failed: Device or resource busy 2025-03-06 16:34:31,728 7f033324db80 INFO /usr/bin/podman: stderr --> failed to wipefs device, will try again to workaround probable race condition ... 2025-03-06 16:34:31,728 7f033324db80 INFO /usr/bin/podman: stderr Traceback (most recent call last): 2025-03-06 16:34:31,729 7f033324db80 INFO /usr/bin/podman: stderr RuntimeError: could not complete wipefs on device: /dev/sdah There are similar logs in ceph-volume.log. Looking at the ceph-osd.2.log around the same time, it seems that osd.2 was being created, but that's the last of it: 2025-03-06T14:38:57.605-0600 7fc9a6185540 4 rocksdb: [db/db_impl/db_impl.cc:446] Shutdown: canceling all background work 2025-03-06T14:38:57.606-0600 7fc9a6185540 4 rocksdb: [db/db_impl/db_impl.cc:625] Shutdown complete 2025-03-06T14:38:57.606-0600 7fc9a6185540 1 bluefs umount 2025-03-06T14:38:57.606-0600 7fc9a6185540 1 bdev(0x55bd46237000 /var/lib/ceph/osd/ceph-2//block) close 2025-03-06T14:38:57.876-0600 7fc9a6185540 1 freelist shutdown 2025-03-06T14:38:57.876-0600 7fc9a6185540 1 bdev(0x55bd46237800 /var/lib/ceph/osd/ceph-2//block) close 2025-03-06T14:38:58.125-0600 7fc9a6185540 0 created object store /var/lib/ceph/osd/ceph-2/ for osd.2 fsid 26315dca-383a-11ee-9d49-00620b4c2392
If you already have an osd service that would pick up this osd, and you zapped it, its creation might have been interrupted.
I think that's exactly what happened.
If you create all your OSDs manually with 'ceph orch daemon add osd...' this theory doesn't make much sense.
This cluster was deployed by a contractor some couple of years ago, and it appears that they have indeed used OSD Spec to create the OSDs, so I think your theory is right (as is Robert's, in the other reply). Do you think zapping the disk with the orchestrator and waiting for it to be used by an automatic OSD creation would work? Should I be worried about remains of this OSD that was attempted to deploy but was interrupted by my manual actions? Thank you - Gustavo
Once stopped:
ceph osd crush remove osd.2 ceph auth del osd.2 ceph osd rm osd.2
While I can't help you with the is-it-gone-or-not part of your journey, the three commands above are correct, but also done in one single step with "ceph osd purge osd.2". So just adding this if anyone else is doing those 3. Forgetting one of them makes for boring debugging later, so it is nice that there is a purge to make all those three needed operations at once. Now if anyone can help you with your actual problem it would be nice. =) -- May the most significant bit of your life be positive.
From: Janne Johansson <icepic.dz@gmail.com> Sent: Friday, March 7, 2025 1:10 AM While I can't help you with the is-it-gone-or-not part of your journey, the three commands above are correct, but also done in one single step with "ceph osd purge osd.2". So just adding this if anyone else is doing those 3. Forgetting one of them makes for boring debugging later, so it is nice that there is a purge to make all those three needed operations at once.
Thank you, Janne! Will keep that in mind!
Now if anyone can help you with your actual problem it would be nice. =)
Some nice folks jumped in to help! Regards - Gustavo
Hi, Am 3/6/25 um 23:04 schrieb Gustavo Garcia Rondina:
ceph osd out osd.2 ceph osd ok-to-stop osd.2
Once Ok to stop, then:
ceph orch daemon stop osd.2
Once stopped:
ceph osd crush remove osd.2 ceph auth del osd.2 ceph osd rm osd.2
As this is a cluster managed by the orchestrator all these commands could have been ceph orch osd rm 2 For the original issue: Do you have an active (managed) OSD service in the orchestrator? If you can please post the output of ceph orch ls osd --export It looks like you did something on the new disk manually while the orchestrator was already creating an OSD. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi Robert
From: Robert Sander <r.sander@heinlein-support.de> Sent: Friday, March 7, 2025 7:02 AM
For the original issue: Do you have an active (managed) OSD service in the orchestrator?
In fact, I do - I hadn't noticed it before, the cluster was configured by a contractor a while ago and my expertise in Ceph still leaves a lot to desire. # ceph orch ls --service-type=osd NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd.osd_spec 167 10m ago 18M node-osd5
If you can please post the output of ceph orch ls osd --export
# ceph orch ls osd --export service_type: osd service_id: osd_spec service_name: osd.osd_spec placement: host_pattern: node-osd5 spec: data_devices: rotational: 1 db_devices: rotational: 0 filter_logic: AND objectstore: bluestore
It looks like you did something on the new disk manually while the orchestrator was already creating an OSD.
I believe that was the case, indeed. Do you think if I zap the disk using the orchestrator with: ceph orch device zap node-osd1 /dev/sdah And wait for it to be picked up for the orchestrator and have the OSD create according to the OSD Spec this would work? My main concern is some half-deployed OSD still lingering, such as that "device2" entry in the crush map. Thank you - Gustavo
Am 3/7/25 um 15:40 schrieb Gustavo Garcia Rondina:
# ceph orch ls osd --export service_type: osd service_id: osd_spec service_name: osd.osd_spec placement: host_pattern: node-osd5 spec: data_devices: rotational: 1 db_devices: rotational: 0 filter_logic: AND objectstore: bluestore
This is just for one host.
ceph orch device zap node-osd1 /dev/sdah
And this is a different host. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
From: Robert Sander <r.sander@heinlein-support.de> Sent: Friday, March 7, 2025 9:11 AM
# ceph orch ls osd --export service_type: osd service_id: osd_spec service_name: osd.osd_spec placement: host_pattern: node-osd5 spec: data_devices: rotational: 1 db_devices: rotational: 0 filter_logic: AND objectstore: bluestore
This is just for one host.
ceph orch device zap node-osd1 /dev/sdah
And this is a different host.
That's the only one we seem to have running: [root@node-admin1 ~]# ceph orch ls | grep osd osd.osd_spec 167 10m ago 18M cds3-osd5 [root@node-admin1 ~]# ceph orch ls --service-type=osd -f yaml service_type: osd service_id: osd_spec service_name: osd.osd_spec placement: host_pattern: node-osd5 spec: data_devices: rotational: 1 db_devices: rotational: 0 filter_logic: AND objectstore: bluestore status: created: '2023-08-23T10:43:40.474777Z' last_refresh: '2025-03-07T15:11:17.457725Z' running: 167 size: 167 The "running" and "size" lines shows all OSDs that we have, could it be that this service is doing all the OSD deployments? Thank you - Gustavo
It's possible that someone just changed the unit.meta file for all OSDs after creation so all of them would show the same service_name (osd.osd_spec). Are all your OSD hosts equipped with the same hardware? So rotational and non-rotational devices? If that is the case, you could just overwrite the host_pattern in the existing spec to apply to all OSD nodes, apply the spec, zap and wipe the faulty OSD and let cephadm recreate it automatically: - ceph orch ls osd --export > osd-spec.yaml - adapt host_pattern in osd-spec.yaml (or use labels, hosts or however you need to distinguish between OSD and non-OSD nodes), save the file. - ceph orch apply -i osd-spec.yaml --dry-run (just to check for any parsing errors) - ceph orch apply -i osd-spec.yaml (this won't do anything to existing OSDs, only deploy new ones if there are available devices matching the specs) - zap & wipe your faulty OSD - watch OSD creation If you're unsure what cephadm will actually do (will it place the DB on the rotational device?), you could set osd_crush_initial_weight to 0 so the OSD isn't immediately populated: ceph config set osd osd_crush_initial_weight 0 If your OSD is created successfully and with its DB on SSD, you can reweight the OSD (this will trigger data movement): ceph osd crush reweight osd.<OSD_ID> <CRUSH_WEIGHT> and then remove the osd_crush_initial_weight config: ceph config rm osd osd_crush_initial_weight Zitat von Gustavo Garcia Rondina <grondina@uchicago.edu>:
From: Robert Sander <r.sander@heinlein-support.de> Sent: Friday, March 7, 2025 9:11 AM
# ceph orch ls osd --export service_type: osd service_id: osd_spec service_name: osd.osd_spec placement: host_pattern: node-osd5 spec: data_devices: rotational: 1 db_devices: rotational: 0 filter_logic: AND objectstore: bluestore
This is just for one host.
ceph orch device zap node-osd1 /dev/sdah
And this is a different host.
That's the only one we seem to have running:
[root@node-admin1 ~]# ceph orch ls | grep osd osd.osd_spec 167 10m ago 18M cds3-osd5
[root@node-admin1 ~]# ceph orch ls --service-type=osd -f yaml service_type: osd service_id: osd_spec service_name: osd.osd_spec placement: host_pattern: node-osd5 spec: data_devices: rotational: 1 db_devices: rotational: 0 filter_logic: AND objectstore: bluestore status: created: '2023-08-23T10:43:40.474777Z' last_refresh: '2025-03-07T15:11:17.457725Z' running: 167 size: 167
The "running" and "size" lines shows all OSDs that we have, could it be that this service is doing all the OSD deployments?
Thank you - Gustavo _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Eugen Block
-
Gustavo Garcia Rondina
-
Janne Johansson
-
Robert Sander