Thanks for your reply Eugen. I’m fairly new to cephadm so I wasn’t aware that we could manage the drives without rebuilding them. However, we thought we’d take advantage of this opportunity to also encrypt the drives, and that does require a rebuild. I have a theory on why the orchestrator is confused. I want to create an osd service for each osd node so I can manage drives on a per-node basis. I started by creating a spec for the first node: service_type: osd service_id: ceph-osd31 placement: hosts: - ceph-osd31 spec: data_devices: rotational: 0 size: '3TB:' encrypted: true filter_logic: AND objectstore: bluestore But I also see a default spec, “osd”, which has placement set to “unmanaged”. `ceph orch ls osd —export` shows the following: service_type: osd service_name: osd unmanaged: true spec: filter_logic: AND objectstore: bluestore --- service_type: osd service_id: ceph-osd31 service_name: osd.ceph-osd31 placement: hosts: - ceph-osd31 spec: data_devices: rotational: 0 size: '3TB:' encrypted: true filter_logic: AND objectstore: bluestore `ceph orch ls osd` shows that I was able to convert 4 drives using my spec: NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd 95 10m ago - <unmanaged> osd.ceph-osd31 4 10m ago 43m ceph-osd31 Despite being able to convert 4 drives, I’m wondering if these specs are conflicting with one another, and that has confused the orchestrator. If so, how do I safely get from where I am now to where I want to be? :-) Cheers, /rjg On Sep 26, 2024, at 3:31 PM, Eugen Block <eblock@nde.ag> wrote: EXTERNAL EMAIL | USE CAUTION Hi, this seems a bit unnecessary to rebuild OSDs just to get them managed. If you apply a spec file that targets your hosts/OSDs, they will appear as managed. So when you would need to replace a drive, you could already utilize the orchestrator to remove and zap the drive. That works just fine. How to get out of your current situation is not entirely clear to me yet. I’ll reread your post tomorrow. Regards, Eugen Zitat von Bob Gibson <rjg@oicr.on.ca>: Hi, We recently converted a legacy cluster running Quincy v17.2.7 to cephadm. The conversion went smoothly and left all osds unmanaged by the orchestrator as expected. We’re now in the process of converting the osds to be managed by the orchestrator. We successfully converted a few of them, but then the orchestrator somehow got confused. `ceph health detail` reports a “stray daemon” for the osd we’re trying to convert, and the orchestrator is unable to refresh its device list so it doesn’t see any available devices. From the perspective of the osd node, the osd has been wiped and is ready to be reinstalled. We’ve also rebooted the node for good measure. `ceph osd tree` shows that the osd has been destroyed, but the orchestrator won’t reinstall it because it thinks the device is still active. The orchestrator device information is stale, but we’re unable to refresh it. The usual recommended workaround of failing over the mgr hasn’t helped. We’ve also tried `ceph orch device ls —refresh` to no avail. In fact after running that command subsequent runs of `ceph orch device ls` produce no output until the mgr is failed over again. Is there a way to force the orchestrator to refresh its list of devices when in this state? If not, can anyone offer any suggestions on how to fix this problem? Cheers, /rjg P.S. Some additional information in case it’s helpful... We’re using the following command to replace existing devices so that they’re managed by the orchestrator: ``` ceph orch osd rm <osd> --replace —zap ``` and we’re currently stuck on osd 88. ``` ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.88 on host ceph-osd31 not managed by cephadm ``` `ceph osd tree` shows that the osd has been destroyed and is ready to be replaced: ``` ceph osd tree-from ceph-osd31 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -46 34.93088 host ceph-osd31 84 ssd 3.49309 osd.84 up 1.00000 1.00000 85 ssd 3.49309 osd.85 up 1.00000 1.00000 86 ssd 3.49309 osd.86 up 1.00000 1.00000 87 ssd 3.49309 osd.87 up 1.00000 1.00000 88 ssd 3.49309 osd.88 destroyed 0 1.00000 89 ssd 3.49309 osd.89 up 1.00000 1.00000 90 ssd 3.49309 osd.90 up 1.00000 1.00000 91 ssd 3.49309 osd.91 up 1.00000 1.00000 92 ssd 3.49309 osd.92 up 1.00000 1.00000 93 ssd 3.49309 osd.93 up 1.00000 1.00000 ``` The cephadm log shows a claim on node `ceph-osd31` for that osd: ``` 2024-09-25T14:15:45.699348-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims -> {'ceph-osd31': ['88']} 2024-09-25T14:15:45.699534-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims for drivegroup ceph-osd31 -> {'ceph-osd31': ['88']} ``` `ceph orch device ls` shows that the device list isn’t refreshing: ``` ceph orch device ls ceph-osd31 HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS ceph-osd31 /dev/sdc ssd INTEL_SSDSC2KG038T8_PHYG039603PE3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdd ssd INTEL_SSDSC2KG038T8_PHYG039600AY3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sde ssd INTEL_SSDSC2KG038T8_PHYG039600CW3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdf ssd INTEL_SSDSC2KG038T8_PHYG039600CM3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdg ssd INTEL_SSDSC2KG038T8_PHYG039600UB3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdh ssd INTEL_SSDSC2KG038T8_PHYG039603753P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdi ssd INTEL_SSDSC2KG038T8_PHYG039603R63P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdj ssd INTEL_SSDSC2KG038TZ_PHYJ4011032M3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdk ssd INTEL_SSDSC2KG038TZ_PHYJ3234010J3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdl ssd INTEL_SSDSC2KG038T8_PHYG039603NS3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ``` `ceph node ls` thinks the osd still exists ``` ceph node ls osd | jq -r '."ceph-osd31"' [ 84, 85, 86, 87, 88, <— this shouldn’t exist 89, 90, 91, 92, 93 ] ``` Each osd node has 10x 3.8 TB ssd drives for osds. On `ceph-osd31`, cephadm doesn’t see osd.88 as expected: ``` cephadm ls --no-detail [ { "style": "cephadm:v1", "name": "osd.93", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.93" }, { "style": "cephadm:v1", "name": "osd.85", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.85" }, { "style": "cephadm:v1", "name": "osd.90", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.90" }, { "style": "cephadm:v1", "name": "osd.92", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.92" }, { "style": "cephadm:v1", "name": "osd.89", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.89" }, { "style": "cephadm:v1", "name": "osd.87", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.87" }, { "style": "cephadm:v1", "name": "osd.86", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.86" }, { "style": "cephadm:v1", "name": "osd.84", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.84" }, { "style": "cephadm:v1", "name": "osd.91", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.91" } ] ``` `lsblk` shows that `/dev/sdg` has been wiped. ``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 223.6G 0 disk |-sda1 8:1 0 94M 0 part `-sda2 8:2 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdb 8:16 0 223.6G 0 disk |-sdb1 8:17 0 94M 0 part `-sdb2 8:18 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdc 8:32 1 3.5T 0 disk `-ceph--03782b4c--9faa--49f5--b554--98e7b8515834-osd--block--ba272724--daa6--45f5--9f69--789cc0bda077 253:3 0 3.5T 0 lvm `-keCkP2-o6h8-jKkw-RKiD-UBFf-A8EL-JDJGPR 253:9 0 3.5T 0 crypt sdd 8:48 1 3.5T 0 disk `-ceph--c07907d8--4a75--4ba3--b5e1--2ebf49ecbdf6-osd--block--58d1d50d--6228--4e6f--9a52--2a305ba00700 253:7 0 3.5T 0 lvm `-WB8Mxn-qCHI-4T01-imiG-hNBR-by60-YuxgfD 253:11 0 3.5T 0 crypt sde 8:64 1 3.5T 0 disk `-ceph--6f9d4df4--7ce6--44a4--a7b1--62c85af8cfe0-osd--block--aabcb30d--0084--490a--969b--78f7af6e94da 253:8 0 3.5T 0 lvm `-g9qErH-vTXY-JQbs-eh61-W0Mn-TAV8-gof4zy 253:12 0 3.5T 0 crypt sdf 8:80 1 3.5T 0 disk `-ceph--d6b728f8--e365--46db--b30f--6c00805c752b-osd--block--88426db7--2322--4807--ac2e--b49929e170d6 253:6 0 3.5T 0 lvm `-LNG2gB-pa0w-gl2v-hVQ3-6qTd-aXsR-Lenri3 253:10 0 3.5T 0 crypt sdg 8:96 1 3.5T 0 disk sdh 8:112 1 3.5T 0 disk `-ceph--de2cfee6--8e0a--4aa0--9e6b--90c09025768c-osd--block--a3b86251--2799--4243--a857--f218fa90f29a 253:2 0 3.5T 0 lvm sdi 8:128 1 3.5T 0 disk `-ceph--30dee450--0fdd--46ea--9eec--6a4c7706df9c-osd--block--bfc090db--dde4--47dd--a1c9--1cd838ea43b3 253:4 0 3.5T 0 lvm sdj 8:144 1 3.5T 0 disk `-ceph--78febcf5--43f4--4820--8dc7--0f6c22816c9f-osd--block--da1e69c7--6427--4562--8290--90bcb9526747 253:0 0 3.5T 0 lvm sdk 8:160 1 3.5T 0 disk `-ceph--fe210281--b1f5--4d5e--9ab0--2f226612af00-osd--block--6bb9f308--e853--4303--83ea--553c3a3513e1 253:1 0 3.5T 0 lvm sdl 8:176 1 3.5T 0 disk `-ceph--9f21c916--f211--4d1b--8214--6ad1cecac810-osd--block--572d850c--c201--4af4--ac42--0ed2a6ed73ed 253:5 0 3.5T 0 lvm ``` _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Oh interesting, I just got into the same situation (I believe) on a test cluster: host1:~ # ceph orch ps | grep unknown osd.1 host6 stopped 72s ago 36m - 4096M <unknown> <unknown> <unknown> osd.13 host6 error 72s ago 36m - 4096M <unknown> <unknown> <unknown> I still had the remainders on the filesystem: host6:~ # ll /var/lib/ceph/543967bc-e586-32b8-bd2c-2d8b8b168f02/osd.1 insgesamt 68 lrwxrwxrwx 1 ceph ceph 111 27. Sep 14:43 block -> /dev/mapper/ceph--0e90997f--456e--4a9b--a8f9--a6f1038c1216-osd--block--81e7f32a--a728--4848--b14d--0b86bb7e1c69 lrwxrwxrwx 1 ceph ceph 108 27. Sep 14:43 block.db -> /dev/mapper/ceph--9ea6e95f--ad43--4e40--8920--2e772b2efa2f-osd--db--f9c57ec1--77c8--4d9a--85df--1dc053a24000 I just removed those two directories to clear the warning, now my orchestrator can deploy OSDs again on that node. Hope that helps! Zitat von Eugen Block <eblock@nde.ag>:

Right, if you need encryption, a rebuild is required. Your procedure has already worked 4 times, so I'd say nothing seems wrong with that per se. Regarding the stuck device list, do you see the mgr logging anything suspicious? Especially when you say that it only returns output after a failover. Those two osd specs are not conflicting since the first is "unmanaged" after adoption. Is there something in 'ceph orch osd rm status'? Can you run 'cephadm ceph-volume inventory' locally on that node? Do you see any hints in the node's syslog? Maybe try a reboot or something?

Zitat von Bob Gibson <rjg@oicr.on.ca>:

...

Thanks for your reply Eugen. I’m fairly new to cephadm so I wasn’t aware that we could manage the drives without rebuilding them. However, we thought we’d take advantage of this opportunity to also encrypt the drives, and that does require a rebuild.

I have a theory on why the orchestrator is confused. I want to create an osd service for each osd node so I can manage drives on a per-node basis.

I started by creating a spec for the first node:

service_type: osd service_id: ceph-osd31 placement: hosts: - ceph-osd31 spec: data_devices: rotational: 0 size: '3TB:' encrypted: true filter_logic: AND objectstore: bluestore

But I also see a default spec, “osd”, which has placement set to “unmanaged”.

`ceph orch ls osd —export` shows the following:

service_type: osd service_name: osd unmanaged: true spec: filter_logic: AND objectstore: bluestore --- service_type: osd service_id: ceph-osd31 service_name: osd.ceph-osd31 placement: hosts: - ceph-osd31 spec: data_devices: rotational: 0 size: '3TB:' encrypted: true filter_logic: AND objectstore: bluestore

`ceph orch ls osd` shows that I was able to convert 4 drives using my spec:

NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd 95 10m ago - <unmanaged> osd.ceph-osd31 4 10m ago 43m ceph-osd31

Despite being able to convert 4 drives, I’m wondering if these specs are conflicting with one another, and that has confused the orchestrator. If so, how do I safely get from where I am now to where I want to be? :-)

Cheers, /rjg

On Sep 26, 2024, at 3:31 PM, Eugen Block <eblock@nde.ag> wrote:

EXTERNAL EMAIL | USE CAUTION

Hi,

this seems a bit unnecessary to rebuild OSDs just to get them managed. If you apply a spec file that targets your hosts/OSDs, they will appear as managed. So when you would need to replace a drive, you could already utilize the orchestrator to remove and zap the drive. That works just fine. How to get out of your current situation is not entirely clear to me yet. I’ll reread your post tomorrow.

Regards, Eugen

Zitat von Bob Gibson <rjg@oicr.on.ca>:

Hi,

We recently converted a legacy cluster running Quincy v17.2.7 to cephadm. The conversion went smoothly and left all osds unmanaged by the orchestrator as expected. We’re now in the process of converting the osds to be managed by the orchestrator. We successfully converted a few of them, but then the orchestrator somehow got confused. `ceph health detail` reports a “stray daemon” for the osd we’re trying to convert, and the orchestrator is unable to refresh its device list so it doesn’t see any available devices.

From the perspective of the osd node, the osd has been wiped and is ready to be reinstalled. We’ve also rebooted the node for good measure. `ceph osd tree` shows that the osd has been destroyed, but the orchestrator won’t reinstall it because it thinks the device is still active. The orchestrator device information is stale, but we’re unable to refresh it. The usual recommended workaround of failing over the mgr hasn’t helped. We’ve also tried `ceph orch device ls —refresh` to no avail. In fact after running that command subsequent runs of `ceph orch device ls` produce no output until the mgr is failed over again.

Is there a way to force the orchestrator to refresh its list of devices when in this state? If not, can anyone offer any suggestions on how to fix this problem?

Cheers, /rjg

P.S. Some additional information in case it’s helpful...

We’re using the following command to replace existing devices so that they’re managed by the orchestrator:

``` ceph orch osd rm <osd> --replace —zap ```

and we’re currently stuck on osd 88.

``` ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.88 on host ceph-osd31 not managed by cephadm ```

`ceph osd tree` shows that the osd has been destroyed and is ready to be replaced:

``` ceph osd tree-from ceph-osd31 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -46 34.93088 host ceph-osd31 84 ssd 3.49309 osd.84 up 1.00000 1.00000 85 ssd 3.49309 osd.85 up 1.00000 1.00000 86 ssd 3.49309 osd.86 up 1.00000 1.00000 87 ssd 3.49309 osd.87 up 1.00000 1.00000 88 ssd 3.49309 osd.88 destroyed 0 1.00000 89 ssd 3.49309 osd.89 up 1.00000 1.00000 90 ssd 3.49309 osd.90 up 1.00000 1.00000 91 ssd 3.49309 osd.91 up 1.00000 1.00000 92 ssd 3.49309 osd.92 up 1.00000 1.00000 93 ssd 3.49309 osd.93 up 1.00000 1.00000 ```

The cephadm log shows a claim on node `ceph-osd31` for that osd:

``` 2024-09-25T14:15:45.699348-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims -> {'ceph-osd31': ['88']} 2024-09-25T14:15:45.699534-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims for drivegroup ceph-osd31 -> {'ceph-osd31': ['88']} ```

`ceph orch device ls` shows that the device list isn’t refreshing:

``` ceph orch device ls ceph-osd31 HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS ceph-osd31 /dev/sdc ssd INTEL_SSDSC2KG038T8_PHYG039603PE3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdd ssd INTEL_SSDSC2KG038T8_PHYG039600AY3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sde ssd INTEL_SSDSC2KG038T8_PHYG039600CW3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdf ssd INTEL_SSDSC2KG038T8_PHYG039600CM3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdg ssd INTEL_SSDSC2KG038T8_PHYG039600UB3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdh ssd INTEL_SSDSC2KG038T8_PHYG039603753P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdi ssd INTEL_SSDSC2KG038T8_PHYG039603R63P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdj ssd INTEL_SSDSC2KG038TZ_PHYJ4011032M3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdk ssd INTEL_SSDSC2KG038TZ_PHYJ3234010J3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdl ssd INTEL_SSDSC2KG038T8_PHYG039603NS3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ```

`ceph node ls` thinks the osd still exists

``` ceph node ls osd | jq -r '."ceph-osd31"' [ 84, 85, 86, 87, 88, <— this shouldn’t exist 89, 90, 91, 92, 93 ] ```

Each osd node has 10x 3.8 TB ssd drives for osds. On `ceph-osd31`, cephadm doesn’t see osd.88 as expected:

``` cephadm ls --no-detail [ { "style": "cephadm:v1", "name": "osd.93", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.93" }, { "style": "cephadm:v1", "name": "osd.85", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.85" }, { "style": "cephadm:v1", "name": "osd.90", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.90" }, { "style": "cephadm:v1", "name": "osd.92", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.92" }, { "style": "cephadm:v1", "name": "osd.89", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.89" }, { "style": "cephadm:v1", "name": "osd.87", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.87" }, { "style": "cephadm:v1", "name": "osd.86", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.86" }, { "style": "cephadm:v1", "name": "osd.84", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.84" }, { "style": "cephadm:v1", "name": "osd.91", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.91" } ] ```

`lsblk` shows that `/dev/sdg` has been wiped.

``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 223.6G 0 disk |-sda1 8:1 0 94M 0 part `-sda2 8:2 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdb 8:16 0 223.6G 0 disk |-sdb1 8:17 0 94M 0 part `-sdb2 8:18 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdc 8:32 1 3.5T 0 disk `-ceph--03782b4c--9faa--49f5--b554--98e7b8515834-osd--block--ba272724--daa6--45f5--9f69--789cc0bda077 253:3 0 3.5T lvm `-keCkP2-o6h8-jKkw-RKiD-UBFf-A8EL-JDJGPR 253:9 0 3.5T 0 crypt sdd 8:48 1 3.5T 0 disk `-ceph--c07907d8--4a75--4ba3--b5e1--2ebf49ecbdf6-osd--block--58d1d50d--6228--4e6f--9a52--2a305ba00700 253:7 0 3.5T lvm `-WB8Mxn-qCHI-4T01-imiG-hNBR-by60-YuxgfD 253:11 0 3.5T 0 crypt sde 8:64 1 3.5T 0 disk `-ceph--6f9d4df4--7ce6--44a4--a7b1--62c85af8cfe0-osd--block--aabcb30d--0084--490a--969b--78f7af6e94da 253:8 0 3.5T lvm `-g9qErH-vTXY-JQbs-eh61-W0Mn-TAV8-gof4zy 253:12 0 3.5T 0 crypt sdf 8:80 1 3.5T 0 disk `-ceph--d6b728f8--e365--46db--b30f--6c00805c752b-osd--block--88426db7--2322--4807--ac2e--b49929e170d6 253:6 0 3.5T lvm `-LNG2gB-pa0w-gl2v-hVQ3-6qTd-aXsR-Lenri3 253:10 0 3.5T 0 crypt sdg 8:96 1 3.5T 0 disk sdh 8:112 1 3.5T 0 disk `-ceph--de2cfee6--8e0a--4aa0--9e6b--90c09025768c-osd--block--a3b86251--2799--4243--a857--f218fa90f29a 253:2 0 3.5T lvm sdi 8:128 1 3.5T 0 disk `-ceph--30dee450--0fdd--46ea--9eec--6a4c7706df9c-osd--block--bfc090db--dde4--47dd--a1c9--1cd838ea43b3 253:4 0 3.5T lvm sdj 8:144 1 3.5T 0 disk `-ceph--78febcf5--43f4--4820--8dc7--0f6c22816c9f-osd--block--da1e69c7--6427--4562--8290--90bcb9526747 253:0 0 3.5T lvm sdk 8:160 1 3.5T 0 disk `-ceph--fe210281--b1f5--4d5e--9ab0--2f226612af00-osd--block--6bb9f308--e853--4303--83ea--553c3a3513e1 253:1 0 3.5T lvm sdl 8:176 1 3.5T 0 disk `-ceph--9f21c916--f211--4d1b--8214--6ad1cecac810-osd--block--572d850c--c201--4af4--ac42--0ed2a6ed73ed 253:5 0 3.5T lvm ```

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Okay, apparently this is not what I was facing. I see two other options right now. The first would be to purge osd.88 from the crush tree entirely. The second approach would be to create an osd manually with ceph-volume, not cephadm ceph-volume, to create a legacy osd (you'd get warnings about a stray daemon). If that works, adopt the osd with cephadm. I don't have a better idea right now. Zitat von Bob Gibson <rjg@oicr.on.ca>:

Here are the contents from the same directory on our osd node:

ceph-osd31.prod.os:/var/lib/ceph/9b3b3539-59a9-4338-8bab-3badfab6e855# ls -l total 412 -rw-r--r-- 1 root root 366903 Sep 14 14:53 cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b drwx------ 3 167 167 4096 Sep 14 15:01 crash drwxr-xr-x 12 root root 4096 Sep 15 12:06 custom_config_files drw-rw---- 2 root root 4096 Sep 23 17:00 home drwx------ 2 167 167 4096 Sep 26 12:47 osd.84 drwx------ 2 167 167 4096 Sep 26 12:47 osd.85 drwx------ 2 167 167 4096 Sep 26 12:47 osd.86 drwx------ 2 167 167 4096 Sep 26 12:47 osd.87 drwx------ 2 167 167 4096 Sep 26 12:47 osd.89 drwx------ 2 167 167 4096 Sep 26 12:47 osd.90 drwx------ 2 167 167 4096 Sep 26 12:47 osd.91 drwx------ 2 167 167 4096 Sep 26 12:47 osd.92 drwx------ 2 167 167 4096 Sep 26 12:47 osd.93 drwx------ 6 root root 4096 Sep 23 15:59 removed

In our case the osd.88 directory is under the subdirectory named “removed”, the same as the other odds which have been converted.

ceph-osd31.prod.os:/var/lib/ceph/9b3b3539-59a9-4338-8bab-3badfab6e855# ls -l removed/osd.88_2024-09-23T19\:59\:42.162302Z/ total 64 lrwxrwxrwx 1 167 167 93 Sep 15 12:10 block -> /dev/ceph-2a13ec6a-a5f0-4773-8254-c38b915c824a/osd-block-7f8f9778-5ae2-47c1-bd03-a92a3a7a1db1 -rw------- 1 167 167 37 Sep 15 12:10 ceph_fsid -rw------- 1 167 167 259 Sep 14 15:14 config -rw------- 1 167 167 37 Sep 15 12:10 fsid -rw------- 1 167 167 56 Sep 15 12:10 keyring -rw------- 1 167 167 6 Sep 15 12:10 ready -rw------- 1 167 167 3 Sep 14 11:11 require_osd_release -rw------- 1 167 167 10 Sep 15 12:10 type -rw------- 1 167 167 38 Sep 14 15:14 unit.configured -rw------- 1 167 167 48 Sep 14 15:14 unit.created -rw------- 1 167 167 26 Sep 14 15:06 unit.image -rw------- 1 167 167 76 Sep 14 15:06 unit.meta -rw------- 1 167 167 1527 Sep 14 15:06 unit.poststop -rw------- 1 167 167 2586 Sep 14 15:06 unit.run -rw------- 1 167 167 334 Sep 14 15:06 unit.stop -rw------- 1 167 167 3 Sep 15 12:10 whoami

On Sep 27, 2024, at 9:30 AM, Eugen Block <eblock@nde.ag> wrote:

EXTERNAL EMAIL | USE CAUTION

Oh interesting, I just got into the same situation (I believe) on a test cluster:

host1:~ # ceph orch ps | grep unknown osd.1 host6 stopped 72s ago 36m - 4096M <unknown> <unknown> <unknown> osd.13 host6 error 72s ago 36m - 4096M <unknown> <unknown> <unknown>

I still had the remainders on the filesystem:

host6:~ # ll /var/lib/ceph/543967bc-e586-32b8-bd2c-2d8b8b168f02/osd.1 insgesamt 68 lrwxrwxrwx 1 ceph ceph 111 27. Sep 14:43 block -> /dev/mapper/ceph--0e90997f--456e--4a9b--a8f9--a6f1038c1216-osd--block--81e7f32a--a728--4848--b14d--0b86bb7e1c69 lrwxrwxrwx 1 ceph ceph 108 27. Sep 14:43 block.db -> /dev/mapper/ceph--9ea6e95f--ad43--4e40--8920--2e772b2efa2f-osd--db--f9c57ec1--77c8--4d9a--85df--1dc053a24000

I just removed those two directories to clear the warning, now my orchestrator can deploy OSDs again on that node.

Hope that helps!

Zitat von Eugen Block <eblock@nde.ag>:

Right, if you need encryption, a rebuild is required. Your procedure has already worked 4 times, so I'd say nothing seems wrong with that per se. Regarding the stuck device list, do you see the mgr logging anything suspicious? Especially when you say that it only returns output after a failover. Those two osd specs are not conflicting since the first is "unmanaged" after adoption. Is there something in 'ceph orch osd rm status'? Can you run 'cephadm ceph-volume inventory' locally on that node? Do you see any hints in the node's syslog? Maybe try a reboot or something?

Zitat von Bob Gibson <rjg@oicr.on.ca>:

Thanks for your reply Eugen. I’m fairly new to cephadm so I wasn’t aware that we could manage the drives without rebuilding them. However, we thought we’d take advantage of this opportunity to also encrypt the drives, and that does require a rebuild.

I have a theory on why the orchestrator is confused. I want to create an osd service for each osd node so I can manage drives on a per-node basis.

I started by creating a spec for the first node:

service_type: osd service_id: ceph-osd31 placement: hosts: - ceph-osd31 spec: data_devices: rotational: 0 size: '3TB:' encrypted: true filter_logic: AND objectstore: bluestore

But I also see a default spec, “osd”, which has placement set to “unmanaged”.

`ceph orch ls osd —export` shows the following:

service_type: osd service_name: osd unmanaged: true spec: filter_logic: AND objectstore: bluestore --- service_type: osd service_id: ceph-osd31 service_name: osd.ceph-osd31 placement: hosts: - ceph-osd31 spec: data_devices: rotational: 0 size: '3TB:' encrypted: true filter_logic: AND objectstore: bluestore

`ceph orch ls osd` shows that I was able to convert 4 drives using my spec:

NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd 95 10m ago - <unmanaged> osd.ceph-osd31 4 10m ago 43m ceph-osd31

Despite being able to convert 4 drives, I’m wondering if these specs are conflicting with one another, and that has confused the orchestrator. If so, how do I safely get from where I am now to where I want to be? :-)

Cheers, /rjg

On Sep 26, 2024, at 3:31 PM, Eugen Block <eblock@nde.ag> wrote:

EXTERNAL EMAIL | USE CAUTION

Hi,

this seems a bit unnecessary to rebuild OSDs just to get them managed. If you apply a spec file that targets your hosts/OSDs, they will appear as managed. So when you would need to replace a drive, you could already utilize the orchestrator to remove and zap the drive. That works just fine. How to get out of your current situation is not entirely clear to me yet. I’ll reread your post tomorrow.

Regards, Eugen

Zitat von Bob Gibson <rjg@oicr.on.ca>:

Hi,

We recently converted a legacy cluster running Quincy v17.2.7 to cephadm. The conversion went smoothly and left all osds unmanaged by the orchestrator as expected. We’re now in the process of converting the osds to be managed by the orchestrator. We successfully converted a few of them, but then the orchestrator somehow got confused. `ceph health detail` reports a “stray daemon” for the osd we’re trying to convert, and the orchestrator is unable to refresh its device list so it doesn’t see any available devices.

From the perspective of the osd node, the osd has been wiped and is ready to be reinstalled. We’ve also rebooted the node for good measure. `ceph osd tree` shows that the osd has been destroyed, but the orchestrator won’t reinstall it because it thinks the device is still active. The orchestrator device information is stale, but we’re unable to refresh it. The usual recommended workaround of failing over the mgr hasn’t helped. We’ve also tried `ceph orch device ls —refresh` to no avail. In fact after running that command subsequent runs of `ceph orch device ls` produce no output until the mgr is failed over again.

Is there a way to force the orchestrator to refresh its list of devices when in this state? If not, can anyone offer any suggestions on how to fix this problem?

Cheers, /rjg

P.S. Some additional information in case it’s helpful...

We’re using the following command to replace existing devices so that they’re managed by the orchestrator:

``` ceph orch osd rm <osd> --replace —zap ```

and we’re currently stuck on osd 88.

``` ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.88 on host ceph-osd31 not managed by cephadm ```

`ceph osd tree` shows that the osd has been destroyed and is ready to be replaced:

``` ceph osd tree-from ceph-osd31 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -46 34.93088 host ceph-osd31 84 ssd 3.49309 osd.84 up 1.00000 1.00000 85 ssd 3.49309 osd.85 up 1.00000 1.00000 86 ssd 3.49309 osd.86 up 1.00000 1.00000 87 ssd 3.49309 osd.87 up 1.00000 1.00000 88 ssd 3.49309 osd.88 destroyed 0 1.00000 89 ssd 3.49309 osd.89 up 1.00000 1.00000 90 ssd 3.49309 osd.90 up 1.00000 1.00000 91 ssd 3.49309 osd.91 up 1.00000 1.00000 92 ssd 3.49309 osd.92 up 1.00000 1.00000 93 ssd 3.49309 osd.93 up 1.00000 1.00000 ```

The cephadm log shows a claim on node `ceph-osd31` for that osd:

``` 2024-09-25T14:15:45.699348-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims -> {'ceph-osd31': ['88']} 2024-09-25T14:15:45.699534-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims for drivegroup ceph-osd31 -> {'ceph-osd31': ['88']} ```

`ceph orch device ls` shows that the device list isn’t refreshing:

``` ceph orch device ls ceph-osd31 HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS ceph-osd31 /dev/sdc ssd INTEL_SSDSC2KG038T8_PHYG039603PE3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdd ssd INTEL_SSDSC2KG038T8_PHYG039600AY3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sde ssd INTEL_SSDSC2KG038T8_PHYG039600CW3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdf ssd INTEL_SSDSC2KG038T8_PHYG039600CM3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdg ssd INTEL_SSDSC2KG038T8_PHYG039600UB3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdh ssd INTEL_SSDSC2KG038T8_PHYG039603753P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdi ssd INTEL_SSDSC2KG038T8_PHYG039603R63P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdj ssd INTEL_SSDSC2KG038TZ_PHYJ4011032M3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdk ssd INTEL_SSDSC2KG038TZ_PHYJ3234010J3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdl ssd INTEL_SSDSC2KG038T8_PHYG039603NS3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ```

`ceph node ls` thinks the osd still exists

``` ceph node ls osd | jq -r '."ceph-osd31"' [ 84, 85, 86, 87, 88, <— this shouldn’t exist 89, 90, 91, 92, 93 ] ```

Each osd node has 10x 3.8 TB ssd drives for osds. On `ceph-osd31`, cephadm doesn’t see osd.88 as expected:

``` cephadm ls --no-detail [ { "style": "cephadm:v1", "name": "osd.93", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.93" }, { "style": "cephadm:v1", "name": "osd.85", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.85" }, { "style": "cephadm:v1", "name": "osd.90", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.90" }, { "style": "cephadm:v1", "name": "osd.92", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.92" }, { "style": "cephadm:v1", "name": "osd.89", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.89" }, { "style": "cephadm:v1", "name": "osd.87", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.87" }, { "style": "cephadm:v1", "name": "osd.86", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.86" }, { "style": "cephadm:v1", "name": "osd.84", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.84" }, { "style": "cephadm:v1", "name": "osd.91", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.91" } ] ```

`lsblk` shows that `/dev/sdg` has been wiped.

``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 223.6G 0 disk |-sda1 8:1 0 94M 0 part `-sda2 8:2 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdb 8:16 0 223.6G 0 disk |-sdb1 8:17 0 94M 0 part `-sdb2 8:18 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdc 8:32 1 3.5T 0 disk `-ceph--03782b4c--9faa--49f5--b554--98e7b8515834-osd--block--ba272724--daa6--45f5--9f69--789cc0bda077 253:3 0 3.5T lvm `-keCkP2-o6h8-jKkw-RKiD-UBFf-A8EL-JDJGPR 253:9 0 3.5T 0 crypt sdd 8:48 1 3.5T 0 disk `-ceph--c07907d8--4a75--4ba3--b5e1--2ebf49ecbdf6-osd--block--58d1d50d--6228--4e6f--9a52--2a305ba00700 253:7 0 3.5T lvm `-WB8Mxn-qCHI-4T01-imiG-hNBR-by60-YuxgfD 253:11 0 3.5T 0 crypt sde 8:64 1 3.5T 0 disk `-ceph--6f9d4df4--7ce6--44a4--a7b1--62c85af8cfe0-osd--block--aabcb30d--0084--490a--969b--78f7af6e94da 253:8 0 3.5T lvm `-g9qErH-vTXY-JQbs-eh61-W0Mn-TAV8-gof4zy 253:12 0 3.5T 0 crypt sdf 8:80 1 3.5T 0 disk `-ceph--d6b728f8--e365--46db--b30f--6c00805c752b-osd--block--88426db7--2322--4807--ac2e--b49929e170d6 253:6 0 3.5T lvm `-LNG2gB-pa0w-gl2v-hVQ3-6qTd-aXsR-Lenri3 253:10 0 3.5T 0 crypt sdg 8:96 1 3.5T 0 disk sdh 8:112 1 3.5T 0 disk `-ceph--de2cfee6--8e0a--4aa0--9e6b--90c09025768c-osd--block--a3b86251--2799--4243--a857--f218fa90f29a 253:2 0 3.5T lvm sdi 8:128 1 3.5T 0 disk `-ceph--30dee450--0fdd--46ea--9eec--6a4c7706df9c-osd--block--bfc090db--dde4--47dd--a1c9--1cd838ea43b3 253:4 0 3.5T lvm sdj 8:144 1 3.5T 0 disk `-ceph--78febcf5--43f4--4820--8dc7--0f6c22816c9f-osd--block--da1e69c7--6427--4562--8290--90bcb9526747 253:0 0 3.5T lvm sdk 8:160 1 3.5T 0 disk `-ceph--fe210281--b1f5--4d5e--9ab0--2f226612af00-osd--block--6bb9f308--e853--4303--83ea--553c3a3513e1 253:1 0 3.5T lvm sdl 8:176 1 3.5T 0 disk `-ceph--9f21c916--f211--4d1b--8214--6ad1cecac810-osd--block--572d850c--c201--4af4--ac42--0ed2a6ed73ed 253:5 0 3.5T lvm ```

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Glad to hear it worked out for you! Zitat von Bob Gibson <rjg@oicr.on.ca>:

I’ve been away on vacation and just got back to this. I’m happy to report that manually recreating the OSD with ceph-volume and then adopting it with cephadm fixed the problem.

Thanks again for your help Eugen!

Cheers, /rjg

...

On Sep 29, 2024, at 10:40 AM, Eugen Block <eblock@nde.ag> wrote:

EXTERNAL EMAIL | USE CAUTION

Okay, apparently this is not what I was facing. I see two other options right now. The first would be to purge osd.88 from the crush tree entirely. The second approach would be to create an osd manually with ceph-volume, not cephadm ceph-volume, to create a legacy osd (you'd get warnings about a stray daemon). If that works, adopt the osd with cephadm. I don't have a better idea right now.

Zitat von Bob Gibson <rjg@oicr.on.ca>:

...

Here are the contents from the same directory on our osd node:

ceph-osd31.prod.os:/var/lib/ceph/9b3b3539-59a9-4338-8bab-3badfab6e855# ls -l total 412 -rw-r--r-- 1 root root 366903 Sep 14 14:53 cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b drwx------ 3 167 167 4096 Sep 14 15:01 crash drwxr-xr-x 12 root root 4096 Sep 15 12:06 custom_config_files drw-rw---- 2 root root 4096 Sep 23 17:00 home drwx------ 2 167 167 4096 Sep 26 12:47 osd.84 drwx------ 2 167 167 4096 Sep 26 12:47 osd.85 drwx------ 2 167 167 4096 Sep 26 12:47 osd.86 drwx------ 2 167 167 4096 Sep 26 12:47 osd.87 drwx------ 2 167 167 4096 Sep 26 12:47 osd.89 drwx------ 2 167 167 4096 Sep 26 12:47 osd.90 drwx------ 2 167 167 4096 Sep 26 12:47 osd.91 drwx------ 2 167 167 4096 Sep 26 12:47 osd.92 drwx------ 2 167 167 4096 Sep 26 12:47 osd.93 drwx------ 6 root root 4096 Sep 23 15:59 removed

In our case the osd.88 directory is under the subdirectory named “removed”, the same as the other odds which have been converted.

ceph-osd31.prod.os:/var/lib/ceph/9b3b3539-59a9-4338-8bab-3badfab6e855# ls -l removed/osd.88_2024-09-23T19\:59\:42.162302Z/ total 64 lrwxrwxrwx 1 167 167 93 Sep 15 12:10 block -> /dev/ceph-2a13ec6a-a5f0-4773-8254-c38b915c824a/osd-block-7f8f9778-5ae2-47c1-bd03-a92a3a7a1db1 -rw------- 1 167 167 37 Sep 15 12:10 ceph_fsid -rw------- 1 167 167 259 Sep 14 15:14 config -rw------- 1 167 167 37 Sep 15 12:10 fsid -rw------- 1 167 167 56 Sep 15 12:10 keyring -rw------- 1 167 167 6 Sep 15 12:10 ready -rw------- 1 167 167 3 Sep 14 11:11 require_osd_release -rw------- 1 167 167 10 Sep 15 12:10 type -rw------- 1 167 167 38 Sep 14 15:14 unit.configured -rw------- 1 167 167 48 Sep 14 15:14 unit.created -rw------- 1 167 167 26 Sep 14 15:06 unit.image -rw------- 1 167 167 76 Sep 14 15:06 unit.meta -rw------- 1 167 167 1527 Sep 14 15:06 unit.poststop -rw------- 1 167 167 2586 Sep 14 15:06 unit.run -rw------- 1 167 167 334 Sep 14 15:06 unit.stop -rw------- 1 167 167 3 Sep 15 12:10 whoami

On Sep 27, 2024, at 9:30 AM, Eugen Block <eblock@nde.ag> wrote:

EXTERNAL EMAIL | USE CAUTION

Oh interesting, I just got into the same situation (I believe) on a test cluster:

host1:~ # ceph orch ps | grep unknown osd.1 host6 stopped 72s ago 36m - 4096M <unknown> <unknown> <unknown> osd.13 host6 error 72s ago 36m - 4096M <unknown> <unknown> <unknown>

I still had the remainders on the filesystem:

host6:~ # ll /var/lib/ceph/543967bc-e586-32b8-bd2c-2d8b8b168f02/osd.1 insgesamt 68 lrwxrwxrwx 1 ceph ceph 111 27. Sep 14:43 block -> /dev/mapper/ceph--0e90997f--456e--4a9b--a8f9--a6f1038c1216-osd--block--81e7f32a--a728--4848--b14d--0b86bb7e1c69 lrwxrwxrwx 1 ceph ceph 108 27. Sep 14:43 block.db -> /dev/mapper/ceph--9ea6e95f--ad43--4e40--8920--2e772b2efa2f-osd--db--f9c57ec1--77c8--4d9a--85df--1dc053a24000

I just removed those two directories to clear the warning, now my orchestrator can deploy OSDs again on that node.

Hope that helps!

Zitat von Eugen Block <eblock@nde.ag>:

Right, if you need encryption, a rebuild is required. Your procedure has already worked 4 times, so I'd say nothing seems wrong with that per se. Regarding the stuck device list, do you see the mgr logging anything suspicious? Especially when you say that it only returns output after a failover. Those two osd specs are not conflicting since the first is "unmanaged" after adoption. Is there something in 'ceph orch osd rm status'? Can you run 'cephadm ceph-volume inventory' locally on that node? Do you see any hints in the node's syslog? Maybe try a reboot or something?

Zitat von Bob Gibson <rjg@oicr.on.ca>:

Thanks for your reply Eugen. I’m fairly new to cephadm so I wasn’t aware that we could manage the drives without rebuilding them. However, we thought we’d take advantage of this opportunity to also encrypt the drives, and that does require a rebuild.

I have a theory on why the orchestrator is confused. I want to create an osd service for each osd node so I can manage drives on a per-node basis.

I started by creating a spec for the first node:

service_type: osd service_id: ceph-osd31 placement: hosts: - ceph-osd31 spec: data_devices: rotational: 0 size: '3TB:' encrypted: true filter_logic: AND objectstore: bluestore

But I also see a default spec, “osd”, which has placement set to “unmanaged”.

`ceph orch ls osd —export` shows the following:

service_type: osd service_name: osd unmanaged: true spec: filter_logic: AND objectstore: bluestore --- service_type: osd service_id: ceph-osd31 service_name: osd.ceph-osd31 placement: hosts: - ceph-osd31 spec: data_devices: rotational: 0 size: '3TB:' encrypted: true filter_logic: AND objectstore: bluestore

`ceph orch ls osd` shows that I was able to convert 4 drives using my spec:

NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd 95 10m ago - <unmanaged> osd.ceph-osd31 4 10m ago 43m ceph-osd31

Despite being able to convert 4 drives, I’m wondering if these specs are conflicting with one another, and that has confused the orchestrator. If so, how do I safely get from where I am now to where I want to be? :-)

Cheers, /rjg

On Sep 26, 2024, at 3:31 PM, Eugen Block <eblock@nde.ag> wrote:

EXTERNAL EMAIL | USE CAUTION

Hi,

this seems a bit unnecessary to rebuild OSDs just to get them managed. If you apply a spec file that targets your hosts/OSDs, they will appear as managed. So when you would need to replace a drive, you could already utilize the orchestrator to remove and zap the drive. That works just fine. How to get out of your current situation is not entirely clear to me yet. I’ll reread your post tomorrow.

Regards, Eugen

Zitat von Bob Gibson <rjg@oicr.on.ca>:

Hi,

We recently converted a legacy cluster running Quincy v17.2.7 to cephadm. The conversion went smoothly and left all osds unmanaged by the orchestrator as expected. We’re now in the process of converting the osds to be managed by the orchestrator. We successfully converted a few of them, but then the orchestrator somehow got confused. `ceph health detail` reports a “stray daemon” for the osd we’re trying to convert, and the orchestrator is unable to refresh its device list so it doesn’t see any available devices.

From the perspective of the osd node, the osd has been wiped and is ready to be reinstalled. We’ve also rebooted the node for good measure. `ceph osd tree` shows that the osd has been destroyed, but the orchestrator won’t reinstall it because it thinks the device is still active. The orchestrator device information is stale, but we’re unable to refresh it. The usual recommended workaround of failing over the mgr hasn’t helped. We’ve also tried `ceph orch device ls —refresh` to no avail. In fact after running that command subsequent runs of `ceph orch device ls` produce no output until the mgr is failed over again.

Is there a way to force the orchestrator to refresh its list of devices when in this state? If not, can anyone offer any suggestions on how to fix this problem?

Cheers, /rjg

P.S. Some additional information in case it’s helpful...

We’re using the following command to replace existing devices so that they’re managed by the orchestrator:

``` ceph orch osd rm <osd> --replace —zap ```

and we’re currently stuck on osd 88.

``` ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.88 on host ceph-osd31 not managed by cephadm ```

`ceph osd tree` shows that the osd has been destroyed and is ready to be replaced:

``` ceph osd tree-from ceph-osd31 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -46 34.93088 host ceph-osd31 84 ssd 3.49309 osd.84 up 1.00000 1.00000 85 ssd 3.49309 osd.85 up 1.00000 1.00000 86 ssd 3.49309 osd.86 up 1.00000 1.00000 87 ssd 3.49309 osd.87 up 1.00000 1.00000 88 ssd 3.49309 osd.88 destroyed 0 1.00000 89 ssd 3.49309 osd.89 up 1.00000 1.00000 90 ssd 3.49309 osd.90 up 1.00000 1.00000 91 ssd 3.49309 osd.91 up 1.00000 1.00000 92 ssd 3.49309 osd.92 up 1.00000 1.00000 93 ssd 3.49309 osd.93 up 1.00000 1.00000 ```

The cephadm log shows a claim on node `ceph-osd31` for that osd:

``` 2024-09-25T14:15:45.699348-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims -> {'ceph-osd31': ['88']} 2024-09-25T14:15:45.699534-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims for drivegroup ceph-osd31 -> {'ceph-osd31': ['88']} ```

`ceph orch device ls` shows that the device list isn’t refreshing:

``` ceph orch device ls ceph-osd31 HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS ceph-osd31 /dev/sdc ssd INTEL_SSDSC2KG038T8_PHYG039603PE3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdd ssd INTEL_SSDSC2KG038T8_PHYG039600AY3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sde ssd INTEL_SSDSC2KG038T8_PHYG039600CW3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdf ssd INTEL_SSDSC2KG038T8_PHYG039600CM3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdg ssd INTEL_SSDSC2KG038T8_PHYG039600UB3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdh ssd INTEL_SSDSC2KG038T8_PHYG039603753P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdi ssd INTEL_SSDSC2KG038T8_PHYG039603R63P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdj ssd INTEL_SSDSC2KG038TZ_PHYJ4011032M3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdk ssd INTEL_SSDSC2KG038TZ_PHYJ3234010J3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdl ssd INTEL_SSDSC2KG038T8_PHYG039603NS3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ```

`ceph node ls` thinks the osd still exists

``` ceph node ls osd | jq -r '."ceph-osd31"' [ 84, 85, 86, 87, 88, <— this shouldn’t exist 89, 90, 91, 92, 93 ] ```

Each osd node has 10x 3.8 TB ssd drives for osds. On `ceph-osd31`, cephadm doesn’t see osd.88 as expected:

``` cephadm ls --no-detail [ { "style": "cephadm:v1", "name": "osd.93", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.93" }, { "style": "cephadm:v1", "name": "osd.85", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.85" }, { "style": "cephadm:v1", "name": "osd.90", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.90" }, { "style": "cephadm:v1", "name": "osd.92", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.92" }, { "style": "cephadm:v1", "name": "osd.89", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.89" }, { "style": "cephadm:v1", "name": "osd.87", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.87" }, { "style": "cephadm:v1", "name": "osd.86", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.86" }, { "style": "cephadm:v1", "name": "osd.84", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.84" }, { "style": "cephadm:v1", "name": "osd.91", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.91" } ] ```

`lsblk` shows that `/dev/sdg` has been wiped.

``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 223.6G 0 disk |-sda1 8:1 0 94M 0 part `-sda2 8:2 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdb 8:16 0 223.6G 0 disk |-sdb1 8:17 0 94M 0 part `-sdb2 8:18 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdc 8:32 1 3.5T 0 disk `-ceph--03782b4c--9faa--49f5--b554--98e7b8515834-osd--block--ba272724--daa6--45f5--9f69--789cc0bda077 253:3 3.5T lvm `-keCkP2-o6h8-jKkw-RKiD-UBFf-A8EL-JDJGPR 253:9 0 3.5T 0 crypt sdd 8:48 1 3.5T 0 disk `-ceph--c07907d8--4a75--4ba3--b5e1--2ebf49ecbdf6-osd--block--58d1d50d--6228--4e6f--9a52--2a305ba00700 253:7 3.5T lvm `-WB8Mxn-qCHI-4T01-imiG-hNBR-by60-YuxgfD 253:11 0 3.5T 0 crypt sde 8:64 1 3.5T 0 disk `-ceph--6f9d4df4--7ce6--44a4--a7b1--62c85af8cfe0-osd--block--aabcb30d--0084--490a--969b--78f7af6e94da 253:8 3.5T lvm `-g9qErH-vTXY-JQbs-eh61-W0Mn-TAV8-gof4zy 253:12 0 3.5T 0 crypt sdf 8:80 1 3.5T 0 disk `-ceph--d6b728f8--e365--46db--b30f--6c00805c752b-osd--block--88426db7--2322--4807--ac2e--b49929e170d6 253:6 3.5T lvm `-LNG2gB-pa0w-gl2v-hVQ3-6qTd-aXsR-Lenri3 253:10 0 3.5T 0 crypt sdg 8:96 1 3.5T 0 disk sdh 8:112 1 3.5T 0 disk `-ceph--de2cfee6--8e0a--4aa0--9e6b--90c09025768c-osd--block--a3b86251--2799--4243--a857--f218fa90f29a 253:2 3.5T lvm sdi 8:128 1 3.5T 0 disk `-ceph--30dee450--0fdd--46ea--9eec--6a4c7706df9c-osd--block--bfc090db--dde4--47dd--a1c9--1cd838ea43b3 253:4 3.5T lvm sdj 8:144 1 3.5T 0 disk `-ceph--78febcf5--43f4--4820--8dc7--0f6c22816c9f-osd--block--da1e69c7--6427--4562--8290--90bcb9526747 253:0 3.5T lvm sdk 8:160 1 3.5T 0 disk `-ceph--fe210281--b1f5--4d5e--9ab0--2f226612af00-osd--block--6bb9f308--e853--4303--83ea--553c3a3513e1 253:1 3.5T lvm sdl 8:176 1 3.5T 0 disk `-ceph--9f21c916--f211--4d1b--8214--6ad1cecac810-osd--block--572d850c--c201--4af4--ac42--0ed2a6ed73ed 253:5 3.5T lvm ```

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Bob, have you tried to restart the active mgr? ( sometimes mgr gets stuck and prevents the orchestrator from working correctly ). Regarding the orchestrator device scan: have a look into the ceph-volume.log on the corresponding host. you will find it under /var/log/ceph/CLUSTER-ID/ceph-volume.log this log is generated by the device scan from the orchestrator. It may also help to have a look at cephadm debug logs - see https://docs.ceph.com/en/latest/cephadm/operations/#watching-cephadm-log-mes... Cheers, tobi Am Mi., 23. Okt. 2024 um 20:15 Uhr schrieb Bob Gibson <rjg@oicr.on.ca>:

Sorry to resurrect this thread, but while I was able to get the cluster healthy again by manually creating the osd, I'm still unable to manage osds using the orchestrator.

The orchestrator is generally working, but It appears to be unable to scan devices. Immediately after failing out the mgr `ceph orch device ls` will display device status from >4 weeks ago, which was when we converted the cluster to be managed by cephadm. Eventually the orchestrator will attempt to refresh its device status. At this point `ceph orch device ls` stops displaying any output at all. I can reproduce this state almost immediately if I run `ceph orch device ls —refresh` to force an immediate refresh. The mgr log shows events like the following just before `ceph orch device ls` stops reporting output (one event for every osd node in the cluster):

"Detected new or changed devices on ceph-osd31”

Here are the osd services in play:

# ceph orch ls osd NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd 95 8m ago - <unmanaged> osd.ceph-osd31 4 8m ago 6d ceph-osd31

# ceph orch ls osd --export service_type: osd service_name: osd unmanaged: true spec: filter_logic: AND objectstore: bluestore --- service_type: osd service_id: ceph-osd31 service_name: osd.ceph-osd31 placement: hosts: - ceph-osd31 spec: data_devices: rotational: 0 size: '3TB:' encrypted: true filter_logic: AND objectstore: bluestore

I tried deleting the default “osd” service in case it was somehow conflicting with my per-node spec, but it looks like that’s not allowed, so I assume any custom osd service specs override the unmanaged default.

# ceph orch rm osd Invalid service 'osd'. Use 'ceph orch ls' to list available services.

My hunch is that some persistent state is corrupted, or there’s something else preventing the orchestrator from successfully refreshing its device status, but I don’t know how to troubleshoot this. Any ideas?

Cheers, /rjg

P.S. @Eugen: When I first started this thread you said it was unnecessary to destroy an osd to convert it from unmanaged to managed. Can you explain how this is done? Although we want to recreate the osds to enable encryption, it would save time, and unnecessary wear on the SSDs, while troubleshooting.

On Oct 16, 2024, at 2:45 PM, Eugen Block <eblock@nde.ag> wrote:

EXTERNAL EMAIL | USE CAUTION

Glad to hear it worked out for you!

Zitat von Bob Gibson <rjg@oicr.on.ca>:

I’ve been away on vacation and just got back to this. I’m happy to report that manually recreating the OSD with ceph-volume and then adopting it with cephadm fixed the problem.

Thanks again for your help Eugen!

Cheers, /rjg

On Sep 29, 2024, at 10:40 AM, Eugen Block <eblock@nde.ag> wrote:

EXTERNAL EMAIL | USE CAUTION

Okay, apparently this is not what I was facing. I see two other options right now. The first would be to purge osd.88 from the crush tree entirely. The second approach would be to create an osd manually with ceph-volume, not cephadm ceph-volume, to create a legacy osd (you'd get warnings about a stray daemon). If that works, adopt the osd with cephadm. I don't have a better idea right now.

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

-- Best Regards, Tobias Fischer Head of Ceph Clyso GmbH p: +49 89 2152527 41 a: Hohenzollernstraße 27 | 80801 München | Germany w: https://clyso.com | e: tobias.fischer@clyso.com We are hiring: https://www.clyso.com/jobs/ --- Geschäftsführer: Dipl. Inf. (FH) Joachim Kraftmayer Unternehmenssitz: Utting am Ammersee Handelsregister beim Amtsgericht: Augsburg Handelsregister-Nummer: HRB 25866 USt. ID-Nr.: DE275430677

Hi, there are a couple of ways to get your OSDs into "managed" state. You can't remove the "unmanaged" service because it's unmanaged. ;-) Just an example from a test cluster where I adopted three OSDs, now they're unmanaged as expected: soc9-ceph:~ # ceph orch ls osd NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd 3 30s ago - <unmanaged> Now I apply a spec with the same "service_name": soc9-ceph:~ # cat osd-unmanaged.yaml service_type: osd service_name: osd unmanaged: false placement: label: osd spec: data_devices: rotational: 1 size: 10G db_devices: rotational: 0 size: 15G filter_logic: AND objectstore: bluestore I explicitly used the "unmanaged: false" statement to show that I want this service_name to be managed: soc9-ceph:~ # ceph orch apply -i osd-unmanaged.yaml Scheduled osd update... soc9-ceph:~ # ceph orch ls osd NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd 3 2m ago 8s label:osd Now all my OSDs are managed. So in this case, this is already my target drivegroup, should I need to replace a drive, this spec would be applied. I would recommend to create a spec that matches your requirements (for example encrypted) so your OSDs are managed, then you could recreate them one by one. Regarding the failing refresh I don't have other ideas than inspecting all the logs, maybe turn on debug mode as Tobi already suggested. I haven't seen that in any cluster yet, so it's hard to tell. Is any of the hosts shown as "offline" in 'ceph orch host ls' output? I wonder if it could be a connection issue, MTU mismatch, apparmor or firewall... Zitat von Bob Gibson <rjg@oicr.on.ca>:

Sorry to resurrect this thread, but while I was able to get the cluster healthy again by manually creating the osd, I'm still unable to manage osds using the orchestrator.

The orchestrator is generally working, but It appears to be unable to scan devices. Immediately after failing out the mgr `ceph orch device ls` will display device status from >4 weeks ago, which was when we converted the cluster to be managed by cephadm. Eventually the orchestrator will attempt to refresh its device status. At this point `ceph orch device ls` stops displaying any output at all. I can reproduce this state almost immediately if I run `ceph orch device ls —refresh` to force an immediate refresh. The mgr log shows events like the following just before `ceph orch device ls` stops reporting output (one event for every osd node in the cluster):

"Detected new or changed devices on ceph-osd31”

Here are the osd services in play:

# ceph orch ls osd NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd 95 8m ago - <unmanaged> osd.ceph-osd31 4 8m ago 6d ceph-osd31

# ceph orch ls osd --export service_type: osd service_name: osd unmanaged: true spec: filter_logic: AND objectstore: bluestore --- service_type: osd service_id: ceph-osd31 service_name: osd.ceph-osd31 placement: hosts: - ceph-osd31 spec: data_devices: rotational: 0 size: '3TB:' encrypted: true filter_logic: AND objectstore: bluestore

I tried deleting the default “osd” service in case it was somehow conflicting with my per-node spec, but it looks like that’s not allowed, so I assume any custom osd service specs override the unmanaged default.

# ceph orch rm osd Invalid service 'osd'. Use 'ceph orch ls' to list available services.

My hunch is that some persistent state is corrupted, or there’s something else preventing the orchestrator from successfully refreshing its device status, but I don’t know how to troubleshoot this. Any ideas?

Cheers, /rjg

P.S. @Eugen: When I first started this thread you said it was unnecessary to destroy an osd to convert it from unmanaged to managed. Can you explain how this is done? Although we want to recreate the osds to enable encryption, it would save time, and unnecessary wear on the SSDs, while troubleshooting.

On Oct 16, 2024, at 2:45 PM, Eugen Block <eblock@nde.ag> wrote:

EXTERNAL EMAIL | USE CAUTION

Glad to hear it worked out for you!

Zitat von Bob Gibson <rjg@oicr.on.ca>:

I’ve been away on vacation and just got back to this. I’m happy to report that manually recreating the OSD with ceph-volume and then adopting it with cephadm fixed the problem.

Thanks again for your help Eugen!

Cheers, /rjg

On Sep 29, 2024, at 10:40 AM, Eugen Block <eblock@nde.ag> wrote:

EXTERNAL EMAIL | USE CAUTION

Okay, apparently this is not what I was facing. I see two other options right now. The first would be to purge osd.88 from the crush tree entirely. The second approach would be to create an osd manually with ceph-volume, not cephadm ceph-volume, to create a legacy osd (you'd get warnings about a stray daemon). If that works, adopt the osd with cephadm. I don't have a better idea right now.

----- Le 23 Oct 24, à 20:14, Bob Gibson rjg@oicr.on.ca a écrit :

Sorry to resurrect this thread, but while I was able to get the cluster healthy again by manually creating the osd, I'm still unable to manage osds using the orchestrator.

The orchestrator is generally working, but It appears to be unable to scan devices. Immediately after failing out the mgr `ceph orch device ls` will display device status from >4 weeks ago, which was when we converted the cluster to be managed by cephadm. Eventually the orchestrator will attempt to refresh its device status. At this point `ceph orch device ls` stops displaying any output at all. I can reproduce this state almost immediately if I run `ceph orch device ls —refresh` to force an immediate refresh. The mgr log shows events like the following just before `ceph orch device ls` stops reporting output (one event for every osd node in the cluster):

"Detected new or changed devices on ceph-osd31”

Here are the osd services in play:

# ceph orch ls osd NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd 95 8m ago - <unmanaged> osd.ceph-osd31 4 8m ago 6d ceph-osd31

# ceph orch ls osd --export service_type: osd service_name: osd unmanaged: true spec: filter_logic: AND objectstore: bluestore --- service_type: osd service_id: ceph-osd31 service_name: osd.ceph-osd31 placement: hosts: - ceph-osd31 spec: data_devices: rotational: 0 size: '3TB:' encrypted: true filter_logic: AND objectstore: bluestore

I tried deleting the default “osd” service in case it was somehow conflicting with my per-node spec, but it looks like that’s not allowed, so I assume any custom osd service specs override the unmanaged default.

# ceph orch rm osd Invalid service 'osd'. Use 'ceph orch ls' to list available services.

Hi Bob, I think this message shows up as this very specific post adoption 'osd' service has already been marked as 'deleted'. Maybe when you ran the command for the first time. The only reason it still shows up on 'ceph orch ls' is that 95 OSDs are still referencing this service in their configuration. Once you'll have edited all OSDs /var/lib/ceph/$(ceph fsid)/osd.xxx/unit.meta files (changed their service_name) and restarted all OSDs (or recreated these 95 OSDs encrypted under another service_name), the 'osd' service will disappear by itself and won't show up anymore on 'ceph orch ls' output. At least this is what I've observed in the past.

My hunch is that some persistent state is corrupted, or there’s something else preventing the orchestrator from successfully refreshing its device status, but I don’t know how to troubleshoot this. Any ideas?

I don't think this is related to the 'osd' service. As suggested by Tobi, enabling cephadm debug will tell you more. Cheers, Frédéric.

Cheers, /rjg

P.S. @Eugen: When I first started this thread you said it was unnecessary to destroy an osd to convert it from unmanaged to managed. Can you explain how this is done? Although we want to recreate the osds to enable encryption, it would save time, and unnecessary wear on the SSDs, while troubleshooting.

On Oct 16, 2024, at 2:45 PM, Eugen Block <eblock@nde.ag> wrote:

EXTERNAL EMAIL | USE CAUTION

Glad to hear it worked out for you!

Zitat von Bob Gibson <rjg@oicr.on.ca>:

I’ve been away on vacation and just got back to this. I’m happy to report that manually recreating the OSD with ceph-volume and then adopting it with cephadm fixed the problem.

Thanks again for your help Eugen!

Cheers, /rjg

On Sep 29, 2024, at 10:40 AM, Eugen Block <eblock@nde.ag> wrote:

EXTERNAL EMAIL | USE CAUTION

Okay, apparently this is not what I was facing. I see two other options right now. The first would be to purge osd.88 from the crush tree entirely. The second approach would be to create an osd manually with ceph-volume, not cephadm ceph-volume, to create a legacy osd (you'd get warnings about a stray daemon). If that works, adopt the osd with cephadm. I don't have a better idea right now.

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

----- Le 25 Oct 24, à 16:31, Bob Gibson rjg@oicr.on.ca a écrit :

HI Frédéric,

...

I think this message shows up as this very specific post adoption 'osd' service has already been marked as 'deleted'. Maybe when you ran the command for the first time. The only reason it still shows up on 'ceph orch ls' is that 95 OSDs are still referencing this service in their configuration.

Once you'll have edited all OSDs /var/lib/ceph/$(ceph fsid)/osd.xxx/unit.meta files (changed their service_name) and restarted all OSDs (or recreated these 95 OSDs encrypted under another service_name), the 'osd' service will disappear by itself and won't show up anymore on 'ceph orch ls' output. At least this is what I've observed in the past.

Yes, as Eugen pointed out, it doesn’t make sense to try to delete an unmanaged service using the orchestrator.

Well actually you **can** delete a service whatever it's status (managed or unmanaged). It works just fine with any osd service other than this specific post adoption 'osd' service. Don't know why. Frédéric.

...

...

My hunch is that some persistent state is corrupted, or there’s something else preventing the orchestrator from successfully refreshing its device status, but I don’t know how to troubleshoot this. Any ideas?

I don't think this is related to the 'osd' service. As suggested by Tobi, enabling cephadm debug will tell you more.

Agreed. I’ll dig through the logs some more today to see if I can spot any problems.

Cheers, /rjg

You're right about deleting the service, of course. I wasn't very clear in my statement, what I actually meant was that it won't be removed entirely until all OSDs report a different spec in their unit.meta file. I forgot to add that info in my last response, that's actually how I've done it several times after adopting a cluster. Thanks for clearing that up! :-) Zitat von Frédéric Nass <frederic.nass@univ-lorraine.fr>:

----- Le 25 Oct 24, à 18:21, Frédéric Nass frederic.nass@univ-lorraine.fr a écrit :

...

----- Le 25 Oct 24, à 16:31, Bob Gibson rjg@oicr.on.ca a écrit :

...

HI Frédéric,

...

I think this message shows up as this very specific post adoption 'osd' service has already been marked as 'deleted'. Maybe when you ran the command for the first time. The only reason it still shows up on 'ceph orch ls' is that 95 OSDs are still referencing this service in their configuration.

Once you'll have edited all OSDs /var/lib/ceph/$(ceph fsid)/osd.xxx/unit.meta files (changed their service_name) and restarted all OSDs (or recreated these 95 OSDs encrypted under another service_name), the 'osd' service will disappear by itself and won't show up anymore on 'ceph orch ls' output. At least this is what I've observed in the past.

Yes, as Eugen pointed out, it doesn’t make sense to try to delete an unmanaged service using the orchestrator.

Well actually you **can** delete a service whatever it's status (managed or unmanaged).

To explain a bit more, see below:

$ ceph orch ls --export osd osd.delete service_type: osd service_id: delete service_name: osd.delete placement: hosts: - test-mom02h01 unmanaged: true spec: data_devices: size: :11G db_devices: size: '12G:' db_slots: 2 filter_logic: AND objectstore: bluestore

This is what you should expect:

$ ceph orch rm osd.delete Error EINVAL: If osd.delete is removed then the following OSDs will remain, --force to proceed anyway host test-mom02h01: osd.11

$ ceph orch rm osd.delete --force <--- ok, let's force it Removed service osd.delete

$ ceph orch ls | grep osd osd.delete 1 95s ago - <unmanaged> <--- still here because used by 1 OSD osd.standard 12 9m ago 8w label:osds

$ ceph orch rm osd.delete Invalid service 'osd.delete'. Use 'ceph orch ls' to list available services. <--- but not for the orchestrator

$ sed -i 's/osd.delete/osd.standard/g' /var/lib/ceph/$(ceph fsid)/osd.11/unit.meta <--- remove this service from osd.11 configuration

$ ceph orch daemon restart osd.11 Scheduled to restart osd.11 on host 'test-mom02h01'

$ ceph orch ls | grep osd osd.standard 13 8m ago 8w label:osds <--- osd.delete service finally gone

The osd.delete service is finally gone right after the last OSD stopped referencing it.

With the very specific post adoption 'osd' service, when you try to delete it, it doesn't complain about existing OSDs referencing it (when it should...) and doesn't require you to use the --force argument. It just deletes the service (that will finally be removed when no more OSDs are using it).

The fact that the 'ceph orch rm' output is not consistent when deleting the 'osd' post adoption service and deleting any other osd service that you create looks like a bug to me.

But anyways, that was just to say you can delete an osd service whatever it's status (managed or unmanaged).

Cheers, Frédéric.

...

It works just fine with any osd service other than this specific post adoption 'osd' service. Don't know why.

Frédéric.

...

...

...

My hunch is that some persistent state is corrupted, or there’s something else preventing the orchestrator from successfully refreshing its device status, but I don’t know how to troubleshoot this. Any ideas?

I don't think this is related to the 'osd' service. As suggested by Tobi, enabling cephadm debug will tell you more.

Agreed. I’ll dig through the logs some more today to see if I can spot any problems.

Cheers, /rjg

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi, I haven't done this in production yet either, but in a test cluster I threw away that config-key and it just gets regenerated. So I suppose one could try that without any bis risk. Just a note, this should also work (get instead of dump): ceph config-key get mgr/cephadm/host.ceph-osd31.devices.0 | jq .devices[].created Zitat von Bob Gibson <rjg@oicr.on.ca>:

I enabled debug logging with `ceph config set mgr mgr/cephadm/log_to_cluster_level debug` and viewed the logs with `ceph -W cephadm --watch-debug`. I can see the orchestrator refreshing the device list, and this is reflected in the `ceph-volume.log` file on the target osd nodes. When I restart the mgr, `ceph orch device ls` reports each device with “5w ago” under the “REFRESHED” column. After the orchestrator attempts to refresh the device list, `ceph orch device ls` stops outputting any data at all until I restart the mgr again.

I discovered that I can query the cached device data using `ceph config-key dump`. On the problematic cluster, the `created` attribute is stale, e.g.

ceph config-key dump | jq -r .'"mgr/cephadm/host.ceph-osd31.devices.0"' | jq .devices[].created "2024-09-23T17:56:44.914535Z" "2024-09-23T17:56:44.914569Z" "2024-09-23T17:56:44.914591Z" "2024-09-23T17:56:44.914612Z" "2024-09-23T17:56:44.914632Z" "2024-09-23T17:56:44.914652Z" "2024-09-23T17:56:44.914672Z" "2024-09-23T17:56:44.914692Z" "2024-09-23T17:56:44.914711Z" "2024-09-23T17:56:44.914732Z"

whereas on working clusters the `created` attribute is set to the time the device information was last cached, e.g.

ceph config-key dump | jq -r .'"mgr/cephadm/host.ceph-osd1.devices.0"' | jq .devices[].created "2024-10-28T21:49:29.510593Z" "2024-10-28T21:49:29.510635Z" "2024-10-28T21:49:29.510657Z" "2024-10-28T21:49:29.510678Z"

It appears that the orchestrator is polling the devices but failing to update the cache for some reason. It would be interesting to see what happens if I removed one of these device entries from the cache, but the cluster is in production so I’m hesitant to poke at it.

We have a maintenance window scheduled in December which will provide an opportunity to perform a complete restart of the cluster. Hopefully that will clean things up. In the meantime, I’ve set all devices to be unmanaged, and the cluster is otherwise healthy, so unless anyone has any other ideas to offer I guess I’ll just leave things as-is until the maintenance window.

Cheers, /rjg

On Oct 25, 2024, at 10:31 AM, Bob Gibson <rjg@oicr.on.ca> wrote:

[…] My hunch is that some persistent state is corrupted, or there’s something else preventing the orchestrator from successfully refreshing its device status, but I don’t know how to troubleshoot this. Any ideas?

I don't think this is related to the 'osd' service. As suggested by Tobi, enabling cephadm debug will tell you more.

Agreed. I’ll dig through the logs some more today to see if I can spot any problems.

Cheers, /rjg

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Eugen, Thanks again for taking the time to help us with this. Here are answers to your questions: Nothing stands out from the mgr logs. Even when `ceph orch device ls` stops reporting, it still shows a claim on the osd in the logs when I run it: Sep 27 09:39:24 ceph-mon3 bash[476409]: debug 2024-09-27T13:39:24.731+0000 7fd4dc6fa700 0 [cephadm INFO root] Found osd claims -> {'ceph-osd31': ['88']} Sep 27 09:39:24 ceph-mon3 bash[476409]: debug 2024-09-27T13:39:24.731+0000 7fd4dc6fa700 0 log_channel(cephadm) log [INF] : Found osd claims -> {'ceph-osd31': ['88']} Sep 27 09:39:24 ceph-mon3 bash[476409]: debug 2024-09-27T13:39:24.731+0000 7fd4dc6fa700 0 [cephadm INFO cephadm.services.osd] Found osd claims for drivegroup ceph-osd31 -> {'ceph-osd31': ['88']} Sep 27 09:39:24 ceph-mon3 bash[476409]: debug 2024-09-27T13:39:24.731+0000 7fd4dc6fa700 0 log_channel(cephadm) log [INF] : Found osd claims for drivegroup ceph-osd31 -> {'ceph-osd31': ['88’]} Here’s a sample of mgr logs right after a mgr failover (I’ve filtered out some noise from pgmap, prometheus, pg_autoscaler, balancer, and progress): Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.006+0000 7f8d2f15c700 1 mgr handle_mgr_map Activating! Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.006+0000 7f8d2f15c700 1 mgr handle_mgr_map I am now activating Sep 27 09:55:18 ceph-mon3 bash[476409]: [27/Sep/2024:13:55:18] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.102+0000 7f8c4baa7700 0 [cephadm DEBUG root] setting log level based on debug_mgr: INFO (2/5) Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.202+0000 7f8c4baa7700 1 mgr load Constructed class from module: cephadm Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.206+0000 7f8c4baa7700 0 [crash DEBUG root] setting log level based on debug_mgr: INFO (2/5) Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.206+0000 7f8c4baa7700 1 mgr load Constructed class from module: crash Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.222+0000 7f8c4baa7700 0 [devicehealth DEBUG root] setting log level based on debug_mgr: INFO (2/5) Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.222+0000 7f8c4baa7700 1 mgr load Constructed class from module: devicehealth Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.222+0000 7f8c3e28c700 0 [devicehealth INFO root] Starting Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.238+0000 7f8c4baa7700 0 [orchestrator DEBUG root] setting log level based on debug_mgr: INFO (2/5) Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.242+0000 7f8c4baa7700 1 mgr load Constructed class from module: orchestrator Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.318+0000 7f8c4baa7700 0 [rbd_support DEBUG root] setting log level based on debug_mgr: INFO (2/5) Sep 27 09:55:18 ceph-mon3 bash[476409]: [27/Sep/2024:13:55:18] ENGINE Bus STARTING Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.346+0000 7f8c31272700 0 [rbd_support INFO root] recovery thread starting Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.346+0000 7f8c31272700 0 [rbd_support INFO root] starting setup Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.354+0000 7f8c4baa7700 1 mgr load Constructed class from module: rbd_support Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.358+0000 7f8c31272700 0 [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.370+0000 7f8c31272700 0 [rbd_support INFO root] load_schedules: rbd, start_after= Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.374+0000 7f8c4baa7700 0 [status DEBUG root] setting log level based on debug_mgr: INFO (2/5) Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.374+0000 7f8c4baa7700 1 mgr load Constructed class from module: status Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.378+0000 7f8c4baa7700 0 [telemetry DEBUG root] setting log level based on debug_mgr: INFO (2/5) Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.378+0000 7f8c4baa7700 1 mgr load Constructed class from module: telemetry Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.382+0000 7f8c4baa7700 0 [volumes DEBUG root] setting log level based on debug_mgr: INFO (2/5) Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.386+0000 7f8c31272700 0 [rbd_support INFO root] load_schedules: images, start_after= Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.390+0000 7f8c31272700 0 [rbd_support INFO root] load_schedules: volumes, start_after= Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.394+0000 7f8c31272700 0 [rbd_support INFO root] load_schedules: vms, start_after= Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.398+0000 7f8c31272700 0 [rbd_support INFO root] load_schedules: backups, start_after= Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.402+0000 7f8c21252700 0 [rbd_support INFO root] MirrorSnapshotScheduleHandler: starting Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.402+0000 7f8c1fa4f700 0 [rbd_support INFO root] PerfHandler: starting Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.422+0000 7f8c31272700 0 [rbd_support INFO root] load_task_task: rbd, start_after= Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.430+0000 7f8c4baa7700 1 mgr load Constructed class from module: volumes Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.434+0000 7f8c31272700 0 [rbd_support INFO root] load_task_task: images, start_after= Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.446+0000 7f8c31272700 0 [rbd_support INFO root] load_task_task: volumes, start_after= Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.458+0000 7f8c31272700 0 [rbd_support INFO root] load_task_task: vms, start_after= Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.870+0000 7f8c3e28c700 0 [devicehealth INFO root] Check health Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.874+0000 7f8c31272700 0 [rbd_support INFO root] load_task_task: backups, start_after= Sep 27 09:55:18 ceph-mon3 bash[476409]: [27/Sep/2024:13:55:18] ENGINE Serving on http://:::9283 Sep 27 09:55:18 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:18.914+0000 7f8c179bf700 0 [rbd_support INFO root] TaskHandler: starting Sep 27 09:55:19 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:19.106+0000 7f8c31272700 0 [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules Sep 27 09:55:19 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:19.122+0000 7f8c31272700 0 [rbd_support INFO root] load_schedules: rbd, start_after= Sep 27 09:55:19 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:19.126+0000 7f8c31272700 0 [rbd_support INFO root] load_schedules: images, start_after= Sep 27 09:55:19 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:19.134+0000 7f8c31272700 0 [rbd_support INFO root] load_schedules: volumes, start_after= Sep 27 09:55:19 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:19.138+0000 7f8c31272700 0 [rbd_support INFO root] load_schedules: vms, start_after= Sep 27 09:55:19 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:19.146+0000 7f8c31272700 0 [rbd_support INFO root] load_schedules: backups, start_after= Sep 27 09:55:19 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:19.150+0000 7f8c171be700 0 [rbd_support INFO root] TrashPurgeScheduleHandler: starting Sep 27 09:55:19 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:19.150+0000 7f8c31272700 0 [rbd_support INFO root] setup complete Sep 27 09:55:19 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:19.638+0000 7f8c43a97700 0 [cephadm INFO cherrypy.error] [27/Sep/2024:13:55:19] ENGINE Bus STARTING Sep 27 09:55:19 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:19.638+0000 7f8c43a97700 0 log_channel(cephadm) log [INF] : [27/Sep/2024:13:55:19] ENGINE Bus STARTING Sep 27 09:55:19 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:19.770+0000 7f8c43a97700 0 [cephadm INFO cherrypy.error] [27/Sep/2024:13:55:19] ENGINE Serving on https://10.5.74.23:7150 Sep 27 09:55:19 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:19.770+0000 7f8c43a97700 0 log_channel(cephadm) log [INF] : [27/Sep/2024:13:55:19] ENGINE Serving on https://10.5.74.23:7150 Sep 27 09:55:19 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:19.770+0000 7f8c43a97700 0 [cephadm INFO cherrypy.error] [27/Sep/2024:13:55:19] ENGINE Bus STARTED Sep 27 09:55:19 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:19.770+0000 7f8c43a97700 0 log_channel(cephadm) log [INF] : [27/Sep/2024:13:55:19] ENGINE Bus STARTED Sep 27 09:55:28 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:28.030+0000 7f8c4b2a6700 2 mgr.server handle_open ignoring open from mgr.ceph-mon1 10.5.74.21:0/3328700921; not ready for session (expect reconnect) Sep 27 09:55:29 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:29.030+0000 7f8c4b2a6700 2 mgr.server handle_open ignoring open from mgr.ceph-mon1 10.5.74.21:0/3328700921; not ready for session (expect reconnect) Sep 27 09:55:36 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:36.410+0000 7f8c42294700 0 [cephadm INFO root] Found osd claims -> {'ceph-osd31': ['88']} Sep 27 09:55:36 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:36.410+0000 7f8c42294700 0 log_channel(cephadm) log [INF] : Found osd claims -> {'ceph-osd31': ['88']} Sep 27 09:55:36 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:36.410+0000 7f8c42294700 0 [cephadm INFO cephadm.services.osd] Found osd claims for drivegroup ceph-osd31 -> {'ceph-osd31': ['88']} Sep 27 09:55:36 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:36.414+0000 7f8c42294700 0 log_channel(cephadm) log [INF] : Found osd claims for drivegroup ceph-osd31 -> {'ceph-osd31': ['88']} Sep 27 09:55:36 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:36.426+0000 7f8c42294700 0 [cephadm INFO root] Found osd claims -> {'ceph-osd31': ['88']} Sep 27 09:55:36 ceph-mon3 bash[476409]: debug 2024-09-27T13:55:36.426+0000 7f8c42294700 0 log_channel(cephadm) log [INF] : Found osd claims -> {'ceph-osd31': ['88']} `ceph orch osd rm status` reports "No OSD remove/replace operations reported” On the osd node: `ceph-volume inventory --list-all --with-lsm /dev/sdg` reports: ====== Device report /dev/sdg ====== path /dev/sdg ceph device False lsm data {} available False rejected reasons Insufficient space (<5GB) device id INTEL_SSDSC2KG038T8_PHYG039600UB3P8EGN vs. a drive which has been successfully converted: ====== Device report /dev/sdf ====== path /dev/sdf ceph device True lsm data {} available False rejected reasons Has a FileSystem, LVM detected, Insufficient space (<10 extents) on vgs, Insufficient space (<5GB) device id INTEL_SSDSC2KG038T8_PHYG039600CM3P8EGN --- Logical Volume --- name osd-block-88426db7-2322-4807-ac2e-b49929e170d6 osd id 87 cluster name ceph type block osd fsid 88426db7-2322-4807-ac2e-b49929e170d6 cluster fsid 9b3b3539-59a9-4338-8bab-3badfab6e855 osdspec affinity ceph-osd31 block uuid LNG2gB-pa0w-gl2v-hVQ3-6qTd-aXsR-Lenri3 We’ve zapped /dev/sdg a few times, initially when we ran the command to fail it out (`ceph orch osd rm 88 --replace —zap`), but also from the osd node itself with `ceph-volume lvm zap /dev/sdg —destroy`. We’ve also zapped the drive manually with: sgdisk --zap-all /dev/sdg wipefs --all --force /dev/sdg dd if=/dev/zero bs=1M count=100 oflag=direct of=/dev/sdg dd bs=512 if=/dev/zero of=/dev/sdg oflag=direct count=204800 seek=$(($(blockdev --getsz /dev/sdg) - 204800)) partprobe /dev/sdg … based on the suggestion here: https://github.com/rook/rook/issues/11474 We’ve rebooted the osd node, with and without the drive inserted. We’re unable to zap the drive from the orchestrator, as expected: ceph orch device zap ceph-osd31 /dev/sdg --force Error EINVAL: Device path '/dev/sdg' not found on host 'ceph-osd31’ Cheers, /rjg On Sep 27, 2024, at 2:07 AM, Eugen Block <eblock@nde.ag> wrote: EXTERNAL EMAIL | USE CAUTION Right, if you need encryption, a rebuild is required. Your procedure has already worked 4 times, so I'd say nothing seems wrong with that per se. Regarding the stuck device list, do you see the mgr logging anything suspicious? Especially when you say that it only returns output after a failover. Those two osd specs are not conflicting since the first is "unmanaged" after adoption. Is there something in 'ceph orch osd rm status'? Can you run 'cephadm ceph-volume inventory' locally on that node? Do you see any hints in the node's syslog? Maybe try a reboot or something? Zitat von Bob Gibson <rjg@oicr.on.ca>: Thanks for your reply Eugen. I’m fairly new to cephadm so I wasn’t aware that we could manage the drives without rebuilding them. However, we thought we’d take advantage of this opportunity to also encrypt the drives, and that does require a rebuild. I have a theory on why the orchestrator is confused. I want to create an osd service for each osd node so I can manage drives on a per-node basis. I started by creating a spec for the first node: service_type: osd service_id: ceph-osd31 placement: hosts: - ceph-osd31 spec: data_devices: rotational: 0 size: '3TB:' encrypted: true filter_logic: AND objectstore: bluestore But I also see a default spec, “osd”, which has placement set to “unmanaged”. `ceph orch ls osd —export` shows the following: service_type: osd service_name: osd unmanaged: true spec: filter_logic: AND objectstore: bluestore --- service_type: osd service_id: ceph-osd31 service_name: osd.ceph-osd31 placement: hosts: - ceph-osd31 spec: data_devices: rotational: 0 size: '3TB:' encrypted: true filter_logic: AND objectstore: bluestore `ceph orch ls osd` shows that I was able to convert 4 drives using my spec: NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd 95 10m ago - <unmanaged> osd.ceph-osd31 4 10m ago 43m ceph-osd31 Despite being able to convert 4 drives, I’m wondering if these specs are conflicting with one another, and that has confused the orchestrator. If so, how do I safely get from where I am now to where I want to be? :-) Cheers, /rjg On Sep 26, 2024, at 3:31 PM, Eugen Block <eblock@nde.ag> wrote: EXTERNAL EMAIL | USE CAUTION Hi, this seems a bit unnecessary to rebuild OSDs just to get them managed. If you apply a spec file that targets your hosts/OSDs, they will appear as managed. So when you would need to replace a drive, you could already utilize the orchestrator to remove and zap the drive. That works just fine. How to get out of your current situation is not entirely clear to me yet. I’ll reread your post tomorrow. Regards, Eugen Zitat von Bob Gibson <rjg@oicr.on.ca>: Hi, We recently converted a legacy cluster running Quincy v17.2.7 to cephadm. The conversion went smoothly and left all osds unmanaged by the orchestrator as expected. We’re now in the process of converting the osds to be managed by the orchestrator. We successfully converted a few of them, but then the orchestrator somehow got confused. `ceph health detail` reports a “stray daemon” for the osd we’re trying to convert, and the orchestrator is unable to refresh its device list so it doesn’t see any available devices. From the perspective of the osd node, the osd has been wiped and is ready to be reinstalled. We’ve also rebooted the node for good measure. `ceph osd tree` shows that the osd has been destroyed, but the orchestrator won’t reinstall it because it thinks the device is still active. The orchestrator device information is stale, but we’re unable to refresh it. The usual recommended workaround of failing over the mgr hasn’t helped. We’ve also tried `ceph orch device ls —refresh` to no avail. In fact after running that command subsequent runs of `ceph orch device ls` produce no output until the mgr is failed over again. Is there a way to force the orchestrator to refresh its list of devices when in this state? If not, can anyone offer any suggestions on how to fix this problem? Cheers, /rjg P.S. Some additional information in case it’s helpful... We’re using the following command to replace existing devices so that they’re managed by the orchestrator: ``` ceph orch osd rm <osd> --replace —zap ``` and we’re currently stuck on osd 88. ``` ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.88 on host ceph-osd31 not managed by cephadm ``` `ceph osd tree` shows that the osd has been destroyed and is ready to be replaced: ``` ceph osd tree-from ceph-osd31 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -46 34.93088 host ceph-osd31 84 ssd 3.49309 osd.84 up 1.00000 1.00000 85 ssd 3.49309 osd.85 up 1.00000 1.00000 86 ssd 3.49309 osd.86 up 1.00000 1.00000 87 ssd 3.49309 osd.87 up 1.00000 1.00000 88 ssd 3.49309 osd.88 destroyed 0 1.00000 89 ssd 3.49309 osd.89 up 1.00000 1.00000 90 ssd 3.49309 osd.90 up 1.00000 1.00000 91 ssd 3.49309 osd.91 up 1.00000 1.00000 92 ssd 3.49309 osd.92 up 1.00000 1.00000 93 ssd 3.49309 osd.93 up 1.00000 1.00000 ``` The cephadm log shows a claim on node `ceph-osd31` for that osd: ``` 2024-09-25T14:15:45.699348-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims -> {'ceph-osd31': ['88']} 2024-09-25T14:15:45.699534-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims for drivegroup ceph-osd31 -> {'ceph-osd31': ['88']} ``` `ceph orch device ls` shows that the device list isn’t refreshing: ``` ceph orch device ls ceph-osd31 HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS ceph-osd31 /dev/sdc ssd INTEL_SSDSC2KG038T8_PHYG039603PE3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdd ssd INTEL_SSDSC2KG038T8_PHYG039600AY3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sde ssd INTEL_SSDSC2KG038T8_PHYG039600CW3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdf ssd INTEL_SSDSC2KG038T8_PHYG039600CM3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdg ssd INTEL_SSDSC2KG038T8_PHYG039600UB3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdh ssd INTEL_SSDSC2KG038T8_PHYG039603753P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdi ssd INTEL_SSDSC2KG038T8_PHYG039603R63P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdj ssd INTEL_SSDSC2KG038TZ_PHYJ4011032M3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdk ssd INTEL_SSDSC2KG038TZ_PHYJ3234010J3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdl ssd INTEL_SSDSC2KG038T8_PHYG039603NS3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ``` `ceph node ls` thinks the osd still exists ``` ceph node ls osd | jq -r '."ceph-osd31"' [ 84, 85, 86, 87, 88, <— this shouldn’t exist 89, 90, 91, 92, 93 ] ``` Each osd node has 10x 3.8 TB ssd drives for osds. On `ceph-osd31`, cephadm doesn’t see osd.88 as expected: ``` cephadm ls --no-detail [ { "style": "cephadm:v1", "name": "osd.93", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.93" }, { "style": "cephadm:v1", "name": "osd.85", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.85" }, { "style": "cephadm:v1", "name": "osd.90", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.90" }, { "style": "cephadm:v1", "name": "osd.92", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.92" }, { "style": "cephadm:v1", "name": "osd.89", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.89" }, { "style": "cephadm:v1", "name": "osd.87", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.87" }, { "style": "cephadm:v1", "name": "osd.86", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.86" }, { "style": "cephadm:v1", "name": "osd.84", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.84" }, { "style": "cephadm:v1", "name": "osd.91", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855@osd.91" } ] ``` `lsblk` shows that `/dev/sdg` has been wiped. ``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 223.6G 0 disk |-sda1 8:1 0 94M 0 part `-sda2 8:2 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdb 8:16 0 223.6G 0 disk |-sdb1 8:17 0 94M 0 part `-sdb2 8:18 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdc 8:32 1 3.5T 0 disk `-ceph--03782b4c--9faa--49f5--b554--98e7b8515834-osd--block--ba272724--daa6--45f5--9f69--789cc0bda077 253:3 0 3.5T lvm `-keCkP2-o6h8-jKkw-RKiD-UBFf-A8EL-JDJGPR 253:9 0 3.5T 0 crypt sdd 8:48 1 3.5T 0 disk `-ceph--c07907d8--4a75--4ba3--b5e1--2ebf49ecbdf6-osd--block--58d1d50d--6228--4e6f--9a52--2a305ba00700 253:7 0 3.5T lvm `-WB8Mxn-qCHI-4T01-imiG-hNBR-by60-YuxgfD 253:11 0 3.5T 0 crypt sde 8:64 1 3.5T 0 disk `-ceph--6f9d4df4--7ce6--44a4--a7b1--62c85af8cfe0-osd--block--aabcb30d--0084--490a--969b--78f7af6e94da 253:8 0 3.5T lvm `-g9qErH-vTXY-JQbs-eh61-W0Mn-TAV8-gof4zy 253:12 0 3.5T 0 crypt sdf 8:80 1 3.5T 0 disk `-ceph--d6b728f8--e365--46db--b30f--6c00805c752b-osd--block--88426db7--2322--4807--ac2e--b49929e170d6 253:6 0 3.5T lvm `-LNG2gB-pa0w-gl2v-hVQ3-6qTd-aXsR-Lenri3 253:10 0 3.5T 0 crypt sdg 8:96 1 3.5T 0 disk sdh 8:112 1 3.5T 0 disk `-ceph--de2cfee6--8e0a--4aa0--9e6b--90c09025768c-osd--block--a3b86251--2799--4243--a857--f218fa90f29a 253:2 0 3.5T lvm sdi 8:128 1 3.5T 0 disk `-ceph--30dee450--0fdd--46ea--9eec--6a4c7706df9c-osd--block--bfc090db--dde4--47dd--a1c9--1cd838ea43b3 253:4 0 3.5T lvm sdj 8:144 1 3.5T 0 disk `-ceph--78febcf5--43f4--4820--8dc7--0f6c22816c9f-osd--block--da1e69c7--6427--4562--8290--90bcb9526747 253:0 0 3.5T lvm sdk 8:160 1 3.5T 0 disk `-ceph--fe210281--b1f5--4d5e--9ab0--2f226612af00-osd--block--6bb9f308--e853--4303--83ea--553c3a3513e1 253:1 0 3.5T lvm sdl 8:176 1 3.5T 0 disk `-ceph--9f21c916--f211--4d1b--8214--6ad1cecac810-osd--block--572d850c--c201--4af4--ac42--0ed2a6ed73ed 253:5 0 3.5T lvm ``` _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io