Hi, did you find an explanation for this? I saw something similar on a customer's cluster. They reprovisioned OSDs (I don't know if any OSD-ID was reused) on one host with smaller disk sizes (size was changed through the raid controller to match the other hosts in that cluster) and they got their old crush weights (reflecting the old disk sizes). In Luminous I remember that changed reweights (not sure about crush weights) were stored somewhere in /var/run/ceph/ but that doesn't seem to be the case anymore and it also would be only relevant until a reboot. I'd also be interested where this information is stored in newer releases and why it's stored in the first place. Regards, Eugen Zitat von Robert Sander <r.sander@heinlein-support.de>:

Hi,

I stumbled across an issue where an OSD the gets redeployed has a CRUSH weight of 0 after cephadm finishes.

I have created a service definition for the orchestrator to automatically deploy OSDs on SSDs:

service_type: osd service_id: SSD_OSDs placement: label: 'osd' data_devices: rotational: 0 size: '100G'

These are my steps to reproduce this in a small test cluster running 15.2.4:

root@ceph01:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 1.63994 root default -18 0.81995 rack rack10 -3 0.40996 host ceph01 8 hdd 0.10699 osd.8 up 1.00000 1.00000 9 hdd 0.10699 osd.9 up 1.00000 1.00000 0 ssd 0.09799 osd.0 up 1.00000 1.00000 1 ssd 0.09798 osd.1 up 1.00000 1.00000 -5 0.40999 host ceph02 10 hdd 0.10699 osd.10 up 1.00000 1.00000 11 hdd 0.10699 osd.11 up 1.00000 1.00000 2 ssd 0.09799 osd.2 up 1.00000 1.00000 3 ssd 0.09799 osd.3 up 1.00000 1.00000 -17 0.81999 rack rack11 -7 0.40999 host ceph03 12 hdd 0.10699 osd.12 up 1.00000 1.00000 13 hdd 0.10699 osd.13 up 1.00000 1.00000 4 ssd 0.09799 osd.4 up 1.00000 1.00000 5 ssd 0.09799 osd.5 up 1.00000 1.00000 -9 0.40999 host ceph04 14 hdd 0.10699 osd.14 up 1.00000 1.00000 15 hdd 0.10699 osd.15 up 1.00000 1.00000 6 ssd 0.09799 osd.6 up 1.00000 1.00000 7 ssd 0.09799 osd.7 up 1.00000 1.00000 root@ceph01:~# ceph osd out 1 marked out osd.1. root@ceph01:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 1.63994 root default -18 0.81995 rack rack10 -3 0.40996 host ceph01 8 hdd 0.10699 osd.8 up 1.00000 1.00000 9 hdd 0.10699 osd.9 up 1.00000 1.00000 0 ssd 0.09799 osd.0 up 1.00000 1.00000 1 ssd 0.09798 osd.1 up 0 1.00000 -5 0.40999 host ceph02 10 hdd 0.10699 osd.10 up 1.00000 1.00000 11 hdd 0.10699 osd.11 up 1.00000 1.00000 2 ssd 0.09799 osd.2 up 1.00000 1.00000 3 ssd 0.09799 osd.3 up 1.00000 1.00000 -17 0.81999 rack rack11 -7 0.40999 host ceph03 12 hdd 0.10699 osd.12 up 1.00000 1.00000 13 hdd 0.10699 osd.13 up 1.00000 1.00000 4 ssd 0.09799 osd.4 up 1.00000 1.00000 5 ssd 0.09799 osd.5 up 1.00000 1.00000 -9 0.40999 host ceph04 14 hdd 0.10699 osd.14 up 1.00000 1.00000 15 hdd 0.10699 osd.15 up 1.00000 1.00000 6 ssd 0.09799 osd.6 up 1.00000 1.00000 7 ssd 0.09799 osd.7 up 1.00000 1.00000 root@ceph01:~# ceph orch osd rm 1 Scheduled OSD(s) for removal

2020-09-10T16:29:58.176991+0200 mgr.ceph02.ouelws [INF] Removing daemon osd.1 from ceph01 2020-09-10T16:30:00.148659+0200 mgr.ceph02.ouelws [INF] Successfully removed OSD <1> on ceph01

root@ceph01:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 1.54196 root default -18 0.72197 rack rack10 -3 0.31198 host ceph01 8 hdd 0.10699 osd.8 up 1.00000 1.00000 9 hdd 0.10699 osd.9 up 1.00000 1.00000 0 ssd 0.09799 osd.0 up 1.00000 1.00000 -5 0.40999 host ceph02 10 hdd 0.10699 osd.10 up 1.00000 1.00000 11 hdd 0.10699 osd.11 up 1.00000 1.00000 2 ssd 0.09799 osd.2 up 1.00000 1.00000 3 ssd 0.09799 osd.3 up 1.00000 1.00000 -17 0.81999 rack rack11 -7 0.40999 host ceph03 12 hdd 0.10699 osd.12 up 1.00000 1.00000 13 hdd 0.10699 osd.13 up 1.00000 1.00000 4 ssd 0.09799 osd.4 up 1.00000 1.00000 5 ssd 0.09799 osd.5 up 1.00000 1.00000 -9 0.40999 host ceph04 14 hdd 0.10699 osd.14 up 1.00000 1.00000 15 hdd 0.10699 osd.15 up 1.00000 1.00000 6 ssd 0.09799 osd.6 up 1.00000 1.00000 7 ssd 0.09799 osd.7 up 1.00000 1.00000 root@ceph01:~# ceph orch device zap ceph01 /dev/sdc --force INFO:cephadm:/usr/bin/docker:stderr --> Zapping: /dev/sdc INFO:cephadm:/usr/bin/docker:stderr --> Zapping lvm member /dev/sdc. lv_path is /dev/ceph-0d19a151-30b6-459e-936a-488f143e11f6/osd-block-d5062900-abe7-413a-9d9a-d1cdda2948eb INFO:cephadm:/usr/bin/docker:stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-0d19a151-30b6-459e-936a-488f143e11f6/osd-block-d5062900-abe7-413a-9d9a-d1cdda2948eb bs=1M count=10 conv=fsync INFO:cephadm:/usr/bin/docker:stderr stderr: 10+0 records in INFO:cephadm:/usr/bin/docker:stderr 10+0 records out INFO:cephadm:/usr/bin/docker:stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0583658 s, 180 MB/s INFO:cephadm:/usr/bin/docker:stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-0d19a151-30b6-459e-936a-488f143e11f6 INFO:cephadm:/usr/bin/docker:stderr Running command: /usr/sbin/vgremove -v -f ceph-0d19a151-30b6-459e-936a-488f143e11f6 INFO:cephadm:/usr/bin/docker:stderr stderr: Removing ceph--0d19a151--30b6--459e--936a--488f143e11f6-osd--block--d5062900--abe7--413a--9d9a--d1cdda2948eb (253:3) INFO:cephadm:/usr/bin/docker:stderr stderr: Archiving volume group "ceph-0d19a151-30b6-459e-936a-488f143e11f6" metadata (seqno 5). INFO:cephadm:/usr/bin/docker:stderr Releasing logical volume "osd-block-d5062900-abe7-413a-9d9a-d1cdda2948eb" INFO:cephadm:/usr/bin/docker:stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-0d19a151-30b6-459e-936a-488f143e11f6" (seqno 6). INFO:cephadm:/usr/bin/docker:stderr stdout: Logical volume "osd-block-d5062900-abe7-413a-9d9a-d1cdda2948eb" successfully removed INFO:cephadm:/usr/bin/docker:stderr stderr: Removing physical volume "/dev/sdc" from volume group "ceph-0d19a151-30b6-459e-936a-488f143e11f6" INFO:cephadm:/usr/bin/docker:stderr stdout: Volume group "ceph-0d19a151-30b6-459e-936a-488f143e11f6" successfully removed INFO:cephadm:/usr/bin/docker:stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/sdc bs=1M count=10 conv=fsync INFO:cephadm:/usr/bin/docker:stderr stderr: 10+0 records in INFO:cephadm:/usr/bin/docker:stderr 10+0 records out INFO:cephadm:/usr/bin/docker:stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.016043 s, 654 MB/s INFO:cephadm:/usr/bin/docker:stderr --> Zapping successful for: <Raw Device: /dev/sdc>

2020-09-10T16:31:15.951617+0200 mgr.ceph02.ouelws [INF] Zap device ceph01:/dev/sdc 2020-09-10T16:31:24.738974+0200 mgr.ceph02.ouelws [INF] Found osd claims for drivegroup SSD_OSDs -> {} 2020-09-10T16:31:24.740489+0200 mgr.ceph02.ouelws [INF] Applying SSD_OSDs on host ceph01... 2020-09-10T16:31:31.549897+0200 mgr.ceph02.ouelws [INF] Deploying daemon osd.1 on ceph01 2020-09-10T16:31:33.057061+0200 mgr.ceph02.ouelws [INF] Applying SSD_OSDs on host ceph02... 2020-09-10T16:31:33.057373+0200 mgr.ceph02.ouelws [INF] Applying SSD_OSDs on host ceph03... 2020-09-10T16:31:33.057519+0200 mgr.ceph02.ouelws [INF] Applying SSD_OSDs on host ceph04... 2020-09-10T16:31:37.569914+0200 mon.ceph01 [INF] osd.1 [v2:10.24.4.128:6810/4173467371,v1:10.24.4.128:6811/4173467371] boot 2020-09-10T16:31:46.531544+0200 mon.ceph01 [INF] Cluster is now healthy

root@ceph01:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 1.54196 root default -18 0.72197 rack rack10 -3 0.31198 host ceph01 8 hdd 0.10699 osd.8 up 1.00000 1.00000 9 hdd 0.10699 osd.9 up 1.00000 1.00000 0 ssd 0.09799 osd.0 up 1.00000 1.00000 1 ssd 0 osd.1 up 1.00000 1.00000 -5 0.40999 host ceph02 10 hdd 0.10699 osd.10 up 1.00000 1.00000 11 hdd 0.10699 osd.11 up 1.00000 1.00000 2 ssd 0.09799 osd.2 up 1.00000 1.00000 3 ssd 0.09799 osd.3 up 1.00000 1.00000 -17 0.81999 rack rack11 -7 0.40999 host ceph03 12 hdd 0.10699 osd.12 up 1.00000 1.00000 13 hdd 0.10699 osd.13 up 1.00000 1.00000 4 ssd 0.09799 osd.4 up 1.00000 1.00000 5 ssd 0.09799 osd.5 up 1.00000 1.00000 -9 0.40999 host ceph04 14 hdd 0.10699 osd.14 up 1.00000 1.00000 15 hdd 0.10699 osd.15 up 1.00000 1.00000 6 ssd 0.09799 osd.6 up 1.00000 1.00000 7 ssd 0.09799 osd.7 up 1.00000 1.00000

Why does osd.1 have a weight of 0 now?

When the OSDs had been initially deployed with the first ceph orch apply command the weights have been correctly set according to their size. Why is there a difference between this process and an OSD (re-)deployed later on?

Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin

http://www.heinlein-support.de

Tel: 030 / 405051-43 Fax: 030 / 405051-19

Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin