Dear team, Kindly help on this , I am completely blocked. Best Regards Michel On Thu, Jan 5, 2023 at 2:45 PM Michel Niyoyita <micou12@gmail.com> wrote:

Dear team,

I have deployed the ceph cluster in production using ceph-ansible on ubuntu OS 20.04 it consists of 3 monitors and 3 osds (each osd has 20 disks , 16 hdd and 4 ssd) after deployment the ceph cluster healthy is OK , but instead of getting 60 disk it appears 48 disks which are hdd only (according to the output of ceph osd df tree) below are output of ceph osd df tree and lsblk command.

hdd: 7.71349 T ssd: 7 T

ceph osd df tree root@ceph-mon3:~# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 370.24731 - 370 TiB 21 TiB 249 MiB 0 B 1.2 GiB 349 TiB 5.66 1.00 - root default -5 123.41577 - 123 TiB 7.0 TiB 83 MiB 0 B 403 MiB 116 TiB 5.66 1.00 - host ceph-osd1 0 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB 7.3 TiB 5.66 1.00 6 up osd.0 3 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB 7.3 TiB 5.66 1.00 8 up osd.3 6 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 35 MiB 7.3 TiB 5.66 1.00 12 up osd.6 9 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB 7.3 TiB 5.66 1.00 4 up osd.9 12 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB 7.3 TiB 5.66 1.00 6 up osd.12 16 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB 7.3 TiB 5.66 1.00 3 up osd.16 18 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB 7.3 TiB 5.66 1.00 5 up osd.18 21 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB 7.3 TiB 5.66 1.00 6 up osd.21 24 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 35 MiB 7.3 TiB 5.66 1.00 7 up osd.24 27 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB 7.3 TiB 5.66 1.00 7 up osd.27 30 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB 7.3 TiB 5.66 1.00 7 up osd.30 33 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB 7.3 TiB 5.66 1.00 7 up osd.33 36 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB 7.3 TiB 5.66 1.00 5 up osd.36 39 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB 7.3 TiB 5.66 1.00 9 up osd.39 42 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB 7.3 TiB 5.66 1.00 6 up osd.42 45 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB 7.3 TiB 5.66 1.00 7 up osd.45 -3 123.41577 - 123 TiB 7.0 TiB 83 MiB 0 B 397 MiB 116 TiB 5.66 1.00 - host ceph-osd2 1 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB 7.3 TiB 5.66 1.00 7 up osd.1 5 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB 7.3 TiB 5.66 1.00 7 up osd.5 8 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB 7.3 TiB 5.66 1.00 6 up osd.8 11 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB 7.3 TiB 5.66 1.00 5 up osd.11 14 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB 7.3 TiB 5.66 1.00 7 up osd.14 15 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB 7.3 TiB 5.66 1.00 12 up osd.15 19 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB 7.3 TiB 5.66 1.00 5 up osd.19 22 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB 7.3 TiB 5.66 1.00 4 up osd.22 25 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB 7.3 TiB 5.66 1.00 2 up osd.25 28 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB 7.3 TiB 5.66 1.00 3 up osd.28 31 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB 7.3 TiB 5.66 1.00 6 up osd.31 34 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB 7.3 TiB 5.66 1.00 7 up osd.34 37 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB 7.3 TiB 5.66 1.00 8 up osd.37 40 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB 7.3 TiB 5.66 1.00 8 up osd.40 43 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB 7.3 TiB 5.66 1.00 6 up osd.43 46 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB 7.3 TiB 5.66 1.00 12 up osd.46 -7 123.41577 - 123 TiB 7.0 TiB 83 MiB 0 B 409 MiB 116 TiB 5.66 1.00 - host ceph-osd3 2 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB 7.3 TiB 5.66 1.00 4 up osd.2 4 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB 7.3 TiB 5.66 1.00 7 up osd.4 7 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB 7.3 TiB 5.66 1.00 5 up osd.7 10 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB 7.3 TiB 5.66 1.00 4 up osd.10 13 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB 7.3 TiB 5.66 1.00 12 up osd.13 17 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB 7.3 TiB 5.66 1.00 6 up osd.17 20 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB 7.3 TiB 5.66 1.00 7 up osd.20 23 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB 7.3 TiB 5.66 1.00 8 up osd.23 26 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB 7.3 TiB 5.66 1.00 8 up osd.26 29 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB 7.3 TiB 5.66 1.00 6 up osd.29 32 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB 7.3 TiB 5.66 1.00 6 up osd.32 35 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB 7.3 TiB 5.66 1.00 8 up osd.35 38 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB 7.3 TiB 5.66 1.00 9 up osd.38 41 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB 7.3 TiB 5.66 1.00 3 up osd.41 44 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB 7.3 TiB 5.66 1.00 4 up osd.44 47 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB 7.3 TiB 5.66 1.00 8 up osd.47 TOTAL 370 TiB 21 TiB 249 MiB 0 B 1.2 GiB 349 TiB 5.66

lsblk out put ========== root@ceph-osd1:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 55.4M 1 loop /snap/core18/1944 loop1 7:1 0 69.9M 1 loop /snap/lxd/19188 loop2 7:2 0 31.1M 1 loop /snap/snapd/10707 sda 8:0 0 7.3T 0 disk └─ceph--f011a10f--3f22--4785--8fe8--1eb4e64d380f-osd--block--1c3eaee6--60a7--4b2d--aff1--dff0a909d48e 253:1 0 7.3T 0 lvm sdb 8:16 0 7.3T 0 disk └─ceph--ba3fef60--84a2--4cd7--9aeb--4af05a17733d-osd--block--a4926772--4913--4ad1--8c01--348e8cb30159 253:3 0 7.3T 0 lvm sdc 8:32 0 7.3T 0 disk └─ceph--14676d53--0a82--47b2--acc2--10ebae5e2cb1-osd--block--b1032109--80e0--4065--923a--80541669c316 253:5 0 7.3T 0 lvm sdd 8:48 0 7.3T 0 disk └─ceph--2e7875da--a136--4b39--b354--21cfa8643951-osd--block--5a1ad840--225c--4d79--ab23--8589f13b9f3d 253:7 0 7.3T 0 lvm sde 8:64 0 7T 0 disk ├─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--be9d5f0f--ac60--42e1--850d--bf2baf913b3b 253:26 0 447.1G 0 lvm ├─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--5e7310c4--2b18--4938--ab6e--dc37e860d9d5 253:28 0 447.1G 0 lvm ├─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--d2d78859--2e60--4ccc--b417--d33ff55837b6 253:30 0 447.1G 0 lvm └─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--31ff57f6--e9d3--4bf1--b08d--95939083b4da 253:32 0 447.1G 0 lvm sdf 8:80 0 7T 0 disk ├─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--7a8d393e--3af3--4c89--a99f--d3efcbb78066 253:18 0 447.1G 0 lvm ├─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--5abd44ca--aed8--48af--8f12--a64a6ae7938c 253:20 0 447.1G 0 lvm ├─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--cd5768aa--b9ce--4c3b--9bc8--38c01b5bdc49 253:22 0 447.1G 0 lvm └─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--364e42ff--6e66--4a76--9df2--b2bb04080cd0 253:24 0 447.1G 0 lvm sdg 8:96 0 7T 0 disk ├─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--3c469d07--0f1d--4544--861a--dd84283332fe 253:10 0 447.1G 0 lvm ├─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--61d08fe1--8d5a--4ef8--ba2d--ba1fc84636db 253:12 0 447.1G 0 lvm ├─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--2b2a8af7--33f5--4ca5--b09b--2f24ebf583b3 253:14 0 447.1G 0 lvm └─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--fd2e105a--4fdf--4f15--a48b--1303717b8961 253:16 0 447.1G 0 lvm sdh 8:112 0 7T 0 disk ├─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--6e289081--5404--4294--a75a--f4a1a8fabe52 253:2 0 447.1G 0 lvm ├─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--e89285a3--4efa--4239--bbb7--a95064ced7f2 253:4 0 447.1G 0 lvm ├─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--0b7f1fd0--bc32--47b7--b8e4--c6f0f9f21047 253:6 0 447.1G 0 lvm └─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--614e085d--ce04--4f5d--a485--ddca51bae65b 253:8 0 447.1G 0 lvm sdi 8:128 0 7.3T 0 disk └─ceph--8343ded8--765b--4038--9855--5e6684b55906-osd--block--054a0d83--4aea--4d4f--b623--77db554e207d 253:9 0 7.3T 0 lvm sdj 8:144 0 7.3T 0 disk └─ceph--88f64697--ab62--4fed--a1b4--88ef3563eaea-osd--block--97189221--a7f6--4d9d--b98a--a474fec95fc7 253:11 0 7.3T 0 lvm sdk 8:160 0 7.3T 0 disk └─ceph--592f15db--face--49d8--b028--15fee31d77cd-osd--block--776b773d--347f--4a5c--8de7--66d4a9d56feb 253:13 0 7.3T 0 lvm sdl 8:176 0 7.3T 0 disk └─ceph--3ef3da28--0aa0--4bfe--80fa--7871a0ebbb2a-osd--block--9b91fccb--3562--4543--916a--6fd0baa00efd 253:15 0 7.3T 0 lvm sdm 8:192 0 7.3T 0 disk └─ceph--604f22a2--bf89--40a8--bd37--bb5c317f0525-osd--block--ff387140--c528--40e2--ac75--3add53fd5bbb 253:17 0 7.3T 0 lvm sdn 8:208 0 7.3T 0 disk └─ceph--b702f897--06ad--43fe--b04b--3e03477ffbc3-osd--block--07cc426f--b4c1--4459--a6bc--81ee7a034060 253:19 0 7.3T 0 lvm sdo 8:224 0 7.3T 0 disk └─ceph--5e1b4c03--f01b--45d7--971f--a02fe109ef13-osd--block--653b2370--dcb0--46bc--a71b--f62af6777feb 253:21 0 7.3T 0 lvm sdp 8:240 0 7.3T 0 disk └─ceph--e3ed1f19--15f6--4bd2--81f0--93f9720d7fad-osd--block--4121fcc9--1c00--4da5--b01e--a041280d330a 253:23 0 7.3T 0 lvm sdq 65:0 0 7.3T 0 disk └─ceph--417cb977--615a--4b86--a4a9--d6af7587b670-osd--block--e77b7eab--7dd8--4300--bd98--27d010c2e17e 253:25 0 7.3T 0 lvm sdr 65:16 0 7.3T 0 disk └─ceph--ad66d2d6--581d--4985--b945--154586b39903-osd--block--6b1b3de2--8e13--4361--bcd0--16ad609177b0 253:27 0 7.3T 0 lvm sds 65:32 0 7.3T 0 disk └─ceph--66aebaac--de6b--42ff--94f1--a489a755e51a-osd--block--a581f3e1--7d86--4f8f--9d0a--de87ef713436 253:29 0 7.3T 0 lvm sdt 65:48 0 7.3T 0 disk └─ceph--1558f2d7--741a--4feb--9e68--a900b823eac7-osd--block--52d32f21--9f3a--491e--a26c--41304d63a0ee 253:31 0 7.3T 0 lvm sdu 65:64 0 223.5G 0 disk ├─sdu1 65:65 0 512M 0 part /boot/efi ├─sdu2 65:66 0 1G 0 part /boot └─sdu3 65:67 0 222G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 111G 0 lvm /

Kindly help me to configure in order to get all disk up and work.

Best Regards

Michel