8 Jan
2023
8 Jan
'23
8:43 p.m.
Dear team,
Kindly help on this , I am completely blocked.
Best Regards
Michel
On Thu, Jan 5, 2023 at 2:45 PM Michel Niyoyita <micou12(a)gmail.com> wrote:
Dear team,
I have deployed the ceph cluster in production using ceph-ansible on
ubuntu OS 20.04 it consists of 3 monitors and 3 osds (each osd has 20 disks
, 16 hdd and 4 ssd) after deployment the ceph cluster healthy is OK , but
instead of getting 60 disk it appears 48 disks which are hdd only
(according to the output of ceph osd df tree) below are output of ceph osd
df tree and lsblk command.
hdd: 7.71349 T
ssd: 7 T
ceph osd df tree
root@ceph-mon3:~# ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE VAR PGS STATUS TYPE NAME
-1 370.24731 - 370 TiB 21 TiB 249 MiB 0 B 1.2 GiB
349 TiB 5.66 1.00 - root default
-5 123.41577 - 123 TiB 7.0 TiB 83 MiB 0 B 403 MiB
116 TiB 5.66 1.00 - host ceph-osd1
0 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB
7.3 TiB 5.66 1.00 6 up osd.0
3 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB
7.3 TiB 5.66 1.00 8 up osd.3
6 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 35 MiB
7.3 TiB 5.66 1.00 12 up osd.6
9 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB
7.3 TiB 5.66 1.00 4 up osd.9
12 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB
7.3 TiB 5.66 1.00 6 up osd.12
16 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB
7.3 TiB 5.66 1.00 3 up osd.16
18 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB
7.3 TiB 5.66 1.00 5 up osd.18
21 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB
7.3 TiB 5.66 1.00 6 up osd.21
24 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 35 MiB
7.3 TiB 5.66 1.00 7 up osd.24
27 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB
7.3 TiB 5.66 1.00 7 up osd.27
30 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB
7.3 TiB 5.66 1.00 7 up osd.30
33 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB
7.3 TiB 5.66 1.00 7 up osd.33
36 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB
7.3 TiB 5.66 1.00 5 up osd.36
39 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB
7.3 TiB 5.66 1.00 9 up osd.39
42 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB
7.3 TiB 5.66 1.00 6 up osd.42
45 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB
7.3 TiB 5.66 1.00 7 up osd.45
-3 123.41577 - 123 TiB 7.0 TiB 83 MiB 0 B 397 MiB
116 TiB 5.66 1.00 - host ceph-osd2
1 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB
7.3 TiB 5.66 1.00 7 up osd.1
5 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB
7.3 TiB 5.66 1.00 7 up osd.5
8 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB
7.3 TiB 5.66 1.00 6 up osd.8
11 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB
7.3 TiB 5.66 1.00 5 up osd.11
14 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB
7.3 TiB 5.66 1.00 7 up osd.14
15 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB
7.3 TiB 5.66 1.00 12 up osd.15
19 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB
7.3 TiB 5.66 1.00 5 up osd.19
22 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB
7.3 TiB 5.66 1.00 4 up osd.22
25 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB
7.3 TiB 5.66 1.00 2 up osd.25
28 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB
7.3 TiB 5.66 1.00 3 up osd.28
31 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB
7.3 TiB 5.66 1.00 6 up osd.31
34 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB
7.3 TiB 5.66 1.00 7 up osd.34
37 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB
7.3 TiB 5.66 1.00 8 up osd.37
40 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB
7.3 TiB 5.66 1.00 8 up osd.40
43 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB
7.3 TiB 5.66 1.00 6 up osd.43
46 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB
7.3 TiB 5.66 1.00 12 up osd.46
-7 123.41577 - 123 TiB 7.0 TiB 83 MiB 0 B 409 MiB
116 TiB 5.66 1.00 - host ceph-osd3
2 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB
7.3 TiB 5.66 1.00 4 up osd.2
4 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB
7.3 TiB 5.66 1.00 7 up osd.4
7 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB
7.3 TiB 5.66 1.00 5 up osd.7
10 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB
7.3 TiB 5.66 1.00 4 up osd.10
13 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB
7.3 TiB 5.66 1.00 12 up osd.13
17 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB
7.3 TiB 5.66 1.00 6 up osd.17
20 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB
7.3 TiB 5.66 1.00 7 up osd.20
23 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB
7.3 TiB 5.66 1.00 8 up osd.23
26 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB
7.3 TiB 5.66 1.00 8 up osd.26
29 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB
7.3 TiB 5.66 1.00 6 up osd.29
32 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB
7.3 TiB 5.66 1.00 6 up osd.32
35 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB
7.3 TiB 5.66 1.00 8 up osd.35
38 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB
7.3 TiB 5.66 1.00 9 up osd.38
41 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB
7.3 TiB 5.66 1.00 3 up osd.41
44 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB
7.3 TiB 5.66 1.00 4 up osd.44
47 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB
7.3 TiB 5.66 1.00 8 up osd.47
TOTAL 370 TiB 21 TiB 249 MiB 0 B 1.2 GiB
349 TiB 5.66
lsblk out put
==========
root@ceph-osd1:~# lsblk
NAME
MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0
7:0 0 55.4M 1 loop /snap/core18/1944
loop1
7:1 0 69.9M 1 loop /snap/lxd/19188
loop2
7:2 0 31.1M 1 loop /snap/snapd/10707
sda
8:0 0 7.3T 0 disk
└─ceph--f011a10f--3f22--4785--8fe8--1eb4e64d380f-osd--block--1c3eaee6--60a7--4b2d--aff1--dff0a909d48e
253:1 0 7.3T 0 lvm
sdb
8:16 0 7.3T 0 disk
└─ceph--ba3fef60--84a2--4cd7--9aeb--4af05a17733d-osd--block--a4926772--4913--4ad1--8c01--348e8cb30159
253:3 0 7.3T 0 lvm
sdc
8:32 0 7.3T 0 disk
└─ceph--14676d53--0a82--47b2--acc2--10ebae5e2cb1-osd--block--b1032109--80e0--4065--923a--80541669c316
253:5 0 7.3T 0 lvm
sdd
8:48 0 7.3T 0 disk
└─ceph--2e7875da--a136--4b39--b354--21cfa8643951-osd--block--5a1ad840--225c--4d79--ab23--8589f13b9f3d
253:7 0 7.3T 0 lvm
sde
8:64 0 7T 0 disk
├─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--be9d5f0f--ac60--42e1--850d--bf2baf913b3b
253:26 0 447.1G 0 lvm
├─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--5e7310c4--2b18--4938--ab6e--dc37e860d9d5
253:28 0 447.1G 0 lvm
├─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--d2d78859--2e60--4ccc--b417--d33ff55837b6
253:30 0 447.1G 0 lvm
└─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--31ff57f6--e9d3--4bf1--b08d--95939083b4da
253:32 0 447.1G 0 lvm
sdf
8:80 0 7T 0 disk
├─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--7a8d393e--3af3--4c89--a99f--d3efcbb78066
253:18 0 447.1G 0 lvm
├─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--5abd44ca--aed8--48af--8f12--a64a6ae7938c
253:20 0 447.1G 0 lvm
├─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--cd5768aa--b9ce--4c3b--9bc8--38c01b5bdc49
253:22 0 447.1G 0 lvm
└─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--364e42ff--6e66--4a76--9df2--b2bb04080cd0
253:24 0 447.1G 0 lvm
sdg
8:96 0 7T 0 disk
├─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--3c469d07--0f1d--4544--861a--dd84283332fe
253:10 0 447.1G 0 lvm
├─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--61d08fe1--8d5a--4ef8--ba2d--ba1fc84636db
253:12 0 447.1G 0 lvm
├─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--2b2a8af7--33f5--4ca5--b09b--2f24ebf583b3
253:14 0 447.1G 0 lvm
└─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--fd2e105a--4fdf--4f15--a48b--1303717b8961
253:16 0 447.1G 0 lvm
sdh
8:112 0 7T 0 disk
├─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--6e289081--5404--4294--a75a--f4a1a8fabe52
253:2 0 447.1G 0 lvm
├─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--e89285a3--4efa--4239--bbb7--a95064ced7f2
253:4 0 447.1G 0 lvm
├─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--0b7f1fd0--bc32--47b7--b8e4--c6f0f9f21047
253:6 0 447.1G 0 lvm
└─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--614e085d--ce04--4f5d--a485--ddca51bae65b
253:8 0 447.1G 0 lvm
sdi
8:128 0 7.3T 0 disk
└─ceph--8343ded8--765b--4038--9855--5e6684b55906-osd--block--054a0d83--4aea--4d4f--b623--77db554e207d
253:9 0 7.3T 0 lvm
sdj
8:144 0 7.3T 0 disk
└─ceph--88f64697--ab62--4fed--a1b4--88ef3563eaea-osd--block--97189221--a7f6--4d9d--b98a--a474fec95fc7
253:11 0 7.3T 0 lvm
sdk
8:160 0 7.3T 0 disk
└─ceph--592f15db--face--49d8--b028--15fee31d77cd-osd--block--776b773d--347f--4a5c--8de7--66d4a9d56feb
253:13 0 7.3T 0 lvm
sdl
8:176 0 7.3T 0 disk
└─ceph--3ef3da28--0aa0--4bfe--80fa--7871a0ebbb2a-osd--block--9b91fccb--3562--4543--916a--6fd0baa00efd
253:15 0 7.3T 0 lvm
sdm
8:192 0 7.3T 0 disk
└─ceph--604f22a2--bf89--40a8--bd37--bb5c317f0525-osd--block--ff387140--c528--40e2--ac75--3add53fd5bbb
253:17 0 7.3T 0 lvm
sdn
8:208 0 7.3T 0 disk
└─ceph--b702f897--06ad--43fe--b04b--3e03477ffbc3-osd--block--07cc426f--b4c1--4459--a6bc--81ee7a034060
253:19 0 7.3T 0 lvm
sdo
8:224 0 7.3T 0 disk
└─ceph--5e1b4c03--f01b--45d7--971f--a02fe109ef13-osd--block--653b2370--dcb0--46bc--a71b--f62af6777feb
253:21 0 7.3T 0 lvm
sdp
8:240 0 7.3T 0 disk
└─ceph--e3ed1f19--15f6--4bd2--81f0--93f9720d7fad-osd--block--4121fcc9--1c00--4da5--b01e--a041280d330a
253:23 0 7.3T 0 lvm
sdq
65:0 0 7.3T 0 disk
└─ceph--417cb977--615a--4b86--a4a9--d6af7587b670-osd--block--e77b7eab--7dd8--4300--bd98--27d010c2e17e
253:25 0 7.3T 0 lvm
sdr
65:16 0 7.3T 0 disk
└─ceph--ad66d2d6--581d--4985--b945--154586b39903-osd--block--6b1b3de2--8e13--4361--bcd0--16ad609177b0
253:27 0 7.3T 0 lvm
sds
65:32 0 7.3T 0 disk
└─ceph--66aebaac--de6b--42ff--94f1--a489a755e51a-osd--block--a581f3e1--7d86--4f8f--9d0a--de87ef713436
253:29 0 7.3T 0 lvm
sdt
65:48 0 7.3T 0 disk
└─ceph--1558f2d7--741a--4feb--9e68--a900b823eac7-osd--block--52d32f21--9f3a--491e--a26c--41304d63a0ee
253:31 0 7.3T 0 lvm
sdu
65:64 0 223.5G 0 disk
├─sdu1
65:65 0 512M 0 part /boot/efi
├─sdu2
65:66 0 1G 0 part /boot
└─sdu3
65:67 0 222G 0 part
└─ubuntu--vg-ubuntu--lv
253:0 0 111G 0 lvm /
Kindly help me to configure in order to get all disk up and work.
Best Regards
Michel