26 May
2021
26 May
'21
6:12 p.m.
Could you share the output of
lsblk -o name,rota,size,type
from the affected osd node?
My spec file is for a tiny lab cluster, in your case the db drive size
should be something like '5T:6T' to specify a range.
How large are the HDDs? Also maybe you should use the option
'filter_logic: AND', but I'm not sure if that's already the default, I
remember that there were issues in Nautilus because the default was
OR. I tried this just recently with a version similar to this, I
believe it was 15.2.8 and it worked for me, but again, it's just a
tiny virtual lab cluster.
Zitat von Kai Stian Olstad <ceph+list(a)olstad.com>om>:
On 26.05.2021 11:16, Eugen Block wrote:
Yes, the LVs are not removed automatically, you
need to free up the
VG, there are a couple of ways to do so, for example remotely:
pacific1:~ # ceph orch device zap pacific4 /dev/vdb --force
or directly on the host with:
pacific1:~ # cephadm ceph-volume lvm zap --destroy /dev/<CEPH_VG>/<DB_LV>
Thanks,
I used the cephadm command and deleted the LV and the VG now has free space
# vgs | egrep "VG|dbs"
VG #PV #LV #SN
Attr VSize VFree
ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b 3 14 0
wz--n- <5.24t 357.74g
But it doesn't seams to be able to use it, because it can find anyting
# ceph orch apply osd -i hdd.yml --dry-run
################
OSDSPEC PREVIEWS
################
+---------+------+-------------+----------+----+-----+
|SERVICE |NAME |HOST |DATA |DB |WAL |
+---------+------+-------------+----------+----+-----+
+---------+------+-------------+----------+----+-----+
I tried adding size as you have in your configuration
db_devices:
rotational: 0
size: '30G:'
Still it was unable to create the OSD.
If I removed the : so it is 30GB exact size, it did find the disk,
but DB is not placed on a SSD since I do not have one with 30 GB
exact size
################
OSDSPEC PREVIEWS
################
+---------+------+-------------+----------+----+-----+
|SERVICE |NAME |HOST |DATA |DB |WAL |
+---------+------+-------------+----------+----+-----+
|osd |hdd |pech-hd-7 |/dev/sdt |- |- |
+---------+------+-------------+----------+----+-----+
To me I looks like Cephadm can't use/find the free space on the VG
and use that as a new LV for the OSD.
--
Kai Stian Olstad
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io 
27 May
27 May
10:30 a.m.
New subject: cephadm: How to replace failed HDD where DB is on SSD
On 26.05.2021 18:12, Eugen Block wrote:
Could you share the output of
lsblk -o name,rota,size,type
from the affected osd node?
# lsblk -o name,rota,size,type
NAME
ROTA SIZE TYPE
loop0
1 71.3M loop
loop1
1 55M loop
loop2
1 29.9M loop
sda
1 223G disk
├─sda1
1 512M part
├─sda2
1 1G part
└─sda3
1 221.5G part
sdb
1 12.5T disk
└─ceph--block--1b5ad7e7--2e24--4315--8a05--7439ab782b45-osd--block--2da790bc--a74c--41da--8772--3b8aac77001c
1 12.5T lvm
sdc
1 12.5T disk
└─ceph--block--44ae73e8--726f--4556--978c--8e7d6570c867-osd--block--daeb5218--c10c--45e1--a864--2d60de44e594
1 12.5T lvm
sdd
1 12.5T disk
└─ceph--block--38e361f5--257f--47a5--85dc--16dbdd5fb905-osd--block--a3e1511f--8644--4c1e--a3dd--f365fcb27fc6
1 12.5T lvm
sde
0 1.8T disk
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--f5d822a2--d42a--4a2c--985f--c65977f4d020
0 357.7G lvm
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--9ed8d8a4--11bc--4669--bb8f--d66806181a7d
0 357.7G lvm
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--0fdff890--f5e4--4762--83f6--8d02ee63c399
0 357.7G lvm
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--facaa59d--1f44--4605--bdb8--5a7d58271323
0 357.7G lvm
└─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--44fbc70f--4169--41aa--9c2f--6638b226065a
0 357.7G lvm
sdf
0 1.8T disk
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--7c82d5e2--6ce1--49c6--9691--2e156a8fd9c0
0 357.7G lvm
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--135169cd--e0c8--4949--a137--c5e7da12bc52
0 357.7G lvm
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--fcb5b6cc--ee95--4dfa--a978--0a768da8bc66
0 357.7G lvm
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--44b74f22--d2f1--485a--971b--d89a82849c6e
0 357.7G lvm
└─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--1dd5ee57--e141--40aa--b4a6--5cb20dc51cc0
0 357.7G lvm
sdg
0 1.8T disk
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--274c882c--da8b--4251--a195--309ee9cbc36f
0 357.7G lvm
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--087b2305--2bbe--4583--b1ca--dda7416efefc
0 357.7G lvm
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--3a97a94b--92bd--41d9--808a--b79aa210fe11
0 357.7G lvm
└─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--441521b4--52ce--4a03--b5b2--da9392d037bc
0 357.7G lvm
sdi
1 12.5T disk
└─ceph--block--1c122cd0--b28a--4409--b17b--535d95029dda-osd--block--9a6456d9--4c23--4585--a6df--85eda27ae651
1 12.5T lvm
sdj
1 12.5T disk
└─ceph--block--4cbe20eb--6aa3--4e4a--bdda--d556144e2a83-osd--block--2fd1bc6c--0eb3--48aa--b495--db0423b5be28
1 12.5T lvm
sdk
1 12.5T disk
└─ceph--block--3a5c3abd--77f6--4823--b313--1f5900470b8f-osd--block--c350620a--32f5--4da1--bda2--423fe3644f17
1 12.5T lvm
sdl
1 12.5T disk
└─ceph--block--6c7d3bea--a19c--4efc--9a36--dc38a06dad9c-osd--block--f23293d3--4ef8--4127--92c3--7235055797df
1 12.5T lvm
sdm
1 12.5T disk
└─ceph--block--7b48b8d4--cca2--4600--a046--7e1bbc03e1d7-osd--block--bfe62b63--f4fc--4f9d--9bc3--028cba0bacc6
1 12.5T lvm
sdn
1 12.5T disk
└─ceph--block--b99edf97--4162--43e6--ae2e--143250e4c4a8-osd--block--c6cd1240--a171--4ac8--8db4--e282d6fd63c3
1 12.5T lvm
sdo
1 12.5T disk
└─ceph--block--5c324c26--f569--4073--8915--b242a6b56e1c-osd--block--9ab9a615--a87b--40b3--90f1--660fa0da5c93
1 12.5T lvm
sdp
1 12.5T disk
└─ceph--block--2ae62d0b--5f71--446d--b1a9--083842227895-osd--block--6f80035f--59b3--4740--9504--c63be09201f6
1 12.5T lvm
sdq
1 12.5T disk
└─ceph--block--76711aee--28a0--4e19--a4da--74492ab56b77-osd--block--f587fb82--30b9--4133--a000--1b7f649c92aa
1 12.5T lvm
sdr
1 12.5T disk
└─ceph--block--bd6f560d--f196--448b--8bc0--840ea381e798-osd--block--37cc1a13--9f4b--4593--85ef--829b33f4c82f
1 12.5T lvm
sds
1 12.5T disk
└─ceph--block--f6826b43--6594--4b1c--ba0d--654d3075656a-osd--block--16b26a2d--8a7b--4c60--abf0--5412d8da1446
1 12.5T lvm
sdt
1 12.5T disk
My spec file is for a tiny lab cluster, in your case
the db drive size
should be something like '5T:6T' to specify a range.
How large are the HDDs?
The VG has 357.74GB of free space of total 5.24TB so I did actually
tried different values like "30G:", "30G", "300G:",
"300G", "357G".
I also tied some crazy high numbers and some ranges, but don't remember
the values. But none of them worked.
The HDD is 15x14TB and SSD 3x1.92TB.
As you can see from lsblk or pvs under, Cephadm created one VG
ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b from the 3 SSD and
created 15 LV on that VG, one for each HDD.
It only has 14 LV now since I zap the LV of the disk that died, the new
disk is /dev/sdt.
Here is also the output of pvs
# pvs
PV VG Fmt
Attr PSize PFree
/dev/sdb ceph-block-1b5ad7e7-2e24-4315-8a05-7439ab782b45 lvm2
a-- 12.47t 0
/dev/sdc ceph-block-44ae73e8-726f-4556-978c-8e7d6570c867 lvm2
a-- 12.47t 0
/dev/sdd ceph-block-38e361f5-257f-47a5-85dc-16dbdd5fb905 lvm2
a-- 12.47t 0
/dev/sde ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b lvm2
a-- <1.75t 16.00m
/dev/sdf ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b lvm2
a-- <1.75t 16.00m
/dev/sdg ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b lvm2
a-- <1.75t 357.71g
/dev/sdi ceph-block-1c122cd0-b28a-4409-b17b-535d95029dda lvm2
a-- 12.47t 0
/dev/sdj ceph-block-4cbe20eb-6aa3-4e4a-bdda-d556144e2a83 lvm2
a-- 12.47t 0
/dev/sdk ceph-block-3a5c3abd-77f6-4823-b313-1f5900470b8f lvm2
a-- 12.47t 0
/dev/sdl ceph-block-6c7d3bea-a19c-4efc-9a36-dc38a06dad9c lvm2
a-- 12.47t 0
/dev/sdm ceph-block-7b48b8d4-cca2-4600-a046-7e1bbc03e1d7 lvm2
a-- 12.47t 0
/dev/sdn ceph-block-b99edf97-4162-43e6-ae2e-143250e4c4a8 lvm2
a-- 12.47t 0
/dev/sdo ceph-block-5c324c26-f569-4073-8915-b242a6b56e1c lvm2
a-- 12.47t 0
/dev/sdp ceph-block-2ae62d0b-5f71-446d-b1a9-083842227895 lvm2
a-- 12.47t 0
/dev/sdq ceph-block-76711aee-28a0-4e19-a4da-74492ab56b77 lvm2
a-- 12.47t 0
/dev/sdr ceph-block-bd6f560d-f196-448b-8bc0-840ea381e798 lvm2
a-- 12.47t 0
/dev/sds ceph-block-f6826b43-6594-4b1c-ba0d-654d3075656a lvm2
a-- 12.47t 0
Also maybe you should use the option
'filter_logic: AND', but I'm not sure if that's already the default, I
remember that there were issues in Nautilus because the default was
OR. I tried this just recently with a version similar to this, I
believe it was 15.2.8 and it worked for me, but again, it's just a
tiny virtual lab cluster.
Yes, AND is default, I tried adding 'filter_logic: AND' but with the
same result.
In you virtual lab cluster do you have multiple HDD sharing the same SSD
as I do?
To me it looks like Cephadm can't find or use the 357.71GB free space on
the VG, it can only find devices that is available.
Here is how my "orch device ls" is for that host
$ ceph orch device ls --wide | egrep "Hostname|hd-7"
Hostname Path Type Vendor Model Size Available
Reject Reasons
pech-hd-7 /dev/sdt hdd WDC WUH721414AL5200 13.7T Yes
pech-hd-7 /dev/sdb hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sdc hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sdd hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sde ssd SAMSUNG MZILT1T9HAJQ0D3 1920G No
LVM detected, locked
pech-hd-7 /dev/sdf ssd SAMSUNG MZILT1T9HAJQ0D3 1920G No
LVM detected, locked
pech-hd-7 /dev/sdg ssd SAMSUNG MZILT1T9HAJQ0D3 1920G No
LVM detected, locked
pech-hd-7 /dev/sdi hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sdj hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sdk hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sdl hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sdm hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sdn hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sdo hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sdp hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sdq hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sdr hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sds hdd SEAGATE ST14000NM0168 13.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
--
Kai Stian Olstad
10:46 a.m.
New subject: cephadm: How to replace failed HDD where DB is on SSD
Hi,
The VG has 357.74GB of free space of total 5.24TB so I
did actually
tried different values like "30G:", "30G", "300G:",
"300G", "357G".
I also tied some crazy high numbers and some ranges, but don't
remember the values. But none of them worked.
the size parameter is filtering the disk size, not the size you want
the db to have (that's block_db_size). Your SSD disk size is 1.8 TB so
your specs could look something like this:
block_db_size: 360G
data_devices:
size: "12T:"
rotational: 1
db_devices:
size: ":2T"
rotational: 0
filter_logic: AND
...
But I was under the impression that this all should of course work
with just the rotational flags, I'm confused that it doesn't. Can you
try with these specs to see if you get the OSD deployed? I'll try
again with Octopus to see if I see similar behaviour.
Zitat von Kai Stian Olstad <ceph+list(a)olstad.com>om>:
> On 26.05.2021 18:12, Eugen Block wrote:
>> Could you share the output of
>>
>> lsblk -o name,rota,size,type
>>
>> from the affected osd node?
>
> # lsblk -o name,rota,size,type
> NAME
> ROTA SIZE TYPE
> loop0
> 1 71.3M loop
> loop1
> 1 55M loop
> loop2
> 1 29.9M loop
> sda
> 1 223G disk
> ├─sda1
> 1 512M part
> ├─sda2
> 1 1G part
> └─sda3
> 1 221.5G part
> sdb
> 1 12.5T disk
>
└─ceph--block--1b5ad7e7--2e24--4315--8a05--7439ab782b45-osd--block--2da790bc--a74c--41da--8772--3b8aac77001c
1 12.5T
> lvm
> sdc
> 1 12.5T disk
>
└─ceph--block--44ae73e8--726f--4556--978c--8e7d6570c867-osd--block--daeb5218--c10c--45e1--a864--2d60de44e594
1 12.5T
> lvm
> sdd
> 1 12.5T disk
>
└─ceph--block--38e361f5--257f--47a5--85dc--16dbdd5fb905-osd--block--a3e1511f--8644--4c1e--a3dd--f365fcb27fc6
1 12.5T
> lvm
> sde
> 0 1.8T disk
>
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--f5d822a2--d42a--4a2c--985f--c65977f4d020
0 357.7G
> lvm
>
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--9ed8d8a4--11bc--4669--bb8f--d66806181a7d
0 357.7G
> lvm
>
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--0fdff890--f5e4--4762--83f6--8d02ee63c399
0 357.7G
> lvm
>
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--facaa59d--1f44--4605--bdb8--5a7d58271323
0 357.7G
> lvm
>
└─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--44fbc70f--4169--41aa--9c2f--6638b226065a
0 357.7G
> lvm
> sdf
> 0 1.8T disk
>
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--7c82d5e2--6ce1--49c6--9691--2e156a8fd9c0
0 357.7G
> lvm
>
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--135169cd--e0c8--4949--a137--c5e7da12bc52
0 357.7G
> lvm
>
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--fcb5b6cc--ee95--4dfa--a978--0a768da8bc66
0 357.7G
> lvm
>
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--44b74f22--d2f1--485a--971b--d89a82849c6e
0 357.7G
> lvm
>
└─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--1dd5ee57--e141--40aa--b4a6--5cb20dc51cc0
0 357.7G
> lvm
> sdg
> 0 1.8T disk
>
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--274c882c--da8b--4251--a195--309ee9cbc36f
0 357.7G
> lvm
>
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--087b2305--2bbe--4583--b1ca--dda7416efefc
0 357.7G
> lvm
>
├─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--3a97a94b--92bd--41d9--808a--b79aa210fe11
0 357.7G
> lvm
>
└─ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--441521b4--52ce--4a03--b5b2--da9392d037bc
0 357.7G
> lvm
> sdi
> 1 12.5T disk
>
└─ceph--block--1c122cd0--b28a--4409--b17b--535d95029dda-osd--block--9a6456d9--4c23--4585--a6df--85eda27ae651
1 12.5T
> lvm
> sdj
> 1 12.5T disk
>
└─ceph--block--4cbe20eb--6aa3--4e4a--bdda--d556144e2a83-osd--block--2fd1bc6c--0eb3--48aa--b495--db0423b5be28
1 12.5T
> lvm
> sdk
> 1 12.5T disk
>
└─ceph--block--3a5c3abd--77f6--4823--b313--1f5900470b8f-osd--block--c350620a--32f5--4da1--bda2--423fe3644f17
1 12.5T
> lvm
> sdl
> 1 12.5T disk
>
└─ceph--block--6c7d3bea--a19c--4efc--9a36--dc38a06dad9c-osd--block--f23293d3--4ef8--4127--92c3--7235055797df
1 12.5T
> lvm
> sdm
> 1 12.5T disk
>
└─ceph--block--7b48b8d4--cca2--4600--a046--7e1bbc03e1d7-osd--block--bfe62b63--f4fc--4f9d--9bc3--028cba0bacc6
1 12.5T
> lvm
> sdn
> 1 12.5T disk
>
└─ceph--block--b99edf97--4162--43e6--ae2e--143250e4c4a8-osd--block--c6cd1240--a171--4ac8--8db4--e282d6fd63c3
1 12.5T
> lvm
> sdo
> 1 12.5T disk
>
└─ceph--block--5c324c26--f569--4073--8915--b242a6b56e1c-osd--block--9ab9a615--a87b--40b3--90f1--660fa0da5c93
1 12.5T
> lvm
> sdp
> 1 12.5T disk
>
└─ceph--block--2ae62d0b--5f71--446d--b1a9--083842227895-osd--block--6f80035f--59b3--4740--9504--c63be09201f6
1 12.5T
> lvm
> sdq
> 1 12.5T disk
>
└─ceph--block--76711aee--28a0--4e19--a4da--74492ab56b77-osd--block--f587fb82--30b9--4133--a000--1b7f649c92aa
1 12.5T
> lvm
> sdr
> 1 12.5T disk
>
└─ceph--block--bd6f560d--f196--448b--8bc0--840ea381e798-osd--block--37cc1a13--9f4b--4593--85ef--829b33f4c82f
1 12.5T
> lvm
> sds
> 1 12.5T disk
>
└─ceph--block--f6826b43--6594--4b1c--ba0d--654d3075656a-osd--block--16b26a2d--8a7b--4c60--abf0--5412d8da1446
1 12.5T
> lvm
> sdt
> 1 12.5T disk
>
>
>> My spec file is for a tiny lab cluster, in your case the db drive size
>> should be something like '5T:6T' to specify a range.
>> How large are the HDDs?
>
The VG has 357.74GB of free space of total 5.24TB so I
did actually
tried different values like "30G:", "30G", "300G:",
"300G", "357G".
I also tied some crazy high numbers and some ranges, but don't
remember the values. But none of them worked.
>
> The HDD is 15x14TB and SSD 3x1.92TB.
> As you can see from lsblk or pvs under, Cephadm created one VG
> ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b from the 3 SSD
> and created 15 LV on that VG, one for each HDD.
> It only has 14 LV now since I zap the LV of the disk that died, the
> new disk is /dev/sdt.
>
> Here is also the output of pvs
> # pvs
> PV VG Fmt
> Attr PSize PFree
> /dev/sdb ceph-block-1b5ad7e7-2e24-4315-8a05-7439ab782b45
> lvm2 a-- 12.47t 0
> /dev/sdc ceph-block-44ae73e8-726f-4556-978c-8e7d6570c867
> lvm2 a-- 12.47t 0
> /dev/sdd ceph-block-38e361f5-257f-47a5-85dc-16dbdd5fb905
> lvm2 a-- 12.47t 0
> /dev/sde ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b
> lvm2 a-- <1.75t 16.00m
> /dev/sdf ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b
> lvm2 a-- <1.75t 16.00m
> /dev/sdg ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b
> lvm2 a-- <1.75t 357.71g
> /dev/sdi ceph-block-1c122cd0-b28a-4409-b17b-535d95029dda
> lvm2 a-- 12.47t 0
> /dev/sdj ceph-block-4cbe20eb-6aa3-4e4a-bdda-d556144e2a83
> lvm2 a-- 12.47t 0
> /dev/sdk ceph-block-3a5c3abd-77f6-4823-b313-1f5900470b8f
> lvm2 a-- 12.47t 0
> /dev/sdl ceph-block-6c7d3bea-a19c-4efc-9a36-dc38a06dad9c
> lvm2 a-- 12.47t 0
> /dev/sdm ceph-block-7b48b8d4-cca2-4600-a046-7e1bbc03e1d7
> lvm2 a-- 12.47t 0
> /dev/sdn ceph-block-b99edf97-4162-43e6-ae2e-143250e4c4a8
> lvm2 a-- 12.47t 0
> /dev/sdo ceph-block-5c324c26-f569-4073-8915-b242a6b56e1c
> lvm2 a-- 12.47t 0
> /dev/sdp ceph-block-2ae62d0b-5f71-446d-b1a9-083842227895
> lvm2 a-- 12.47t 0
> /dev/sdq ceph-block-76711aee-28a0-4e19-a4da-74492ab56b77
> lvm2 a-- 12.47t 0
> /dev/sdr ceph-block-bd6f560d-f196-448b-8bc0-840ea381e798
> lvm2 a-- 12.47t 0
> /dev/sds ceph-block-f6826b43-6594-4b1c-ba0d-654d3075656a
> lvm2 a-- 12.47t 0
>
>
>> Also maybe you should use the option
>> 'filter_logic: AND', but I'm not sure if that's already the
default, I
>> remember that there were issues in Nautilus because the default was
>> OR. I tried this just recently with a version similar to this, I
>> believe it was 15.2.8 and it worked for me, but again, it's just a
>> tiny virtual lab cluster.
>
> Yes, AND is default, I tried adding 'filter_logic: AND' but with the
> same result.
>
> In you virtual lab cluster do you have multiple HDD sharing the same
> SSD as I do?
>
> To me it looks like Cephadm can't find or use the 357.71GB free
> space on the VG, it can only find devices that is available.
> Here is how my "orch device ls" is for that host
>
> $ ceph orch device ls --wide | egrep "Hostname|hd-7"
> Hostname Path Type Vendor Model Size
> Available Reject Reasons
> pech-hd-7 /dev/sdt hdd WDC WUH721414AL5200 13.7T Yes
> pech-hd-7 /dev/sdb hdd SEAGATE ST14000NM0168 13.7T No
> Insufficient space (<10 extents) on vgs, LVM detected, locked
> pech-hd-7 /dev/sdc hdd SEAGATE ST14000NM0168 13.7T No
> Insufficient space (<10 extents) on vgs, LVM detected, locked
> pech-hd-7 /dev/sdd hdd SEAGATE ST14000NM0168 13.7T No
> Insufficient space (<10 extents) on vgs, LVM detected, locked
> pech-hd-7 /dev/sde ssd SAMSUNG MZILT1T9HAJQ0D3 1920G No
> LVM detected, locked
> pech-hd-7 /dev/sdf ssd SAMSUNG MZILT1T9HAJQ0D3 1920G No
> LVM detected, locked
> pech-hd-7 /dev/sdg ssd SAMSUNG MZILT1T9HAJQ0D3 1920G No
> LVM detected, locked
> pech-hd-7 /dev/sdi hdd SEAGATE ST14000NM0168 13.7T No
> Insufficient space (<10 extents) on vgs, LVM detected, locked
> pech-hd-7 /dev/sdj hdd SEAGATE ST14000NM0168 13.7T No
> Insufficient space (<10 extents) on vgs, LVM detected, locked
> pech-hd-7 /dev/sdk hdd SEAGATE ST14000NM0168 13.7T No
> Insufficient space (<10 extents) on vgs, LVM detected, locked
> pech-hd-7 /dev/sdl hdd SEAGATE ST14000NM0168 13.7T No
> Insufficient space (<10 extents) on vgs, LVM detected, locked
> pech-hd-7 /dev/sdm hdd SEAGATE ST14000NM0168 13.7T No
> Insufficient space (<10 extents) on vgs, LVM detected, locked
> pech-hd-7 /dev/sdn hdd SEAGATE ST14000NM0168 13.7T No
> Insufficient space (<10 extents) on vgs, LVM detected, locked
> pech-hd-7 /dev/sdo hdd SEAGATE ST14000NM0168 13.7T No
> Insufficient space (<10 extents) on vgs, LVM detected, locked
> pech-hd-7 /dev/sdp hdd SEAGATE ST14000NM0168 13.7T No
> Insufficient space (<10 extents) on vgs, LVM detected, locked
> pech-hd-7 /dev/sdq hdd SEAGATE ST14000NM0168 13.7T No
> Insufficient space (<10 extents) on vgs, LVM detected, locked
> pech-hd-7 /dev/sdr hdd SEAGATE ST14000NM0168 13.7T No
> Insufficient space (<10 extents) on vgs, LVM detected, locked
> pech-hd-7 /dev/sds hdd SEAGATE ST14000NM0168 13.7T No
> Insufficient space (<10 extents) on vgs, LVM detected, locked
>
> --
> Kai Stian Olstad
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:12 a.m.
New subject: cephadm: How to replace failed HDD where DB is on SSD
On 27.05.2021 10:46, Eugen Block wrote:
Hi,
I tried this one
hdd-test-from-eugen.yml
-----------------------
service_type: osd
service_id: hdd
placement:
host_pattern: 'pech-hd-*'
block_db_size: 360G
data_devices:
size: "12T:"
rotational: 1
db_devices:
size: ":2T"
rotational: 0
filter_logic: AND
But it doesn't find any disk.
I also tried this, but with the same result.
service_type: osd
service_id: hdd
placement:
host_pattern: 'pech-hd-*'
block_db_size: 360G
data_devices:
rotational: 1
db_devices:
rotational: 0
filter_logic: AND
The VG has 357.74GB of free space of total 5.24TB
so I did actually
tried different values like "30G:", "30G", "300G:",
"300G", "357G".
I also tied some crazy high numbers and some ranges, but don't
remember the values. But none of them worked.
the size parameter is filtering the disk size, not the size you want
the db to have (that's block_db_size). Your SSD disk size is 1.8 TB so
your specs could look something like this:
block_db_size: 360G
data_devices:
size: "12T:"
rotational: 1
db_devices:
size: ":2T"
rotational: 0
filter_logic: AND
...
But I was under the impression that this all should of course work
with just the rotational flags, I'm confused that it doesn't. Can you
try with these specs to see if you get the OSD deployed? I'll try again with Octopus to see if I see
similar behaviour.
Very much appreciated, thanks.
--
Kai Stian Olstad
1074
days inactive
1075
days old
3 comments
2 participants
participants (2)
-
Eugen Block -
Kai Stian Olstad