25 May
2021
25 May
'21
2:50 p.m.
Hi
The server run 15.2.9 and has 15 HDD and 3 SSD.
The OSDs was created with this YAML file
hdd.yml
--------
service_type: osd
service_id: hdd
placement:
host_pattern: 'pech-hd-*'
data_devices:
rotational: 1
db_devices:
rotational: 0
The result was that the 3 SSD is added to 1 VG with 15 LV on it.
# vgs | egrep "VG|dbs"
VG #PV #LV #SN Attr
VSize VFree
ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b 3 15 0 wz--n-
<5.24t 48.00m
One of the osd failed and I run rm with replace
# ceph orch osd rm 178 --replace
and the result is
# ceph osd tree | grep "ID|destroyed"
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT
PRI-AFF
178 hdd 12.82390 osd.178 destroyed 0
1.00000
But I'm not able to replace the disk with the same YAML file as shown
above.
# ceph orch apply osd -i hdd.yml --dry-run
################
OSDSPEC PREVIEWS
################
+---------+------+------+------+----+-----+
|SERVICE |NAME |HOST |DATA |DB |WAL |
+---------+------+------+------+----+-----+
+---------+------+------+------+----+-----+
I guess this is the wrong way to do it, but I can't find the answer in
the documentation.
So how can I replace this failed disk in Cephadm?
--
Kai Stian Olstad
26 May
26 May
9:22 a.m.
Hi,
did you wipe the LV on the SSD that was assigned to the failed HDD? I
just did that on a fresh Pacific install successfully, a couple of
weeks ago it also worked on an Octopus cluster. Note that I have a few
filters in my specs file but that shouldn't make a difference, I
believe.
pacific1:~ # cat osd-specs.yml
block_db_size: 4G
data_devices:
size: "10G:"
rotational: 1
db_devices:
size: "20G:"
rotational: 0
filter_logic: AND
service_id: ssd-hdd-mix
service_type: osd
service_name: osd.with.ssd.db
placement:
hosts:
- pacific1
- pacific2
Zitat von Kai Stian Olstad <ceph+list(a)olstad.com>om>:
Hi
The server run 15.2.9 and has 15 HDD and 3 SSD.
The OSDs was created with this YAML file
hdd.yml
--------
service_type: osd
service_id: hdd
placement:
host_pattern: 'pech-hd-*'
data_devices:
rotational: 1
db_devices:
rotational: 0
The result was that the 3 SSD is added to 1 VG with 15 LV on it.
# vgs | egrep "VG|dbs"
VG #PV #LV #SN
Attr VSize VFree
ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b 3 15 0
wz--n- <5.24t 48.00m
One of the osd failed and I run rm with replace
# ceph orch osd rm 178 --replace
and the result is
# ceph osd tree | grep "ID|destroyed"
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
178 hdd 12.82390 osd.178 destroyed 0 1.00000
But I'm not able to replace the disk with the same YAML file as shown above.
# ceph orch apply osd -i hdd.yml --dry-run
################
OSDSPEC PREVIEWS
################
+---------+------+------+------+----+-----+
|SERVICE |NAME |HOST |DATA |DB |WAL |
+---------+------+------+------+----+-----+
+---------+------+------+------+----+-----+
I guess this is the wrong way to do it, but I can't find the answer
in the documentation.
So how can I replace this failed disk in Cephadm?
--
Kai Stian Olstad
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
12:09 p.m.
On 26.05.2021 08:22, Eugen Block wrote:
Hi,
did you wipe the LV on the SSD that was assigned to the failed HDD? I
just did that on a fresh Pacific install successfully, a couple of
weeks ago it also worked on an Octopus cluster.
No, I did not wipe the LV.
Not sure what you mean by wipe, so I tried overwriting the LV with
/dev/zero, but that did solve it.
So I guess with wipe do you mean delete the LV with lvremove?
--
Kai Stian Olstad
12:16 p.m.
Yes, the LVs are not removed automatically, you need to free up the
VG, there are a couple of ways to do so, for example remotely:
pacific1:~ # ceph orch device zap pacific4 /dev/vdb --force
or directly on the host with:
pacific1:~ # cephadm ceph-volume lvm zap --destroy /dev/<CEPH_VG>/<DB_LV>
Zitat von Kai Stian Olstad <ceph+list(a)olstad.com>om>:
On 26.05.2021 08:22, Eugen Block wrote:
Hi,
did you wipe the LV on the SSD that was assigned to the failed HDD? I
just did that on a fresh Pacific install successfully, a couple of
weeks ago it also worked on an Octopus cluster.
No, I did not wipe the LV.
Not sure what you mean by wipe, so I tried overwriting the LV with
/dev/zero, but that did solve it.
So I guess with wipe do you mean delete the LV with lvremove?
--
Kai Stian Olstad
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
4:41 p.m.
On 26.05.2021 11:16, Eugen Block wrote:
Yes, the LVs are not removed automatically, you need
to free up the
VG, there are a couple of ways to do so, for example remotely:
pacific1:~ # ceph orch device zap pacific4 /dev/vdb --force
or directly on the host with:
pacific1:~ # cephadm ceph-volume lvm zap --destroy
/dev/<CEPH_VG>/<DB_LV>
Thanks,
I used the cephadm command and deleted the LV and the VG now has free
space
# vgs | egrep "VG|dbs"
VG #PV #LV #SN Attr
VSize VFree
ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b 3 14 0 wz--n-
<5.24t 357.74g
But it doesn't seams to be able to use it, because it can find anyting
# ceph orch apply osd -i hdd.yml --dry-run
################
OSDSPEC PREVIEWS
################
+---------+------+-------------+----------+----+-----+
|SERVICE |NAME |HOST |DATA |DB |WAL |
+---------+------+-------------+----------+----+-----+
+---------+------+-------------+----------+----+-----+
I tried adding size as you have in your configuration
db_devices:
rotational: 0
size: '30G:'
Still it was unable to create the OSD.
If I removed the : so it is 30GB exact size, it did find the disk, but
DB is not placed on a SSD since I do not have one with 30 GB exact size
################
OSDSPEC PREVIEWS
################
+---------+------+-------------+----------+----+-----+
|SERVICE |NAME |HOST |DATA |DB |WAL |
+---------+------+-------------+----------+----+-----+
|osd |hdd |pech-hd-7 |/dev/sdt |- |- |
+---------+------+-------------+----------+----+-----+
To me I looks like Cephadm can't use/find the free space on the VG and
use that as a new LV for the OSD.
--
Kai Stian Olstad
11:14 p.m.
We've found that after doing the osd rm, you can use: "ceph-volume lvm
zap --osd-id 178 --destroy" on the server with that OSD as per:
https://docs.ceph.com/en/latest/ceph-volume/lvm/zap/#removing-devices
and it will clean things up so they work as expected.
On Tue, May 25, 2021 at 6:51 AM Kai Stian Olstad <ceph+list(a)olstad.com> wrote:
>
> Hi
>
> The server run 15.2.9 and has 15 HDD and 3 SSD.
> The OSDs was created with this YAML file
>
> hdd.yml
> --------
> service_type: osd
> service_id: hdd
> placement:
> host_pattern: 'pech-hd-*'
> data_devices:
> rotational: 1
> db_devices:
> rotational: 0
>
>
> The result was that the 3 SSD is added to 1 VG with 15 LV on it.
>
> # vgs | egrep "VG|dbs"
> VG #PV #LV #SN Attr
> VSize VFree
> ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b 3 15 0 wz--n-
> <5.24t 48.00m
>
>
> One of the osd failed and I run rm with replace
>
> # ceph orch osd rm 178 --replace
>
> and the result is
>
> # ceph osd tree | grep "ID|destroyed"
> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT
> PRI-AFF
> 178 hdd 12.82390 osd.178 destroyed 0
> 1.00000
>
>
> But I'm not able to replace the disk with the same YAML file as shown
> above.
>
>
> # ceph orch apply osd -i hdd.yml --dry-run
> ################
> OSDSPEC PREVIEWS
> ################
> +---------+------+------+------+----+-----+
> |SERVICE |NAME |HOST |DATA |DB |WAL |
> +---------+------+------+------+----+-----+
> +---------+------+------+------+----+-----+
>
> I guess this is the wrong way to do it, but I can't find the answer in
> the documentation.
> So how can I replace this failed disk in Cephadm?
>
>
> --
> Kai Stian Olstad
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
27 May
27 May
11:44 a.m.
On 26.05.2021 22:14, David Orman wrote:
We've found that after doing the osd rm, you can
use: "ceph-volume lvm
zap --osd-id 178 --destroy" on the server with that OSD as per:
https://docs.ceph.com/en/latest/ceph-volume/lvm/zap/#removing-devices
and it will clean things up so they work as expected.
With the help of Eugen I did run "cephadm ceph-volume lvm zap --destroy
<LV>" and the LV is gone.
I think that is the same result as "ceph-volume lvm zap --osd-id 178
--destroy" would give me?
I now have 357GB free space on the VG, but Cephadm doesn't find and use
this space.
Above it the result of the zap command and it show the LV is deleted.
$ sudo cephadm ceph-volume lvm zap --destroy
/dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69
INFO:cephadm:Inferring fsid 3614abcc-201c-11eb-995a-2794bcc75ae0
INFO:cephadm:Using recent ceph image ceph:v15.2.9
INFO:cephadm:/usr/bin/podman:stderr --> Zapping:
/dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69
INFO:cephadm:/usr/bin/podman:stderr Running command: /usr/bin/dd
if=/dev/zero
of=/dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69
bs=1M count=10 conv=fsync
INFO:cephadm:/usr/bin/podman:stderr stderr: 10+0 records in
INFO:cephadm:/usr/bin/podman:stderr 10+0 records out
INFO:cephadm:/usr/bin/podman:stderr stderr: 10485760 bytes (10 MB, 10
MiB) copied, 0.0195532 s, 536 MB/s
INFO:cephadm:/usr/bin/podman:stderr --> More than 1 LV left in VG, will
proceed to destroy LV only
INFO:cephadm:/usr/bin/podman:stderr --> Removing LV because --destroy
was given:
/dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69
INFO:cephadm:/usr/bin/podman:stderr Running command: /usr/sbin/lvremove
-v -f
/dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69
INFO:cephadm:/usr/bin/podman:stderr stdout: Logical volume
"osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69" successfully removed
INFO:cephadm:/usr/bin/podman:stderr stderr: Removing
ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--449bd001--eb32--46de--ab80--a1cbcd293d69
(253:3)
INFO:cephadm:/usr/bin/podman:stderr stderr: Archiving volume group
"ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b" metadata (seqno
61).
INFO:cephadm:/usr/bin/podman:stderr stderr: Releasing logical volume
"osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69"
INFO:cephadm:/usr/bin/podman:stderr stderr: Creating volume group
backup
"/etc/lvm/backup/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b"
(seqno 62).
INFO:cephadm:/usr/bin/podman:stderr --> Zapping successful for: <LV:
/dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69>
--
Kai Stian Olstad
12:17 p.m.
That's not how it's supposed to work. I tried the same on an Octopus
cluster and removed all filters except:
data_devices:
rotational: 1
db_devices:
rotational: 0
My Octopus test osd nodes have two HDDs and one SSD, I removed all
OSDs and redeployed on one node. This spec file results in three
standalone OSDs! Without the other filters this won't work as
expected, it seems. I'll try again on Pacific with the same test and
see where that goes.
Zitat von Kai Stian Olstad <ceph+list(a)olstad.com>om>:
On 26.05.2021 22:14, David Orman wrote:
We've found that after doing the osd rm, you
can use: "ceph-volume lvm
zap --osd-id 178 --destroy" on the server with that OSD as per:
https://docs.ceph.com/en/latest/ceph-volume/lvm/zap/#removing-devices
and it will clean things up so they work as expected.
With the help of Eugen I did run "cephadm ceph-volume lvm zap
--destroy <LV>" and the LV is gone.
I think that is the same result as "ceph-volume lvm zap --osd-id 178
--destroy" would give me?
I now have 357GB free space on the VG, but Cephadm doesn't find and
use this space.
Above it the result of the zap command and it show the LV is deleted.
$ sudo cephadm ceph-volume lvm zap --destroy
/dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69
INFO:cephadm:Inferring fsid 3614abcc-201c-11eb-995a-2794bcc75ae0
INFO:cephadm:Using recent ceph image ceph:v15.2.9
INFO:cephadm:/usr/bin/podman:stderr --> Zapping:
/dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69
INFO:cephadm:/usr/bin/podman:stderr Running command: /usr/bin/dd
if=/dev/zero
of=/dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69
bs=1M count=10
conv=fsync
INFO:cephadm:/usr/bin/podman:stderr stderr: 10+0 records in
INFO:cephadm:/usr/bin/podman:stderr 10+0 records out
INFO:cephadm:/usr/bin/podman:stderr stderr: 10485760 bytes (10 MB,
10 MiB) copied, 0.0195532 s, 536 MB/s
INFO:cephadm:/usr/bin/podman:stderr --> More than 1 LV left in VG,
will proceed to destroy LV only
INFO:cephadm:/usr/bin/podman:stderr --> Removing LV because
--destroy was given:
/dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69
INFO:cephadm:/usr/bin/podman:stderr Running command:
/usr/sbin/lvremove -v -f
/dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69
INFO:cephadm:/usr/bin/podman:stderr stdout: Logical volume
"osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69" successfully
removed
INFO:cephadm:/usr/bin/podman:stderr stderr: Removing
ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--449bd001--eb32--46de--ab80--a1cbcd293d69
(253:3)
INFO:cephadm:/usr/bin/podman:stderr stderr: Archiving volume group
"ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b" metadata
(seqno 61).
INFO:cephadm:/usr/bin/podman:stderr stderr: Releasing logical
volume "osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69"
INFO:cephadm:/usr/bin/podman:stderr stderr: Creating volume group
backup
"/etc/lvm/backup/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b" (seqno
62).
INFO:cephadm:/usr/bin/podman:stderr --> Zapping successful for: <LV:
/dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69>
--
Kai Stian Olstad
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
12:32 p.m.
On 27.05.2021 11:17, Eugen Block wrote:
That's not how it's supposed to work. I tried
the same on an Octopus
cluster and removed all filters except:
data_devices:
rotational: 1
db_devices:
rotational: 0
My Octopus test osd nodes have two HDDs and one SSD, I removed all
OSDs and redeployed on one node. This spec file results in three
standalone OSDs! Without the other filters this won't work as
expected, it seems. I'll try again on Pacific with the same test and
see where that goes.
This spec did worked for me when I initially deployed with Octopus
15.2.5.
--
Kai Stian Olstad
12:53 p.m.
This test was on ceph version 15.2.8.
On Pacific (ceph version 16.2.4) this also works for me for initial
deployment of an entire host:
+---------+-------------+----------+----------+----------+-----+
|SERVICE |NAME |HOST |DATA |DB |WAL |
+---------+-------------+----------+----------+----------+-----+
|osd |ssd-hdd-mix |pacific1 |/dev/vdb |/dev/vdd |- |
|osd |ssd-hdd-mix |pacific1 |/dev/vdc |/dev/vdd |- |
+---------+-------------+----------+----------+----------+-----+
But it doesn't work if I remove one OSD, just like you describe. This
is what ceph-volume reports:
---snip---
[ceph: root@pacific1 /]# ceph-volume lvm batch --report /dev/vdc
--db-devices /dev/vdd --block-db-size 3G
--> passed data devices: 1 physical, 0 LVM
--> relative data size: 1.0
--> passed block_db devices: 1 physical, 0 LVM
--> 1 fast devices were passed, but none are available
Total OSDs: 0
Type Path
LV Size % of device
---snip---
I know that this has already worked in Octopus, I did test it
successfully not long ago.
Zitat von Kai Stian Olstad <ceph+list(a)olstad.com>om>:
On 27.05.2021 11:17, Eugen Block wrote:
That's not how it's supposed to work. I
tried the same on an Octopus
cluster and removed all filters except:
data_devices:
rotational: 1
db_devices:
rotational: 0
My Octopus test osd nodes have two HDDs and one SSD, I removed all
OSDs and redeployed on one node. This spec file results in three
standalone OSDs! Without the other filters this won't work as
expected, it seems. I'll try again on Pacific with the same test and
see where that goes.
This spec did worked for me when I initially deployed with Octopus 15.2.5.
--
Kai Stian Olstad
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:20 p.m.
On 27.05.2021 11:53, Eugen Block wrote:
This test was on ceph version 15.2.8.
On Pacific (ceph version 16.2.4) this also works for me for initial
deployment of an entire host:
+---------+-------------+----------+----------+----------+-----+
|SERVICE |NAME |HOST |DATA |DB |WAL |
+---------+-------------+----------+----------+----------+-----+
|osd |ssd-hdd-mix |pacific1 |/dev/vdb |/dev/vdd |- |
|osd |ssd-hdd-mix |pacific1 |/dev/vdc |/dev/vdd |- |
+---------+-------------+----------+----------+----------+-----+
But it doesn't work if I remove one OSD, just like you describe. This
is what ceph-volume reports:
---snip---
[ceph: root@pacific1 /]# ceph-volume lvm batch --report /dev/vdc
--db-devices /dev/vdd --block-db-size 3G
--> passed data devices: 1 physical, 0 LVM
--> relative data size: 1.0
--> passed block_db devices: 1 physical, 0 LVM
--> 1 fast devices were passed, but none are available
Total OSDs: 0
Type Path
LV Size % of device
---snip---
I know that this has already worked in Octopus, I did test it
successfully not long ago.
Thank you for trying, so it looks like a bug.
Searching through the issue tracker I find few issues related to
replacing OSD, but it doesn't look like they get much attention.
I tried to find a way to add the disk manually, did not find any
documentation about it, but looking at the source code, some issues with
some trial and error I ended up with this.
Since the LV is deleted I recreated it with the same name.
# lvcreate -l 91570 -n osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69
ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b
In "cephadm shell"
# cephadm shell
# ceph auth get client.bootstrap-osd
/var/lib/ceph/bootstrap-osd/ceph.keyring
#
ceph-volume lvm prepare --bluestore --no-systemd --data /dev/sdt
--block.db
ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69
Need to have a json file for the "cephadm deploy"
# printf '{\n"config": "%s",\n"keyring":
"%s"\n}\n' "$(ceph config
generate-minimal-conf | sed -e ':a;N;$!ba;s/\n/\\n/g' -e 's/\t/\\t/g' -e
's/$/\\n/')" "$(ceph auth get osd.178 | head -n 2 | sed -e
':a;N;$!ba;s/\n/\\n/g' -e 's/\t/\\t/g' -e 's/$/\\n/')"
config-osd.178.json
Exit cephadm shell and run
# cephadm --image ceph:v15.2.9 deploy --fsid
3614abcc-201c-11eb-995a-2794bcc75ae0 --config-json
/var/lib/ceph/3614abcc-201c-11eb-995a-2794bcc75ae0/home/config-osd.178.json
--osd-fsid 9227e8ae-92eb-429e-9c7f-d4a2b75afb8e
And the OSD is back, but the VG name on the HDD is missing block in it's
name, just a cosmetic thing so I leave it as is.
LV VG
Attr LSize
osd-block-9227e8ae-92eb-429e-9c7f-d4a2b75afb8e
ceph-46f42262-d3dc-4dc3-8952-eec3e4a2c178 -wi-ao---- 12.47t
osd-block-2da790bc-a74c-41da-8772-3b8aac77001c
ceph-block-1b5ad7e7-2e24-4315-8a05-7439ab782b45 -wi-ao---- 12.47t
The fist one is the new OSD and the second one is one that cephadm
itself created.
--
Kai Stian Olstad
1071
days inactive
1073
days old
10 comments
3 participants
participants (3)
-
David Orman -
Eugen Block -
Kai Stian Olstad