cephadm: How to replace failed HDD where DB is on SSD
Hi The server run 15.2.9 and has 15 HDD and 3 SSD. The OSDs was created with this YAML file hdd.yml -------- service_type: osd service_id: hdd placement: host_pattern: 'pech-hd-*' data_devices: rotational: 1 db_devices: rotational: 0 The result was that the 3 SSD is added to 1 VG with 15 LV on it. # vgs | egrep "VG|dbs" VG #PV #LV #SN Attr VSize VFree ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b 3 15 0 wz--n- <5.24t 48.00m One of the osd failed and I run rm with replace # ceph orch osd rm 178 --replace and the result is # ceph osd tree | grep "ID|destroyed" ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 178 hdd 12.82390 osd.178 destroyed 0 1.00000 But I'm not able to replace the disk with the same YAML file as shown above. # ceph orch apply osd -i hdd.yml --dry-run ################ OSDSPEC PREVIEWS ################ +---------+------+------+------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+------+------+------+----+-----+ +---------+------+------+------+----+-----+ I guess this is the wrong way to do it, but I can't find the answer in the documentation. So how can I replace this failed disk in Cephadm? -- Kai Stian Olstad
Hi, did you wipe the LV on the SSD that was assigned to the failed HDD? I just did that on a fresh Pacific install successfully, a couple of weeks ago it also worked on an Octopus cluster. Note that I have a few filters in my specs file but that shouldn't make a difference, I believe. pacific1:~ # cat osd-specs.yml block_db_size: 4G data_devices: size: "10G:" rotational: 1 db_devices: size: "20G:" rotational: 0 filter_logic: AND service_id: ssd-hdd-mix service_type: osd service_name: osd.with.ssd.db placement: hosts: - pacific1 - pacific2 Zitat von Kai Stian Olstad <ceph+list@olstad.com>:
Hi
The server run 15.2.9 and has 15 HDD and 3 SSD. The OSDs was created with this YAML file
hdd.yml -------- service_type: osd service_id: hdd placement: host_pattern: 'pech-hd-*' data_devices: rotational: 1 db_devices: rotational: 0
The result was that the 3 SSD is added to 1 VG with 15 LV on it.
# vgs | egrep "VG|dbs" VG #PV #LV #SN Attr VSize VFree ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b 3 15 0 wz--n- <5.24t 48.00m
One of the osd failed and I run rm with replace
# ceph orch osd rm 178 --replace
and the result is
# ceph osd tree | grep "ID|destroyed" ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 178 hdd 12.82390 osd.178 destroyed 0 1.00000
But I'm not able to replace the disk with the same YAML file as shown above.
# ceph orch apply osd -i hdd.yml --dry-run ################ OSDSPEC PREVIEWS ################ +---------+------+------+------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+------+------+------+----+-----+ +---------+------+------+------+----+-----+
I guess this is the wrong way to do it, but I can't find the answer in the documentation. So how can I replace this failed disk in Cephadm?
-- Kai Stian Olstad _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 26.05.2021 08:22, Eugen Block wrote:
Hi,
did you wipe the LV on the SSD that was assigned to the failed HDD? I just did that on a fresh Pacific install successfully, a couple of weeks ago it also worked on an Octopus cluster.
No, I did not wipe the LV. Not sure what you mean by wipe, so I tried overwriting the LV with /dev/zero, but that did solve it. So I guess with wipe do you mean delete the LV with lvremove? -- Kai Stian Olstad
Yes, the LVs are not removed automatically, you need to free up the VG, there are a couple of ways to do so, for example remotely: pacific1:~ # ceph orch device zap pacific4 /dev/vdb --force or directly on the host with: pacific1:~ # cephadm ceph-volume lvm zap --destroy /dev/<CEPH_VG>/<DB_LV> Zitat von Kai Stian Olstad <ceph+list@olstad.com>:
On 26.05.2021 08:22, Eugen Block wrote:
Hi,
did you wipe the LV on the SSD that was assigned to the failed HDD? I just did that on a fresh Pacific install successfully, a couple of weeks ago it also worked on an Octopus cluster.
No, I did not wipe the LV. Not sure what you mean by wipe, so I tried overwriting the LV with /dev/zero, but that did solve it. So I guess with wipe do you mean delete the LV with lvremove?
-- Kai Stian Olstad _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 26.05.2021 11:16, Eugen Block wrote:
Yes, the LVs are not removed automatically, you need to free up the VG, there are a couple of ways to do so, for example remotely:
pacific1:~ # ceph orch device zap pacific4 /dev/vdb --force
or directly on the host with:
pacific1:~ # cephadm ceph-volume lvm zap --destroy /dev/<CEPH_VG>/<DB_LV>
Thanks, I used the cephadm command and deleted the LV and the VG now has free space # vgs | egrep "VG|dbs" VG #PV #LV #SN Attr VSize VFree ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b 3 14 0 wz--n- <5.24t 357.74g But it doesn't seams to be able to use it, because it can find anyting # ceph orch apply osd -i hdd.yml --dry-run ################ OSDSPEC PREVIEWS ################ +---------+------+-------------+----------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+------+-------------+----------+----+-----+ +---------+------+-------------+----------+----+-----+ I tried adding size as you have in your configuration db_devices: rotational: 0 size: '30G:' Still it was unable to create the OSD. If I removed the : so it is 30GB exact size, it did find the disk, but DB is not placed on a SSD since I do not have one with 30 GB exact size ################ OSDSPEC PREVIEWS ################ +---------+------+-------------+----------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+------+-------------+----------+----+-----+ |osd |hdd |pech-hd-7 |/dev/sdt |- |- | +---------+------+-------------+----------+----+-----+ To me I looks like Cephadm can't use/find the free space on the VG and use that as a new LV for the OSD. -- Kai Stian Olstad
We've found that after doing the osd rm, you can use: "ceph-volume lvm zap --osd-id 178 --destroy" on the server with that OSD as per: https://docs.ceph.com/en/latest/ceph-volume/lvm/zap/#removing-devices and it will clean things up so they work as expected. On Tue, May 25, 2021 at 6:51 AM Kai Stian Olstad <ceph+list@olstad.com> wrote:
Hi
The server run 15.2.9 and has 15 HDD and 3 SSD. The OSDs was created with this YAML file
hdd.yml -------- service_type: osd service_id: hdd placement: host_pattern: 'pech-hd-*' data_devices: rotational: 1 db_devices: rotational: 0
The result was that the 3 SSD is added to 1 VG with 15 LV on it.
# vgs | egrep "VG|dbs" VG #PV #LV #SN Attr VSize VFree ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b 3 15 0 wz--n- <5.24t 48.00m
One of the osd failed and I run rm with replace
# ceph orch osd rm 178 --replace
and the result is
# ceph osd tree | grep "ID|destroyed" ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 178 hdd 12.82390 osd.178 destroyed 0 1.00000
But I'm not able to replace the disk with the same YAML file as shown above.
# ceph orch apply osd -i hdd.yml --dry-run ################ OSDSPEC PREVIEWS ################ +---------+------+------+------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+------+------+------+----+-----+ +---------+------+------+------+----+-----+
I guess this is the wrong way to do it, but I can't find the answer in the documentation. So how can I replace this failed disk in Cephadm?
-- Kai Stian Olstad _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 26.05.2021 22:14, David Orman wrote:
We've found that after doing the osd rm, you can use: "ceph-volume lvm zap --osd-id 178 --destroy" on the server with that OSD as per: https://docs.ceph.com/en/latest/ceph-volume/lvm/zap/#removing-devices and it will clean things up so they work as expected.
With the help of Eugen I did run "cephadm ceph-volume lvm zap --destroy <LV>" and the LV is gone. I think that is the same result as "ceph-volume lvm zap --osd-id 178 --destroy" would give me? I now have 357GB free space on the VG, but Cephadm doesn't find and use this space. Above it the result of the zap command and it show the LV is deleted. $ sudo cephadm ceph-volume lvm zap --destroy /dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69 INFO:cephadm:Inferring fsid 3614abcc-201c-11eb-995a-2794bcc75ae0 INFO:cephadm:Using recent ceph image ceph:v15.2.9 INFO:cephadm:/usr/bin/podman:stderr --> Zapping: /dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69 INFO:cephadm:/usr/bin/podman:stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69 bs=1M count=10 conv=fsync INFO:cephadm:/usr/bin/podman:stderr stderr: 10+0 records in INFO:cephadm:/usr/bin/podman:stderr 10+0 records out INFO:cephadm:/usr/bin/podman:stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0195532 s, 536 MB/s INFO:cephadm:/usr/bin/podman:stderr --> More than 1 LV left in VG, will proceed to destroy LV only INFO:cephadm:/usr/bin/podman:stderr --> Removing LV because --destroy was given: /dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69 INFO:cephadm:/usr/bin/podman:stderr Running command: /usr/sbin/lvremove -v -f /dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69 INFO:cephadm:/usr/bin/podman:stderr stdout: Logical volume "osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69" successfully removed INFO:cephadm:/usr/bin/podman:stderr stderr: Removing ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--449bd001--eb32--46de--ab80--a1cbcd293d69 (253:3) INFO:cephadm:/usr/bin/podman:stderr stderr: Archiving volume group "ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b" metadata (seqno 61). INFO:cephadm:/usr/bin/podman:stderr stderr: Releasing logical volume "osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69" INFO:cephadm:/usr/bin/podman:stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b" (seqno 62). INFO:cephadm:/usr/bin/podman:stderr --> Zapping successful for: <LV: /dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69> -- Kai Stian Olstad
That's not how it's supposed to work. I tried the same on an Octopus cluster and removed all filters except: data_devices: rotational: 1 db_devices: rotational: 0 My Octopus test osd nodes have two HDDs and one SSD, I removed all OSDs and redeployed on one node. This spec file results in three standalone OSDs! Without the other filters this won't work as expected, it seems. I'll try again on Pacific with the same test and see where that goes. Zitat von Kai Stian Olstad <ceph+list@olstad.com>:
On 26.05.2021 22:14, David Orman wrote:
We've found that after doing the osd rm, you can use: "ceph-volume lvm zap --osd-id 178 --destroy" on the server with that OSD as per: https://docs.ceph.com/en/latest/ceph-volume/lvm/zap/#removing-devices and it will clean things up so they work as expected.
With the help of Eugen I did run "cephadm ceph-volume lvm zap --destroy <LV>" and the LV is gone. I think that is the same result as "ceph-volume lvm zap --osd-id 178 --destroy" would give me?
I now have 357GB free space on the VG, but Cephadm doesn't find and use this space. Above it the result of the zap command and it show the LV is deleted.
$ sudo cephadm ceph-volume lvm zap --destroy /dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69 INFO:cephadm:Inferring fsid 3614abcc-201c-11eb-995a-2794bcc75ae0 INFO:cephadm:Using recent ceph image ceph:v15.2.9 INFO:cephadm:/usr/bin/podman:stderr --> Zapping: /dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69 INFO:cephadm:/usr/bin/podman:stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69 bs=1M count=10 conv=fsync INFO:cephadm:/usr/bin/podman:stderr stderr: 10+0 records in INFO:cephadm:/usr/bin/podman:stderr 10+0 records out INFO:cephadm:/usr/bin/podman:stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0195532 s, 536 MB/s INFO:cephadm:/usr/bin/podman:stderr --> More than 1 LV left in VG, will proceed to destroy LV only INFO:cephadm:/usr/bin/podman:stderr --> Removing LV because --destroy was given: /dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69 INFO:cephadm:/usr/bin/podman:stderr Running command: /usr/sbin/lvremove -v -f /dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69 INFO:cephadm:/usr/bin/podman:stderr stdout: Logical volume "osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69" successfully removed INFO:cephadm:/usr/bin/podman:stderr stderr: Removing ceph--block--dbs--563432b7--f52d--4cfe--b952--11542594843b-osd--block--db--449bd001--eb32--46de--ab80--a1cbcd293d69 (253:3) INFO:cephadm:/usr/bin/podman:stderr stderr: Archiving volume group "ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b" metadata (seqno 61). INFO:cephadm:/usr/bin/podman:stderr stderr: Releasing logical volume "osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69" INFO:cephadm:/usr/bin/podman:stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b" (seqno 62). INFO:cephadm:/usr/bin/podman:stderr --> Zapping successful for: <LV: /dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69>
-- Kai Stian Olstad _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 27.05.2021 11:17, Eugen Block wrote:
That's not how it's supposed to work. I tried the same on an Octopus cluster and removed all filters except:
data_devices: rotational: 1 db_devices: rotational: 0
My Octopus test osd nodes have two HDDs and one SSD, I removed all OSDs and redeployed on one node. This spec file results in three standalone OSDs! Without the other filters this won't work as expected, it seems. I'll try again on Pacific with the same test and see where that goes.
This spec did worked for me when I initially deployed with Octopus 15.2.5. -- Kai Stian Olstad
This test was on ceph version 15.2.8. On Pacific (ceph version 16.2.4) this also works for me for initial deployment of an entire host: +---------+-------------+----------+----------+----------+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+-------------+----------+----------+----------+-----+ |osd |ssd-hdd-mix |pacific1 |/dev/vdb |/dev/vdd |- | |osd |ssd-hdd-mix |pacific1 |/dev/vdc |/dev/vdd |- | +---------+-------------+----------+----------+----------+-----+ But it doesn't work if I remove one OSD, just like you describe. This is what ceph-volume reports: ---snip--- [ceph: root@pacific1 /]# ceph-volume lvm batch --report /dev/vdc --db-devices /dev/vdd --block-db-size 3G --> passed data devices: 1 physical, 0 LVM --> relative data size: 1.0 --> passed block_db devices: 1 physical, 0 LVM --> 1 fast devices were passed, but none are available Total OSDs: 0 Type Path LV Size % of device ---snip--- I know that this has already worked in Octopus, I did test it successfully not long ago. Zitat von Kai Stian Olstad <ceph+list@olstad.com>:
On 27.05.2021 11:17, Eugen Block wrote:
That's not how it's supposed to work. I tried the same on an Octopus cluster and removed all filters except:
data_devices: rotational: 1 db_devices: rotational: 0
My Octopus test osd nodes have two HDDs and one SSD, I removed all OSDs and redeployed on one node. This spec file results in three standalone OSDs! Without the other filters this won't work as expected, it seems. I'll try again on Pacific with the same test and see where that goes.
This spec did worked for me when I initially deployed with Octopus 15.2.5.
-- Kai Stian Olstad _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 27.05.2021 11:53, Eugen Block wrote:
This test was on ceph version 15.2.8.
On Pacific (ceph version 16.2.4) this also works for me for initial deployment of an entire host:
+---------+-------------+----------+----------+----------+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+-------------+----------+----------+----------+-----+ |osd |ssd-hdd-mix |pacific1 |/dev/vdb |/dev/vdd |- | |osd |ssd-hdd-mix |pacific1 |/dev/vdc |/dev/vdd |- | +---------+-------------+----------+----------+----------+-----+
But it doesn't work if I remove one OSD, just like you describe. This is what ceph-volume reports:
---snip--- [ceph: root@pacific1 /]# ceph-volume lvm batch --report /dev/vdc --db-devices /dev/vdd --block-db-size 3G --> passed data devices: 1 physical, 0 LVM --> relative data size: 1.0 --> passed block_db devices: 1 physical, 0 LVM --> 1 fast devices were passed, but none are available
Total OSDs: 0
Type Path LV Size % of device ---snip---
I know that this has already worked in Octopus, I did test it successfully not long ago.
Thank you for trying, so it looks like a bug. Searching through the issue tracker I find few issues related to replacing OSD, but it doesn't look like they get much attention. I tried to find a way to add the disk manually, did not find any documentation about it, but looking at the source code, some issues with some trial and error I ended up with this. Since the LV is deleted I recreated it with the same name. # lvcreate -l 91570 -n osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69 ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b In "cephadm shell" # cephadm shell # ceph auth get client.bootstrap-osd
/var/lib/ceph/bootstrap-osd/ceph.keyring # ceph-volume lvm prepare --bluestore --no-systemd --data /dev/sdt --block.db ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69
Need to have a json file for the "cephadm deploy" # printf '{\n"config": "%s",\n"keyring": "%s"\n}\n' "$(ceph config generate-minimal-conf | sed -e ':a;N;$!ba;s/\n/\\n/g' -e 's/\t/\\t/g' -e 's/$/\\n/')" "$(ceph auth get osd.178 | head -n 2 | sed -e ':a;N;$!ba;s/\n/\\n/g' -e 's/\t/\\t/g' -e 's/$/\\n/')"
config-osd.178.json
Exit cephadm shell and run # cephadm --image ceph:v15.2.9 deploy --fsid 3614abcc-201c-11eb-995a-2794bcc75ae0 --config-json /var/lib/ceph/3614abcc-201c-11eb-995a-2794bcc75ae0/home/config-osd.178.json --osd-fsid 9227e8ae-92eb-429e-9c7f-d4a2b75afb8e And the OSD is back, but the VG name on the HDD is missing block in it's name, just a cosmetic thing so I leave it as is. LV VG Attr LSize osd-block-9227e8ae-92eb-429e-9c7f-d4a2b75afb8e ceph-46f42262-d3dc-4dc3-8952-eec3e4a2c178 -wi-ao---- 12.47t osd-block-2da790bc-a74c-41da-8772-3b8aac77001c ceph-block-1b5ad7e7-2e24-4315-8a05-7439ab782b45 -wi-ao---- 12.47t The fist one is the new OSD and the second one is one that cephadm itself created. -- Kai Stian Olstad
participants (3)
-
David Orman
-
Eugen Block
-
Kai Stian Olstad