Merge DB/WAL back to the main device?
Hi all, while reading a sibling thread about moving DB/WAL to a separate device, I wonder whether is it possible to go the other way round as well, i.e. to remove a metadata device from an OSD and merge metadata back to the main storage? What I am trying to do: My OSD nodes are 1U boxes with 4 drive bays, two of which support NVMe. They have two small-ish NVMe drives and two large HDDs. On NVMe, there is a partition for an OS (configured as RAID-1 across both NVMe drives), and the rest of each NVMe is used as metadata for one of the HDD-based OSDs. So I have two OSDs per node. Now I am considering replacing those small NVMes with much larger ones, and use part of them for an OS as before, part of them as metadata for the HDD-based OSDs as before, and the rest as new NVMe-only OSDs. Can this be done without tearing the original HDD+NVMe-based OSDs down and recreating them again? Being able to remove a metadata device from the OSD would help in this case. Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall
Hi, in this case I would recommend to use the ceph-bluestore-tool instead of ceph-volume because it cleans up (removes the block.db symlink and labels). These are the steps to revert that change: # before reverting two devices are in use: soc9-ceph:~ # ceph osd metadata 0 | grep devices "bluefs_db_devices": "vdf", "bluestore_bdev_devices": "vdb", "devices": "vdb,vdf", "objectstore_numa_unknown_devices": "vdb,vdf", 1. soc9-ceph:~ # ceph orch daemon stop osd.0 2. soc9-ceph:~ # cephadm shell --name osd.0 3. [ceph: root@soc9-ceph /]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-0/ --command bluefs-bdev-migrate --devs-source /var/lib/ceph/osd/ceph-0/block.db --dev-target /var/lib/ceph/osd/ceph-0/block inferring bluefs devices from bluestore path device removed:1 /var/lib/ceph/osd/ceph-0/block.db 4. IMPORTANT: Remove the DB LV before you start the OSD, otherwise the OSD will start to use it again because of the LV tags. Alternatively, delete the LV tags of the DB LV before starting the OSD. 5. soc9-ceph:~ # ceph orch daemon start osd.0 # check result soc9-ceph:~ # ceph osd metadata 0 | grep devices "bluestore_bdev_devices": "vdb", "devices": "vdb", "objectstore_numa_unknown_devices": "vdb", Regards, Eugen Zitat von Jan Kasprzak <kas@fi.muni.cz>:
Hi all,
while reading a sibling thread about moving DB/WAL to a separate device, I wonder whether is it possible to go the other way round as well, i.e. to remove a metadata device from an OSD and merge metadata back to the main storage?
What I am trying to do:
My OSD nodes are 1U boxes with 4 drive bays, two of which support NVMe. They have two small-ish NVMe drives and two large HDDs. On NVMe, there is a partition for an OS (configured as RAID-1 across both NVMe drives), and the rest of each NVMe is used as metadata for one of the HDD-based OSDs. So I have two OSDs per node.
Now I am considering replacing those small NVMes with much larger ones, and use part of them for an OS as before, part of them as metadata for the HDD-based OSDs as before, and the rest as new NVMe-only OSDs. Can this be done without tearing the original HDD+NVMe-based OSDs down and recreating them again?
Being able to remove a metadata device from the OSD would help in this case.
Thanks!
-Yenya
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, as we are just trying somthing similar i will jump into this as well: As our initial setup for the WAL/DB was on too small SSDs we need to move it back to the HDD and replace the SSD with larger ones (and after that we decide if we just keep it that way - it an archive cluster without bit performance demands - and use the SSDs for a full flash data pool or remigrate the DB to the larger SSDs) So I just tested how this might go and use ceph-volume: My steps are inline below: On 04.02.25 09:52, Eugen Block wrote:
Hi,
in this case I would recommend to use the ceph-bluestore-tool instead of ceph-volume because it cleans up (removes the block.db symlink and labels). These are the steps to revert that change:
# before reverting two devices are in use:
soc9-ceph:~ # ceph osd metadata 0 | grep devices "bluefs_db_devices": "vdf", "bluestore_bdev_devices": "vdb", "devices": "vdb,vdf", "objectstore_numa_unknown_devices": "vdb,vdf",
0. # ceph osd set noout block + block.db links before: # ls -l /var/lib/ceph/osd/ceph-1406/ total 60 lrwxrwxrwx 1 ceph ceph 111 Jan 28 17:23 block -> /dev/mapper/ceph--7c3251f6--3307--4636--92c2--0dd4da36d929-osd--block--8b3d79bc--2f32--46fe--84d4--cea620c5990a lrwxrwxrwx 1 ceph ceph 108 Jan 28 17:23 block.db -> /dev/mapper/ceph--bdd19323--8061--4253--a14a--d6d2a3bb13c2-osd--db--471ed787--0eda--4862--8cdc--fb9f1746d4ee (...)
1. soc9-ceph:~ # ceph orch daemon stop osd.0
1. Here I stoped the osd via systemctl: # systemctl stop ceph-e776dd57-7fc6-11ee-9f23-9bb83aca7b4b@osd.1406.service
2. soc9-ceph:~ # cephadm shell --name osd.0
2. # cephadm shell --name osd.1406
3. [ceph: root@soc9-ceph /]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-0/ --command bluefs-bdev-migrate --devs-source /var/lib/ceph/osd/ceph-0/block.db --dev-target /var/lib/ceph/osd/ceph-0/block inferring bluefs devices from bluestore path device removed:1 /var/lib/ceph/osd/ceph-0/block.db
3. Here I used ceph-volume: # ceph-volume lvm migrate --osd-id 1406 --osd-fsid 8b3d79bc-2f32-46fe-84d4-cea620c5990a --from db wal --target ceph-7c3251f6-3307-4636-92c2-0dd4da36d929/osd-block-8b3d79bc-2f32-46fe-84d4-cea620c5990a --> Migrate to existing, Source: ['--devs-source', '/var/lib/ceph/osd/ceph-1406/block.db'] Target: /var/lib/ceph/osd/ceph-1406/block --> Migration successful.
4. IMPORTANT: Remove the DB LV before you start the OSD, otherwise the OSD will start to use it again because of the LV tags. Alternatively, delete the LV tags of the DB LV before starting the OSD.
4. This I also did not remove - as the 'block.db' link disappearred from the config dir I assumed it was removed: # ls -l /var/lib/ceph/osd/ceph-1406/ total 60 lrwxrwxrwx 1 ceph ceph 111 Jan 28 17:23 block -> /dev/mapper/ceph--7c3251f6--3307--4636--92c2--0dd4da36d929-osd--block--8b3d79bc--2f32--46fe--84d4--cea620c5990a (..)
5. soc9-ceph:~ # ceph orch daemon start osd.0
5. Exit from container + restart: # systemctl start ceph-e776dd57-7fc6-11ee-9f23-9bb83aca7b4b@osd.1406.service
# check result
soc9-ceph:~ # ceph osd metadata 0 | grep devices "bluestore_bdev_devices": "vdb", "devices": "vdb", "objectstore_numa_unknown_devices": "vdb",
This I did just now - and it seem that the DB device is still listed there: # ceph osd metadata 1406 | grep devices "bluefs_db_devices": "md10", "bluestore_bdev_devices": "sdy", "devices": "md10,sdy", "objectstore_numa_unknown_devices": "md10,sdy", -> So is this OSD now using the old LVM device without having the 'block.db' link in its config directory? I will check with another OSD with the ceph-bluestore-tool (which I actually tried first, but looking at the history I managed to have a typo in my '---path' which probably caused the error I got (can't migrate /var/lib/ceph/osd/ceph-1406/block, not a valid bluefs volume - I managed to have the 'bolck' already in the --path and did not realize that) And is there a good way to straighten out the current setup of osd.1406 from its current state? Regards, Holger
Regards, Eugen
Zitat von Jan Kasprzak <kas@fi.muni.cz>:
Hi all,
while reading a sibling thread about moving DB/WAL to a separate device, I wonder whether is it possible to go the other way round as well, i.e. to remove a metadata device from an OSD and merge metadata back to the main storage?
What I am trying to do:
My OSD nodes are 1U boxes with 4 drive bays, two of which support NVMe. They have two small-ish NVMe drives and two large HDDs. On NVMe, there is a partition for an OS (configured as RAID-1 across both NVMe drives), and the rest of each NVMe is used as metadata for one of the HDD-based OSDs. So I have two OSDs per node.
Now I am considering replacing those small NVMes with much larger ones, and use part of them for an OS as before, part of them as metadata for the HDD-based OSDs as before, and the rest as new NVMe-only OSDs. Can this be done without tearing the original HDD+NVMe-based OSDs down and recreating them again?
Being able to remove a metadata device from the OSD would help in this case.
Thanks!
-Yenya
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naundorf@rz.uni-kiel.de
Hi Holger, oh yeah, the noout flag is important of course, I missed that in this response, although I mentioned it in the other thread. :-)
-> So is this OSD now using the old LVM device without having the 'block.db' link in its config directory?
No, the block.db symlink is most likely still present after OSD restart, check again: ls -l /var/lib/ceph/e776dd57-7fc6-11ee-9f23-9bb83aca7b4b/osd.1406/block.db
And is there a good way to straighten out the current setup of osd.1406 from its current state?
Try my recommended steps again for that OSD, it should work if you use the ceph-bluestore-tool and remove the LV before you start the OSD. Zitat von Holger Naundorf <naundorf@rz.uni-kiel.de>:
Hello, as we are just trying somthing similar i will jump into this as well: As our initial setup for the WAL/DB was on too small SSDs we need to move it back to the HDD and replace the SSD with larger ones (and after that we decide if we just keep it that way - it an archive cluster without bit performance demands - and use the SSDs for a full flash data pool or remigrate the DB to the larger SSDs)
So I just tested how this might go and use ceph-volume:
My steps are inline below:
On 04.02.25 09:52, Eugen Block wrote:
Hi,
in this case I would recommend to use the ceph-bluestore-tool instead of ceph-volume because it cleans up (removes the block.db symlink and labels). These are the steps to revert that change:
# before reverting two devices are in use:
soc9-ceph:~ # ceph osd metadata 0 | grep devices "bluefs_db_devices": "vdf", "bluestore_bdev_devices": "vdb", "devices": "vdb,vdf", "objectstore_numa_unknown_devices": "vdb,vdf",
0. # ceph osd set noout
block + block.db links before:
# ls -l /var/lib/ceph/osd/ceph-1406/ total 60 lrwxrwxrwx 1 ceph ceph 111 Jan 28 17:23 block -> /dev/mapper/ceph--7c3251f6--3307--4636--92c2--0dd4da36d929-osd--block--8b3d79bc--2f32--46fe--84d4--cea620c5990a
lrwxrwxrwx 1 ceph ceph 108 Jan 28 17:23 block.db -> /dev/mapper/ceph--bdd19323--8061--4253--a14a--d6d2a3bb13c2-osd--db--471ed787--0eda--4862--8cdc--fb9f1746d4ee
(...)
1. soc9-ceph:~ # ceph orch daemon stop osd.0
1. Here I stoped the osd via systemctl:
# systemctl stop ceph-e776dd57-7fc6-11ee-9f23-9bb83aca7b4b@osd.1406.service
2. soc9-ceph:~ # cephadm shell --name osd.0
2. # cephadm shell --name osd.1406
3. [ceph: root@soc9-ceph /]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-0/ --command bluefs-bdev-migrate --devs-source /var/lib/ceph/osd/ceph-0/block.db --dev-target /var/lib/ceph/osd/ceph-0/block inferring bluefs devices from bluestore path device removed:1 /var/lib/ceph/osd/ceph-0/block.db
3. Here I used ceph-volume:
# ceph-volume lvm migrate --osd-id 1406 --osd-fsid 8b3d79bc-2f32-46fe-84d4-cea620c5990a --from db wal --target ceph-7c3251f6-3307-4636-92c2-0dd4da36d929/osd-block-8b3d79bc-2f32-46fe-84d4-cea620c5990a --> Migrate to existing, Source: ['--devs-source', '/var/lib/ceph/osd/ceph-1406/block.db'] Target: /var/lib/ceph/osd/ceph-1406/block --> Migration successful.
4. IMPORTANT: Remove the DB LV before you start the OSD, otherwise the OSD will start to use it again because of the LV tags. Alternatively, delete the LV tags of the DB LV before starting the OSD.
4. This I also did not remove - as the 'block.db' link disappearred from the config dir I assumed it was removed:
# ls -l /var/lib/ceph/osd/ceph-1406/ total 60 lrwxrwxrwx 1 ceph ceph 111 Jan 28 17:23 block -> /dev/mapper/ceph--7c3251f6--3307--4636--92c2--0dd4da36d929-osd--block--8b3d79bc--2f32--46fe--84d4--cea620c5990a (..)
5. soc9-ceph:~ # ceph orch daemon start osd.0
5. Exit from container + restart:
# systemctl start ceph-e776dd57-7fc6-11ee-9f23-9bb83aca7b4b@osd.1406.service
# check result
soc9-ceph:~ # ceph osd metadata 0 | grep devices "bluestore_bdev_devices": "vdb", "devices": "vdb", "objectstore_numa_unknown_devices": "vdb",
This I did just now - and it seem that the DB device is still listed there:
# ceph osd metadata 1406 | grep devices "bluefs_db_devices": "md10", "bluestore_bdev_devices": "sdy", "devices": "md10,sdy", "objectstore_numa_unknown_devices": "md10,sdy",
-> So is this OSD now using the old LVM device without having the 'block.db' link in its config directory?
I will check with another OSD with the ceph-bluestore-tool (which I actually tried first, but looking at the history I managed to have a typo in my '---path' which probably caused the error I got (can't migrate /var/lib/ceph/osd/ceph-1406/block, not a valid bluefs volume - I managed to have the 'bolck' already in the --path and did not realize that)
And is there a good way to straighten out the current setup of osd.1406 from its current state?
Regards, Holger
Regards, Eugen
Zitat von Jan Kasprzak <kas@fi.muni.cz>:
Hi all,
while reading a sibling thread about moving DB/WAL to a separate device, I wonder whether is it possible to go the other way round as well, i.e. to remove a metadata device from an OSD and merge metadata back to the main storage?
What I am trying to do:
My OSD nodes are 1U boxes with 4 drive bays, two of which support NVMe. They have two small-ish NVMe drives and two large HDDs. On NVMe, there is a partition for an OS (configured as RAID-1 across both NVMe drives), and the rest of each NVMe is used as metadata for one of the HDD-based OSDs. So I have two OSDs per node.
Now I am considering replacing those small NVMes with much larger ones, and use part of them for an OS as before, part of them as metadata for the HDD-based OSDs as before, and the rest as new NVMe-only OSDs. Can this be done without tearing the original HDD+NVMe-based OSDs down and recreating them again?
Being able to remove a metadata device from the OSD would help in this case.
Thanks!
-Yenya
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naundorf@rz.uni-kiel.de
On 04.02.25 12:54, Eugen Block wrote:
Hi Holger,
oh yeah, the noout flag is important of course, I missed that in this response, although I mentioned it in the other thread. :-)
-> So is this OSD now using the old LVM device without having the 'block.db' link in its config directory?
No, the block.db symlink is most likely still present after OSD restart, check again:
ls -l /var/lib/ceph/e776dd57-7fc6-11ee-9f23-9bb83aca7b4b/osd.1406/block.db
Yes, it was gone directly after I did the ceph-volume migrate, but popped back up after the OSD was back up.
And is there a good way to straighten out the current setup of osd.1406 from its current state?
Try my recommended steps again for that OSD, it should work if you use the ceph-bluestore-tool and remove the LV before you start the OSD.
It did, thanks for the pointers. I am just wondering, would the 'ceph-volume' method result in the same outcome if I had removed the LVM before the restart? Regards, Holger
Zitat von Holger Naundorf <naundorf@rz.uni-kiel.de>:
Hello, as we are just trying somthing similar i will jump into this as well: As our initial setup for the WAL/DB was on too small SSDs we need to move it back to the HDD and replace the SSD with larger ones (and after that we decide if we just keep it that way - it an archive cluster without bit performance demands - and use the SSDs for a full flash data pool or remigrate the DB to the larger SSDs)
So I just tested how this might go and use ceph-volume:
My steps are inline below:
On 04.02.25 09:52, Eugen Block wrote:
Hi,
in this case I would recommend to use the ceph-bluestore-tool instead of ceph-volume because it cleans up (removes the block.db symlink and labels). These are the steps to revert that change:
# before reverting two devices are in use:
soc9-ceph:~ # ceph osd metadata 0 | grep devices "bluefs_db_devices": "vdf", "bluestore_bdev_devices": "vdb", "devices": "vdb,vdf", "objectstore_numa_unknown_devices": "vdb,vdf",
0. # ceph osd set noout
block + block.db links before:
# ls -l /var/lib/ceph/osd/ceph-1406/ total 60 lrwxrwxrwx 1 ceph ceph 111 Jan 28 17:23 block -> /dev/mapper/ceph--7c3251f6--3307--4636--92c2--0dd4da36d929-osd--block--8b3d79bc--2f32--46fe--84d4--cea620c5990a
lrwxrwxrwx 1 ceph ceph 108 Jan 28 17:23 block.db -> /dev/mapper/ceph--bdd19323--8061--4253--a14a--d6d2a3bb13c2-osd--db--471ed787--0eda--4862--8cdc--fb9f1746d4ee
(...)
1. soc9-ceph:~ # ceph orch daemon stop osd.0
1. Here I stoped the osd via systemctl:
# systemctl stop ceph-e776dd57-7fc6-11ee-9f23-9bb83aca7b4b@osd.1406.service
2. soc9-ceph:~ # cephadm shell --name osd.0
2. # cephadm shell --name osd.1406
3. [ceph: root@soc9-ceph /]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-0/ --command bluefs-bdev-migrate --devs-source /var/lib/ceph/osd/ceph-0/block.db --dev-target /var/lib/ceph/osd/ceph-0/block inferring bluefs devices from bluestore path device removed:1 /var/lib/ceph/osd/ceph-0/block.db
3. Here I used ceph-volume:
# ceph-volume lvm migrate --osd-id 1406 --osd-fsid 8b3d79bc-2f32-46fe-84d4-cea620c5990a --from db wal --target ceph-7c3251f6-3307-4636-92c2-0dd4da36d929/osd-block-8b3d79bc-2f32-46fe-84d4-cea620c5990a --> Migrate to existing, Source: ['--devs-source', '/var/lib/ceph/osd/ceph-1406/block.db'] Target: /var/lib/ceph/osd/ceph-1406/block --> Migration successful.
4. IMPORTANT: Remove the DB LV before you start the OSD, otherwise the OSD will start to use it again because of the LV tags. Alternatively, delete the LV tags of the DB LV before starting the OSD.
4. This I also did not remove - as the 'block.db' link disappearred from the config dir I assumed it was removed:
# ls -l /var/lib/ceph/osd/ceph-1406/ total 60 lrwxrwxrwx 1 ceph ceph 111 Jan 28 17:23 block -> /dev/mapper/ceph--7c3251f6--3307--4636--92c2--0dd4da36d929-osd--block--8b3d79bc--2f32--46fe--84d4--cea620c5990a (..)
5. soc9-ceph:~ # ceph orch daemon start osd.0
5. Exit from container + restart:
# systemctl start ceph-e776dd57-7fc6-11ee-9f23-9bb83aca7b4b@osd.1406.service
# check result
soc9-ceph:~ # ceph osd metadata 0 | grep devices "bluestore_bdev_devices": "vdb", "devices": "vdb", "objectstore_numa_unknown_devices": "vdb",
This I did just now - and it seem that the DB device is still listed there:
# ceph osd metadata 1406 | grep devices "bluefs_db_devices": "md10", "bluestore_bdev_devices": "sdy", "devices": "md10,sdy", "objectstore_numa_unknown_devices": "md10,sdy",
-> So is this OSD now using the old LVM device without having the 'block.db' link in its config directory?
I will check with another OSD with the ceph-bluestore-tool (which I actually tried first, but looking at the history I managed to have a typo in my '---path' which probably caused the error I got (can't migrate /var/lib/ceph/osd/ceph-1406/block, not a valid bluefs volume - I managed to have the 'bolck' already in the --path and did not realize that)
And is there a good way to straighten out the current setup of osd.1406 from its current state?
Regards, Holger
Regards, Eugen
Zitat von Jan Kasprzak <kas@fi.muni.cz>:
Hi all,
while reading a sibling thread about moving DB/WAL to a separate device, I wonder whether is it possible to go the other way round as well, i.e. to remove a metadata device from an OSD and merge metadata back to the main storage?
What I am trying to do:
My OSD nodes are 1U boxes with 4 drive bays, two of which support NVMe. They have two small-ish NVMe drives and two large HDDs. On NVMe, there is a partition for an OS (configured as RAID-1 across both NVMe drives), and the rest of each NVMe is used as metadata for one of the HDD-based OSDs. So I have two OSDs per node.
Now I am considering replacing those small NVMes with much larger ones, and use part of them for an OS as before, part of them as metadata for the HDD-based OSDs as before, and the rest as new NVMe-only OSDs. Can this be done without tearing the original HDD+NVMe-based OSDs down and recreating them again?
Being able to remove a metadata device from the OSD would help in this case.
Thanks!
-Yenya
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naundorf@rz.uni-kiel.de
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naundorf@rz.uni-kiel.de
It looks like it works with ceph-volume as well when I remove the LV before starting the OSD. I just tried that in my lab cluster. Zitat von Holger Naundorf <naundorf@rz.uni-kiel.de>:
On 04.02.25 12:54, Eugen Block wrote:
Hi Holger,
oh yeah, the noout flag is important of course, I missed that in this response, although I mentioned it in the other thread. :-)
-> So is this OSD now using the old LVM device without having the 'block.db' link in its config directory?
No, the block.db symlink is most likely still present after OSD restart, check again:
ls -l /var/lib/ceph/e776dd57-7fc6-11ee-9f23-9bb83aca7b4b/osd.1406/block.db
Yes, it was gone directly after I did the ceph-volume migrate, but popped back up after the OSD was back up.
And is there a good way to straighten out the current setup of osd.1406 from its current state?
Try my recommended steps again for that OSD, it should work if you use the ceph-bluestore-tool and remove the LV before you start the OSD.
It did, thanks for the pointers.
I am just wondering, would the 'ceph-volume' method result in the same outcome if I had removed the LVM before the restart?
Regards, Holger
Zitat von Holger Naundorf <naundorf@rz.uni-kiel.de>:
Hello, as we are just trying somthing similar i will jump into this as well: As our initial setup for the WAL/DB was on too small SSDs we need to move it back to the HDD and replace the SSD with larger ones (and after that we decide if we just keep it that way - it an archive cluster without bit performance demands - and use the SSDs for a full flash data pool or remigrate the DB to the larger SSDs)
So I just tested how this might go and use ceph-volume:
My steps are inline below:
On 04.02.25 09:52, Eugen Block wrote:
Hi,
in this case I would recommend to use the ceph-bluestore-tool instead of ceph-volume because it cleans up (removes the block.db symlink and labels). These are the steps to revert that change:
# before reverting two devices are in use:
soc9-ceph:~ # ceph osd metadata 0 | grep devices "bluefs_db_devices": "vdf", "bluestore_bdev_devices": "vdb", "devices": "vdb,vdf", "objectstore_numa_unknown_devices": "vdb,vdf",
0. # ceph osd set noout
block + block.db links before:
# ls -l /var/lib/ceph/osd/ceph-1406/ total 60 lrwxrwxrwx 1 ceph ceph 111 Jan 28 17:23 block -> /dev/mapper/ceph--7c3251f6--3307--4636--92c2--0dd4da36d929-osd--block--8b3d79bc--2f32--46fe--84d4--cea620c5990a
lrwxrwxrwx 1 ceph ceph 108 Jan 28 17:23 block.db -> /dev/mapper/ceph--bdd19323--8061--4253--a14a--d6d2a3bb13c2-osd--db--471ed787--0eda--4862--8cdc--fb9f1746d4ee
(...)
1. soc9-ceph:~ # ceph orch daemon stop osd.0
1. Here I stoped the osd via systemctl:
# systemctl stop ceph-e776dd57-7fc6-11ee-9f23-9bb83aca7b4b@osd.1406.service
2. soc9-ceph:~ # cephadm shell --name osd.0
2. # cephadm shell --name osd.1406
3. [ceph: root@soc9-ceph /]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-0/ --command bluefs-bdev-migrate --devs-source /var/lib/ceph/osd/ceph-0/block.db --dev-target /var/lib/ceph/osd/ceph-0/block inferring bluefs devices from bluestore path device removed:1 /var/lib/ceph/osd/ceph-0/block.db
3. Here I used ceph-volume:
# ceph-volume lvm migrate --osd-id 1406 --osd-fsid 8b3d79bc-2f32-46fe-84d4-cea620c5990a --from db wal --target ceph-7c3251f6-3307-4636-92c2-0dd4da36d929/osd-block-8b3d79bc-2f32-46fe-84d4-cea620c5990a --> Migrate to existing, Source: ['--devs-source', '/var/lib/ceph/osd/ceph-1406/block.db'] Target: /var/lib/ceph/osd/ceph-1406/block --> Migration successful.
4. IMPORTANT: Remove the DB LV before you start the OSD, otherwise the OSD will start to use it again because of the LV tags. Alternatively, delete the LV tags of the DB LV before starting the OSD.
4. This I also did not remove - as the 'block.db' link disappearred from the config dir I assumed it was removed:
# ls -l /var/lib/ceph/osd/ceph-1406/ total 60 lrwxrwxrwx 1 ceph ceph 111 Jan 28 17:23 block -> /dev/mapper/ceph--7c3251f6--3307--4636--92c2--0dd4da36d929-osd--block--8b3d79bc--2f32--46fe--84d4--cea620c5990a (..)
5. soc9-ceph:~ # ceph orch daemon start osd.0
5. Exit from container + restart:
# systemctl start ceph-e776dd57-7fc6-11ee-9f23-9bb83aca7b4b@osd.1406.service
# check result
soc9-ceph:~ # ceph osd metadata 0 | grep devices "bluestore_bdev_devices": "vdb", "devices": "vdb", "objectstore_numa_unknown_devices": "vdb",
This I did just now - and it seem that the DB device is still listed there:
# ceph osd metadata 1406 | grep devices "bluefs_db_devices": "md10", "bluestore_bdev_devices": "sdy", "devices": "md10,sdy", "objectstore_numa_unknown_devices": "md10,sdy",
-> So is this OSD now using the old LVM device without having the 'block.db' link in its config directory?
I will check with another OSD with the ceph-bluestore-tool (which I actually tried first, but looking at the history I managed to have a typo in my '---path' which probably caused the error I got (can't migrate /var/lib/ceph/osd/ceph-1406/block, not a valid bluefs volume - I managed to have the 'bolck' already in the --path and did not realize that)
And is there a good way to straighten out the current setup of osd.1406 from its current state?
Regards, Holger
Regards, Eugen
Zitat von Jan Kasprzak <kas@fi.muni.cz>:
Hi all,
while reading a sibling thread about moving DB/WAL to a separate device, I wonder whether is it possible to go the other way round as well, i.e. to remove a metadata device from an OSD and merge metadata back to the main storage?
What I am trying to do:
My OSD nodes are 1U boxes with 4 drive bays, two of which support NVMe. They have two small-ish NVMe drives and two large HDDs. On NVMe, there is a partition for an OS (configured as RAID-1 across both NVMe drives), and the rest of each NVMe is used as metadata for one of the HDD-based OSDs. So I have two OSDs per node.
Now I am considering replacing those small NVMes with much larger ones, and use part of them for an OS as before, part of them as metadata for the HDD-based OSDs as before, and the rest as new NVMe-only OSDs. Can this be done without tearing the original HDD+NVMe-based OSDs down and recreating them again?
Being able to remove a metadata device from the OSD would help in this case.
Thanks!
-Yenya
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naundorf@rz.uni-kiel.de
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naundorf@rz.uni-kiel.de
Holger, Eugen, thanks! I tried the ceph-volume approach, and it worked. The only strange thing was that "ceph osd metadata $ID | grep devices" reports "bluefs_db_devices": "nvme1n1", while in fact the db+wal LV/PV/VG is on a _partition_ of that device, nvme1n1p4. Another problem is how to remove the volume group on that device. I expected "ceph-volume zap /dev/nvme1n1p4" to be able to do it, but it failed with: stderr: wipefs: error: /dev/nvme1n1p4: probing initialization failed: Device or resource busy --> failed to wipefs device, will try again to workaround probable race condition In the end I did osd# vgchange -a n ceph-1cbc1f4d-9043-4a10-844b-c7fe28b4a333 osd# vgremove ceph-1cbc1f4d-9043-4a10-844b-c7fe28b4a333 which worked. Can ceph-volume zap be used for this as well? FWIW, all the commands I did: mon# ceph osd metadata 8 | grep devices "bluefs_db_devices": "nvme1n1", "bluestore_bdev_devices": "sda", "devices": "nvme1n1,sda", "objectstore_numa_unknown_devices": "sda", mon# ceph osd set noout osd# ls -l /var/lib/ceph/osd/ceph-8 lrwxrwxrwx. 1 ceph ceph 93 Feb 3 22:11 block -> /dev/ceph-179dc46e-f620-4b54-b663-24e5be779b3b/osd-block-d38e32da-fc81-4744-ad3a-750ba2cf3bc6 lrwxrwxrwx. 1 ceph ceph 90 Feb 3 22:11 block.db -> /dev/ceph-1cbc1f4d-9043-4a10-844b-c7fe28b4a333/osd-db-ce2a1653-3cf3-4eac-b74c-3d49bb4ef170 osd# systemctl stop ceph-osd@8.service osd# ceph-volume lvm migrate --osd-id 8 --osd-fsid d38e32da-fc81-4744-ad3a-750 osd# ba2cf3bc6 --from db wal --target ceph-179dc46e-f620-4b54-b663-24e5be779b3b/osd-block-d38e32da-fc81-4744-ad3a-750ba2cf3bc6 osd# vgchange -a n ceph-1cbc1f4d-9043-4a10-844b-c7fe28b4a333 osd# vgremove ceph-1cbc1f4d-9043-4a10-844b-c7fe28b4a333 osd# systemctl start ceph-osd@8.service mon# ceph osd unset noout mon# watch ceph -s # till it reports HEALTH_OK Thanks, -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall
Hi,
while in fact the db+wal LV/PV/VG is on a _partition_ of that device, nvme1n1p4.
the metadata.devices only contains the main device, not the partition.
which worked. Can ceph-volume zap be used for this as well?
Usually, I use 'ceph-volume lvm zap --destroy...', have you tried it with the keyword 'lvm' as well? I decided to write a short blog post [0] to describe both migrating to a new DB device and back from DB to the main device. [0] https://heiterbiswolkig.blogs.nde.ag/2025/02/05/cephadm-migrate-db-wal-to-ne... Zitat von Jan Kasprzak <kas@fi.muni.cz>:
Holger, Eugen,
thanks! I tried the ceph-volume approach, and it worked. The only strange thing was that "ceph osd metadata $ID | grep devices" reports
"bluefs_db_devices": "nvme1n1",
while in fact the db+wal LV/PV/VG is on a _partition_ of that device, nvme1n1p4.
Another problem is how to remove the volume group on that device. I expected "ceph-volume zap /dev/nvme1n1p4" to be able to do it, but it failed with:
stderr: wipefs: error: /dev/nvme1n1p4: probing initialization failed: Device or resource busy --> failed to wipefs device, will try again to workaround probable race condition
In the end I did
osd# vgchange -a n ceph-1cbc1f4d-9043-4a10-844b-c7fe28b4a333 osd# vgremove ceph-1cbc1f4d-9043-4a10-844b-c7fe28b4a333
which worked. Can ceph-volume zap be used for this as well?
FWIW, all the commands I did:
mon# ceph osd metadata 8 | grep devices "bluefs_db_devices": "nvme1n1", "bluestore_bdev_devices": "sda", "devices": "nvme1n1,sda", "objectstore_numa_unknown_devices": "sda", mon# ceph osd set noout
osd# ls -l /var/lib/ceph/osd/ceph-8 lrwxrwxrwx. 1 ceph ceph 93 Feb 3 22:11 block -> /dev/ceph-179dc46e-f620-4b54-b663-24e5be779b3b/osd-block-d38e32da-fc81-4744-ad3a-750ba2cf3bc6 lrwxrwxrwx. 1 ceph ceph 90 Feb 3 22:11 block.db -> /dev/ceph-1cbc1f4d-9043-4a10-844b-c7fe28b4a333/osd-db-ce2a1653-3cf3-4eac-b74c-3d49bb4ef170 osd# systemctl stop ceph-osd@8.service osd# ceph-volume lvm migrate --osd-id 8 --osd-fsid d38e32da-fc81-4744-ad3a-750 osd# ba2cf3bc6 --from db wal --target ceph-179dc46e-f620-4b54-b663-24e5be779b3b/osd-block-d38e32da-fc81-4744-ad3a-750ba2cf3bc6 osd# vgchange -a n ceph-1cbc1f4d-9043-4a10-844b-c7fe28b4a333 osd# vgremove ceph-1cbc1f4d-9043-4a10-844b-c7fe28b4a333 osd# systemctl start ceph-osd@8.service
mon# ceph osd unset noout mon# watch ceph -s # till it reports HEALTH_OK
Thanks,
-Yenya
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 4/2/25 15:46, Jan Kasprzak wrote:
I wonder whether is it possible to go the other way round as well, i.e. to remove a metadata device from an OSD and merge metadata back to the main storage?
Yes. I actually used ceph-volume last time I did this, so I wonder if I should be using a cephadm sequence to do so. I used https://docs.clyso.com/blog/ceph-volume-create-wal-db-on-separate-device-for... to figure out steps on test cluster, plus I used this to gather more info: ceph-bluestore-tool show-label --dev /dev/ceph-$fsid/osd-block-$osduuid | jq .[] `ceph-bluestore-tool bluefs-bdev-new-db` at https://docs.ceph.com/en/latest/man/8/ceph-bluestore-tool/ looks useful, but again I wonder about cephadm. HTH, Greg.
participants (4)
-
Eugen Block
-
Gregory Orange
-
Holger Naundorf
-
Jan Kasprzak