Re: cephadm: Move DB/WAL from HDD to SSD
Hello, I ran the commands from the "regular" cephadm shell ('cephadm shell --') but did not specify an OSD (I wasn't aware you could do that) When you ran the ceph-volume command, did you do it within the container of the OSD? Running it with the container of the OSD would make sense as to why I was getting the errors I was getting. I can give it a try and let you know how it goes, and if necessary go from there. Sent from my mobile device. Please excuse brevity and ttpos.
Hi,
When you ran the ceph-volume command, did you do it within the container of the OSD? Running it with the container of the OSD would make sense as to why I was getting the errors I was getting.
Followup question: if I do seem to have to go the route of destroying the OSD and re-creating it with the flags for point the DB/WAL to the SSD
yes, I ran the commands within the OSD container. Don't forget to run the "migrate" command as well after you created a new db. Just one more remark about your followup question in the initial post: partition, the syntax I have seen is this:
ceph-volume lvm prepare --bluestore --block.db --block.wal --data VOLUME_GROUP/LOGICAL_VOLUME
Beside the fact that you assume correctly and the command lacks the db/wal device specification, I strongly recommend to let cephadm handle the entire process of OSD creation, just ensure you have a fitting spec file. Don't prepare any LVs manually beforehand, except for such a case as right now where you want to move DBs out of the main device. But if you want to create OSDs from scratch, use the orchestrator, it does all of the work. And don't forget the --dry-run flag if you want to apply a new spec (ceph orch apply -i osd-spec.yaml --dry-run). Regards, Eugen Zitat von Alan Murrell <Alan@t-net.ca>:
Hello,
I ran the commands from the "regular" cephadm shell ('cephadm shell --') but did not specify an OSD (I wasn't aware you could do that)
When you ran the ceph-volume command, did you do it within the container of the OSD? Running it with the container of the OSD would make sense as to why I was getting the errors I was getting.
I can give it a try and let you know how it goes, and if necessary go from there.
Sent from my mobile device. Please excuse brevity and ttpos.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Do I still have to stop the osd.X systemctl service on the host before running the commands, like in that article I linked, or is that unnecessary? -----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: February 2, 2025 23:45 To: ceph-users@ceph.io Subject: [ceph-users] Re: cephadm: Move DB/WAL from HDD to SSD *** This is an EXTERNAL email. Please exercise caution. DO NOT open attachments or click links from unknown senders or unexpected email. *** Hi,
When you ran the ceph-volume command, did you do it within the container of the OSD? Running it with the container of the OSD would make sense as to why I was getting the errors I was getting.
yes, I ran the commands within the OSD container. Don't forget to run the "migrate" command as well after you created a new db. Just one more remark about your followup question in the initial post:
Followup question: if I do seem to have to go the route of destroying the OSD and re-creating it with the flags for point the DB/WAL to the SSD partition, the syntax I have seen is this: ceph-volume lvm prepare --bluestore --block.db --block.wal --data VOLUME_GROUP/LOGICAL_VOLUME
Beside the fact that you assume correctly and the command lacks the db/wal device specification, I strongly recommend to let cephadm handle the entire process of OSD creation, just ensure you have a fitting spec file. Don't prepare any LVs manually beforehand, except for such a case as right now where you want to move DBs out of the main device. But if you want to create OSDs from scratch, use the orchestrator, it does all of the work. And don't forget the --dry-run flag if you want to apply a new spec (ceph orch apply -i osd-spec.yaml --dry-run). Regards, Eugen Zitat von Alan Murrell <Alan@t-net.ca>:
Hello,
I ran the commands from the "regular" cephadm shell ('cephadm shell --') but did not specify an OSD (I wasn't aware you could do that)
When you ran the ceph-volume command, did you do it within the container of the OSD? Running it with the container of the OSD would make sense as to why I was getting the errors I was getting.
I can give it a try and let you know how it goes, and if necessary go from there.
Sent from my mobile device. Please excuse brevity and ttpos.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Definitely stop the daemon (and set the noout flag), otherwise you won’t be able to modify anything. Zitat von Alan Murrell <Alan@t-net.ca>:
Do I still have to stop the osd.X systemctl service on the host before running the commands, like in that article I linked, or is that unnecessary?
-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: February 2, 2025 23:45 To: ceph-users@ceph.io Subject: [ceph-users] Re: cephadm: Move DB/WAL from HDD to SSD
*** This is an EXTERNAL email. Please exercise caution. DO NOT open attachments or click links from unknown senders or unexpected email. ***
Hi,
When you ran the ceph-volume command, did you do it within the container of the OSD? Running it with the container of the OSD would make sense as to why I was getting the errors I was getting.
yes, I ran the commands within the OSD container. Don't forget to run the "migrate" command as well after you created a new db.
Just one more remark about your followup question in the initial post:
Followup question: if I do seem to have to go the route of destroying the OSD and re-creating it with the flags for point the DB/WAL to the SSD partition, the syntax I have seen is this: ceph-volume lvm prepare --bluestore --block.db --block.wal --data VOLUME_GROUP/LOGICAL_VOLUME
Beside the fact that you assume correctly and the command lacks the db/wal device specification, I strongly recommend to let cephadm handle the entire process of OSD creation, just ensure you have a fitting spec file. Don't prepare any LVs manually beforehand, except for such a case as right now where you want to move DBs out of the main device. But if you want to create OSDs from scratch, use the orchestrator, it does all of the work. And don't forget the --dry-run flag if you want to apply a new spec (ceph orch apply -i osd-spec.yaml --dry-run).
Regards, Eugen
Zitat von Alan Murrell <Alan@t-net.ca>:
Hello,
I ran the commands from the "regular" cephadm shell ('cephadm shell --') but did not specify an OSD (I wasn't aware you could do that)
When you ran the ceph-volume command, did you do it within the container of the OSD? Running it with the container of the OSD would make sense as to why I was getting the errors I was getting.
I can give it a try and let you know how it goes, and if necessary go from there.
Sent from my mobile device. Please excuse brevity and ttpos.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Definitely stop the daemon (and set the noout flag), otherwise you won’t be able to modify anything.
OK, so then basically just run through the steps/commands in that article that I linked in my original post, but just make sure to run the 'ceph-volume' commands from within the specific OSD's container, yes? (Sorry, I just want to be 100% certain before trying this again 😊 )
Exactly, yes. Zitat von Alan Murrell <Alan@t-net.ca>:
Definitely stop the daemon (and set the noout flag), otherwise you won’t be able to modify anything.
OK, so then basically just run through the steps/commands in that article that I linked in my original post, but just make sure to run the 'ceph-volume' commands from within the specific OSD's container, yes?
(Sorry, I just want to be 100% certain before trying this again 😊 ) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Ok, just gave it a try and I am still running into an error. Here is exactly what I did: I logged on to my host where osd.10 is Deleted my current VG and LV's on my NVME that will hold the WAL/DB, as I kind of liked what you used. My VG is called "cephdb03" and my LVs are called "ceph-osd-dbX", where "X" is 1 through 4. Ran the command to stop osd.10 service: systemctl stop ceph-474264fe-b00e-11ee-b586-ac1f6b0ff21a@osd.10 connected to the general cephadm shell and ran: ceph-volume lvm activate --all --no-systemd Exited the general shel and entered the container for OSD 10: cephadm shell name osd.10 Ran the ceph-volume command to create the new DB on cephdb03/ceph-osd-db1 : ceph-volume lvm new-db --osd-id 10 --osd-fsid 474264fe-b00e-11ee-b586-ac1f6b0ff21a --target cephdb03/ceph-osd-db1 Got the following error: --> Unable to find any LV for source OSD: id:10 fsid:474264fe-b00e-11ee-b586-ac1f6b0ff21a Unexpected error, terminating
I'm not sure why it fails, but it seems like you deviate a bit from the instructions. If you want to migrate the db to a new device, you need to specify an existing VG and LV, in this case it is not created for you. And I'm not really sure why you run 'ceph-volume lvm activate --all --no-systemd', that's not necessary. So I'll try to provide a complete list of steps, hopefully that works for you as it does for me: 1. soc9-ceph:~ # vgcreate ceph-db /dev/vdf 2. soc9-ceph:~ # lvcreate -L 5G -n ceph-osd0-db ceph-db (mind the LV size, just a test cluster here) 3. soc9-ceph:~ # ceph orch daemon stop osd.0 4. soc9-ceph:~ # cephadm shell --name osd.0 5. [ceph: root@soc9-ceph /]# ceph-volume lvm new-db --osd-id 0 --osd-fsid fb69ba54-4d56-4c90-a855-6b350d186df5 --target ceph-db/ceph-osd0-db 6. [ceph: root@soc9-ceph /]# ceph-volume lvm migrate --osd-id 0 --osd-fsid fb69ba54-4d56-4c90-a855-6b350d186df5 --from /var/lib/ceph/osd/ceph-0/block --target ceph-db/ceph-osd0-db 7. Exit shell 8. soc9-ceph:~ # ceph orch daemon start osd.0 9. Verify db config: soc9-ceph:~ # ceph tell osd.0 perf dump bluefs | jq -r '.[].db_total_bytes,.[].db_used_bytes' 5368700928 47185920 So as you see, the OSD has picked up the new db device and uses 47 MB (it's an empty test cluster). Also note that this is a single-node cluster, so the orchestrator commands and shell commands are all executed on the same host. Let me know how it goes. Zitat von Alan Murrell <Alan@t-net.ca>:
Ok, just gave it a try and I am still running into an error. Here is exactly what I did:
I logged on to my host where osd.10 is
Deleted my current VG and LV's on my NVME that will hold the WAL/DB, as I kind of liked what you used. My VG is called "cephdb03" and my LVs are called "ceph-osd-dbX", where "X" is 1 through 4.
Ran the command to stop osd.10 service:
systemctl stop ceph-474264fe-b00e-11ee-b586-ac1f6b0ff21a@osd.10
connected to the general cephadm shell and ran:
ceph-volume lvm activate --all --no-systemd
Exited the general shel and entered the container for OSD 10:
cephadm shell name osd.10
Ran the ceph-volume command to create the new DB on cephdb03/ceph-osd-db1 :
ceph-volume lvm new-db --osd-id 10 --osd-fsid 474264fe-b00e-11ee-b586-ac1f6b0ff21a --target cephdb03/ceph-osd-db1
Got the following error:
--> Unable to find any LV for source OSD: id:10 fsid:474264fe-b00e-11ee-b586-ac1f6b0ff21a Unexpected error, terminating
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eugen, It is still failing with exactly the same error, which seems to indicate it cannot find a Logical Volume for osd.10.
If you want to migrate the db to a new device, you need to specify an existing VG and LV, in this case it is not created for you.
I had actually already done that with the vgcreate and lvcreate commands. Here the exact commands I did: vgcreate cephdb03 /dev/nvme5n1 lvcreate -L 232G -n ceph-osd-db1 lvcreate -L 232G -n ceph-osd-db2 lvcreate -L 232G -n ceph-osd-db3 lvcreate -L 232G -n ceph-osd-db4 So I did already have the LVM volumes created.
And I'm not really sure why you run 'ceph-volume lvm activate --all --no-systemd', that's not necessary
That is what was in the article I linked in my first post: ' https://docs.clyso.com/blog/ceph-volume-create-wal-db-on-separate-device-for...' I followed the rest of your steps exactly: From general cephadm shell (which I was logged in to from another node - 'cephadm shell --', though I am not sure if I could have done this from the OSD 10 shell also?): ceph orch daemon stop osd.10 I confirmed OSD 10 was marked "Down" in the web admin. Then from the host the OSD and NVME WAL/DB drive is on: root@cephnone03:~# cephadm shell --name osd.10 root@cephnode03:/# ceph-volume lvm new-db --osd-id 10 --osd-fsid 474264fe-b00e-11ee-b586-ac1f6b0ff21a --target cephdb03/ceph-osd-db1 and then that is where I get the error. My OSDs are definitely using LVM and I can see them if I run 'lvdisplay' or 'lvs', so I am not sure why ceph-volume says it can't find a LV for OSD 10. -----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: February 3, 2025 11:30 To: ceph-users@ceph.io Subject: [ceph-users] Re: cephadm: Move DB/WAL from HDD to SSD *** This is an EXTERNAL email. Please exercise caution. DO NOT open attachments or click links from unknown senders or unexpected email. *** I'm not sure why it fails, but it seems like you deviate a bit from the instructions. If you want to migrate the db to a new device, you need to specify an existing VG and LV, in this case it is not created for you. And I'm not really sure why you run 'ceph-volume lvm activate --all --no-systemd', that's not necessary. So I'll try to provide a complete list of steps, hopefully that works for you as it does for me: 1. soc9-ceph:~ # vgcreate ceph-db /dev/vdf 2. soc9-ceph:~ # lvcreate -L 5G -n ceph-osd0-db ceph-db (mind the LV size, just a test cluster here) 3. soc9-ceph:~ # ceph orch daemon stop osd.0 4. soc9-ceph:~ # cephadm shell --name osd.0 5. [ceph: root@soc9-ceph /]# ceph-volume lvm new-db --osd-id 0 --osd-fsid fb69ba54-4d56-4c90-a855-6b350d186df5 --target ceph-db/ceph-osd0-db 6. [ceph: root@soc9-ceph /]# ceph-volume lvm migrate --osd-id 0 --osd-fsid fb69ba54-4d56-4c90-a855-6b350d186df5 --from /var/lib/ceph/osd/ceph-0/block --target ceph-db/ceph-osd0-db 7. Exit shell 8. soc9-ceph:~ # ceph orch daemon start osd.0 9. Verify db config: soc9-ceph:~ # ceph tell osd.0 perf dump bluefs | jq -r '.[].db_total_bytes,.[].db_used_bytes' 5368700928 47185920 So as you see, the OSD has picked up the new db device and uses 47 MB (it's an empty test cluster). Also note that this is a single-node cluster, so the orchestrator commands and shell commands are all executed on the same host. Let me know how it goes. Zitat von Alan Murrell <Alan@t-net.ca>:
Ok, just gave it a try and I am still running into an error. Here is exactly what I did:
I logged on to my host where osd.10 is
Deleted my current VG and LV's on my NVME that will hold the WAL/DB, as I kind of liked what you used. My VG is called "cephdb03" and my LVs are called "ceph-osd-dbX", where "X" is 1 through 4.
Ran the command to stop osd.10 service:
systemctl stop ceph-474264fe-b00e-11ee-b586-ac1f6b0ff21a@osd.10
connected to the general cephadm shell and ran:
ceph-volume lvm activate --all --no-systemd
Exited the general shel and entered the container for OSD 10:
cephadm shell name osd.10
Ran the ceph-volume command to create the new DB on cephdb03/ceph-osd-db1 :
ceph-volume lvm new-db --osd-id 10 --osd-fsid 474264fe-b00e-11ee-b586-ac1f6b0ff21a --target cephdb03/ceph-osd-db1
Got the following error:
--> Unable to find any LV for source OSD: id:10 fsid:474264fe-b00e-11ee-b586-ac1f6b0ff21a Unexpected error, terminating
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Oh, was my formatting really that bad as it looks in your response? I apologize for that! Can you show the output of: cephadm ceph-volume lvm list 10 Zitat von Alan Murrell <Alan@t-net.ca>:
Hi Eugen,
It is still failing with exactly the same error, which seems to indicate it cannot find a Logical Volume for osd.10.
If you want to migrate the db to a new device, you need to specify an existing VG and LV, in this case it is not created for you.
I had actually already done that with the vgcreate and lvcreate commands. Here the exact commands I did:
vgcreate cephdb03 /dev/nvme5n1 lvcreate -L 232G -n ceph-osd-db1 lvcreate -L 232G -n ceph-osd-db2 lvcreate -L 232G -n ceph-osd-db3 lvcreate -L 232G -n ceph-osd-db4
So I did already have the LVM volumes created.
And I'm not really sure why you run 'ceph-volume lvm activate --all --no-systemd', that's not necessary
That is what was in the article I linked in my first post: ' https://docs.clyso.com/blog/ceph-volume-create-wal-db-on-separate-device-for...'
I followed the rest of your steps exactly:
From general cephadm shell (which I was logged in to from another node - 'cephadm shell --', though I am not sure if I could have done this from the OSD 10 shell also?):
ceph orch daemon stop osd.10
I confirmed OSD 10 was marked "Down" in the web admin. Then from the host the OSD and NVME WAL/DB drive is on:
root@cephnone03:~# cephadm shell --name osd.10 root@cephnode03:/# ceph-volume lvm new-db --osd-id 10 --osd-fsid 474264fe-b00e-11ee-b586-ac1f6b0ff21a --target cephdb03/ceph-osd-db1
and then that is where I get the error.
My OSDs are definitely using LVM and I can see them if I run 'lvdisplay' or 'lvs', so I am not sure why ceph-volume says it can't find a LV for OSD 10.
-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: February 3, 2025 11:30 To: ceph-users@ceph.io Subject: [ceph-users] Re: cephadm: Move DB/WAL from HDD to SSD
*** This is an EXTERNAL email. Please exercise caution. DO NOT open attachments or click links from unknown senders or unexpected email. ***
I'm not sure why it fails, but it seems like you deviate a bit from the instructions. If you want to migrate the db to a new device, you need to specify an existing VG and LV, in this case it is not created for you. And I'm not really sure why you run 'ceph-volume lvm activate --all --no-systemd', that's not necessary. So I'll try to provide a complete list of steps, hopefully that works for you as it does for me:
1. soc9-ceph:~ # vgcreate ceph-db /dev/vdf 2. soc9-ceph:~ # lvcreate -L 5G -n ceph-osd0-db ceph-db (mind the LV size, just a test cluster here) 3. soc9-ceph:~ # ceph orch daemon stop osd.0 4. soc9-ceph:~ # cephadm shell --name osd.0 5. [ceph: root@soc9-ceph /]# ceph-volume lvm new-db --osd-id 0 --osd-fsid fb69ba54-4d56-4c90-a855-6b350d186df5 --target ceph-db/ceph-osd0-db 6. [ceph: root@soc9-ceph /]# ceph-volume lvm migrate --osd-id 0 --osd-fsid fb69ba54-4d56-4c90-a855-6b350d186df5 --from /var/lib/ceph/osd/ceph-0/block --target ceph-db/ceph-osd0-db 7. Exit shell 8. soc9-ceph:~ # ceph orch daemon start osd.0 9. Verify db config: soc9-ceph:~ # ceph tell osd.0 perf dump bluefs | jq -r '.[].db_total_bytes,.[].db_used_bytes' 5368700928 47185920
So as you see, the OSD has picked up the new db device and uses 47 MB (it's an empty test cluster). Also note that this is a single-node cluster, so the orchestrator commands and shell commands are all executed on the same host.
Let me know how it goes.
Zitat von Alan Murrell <Alan@t-net.ca>:
Ok, just gave it a try and I am still running into an error. Here is exactly what I did:
I logged on to my host where osd.10 is
Deleted my current VG and LV's on my NVME that will hold the WAL/DB, as I kind of liked what you used. My VG is called "cephdb03" and my LVs are called "ceph-osd-dbX", where "X" is 1 through 4.
Ran the command to stop osd.10 service:
systemctl stop ceph-474264fe-b00e-11ee-b586-ac1f6b0ff21a@osd.10
connected to the general cephadm shell and ran:
ceph-volume lvm activate --all --no-systemd
Exited the general shel and entered the container for OSD 10:
cephadm shell name osd.10
Ran the ceph-volume command to create the new DB on cephdb03/ceph-osd-db1 :
ceph-volume lvm new-db --osd-id 10 --osd-fsid 474264fe-b00e-11ee-b586-ac1f6b0ff21a --target cephdb03/ceph-osd-db1
Got the following error:
--> Unable to find any LV for source OSD: id:10 fsid:474264fe-b00e-11ee-b586-ac1f6b0ff21a Unexpected error, terminating
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Oh, was my formatting really that bad as it looks in your response? I apologize for that!
It was, actually; all the commands were one after the other. It's OK; I was able to make it out. Might have something to do with how my MUA (Outlook) formatted it.
Can you show the output of:
cephadm ceph-volume lvm list 10
Copied-and-pasted from my terminal: root@cephnode03:~# cephadm ceph-volume lvm list 10 Inferring fsid 474264fe-b00e-11ee-b586-ac1f6b0ff21a ====== osd.10 ====== [block] /dev/ceph-0c438d0c-f25a-41b2-b478-b0f98558f585/osd-block-fb0e0a45-75a0-4400-9b1f-7568f185544c block device /dev/ceph-0c438d0c-f25a-41b2-b478-b0f98558f585/osd-block-fb0e0a45-75a0-4400-9b1f-7568f185544c block uuid iHgcum-NAV1-cLy3-PLa3-D8jp-5qXe-L6Vy8u cephx lockbox secret cluster fsid 474264fe-b00e-11ee-b586-ac1f6b0ff21a cluster name ceph crush device class encrypted 0 osd fsid fb0e0a45-75a0-4400-9b1f-7568f185544c osd id 10 osdspec affinity cost_capacity type block vdo 0 devices /dev/sda
Alright, and does either cephadm.log (/var/log/ceph/cephadm.log) or ceph-volume.log (/var/log/ceph/{FSID}/ceph-volume.log) provide more details? Which OS is the host running on? Zitat von Alan Murrell <Alan@t-net.ca>:
Oh, was my formatting really that bad as it looks in your response? I apologize for that!
It was, actually; all the commands were one after the other. It's OK; I was able to make it out. Might have something to do with how my MUA (Outlook) formatted it.
Can you show the output of:
cephadm ceph-volume lvm list 10
Copied-and-pasted from my terminal:
root@cephnode03:~# cephadm ceph-volume lvm list 10 Inferring fsid 474264fe-b00e-11ee-b586-ac1f6b0ff21a
====== osd.10 ======
[block] /dev/ceph-0c438d0c-f25a-41b2-b478-b0f98558f585/osd-block-fb0e0a45-75a0-4400-9b1f-7568f185544c
block device /dev/ceph-0c438d0c-f25a-41b2-b478-b0f98558f585/osd-block-fb0e0a45-75a0-4400-9b1f-7568f185544c block uuid iHgcum-NAV1-cLy3-PLa3-D8jp-5qXe-L6Vy8u cephx lockbox secret cluster fsid 474264fe-b00e-11ee-b586-ac1f6b0ff21a cluster name ceph crush device class encrypted 0 osd fsid fb0e0a45-75a0-4400-9b1f-7568f185544c osd id 10 osdspec affinity cost_capacity type block vdo 0 devices /dev/sda
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Alright, and does either cephadm.log (/var/log/ceph/cephadm.log) or ceph-volume.log (/var/log/ceph/{FSID}/ceph-volume.log) provide more details? Which OS is the host running on?
The host OS is Debian 12. In the cephadm.log file, I don't see anything around the time I did the ceph-volume commands that indicate any errors. Here is a copy-and-paste from the log for that period: --- START --- cephadm ['--timeout', '895', 'gather-facts'] 2025-02-03 10:07:35,254 7fb630b9e2c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2025-02-03 10:08:45,086 7f09e25412c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2025-02-03 10:09:46,607 7f8f6c0f82c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2025-02-03 10:10:47,912 7fdb3b6412c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2025-02-03 10:11:49,418 7f5c587942c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2025-02-03 10:11:50,567 7fbfd40af2c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--image', 'quay.io/ceph/ceph@sha256:f8d467dcf49d13b8ea42229d89be642581 110175d8ce36e216aefc9b32b0854d', '--timeout', '895', 'list-networks'] 2025-02-03 10:11:51,125 7f39826592c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--image', 'quay.io/ceph/ceph@sha256:f8d467dcf49d13b8ea42229d89be642581 110175d8ce36e216aefc9b32b0854d', '--timeout', '895', 'ceph-volume', '--fsid', '4 74264fe-b00e-11ee-b586-ac1f6b0ff21a', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2025-02-03 10:11:55,935 7f39826592c0 INFO Inferring config /var/lib/ceph/474264f e-b00e-11ee-b586-ac1f6b0ff21a/mon.cephnode03/config 2025-02-03 10:11:55,935 7f39826592c0 DEBUG Using specified fsid: 474264fe-b00e-1 1ee-b586-ac1f6b0ff21a 2025-02-03 10:12:02,524 7fd2367902c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--env', 'CEPH_VOLUME_OSDSPEC_AFFINITY=cost_capacity', '--image', 'quay .io/ceph/ceph@sha256:f8d467dcf49d13b8ea42229d89be642581110175d8ce36e216aefc9b32b 0854d', '--timeout', '895', 'ceph-volume', '--fsid', '474264fe-b00e-11ee-b586-ac 1f6b0ff21a', '--config-json', '-', '--', 'lvm', 'batch', '--no-auto', '/dev/sda' , '/dev/sdb', '/dev/sdc', '/dev/sdd', '--yes', '--no-systemd'] 2025-02-03 10:12:07,346 7fd2367902c0 INFO Inferring config /var/lib/ceph/474264f e-b00e-11ee-b586-ac1f6b0ff21a/mon.cephnode03/config 2025-02-03 10:12:07,346 7fd2367902c0 DEBUG Using specified fsid: 474264fe-b00e-1 1ee-b586-ac1f6b0ff21a 2025-02-03 10:12:10,321 7f56b8d692c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--image', 'quay.io/ceph/ceph@sha256:f8d467dcf49d13b8ea42229d89be642581 110175d8ce36e216aefc9b32b0854d', '--timeout', '895', 'ceph-volume', '--fsid', '4 74264fe-b00e-11ee-b586-ac1f6b0ff21a', '--', 'lvm', 'list', '--format', 'json'] 2025-02-03 10:12:15,130 7f56b8d692c0 INFO Inferring config /var/lib/ceph/474264f e-b00e-11ee-b586-ac1f6b0ff21a/mon.cephnode03/config 2025-02-03 10:12:15,130 7f56b8d692c0 DEBUG Using specified fsid: 474264fe-b00e-1 1ee-b586-ac1f6b0ff21a 2025-02-03 10:12:16,773 7fb154ac02c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--image', 'quay.io/ceph/ceph@sha256:f8d467dcf49d13b8ea42229d89be642581 110175d8ce36e216aefc9b32b0854d', '--timeout', '895', 'ceph-volume', '--fsid', '4 74264fe-b00e-11ee-b586-ac1f6b0ff21a', '--', 'raw', 'list', '--format', 'json'] 2025-02-03 10:12:21,609 7fb154ac02c0 INFO Inferring config /var/lib/ceph/474264f e-b00e-11ee-b586-ac1f6b0ff21a/mon.cephnode03/config 2025-02-03 10:12:21,609 7fb154ac02c0 DEBUG Using specified fsid: 474264fe-b00e-1 1ee-b586-ac1f6b0ff21a 2025-02-03 10:13:24,527 7f62a01072c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2025-02-03 10:14:26,085 7fd1891bd2c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2025-02-03 10:15:27,587 7f336fc182c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2025-02-03 10:16:29,014 7f8f959312c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--image', 'quay.io/ceph/ceph@sha256:f8d467dcf49d13b8ea42229d89be642581 110175d8ce36e216aefc9b32b0854d', '--timeout', '895', 'ls'] 2025-02-03 10:16:35,992 7f84132152c0 DEBUG ------------------------------------- ------------------------------------------- --- END --- The ceph-volume.log file doesn't seem to contain anything either. It skips from 10:01am to 10:32am, whereas I did my first try at about 10:10am.
Oh, I think I found the error. You seem to provide the cluster FSID, not the OSD FSID. Your OSD FSID is: fb0e0a45-75a0-4400-9b1f-7568f185544c Check the lvm list output. Try this: ceph-volume lvm new-db --osd-id 10 --osd-fsid fb0e0a45-75a0-4400-9b1f-7568f185544c --target cephdb03/ceph-osd-db1 Zitat von Alan Murrell <Alan@t-net.ca>:
Alright, and does either cephadm.log (/var/log/ceph/cephadm.log) or ceph-volume.log (/var/log/ceph/{FSID}/ceph-volume.log) provide more details? Which OS is the host running on?
The host OS is Debian 12.
In the cephadm.log file, I don't see anything around the time I did the ceph-volume commands that indicate any errors. Here is a copy-and-paste from the log for that period:
--- START --- cephadm ['--timeout', '895', 'gather-facts'] 2025-02-03 10:07:35,254 7fb630b9e2c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2025-02-03 10:08:45,086 7f09e25412c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2025-02-03 10:09:46,607 7f8f6c0f82c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2025-02-03 10:10:47,912 7fdb3b6412c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2025-02-03 10:11:49,418 7f5c587942c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2025-02-03 10:11:50,567 7fbfd40af2c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--image', 'quay.io/ceph/ceph@sha256:f8d467dcf49d13b8ea42229d89be642581 110175d8ce36e216aefc9b32b0854d', '--timeout', '895', 'list-networks'] 2025-02-03 10:11:51,125 7f39826592c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--image', 'quay.io/ceph/ceph@sha256:f8d467dcf49d13b8ea42229d89be642581 110175d8ce36e216aefc9b32b0854d', '--timeout', '895', 'ceph-volume', '--fsid', '4 74264fe-b00e-11ee-b586-ac1f6b0ff21a', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2025-02-03 10:11:55,935 7f39826592c0 INFO Inferring config /var/lib/ceph/474264f e-b00e-11ee-b586-ac1f6b0ff21a/mon.cephnode03/config 2025-02-03 10:11:55,935 7f39826592c0 DEBUG Using specified fsid: 474264fe-b00e-1 1ee-b586-ac1f6b0ff21a 2025-02-03 10:12:02,524 7fd2367902c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--env', 'CEPH_VOLUME_OSDSPEC_AFFINITY=cost_capacity', '--image', 'quay .io/ceph/ceph@sha256:f8d467dcf49d13b8ea42229d89be642581110175d8ce36e216aefc9b32b 0854d', '--timeout', '895', 'ceph-volume', '--fsid', '474264fe-b00e-11ee-b586-ac 1f6b0ff21a', '--config-json', '-', '--', 'lvm', 'batch', '--no-auto', '/dev/sda' , '/dev/sdb', '/dev/sdc', '/dev/sdd', '--yes', '--no-systemd'] 2025-02-03 10:12:07,346 7fd2367902c0 INFO Inferring config /var/lib/ceph/474264f e-b00e-11ee-b586-ac1f6b0ff21a/mon.cephnode03/config 2025-02-03 10:12:07,346 7fd2367902c0 DEBUG Using specified fsid: 474264fe-b00e-1 1ee-b586-ac1f6b0ff21a 2025-02-03 10:12:10,321 7f56b8d692c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--image', 'quay.io/ceph/ceph@sha256:f8d467dcf49d13b8ea42229d89be642581 110175d8ce36e216aefc9b32b0854d', '--timeout', '895', 'ceph-volume', '--fsid', '4 74264fe-b00e-11ee-b586-ac1f6b0ff21a', '--', 'lvm', 'list', '--format', 'json'] 2025-02-03 10:12:15,130 7f56b8d692c0 INFO Inferring config /var/lib/ceph/474264f e-b00e-11ee-b586-ac1f6b0ff21a/mon.cephnode03/config 2025-02-03 10:12:15,130 7f56b8d692c0 DEBUG Using specified fsid: 474264fe-b00e-1 1ee-b586-ac1f6b0ff21a 2025-02-03 10:12:16,773 7fb154ac02c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--image', 'quay.io/ceph/ceph@sha256:f8d467dcf49d13b8ea42229d89be642581 110175d8ce36e216aefc9b32b0854d', '--timeout', '895', 'ceph-volume', '--fsid', '4 74264fe-b00e-11ee-b586-ac1f6b0ff21a', '--', 'raw', 'list', '--format', 'json'] 2025-02-03 10:12:21,609 7fb154ac02c0 INFO Inferring config /var/lib/ceph/474264f e-b00e-11ee-b586-ac1f6b0ff21a/mon.cephnode03/config 2025-02-03 10:12:21,609 7fb154ac02c0 DEBUG Using specified fsid: 474264fe-b00e-1 1ee-b586-ac1f6b0ff21a 2025-02-03 10:13:24,527 7f62a01072c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2025-02-03 10:14:26,085 7fd1891bd2c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2025-02-03 10:15:27,587 7f336fc182c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2025-02-03 10:16:29,014 7f8f959312c0 DEBUG ------------------------------------- ------------------------------------------- cephadm ['--image', 'quay.io/ceph/ceph@sha256:f8d467dcf49d13b8ea42229d89be642581 110175d8ce36e216aefc9b32b0854d', '--timeout', '895', 'ls'] 2025-02-03 10:16:35,992 7f84132152c0 DEBUG ------------------------------------- ------------------------------------------- --- END ---
The ceph-volume.log file doesn't seem to contain anything either. It skips from 10:01am to 10:32am, whereas I did my first try at about 10:10am. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Eugen Block NDE Netzdesign und -entwicklung AG voice: +49 40 5595175 Postfach 61 03 15 e-mail: eblock@nde.ag D-22423 Hamburg Vorstand: Jens-U. Mozdzen Aufsichtsratsvorsitzende: Angelika Torlée-Mozdzen Sitz und Registergericht: Hamburg, HRB 90934 USt-IdNr: DE 814 013 983
ceph-volume lvm new-db --osd-id 10 --osd-fsid fb0e0a45-75a0-4400-9b1f-7568f185544c --target cephdb03/ceph-osd-db1
OMG, that worked. I hit a snag on the "migrate" command, but I think I know what I need to do, but wanted to confirm. When I ran: ceph-volume lvm migrate --osd-id 10 --osd-fsid fb0e0a45-75a0-4400-9b1f-7568f185544c --from /var/lib/ceph/osd/ceph-10/block --target cephdb03/ceph-osd-db1 I got the following error: ceph-volume lvm migrate: error: argument --from: invalid choice: '/var/lib/ceph/osd/ceph-10/block' (choose from 'data', 'db', 'wal') so I *think* I need to change the command to: ceph-volume lvm migrate --osd-id 10 --osd-fsid fb0e0a45-75a0-4400-9b1f-7568f185544c --from db --target cephdb03/ceph-osd-db1 and I think that takes care of both the 'db' and the 'wal'? I recall reading in another help doc that specifying just the 'db' without also specifying 'wal' puts (or moves) the 'db' and the 'wal' on to the same device?
Response inline. Zitat von Alan Murrell <Alan@t-net.ca>:
ceph-volume lvm new-db --osd-id 10 --osd-fsid fb0e0a45-75a0-4400-9b1f-7568f185544c --target cephdb03/ceph-osd-db1
OMG, that worked. I hit a snag on the "migrate" command, but I think I know what I need to do, but wanted to confirm.
When I ran:
ceph-volume lvm migrate --osd-id 10 --osd-fsid fb0e0a45-75a0-4400-9b1f-7568f185544c --from /var/lib/ceph/osd/ceph-10/block --target cephdb03/ceph-osd-db1
I got the following error:
ceph-volume lvm migrate: error: argument --from: invalid choice: '/var/lib/ceph/osd/ceph-10/block' (choose from 'data', 'db', 'wal')
so I *think* I need to change the command to:
ceph-volume lvm migrate --osd-id 10 --osd-fsid fb0e0a45-75a0-4400-9b1f-7568f185544c --from db --target cephdb03/ceph-osd-db1
You didn’t have a dedicated db yet, you want to migrate it out of the data device to the new block.db, so it’s --from data.
and I think that takes care of both the 'db' and the 'wal'? I recall reading in another help doc that specifying just the 'db' without also specifying 'wal' puts (or moves) the 'db' and the 'wal' on to the same device?
Correct, wal is part of the db if not specified otherwise.
OK, I think I am good now. I have completed the rest of the steps (exit shell, start the osd.10 daemon back up again, waited for it to be marked as "Up") and then I ran: ceph osd metadata 10 and all the bluestore_db items are pointing to the SSD now. OK, so now to note these steps/commands in to "Ceph Notes" and the repeat the process for the other HDDs on this node and my other four 😊 Thank you *so* much for your help (and patience!)
I'm glad it worked. I thought those steps had already been added to the docs (cephadm specific), but I couldn't find them either. I'll ping Zac about it. Zitat von Alan Murrell <Alan@t-net.ca>:
OK, I think I am good now. I have completed the rest of the steps (exit shell, start the osd.10 daemon back up again, waited for it to be marked as "Up") and then I ran:
ceph osd metadata 10
and all the bluestore_db items are pointing to the SSD now.
OK, so now to note these steps/commands in to "Ceph Notes" and the repeat the process for the other HDDs on this node and my other four 😊
Thank you *so* much for your help (and patience!) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I strongly recommend to let cephadm handle the entire process of OSD creation, just ensure you have a fitting spec file. Don't prepare any LVs manually beforehand, except for such a case as right now where you want to move DBs out of the main device.
So if I create a spec file so Ceph automatically puts the DBs of the rotational devices onto a specified SSD (whether I do it by size, vendor, model, or all three), Ceph will crate the necessary LVs on the SSD automatically? How does it determine the size to use (or would that also get specified in the spec file)? What if I have an SSD with he LVs already created? I assume Ceph wouldn't see that as a valid device for the DB/WAL?
Hi, response inline... Zitat von Alan Murrell <Alan@t-net.ca>:
I strongly recommend to let cephadm handle the entire process of OSD creation, just ensure you have a fitting spec file. Don't prepare any LVs manually beforehand, except for such a case as right now where you want to move DBs out of the main device.
So if I create a spec file so Ceph automatically puts the DBs of the rotational devices onto a specified SSD (whether I do it by size, vendor, model, or all three), Ceph will crate the necessary LVs on the SSD automatically? How does it determine the size to use (or would that also get specified in the spec file)?
Yes, Ceph creates VGs/LVs automatically if there's enough free space. You can also specify the db/wal sizes in your spec file.
What if I have an SSD with he LVs already created? I assume Ceph wouldn't see that as a valid device for the DB/WAL?
I haven't tested each possible scenario in detail, but in general, cephadm can re-use freed up space of the SSD, for example after HDD replacement which also requires to zap the respective block.db LV. I'm not familiar which filters ceph-volume uses to determine if it can use the existing SSD.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Alan Murrell
-
Eugen Block