which is definitive: /var/lib/ceph symlinks or ceph-volume?
I am in a situation where I see conflicting information. On the one hand, ls -l /var/lib/ceph/osd/ceph-7 shows a symlink for block device, but no block.db On the other hand, ceph-volume lvm list claims that there is a separate db device registered for osd 7 how can I know which one is correct? (This is currently ceph nautilus) -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown@medata.com| www.medata.com
Is the OSD up and running? Do you see IO on the dedicated block.db LV when running a 'ceph tell osd.7 bench'? It sounds very strange, I haven't seen that. Zitat von Philip Brown <pbrown@medata.com>:
I am in a situation where I see conflicting information. On the one hand, ls -l /var/lib/ceph/osd/ceph-7 shows a symlink for block device, but no block.db
On the other hand,
ceph-volume lvm list claims that there is a separate db device registered for osd 7
how can I know which one is correct?
(This is currently ceph nautilus)
-- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown@medata.com| www.medata.com _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
It was up and running. I should mention that the situation came about when I was testing OSD rebuild. So I had a previously autogenerated hybrid OSD, with SSD db, and HDD block dev. I then stopped the OSD, wiped HDD, .... did "ceph osd rm" and ceph auth rm. and then rebuilt with ceph-volume.. with the wrong flags. I belive I used: ceph-volume lvm create --block /dev/xxx --journal /dev/blahblah After that, ceph-volume lvm list claimed it was using the ssd again as db device. but ls -l on /var/lib/ceph/osd/XXXX only showed symlink for block dev. This was on nautilus. ----- Original Message ----- From: "Eugen Block" <eblock@nde.ag> To: "ceph-users" <ceph-users@ceph.io> Sent: Tuesday, April 6, 2021 5:20:40 AM Subject: [ceph-users] Re: which is definitive: /var/lib/ceph symlinks or ceph-volume? Is the OSD up and running? Do you see IO on the dedicated block.db LV when running a 'ceph tell osd.7 bench'? It sounds very strange, I haven't seen that. Zitat von Philip Brown <pbrown@medata.com>:
I am in a situation where I see conflicting information. On the one hand, ls -l /var/lib/ceph/osd/ceph-7 shows a symlink for block device, but no block.db
On the other hand,
ceph-volume lvm list claims that there is a separate db device registered for osd 7
how can I know which one is correct?
(This is currently ceph nautilus)
-- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown@medata.com| www.medata.com _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Did you recreate the OSD with the same UUID? I'm guessing ceph-volume just reads LV tags and thinks there's a block.db for that OSD. Before creating the OSD you should also wipe the LV containing the block.db for the failed OSD. Zitat von Philip Brown <pbrown@medata.com>:
It was up and running.
I should mention that the situation came about when I was testing OSD rebuild.
So I had a previously autogenerated hybrid OSD, with SSD db, and HDD block dev.
I then stopped the OSD, wiped HDD, .... did "ceph osd rm" and ceph auth rm. and then rebuilt with ceph-volume.. with the wrong flags. I belive I used:
ceph-volume lvm create --block /dev/xxx --journal /dev/blahblah
After that, ceph-volume lvm list claimed it was using the ssd again as db device. but ls -l on /var/lib/ceph/osd/XXXX only showed symlink for block dev.
This was on nautilus.
----- Original Message ----- From: "Eugen Block" <eblock@nde.ag> To: "ceph-users" <ceph-users@ceph.io> Sent: Tuesday, April 6, 2021 5:20:40 AM Subject: [ceph-users] Re: which is definitive: /var/lib/ceph symlinks or ceph-volume?
Is the OSD up and running? Do you see IO on the dedicated block.db LV when running a 'ceph tell osd.7 bench'? It sounds very strange, I haven't seen that.
Zitat von Philip Brown <pbrown@medata.com>:
I am in a situation where I see conflicting information. On the one hand, ls -l /var/lib/ceph/osd/ceph-7 shows a symlink for block device, but no block.db
On the other hand,
ceph-volume lvm list claims that there is a separate db device registered for osd 7
how can I know which one is correct?
(This is currently ceph nautilus)
-- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown@medata.com| www.medata.com _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Where does it read it from? does it keep it in the block.db lv, or the block dev lv, or both? I removed the vg from the block dev and did wipefs, if I recall. ----- Original Message ----- From: "Eugen Block" <eblock@nde.ag> To: "Philip Brown" <pbrown@medata.com> Cc: "ceph-users" <ceph-users@ceph.io> Sent: Tuesday, April 6, 2021 9:06:50 AM Subject: [BULK] Re: [ceph-users] Re: which is definitive: /var/lib/ceph symlinks or ceph-volume? Did you recreate the OSD with the same UUID? I'm guessing ceph-volume just reads LV tags and thinks there's a block.db for that OSD. Before creating the OSD you should also wipe the LV containing the block.db for the failed OSD. Zitat von Philip Brown <pbrown@medata.com>:
It was up and running.
I should mention that the situation came about when I was testing OSD rebuild.
So I had a previously autogenerated hybrid OSD, with SSD db, and HDD block dev.
I then stopped the OSD, wiped HDD, .... did "ceph osd rm" and ceph auth rm. and then rebuilt with ceph-volume.. with the wrong flags. I belive I used:
ceph-volume lvm create --block /dev/xxx --journal /dev/blahblah
After that, ceph-volume lvm list claimed it was using the ssd again as db device. but ls -l on /var/lib/ceph/osd/XXXX only showed symlink for block dev.
This was on nautilus.
----- Original Message ----- From: "Eugen Block" <eblock@nde.ag> To: "ceph-users" <ceph-users@ceph.io> Sent: Tuesday, April 6, 2021 5:20:40 AM Subject: [ceph-users] Re: which is definitive: /var/lib/ceph symlinks or ceph-volume?
Is the OSD up and running? Do you see IO on the dedicated block.db LV when running a 'ceph tell osd.7 bench'? It sounds very strange, I haven't seen that.
Zitat von Philip Brown <pbrown@medata.com>:
I am in a situation where I see conflicting information. On the one hand, ls -l /var/lib/ceph/osd/ceph-7 shows a symlink for block device, but no block.db
On the other hand,
ceph-volume lvm list claims that there is a separate db device registered for osd 7
how can I know which one is correct?
(This is currently ceph nautilus)
-- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown@medata.com| www.medata.com _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Here's an example of an OSD with separate block.db: ---snip--- osd-db-0cd3d532-a03a-4a17-bc0c-ae3e8c710ace ceph.block_device=/dev/ceph-63cf931a-f7a8-4ab8-bdd2-4c01c8e10f4d/osd-block-a30f85ab-b2b9-4f75-a1f9-48c2ef6db1c1,ceph.block_uuid=8wWfCT-4aQw-2VKR-I1Lf-a0e3-rRRA-v3pLqL,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=7bdffde0-623f-11eb-b3db-fa163e672db2,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/ceph-ce9e4b47-1007-453c-b11e-46212474c130/osd-db-0cd3d532-a03a-4a17-bc0c-ae3e8c710ace,ceph.db_uuid=UTO5eF-WmfP-vn32-w7Cg-1dS0-RVIT-5egFoM,ceph.encrypted=0,ceph.osd_fsid=a30f85ab-b2b9-4f75-a1f9-48c2ef6db1c1,ceph.osd_id=6,ceph.osdspec_affinity=default,ceph.type=db,ceph.vdo=0 osd-block-a30f85ab-b2b9-4f75-a1f9-48c2ef6db1c1 ceph.block_device=/dev/ceph-63cf931a-f7a8-4ab8-bdd2-4c01c8e10f4d/osd-block-a30f85ab-b2b9-4f75-a1f9-48c2ef6db1c1,ceph.block_uuid=8wWfCT-4aQw-2VKR-I1Lf-a0e3-rRRA-v3pLqL,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=7bdffde0-623f-11eb-b3db-fa163e672db2,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/ceph-ce9e4b47-1007-453c-b11e-46212474c130/osd-db-0cd3d532-a03a-4a17-bc0c-ae3e8c710ace,ceph.db_uuid=UTO5eF-WmfP-vn32-w7Cg-1dS0-RVIT-5egFoM,ceph.encrypted=0,ceph.osd_fsid=a30f85ab-b2b9-4f75-a1f9-48c2ef6db1c1,ceph.osd_id=6,ceph.osdspec_affinity=default,ceph.type=block,ceph.vdo=0 ---snip--- As you can see the information is identical, so in order to wipe an OSD entirely you also need to wipe the db device, which ceph-volume can do for you: ceph-volume lvm zap --destroy /dev/ceph-ce9e4b47-1007-453c-b11e-46212474c130/osd-db-0cd3d532-a03a-4a17-bc0c-ae3e8c710ace Zitat von Philip Brown <pbrown@medata.com>:
Where does it read it from? does it keep it in the block.db lv, or the block dev lv, or both?
I removed the vg from the block dev and did wipefs, if I recall.
----- Original Message ----- From: "Eugen Block" <eblock@nde.ag> To: "Philip Brown" <pbrown@medata.com> Cc: "ceph-users" <ceph-users@ceph.io> Sent: Tuesday, April 6, 2021 9:06:50 AM Subject: [BULK] Re: [ceph-users] Re: which is definitive: /var/lib/ceph symlinks or ceph-volume?
Did you recreate the OSD with the same UUID? I'm guessing ceph-volume just reads LV tags and thinks there's a block.db for that OSD. Before creating the OSD you should also wipe the LV containing the block.db for the failed OSD.
Zitat von Philip Brown <pbrown@medata.com>:
It was up and running.
I should mention that the situation came about when I was testing OSD rebuild.
So I had a previously autogenerated hybrid OSD, with SSD db, and HDD block dev.
I then stopped the OSD, wiped HDD, .... did "ceph osd rm" and ceph auth rm. and then rebuilt with ceph-volume.. with the wrong flags. I belive I used:
ceph-volume lvm create --block /dev/xxx --journal /dev/blahblah
After that, ceph-volume lvm list claimed it was using the ssd again as db device. but ls -l on /var/lib/ceph/osd/XXXX only showed symlink for block dev.
This was on nautilus.
----- Original Message ----- From: "Eugen Block" <eblock@nde.ag> To: "ceph-users" <ceph-users@ceph.io> Sent: Tuesday, April 6, 2021 5:20:40 AM Subject: [ceph-users] Re: which is definitive: /var/lib/ceph symlinks or ceph-volume?
Is the OSD up and running? Do you see IO on the dedicated block.db LV when running a 'ceph tell osd.7 bench'? It sounds very strange, I haven't seen that.
Zitat von Philip Brown <pbrown@medata.com>:
I am in a situation where I see conflicting information. On the one hand, ls -l /var/lib/ceph/osd/ceph-7 shows a symlink for block device, but no block.db
On the other hand,
ceph-volume lvm list claims that there is a separate db device registered for osd 7
how can I know which one is correct?
(This is currently ceph nautilus)
-- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown@medata.com| www.medata.com _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Eugen Block
-
Philip Brown