Hi, What's the correct way to migrate an OSD wal/db from a fast device to the (slow) block device? I have an osd with wal/db on a fast LV device and block on a slow LV device. I want to move the wal/db onto the block device so I can reconfigure the fast device before moving the wal/db back to the fast device. This link says to use "ceph-volume lvm migrate" (I'm on pacific, but the quincy and reef docs are the same): https://docs.ceph.com/en/pacific/ceph-volume/lvm/migrate/ I tried: $ cephadm unit --fsid ${fsid} --name osd.${osdid} stop $ cephadm shell --fsid ${fsid} --name osd.${osdid} -- \ ceph-volume lvm migrate --osd-id ${osdid} --osd-fsid ${osd_fsid} \ --from db wal --target ${block_vglv} $ systemctl stop ${osd_service} $ systemctl start ${osd_service} "cephadm ceph-volume lvm list" now shows only the (slow) block device whereas before the migrate it was showing both the block and db devices. However "lsof" shows the new osd process still has the original fast wal/db device open and "iostat" shows this device is still getting i/o. Also: $ ls -l /var/lib/ceph/${fsid}/osd.${osdid}/block* ...shows both the "block" and "block.db" symlinks to the original separate devices. And there are now no lv_tags on the original wal/db LV: $ lvs -o lv_tags ${original_db_vg_lv} Now I'm concerned there's device mismatch for this osd: "cephadm ceph-volume lvm list" believes there's no separate wal/db, but the osd is currently *using* the original separate wal/db. I guess if the server were to restart this osd would be in all sorts of trouble. What's going on there, and what can be done to fix it? Is it a matter of recreating the tags on the original db device? (But then what happens to whatever did get migrated to the block device - e.g. is that space lost?) Or is it a matter of using ceph-bluestore-tool to do a bluefs-bdev-migrate, e.g. something like: $ cephadm unit --fsid ${fsid} --name osd.${osdid} stop $ osddir=/var/lib/ceph/osd/ceph-${osdid} $ cephadm shell --fsid ${fsid} --name osd.${osdid} -- \ ceph-bluestore-tool --path ${osddir} --devs-source ${osddir}/block.db \ --dev-target ${osddir}/block bluefs-bdev-migrate $ rm /var/lib/ceph/${fsid}/osd.${osdid}/block.db $ systemctl stop ${osd_service} $ systemctl start ${osd_service} Or... something else? And how *should* moving the wal/db be done? Cheers, Chris
Hi, AFAIU, you can’t migrate back to the slow device. It’s either migrating from the slow device to a fast device or remove between fast devices. I’m not aware that your scenario was considered in that tool. The docs don’t specifically say that, but they also don’t mention going back to slow device only. Someone please correct me, but I’d say you’ll have to rebuild that OSD to detach it from the fast device. Regards, Eugen Zitat von Chris Dunlop <chris@onthe.net.au>:
Hi,
What's the correct way to migrate an OSD wal/db from a fast device to the (slow) block device?
I have an osd with wal/db on a fast LV device and block on a slow LV device. I want to move the wal/db onto the block device so I can reconfigure the fast device before moving the wal/db back to the fast device.
This link says to use "ceph-volume lvm migrate" (I'm on pacific, but the quincy and reef docs are the same):
https://docs.ceph.com/en/pacific/ceph-volume/lvm/migrate/
I tried:
$ cephadm unit --fsid ${fsid} --name osd.${osdid} stop $ cephadm shell --fsid ${fsid} --name osd.${osdid} -- \ ceph-volume lvm migrate --osd-id ${osdid} --osd-fsid ${osd_fsid} \ --from db wal --target ${block_vglv} $ systemctl stop ${osd_service} $ systemctl start ${osd_service}
"cephadm ceph-volume lvm list" now shows only the (slow) block device whereas before the migrate it was showing both the block and db devices. However "lsof" shows the new osd process still has the original fast wal/db device open and "iostat" shows this device is still getting i/o.
Also:
$ ls -l /var/lib/ceph/${fsid}/osd.${osdid}/block*
...shows both the "block" and "block.db" symlinks to the original separate devices.
And there are now no lv_tags on the original wal/db LV:
$ lvs -o lv_tags ${original_db_vg_lv}
Now I'm concerned there's device mismatch for this osd: "cephadm ceph-volume lvm list" believes there's no separate wal/db, but the osd is currently *using* the original separate wal/db.
I guess if the server were to restart this osd would be in all sorts of trouble.
What's going on there, and what can be done to fix it? Is it a matter of recreating the tags on the original db device? (But then what happens to whatever did get migrated to the block device - e.g. is that space lost?) Or is it a matter of using ceph-bluestore-tool to do a bluefs-bdev-migrate, e.g. something like:
$ cephadm unit --fsid ${fsid} --name osd.${osdid} stop $ osddir=/var/lib/ceph/osd/ceph-${osdid} $ cephadm shell --fsid ${fsid} --name osd.${osdid} -- \ ceph-bluestore-tool --path ${osddir} --devs-source ${osddir}/block.db \ --dev-target ${osddir}/block bluefs-bdev-migrate $ rm /var/lib/ceph/${fsid}/osd.${osdid}/block.db $ systemctl stop ${osd_service} $ systemctl start ${osd_service}
Or... something else?
And how *should* moving the wal/db be done?
Cheers,
Chris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eugen, this scenario is supported, see the last example on the relevant doc page: Moves BlueFS data from main, DB and WAL devices to main device, WAL and DB are removed: ceph-volume lvm migrate --osd-id 1 --osd-fsid <uuid> --from db wal --target vgname/data Thanks, Igor On 11/15/2023 11:20 AM, Eugen Block wrote:
Hi,
AFAIU, you can’t migrate back to the slow device. It’s either migrating from the slow device to a fast device or remove between fast devices. I’m not aware that your scenario was considered in that tool. The docs don’t specifically say that, but they also don’t mention going back to slow device only. Someone please correct me, but I’d say you’ll have to rebuild that OSD to detach it from the fast device.
Regards, Eugen
Zitat von Chris Dunlop <chris@onthe.net.au>:
Hi,
What's the correct way to migrate an OSD wal/db from a fast device to the (slow) block device?
I have an osd with wal/db on a fast LV device and block on a slow LV device. I want to move the wal/db onto the block device so I can reconfigure the fast device before moving the wal/db back to the fast device.
This link says to use "ceph-volume lvm migrate" (I'm on pacific, but the quincy and reef docs are the same):
https://docs.ceph.com/en/pacific/ceph-volume/lvm/migrate/
I tried:
$ cephadm unit --fsid ${fsid} --name osd.${osdid} stop $ cephadm shell --fsid ${fsid} --name osd.${osdid} -- \ ceph-volume lvm migrate --osd-id ${osdid} --osd-fsid ${osd_fsid} \ --from db wal --target ${block_vglv} $ systemctl stop ${osd_service} $ systemctl start ${osd_service}
"cephadm ceph-volume lvm list" now shows only the (slow) block device whereas before the migrate it was showing both the block and db devices. However "lsof" shows the new osd process still has the original fast wal/db device open and "iostat" shows this device is still getting i/o.
Also:
$ ls -l /var/lib/ceph/${fsid}/osd.${osdid}/block*
...shows both the "block" and "block.db" symlinks to the original separate devices.
And there are now no lv_tags on the original wal/db LV:
$ lvs -o lv_tags ${original_db_vg_lv}
Now I'm concerned there's device mismatch for this osd: "cephadm ceph-volume lvm list" believes there's no separate wal/db, but the osd is currently *using* the original separate wal/db.
I guess if the server were to restart this osd would be in all sorts of trouble.
What's going on there, and what can be done to fix it? Is it a matter of recreating the tags on the original db device? (But then what happens to whatever did get migrated to the block device - e.g. is that space lost?) Or is it a matter of using ceph-bluestore-tool to do a bluefs-bdev-migrate, e.g. something like:
$ cephadm unit --fsid ${fsid} --name osd.${osdid} stop $ osddir=/var/lib/ceph/osd/ceph-${osdid} $ cephadm shell --fsid ${fsid} --name osd.${osdid} -- \ ceph-bluestore-tool --path ${osddir} --devs-source ${osddir}/block.db \ --dev-target ${osddir}/block bluefs-bdev-migrate $ rm /var/lib/ceph/${fsid}/osd.${osdid}/block.db $ systemctl stop ${osd_service} $ systemctl start ${osd_service}
Or... something else?
And how *should* moving the wal/db be done?
Cheers,
Chris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Igor Fedotov Ceph Lead Developer Looking for help with your Ceph cluster? Contact us athttps://croit.io croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx
Oh right, I responded from my mobile phone and missed the examples. Thanks for the clarification! OP did stop the OSD according to his output:
$ cephadm unit --fsid ${fsid} --name osd.${osdid} stop
But there might have been an error anyway, I guess. Zitat von Igor Fedotov <igor.fedotov@croit.io>:
Hi Eugen,
this scenario is supported, see the last example on the relevant doc page:
Moves BlueFS data from main, DB and WAL devices to main device, WAL and DB are removed:
ceph-volume lvm migrate --osd-id 1 --osd-fsid <uuid> --from db wal --target vgname/data
Thanks, Igor
On 11/15/2023 11:20 AM, Eugen Block wrote:
Hi,
AFAIU, you can’t migrate back to the slow device. It’s either migrating from the slow device to a fast device or remove between fast devices. I’m not aware that your scenario was considered in that tool. The docs don’t specifically say that, but they also don’t mention going back to slow device only. Someone please correct me, but I’d say you’ll have to rebuild that OSD to detach it from the fast device.
Regards, Eugen
Zitat von Chris Dunlop <chris@onthe.net.au>:
Hi,
What's the correct way to migrate an OSD wal/db from a fast device to the (slow) block device?
I have an osd with wal/db on a fast LV device and block on a slow LV device. I want to move the wal/db onto the block device so I can reconfigure the fast device before moving the wal/db back to the fast device.
This link says to use "ceph-volume lvm migrate" (I'm on pacific, but the quincy and reef docs are the same):
https://docs.ceph.com/en/pacific/ceph-volume/lvm/migrate/
I tried:
$ cephadm unit --fsid ${fsid} --name osd.${osdid} stop $ cephadm shell --fsid ${fsid} --name osd.${osdid} -- \ ceph-volume lvm migrate --osd-id ${osdid} --osd-fsid ${osd_fsid} \ --from db wal --target ${block_vglv} $ systemctl stop ${osd_service} $ systemctl start ${osd_service}
"cephadm ceph-volume lvm list" now shows only the (slow) block device whereas before the migrate it was showing both the block and db devices. However "lsof" shows the new osd process still has the original fast wal/db device open and "iostat" shows this device is still getting i/o.
Also:
$ ls -l /var/lib/ceph/${fsid}/osd.${osdid}/block*
...shows both the "block" and "block.db" symlinks to the original separate devices.
And there are now no lv_tags on the original wal/db LV:
$ lvs -o lv_tags ${original_db_vg_lv}
Now I'm concerned there's device mismatch for this osd: "cephadm ceph-volume lvm list" believes there's no separate wal/db, but the osd is currently *using* the original separate wal/db.
I guess if the server were to restart this osd would be in all sorts of trouble.
What's going on there, and what can be done to fix it? Is it a matter of recreating the tags on the original db device? (But then what happens to whatever did get migrated to the block device - e.g. is that space lost?) Or is it a matter of using ceph-bluestore-tool to do a bluefs-bdev-migrate, e.g. something like:
$ cephadm unit --fsid ${fsid} --name osd.${osdid} stop $ osddir=/var/lib/ceph/osd/ceph-${osdid} $ cephadm shell --fsid ${fsid} --name osd.${osdid} -- \ ceph-bluestore-tool --path ${osddir} --devs-source ${osddir}/block.db \ --dev-target ${osddir}/block bluefs-bdev-migrate $ rm /var/lib/ceph/${fsid}/osd.${osdid}/block.db $ systemctl stop ${osd_service} $ systemctl start ${osd_service}
Or... something else?
And how *should* moving the wal/db be done?
Cheers,
Chris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Igor Fedotov Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us athttps://croit.io
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx
Hi Chris, haven't checked you actions thoroughly but migration to be done on a down OSD which is apparently not the case here. May be that's a culprit and we/you somehow missed the relevant error during the migration process? Thanks, Igor On 11/15/2023 5:33 AM, Chris Dunlop wrote:
Hi,
What's the correct way to migrate an OSD wal/db from a fast device to the (slow) block device?
I have an osd with wal/db on a fast LV device and block on a slow LV device. I want to move the wal/db onto the block device so I can reconfigure the fast device before moving the wal/db back to the fast device.
This link says to use "ceph-volume lvm migrate" (I'm on pacific, but the quincy and reef docs are the same):
https://docs.ceph.com/en/pacific/ceph-volume/lvm/migrate/
I tried:
$ cephadm unit --fsid ${fsid} --name osd.${osdid} stop $ cephadm shell --fsid ${fsid} --name osd.${osdid} -- \ ceph-volume lvm migrate --osd-id ${osdid} --osd-fsid ${osd_fsid} \ --from db wal --target ${block_vglv} $ systemctl stop ${osd_service} $ systemctl start ${osd_service}
"cephadm ceph-volume lvm list" now shows only the (slow) block device whereas before the migrate it was showing both the block and db devices. However "lsof" shows the new osd process still has the original fast wal/db device open and "iostat" shows this device is still getting i/o.
Also:
$ ls -l /var/lib/ceph/${fsid}/osd.${osdid}/block*
...shows both the "block" and "block.db" symlinks to the original separate devices.
And there are now no lv_tags on the original wal/db LV:
$ lvs -o lv_tags ${original_db_vg_lv}
Now I'm concerned there's device mismatch for this osd: "cephadm ceph-volume lvm list" believes there's no separate wal/db, but the osd is currently *using* the original separate wal/db.
I guess if the server were to restart this osd would be in all sorts of trouble.
What's going on there, and what can be done to fix it? Is it a matter of recreating the tags on the original db device? (But then what happens to whatever did get migrated to the block device - e.g. is that space lost?) Or is it a matter of using ceph-bluestore-tool to do a bluefs-bdev-migrate, e.g. something like:
$ cephadm unit --fsid ${fsid} --name osd.${osdid} stop $ osddir=/var/lib/ceph/osd/ceph-${osdid} $ cephadm shell --fsid ${fsid} --name osd.${osdid} -- \ ceph-bluestore-tool --path ${osddir} --devs-source ${osddir}/block.db \ --dev-target ${osddir}/block bluefs-bdev-migrate $ rm /var/lib/ceph/${fsid}/osd.${osdid}/block.db $ systemctl stop ${osd_service} $ systemctl start ${osd_service}
Or... something else?
And how *should* moving the wal/db be done?
Cheers,
Chris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Igor Fedotov Ceph Lead Developer Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
Hi Igor, On Wed, Nov 15, 2023 at 12:30:57PM +0300, Igor Fedotov wrote:
Hi Chris,
haven't checked you actions thoroughly but migration to be done on a down OSD which is apparently not the case here.
May be that's a culprit and we/you somehow missed the relevant error during the migration process?
The migration was done with the container still running, but the osd process was stopped within the container, like so: $ cephadm unit --fsid ${fsid} --name osd.${osdid} stop I've confirmed that command indeed stops the ceph-osd process. I restored the tags on both the db and block LVs (the db LV had all it's tags removed, and the block LV had the db_device and db_uuid tags removed during the previous "lvm migrate" attempt) and confirmed "ceph-volume lvm list" then returned the same as before the previous "lvm migrate" attempt. (I'm pretty sure "ceph-volume lvm list" just reads the tags direct from the LVs and presents them in a formatted output.) I then tried the migrate again, this time stopping the container before the migrate: $ systemctl stop "${osd_service}" $ cephadm shell --fsid "${fsid}" --name "osd.${osd}" -- \ ceph-volume lvm migrate --osd-id "${osd}" --osd-fsid "${osd_fsid}" \ --from db wal --target "${vg_lv}" $ systemctl start "${osd_service}" Unfortunately that had precisely the same result: - "lsof" shows the new osd process still has the original fast wal/db device open - "iostat" shows this device is still getting i/o - both "ceph-volume lvm list" and "lvs -o tag" show all the tags have been removed from the db device, and the db_device and db_uuid tags have been removed from the block device. Notably, whilst the "lvm migrate" is running, "iostat" on the db device shows very high read activity (and no write activity), so it's certainly reading whatever is on there, presumably to copy the data to the block device. However even after the migrate something is making the osd start up with the original db device rather than using the block device for the db. Any ideas? Cheers, Chris
Hi Igor, The immediate answer is to use "ceph-volume lvm zap" on the db LV after running the migrate. But for the longer term I think the "lvm zap" should be included in the "lvm migrate" process. I.e. this works to migrate a separate wal/db to the block device: # # WARNING! DO NOT ZAP AFTER STARTING THE OSD!! # $ cephadm ceph-volume lvm list "${osd}" > ~/"osd.${osd}.list" $ systemctl stop "${osd_service}" $ cephadm shell --fsid "${fsid}" --name "osd.${osd}" -- \ ceph-volume lvm migrate --osd-id "${osd}" --osd-fsid "${osd_fsid}" \ --from db wal --target "${vg_lv}" $ cephadm shell --fsid "${fsid}" --name "osd.${osd}" -- \ ceph-volume lvm zap "${db_lv}" $ systemctl start "${osd_service}" WARNING! If you don't do the zap before starting the osd, the osd will be running with the db still on the LV. If you then stop the osd and zap the LV and start the osd again, you'll be running on the original db as it was copied to the block device before the migrate, which will be missing any updates done in the meantime. I don't know what problems that might cause. In this situation I've restored the LV tags (i.e. all tags on the db LV, the db_device and db_uuid tags on the block LV) using the info from ~/osd.${osd}.list (otherwise the migrate fails!) and then gone through the migrate process again. The problem is, it turns out the osd is being activated as a "raw" device rather than an "lvm" device, and the "raw" db device (which is actually an lvm LV) still has a bluestore label on it after the migrate, so it's still seen as a component of the osd. E.g. before the migrate, both of these show the osd with the separate db: $ cephadm ceph-volume lvm list $ cephadm ceph-volume raw list After the migrate (without zap), the "lvm list" does NOT show the separate db (because the appropriate LV tags have been removed), but the "raw list" still shows the osd with the separate db. And the osd is being activated as a "raw" device, both before and after the migrate. E.g. extract from the journal before the migrate: Nov 15 22:39:05 k12 bash[3829222]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-25 Nov 15 22:39:05 k12 bash[3829222]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-25 --no-mon-config --dev /dev/mapper/ceph--5ccbb386--142b--4bf7-- Nov 15 22:39:05 k12 bash[3829222]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/ceph--5ccbb386--142b--4bf7--a180--04bcf9a1f61b-osd--block--7710024b--ec71--4fd3--b94c--c4c4b9af2d2 Nov 15 22:39:05 k12 bash[3829222]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Nov 15 22:39:05 k12 bash[3829222]: Running command: /usr/bin/ln -s /dev/mapper/ceph--5ccbb386--142b--4bf7--a180--04bcf9a1f61b-osd--block--7710024b--ec71--4fd3--b94c--c4c4b9af2d21 /var/lib/ce Nov 15 22:39:05 k12 bash[3829222]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/ceph--d4b1e932--4557--4b88--bed2--9305a07e76eb-osd--db--6a507f57--884c--4947--a147--cd50f98f1a23 Nov 15 22:39:05 k12 bash[3829222]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 Nov 15 22:39:05 k12 bash[3829222]: Running command: /usr/bin/ln -s /dev/mapper/ceph--d4b1e932--4557--4b88--bed2--9305a07e76eb-osd--db--6a507f57--884c--4947--a147--cd50f98f1a23 /var/lib/ceph/ Nov 15 22:39:05 k12 bash[3829222]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-25 Nov 15 22:39:05 k12 bash[3829222]: --> ceph-volume raw activate successful for osd ID: 25 After a migrate without a zap - note there are still two mapper/lv devices found, which includes the now-unwanted db LV: Nov 16 09:08:31 k12 bash[4012506]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-25 Nov 16 09:08:31 k12 bash[4012506]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-25 --no-mon-config --dev /dev/mapper/ceph--5ccbb386--142b--4bf7-- Nov 16 09:08:31 k12 bash[4012506]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/ceph--5ccbb386--142b--4bf7--a180--04bcf9a1f61b-osd--block--7710024b--ec71--4fd3--b94c--c4c4b9af2d2 Nov 16 09:08:31 k12 bash[4012506]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Nov 16 09:08:31 k12 bash[4012506]: Running command: /usr/bin/ln -s /dev/mapper/ceph--5ccbb386--142b--4bf7--a180--04bcf9a1f61b-osd--block--7710024b--ec71--4fd3--b94c--c4c4b9af2d21 /var/lib/ce Nov 16 09:08:31 k12 bash[4012506]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/ceph--d4b1e932--4557--4b88--bed2--9305a07e76eb-osd--db--6a507f57--884c--4947--a147--cd50f98f1a23 Nov 16 09:08:31 k12 bash[4012506]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 Nov 16 09:08:31 k12 bash[4012506]: Running command: /usr/bin/ln -s /dev/mapper/ceph--d4b1e932--4557--4b88--bed2--9305a07e76eb-osd--db--6a507f57--884c--4947--a147--cd50f98f1a23 /var/lib/ceph/ Nov 16 09:08:31 k12 bash[4012506]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-25 Nov 16 09:08:31 k12 bash[4012506]: --> ceph-volume raw activate successful for osd ID: 25 After a migrate and zap - note there's now only a single mapper/lv device found, i.e. we've successfully stopped using the separate db device: Nov 16 12:33:39 k12 bash[4091471]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-25 Nov 16 12:33:39 k12 bash[4091471]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-25 --no-mon-config --dev /dev/mapper/ceph--5ccbb386--142b--4bf7-- Nov 16 12:33:39 k12 bash[4091471]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/ceph--5ccbb386--142b--4bf7--a180--04bcf9a1f61b-osd--block--7710024b--ec71--4fd3--b94c--c4c4b9af2d2 Nov 16 12:33:39 k12 bash[4091471]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Nov 16 12:33:39 k12 bash[4091471]: Running command: /usr/bin/ln -s /dev/mapper/ceph--5ccbb386--142b--4bf7--a180--04bcf9a1f61b-osd--block--7710024b--ec71--4fd3--b94c--c4c4b9af2d21 /var/lib/ce Nov 16 12:33:39 k12 bash[4091471]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-25 Nov 16 12:33:39 k12 bash[4091471]: --> ceph-volume raw activate successful for osd ID: 25 Wrapping up... I think the "lvm zap" should be included in the "ceph-volume lvm migrate" process, and perhaps "ceph-volume activate" changed to NOT detect LVs as raw devices so they're correctly activated as "lvm" devices. Another oddity that unfortunately extended the time taken to analyse this issue... why does "ceph-volume raw list ${osd}" NOT show lvm osds, when plain "ceph-volume raw list" shows them? Cheers, Chris
participants (3)
-
Chris Dunlop
-
Eugen Block
-
Igor Fedotov