Bluestore migration: per-osd device copy
Hi, The docs have scant detail on doing a migration to bluestore using a per-osd device copy: https://docs.ceph.com/en/latest/rados/operations/bluestore-migration/#per-os... This mentions "using the copy function of ceph-objectstore-tool", but ceph-objectstore-tool doesn't have a copy function (all the way from v9 to current). Has anyone actually tried doing this? Is there any further detail available on what is involved, e.g. a broad outline of the steps? Of course, detailed instructions would be even better, even if accompanied by "here be dragons!" warnings. Cheers, Chris
Hi, I think by "copy function" would be the "bluefs-bdev-migrate" command from ceph-bluestore-tool, this is an excerpt from the man paage: ---snip--- bluefs-bdev-migrate --dev-target new-device --devs-source device1 [--devs-source device2] Moves BlueFS data from source device(s) to the target one, source devices (except the main one) are removed on success. Target device can be both already attached or new device. In the latter case it's added to OSD replacing one of the source devices. Following replacement rules apply (in the order of precedence, stop on the first match): · if source list has DB volume - target device replaces it. · if source list has WAL volume - target device replace it. · if source list has slow volume only - operation isn't permitted, requires explicit allocation via new-db/new-wal command. ---snip--- There have been several threads about the process on this list, I don't have a link at hand right now. Regards, Eugen Zitat von Chris Dunlop <chris@onthe.net.au>:
Hi,
The docs have scant detail on doing a migration to bluestore using a per-osd device copy:
https://docs.ceph.com/en/latest/rados/operations/bluestore-migration/#per-os...
This mentions "using the copy function of ceph-objectstore-tool", but ceph-objectstore-tool doesn't have a copy function (all the way from v9 to current). Has anyone actually tried doing this?
Is there any further detail available on what is involved, e.g. a broad outline of the steps?
Of course, detailed instructions would be even better, even if accompanied by "here be dragons!" warnings.
Cheers,
Chris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eugen, Reminder: I'm looking for guidance / hints on how to migrate from filestore to bluestore using a "per-osd device copy": https://docs.ceph.com/en/latest/rados/operations/bluestore-migration/#per-os... On Fri, Oct 09, 2020 at 07:03:33AM +0000, Eugen Block wrote:
I think by "copy function" would be the "bluefs-bdev-migrate" command from ceph-bluestore-tool, this is an excerpt from the man paage:
It looks like ceph-bluestore-tool is purely about dealing with bluestore osds, there's no facility to migrate from filestore to bluestore.
There have been several threads about the process on this list, I don't have a link at hand right now.
I've not been able to find any threads relating to migrating from filestore to bluestore using a per-osd device copy - i.e. where the osd data is copied directly from an original filestore store to a new bluestore store, then the new store "becomes" the original osd. The advantage of doing this would be to remove the rebalancing involved in the simple method of creating a new bluestore osd and marking out the old filestore osd (then rinse and repeat until all filestores have been replaced). Cheers, Chris
I really should read these emails more carefully... Sorry, thanks for pointing that out. I haven't done the filestore migration per OSD. I created a filestore OSD in my lab setup to play around with ceph-objectstore-tool but I couldn't find anything except for '--op dup' but it's not really clear what exactly that does. I figured it could be something like "duplicate" and tried a couple of different approaches but none of them succeeded. Maybe someone else has more insights. Zitat von Chris Dunlop <chris@onthe.net.au>:
Hi Eugen,
Reminder: I'm looking for guidance / hints on how to migrate from filestore to bluestore using a "per-osd device copy":
https://docs.ceph.com/en/latest/rados/operations/bluestore-migration/#per-os...
On Fri, Oct 09, 2020 at 07:03:33AM +0000, Eugen Block wrote:
I think by "copy function" would be the "bluefs-bdev-migrate" command from ceph-bluestore-tool, this is an excerpt from the man paage:
It looks like ceph-bluestore-tool is purely about dealing with bluestore osds, there's no facility to migrate from filestore to bluestore.
There have been several threads about the process on this list, I don't have a link at hand right now.
I've not been able to find any threads relating to migrating from filestore to bluestore using a per-osd device copy - i.e. where the osd data is copied directly from an original filestore store to a new bluestore store, then the new store "becomes" the original osd.
The advantage of doing this would be to remove the rebalancing involved in the simple method of creating a new bluestore osd and marking out the old filestore osd (then rinse and repeat until all filestores have been replaced).
Cheers,
Chris
Poking through the source I *think* the doc should indeed refer to the “dup” function, vs “copy”. That said, arguably we shouldn’t have a section in the docs that says "there’s this thing you can do but we aren’t going to tell you how”. Looking at the history / blame info, which only seems to go back partway, I suspect that this came along with the preceeding bits re repaving entire nodes. The understandable “We really should mention this but I’m tired and out of time so I’ll put in this placeholder and circle back RSN” effect. I’ve entered https://tracker.ceph.com/issues/47839 to track this. I really want to dive in, test this out and document it properly right now but I’m short on time today. Perhaps within a couple of weeks; the preparation of partitions / LVs smacks of a rabbit hole and I’m trying to not overcommit. — aad
I really should read these emails more carefully... Sorry, thanks for pointing that out. I haven't done the filestore migration per OSD. I created a filestore OSD in my lab setup to play around with ceph-objectstore-tool but I couldn't find anything except for '--op dup' but it's not really clear what exactly that does. I figured it could be something like "duplicate" and tried a couple of different approaches but none of them succeeded.
Maybe someone else has more insights.
Zitat von Chris Dunlop <chris@onthe.net.au>:
Hi Eugen,
Reminder: I'm looking for guidance / hints on how to migrate from filestore to bluestore using a "per-osd device copy":
https://docs.ceph.com/en/latest/rados/operations/bluestore-migration/#per-os...
On Fri, Oct 09, 2020 at 07:03:33AM +0000, Eugen Block wrote:
I think by "copy function" would be the "bluefs-bdev-migrate" command from ceph-bluestore-tool, this is an excerpt from the man paage:
It looks like ceph-bluestore-tool is purely about dealing with bluestore osds, there's no facility to migrate from filestore to bluestore.
There have been several threads about the process on this list, I don't have a link at hand right now.
I've not been able to find any threads relating to migrating from filestore to bluestore using a per-osd device copy - i.e. where the osd data is copied directly from an original filestore store to a new bluestore store, then the new store "becomes" the original osd.
The advantage of doing this would be to remove the rebalancing involved in the simple method of creating a new bluestore osd and marking out the old filestore osd (then rinse and repeat until all filestores have been replaced).
Cheers,
Chris
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Anthony, Thanks for looking into this and opening the ticket - I'll keep an eye on it. For prepping the LVMs etc. I was thinking could probably use 'ceph-volume lvm prepare' then fixing up the relevant LV tags with the appropriate values from the origin osd. Cheers, Chris On Mon, Oct 12, 2020 at 11:16:43AM -0700, Anthony D'Atri wrote:
Poking through the source I *think* the doc should indeed refer to the “dup” function, vs “copy”. That said, arguably we shouldn’t have a section in the docs that says "there’s this thing you can do but we aren’t going to tell you how”.
Looking at the history / blame info, which only seems to go back partway, I suspect that this came along with the preceeding bits re repaving entire nodes. The understandable “We really should mention this but I’m tired and out of time so I’ll put in this placeholder and circle back RSN” effect.
I’ve entered https://tracker.ceph.com/issues/47839 to track this. I really want to dive in, test this out and document it properly right now but I’m short on time today. Perhaps within a couple of weeks; the preparation of partitions / LVs smacks of a rabbit hole and I’m trying to not overcommit.
— aad
I really should read these emails more carefully... Sorry, thanks for pointing that out. I haven't done the filestore migration per OSD. I created a filestore OSD in my lab setup to play around with ceph-objectstore-tool but I couldn't find anything except for '--op dup' but it's not really clear what exactly that does. I figured it could be something like "duplicate" and tried a couple of different approaches but none of them succeeded.
Maybe someone else has more insights.
Zitat von Chris Dunlop <chris@onthe.net.au>:
Hi Eugen,
Reminder: I'm looking for guidance / hints on how to migrate from filestore to bluestore using a "per-osd device copy":
https://docs.ceph.com/en/latest/rados/operations/bluestore-migration/#per-os...
On Fri, Oct 09, 2020 at 07:03:33AM +0000, Eugen Block wrote:
I think by "copy function" would be the "bluefs-bdev-migrate" command from ceph-bluestore-tool, this is an excerpt from the man paage:
It looks like ceph-bluestore-tool is purely about dealing with bluestore osds, there's no facility to migrate from filestore to bluestore.
There have been several threads about the process on this list, I don't have a link at hand right now.
I've not been able to find any threads relating to migrating from filestore to bluestore using a per-osd device copy - i.e. where the osd data is copied directly from an original filestore store to a new bluestore store, then the new store "becomes" the original osd.
The advantage of doing this would be to remove the rebalancing involved in the simple method of creating a new bluestore osd and marking out the old filestore osd (then rinse and repeat until all filestores have been replaced).
Cheers,
Chris
Just to follow up... I was able to use a per-osd device copy to migrate nearly 80 FileStore osds to BlueStore using a version of the script in: https://tracker.ceph.com/issues/47839 Cheers, Chris On Fri, Oct 09, 2020 at 12:05:32PM +1100, Chris Dunlop wrote:
Hi,
The docs have scant detail on doing a migration to bluestore using a per-osd device copy:
https://docs.ceph.com/en/latest/rados/operations/bluestore-migration/#per-os...
This mentions "using the copy function of ceph-objectstore-tool", but ceph-objectstore-tool doesn't have a copy function (all the way from v9 to current).
Has anyone actually tried doing this?
Is there any further detail available on what is involved, e.g. a broad outline of the steps?
Of course, detailed instructions would be even better, even if accompanied by "here be dragons!" warnings.
Cheers,
Chris
participants (3)
-
Anthony D'Atri
-
Chris Dunlop
-
Eugen Block