Hi, I think by "copy function" would be the "bluefs-bdev-migrate" command from ceph-bluestore-tool, this is an excerpt from the man paage: ---snip--- bluefs-bdev-migrate --dev-target new-device --devs-source device1 [--devs-source device2] Moves BlueFS data from source device(s) to the target one, source devices (except the main one) are removed on success. Target device can be both already attached or new device. In the latter case it's added to OSD replacing one of the source devices. Following replacement rules apply (in the order of precedence, stop on the first match): · if source list has DB volume - target device replaces it. · if source list has WAL volume - target device replace it. · if source list has slow volume only - operation isn't permitted, requires explicit allocation via new-db/new-wal command. ---snip--- There have been several threads about the process on this list, I don't have a link at hand right now. Regards, Eugen Zitat von Chris Dunlop <chris(a)onthe.net.au>au>:

I really should read these emails more carefully... Sorry, thanks for pointing that out. I haven't done the filestore migration per OSD. I created a filestore OSD in my lab setup to play around with ceph-objectstore-tool but I couldn't find anything except for '--op dup' but it's not really clear what exactly that does. I figured it could be something like "duplicate" and tried a couple of different approaches but none of them succeeded. Maybe someone else has more insights. Zitat von Chris Dunlop <chris(a)onthe.net.au>au>:

Hi Anthony, Thanks for looking into this and opening the ticket - I'll keep an eye on it. For prepping the LVMs etc. I was thinking could probably use 'ceph-volume lvm prepare' then fixing up the relevant LV tags with the appropriate values from the origin osd. Cheers, Chris On Mon, Oct 12, 2020 at 11:16:43AM -0700, Anthony D'Atri wrote: > > Poking through the source I *think* the doc should indeed refer to the “dup” function, vs “copy”. That said, arguably we shouldn’t have a section in the docs that says "there’s this thing you can do but we aren’t going to tell you how”. > > Looking at the history / blame info, which only seems to go back partway, I suspect that this came along with the preceeding bits re repaving entire nodes. The understandable “We really should mention this but I’m tired and out of time so I’ll put in this placeholder and circle back RSN” effect. > > I’ve entered https://tracker.ceph.com/issues/47839 to track this. I really want to dive in, test this out and document it properly right now but I’m short on time today. Perhaps within a couple of weeks; the preparation of partitions / LVs smacks of a rabbit hole and I’m trying to not overcommit. > > — aad > >> I really should read these emails more carefully... Sorry, thanks for pointing that out. I haven't done the filestore migration per OSD. I created a filestore OSD in my lab setup to play around with ceph-objectstore-tool but I couldn't find anything except for '--op dup' but it's not really clear what exactly that does. I figured it could be something like "duplicate" and tried a couple of different approaches but none of them succeeded. >> >> Maybe someone else has more insights. >> >> >> Zitat von Chris Dunlop <chris(a)onthe.net.au>au>: >> >>> Hi Eugen, >>> >>> Reminder: I'm looking for guidance / hints on how to migrate from filestore to bluestore using a "per-osd device copy": >>> >>> https://docs.ceph.com/en/latest/rados/operations/bluestore-migration/#per-o… >>> >>> On Fri, Oct 09, 2020 at 07:03:33AM +0000, Eugen Block wrote: >>>> I think by "copy function" would be the "bluefs-bdev-migrate" command from ceph-bluestore-tool, this is an excerpt from the man paage: >>> >>> It looks like ceph-bluestore-tool is purely about dealing with bluestore osds, there's no facility to migrate from filestore to bluestore. >>> >>>> There have been several threads about the process on this list, I don't have a link at hand right now. >>> >>> I've not been able to find any threads relating to migrating from filestore to bluestore using a per-osd device copy - i.e. where the osd data is copied directly from an original filestore store to a new bluestore store, then the new store "becomes" the original osd. >>> >>> The advantage of doing this would be to remove the rebalancing involved in the simple method of creating a new bluestore osd and marking out the old filestore osd (then rinse and repeat until all filestores have been replaced). >>> >>> Cheers, >>> >>> Chris