Hi Lucian, Thanks for your reply :) Our two main requirements are: 1) Security (...it's great news you can fix this.) 2) Stability (which is understandably harder) Our testing so far has been on a Windows 10 Pro VM (8 cores, 8GB RAM) We mount one share from a real WIndows 10 system as Z (hardware RAID, battery backed 400TB) Then use ceph-dokan to mount /cephfs on the VM as X We then use robosync to copy data from the Windows server to the cephfs mount C:\>robocopy z:\TestDatasets "X:\jog\Kates Data" /np /mt:128 /log:c:/Users/unixadmin/desktop/robolog.txt /E After copying 5TB from the WIndows server to /cephfs the ceph-dokan mount crashes. C:\WINDOWS\system32>ceph-dokan -l x -o ceph_conf_read_file OK ceph_mount OK ceph_getcwd [/] /home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = short unsigned int; C = std::map]' thread 12 time 2021-02-11T16:56:04.874874GMT Standard Time /home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: 527: FAILED ceph_assert(p->first <= start)  ceph version IT-NOTFOUND (f762f7c3be560c11ea0dd51896c976c45137f5ed) pacific (dev) restarting robocopy with a lower thread count results in another crash. Finally, one other feature request: ceph-dokan reports it's version as "16.0.0" perhaps a "-V" or "--version" switch would be useful? We are using https://github.com/dokan-dev/dokany/releases/download/v1.4.1.1000/DokanSetup... best regards, Jake On 15/02/2021 15:09, Lucian Petrut wrote:

Hi,

Thanks for trying it out. Indeed, this option is currently missing from ceph-dokan but it’s an easy thing to add. We’ll take care of it as soon as possible, hopefully it will be included in the Pacific release.

Please let us know if you have any other suggestions.

Regards,

Lucian Petrut

*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk> *Sent: *Thursday, February 11, 2021 3:00 PM *To: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com>; Lucian Petrut <mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>; dev@ceph.io <mailto:dev@ceph.io> *Subject: *Re: Windows port

Hi,

We have been testing ceph-dokan, based on the guide here: <https://documentation.suse.com/ses/7/single-html/ses-windows/index.html#windows-cephfs>

And watching <https://www.youtube.com/watch?v=BWZIwXLcNts&ab_channel=SUSE>

Initial tests on a Windows 10 VM show good write speed - around 600MB/s, which is faster than our samba server.

What worries us, is using the "root" ceph.client.admin.keyring on a Windows system, as it gives access to the entire cephfs cluster - which in our case is 5PB.

I'd really like this to work, as it would let user administrated Windows systems that control microscopes to save data directly to cephfs, so that we can process the data on our HPC cluster.

I'd normally use cephx, and make a key that allows access to a directory off the root.

e.g.

[root@ceph-s1 users]# ceph auth get client.x_lab exported keyring for client.x_lab [client.x_lab]         key = xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==         caps mds = "allow r path=/users/, allow rw path=/users/x_lab"         caps mon = "allow r"         caps osd = "allow class-read object_prefix rbd_children, allow rw pool=ec82pool"

The real key works fine on linux, but when we try this key with ceph-dokan, and specify the ceph directory (x_lab) as a ceph path, there is no option to specify the user - is this hard-coded as admin?

Have I just missed something? Or is this a missing feature?

anyhow, ceph-dokan looks like it could be quite useful, thank you Cloudbase :)

best regards,

Jake

-- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK.

...

Hi,

We’ve just pushed the final part of the Windows PR series[1], allowing RBD images as well as CephFS to be mounted on Windows.

There’s a comprehensive guide[2], describing the build, installation, configuration and usage steps.

2 out of 12 PRs have been merged already, we look forward to merging

On 4/30/20 1:54 PM, Lucian Petrut wrote: the

...

others as well.

Lucian Petrut

Cloudbase Solutions

[1] https://github.com/ceph/ceph/pull/34859 <https://github.com/ceph/ceph/pull/34859>

[2]

https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst <https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst>

...

*From: *Lucian Petrut

...

*Sent: *Monday, December 16, 2019 10:12 AM *To: *dev@ceph.io <mailto:dev@ceph.io> *Subject: *Windows port

Hi,

We're happy to announce that a couple of weeks ago, we've submitted a few Github pull requests[1][2][3] adding initial Windows support. A big thank you to the people that have already reviewed the patches.

To bring some context about the scope and current status of our work: we're mostly targeting the client side, allowing Windows hosts to consume rados, rbd and cephfs resources.

We have Windows binaries capable of writing to rados pools[4]. We're using mingw to build the ceph components, mostly due to the fact

<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C> that it

...

requires the minimum amount of changes to cross compile ceph for Windows. However, we're soon going to switch to MSVC/Clang due to mingw limitations and long standing bugs[5][6]. Porting the unit tests is also something that we're currently working on.

The next step will be implementing a virtual miniport driver so that RBD volumes can be exposed to Windows hosts and Hyper-V guests. We're hoping to leverage librbd as much as possible as part of a daemon that will communicate with the driver. We're also aiming at cephfs and considering using Dokan, which is FUSE compatible.

Merging the open PRs would allow us to move forward, focusing on the drivers and avoiding rebase issues. Any help on that is greatly appreciated.

Last but not least, I'd like to thank Suse, who's sponsoring this effort!

Lucian Petrut

Cloudbase Solutions

[1] https://github.com/ceph/ceph/pull/31981

[2] https://github.com/ceph/ceph/pull/32027

[3] https://github.com/ceph/rocksdb/pull/42

[4] http://paste.openstack.org/raw/787534/

[5] https://sourceforge.net/p/mingw-w64/bugs/816/

[6] https://sourceforge.net/p/mingw-w64/bugs/527/

_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io

Note: I am working from home until further notice. For help, contact unixadmin@mrc-lmb.cam.ac.uk -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539

Hi Lucian, Many thanks for looking into this, completely understand that it is just a PoC, but just to let you know that the crash still occurs with your nightly build. C:\Users\unixadmin>ceph-dokan -l x -o ceph_conf_read_file OK 2021-02-17T12:40:59.637GMT Standard Time 1 -1 asok(0xb36ef20) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to 'C:/ProgramData/ceph/client.admin.4136.asok': (13) Permission denied ceph_mount OK ceph_getcwd [/] ../src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = short unsigned int; C = std::map]' thread 11 time 2021-02-17T13:58:21.387427GMT Standard Time ../src/include/interval_set.h: 538: FAILED ceph_assert(p->first <= start)  ceph version 15.0.0-21905-g8f11930301 (8f11930301a9ecd5315a8a69a9874eb4428f9366) pacific (rc) best regards, Jake On 17/02/2021 10:56, Lucian Petrut wrote:

Hi,

That crash is indeed a concern. We’ll look into it as soon as possible.

A small disclaimer: our main focus was RBD, ceph-dokan was more of a PoC, as mentioned by the docs. Nevertheless, it seems to be highly demanded so we’re going to invest more time in it.

FWIW, here’s our nightly Ceph MSI build: https://cloudbase.it/downloads/ceph_v16_0_0_beta.msi <https://cloudbase.it/downloads/ceph_v16_0_0_beta.msi>. It provides some important RBD fixes (though you’re probably more interested in cephfs).

I’ll come with a follow up as soon as those issues are fixed.

Thanks,

Lucian

*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk> *Sent: *Wednesday, February 17, 2021 12:25 PM *To: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com>; dev@ceph.io <mailto:dev@ceph.io> *Subject: *Re: Windows port

Hi Lucian,

Thanks for your reply :)

Our two main requirements are:

1) Security (...it's great news you can fix this.)

2) Stability (which is understandably harder)

Our testing so far has been on a Windows 10 Pro VM (8 cores, 8GB RAM)

We mount one share from a real WIndows 10 system as Z (hardware RAID, battery backed 400TB)

Then use ceph-dokan to mount /cephfs on the VM as X

We then use robosync to copy data from the Windows server to the cephfs mount

C:\>robocopy z:\TestDatasets "X:\jog\Kates Data" /np /mt:128 /log:c:/Users/unixadmin/desktop/robolog.txt /E

After copying 5TB from the WIndows server to /cephfs the ceph-dokan mount crashes.

C:\WINDOWS\system32>ceph-dokan -l x -o ceph_conf_read_file OK ceph_mount OK ceph_getcwd [/] /home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = short unsigned int; C = std::map]' thread 12 time 2021-02-11T16:56:04.874874GMT Standard Time /home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: 527: FAILED ceph_assert(p->first <= start)  ceph version IT-NOTFOUND (f762f7c3be560c11ea0dd51896c976c45137f5ed) pacific (dev)

restarting robocopy with a lower thread count results in another crash.

Finally, one other feature request: ceph-dokan reports it's version as "16.0.0" perhaps a "-V" or "--version" switch would be useful?

We are using https://github.com/dokan-dev/dokany/releases/download/v1.4.1.1000/DokanSetup...

best regards,

Jake

On 15/02/2021 15:09, Lucian Petrut wrote:

Hi,

Thanks for trying it out. Indeed, this option is currently missing from ceph-dokan but it’s an easy thing to add. We’ll take care of it as soon as possible, hopefully it will be included in the Pacific release.

Please let us know if you have any other suggestions.

Regards,

Lucian Petrut

*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk> *Sent: *Thursday, February 11, 2021 3:00 PM *To: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com>; Lucian Petrut <mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>; dev@ceph.io <mailto:dev@ceph.io> *Subject: *Re: Windows port

Hi,

We have been testing ceph-dokan, based on the guide here: <https://documentation.suse.com/ses/7/single-html/ses-windows/index.html#windows-cephfs>

And watching <https://www.youtube.com/watch?v=BWZIwXLcNts&ab_channel=SUSE>

Initial tests on a Windows 10 VM show good write speed - around 600MB/s, which is faster than our samba server.

What worries us, is using the "root" ceph.client.admin.keyring on a Windows system, as it gives access to the entire cephfs cluster - which in our case is 5PB.

I'd really like this to work, as it would let user administrated Windows systems that control microscopes to save data directly to cephfs, so that we can process the data on our HPC cluster.

I'd normally use cephx, and make a key that allows access to a directory off the root.

e.g.

[root@ceph-s1 users]# ceph auth get client.x_lab exported keyring for client.x_lab [client.x_lab]         key = xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==         caps mds = "allow r path=/users/, allow rw path=/users/x_lab"         caps mon = "allow r"         caps osd = "allow class-read object_prefix rbd_children, allow rw pool=ec82pool"

The real key works fine on linux, but when we try this key with ceph-dokan, and specify the ceph directory (x_lab) as a ceph path, there is no option to specify the user - is this hard-coded as admin?

Have I just missed something? Or is this a missing feature?

anyhow, ceph-dokan looks like it could be quite useful, thank you Cloudbase :)

best regards,

Jake

-- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK.

On 4/30/20 1:54 PM, Lucian Petrut wrote: > Hi, > > We’ve just pushed the final part of the Windows PR series[1], allowing > RBD images as well as CephFS to be mounted on Windows. > > There’s a comprehensive guide[2], describing the build, installation, > configuration and usage steps. > > 2 out of 12 PRs have been merged already, we look forward to merging the > others as well. > > Lucian Petrut > > Cloudbase Solutions > > [1] https://github.com/ceph/ceph/pull/34859 > <https://github.com/ceph/ceph/pull/34859> > > [2] > https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst <https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst> > > *From: *Lucian Petrut > <mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C> > *Sent: *Monday, December 16, 2019 10:12 AM > *To: *dev@ceph.io <mailto:dev@ceph.io> > *Subject: *Windows port > > Hi, > > We're happy to announce that a couple of weeks ago, we've submitted a > few Github pull requests[1][2][3] adding initial Windows support. A big > thank you to the people that have already reviewed the patches. > > To bring some context about the scope and current status of our work: > we're mostly targeting the client side, allowing Windows hosts to > consume rados, rbd and cephfs resources. > > We have Windows binaries capable of writing to rados pools[4]. We're > using mingw to build the ceph components, mostly due to the fact that it > requires the minimum amount of changes to cross compile ceph for > Windows. However, we're soon going to switch to MSVC/Clang due to mingw > limitations and long standing bugs[5][6]. Porting the unit tests is also > something that we're currently working on. > > The next step will be implementing a virtual miniport driver so that RBD > volumes can be exposed to Windows hosts and Hyper-V guests. We're hoping > to leverage librbd as much as possible as part of a daemon that will > communicate with the driver. We're also aiming at cephfs and considering > using Dokan, which is FUSE compatible. > > Merging the open PRs would allow us to move forward, focusing on the > drivers and avoiding rebase issues. Any help on that is greatly appreciated. > > Last but not least, I'd like to thank Suse, who's sponsoring this effort! > > Lucian Petrut > > Cloudbase Solutions > > [1] https://github.com/ceph/ceph/pull/31981 > > [2] https://github.com/ceph/ceph/pull/32027 > > [3] https://github.com/ceph/rocksdb/pull/42 > > [4] http://paste.openstack.org/raw/787534/ > > [5] https://sourceforge.net/p/mingw-w64/bugs/816/ > > [6] https://sourceforge.net/p/mingw-w64/bugs/527/ > > > _______________________________________________ > Dev mailing list -- dev@ceph.io <mailto:dev@ceph.io> > To unsubscribe send an email to dev-leave@ceph.io <mailto:dev-leave@ceph.io> >

Note: I am working from home until further notice. For help, contactunixadmin@mrc-lmb.cam.ac.uk <mailto:unixadmin@mrc-lmb.cam.ac.uk> -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539

Note: I am working from home until further notice. For help, contact unixadmin@mrc-lmb.cam.ac.uk -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539

Hi, Nevermind, I managed to get a crash after increasing the number of threads: http://paste.openstack.org/raw/802827/ I’ll try to get a fix as soon as possible. Thanks again for all the info! Lucian From: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com> Sent: Friday, February 19, 2021 3:56 PM To: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>; dev@ceph.io<mailto:dev@ceph.io> Subject: RE: Windows port Hi, I couldn’t reproduce the issue yet. Judging by the exception message, it seems to be related to libcephfs’s inode handling. The latest msi installer includes debug symbols. Could you please send us an archive with a crash dump [1] and ideally the msi installer as well? Thanks, Lucian [1] https://docs.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dump... From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk> Sent: Wednesday, February 17, 2021 4:42 PM To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>; dev@ceph.io<mailto:dev@ceph.io> Subject: Re: Windows port Hi Lucian, Many thanks for looking into this, completely understand that it is just a PoC, but just to let you know that the crash still occurs with your nightly build. C:\Users\unixadmin>ceph-dokan -l x -o ceph_conf_read_file OK 2021-02-17T12:40:59.637GMT Standard Time 1 -1 asok(0xb36ef20) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to 'C:/ProgramData/ceph/client.admin.4136.asok': (13) Permission denied ceph_mount OK ceph_getcwd [/] ../src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = short unsigned int; C = std::map]' thread 11 time 2021-02-17T13:58:21.387427GMT Standard Time ../src/include/interval_set.h: 538: FAILED ceph_assert(p->first <= start) ceph version 15.0.0-21905-g8f11930301 (8f11930301a9ecd5315a8a69a9874eb4428f9366) pacific (rc) best regards, Jake On 17/02/2021 10:56, Lucian Petrut wrote: Hi, That crash is indeed a concern. We’ll look into it as soon as possible. A small disclaimer: our main focus was RBD, ceph-dokan was more of a PoC, as mentioned by the docs. Nevertheless, it seems to be highly demanded so we’re going to invest more time in it. FWIW, here’s our nightly Ceph MSI build: https://cloudbase.it/downloads/ceph_v16_0_0_beta.msi. It provides some important RBD fixes (though you’re probably more interested in cephfs). I’ll come with a follow up as soon as those issues are fixed. Thanks, Lucian From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk> Sent: Wednesday, February 17, 2021 12:25 PM To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>; dev@ceph.io<mailto:dev@ceph.io> Subject: Re: Windows port Hi Lucian, Thanks for your reply :) Our two main requirements are: 1) Security (...it's great news you can fix this.) 2) Stability (which is understandably harder) Our testing so far has been on a Windows 10 Pro VM (8 cores, 8GB RAM) We mount one share from a real WIndows 10 system as Z (hardware RAID, battery backed 400TB) Then use ceph-dokan to mount /cephfs on the VM as X We then use robosync to copy data from the Windows server to the cephfs mount C:\>robocopy z:\TestDatasets "X:\jog\Kates Data" /np /mt:128 /log:c:/Users/unixadmin/desktop/robolog.txt /E After copying 5TB from the WIndows server to /cephfs the ceph-dokan mount crashes. C:\WINDOWS\system32>ceph-dokan -l x -o ceph_conf_read_file OK ceph_mount OK ceph_getcwd [/] /home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = short unsigned int; C = std::map]' thread 12 time 2021-02-11T16:56:04.874874GMT Standard Time /home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: 527: FAILED ceph_assert(p->first <= start) ceph version IT-NOTFOUND (f762f7c3be560c11ea0dd51896c976c45137f5ed) pacific (dev) restarting robocopy with a lower thread count results in another crash. Finally, one other feature request: ceph-dokan reports it's version as "16.0.0" perhaps a "-V" or "--version" switch would be useful? We are using https://github.com/dokan-dev/dokany/releases/download/v1.4.1.1000/DokanSetup... best regards, Jake On 15/02/2021 15:09, Lucian Petrut wrote: Hi, Thanks for trying it out. Indeed, this option is currently missing from ceph-dokan but it’s an easy thing to add. We’ll take care of it as soon as possible, hopefully it will be included in the Pacific release. Please let us know if you have any other suggestions. Regards, Lucian Petrut From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk> Sent: Thursday, February 11, 2021 3:00 PM To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>; Lucian Petrut<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>; dev@ceph.io<mailto:dev@ceph.io> Subject: Re: Windows port Hi, We have been testing ceph-dokan, based on the guide here: <https://documentation.suse.com/ses/7/single-html/ses-windows/index.html#windows-cephfs> And watching <https://www.youtube.com/watch?v=BWZIwXLcNts&ab_channel=SUSE> Initial tests on a Windows 10 VM show good write speed - around 600MB/s, which is faster than our samba server. What worries us, is using the "root" ceph.client.admin.keyring on a Windows system, as it gives access to the entire cephfs cluster - which in our case is 5PB. I'd really like this to work, as it would let user administrated Windows systems that control microscopes to save data directly to cephfs, so that we can process the data on our HPC cluster. I'd normally use cephx, and make a key that allows access to a directory off the root. e.g. [root@ceph-s1 users]# ceph auth get client.x_lab exported keyring for client.x_lab [client.x_lab] key = xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX== caps mds = "allow r path=/users/, allow rw path=/users/x_lab" caps mon = "allow r" caps osd = "allow class-read object_prefix rbd_children, allow rw pool=ec82pool" The real key works fine on linux, but when we try this key with ceph-dokan, and specify the ceph directory (x_lab) as a ceph path, there is no option to specify the user - is this hard-coded as admin? Have I just missed something? Or is this a missing feature? anyhow, ceph-dokan looks like it could be quite useful, thank you Cloudbase :) best regards, Jake -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. On 4/30/20 1:54 PM, Lucian Petrut wrote:

Hi,

We’ve just pushed the final part of the Windows PR series[1], allowing RBD images as well as CephFS to be mounted on Windows.

There’s a comprehensive guide[2], describing the build, installation, configuration and usage steps.

2 out of 12 PRs have been merged already, we look forward to merging the others as well.

Lucian Petrut

Cloudbase Solutions

[1] https://github.com/ceph/ceph/pull/34859 <https://github.com/ceph/ceph/pull/34859>

[2] https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst <https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst>

*From: *Lucian Petrut <mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C> *Sent: *Monday, December 16, 2019 10:12 AM *To: *dev@ceph.io <mailto:dev@ceph.io> *Subject: *Windows port

Hi,

We're happy to announce that a couple of weeks ago, we've submitted a few Github pull requests[1][2][3] adding initial Windows support. A big thank you to the people that have already reviewed the patches.

To bring some context about the scope and current status of our work: we're mostly targeting the client side, allowing Windows hosts to consume rados, rbd and cephfs resources.

We have Windows binaries capable of writing to rados pools[4]. We're using mingw to build the ceph components, mostly due to the fact that it requires the minimum amount of changes to cross compile ceph for Windows. However, we're soon going to switch to MSVC/Clang due to mingw limitations and long standing bugs[5][6]. Porting the unit tests is also something that we're currently working on.

The next step will be implementing a virtual miniport driver so that RBD volumes can be exposed to Windows hosts and Hyper-V guests. We're hoping to leverage librbd as much as possible as part of a daemon that will communicate with the driver. We're also aiming at cephfs and considering using Dokan, which is FUSE compatible.

Merging the open PRs would allow us to move forward, focusing on the drivers and avoiding rebase issues. Any help on that is greatly appreciated.

Last but not least, I'd like to thank Suse, who's sponsoring this effort!

Lucian Petrut

Cloudbase Solutions

[1] https://github.com/ceph/ceph/pull/31981

[2] https://github.com/ceph/ceph/pull/32027

[3] https://github.com/ceph/rocksdb/pull/42

[4] http://paste.openstack.org/raw/787534/

[5] https://sourceforge.net/p/mingw-w64/bugs/816/

[6] https://sourceforge.net/p/mingw-w64/bugs/527/

_______________________________________________ Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io> To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io>

Note: I am working from home until further notice. For help, contact unixadmin@mrc-lmb.cam.ac.uk<mailto:unixadmin@mrc-lmb.cam.ac.uk> -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539 Note: I am working from home until further notice. For help, contact unixadmin@mrc-lmb.cam.ac.uk<mailto:unixadmin@mrc-lmb.cam.ac.uk> -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539

Hi Lucian, many thanks for this - I'll test and get back to you :) best regards, Jake On 22/02/2021 15:12, Lucian Petrut wrote:

Hi,

I’ve fixed the crash. It was due to Ceph’s inode emulation, which can be skipped as Windows supports 64B file identifiers.

The fix is included in the latest msi.

I’ll prepare a few more cephfs related fixes, including the user option.

Thanks,

Lucian

*From: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com> *Sent: *Friday, February 19, 2021 4:51 PM *To: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk>; dev@ceph.io <mailto:dev@ceph.io> *Subject: *RE: Windows port

Hi,

Nevermind, I managed to get a crash after increasing the number of threads: http://paste.openstack.org/raw/802827/

I’ll try to get a fix as soon as possible. Thanks again for all the info!

Lucian

*From: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com> *Sent: *Friday, February 19, 2021 3:56 PM *To: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk>; dev@ceph.io <mailto:dev@ceph.io> *Subject: *RE: Windows port

Hi,

I couldn’t reproduce the issue yet. Judging by the exception message, it seems to be related to libcephfs’s inode handling.

The latest msi installer includes debug symbols. Could you please send us an archive with a crash dump [1] and ideally the msi installer as well?

Thanks,

Lucian

[1] https://docs.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dump...

*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk> *Sent: *Wednesday, February 17, 2021 4:42 PM *To: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com>; dev@ceph.io <mailto:dev@ceph.io> *Subject: *Re: Windows port

Hi Lucian,

Many thanks for looking into this, completely understand that it is just a PoC, but just to let you know that the crash still occurs with your nightly build.

C:\Users\unixadmin>ceph-dokan -l x -o ceph_conf_read_file OK 2021-02-17T12:40:59.637GMT Standard Time 1 -1 asok(0xb36ef20) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to 'C:/ProgramData/ceph/client.admin.4136.asok': (13) Permission denied ceph_mount OK ceph_getcwd [/] ../src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = short unsigned int; C = std::map]' thread 11 time 2021-02-17T13:58:21.387427GMT Standard Time ../src/include/interval_set.h: 538: FAILED ceph_assert(p->first <= start)  ceph version 15.0.0-21905-g8f11930301 (8f11930301a9ecd5315a8a69a9874eb4428f9366) pacific (rc)

best regards,

Jake

On 17/02/2021 10:56, Lucian Petrut wrote:

Hi,

That crash is indeed a concern. We’ll look into it as soon as possible.

A small disclaimer: our main focus was RBD, ceph-dokan was more of a PoC, as mentioned by the docs. Nevertheless, it seems to be highly demanded so we’re going to invest more time in it.

FWIW, here’s our nightly Ceph MSI build: https://cloudbase.it/downloads/ceph_v16_0_0_beta.msi <https://cloudbase.it/downloads/ceph_v16_0_0_beta.msi>. It provides some important RBD fixes (though you’re probably more interested in cephfs).

I’ll come with a follow up as soon as those issues are fixed.

Thanks,

Lucian

*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk> *Sent: *Wednesday, February 17, 2021 12:25 PM *To: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com>; dev@ceph.io <mailto:dev@ceph.io> *Subject: *Re: Windows port

Hi Lucian,

Thanks for your reply :)

Our two main requirements are:

1) Security (...it's great news you can fix this.)

2) Stability (which is understandably harder)

Our testing so far has been on a Windows 10 Pro VM (8 cores, 8GB RAM)

We mount one share from a real WIndows 10 system as Z (hardware RAID, battery backed 400TB)

Then use ceph-dokan to mount /cephfs on the VM as X

We then use robosync to copy data from the Windows server to the cephfs mount

C:\>robocopy z:\TestDatasets "X:\jog\Kates Data" /np /mt:128 /log:c:/Users/unixadmin/desktop/robolog.txt /E

After copying 5TB from the WIndows server to /cephfs the ceph-dokan mount crashes.

C:\WINDOWS\system32>ceph-dokan -l x -o ceph_conf_read_file OK ceph_mount OK ceph_getcwd [/] /home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = short unsigned int; C = std::map]' thread 12 time 2021-02-11T16:56:04.874874GMT Standard Time /home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: 527: FAILED ceph_assert(p->first <= start)  ceph version IT-NOTFOUND (f762f7c3be560c11ea0dd51896c976c45137f5ed) pacific (dev)

restarting robocopy with a lower thread count results in another crash.

Finally, one other feature request: ceph-dokan reports it's version as "16.0.0" perhaps a "-V" or "--version" switch would be useful?

We are using https://github.com/dokan-dev/dokany/releases/download/v1.4.1.1000/DokanSetup...

best regards,

Jake

On 15/02/2021 15:09, Lucian Petrut wrote:

Hi,

Thanks for trying it out. Indeed, this option is currently missing from ceph-dokan but it’s an easy thing to add. We’ll take care of it as soon as possible, hopefully it will be included in the Pacific release.

Please let us know if you have any other suggestions.

Regards,

Lucian Petrut

*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk> *Sent: *Thursday, February 11, 2021 3:00 PM *To: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com>; Lucian Petrut <mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>; dev@ceph.io <mailto:dev@ceph.io> *Subject: *Re: Windows port

Hi,

We have been testing ceph-dokan, based on the guide here: <https://documentation.suse.com/ses/7/single-html/ses-windows/index.html#windows-cephfs>

And watching <https://www.youtube.com/watch?v=BWZIwXLcNts&ab_channel=SUSE>

Initial tests on a Windows 10 VM show good write speed - around 600MB/s, which is faster than our samba server.

What worries us, is using the "root" ceph.client.admin.keyring on a Windows system, as it gives access to the entire cephfs cluster - which in our case is 5PB.

I'd really like this to work, as it would let user administrated Windows systems that control microscopes to save data directly to cephfs, so that we can process the data on our HPC cluster.

I'd normally use cephx, and make a key that allows access to a directory off the root.

e.g.

[root@ceph-s1 users]# ceph auth get client.x_lab exported keyring for client.x_lab [client.x_lab]         key = xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==         caps mds = "allow r path=/users/, allow rw path=/users/x_lab"         caps mon = "allow r"         caps osd = "allow class-read object_prefix rbd_children, allow rw pool=ec82pool"

The real key works fine on linux, but when we try this key with ceph-dokan, and specify the ceph directory (x_lab) as a ceph path, there is no option to specify the user - is this hard-coded as admin?

Have I just missed something? Or is this a missing feature?

anyhow, ceph-dokan looks like it could be quite useful, thank you Cloudbase :)

best regards,

Jake

-- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK.

On 4/30/20 1:54 PM, Lucian Petrut wrote: > Hi, > > We’ve just pushed the final part of the Windows PR series[1], allowing > RBD images as well as CephFS to be mounted on Windows. > > There’s a comprehensive guide[2], describing the build, installation, > configuration and usage steps. > > 2 out of 12 PRs have been merged already, we look forward to merging the > others as well. > > Lucian Petrut > > Cloudbase Solutions > > [1] https://github.com/ceph/ceph/pull/34859 > <https://github.com/ceph/ceph/pull/34859> > > [2] > https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst <https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst> > > *From: *Lucian Petrut > <mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C> > *Sent: *Monday, December 16, 2019 10:12 AM > *To: *dev@ceph.io <mailto:dev@ceph.io> > *Subject: *Windows port > > Hi, > > We're happy to announce that a couple of weeks ago, we've submitted a > few Github pull requests[1][2][3] adding initial Windows support. A big > thank you to the people that have already reviewed the patches. > > To bring some context about the scope and current status of our work: > we're mostly targeting the client side, allowing Windows hosts to > consume rados, rbd and cephfs resources. > > We have Windows binaries capable of writing to rados pools[4]. We're > using mingw to build the ceph components, mostly due to the fact that it > requires the minimum amount of changes to cross compile ceph for > Windows. However, we're soon going to switch to MSVC/Clang due to mingw > limitations and long standing bugs[5][6]. Porting the unit tests is also > something that we're currently working on. > > The next step will be implementing a virtual miniport driver so that RBD > volumes can be exposed to Windows hosts and Hyper-V guests. We're hoping > to leverage librbd as much as possible as part of a daemon that will > communicate with the driver. We're also aiming at cephfs and considering > using Dokan, which is FUSE compatible. > > Merging the open PRs would allow us to move forward, focusing on the > drivers and avoiding rebase issues. Any help on that is greatly appreciated. > > Last but not least, I'd like to thank Suse, who's sponsoring this effort! > > Lucian Petrut > > Cloudbase Solutions > > [1] https://github.com/ceph/ceph/pull/31981 > > [2] https://github.com/ceph/ceph/pull/32027 > > [3] https://github.com/ceph/rocksdb/pull/42 > > [4] http://paste.openstack.org/raw/787534/ > > [5] https://sourceforge.net/p/mingw-w64/bugs/816/ > > [6] https://sourceforge.net/p/mingw-w64/bugs/527/ > > > _______________________________________________ > Dev mailing list -- dev@ceph.io <mailto:dev@ceph.io> > To unsubscribe send an email to dev-leave@ceph.io <mailto:dev-leave@ceph.io> >

Note: I am working from home until further notice.

For help, contactunixadmin@mrc-lmb.cam.ac.uk <mailto:unixadmin@mrc-lmb.cam.ac.uk>

--

Dr Jake Grimmett

Head Of Scientific Computing

MRC Laboratory of Molecular Biology

Francis Crick Avenue,

Cambridge CB2 0QH, UK.

Phone 01223 267019

Mobile 0776 9886539

Note: I am working from home until further notice. For help, contactunixadmin@mrc-lmb.cam.ac.uk <mailto:unixadmin@mrc-lmb.cam.ac.uk> -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539

_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io

Note: I am working from home until further notice. For help, contact unixadmin@mrc-lmb.cam.ac.uk -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539

Hi, That’s great, thanks for the confirmation. I hope I’ll get to push a larger change by the end of the week, covering the user option as well as a few other fixes. Regards, Lucian From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk> Sent: Tuesday, February 23, 2021 11:48 AM To: dev@ceph.io<mailto:dev@ceph.io>; Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com> Subject: Re: Windows port Hi Lucian, Good news - your fix works; I was able to write 7TB from a Windows 10 VM to cephfs last night. I'll carry on testing the Windows client with large workloads using robocopy. thanks again for working on this, a cephx "user" option should provide the core requirements we need for a usable system :) best regards, Jake On 22/02/2021 16:29, Jake Grimmett wrote: Hi Lucian, many thanks for this - I'll test and get back to you :) best regards, Jake On 22/02/2021 15:12, Lucian Petrut wrote: Hi, I’ve fixed the crash. It was due to Ceph’s inode emulation, which can be skipped as Windows supports 64B file identifiers. The fix is included in the latest msi. I’ll prepare a few more cephfs related fixes, including the user option. Thanks, Lucian From: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com> Sent: Friday, February 19, 2021 4:51 PM To: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>; dev@ceph.io<mailto:dev@ceph.io> Subject: RE: Windows port Hi, Nevermind, I managed to get a crash after increasing the number of threads: http://paste.openstack.org/raw/802827/ I’ll try to get a fix as soon as possible. Thanks again for all the info! Lucian From: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com> Sent: Friday, February 19, 2021 3:56 PM To: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>; dev@ceph.io<mailto:dev@ceph.io> Subject: RE: Windows port Hi, I couldn’t reproduce the issue yet. Judging by the exception message, it seems to be related to libcephfs’s inode handling. The latest msi installer includes debug symbols. Could you please send us an archive with a crash dump [1] and ideally the msi installer as well? Thanks, Lucian [1] https://docs.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dump... From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk> Sent: Wednesday, February 17, 2021 4:42 PM To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>; dev@ceph.io<mailto:dev@ceph.io> Subject: Re: Windows port Hi Lucian, Many thanks for looking into this, completely understand that it is just a PoC, but just to let you know that the crash still occurs with your nightly build. C:\Users\unixadmin>ceph-dokan -l x -o ceph_conf_read_file OK 2021-02-17T12:40:59.637GMT Standard Time 1 -1 asok(0xb36ef20) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to 'C:/ProgramData/ceph/client.admin.4136.asok': (13) Permission denied ceph_mount OK ceph_getcwd [/] ../src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = short unsigned int; C = std::map]' thread 11 time 2021-02-17T13:58:21.387427GMT Standard Time ../src/include/interval_set.h: 538: FAILED ceph_assert(p->first <= start) ceph version 15.0.0-21905-g8f11930301 (8f11930301a9ecd5315a8a69a9874eb4428f9366) pacific (rc) best regards, Jake On 17/02/2021 10:56, Lucian Petrut wrote: Hi, That crash is indeed a concern. We’ll look into it as soon as possible. A small disclaimer: our main focus was RBD, ceph-dokan was more of a PoC, as mentioned by the docs. Nevertheless, it seems to be highly demanded so we’re going to invest more time in it. FWIW, here’s our nightly Ceph MSI build: https://cloudbase.it/downloads/ceph_v16_0_0_beta.msi. It provides some important RBD fixes (though you’re probably more interested in cephfs). I’ll come with a follow up as soon as those issues are fixed. Thanks, Lucian From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk> Sent: Wednesday, February 17, 2021 12:25 PM To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>; dev@ceph.io<mailto:dev@ceph.io> Subject: Re: Windows port Hi Lucian, Thanks for your reply :) Our two main requirements are: 1) Security (...it's great news you can fix this.) 2) Stability (which is understandably harder) Our testing so far has been on a Windows 10 Pro VM (8 cores, 8GB RAM) We mount one share from a real WIndows 10 system as Z (hardware RAID, battery backed 400TB) Then use ceph-dokan to mount /cephfs on the VM as X We then use robosync to copy data from the Windows server to the cephfs mount C:\>robocopy z:\TestDatasets "X:\jog\Kates Data" /np /mt:128 /log:c:/Users/unixadmin/desktop/robolog.txt /E After copying 5TB from the WIndows server to /cephfs the ceph-dokan mount crashes. C:\WINDOWS\system32>ceph-dokan -l x -o ceph_conf_read_file OK ceph_mount OK ceph_getcwd [/] /home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = short unsigned int; C = std::map]' thread 12 time 2021-02-11T16:56:04.874874GMT Standard Time /home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: 527: FAILED ceph_assert(p->first <= start) ceph version IT-NOTFOUND (f762f7c3be560c11ea0dd51896c976c45137f5ed) pacific (dev) restarting robocopy with a lower thread count results in another crash. Finally, one other feature request: ceph-dokan reports it's version as "16.0.0" perhaps a "-V" or "--version" switch would be useful? We are using https://github.com/dokan-dev/dokany/releases/download/v1.4.1.1000/DokanSetup... best regards, Jake On 15/02/2021 15:09, Lucian Petrut wrote: Hi, Thanks for trying it out. Indeed, this option is currently missing from ceph-dokan but it’s an easy thing to add. We’ll take care of it as soon as possible, hopefully it will be included in the Pacific release. Please let us know if you have any other suggestions. Regards, Lucian Petrut From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk> Sent: Thursday, February 11, 2021 3:00 PM To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>; Lucian Petrut<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>; dev@ceph.io<mailto:dev@ceph.io> Subject: Re: Windows port Hi, We have been testing ceph-dokan, based on the guide here: <https://documentation.suse.com/ses/7/single-html/ses-windows/index.html#windows-cephfs> And watching <https://www.youtube.com/watch?v=BWZIwXLcNts&ab_channel=SUSE> Initial tests on a Windows 10 VM show good write speed - around 600MB/s, which is faster than our samba server. What worries us, is using the "root" ceph.client.admin.keyring on a Windows system, as it gives access to the entire cephfs cluster - which in our case is 5PB. I'd really like this to work, as it would let user administrated Windows systems that control microscopes to save data directly to cephfs, so that we can process the data on our HPC cluster. I'd normally use cephx, and make a key that allows access to a directory off the root. e.g. [root@ceph-s1 users]# ceph auth get client.x_lab exported keyring for client.x_lab [client.x_lab] key = xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX== caps mds = "allow r path=/users/, allow rw path=/users/x_lab" caps mon = "allow r" caps osd = "allow class-read object_prefix rbd_children, allow rw pool=ec82pool" The real key works fine on linux, but when we try this key with ceph-dokan, and specify the ceph directory (x_lab) as a ceph path, there is no option to specify the user - is this hard-coded as admin? Have I just missed something? Or is this a missing feature? anyhow, ceph-dokan looks like it could be quite useful, thank you Cloudbase :) best regards, Jake -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. On 4/30/20 1:54 PM, Lucian Petrut wrote:

Hi,

We’ve just pushed the final part of the Windows PR series[1], allowing RBD images as well as CephFS to be mounted on Windows.

There’s a comprehensive guide[2], describing the build, installation, configuration and usage steps.

2 out of 12 PRs have been merged already, we look forward to merging the others as well.

Lucian Petrut

Cloudbase Solutions

[1] https://github.com/ceph/ceph/pull/34859 <https://github.com/ceph/ceph/pull/34859>

[2] https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst <https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst>

*From: *Lucian Petrut <mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C> *Sent: *Monday, December 16, 2019 10:12 AM *To: *dev@ceph.io <mailto:dev@ceph.io> *Subject: *Windows port

Hi,

We're happy to announce that a couple of weeks ago, we've submitted a few Github pull requests[1][2][3] adding initial Windows support. A big thank you to the people that have already reviewed the patches.

To bring some context about the scope and current status of our work: we're mostly targeting the client side, allowing Windows hosts to consume rados, rbd and cephfs resources.

We have Windows binaries capable of writing to rados pools[4]. We're using mingw to build the ceph components, mostly due to the fact that it requires the minimum amount of changes to cross compile ceph for Windows. However, we're soon going to switch to MSVC/Clang due to mingw limitations and long standing bugs[5][6]. Porting the unit tests is also something that we're currently working on.

The next step will be implementing a virtual miniport driver so that RBD volumes can be exposed to Windows hosts and Hyper-V guests. We're hoping to leverage librbd as much as possible as part of a daemon that will communicate with the driver. We're also aiming at cephfs and considering using Dokan, which is FUSE compatible.

Merging the open PRs would allow us to move forward, focusing on the drivers and avoiding rebase issues. Any help on that is greatly appreciated.

Last but not least, I'd like to thank Suse, who's sponsoring this effort!

Lucian Petrut

Cloudbase Solutions

[1] https://github.com/ceph/ceph/pull/31981

[2] https://github.com/ceph/ceph/pull/32027

[3] https://github.com/ceph/rocksdb/pull/42

[4] http://paste.openstack.org/raw/787534/

[5] https://sourceforge.net/p/mingw-w64/bugs/816/

[6] https://sourceforge.net/p/mingw-w64/bugs/527/

_______________________________________________ Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io> To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io>

Note: I am working from home until further notice. For help, contact unixadmin@mrc-lmb.cam.ac.uk<mailto:unixadmin@mrc-lmb.cam.ac.uk> -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539 Note: I am working from home until further notice. For help, contact unixadmin@mrc-lmb.cam.ac.uk<mailto:unixadmin@mrc-lmb.cam.ac.uk> -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539 _______________________________________________ Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io> To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io> Note: I am working from home until further notice. For help, contact unixadmin@mrc-lmb.cam.ac.uk<mailto:unixadmin@mrc-lmb.cam.ac.uk> -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539 _______________________________________________ Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io> To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io> Note: I am working from home until further notice. For help, contact unixadmin@mrc-lmb.cam.ac.uk<mailto:unixadmin@mrc-lmb.cam.ac.uk> -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539

Hi, That’s great. I’ll push the user option setting ASAP. About the quotas, you mean Windows quotas or Ceph quotas? Regars, Lucian From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk> Sent: Thursday, February 25, 2021 11:37 AM To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>; dev@ceph.io<mailto:dev@ceph.io> Subject: Re: Windows port Hi Lucian, So I've now copied 32TB of data from a Windows server to our test ceph cluster, without any crashes. Average speed over one 16TB copy was 468MB/s, i.e. just under 11 hours to transfer 16TB, with a data set that has 1,144,495 files. The transfer was from Windows 10 server (hardware RAID) > Windows 10 VM (with the cephfs driver installed) > cephfs test cluster. If you can get the user option working, I'll put the driver on a physical Windows 10 system, and see how fast a direct transfer is. One other thing that would be useful, is over-quota error handling. If I try writing and exceed the quota, the mount just hangs. If I increase the quota, the mount recovers, but it would be nice if we had a quota error in Windows. Test cluster consists of 6 Dell C2100 nodes, bought in 2012. Each has 10 x 900GB 10k HDD, we use EC 4+2 for the cephfs pool, metadata is on 3 x NVMe. Dual Xeon X5650, 12 threads, 96GB RAM, 2x10GB bond. Ceph 15.2.8, Scientific Linux 7.9. best regards, Jake On 2/23/21 11:49 AM, Lucian Petrut wrote:

Hi,

That’s great, thanks for the confirmation.

I hope I’ll get to push a larger change by the end of the week, covering the user option as well as a few other fixes.

Regards,

Lucian

*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk> *Sent: *Tuesday, February 23, 2021 11:48 AM *To: *dev@ceph.io <mailto:dev@ceph.io>; Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com> *Subject: *Re: Windows port

Hi Lucian,

Good news - your fix works; I was able to write 7TB from a Windows 10 VM to cephfs last night.

I'll carry on testing the Windows client with large workloads using robocopy.

thanks again for working on this, a cephx "user" option should provide the core requirements we need for a usable system :)

best regards,

Jake

Note: I am working from home until further notice. For help, contact unixadmin@mrc-lmb.cam.ac.uk -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539

Hi, Makes sense. I’ll look into it ASAP, thanks for bringing it up! Lucian From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk> Sent: Thursday, February 25, 2021 2:21 PM To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>; dev@ceph.io<mailto:dev@ceph.io> Subject: Re: Windows port Hi Lucian, Exceeding the cephfs quota seems to "upset" Windows: e.g. on the cephfs system # setfattr -n ceph.quota.max_bytes -v 1000 /ctestfs/quota And then on the Windows system PS Z:\Jake_tests> copy .\4GB_file X:\quota\ (Shell hangs) back on the cephfs system # setfattr -n ceph.quota.max_bytes -v 1000000000 /ctestfs/quota Shell (eventually) returns on Windows, after much CTRL+C I can then repeat the "copy .\4GB_file X:\quota\" successfully It would be ideal if Windows gave an error, such as "no space" or "out of quota" rather than hanging... thanks again for your work :) Jake On 2/25/21 10:51 AM, Lucian Petrut wrote:

Hi,

That’s great. I’ll push the user option setting ASAP.

About the quotas, you mean Windows quotas or Ceph quotas?

Regars,

Lucian

*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk> *Sent: *Thursday, February 25, 2021 11:37 AM *To: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com>; dev@ceph.io <mailto:dev@ceph.io> *Subject: *Re: Windows port

Hi Lucian,

So I've now copied 32TB of data from a Windows server to our test ceph cluster, without any crashes.

Average speed over one 16TB copy was 468MB/s, i.e. just under 11 hours to transfer 16TB, with a data set that has 1,144,495 files.

The transfer was from Windows 10 server (hardware RAID) > Windows 10 VM (with the cephfs driver installed) > cephfs test cluster.

If you can get the user option working, I'll put the driver on a physical Windows 10 system, and see how fast a direct transfer is.

One other thing that would be useful, is over-quota error handling. If I try writing and exceed the quota, the mount just hangs.

If I increase the quota, the mount recovers, but it would be nice if we had a quota error in Windows.

Test cluster consists of 6 Dell C2100 nodes, bought in 2012. Each has 10 x 900GB 10k HDD, we use EC 4+2 for the cephfs pool, metadata is on 3 x NVMe. Dual Xeon X5650, 12 threads, 96GB RAM, 2x10GB bond. Ceph 15.2.8, Scientific Linux 7.9.

best regards,

Jake

On 2/23/21 11:49 AM, Lucian Petrut wrote:

...

Hi,

That’s great, thanks for the confirmation.

I hope I’ll get to push a larger change by the end of the week, covering the user option as well as a few other fixes.

Regards,

Lucian

*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk <mailto:jog@mrc-lmb.cam.ac.uk>> *Sent: *Tuesday, February 23, 2021 11:48 AM *To: *dev@ceph.io <mailto:dev@ceph.io <mailto:dev@ceph.io>>; Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com <mailto:lpetrut@cloudbasesolutions.com>> *Subject: *Re: Windows port

Hi Lucian,

Good news - your fix works; I was able to write 7TB from a Windows 10 VM to cephfs last night.

I'll carry on testing the Windows client with large workloads using robocopy.

thanks again for working on this, a cephx "user" option should provide the core requirements we need for a usable system :)

best regards,

Jake

Note: I am working from home until further notice. For help, contact unixadmin@mrc-lmb.cam.ac.uk -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539

Note: I am working from home until further notice. For help, contact unixadmin@mrc-lmb.cam.ac.uk -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539

Hi Lucian, Many thanks for adding user level authentication, it works - yippee! The "out of quota" error on Windows worked as expected when re-tested :) The new ceph-dokan paramerers confused me until I re-read the cephx docs. --id ID set ID portion of my name --name/-n TYPE.ID set name I'm sure you'll provide examples, but here is my usage / config PS C:\> ceph-dokan -l x --conf C:/ProgramData/ceph/ceph.testauth.conf --name client.test -x /testauthdir (is equivalent to...) PS C:\> ceph-dokan -l x --conf C:/ProgramData/ceph/ceph.testauth.conf --id test -x /testauthdir where: PS C:\ProgramData\ceph> type .\ceph.testauth.conf [global] log to stderr = true run dir = C:/ProgramData/ceph crash dir = C:/ProgramData/ceph [client] keyring = C:/ProgramData/ceph/test.keyring log file = C:/ProgramData/ceph/$name.$pid.log admin socket = C:/ProgramData/ceph/$name.$pid.asok [global] ; Specify IP addresses for monitor nodes as in the following example: ; mon host = [v2:10.1.2.39:3300,v1:10.1.2.39:6789] [v2:10.1.0.96:3300,v1:10.1.0.96:6789] [v2:10.1.0.98:3300,v1:1.1.0.98:6789] PS C:\ProgramData\ceph> type .\test.keyring [client.test] key = AQBFFSVgvypAIBAA8CHRkwdVJ6i9u+9yKiq1hQ== caps mds = "allow r path=/, allow rw path=/testauthdir" caps mon = "allow r" caps osd = "allow class-read object_prefix rbd_children, allow rw pool=ec42pool" test.keyring is the output of "ceph auth get client.test" on the ceph server I'll now test this more thoroughly, best regards, and thanks again for doing this, it's really useful :) Jake Note: I am working from home until further notice. For help, contact unixadmin@mrc-lmb.cam.ac.uk -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539

Hi Lucian Driver is excellent, however I'm not seeing cephfs recursive dir stats being updated. For example, 24 hours after copying 16TB to cephfs: # ls -lh drwxrwxrwx 1 root root 1.9T Mar 2 14:47 testauthdir # ls -lR testauthdir >/dev/null # ls -lh drwxrwxrwx 1 root root 16TB Mar 5 11:38 testauthdir In earlier tests I think recursive dir stats was working, though it might be related to us upgrading ceph from Nautilus to Octopus (which we did to try and stop the earlier crashes) Our current ceph version is 15.2.8 We use recursive stats to efficiently replicate the filesystem to a second cephfs cluster; it's a very useful feature. The new cephfs-mirror tool in Pacific might rely on this feature too. thanks again for your work on this :) Jake On 3/2/21 12:38 PM, Lucian Petrut wrote:

Hi,

Thanks for providing valuable feedback. Please let us know if anything else comes up.

Regards,

Lucian

Note: I am working from home until further notice. For help, contact unixadmin@mrc-lmb.cam.ac.uk -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539

Hi, I gave it a quick try, the recursive stats do get updated although it might take a while for that to happen. I think MDS load can also have an impact on this. Also, you might want to check if mds_dirstat_min_interval is set, that would delay the recursive stat propagation. Regards, Lucian From: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com> Sent: Friday, March 5, 2021 3:39 PM To: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>; dev@ceph.io<mailto:dev@ceph.io> Subject: RE: Windows port Hi, Interesting. AFAIK the rbytes tracking happens on the MDS side but I’ll look into it. Btw, just so you know, I did include a few unrelated fixes in the latest MSI. Regards, Lucian From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk> Sent: Friday, March 5, 2021 1:55 PM To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>; dev@ceph.io<mailto:dev@ceph.io> Subject: Re: Windows port Hi Lucian Driver is excellent, however I'm not seeing cephfs recursive dir stats being updated. For example, 24 hours after copying 16TB to cephfs: # ls -lh drwxrwxrwx 1 root root 1.9T Mar 2 14:47 testauthdir # ls -lR testauthdir >/dev/null # ls -lh drwxrwxrwx 1 root root 16TB Mar 5 11:38 testauthdir In earlier tests I think recursive dir stats was working, though it might be related to us upgrading ceph from Nautilus to Octopus (which we did to try and stop the earlier crashes) Our current ceph version is 15.2.8 We use recursive stats to efficiently replicate the filesystem to a second cephfs cluster; it's a very useful feature. The new cephfs-mirror tool in Pacific might rely on this feature too. thanks again for your work on this :) Jake On 3/2/21 12:38 PM, Lucian Petrut wrote:

Hi,

Thanks for providing valuable feedback. Please let us know if anything else comes up.

Regards,

Lucian

Note: I am working from home until further notice. For help, contact unixadmin@mrc-lmb.cam.ac.uk -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539