11 Feb
2021
11 Feb
'21
8 p.m.
Hi,
We have been testing ceph-dokan, based on the guide here:
<https://documentation.suse.com/ses/7/single-html/ses-windows/index.html#windows-cephfs>
And watching <https://www.youtube.com/watch?v=BWZIwXLcNts&ab_channel=SUSE>
Initial tests on a Windows 10 VM show good write speed - around 600MB/s,
which is faster than our samba server.
What worries us, is using the "root" ceph.client.admin.keyring on a
Windows system, as it gives access to the entire cephfs cluster - which
in our case is 5PB.
I'd really like this to work, as it would let user administrated Windows
systems that control microscopes to save data directly to cephfs, so
that we can process the data on our HPC cluster.
I'd normally use cephx, and make a key that allows access to a directory
off the root.
e.g.
[root@ceph-s1 users]# ceph auth get client.x_lab
exported keyring for client.x_lab
[client.x_lab]
key = xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==
caps mds = "allow r path=/users/, allow rw path=/users/x_lab"
caps mon = "allow r"
caps osd = "allow class-read object_prefix rbd_children, allow rw
pool=ec82pool"
The real key works fine on linux, but when we try this key with
ceph-dokan, and specify the ceph directory (x_lab) as a ceph path, there
is no option to specify the user - is this hard-coded as admin?
Have I just missed something? Or is this a missing feature?
anyhow, ceph-dokan looks like it could be quite useful,
thank you Cloudbase :)
best regards,
Jake
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
On 4/30/20 1:54 PM, Lucian Petrut wrote:
Hi,
We’ve just pushed the final part of the Windows PR series[1], allowing
RBD images as well as CephFS to be mounted on Windows.
There’s a comprehensive guide[2], describing the build, installation,
configuration and usage steps.
2 out of 12 PRs have been merged already, we look forward to merging the
others as well.
Lucian Petrut
Cloudbase Solutions
[1] https://github.com/ceph/ceph/pull/34859
<https://github.com/ceph/ceph/pull/34859>
[2]
https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst
<https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst>
*From: *Lucian Petrut
<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>
*Sent: *Monday, December 16, 2019 10:12 AM
*To: *dev(a)ceph.io <mailto:dev@ceph.io>
*Subject: *Windows port
Hi,
We're happy to announce that a couple of weeks ago, we've submitted a
few Github pull requests[1][2][3] adding initial Windows support. A big
thank you to the people that have already reviewed the patches.
To bring some context about the scope and current status of our work:
we're mostly targeting the client side, allowing Windows hosts to
consume rados, rbd and cephfs resources.
We have Windows binaries capable of writing to rados pools[4]. We're
using mingw to build the ceph components, mostly due to the fact that it
requires the minimum amount of changes to cross compile ceph for
Windows. However, we're soon going to switch to MSVC/Clang due to mingw
limitations and long standing bugs[5][6]. Porting the unit tests is also
something that we're currently working on.
The next step will be implementing a virtual miniport driver so that RBD
volumes can be exposed to Windows hosts and Hyper-V guests. We're hoping
to leverage librbd as much as possible as part of a daemon that will
communicate with the driver. We're also aiming at cephfs and considering
using Dokan, which is FUSE compatible.
Merging the open PRs would allow us to move forward, focusing on the
drivers and avoiding rebase issues. Any help on that is greatly appreciated.
Last but not least, I'd like to thank Suse, who's sponsoring this effort!
Lucian Petrut
Cloudbase Solutions
[1] https://github.com/ceph/ceph/pull/31981
[2] https://github.com/ceph/ceph/pull/32027
[3] https://github.com/ceph/rocksdb/pull/42
[4] http://paste.openstack.org/raw/787534/
[5] https://sourceforge.net/p/mingw-w64/bugs/816/
[6] https://sourceforge.net/p/mingw-w64/bugs/527/
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
15 Feb
15 Feb
10:09 p.m.
New subject: Windows port
Hi,
Thanks for trying it out. Indeed, this option is currently missing from ceph-dokan but
it’s an easy thing to add. We’ll take care of it as soon as possible, hopefully it will be
included in the Pacific release.
Please let us know if you have any other suggestions.
Regards,
Lucian Petrut
From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>
Sent: Thursday, February 11, 2021 3:00 PM
To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>; Lucian
Petrut<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: Re: Windows port
Hi,
We have been testing ceph-dokan, based on the guide here:
<https://documentation.suse.com/ses/7/single-html/ses-windows/index.html#windows-cephfs>
And watching <https://www.youtube.com/watch?v=BWZIwXLcNts&ab_channel=SUSE>
Initial tests on a Windows 10 VM show good write speed - around 600MB/s,
which is faster than our samba server.
What worries us, is using the "root" ceph.client.admin.keyring on a
Windows system, as it gives access to the entire cephfs cluster - which
in our case is 5PB.
I'd really like this to work, as it would let user administrated Windows
systems that control microscopes to save data directly to cephfs, so
that we can process the data on our HPC cluster.
I'd normally use cephx, and make a key that allows access to a directory
off the root.
e.g.
[root@ceph-s1 users]# ceph auth get client.x_lab
exported keyring for client.x_lab
[client.x_lab]
key = xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==
caps mds = "allow r path=/users/, allow rw path=/users/x_lab"
caps mon = "allow r"
caps osd = "allow class-read object_prefix rbd_children, allow rw
pool=ec82pool"
The real key works fine on linux, but when we try this key with
ceph-dokan, and specify the ceph directory (x_lab) as a ceph path, there
is no option to specify the user - is this hard-coded as admin?
Have I just missed something? Or is this a missing feature?
anyhow, ceph-dokan looks like it could be quite useful,
thank you Cloudbase :)
best regards,
Jake
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
On 4/30/20 1:54 PM, Lucian Petrut wrote:
Hi,
We’ve just pushed the final part of the Windows PR series[1], allowing
RBD images as well as CephFS to be mounted on Windows.
There’s a comprehensive guide[2], describing the build, installation,
configuration and usage steps.
2 out of 12 PRs have been merged already, we look forward to merging the
others as well.
Lucian Petrut
Cloudbase Solutions
[1] https://github.com/ceph/ceph/pull/34859
<https://github.com/ceph/ceph/pull/34859>
[2]
https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst
<https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst>
*From: *Lucian Petrut
<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>
*Sent: *Monday, December 16, 2019 10:12 AM
*To: *dev(a)ceph.io <mailto:dev@ceph.io>
*Subject: *Windows port
Hi,
We're happy to announce that a couple of weeks ago, we've submitted a
few Github pull requests[1][2][3] adding initial Windows support. A big
thank you to the people that have already reviewed the patches.
To bring some context about the scope and current status of our work:
we're mostly targeting the client side, allowing Windows hosts to
consume rados, rbd and cephfs resources.
We have Windows binaries capable of writing to rados pools[4]. We're
using mingw to build the ceph components, mostly due to the fact that it
requires the minimum amount of changes to cross compile ceph for
Windows. However, we're soon going to switch to MSVC/Clang due to mingw
limitations and long standing bugs[5][6]. Porting the unit tests is also
something that we're currently working on.
The next step will be implementing a virtual miniport driver so that RBD
volumes can be exposed to Windows hosts and Hyper-V guests. We're hoping
to leverage librbd as much as possible as part of a daemon that will
communicate with the driver. We're also aiming at cephfs and considering
using Dokan, which is FUSE compatible.
Merging the open PRs would allow us to move forward, focusing on the
drivers and avoiding rebase issues. Any help on that is greatly appreciated.
Last but not least, I'd like to thank Suse, who's sponsoring this effort!
Lucian Petrut
Cloudbase Solutions
[1] https://github.com/ceph/ceph/pull/31981
[2] https://github.com/ceph/ceph/pull/32027
[3] https://github.com/ceph/rocksdb/pull/42
[4] http://paste.openstack.org/raw/787534/
[5] https://sourceforge.net/p/mingw-w64/bugs/816/
[6] https://sourceforge.net/p/mingw-w64/bugs/527/
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
17 Feb
17 Feb
5:25 p.m.
New subject: Windows port
Hi Lucian,
Thanks for your reply :)
Our two main requirements are:
1) Security (...it's great news you can fix this.)
2) Stability (which is understandably harder)
Our testing so far has been on a Windows 10 Pro VM (8 cores, 8GB RAM)
We mount one share from a real WIndows 10 system as Z (hardware RAID,
battery backed 400TB)
Then use ceph-dokan to mount /cephfs on the VM as X
We then use robosync to copy data from the Windows server to the cephfs
mount
C:\>robocopy z:\TestDatasets "X:\jog\Kates Data" /np /mt:128
/log:c:/Users/unixadmin/desktop/robolog.txt /E
After copying 5TB from the WIndows server to /cephfs the ceph-dokan
mount crashes.
C:\WINDOWS\system32>ceph-dokan -l x -o
ceph_conf_read_file OK
ceph_mount OK
ceph_getcwd [/]
/home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: In function
'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T
= short unsigned int; C = std::map]' thread 12 time
2021-02-11T16:56:04.874874GMT Standard Time
/home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: 527: FAILED
ceph_assert(p->first <= start)
ceph version IT-NOTFOUND (f762f7c3be560c11ea0dd51896c976c45137f5ed)
pacific (dev)
restarting robocopy with a lower thread count results in another crash.
Finally, one other feature request: ceph-dokan reports it's version as
"16.0.0" perhaps a "-V" or "--version" switch would be
useful?
We are using
https://github.com/dokan-dev/dokany/releases/download/v1.4.1.1000/DokanSetu…
best regards,
Jake
On 15/02/2021 15:09, Lucian Petrut wrote:
Hi,
Thanks for trying it out. Indeed, this option is currently missing
from ceph-dokan but it’s an easy thing to add. We’ll take care of it
as soon as possible, hopefully it will be included in the Pacific release.
Please let us know if you have any other suggestions.
Regards,
Lucian Petrut
*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk>
*Sent: *Thursday, February 11, 2021 3:00 PM
*To: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com>; Lucian
Petrut
<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>;
dev(a)ceph.io <mailto:dev@ceph.io>
*Subject: *Re: Windows port
Hi,
We have been testing ceph-dokan, based on the guide here:
<https://documentation.suse.com/ses/7/single-html/ses-windows/index.html#windows-cephfs>
And watching <https://www.youtube.com/watch?v=BWZIwXLcNts&ab_channel=SUSE>
Initial tests on a Windows 10 VM show good write speed - around 600MB/s,
which is faster than our samba server.
What worries us, is using the "root" ceph.client.admin.keyring on a
Windows system, as it gives access to the entire cephfs cluster - which
in our case is 5PB.
I'd really like this to work, as it would let user administrated Windows
systems that control microscopes to save data directly to cephfs, so
that we can process the data on our HPC cluster.
I'd normally use cephx, and make a key that allows access to a directory
off the root.
e.g.
[root@ceph-s1 users]# ceph auth get client.x_lab
exported keyring for client.x_lab
[client.x_lab]
key = xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==
caps mds = "allow r path=/users/, allow rw path=/users/x_lab"
caps mon = "allow r"
caps osd = "allow class-read object_prefix rbd_children, allow rw
pool=ec82pool"
The real key works fine on linux, but when we try this key with
ceph-dokan, and specify the ceph directory (x_lab) as a ceph path, there
is no option to specify the user - is this hard-coded as admin?
Have I just missed something? Or is this a missing feature?
anyhow, ceph-dokan looks like it could be quite useful,
thank you Cloudbase :)
best regards,
Jake
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
On 4/30/20 1:54 PM, Lucian Petrut wrote:
Note: I am working from home until further notice.
For help, contact unixadmin(a)mrc-lmb.cam.ac.uk
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
Hi,
We’ve just pushed the final part of the Windows PR series[1], allowing
RBD images as well as CephFS to be mounted on Windows.
There’s a comprehensive guide[2], describing the build, installation,
configuration and usage steps.
2 out of 12 PRs have been merged already, we look forward to merging
the
others as well.
Lucian Petrut
Cloudbase Solutions
[1] https://github.com/ceph/ceph/pull/34859
<https://github.com/ceph/ceph/pull/34859>
[2]
https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst
<https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst>
*From: *Lucian Petrut
<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>
*Sent: *Monday, December 16, 2019 10:12 AM
*To: *dev(a)ceph.io <mailto:dev@ceph.io>
*Subject: *Windows port
Hi,
We're happy to announce that a couple of weeks ago, we've submitted a
few Github pull requests[1][2][3] adding initial Windows support. A big
thank you to the people that have already reviewed the patches.
To bring some context about the scope and current status of our work:
we're mostly targeting the client side, allowing Windows hosts to
consume rados, rbd and cephfs resources.
We have Windows binaries capable of writing to rados pools[4]. We're
using mingw to build the ceph components, mostly due to the fact
that it
requires the minimum amount of changes to cross
compile ceph for
Windows. However, we're soon going to switch to MSVC/Clang due to mingw
limitations and long standing bugs[5][6]. Porting the unit tests is
also
something that we're currently working on.
The next step will be implementing a virtual miniport driver so that
RBD
volumes can be exposed to Windows hosts and
Hyper-V guests. We're
hoping
to leverage librbd as much as possible as part of
a daemon that will
communicate with the driver. We're also aiming at cephfs and
considering
using Dokan, which is FUSE compatible.
Merging the open PRs would allow us to move forward, focusing on the
drivers and avoiding rebase issues. Any help on that is greatly
appreciated.
Last but not least, I'd like to thank Suse, who's sponsoring this
effort!
Lucian Petrut
Cloudbase Solutions
[1] https://github.com/ceph/ceph/pull/31981
[2] https://github.com/ceph/ceph/pull/32027
[3] https://github.com/ceph/rocksdb/pull/42
[4] http://paste.openstack.org/raw/787534/
[5] https://sourceforge.net/p/mingw-w64/bugs/816/
[6] https://sourceforge.net/p/mingw-w64/bugs/527/
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
5:56 p.m.
New subject: Windows port
Hi,
That crash is indeed a concern. We’ll look into it as soon as possible.
A small disclaimer: our main focus was RBD, ceph-dokan was more of a PoC, as mentioned by
the docs. Nevertheless, it seems to be highly demanded so we’re going to invest more time
in it.
FWIW, here’s our nightly Ceph MSI build:
https://cloudbase.it/downloads/ceph_v16_0_0_beta.msi. It provides some important RBD fixes
(though you’re probably more interested in cephfs).
I’ll come with a follow up as soon as those issues are fixed.
Thanks,
Lucian
From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>
Sent: Wednesday, February 17, 2021 12:25 PM
To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: Re: Windows port
Hi Lucian,
Thanks for your reply :)
Our two main requirements are:
1) Security (...it's great news you can fix this.)
2) Stability (which is understandably harder)
Our testing so far has been on a Windows 10 Pro VM (8 cores, 8GB RAM)
We mount one share from a real WIndows 10 system as Z (hardware RAID, battery backed
400TB)
Then use ceph-dokan to mount /cephfs on the VM as X
We then use robosync to copy data from the Windows server to the cephfs mount
C:\>robocopy z:\TestDatasets "X:\jog\Kates Data" /np /mt:128
/log:c:/Users/unixadmin/desktop/robolog.txt /E
After copying 5TB from the WIndows server to /cephfs the ceph-dokan mount crashes.
C:\WINDOWS\system32>ceph-dokan -l x -o
ceph_conf_read_file OK
ceph_mount OK
ceph_getcwd [/]
/home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: In function 'void
interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = short
unsigned int; C = std::map]' thread 12 time 2021-02-11T16:56:04.874874GMT Standard
Time
/home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: 527: FAILED
ceph_assert(p->first <= start)
ceph version IT-NOTFOUND (f762f7c3be560c11ea0dd51896c976c45137f5ed) pacific (dev)
restarting robocopy with a lower thread count results in another crash.
Finally, one other feature request: ceph-dokan reports it's version as
"16.0.0" perhaps a "-V" or "--version" switch would be
useful?
We are using
https://github.com/dokan-dev/dokany/releases/download/v1.4.1.1000/DokanSetu…
best regards,
Jake
On 15/02/2021 15:09, Lucian Petrut wrote:
Hi,
Thanks for trying it out. Indeed, this option is currently missing from ceph-dokan but
it’s an easy thing to add. We’ll take care of it as soon as possible, hopefully it will be
included in the Pacific release.
Please let us know if you have any other suggestions.
Regards,
Lucian Petrut
From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>
Sent: Thursday, February 11, 2021 3:00 PM
To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>; Lucian
Petrut<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: Re: Windows port
Hi,
We have been testing ceph-dokan, based on the guide here:
<https://documentation.suse.com/ses/7/single-html/ses-windows/index.html#windows-cephfs>
And watching <https://www.youtube.com/watch?v=BWZIwXLcNts&ab_channel=SUSE>
Initial tests on a Windows 10 VM show good write speed - around 600MB/s,
which is faster than our samba server.
What worries us, is using the "root" ceph.client.admin.keyring on a
Windows system, as it gives access to the entire cephfs cluster - which
in our case is 5PB.
I'd really like this to work, as it would let user administrated Windows
systems that control microscopes to save data directly to cephfs, so
that we can process the data on our HPC cluster.
I'd normally use cephx, and make a key that allows access to a directory
off the root.
e.g.
[root@ceph-s1 users]# ceph auth get client.x_lab
exported keyring for client.x_lab
[client.x_lab]
key = xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==
caps mds = "allow r path=/users/, allow rw path=/users/x_lab"
caps mon = "allow r"
caps osd = "allow class-read object_prefix rbd_children, allow rw
pool=ec82pool"
The real key works fine on linux, but when we try this key with
ceph-dokan, and specify the ceph directory (x_lab) as a ceph path, there
is no option to specify the user - is this hard-coded as admin?
Have I just missed something? Or is this a missing feature?
anyhow, ceph-dokan looks like it could be quite useful,
thank you Cloudbase :)
best regards,
Jake
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
On 4/30/20 1:54 PM, Lucian Petrut wrote:
Hi,
We’ve just pushed the final part of the Windows PR series[1], allowing
RBD images as well as CephFS to be mounted on Windows.
There’s a comprehensive guide[2], describing the build, installation,
configuration and usage steps.
2 out of 12 PRs have been merged already, we look forward to merging the
others as well.
Lucian Petrut
Cloudbase Solutions
[1] https://github.com/ceph/ceph/pull/34859
<https://github.com/ceph/ceph/pull/34859>
[2]
https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst
<https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst>
*From: *Lucian Petrut
<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>
*Sent: *Monday, December 16, 2019 10:12 AM
*To: *dev(a)ceph.io <mailto:dev@ceph.io>
*Subject: *Windows port
Hi,
We're happy to announce that a couple of weeks ago, we've submitted a
few Github pull requests[1][2][3] adding initial Windows support. A big
thank you to the people that have already reviewed the patches.
To bring some context about the scope and current status of our work:
we're mostly targeting the client side, allowing Windows hosts to
consume rados, rbd and cephfs resources.
We have Windows binaries capable of writing to rados pools[4]. We're
using mingw to build the ceph components, mostly due to the fact that it
requires the minimum amount of changes to cross compile ceph for
Windows. However, we're soon going to switch to MSVC/Clang due to mingw
limitations and long standing bugs[5][6]. Porting the unit tests is also
something that we're currently working on.
The next step will be implementing a virtual miniport driver so that RBD
volumes can be exposed to Windows hosts and Hyper-V guests. We're hoping
to leverage librbd as much as possible as part of a daemon that will
communicate with the driver. We're also aiming at cephfs and considering
using Dokan, which is FUSE compatible.
Merging the open PRs would allow us to move forward, focusing on the
drivers and avoiding rebase issues. Any help on that is greatly appreciated.
Last but not least, I'd like to thank Suse, who's sponsoring this effort!
Lucian Petrut
Cloudbase Solutions
[1] https://github.com/ceph/ceph/pull/31981
[2] https://github.com/ceph/ceph/pull/32027
[3] https://github.com/ceph/rocksdb/pull/42
[4] http://paste.openstack.org/raw/787534/
[5] https://sourceforge.net/p/mingw-w64/bugs/816/
[6] https://sourceforge.net/p/mingw-w64/bugs/527/
_______________________________________________
Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io>
To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io>
Note: I am working from home until further notice.
For help, contact unixadmin@mrc-lmb.cam.ac.uk<mailto:unixadmin@mrc-lmb.cam.ac.uk>
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
9:42 p.m.
New subject: Windows port
Hi Lucian,
Many thanks for looking into this, completely understand that it is just
a PoC, but just to let you know that the crash still occurs with your
nightly build.
C:\Users\unixadmin>ceph-dokan -l x -o
ceph_conf_read_file OK
2021-02-17T12:40:59.637GMT Standard Time 1 -1 asok(0xb36ef20)
AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed
to bind the UNIX domain socket to
'C:/ProgramData/ceph/client.admin.4136.asok': (13) Permission denied
ceph_mount OK
ceph_getcwd [/]
../src/include/interval_set.h: In function 'void interval_set<T,
C>::erase(T, T, std::function<bool(T, T)>) [with T = short unsigned int;
C = std::map]' thread 11 time 2021-02-17T13:58:21.387427GMT Standard Time
../src/include/interval_set.h: 538: FAILED ceph_assert(p->first <= start)
ceph version 15.0.0-21905-g8f11930301
(8f11930301a9ecd5315a8a69a9874eb4428f9366) pacific (rc)
best regards,
Jake
On 17/02/2021 10:56, Lucian Petrut wrote:
Hi,
That crash is indeed a concern. We’ll look into it as soon as possible.
A small disclaimer: our main focus was RBD, ceph-dokan was more of a
PoC, as mentioned by the docs. Nevertheless, it seems to be highly
demanded so we’re going to invest more time in it.
FWIW, here’s our nightly Ceph MSI build:
https://cloudbase.it/downloads/ceph_v16_0_0_beta.msi
<https://cloudbase.it/downloads/ceph_v16_0_0_beta.msi>. It provides
some important RBD fixes (though you’re probably more interested in
cephfs).
I’ll come with a follow up as soon as those issues are fixed.
Thanks,
Lucian
*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk>
*Sent: *Wednesday, February 17, 2021 12:25 PM
*To: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com>;
dev(a)ceph.io <mailto:dev@ceph.io>
*Subject: *Re: Windows port
Hi Lucian,
Thanks for your reply :)
Our two main requirements are:
1) Security (...it's great news you can fix this.)
2) Stability (which is understandably harder)
Our testing so far has been on a Windows 10 Pro VM (8 cores, 8GB RAM)
We mount one share from a real WIndows 10 system as Z (hardware RAID,
battery backed 400TB)
Then use ceph-dokan to mount /cephfs on the VM as X
We then use robosync to copy data from the Windows server to the
cephfs mount
C:\>robocopy z:\TestDatasets "X:\jog\Kates Data" /np /mt:128
/log:c:/Users/unixadmin/desktop/robolog.txt /E
After copying 5TB from the WIndows server to /cephfs the ceph-dokan
mount crashes.
C:\WINDOWS\system32>ceph-dokan -l x -o
ceph_conf_read_file OK
ceph_mount OK
ceph_getcwd [/]
/home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: In
function 'void interval_set<T, C>::erase(T, T, std::function<bool(T,
T)>) [with T = short unsigned int; C = std::map]' thread 12 time
2021-02-11T16:56:04.874874GMT Standard Time
/home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: 527:
FAILED ceph_assert(p->first <= start)
ceph version IT-NOTFOUND (f762f7c3be560c11ea0dd51896c976c45137f5ed)
pacific (dev)
restarting robocopy with a lower thread count results in another crash.
Finally, one other feature request: ceph-dokan reports it's version as
"16.0.0" perhaps a "-V" or "--version" switch would be
useful?
We are using
https://github.com/dokan-dev/dokany/releases/download/v1.4.1.1000/DokanSetu…
best regards,
Jake
On 15/02/2021 15:09, Lucian Petrut wrote:
Hi,
Thanks for trying it out. Indeed, this option is currently missing
from ceph-dokan but it’s an easy thing to add. We’ll take care of
it as soon as possible, hopefully it will be included in the
Pacific release.
Please let us know if you have any other suggestions.
Regards,
Lucian Petrut
*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk>
*Sent: *Thursday, February 11, 2021 3:00 PM
*To: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com>;
Lucian Petrut
<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>;
dev(a)ceph.io <mailto:dev@ceph.io>
*Subject: *Re: Windows port
Hi,
We have been testing ceph-dokan, based on the guide here:
<https://documentation.suse.com/ses/7/single-html/ses-windows/index.html#windows-cephfs>
And watching
<https://www.youtube.com/watch?v=BWZIwXLcNts&ab_channel=SUSE>
Initial tests on a Windows 10 VM show good write speed - around
600MB/s,
which is faster than our samba server.
What worries us, is using the "root" ceph.client.admin.keyring on a
Windows system, as it gives access to the entire cephfs cluster -
which
in our case is 5PB.
I'd really like this to work, as it would let user administrated
Windows
systems that control microscopes to save data directly to cephfs, so
that we can process the data on our HPC cluster.
I'd normally use cephx, and make a key that allows access to a
directory
off the root.
e.g.
[root@ceph-s1 users]# ceph auth get client.x_lab
exported keyring for client.x_lab
[client.x_lab]
key = xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==
caps mds = "allow r path=/users/, allow rw path=/users/x_lab"
caps mon = "allow r"
caps osd = "allow class-read object_prefix rbd_children,
allow rw
pool=ec82pool"
The real key works fine on linux, but when we try this key with
ceph-dokan, and specify the ceph directory (x_lab) as a ceph path,
there
is no option to specify the user - is this hard-coded as admin?
Have I just missed something? Or is this a missing feature?
anyhow, ceph-dokan looks like it could be quite useful,
thank you Cloudbase :)
best regards,
Jake
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
On 4/30/20 1:54 PM, Lucian Petrut wrote:
Note: I am working from home until further notice.
For help, contactunixadmin(a)mrc-lmb.cam.ac.uk <mailto:unixadmin@mrc-lmb.cam.ac.uk>
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
Note: I am working from home until further notice.
For help, contact unixadmin(a)mrc-lmb.cam.ac.uk
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
Hi,
We’ve just pushed the final part of the Windows PR series[1],
allowing
RBD images as well as CephFS to be mounted on
Windows.
There’s a comprehensive guide[2], describing the build,
installation,
configuration and usage steps.
2 out of 12 PRs have been merged already, we look forward to
merging the
others as well.
Lucian Petrut
Cloudbase Solutions
[1] https://github.com/ceph/ceph/pull/34859
<https://github.com/ceph/ceph/pull/34859>
[2]
https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst
<https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst>
*From: *Lucian Petrut
<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>
*Sent: *Monday, December 16, 2019 10:12 AM
*To: *dev(a)ceph.io <mailto:dev@ceph.io>
*Subject: *Windows port
Hi,
We're happy to announce that a couple of weeks ago, we've
submitted a
few Github pull requests[1][2][3] adding initial
Windows
support. A big
thank you to the people that have already
reviewed the patches.
To bring some context about the scope and current status of our
work:
we're mostly targeting the client side,
allowing Windows hosts to
consume rados, rbd and cephfs resources.
We have Windows binaries capable of writing to rados pools[4].
We're
using mingw to build the ceph components, mostly
due to the fact
that it
requires the minimum amount of changes to cross
compile ceph for
Windows. However, we're soon going to switch to MSVC/Clang due
to mingw
limitations and long standing bugs[5][6]. Porting
the unit tests
is also
something that we're currently working on.
The next step will be implementing a virtual miniport driver so
that RBD
volumes can be exposed to Windows hosts and
Hyper-V guests.
We're hoping
to leverage librbd as much as possible as part of
a daemon that
will
communicate with the driver. We're also
aiming at cephfs and
considering
using Dokan, which is FUSE compatible.
Merging the open PRs would allow us to move forward, focusing on
the
drivers and avoiding rebase issues. Any help on
that is greatly
appreciated.
Last but not least, I'd like to thank Suse, who's sponsoring
this
effort!
Lucian Petrut
Cloudbase Solutions
[1] https://github.com/ceph/ceph/pull/31981
[2] https://github.com/ceph/ceph/pull/32027
[3] https://github.com/ceph/rocksdb/pull/42
[4] http://paste.openstack.org/raw/787534/
[5] https://sourceforge.net/p/mingw-w64/bugs/816/
[6] https://sourceforge.net/p/mingw-w64/bugs/527/
_______________________________________________
Dev mailing list -- dev(a)ceph.io <mailto:dev@ceph.io>
To unsubscribe send an email to dev-leave(a)ceph.io
<mailto:dev-leave@ceph.io>
19 Feb
19 Feb
8:56 p.m.
New subject: Windows port
Hi,
I couldn’t reproduce the issue yet. Judging by the exception message, it seems to be
related to libcephfs’s inode handling.
The latest msi installer includes debug symbols. Could you please send us an archive with
a crash dump [1] and ideally the msi installer as well?
Thanks,
Lucian
[1] https://docs.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dum…
From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>
Sent: Wednesday, February 17, 2021 4:42 PM
To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: Re: Windows port
Hi Lucian,
Many thanks for looking into this, completely understand that it is just a PoC, but just
to let you know that the crash still occurs with your nightly build.
C:\Users\unixadmin>ceph-dokan -l x -o
ceph_conf_read_file OK
2021-02-17T12:40:59.637GMT Standard Time 1 -1 asok(0xb36ef20) AdminSocketConfigObs::init:
failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to
'C:/ProgramData/ceph/client.admin.4136.asok': (13) Permission denied
ceph_mount OK
ceph_getcwd [/]
../src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T,
std::function<bool(T, T)>) [with T = short unsigned int; C = std::map]' thread
11 time 2021-02-17T13:58:21.387427GMT Standard Time
../src/include/interval_set.h: 538: FAILED ceph_assert(p->first <= start)
ceph version 15.0.0-21905-g8f11930301 (8f11930301a9ecd5315a8a69a9874eb4428f9366) pacific
(rc)
best regards,
Jake
On 17/02/2021 10:56, Lucian Petrut wrote:
Hi,
That crash is indeed a concern. We’ll look into it as soon as possible.
A small disclaimer: our main focus was RBD, ceph-dokan was more of a PoC, as mentioned by
the docs. Nevertheless, it seems to be highly demanded so we’re going to invest more time
in it.
FWIW, here’s our nightly Ceph MSI build:
https://cloudbase.it/downloads/ceph_v16_0_0_beta.msi. It provides some important RBD fixes
(though you’re probably more interested in cephfs).
I’ll come with a follow up as soon as those issues are fixed.
Thanks,
Lucian
From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>
Sent: Wednesday, February 17, 2021 12:25 PM
To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: Re: Windows port
Hi Lucian,
Thanks for your reply :)
Our two main requirements are:
1) Security (...it's great news you can fix this.)
2) Stability (which is understandably harder)
Our testing so far has been on a Windows 10 Pro VM (8 cores, 8GB RAM)
We mount one share from a real WIndows 10 system as Z (hardware RAID, battery backed
400TB)
Then use ceph-dokan to mount /cephfs on the VM as X
We then use robosync to copy data from the Windows server to the cephfs mount
C:\>robocopy z:\TestDatasets "X:\jog\Kates Data" /np /mt:128
/log:c:/Users/unixadmin/desktop/robolog.txt /E
After copying 5TB from the WIndows server to /cephfs the ceph-dokan mount crashes.
C:\WINDOWS\system32>ceph-dokan -l x -o
ceph_conf_read_file OK
ceph_mount OK
ceph_getcwd [/]
/home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: In function 'void
interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = short
unsigned int; C = std::map]' thread 12 time 2021-02-11T16:56:04.874874GMT Standard
Time
/home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: 527: FAILED
ceph_assert(p->first <= start)
ceph version IT-NOTFOUND (f762f7c3be560c11ea0dd51896c976c45137f5ed) pacific (dev)
restarting robocopy with a lower thread count results in another crash.
Finally, one other feature request: ceph-dokan reports it's version as
"16.0.0" perhaps a "-V" or "--version" switch would be
useful?
We are using
https://github.com/dokan-dev/dokany/releases/download/v1.4.1.1000/DokanSetu…
best regards,
Jake
On 15/02/2021 15:09, Lucian Petrut wrote:
Hi,
Thanks for trying it out. Indeed, this option is currently missing from ceph-dokan but
it’s an easy thing to add. We’ll take care of it as soon as possible, hopefully it will be
included in the Pacific release.
Please let us know if you have any other suggestions.
Regards,
Lucian Petrut
From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>
Sent: Thursday, February 11, 2021 3:00 PM
To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>; Lucian
Petrut<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: Re: Windows port
Hi,
We have been testing ceph-dokan, based on the guide here:
<https://documentation.suse.com/ses/7/single-html/ses-windows/index.html#windows-cephfs>
And watching <https://www.youtube.com/watch?v=BWZIwXLcNts&ab_channel=SUSE>
Initial tests on a Windows 10 VM show good write speed - around 600MB/s,
which is faster than our samba server.
What worries us, is using the "root" ceph.client.admin.keyring on a
Windows system, as it gives access to the entire cephfs cluster - which
in our case is 5PB.
I'd really like this to work, as it would let user administrated Windows
systems that control microscopes to save data directly to cephfs, so
that we can process the data on our HPC cluster.
I'd normally use cephx, and make a key that allows access to a directory
off the root.
e.g.
[root@ceph-s1 users]# ceph auth get client.x_lab
exported keyring for client.x_lab
[client.x_lab]
key = xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==
caps mds = "allow r path=/users/, allow rw path=/users/x_lab"
caps mon = "allow r"
caps osd = "allow class-read object_prefix rbd_children, allow rw
pool=ec82pool"
The real key works fine on linux, but when we try this key with
ceph-dokan, and specify the ceph directory (x_lab) as a ceph path, there
is no option to specify the user - is this hard-coded as admin?
Have I just missed something? Or is this a missing feature?
anyhow, ceph-dokan looks like it could be quite useful,
thank you Cloudbase :)
best regards,
Jake
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
On 4/30/20 1:54 PM, Lucian Petrut wrote:
Hi,
We’ve just pushed the final part of the Windows PR series[1], allowing
RBD images as well as CephFS to be mounted on Windows.
There’s a comprehensive guide[2], describing the build, installation,
configuration and usage steps.
2 out of 12 PRs have been merged already, we look forward to merging the
others as well.
Lucian Petrut
Cloudbase Solutions
[1] https://github.com/ceph/ceph/pull/34859
<https://github.com/ceph/ceph/pull/34859>
[2]
https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst
<https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst>
*From: *Lucian Petrut
<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>
*Sent: *Monday, December 16, 2019 10:12 AM
*To: *dev(a)ceph.io <mailto:dev@ceph.io>
*Subject: *Windows port
Hi,
We're happy to announce that a couple of weeks ago, we've submitted a
few Github pull requests[1][2][3] adding initial Windows support. A big
thank you to the people that have already reviewed the patches.
To bring some context about the scope and current status of our work:
we're mostly targeting the client side, allowing Windows hosts to
consume rados, rbd and cephfs resources.
We have Windows binaries capable of writing to rados pools[4]. We're
using mingw to build the ceph components, mostly due to the fact that it
requires the minimum amount of changes to cross compile ceph for
Windows. However, we're soon going to switch to MSVC/Clang due to mingw
limitations and long standing bugs[5][6]. Porting the unit tests is also
something that we're currently working on.
The next step will be implementing a virtual miniport driver so that RBD
volumes can be exposed to Windows hosts and Hyper-V guests. We're hoping
to leverage librbd as much as possible as part of a daemon that will
communicate with the driver. We're also aiming at cephfs and considering
using Dokan, which is FUSE compatible.
Merging the open PRs would allow us to move forward, focusing on the
drivers and avoiding rebase issues. Any help on that is greatly appreciated.
Last but not least, I'd like to thank Suse, who's sponsoring this effort!
Lucian Petrut
Cloudbase Solutions
[1] https://github.com/ceph/ceph/pull/31981
[2] https://github.com/ceph/ceph/pull/32027
[3] https://github.com/ceph/rocksdb/pull/42
[4] http://paste.openstack.org/raw/787534/
[5] https://sourceforge.net/p/mingw-w64/bugs/816/
[6] https://sourceforge.net/p/mingw-w64/bugs/527/
_______________________________________________
Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io>
To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io>
Note: I am working from home until further notice.
For help, contact unixadmin@mrc-lmb.cam.ac.uk<mailto:unixadmin@mrc-lmb.cam.ac.uk>
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
Note: I am working from home until further notice.
For help, contact unixadmin@mrc-lmb.cam.ac.uk<mailto:unixadmin@mrc-lmb.cam.ac.uk>
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
9:51 p.m.
New subject: Windows port
Hi,
Nevermind, I managed to get a crash after increasing the number of threads:
http://paste.openstack.org/raw/802827/
I’ll try to get a fix as soon as possible. Thanks again for all the info!
Lucian
From: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>
Sent: Friday, February 19, 2021 3:56 PM
To: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: RE: Windows port
Hi,
I couldn’t reproduce the issue yet. Judging by the exception message, it seems to be
related to libcephfs’s inode handling.
The latest msi installer includes debug symbols. Could you please send us an archive with
a crash dump [1] and ideally the msi installer as well?
Thanks,
Lucian
[1] https://docs.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dum…
From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>
Sent: Wednesday, February 17, 2021 4:42 PM
To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: Re: Windows port
Hi Lucian,
Many thanks for looking into this, completely understand that it is just a PoC, but just
to let you know that the crash still occurs with your nightly build.
C:\Users\unixadmin>ceph-dokan -l x -o
ceph_conf_read_file OK
2021-02-17T12:40:59.637GMT Standard Time 1 -1 asok(0xb36ef20) AdminSocketConfigObs::init:
failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to
'C:/ProgramData/ceph/client.admin.4136.asok': (13) Permission denied
ceph_mount OK
ceph_getcwd [/]
../src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T,
std::function<bool(T, T)>) [with T = short unsigned int; C = std::map]' thread
11 time 2021-02-17T13:58:21.387427GMT Standard Time
../src/include/interval_set.h: 538: FAILED ceph_assert(p->first <= start)
ceph version 15.0.0-21905-g8f11930301 (8f11930301a9ecd5315a8a69a9874eb4428f9366) pacific
(rc)
best regards,
Jake
On 17/02/2021 10:56, Lucian Petrut wrote:
Hi,
That crash is indeed a concern. We’ll look into it as soon as possible.
A small disclaimer: our main focus was RBD, ceph-dokan was more of a PoC, as mentioned by
the docs. Nevertheless, it seems to be highly demanded so we’re going to invest more time
in it.
FWIW, here’s our nightly Ceph MSI build:
https://cloudbase.it/downloads/ceph_v16_0_0_beta.msi. It provides some important RBD fixes
(though you’re probably more interested in cephfs).
I’ll come with a follow up as soon as those issues are fixed.
Thanks,
Lucian
From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>
Sent: Wednesday, February 17, 2021 12:25 PM
To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: Re: Windows port
Hi Lucian,
Thanks for your reply :)
Our two main requirements are:
1) Security (...it's great news you can fix this.)
2) Stability (which is understandably harder)
Our testing so far has been on a Windows 10 Pro VM (8 cores, 8GB RAM)
We mount one share from a real WIndows 10 system as Z (hardware RAID, battery backed
400TB)
Then use ceph-dokan to mount /cephfs on the VM as X
We then use robosync to copy data from the Windows server to the cephfs mount
C:\>robocopy z:\TestDatasets "X:\jog\Kates Data" /np /mt:128
/log:c:/Users/unixadmin/desktop/robolog.txt /E
After copying 5TB from the WIndows server to /cephfs the ceph-dokan mount crashes.
C:\WINDOWS\system32>ceph-dokan -l x -o
ceph_conf_read_file OK
ceph_mount OK
ceph_getcwd [/]
/home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: In function 'void
interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = short
unsigned int; C = std::map]' thread 12 time 2021-02-11T16:56:04.874874GMT Standard
Time
/home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: 527: FAILED
ceph_assert(p->first <= start)
ceph version IT-NOTFOUND (f762f7c3be560c11ea0dd51896c976c45137f5ed) pacific (dev)
restarting robocopy with a lower thread count results in another crash.
Finally, one other feature request: ceph-dokan reports it's version as
"16.0.0" perhaps a "-V" or "--version" switch would be
useful?
We are using
https://github.com/dokan-dev/dokany/releases/download/v1.4.1.1000/DokanSetu…
best regards,
Jake
On 15/02/2021 15:09, Lucian Petrut wrote:
Hi,
Thanks for trying it out. Indeed, this option is currently missing from ceph-dokan but
it’s an easy thing to add. We’ll take care of it as soon as possible, hopefully it will be
included in the Pacific release.
Please let us know if you have any other suggestions.
Regards,
Lucian Petrut
From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>
Sent: Thursday, February 11, 2021 3:00 PM
To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>; Lucian
Petrut<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: Re: Windows port
Hi,
We have been testing ceph-dokan, based on the guide here:
<https://documentation.suse.com/ses/7/single-html/ses-windows/index.html#windows-cephfs>
And watching <https://www.youtube.com/watch?v=BWZIwXLcNts&ab_channel=SUSE>
Initial tests on a Windows 10 VM show good write speed - around 600MB/s,
which is faster than our samba server.
What worries us, is using the "root" ceph.client.admin.keyring on a
Windows system, as it gives access to the entire cephfs cluster - which
in our case is 5PB.
I'd really like this to work, as it would let user administrated Windows
systems that control microscopes to save data directly to cephfs, so
that we can process the data on our HPC cluster.
I'd normally use cephx, and make a key that allows access to a directory
off the root.
e.g.
[root@ceph-s1 users]# ceph auth get client.x_lab
exported keyring for client.x_lab
[client.x_lab]
key = xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==
caps mds = "allow r path=/users/, allow rw path=/users/x_lab"
caps mon = "allow r"
caps osd = "allow class-read object_prefix rbd_children, allow rw
pool=ec82pool"
The real key works fine on linux, but when we try this key with
ceph-dokan, and specify the ceph directory (x_lab) as a ceph path, there
is no option to specify the user - is this hard-coded as admin?
Have I just missed something? Or is this a missing feature?
anyhow, ceph-dokan looks like it could be quite useful,
thank you Cloudbase :)
best regards,
Jake
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
On 4/30/20 1:54 PM, Lucian Petrut wrote:
Hi,
We’ve just pushed the final part of the Windows PR series[1], allowing
RBD images as well as CephFS to be mounted on Windows.
There’s a comprehensive guide[2], describing the build, installation,
configuration and usage steps.
2 out of 12 PRs have been merged already, we look forward to merging the
others as well.
Lucian Petrut
Cloudbase Solutions
[1] https://github.com/ceph/ceph/pull/34859
<https://github.com/ceph/ceph/pull/34859>
[2]
https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst
<https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst>
*From: *Lucian Petrut
<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>
*Sent: *Monday, December 16, 2019 10:12 AM
*To: *dev(a)ceph.io <mailto:dev@ceph.io>
*Subject: *Windows port
Hi,
We're happy to announce that a couple of weeks ago, we've submitted a
few Github pull requests[1][2][3] adding initial Windows support. A big
thank you to the people that have already reviewed the patches.
To bring some context about the scope and current status of our work:
we're mostly targeting the client side, allowing Windows hosts to
consume rados, rbd and cephfs resources.
We have Windows binaries capable of writing to rados pools[4]. We're
using mingw to build the ceph components, mostly due to the fact that it
requires the minimum amount of changes to cross compile ceph for
Windows. However, we're soon going to switch to MSVC/Clang due to mingw
limitations and long standing bugs[5][6]. Porting the unit tests is also
something that we're currently working on.
The next step will be implementing a virtual miniport driver so that RBD
volumes can be exposed to Windows hosts and Hyper-V guests. We're hoping
to leverage librbd as much as possible as part of a daemon that will
communicate with the driver. We're also aiming at cephfs and considering
using Dokan, which is FUSE compatible.
Merging the open PRs would allow us to move forward, focusing on the
drivers and avoiding rebase issues. Any help on that is greatly appreciated.
Last but not least, I'd like to thank Suse, who's sponsoring this effort!
Lucian Petrut
Cloudbase Solutions
[1] https://github.com/ceph/ceph/pull/31981
[2] https://github.com/ceph/ceph/pull/32027
[3] https://github.com/ceph/rocksdb/pull/42
[4] http://paste.openstack.org/raw/787534/
[5] https://sourceforge.net/p/mingw-w64/bugs/816/
[6] https://sourceforge.net/p/mingw-w64/bugs/527/
_______________________________________________
Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io>
To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io>
Note: I am working from home until further notice.
For help, contact unixadmin@mrc-lmb.cam.ac.uk<mailto:unixadmin@mrc-lmb.cam.ac.uk>
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
Note: I am working from home until further notice.
For help, contact unixadmin@mrc-lmb.cam.ac.uk<mailto:unixadmin@mrc-lmb.cam.ac.uk>
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
22 Feb
22 Feb
10:12 p.m.
New subject: Windows port
Hi,
I’ve fixed the crash. It was due to Ceph’s inode emulation, which can be skipped as
Windows supports 64B file identifiers.
The fix is included in the latest msi.
I’ll prepare a few more cephfs related fixes, including the user option.
Thanks,
Lucian
From: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>
Sent: Friday, February 19, 2021 4:51 PM
To: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: RE: Windows port
Hi,
Nevermind, I managed to get a crash after increasing the number of threads:
http://paste.openstack.org/raw/802827/
I’ll try to get a fix as soon as possible. Thanks again for all the info!
Lucian
From: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>
Sent: Friday, February 19, 2021 3:56 PM
To: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: RE: Windows port
Hi,
I couldn’t reproduce the issue yet. Judging by the exception message, it seems to be
related to libcephfs’s inode handling.
The latest msi installer includes debug symbols. Could you please send us an archive with
a crash dump [1] and ideally the msi installer as well?
Thanks,
Lucian
[1] https://docs.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dum…
From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>
Sent: Wednesday, February 17, 2021 4:42 PM
To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: Re: Windows port
Hi Lucian,
Many thanks for looking into this, completely understand that it is just a PoC, but just
to let you know that the crash still occurs with your nightly build.
C:\Users\unixadmin>ceph-dokan -l x -o
ceph_conf_read_file OK
2021-02-17T12:40:59.637GMT Standard Time 1 -1 asok(0xb36ef20) AdminSocketConfigObs::init:
failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to
'C:/ProgramData/ceph/client.admin.4136.asok': (13) Permission denied
ceph_mount OK
ceph_getcwd [/]
../src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T,
std::function<bool(T, T)>) [with T = short unsigned int; C = std::map]' thread
11 time 2021-02-17T13:58:21.387427GMT Standard Time
../src/include/interval_set.h: 538: FAILED ceph_assert(p->first <= start)
ceph version 15.0.0-21905-g8f11930301 (8f11930301a9ecd5315a8a69a9874eb4428f9366) pacific
(rc)
best regards,
Jake
On 17/02/2021 10:56, Lucian Petrut wrote:
Hi,
That crash is indeed a concern. We’ll look into it as soon as possible.
A small disclaimer: our main focus was RBD, ceph-dokan was more of a PoC, as mentioned by
the docs. Nevertheless, it seems to be highly demanded so we’re going to invest more time
in it.
FWIW, here’s our nightly Ceph MSI build:
https://cloudbase.it/downloads/ceph_v16_0_0_beta.msi. It provides some important RBD fixes
(though you’re probably more interested in cephfs).
I’ll come with a follow up as soon as those issues are fixed.
Thanks,
Lucian
From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>
Sent: Wednesday, February 17, 2021 12:25 PM
To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: Re: Windows port
Hi Lucian,
Thanks for your reply :)
Our two main requirements are:
1) Security (...it's great news you can fix this.)
2) Stability (which is understandably harder)
Our testing so far has been on a Windows 10 Pro VM (8 cores, 8GB RAM)
We mount one share from a real WIndows 10 system as Z (hardware RAID, battery backed
400TB)
Then use ceph-dokan to mount /cephfs on the VM as X
We then use robosync to copy data from the Windows server to the cephfs mount
C:\>robocopy z:\TestDatasets "X:\jog\Kates Data" /np /mt:128
/log:c:/Users/unixadmin/desktop/robolog.txt /E
After copying 5TB from the WIndows server to /cephfs the ceph-dokan mount crashes.
C:\WINDOWS\system32>ceph-dokan -l x -o
ceph_conf_read_file OK
ceph_mount OK
ceph_getcwd [/]
/home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: In function 'void
interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = short
unsigned int; C = std::map]' thread 12 time 2021-02-11T16:56:04.874874GMT Standard
Time
/home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: 527: FAILED
ceph_assert(p->first <= start)
ceph version IT-NOTFOUND (f762f7c3be560c11ea0dd51896c976c45137f5ed) pacific (dev)
restarting robocopy with a lower thread count results in another crash.
Finally, one other feature request: ceph-dokan reports it's version as
"16.0.0" perhaps a "-V" or "--version" switch would be
useful?
We are using
https://github.com/dokan-dev/dokany/releases/download/v1.4.1.1000/DokanSetu…
best regards,
Jake
On 15/02/2021 15:09, Lucian Petrut wrote:
Hi,
Thanks for trying it out. Indeed, this option is currently missing from ceph-dokan but
it’s an easy thing to add. We’ll take care of it as soon as possible, hopefully it will be
included in the Pacific release.
Please let us know if you have any other suggestions.
Regards,
Lucian Petrut
From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>
Sent: Thursday, February 11, 2021 3:00 PM
To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>; Lucian
Petrut<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: Re: Windows port
Hi,
We have been testing ceph-dokan, based on the guide here:
<https://documentation.suse.com/ses/7/single-html/ses-windows/index.html#windows-cephfs>
And watching <https://www.youtube.com/watch?v=BWZIwXLcNts&ab_channel=SUSE>
Initial tests on a Windows 10 VM show good write speed - around 600MB/s,
which is faster than our samba server.
What worries us, is using the "root" ceph.client.admin.keyring on a
Windows system, as it gives access to the entire cephfs cluster - which
in our case is 5PB.
I'd really like this to work, as it would let user administrated Windows
systems that control microscopes to save data directly to cephfs, so
that we can process the data on our HPC cluster.
I'd normally use cephx, and make a key that allows access to a directory
off the root.
e.g.
[root@ceph-s1 users]# ceph auth get client.x_lab
exported keyring for client.x_lab
[client.x_lab]
key = xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==
caps mds = "allow r path=/users/, allow rw path=/users/x_lab"
caps mon = "allow r"
caps osd = "allow class-read object_prefix rbd_children, allow rw
pool=ec82pool"
The real key works fine on linux, but when we try this key with
ceph-dokan, and specify the ceph directory (x_lab) as a ceph path, there
is no option to specify the user - is this hard-coded as admin?
Have I just missed something? Or is this a missing feature?
anyhow, ceph-dokan looks like it could be quite useful,
thank you Cloudbase :)
best regards,
Jake
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
On 4/30/20 1:54 PM, Lucian Petrut wrote:
Hi,
We’ve just pushed the final part of the Windows PR series[1], allowing
RBD images as well as CephFS to be mounted on Windows.
There’s a comprehensive guide[2], describing the build, installation,
configuration and usage steps.
2 out of 12 PRs have been merged already, we look forward to merging the
others as well.
Lucian Petrut
Cloudbase Solutions
[1] https://github.com/ceph/ceph/pull/34859
<https://github.com/ceph/ceph/pull/34859>
[2]
https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst
<https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst>
*From: *Lucian Petrut
<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>
*Sent: *Monday, December 16, 2019 10:12 AM
*To: *dev(a)ceph.io <mailto:dev@ceph.io>
*Subject: *Windows port
Hi,
We're happy to announce that a couple of weeks ago, we've submitted a
few Github pull requests[1][2][3] adding initial Windows support. A big
thank you to the people that have already reviewed the patches.
To bring some context about the scope and current status of our work:
we're mostly targeting the client side, allowing Windows hosts to
consume rados, rbd and cephfs resources.
We have Windows binaries capable of writing to rados pools[4]. We're
using mingw to build the ceph components, mostly due to the fact that it
requires the minimum amount of changes to cross compile ceph for
Windows. However, we're soon going to switch to MSVC/Clang due to mingw
limitations and long standing bugs[5][6]. Porting the unit tests is also
something that we're currently working on.
The next step will be implementing a virtual miniport driver so that RBD
volumes can be exposed to Windows hosts and Hyper-V guests. We're hoping
to leverage librbd as much as possible as part of a daemon that will
communicate with the driver. We're also aiming at cephfs and considering
using Dokan, which is FUSE compatible.
Merging the open PRs would allow us to move forward, focusing on the
drivers and avoiding rebase issues. Any help on that is greatly appreciated.
Last but not least, I'd like to thank Suse, who's sponsoring this effort!
Lucian Petrut
Cloudbase Solutions
[1] https://github.com/ceph/ceph/pull/31981
[2] https://github.com/ceph/ceph/pull/32027
[3] https://github.com/ceph/rocksdb/pull/42
[4] http://paste.openstack.org/raw/787534/
[5] https://sourceforge.net/p/mingw-w64/bugs/816/
[6] https://sourceforge.net/p/mingw-w64/bugs/527/
_______________________________________________
Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io>
To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io>
Note: I am working from home until further notice.
For help, contact unixadmin@mrc-lmb.cam.ac.uk<mailto:unixadmin@mrc-lmb.cam.ac.uk>
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
Note: I am working from home until further notice.
For help, contact unixadmin@mrc-lmb.cam.ac.uk<mailto:unixadmin@mrc-lmb.cam.ac.uk>
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
11:29 p.m.
New subject: Windows port
Hi Lucian,
many thanks for this - I'll test and get back to you :)
best regards,
Jake
On 22/02/2021 15:12, Lucian Petrut wrote:
Hi,
I’ve fixed the crash. It was due to Ceph’s inode emulation, which can
be skipped as Windows supports 64B file identifiers.
The fix is included in the latest msi.
I’ll prepare a few more cephfs related fixes, including the user option.
Thanks,
Lucian
*From: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com>
*Sent: *Friday, February 19, 2021 4:51 PM
*To: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk>; dev(a)ceph.io
<mailto:dev@ceph.io>
*Subject: *RE: Windows port
Hi,
Nevermind, I managed to get a crash after increasing the number of
threads: http://paste.openstack.org/raw/802827/
I’ll try to get a fix as soon as possible. Thanks again for all the info!
Lucian
*From: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com>
*Sent: *Friday, February 19, 2021 3:56 PM
*To: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk>; dev(a)ceph.io
<mailto:dev@ceph.io>
*Subject: *RE: Windows port
Hi,
I couldn’t reproduce the issue yet. Judging by the exception message,
it seems to be related to libcephfs’s inode handling.
The latest msi installer includes debug symbols. Could you please send
us an archive with a crash dump [1] and ideally the msi installer as well?
Thanks,
Lucian
[1]
https://docs.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dum…
*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk>
*Sent: *Wednesday, February 17, 2021 4:42 PM
*To: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com>;
dev(a)ceph.io <mailto:dev@ceph.io>
*Subject: *Re: Windows port
Hi Lucian,
Many thanks for looking into this, completely understand that it is
just a PoC, but just to let you know that the crash still occurs with
your nightly build.
C:\Users\unixadmin>ceph-dokan -l x -o
ceph_conf_read_file OK
2021-02-17T12:40:59.637GMT Standard Time 1 -1 asok(0xb36ef20)
AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen:
failed to bind the UNIX domain socket to
'C:/ProgramData/ceph/client.admin.4136.asok': (13) Permission denied
ceph_mount OK
ceph_getcwd [/]
../src/include/interval_set.h: In function 'void interval_set<T,
C>::erase(T, T, std::function<bool(T, T)>) [with T = short unsigned
int; C = std::map]' thread 11 time 2021-02-17T13:58:21.387427GMT
Standard Time
../src/include/interval_set.h: 538: FAILED ceph_assert(p->first <= start)
ceph version 15.0.0-21905-g8f11930301
(8f11930301a9ecd5315a8a69a9874eb4428f9366) pacific (rc)
best regards,
Jake
On 17/02/2021 10:56, Lucian Petrut wrote:
Hi,
That crash is indeed a concern. We’ll look into it as soon as
possible.
A small disclaimer: our main focus was RBD, ceph-dokan was more of
a PoC, as mentioned by the docs. Nevertheless, it seems to be
highly demanded so we’re going to invest more time in it.
FWIW, here’s our nightly Ceph MSI build:
https://cloudbase.it/downloads/ceph_v16_0_0_beta.msi
<https://cloudbase.it/downloads/ceph_v16_0_0_beta.msi>. It
provides some important RBD fixes (though you’re probably more
interested in cephfs).
I’ll come with a follow up as soon as those issues are fixed.
Thanks,
Lucian
*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk>
*Sent: *Wednesday, February 17, 2021 12:25 PM
*To: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com>;
dev(a)ceph.io <mailto:dev@ceph.io>
*Subject: *Re: Windows port
Hi Lucian,
Thanks for your reply :)
Our two main requirements are:
1) Security (...it's great news you can fix this.)
2) Stability (which is understandably harder)
Our testing so far has been on a Windows 10 Pro VM (8 cores, 8GB RAM)
We mount one share from a real WIndows 10 system as Z (hardware
RAID, battery backed 400TB)
Then use ceph-dokan to mount /cephfs on the VM as X
We then use robosync to copy data from the Windows server to the
cephfs mount
C:\>robocopy z:\TestDatasets "X:\jog\Kates Data" /np /mt:128
/log:c:/Users/unixadmin/desktop/robolog.txt /E
After copying 5TB from the WIndows server to /cephfs the
ceph-dokan mount crashes.
C:\WINDOWS\system32>ceph-dokan -l x -o
ceph_conf_read_file OK
ceph_mount OK
ceph_getcwd [/]
/home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: In
function 'void interval_set<T, C>::erase(T, T,
std::function<bool(T, T)>) [with T = short unsigned int; C =
std::map]' thread 12 time 2021-02-11T16:56:04.874874GMT Standard Time
/home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: 527:
FAILED ceph_assert(p->first <= start)
ceph version IT-NOTFOUND
(f762f7c3be560c11ea0dd51896c976c45137f5ed) pacific (dev)
restarting robocopy with a lower thread count results in another
crash.
Finally, one other feature request: ceph-dokan reports it's
version as "16.0.0" perhaps a "-V" or "--version"
switch would be
useful?
We are using
https://github.com/dokan-dev/dokany/releases/download/v1.4.1.1000/DokanSetu…
best regards,
Jake
On 15/02/2021 15:09, Lucian Petrut wrote:
Hi,
Thanks for trying it out. Indeed, this option is currently
missing from ceph-dokan but it’s an easy thing to add. We’ll
take care of it as soon as possible, hopefully it will be
included in the Pacific release.
Please let us know if you have any other suggestions.
Regards,
Lucian Petrut
*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk>
*Sent: *Thursday, February 11, 2021 3:00 PM
*To: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com>;
Lucian Petrut
<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>;
dev(a)ceph.io <mailto:dev@ceph.io>
*Subject: *Re: Windows port
Hi,
We have been testing ceph-dokan, based on the guide here:
<https://documentation.suse.com/ses/7/single-html/ses-windows/index.html#windows-cephfs>
And watching
<https://www.youtube.com/watch?v=BWZIwXLcNts&ab_channel=SUSE>
Initial tests on a Windows 10 VM show good write speed -
around 600MB/s,
which is faster than our samba server.
What worries us, is using the "root" ceph.client.admin.keyring
on a
Windows system, as it gives access to the entire cephfs
cluster - which
in our case is 5PB.
I'd really like this to work, as it would let user
administrated Windows
systems that control microscopes to save data directly to
cephfs, so
that we can process the data on our HPC cluster.
I'd normally use cephx, and make a key that allows access to a
directory
off the root.
e.g.
[root@ceph-s1 users]# ceph auth get client.x_lab
exported keyring for client.x_lab
[client.x_lab]
key = xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==
caps mds = "allow r path=/users/, allow rw
path=/users/x_lab"
caps mon = "allow r"
caps osd = "allow class-read object_prefix
rbd_children, allow rw
pool=ec82pool"
The real key works fine on linux, but when we try this key with
ceph-dokan, and specify the ceph directory (x_lab) as a ceph
path, there
is no option to specify the user - is this hard-coded as admin?
Have I just missed something? Or is this a missing feature?
anyhow, ceph-dokan looks like it could be quite useful,
thank you Cloudbase :)
best regards,
Jake
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
On 4/30/20 1:54 PM, Lucian Petrut wrote:
Note: I am working from home until further notice.
For help, contactunixadmin(a)mrc-lmb.cam.ac.uk
<mailto:unixadmin@mrc-lmb.cam.ac.uk>
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
Note: I am working from home until further notice.
For help, contactunixadmin(a)mrc-lmb.cam.ac.uk <mailto:unixadmin@mrc-lmb.cam.ac.uk>
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
Note: I am working from home until further notice.
For help, contact unixadmin(a)mrc-lmb.cam.ac.uk
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
Hi,
We’ve just pushed the final part of the Windows PR
series[1], allowing
RBD images as well as CephFS to be mounted on
Windows.
There’s a comprehensive guide[2], describing the build,
installation,
configuration and usage steps.
2 out of 12 PRs have been merged already, we look forward to
merging the
others as well.
Lucian Petrut
Cloudbase Solutions
[1] https://github.com/ceph/ceph/pull/34859
<https://github.com/ceph/ceph/pull/34859>
[2]
https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst
<https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst>
*From: *Lucian Petrut
<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>
*Sent: *Monday, December 16, 2019 10:12 AM
*To: *dev(a)ceph.io <mailto:dev@ceph.io>
*Subject: *Windows port
Hi,
We're happy to announce that a couple of weeks ago, we've
submitted a
few Github pull requests[1][2][3] adding initial
Windows
support. A big
thank you to the people that have already
reviewed the patches.
To bring some context about the scope and current status of
our work:
we're mostly targeting the client side,
allowing Windows
hosts to
consume rados, rbd and cephfs resources.
We have Windows binaries capable of writing to rados
pools[4]. We're
using mingw to build the ceph components, mostly
due to the
fact that it
requires the minimum amount of changes to cross
compile ceph
for
Windows. However, we're soon going to switch
to MSVC/Clang
due to mingw
limitations and long standing bugs[5][6]. Porting
the unit
tests is also
something that we're currently working on.
The next step will be implementing a virtual miniport driver
so that RBD
volumes can be exposed to Windows hosts and
Hyper-V guests.
We're hoping
to leverage librbd as much as possible as part of
a daemon
that will
communicate with the driver. We're also
aiming at cephfs and
considering
using Dokan, which is FUSE compatible.
Merging the open PRs would allow us to move forward,
focusing on the
drivers and avoiding rebase issues. Any help on
that is
greatly appreciated.
Last but not least, I'd like to thank Suse, who's sponsoring
this
effort!
Lucian Petrut
Cloudbase Solutions
[1] https://github.com/ceph/ceph/pull/31981
[2] https://github.com/ceph/ceph/pull/32027
[3] https://github.com/ceph/rocksdb/pull/42
[4] http://paste.openstack.org/raw/787534/
[5] https://sourceforge.net/p/mingw-w64/bugs/816/
[6] https://sourceforge.net/p/mingw-w64/bugs/527/
_______________________________________________
Dev mailing list -- dev(a)ceph.io <mailto:dev@ceph.io>
To unsubscribe send an email to dev-leave(a)ceph.io
<mailto:dev-leave@ceph.io>
23 Feb
23 Feb
4:48 p.m.
New subject: Windows port
Hi Lucian,
Good news - your fix works; I was able to write 7TB from a Windows 10 VM
to cephfs last night.
I'll carry on testing the Windows client with large workloads using
robocopy.
thanks again for working on this, a cephx "user" option should provide
the core requirements we need for a usable system :)
best regards,
Jake
On 22/02/2021 16:29, Jake Grimmett wrote:
Hi Lucian,
many thanks for this - I'll test and get back to you :)
best regards,
Jake
On 22/02/2021 15:12, Lucian Petrut wrote:
Note: I am working from home until further notice.
For help, contact unixadmin(a)mrc-lmb.cam.ac.uk
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
Hi,
I’ve fixed the crash. It was due to Ceph’s inode emulation, which can
be skipped as Windows supports 64B file identifiers.
The fix is included in the latest msi.
I’ll prepare a few more cephfs related fixes, including the user option.
Thanks,
Lucian
*From: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com>
*Sent: *Friday, February 19, 2021 4:51 PM
*To: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk>; dev(a)ceph.io
<mailto:dev@ceph.io>
*Subject: *RE: Windows port
Hi,
Nevermind, I managed to get a crash after increasing the number of
threads: http://paste.openstack.org/raw/802827/
I’ll try to get a fix as soon as possible. Thanks again for all the info!
Lucian
*From: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com>
*Sent: *Friday, February 19, 2021 3:56 PM
*To: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk>; dev(a)ceph.io
<mailto:dev@ceph.io>
*Subject: *RE: Windows port
Hi,
I couldn’t reproduce the issue yet. Judging by the exception message,
it seems to be related to libcephfs’s inode handling.
The latest msi installer includes debug symbols. Could you please
send us an archive with a crash dump [1] and ideally the msi
installer as well?
Thanks,
Lucian
[1]
https://docs.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dum…
*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk>
*Sent: *Wednesday, February 17, 2021 4:42 PM
*To: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com>;
dev(a)ceph.io <mailto:dev@ceph.io>
*Subject: *Re: Windows port
Hi Lucian,
Many thanks for looking into this, completely understand that it is
just a PoC, but just to let you know that the crash still occurs with
your nightly build.
C:\Users\unixadmin>ceph-dokan -l x -o
ceph_conf_read_file OK
2021-02-17T12:40:59.637GMT Standard Time 1 -1 asok(0xb36ef20)
AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen:
failed to bind the UNIX domain socket to
'C:/ProgramData/ceph/client.admin.4136.asok': (13) Permission denied
ceph_mount OK
ceph_getcwd [/]
../src/include/interval_set.h: In function 'void interval_set<T,
C>::erase(T, T, std::function<bool(T, T)>) [with T = short unsigned
int; C = std::map]' thread 11 time 2021-02-17T13:58:21.387427GMT
Standard Time
../src/include/interval_set.h: 538: FAILED ceph_assert(p->first <= start)
ceph version 15.0.0-21905-g8f11930301
(8f11930301a9ecd5315a8a69a9874eb4428f9366) pacific (rc)
best regards,
Jake
On 17/02/2021 10:56, Lucian Petrut wrote:
Hi,
That crash is indeed a concern. We’ll look into it as soon as
possible.
A small disclaimer: our main focus was RBD, ceph-dokan was more
of a PoC, as mentioned by the docs. Nevertheless, it seems to be
highly demanded so we’re going to invest more time in it.
FWIW, here’s our nightly Ceph MSI build:
https://cloudbase.it/downloads/ceph_v16_0_0_beta.msi
<https://cloudbase.it/downloads/ceph_v16_0_0_beta.msi>. It
provides some important RBD fixes (though you’re probably more
interested in cephfs).
I’ll come with a follow up as soon as those issues are fixed.
Thanks,
Lucian
*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk>
*Sent: *Wednesday, February 17, 2021 12:25 PM
*To: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com>;
dev(a)ceph.io <mailto:dev@ceph.io>
*Subject: *Re: Windows port
Hi Lucian,
Thanks for your reply :)
Our two main requirements are:
1) Security (...it's great news you can fix this.)
2) Stability (which is understandably harder)
Our testing so far has been on a Windows 10 Pro VM (8 cores, 8GB RAM)
We mount one share from a real WIndows 10 system as Z (hardware
RAID, battery backed 400TB)
Then use ceph-dokan to mount /cephfs on the VM as X
We then use robosync to copy data from the Windows server to the
cephfs mount
C:\>robocopy z:\TestDatasets "X:\jog\Kates Data" /np /mt:128
/log:c:/Users/unixadmin/desktop/robolog.txt /E
After copying 5TB from the WIndows server to /cephfs the
ceph-dokan mount crashes.
C:\WINDOWS\system32>ceph-dokan -l x -o
ceph_conf_read_file OK
ceph_mount OK
ceph_getcwd [/]
/home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: In
function 'void interval_set<T, C>::erase(T, T,
std::function<bool(T, T)>) [with T = short unsigned int; C =
std::map]' thread 12 time 2021-02-11T16:56:04.874874GMT Standard Time
/home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: 527:
FAILED ceph_assert(p->first <= start)
ceph version IT-NOTFOUND
(f762f7c3be560c11ea0dd51896c976c45137f5ed) pacific (dev)
restarting robocopy with a lower thread count results in another
crash.
Finally, one other feature request: ceph-dokan reports it's
version as "16.0.0" perhaps a "-V" or "--version"
switch would be
useful?
We are using
https://github.com/dokan-dev/dokany/releases/download/v1.4.1.1000/DokanSetu…
best regards,
Jake
On 15/02/2021 15:09, Lucian Petrut wrote:
Hi,
Thanks for trying it out. Indeed, this option is currently
missing from ceph-dokan but it’s an easy thing to add. We’ll
take care of it as soon as possible, hopefully it will be
included in the Pacific release.
Please let us know if you have any other suggestions.
Regards,
Lucian Petrut
*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk>
*Sent: *Thursday, February 11, 2021 3:00 PM
*To: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com>;
Lucian Petrut
<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>;
dev(a)ceph.io <mailto:dev@ceph.io>
*Subject: *Re: Windows port
Hi,
We have been testing ceph-dokan, based on the guide here:
<https://documentation.suse.com/ses/7/single-html/ses-windows/index.html#windows-cephfs>
And watching
<https://www.youtube.com/watch?v=BWZIwXLcNts&ab_channel=SUSE>
Initial tests on a Windows 10 VM show good write speed -
around 600MB/s,
which is faster than our samba server.
What worries us, is using the "root"
ceph.client.admin.keyring on a
Windows system, as it gives access to the entire cephfs
cluster - which
in our case is 5PB.
I'd really like this to work, as it would let user
administrated Windows
systems that control microscopes to save data directly to
cephfs, so
that we can process the data on our HPC cluster.
I'd normally use cephx, and make a key that allows access to
a directory
off the root.
e.g.
[root@ceph-s1 users]# ceph auth get client.x_lab
exported keyring for client.x_lab
[client.x_lab]
key = xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==
caps mds = "allow r path=/users/, allow rw
path=/users/x_lab"
caps mon = "allow r"
caps osd = "allow class-read object_prefix
rbd_children, allow rw
pool=ec82pool"
The real key works fine on linux, but when we try this key with
ceph-dokan, and specify the ceph directory (x_lab) as a ceph
path, there
is no option to specify the user - is this hard-coded as admin?
Have I just missed something? Or is this a missing feature?
anyhow, ceph-dokan looks like it could be quite useful,
thank you Cloudbase :)
best regards,
Jake
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
On 4/30/20 1:54 PM, Lucian Petrut wrote:
Note: I am working from home until further notice.
For help, contactunixadmin(a)mrc-lmb.cam.ac.uk
<mailto:unixadmin@mrc-lmb.cam.ac.uk>
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
Note: I am working from home until further notice.
For help, contactunixadmin(a)mrc-lmb.cam.ac.uk <mailto:unixadmin@mrc-lmb.cam.ac.uk>
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
_______________________________________________
Dev mailing list --dev(a)ceph.io
To unsubscribe send an email todev-leave(a)ceph.io
Note: I am working from home
until further notice.
For help, contactunixadmin(a)mrc-lmb.cam.ac.uk
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io Hi,
We’ve just pushed the final part of the Windows PR
series[1], allowing
RBD images as well as CephFS to be mounted on
Windows.
There’s a comprehensive guide[2], describing the build,
installation,
configuration and usage steps.
2 out of 12 PRs have been merged already, we look forward
to merging the
others as well.
Lucian Petrut
Cloudbase Solutions
[1] https://github.com/ceph/ceph/pull/34859
<https://github.com/ceph/ceph/pull/34859>
[2]
https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst
<https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst>
*From: *Lucian Petrut
<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>
*Sent: *Monday, December 16, 2019 10:12 AM
*To: *dev(a)ceph.io <mailto:dev@ceph.io>
*Subject: *Windows port
Hi,
We're happy to announce that a couple of weeks ago, we've
submitted a
few Github pull requests[1][2][3] adding initial
Windows
support. A big
thank you to the people that have already
reviewed the patches.
To bring some context about the scope and current status of
our work:
we're mostly targeting the client side,
allowing Windows
hosts to
consume rados, rbd and cephfs resources.
We have Windows binaries capable of writing to rados
pools[4]. We're
using mingw to build the ceph components, mostly
due to the
fact that it
requires the minimum amount of changes to cross
compile
ceph for
Windows. However, we're soon going to switch
to MSVC/Clang
due to mingw
limitations and long standing bugs[5][6]. Porting
the unit
tests is also
something that we're currently working on.
The next step will be implementing a virtual miniport
driver so that RBD
volumes can be exposed to Windows hosts and
Hyper-V guests.
We're hoping
to leverage librbd as much as possible as part of
a daemon
that will
communicate with the driver. We're also
aiming at cephfs
and considering
using Dokan, which is FUSE compatible.
Merging the open PRs would allow us to move forward,
focusing on the
drivers and avoiding rebase issues. Any help on
that is
greatly appreciated.
Last but not least, I'd like to thank Suse, who's
sponsoring this
effort!
Lucian Petrut
Cloudbase Solutions
[1] https://github.com/ceph/ceph/pull/31981
[2] https://github.com/ceph/ceph/pull/32027
[3] https://github.com/ceph/rocksdb/pull/42
[4] http://paste.openstack.org/raw/787534/
[5] https://sourceforge.net/p/mingw-w64/bugs/816/
[6] https://sourceforge.net/p/mingw-w64/bugs/527/
_______________________________________________
Dev mailing list -- dev(a)ceph.io <mailto:dev@ceph.io>
To unsubscribe send an email to dev-leave(a)ceph.io
<mailto:dev-leave@ceph.io>
6:49 p.m.
New subject: Windows port
Hi,
That’s great, thanks for the confirmation.
I hope I’ll get to push a larger change by the end of the week, covering the user option
as well as a few other fixes.
Regards,
Lucian
From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>
Sent: Tuesday, February 23, 2021 11:48 AM
To: dev@ceph.io<mailto:dev@ceph.io>; Lucian
Petrut<mailto:lpetrut@cloudbasesolutions.com>
Subject: Re: Windows port
Hi Lucian,
Good news - your fix works; I was able to write 7TB from a Windows 10 VM to cephfs last
night.
I'll carry on testing the Windows client with large workloads using robocopy.
thanks again for working on this, a cephx "user" option should provide the core
requirements we need for a usable system :)
best regards,
Jake
On 22/02/2021 16:29, Jake Grimmett wrote:
Hi Lucian,
many thanks for this - I'll test and get back to you :)
best regards,
Jake
On 22/02/2021 15:12, Lucian Petrut wrote:
Hi,
I’ve fixed the crash. It was due to Ceph’s inode emulation, which can be skipped as
Windows supports 64B file identifiers.
The fix is included in the latest msi.
I’ll prepare a few more cephfs related fixes, including the user option.
Thanks,
Lucian
From: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>
Sent: Friday, February 19, 2021 4:51 PM
To: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: RE: Windows port
Hi,
Nevermind, I managed to get a crash after increasing the number of threads:
http://paste.openstack.org/raw/802827/
I’ll try to get a fix as soon as possible. Thanks again for all the info!
Lucian
From: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>
Sent: Friday, February 19, 2021 3:56 PM
To: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: RE: Windows port
Hi,
I couldn’t reproduce the issue yet. Judging by the exception message, it seems to be
related to libcephfs’s inode handling.
The latest msi installer includes debug symbols. Could you please send us an archive with
a crash dump [1] and ideally the msi installer as well?
Thanks,
Lucian
[1] https://docs.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dum…
From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>
Sent: Wednesday, February 17, 2021 4:42 PM
To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: Re: Windows port
Hi Lucian,
Many thanks for looking into this, completely understand that it is just a PoC, but just
to let you know that the crash still occurs with your nightly build.
C:\Users\unixadmin>ceph-dokan -l x -o
ceph_conf_read_file OK
2021-02-17T12:40:59.637GMT Standard Time 1 -1 asok(0xb36ef20) AdminSocketConfigObs::init:
failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to
'C:/ProgramData/ceph/client.admin.4136.asok': (13) Permission denied
ceph_mount OK
ceph_getcwd [/]
../src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T,
std::function<bool(T, T)>) [with T = short unsigned int; C = std::map]' thread
11 time 2021-02-17T13:58:21.387427GMT Standard Time
../src/include/interval_set.h: 538: FAILED ceph_assert(p->first <= start)
ceph version 15.0.0-21905-g8f11930301 (8f11930301a9ecd5315a8a69a9874eb4428f9366) pacific
(rc)
best regards,
Jake
On 17/02/2021 10:56, Lucian Petrut wrote:
Hi,
That crash is indeed a concern. We’ll look into it as soon as possible.
A small disclaimer: our main focus was RBD, ceph-dokan was more of a PoC, as mentioned by
the docs. Nevertheless, it seems to be highly demanded so we’re going to invest more time
in it.
FWIW, here’s our nightly Ceph MSI build:
https://cloudbase.it/downloads/ceph_v16_0_0_beta.msi. It provides some important RBD fixes
(though you’re probably more interested in cephfs).
I’ll come with a follow up as soon as those issues are fixed.
Thanks,
Lucian
From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>
Sent: Wednesday, February 17, 2021 12:25 PM
To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: Re: Windows port
Hi Lucian,
Thanks for your reply :)
Our two main requirements are:
1) Security (...it's great news you can fix this.)
2) Stability (which is understandably harder)
Our testing so far has been on a Windows 10 Pro VM (8 cores, 8GB RAM)
We mount one share from a real WIndows 10 system as Z (hardware RAID, battery backed
400TB)
Then use ceph-dokan to mount /cephfs on the VM as X
We then use robosync to copy data from the Windows server to the cephfs mount
C:\>robocopy z:\TestDatasets "X:\jog\Kates Data" /np /mt:128
/log:c:/Users/unixadmin/desktop/robolog.txt /E
After copying 5TB from the WIndows server to /cephfs the ceph-dokan mount crashes.
C:\WINDOWS\system32>ceph-dokan -l x -o
ceph_conf_read_file OK
ceph_mount OK
ceph_getcwd [/]
/home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: In function 'void
interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = short
unsigned int; C = std::map]' thread 12 time 2021-02-11T16:56:04.874874GMT Standard
Time
/home/abuild/rpmbuild/BUILD/ceph/src/include/interval_set.h: 527: FAILED
ceph_assert(p->first <= start)
ceph version IT-NOTFOUND (f762f7c3be560c11ea0dd51896c976c45137f5ed) pacific (dev)
restarting robocopy with a lower thread count results in another crash.
Finally, one other feature request: ceph-dokan reports it's version as
"16.0.0" perhaps a "-V" or "--version" switch would be
useful?
We are using
https://github.com/dokan-dev/dokany/releases/download/v1.4.1.1000/DokanSetu…
best regards,
Jake
On 15/02/2021 15:09, Lucian Petrut wrote:
Hi,
Thanks for trying it out. Indeed, this option is currently missing from ceph-dokan but
it’s an easy thing to add. We’ll take care of it as soon as possible, hopefully it will be
included in the Pacific release.
Please let us know if you have any other suggestions.
Regards,
Lucian Petrut
From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>
Sent: Thursday, February 11, 2021 3:00 PM
To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>; Lucian
Petrut<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: Re: Windows port
Hi,
We have been testing ceph-dokan, based on the guide here:
<https://documentation.suse.com/ses/7/single-html/ses-windows/index.html#windows-cephfs>
And watching <https://www.youtube.com/watch?v=BWZIwXLcNts&ab_channel=SUSE>
Initial tests on a Windows 10 VM show good write speed - around 600MB/s,
which is faster than our samba server.
What worries us, is using the "root" ceph.client.admin.keyring on a
Windows system, as it gives access to the entire cephfs cluster - which
in our case is 5PB.
I'd really like this to work, as it would let user administrated Windows
systems that control microscopes to save data directly to cephfs, so
that we can process the data on our HPC cluster.
I'd normally use cephx, and make a key that allows access to a directory
off the root.
e.g.
[root@ceph-s1 users]# ceph auth get client.x_lab
exported keyring for client.x_lab
[client.x_lab]
key = xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==
caps mds = "allow r path=/users/, allow rw path=/users/x_lab"
caps mon = "allow r"
caps osd = "allow class-read object_prefix rbd_children, allow rw
pool=ec82pool"
The real key works fine on linux, but when we try this key with
ceph-dokan, and specify the ceph directory (x_lab) as a ceph path, there
is no option to specify the user - is this hard-coded as admin?
Have I just missed something? Or is this a missing feature?
anyhow, ceph-dokan looks like it could be quite useful,
thank you Cloudbase :)
best regards,
Jake
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
On 4/30/20 1:54 PM, Lucian Petrut wrote:
Hi,
We’ve just pushed the final part of the Windows PR series[1], allowing
RBD images as well as CephFS to be mounted on Windows.
There’s a comprehensive guide[2], describing the build, installation,
configuration and usage steps.
2 out of 12 PRs have been merged already, we look forward to merging the
others as well.
Lucian Petrut
Cloudbase Solutions
[1] https://github.com/ceph/ceph/pull/34859
<https://github.com/ceph/ceph/pull/34859>
[2]
https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst
<https://github.com/petrutlucian94/ceph/blob/windows.12/README.windows.rst>
*From: *Lucian Petrut
<mailto:/O=CLOUDBASE/OU=EXCHANGE%20ADMINISTRATIVE%20GROUP%20(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=LUCIAN%20PETRUT77C>
*Sent: *Monday, December 16, 2019 10:12 AM
*To: *dev(a)ceph.io <mailto:dev@ceph.io>
*Subject: *Windows port
Hi,
We're happy to announce that a couple of weeks ago, we've submitted a
few Github pull requests[1][2][3] adding initial Windows support. A big
thank you to the people that have already reviewed the patches.
To bring some context about the scope and current status of our work:
we're mostly targeting the client side, allowing Windows hosts to
consume rados, rbd and cephfs resources.
We have Windows binaries capable of writing to rados pools[4]. We're
using mingw to build the ceph components, mostly due to the fact that it
requires the minimum amount of changes to cross compile ceph for
Windows. However, we're soon going to switch to MSVC/Clang due to mingw
limitations and long standing bugs[5][6]. Porting the unit tests is also
something that we're currently working on.
The next step will be implementing a virtual miniport driver so that RBD
volumes can be exposed to Windows hosts and Hyper-V guests. We're hoping
to leverage librbd as much as possible as part of a daemon that will
communicate with the driver. We're also aiming at cephfs and considering
using Dokan, which is FUSE compatible.
Merging the open PRs would allow us to move forward, focusing on the
drivers and avoiding rebase issues. Any help on that is greatly appreciated.
Last but not least, I'd like to thank Suse, who's sponsoring this effort!
Lucian Petrut
Cloudbase Solutions
[1] https://github.com/ceph/ceph/pull/31981
[2] https://github.com/ceph/ceph/pull/32027
[3] https://github.com/ceph/rocksdb/pull/42
[4] http://paste.openstack.org/raw/787534/
[5] https://sourceforge.net/p/mingw-w64/bugs/816/
[6] https://sourceforge.net/p/mingw-w64/bugs/527/
_______________________________________________
Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io>
To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io>
Note: I am working from home until further notice.
For help, contact unixadmin@mrc-lmb.cam.ac.uk<mailto:unixadmin@mrc-lmb.cam.ac.uk>
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
Note: I am working from home until further notice.
For help, contact unixadmin@mrc-lmb.cam.ac.uk<mailto:unixadmin@mrc-lmb.cam.ac.uk>
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
_______________________________________________
Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io>
To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io>
Note: I am working from home until further notice.
For help, contact unixadmin@mrc-lmb.cam.ac.uk<mailto:unixadmin@mrc-lmb.cam.ac.uk>
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
_______________________________________________
Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io>
To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io>
Note: I am working from home until further notice.
For help, contact unixadmin@mrc-lmb.cam.ac.uk<mailto:unixadmin@mrc-lmb.cam.ac.uk>
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
25 Feb
25 Feb
4:37 p.m.
New subject: Windows port
Hi Lucian,
So I've now copied 32TB of data from a Windows server to our test ceph
cluster, without any crashes.
Average speed over one 16TB copy was 468MB/s, i.e. just under 11 hours
to transfer 16TB, with a data set that has 1,144,495 files.
The transfer was from Windows 10 server (hardware RAID) > Windows 10 VM
(with the cephfs driver installed) > cephfs test cluster.
If you can get the user option working, I'll put the driver on a
physical Windows 10 system, and see how fast a direct transfer is.
One other thing that would be useful, is over-quota error handling.
If I try writing and exceed the quota, the mount just hangs.
If I increase the quota, the mount recovers, but it would be nice if we
had a quota error in Windows.
Test cluster consists of 6 Dell C2100 nodes, bought in 2012.
Each has 10 x 900GB 10k HDD, we use EC 4+2 for the cephfs pool, metadata
is on 3 x NVMe. Dual Xeon X5650, 12 threads, 96GB RAM, 2x10GB bond.
Ceph 15.2.8, Scientific Linux 7.9.
best regards,
Jake
On 2/23/21 11:49 AM, Lucian Petrut wrote:
Hi,
That’s great, thanks for the confirmation.
I hope I’ll get to push a larger change by the end of the week, covering
the user option as well as a few other fixes.
Regards,
Lucian
*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk>
*Sent: *Tuesday, February 23, 2021 11:48 AM
*To: *dev(a)ceph.io <mailto:dev@ceph.io>; Lucian Petrut
<mailto:lpetrut@cloudbasesolutions.com>
*Subject: *Re: Windows port
Hi Lucian,
Good news - your fix works; I was able to write 7TB from a Windows 10 VM
to cephfs last night.
I'll carry on testing the Windows client with large workloads using
robocopy.
thanks again for working on this, a cephx "user" option should provide
the core requirements we need for a usable system :)
best regards,
Jake
Note: I am working from home until further notice.
For help, contact unixadmin(a)mrc-lmb.cam.ac.uk
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
5:51 p.m.
New subject: Windows port
Hi,
That’s great. I’ll push the user option setting ASAP.
About the quotas, you mean Windows quotas or Ceph quotas?
Regars,
Lucian
From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>
Sent: Thursday, February 25, 2021 11:37 AM
To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: Re: Windows port
Hi Lucian,
So I've now copied 32TB of data from a Windows server to our test ceph
cluster, without any crashes.
Average speed over one 16TB copy was 468MB/s, i.e. just under 11 hours
to transfer 16TB, with a data set that has 1,144,495 files.
The transfer was from Windows 10 server (hardware RAID) > Windows 10 VM
(with the cephfs driver installed) > cephfs test cluster.
If you can get the user option working, I'll put the driver on a
physical Windows 10 system, and see how fast a direct transfer is.
One other thing that would be useful, is over-quota error handling.
If I try writing and exceed the quota, the mount just hangs.
If I increase the quota, the mount recovers, but it would be nice if we
had a quota error in Windows.
Test cluster consists of 6 Dell C2100 nodes, bought in 2012.
Each has 10 x 900GB 10k HDD, we use EC 4+2 for the cephfs pool, metadata
is on 3 x NVMe. Dual Xeon X5650, 12 threads, 96GB RAM, 2x10GB bond.
Ceph 15.2.8, Scientific Linux 7.9.
best regards,
Jake
On 2/23/21 11:49 AM, Lucian Petrut wrote:
Hi,
That’s great, thanks for the confirmation.
I hope I’ll get to push a larger change by the end of the week, covering
the user option as well as a few other fixes.
Regards,
Lucian
*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk>
*Sent: *Tuesday, February 23, 2021 11:48 AM
*To: *dev(a)ceph.io <mailto:dev@ceph.io>; Lucian Petrut
<mailto:lpetrut@cloudbasesolutions.com>
*Subject: *Re: Windows port
Hi Lucian,
Good news - your fix works; I was able to write 7TB from a Windows 10 VM
to cephfs last night.
I'll carry on testing the Windows client with large workloads using
robocopy.
thanks again for working on this, a cephx "user" option should provide
the core requirements we need for a usable system :)
best regards,
Jake
Note: I am working from home until further notice.
For help, contact unixadmin(a)mrc-lmb.cam.ac.uk
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
7:21 p.m.
New subject: Windows port
Hi Lucian,
Exceeding the cephfs quota seems to "upset" Windows:
e.g. on the cephfs system
# setfattr -n ceph.quota.max_bytes -v 1000 /ctestfs/quota
And then on the Windows system
PS Z:\Jake_tests> copy .\4GB_file X:\quota\
(Shell hangs)
back on the cephfs system
# setfattr -n ceph.quota.max_bytes -v 1000000000 /ctestfs/quota
Shell (eventually) returns on Windows, after much CTRL+C
I can then repeat the "copy .\4GB_file X:\quota\" successfully
It would be ideal if Windows gave an error, such as "no space" or "out
of quota" rather than hanging...
thanks again for your work :)
Jake
On 2/25/21 10:51 AM, Lucian Petrut wrote:
Hi,
That’s great. I’ll push the user option setting ASAP.
About the quotas, you mean Windows quotas or Ceph quotas?
Regars,
Lucian
*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk>
*Sent: *Thursday, February 25, 2021 11:37 AM
*To: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com>; dev(a)ceph.io
<mailto:dev@ceph.io>
*Subject: *Re: Windows port
Hi Lucian,
So I've now copied 32TB of data from a Windows server to our test ceph
cluster, without any crashes.
Average speed over one 16TB copy was 468MB/s, i.e. just under 11 hours
to transfer 16TB, with a data set that has 1,144,495 files.
The transfer was from Windows 10 server (hardware RAID) > Windows 10 VM
(with the cephfs driver installed) > cephfs test cluster.
If you can get the user option working, I'll put the driver on a
physical Windows 10 system, and see how fast a direct transfer is.
One other thing that would be useful, is over-quota error handling.
If I try writing and exceed the quota, the mount just hangs.
If I increase the quota, the mount recovers, but it would be nice if we
had a quota error in Windows.
Test cluster consists of 6 Dell C2100 nodes, bought in 2012.
Each has 10 x 900GB 10k HDD, we use EC 4+2 for the cephfs pool, metadata
is on 3 x NVMe. Dual Xeon X5650, 12 threads, 96GB RAM, 2x10GB bond.
Ceph 15.2.8, Scientific Linux 7.9.
best regards,
Jake
On 2/23/21 11:49 AM, Lucian Petrut wrote:
Note: I am working from home until further notice.
For help, contact unixadmin(a)mrc-lmb.cam.ac.uk
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
Hi,
That’s great, thanks for the confirmation.
I hope I’ll get to push a larger change by the end of the week, covering
the user option as well as a few other fixes.
Regards,
Lucian
*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk
<mailto:jog@mrc-lmb.cam.ac.uk>>
*Sent: *Tuesday, February 23, 2021 11:48 AM
*To: *dev(a)ceph.io <mailto:dev@ceph.io <mailto:dev@ceph.io>>; Lucian
Petrut
<mailto:lpetrut@cloudbasesolutions.com
<mailto:lpetrut@cloudbasesolutions.com>>
*Subject: *Re: Windows port
Hi Lucian,
Good news - your fix works; I was able to write 7TB from a Windows 10 VM
to cephfs last night.
I'll carry on testing the Windows client with large workloads using
robocopy.
thanks again for working on this, a cephx "user" option should provide
the core requirements we need for a usable system :)
best regards,
Jake
Note: I am working from home until further notice.
For help, contact unixadmin(a)mrc-lmb.cam.ac.uk
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
7:27 p.m.
New subject: Windows port
Hi,
Makes sense. I’ll look into it ASAP, thanks for bringing it up!
Lucian
From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>
Sent: Thursday, February 25, 2021 2:21 PM
To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: Re: Windows port
Hi Lucian,
Exceeding the cephfs quota seems to "upset" Windows:
e.g. on the cephfs system
# setfattr -n ceph.quota.max_bytes -v 1000 /ctestfs/quota
And then on the Windows system
PS Z:\Jake_tests> copy .\4GB_file X:\quota\
(Shell hangs)
back on the cephfs system
# setfattr -n ceph.quota.max_bytes -v 1000000000 /ctestfs/quota
Shell (eventually) returns on Windows, after much CTRL+C
I can then repeat the "copy .\4GB_file X:\quota\" successfully
It would be ideal if Windows gave an error, such as "no space" or "out
of quota" rather than hanging...
thanks again for your work :)
Jake
On 2/25/21 10:51 AM, Lucian Petrut wrote:
Hi,
That’s great. I’ll push the user option setting ASAP.
About the quotas, you mean Windows quotas or Ceph quotas?
Regars,
Lucian
*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk>
*Sent: *Thursday, February 25, 2021 11:37 AM
*To: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com>; dev(a)ceph.io
<mailto:dev@ceph.io>
*Subject: *Re: Windows port
Hi Lucian,
So I've now copied 32TB of data from a Windows server to our test ceph
cluster, without any crashes.
Average speed over one 16TB copy was 468MB/s, i.e. just under 11 hours
to transfer 16TB, with a data set that has 1,144,495 files.
The transfer was from Windows 10 server (hardware RAID) > Windows 10 VM
(with the cephfs driver installed) > cephfs test cluster.
If you can get the user option working, I'll put the driver on a
physical Windows 10 system, and see how fast a direct transfer is.
One other thing that would be useful, is over-quota error handling.
If I try writing and exceed the quota, the mount just hangs.
If I increase the quota, the mount recovers, but it would be nice if we
had a quota error in Windows.
Test cluster consists of 6 Dell C2100 nodes, bought in 2012.
Each has 10 x 900GB 10k HDD, we use EC 4+2 for the cephfs pool, metadata
is on 3 x NVMe. Dual Xeon X5650, 12 threads, 96GB RAM, 2x10GB bond.
Ceph 15.2.8, Scientific Linux 7.9.
best regards,
Jake
On 2/23/21 11:49 AM, Lucian Petrut wrote:
Note: I am working from home until further notice.
For help, contact unixadmin(a)mrc-lmb.cam.ac.uk
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
Hi,
That’s great, thanks for the confirmation.
I hope I’ll get to push a larger change by the end of the week, covering
the user option as well as a few other fixes.
Regards,
Lucian
*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk
<mailto:jog@mrc-lmb.cam.ac.uk>>
*Sent: *Tuesday, February 23, 2021 11:48 AM
*To: *dev(a)ceph.io <mailto:dev@ceph.io <mailto:dev@ceph.io>>; Lucian
Petrut
<mailto:lpetrut@cloudbasesolutions.com
<mailto:lpetrut@cloudbasesolutions.com>>
*Subject: *Re: Windows port
Hi Lucian,
Good news - your fix works; I was able to write 7TB from a Windows 10 VM
to cephfs last night.
I'll carry on testing the Windows client with large workloads using
robocopy.
thanks again for working on this, a cephx "user" option should provide
the core requirements we need for a usable system :)
best regards,
Jake
Note: I am working from home until further notice.
For help, contact unixadmin(a)mrc-lmb.cam.ac.uk
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
26 Feb
26 Feb
11:48 p.m.
New subject: Windows port
Hi,
The quotas work fine on my env:
PS x:\quota> echo exceeding_quota > test
out-file : Not enough quota is available to process this command.
ceph-dokan: WinCephWriteFile /quota/test: ceph_write failed. Error: -311. Offset:
0 Buffer length: 36
I’m wondering if this has something to do with the ceph cluster version. Are you using
Ceph Octopus?
About the –user/--id option, I’ve just pushed a larger change that covers this, also
improving the CLI syntax and logging. Please try out the latest MSI.
In case you’ll need to change the ceph-dokan log level, please note that we’re now using
the “client” ceph log subsystem.
Thanks,
Lucian
From: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>
Sent: Thursday, February 25, 2021 2:27 PM
To: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: RE: Windows port
Hi,
Makes sense. I’ll look into it ASAP, thanks for bringing it up!
Lucian
From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>
Sent: Thursday, February 25, 2021 2:21 PM
To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: Re: Windows port
Hi Lucian,
Exceeding the cephfs quota seems to "upset" Windows:
e.g. on the cephfs system
# setfattr -n ceph.quota.max_bytes -v 1000 /ctestfs/quota
And then on the Windows system
PS Z:\Jake_tests> copy .\4GB_file X:\quota\
(Shell hangs)
back on the cephfs system
# setfattr -n ceph.quota.max_bytes -v 1000000000 /ctestfs/quota
Shell (eventually) returns on Windows, after much CTRL+C
I can then repeat the "copy .\4GB_file X:\quota\" successfully
It would be ideal if Windows gave an error, such as "no space" or "out
of quota" rather than hanging...
thanks again for your work :)
Jake
On 2/25/21 10:51 AM, Lucian Petrut wrote:
Hi,
That’s great. I’ll push the user option setting ASAP.
About the quotas, you mean Windows quotas or Ceph quotas?
Regars,
Lucian
*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk>
*Sent: *Thursday, February 25, 2021 11:37 AM
*To: *Lucian Petrut <mailto:lpetrut@cloudbasesolutions.com>; dev(a)ceph.io
<mailto:dev@ceph.io>
*Subject: *Re: Windows port
Hi Lucian,
So I've now copied 32TB of data from a Windows server to our test ceph
cluster, without any crashes.
Average speed over one 16TB copy was 468MB/s, i.e. just under 11 hours
to transfer 16TB, with a data set that has 1,144,495 files.
The transfer was from Windows 10 server (hardware RAID) > Windows 10 VM
(with the cephfs driver installed) > cephfs test cluster.
If you can get the user option working, I'll put the driver on a
physical Windows 10 system, and see how fast a direct transfer is.
One other thing that would be useful, is over-quota error handling.
If I try writing and exceed the quota, the mount just hangs.
If I increase the quota, the mount recovers, but it would be nice if we
had a quota error in Windows.
Test cluster consists of 6 Dell C2100 nodes, bought in 2012.
Each has 10 x 900GB 10k HDD, we use EC 4+2 for the cephfs pool, metadata
is on 3 x NVMe. Dual Xeon X5650, 12 threads, 96GB RAM, 2x10GB bond.
Ceph 15.2.8, Scientific Linux 7.9.
best regards,
Jake
On 2/23/21 11:49 AM, Lucian Petrut wrote:
Note: I am working from home until further notice.
For help, contact unixadmin(a)mrc-lmb.cam.ac.uk
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
Hi,
That’s great, thanks for the confirmation.
I hope I’ll get to push a larger change by the end of the week, covering
the user option as well as a few other fixes.
Regards,
Lucian
*From: *Jake Grimmett <mailto:jog@mrc-lmb.cam.ac.uk
<mailto:jog@mrc-lmb.cam.ac.uk>>
*Sent: *Tuesday, February 23, 2021 11:48 AM
*To: *dev(a)ceph.io <mailto:dev@ceph.io <mailto:dev@ceph.io>>; Lucian
Petrut
<mailto:lpetrut@cloudbasesolutions.com
<mailto:lpetrut@cloudbasesolutions.com>>
*Subject: *Re: Windows port
Hi Lucian,
Good news - your fix works; I was able to write 7TB from a Windows 10 VM
to cephfs last night.
I'll carry on testing the Windows client with large workloads using
robocopy.
thanks again for working on this, a cephx "user" option should provide
the core requirements we need for a usable system :)
best regards,
Jake
Note: I am working from home until further notice.
For help, contact unixadmin(a)mrc-lmb.cam.ac.uk
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
2 Mar
2 Mar
7:22 p.m.
New subject: Windows port
Hi Lucian,
Many thanks for adding user level authentication, it works - yippee!
The "out of quota" error on Windows worked as expected when re-tested :)
The new ceph-dokan paramerers confused me until I re-read the cephx docs.
--id ID set ID portion of my name
--name/-n TYPE.ID set name
I'm sure you'll provide examples, but here is my usage / config
PS C:\> ceph-dokan -l x --conf C:/ProgramData/ceph/ceph.testauth.conf
--name client.test -x /testauthdir
(is equivalent to...)
PS C:\> ceph-dokan -l x --conf C:/ProgramData/ceph/ceph.testauth.conf
--id test -x /testauthdir
where:
PS C:\ProgramData\ceph> type .\ceph.testauth.conf
[global]
log to stderr = true
run dir = C:/ProgramData/ceph
crash dir = C:/ProgramData/ceph
[client]
keyring = C:/ProgramData/ceph/test.keyring
log file = C:/ProgramData/ceph/$name.$pid.log
admin socket = C:/ProgramData/ceph/$name.$pid.asok
[global]
; Specify IP addresses for monitor nodes as in the following
example: ;
mon host = [v2:10.1.2.39:3300,v1:10.1.2.39:6789]
[v2:10.1.0.96:3300,v1:10.1.0.96:6789] [v2:10.1.0.98:3300,v1:1.1.0.98:6789]
PS C:\ProgramData\ceph> type .\test.keyring
[client.test]
key = AQBFFSVgvypAIBAA8CHRkwdVJ6i9u+9yKiq1hQ==
caps mds = "allow r path=/, allow rw path=/testauthdir"
caps mon = "allow r"
caps osd = "allow class-read object_prefix rbd_children, allow
rw pool=ec42pool"
test.keyring is the output of "ceph auth get client.test" on the ceph server
I'll now test this more thoroughly,
best regards, and thanks again for doing this, it's really useful :)
Jake
Note: I am working from home until further notice.
For help, contact unixadmin(a)mrc-lmb.cam.ac.uk
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
7:38 p.m.
New subject: Windows port
Hi,
Thanks for providing valuable feedback. Please let us know if anything else comes up.
Regards,
Lucian
From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>
Sent: Tuesday, March 2, 2021 2:22 PM
To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: Re: Windows port
Hi Lucian,
Many thanks for adding user level authentication, it works - yippee!
The "out of quota" error on Windows worked as expected when re-tested :)
The new ceph-dokan paramerers confused me until I re-read the cephx docs.
--id ID set ID portion of my name
--name/-n TYPE.ID set name
I'm sure you'll provide examples, but here is my usage / config
PS C:\> ceph-dokan -l x --conf C:/ProgramData/ceph/ceph.testauth.conf
--name client.test -x /testauthdir
(is equivalent to...)
PS C:\> ceph-dokan -l x --conf C:/ProgramData/ceph/ceph.testauth.conf
--id test -x /testauthdir
where:
PS C:\ProgramData\ceph> type .\ceph.testauth.conf
[global]
log to stderr = true
run dir = C:/ProgramData/ceph
crash dir = C:/ProgramData/ceph
[client]
keyring = C:/ProgramData/ceph/test.keyring
log file = C:/ProgramData/ceph/$name.$pid.log
admin socket = C:/ProgramData/ceph/$name.$pid.asok
[global]
; Specify IP addresses for monitor nodes as in the following
example: ;
mon host = [v2:10.1.2.39:3300,v1:10.1.2.39:6789]
[v2:10.1.0.96:3300,v1:10.1.0.96:6789] [v2:10.1.0.98:3300,v1:1.1.0.98:6789]
PS C:\ProgramData\ceph> type .\test.keyring
[client.test]
key = AQBFFSVgvypAIBAA8CHRkwdVJ6i9u+9yKiq1hQ==
caps mds = "allow r path=/, allow rw path=/testauthdir"
caps mon = "allow r"
caps osd = "allow class-read object_prefix rbd_children, allow
rw pool=ec42pool"
test.keyring is the output of "ceph auth get client.test" on the ceph server
I'll now test this more thoroughly,
best regards, and thanks again for doing this, it's really useful :)
Jake
Note: I am working from home until further notice.
For help, contact unixadmin(a)mrc-lmb.cam.ac.uk
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
5 Mar
5 Mar
6:55 p.m.
New subject: Windows port
Hi Lucian
Driver is excellent, however I'm not seeing cephfs recursive dir stats
being updated.
For example, 24 hours after copying 16TB to cephfs:
# ls -lh
drwxrwxrwx 1 root root 1.9T Mar 2 14:47 testauthdir
# ls -lR testauthdir >/dev/null
# ls -lh
drwxrwxrwx 1 root root 16TB Mar 5 11:38 testauthdir
In earlier tests I think recursive dir stats was working, though it
might be related to us upgrading ceph from Nautilus to Octopus (which we
did to try and stop the earlier crashes)
Our current ceph version is 15.2.8
We use recursive stats to efficiently replicate the filesystem to a
second cephfs cluster; it's a very useful feature.
The new cephfs-mirror tool in Pacific might rely on this feature too.
thanks again for your work on this :)
Jake
On 3/2/21 12:38 PM, Lucian Petrut wrote:
Hi,
Thanks for providing valuable feedback. Please let us know if anything
else comes up.
Regards,
Lucian
Note: I am working from home until further notice.
For help, contact unixadmin(a)mrc-lmb.cam.ac.uk
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
8:38 p.m.
New subject: Windows port
Hi,
Interesting. AFAIK the rbytes tracking happens on the MDS side but I’ll look into it.
Btw, just so you know, I did include a few unrelated fixes in the latest MSI.
Regards,
Lucian
From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>
Sent: Friday, March 5, 2021 1:55 PM
To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: Re: Windows port
Hi Lucian
Driver is excellent, however I'm not seeing cephfs recursive dir stats
being updated.
For example, 24 hours after copying 16TB to cephfs:
# ls -lh
drwxrwxrwx 1 root root 1.9T Mar 2 14:47 testauthdir
# ls -lR testauthdir >/dev/null
# ls -lh
drwxrwxrwx 1 root root 16TB Mar 5 11:38 testauthdir
In earlier tests I think recursive dir stats was working, though it
might be related to us upgrading ceph from Nautilus to Octopus (which we
did to try and stop the earlier crashes)
Our current ceph version is 15.2.8
We use recursive stats to efficiently replicate the filesystem to a
second cephfs cluster; it's a very useful feature.
The new cephfs-mirror tool in Pacific might rely on this feature too.
thanks again for your work on this :)
Jake
On 3/2/21 12:38 PM, Lucian Petrut wrote:
Hi,
Thanks for providing valuable feedback. Please let us know if anything
else comes up.
Regards,
Lucian
Note: I am working from home until further notice.
For help, contact unixadmin(a)mrc-lmb.cam.ac.uk
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
9:59 p.m.
New subject: Windows port
Hi,
I gave it a quick try, the recursive stats do get updated although it might take a while
for that to happen. I think MDS load can also have an impact on this.
Also, you might want to check if mds_dirstat_min_interval is set, that would delay the
recursive stat propagation.
Regards,
Lucian
From: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>
Sent: Friday, March 5, 2021 3:39 PM
To: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: RE: Windows port
Hi,
Interesting. AFAIK the rbytes tracking happens on the MDS side but I’ll look into it.
Btw, just so you know, I did include a few unrelated fixes in the latest MSI.
Regards,
Lucian
From: Jake Grimmett<mailto:jog@mrc-lmb.cam.ac.uk>
Sent: Friday, March 5, 2021 1:55 PM
To: Lucian Petrut<mailto:lpetrut@cloudbasesolutions.com>;
dev@ceph.io<mailto:dev@ceph.io>
Subject: Re: Windows port
Hi Lucian
Driver is excellent, however I'm not seeing cephfs recursive dir stats
being updated.
For example, 24 hours after copying 16TB to cephfs:
# ls -lh
drwxrwxrwx 1 root root 1.9T Mar 2 14:47 testauthdir
# ls -lR testauthdir >/dev/null
# ls -lh
drwxrwxrwx 1 root root 16TB Mar 5 11:38 testauthdir
In earlier tests I think recursive dir stats was working, though it
might be related to us upgrading ceph from Nautilus to Octopus (which we
did to try and stop the earlier crashes)
Our current ceph version is 15.2.8
We use recursive stats to efficiently replicate the filesystem to a
second cephfs cluster; it's a very useful feature.
The new cephfs-mirror tool in Pacific might rely on this feature too.
thanks again for your work on this :)
Jake
On 3/2/21 12:38 PM, Lucian Petrut wrote:
Hi,
Thanks for providing valuable feedback. Please let us know if anything
else comes up.
Regards,
Lucian
Note: I am working from home until further notice.
For help, contact unixadmin(a)mrc-lmb.cam.ac.uk
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
11:44 p.m.
New subject: Windows port
On Fri, Mar 5, 2021 at 9:00 AM Lucian Petrut
<lpetrut(a)cloudbasesolutions.com> wrote:
Hi,
I gave it a quick try, the recursive stats do get updated although it might take a while
for that to happen. I think MDS load can also have an impact on this.
Also, you might want to check if mds_dirstat_min_interval is set, that would delay the
recursive stat propagation.
Having the rstats changes reflected on the client is a general issue
with the client/mds protocol, as there is no explicit forced update or
way to ensure that the client's view is up to date. I don't think
this is anything related to the windows dokan client.
932
days inactive
954
days old
21 comments
3 participants
participants (3)
-
Jake Grimmett -
Lucian Petrut -
Sage Weil