19 Jun
2023
19 Jun
'23
2:47 p.m.
Hello guys,
We have a Ceph cluster that runs just fine with Ceph Octopus; we use RBD
for some workloads, RadosGW (via S3) for others, and iSCSI for some Windows
clients.
Recently, we had the need to add some VMWare clusters as clients for the
iSCSI GW and also Windows systems with the use of Clustered Storage Volumes
(CSV), and we are facing a weird situation. In windows for instance, the
iSCSI block can be mounted, formatted and consumed by all nodes, but when
we add in the CSV it fails with some generic exception. The same happens in
VMWare, when we try to use it with VMFS it fails.
We do not seem to find the root cause for these errors. However, the errors
seem to be linked to the situation of multiple nodes consuming the same
block by shared file systems. Have you guys seen this before?
Are we missing some basic configuration in the iSCSI GW?
19 Jun
19 Jun
3:38 p.m.
On 19.06.23 13:47, Work Ceph wrote:
Recently, we had the need to add some VMWare clusters
as clients for the
iSCSI GW and also Windows systems with the use of Clustered Storage Volumes
(CSV), and we are facing a weird situation. In windows for instance, the
iSCSI block can be mounted, formatted and consumed by all nodes, but when
we add in the CSV it fails with some generic exception. The same happens in
VMWare, when we try to use it with VMFS it fails.
The iSCSI target used does not support SCSI persistent group
reservations when in multipath mode.
https://docs.ceph.com/en/quincy/rbd/iscsi-initiators/
AFAIK VMware uses these in VMFS.
Regards
--
Robert Sander
Heinlein Support GmbH
Linux: Akademie - Support - Hosting
http://www.heinlein-support.de
Tel: 030-405051-43
Fax: 030-405051-19
Zwangsangaben lt. §35a GmbHG:
HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin
4:05 p.m.
Windows Clustered Shared Volumes and Failover Clustering require the
support of clustered persistence reservations by the block device to
coordinate access by multiple hosts. The default iSCSI implementation in
Ceph does not support this, you can use the iSCSI implementation in
PetaSAN project:
www.petasan.org
which supports this feature and provides a high performance
implementation. We currently use Ceph 17.2.5
On 19/06/2023 14:47, Work Ceph wrote:
Hello guys,
We have a Ceph cluster that runs just fine with Ceph Octopus; we use RBD
for some workloads, RadosGW (via S3) for others, and iSCSI for some Windows
clients.
Recently, we had the need to add some VMWare clusters as clients for the
iSCSI GW and also Windows systems with the use of Clustered Storage Volumes
(CSV), and we are facing a weird situation. In windows for instance, the
iSCSI block can be mounted, formatted and consumed by all nodes, but when
we add in the CSV it fails with some generic exception. The same happens in
VMWare, when we try to use it with VMFS it fails.
We do not seem to find the root cause for these errors. However, the errors
seem to be linked to the situation of multiple nodes consuming the same
block by shared file systems. Have you guys seen this before?
Are we missing some basic configuration in the iSCSI GW?
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
20 Jun
20 Jun
1:16 a.m.
I see, thanks for the feedback guys!
It is interesting that Ceph Manager does not allow us to export iSCSI
blocks without selecting 2 or more iSCSI portals. Therefore, we will always
use at least two, and as a consequence that feature is not going to be
supported. Can I export an RBD image via iSCSI gateway using only one
portal via GwCli?
@Maged Mokhtar, I am not sure I follow. Do you guys have an iSCSI
implementation that we can use to somehow replace the default iSCSI server
in the default Ceph iSCSI Gateway? I didn't quite understand what the
petasan project is, and if it is an OpenSource solution that we can somehow
just pick/select/use one of its modules (e.g. just the iSCSI
implementation) that you guys have.
On Mon, Jun 19, 2023 at 10:07 AM Maged Mokhtar <mmokhtar(a)petasan.org> wrote:
Windows Clustered Shared Volumes and Failover
Clustering require the
support of clustered persistence reservations by the block device to
coordinate access by multiple hosts. The default iSCSI implementation in
Ceph does not support this, you can use the iSCSI implementation in
PetaSAN project:
www.petasan.org
which supports this feature and provides a high performance
implementation. We currently use Ceph 17.2.5
On 19/06/2023 14:47, Work Ceph wrote:
Hello guys,
We have a Ceph cluster that runs just fine with Ceph Octopus; we use RBD
for some workloads, RadosGW (via S3) for others, and iSCSI for some
Windows
clients.
Recently, we had the need to add some VMWare clusters as clients for the
iSCSI GW and also Windows systems with the use of Clustered Storage
Volumes
(CSV), and we are facing a weird situation. In
windows for instance, the
iSCSI block can be mounted, formatted and consumed by all nodes, but when
we add in the CSV it fails with some generic exception. The same happens
in
VMWare, when we try to use it with VMFS it
fails.
We do not seem to find the root cause for these errors. However, the
errors
seem to be linked to the situation of multiple
nodes consuming the same
block by shared file systems. Have you guys seen this before?
Are we missing some basic configuration in the iSCSI GW?
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:58 a.m.
As a sidenote: there's the windows rbd driver which will get you waaaay
more performance. It's labeled beta, but it seems to work fine for a lot
of people. If you have a testlab you could try that.
Angelo.
On 19/06/2023 18:16, Work Ceph wrote:
> I see, thanks for the feedback guys!
>
> It is interesting that Ceph Manager does not allow us to export iSCSI
> blocks without selecting 2 or more iSCSI portals. Therefore, we will always
> use at least two, and as a consequence that feature is not going to be
> supported. Can I export an RBD image via iSCSI gateway using only one
> portal via GwCli?
>
> @Maged Mokhtar, I am not sure I follow. Do you guys have an iSCSI
> implementation that we can use to somehow replace the default iSCSI server
> in the default Ceph iSCSI Gateway? I didn't quite understand what the
> petasan project is, and if it is an OpenSource solution that we can somehow
> just pick/select/use one of its modules (e.g. just the iSCSI
> implementation) that you guys have.
>
> On Mon, Jun 19, 2023 at 10:07 AM Maged Mokhtar <mmokhtar(a)petasan.org> wrote:
>
>> Windows Clustered Shared Volumes and Failover Clustering require the
>> support of clustered persistence reservations by the block device to
>> coordinate access by multiple hosts. The default iSCSI implementation in
>> Ceph does not support this, you can use the iSCSI implementation in
>> PetaSAN project:
>>
>> www.petasan.org
>>
>> which supports this feature and provides a high performance
>> implementation. We currently use Ceph 17.2.5
>>
>>
>> On 19/06/2023 14:47, Work Ceph wrote:
>>> Hello guys,
>>>
>>> We have a Ceph cluster that runs just fine with Ceph Octopus; we use RBD
>>> for some workloads, RadosGW (via S3) for others, and iSCSI for some
>> Windows
>>> clients.
>>>
>>> Recently, we had the need to add some VMWare clusters as clients for the
>>> iSCSI GW and also Windows systems with the use of Clustered Storage
>> Volumes
>>> (CSV), and we are facing a weird situation. In windows for instance, the
>>> iSCSI block can be mounted, formatted and consumed by all nodes, but when
>>> we add in the CSV it fails with some generic exception. The same happens
>> in
>>> VMWare, when we try to use it with VMFS it fails.
>>>
>>> We do not seem to find the root cause for these errors. However, the
>> errors
>>> seem to be linked to the situation of multiple nodes consuming the same
>>> block by shared file systems. Have you guys seen this before?
>>>
>>> Are we missing some basic configuration in the iSCSI GW?
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
21 Jun
21 Jun
12:02 p.m.
On 20/06/2023 01:16, Work Ceph wrote:
I see, thanks for the feedback guys!
It is interesting that Ceph Manager does not allow us to export iSCSI
blocks without selecting 2 or more iSCSI portals. Therefore, we will
always use at least two, and as a consequence that feature is not
going to be supported. Can I export an RBD image via iSCSI gateway
using only one portal via GwCli?
@Maged Mokhtar, I am not sure I follow. Do you guys have an iSCSI
implementation that we can use to somehow replace the default iSCSI
server in the default Ceph iSCSI Gateway? I didn't quite understand
what the petasan project is, and if it is an OpenSource solution that
we can somehow just pick/select/use one of its modules (e.g. just the
iSCSI implementation) that you guys have.
For sure PetaSAN is open source..you should see this from the home page :)
we use Consul
https://www.consul.io/use-cases/multi-platform-service-mesh
to scale-out the service/protocol layers above Ceph in a scale-out
active/active fashion.
Most of our target use cases are non linux, such as VMWare and Windows,
we provide easy to use deployment and management.
For iSCSI, we use kernel/LIO rbd backstore originally developed by SUSE
Enterprise storge. We have done some changes to send persistence
reservations using the Ceph watch/notify, we also added changes to
coordinate pre-snapshot quiescing/flushing across different gateways. We
ported rbd backstore to 5.14 kernel.
You should be able to use the iSCSI gateway by itself on existing non
PetaSAN clusters but it is not a setup we support. You would use the LIO
targercli to script the setup. There are some things to take care of
such as setting the disk serial wwn to be the same across the different
gateways serving the same image, setting up the multiple tpgs (target
portal groups) for an image but only enabling the tpgs for local node.
This setup will be using multi path MPIO to provide HA. Again it is not
a setup we support, you could try it yourself in a test environment, you
can also setup a test PetaSAN setup and examine the LIO configuration
using targetcli. You can send me email if you need any clarifications.
Cheers /Maged
23 Jun
23 Jun
6:31 p.m.
Thanks for the help so far guys!
Has anybody used (made it work) the default ceph-iscsi implementation with
VMware and/or Windows CSV storage system with a single target/portal in
iSCSI?
On Wed, Jun 21, 2023 at 6:02 AM Maged Mokhtar <mmokhtar(a)petasan.org> wrote:
On 20/06/2023 01:16, Work Ceph wrote:
I see, thanks for the feedback guys!
It is interesting that Ceph Manager does not allow us to export iSCSI
blocks without selecting 2 or more iSCSI portals. Therefore, we will
always use at least two, and as a consequence that feature is not
going to be supported. Can I export an RBD image via iSCSI gateway
using only one portal via GwCli?
@Maged Mokhtar, I am not sure I follow. Do you guys have an iSCSI
implementation that we can use to somehow replace the default iSCSI
server in the default Ceph iSCSI Gateway? I didn't quite understand
what the petasan project is, and if it is an OpenSource solution that
we can somehow just pick/select/use one of its modules (e.g. just the
iSCSI implementation) that you guys have.
For sure PetaSAN is open source..you should see this from the home page :)
we use Consul
https://www.consul.io/use-cases/multi-platform-service-mesh
to scale-out the service/protocol layers above Ceph in a scale-out
active/active fashion.
Most of our target use cases are non linux, such as VMWare and Windows,
we provide easy to use deployment and management.
For iSCSI, we use kernel/LIO rbd backstore originally developed by SUSE
Enterprise storge. We have done some changes to send persistence
reservations using the Ceph watch/notify, we also added changes to
coordinate pre-snapshot quiescing/flushing across different gateways. We
ported rbd backstore to 5.14 kernel.
You should be able to use the iSCSI gateway by itself on existing non
PetaSAN clusters but it is not a setup we support. You would use the LIO
targercli to script the setup. There are some things to take care of
such as setting the disk serial wwn to be the same across the different
gateways serving the same image, setting up the multiple tpgs (target
portal groups) for an image but only enabling the tpgs for local node.
This setup will be using multi path MPIO to provide HA. Again it is not
a setup we support, you could try it yourself in a test environment, you
can also setup a test PetaSAN setup and examine the LIO configuration
using targetcli. You can send me email if you need any clarifications.
Cheers /Maged
4 Jul
4 Jul
5:52 p.m.
Thank you guys for the help here! We discovered the issue. We deployed the
whole system using Ubuntu, and it seems that when the TCMU-runner is
installed, some folders are not created, and as a consequence, the iSCSI
reservations do not work as they (the iSCSI reservation) write in a file
the reservations created by the clients.
The folders that we needed to be created in the iSCSI GW systems are the
following:
- /var/target/pr
- /var/target/alua
By the way, that is why only one GW is supported when using iSCSI
reservation. Just because they are written in a file in the local file
system. If they were written to the Ceph system in a Rados object we would
not have that limitation, or if they were written in a shared folder path
that is mounted in all iSCSI GW. We will validate this setup to see if they
would work just fine for Multipath with iSCSI reservation.
On Fri, Jun 23, 2023 at 12:31 PM Work Ceph <work.ceph.user.mailing(a)gmail.com>
wrote:
Thanks for the help so far guys!
Has anybody used (made it work) the default ceph-iscsi implementation with
VMware and/or Windows CSV storage system with a single target/portal in
iSCSI?
On Wed, Jun 21, 2023 at 6:02 AM Maged Mokhtar <mmokhtar(a)petasan.org>
wrote:
>
> On 20/06/2023 01:16, Work Ceph wrote:
> > I see, thanks for the feedback guys!
> >
> > It is interesting that Ceph Manager does not allow us to export iSCSI
> > blocks without selecting 2 or more iSCSI portals. Therefore, we will
> > always use at least two, and as a consequence that feature is not
> > going to be supported. Can I export an RBD image via iSCSI gateway
> > using only one portal via GwCli?
> >
> > @Maged Mokhtar, I am not sure I follow. Do you guys have an iSCSI
> > implementation that we can use to somehow replace the default iSCSI
> > server in the default Ceph iSCSI Gateway? I didn't quite understand
> > what the petasan project is, and if it is an OpenSource solution that
> > we can somehow just pick/select/use one of its modules (e.g. just the
> > iSCSI implementation) that you guys have.
> >
>
> For sure PetaSAN is open source..you should see this from the home page :)
> we use Consul
> https://www.consul.io/use-cases/multi-platform-service-mesh
> to scale-out the service/protocol layers above Ceph in a scale-out
> active/active fashion.
> Most of our target use cases are non linux, such as VMWare and Windows,
> we provide easy to use deployment and management.
>
> For iSCSI, we use kernel/LIO rbd backstore originally developed by SUSE
> Enterprise storge. We have done some changes to send persistence
> reservations using the Ceph watch/notify, we also added changes to
> coordinate pre-snapshot quiescing/flushing across different gateways. We
> ported rbd backstore to 5.14 kernel.
>
> You should be able to use the iSCSI gateway by itself on existing non
> PetaSAN clusters but it is not a setup we support. You would use the LIO
> targercli to script the setup. There are some things to take care of
> such as setting the disk serial wwn to be the same across the different
> gateways serving the same image, setting up the multiple tpgs (target
> portal groups) for an image but only enabling the tpgs for local node.
> This setup will be using multi path MPIO to provide HA. Again it is not
> a setup we support, you could try it yourself in a test environment, you
> can also setup a test PetaSAN setup and examine the LIO configuration
> using targetcli. You can send me email if you need any clarifications.
>
> Cheers /Maged
>
>
>
321
days inactive
336
days old
7 comments
4 participants
participants (4)
-
Angelo Hongens -
Maged Mokhtar -
Robert Sander -
Work Ceph