I see, thanks for the feedback guys! It is interesting that Ceph Manager does not allow us to export iSCSI blocks without selecting 2 or more iSCSI portals. Therefore, we will always use at least two, and as a consequence that feature is not going to be supported. Can I export an RBD image via iSCSI gateway using only one portal via GwCli? @Maged Mokhtar, I am not sure I follow. Do you guys have an iSCSI implementation that we can use to somehow replace the default iSCSI server in the default Ceph iSCSI Gateway? I didn't quite understand what the petasan project is, and if it is an OpenSource solution that we can somehow just pick/select/use one of its modules (e.g. just the iSCSI implementation) that you guys have. On Mon, Jun 19, 2023 at 10:07 AM Maged Mokhtar <mmokhtar@petasan.org> wrote:

Windows Clustered Shared Volumes and Failover Clustering require the support of clustered persistence reservations by the block device to coordinate access by multiple hosts. The default iSCSI implementation in Ceph does not support this, you can use the iSCSI implementation in PetaSAN project:

www.petasan.org

which supports this feature and provides a high performance implementation. We currently use Ceph 17.2.5

On 19/06/2023 14:47, Work Ceph wrote:

...

Hello guys,

We have a Ceph cluster that runs just fine with Ceph Octopus; we use RBD for some workloads, RadosGW (via S3) for others, and iSCSI for some Windows clients.

Recently, we had the need to add some VMWare clusters as clients for the iSCSI GW and also Windows systems with the use of Clustered Storage Volumes (CSV), and we are facing a weird situation. In windows for instance, the iSCSI block can be mounted, formatted and consumed by all nodes, but when we add in the CSV it fails with some generic exception. The same happens in VMWare, when we try to use it with VMFS it fails.

We do not seem to find the root cause for these errors. However, the errors seem to be linked to the situation of multiple nodes consuming the same block by shared file systems. Have you guys seen this before?

Are we missing some basic configuration in the iSCSI GW? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

On 20/06/2023 01:16, Work Ceph wrote:

I see, thanks for the feedback guys!

It is interesting that Ceph Manager does not allow us to export iSCSI blocks without selecting 2 or more iSCSI portals. Therefore, we will always use at least two, and as a consequence that feature is not going to be supported. Can I export an RBD image via iSCSI gateway using only one portal via GwCli?

@Maged Mokhtar, I am not sure I follow. Do you guys have an iSCSI implementation that we can use to somehow replace the default iSCSI server in the default Ceph iSCSI Gateway? I didn't quite understand what the petasan project is, and if it is an OpenSource solution that we can somehow just pick/select/use one of its modules (e.g. just the iSCSI implementation) that you guys have.

For sure PetaSAN is open source..you should see this from the home page :) we use Consul https://www.consul.io/use-cases/multi-platform-service-mesh to scale-out the service/protocol layers above Ceph in a scale-out active/active fashion. Most of our target use cases are non linux, such as VMWare and Windows, we provide easy to use deployment and management. For iSCSI, we use kernel/LIO rbd backstore originally developed by SUSE Enterprise storge. We have done some changes to send persistence reservations using the Ceph watch/notify, we also added changes to coordinate pre-snapshot quiescing/flushing across different gateways. We ported rbd backstore to 5.14 kernel. You should be able to use the iSCSI gateway by itself on existing non PetaSAN clusters but it is not a setup we support. You would use the LIO targercli to script the setup. There are some things to take care of such as setting the disk serial wwn to be the same across the different gateways serving the same image, setting up the multiple tpgs (target portal groups) for an image but only enabling the tpgs for local node. This setup will be using multi path MPIO to provide HA. Again it is not a setup we support, you could try it yourself in a test environment, you can also setup a test PetaSAN setup and examine the LIO configuration using targetcli. You can send me email if you need any clarifications. Cheers /Maged

Thank you guys for the help here! We discovered the issue. We deployed the whole system using Ubuntu, and it seems that when the TCMU-runner is installed, some folders are not created, and as a consequence, the iSCSI reservations do not work as they (the iSCSI reservation) write in a file the reservations created by the clients. The folders that we needed to be created in the iSCSI GW systems are the following: - /var/target/pr - /var/target/alua By the way, that is why only one GW is supported when using iSCSI reservation. Just because they are written in a file in the local file system. If they were written to the Ceph system in a Rados object we would not have that limitation, or if they were written in a shared folder path that is mounted in all iSCSI GW. We will validate this setup to see if they would work just fine for Multipath with iSCSI reservation. On Fri, Jun 23, 2023 at 12:31 PM Work Ceph <work.ceph.user.mailing@gmail.com> wrote:

Thanks for the help so far guys!

Has anybody used (made it work) the default ceph-iscsi implementation with VMware and/or Windows CSV storage system with a single target/portal in iSCSI?

On Wed, Jun 21, 2023 at 6:02 AM Maged Mokhtar <mmokhtar@petasan.org> wrote:

...

On 20/06/2023 01:16, Work Ceph wrote:

...

I see, thanks for the feedback guys!

It is interesting that Ceph Manager does not allow us to export iSCSI blocks without selecting 2 or more iSCSI portals. Therefore, we will always use at least two, and as a consequence that feature is not going to be supported. Can I export an RBD image via iSCSI gateway using only one portal via GwCli?

@Maged Mokhtar, I am not sure I follow. Do you guys have an iSCSI implementation that we can use to somehow replace the default iSCSI server in the default Ceph iSCSI Gateway? I didn't quite understand what the petasan project is, and if it is an OpenSource solution that we can somehow just pick/select/use one of its modules (e.g. just the iSCSI implementation) that you guys have.

For sure PetaSAN is open source..you should see this from the home page :) we use Consul https://www.consul.io/use-cases/multi-platform-service-mesh to scale-out the service/protocol layers above Ceph in a scale-out active/active fashion. Most of our target use cases are non linux, such as VMWare and Windows, we provide easy to use deployment and management.

For iSCSI, we use kernel/LIO rbd backstore originally developed by SUSE Enterprise storge. We have done some changes to send persistence reservations using the Ceph watch/notify, we also added changes to coordinate pre-snapshot quiescing/flushing across different gateways. We ported rbd backstore to 5.14 kernel.

You should be able to use the iSCSI gateway by itself on existing non PetaSAN clusters but it is not a setup we support. You would use the LIO targercli to script the setup. There are some things to take care of such as setting the disk serial wwn to be the same across the different gateways serving the same image, setting up the multiple tpgs (target portal groups) for an image but only enabling the tpgs for local node. This setup will be using multi path MPIO to provide HA. Again it is not a setup we support, you could try it yourself in a test environment, you can also setup a test PetaSAN setup and examine the LIO configuration using targetcli. You can send me email if you need any clarifications.

Cheers /Maged