Good morning, I wanted to inquire about the status of the Ceph iSCSI gateway service. We currently have several machines installed with this technology that are working correctly, although I have seen that it appears to be discontinued since 2022. My question is whether to continue down this path, adding machines, or whether it will be discontinued in new Ceph distributions. Best regards and thanks in advance. -- ================================================================ Ibán Cabrillo Bartolomé Instituto de Física de Cantabria (IFCA-CSIC) Santander, Spain Tel: +34942200969/+34669930421 Responsible for advanced computing service (RSC) ========================================================================================= ========================================================================================= All our suppliers must know and accept IFCA policy available at: https://confluence.ifca.es/display/IC/Information+Security+Policy+for+Extern... ==========================================================================================
ISCSI is still being used in the LRC (long running cluster) that is a storage backend for parts of the ceph team's infrastructure, so I don't think it's going to disappear in the near future. I believe the plan is to eventually swap over to nvmeof instead ( https://docs.ceph.com/en/reef/rbd/nvmeof-overview/) but nvmeof isn't ready in squid iirc. On Mon, Feb 10, 2025 at 6:27 AM Iban Cabrillo <cabrillo@ifca.unican.es> wrote:
Good morning,
I wanted to inquire about the status of the Ceph iSCSI gateway service. We currently have several machines installed with this technology that are working correctly,
although I have seen that it appears to be discontinued since 2022. My question is whether to continue down this path, adding machines, or whether it will be discontinued in new Ceph distributions.
Best regards and thanks in advance. --
================================================================ Ibán Cabrillo Bartolomé Instituto de Física de Cantabria (IFCA-CSIC) Santander, Spain Tel: +34942200969/+34669930421 Responsible for advanced computing service (RSC) =========================================================================================
=========================================================================================
All our suppliers must know and accept IFCA policy available at:
https://confluence.ifca.es/display/IC/Information+Security+Policy+for+Extern... ==========================================================================================
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, I found that there is also a third-party iSCSI implementation that claims to support RBD: https://github.com/fujita/tgt I use it in a non-Ceph context, and it works well. Question: Does anyone have any experience running tgt as an alternative to tcmu-runner? If so, which clients (iSCSI initiators) do you use? On Mon, Feb 10, 2025 at 11:22 PM Adam King <adking@redhat.com> wrote:
ISCSI is still being used in the LRC (long running cluster) that is a storage backend for parts of the ceph team's infrastructure, so I don't think it's going to disappear in the near future. I believe the plan is to eventually swap over to nvmeof instead ( https://docs.ceph.com/en/reef/rbd/nvmeof-overview/) but nvmeof isn't ready in squid iirc.
On Mon, Feb 10, 2025 at 6:27 AM Iban Cabrillo <cabrillo@ifca.unican.es> wrote:
Good morning,
I wanted to inquire about the status of the Ceph iSCSI gateway service. We currently have several machines installed with this technology that are working correctly,
although I have seen that it appears to be discontinued since 2022. My question is whether to continue down this path, adding machines, or whether it will be discontinued in new Ceph distributions.
Best regards and thanks in advance. --
================================================================ Ibán Cabrillo Bartolomé Instituto de Física de Cantabria (IFCA-CSIC) Santander, Spain Tel: +34942200969/+34669930421 Responsible for advanced computing service (RSC) =========================================================================================
=========================================================================================
All our suppliers must know and accept IFCA policy available at:
https://confluence.ifca.es/display/IC/Information+Security+Policy+for+Extern... ==========================================================================================
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
The iSCSI gateway is likely to disappear in the future and is definitely in minimal maintenance mode right now. However, as with all removed features, if we do that it will have plenty of warning — we will announce it is deprecated in a major release without changing or removing it, and then remove it in a subsequent major release. -Greg On Mon, Feb 10, 2025 at 7:22 AM Adam King <adking@redhat.com> wrote:
ISCSI is still being used in the LRC (long running cluster) that is a storage backend for parts of the ceph team's infrastructure, so I don't think it's going to disappear in the near future. I believe the plan is to eventually swap over to nvmeof instead ( https://docs.ceph.com/en/reef/rbd/nvmeof-overview/) but nvmeof isn't ready in squid iirc.
On Mon, Feb 10, 2025 at 6:27 AM Iban Cabrillo <cabrillo@ifca.unican.es> wrote:
Good morning,
I wanted to inquire about the status of the Ceph iSCSI gateway service. We currently have several machines installed with this technology that are working correctly,
although I have seen that it appears to be discontinued since 2022. My question is whether to continue down this path, adding machines, or whether it will be discontinued in new Ceph distributions.
Best regards and thanks in advance. --
================================================================ Ibán Cabrillo Bartolomé Instituto de Física de Cantabria (IFCA-CSIC) Santander, Spain Tel: +34942200969/+34669930421 Responsible for advanced computing service (RSC) =========================================================================================
=========================================================================================
All our suppliers must know and accept IFCA policy available at:
https://confluence.ifca.es/display/IC/Information+Security+Policy+for+Extern... ==========================================================================================
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Okay Gregory, Bad news for me, I will have to find another way. The truth is that from the operational and long-term maintenance point of view it is practically transparent. This option fitted very well in our system, since the Ceph cluster is easy to maintain, while for example the iSCSI cabinets have to be migrated every certain time. In our case where the use of block access is not intensive, the truth is that we were not having a bad performance, it is also true that until now I had only exported 4 blocks with this system. Regards, I -- ================================================================ Ibán Cabrillo Bartolomé Instituto de Física de Cantabria (IFCA-CSIC) Santander, Spain Tel: +34942200969/+34669930421 Responsible for advanced computing service (RSC) ========================================================================================= ========================================================================================= All our suppliers must know and accept IFCA policy available at: https://confluence.ifca.es/display/IC/Information+Security+Policy+for+Extern... ==========================================================================================
Hi Iban, have you thought about replacing the iSCSI gateway with the NVMeOF gateway? Regards, Joachim joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677 Am Mi., 12. Feb. 2025 um 08:53 Uhr schrieb Iban Cabrillo < cabrillo@ifca.unican.es>:
Okay Gregory, Bad news for me, I will have to find another way. The truth is that from the operational and long-term maintenance point of view it is practically transparent. This option fitted very well in our system, since the Ceph cluster is easy to maintain, while for example the iSCSI cabinets have to be migrated every certain time.
In our case where the use of block access is not intensive, the truth is that we were not having a bad performance, it is also true that until now I had only exported 4 blocks with this system.
Regards, I
-- ================================================================ Ibán Cabrillo Bartolomé Instituto de Física de Cantabria (IFCA-CSIC) Santander, Spain Tel: +34942200969/+34669930421 Responsible for advanced computing service (RSC)
=========================================================================================
========================================================================================= All our suppliers must know and accept IFCA policy available at:
https://confluence.ifca.es/display/IC/Information+Security+Policy+for+Extern...
========================================================================================== _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
You can compile this iscsi stuff not? I did this recently with a recent ceph version. But noticed it was performing less than just a rbd mount.
Okay Gregory, Bad news for me, I will have to find another way. The truth is that from the operational and long-term maintenance point of view it is practically transparent. This option fitted very well in our system, since the Ceph cluster is easy to maintain, while for example the iSCSI cabinets have to be migrated every certain time.
In our case where the use of block access is not intensive, the truth is that we were not having a bad performance, it is also true that until now I had only exported 4 blocks with this system.
On 10-02-2025 12:26, Iban Cabrillo wrote:
Good morning,
I wanted to inquire about the status of the Ceph iSCSI gateway service. We currently have several machines installed with this technology that are working correctly,
although I have seen that it appears to be discontinued since 2022. My question is whether to continue down this path, adding machines, or whether it will be discontinued in new Ceph distributions.
Best regards and thanks in advance.
We are using LIO (targetcli) with rbd-nbd (can also be used with krbd). We have made it HA by modifying CTDB scripts (Samba HA solution) in a three node setup. To prevent split-brain a recovery lock on a ceph object is used (recovery lock = !/usr/local/bin/ctdb_mutex_ceph_rados_helper ceph client.$client poolname ctdb_sgw_rbd_recovery_lock 10). This setup also supports SCSI persistent reservations, a requirement for a version of MSSQL cluster that uses shared storage. This does not do any fancy stuff like tcmu-runner can do with ALUA (multi-path). So you are bound to a single node for performance. But for us this is more than good enough. As soon as NVMeOF supports persistent SCSI reservations we will make the shift to NVMeOF [1]. Gr. Stefan [1]: https://github.com/ceph/ceph-nvmeof/issues/41
Hi Iban, If using NVME-oF, which is meant to replace iSCSI in Ceph, is not feasible for you, you could take a look at PetaSAN's iSCSI implementation. We've been successfully using PetaSAN [1] iSCSI gateways with an external Ceph cluster (not the one deployed by PetaSAN itself) on VMware hypervisors (135 VMs, 75TB) for a year and a half. We've adopted PetaSAN after conducting extensive resilience, stability, and performance tests and identifying a few quickly-fixed bugs (memory leak and XCOPY VAAI malfunction). It offers great performance and scalability. Still working great. Regards, Frédéric. [1] https://www.petasan.org ----- Le 10 Fév 25, à 12:26, Iban Cabrillo cabrillo@ifca.unican.es a écrit :
Good morning,
I wanted to inquire about the status of the Ceph iSCSI gateway service. We currently have several machines installed with this technology that are working correctly,
although I have seen that it appears to be discontinued since 2022. My question is whether to continue down this path, adding machines, or whether it will be discontinued in new Ceph distributions.
Best regards and thanks in advance. --
================================================================ Ibán Cabrillo Bartolomé Instituto de Física de Cantabria (IFCA-CSIC) Santander, Spain Tel: +34942200969/+34669930421 Responsible for advanced computing service (RSC) ========================================================================================= ========================================================================================= All our suppliers must know and accept IFCA policy available at:
https://confluence.ifca.es/display/IC/Information+Security+Policy+for+Extern... ==========================================================================================
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frederic,
We've been successfully using PetaSAN [1] iSCSI gateways with an external Ceph cluster (not the one deployed by PetaSAN itself) on VMware hypervisors (135 VMs, 75TB) for a year and a half.
I am interested in this. We are currently using a TrueNAS VM on our Proxmox cluster that is exporting its vDisk via iSCSI, which is less than ideal, after find extremely poor performance with the iSCSI in Ceph Currently just using that iSCSI solution for a backup repository for Veeam, but we would like to use iSCSI with our Ceph cluster to move some of our VMs on VMware (that we haven't been able to migrate to the Proxmox cluster yet) to it so we can clear off one or two of our traditional SANs and connect them to our Proxmox cluster as additional storage in addition to our Ceph cluster. Is your Ceph cluster on bare metal or deployed using cephadm? Is there a document or guide you followed to install PeteSAN with using an external Ceph cluster? All I can seem to find at the moment are for deploying PetaSAN using its own Ceph implementation.
Hi Alan, We use containers deployed by the orchestrator and cephadm on a bare-metal nodes, but this is not important for PetaSAN since it doesn't manage our external cluster. I synthesized my notes here [1] this week. If you follow the guide, please share your experience and feel free to open an issue if you see any points that could be improved. Best regards, Frédéric. [1] https://github.com/frednass/ceph_petasan ----- Le 17 Fév 25, à 15:28, Alan Murrell Alan@t-net.ca a écrit :
Hi Frederic,
We've been successfully using PetaSAN [1] iSCSI gateways with an external Ceph cluster (not the one deployed by PetaSAN itself) on VMware hypervisors (135 VMs, 75TB) for a year and a half.
I am interested in this.
We are currently using a TrueNAS VM on our Proxmox cluster that is exporting its vDisk via iSCSI, which is less than ideal, after find extremely poor performance with the iSCSI in Ceph Currently just using that iSCSI solution for a backup repository for Veeam, but we would like to use iSCSI with our Ceph cluster to move some of our VMs on VMware (that we haven't been able to migrate to the Proxmox cluster yet) to it so we can clear off one or two of our traditional SANs and connect them to our Proxmox cluster as additional storage in addition to our Ceph cluster.
Is your Ceph cluster on bare metal or deployed using cephadm? Is there a document or guide you followed to install PeteSAN with using an external Ceph cluster? All I can seem to find at the moment are for deploying PetaSAN using its own Ceph implementation. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I synthesized my notes here [1] this week. If you follow the guide, please share your experience and feel free to open an issue if you see any points that could be improved.
Best regards, Frédéric.
Thanks Frederic for the excellent guide. We at PetaSAN project will try to use it to integrate with external existing Ceph clusters. Hopefully we can improve things in future releases to make this more streamlined. /maged
participants (10)
-
Adam King
-
Alan Murrell
-
Alexander Patrakov
-
Frédéric Nass
-
Gregory Farnum
-
Iban Cabrillo
-
Joachim Kraftmayer
-
Maged Mokhtar
-
Marc
-
Stefan Kooman