Doubt about ceph-iscsi and Vmware
Hi, I'm testing Ceph with Vmware, using Ceph-iscsi gateway. I reading documentation* and have doubts some points: - If I understanded, in general terms, for each VMFS datastore in VMware will match the an RBD image. (consequently in an RBD image I will possible have many VMWare disks). Its correct? - In documentation is this: "gwcli requires a pool with the name rbd, so it can store metadata like the iSCSI configuration". In part 4 of "Configuration", have: "Add a RBD image with the name disk_1 in the pool rbd". In this part, the use of "rbd" pool is a example and I could use any pool for storage of image, or the pool should be "rbd"? Resuming: gwcli require "rbd" pool for metadata and I could use any pool for image, or i will use just "rbd pool" for storage image and metadata? - How much memory ceph-iscsi use? Which is a good number of RAM? Regards Gesiel * https://docs.ceph.com/docs/master/rbd/iscsi-target-cli/
On Fri, Sep 20, 2019 at 8:55 PM Gesiel Galvão Bernardes <gesiel.bernardes@gmail.com> wrote:
Hi, I'm testing Ceph with Vmware, using Ceph-iscsi gateway. I reading documentation* and have doubts some points:
- If I understanded, in general terms, for each VMFS datastore in VMware will match the an RBD image. (consequently in an RBD image I will possible have many VMWare disks). Its correct?
yes
- In documentation is this: "gwcli requires a pool with the name rbd, so it can store metadata like the iSCSI configuration". In part 4 of "Configuration", have: "Add a RBD image with the name disk_1 in the pool rbd". In this part, the use of "rbd" pool is a example and I could use any pool for storage of image, or the pool should be "rbd"?
that's just an example, yes
Resuming: gwcli require "rbd" pool for metadata and I could use any pool for image, or i will use just "rbd pool" for storage image and metadata?
you can even store the metadata elsewhere in newer versions, see options in the config file
- How much memory ceph-iscsi use? Which is a good number of RAM?
since it can't cache anything: virtually nothing
Regards Gesiel
* https://docs.ceph.com/docs/master/rbd/iscsi-target-cli/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Gesiel, Some iscsi settings are stored in an object, this object is stored in the rbd pool. Hnece the rbd pool is required. Your LUN's are mapped to {pool}/{rbdimage}. You should treat these as you treat pools and rbd images in general. In smallish deployments I try to keep it simple and make 1 pool for each deviceclass and make LUN's as big as possible, while still allowing for setting 1 LUN in maintenance mode in the datastore cluster within vSphere, in case we need to re-format as part of vmfs upgrade. Remember to set PSP and recovery timeout properly. From the LUN level and up, you just treat the storage as any other iscsi storage connected to vSphere. the iGW consumes RAM pr. LUN export... I can't remember the default settings but we are talking about single-digit Gb qith tens of LUN exported, so it's fairly lightweight. /Heðin On frí, 2019-09-20 at 15:52 -0300, Gesiel Galvão Bernardes wrote:
Hi, I'm testing Ceph with Vmware, using Ceph-iscsi gateway. I reading documentation* and have doubts some points:
- If I understanded, in general terms, for each VMFS datastore in VMware will match the an RBD image. (consequently in an RBD image I will possible have many VMWare disks). Its correct?
- In documentation is this: "gwcli requires a pool with the name rbd, so it can store metadata like the iSCSI configuration". In part 4 of "Configuration", have: "Add a RBD image with the name disk_1 in the pool rbd". In this part, the use of "rbd" pool is a example and I could use any pool for storage of image, or the pool should be "rbd"? Resuming: gwcli require "rbd" pool for metadata and I could use any pool for image, or i will use just "rbd pool" for storage image and metadata?
- How much memory ceph-iscsi use? Which is a good number of RAM?
Regards Gesiel
* https://docs.ceph.com/docs/master/rbd/iscsi-target-cli/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 09/20/2019 01:52 PM, Gesiel Galvão Bernardes wrote:
Hi, I'm testing Ceph with Vmware, using Ceph-iscsi gateway. I reading documentation* and have doubts some points:
- If I understanded, in general terms, for each VMFS datastore in VMware will match the an RBD image. (consequently in an RBD image I will possible have many VMWare disks). Its correct?
- In documentation is this: "gwcli requires a pool with the name rbd, so it can store metadata like the iSCSI configuration". In part 4 of "Configuration", have: "Add a RBD image with the name disk_1 in the pool rbd". In this part, the use of "rbd" pool is a example and I could use any pool for storage of image, or the pool should be "rbd"? Resuming: gwcli require "rbd" pool for metadata and I could use any pool for image, or i will use just "rbd pool" for storage image and metadata?
- How much memory ceph-iscsi use? Which is a good number of RAM?
The major memory use is: 1. In RHEL 7.5 kernels and older we allocate max_data_area_mb of kernel memory per device. The default value for that is 8. You can use gwcli to configure it. It is allocated when the device is created. In newer kernels, there is pool of memory and each device can use up to max_data_area_mb worth of it. The per device default is the same and you can change it with gwcli. The total pool limit is 2 GB. There is a sysfs file: /sys/module/target_core_user/parameters/global_max_data_area_mb that can be used to change it. 2. Each device uses about 20 MB of memory in userspace. This is not configurable.
Thank you everyone for the explanations. Still on this subject: I created an host and attached a disk. For a second host, for use a shared iscsi storage, I just need add disk to second client? I tried this:
disk add pool1/vmware_iscsi1 Warning: 'pool1/vmware_iscsi1' mapped to 1 other client(s) ok
It's a problem, or is correct? Regards Gesiel Em sex, 20 de set de 2019 às 20:13, Mike Christie <mchristi@redhat.com> escreveu:
On 09/20/2019 01:52 PM, Gesiel Galvão Bernardes wrote:
Hi, I'm testing Ceph with Vmware, using Ceph-iscsi gateway. I reading documentation* and have doubts some points:
- If I understanded, in general terms, for each VMFS datastore in VMware will match the an RBD image. (consequently in an RBD image I will possible have many VMWare disks). Its correct?
- In documentation is this: "gwcli requires a pool with the name rbd, so it can store metadata like the iSCSI configuration". In part 4 of "Configuration", have: "Add a RBD image with the name disk_1 in the pool rbd". In this part, the use of "rbd" pool is a example and I could use any pool for storage of image, or the pool should be "rbd"? Resuming: gwcli require "rbd" pool for metadata and I could use any pool for image, or i will use just "rbd pool" for storage image and metadata?
- How much memory ceph-iscsi use? Which is a good number of RAM?
The major memory use is:
1. In RHEL 7.5 kernels and older we allocate max_data_area_mb of kernel memory per device. The default value for that is 8. You can use gwcli to configure it. It is allocated when the device is created. In newer kernels, there is pool of memory and each device can use up to max_data_area_mb worth of it. The per device default is the same and you can change it with gwcli. The total pool limit is 2 GB. There is a sysfs file:
/sys/module/target_core_user/parameters/global_max_data_area_mb
that can be used to change it.
2. Each device uses about 20 MB of memory in userspace. This is not configurable.
On 10/01/2019 07:32 AM, Gesiel Galvão Bernardes wrote:
Thank you everyone for the explanations.
Still on this subject: I created an host and attached a disk. For a second host, for use a shared iscsi storage, I just need add disk to second client? I tried this:
disk add pool1/vmware_iscsi1 Warning: 'pool1/vmware_iscsi1' mapped to 1 other client(s) ok
It's a problem, or is correct?
Normally you would do this in a host group.
Regards Gesiel
Em sex, 20 de set de 2019 às 20:13, Mike Christie <mchristi@redhat.com <mailto:mchristi@redhat.com>> escreveu:
On 09/20/2019 01:52 PM, Gesiel Galvão Bernardes wrote: > Hi, > I'm testing Ceph with Vmware, using Ceph-iscsi gateway. I reading > documentation* and have doubts some points: > > - If I understanded, in general terms, for each VMFS datastore in VMware > will match the an RBD image. (consequently in an RBD image I will > possible have many VMWare disks). Its correct? > > - In documentation is this: "gwcli requires a pool with the name rbd, so > it can store metadata like the iSCSI configuration". In part 4 of > "Configuration", have: "Add a RBD image with the name disk_1 in the pool > rbd". In this part, the use of "rbd" pool is a example and I could use > any pool for storage of image, or the pool should be "rbd"? > Resuming: gwcli require "rbd" pool for metadata and I could use any pool > for image, or i will use just "rbd pool" for storage image and metadata? > > - How much memory ceph-iscsi use? Which is a good number of RAM? >
The major memory use is:
1. In RHEL 7.5 kernels and older we allocate max_data_area_mb of kernel memory per device. The default value for that is 8. You can use gwcli to configure it. It is allocated when the device is created. In newer kernels, there is pool of memory and each device can use up to max_data_area_mb worth of it. The per device default is the same and you can change it with gwcli. The total pool limit is 2 GB. There is a sysfs file:
/sys/module/target_core_user/parameters/global_max_data_area_mb
that can be used to change it.
2. Each device uses about 20 MB of memory in userspace. This is not configurable.
participants (4)
-
Gesiel Galvão Bernardes
-
Heðin Ejdesgaard Møller
-
Mike Christie
-
Paul Emmerich