15 Apr
2024
15 Apr
'24
9:36 a.m.
Hi,
In AWS EBS gp3, AWS says that small volume size cannot achieve best
performance. I think it's a feature or tendency of general
distributed storages including Ceph. Is that right in Ceph block storage? I
read many docs on ceph community. I never heard of Ceph storage.
https://docs.aws.amazon.com/ebs/latest/userguide/general-purpose.html
Regard,
--
Mitsumasa KONDO
15 Apr
15 Apr
11:59 a.m.
Hi,
Volume size doesn't affect performance, cloud providers apply a limit to ensure they
can deliver expected performances to all their customers.
Étienne
________________________________
From: Mitsumasa KONDO <kondo.mitsumasa(a)gmail.com>
Sent: Monday, 15 April 2024 06:06
To: ceph-users(a)ceph.io <ceph-users(a)ceph.io>
Subject: [ceph-users] Performance of volume size, not a block size
[Some people who received this message don't often get email from
kondo.mitsumasa(a)gmail.com. Learn why this is important at
https://aka.ms/LearnAboutSenderIdentification ]
Hi,
In AWS EBS gp3, AWS says that small volume size cannot achieve best
performance. I think it's a feature or tendency of general
distributed storages including Ceph. Is that right in Ceph block storage? I
read many docs on ceph community. I never heard of Ceph storage.
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.aws.…
Regard,
--
Mitsumasa KONDO
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
4:37 p.m.
Hi Menguy-san,
Thank you for your reply. Users who use large IO with tiny volumes are a
nuisance to cloud providers.
I confirmed my ceph cluster with 40 SSDs. Each OSD on 1TB SSD has about 50
placement groups in my cluster. Therefore, each PG has approximately 20GB
of space.
If we create a small 8GB volume, I had a feeling it wouldn't be distributed
well, but it will be distributed well.
Regards,
--
Mitsumasa KONDO
2024年4月15日(月) 15:29 Etienne Menguy <etienne.menguy(a)ubisoft.com>om>:
Hi,
Volume size doesn't affect performance, cloud providers apply a limit to
ensure they can deliver expected performances to all their customers.
Étienne
------------------------------
*From:* Mitsumasa KONDO <kondo.mitsumasa(a)gmail.com>
*Sent:* Monday, 15 April 2024 06:06
*To:* ceph-users(a)ceph.io <ceph-users(a)ceph.io>
*Subject:* [ceph-users] Performance of volume size, not a block size
[Some people who received this message don't often get email from
kondo.mitsumasa(a)gmail.com. Learn why this is important at
https://aka.ms/LearnAboutSenderIdentification ]
Hi,
In AWS EBS gp3, AWS says that small volume size cannot achieve best
performance. I think it's a feature or tendency of general
distributed storages including Ceph. Is that right in Ceph block storage? I
read many docs on ceph community. I never heard of Ceph storage.
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.aws.…
<https://docs.aws.amazon.com/ebs/latest/userguide/general-purpose.html>
Regard,
--
Mitsumasa KONDO
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:28 p.m.
If you're using SATA/SAS SSDs I would aim for 150-200 PGs per OSD as shown by `ceph
osd df`.
If NVMe, 200-300 unless you're starved for RAM.
On Apr 15, 2024, at 07:07, Mitsumasa KONDO
<kondo.mitsumasa(a)gmail.com> wrote:
Hi Menguy-san,
Thank you for your reply. Users who use large IO with tiny volumes are a
nuisance to cloud providers.
I confirmed my ceph cluster with 40 SSDs. Each OSD on 1TB SSD has about 50
placement groups in my cluster. Therefore, each PG has approximately 20GB
of space.
If we create a small 8GB volume, I had a feeling it wouldn't be distributed
well, but it will be distributed well.
Regards,
--
Mitsumasa KONDO
2024年4月15日(月) 15:29 Etienne Menguy <etienne.menguy(a)ubisoft.com>om>:
Hi,
Volume size doesn't affect performance, cloud providers apply a limit to
ensure they can deliver expected performances to all their customers.
Étienne
------------------------------
*From:* Mitsumasa KONDO <kondo.mitsumasa(a)gmail.com>
*Sent:* Monday, 15 April 2024 06:06
*To:* ceph-users(a)ceph.io <ceph-users(a)ceph.io>
*Subject:* [ceph-users] Performance of volume size, not a block size
[Some people who received this message don't often get email from
kondo.mitsumasa(a)gmail.com. Learn why this is important at
https://aka.ms/LearnAboutSenderIdentification ]
Hi,
In AWS EBS gp3, AWS says that small volume size cannot achieve best
performance. I think it's a feature or tendency of general
distributed storages including Ceph. Is that right in Ceph block storage? I
read many docs on ceph community. I never heard of Ceph storage.
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.aws.…
<https://docs.aws.amazon.com/ebs/latest/userguide/general-purpose.html>
Regard,
--
Mitsumasa KONDO
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
16 Apr
16 Apr
8:47 a.m.
Hi Anthony-san,
Thank you for your advice. I confirm my settings of my ceph cluster.
Autoscaler mode is on, so I had thought it's the best PGs. But the
autoscaler feature doesn't affect OSD's PGs. It's just for PG_NUM in
storage pools. Is that right?
Regards,
--
Mitsumasa KONDO
2024年4月15日(月) 22:58 Anthony D'Atri <anthony.datri(a)gmail.com>om>:
If you're using SATA/SAS SSDs I would aim for
150-200 PGs per OSD as shown
by `ceph osd df`.
If NVMe, 200-300 unless you're starved for RAM.
On Apr 15, 2024, at 07:07, Mitsumasa KONDO
<kondo.mitsumasa(a)gmail.com>
wrote:
Hi Menguy-san,
Thank you for your reply. Users who use large IO with tiny volumes are a
nuisance to cloud providers.
I confirmed my ceph cluster with 40 SSDs. Each OSD on 1TB SSD has about
50
placement groups in my cluster. Therefore, each
PG has approximately 20GB
of space.
If we create a small 8GB volume, I had a feeling it wouldn't be
distributed
well, but it will be distributed well.
Regards,
--
Mitsumasa KONDO
2024年4月15日(月) 15:29 Etienne Menguy <etienne.menguy(a)ubisoft.com>om>:
> Hi,
>
> Volume size doesn't affect performance, cloud providers apply a limit to
> ensure they can deliver expected performances to all their customers.
>
> Étienne
> ------------------------------
> *From:* Mitsumasa KONDO <kondo.mitsumasa(a)gmail.com>
> *Sent:* Monday, 15 April 2024 06:06
> *To:* ceph-users(a)ceph.io <ceph-users(a)ceph.io>
> *Subject:* [ceph-users] Performance of volume size, not a block size
>
> [Some people who received this message don't often get email from
> kondo.mitsumasa(a)gmail.com. Learn why this is important at
> https://aka.ms/LearnAboutSenderIdentification ]
>
> Hi,
>
> In AWS EBS gp3, AWS says that small volume size cannot achieve best
> performance. I think it's a feature or tendency of general
> distributed storages including Ceph. Is that right in Ceph block
storage? I
> read many docs on ceph community. I never
heard of Ceph storage.
>
>
>
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.aws.…
<https://docs.aws.amazon.com/ebs/latest/userguide/general-purpose.html>
Regard,
--
Mitsumasa KONDO
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io 
12:05 p.m.
Den mån 15 apr. 2024 kl 13:09 skrev Mitsumasa KONDO <kondo.mitsumasa(a)gmail.com>om>:
Hi Menguy-san,
Thank you for your reply. Users who use large IO with tiny volumes are a
nuisance to cloud providers.
I confirmed my ceph cluster with 40 SSDs. Each OSD on 1TB SSD has about 50
placement groups in my cluster. Therefore, each PG has approximately 20GB
of space.
If we create a small 8GB volume, I had a feeling it wouldn't be distributed
well, but it will be distributed well.
RBD images get split into 2 or 4M pieces when stored in ceph, so an 8G
RBD image will be split into 2048-or-4096 separate pieces that end up
"randomly" on the PGs the pool is based on, which means that if you
read or write the whole RBD image from start to end, you are going to
spread the load to all OSDs.
I think it works something like this, you ask librbd for an 8G image
named "myimage", and underneath it makes myimage.0, myimage.1, 2,3,4
and so on. The PG placement will depend on the object name, which of
course differs for all the pieces, and hence they end up on different
PGs, thereby spreading the load. If ceph did not do this, then you
could never make an RBD image that was larger than "smallest free
space on any of the pools OSDs" but also, it would mean that the RBD
client would be talking to the same single OSD for everything, and
that would not be a good way to use a clusters resources evenly.
--
May the most significant bit of your life be positive.
17 Apr
17 Apr
3:56 p.m.
Hi Johansson-san,
Thank you very much for your detailed explanation. I read some documents in
Ceph community. So I generally understand. Thank you very much for all the
useful advice. The characteristics of distributed storage seem to be quite
complex, so I will investigate various things when I have time.
Regards,
--
Mitsumasa KONDO
2024年4月16日(火) 15:35 Janne Johansson <icepic.dz(a)gmail.com>om>:
Den mån 15 apr. 2024 kl 13:09 skrev Mitsumasa KONDO
<
kondo.mitsumasa(a)gmail.com>gt;:
Hi Menguy-san,
Thank you for your reply. Users who use large IO with tiny volumes are a
nuisance to cloud providers.
I confirmed my ceph cluster with 40 SSDs. Each OSD on 1TB SSD has about
50
placement groups in my cluster. Therefore, each
PG has approximately 20GB
of space.
If we create a small 8GB volume, I had a feeling it wouldn't be
distributed
well, but it will be distributed well.
RBD images get split into 2 or 4M pieces when stored in ceph, so an 8G
RBD image will be split into 2048-or-4096 separate pieces that end up
"randomly" on the PGs the pool is based on, which means that if you
read or write the whole RBD image from start to end, you are going to
spread the load to all OSDs.
I think it works something like this, you ask librbd for an 8G image
named "myimage", and underneath it makes myimage.0, myimage.1, 2,3,4
and so on. The PG placement will depend on the object name, which of
course differs for all the pieces, and hence they end up on different
PGs, thereby spreading the load. If ceph did not do this, then you
could never make an RBD image that was larger than "smallest free
space on any of the pools OSDs" but also, it would mean that the RBD
client would be talking to the same single OSD for everything, and
that would not be a good way to use a clusters resources evenly.
--
May the most significant bit of your life be positive.
15
days inactive
17
days old
6 comments
4 participants
participants (4)
-
Anthony D'Atri -
Etienne Menguy -
Janne Johansson -
Mitsumasa KONDO