27 Aug
2020
27 Aug
'20
8:36 a.m.
Hi ,
I want to know the number of connections in CEPH. I think the connection is mainly OSD
connection.
Is the following statement correct?
Each OSD is connected with other OSDs, and there may be more than one connection between
two OSDs.
If there is only one connection per OSD, the number of the connection is N(N-1)/2,. If
there are k connections per OSD, the number of the connection is kN(N-1)/2.
Thanks for your help.
Best regards.
Congmin Yin
28 Aug
28 Aug
7:56 p.m.
New subject: question: the number of connections in ceph
I you're asking about the osd<->osd connection count for a particular
osd, it'll be proportional (two hearbeat connections and a cluster
connection per pair I think) to the number of osds that osd shares a
pg with (num_pgs/num_osds * (pool_size - 1)).
-Sam
On Thu, Aug 27, 2020 at 1:38 AM Yin, Congmin <congmin.yin(a)intel.com> wrote:
Hi ,
I want to know the number of connections in CEPH. I think the connection is mainly OSD
connection.
Is the following statement correct?
Each OSD is connected with other OSDs, and there may be more than one connection between
two OSDs.
If there is only one connection per OSD, the number of the connection is N(N-1)/2,. If
there are k connections per OSD, the number of the connection is kN(N-1)/2.
Thanks for your help.
Best regards.
Congmin Yin
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
8:24 p.m.
New subject: question: the number of connections in ceph
Last time I read through the code, there were a limited number of “extra” peerings added
to the acting set OSDs too.
But it’s a bit unclear what a “connection” means. And if that includes clients.
On Aug 28, 2020, at 12:56 PM, Sam Just
<sjust(a)redhat.com> wrote:
I you're asking about the osd<->osd connection count for a particular
osd, it'll be proportional (two hearbeat connections and a cluster
connection per pair I think) to the number of osds that osd shares a
pg with (num_pgs/num_osds * (pool_size - 1)).
-Sam
On Thu, Aug 27, 2020 at 1:38 AM Yin, Congmin <congmin.yin(a)intel.com> wrote:
Hi ,
I want to know the number of connections in CEPH. I think the connection is mainly OSD
connection.
Is the following statement correct?
Each OSD is connected with other OSDs, and there may be more than one connection between
two OSDs.
If there is only one connection per OSD, the number of the connection is N(N-1)/2,. If
there are k connections per OSD, the number of the connection is kN(N-1)/2.
Thanks for your help.
Best regards.
Congmin Yin
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
31 Aug
31 Aug
5:25 a.m.
New subject: question: the number of connections in ceph
Hi Anthony,
My main question is the number of connections between OSDs. Of course, CEPH cluster will
have other additional connections.
For example, if I have three cluster servers with 3 replication, and there are 12 OSDs on
them. Then I test with FIO that there are 8 images in a pool. And the pool pgs num is 32.
What is the theoretical connection tree between OSDs
@Sam Just For the example above, per OSD connection number: (num_pgs/num_osds *
(pool_size - 1))= (32/12*(3-1))=5.3? Does this have nothing to do with the number of
threads and images started by the test?
Best regards.
Congmin Yin
-----Original Message-----
From: Anthony D'Atri <anthony.datri(a)gmail.com>
Sent: Saturday, August 29, 2020 4:24 AM
To: Sam Just <sjust(a)redhat.com>
Cc: Yin, Congmin <congmin.yin(a)intel.com>om>; dev(a)ceph.io
Subject: Re: question: the number of connections in ceph
Last time I read through the code, there were a limited number of “extra” peerings added
to the acting set OSDs too.
But it’s a bit unclear what a “connection” means. And if that includes clients.
On Aug 28, 2020, at 12:56 PM, Sam Just
<sjust(a)redhat.com> wrote:
I you're asking about the osd<->osd connection count for a particular
osd, it'll be proportional (two hearbeat connections and a cluster
connection per pair I think) to the number of osds that osd shares a
pg with (num_pgs/num_osds * (pool_size - 1)).
-Sam
On Thu, Aug 27, 2020 at 1:38 AM Yin, Congmin <congmin.yin(a)intel.com> wrote:
Hi ,
I want to know the number of connections in CEPH. I think the connection is mainly OSD
connection.
Is the following statement correct?
Each OSD is connected with other OSDs, and there may be more than one connection between
two OSDs.
If there is only one connection per OSD, the number of the connection is N(N-1)/2,. If
there are k connections per OSD, the number of the connection is kN(N-1)/2.
Thanks for your help.
Best regards.
Congmin Yin
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
7:24 p.m.
New subject: question: the number of connections in ceph
Mmm, I did the math wrong (I think I left out a factor of pool_size).
For 32 pgs over 3 hosts with 12 osds each, each osd should have ~3 pgs
(check ceph pg dump), I think. Each osd should therefore have ~6
peers on the other two hosts (again, ceph pg dump). At 3 connections
per peer, that would be 18 connections. You should be able to observe
this empirically though -- I've probably got the number of
connections/pair wrong.
Btw, 32 pgs for 36 osds is *way* too few. I think there are some
tuning guides around you'll probably want to look at.
-Sam
On Sun, Aug 30, 2020 at 10:26 PM Yin, Congmin <congmin.yin(a)intel.com> wrote:
Hi Anthony,
My main question is the number of connections between OSDs. Of course, CEPH cluster will
have other additional connections.
For example, if I have three cluster servers with 3 replication, and there are 12 OSDs
on them. Then I test with FIO that there are 8 images in a pool. And the pool pgs num is
32. What is the theoretical connection tree between OSDs
@Sam Just For the example above, per OSD connection number: (num_pgs/num_osds *
(pool_size - 1))= (32/12*(3-1))=5.3? Does this have nothing to do with the number of
threads and images started by the test?
Best regards.
Congmin Yin
-----Original Message-----
From: Anthony D'Atri <anthony.datri(a)gmail.com>
Sent: Saturday, August 29, 2020 4:24 AM
To: Sam Just <sjust(a)redhat.com>
Cc: Yin, Congmin <congmin.yin(a)intel.com>om>; dev(a)ceph.io
Subject: Re: question: the number of connections in ceph
Last time I read through the code, there were a limited number of “extra” peerings added
to the acting set OSDs too.
But it’s a bit unclear what a “connection” means. And if that includes clients.
On Aug 28, 2020, at 12:56 PM, Sam Just
<sjust(a)redhat.com> wrote:
I you're asking about the osd<->osd connection count for a particular
osd, it'll be proportional (two hearbeat connections and a cluster
connection per pair I think) to the number of osds that osd shares a
pg with (num_pgs/num_osds * (pool_size - 1)).
-Sam
On Thu, Aug 27, 2020 at 1:38 AM Yin, Congmin <congmin.yin(a)intel.com> wrote:
Hi ,
I want to know the number of connections in CEPH. I think the connection is mainly OSD
connection.
Is the following statement correct?
Each OSD is connected with other OSDs, and there may be more than one connection between
two OSDs.
If there is only one connection per OSD, the number of the connection is N(N-1)/2,. If
there are k connections per OSD, the number of the connection is kN(N-1)/2.
Thanks for your help.
Best regards.
Congmin Yin
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
1 Sep
1 Sep
1:13 a.m.
New subject: question: the number of connections in ceph
Ok, I see. Thanks very much.
-----Original Message-----
From: Sam Just <sjust(a)redhat.com>
Sent: Tuesday, September 1, 2020 3:25 AM
To: Yin, Congmin <congmin.yin(a)intel.com>
Cc: Anthony D'Atri <anthony.datri(a)gmail.com>om>; dev(a)ceph.io; Zhu, Vivian
<vivian.zhu(a)intel.com>
Subject: Re: question: the number of connections in ceph
Mmm, I did the math wrong (I think I left out a factor of pool_size).
For 32 pgs over 3 hosts with 12 osds each, each osd should have ~3 pgs (check ceph pg
dump), I think. Each osd should therefore have ~6 peers on the other two hosts (again,
ceph pg dump). At 3 connections per peer, that would be 18 connections. You should be
able to observe this empirically though -- I've probably got the number of
connections/pair wrong.
Btw, 32 pgs for 36 osds is *way* too few. I think there are some tuning guides around
you'll probably want to look at.
-Sam
On Sun, Aug 30, 2020 at 10:26 PM Yin, Congmin <congmin.yin(a)intel.com> wrote:
Hi Anthony,
My main question is the number of connections between OSDs. Of course, CEPH cluster will
have other additional connections.
For example, if I have three cluster servers with 3 replication, and
there are 12 OSDs on them. Then I test with FIO that there are 8
images in a pool. And the pool pgs num is 32. What is the theoretical
connection tree between OSDs
@Sam Just For the example above, per OSD connection number: (num_pgs/num_osds *
(pool_size - 1))= (32/12*(3-1))=5.3? Does this have nothing to do with the number of
threads and images started by the test?
Best regards.
Congmin Yin
-----Original Message-----
From: Anthony D'Atri <anthony.datri(a)gmail.com>
Sent: Saturday, August 29, 2020 4:24 AM
To: Sam Just <sjust(a)redhat.com>
Cc: Yin, Congmin <congmin.yin(a)intel.com>om>; dev(a)ceph.io
Subject: Re: question: the number of connections in ceph
Last time I read through the code, there were a limited number of “extra” peerings added
to the acting set OSDs too.
But it’s a bit unclear what a “connection” means. And if that includes clients.
On Aug 28, 2020, at 12:56 PM, Sam Just
<sjust(a)redhat.com> wrote:
I you're asking about the osd<->osd connection count for a
particular osd, it'll be proportional (two hearbeat connections and
a cluster connection per pair I think) to the number of osds that
osd shares a pg with (num_pgs/num_osds * (pool_size - 1)).
-Sam
On Thu, Aug 27, 2020 at 1:38 AM Yin, Congmin <congmin.yin(a)intel.com> wrote:
Hi ,
I want to know the number of connections in CEPH. I think the connection is mainly OSD
connection.
Is the following statement correct?
Each OSD is connected with other OSDs, and there may be more than one connection between
two OSDs.
If there is only one connection per OSD, the number of the connection is N(N-1)/2,. If
there are k connections per OSD, the number of the connection is kN(N-1)/2.
Thanks for your help.
Best regards.
Congmin Yin
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
1128
days inactive
1133
days old
5 comments
3 participants
participants (3)
-
Anthony D'Atri -
Sam Just -
Yin, Congmin