Ok, I see. Thanks very much. -----Original Message----- From: Sam Just <sjust@redhat.com> Sent: Tuesday, September 1, 2020 3:25 AM To: Yin, Congmin <congmin.yin@intel.com> Cc: Anthony D'Atri <anthony.datri@gmail.com>; dev@ceph.io; Zhu, Vivian <vivian.zhu@intel.com> Subject: Re: question: the number of connections in ceph Mmm, I did the math wrong (I think I left out a factor of pool_size). For 32 pgs over 3 hosts with 12 osds each, each osd should have ~3 pgs (check ceph pg dump), I think. Each osd should therefore have ~6 peers on the other two hosts (again, ceph pg dump). At 3 connections per peer, that would be 18 connections. You should be able to observe this empirically though -- I've probably got the number of connections/pair wrong. Btw, 32 pgs for 36 osds is *way* too few. I think there are some tuning guides around you'll probably want to look at. -Sam On Sun, Aug 30, 2020 at 10:26 PM Yin, Congmin <congmin.yin@intel.com> wrote:
Hi Anthony, My main question is the number of connections between OSDs. Of course, CEPH cluster will have other additional connections. For example, if I have three cluster servers with 3 replication, and there are 12 OSDs on them. Then I test with FIO that there are 8 images in a pool. And the pool pgs num is 32. What is the theoretical connection tree between OSDs
@Sam Just For the example above, per OSD connection number: (num_pgs/num_osds * (pool_size - 1))= (32/12*(3-1))=5.3? Does this have nothing to do with the number of threads and images started by the test?
Best regards. Congmin Yin
-----Original Message----- From: Anthony D'Atri <anthony.datri@gmail.com> Sent: Saturday, August 29, 2020 4:24 AM To: Sam Just <sjust@redhat.com> Cc: Yin, Congmin <congmin.yin@intel.com>; dev@ceph.io Subject: Re: question: the number of connections in ceph
Last time I read through the code, there were a limited number of “extra” peerings added to the acting set OSDs too.
But it’s a bit unclear what a “connection” means. And if that includes clients.
On Aug 28, 2020, at 12:56 PM, Sam Just <sjust@redhat.com> wrote:
I you're asking about the osd<->osd connection count for a particular osd, it'll be proportional (two hearbeat connections and a cluster connection per pair I think) to the number of osds that osd shares a pg with (num_pgs/num_osds * (pool_size - 1)). -Sam
On Thu, Aug 27, 2020 at 1:38 AM Yin, Congmin <congmin.yin@intel.com> wrote:
Hi ,
I want to know the number of connections in CEPH. I think the connection is mainly OSD connection.
Is the following statement correct?
Each OSD is connected with other OSDs, and there may be more than one connection between two OSDs.
If there is only one connection per OSD, the number of the connection is N(N-1)/2,. If there are k connections per OSD, the number of the connection is kN(N-1)/2.
Thanks for your help.
Best regards.
Congmin Yin
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io