Is there any rules for computing RAM requeirements in terms of the number of PGs? Just curious abount what is the fundamental limitations on the number of PGs per OSD for bigger capacity HDD best regards, Samuel huxiaoyu(a)horebdata.cn From: Anthony D'Atri Date: 2020-09-05 20:00 To: huxiaoyu(a)horebdata.cn CC: ceph-users Subject: Re: [ceph-users] PG number per OSD One factor is RAM usage, that was IIRC the motivation for the lowering of the recommendation of the ratio from 200 to 100. Memory needs also increase during recovery and backfill. When calculating, be sure to consider repllicas. ratio = (pgp_num x replication) / num_osds As HDDs grow the interface though isn’t becoming faster (with SATA at least), and there are only so many IOPS and MB/s that you’re going to get out of one no matter how you slice it. Everything always depends on your use-case and workload, but I suspect that often the bottleneck is the drive, not PG or OSD serialization. For example, do you prize IOPS more, latency, or MB/s? If you don’t care about latency, then you can drive your HDDs harder and get more MB/s throughput out of them, though your average latency might climb to 100ms. Which eg. RBD VM clients probably wouldn’t be too happy about, but which an object service *might* tolerate. Basically in the absence of more info, I would personally suggest aiming at the 150-200 average range, with pgp_num a power of 2. If you aim a bit high, the ratio will come down a bit when you add nodes/OSDs to your cluster to gain capacity. Be sure to balance usage and watch your mon_max_pg_per_osd setting — allowing some headroom for natural variation and for when components fail. YMMV. — aad _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io