6 pgs not deep-scrubbed in time
Hello team, I have a cluster in production composed by 3 osds servers with 20 disks each deployed using ceph-ansibleand ubuntu OS , and the version is pacific . These days is in WARN state caused by pgs which are not deep-scrubbed in time . I tried to deep-scrubbed some pg manually but seems that the cluster can be slow, would like your assistance in order that my cluster can be in HEALTH_OK state as before without any interuption of service . The cluster is used as openstack backend storage. Best Regards Michel ceph -s cluster: id: cb0caedc-eb5b-42d1-a34f-96facfda8c27 health: HEALTH_WARN 6 pgs not deep-scrubbed in time services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M) mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1 osd: 48 osds: 48 up (since 11M), 48 in (since 11M) rgw: 6 daemons active (6 hosts, 1 zones) data: pools: 10 pools, 385 pgs objects: 5.97M objects, 23 TiB usage: 151 TiB used, 282 TiB / 433 TiB avail pgs: 381 active+clean 4 active+clean+scrubbing+deep io: client: 59 MiB/s rd, 860 MiB/s wr, 155 op/s rd, 665 op/s wr root@ceph-osd3:~# ceph health detail HEALTH_WARN 6 pgs not deep-scrubbed in time [WRN] PG_NOT_DEEP_SCRUBBED: 6 pgs not deep-scrubbed in time pg 6.78 not deep-scrubbed since 2024-01-11T16:07:54.875746+0200 pg 6.60 not deep-scrubbed since 2024-01-13T19:44:26.922000+0200 pg 6.5c not deep-scrubbed since 2024-01-13T09:07:24.780936+0200 pg 4.12 not deep-scrubbed since 2024-01-13T09:09:22.176240+0200 pg 10.d not deep-scrubbed since 2024-01-12T08:04:02.078062+0200 pg 5.f not deep-scrubbed since 2024-01-12T06:06:00.970665+0200
We had the same problem. It turned out that one disk was slowly dying. It was easy to identify by the commands (in your case): ceph pg dump | grep -F 6.78 ceph pg dump | grep -F 6.60 … This command shows the OSDs of a PG in square brackets. If is there always the same number, then you've found the OSD which causes the slow scrubs. Am Fr., 26. Jan. 2024 um 07:45 Uhr schrieb Michel Niyoyita < micou12@gmail.com>:
Hello team,
I have a cluster in production composed by 3 osds servers with 20 disks each deployed using ceph-ansibleand ubuntu OS , and the version is pacific . These days is in WARN state caused by pgs which are not deep-scrubbed in time . I tried to deep-scrubbed some pg manually but seems that the cluster can be slow, would like your assistance in order that my cluster can be in HEALTH_OK state as before without any interuption of service . The cluster is used as openstack backend storage.
Best Regards
Michel
ceph -s cluster: id: cb0caedc-eb5b-42d1-a34f-96facfda8c27 health: HEALTH_WARN 6 pgs not deep-scrubbed in time
services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M) mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1 osd: 48 osds: 48 up (since 11M), 48 in (since 11M) rgw: 6 daemons active (6 hosts, 1 zones)
data: pools: 10 pools, 385 pgs objects: 5.97M objects, 23 TiB usage: 151 TiB used, 282 TiB / 433 TiB avail pgs: 381 active+clean 4 active+clean+scrubbing+deep
io: client: 59 MiB/s rd, 860 MiB/s wr, 155 op/s rd, 665 op/s wr
root@ceph-osd3:~# ceph health detail HEALTH_WARN 6 pgs not deep-scrubbed in time [WRN] PG_NOT_DEEP_SCRUBBED: 6 pgs not deep-scrubbed in time pg 6.78 not deep-scrubbed since 2024-01-11T16:07:54.875746+0200 pg 6.60 not deep-scrubbed since 2024-01-13T19:44:26.922000+0200 pg 6.5c not deep-scrubbed since 2024-01-13T09:07:24.780936+0200 pg 4.12 not deep-scrubbed since 2024-01-13T09:09:22.176240+0200 pg 10.d not deep-scrubbed since 2024-01-12T08:04:02.078062+0200 pg 5.f not deep-scrubbed since 2024-01-12T06:06:00.970665+0200 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
It seems that are different OSDs as shown here . how have you managed to sort this out? ceph pg dump | grep -F 6.78 dumped all 6.78 44268 0 0 0 0 178679640118 0 0 10099 10099 active+clean 2024-01-26T03:51:26.781438+0200 107547'115445304 107547:225274427 [12,36,37] 12 [12,36,37] 12 106977'114532385 2024-01-24T08:37:53.597331+0200 101161'109078277 2024-01-11T16:07:54.875746+0200 0 root@ceph-osd3:~# ceph pg dump | grep -F 6.60 dumped all 6.60 44449 0 0 0 0 179484338742 716 36 10097 10097 active+clean 2024-01-26T03:50:44.579831+0200 107547'153238805 107547:287193139 [32,5,29] 32 [32,5,29] 32 107231'152689835 2024-01-25T02:34:01.849966+0200 102171'147920798 2024-01-13T19:44:26.922000+0200 0 6.3a 44807 0 0 0 0 180969005694 0 0 10093 10093 active+clean 2024-01-26T03:53:28.837685+0200 107547'114765984 107547:238170093 [22,13,11] 22 [22,13,11] 22 106945'113739877 2024-01-24T04:10:17.224982+0200 102863'109559444 2024-01-15T05:31:36.606478+0200 0 root@ceph-osd3:~# ceph pg dump | grep -F 6.5c 6.5c 44277 0 0 0 0 178764978230 0 0 10051 10051 active+clean 2024-01-26T03:55:23.339584+0200 107547'126480090 107547:264432655 [22,37,30] 22 [22,37,30] 22 107205'125858697 2024-01-24T22:32:10.365869+0200 101941'120957992 2024-01-13T09:07:24.780936+0200 0 dumped all root@ceph-osd3:~# ceph pg dump | grep -F 4.12 dumped all 4.12 0 0 0 0 0 0 0 0 0 0 active+clean 2024-01-24T08:36:48.284388+0200 0'0 107546:152711 [22,19,7] 22 [22,19,7] 22 0'0 2024-01-24T08:36:48.284307+0200 0'0 2024-01-13T09:09:22.176240+0200 0 root@ceph-osd3:~# ceph pg dump | grep -F 10.d dumped all 10.d 0 0 0 0 0 0 0 0 0 0 active+clean 2024-01-24T04:04:33.641541+0200 0'0 107546:142651 [14,28,1] 14 [14,28,1] 14 0'0 2024-01-24T04:04:33.641451+0200 0'0 2024-01-12T08:04:02.078062+0200 0 root@ceph-osd3:~# ceph pg dump | grep -F 5.f dumped all 5.f 0 0 0 0 0 0 0 0 0 0 active+clean 2024-01-25T08:19:04.148941+0200 0'0 107546:161331 [11,24,35] 11 [11,24,35] 11 0'0 2024-01-25T08:19:04.148837+0200 0'0 2024-01-12T06:06:00.970665+0200 0 On Fri, Jan 26, 2024 at 8:58 AM E Taka <0etaka0@gmail.com> wrote:
We had the same problem. It turned out that one disk was slowly dying. It was easy to identify by the commands (in your case):
ceph pg dump | grep -F 6.78 ceph pg dump | grep -F 6.60 …
This command shows the OSDs of a PG in square brackets. If is there always the same number, then you've found the OSD which causes the slow scrubs.
Am Fr., 26. Jan. 2024 um 07:45 Uhr schrieb Michel Niyoyita < micou12@gmail.com>:
Hello team,
I have a cluster in production composed by 3 osds servers with 20 disks each deployed using ceph-ansibleand ubuntu OS , and the version is pacific . These days is in WARN state caused by pgs which are not deep-scrubbed in time . I tried to deep-scrubbed some pg manually but seems that the cluster can be slow, would like your assistance in order that my cluster can be in HEALTH_OK state as before without any interuption of service . The cluster is used as openstack backend storage.
Best Regards
Michel
ceph -s cluster: id: cb0caedc-eb5b-42d1-a34f-96facfda8c27 health: HEALTH_WARN 6 pgs not deep-scrubbed in time
services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M) mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1 osd: 48 osds: 48 up (since 11M), 48 in (since 11M) rgw: 6 daemons active (6 hosts, 1 zones)
data: pools: 10 pools, 385 pgs objects: 5.97M objects, 23 TiB usage: 151 TiB used, 282 TiB / 433 TiB avail pgs: 381 active+clean 4 active+clean+scrubbing+deep
io: client: 59 MiB/s rd, 860 MiB/s wr, 155 op/s rd, 665 op/s wr
root@ceph-osd3:~# ceph health detail HEALTH_WARN 6 pgs not deep-scrubbed in time [WRN] PG_NOT_DEEP_SCRUBBED: 6 pgs not deep-scrubbed in time pg 6.78 not deep-scrubbed since 2024-01-11T16:07:54.875746+0200 pg 6.60 not deep-scrubbed since 2024-01-13T19:44:26.922000+0200 pg 6.5c not deep-scrubbed since 2024-01-13T09:07:24.780936+0200 pg 4.12 not deep-scrubbed since 2024-01-13T09:09:22.176240+0200 pg 10.d not deep-scrubbed since 2024-01-12T08:04:02.078062+0200 pg 5.f not deep-scrubbed since 2024-01-12T06:06:00.970665+0200 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
22 is more often there than the others. Other operations may be blocked because of a deep-scrub is not finished yet. I would remove OSD 22, just to be sure about this: ceph orch osd rm osd.22 If this does not help, just add it again. Am Fr., 26. Jan. 2024 um 08:05 Uhr schrieb Michel Niyoyita < micou12@gmail.com>:
It seems that are different OSDs as shown here . how have you managed to sort this out?
ceph pg dump | grep -F 6.78 dumped all 6.78 44268 0 0 0 0 178679640118 0 0 10099 10099 active+clean 2024-01-26T03:51:26.781438+0200 107547'115445304 107547:225274427 [12,36,37] 12 [12,36,37] 12 106977'114532385 2024-01-24T08:37:53.597331+0200 101161'109078277 2024-01-11T16:07:54.875746+0200 0 root@ceph-osd3:~# ceph pg dump | grep -F 6.60 dumped all 6.60 44449 0 0 0 0 179484338742 716 36 10097 10097 active+clean 2024-01-26T03:50:44.579831+0200 107547'153238805 107547:287193139 [32,5,29] 32 [32,5,29] 32 107231'152689835 2024-01-25T02:34:01.849966+0200 102171'147920798 2024-01-13T19:44:26.922000+0200 0 6.3a 44807 0 0 0 0 180969005694 0 0 10093 10093 active+clean 2024-01-26T03:53:28.837685+0200 107547'114765984 107547:238170093 [22,13,11] 22 [22,13,11] 22 106945'113739877 2024-01-24T04:10:17.224982+0200 102863'109559444 2024-01-15T05:31:36.606478+0200 0 root@ceph-osd3:~# ceph pg dump | grep -F 6.5c 6.5c 44277 0 0 0 0 178764978230 0 0 10051 10051 active+clean 2024-01-26T03:55:23.339584+0200 107547'126480090 107547:264432655 [22,37,30] 22 [22,37,30] 22 107205'125858697 2024-01-24T22:32:10.365869+0200 101941'120957992 2024-01-13T09:07:24.780936+0200 0 dumped all root@ceph-osd3:~# ceph pg dump | grep -F 4.12 dumped all 4.12 0 0 0 0 0 0 0 0 0 0 active+clean 2024-01-24T08:36:48.284388+0200 0'0 107546:152711 [22,19,7] 22 [22,19,7] 22 0'0 2024-01-24T08:36:48.284307+0200 0'0 2024-01-13T09:09:22.176240+0200 0 root@ceph-osd3:~# ceph pg dump | grep -F 10.d dumped all 10.d 0 0 0 0 0 0 0 0 0 0 active+clean 2024-01-24T04:04:33.641541+0200 0'0 107546:142651 [14,28,1] 14 [14,28,1] 14 0'0 2024-01-24T04:04:33.641451+0200 0'0 2024-01-12T08:04:02.078062+0200 0 root@ceph-osd3:~# ceph pg dump | grep -F 5.f dumped all 5.f 0 0 0 0 0 0 0 0 0 0 active+clean 2024-01-25T08:19:04.148941+0200 0'0 107546:161331 [11,24,35] 11 [11,24,35] 11 0'0 2024-01-25T08:19:04.148837+0200 0'0 2024-01-12T06:06:00.970665+0200 0
On Fri, Jan 26, 2024 at 8:58 AM E Taka <0etaka0@gmail.com> wrote:
We had the same problem. It turned out that one disk was slowly dying. It was easy to identify by the commands (in your case):
ceph pg dump | grep -F 6.78 ceph pg dump | grep -F 6.60 …
This command shows the OSDs of a PG in square brackets. If is there always the same number, then you've found the OSD which causes the slow scrubs.
Am Fr., 26. Jan. 2024 um 07:45 Uhr schrieb Michel Niyoyita < micou12@gmail.com>:
Hello team,
I have a cluster in production composed by 3 osds servers with 20 disks each deployed using ceph-ansibleand ubuntu OS , and the version is pacific . These days is in WARN state caused by pgs which are not deep-scrubbed in time . I tried to deep-scrubbed some pg manually but seems that the cluster can be slow, would like your assistance in order that my cluster can be in HEALTH_OK state as before without any interuption of service . The cluster is used as openstack backend storage.
Best Regards
Michel
ceph -s cluster: id: cb0caedc-eb5b-42d1-a34f-96facfda8c27 health: HEALTH_WARN 6 pgs not deep-scrubbed in time
services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M) mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1 osd: 48 osds: 48 up (since 11M), 48 in (since 11M) rgw: 6 daemons active (6 hosts, 1 zones)
data: pools: 10 pools, 385 pgs objects: 5.97M objects, 23 TiB usage: 151 TiB used, 282 TiB / 433 TiB avail pgs: 381 active+clean 4 active+clean+scrubbing+deep
io: client: 59 MiB/s rd, 860 MiB/s wr, 155 op/s rd, 665 op/s wr
root@ceph-osd3:~# ceph health detail HEALTH_WARN 6 pgs not deep-scrubbed in time [WRN] PG_NOT_DEEP_SCRUBBED: 6 pgs not deep-scrubbed in time pg 6.78 not deep-scrubbed since 2024-01-11T16:07:54.875746+0200 pg 6.60 not deep-scrubbed since 2024-01-13T19:44:26.922000+0200 pg 6.5c not deep-scrubbed since 2024-01-13T09:07:24.780936+0200 pg 4.12 not deep-scrubbed since 2024-01-13T09:09:22.176240+0200 pg 10.d not deep-scrubbed since 2024-01-12T08:04:02.078062+0200 pg 5.f not deep-scrubbed since 2024-01-12T06:06:00.970665+0200 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Now they are increasing , Friday I tried to deep-scrubbing manually and they have been successfully done , but Monday morning I found that they are increasing to 37 , is it the best to deep-scrubbing manually while we are using the cluster? if not what is the best to do in order to address that . Best Regards. Michel ceph -s cluster: id: cb0caedc-eb5b-42d1-a34f-96facfda8c27 health: HEALTH_WARN 37 pgs not deep-scrubbed in time services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M) mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1 osd: 48 osds: 48 up (since 11M), 48 in (since 11M) rgw: 6 daemons active (6 hosts, 1 zones) data: pools: 10 pools, 385 pgs objects: 6.00M objects, 23 TiB usage: 151 TiB used, 282 TiB / 433 TiB avail pgs: 381 active+clean 4 active+clean+scrubbing+deep io: client: 265 MiB/s rd, 786 MiB/s wr, 3.87k op/s rd, 699 op/s wr On Sun, Jan 28, 2024 at 6:14 PM E Taka <0etaka0@gmail.com> wrote:
22 is more often there than the others. Other operations may be blocked because of a deep-scrub is not finished yet. I would remove OSD 22, just to be sure about this: ceph orch osd rm osd.22
If this does not help, just add it again.
Am Fr., 26. Jan. 2024 um 08:05 Uhr schrieb Michel Niyoyita < micou12@gmail.com>:
It seems that are different OSDs as shown here . how have you managed to sort this out?
ceph pg dump | grep -F 6.78 dumped all 6.78 44268 0 0 0 0 178679640118 0 0 10099 10099 active+clean 2024-01-26T03:51:26.781438+0200 107547'115445304 107547:225274427 [12,36,37] 12 [12,36,37] 12 106977'114532385 2024-01-24T08:37:53.597331+0200 101161'109078277 2024-01-11T16:07:54.875746+0200 0 root@ceph-osd3:~# ceph pg dump | grep -F 6.60 dumped all 6.60 44449 0 0 0 0 179484338742 716 36 10097 10097 active+clean 2024-01-26T03:50:44.579831+0200 107547'153238805 107547:287193139 [32,5,29] 32 [32,5,29] 32 107231'152689835 2024-01-25T02:34:01.849966+0200 102171'147920798 2024-01-13T19:44:26.922000+0200 0 6.3a 44807 0 0 0 0 180969005694 0 0 10093 10093 active+clean 2024-01-26T03:53:28.837685+0200 107547'114765984 107547:238170093 [22,13,11] 22 [22,13,11] 22 106945'113739877 2024-01-24T04:10:17.224982+0200 102863'109559444 2024-01-15T05:31:36.606478+0200 0 root@ceph-osd3:~# ceph pg dump | grep -F 6.5c 6.5c 44277 0 0 0 0 178764978230 0 0 10051 10051 active+clean 2024-01-26T03:55:23.339584+0200 107547'126480090 107547:264432655 [22,37,30] 22 [22,37,30] 22 107205'125858697 2024-01-24T22:32:10.365869+0200 101941'120957992 2024-01-13T09:07:24.780936+0200 0 dumped all root@ceph-osd3:~# ceph pg dump | grep -F 4.12 dumped all 4.12 0 0 0 0 0 0 0 0 0 0 active+clean 2024-01-24T08:36:48.284388+0200 0'0 107546:152711 [22,19,7] 22 [22,19,7] 22 0'0 2024-01-24T08:36:48.284307+0200 0'0 2024-01-13T09:09:22.176240+0200 0 root@ceph-osd3:~# ceph pg dump | grep -F 10.d dumped all 10.d 0 0 0 0 0 0 0 0 0 0 active+clean 2024-01-24T04:04:33.641541+0200 0'0 107546:142651 [14,28,1] 14 [14,28,1] 14 0'0 2024-01-24T04:04:33.641451+0200 0'0 2024-01-12T08:04:02.078062+0200 0 root@ceph-osd3:~# ceph pg dump | grep -F 5.f dumped all 5.f 0 0 0 0 0 0 0 0 0 0 active+clean 2024-01-25T08:19:04.148941+0200 0'0 107546:161331 [11,24,35] 11 [11,24,35] 11 0'0 2024-01-25T08:19:04.148837+0200 0'0 2024-01-12T06:06:00.970665+0200 0
On Fri, Jan 26, 2024 at 8:58 AM E Taka <0etaka0@gmail.com> wrote:
We had the same problem. It turned out that one disk was slowly dying. It was easy to identify by the commands (in your case):
ceph pg dump | grep -F 6.78 ceph pg dump | grep -F 6.60 …
This command shows the OSDs of a PG in square brackets. If is there always the same number, then you've found the OSD which causes the slow scrubs.
Am Fr., 26. Jan. 2024 um 07:45 Uhr schrieb Michel Niyoyita < micou12@gmail.com>:
Hello team,
I have a cluster in production composed by 3 osds servers with 20 disks each deployed using ceph-ansibleand ubuntu OS , and the version is pacific . These days is in WARN state caused by pgs which are not deep-scrubbed in time . I tried to deep-scrubbed some pg manually but seems that the cluster can be slow, would like your assistance in order that my cluster can be in HEALTH_OK state as before without any interuption of service . The cluster is used as openstack backend storage.
Best Regards
Michel
ceph -s cluster: id: cb0caedc-eb5b-42d1-a34f-96facfda8c27 health: HEALTH_WARN 6 pgs not deep-scrubbed in time
services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M) mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1 osd: 48 osds: 48 up (since 11M), 48 in (since 11M) rgw: 6 daemons active (6 hosts, 1 zones)
data: pools: 10 pools, 385 pgs objects: 5.97M objects, 23 TiB usage: 151 TiB used, 282 TiB / 433 TiB avail pgs: 381 active+clean 4 active+clean+scrubbing+deep
io: client: 59 MiB/s rd, 860 MiB/s wr, 155 op/s rd, 665 op/s wr
root@ceph-osd3:~# ceph health detail HEALTH_WARN 6 pgs not deep-scrubbed in time [WRN] PG_NOT_DEEP_SCRUBBED: 6 pgs not deep-scrubbed in time pg 6.78 not deep-scrubbed since 2024-01-11T16:07:54.875746+0200 pg 6.60 not deep-scrubbed since 2024-01-13T19:44:26.922000+0200 pg 6.5c not deep-scrubbed since 2024-01-13T09:07:24.780936+0200 pg 4.12 not deep-scrubbed since 2024-01-13T09:09:22.176240+0200 pg 10.d not deep-scrubbed since 2024-01-12T08:04:02.078062+0200 pg 5.f not deep-scrubbed since 2024-01-12T06:06:00.970665+0200 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Michel, are your OSDs HDD or SSD? If they are HDD, its possible that they can't handle the deep-scrub load with default settings. In that case, have a look at this post https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YUHWQCDAKP5M... for some basic tuning info and a script to check your scrub stamp distribution. You should also get rid of slow/failing ones. Look at smartctl output and throw out disks with remapped sectors, uncorrectable r/w errors or unusually many corrected read-write-errors (assuming you have disks with ECC). A basic calculation for deep-scrub is as follows: max number of PGs that can be scrubbed at the same time: A=#OSDs/replication factor (rounded down). Take the B=deep-scrub times from the OSD logs (grep for deep-scrub) in minutes. Your pool can deep-scrub at a max A*24*(60/B) PGs per day. For reasonable operations you should not do more than 50% of that. With that you can calculate how many days it needs to deep-scrub your PGs. Usual reasons for slow deep-scrub progress is too few PGs. With replication factor 3 and 48 OSDs you have a PG budget of ca. 3200 (ca 200/OSD) but use only 385. You should consider increasing the PG count for pools with lots of data. This should already relax the situation somewhat. Then do the calc above and tune deep-scrub times per pool such that they match with disk performance. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Michel Niyoyita <micou12@gmail.com> Sent: Monday, January 29, 2024 7:42 AM To: E Taka Cc: ceph-users Subject: [ceph-users] Re: 6 pgs not deep-scrubbed in time Now they are increasing , Friday I tried to deep-scrubbing manually and they have been successfully done , but Monday morning I found that they are increasing to 37 , is it the best to deep-scrubbing manually while we are using the cluster? if not what is the best to do in order to address that . Best Regards. Michel ceph -s cluster: id: cb0caedc-eb5b-42d1-a34f-96facfda8c27 health: HEALTH_WARN 37 pgs not deep-scrubbed in time services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M) mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1 osd: 48 osds: 48 up (since 11M), 48 in (since 11M) rgw: 6 daemons active (6 hosts, 1 zones) data: pools: 10 pools, 385 pgs objects: 6.00M objects, 23 TiB usage: 151 TiB used, 282 TiB / 433 TiB avail pgs: 381 active+clean 4 active+clean+scrubbing+deep io: client: 265 MiB/s rd, 786 MiB/s wr, 3.87k op/s rd, 699 op/s wr On Sun, Jan 28, 2024 at 6:14 PM E Taka <0etaka0@gmail.com> wrote:
22 is more often there than the others. Other operations may be blocked because of a deep-scrub is not finished yet. I would remove OSD 22, just to be sure about this: ceph orch osd rm osd.22
If this does not help, just add it again.
Am Fr., 26. Jan. 2024 um 08:05 Uhr schrieb Michel Niyoyita < micou12@gmail.com>:
It seems that are different OSDs as shown here . how have you managed to sort this out?
ceph pg dump | grep -F 6.78 dumped all 6.78 44268 0 0 0 0 178679640118 0 0 10099 10099 active+clean 2024-01-26T03:51:26.781438+0200 107547'115445304 107547:225274427 [12,36,37] 12 [12,36,37] 12 106977'114532385 2024-01-24T08:37:53.597331+0200 101161'109078277 2024-01-11T16:07:54.875746+0200 0 root@ceph-osd3:~# ceph pg dump | grep -F 6.60 dumped all 6.60 44449 0 0 0 0 179484338742 716 36 10097 10097 active+clean 2024-01-26T03:50:44.579831+0200 107547'153238805 107547:287193139 [32,5,29] 32 [32,5,29] 32 107231'152689835 2024-01-25T02:34:01.849966+0200 102171'147920798 2024-01-13T19:44:26.922000+0200 0 6.3a 44807 0 0 0 0 180969005694 0 0 10093 10093 active+clean 2024-01-26T03:53:28.837685+0200 107547'114765984 107547:238170093 [22,13,11] 22 [22,13,11] 22 106945'113739877 2024-01-24T04:10:17.224982+0200 102863'109559444 2024-01-15T05:31:36.606478+0200 0 root@ceph-osd3:~# ceph pg dump | grep -F 6.5c 6.5c 44277 0 0 0 0 178764978230 0 0 10051 10051 active+clean 2024-01-26T03:55:23.339584+0200 107547'126480090 107547:264432655 [22,37,30] 22 [22,37,30] 22 107205'125858697 2024-01-24T22:32:10.365869+0200 101941'120957992 2024-01-13T09:07:24.780936+0200 0 dumped all root@ceph-osd3:~# ceph pg dump | grep -F 4.12 dumped all 4.12 0 0 0 0 0 0 0 0 0 0 active+clean 2024-01-24T08:36:48.284388+0200 0'0 107546:152711 [22,19,7] 22 [22,19,7] 22 0'0 2024-01-24T08:36:48.284307+0200 0'0 2024-01-13T09:09:22.176240+0200 0 root@ceph-osd3:~# ceph pg dump | grep -F 10.d dumped all 10.d 0 0 0 0 0 0 0 0 0 0 active+clean 2024-01-24T04:04:33.641541+0200 0'0 107546:142651 [14,28,1] 14 [14,28,1] 14 0'0 2024-01-24T04:04:33.641451+0200 0'0 2024-01-12T08:04:02.078062+0200 0 root@ceph-osd3:~# ceph pg dump | grep -F 5.f dumped all 5.f 0 0 0 0 0 0 0 0 0 0 active+clean 2024-01-25T08:19:04.148941+0200 0'0 107546:161331 [11,24,35] 11 [11,24,35] 11 0'0 2024-01-25T08:19:04.148837+0200 0'0 2024-01-12T06:06:00.970665+0200 0
On Fri, Jan 26, 2024 at 8:58 AM E Taka <0etaka0@gmail.com> wrote:
We had the same problem. It turned out that one disk was slowly dying. It was easy to identify by the commands (in your case):
ceph pg dump | grep -F 6.78 ceph pg dump | grep -F 6.60 …
This command shows the OSDs of a PG in square brackets. If is there always the same number, then you've found the OSD which causes the slow scrubs.
Am Fr., 26. Jan. 2024 um 07:45 Uhr schrieb Michel Niyoyita < micou12@gmail.com>:
Hello team,
I have a cluster in production composed by 3 osds servers with 20 disks each deployed using ceph-ansibleand ubuntu OS , and the version is pacific . These days is in WARN state caused by pgs which are not deep-scrubbed in time . I tried to deep-scrubbed some pg manually but seems that the cluster can be slow, would like your assistance in order that my cluster can be in HEALTH_OK state as before without any interuption of service . The cluster is used as openstack backend storage.
Best Regards
Michel
ceph -s cluster: id: cb0caedc-eb5b-42d1-a34f-96facfda8c27 health: HEALTH_WARN 6 pgs not deep-scrubbed in time
services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M) mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1 osd: 48 osds: 48 up (since 11M), 48 in (since 11M) rgw: 6 daemons active (6 hosts, 1 zones)
data: pools: 10 pools, 385 pgs objects: 5.97M objects, 23 TiB usage: 151 TiB used, 282 TiB / 433 TiB avail pgs: 381 active+clean 4 active+clean+scrubbing+deep
io: client: 59 MiB/s rd, 860 MiB/s wr, 155 op/s rd, 665 op/s wr
root@ceph-osd3:~# ceph health detail HEALTH_WARN 6 pgs not deep-scrubbed in time [WRN] PG_NOT_DEEP_SCRUBBED: 6 pgs not deep-scrubbed in time pg 6.78 not deep-scrubbed since 2024-01-11T16:07:54.875746+0200 pg 6.60 not deep-scrubbed since 2024-01-13T19:44:26.922000+0200 pg 6.5c not deep-scrubbed since 2024-01-13T09:07:24.780936+0200 pg 4.12 not deep-scrubbed since 2024-01-13T09:09:22.176240+0200 pg 10.d not deep-scrubbed since 2024-01-12T08:04:02.078062+0200 pg 5.f not deep-scrubbed since 2024-01-12T06:06:00.970665+0200 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thank you Frank , All disks are HDDs . Would like to know if I can increase the number of PGs live in production without a negative impact to the cluster. if yes which commands to use . Thank you very much for your prompt reply. Michel On Mon, Jan 29, 2024 at 10:59 AM Frank Schilder <frans@dtu.dk> wrote:
Hi Michel,
are your OSDs HDD or SSD? If they are HDD, its possible that they can't handle the deep-scrub load with default settings. In that case, have a look at this post https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YUHWQCDAKP5M... for some basic tuning info and a script to check your scrub stamp distribution.
You should also get rid of slow/failing ones. Look at smartctl output and throw out disks with remapped sectors, uncorrectable r/w errors or unusually many corrected read-write-errors (assuming you have disks with ECC).
A basic calculation for deep-scrub is as follows: max number of PGs that can be scrubbed at the same time: A=#OSDs/replication factor (rounded down). Take the B=deep-scrub times from the OSD logs (grep for deep-scrub) in minutes. Your pool can deep-scrub at a max A*24*(60/B) PGs per day. For reasonable operations you should not do more than 50% of that. With that you can calculate how many days it needs to deep-scrub your PGs.
Usual reasons for slow deep-scrub progress is too few PGs. With replication factor 3 and 48 OSDs you have a PG budget of ca. 3200 (ca 200/OSD) but use only 385. You should consider increasing the PG count for pools with lots of data. This should already relax the situation somewhat. Then do the calc above and tune deep-scrub times per pool such that they match with disk performance.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Michel Niyoyita <micou12@gmail.com> Sent: Monday, January 29, 2024 7:42 AM To: E Taka Cc: ceph-users Subject: [ceph-users] Re: 6 pgs not deep-scrubbed in time
Now they are increasing , Friday I tried to deep-scrubbing manually and they have been successfully done , but Monday morning I found that they are increasing to 37 , is it the best to deep-scrubbing manually while we are using the cluster? if not what is the best to do in order to address that .
Best Regards.
Michel
ceph -s cluster: id: cb0caedc-eb5b-42d1-a34f-96facfda8c27 health: HEALTH_WARN 37 pgs not deep-scrubbed in time
services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M) mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1 osd: 48 osds: 48 up (since 11M), 48 in (since 11M) rgw: 6 daemons active (6 hosts, 1 zones)
data: pools: 10 pools, 385 pgs objects: 6.00M objects, 23 TiB usage: 151 TiB used, 282 TiB / 433 TiB avail pgs: 381 active+clean 4 active+clean+scrubbing+deep
io: client: 265 MiB/s rd, 786 MiB/s wr, 3.87k op/s rd, 699 op/s wr
On Sun, Jan 28, 2024 at 6:14 PM E Taka <0etaka0@gmail.com> wrote:
22 is more often there than the others. Other operations may be blocked because of a deep-scrub is not finished yet. I would remove OSD 22, just to be sure about this: ceph orch osd rm osd.22
If this does not help, just add it again.
Am Fr., 26. Jan. 2024 um 08:05 Uhr schrieb Michel Niyoyita < micou12@gmail.com>:
It seems that are different OSDs as shown here . how have you managed to sort this out?
ceph pg dump | grep -F 6.78 dumped all 6.78 44268 0 0 0 0 178679640118 0 0 10099 10099 active+clean 2024-01-26T03:51:26.781438+0200 107547'115445304 107547:225274427 [12,36,37] 12 [12,36,37] 12 106977'114532385 2024-01-24T08:37:53.597331+0200 101161'109078277 2024-01-11T16:07:54.875746+0200 0 root@ceph-osd3:~# ceph pg dump | grep -F 6.60 dumped all 6.60 44449 0 0 0 0 179484338742 716 36 10097 10097 active+clean 2024-01-26T03:50:44.579831+0200 107547'153238805 107547:287193139 [32,5,29] 32 [32,5,29] 32 107231'152689835 2024-01-25T02:34:01.849966+0200 102171'147920798 2024-01-13T19:44:26.922000+0200 0 6.3a 44807 0 0 0 0 180969005694 0 0 10093 10093 active+clean 2024-01-26T03:53:28.837685+0200 107547'114765984 107547:238170093 [22,13,11] 22 [22,13,11] 22 106945'113739877 2024-01-24T04:10:17.224982+0200 102863'109559444 2024-01-15T05:31:36.606478+0200 0 root@ceph-osd3:~# ceph pg dump | grep -F 6.5c 6.5c 44277 0 0 0 0 178764978230 0 0 10051 10051 active+clean 2024-01-26T03:55:23.339584+0200 107547'126480090 107547:264432655 [22,37,30] 22 [22,37,30] 22 107205'125858697 2024-01-24T22:32:10.365869+0200 101941'120957992 2024-01-13T09:07:24.780936+0200 0 dumped all root@ceph-osd3:~# ceph pg dump | grep -F 4.12 dumped all 4.12 0 0 0 0 0 0 0 0 0 0 active+clean 2024-01-24T08:36:48.284388+0200 0'0 107546:152711 [22,19,7] 22 [22,19,7] 22 0'0 2024-01-24T08:36:48.284307+0200 0'0 2024-01-13T09:09:22.176240+0200 0 root@ceph-osd3:~# ceph pg dump | grep -F 10.d dumped all 10.d 0 0 0 0 0 0 0 0 0 0 active+clean 2024-01-24T04:04:33.641541+0200 0'0 107546:142651 [14,28,1] 14 [14,28,1] 14 0'0 2024-01-24T04:04:33.641451+0200 0'0 2024-01-12T08:04:02.078062+0200 0 root@ceph-osd3:~# ceph pg dump | grep -F 5.f dumped all 5.f 0 0 0 0 0 0 0 0 0 0 active+clean 2024-01-25T08:19:04.148941+0200 0'0 107546:161331 [11,24,35] 11 [11,24,35] 11 0'0 2024-01-25T08:19:04.148837+0200 0'0 2024-01-12T06:06:00.970665+0200 0
On Fri, Jan 26, 2024 at 8:58 AM E Taka <0etaka0@gmail.com> wrote:
We had the same problem. It turned out that one disk was slowly dying. It was easy to identify by the commands (in your case):
ceph pg dump | grep -F 6.78 ceph pg dump | grep -F 6.60 …
This command shows the OSDs of a PG in square brackets. If is there always the same number, then you've found the OSD which causes the slow scrubs.
Am Fr., 26. Jan. 2024 um 07:45 Uhr schrieb Michel Niyoyita < micou12@gmail.com>:
Hello team,
I have a cluster in production composed by 3 osds servers with 20 disks each deployed using ceph-ansibleand ubuntu OS , and the version is pacific . These days is in WARN state caused by pgs which are not deep-scrubbed in time . I tried to deep-scrubbed some pg manually but seems that the cluster can be slow, would like your assistance in order that my cluster can be in HEALTH_OK state as before without any interuption of service . The cluster is used as openstack backend storage.
Best Regards
Michel
ceph -s cluster: id: cb0caedc-eb5b-42d1-a34f-96facfda8c27 health: HEALTH_WARN 6 pgs not deep-scrubbed in time
services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M) mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1 osd: 48 osds: 48 up (since 11M), 48 in (since 11M) rgw: 6 daemons active (6 hosts, 1 zones)
data: pools: 10 pools, 385 pgs objects: 5.97M objects, 23 TiB usage: 151 TiB used, 282 TiB / 433 TiB avail pgs: 381 active+clean 4 active+clean+scrubbing+deep
io: client: 59 MiB/s rd, 860 MiB/s wr, 155 op/s rd, 665 op/s wr
root@ceph-osd3:~# ceph health detail HEALTH_WARN 6 pgs not deep-scrubbed in time [WRN] PG_NOT_DEEP_SCRUBBED: 6 pgs not deep-scrubbed in time pg 6.78 not deep-scrubbed since 2024-01-11T16:07:54.875746+0200 pg 6.60 not deep-scrubbed since 2024-01-13T19:44:26.922000+0200 pg 6.5c not deep-scrubbed since 2024-01-13T09:07:24.780936+0200 pg 4.12 not deep-scrubbed since 2024-01-13T09:09:22.176240+0200 pg 10.d not deep-scrubbed since 2024-01-12T08:04:02.078062+0200 pg 5.f not deep-scrubbed since 2024-01-12T06:06:00.970665+0200 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita <micou12@gmail.com>:
Thank you Frank ,
All disks are HDDs . Would like to know if I can increase the number of PGs live in production without a negative impact to the cluster. if yes which commands to use .
Yes. "ceph osd pool set <poolname> pg_num <number larger than before>" where the number usually should be a power of two that leads to a number of PGs per OSD between 100-200. -- May the most significant bit of your life be positive.
Thank you Janne , no need of setting some flags like ceph osd set nodeep-scrub ??? Thank you On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson <icepic.dz@gmail.com> wrote:
Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita <micou12@gmail.com>:
Thank you Frank ,
All disks are HDDs . Would like to know if I can increase the number of
PGs
live in production without a negative impact to the cluster. if yes which commands to use .
Yes. "ceph osd pool set <poolname> pg_num <number larger than before>" where the number usually should be a power of two that leads to a number of PGs per OSD between 100-200.
-- May the most significant bit of your life be positive.
This is how it is set , if you suggest to make some changes please advises. Thank you. ceph osd pool ls detail pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 1407 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr_devicehealth pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 flags hashpspool stripe_width 0 application rgw pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1394 flags hashpspool stripe_width 0 application rgw pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1395 flags hashpspool stripe_width 0 application rgw pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 108802 lfor 0/0/14812 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps_queue [22d7~3,11561~2,11571~1,11573~1c,11594~6,1159b~f,115b0~1,115b3~1,115c3~1,115f3~1,115f5~e,11613~6,1161f~c,11637~1b,11660~1,11663~2,11673~1,116d1~c,116f5~10,11721~c] pool 7 'images' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 94609 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd pool 8 'backups' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1399 flags hashpspool stripe_width 0 application rbd pool 9 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 108783 lfor 0/561/559 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps_queue [3fa~1,3fc~3,400~1,402~1] pool 10 'testbench' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20931 lfor 0/20931/20929 flags hashpspool stripe_width 0 On Mon, Jan 29, 2024 at 2:09 PM Michel Niyoyita <micou12@gmail.com> wrote:
Thank you Janne ,
no need of setting some flags like ceph osd set nodeep-scrub ???
Thank you
On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson <icepic.dz@gmail.com> wrote:
Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita <micou12@gmail.com>:
Thank you Frank ,
All disks are HDDs . Would like to know if I can increase the number of
PGs
live in production without a negative impact to the cluster. if yes which commands to use .
Yes. "ceph osd pool set <poolname> pg_num <number larger than before>" where the number usually should be a power of two that leads to a number of PGs per OSD between 100-200.
-- May the most significant bit of your life be positive.
You will have to look at the output of "ceph df" and make a decision to balance "objects per PG" and "GB per PG". Increase he PG count for the pools with the worst of these two numbers most such that it balances out as much as possible. If you have pools that see significantly more user-IO than others, prioritise these. You will have to find out for your specific cluster, we can only give general guidelines. Make changes, run benchmarks, re-evaluate. Take the time for it. The better you know your cluster and your users, the better the end result will be. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Michel Niyoyita <micou12@gmail.com> Sent: Monday, January 29, 2024 2:04 PM To: Janne Johansson Cc: Frank Schilder; E Taka; ceph-users Subject: Re: [ceph-users] Re: 6 pgs not deep-scrubbed in time This is how it is set , if you suggest to make some changes please advises. Thank you. ceph osd pool ls detail pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 1407 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr_devicehealth pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 flags hashpspool stripe_width 0 application rgw pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1394 flags hashpspool stripe_width 0 application rgw pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1395 flags hashpspool stripe_width 0 application rgw pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 108802 lfor 0/0/14812 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps_queue [22d7~3,11561~2,11571~1,11573~1c,11594~6,1159b~f,115b0~1,115b3~1,115c3~1,115f3~1,115f5~e,11613~6,1161f~c,11637~1b,11660~1,11663~2,11673~1,116d1~c,116f5~10,11721~c] pool 7 'images' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 94609 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd pool 8 'backups' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1399 flags hashpspool stripe_width 0 application rbd pool 9 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 108783 lfor 0/561/559 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps_queue [3fa~1,3fc~3,400~1,402~1] pool 10 'testbench' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20931 lfor 0/20931/20929 flags hashpspool stripe_width 0 On Mon, Jan 29, 2024 at 2:09 PM Michel Niyoyita <micou12@gmail.com<mailto:micou12@gmail.com>> wrote: Thank you Janne , no need of setting some flags like ceph osd set nodeep-scrub ??? Thank you On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson <icepic.dz@gmail.com<mailto:icepic.dz@gmail.com>> wrote: Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita <micou12@gmail.com<mailto:micou12@gmail.com>>:
Thank you Frank ,
All disks are HDDs . Would like to know if I can increase the number of PGs live in production without a negative impact to the cluster. if yes which commands to use .
Yes. "ceph osd pool set <poolname> pg_num <number larger than before>" where the number usually should be a power of two that leads to a number of PGs per OSD between 100-200. -- May the most significant bit of your life be positive.
Dear team, below is the output of ceph df command and the ceph version I am running ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 433 TiB 282 TiB 151 TiB 151 TiB 34.82 TOTAL 433 TiB 282 TiB 151 TiB 151 TiB 34.82 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 1.1 MiB 3 3.2 MiB 0 73 TiB .rgw.root 2 32 3.7 KiB 8 96 KiB 0 73 TiB default.rgw.log 3 32 3.6 KiB 209 408 KiB 0 73 TiB default.rgw.control 4 32 0 B 8 0 B 0 73 TiB default.rgw.meta 5 32 382 B 2 24 KiB 0 73 TiB volumes 6 128 21 TiB 5.68M 62 TiB 22.09 73 TiB images 7 32 878 GiB 112.50k 2.6 TiB 1.17 73 TiB backups 8 32 0 B 0 0 B 0 73 TiB vms 9 32 881 GiB 174.30k 2.5 TiB 1.13 73 TiB testbench 10 32 0 B 0 0 B 0 73 TiB root@ceph-mon1:~# ceph --version ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) root@ceph-mon1:~# please advise accordingly Michel On Mon, Jan 29, 2024 at 9:48 PM Frank Schilder <frans@dtu.dk> wrote:
You will have to look at the output of "ceph df" and make a decision to balance "objects per PG" and "GB per PG". Increase he PG count for the pools with the worst of these two numbers most such that it balances out as much as possible. If you have pools that see significantly more user-IO than others, prioritise these.
You will have to find out for your specific cluster, we can only give general guidelines. Make changes, run benchmarks, re-evaluate. Take the time for it. The better you know your cluster and your users, the better the end result will be.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Michel Niyoyita <micou12@gmail.com> Sent: Monday, January 29, 2024 2:04 PM To: Janne Johansson Cc: Frank Schilder; E Taka; ceph-users Subject: Re: [ceph-users] Re: 6 pgs not deep-scrubbed in time
This is how it is set , if you suggest to make some changes please advises.
Thank you.
ceph osd pool ls detail pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 1407 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr_devicehealth pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 flags hashpspool stripe_width 0 application rgw pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1394 flags hashpspool stripe_width 0 application rgw pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1395 flags hashpspool stripe_width 0 application rgw pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 108802 lfor 0/0/14812 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps_queue [22d7~3,11561~2,11571~1,11573~1c,11594~6,1159b~f,115b0~1,115b3~1,115c3~1,115f3~1,115f5~e,11613~6,1161f~c,11637~1b,11660~1,11663~2,11673~1,116d1~c,116f5~10,11721~c] pool 7 'images' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 94609 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd pool 8 'backups' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1399 flags hashpspool stripe_width 0 application rbd pool 9 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 108783 lfor 0/561/559 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps_queue [3fa~1,3fc~3,400~1,402~1] pool 10 'testbench' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20931 lfor 0/20931/20929 flags hashpspool stripe_width 0
On Mon, Jan 29, 2024 at 2:09 PM Michel Niyoyita <micou12@gmail.com<mailto: micou12@gmail.com>> wrote: Thank you Janne ,
no need of setting some flags like ceph osd set nodeep-scrub ???
Thank you
On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson <icepic.dz@gmail.com <mailto:icepic.dz@gmail.com>> wrote: Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita <micou12@gmail.com <mailto:micou12@gmail.com>>:
Thank you Frank ,
All disks are HDDs . Would like to know if I can increase the number of
PGs
live in production without a negative impact to the cluster. if yes which commands to use .
Yes. "ceph osd pool set <poolname> pg_num <number larger than before>" where the number usually should be a power of two that leads to a number of PGs per OSD between 100-200.
-- May the most significant bit of your life be positive.
I now concur you should increase the pg_num as a first step for this cluster. Disable the pg autoscaler for and increase the volumes pool to pg_num 256. Then likely re-asses and make the next power of 2 jump to 512 and probably beyond. Keep in mind this is not going to fix your short term deep-scrub issue in fact it will increase the number of not scrubbed in time PGs until the pg_num change is complete. This is because OSDs dont scrub when they are backfilling. I would sit on 256 for a couple weeks and let scrubs happen then continue past 256. with the ultimate target of around 100-200 PGs per OSD which "ceph osd df tree" will show you in the PGs column. Respectfully, *Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham> On Tue, Jan 30, 2024 at 3:16 AM Michel Niyoyita <micou12@gmail.com> wrote:
Dear team,
below is the output of ceph df command and the ceph version I am running
ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 433 TiB 282 TiB 151 TiB 151 TiB 34.82 TOTAL 433 TiB 282 TiB 151 TiB 151 TiB 34.82
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 1.1 MiB 3 3.2 MiB 0 73 TiB .rgw.root 2 32 3.7 KiB 8 96 KiB 0 73 TiB default.rgw.log 3 32 3.6 KiB 209 408 KiB 0 73 TiB default.rgw.control 4 32 0 B 8 0 B 0 73 TiB default.rgw.meta 5 32 382 B 2 24 KiB 0 73 TiB volumes 6 128 21 TiB 5.68M 62 TiB 22.09 73 TiB images 7 32 878 GiB 112.50k 2.6 TiB 1.17 73 TiB backups 8 32 0 B 0 0 B 0 73 TiB vms 9 32 881 GiB 174.30k 2.5 TiB 1.13 73 TiB testbench 10 32 0 B 0 0 B 0 73 TiB root@ceph-mon1:~# ceph --version ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) root@ceph-mon1:~#
please advise accordingly
Michel
On Mon, Jan 29, 2024 at 9:48 PM Frank Schilder <frans@dtu.dk> wrote:
You will have to look at the output of "ceph df" and make a decision to balance "objects per PG" and "GB per PG". Increase he PG count for the pools with the worst of these two numbers most such that it balances out as much as possible. If you have pools that see significantly more user-IO than others, prioritise these.
You will have to find out for your specific cluster, we can only give general guidelines. Make changes, run benchmarks, re-evaluate. Take the time for it. The better you know your cluster and your users, the better the end result will be.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Michel Niyoyita <micou12@gmail.com> Sent: Monday, January 29, 2024 2:04 PM To: Janne Johansson Cc: Frank Schilder; E Taka; ceph-users Subject: Re: [ceph-users] Re: 6 pgs not deep-scrubbed in time
This is how it is set , if you suggest to make some changes please advises.
Thank you.
ceph osd pool ls detail pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 1407 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr_devicehealth pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 flags hashpspool stripe_width 0 application rgw pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1394 flags hashpspool stripe_width 0 application rgw pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1395 flags hashpspool stripe_width 0 application rgw pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 108802 lfor 0/0/14812 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps_queue
[22d7~3,11561~2,11571~1,11573~1c,11594~6,1159b~f,115b0~1,115b3~1,115c3~1,115f3~1,115f5~e,11613~6,1161f~c,11637~1b,11660~1,11663~2,11673~1,116d1~c,116f5~10,11721~c]
pool 7 'images' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 94609 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd pool 8 'backups' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1399 flags hashpspool stripe_width 0 application rbd pool 9 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 108783 lfor 0/561/559 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps_queue [3fa~1,3fc~3,400~1,402~1] pool 10 'testbench' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20931 lfor 0/20931/20929 flags hashpspool stripe_width 0
On Mon, Jan 29, 2024 at 2:09 PM Michel Niyoyita <micou12@gmail.com <mailto: micou12@gmail.com>> wrote: Thank you Janne ,
no need of setting some flags like ceph osd set nodeep-scrub ???
Thank you
On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson <icepic.dz@gmail.com <mailto:icepic.dz@gmail.com>> wrote: Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita <micou12@gmail.com <mailto:micou12@gmail.com>>:
Thank you Frank ,
All disks are HDDs . Would like to know if I can increase the number of
PGs
live in production without a negative impact to the cluster. if yes which commands to use .
Yes. "ceph osd pool set <poolname> pg_num <number larger than before>" where the number usually should be a power of two that leads to a number of PGs per OSD between 100-200.
-- May the most significant bit of your life be positive.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for your advices Wes, below is what ceph osd df tree shows , the increase of pg_num of the production cluster will not affect the performance or crush ? how long it can takes to finish? ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 433.11841 - 433 TiB 151 TiB 67 TiB 364 MiB 210 GiB 282 TiB 34.86 1.00 - root default -3 144.37280 - 144 TiB 50 TiB 22 TiB 121 MiB 70 GiB 94 TiB 34.86 1.00 - host ceph-osd1 2 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1021 GiB 5.4 MiB 3.7 GiB 6.3 TiB 30.40 0.87 19 up osd.2 3 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 931 GiB 4.1 MiB 3.5 GiB 6.4 TiB 29.43 0.84 29 up osd.3 6 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.5 TiB 8.1 MiB 4.5 GiB 5.8 TiB 36.09 1.04 20 up osd.6 9 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.0 TiB 6.6 MiB 3.8 GiB 6.2 TiB 30.97 0.89 23 up osd.9 12 hdd 9.02330 1.00000 9.0 TiB 4.0 TiB 2.3 TiB 13 MiB 6.1 GiB 5.0 TiB 44.68 1.28 30 up osd.12 15 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.8 TiB 9.2 MiB 5.2 GiB 5.5 TiB 38.99 1.12 30 up osd.15 18 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.5 MiB 4.0 GiB 6.1 TiB 32.80 0.94 21 up osd.18 22 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.9 TiB 10 MiB 5.4 GiB 5.4 TiB 40.25 1.15 22 up osd.22 25 hdd 9.02330 1.00000 9.0 TiB 3.9 TiB 2.1 TiB 12 MiB 5.7 GiB 5.1 TiB 42.94 1.23 22 up osd.25 28 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.5 MiB 4.1 GiB 5.9 TiB 34.87 1.00 21 up osd.28 32 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1017 GiB 4.8 MiB 3.7 GiB 6.3 TiB 30.36 0.87 27 up osd.32 35 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.3 TiB 7.2 MiB 4.2 GiB 6.0 TiB 33.73 0.97 21 up osd.35 38 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.3 MiB 4.1 GiB 5.9 TiB 34.57 0.99 24 up osd.38 41 hdd 9.02330 1.00000 9.0 TiB 2.9 TiB 1.2 TiB 6.2 MiB 4.0 GiB 6.1 TiB 32.49 0.93 24 up osd.41 44 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.3 MiB 4.4 GiB 5.9 TiB 34.87 1.00 29 up osd.44 47 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1016 GiB 5.4 MiB 3.6 GiB 6.3 TiB 30.35 0.87 23 up osd.47 -7 144.37280 - 144 TiB 50 TiB 22 TiB 122 MiB 70 GiB 94 TiB 34.86 1.00 - host ceph-osd2 1 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.7 MiB 3.8 GiB 6.2 TiB 31.00 0.89 27 up osd.1 5 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.5 TiB 7.3 MiB 4.5 GiB 5.8 TiB 35.45 1.02 27 up osd.5 8 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.6 TiB 8.3 MiB 4.7 GiB 5.7 TiB 36.85 1.06 30 up osd.8 10 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.5 MiB 4.5 GiB 5.9 TiB 34.87 1.00 20 up osd.10 13 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB 5.3 GiB 5.4 TiB 39.63 1.14 27 up osd.13 16 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 6.0 MiB 3.8 GiB 6.2 TiB 31.01 0.89 19 up osd.16 19 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.4 MiB 4.0 GiB 6.1 TiB 32.77 0.94 21 up osd.19 21 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.5 MiB 3.7 GiB 6.2 TiB 31.58 0.91 26 up osd.21 24 hdd 9.02330 1.00000 9.0 TiB 2.6 TiB 855 GiB 4.7 MiB 3.3 GiB 6.4 TiB 28.61 0.82 19 up osd.24 27 hdd 9.02330 1.00000 9.0 TiB 3.7 TiB 1.9 TiB 10 MiB 5.2 GiB 5.3 TiB 40.84 1.17 24 up osd.27 30 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.4 TiB 7.5 MiB 4.5 GiB 5.9 TiB 35.16 1.01 22 up osd.30 33 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 8.6 MiB 4.3 GiB 5.9 TiB 34.59 0.99 23 up osd.33 36 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 10 MiB 5.0 GiB 5.6 TiB 38.17 1.09 25 up osd.36 39 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 8.5 MiB 5.1 GiB 5.6 TiB 37.79 1.08 31 up osd.39 42 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB 5.2 GiB 5.4 TiB 39.68 1.14 23 up osd.42 45 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 964 GiB 5.1 MiB 3.5 GiB 6.3 TiB 29.78 0.85 21 up osd.45 -5 144.37280 - 144 TiB 50 TiB 22 TiB 121 MiB 70 GiB 94 TiB 34.86 1.00 - host ceph-osd3 0 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 934 GiB 4.9 MiB 3.4 GiB 6.4 TiB 29.47 0.85 21 up osd.0 4 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.5 MiB 4.1 GiB 6.1 TiB 32.73 0.94 22 up osd.4 7 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.8 TiB 9.2 MiB 5.1 GiB 5.5 TiB 39.02 1.12 30 up osd.7 11 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.9 TiB 10 MiB 5.1 GiB 5.4 TiB 39.97 1.15 27 up osd.11 14 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.7 TiB 10 MiB 5.1 GiB 5.6 TiB 38.24 1.10 27 up osd.14 17 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.4 MiB 4.1 GiB 6.0 TiB 33.09 0.95 23 up osd.17 20 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.6 MiB 3.8 GiB 6.2 TiB 31.55 0.90 20 up osd.20 23 hdd 9.02330 1.00000 9.0 TiB 2.6 TiB 828 GiB 4.0 MiB 3.3 GiB 6.5 TiB 28.32 0.81 23 up osd.23 26 hdd 9.02330 1.00000 9.0 TiB 2.9 TiB 1.2 TiB 5.8 MiB 3.8 GiB 6.1 TiB 32.12 0.92 26 up osd.26 29 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB 5.1 GiB 5.4 TiB 39.73 1.14 24 up osd.29 31 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.8 MiB 3.7 GiB 6.2 TiB 31.56 0.91 22 up osd.31 34 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.5 TiB 8.2 MiB 4.6 GiB 5.7 TiB 36.29 1.04 23 up osd.34 37 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.5 TiB 8.2 MiB 4.5 GiB 5.8 TiB 35.51 1.02 20 up osd.37 40 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 9.3 MiB 4.9 GiB 5.6 TiB 38.16 1.09 25 up osd.40 43 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.6 TiB 8.5 MiB 4.8 GiB 5.7 TiB 37.19 1.07 29 up osd.43 46 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 8.4 MiB 4.4 GiB 5.9 TiB 34.85 1.00 23 up osd.46 TOTAL 433 TiB 151 TiB 67 TiB 364 MiB 210 GiB 282 TiB 34.86 MIN/MAX VAR: 0.81/1.28 STDDEV: 3.95 Michel On Tue, Jan 30, 2024 at 4:18 PM Wesley Dillingham <wes@wesdillingham.com> wrote:
I now concur you should increase the pg_num as a first step for this cluster. Disable the pg autoscaler for and increase the volumes pool to pg_num 256. Then likely re-asses and make the next power of 2 jump to 512 and probably beyond.
Keep in mind this is not going to fix your short term deep-scrub issue in fact it will increase the number of not scrubbed in time PGs until the pg_num change is complete. This is because OSDs dont scrub when they are backfilling.
I would sit on 256 for a couple weeks and let scrubs happen then continue past 256.
with the ultimate target of around 100-200 PGs per OSD which "ceph osd df tree" will show you in the PGs column.
Respectfully,
*Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Jan 30, 2024 at 3:16 AM Michel Niyoyita <micou12@gmail.com> wrote:
Dear team,
below is the output of ceph df command and the ceph version I am running
ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 433 TiB 282 TiB 151 TiB 151 TiB 34.82 TOTAL 433 TiB 282 TiB 151 TiB 151 TiB 34.82
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 1.1 MiB 3 3.2 MiB 0 73 TiB .rgw.root 2 32 3.7 KiB 8 96 KiB 0 73 TiB default.rgw.log 3 32 3.6 KiB 209 408 KiB 0 73 TiB default.rgw.control 4 32 0 B 8 0 B 0 73 TiB default.rgw.meta 5 32 382 B 2 24 KiB 0 73 TiB volumes 6 128 21 TiB 5.68M 62 TiB 22.09 73 TiB images 7 32 878 GiB 112.50k 2.6 TiB 1.17 73 TiB backups 8 32 0 B 0 0 B 0 73 TiB vms 9 32 881 GiB 174.30k 2.5 TiB 1.13 73 TiB testbench 10 32 0 B 0 0 B 0 73 TiB root@ceph-mon1:~# ceph --version ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) root@ceph-mon1:~#
please advise accordingly
Michel
On Mon, Jan 29, 2024 at 9:48 PM Frank Schilder <frans@dtu.dk> wrote:
You will have to look at the output of "ceph df" and make a decision to balance "objects per PG" and "GB per PG". Increase he PG count for the pools with the worst of these two numbers most such that it balances out as much as possible. If you have pools that see significantly more user-IO than others, prioritise these.
You will have to find out for your specific cluster, we can only give general guidelines. Make changes, run benchmarks, re-evaluate. Take the time for it. The better you know your cluster and your users, the better the end result will be.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Michel Niyoyita <micou12@gmail.com> Sent: Monday, January 29, 2024 2:04 PM To: Janne Johansson Cc: Frank Schilder; E Taka; ceph-users Subject: Re: [ceph-users] Re: 6 pgs not deep-scrubbed in time
This is how it is set , if you suggest to make some changes please advises.
Thank you.
ceph osd pool ls detail pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 1407 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr_devicehealth pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 flags hashpspool stripe_width 0 application rgw pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1394 flags hashpspool stripe_width 0 application rgw pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1395 flags hashpspool stripe_width 0 application rgw pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 108802 lfor 0/0/14812 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps_queue
[22d7~3,11561~2,11571~1,11573~1c,11594~6,1159b~f,115b0~1,115b3~1,115c3~1,115f3~1,115f5~e,11613~6,1161f~c,11637~1b,11660~1,11663~2,11673~1,116d1~c,116f5~10,11721~c]
pool 7 'images' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 94609 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd pool 8 'backups' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1399 flags hashpspool stripe_width 0 application rbd pool 9 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 108783 lfor 0/561/559 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps_queue [3fa~1,3fc~3,400~1,402~1] pool 10 'testbench' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20931 lfor 0/20931/20929 flags hashpspool stripe_width 0
On Mon, Jan 29, 2024 at 2:09 PM Michel Niyoyita <micou12@gmail.com <mailto: micou12@gmail.com>> wrote: Thank you Janne ,
no need of setting some flags like ceph osd set nodeep-scrub ???
Thank you
On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson <icepic.dz@gmail.com <mailto:icepic.dz@gmail.com>> wrote: Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita <micou12@gmail.com <mailto:micou12@gmail.com>>:
Thank you Frank ,
All disks are HDDs . Would like to know if I can increase the number
of PGs
live in production without a negative impact to the cluster. if yes which commands to use .
Yes. "ceph osd pool set <poolname> pg_num <number larger than before>" where the number usually should be a power of two that leads to a number of PGs per OSD between 100-200.
-- May the most significant bit of your life be positive.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
It will take a couple weeks to a couple months to complete is my best guess on 10TB spinners at ~40% full. The cluster should be usable throughout the process. Keep in mind, you should disable the pg autoscaler on any pool which you are manually adjusting the pg_num for. Increasing the pg_num is called "pg splitting" you can google around for this to see how it will work etc. There are a few knobs to increase or decrease the aggressiveness of the pg split, primarily these are osd_max_backfills and target_max_misplaced_ratio. You can monitor the progress of the split by looking at "ceph osd pool ls detail" for the pool you are splitting, for this pool pgp_num will slowly increase up until it reaches the pg_num / pg_num_target. IMO this blog post best covers the issue which you are looking to undertake: https://ceph.io/en/news/blog/2019/new-in-nautilus-pg-merging-and-autotuning/ Respectfully, *Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham> On Tue, Jan 30, 2024 at 9:38 AM Michel Niyoyita <micou12@gmail.com> wrote:
Thanks for your advices Wes, below is what ceph osd df tree shows , the increase of pg_num of the production cluster will not affect the performance or crush ? how long it can takes to finish?
ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 433.11841 - 433 TiB 151 TiB 67 TiB 364 MiB 210 GiB 282 TiB 34.86 1.00 - root default -3 144.37280 - 144 TiB 50 TiB 22 TiB 121 MiB 70 GiB 94 TiB 34.86 1.00 - host ceph-osd1 2 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1021 GiB 5.4 MiB 3.7 GiB 6.3 TiB 30.40 0.87 19 up osd.2 3 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 931 GiB 4.1 MiB 3.5 GiB 6.4 TiB 29.43 0.84 29 up osd.3 6 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.5 TiB 8.1 MiB 4.5 GiB 5.8 TiB 36.09 1.04 20 up osd.6 9 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.0 TiB 6.6 MiB 3.8 GiB 6.2 TiB 30.97 0.89 23 up osd.9 12 hdd 9.02330 1.00000 9.0 TiB 4.0 TiB 2.3 TiB 13 MiB 6.1 GiB 5.0 TiB 44.68 1.28 30 up osd.12 15 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.8 TiB 9.2 MiB 5.2 GiB 5.5 TiB 38.99 1.12 30 up osd.15 18 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.5 MiB 4.0 GiB 6.1 TiB 32.80 0.94 21 up osd.18 22 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.9 TiB 10 MiB 5.4 GiB 5.4 TiB 40.25 1.15 22 up osd.22 25 hdd 9.02330 1.00000 9.0 TiB 3.9 TiB 2.1 TiB 12 MiB 5.7 GiB 5.1 TiB 42.94 1.23 22 up osd.25 28 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.5 MiB 4.1 GiB 5.9 TiB 34.87 1.00 21 up osd.28 32 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1017 GiB 4.8 MiB 3.7 GiB 6.3 TiB 30.36 0.87 27 up osd.32 35 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.3 TiB 7.2 MiB 4.2 GiB 6.0 TiB 33.73 0.97 21 up osd.35 38 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.3 MiB 4.1 GiB 5.9 TiB 34.57 0.99 24 up osd.38 41 hdd 9.02330 1.00000 9.0 TiB 2.9 TiB 1.2 TiB 6.2 MiB 4.0 GiB 6.1 TiB 32.49 0.93 24 up osd.41 44 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.3 MiB 4.4 GiB 5.9 TiB 34.87 1.00 29 up osd.44 47 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1016 GiB 5.4 MiB 3.6 GiB 6.3 TiB 30.35 0.87 23 up osd.47 -7 144.37280 - 144 TiB 50 TiB 22 TiB 122 MiB 70 GiB 94 TiB 34.86 1.00 - host ceph-osd2 1 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.7 MiB 3.8 GiB 6.2 TiB 31.00 0.89 27 up osd.1 5 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.5 TiB 7.3 MiB 4.5 GiB 5.8 TiB 35.45 1.02 27 up osd.5 8 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.6 TiB 8.3 MiB 4.7 GiB 5.7 TiB 36.85 1.06 30 up osd.8 10 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.5 MiB 4.5 GiB 5.9 TiB 34.87 1.00 20 up osd.10 13 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB 5.3 GiB 5.4 TiB 39.63 1.14 27 up osd.13 16 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 6.0 MiB 3.8 GiB 6.2 TiB 31.01 0.89 19 up osd.16 19 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.4 MiB 4.0 GiB 6.1 TiB 32.77 0.94 21 up osd.19 21 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.5 MiB 3.7 GiB 6.2 TiB 31.58 0.91 26 up osd.21 24 hdd 9.02330 1.00000 9.0 TiB 2.6 TiB 855 GiB 4.7 MiB 3.3 GiB 6.4 TiB 28.61 0.82 19 up osd.24 27 hdd 9.02330 1.00000 9.0 TiB 3.7 TiB 1.9 TiB 10 MiB 5.2 GiB 5.3 TiB 40.84 1.17 24 up osd.27 30 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.4 TiB 7.5 MiB 4.5 GiB 5.9 TiB 35.16 1.01 22 up osd.30 33 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 8.6 MiB 4.3 GiB 5.9 TiB 34.59 0.99 23 up osd.33 36 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 10 MiB 5.0 GiB 5.6 TiB 38.17 1.09 25 up osd.36 39 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 8.5 MiB 5.1 GiB 5.6 TiB 37.79 1.08 31 up osd.39 42 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB 5.2 GiB 5.4 TiB 39.68 1.14 23 up osd.42 45 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 964 GiB 5.1 MiB 3.5 GiB 6.3 TiB 29.78 0.85 21 up osd.45 -5 144.37280 - 144 TiB 50 TiB 22 TiB 121 MiB 70 GiB 94 TiB 34.86 1.00 - host ceph-osd3 0 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 934 GiB 4.9 MiB 3.4 GiB 6.4 TiB 29.47 0.85 21 up osd.0 4 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.5 MiB 4.1 GiB 6.1 TiB 32.73 0.94 22 up osd.4 7 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.8 TiB 9.2 MiB 5.1 GiB 5.5 TiB 39.02 1.12 30 up osd.7 11 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.9 TiB 10 MiB 5.1 GiB 5.4 TiB 39.97 1.15 27 up osd.11 14 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.7 TiB 10 MiB 5.1 GiB 5.6 TiB 38.24 1.10 27 up osd.14 17 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.4 MiB 4.1 GiB 6.0 TiB 33.09 0.95 23 up osd.17 20 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.6 MiB 3.8 GiB 6.2 TiB 31.55 0.90 20 up osd.20 23 hdd 9.02330 1.00000 9.0 TiB 2.6 TiB 828 GiB 4.0 MiB 3.3 GiB 6.5 TiB 28.32 0.81 23 up osd.23 26 hdd 9.02330 1.00000 9.0 TiB 2.9 TiB 1.2 TiB 5.8 MiB 3.8 GiB 6.1 TiB 32.12 0.92 26 up osd.26 29 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB 5.1 GiB 5.4 TiB 39.73 1.14 24 up osd.29 31 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.8 MiB 3.7 GiB 6.2 TiB 31.56 0.91 22 up osd.31 34 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.5 TiB 8.2 MiB 4.6 GiB 5.7 TiB 36.29 1.04 23 up osd.34 37 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.5 TiB 8.2 MiB 4.5 GiB 5.8 TiB 35.51 1.02 20 up osd.37 40 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 9.3 MiB 4.9 GiB 5.6 TiB 38.16 1.09 25 up osd.40 43 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.6 TiB 8.5 MiB 4.8 GiB 5.7 TiB 37.19 1.07 29 up osd.43 46 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 8.4 MiB 4.4 GiB 5.9 TiB 34.85 1.00 23 up osd.46 TOTAL 433 TiB 151 TiB 67 TiB 364 MiB 210 GiB 282 TiB 34.86 MIN/MAX VAR: 0.81/1.28 STDDEV: 3.95
Michel
On Tue, Jan 30, 2024 at 4:18 PM Wesley Dillingham <wes@wesdillingham.com> wrote:
I now concur you should increase the pg_num as a first step for this cluster. Disable the pg autoscaler for and increase the volumes pool to pg_num 256. Then likely re-asses and make the next power of 2 jump to 512 and probably beyond.
Keep in mind this is not going to fix your short term deep-scrub issue in fact it will increase the number of not scrubbed in time PGs until the pg_num change is complete. This is because OSDs dont scrub when they are backfilling.
I would sit on 256 for a couple weeks and let scrubs happen then continue past 256.
with the ultimate target of around 100-200 PGs per OSD which "ceph osd df tree" will show you in the PGs column.
Respectfully,
*Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Jan 30, 2024 at 3:16 AM Michel Niyoyita <micou12@gmail.com> wrote:
Dear team,
below is the output of ceph df command and the ceph version I am running
ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 433 TiB 282 TiB 151 TiB 151 TiB 34.82 TOTAL 433 TiB 282 TiB 151 TiB 151 TiB 34.82
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 1.1 MiB 3 3.2 MiB 0 73 TiB .rgw.root 2 32 3.7 KiB 8 96 KiB 0 73 TiB default.rgw.log 3 32 3.6 KiB 209 408 KiB 0 73 TiB default.rgw.control 4 32 0 B 8 0 B 0 73 TiB default.rgw.meta 5 32 382 B 2 24 KiB 0 73 TiB volumes 6 128 21 TiB 5.68M 62 TiB 22.09 73 TiB images 7 32 878 GiB 112.50k 2.6 TiB 1.17 73 TiB backups 8 32 0 B 0 0 B 0 73 TiB vms 9 32 881 GiB 174.30k 2.5 TiB 1.13 73 TiB testbench 10 32 0 B 0 0 B 0 73 TiB root@ceph-mon1:~# ceph --version ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) root@ceph-mon1:~#
please advise accordingly
Michel
On Mon, Jan 29, 2024 at 9:48 PM Frank Schilder <frans@dtu.dk> wrote:
You will have to look at the output of "ceph df" and make a decision to balance "objects per PG" and "GB per PG". Increase he PG count for the pools with the worst of these two numbers most such that it balances out as much as possible. If you have pools that see significantly more user-IO than others, prioritise these.
You will have to find out for your specific cluster, we can only give general guidelines. Make changes, run benchmarks, re-evaluate. Take the time for it. The better you know your cluster and your users, the better the end result will be.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Michel Niyoyita <micou12@gmail.com> Sent: Monday, January 29, 2024 2:04 PM To: Janne Johansson Cc: Frank Schilder; E Taka; ceph-users Subject: Re: [ceph-users] Re: 6 pgs not deep-scrubbed in time
This is how it is set , if you suggest to make some changes please advises.
Thank you.
ceph osd pool ls detail pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 1407 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr_devicehealth pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 flags hashpspool stripe_width 0 application rgw pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1394 flags hashpspool stripe_width 0 application rgw pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1395 flags hashpspool stripe_width 0 application rgw pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 108802 lfor 0/0/14812 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps_queue
[22d7~3,11561~2,11571~1,11573~1c,11594~6,1159b~f,115b0~1,115b3~1,115c3~1,115f3~1,115f5~e,11613~6,1161f~c,11637~1b,11660~1,11663~2,11673~1,116d1~c,116f5~10,11721~c]
pool 7 'images' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 94609 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd pool 8 'backups' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1399 flags hashpspool stripe_width 0 application rbd pool 9 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 108783 lfor 0/561/559 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps_queue [3fa~1,3fc~3,400~1,402~1] pool 10 'testbench' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20931 lfor 0/20931/20929 flags hashpspool stripe_width 0
On Mon, Jan 29, 2024 at 2:09 PM Michel Niyoyita <micou12@gmail.com <mailto: micou12@gmail.com>> wrote: Thank you Janne ,
no need of setting some flags like ceph osd set nodeep-scrub ???
Thank you
On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson <icepic.dz@gmail.com <mailto:icepic.dz@gmail.com>> wrote: Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita <micou12@gmail.com <mailto:micou12@gmail.com>>:
Thank you Frank ,
All disks are HDDs . Would like to know if I can increase the number
of PGs
live in production without a negative impact to the cluster. if yes which commands to use .
Yes. "ceph osd pool set <poolname> pg_num <number larger than before>" where the number usually should be a power of two that leads to a number of PGs per OSD between 100-200.
-- May the most significant bit of your life be positive.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I tried that on one of my pool (pool id 3) but the number of pgs not deep-scrubbed in time increased also from 55 to 100 but the number of PGs was increased. I set also autoscale to off mode. before continue to other pools would like to ask if so far there is no negative impact. ceph -s cluster: id: cb0caedc-eb5b-42d1-a34f-96facfda8c27 health: HEALTH_WARN 100 pgs not deep-scrubbed in time services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M) mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1 osd: 48 osds: 48 up (since 11M), 48 in (since 12M) rgw: 6 daemons active (6 hosts, 1 zones) data: pools: 10 pools, 609 pgs objects: 6.03M objects, 23 TiB usage: 151 TiB used, 282 TiB / 433 TiB avail pgs: 603 active+clean 4 active+clean+scrubbing+deep 2 active+clean+scrubbing io: client: 96 MiB/s rd, 573 MiB/s wr, 576 op/s rd, 648 op/s wr root@ceph-osd3:/var/log# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 433 TiB 282 TiB 151 TiB 151 TiB 34.93 TOTAL 433 TiB 282 TiB 151 TiB 151 TiB 34.93 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 1.1 MiB 3 3.2 MiB 0 72 TiB .rgw.root 2 32 3.7 KiB 8 96 KiB 0 72 TiB default.rgw.log 3 256 3.6 KiB 204 408 KiB 0 72 TiB default.rgw.control 4 32 0 B 8 0 B 0 72 TiB default.rgw.meta 5 32 382 B 2 24 KiB 0 72 TiB volumes 6 128 21 TiB 5.74M 62 TiB 22.30 72 TiB images 7 32 878 GiB 112.50k 2.6 TiB 1.17 72 TiB backups 8 32 0 B 0 0 B 0 72 TiB vms 9 32 870 GiB 170.73k 2.5 TiB 1.13 72 TiB testbench 10 32 0 B 0 0 B 0 72 TiB On Tue, Jan 30, 2024 at 5:05 PM Wesley Dillingham <wes@wesdillingham.com> wrote:
It will take a couple weeks to a couple months to complete is my best guess on 10TB spinners at ~40% full. The cluster should be usable throughout the process.
Keep in mind, you should disable the pg autoscaler on any pool which you are manually adjusting the pg_num for. Increasing the pg_num is called "pg splitting" you can google around for this to see how it will work etc.
There are a few knobs to increase or decrease the aggressiveness of the pg split, primarily these are osd_max_backfills and target_max_misplaced_ratio.
You can monitor the progress of the split by looking at "ceph osd pool ls detail" for the pool you are splitting, for this pool pgp_num will slowly increase up until it reaches the pg_num / pg_num_target.
IMO this blog post best covers the issue which you are looking to undertake: https://ceph.io/en/news/blog/2019/new-in-nautilus-pg-merging-and-autotuning/
Respectfully,
*Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Jan 30, 2024 at 9:38 AM Michel Niyoyita <micou12@gmail.com> wrote:
Thanks for your advices Wes, below is what ceph osd df tree shows , the increase of pg_num of the production cluster will not affect the performance or crush ? how long it can takes to finish?
ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 433.11841 - 433 TiB 151 TiB 67 TiB 364 MiB 210 GiB 282 TiB 34.86 1.00 - root default -3 144.37280 - 144 TiB 50 TiB 22 TiB 121 MiB 70 GiB 94 TiB 34.86 1.00 - host ceph-osd1 2 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1021 GiB 5.4 MiB 3.7 GiB 6.3 TiB 30.40 0.87 19 up osd.2 3 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 931 GiB 4.1 MiB 3.5 GiB 6.4 TiB 29.43 0.84 29 up osd.3 6 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.5 TiB 8.1 MiB 4.5 GiB 5.8 TiB 36.09 1.04 20 up osd.6 9 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.0 TiB 6.6 MiB 3.8 GiB 6.2 TiB 30.97 0.89 23 up osd.9 12 hdd 9.02330 1.00000 9.0 TiB 4.0 TiB 2.3 TiB 13 MiB 6.1 GiB 5.0 TiB 44.68 1.28 30 up osd.12 15 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.8 TiB 9.2 MiB 5.2 GiB 5.5 TiB 38.99 1.12 30 up osd.15 18 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.5 MiB 4.0 GiB 6.1 TiB 32.80 0.94 21 up osd.18 22 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.9 TiB 10 MiB 5.4 GiB 5.4 TiB 40.25 1.15 22 up osd.22 25 hdd 9.02330 1.00000 9.0 TiB 3.9 TiB 2.1 TiB 12 MiB 5.7 GiB 5.1 TiB 42.94 1.23 22 up osd.25 28 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.5 MiB 4.1 GiB 5.9 TiB 34.87 1.00 21 up osd.28 32 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1017 GiB 4.8 MiB 3.7 GiB 6.3 TiB 30.36 0.87 27 up osd.32 35 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.3 TiB 7.2 MiB 4.2 GiB 6.0 TiB 33.73 0.97 21 up osd.35 38 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.3 MiB 4.1 GiB 5.9 TiB 34.57 0.99 24 up osd.38 41 hdd 9.02330 1.00000 9.0 TiB 2.9 TiB 1.2 TiB 6.2 MiB 4.0 GiB 6.1 TiB 32.49 0.93 24 up osd.41 44 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.3 MiB 4.4 GiB 5.9 TiB 34.87 1.00 29 up osd.44 47 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1016 GiB 5.4 MiB 3.6 GiB 6.3 TiB 30.35 0.87 23 up osd.47 -7 144.37280 - 144 TiB 50 TiB 22 TiB 122 MiB 70 GiB 94 TiB 34.86 1.00 - host ceph-osd2 1 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.7 MiB 3.8 GiB 6.2 TiB 31.00 0.89 27 up osd.1 5 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.5 TiB 7.3 MiB 4.5 GiB 5.8 TiB 35.45 1.02 27 up osd.5 8 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.6 TiB 8.3 MiB 4.7 GiB 5.7 TiB 36.85 1.06 30 up osd.8 10 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.5 MiB 4.5 GiB 5.9 TiB 34.87 1.00 20 up osd.10 13 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB 5.3 GiB 5.4 TiB 39.63 1.14 27 up osd.13 16 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 6.0 MiB 3.8 GiB 6.2 TiB 31.01 0.89 19 up osd.16 19 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.4 MiB 4.0 GiB 6.1 TiB 32.77 0.94 21 up osd.19 21 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.5 MiB 3.7 GiB 6.2 TiB 31.58 0.91 26 up osd.21 24 hdd 9.02330 1.00000 9.0 TiB 2.6 TiB 855 GiB 4.7 MiB 3.3 GiB 6.4 TiB 28.61 0.82 19 up osd.24 27 hdd 9.02330 1.00000 9.0 TiB 3.7 TiB 1.9 TiB 10 MiB 5.2 GiB 5.3 TiB 40.84 1.17 24 up osd.27 30 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.4 TiB 7.5 MiB 4.5 GiB 5.9 TiB 35.16 1.01 22 up osd.30 33 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 8.6 MiB 4.3 GiB 5.9 TiB 34.59 0.99 23 up osd.33 36 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 10 MiB 5.0 GiB 5.6 TiB 38.17 1.09 25 up osd.36 39 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 8.5 MiB 5.1 GiB 5.6 TiB 37.79 1.08 31 up osd.39 42 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB 5.2 GiB 5.4 TiB 39.68 1.14 23 up osd.42 45 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 964 GiB 5.1 MiB 3.5 GiB 6.3 TiB 29.78 0.85 21 up osd.45 -5 144.37280 - 144 TiB 50 TiB 22 TiB 121 MiB 70 GiB 94 TiB 34.86 1.00 - host ceph-osd3 0 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 934 GiB 4.9 MiB 3.4 GiB 6.4 TiB 29.47 0.85 21 up osd.0 4 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.5 MiB 4.1 GiB 6.1 TiB 32.73 0.94 22 up osd.4 7 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.8 TiB 9.2 MiB 5.1 GiB 5.5 TiB 39.02 1.12 30 up osd.7 11 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.9 TiB 10 MiB 5.1 GiB 5.4 TiB 39.97 1.15 27 up osd.11 14 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.7 TiB 10 MiB 5.1 GiB 5.6 TiB 38.24 1.10 27 up osd.14 17 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.4 MiB 4.1 GiB 6.0 TiB 33.09 0.95 23 up osd.17 20 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.6 MiB 3.8 GiB 6.2 TiB 31.55 0.90 20 up osd.20 23 hdd 9.02330 1.00000 9.0 TiB 2.6 TiB 828 GiB 4.0 MiB 3.3 GiB 6.5 TiB 28.32 0.81 23 up osd.23 26 hdd 9.02330 1.00000 9.0 TiB 2.9 TiB 1.2 TiB 5.8 MiB 3.8 GiB 6.1 TiB 32.12 0.92 26 up osd.26 29 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB 5.1 GiB 5.4 TiB 39.73 1.14 24 up osd.29 31 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.8 MiB 3.7 GiB 6.2 TiB 31.56 0.91 22 up osd.31 34 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.5 TiB 8.2 MiB 4.6 GiB 5.7 TiB 36.29 1.04 23 up osd.34 37 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.5 TiB 8.2 MiB 4.5 GiB 5.8 TiB 35.51 1.02 20 up osd.37 40 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 9.3 MiB 4.9 GiB 5.6 TiB 38.16 1.09 25 up osd.40 43 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.6 TiB 8.5 MiB 4.8 GiB 5.7 TiB 37.19 1.07 29 up osd.43 46 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 8.4 MiB 4.4 GiB 5.9 TiB 34.85 1.00 23 up osd.46 TOTAL 433 TiB 151 TiB 67 TiB 364 MiB 210 GiB 282 TiB 34.86 MIN/MAX VAR: 0.81/1.28 STDDEV: 3.95
Michel
On Tue, Jan 30, 2024 at 4:18 PM Wesley Dillingham <wes@wesdillingham.com> wrote:
I now concur you should increase the pg_num as a first step for this cluster. Disable the pg autoscaler for and increase the volumes pool to pg_num 256. Then likely re-asses and make the next power of 2 jump to 512 and probably beyond.
Keep in mind this is not going to fix your short term deep-scrub issue in fact it will increase the number of not scrubbed in time PGs until the pg_num change is complete. This is because OSDs dont scrub when they are backfilling.
I would sit on 256 for a couple weeks and let scrubs happen then continue past 256.
with the ultimate target of around 100-200 PGs per OSD which "ceph osd df tree" will show you in the PGs column.
Respectfully,
*Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Jan 30, 2024 at 3:16 AM Michel Niyoyita <micou12@gmail.com> wrote:
Dear team,
below is the output of ceph df command and the ceph version I am running
ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 433 TiB 282 TiB 151 TiB 151 TiB 34.82 TOTAL 433 TiB 282 TiB 151 TiB 151 TiB 34.82
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 1.1 MiB 3 3.2 MiB 0 73 TiB .rgw.root 2 32 3.7 KiB 8 96 KiB 0 73 TiB default.rgw.log 3 32 3.6 KiB 209 408 KiB 0 73 TiB default.rgw.control 4 32 0 B 8 0 B 0 73 TiB default.rgw.meta 5 32 382 B 2 24 KiB 0 73 TiB volumes 6 128 21 TiB 5.68M 62 TiB 22.09 73 TiB images 7 32 878 GiB 112.50k 2.6 TiB 1.17 73 TiB backups 8 32 0 B 0 0 B 0 73 TiB vms 9 32 881 GiB 174.30k 2.5 TiB 1.13 73 TiB testbench 10 32 0 B 0 0 B 0 73 TiB root@ceph-mon1:~# ceph --version ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) root@ceph-mon1:~#
please advise accordingly
Michel
On Mon, Jan 29, 2024 at 9:48 PM Frank Schilder <frans@dtu.dk> wrote:
You will have to look at the output of "ceph df" and make a decision to balance "objects per PG" and "GB per PG". Increase he PG count for the pools with the worst of these two numbers most such that it balances out as much as possible. If you have pools that see significantly more user-IO than others, prioritise these.
You will have to find out for your specific cluster, we can only give general guidelines. Make changes, run benchmarks, re-evaluate. Take the time for it. The better you know your cluster and your users, the better the end result will be.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Michel Niyoyita <micou12@gmail.com> Sent: Monday, January 29, 2024 2:04 PM To: Janne Johansson Cc: Frank Schilder; E Taka; ceph-users Subject: Re: [ceph-users] Re: 6 pgs not deep-scrubbed in time
This is how it is set , if you suggest to make some changes please advises.
Thank you.
ceph osd pool ls detail pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 1407 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr_devicehealth pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 flags hashpspool stripe_width 0 application rgw pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1394 flags hashpspool stripe_width 0 application rgw pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1395 flags hashpspool stripe_width 0 application rgw pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 108802 lfor 0/0/14812 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps_queue
[22d7~3,11561~2,11571~1,11573~1c,11594~6,1159b~f,115b0~1,115b3~1,115c3~1,115f3~1,115f5~e,11613~6,1161f~c,11637~1b,11660~1,11663~2,11673~1,116d1~c,116f5~10,11721~c]
pool 7 'images' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 94609 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd pool 8 'backups' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1399 flags hashpspool stripe_width 0 application rbd pool 9 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 108783 lfor 0/561/559 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps_queue [3fa~1,3fc~3,400~1,402~1] pool 10 'testbench' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20931 lfor 0/20931/20929 flags hashpspool stripe_width 0
On Mon, Jan 29, 2024 at 2:09 PM Michel Niyoyita <micou12@gmail.com <mailto: micou12@gmail.com>> wrote: Thank you Janne ,
no need of setting some flags like ceph osd set nodeep-scrub ???
Thank you
On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson <icepic.dz@gmail.com <mailto:icepic.dz@gmail.com>> wrote: Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita < micou12@gmail.com <mailto:micou12@gmail.com>>:
Thank you Frank ,
All disks are HDDs . Would like to know if I can increase the
number of PGs
live in production without a negative impact to the cluster. if yes which commands to use .
Yes. "ceph osd pool set <poolname> pg_num <number larger than before>" where the number usually should be a power of two that leads to a number of PGs per OSD between 100-200.
-- May the most significant bit of your life be positive.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
You may want to consider upgrading to 16.2.14 before you do the pg split. Respectfully, *Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham> On Tue, Jan 30, 2024 at 10:18 AM Michel Niyoyita <micou12@gmail.com> wrote:
I tried that on one of my pool (pool id 3) but the number of pgs not deep-scrubbed in time increased also from 55 to 100 but the number of PGs was increased. I set also autoscale to off mode. before continue to other pools would like to ask if so far there is no negative impact.
ceph -s cluster: id: cb0caedc-eb5b-42d1-a34f-96facfda8c27 health: HEALTH_WARN 100 pgs not deep-scrubbed in time
services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M) mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1 osd: 48 osds: 48 up (since 11M), 48 in (since 12M) rgw: 6 daemons active (6 hosts, 1 zones)
data: pools: 10 pools, 609 pgs objects: 6.03M objects, 23 TiB usage: 151 TiB used, 282 TiB / 433 TiB avail pgs: 603 active+clean 4 active+clean+scrubbing+deep 2 active+clean+scrubbing
io: client: 96 MiB/s rd, 573 MiB/s wr, 576 op/s rd, 648 op/s wr
root@ceph-osd3:/var/log# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 433 TiB 282 TiB 151 TiB 151 TiB 34.93 TOTAL 433 TiB 282 TiB 151 TiB 151 TiB 34.93
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 1.1 MiB 3 3.2 MiB 0 72 TiB .rgw.root 2 32 3.7 KiB 8 96 KiB 0 72 TiB default.rgw.log 3 256 3.6 KiB 204 408 KiB 0 72 TiB default.rgw.control 4 32 0 B 8 0 B 0 72 TiB default.rgw.meta 5 32 382 B 2 24 KiB 0 72 TiB volumes 6 128 21 TiB 5.74M 62 TiB 22.30 72 TiB images 7 32 878 GiB 112.50k 2.6 TiB 1.17 72 TiB backups 8 32 0 B 0 0 B 0 72 TiB vms 9 32 870 GiB 170.73k 2.5 TiB 1.13 72 TiB testbench 10 32 0 B 0 0 B 0 72 TiB
On Tue, Jan 30, 2024 at 5:05 PM Wesley Dillingham <wes@wesdillingham.com> wrote:
It will take a couple weeks to a couple months to complete is my best guess on 10TB spinners at ~40% full. The cluster should be usable throughout the process.
Keep in mind, you should disable the pg autoscaler on any pool which you are manually adjusting the pg_num for. Increasing the pg_num is called "pg splitting" you can google around for this to see how it will work etc.
There are a few knobs to increase or decrease the aggressiveness of the pg split, primarily these are osd_max_backfills and target_max_misplaced_ratio.
You can monitor the progress of the split by looking at "ceph osd pool ls detail" for the pool you are splitting, for this pool pgp_num will slowly increase up until it reaches the pg_num / pg_num_target.
IMO this blog post best covers the issue which you are looking to undertake: https://ceph.io/en/news/blog/2019/new-in-nautilus-pg-merging-and-autotuning/
Respectfully,
*Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Jan 30, 2024 at 9:38 AM Michel Niyoyita <micou12@gmail.com> wrote:
Thanks for your advices Wes, below is what ceph osd df tree shows , the increase of pg_num of the production cluster will not affect the performance or crush ? how long it can takes to finish?
ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 433.11841 - 433 TiB 151 TiB 67 TiB 364 MiB 210 GiB 282 TiB 34.86 1.00 - root default -3 144.37280 - 144 TiB 50 TiB 22 TiB 121 MiB 70 GiB 94 TiB 34.86 1.00 - host ceph-osd1 2 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1021 GiB 5.4 MiB 3.7 GiB 6.3 TiB 30.40 0.87 19 up osd.2 3 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 931 GiB 4.1 MiB 3.5 GiB 6.4 TiB 29.43 0.84 29 up osd.3 6 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.5 TiB 8.1 MiB 4.5 GiB 5.8 TiB 36.09 1.04 20 up osd.6 9 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.0 TiB 6.6 MiB 3.8 GiB 6.2 TiB 30.97 0.89 23 up osd.9 12 hdd 9.02330 1.00000 9.0 TiB 4.0 TiB 2.3 TiB 13 MiB 6.1 GiB 5.0 TiB 44.68 1.28 30 up osd.12 15 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.8 TiB 9.2 MiB 5.2 GiB 5.5 TiB 38.99 1.12 30 up osd.15 18 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.5 MiB 4.0 GiB 6.1 TiB 32.80 0.94 21 up osd.18 22 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.9 TiB 10 MiB 5.4 GiB 5.4 TiB 40.25 1.15 22 up osd.22 25 hdd 9.02330 1.00000 9.0 TiB 3.9 TiB 2.1 TiB 12 MiB 5.7 GiB 5.1 TiB 42.94 1.23 22 up osd.25 28 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.5 MiB 4.1 GiB 5.9 TiB 34.87 1.00 21 up osd.28 32 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1017 GiB 4.8 MiB 3.7 GiB 6.3 TiB 30.36 0.87 27 up osd.32 35 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.3 TiB 7.2 MiB 4.2 GiB 6.0 TiB 33.73 0.97 21 up osd.35 38 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.3 MiB 4.1 GiB 5.9 TiB 34.57 0.99 24 up osd.38 41 hdd 9.02330 1.00000 9.0 TiB 2.9 TiB 1.2 TiB 6.2 MiB 4.0 GiB 6.1 TiB 32.49 0.93 24 up osd.41 44 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.3 MiB 4.4 GiB 5.9 TiB 34.87 1.00 29 up osd.44 47 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1016 GiB 5.4 MiB 3.6 GiB 6.3 TiB 30.35 0.87 23 up osd.47 -7 144.37280 - 144 TiB 50 TiB 22 TiB 122 MiB 70 GiB 94 TiB 34.86 1.00 - host ceph-osd2 1 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.7 MiB 3.8 GiB 6.2 TiB 31.00 0.89 27 up osd.1 5 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.5 TiB 7.3 MiB 4.5 GiB 5.8 TiB 35.45 1.02 27 up osd.5 8 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.6 TiB 8.3 MiB 4.7 GiB 5.7 TiB 36.85 1.06 30 up osd.8 10 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.5 MiB 4.5 GiB 5.9 TiB 34.87 1.00 20 up osd.10 13 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB 5.3 GiB 5.4 TiB 39.63 1.14 27 up osd.13 16 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 6.0 MiB 3.8 GiB 6.2 TiB 31.01 0.89 19 up osd.16 19 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.4 MiB 4.0 GiB 6.1 TiB 32.77 0.94 21 up osd.19 21 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.5 MiB 3.7 GiB 6.2 TiB 31.58 0.91 26 up osd.21 24 hdd 9.02330 1.00000 9.0 TiB 2.6 TiB 855 GiB 4.7 MiB 3.3 GiB 6.4 TiB 28.61 0.82 19 up osd.24 27 hdd 9.02330 1.00000 9.0 TiB 3.7 TiB 1.9 TiB 10 MiB 5.2 GiB 5.3 TiB 40.84 1.17 24 up osd.27 30 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.4 TiB 7.5 MiB 4.5 GiB 5.9 TiB 35.16 1.01 22 up osd.30 33 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 8.6 MiB 4.3 GiB 5.9 TiB 34.59 0.99 23 up osd.33 36 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 10 MiB 5.0 GiB 5.6 TiB 38.17 1.09 25 up osd.36 39 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 8.5 MiB 5.1 GiB 5.6 TiB 37.79 1.08 31 up osd.39 42 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB 5.2 GiB 5.4 TiB 39.68 1.14 23 up osd.42 45 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 964 GiB 5.1 MiB 3.5 GiB 6.3 TiB 29.78 0.85 21 up osd.45 -5 144.37280 - 144 TiB 50 TiB 22 TiB 121 MiB 70 GiB 94 TiB 34.86 1.00 - host ceph-osd3 0 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 934 GiB 4.9 MiB 3.4 GiB 6.4 TiB 29.47 0.85 21 up osd.0 4 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.5 MiB 4.1 GiB 6.1 TiB 32.73 0.94 22 up osd.4 7 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.8 TiB 9.2 MiB 5.1 GiB 5.5 TiB 39.02 1.12 30 up osd.7 11 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.9 TiB 10 MiB 5.1 GiB 5.4 TiB 39.97 1.15 27 up osd.11 14 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.7 TiB 10 MiB 5.1 GiB 5.6 TiB 38.24 1.10 27 up osd.14 17 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.4 MiB 4.1 GiB 6.0 TiB 33.09 0.95 23 up osd.17 20 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.6 MiB 3.8 GiB 6.2 TiB 31.55 0.90 20 up osd.20 23 hdd 9.02330 1.00000 9.0 TiB 2.6 TiB 828 GiB 4.0 MiB 3.3 GiB 6.5 TiB 28.32 0.81 23 up osd.23 26 hdd 9.02330 1.00000 9.0 TiB 2.9 TiB 1.2 TiB 5.8 MiB 3.8 GiB 6.1 TiB 32.12 0.92 26 up osd.26 29 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB 5.1 GiB 5.4 TiB 39.73 1.14 24 up osd.29 31 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.8 MiB 3.7 GiB 6.2 TiB 31.56 0.91 22 up osd.31 34 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.5 TiB 8.2 MiB 4.6 GiB 5.7 TiB 36.29 1.04 23 up osd.34 37 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.5 TiB 8.2 MiB 4.5 GiB 5.8 TiB 35.51 1.02 20 up osd.37 40 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 9.3 MiB 4.9 GiB 5.6 TiB 38.16 1.09 25 up osd.40 43 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.6 TiB 8.5 MiB 4.8 GiB 5.7 TiB 37.19 1.07 29 up osd.43 46 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 8.4 MiB 4.4 GiB 5.9 TiB 34.85 1.00 23 up osd.46 TOTAL 433 TiB 151 TiB 67 TiB 364 MiB 210 GiB 282 TiB 34.86 MIN/MAX VAR: 0.81/1.28 STDDEV: 3.95
Michel
On Tue, Jan 30, 2024 at 4:18 PM Wesley Dillingham <wes@wesdillingham.com> wrote:
I now concur you should increase the pg_num as a first step for this cluster. Disable the pg autoscaler for and increase the volumes pool to pg_num 256. Then likely re-asses and make the next power of 2 jump to 512 and probably beyond.
Keep in mind this is not going to fix your short term deep-scrub issue in fact it will increase the number of not scrubbed in time PGs until the pg_num change is complete. This is because OSDs dont scrub when they are backfilling.
I would sit on 256 for a couple weeks and let scrubs happen then continue past 256.
with the ultimate target of around 100-200 PGs per OSD which "ceph osd df tree" will show you in the PGs column.
Respectfully,
*Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Jan 30, 2024 at 3:16 AM Michel Niyoyita <micou12@gmail.com> wrote:
Dear team,
below is the output of ceph df command and the ceph version I am running
ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 433 TiB 282 TiB 151 TiB 151 TiB 34.82 TOTAL 433 TiB 282 TiB 151 TiB 151 TiB 34.82
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 1.1 MiB 3 3.2 MiB 0 73 TiB .rgw.root 2 32 3.7 KiB 8 96 KiB 0 73 TiB default.rgw.log 3 32 3.6 KiB 209 408 KiB 0 73 TiB default.rgw.control 4 32 0 B 8 0 B 0 73 TiB default.rgw.meta 5 32 382 B 2 24 KiB 0 73 TiB volumes 6 128 21 TiB 5.68M 62 TiB 22.09 73 TiB images 7 32 878 GiB 112.50k 2.6 TiB 1.17 73 TiB backups 8 32 0 B 0 0 B 0 73 TiB vms 9 32 881 GiB 174.30k 2.5 TiB 1.13 73 TiB testbench 10 32 0 B 0 0 B 0 73 TiB root@ceph-mon1:~# ceph --version ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) root@ceph-mon1:~#
please advise accordingly
Michel
On Mon, Jan 29, 2024 at 9:48 PM Frank Schilder <frans@dtu.dk> wrote:
You will have to look at the output of "ceph df" and make a decision to balance "objects per PG" and "GB per PG". Increase he PG count for the pools with the worst of these two numbers most such that it balances out as much as possible. If you have pools that see significantly more user-IO than others, prioritise these.
You will have to find out for your specific cluster, we can only give general guidelines. Make changes, run benchmarks, re-evaluate. Take the time for it. The better you know your cluster and your users, the better the end result will be.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Michel Niyoyita <micou12@gmail.com> Sent: Monday, January 29, 2024 2:04 PM To: Janne Johansson Cc: Frank Schilder; E Taka; ceph-users Subject: Re: [ceph-users] Re: 6 pgs not deep-scrubbed in time
This is how it is set , if you suggest to make some changes please advises.
Thank you.
ceph osd pool ls detail pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 1407 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr_devicehealth pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 flags hashpspool stripe_width 0 application rgw pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1394 flags hashpspool stripe_width 0 application rgw pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1395 flags hashpspool stripe_width 0 application rgw pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 108802 lfor 0/0/14812 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps_queue
[22d7~3,11561~2,11571~1,11573~1c,11594~6,1159b~f,115b0~1,115b3~1,115c3~1,115f3~1,115f5~e,11613~6,1161f~c,11637~1b,11660~1,11663~2,11673~1,116d1~c,116f5~10,11721~c]
pool 7 'images' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 94609 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd pool 8 'backups' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1399 flags hashpspool stripe_width 0 application rbd pool 9 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 108783 lfor 0/561/559 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps_queue [3fa~1,3fc~3,400~1,402~1] pool 10 'testbench' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20931 lfor 0/20931/20929 flags hashpspool stripe_width 0
On Mon, Jan 29, 2024 at 2:09 PM Michel Niyoyita <micou12@gmail.com <mailto: micou12@gmail.com>> wrote: Thank you Janne ,
no need of setting some flags like ceph osd set nodeep-scrub ???
Thank you
On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson <icepic.dz@gmail.com <mailto:icepic.dz@gmail.com>> wrote: Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita < micou12@gmail.com <mailto:micou12@gmail.com>>: > > Thank you Frank , > > All disks are HDDs . Would like to know if I can increase the number of PGs > live in production without a negative impact to the cluster. if yes which > commands to use .
Yes. "ceph osd pool set <poolname> pg_num <number larger than before>" where the number usually should be a power of two that leads to a number of PGs per OSD between 100-200.
-- May the most significant bit of your life be positive.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
actually it seems the issue I had in mind was fixed in 16.2.11 so you should be fine. Respectfully, *Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham> On Tue, Jan 30, 2024 at 10:34 AM Wesley Dillingham <wes@wesdillingham.com> wrote:
You may want to consider upgrading to 16.2.14 before you do the pg split.
Respectfully,
*Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Jan 30, 2024 at 10:18 AM Michel Niyoyita <micou12@gmail.com> wrote:
I tried that on one of my pool (pool id 3) but the number of pgs not deep-scrubbed in time increased also from 55 to 100 but the number of PGs was increased. I set also autoscale to off mode. before continue to other pools would like to ask if so far there is no negative impact.
ceph -s cluster: id: cb0caedc-eb5b-42d1-a34f-96facfda8c27 health: HEALTH_WARN 100 pgs not deep-scrubbed in time
services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M) mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1 osd: 48 osds: 48 up (since 11M), 48 in (since 12M) rgw: 6 daemons active (6 hosts, 1 zones)
data: pools: 10 pools, 609 pgs objects: 6.03M objects, 23 TiB usage: 151 TiB used, 282 TiB / 433 TiB avail pgs: 603 active+clean 4 active+clean+scrubbing+deep 2 active+clean+scrubbing
io: client: 96 MiB/s rd, 573 MiB/s wr, 576 op/s rd, 648 op/s wr
root@ceph-osd3:/var/log# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 433 TiB 282 TiB 151 TiB 151 TiB 34.93 TOTAL 433 TiB 282 TiB 151 TiB 151 TiB 34.93
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 1.1 MiB 3 3.2 MiB 0 72 TiB .rgw.root 2 32 3.7 KiB 8 96 KiB 0 72 TiB default.rgw.log 3 256 3.6 KiB 204 408 KiB 0 72 TiB default.rgw.control 4 32 0 B 8 0 B 0 72 TiB default.rgw.meta 5 32 382 B 2 24 KiB 0 72 TiB volumes 6 128 21 TiB 5.74M 62 TiB 22.30 72 TiB images 7 32 878 GiB 112.50k 2.6 TiB 1.17 72 TiB backups 8 32 0 B 0 0 B 0 72 TiB vms 9 32 870 GiB 170.73k 2.5 TiB 1.13 72 TiB testbench 10 32 0 B 0 0 B 0 72 TiB
On Tue, Jan 30, 2024 at 5:05 PM Wesley Dillingham <wes@wesdillingham.com> wrote:
It will take a couple weeks to a couple months to complete is my best guess on 10TB spinners at ~40% full. The cluster should be usable throughout the process.
Keep in mind, you should disable the pg autoscaler on any pool which you are manually adjusting the pg_num for. Increasing the pg_num is called "pg splitting" you can google around for this to see how it will work etc.
There are a few knobs to increase or decrease the aggressiveness of the pg split, primarily these are osd_max_backfills and target_max_misplaced_ratio.
You can monitor the progress of the split by looking at "ceph osd pool ls detail" for the pool you are splitting, for this pool pgp_num will slowly increase up until it reaches the pg_num / pg_num_target.
IMO this blog post best covers the issue which you are looking to undertake: https://ceph.io/en/news/blog/2019/new-in-nautilus-pg-merging-and-autotuning/
Respectfully,
*Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Jan 30, 2024 at 9:38 AM Michel Niyoyita <micou12@gmail.com> wrote:
Thanks for your advices Wes, below is what ceph osd df tree shows , the increase of pg_num of the production cluster will not affect the performance or crush ? how long it can takes to finish?
ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 433.11841 - 433 TiB 151 TiB 67 TiB 364 MiB 210 GiB 282 TiB 34.86 1.00 - root default -3 144.37280 - 144 TiB 50 TiB 22 TiB 121 MiB 70 GiB 94 TiB 34.86 1.00 - host ceph-osd1 2 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1021 GiB 5.4 MiB 3.7 GiB 6.3 TiB 30.40 0.87 19 up osd.2 3 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 931 GiB 4.1 MiB 3.5 GiB 6.4 TiB 29.43 0.84 29 up osd.3 6 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.5 TiB 8.1 MiB 4.5 GiB 5.8 TiB 36.09 1.04 20 up osd.6 9 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.0 TiB 6.6 MiB 3.8 GiB 6.2 TiB 30.97 0.89 23 up osd.9 12 hdd 9.02330 1.00000 9.0 TiB 4.0 TiB 2.3 TiB 13 MiB 6.1 GiB 5.0 TiB 44.68 1.28 30 up osd.12 15 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.8 TiB 9.2 MiB 5.2 GiB 5.5 TiB 38.99 1.12 30 up osd.15 18 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.5 MiB 4.0 GiB 6.1 TiB 32.80 0.94 21 up osd.18 22 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.9 TiB 10 MiB 5.4 GiB 5.4 TiB 40.25 1.15 22 up osd.22 25 hdd 9.02330 1.00000 9.0 TiB 3.9 TiB 2.1 TiB 12 MiB 5.7 GiB 5.1 TiB 42.94 1.23 22 up osd.25 28 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.5 MiB 4.1 GiB 5.9 TiB 34.87 1.00 21 up osd.28 32 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1017 GiB 4.8 MiB 3.7 GiB 6.3 TiB 30.36 0.87 27 up osd.32 35 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.3 TiB 7.2 MiB 4.2 GiB 6.0 TiB 33.73 0.97 21 up osd.35 38 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.3 MiB 4.1 GiB 5.9 TiB 34.57 0.99 24 up osd.38 41 hdd 9.02330 1.00000 9.0 TiB 2.9 TiB 1.2 TiB 6.2 MiB 4.0 GiB 6.1 TiB 32.49 0.93 24 up osd.41 44 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.3 MiB 4.4 GiB 5.9 TiB 34.87 1.00 29 up osd.44 47 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1016 GiB 5.4 MiB 3.6 GiB 6.3 TiB 30.35 0.87 23 up osd.47 -7 144.37280 - 144 TiB 50 TiB 22 TiB 122 MiB 70 GiB 94 TiB 34.86 1.00 - host ceph-osd2 1 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.7 MiB 3.8 GiB 6.2 TiB 31.00 0.89 27 up osd.1 5 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.5 TiB 7.3 MiB 4.5 GiB 5.8 TiB 35.45 1.02 27 up osd.5 8 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.6 TiB 8.3 MiB 4.7 GiB 5.7 TiB 36.85 1.06 30 up osd.8 10 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.5 MiB 4.5 GiB 5.9 TiB 34.87 1.00 20 up osd.10 13 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB 5.3 GiB 5.4 TiB 39.63 1.14 27 up osd.13 16 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 6.0 MiB 3.8 GiB 6.2 TiB 31.01 0.89 19 up osd.16 19 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.4 MiB 4.0 GiB 6.1 TiB 32.77 0.94 21 up osd.19 21 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.5 MiB 3.7 GiB 6.2 TiB 31.58 0.91 26 up osd.21 24 hdd 9.02330 1.00000 9.0 TiB 2.6 TiB 855 GiB 4.7 MiB 3.3 GiB 6.4 TiB 28.61 0.82 19 up osd.24 27 hdd 9.02330 1.00000 9.0 TiB 3.7 TiB 1.9 TiB 10 MiB 5.2 GiB 5.3 TiB 40.84 1.17 24 up osd.27 30 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.4 TiB 7.5 MiB 4.5 GiB 5.9 TiB 35.16 1.01 22 up osd.30 33 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 8.6 MiB 4.3 GiB 5.9 TiB 34.59 0.99 23 up osd.33 36 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 10 MiB 5.0 GiB 5.6 TiB 38.17 1.09 25 up osd.36 39 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 8.5 MiB 5.1 GiB 5.6 TiB 37.79 1.08 31 up osd.39 42 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB 5.2 GiB 5.4 TiB 39.68 1.14 23 up osd.42 45 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 964 GiB 5.1 MiB 3.5 GiB 6.3 TiB 29.78 0.85 21 up osd.45 -5 144.37280 - 144 TiB 50 TiB 22 TiB 121 MiB 70 GiB 94 TiB 34.86 1.00 - host ceph-osd3 0 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 934 GiB 4.9 MiB 3.4 GiB 6.4 TiB 29.47 0.85 21 up osd.0 4 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.5 MiB 4.1 GiB 6.1 TiB 32.73 0.94 22 up osd.4 7 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.8 TiB 9.2 MiB 5.1 GiB 5.5 TiB 39.02 1.12 30 up osd.7 11 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.9 TiB 10 MiB 5.1 GiB 5.4 TiB 39.97 1.15 27 up osd.11 14 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.7 TiB 10 MiB 5.1 GiB 5.6 TiB 38.24 1.10 27 up osd.14 17 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.4 MiB 4.1 GiB 6.0 TiB 33.09 0.95 23 up osd.17 20 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.6 MiB 3.8 GiB 6.2 TiB 31.55 0.90 20 up osd.20 23 hdd 9.02330 1.00000 9.0 TiB 2.6 TiB 828 GiB 4.0 MiB 3.3 GiB 6.5 TiB 28.32 0.81 23 up osd.23 26 hdd 9.02330 1.00000 9.0 TiB 2.9 TiB 1.2 TiB 5.8 MiB 3.8 GiB 6.1 TiB 32.12 0.92 26 up osd.26 29 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB 5.1 GiB 5.4 TiB 39.73 1.14 24 up osd.29 31 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.8 MiB 3.7 GiB 6.2 TiB 31.56 0.91 22 up osd.31 34 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.5 TiB 8.2 MiB 4.6 GiB 5.7 TiB 36.29 1.04 23 up osd.34 37 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.5 TiB 8.2 MiB 4.5 GiB 5.8 TiB 35.51 1.02 20 up osd.37 40 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 9.3 MiB 4.9 GiB 5.6 TiB 38.16 1.09 25 up osd.40 43 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.6 TiB 8.5 MiB 4.8 GiB 5.7 TiB 37.19 1.07 29 up osd.43 46 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 8.4 MiB 4.4 GiB 5.9 TiB 34.85 1.00 23 up osd.46 TOTAL 433 TiB 151 TiB 67 TiB 364 MiB 210 GiB 282 TiB 34.86 MIN/MAX VAR: 0.81/1.28 STDDEV: 3.95
Michel
On Tue, Jan 30, 2024 at 4:18 PM Wesley Dillingham < wes@wesdillingham.com> wrote:
I now concur you should increase the pg_num as a first step for this cluster. Disable the pg autoscaler for and increase the volumes pool to pg_num 256. Then likely re-asses and make the next power of 2 jump to 512 and probably beyond.
Keep in mind this is not going to fix your short term deep-scrub issue in fact it will increase the number of not scrubbed in time PGs until the pg_num change is complete. This is because OSDs dont scrub when they are backfilling.
I would sit on 256 for a couple weeks and let scrubs happen then continue past 256.
with the ultimate target of around 100-200 PGs per OSD which "ceph osd df tree" will show you in the PGs column.
Respectfully,
*Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Jan 30, 2024 at 3:16 AM Michel Niyoyita <micou12@gmail.com> wrote:
Dear team,
below is the output of ceph df command and the ceph version I am running
ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 433 TiB 282 TiB 151 TiB 151 TiB 34.82 TOTAL 433 TiB 282 TiB 151 TiB 151 TiB 34.82
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 1.1 MiB 3 3.2 MiB 0 73 TiB .rgw.root 2 32 3.7 KiB 8 96 KiB 0 73 TiB default.rgw.log 3 32 3.6 KiB 209 408 KiB 0 73 TiB default.rgw.control 4 32 0 B 8 0 B 0 73 TiB default.rgw.meta 5 32 382 B 2 24 KiB 0 73 TiB volumes 6 128 21 TiB 5.68M 62 TiB 22.09 73 TiB images 7 32 878 GiB 112.50k 2.6 TiB 1.17 73 TiB backups 8 32 0 B 0 0 B 0 73 TiB vms 9 32 881 GiB 174.30k 2.5 TiB 1.13 73 TiB testbench 10 32 0 B 0 0 B 0 73 TiB root@ceph-mon1:~# ceph --version ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) root@ceph-mon1:~#
please advise accordingly
Michel
On Mon, Jan 29, 2024 at 9:48 PM Frank Schilder <frans@dtu.dk> wrote:
> You will have to look at the output of "ceph df" and make a decision to > balance "objects per PG" and "GB per PG". Increase he PG count for the > pools with the worst of these two numbers most such that it balances out as > much as possible. If you have pools that see significantly more user-IO > than others, prioritise these. > > You will have to find out for your specific cluster, we can only give > general guidelines. Make changes, run benchmarks, re-evaluate. Take the > time for it. The better you know your cluster and your users, the better > the end result will be. > > Best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Michel Niyoyita <micou12@gmail.com> > Sent: Monday, January 29, 2024 2:04 PM > To: Janne Johansson > Cc: Frank Schilder; E Taka; ceph-users > Subject: Re: [ceph-users] Re: 6 pgs not deep-scrubbed in time > > This is how it is set , if you suggest to make some changes please advises. > > Thank you. > > > ceph osd pool ls detail > pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 > object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 1407 > flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application > mgr_devicehealth > pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 flags > hashpspool stripe_width 0 application rgw > pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change > 1394 flags hashpspool stripe_width 0 application rgw > pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change > 1395 flags hashpspool stripe_width 0 application rgw > pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change > 1396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw > pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash > rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 108802 lfor > 0/0/14812 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd > removed_snaps_queue > [22d7~3,11561~2,11571~1,11573~1c,11594~6,1159b~f,115b0~1,115b3~1,115c3~1,115f3~1,115f5~e,11613~6,1161f~c,11637~1b,11660~1,11663~2,11673~1,116d1~c,116f5~10,11721~c] > pool 7 'images' replicated size 3 min_size 2 crush_rule 0 object_hash > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 94609 flags > hashpspool,selfmanaged_snaps stripe_width 0 application rbd > pool 8 'backups' replicated size 3 min_size 2 crush_rule 0 object_hash > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1399 flags > hashpspool stripe_width 0 application rbd > pool 9 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 108783 lfor > 0/561/559 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd > removed_snaps_queue [3fa~1,3fc~3,400~1,402~1] > pool 10 'testbench' replicated size 3 min_size 2 crush_rule 0 object_hash > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20931 lfor > 0/20931/20929 flags hashpspool stripe_width 0 > > > On Mon, Jan 29, 2024 at 2:09 PM Michel Niyoyita <micou12@gmail.com <mailto: > micou12@gmail.com>> wrote: > Thank you Janne , > > no need of setting some flags like ceph osd set nodeep-scrub ??? > > Thank you > > On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson < icepic.dz@gmail.com > <mailto:icepic.dz@gmail.com>> wrote: > Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita < micou12@gmail.com > <mailto:micou12@gmail.com>>: > > > > Thank you Frank , > > > > All disks are HDDs . Would like to know if I can increase the number of > PGs > > live in production without a negative impact to the cluster. if yes which > > commands to use . > > Yes. "ceph osd pool set <poolname> pg_num <number larger than before>" > where the number usually should be a power of two that leads to a > number of PGs per OSD between 100-200. > > -- > May the most significant bit of your life be positive. > _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks Very much Wesley, We have decided to restart one host among three osds hosts. before doing that I need the advices of the team . these are flags I want to set before restart. 'ceph osd set noout' 'ceph osd set nobackfill' 'ceph osd set norecover' 'ceph osd set norebalance' 'ceph osd set nodown' 'ceph osd set pause' 'ceph osd set nodeep-scrub' 'ceph osd set noscrub' Would like to ask if this can be enough to set and restart the host safely . the cluster has 3 as replicas. will the cluster still be accessible while restart the hosts? after restarting I will unset the flags. Kindly advise. Michel On Tue, 30 Jan 2024, 17:44 Wesley Dillingham, <wes@wesdillingham.com> wrote:
actually it seems the issue I had in mind was fixed in 16.2.11 so you should be fine.
Respectfully,
*Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Jan 30, 2024 at 10:34 AM Wesley Dillingham <wes@wesdillingham.com> wrote:
You may want to consider upgrading to 16.2.14 before you do the pg split.
Respectfully,
*Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Jan 30, 2024 at 10:18 AM Michel Niyoyita <micou12@gmail.com> wrote:
I tried that on one of my pool (pool id 3) but the number of pgs not deep-scrubbed in time increased also from 55 to 100 but the number of PGs was increased. I set also autoscale to off mode. before continue to other pools would like to ask if so far there is no negative impact.
ceph -s cluster: id: cb0caedc-eb5b-42d1-a34f-96facfda8c27 health: HEALTH_WARN 100 pgs not deep-scrubbed in time
services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M) mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1 osd: 48 osds: 48 up (since 11M), 48 in (since 12M) rgw: 6 daemons active (6 hosts, 1 zones)
data: pools: 10 pools, 609 pgs objects: 6.03M objects, 23 TiB usage: 151 TiB used, 282 TiB / 433 TiB avail pgs: 603 active+clean 4 active+clean+scrubbing+deep 2 active+clean+scrubbing
io: client: 96 MiB/s rd, 573 MiB/s wr, 576 op/s rd, 648 op/s wr
root@ceph-osd3:/var/log# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 433 TiB 282 TiB 151 TiB 151 TiB 34.93 TOTAL 433 TiB 282 TiB 151 TiB 151 TiB 34.93
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 1.1 MiB 3 3.2 MiB 0 72 TiB .rgw.root 2 32 3.7 KiB 8 96 KiB 0 72 TiB default.rgw.log 3 256 3.6 KiB 204 408 KiB 0 72 TiB default.rgw.control 4 32 0 B 8 0 B 0 72 TiB default.rgw.meta 5 32 382 B 2 24 KiB 0 72 TiB volumes 6 128 21 TiB 5.74M 62 TiB 22.30 72 TiB images 7 32 878 GiB 112.50k 2.6 TiB 1.17 72 TiB backups 8 32 0 B 0 0 B 0 72 TiB vms 9 32 870 GiB 170.73k 2.5 TiB 1.13 72 TiB testbench 10 32 0 B 0 0 B 0 72 TiB
On Tue, Jan 30, 2024 at 5:05 PM Wesley Dillingham <wes@wesdillingham.com> wrote:
It will take a couple weeks to a couple months to complete is my best guess on 10TB spinners at ~40% full. The cluster should be usable throughout the process.
Keep in mind, you should disable the pg autoscaler on any pool which you are manually adjusting the pg_num for. Increasing the pg_num is called "pg splitting" you can google around for this to see how it will work etc.
There are a few knobs to increase or decrease the aggressiveness of the pg split, primarily these are osd_max_backfills and target_max_misplaced_ratio.
You can monitor the progress of the split by looking at "ceph osd pool ls detail" for the pool you are splitting, for this pool pgp_num will slowly increase up until it reaches the pg_num / pg_num_target.
IMO this blog post best covers the issue which you are looking to undertake: https://ceph.io/en/news/blog/2019/new-in-nautilus-pg-merging-and-autotuning/
Respectfully,
*Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Jan 30, 2024 at 9:38 AM Michel Niyoyita <micou12@gmail.com> wrote:
Thanks for your advices Wes, below is what ceph osd df tree shows , the increase of pg_num of the production cluster will not affect the performance or crush ? how long it can takes to finish?
ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 433.11841 - 433 TiB 151 TiB 67 TiB 364 MiB 210 GiB 282 TiB 34.86 1.00 - root default -3 144.37280 - 144 TiB 50 TiB 22 TiB 121 MiB 70 GiB 94 TiB 34.86 1.00 - host ceph-osd1 2 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1021 GiB 5.4 MiB 3.7 GiB 6.3 TiB 30.40 0.87 19 up osd.2 3 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 931 GiB 4.1 MiB 3.5 GiB 6.4 TiB 29.43 0.84 29 up osd.3 6 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.5 TiB 8.1 MiB 4.5 GiB 5.8 TiB 36.09 1.04 20 up osd.6 9 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.0 TiB 6.6 MiB 3.8 GiB 6.2 TiB 30.97 0.89 23 up osd.9 12 hdd 9.02330 1.00000 9.0 TiB 4.0 TiB 2.3 TiB 13 MiB 6.1 GiB 5.0 TiB 44.68 1.28 30 up osd.12 15 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.8 TiB 9.2 MiB 5.2 GiB 5.5 TiB 38.99 1.12 30 up osd.15 18 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.5 MiB 4.0 GiB 6.1 TiB 32.80 0.94 21 up osd.18 22 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.9 TiB 10 MiB 5.4 GiB 5.4 TiB 40.25 1.15 22 up osd.22 25 hdd 9.02330 1.00000 9.0 TiB 3.9 TiB 2.1 TiB 12 MiB 5.7 GiB 5.1 TiB 42.94 1.23 22 up osd.25 28 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.5 MiB 4.1 GiB 5.9 TiB 34.87 1.00 21 up osd.28 32 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1017 GiB 4.8 MiB 3.7 GiB 6.3 TiB 30.36 0.87 27 up osd.32 35 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.3 TiB 7.2 MiB 4.2 GiB 6.0 TiB 33.73 0.97 21 up osd.35 38 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.3 MiB 4.1 GiB 5.9 TiB 34.57 0.99 24 up osd.38 41 hdd 9.02330 1.00000 9.0 TiB 2.9 TiB 1.2 TiB 6.2 MiB 4.0 GiB 6.1 TiB 32.49 0.93 24 up osd.41 44 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.3 MiB 4.4 GiB 5.9 TiB 34.87 1.00 29 up osd.44 47 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1016 GiB 5.4 MiB 3.6 GiB 6.3 TiB 30.35 0.87 23 up osd.47 -7 144.37280 - 144 TiB 50 TiB 22 TiB 122 MiB 70 GiB 94 TiB 34.86 1.00 - host ceph-osd2 1 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.7 MiB 3.8 GiB 6.2 TiB 31.00 0.89 27 up osd.1 5 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.5 TiB 7.3 MiB 4.5 GiB 5.8 TiB 35.45 1.02 27 up osd.5 8 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.6 TiB 8.3 MiB 4.7 GiB 5.7 TiB 36.85 1.06 30 up osd.8 10 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.5 MiB 4.5 GiB 5.9 TiB 34.87 1.00 20 up osd.10 13 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB 5.3 GiB 5.4 TiB 39.63 1.14 27 up osd.13 16 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 6.0 MiB 3.8 GiB 6.2 TiB 31.01 0.89 19 up osd.16 19 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.4 MiB 4.0 GiB 6.1 TiB 32.77 0.94 21 up osd.19 21 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.5 MiB 3.7 GiB 6.2 TiB 31.58 0.91 26 up osd.21 24 hdd 9.02330 1.00000 9.0 TiB 2.6 TiB 855 GiB 4.7 MiB 3.3 GiB 6.4 TiB 28.61 0.82 19 up osd.24 27 hdd 9.02330 1.00000 9.0 TiB 3.7 TiB 1.9 TiB 10 MiB 5.2 GiB 5.3 TiB 40.84 1.17 24 up osd.27 30 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.4 TiB 7.5 MiB 4.5 GiB 5.9 TiB 35.16 1.01 22 up osd.30 33 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 8.6 MiB 4.3 GiB 5.9 TiB 34.59 0.99 23 up osd.33 36 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 10 MiB 5.0 GiB 5.6 TiB 38.17 1.09 25 up osd.36 39 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 8.5 MiB 5.1 GiB 5.6 TiB 37.79 1.08 31 up osd.39 42 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB 5.2 GiB 5.4 TiB 39.68 1.14 23 up osd.42 45 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 964 GiB 5.1 MiB 3.5 GiB 6.3 TiB 29.78 0.85 21 up osd.45 -5 144.37280 - 144 TiB 50 TiB 22 TiB 121 MiB 70 GiB 94 TiB 34.86 1.00 - host ceph-osd3 0 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 934 GiB 4.9 MiB 3.4 GiB 6.4 TiB 29.47 0.85 21 up osd.0 4 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.5 MiB 4.1 GiB 6.1 TiB 32.73 0.94 22 up osd.4 7 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.8 TiB 9.2 MiB 5.1 GiB 5.5 TiB 39.02 1.12 30 up osd.7 11 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.9 TiB 10 MiB 5.1 GiB 5.4 TiB 39.97 1.15 27 up osd.11 14 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.7 TiB 10 MiB 5.1 GiB 5.6 TiB 38.24 1.10 27 up osd.14 17 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.4 MiB 4.1 GiB 6.0 TiB 33.09 0.95 23 up osd.17 20 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.6 MiB 3.8 GiB 6.2 TiB 31.55 0.90 20 up osd.20 23 hdd 9.02330 1.00000 9.0 TiB 2.6 TiB 828 GiB 4.0 MiB 3.3 GiB 6.5 TiB 28.32 0.81 23 up osd.23 26 hdd 9.02330 1.00000 9.0 TiB 2.9 TiB 1.2 TiB 5.8 MiB 3.8 GiB 6.1 TiB 32.12 0.92 26 up osd.26 29 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB 5.1 GiB 5.4 TiB 39.73 1.14 24 up osd.29 31 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.8 MiB 3.7 GiB 6.2 TiB 31.56 0.91 22 up osd.31 34 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.5 TiB 8.2 MiB 4.6 GiB 5.7 TiB 36.29 1.04 23 up osd.34 37 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.5 TiB 8.2 MiB 4.5 GiB 5.8 TiB 35.51 1.02 20 up osd.37 40 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 9.3 MiB 4.9 GiB 5.6 TiB 38.16 1.09 25 up osd.40 43 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.6 TiB 8.5 MiB 4.8 GiB 5.7 TiB 37.19 1.07 29 up osd.43 46 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 8.4 MiB 4.4 GiB 5.9 TiB 34.85 1.00 23 up osd.46 TOTAL 433 TiB 151 TiB 67 TiB 364 MiB 210 GiB 282 TiB 34.86 MIN/MAX VAR: 0.81/1.28 STDDEV: 3.95
Michel
On Tue, Jan 30, 2024 at 4:18 PM Wesley Dillingham < wes@wesdillingham.com> wrote:
I now concur you should increase the pg_num as a first step for this cluster. Disable the pg autoscaler for and increase the volumes pool to pg_num 256. Then likely re-asses and make the next power of 2 jump to 512 and probably beyond.
Keep in mind this is not going to fix your short term deep-scrub issue in fact it will increase the number of not scrubbed in time PGs until the pg_num change is complete. This is because OSDs dont scrub when they are backfilling.
I would sit on 256 for a couple weeks and let scrubs happen then continue past 256.
with the ultimate target of around 100-200 PGs per OSD which "ceph osd df tree" will show you in the PGs column.
Respectfully,
*Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Jan 30, 2024 at 3:16 AM Michel Niyoyita <micou12@gmail.com> wrote:
> Dear team, > > below is the output of ceph df command and the ceph version I am > running > > ceph df > --- RAW STORAGE --- > CLASS SIZE AVAIL USED RAW USED %RAW USED > hdd 433 TiB 282 TiB 151 TiB 151 TiB 34.82 > TOTAL 433 TiB 282 TiB 151 TiB 151 TiB 34.82 > > --- POOLS --- > POOL ID PGS STORED OBJECTS USED %USED > MAX AVAIL > device_health_metrics 1 1 1.1 MiB 3 3.2 MiB 0 > 73 TiB > .rgw.root 2 32 3.7 KiB 8 96 KiB 0 > 73 TiB > default.rgw.log 3 32 3.6 KiB 209 408 KiB 0 > 73 TiB > default.rgw.control 4 32 0 B 8 0 B 0 > 73 TiB > default.rgw.meta 5 32 382 B 2 24 KiB 0 > 73 TiB > volumes 6 128 21 TiB 5.68M 62 TiB 22.09 > 73 TiB > images 7 32 878 GiB 112.50k 2.6 TiB 1.17 > 73 TiB > backups 8 32 0 B 0 0 B 0 > 73 TiB > vms 9 32 881 GiB 174.30k 2.5 TiB 1.13 > 73 TiB > testbench 10 32 0 B 0 0 B 0 > 73 TiB > root@ceph-mon1:~# ceph --version > ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) > pacific > (stable) > root@ceph-mon1:~# > > please advise accordingly > > Michel > > On Mon, Jan 29, 2024 at 9:48 PM Frank Schilder <frans@dtu.dk> wrote: > > > You will have to look at the output of "ceph df" and make a > decision to > > balance "objects per PG" and "GB per PG". Increase he PG count for > the > > pools with the worst of these two numbers most such that it > balances out as > > much as possible. If you have pools that see significantly more > user-IO > > than others, prioritise these. > > > > You will have to find out for your specific cluster, we can only > give > > general guidelines. Make changes, run benchmarks, re-evaluate. > Take the > > time for it. The better you know your cluster and your users, the > better > > the end result will be. > > > > Best regards, > > ================= > > Frank Schilder > > AIT Risø Campus > > Bygning 109, rum S14 > > > > ________________________________________ > > From: Michel Niyoyita <micou12@gmail.com> > > Sent: Monday, January 29, 2024 2:04 PM > > To: Janne Johansson > > Cc: Frank Schilder; E Taka; ceph-users > > Subject: Re: [ceph-users] Re: 6 pgs not deep-scrubbed in time > > > > This is how it is set , if you suggest to make some changes please > advises. > > > > Thank you. > > > > > > ceph osd pool ls detail > > pool 1 'device_health_metrics' replicated size 3 min_size 2 > crush_rule 0 > > object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on > last_change 1407 > > flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 > application > > mgr_devicehealth > > pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 > object_hash > > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 > flags > > hashpspool stripe_width 0 application rgw > > pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 > > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on > last_change > > 1394 flags hashpspool stripe_width 0 application rgw > > pool 4 'default.rgw.control' replicated size 3 min_size 2 > crush_rule 0 > > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on > last_change > > 1395 flags hashpspool stripe_width 0 application rgw > > pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 > > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on > last_change > > 1396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 > application rgw > > pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 > object_hash > > rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change > 108802 lfor > > 0/0/14812 flags hashpspool,selfmanaged_snaps stripe_width 0 > application rbd > > removed_snaps_queue > > > [22d7~3,11561~2,11571~1,11573~1c,11594~6,1159b~f,115b0~1,115b3~1,115c3~1,115f3~1,115f5~e,11613~6,1161f~c,11637~1b,11660~1,11663~2,11673~1,116d1~c,116f5~10,11721~c] > > pool 7 'images' replicated size 3 min_size 2 crush_rule 0 > object_hash > > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 94609 > flags > > hashpspool,selfmanaged_snaps stripe_width 0 application rbd > > pool 8 'backups' replicated size 3 min_size 2 crush_rule 0 > object_hash > > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1399 > flags > > hashpspool stripe_width 0 application rbd > > pool 9 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash > > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 108783 > lfor > > 0/561/559 flags hashpspool,selfmanaged_snaps stripe_width 0 > application rbd > > removed_snaps_queue [3fa~1,3fc~3,400~1,402~1] > > pool 10 'testbench' replicated size 3 min_size 2 crush_rule 0 > object_hash > > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20931 > lfor > > 0/20931/20929 flags hashpspool stripe_width 0 > > > > > > On Mon, Jan 29, 2024 at 2:09 PM Michel Niyoyita <micou12@gmail.com > <mailto: > > micou12@gmail.com>> wrote: > > Thank you Janne , > > > > no need of setting some flags like ceph osd set nodeep-scrub ??? > > > > Thank you > > > > On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson < > icepic.dz@gmail.com > > <mailto:icepic.dz@gmail.com>> wrote: > > Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita < > micou12@gmail.com > > <mailto:micou12@gmail.com>>: > > > > > > Thank you Frank , > > > > > > All disks are HDDs . Would like to know if I can increase the > number of > > PGs > > > live in production without a negative impact to the cluster. if > yes which > > > commands to use . > > > > Yes. "ceph osd pool set <poolname> pg_num <number larger than > before>" > > where the number usually should be a power of two that leads to a > > number of PGs per OSD between 100-200. > > > > -- > > May the most significant bit of your life be positive. > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io >
pause and nodown is not a good option to set, that will certainly make clients stop IO. Pause will stop it immediately, and nodown will stop IO when the OSD processes stop running on this host. When we do service on a host, we set "noout" and "nobackfill", that is enough for reboots, OS upgrades and simple disk exchanges. The PGs on this one host will be degraded during the down period, but IO continues. Of course this is when the cluster was healthy to begin with (not counting "not scrubbed in time" warnings, they don't matter in this case.) Den tors 1 feb. 2024 kl 12:21 skrev Michel Niyoyita <micou12@gmail.com>:
Thanks Very much Wesley,
We have decided to restart one host among three osds hosts. before doing that I need the advices of the team . these are flags I want to set before restart.
'ceph osd set noout' 'ceph osd set nobackfill' 'ceph osd set norecover' 'ceph osd set norebalance' 'ceph osd set nodown' 'ceph osd set pause' 'ceph osd set nodeep-scrub' 'ceph osd set noscrub'
Would like to ask if this can be enough to set and restart the host safely . the cluster has 3 as replicas.
will the cluster still be accessible while restart the hosts? after restarting I will unset the flags.
Kindly advise.
Michel
On Tue, 30 Jan 2024, 17:44 Wesley Dillingham, <wes@wesdillingham.com> wrote:
actually it seems the issue I had in mind was fixed in 16.2.11 so you should be fine.
Respectfully,
*Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Jan 30, 2024 at 10:34 AM Wesley Dillingham <wes@wesdillingham.com> wrote:
You may want to consider upgrading to 16.2.14 before you do the pg split.
Respectfully,
*Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Jan 30, 2024 at 10:18 AM Michel Niyoyita <micou12@gmail.com> wrote:
I tried that on one of my pool (pool id 3) but the number of pgs not deep-scrubbed in time increased also from 55 to 100 but the number of PGs was increased. I set also autoscale to off mode. before continue to other pools would like to ask if so far there is no negative impact.
ceph -s cluster: id: cb0caedc-eb5b-42d1-a34f-96facfda8c27 health: HEALTH_WARN 100 pgs not deep-scrubbed in time
services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M) mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1 osd: 48 osds: 48 up (since 11M), 48 in (since 12M) rgw: 6 daemons active (6 hosts, 1 zones)
data: pools: 10 pools, 609 pgs objects: 6.03M objects, 23 TiB usage: 151 TiB used, 282 TiB / 433 TiB avail pgs: 603 active+clean 4 active+clean+scrubbing+deep 2 active+clean+scrubbing
io: client: 96 MiB/s rd, 573 MiB/s wr, 576 op/s rd, 648 op/s wr
root@ceph-osd3:/var/log# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 433 TiB 282 TiB 151 TiB 151 TiB 34.93 TOTAL 433 TiB 282 TiB 151 TiB 151 TiB 34.93
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 1.1 MiB 3 3.2 MiB 0 72 TiB .rgw.root 2 32 3.7 KiB 8 96 KiB 0 72 TiB default.rgw.log 3 256 3.6 KiB 204 408 KiB 0 72 TiB default.rgw.control 4 32 0 B 8 0 B 0 72 TiB default.rgw.meta 5 32 382 B 2 24 KiB 0 72 TiB volumes 6 128 21 TiB 5.74M 62 TiB 22.30 72 TiB images 7 32 878 GiB 112.50k 2.6 TiB 1.17 72 TiB backups 8 32 0 B 0 0 B 0 72 TiB vms 9 32 870 GiB 170.73k 2.5 TiB 1.13 72 TiB testbench 10 32 0 B 0 0 B 0 72 TiB
On Tue, Jan 30, 2024 at 5:05 PM Wesley Dillingham <wes@wesdillingham.com> wrote:
It will take a couple weeks to a couple months to complete is my best guess on 10TB spinners at ~40% full. The cluster should be usable throughout the process.
Keep in mind, you should disable the pg autoscaler on any pool which you are manually adjusting the pg_num for. Increasing the pg_num is called "pg splitting" you can google around for this to see how it will work etc.
There are a few knobs to increase or decrease the aggressiveness of the pg split, primarily these are osd_max_backfills and target_max_misplaced_ratio.
You can monitor the progress of the split by looking at "ceph osd pool ls detail" for the pool you are splitting, for this pool pgp_num will slowly increase up until it reaches the pg_num / pg_num_target.
IMO this blog post best covers the issue which you are looking to undertake: https://ceph.io/en/news/blog/2019/new-in-nautilus-pg-merging-and-autotuning/
Respectfully,
*Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Jan 30, 2024 at 9:38 AM Michel Niyoyita <micou12@gmail.com> wrote:
Thanks for your advices Wes, below is what ceph osd df tree shows , the increase of pg_num of the production cluster will not affect the performance or crush ? how long it can takes to finish?
ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 433.11841 - 433 TiB 151 TiB 67 TiB 364 MiB 210 GiB 282 TiB 34.86 1.00 - root default -3 144.37280 - 144 TiB 50 TiB 22 TiB 121 MiB 70 GiB 94 TiB 34.86 1.00 - host ceph-osd1 2 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1021 GiB 5.4 MiB 3.7 GiB 6.3 TiB 30.40 0.87 19 up osd.2 3 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 931 GiB 4.1 MiB 3.5 GiB 6.4 TiB 29.43 0.84 29 up osd.3 6 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.5 TiB 8.1 MiB 4.5 GiB 5.8 TiB 36.09 1.04 20 up osd.6 9 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.0 TiB 6.6 MiB 3.8 GiB 6.2 TiB 30.97 0.89 23 up osd.9 12 hdd 9.02330 1.00000 9.0 TiB 4.0 TiB 2.3 TiB 13 MiB 6.1 GiB 5.0 TiB 44.68 1.28 30 up osd.12 15 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.8 TiB 9.2 MiB 5.2 GiB 5.5 TiB 38.99 1.12 30 up osd.15 18 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.5 MiB 4.0 GiB 6.1 TiB 32.80 0.94 21 up osd.18 22 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.9 TiB 10 MiB 5.4 GiB 5.4 TiB 40.25 1.15 22 up osd.22 25 hdd 9.02330 1.00000 9.0 TiB 3.9 TiB 2.1 TiB 12 MiB 5.7 GiB 5.1 TiB 42.94 1.23 22 up osd.25 28 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.5 MiB 4.1 GiB 5.9 TiB 34.87 1.00 21 up osd.28 32 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1017 GiB 4.8 MiB 3.7 GiB 6.3 TiB 30.36 0.87 27 up osd.32 35 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.3 TiB 7.2 MiB 4.2 GiB 6.0 TiB 33.73 0.97 21 up osd.35 38 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.3 MiB 4.1 GiB 5.9 TiB 34.57 0.99 24 up osd.38 41 hdd 9.02330 1.00000 9.0 TiB 2.9 TiB 1.2 TiB 6.2 MiB 4.0 GiB 6.1 TiB 32.49 0.93 24 up osd.41 44 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.3 MiB 4.4 GiB 5.9 TiB 34.87 1.00 29 up osd.44 47 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1016 GiB 5.4 MiB 3.6 GiB 6.3 TiB 30.35 0.87 23 up osd.47 -7 144.37280 - 144 TiB 50 TiB 22 TiB 122 MiB 70 GiB 94 TiB 34.86 1.00 - host ceph-osd2 1 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.7 MiB 3.8 GiB 6.2 TiB 31.00 0.89 27 up osd.1 5 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.5 TiB 7.3 MiB 4.5 GiB 5.8 TiB 35.45 1.02 27 up osd.5 8 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.6 TiB 8.3 MiB 4.7 GiB 5.7 TiB 36.85 1.06 30 up osd.8 10 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.5 MiB 4.5 GiB 5.9 TiB 34.87 1.00 20 up osd.10 13 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB 5.3 GiB 5.4 TiB 39.63 1.14 27 up osd.13 16 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 6.0 MiB 3.8 GiB 6.2 TiB 31.01 0.89 19 up osd.16 19 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.4 MiB 4.0 GiB 6.1 TiB 32.77 0.94 21 up osd.19 21 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.5 MiB 3.7 GiB 6.2 TiB 31.58 0.91 26 up osd.21 24 hdd 9.02330 1.00000 9.0 TiB 2.6 TiB 855 GiB 4.7 MiB 3.3 GiB 6.4 TiB 28.61 0.82 19 up osd.24 27 hdd 9.02330 1.00000 9.0 TiB 3.7 TiB 1.9 TiB 10 MiB 5.2 GiB 5.3 TiB 40.84 1.17 24 up osd.27 30 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.4 TiB 7.5 MiB 4.5 GiB 5.9 TiB 35.16 1.01 22 up osd.30 33 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 8.6 MiB 4.3 GiB 5.9 TiB 34.59 0.99 23 up osd.33 36 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 10 MiB 5.0 GiB 5.6 TiB 38.17 1.09 25 up osd.36 39 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 8.5 MiB 5.1 GiB 5.6 TiB 37.79 1.08 31 up osd.39 42 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB 5.2 GiB 5.4 TiB 39.68 1.14 23 up osd.42 45 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 964 GiB 5.1 MiB 3.5 GiB 6.3 TiB 29.78 0.85 21 up osd.45 -5 144.37280 - 144 TiB 50 TiB 22 TiB 121 MiB 70 GiB 94 TiB 34.86 1.00 - host ceph-osd3 0 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 934 GiB 4.9 MiB 3.4 GiB 6.4 TiB 29.47 0.85 21 up osd.0 4 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.5 MiB 4.1 GiB 6.1 TiB 32.73 0.94 22 up osd.4 7 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.8 TiB 9.2 MiB 5.1 GiB 5.5 TiB 39.02 1.12 30 up osd.7 11 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.9 TiB 10 MiB 5.1 GiB 5.4 TiB 39.97 1.15 27 up osd.11 14 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.7 TiB 10 MiB 5.1 GiB 5.6 TiB 38.24 1.10 27 up osd.14 17 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.4 MiB 4.1 GiB 6.0 TiB 33.09 0.95 23 up osd.17 20 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.6 MiB 3.8 GiB 6.2 TiB 31.55 0.90 20 up osd.20 23 hdd 9.02330 1.00000 9.0 TiB 2.6 TiB 828 GiB 4.0 MiB 3.3 GiB 6.5 TiB 28.32 0.81 23 up osd.23 26 hdd 9.02330 1.00000 9.0 TiB 2.9 TiB 1.2 TiB 5.8 MiB 3.8 GiB 6.1 TiB 32.12 0.92 26 up osd.26 29 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB 5.1 GiB 5.4 TiB 39.73 1.14 24 up osd.29 31 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.8 MiB 3.7 GiB 6.2 TiB 31.56 0.91 22 up osd.31 34 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.5 TiB 8.2 MiB 4.6 GiB 5.7 TiB 36.29 1.04 23 up osd.34 37 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.5 TiB 8.2 MiB 4.5 GiB 5.8 TiB 35.51 1.02 20 up osd.37 40 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 9.3 MiB 4.9 GiB 5.6 TiB 38.16 1.09 25 up osd.40 43 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.6 TiB 8.5 MiB 4.8 GiB 5.7 TiB 37.19 1.07 29 up osd.43 46 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 8.4 MiB 4.4 GiB 5.9 TiB 34.85 1.00 23 up osd.46 TOTAL 433 TiB 151 TiB 67 TiB 364 MiB 210 GiB 282 TiB 34.86 MIN/MAX VAR: 0.81/1.28 STDDEV: 3.95
Michel
On Tue, Jan 30, 2024 at 4:18 PM Wesley Dillingham < wes@wesdillingham.com> wrote:
> I now concur you should increase the pg_num as a first step for this > cluster. Disable the pg autoscaler for and increase the volumes pool to > pg_num 256. Then likely re-asses and make the next power of 2 jump to 512 > and probably beyond. > > Keep in mind this is not going to fix your short term deep-scrub > issue in fact it will increase the number of not scrubbed in time PGs until > the pg_num change is complete. This is because OSDs dont scrub when they > are backfilling. > > I would sit on 256 for a couple weeks and let scrubs happen then > continue past 256. > > with the ultimate target of around 100-200 PGs per OSD which "ceph > osd df tree" will show you in the PGs column. > > Respectfully, > > *Wes Dillingham* > wes@wesdillingham.com > LinkedIn <http://www.linkedin.com/in/wesleydillingham> > > > On Tue, Jan 30, 2024 at 3:16 AM Michel Niyoyita <micou12@gmail.com> > wrote: > >> Dear team, >> >> below is the output of ceph df command and the ceph version I am >> running >> >> ceph df >> --- RAW STORAGE --- >> CLASS SIZE AVAIL USED RAW USED %RAW USED >> hdd 433 TiB 282 TiB 151 TiB 151 TiB 34.82 >> TOTAL 433 TiB 282 TiB 151 TiB 151 TiB 34.82 >> >> --- POOLS --- >> POOL ID PGS STORED OBJECTS USED %USED >> MAX AVAIL >> device_health_metrics 1 1 1.1 MiB 3 3.2 MiB 0 >> 73 TiB >> .rgw.root 2 32 3.7 KiB 8 96 KiB 0 >> 73 TiB >> default.rgw.log 3 32 3.6 KiB 209 408 KiB 0 >> 73 TiB >> default.rgw.control 4 32 0 B 8 0 B 0 >> 73 TiB >> default.rgw.meta 5 32 382 B 2 24 KiB 0 >> 73 TiB >> volumes 6 128 21 TiB 5.68M 62 TiB 22.09 >> 73 TiB >> images 7 32 878 GiB 112.50k 2.6 TiB 1.17 >> 73 TiB >> backups 8 32 0 B 0 0 B 0 >> 73 TiB >> vms 9 32 881 GiB 174.30k 2.5 TiB 1.13 >> 73 TiB >> testbench 10 32 0 B 0 0 B 0 >> 73 TiB >> root@ceph-mon1:~# ceph --version >> ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >> pacific >> (stable) >> root@ceph-mon1:~# >> >> please advise accordingly >> >> Michel >> >> On Mon, Jan 29, 2024 at 9:48 PM Frank Schilder <frans@dtu.dk> wrote: >> >> > You will have to look at the output of "ceph df" and make a >> decision to >> > balance "objects per PG" and "GB per PG". Increase he PG count for >> the >> > pools with the worst of these two numbers most such that it >> balances out as >> > much as possible. If you have pools that see significantly more >> user-IO >> > than others, prioritise these. >> > >> > You will have to find out for your specific cluster, we can only >> give >> > general guidelines. Make changes, run benchmarks, re-evaluate. >> Take the >> > time for it. The better you know your cluster and your users, the >> better >> > the end result will be. >> > >> > Best regards, >> > ================= >> > Frank Schilder >> > AIT Risø Campus >> > Bygning 109, rum S14 >> > >> > ________________________________________ >> > From: Michel Niyoyita <micou12@gmail.com> >> > Sent: Monday, January 29, 2024 2:04 PM >> > To: Janne Johansson >> > Cc: Frank Schilder; E Taka; ceph-users >> > Subject: Re: [ceph-users] Re: 6 pgs not deep-scrubbed in time >> > >> > This is how it is set , if you suggest to make some changes please >> advises. >> > >> > Thank you. >> > >> > >> > ceph osd pool ls detail >> > pool 1 'device_health_metrics' replicated size 3 min_size 2 >> crush_rule 0 >> > object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on >> last_change 1407 >> > flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 >> application >> > mgr_devicehealth >> > pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 >> object_hash >> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 >> flags >> > hashpspool stripe_width 0 application rgw >> > pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 >> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >> last_change >> > 1394 flags hashpspool stripe_width 0 application rgw >> > pool 4 'default.rgw.control' replicated size 3 min_size 2 >> crush_rule 0 >> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >> last_change >> > 1395 flags hashpspool stripe_width 0 application rgw >> > pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 >> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >> last_change >> > 1396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 >> application rgw >> > pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 >> object_hash >> > rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change >> 108802 lfor >> > 0/0/14812 flags hashpspool,selfmanaged_snaps stripe_width 0 >> application rbd >> > removed_snaps_queue >> > >> [22d7~3,11561~2,11571~1,11573~1c,11594~6,1159b~f,115b0~1,115b3~1,115c3~1,115f3~1,115f5~e,11613~6,1161f~c,11637~1b,11660~1,11663~2,11673~1,116d1~c,116f5~10,11721~c] >> > pool 7 'images' replicated size 3 min_size 2 crush_rule 0 >> object_hash >> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 94609 >> flags >> > hashpspool,selfmanaged_snaps stripe_width 0 application rbd >> > pool 8 'backups' replicated size 3 min_size 2 crush_rule 0 >> object_hash >> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1399 >> flags >> > hashpspool stripe_width 0 application rbd >> > pool 9 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash >> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 108783 >> lfor >> > 0/561/559 flags hashpspool,selfmanaged_snaps stripe_width 0 >> application rbd >> > removed_snaps_queue [3fa~1,3fc~3,400~1,402~1] >> > pool 10 'testbench' replicated size 3 min_size 2 crush_rule 0 >> object_hash >> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20931 >> lfor >> > 0/20931/20929 flags hashpspool stripe_width 0 >> > >> > >> > On Mon, Jan 29, 2024 at 2:09 PM Michel Niyoyita <micou12@gmail.com >> <mailto: >> > micou12@gmail.com>> wrote: >> > Thank you Janne , >> > >> > no need of setting some flags like ceph osd set nodeep-scrub ??? >> > >> > Thank you >> > >> > On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson < >> icepic.dz@gmail.com >> > <mailto:icepic.dz@gmail.com>> wrote: >> > Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita < >> micou12@gmail.com >> > <mailto:micou12@gmail.com>>: >> > > >> > > Thank you Frank , >> > > >> > > All disks are HDDs . Would like to know if I can increase the >> number of >> > PGs >> > > live in production without a negative impact to the cluster. if >> yes which >> > > commands to use . >> > >> > Yes. "ceph osd pool set <poolname> pg_num <number larger than >> before>" >> > where the number usually should be a power of two that leads to a >> > number of PGs per OSD between 100-200. >> > >> > -- >> > May the most significant bit of your life be positive. >> > >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io >> >
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- May the most significant bit of your life be positive.
Thank you very much Janne. On Thu, 1 Feb 2024, 15:21 Janne Johansson, <icepic.dz@gmail.com> wrote:
pause and nodown is not a good option to set, that will certainly make clients stop IO. Pause will stop it immediately, and nodown will stop IO when the OSD processes stop running on this host.
When we do service on a host, we set "noout" and "nobackfill", that is enough for reboots, OS upgrades and simple disk exchanges. The PGs on this one host will be degraded during the down period, but IO continues. Of course this is when the cluster was healthy to begin with (not counting "not scrubbed in time" warnings, they don't matter in this case.)
Den tors 1 feb. 2024 kl 12:21 skrev Michel Niyoyita <micou12@gmail.com>:
Thanks Very much Wesley,
We have decided to restart one host among three osds hosts. before doing that I need the advices of the team . these are flags I want to set
restart.
'ceph osd set noout' 'ceph osd set nobackfill' 'ceph osd set norecover' 'ceph osd set norebalance' 'ceph osd set nodown' 'ceph osd set pause' 'ceph osd set nodeep-scrub' 'ceph osd set noscrub'
Would like to ask if this can be enough to set and restart the host safely . the cluster has 3 as replicas.
will the cluster still be accessible while restart the hosts? after restarting I will unset the flags.
Kindly advise.
Michel
On Tue, 30 Jan 2024, 17:44 Wesley Dillingham, <wes@wesdillingham.com> wrote:
actually it seems the issue I had in mind was fixed in 16.2.11 so you should be fine.
Respectfully,
*Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Jan 30, 2024 at 10:34 AM Wesley Dillingham < wes@wesdillingham.com> wrote:
You may want to consider upgrading to 16.2.14 before you do the pg split.
Respectfully,
*Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Jan 30, 2024 at 10:18 AM Michel Niyoyita <micou12@gmail.com> wrote:
I tried that on one of my pool (pool id 3) but the number of pgs not deep-scrubbed in time increased also from 55 to 100 but the number of PGs was increased. I set also autoscale to off mode. before continue to other pools would like to ask if so far there is no negative impact.
ceph -s cluster: id: cb0caedc-eb5b-42d1-a34f-96facfda8c27 health: HEALTH_WARN 100 pgs not deep-scrubbed in time
services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M) mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1 osd: 48 osds: 48 up (since 11M), 48 in (since 12M) rgw: 6 daemons active (6 hosts, 1 zones)
data: pools: 10 pools, 609 pgs objects: 6.03M objects, 23 TiB usage: 151 TiB used, 282 TiB / 433 TiB avail pgs: 603 active+clean 4 active+clean+scrubbing+deep 2 active+clean+scrubbing
io: client: 96 MiB/s rd, 573 MiB/s wr, 576 op/s rd, 648 op/s wr
root@ceph-osd3:/var/log# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 433 TiB 282 TiB 151 TiB 151 TiB 34.93 TOTAL 433 TiB 282 TiB 151 TiB 151 TiB 34.93
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 1.1 MiB 3 3.2 MiB 0 72 TiB .rgw.root 2 32 3.7 KiB 8 96 KiB 0 72 TiB default.rgw.log 3 256 3.6 KiB 204 408 KiB 0 72 TiB default.rgw.control 4 32 0 B 8 0 B 0 72 TiB default.rgw.meta 5 32 382 B 2 24 KiB 0 72 TiB volumes 6 128 21 TiB 5.74M 62 TiB 22.30 72 TiB images 7 32 878 GiB 112.50k 2.6 TiB 1.17 72 TiB backups 8 32 0 B 0 0 B 0 72 TiB vms 9 32 870 GiB 170.73k 2.5 TiB 1.13 72 TiB testbench 10 32 0 B 0 0 B 0 72 TiB
On Tue, Jan 30, 2024 at 5:05 PM Wesley Dillingham < wes@wesdillingham.com> wrote:
It will take a couple weeks to a couple months to complete is my best guess on 10TB spinners at ~40% full. The cluster should be usable throughout the process.
Keep in mind, you should disable the pg autoscaler on any pool which you are manually adjusting the pg_num for. Increasing the pg_num is called "pg splitting" you can google around for this to see how it will work etc.
There are a few knobs to increase or decrease the aggressiveness of
pg split, primarily these are osd_max_backfills and target_max_misplaced_ratio.
You can monitor the progress of the split by looking at "ceph osd
ls detail" for the pool you are splitting, for this pool pgp_num will slowly increase up until it reaches the pg_num / pg_num_target.
IMO this blog post best covers the issue which you are looking to undertake:
https://ceph.io/en/news/blog/2019/new-in-nautilus-pg-merging-and-autotuning/
Respectfully,
*Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Jan 30, 2024 at 9:38 AM Michel Niyoyita <micou12@gmail.com> wrote:
> Thanks for your advices Wes, below is what ceph osd df tree shows , > the increase of pg_num of the production cluster will not affect
> performance or crush ? how long it can takes to finish? > > ceph osd df tree > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP > META AVAIL %USE VAR PGS STATUS TYPE NAME > -1 433.11841 - 433 TiB 151 TiB 67 TiB 364 MiB > 210 GiB 282 TiB 34.86 1.00 - root default > -3 144.37280 - 144 TiB 50 TiB 22 TiB 121 MiB > 70 GiB 94 TiB 34.86 1.00 - host ceph-osd1 > 2 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1021 GiB 5.4 MiB > 3.7 GiB 6.3 TiB 30.40 0.87 19 up osd.2 > 3 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 931 GiB 4.1 MiB > 3.5 GiB 6.4 TiB 29.43 0.84 29 up osd.3 > 6 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.5 TiB 8.1 MiB > 4.5 GiB 5.8 TiB 36.09 1.04 20 up osd.6 > 9 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.0 TiB 6.6 MiB > 3.8 GiB 6.2 TiB 30.97 0.89 23 up osd.9 > 12 hdd 9.02330 1.00000 9.0 TiB 4.0 TiB 2.3 TiB 13 MiB > 6.1 GiB 5.0 TiB 44.68 1.28 30 up osd.12 > 15 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.8 TiB 9.2 MiB > 5.2 GiB 5.5 TiB 38.99 1.12 30 up osd.15 > 18 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.5 MiB > 4.0 GiB 6.1 TiB 32.80 0.94 21 up osd.18 > 22 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.9 TiB 10 MiB > 5.4 GiB 5.4 TiB 40.25 1.15 22 up osd.22 > 25 hdd 9.02330 1.00000 9.0 TiB 3.9 TiB 2.1 TiB 12 MiB > 5.7 GiB 5.1 TiB 42.94 1.23 22 up osd.25 > 28 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.5 MiB > 4.1 GiB 5.9 TiB 34.87 1.00 21 up osd.28 > 32 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1017 GiB 4.8 MiB > 3.7 GiB 6.3 TiB 30.36 0.87 27 up osd.32 > 35 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.3 TiB 7.2 MiB > 4.2 GiB 6.0 TiB 33.73 0.97 21 up osd.35 > 38 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.3 MiB > 4.1 GiB 5.9 TiB 34.57 0.99 24 up osd.38 > 41 hdd 9.02330 1.00000 9.0 TiB 2.9 TiB 1.2 TiB 6.2 MiB > 4.0 GiB 6.1 TiB 32.49 0.93 24 up osd.41 > 44 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.3 MiB > 4.4 GiB 5.9 TiB 34.87 1.00 29 up osd.44 > 47 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1016 GiB 5.4 MiB > 3.6 GiB 6.3 TiB 30.35 0.87 23 up osd.47 > -7 144.37280 - 144 TiB 50 TiB 22 TiB 122 MiB > 70 GiB 94 TiB 34.86 1.00 - host ceph-osd2 > 1 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.7 MiB > 3.8 GiB 6.2 TiB 31.00 0.89 27 up osd.1 > 5 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.5 TiB 7.3 MiB > 4.5 GiB 5.8 TiB 35.45 1.02 27 up osd.5 > 8 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.6 TiB 8.3 MiB > 4.7 GiB 5.7 TiB 36.85 1.06 30 up osd.8 > 10 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.5 MiB > 4.5 GiB 5.9 TiB 34.87 1.00 20 up osd.10 > 13 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB > 5.3 GiB 5.4 TiB 39.63 1.14 27 up osd.13 > 16 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 6.0 MiB > 3.8 GiB 6.2 TiB 31.01 0.89 19 up osd.16 > 19 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.4 MiB > 4.0 GiB 6.1 TiB 32.77 0.94 21 up osd.19 > 21 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.5 MiB > 3.7 GiB 6.2 TiB 31.58 0.91 26 up osd.21 > 24 hdd 9.02330 1.00000 9.0 TiB 2.6 TiB 855 GiB 4.7 MiB > 3.3 GiB 6.4 TiB 28.61 0.82 19 up osd.24 > 27 hdd 9.02330 1.00000 9.0 TiB 3.7 TiB 1.9 TiB 10 MiB > 5.2 GiB 5.3 TiB 40.84 1.17 24 up osd.27 > 30 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.4 TiB 7.5 MiB > 4.5 GiB 5.9 TiB 35.16 1.01 22 up osd.30 > 33 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 8.6 MiB > 4.3 GiB 5.9 TiB 34.59 0.99 23 up osd.33 > 36 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 10 MiB > 5.0 GiB 5.6 TiB 38.17 1.09 25 up osd.36 > 39 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 8.5 MiB > 5.1 GiB 5.6 TiB 37.79 1.08 31 up osd.39 > 42 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB > 5.2 GiB 5.4 TiB 39.68 1.14 23 up osd.42 > 45 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 964 GiB 5.1 MiB > 3.5 GiB 6.3 TiB 29.78 0.85 21 up osd.45 > -5 144.37280 - 144 TiB 50 TiB 22 TiB 121 MiB > 70 GiB 94 TiB 34.86 1.00 - host ceph-osd3 > 0 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 934 GiB 4.9 MiB > 3.4 GiB 6.4 TiB 29.47 0.85 21 up osd.0 > 4 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.5 MiB > 4.1 GiB 6.1 TiB 32.73 0.94 22 up osd.4 > 7 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.8 TiB 9.2 MiB > 5.1 GiB 5.5 TiB 39.02 1.12 30 up osd.7 > 11 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.9 TiB 10 MiB > 5.1 GiB 5.4 TiB 39.97 1.15 27 up osd.11 > 14 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.7 TiB 10 MiB > 5.1 GiB 5.6 TiB 38.24 1.10 27 up osd.14 > 17 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.4 MiB > 4.1 GiB 6.0 TiB 33.09 0.95 23 up osd.17 > 20 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.6 MiB > 3.8 GiB 6.2 TiB 31.55 0.90 20 up osd.20 > 23 hdd 9.02330 1.00000 9.0 TiB 2.6 TiB 828 GiB 4.0 MiB > 3.3 GiB 6.5 TiB 28.32 0.81 23 up osd.23 > 26 hdd 9.02330 1.00000 9.0 TiB 2.9 TiB 1.2 TiB 5.8 MiB > 3.8 GiB 6.1 TiB 32.12 0.92 26 up osd.26 > 29 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB > 5.1 GiB 5.4 TiB 39.73 1.14 24 up osd.29 > 31 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.8 MiB > 3.7 GiB 6.2 TiB 31.56 0.91 22 up osd.31 > 34 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.5 TiB 8.2 MiB > 4.6 GiB 5.7 TiB 36.29 1.04 23 up osd.34 > 37 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.5 TiB 8.2 MiB > 4.5 GiB 5.8 TiB 35.51 1.02 20 up osd.37 > 40 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 9.3 MiB > 4.9 GiB 5.6 TiB 38.16 1.09 25 up osd.40 > 43 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.6 TiB 8.5 MiB > 4.8 GiB 5.7 TiB 37.19 1.07 29 up osd.43 > 46 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 8.4 MiB > 4.4 GiB 5.9 TiB 34.85 1.00 23 up osd.46 > TOTAL 433 TiB 151 TiB 67 TiB 364 MiB > 210 GiB 282 TiB 34.86 > MIN/MAX VAR: 0.81/1.28 STDDEV: 3.95 > > > Michel > > > On Tue, Jan 30, 2024 at 4:18 PM Wesley Dillingham < > wes@wesdillingham.com> wrote: > >> I now concur you should increase the pg_num as a first step for
>> cluster. Disable the pg autoscaler for and increase the volumes
>> pg_num 256. Then likely re-asses and make the next power of 2 jump to 512 >> and probably beyond. >> >> Keep in mind this is not going to fix your short term deep-scrub >> issue in fact it will increase the number of not scrubbed in time PGs until >> the pg_num change is complete. This is because OSDs dont scrub when they >> are backfilling. >> >> I would sit on 256 for a couple weeks and let scrubs happen then >> continue past 256. >> >> with the ultimate target of around 100-200 PGs per OSD which "ceph >> osd df tree" will show you in the PGs column. >> >> Respectfully, >> >> *Wes Dillingham* >> wes@wesdillingham.com >> LinkedIn <http://www.linkedin.com/in/wesleydillingham> >> >> >> On Tue, Jan 30, 2024 at 3:16 AM Michel Niyoyita < micou12@gmail.com> >> wrote: >> >>> Dear team, >>> >>> below is the output of ceph df command and the ceph version I am >>> running >>> >>> ceph df >>> --- RAW STORAGE --- >>> CLASS SIZE AVAIL USED RAW USED %RAW USED >>> hdd 433 TiB 282 TiB 151 TiB 151 TiB 34.82 >>> TOTAL 433 TiB 282 TiB 151 TiB 151 TiB 34.82 >>> >>> --- POOLS --- >>> POOL ID PGS STORED OBJECTS USED %USED >>> MAX AVAIL >>> device_health_metrics 1 1 1.1 MiB 3 3.2 MiB 0 >>> 73 TiB >>> .rgw.root 2 32 3.7 KiB 8 96 KiB 0 >>> 73 TiB >>> default.rgw.log 3 32 3.6 KiB 209 408 KiB 0 >>> 73 TiB >>> default.rgw.control 4 32 0 B 8 0 B 0 >>> 73 TiB >>> default.rgw.meta 5 32 382 B 2 24 KiB 0 >>> 73 TiB >>> volumes 6 128 21 TiB 5.68M 62 TiB 22.09 >>> 73 TiB >>> images 7 32 878 GiB 112.50k 2.6 TiB 1.17 >>> 73 TiB >>> backups 8 32 0 B 0 0 B 0 >>> 73 TiB >>> vms 9 32 881 GiB 174.30k 2.5 TiB 1.13 >>> 73 TiB >>> testbench 10 32 0 B 0 0 B 0 >>> 73 TiB >>> root@ceph-mon1:~# ceph --version >>> ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >>> pacific >>> (stable) >>> root@ceph-mon1:~# >>> >>> please advise accordingly >>> >>> Michel >>> >>> On Mon, Jan 29, 2024 at 9:48 PM Frank Schilder <frans@dtu.dk> wrote: >>> >>> > You will have to look at the output of "ceph df" and make a >>> decision to >>> > balance "objects per PG" and "GB per PG". Increase he PG count for >>> the >>> > pools with the worst of these two numbers most such that it >>> balances out as >>> > much as possible. If you have pools that see significantly more >>> user-IO >>> > than others, prioritise these. >>> > >>> > You will have to find out for your specific cluster, we can only >>> give >>> > general guidelines. Make changes, run benchmarks, re-evaluate. >>> Take the >>> > time for it. The better you know your cluster and your users,
>>> better >>> > the end result will be. >>> > >>> > Best regards, >>> > ================= >>> > Frank Schilder >>> > AIT Risø Campus >>> > Bygning 109, rum S14 >>> > >>> > ________________________________________ >>> > From: Michel Niyoyita <micou12@gmail.com> >>> > Sent: Monday, January 29, 2024 2:04 PM >>> > To: Janne Johansson >>> > Cc: Frank Schilder; E Taka; ceph-users >>> > Subject: Re: [ceph-users] Re: 6 pgs not deep-scrubbed in time >>> > >>> > This is how it is set , if you suggest to make some changes
before the pool the this pool to the please
>>> advises. >>> > >>> > Thank you. >>> > >>> > >>> > ceph osd pool ls detail >>> > pool 1 'device_health_metrics' replicated size 3 min_size 2 >>> crush_rule 0 >>> > object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on >>> last_change 1407 >>> > flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 >>> application >>> > mgr_devicehealth >>> > pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 >>> object_hash >>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 >>> flags >>> > hashpspool stripe_width 0 application rgw >>> > pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 >>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >>> last_change >>> > 1394 flags hashpspool stripe_width 0 application rgw >>> > pool 4 'default.rgw.control' replicated size 3 min_size 2 >>> crush_rule 0 >>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >>> last_change >>> > 1395 flags hashpspool stripe_width 0 application rgw >>> > pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 >>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >>> last_change >>> > 1396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 >>> application rgw >>> > pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 >>> object_hash >>> > rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change >>> 108802 lfor >>> > 0/0/14812 flags hashpspool,selfmanaged_snaps stripe_width 0 >>> application rbd >>> > removed_snaps_queue >>> > >>> [22d7~3,11561~2,11571~1,11573~1c,11594~6,1159b~f,115b0~1,115b3~1,115c3~1,115f3~1,115f5~e,11613~6,1161f~c,11637~1b,11660~1,11663~2,11673~1,116d1~c,116f5~10,11721~c] >>> > pool 7 'images' replicated size 3 min_size 2 crush_rule 0 >>> object_hash >>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 94609 >>> flags >>> > hashpspool,selfmanaged_snaps stripe_width 0 application rbd >>> > pool 8 'backups' replicated size 3 min_size 2 crush_rule 0 >>> object_hash >>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1399 >>> flags >>> > hashpspool stripe_width 0 application rbd >>> > pool 9 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash >>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 108783 >>> lfor >>> > 0/561/559 flags hashpspool,selfmanaged_snaps stripe_width 0 >>> application rbd >>> > removed_snaps_queue [3fa~1,3fc~3,400~1,402~1] >>> > pool 10 'testbench' replicated size 3 min_size 2 crush_rule 0 >>> object_hash >>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20931 >>> lfor >>> > 0/20931/20929 flags hashpspool stripe_width 0 >>> > >>> > >>> > On Mon, Jan 29, 2024 at 2:09 PM Michel Niyoyita < micou12@gmail.com >>> <mailto: >>> > micou12@gmail.com>> wrote: >>> > Thank you Janne , >>> > >>> > no need of setting some flags like ceph osd set nodeep-scrub ??? >>> > >>> > Thank you >>> > >>> > On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson < >>> icepic.dz@gmail.com >>> > <mailto:icepic.dz@gmail.com>> wrote: >>> > Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita < >>> micou12@gmail.com >>> > <mailto:micou12@gmail.com>>: >>> > > >>> > > Thank you Frank , >>> > > >>> > > All disks are HDDs . Would like to know if I can increase the >>> number of >>> > PGs >>> > > live in production without a negative impact to the cluster. if >>> yes which >>> > > commands to use . >>> > >>> > Yes. "ceph osd pool set <poolname> pg_num <number larger than >>> before>" >>> > where the number usually should be a power of two that leads to a >>> > number of PGs per OSD between 100-200. >>> > >>> > -- >>> > May the most significant bit of your life be positive. >>> > >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> >>
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- May the most significant bit of your life be positive.
And as said before still it is in warning state with pgs not deep-scrubed in time . Hope this can be ignored and set those two flags "noout and nobackfill" then reboot . Thank you again Sir On Thu, 1 Feb 2024, 16:11 Michel Niyoyita, <micou12@gmail.com> wrote:
Thank you very much Janne.
On Thu, 1 Feb 2024, 15:21 Janne Johansson, <icepic.dz@gmail.com> wrote:
pause and nodown is not a good option to set, that will certainly make clients stop IO. Pause will stop it immediately, and nodown will stop IO when the OSD processes stop running on this host.
When we do service on a host, we set "noout" and "nobackfill", that is enough for reboots, OS upgrades and simple disk exchanges. The PGs on this one host will be degraded during the down period, but IO continues. Of course this is when the cluster was healthy to begin with (not counting "not scrubbed in time" warnings, they don't matter in this case.)
Den tors 1 feb. 2024 kl 12:21 skrev Michel Niyoyita <micou12@gmail.com>:
Thanks Very much Wesley,
We have decided to restart one host among three osds hosts. before doing that I need the advices of the team . these are flags I want to set
restart.
'ceph osd set noout' 'ceph osd set nobackfill' 'ceph osd set norecover' 'ceph osd set norebalance' 'ceph osd set nodown' 'ceph osd set pause' 'ceph osd set nodeep-scrub' 'ceph osd set noscrub'
Would like to ask if this can be enough to set and restart the host safely . the cluster has 3 as replicas.
will the cluster still be accessible while restart the hosts? after restarting I will unset the flags.
Kindly advise.
Michel
On Tue, 30 Jan 2024, 17:44 Wesley Dillingham, <wes@wesdillingham.com> wrote:
actually it seems the issue I had in mind was fixed in 16.2.11 so you should be fine.
Respectfully,
*Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Jan 30, 2024 at 10:34 AM Wesley Dillingham < wes@wesdillingham.com> wrote:
You may want to consider upgrading to 16.2.14 before you do the pg split.
Respectfully,
*Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Jan 30, 2024 at 10:18 AM Michel Niyoyita <micou12@gmail.com> wrote:
I tried that on one of my pool (pool id 3) but the number of pgs not deep-scrubbed in time increased also from 55 to 100 but the number of PGs was increased. I set also autoscale to off mode. before continue to other pools would like to ask if so far there is no negative impact.
ceph -s cluster: id: cb0caedc-eb5b-42d1-a34f-96facfda8c27 health: HEALTH_WARN 100 pgs not deep-scrubbed in time
services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M) mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1 osd: 48 osds: 48 up (since 11M), 48 in (since 12M) rgw: 6 daemons active (6 hosts, 1 zones)
data: pools: 10 pools, 609 pgs objects: 6.03M objects, 23 TiB usage: 151 TiB used, 282 TiB / 433 TiB avail pgs: 603 active+clean 4 active+clean+scrubbing+deep 2 active+clean+scrubbing
io: client: 96 MiB/s rd, 573 MiB/s wr, 576 op/s rd, 648 op/s wr
root@ceph-osd3:/var/log# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 433 TiB 282 TiB 151 TiB 151 TiB 34.93 TOTAL 433 TiB 282 TiB 151 TiB 151 TiB 34.93
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 1.1 MiB 3 3.2 MiB 0 72 TiB .rgw.root 2 32 3.7 KiB 8 96 KiB 0 72 TiB default.rgw.log 3 256 3.6 KiB 204 408 KiB 0 72 TiB default.rgw.control 4 32 0 B 8 0 B 0 72 TiB default.rgw.meta 5 32 382 B 2 24 KiB 0 72 TiB volumes 6 128 21 TiB 5.74M 62 TiB 22.30 72 TiB images 7 32 878 GiB 112.50k 2.6 TiB 1.17 72 TiB backups 8 32 0 B 0 0 B 0 72 TiB vms 9 32 870 GiB 170.73k 2.5 TiB 1.13 72 TiB testbench 10 32 0 B 0 0 B 0 72 TiB
On Tue, Jan 30, 2024 at 5:05 PM Wesley Dillingham < wes@wesdillingham.com> wrote:
> It will take a couple weeks to a couple months to complete is my best > guess on 10TB spinners at ~40% full. The cluster should be usable > throughout the process. > > Keep in mind, you should disable the pg autoscaler on any pool which > you are manually adjusting the pg_num for. Increasing the pg_num is called > "pg splitting" you can google around for this to see how it will work etc. > > There are a few knobs to increase or decrease the aggressiveness of the > pg split, primarily these are osd_max_backfills and > target_max_misplaced_ratio. > > You can monitor the progress of the split by looking at "ceph osd
> ls detail" for the pool you are splitting, for this pool pgp_num will > slowly increase up until it reaches the pg_num / pg_num_target. > > IMO this blog post best covers the issue which you are looking to > undertake: > https://ceph.io/en/news/blog/2019/new-in-nautilus-pg-merging-and-autotuning/ > > Respectfully, > > *Wes Dillingham* > wes@wesdillingham.com > LinkedIn <http://www.linkedin.com/in/wesleydillingham> > > > On Tue, Jan 30, 2024 at 9:38 AM Michel Niyoyita <micou12@gmail.com
> wrote: > >> Thanks for your advices Wes, below is what ceph osd df tree shows , >> the increase of pg_num of the production cluster will not affect
>> performance or crush ? how long it can takes to finish? >> >> ceph osd df tree >> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP >> META AVAIL %USE VAR PGS STATUS TYPE NAME >> -1 433.11841 - 433 TiB 151 TiB 67 TiB 364 MiB >> 210 GiB 282 TiB 34.86 1.00 - root default >> -3 144.37280 - 144 TiB 50 TiB 22 TiB 121 MiB >> 70 GiB 94 TiB 34.86 1.00 - host ceph-osd1 >> 2 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1021 GiB 5.4 MiB >> 3.7 GiB 6.3 TiB 30.40 0.87 19 up osd.2 >> 3 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 931 GiB 4.1 MiB >> 3.5 GiB 6.4 TiB 29.43 0.84 29 up osd.3 >> 6 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.5 TiB 8.1 MiB >> 4.5 GiB 5.8 TiB 36.09 1.04 20 up osd.6 >> 9 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.0 TiB 6.6 MiB >> 3.8 GiB 6.2 TiB 30.97 0.89 23 up osd.9 >> 12 hdd 9.02330 1.00000 9.0 TiB 4.0 TiB 2.3 TiB 13 MiB >> 6.1 GiB 5.0 TiB 44.68 1.28 30 up osd.12 >> 15 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.8 TiB 9.2 MiB >> 5.2 GiB 5.5 TiB 38.99 1.12 30 up osd.15 >> 18 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.5 MiB >> 4.0 GiB 6.1 TiB 32.80 0.94 21 up osd.18 >> 22 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.9 TiB 10 MiB >> 5.4 GiB 5.4 TiB 40.25 1.15 22 up osd.22 >> 25 hdd 9.02330 1.00000 9.0 TiB 3.9 TiB 2.1 TiB 12 MiB >> 5.7 GiB 5.1 TiB 42.94 1.23 22 up osd.25 >> 28 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.5 MiB >> 4.1 GiB 5.9 TiB 34.87 1.00 21 up osd.28 >> 32 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1017 GiB 4.8 MiB >> 3.7 GiB 6.3 TiB 30.36 0.87 27 up osd.32 >> 35 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.3 TiB 7.2 MiB >> 4.2 GiB 6.0 TiB 33.73 0.97 21 up osd.35 >> 38 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.3 MiB >> 4.1 GiB 5.9 TiB 34.57 0.99 24 up osd.38 >> 41 hdd 9.02330 1.00000 9.0 TiB 2.9 TiB 1.2 TiB 6.2 MiB >> 4.0 GiB 6.1 TiB 32.49 0.93 24 up osd.41 >> 44 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.3 MiB >> 4.4 GiB 5.9 TiB 34.87 1.00 29 up osd.44 >> 47 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1016 GiB 5.4 MiB >> 3.6 GiB 6.3 TiB 30.35 0.87 23 up osd.47 >> -7 144.37280 - 144 TiB 50 TiB 22 TiB 122 MiB >> 70 GiB 94 TiB 34.86 1.00 - host ceph-osd2 >> 1 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.7 MiB >> 3.8 GiB 6.2 TiB 31.00 0.89 27 up osd.1 >> 5 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.5 TiB 7.3 MiB >> 4.5 GiB 5.8 TiB 35.45 1.02 27 up osd.5 >> 8 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.6 TiB 8.3 MiB >> 4.7 GiB 5.7 TiB 36.85 1.06 30 up osd.8 >> 10 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.5 MiB >> 4.5 GiB 5.9 TiB 34.87 1.00 20 up osd.10 >> 13 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB >> 5.3 GiB 5.4 TiB 39.63 1.14 27 up osd.13 >> 16 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 6.0 MiB >> 3.8 GiB 6.2 TiB 31.01 0.89 19 up osd.16 >> 19 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.4 MiB >> 4.0 GiB 6.1 TiB 32.77 0.94 21 up osd.19 >> 21 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.5 MiB >> 3.7 GiB 6.2 TiB 31.58 0.91 26 up osd.21 >> 24 hdd 9.02330 1.00000 9.0 TiB 2.6 TiB 855 GiB 4.7 MiB >> 3.3 GiB 6.4 TiB 28.61 0.82 19 up osd.24 >> 27 hdd 9.02330 1.00000 9.0 TiB 3.7 TiB 1.9 TiB 10 MiB >> 5.2 GiB 5.3 TiB 40.84 1.17 24 up osd.27 >> 30 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.4 TiB 7.5 MiB >> 4.5 GiB 5.9 TiB 35.16 1.01 22 up osd.30 >> 33 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 8.6 MiB >> 4.3 GiB 5.9 TiB 34.59 0.99 23 up osd.33 >> 36 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 10 MiB >> 5.0 GiB 5.6 TiB 38.17 1.09 25 up osd.36 >> 39 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 8.5 MiB >> 5.1 GiB 5.6 TiB 37.79 1.08 31 up osd.39 >> 42 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB >> 5.2 GiB 5.4 TiB 39.68 1.14 23 up osd.42 >> 45 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 964 GiB 5.1 MiB >> 3.5 GiB 6.3 TiB 29.78 0.85 21 up osd.45 >> -5 144.37280 - 144 TiB 50 TiB 22 TiB 121 MiB >> 70 GiB 94 TiB 34.86 1.00 - host ceph-osd3 >> 0 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 934 GiB 4.9 MiB >> 3.4 GiB 6.4 TiB 29.47 0.85 21 up osd.0 >> 4 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.5 MiB >> 4.1 GiB 6.1 TiB 32.73 0.94 22 up osd.4 >> 7 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.8 TiB 9.2 MiB >> 5.1 GiB 5.5 TiB 39.02 1.12 30 up osd.7 >> 11 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.9 TiB 10 MiB >> 5.1 GiB 5.4 TiB 39.97 1.15 27 up osd.11 >> 14 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.7 TiB 10 MiB >> 5.1 GiB 5.6 TiB 38.24 1.10 27 up osd.14 >> 17 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.4 MiB >> 4.1 GiB 6.0 TiB 33.09 0.95 23 up osd.17 >> 20 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.6 MiB >> 3.8 GiB 6.2 TiB 31.55 0.90 20 up osd.20 >> 23 hdd 9.02330 1.00000 9.0 TiB 2.6 TiB 828 GiB 4.0 MiB >> 3.3 GiB 6.5 TiB 28.32 0.81 23 up osd.23 >> 26 hdd 9.02330 1.00000 9.0 TiB 2.9 TiB 1.2 TiB 5.8 MiB >> 3.8 GiB 6.1 TiB 32.12 0.92 26 up osd.26 >> 29 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB >> 5.1 GiB 5.4 TiB 39.73 1.14 24 up osd.29 >> 31 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.8 MiB >> 3.7 GiB 6.2 TiB 31.56 0.91 22 up osd.31 >> 34 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.5 TiB 8.2 MiB >> 4.6 GiB 5.7 TiB 36.29 1.04 23 up osd.34 >> 37 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.5 TiB 8.2 MiB >> 4.5 GiB 5.8 TiB 35.51 1.02 20 up osd.37 >> 40 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 9.3 MiB >> 4.9 GiB 5.6 TiB 38.16 1.09 25 up osd.40 >> 43 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.6 TiB 8.5 MiB >> 4.8 GiB 5.7 TiB 37.19 1.07 29 up osd.43 >> 46 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 8.4 MiB >> 4.4 GiB 5.9 TiB 34.85 1.00 23 up osd.46 >> TOTAL 433 TiB 151 TiB 67 TiB 364 MiB >> 210 GiB 282 TiB 34.86 >> MIN/MAX VAR: 0.81/1.28 STDDEV: 3.95 >> >> >> Michel >> >> >> On Tue, Jan 30, 2024 at 4:18 PM Wesley Dillingham < >> wes@wesdillingham.com> wrote: >> >>> I now concur you should increase the pg_num as a first step for
>>> cluster. Disable the pg autoscaler for and increase the volumes
>>> pg_num 256. Then likely re-asses and make the next power of 2 jump to 512 >>> and probably beyond. >>> >>> Keep in mind this is not going to fix your short term deep-scrub >>> issue in fact it will increase the number of not scrubbed in time PGs until >>> the pg_num change is complete. This is because OSDs dont scrub when they >>> are backfilling. >>> >>> I would sit on 256 for a couple weeks and let scrubs happen then >>> continue past 256. >>> >>> with the ultimate target of around 100-200 PGs per OSD which "ceph >>> osd df tree" will show you in the PGs column. >>> >>> Respectfully, >>> >>> *Wes Dillingham* >>> wes@wesdillingham.com >>> LinkedIn <http://www.linkedin.com/in/wesleydillingham> >>> >>> >>> On Tue, Jan 30, 2024 at 3:16 AM Michel Niyoyita < micou12@gmail.com> >>> wrote: >>> >>>> Dear team, >>>> >>>> below is the output of ceph df command and the ceph version I am >>>> running >>>> >>>> ceph df >>>> --- RAW STORAGE --- >>>> CLASS SIZE AVAIL USED RAW USED %RAW USED >>>> hdd 433 TiB 282 TiB 151 TiB 151 TiB 34.82 >>>> TOTAL 433 TiB 282 TiB 151 TiB 151 TiB 34.82 >>>> >>>> --- POOLS --- >>>> POOL ID PGS STORED OBJECTS USED %USED >>>> MAX AVAIL >>>> device_health_metrics 1 1 1.1 MiB 3 3.2 MiB 0 >>>> 73 TiB >>>> .rgw.root 2 32 3.7 KiB 8 96 KiB 0 >>>> 73 TiB >>>> default.rgw.log 3 32 3.6 KiB 209 408 KiB 0 >>>> 73 TiB >>>> default.rgw.control 4 32 0 B 8 0 B 0 >>>> 73 TiB >>>> default.rgw.meta 5 32 382 B 2 24 KiB 0 >>>> 73 TiB >>>> volumes 6 128 21 TiB 5.68M 62 TiB 22.09 >>>> 73 TiB >>>> images 7 32 878 GiB 112.50k 2.6 TiB 1.17 >>>> 73 TiB >>>> backups 8 32 0 B 0 0 B 0 >>>> 73 TiB >>>> vms 9 32 881 GiB 174.30k 2.5 TiB 1.13 >>>> 73 TiB >>>> testbench 10 32 0 B 0 0 B 0 >>>> 73 TiB >>>> root@ceph-mon1:~# ceph --version >>>> ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >>>> pacific >>>> (stable) >>>> root@ceph-mon1:~# >>>> >>>> please advise accordingly >>>> >>>> Michel >>>> >>>> On Mon, Jan 29, 2024 at 9:48 PM Frank Schilder <frans@dtu.dk> wrote: >>>> >>>> > You will have to look at the output of "ceph df" and make a >>>> decision to >>>> > balance "objects per PG" and "GB per PG". Increase he PG count for >>>> the >>>> > pools with the worst of these two numbers most such that it >>>> balances out as >>>> > much as possible. If you have pools that see significantly more >>>> user-IO >>>> > than others, prioritise these. >>>> > >>>> > You will have to find out for your specific cluster, we can only >>>> give >>>> > general guidelines. Make changes, run benchmarks, re-evaluate. >>>> Take the >>>> > time for it. The better you know your cluster and your users,
>>>> better >>>> > the end result will be. >>>> > >>>> > Best regards, >>>> > ================= >>>> > Frank Schilder >>>> > AIT Risø Campus >>>> > Bygning 109, rum S14 >>>> > >>>> > ________________________________________ >>>> > From: Michel Niyoyita <micou12@gmail.com> >>>> > Sent: Monday, January 29, 2024 2:04 PM >>>> > To: Janne Johansson >>>> > Cc: Frank Schilder; E Taka; ceph-users >>>> > Subject: Re: [ceph-users] Re: 6 pgs not deep-scrubbed in time >>>> > >>>> > This is how it is set , if you suggest to make some changes
>>>> advises. >>>> > >>>> > Thank you. >>>> > >>>> > >>>> > ceph osd pool ls detail >>>> > pool 1 'device_health_metrics' replicated size 3 min_size 2 >>>> crush_rule 0 >>>> > object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on >>>> last_change 1407 >>>> > flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 >>>> application >>>> > mgr_devicehealth >>>> > pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 >>>> object_hash >>>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 >>>> flags >>>> > hashpspool stripe_width 0 application rgw >>>> > pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 >>>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >>>> last_change >>>> > 1394 flags hashpspool stripe_width 0 application rgw >>>> > pool 4 'default.rgw.control' replicated size 3 min_size 2 >>>> crush_rule 0 >>>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >>>> last_change >>>> > 1395 flags hashpspool stripe_width 0 application rgw >>>> > pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 >>>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >>>> last_change >>>> > 1396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 >>>> application rgw >>>> > pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 >>>> object_hash >>>> > rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change >>>> 108802 lfor >>>> > 0/0/14812 flags hashpspool,selfmanaged_snaps stripe_width 0 >>>> application rbd >>>> > removed_snaps_queue >>>> > >>>> [22d7~3,11561~2,11571~1,11573~1c,11594~6,1159b~f,115b0~1,115b3~1,115c3~1,115f3~1,115f5~e,11613~6,1161f~c,11637~1b,11660~1,11663~2,11673~1,116d1~c,116f5~10,11721~c] >>>> > pool 7 'images' replicated size 3 min_size 2 crush_rule 0 >>>> object_hash >>>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 94609 >>>> flags >>>> > hashpspool,selfmanaged_snaps stripe_width 0 application rbd >>>> > pool 8 'backups' replicated size 3 min_size 2 crush_rule 0 >>>> object_hash >>>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1399 >>>> flags >>>> > hashpspool stripe_width 0 application rbd >>>> > pool 9 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash >>>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 108783 >>>> lfor >>>> > 0/561/559 flags hashpspool,selfmanaged_snaps stripe_width 0 >>>> application rbd >>>> > removed_snaps_queue [3fa~1,3fc~3,400~1,402~1] >>>> > pool 10 'testbench' replicated size 3 min_size 2 crush_rule 0 >>>> object_hash >>>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20931 >>>> lfor >>>> > 0/20931/20929 flags hashpspool stripe_width 0 >>>> > >>>> > >>>> > On Mon, Jan 29, 2024 at 2:09 PM Michel Niyoyita < micou12@gmail.com >>>> <mailto: >>>> > micou12@gmail.com>> wrote: >>>> > Thank you Janne , >>>> > >>>> > no need of setting some flags like ceph osd set nodeep-scrub ??? >>>> > >>>> > Thank you >>>> > >>>> > On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson < >>>> icepic.dz@gmail.com >>>> > <mailto:icepic.dz@gmail.com>> wrote: >>>> > Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita < >>>> micou12@gmail.com >>>> > <mailto:micou12@gmail.com>>: >>>> > > >>>> > > Thank you Frank , >>>> > > >>>> > > All disks are HDDs . Would like to know if I can increase
before pool the this pool to the please the
>>>> number of >>>> > PGs >>>> > > live in production without a negative impact to the cluster. if >>>> yes which >>>> > > commands to use . >>>> > >>>> > Yes. "ceph osd pool set <poolname> pg_num <number larger than >>>> before>" >>>> > where the number usually should be a power of two that leads to a >>>> > number of PGs per OSD between 100-200. >>>> > >>>> > -- >>>> > May the most significant bit of your life be positive. >>>> > >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> >>>
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- May the most significant bit of your life be positive.
I would just set noout for the duration of the reboot no other flags really needed. There is a better option to limit that flag to just the host being rebooted. which is "set-group noout <host>" where <host> is the servers name in CRUSH. Just the global noout will suffice though. Anyways... your not scrubbed in time warnings arent going away anytime in the short term until you finish the pg split. In fact, they will get more numerous until the pg split finishes (did you start that?). If you want to get rid of the "cosmetic" issue of the warning you can adjust the interval in which the warning comes up, but I would suggest you leave it since you are trying to address the root of the situation and want to see the resolution. Respectfully, *Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham> On Thu, Feb 1, 2024 at 9:16 AM Michel Niyoyita <micou12@gmail.com> wrote:
And as said before still it is in warning state with pgs not deep-scrubed in time . Hope this can be ignored and set those two flags "noout and nobackfill" then reboot .
Thank you again Sir
On Thu, 1 Feb 2024, 16:11 Michel Niyoyita, <micou12@gmail.com> wrote:
Thank you very much Janne.
On Thu, 1 Feb 2024, 15:21 Janne Johansson, <icepic.dz@gmail.com> wrote:
pause and nodown is not a good option to set, that will certainly make clients stop IO. Pause will stop it immediately, and nodown will stop IO when the OSD processes stop running on this host.
When we do service on a host, we set "noout" and "nobackfill", that is enough for reboots, OS upgrades and simple disk exchanges. The PGs on this one host will be degraded during the down period, but IO continues. Of course this is when the cluster was healthy to begin with (not counting "not scrubbed in time" warnings, they don't matter in this case.)
Den tors 1 feb. 2024 kl 12:21 skrev Michel Niyoyita <micou12@gmail.com>:
Thanks Very much Wesley,
We have decided to restart one host among three osds hosts. before
that I need the advices of the team . these are flags I want to set before restart.
'ceph osd set noout' 'ceph osd set nobackfill' 'ceph osd set norecover' 'ceph osd set norebalance' 'ceph osd set nodown' 'ceph osd set pause' 'ceph osd set nodeep-scrub' 'ceph osd set noscrub'
Would like to ask if this can be enough to set and restart the host safely . the cluster has 3 as replicas.
will the cluster still be accessible while restart the hosts? after restarting I will unset the flags.
Kindly advise.
Michel
On Tue, 30 Jan 2024, 17:44 Wesley Dillingham, <wes@wesdillingham.com> wrote:
actually it seems the issue I had in mind was fixed in 16.2.11 so you should be fine.
Respectfully,
*Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Jan 30, 2024 at 10:34 AM Wesley Dillingham < wes@wesdillingham.com> wrote:
You may want to consider upgrading to 16.2.14 before you do the pg split.
Respectfully,
*Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Jan 30, 2024 at 10:18 AM Michel Niyoyita <micou12@gmail.com
wrote:
> I tried that on one of my pool (pool id 3) but the number of pgs not > deep-scrubbed in time increased also from 55 to 100 but the number of PGs > was increased. I set also autoscale to off mode. before continue to other > pools would like to ask if so far there is no negative impact. > > ceph -s > cluster: > id: cb0caedc-eb5b-42d1-a34f-96facfda8c27 > health: HEALTH_WARN > 100 pgs not deep-scrubbed in time > > services: > mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M) > mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1 > osd: 48 osds: 48 up (since 11M), 48 in (since 12M) > rgw: 6 daemons active (6 hosts, 1 zones) > > data: > pools: 10 pools, 609 pgs > objects: 6.03M objects, 23 TiB > usage: 151 TiB used, 282 TiB / 433 TiB avail > pgs: 603 active+clean > 4 active+clean+scrubbing+deep > 2 active+clean+scrubbing > > io: > client: 96 MiB/s rd, 573 MiB/s wr, 576 op/s rd, 648 op/s wr > > root@ceph-osd3:/var/log# ceph df > --- RAW STORAGE --- > CLASS SIZE AVAIL USED RAW USED %RAW USED > hdd 433 TiB 282 TiB 151 TiB 151 TiB 34.93 > TOTAL 433 TiB 282 TiB 151 TiB 151 TiB 34.93 > > --- POOLS --- > POOL ID PGS STORED OBJECTS USED %USED MAX > AVAIL > device_health_metrics 1 1 1.1 MiB 3 3.2 MiB 0 72 > TiB > .rgw.root 2 32 3.7 KiB 8 96 KiB 0 72 > TiB > default.rgw.log 3 256 3.6 KiB 204 408 KiB 0 72 > TiB > default.rgw.control 4 32 0 B 8 0 B 0 72 > TiB > default.rgw.meta 5 32 382 B 2 24 KiB 0 72 > TiB > volumes 6 128 21 TiB 5.74M 62 TiB 22.30 72 > TiB > images 7 32 878 GiB 112.50k 2.6 TiB 1.17 72 > TiB > backups 8 32 0 B 0 0 B 0 72 > TiB > vms 9 32 870 GiB 170.73k 2.5 TiB 1.13 72 > TiB > testbench 10 32 0 B 0 0 B 0 72 > TiB > > On Tue, Jan 30, 2024 at 5:05 PM Wesley Dillingham < wes@wesdillingham.com> > wrote: > >> It will take a couple weeks to a couple months to complete is my best >> guess on 10TB spinners at ~40% full. The cluster should be usable >> throughout the process. >> >> Keep in mind, you should disable the pg autoscaler on any pool which >> you are manually adjusting the pg_num for. Increasing the pg_num is called >> "pg splitting" you can google around for this to see how it will work etc. >> >> There are a few knobs to increase or decrease the aggressiveness of the >> pg split, primarily these are osd_max_backfills and >> target_max_misplaced_ratio. >> >> You can monitor the progress of the split by looking at "ceph osd
>> ls detail" for the pool you are splitting, for this pool pgp_num will >> slowly increase up until it reaches the pg_num / pg_num_target. >> >> IMO this blog post best covers the issue which you are looking to >> undertake: >> https://ceph.io/en/news/blog/2019/new-in-nautilus-pg-merging-and-autotuning/ >> >> Respectfully, >> >> *Wes Dillingham* >> wes@wesdillingham.com >> LinkedIn <http://www.linkedin.com/in/wesleydillingham> >> >> >> On Tue, Jan 30, 2024 at 9:38 AM Michel Niyoyita < micou12@gmail.com> >> wrote: >> >>> Thanks for your advices Wes, below is what ceph osd df tree shows , >>> the increase of pg_num of the production cluster will not affect
>>> performance or crush ? how long it can takes to finish? >>> >>> ceph osd df tree >>> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP >>> META AVAIL %USE VAR PGS STATUS TYPE NAME >>> -1 433.11841 - 433 TiB 151 TiB 67 TiB 364 MiB >>> 210 GiB 282 TiB 34.86 1.00 - root default >>> -3 144.37280 - 144 TiB 50 TiB 22 TiB 121 MiB >>> 70 GiB 94 TiB 34.86 1.00 - host ceph-osd1 >>> 2 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1021 GiB 5.4 MiB >>> 3.7 GiB 6.3 TiB 30.40 0.87 19 up osd.2 >>> 3 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 931 GiB 4.1 MiB >>> 3.5 GiB 6.4 TiB 29.43 0.84 29 up osd.3 >>> 6 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.5 TiB 8.1 MiB >>> 4.5 GiB 5.8 TiB 36.09 1.04 20 up osd.6 >>> 9 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.0 TiB 6.6 MiB >>> 3.8 GiB 6.2 TiB 30.97 0.89 23 up osd.9 >>> 12 hdd 9.02330 1.00000 9.0 TiB 4.0 TiB 2.3 TiB 13 MiB >>> 6.1 GiB 5.0 TiB 44.68 1.28 30 up osd.12 >>> 15 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.8 TiB 9.2 MiB >>> 5.2 GiB 5.5 TiB 38.99 1.12 30 up osd.15 >>> 18 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.5 MiB >>> 4.0 GiB 6.1 TiB 32.80 0.94 21 up osd.18 >>> 22 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.9 TiB 10 MiB >>> 5.4 GiB 5.4 TiB 40.25 1.15 22 up osd.22 >>> 25 hdd 9.02330 1.00000 9.0 TiB 3.9 TiB 2.1 TiB 12 MiB >>> 5.7 GiB 5.1 TiB 42.94 1.23 22 up osd.25 >>> 28 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.5 MiB >>> 4.1 GiB 5.9 TiB 34.87 1.00 21 up osd.28 >>> 32 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1017 GiB 4.8 MiB >>> 3.7 GiB 6.3 TiB 30.36 0.87 27 up osd.32 >>> 35 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.3 TiB 7.2 MiB >>> 4.2 GiB 6.0 TiB 33.73 0.97 21 up osd.35 >>> 38 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.3 MiB >>> 4.1 GiB 5.9 TiB 34.57 0.99 24 up osd.38 >>> 41 hdd 9.02330 1.00000 9.0 TiB 2.9 TiB 1.2 TiB 6.2 MiB >>> 4.0 GiB 6.1 TiB 32.49 0.93 24 up osd.41 >>> 44 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.3 MiB >>> 4.4 GiB 5.9 TiB 34.87 1.00 29 up osd.44 >>> 47 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 1016 GiB 5.4 MiB >>> 3.6 GiB 6.3 TiB 30.35 0.87 23 up osd.47 >>> -7 144.37280 - 144 TiB 50 TiB 22 TiB 122 MiB >>> 70 GiB 94 TiB 34.86 1.00 - host ceph-osd2 >>> 1 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.7 MiB >>> 3.8 GiB 6.2 TiB 31.00 0.89 27 up osd.1 >>> 5 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.5 TiB 7.3 MiB >>> 4.5 GiB 5.8 TiB 35.45 1.02 27 up osd.5 >>> 8 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.6 TiB 8.3 MiB >>> 4.7 GiB 5.7 TiB 36.85 1.06 30 up osd.8 >>> 10 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 7.5 MiB >>> 4.5 GiB 5.9 TiB 34.87 1.00 20 up osd.10 >>> 13 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB >>> 5.3 GiB 5.4 TiB 39.63 1.14 27 up osd.13 >>> 16 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 6.0 MiB >>> 3.8 GiB 6.2 TiB 31.01 0.89 19 up osd.16 >>> 19 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.4 MiB >>> 4.0 GiB 6.1 TiB 32.77 0.94 21 up osd.19 >>> 21 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.5 MiB >>> 3.7 GiB 6.2 TiB 31.58 0.91 26 up osd.21 >>> 24 hdd 9.02330 1.00000 9.0 TiB 2.6 TiB 855 GiB 4.7 MiB >>> 3.3 GiB 6.4 TiB 28.61 0.82 19 up osd.24 >>> 27 hdd 9.02330 1.00000 9.0 TiB 3.7 TiB 1.9 TiB 10 MiB >>> 5.2 GiB 5.3 TiB 40.84 1.17 24 up osd.27 >>> 30 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.4 TiB 7.5 MiB >>> 4.5 GiB 5.9 TiB 35.16 1.01 22 up osd.30 >>> 33 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 8.6 MiB >>> 4.3 GiB 5.9 TiB 34.59 0.99 23 up osd.33 >>> 36 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 10 MiB >>> 5.0 GiB 5.6 TiB 38.17 1.09 25 up osd.36 >>> 39 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 8.5 MiB >>> 5.1 GiB 5.6 TiB 37.79 1.08 31 up osd.39 >>> 42 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB >>> 5.2 GiB 5.4 TiB 39.68 1.14 23 up osd.42 >>> 45 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 964 GiB 5.1 MiB >>> 3.5 GiB 6.3 TiB 29.78 0.85 21 up osd.45 >>> -5 144.37280 - 144 TiB 50 TiB 22 TiB 121 MiB >>> 70 GiB 94 TiB 34.86 1.00 - host ceph-osd3 >>> 0 hdd 9.02330 1.00000 9.0 TiB 2.7 TiB 934 GiB 4.9 MiB >>> 3.4 GiB 6.4 TiB 29.47 0.85 21 up osd.0 >>> 4 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.5 MiB >>> 4.1 GiB 6.1 TiB 32.73 0.94 22 up osd.4 >>> 7 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.8 TiB 9.2 MiB >>> 5.1 GiB 5.5 TiB 39.02 1.12 30 up osd.7 >>> 11 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.9 TiB 10 MiB >>> 5.1 GiB 5.4 TiB 39.97 1.15 27 up osd.11 >>> 14 hdd 9.02330 1.00000 9.0 TiB 3.5 TiB 1.7 TiB 10 MiB >>> 5.1 GiB 5.6 TiB 38.24 1.10 27 up osd.14 >>> 17 hdd 9.02330 1.00000 9.0 TiB 3.0 TiB 1.2 TiB 6.4 MiB >>> 4.1 GiB 6.0 TiB 33.09 0.95 23 up osd.17 >>> 20 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.6 MiB >>> 3.8 GiB 6.2 TiB 31.55 0.90 20 up osd.20 >>> 23 hdd 9.02330 1.00000 9.0 TiB 2.6 TiB 828 GiB 4.0 MiB >>> 3.3 GiB 6.5 TiB 28.32 0.81 23 up osd.23 >>> 26 hdd 9.02330 1.00000 9.0 TiB 2.9 TiB 1.2 TiB 5.8 MiB >>> 3.8 GiB 6.1 TiB 32.12 0.92 26 up osd.26 >>> 29 hdd 9.02330 1.00000 9.0 TiB 3.6 TiB 1.8 TiB 10 MiB >>> 5.1 GiB 5.4 TiB 39.73 1.14 24 up osd.29 >>> 31 hdd 9.02330 1.00000 9.0 TiB 2.8 TiB 1.1 TiB 5.8 MiB >>> 3.7 GiB 6.2 TiB 31.56 0.91 22 up osd.31 >>> 34 hdd 9.02330 1.00000 9.0 TiB 3.3 TiB 1.5 TiB 8.2 MiB >>> 4.6 GiB 5.7 TiB 36.29 1.04 23 up osd.34 >>> 37 hdd 9.02330 1.00000 9.0 TiB 3.2 TiB 1.5 TiB 8.2 MiB >>> 4.5 GiB 5.8 TiB 35.51 1.02 20 up osd.37 >>> 40 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.7 TiB 9.3 MiB >>> 4.9 GiB 5.6 TiB 38.16 1.09 25 up osd.40 >>> 43 hdd 9.02330 1.00000 9.0 TiB 3.4 TiB 1.6 TiB 8.5 MiB >>> 4.8 GiB 5.7 TiB 37.19 1.07 29 up osd.43 >>> 46 hdd 9.02330 1.00000 9.0 TiB 3.1 TiB 1.4 TiB 8.4 MiB >>> 4.4 GiB 5.9 TiB 34.85 1.00 23 up osd.46 >>> TOTAL 433 TiB 151 TiB 67 TiB 364 MiB >>> 210 GiB 282 TiB 34.86 >>> MIN/MAX VAR: 0.81/1.28 STDDEV: 3.95 >>> >>> >>> Michel >>> >>> >>> On Tue, Jan 30, 2024 at 4:18 PM Wesley Dillingham < >>> wes@wesdillingham.com> wrote: >>> >>>> I now concur you should increase the pg_num as a first step for
>>>> cluster. Disable the pg autoscaler for and increase the volumes
>>>> pg_num 256. Then likely re-asses and make the next power of 2 jump to 512 >>>> and probably beyond. >>>> >>>> Keep in mind this is not going to fix your short term deep-scrub >>>> issue in fact it will increase the number of not scrubbed in time PGs until >>>> the pg_num change is complete. This is because OSDs dont scrub when they >>>> are backfilling. >>>> >>>> I would sit on 256 for a couple weeks and let scrubs happen then >>>> continue past 256. >>>> >>>> with the ultimate target of around 100-200 PGs per OSD which "ceph >>>> osd df tree" will show you in the PGs column. >>>> >>>> Respectfully, >>>> >>>> *Wes Dillingham* >>>> wes@wesdillingham.com >>>> LinkedIn <http://www.linkedin.com/in/wesleydillingham> >>>> >>>> >>>> On Tue, Jan 30, 2024 at 3:16 AM Michel Niyoyita < micou12@gmail.com> >>>> wrote: >>>> >>>>> Dear team, >>>>> >>>>> below is the output of ceph df command and the ceph version I am >>>>> running >>>>> >>>>> ceph df >>>>> --- RAW STORAGE --- >>>>> CLASS SIZE AVAIL USED RAW USED %RAW USED >>>>> hdd 433 TiB 282 TiB 151 TiB 151 TiB 34.82 >>>>> TOTAL 433 TiB 282 TiB 151 TiB 151 TiB 34.82 >>>>> >>>>> --- POOLS --- >>>>> POOL ID PGS STORED OBJECTS USED %USED >>>>> MAX AVAIL >>>>> device_health_metrics 1 1 1.1 MiB 3 3.2 MiB 0 >>>>> 73 TiB >>>>> .rgw.root 2 32 3.7 KiB 8 96 KiB 0 >>>>> 73 TiB >>>>> default.rgw.log 3 32 3.6 KiB 209 408 KiB 0 >>>>> 73 TiB >>>>> default.rgw.control 4 32 0 B 8 0 B 0 >>>>> 73 TiB >>>>> default.rgw.meta 5 32 382 B 2 24 KiB 0 >>>>> 73 TiB >>>>> volumes 6 128 21 TiB 5.68M 62 TiB 22.09 >>>>> 73 TiB >>>>> images 7 32 878 GiB 112.50k 2.6 TiB 1.17 >>>>> 73 TiB >>>>> backups 8 32 0 B 0 0 B 0 >>>>> 73 TiB >>>>> vms 9 32 881 GiB 174.30k 2.5 TiB 1.13 >>>>> 73 TiB >>>>> testbench 10 32 0 B 0 0 B 0 >>>>> 73 TiB >>>>> root@ceph-mon1:~# ceph --version >>>>> ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >>>>> pacific >>>>> (stable) >>>>> root@ceph-mon1:~# >>>>> >>>>> please advise accordingly >>>>> >>>>> Michel >>>>> >>>>> On Mon, Jan 29, 2024 at 9:48 PM Frank Schilder <frans@dtu.dk> wrote: >>>>> >>>>> > You will have to look at the output of "ceph df" and make a >>>>> decision to >>>>> > balance "objects per PG" and "GB per PG". Increase he PG count for >>>>> the >>>>> > pools with the worst of these two numbers most such that it >>>>> balances out as >>>>> > much as possible. If you have pools that see significantly more >>>>> user-IO >>>>> > than others, prioritise these. >>>>> > >>>>> > You will have to find out for your specific cluster, we can only >>>>> give >>>>> > general guidelines. Make changes, run benchmarks, re-evaluate. >>>>> Take the >>>>> > time for it. The better you know your cluster and your users, the >>>>> better >>>>> > the end result will be. >>>>> > >>>>> > Best regards, >>>>> > ================= >>>>> > Frank Schilder >>>>> > AIT Risø Campus >>>>> > Bygning 109, rum S14 >>>>> > >>>>> > ________________________________________ >>>>> > From: Michel Niyoyita <micou12@gmail.com> >>>>> > Sent: Monday, January 29, 2024 2:04 PM >>>>> > To: Janne Johansson >>>>> > Cc: Frank Schilder; E Taka; ceph-users >>>>> > Subject: Re: [ceph-users] Re: 6 pgs not deep-scrubbed in time >>>>> > >>>>> > This is how it is set , if you suggest to make some changes
>>>>> advises. >>>>> > >>>>> > Thank you. >>>>> > >>>>> > >>>>> > ceph osd pool ls detail >>>>> > pool 1 'device_health_metrics' replicated size 3 min_size 2 >>>>> crush_rule 0 >>>>> > object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on >>>>> last_change 1407 >>>>> > flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 >>>>> application >>>>> > mgr_devicehealth >>>>> > pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 >>>>> object_hash >>>>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 >>>>> flags >>>>> > hashpspool stripe_width 0 application rgw >>>>> > pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 >>>>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >>>>> last_change >>>>> > 1394 flags hashpspool stripe_width 0 application rgw >>>>> > pool 4 'default.rgw.control' replicated size 3 min_size 2 >>>>> crush_rule 0 >>>>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >>>>> last_change >>>>> > 1395 flags hashpspool stripe_width 0 application rgw >>>>> > pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 >>>>> > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on >>>>> last_change >>>>> > 1396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 >>>>> application rgw >>>>> > pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 >>>>> object_hash >>>>> > rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change >>>>> 108802 lfor >>>>> > 0/0/14812 flags hashpspool,selfmanaged_snaps stripe_width 0 >>>>> application rbd >>>>> > removed_snaps_queue >>>>> > >>>>> [22d7~3,11561~2,11571~1,11573~1c,11594~6,1159b~f,115b0~1,115b3~1,115c3~1,115f3~1,115f5~e,11613~6,1161f~c,11637~1b,11660~1,11663~2,11673~1,116d1~c,116f5~10,11721~c] >>>>> > pool 7 'images' replicated size 3 min_size 2 crush_rule 0 >>>>> object_hash >>>>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 94609 >>>>> flags >>>>> > hashpspool,selfmanaged_snaps stripe_width 0 application rbd >>>>> > pool 8 'backups' replicated size 3 min_size 2 crush_rule 0 >>>>> object_hash >>>>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1399 >>>>> flags >>>>> > hashpspool stripe_width 0 application rbd >>>>> > pool 9 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash >>>>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 108783 >>>>> lfor >>>>> > 0/561/559 flags hashpspool,selfmanaged_snaps stripe_width 0 >>>>> application rbd >>>>> > removed_snaps_queue [3fa~1,3fc~3,400~1,402~1] >>>>> > pool 10 'testbench' replicated size 3 min_size 2 crush_rule 0 >>>>> object_hash >>>>> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20931 >>>>> lfor >>>>> > 0/20931/20929 flags hashpspool stripe_width 0 >>>>> > >>>>> > >>>>> > On Mon, Jan 29, 2024 at 2:09 PM Michel Niyoyita < micou12@gmail.com >>>>> <mailto: >>>>> > micou12@gmail.com>> wrote: >>>>> > Thank you Janne , >>>>> > >>>>> > no need of setting some flags like ceph osd set nodeep-scrub ??? >>>>> > >>>>> > Thank you >>>>> > >>>>> > On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson < >>>>> icepic.dz@gmail.com >>>>> > <mailto:icepic.dz@gmail.com>> wrote: >>>>> > Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita < >>>>> micou12@gmail.com >>>>> > <mailto:micou12@gmail.com>>: >>>>> > > >>>>> > > Thank you Frank , >>>>> > > >>>>> > > All disks are HDDs . Would like to know if I can increase
doing pool the this pool to please the
>>>>> number of >>>>> > PGs >>>>> > > live in production without a negative impact to the cluster. if >>>>> yes which >>>>> > > commands to use . >>>>> > >>>>> > Yes. "ceph osd pool set <poolname> pg_num <number larger than >>>>> before>" >>>>> > where the number usually should be a power of two that leads to a >>>>> > number of PGs per OSD between 100-200. >>>>> > >>>>> > -- >>>>> > May the most significant bit of your life be positive. >>>>> > >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>> >>>>
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- May the most significant bit of your life be positive.
Make sure you're on a fairly recent version of Ceph before doing this, though. Josh On Mon, Jan 29, 2024 at 5:05 AM Janne Johansson <icepic.dz@gmail.com> wrote:
Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita <micou12@gmail.com>:
Thank you Frank ,
All disks are HDDs . Would like to know if I can increase the number of PGs live in production without a negative impact to the cluster. if yes which commands to use .
Yes. "ceph osd pool set <poolname> pg_num <number larger than before>" where the number usually should be a power of two that leads to a number of PGs per OSD between 100-200.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I am running ceph pacific , version 16 , ubuntu 20 OS , deployed using ceph-ansible. Michel On Mon, Jan 29, 2024 at 4:47 PM Josh Baergen <jbaergen@digitalocean.com> wrote:
Make sure you're on a fairly recent version of Ceph before doing this, though.
Josh
On Mon, Jan 29, 2024 at 5:05 AM Janne Johansson <icepic.dz@gmail.com> wrote:
Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita <micou12@gmail.com>:
Thank you Frank ,
All disks are HDDs . Would like to know if I can increase the number
of PGs
live in production without a negative impact to the cluster. if yes which commands to use .
Yes. "ceph osd pool set <poolname> pg_num <number larger than before>" where the number usually should be a power of two that leads to a number of PGs per OSD between 100-200.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
You need to be running at least 16.2.11 on the OSDs so that you have the fix for https://tracker.ceph.com/issues/55631. On Mon, Jan 29, 2024 at 8:07 AM Michel Niyoyita <micou12@gmail.com> wrote:
I am running ceph pacific , version 16 , ubuntu 20 OS , deployed using ceph-ansible.
Michel
On Mon, Jan 29, 2024 at 4:47 PM Josh Baergen <jbaergen@digitalocean.com> wrote:
Make sure you're on a fairly recent version of Ceph before doing this, though.
Josh
On Mon, Jan 29, 2024 at 5:05 AM Janne Johansson <icepic.dz@gmail.com> wrote:
Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita <micou12@gmail.com>:
Thank you Frank ,
All disks are HDDs . Would like to know if I can increase the number of PGs live in production without a negative impact to the cluster. if yes which commands to use .
Yes. "ceph osd pool set <poolname> pg_num <number larger than before>" where the number usually should be a power of two that leads to a number of PGs per OSD between 100-200.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Respond back with "ceph versions" output If your sole goal is to eliminate the not scrubbed in time errors you can increase the aggressiveness of scrubbing by setting: osd_max_scrubs = 2 The default in pacific is 1. if you are going to start tinkering manually with the pg_num you will want to turn off the pg autoscaler on the pools you are touching. reducing the size of your PGs may make sense and help with scrubbing but if the pool has a lot of data it will take a long long time to finish. Respectfully, *Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham> On Mon, Jan 29, 2024 at 10:08 AM Michel Niyoyita <micou12@gmail.com> wrote:
I am running ceph pacific , version 16 , ubuntu 20 OS , deployed using ceph-ansible.
Michel
On Mon, Jan 29, 2024 at 4:47 PM Josh Baergen <jbaergen@digitalocean.com> wrote:
Make sure you're on a fairly recent version of Ceph before doing this, though.
Josh
On Mon, Jan 29, 2024 at 5:05 AM Janne Johansson <icepic.dz@gmail.com> wrote:
Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita <micou12@gmail.com
:
Thank you Frank ,
All disks are HDDs . Would like to know if I can increase the number
of PGs
live in production without a negative impact to the cluster. if yes which commands to use .
Yes. "ceph osd pool set <poolname> pg_num <number larger than before>" where the number usually should be a power of two that leads to a number of PGs per OSD between 100-200.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Setting osd_max_scrubs = 2 for HDD OSDs was a mistake I made. The result was that PGs needed a bit more than twice as long to deep-scrub. Net effect: high scrub load, much less user IO and, last but not least, the "not deep-scrubbed in time" problem got worse, because (2+eps)/2 > 1. For spinners a consideration looking at the actually available drive performance is required, plus a few things more, like PG count, distribution etc. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Wesley Dillingham <wes@wesdillingham.com> Sent: Monday, January 29, 2024 7:14 PM To: Michel Niyoyita Cc: Josh Baergen; E Taka; ceph-users Subject: [ceph-users] Re: 6 pgs not deep-scrubbed in time Respond back with "ceph versions" output If your sole goal is to eliminate the not scrubbed in time errors you can increase the aggressiveness of scrubbing by setting: osd_max_scrubs = 2 The default in pacific is 1. if you are going to start tinkering manually with the pg_num you will want to turn off the pg autoscaler on the pools you are touching. reducing the size of your PGs may make sense and help with scrubbing but if the pool has a lot of data it will take a long long time to finish. Respectfully, *Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham> On Mon, Jan 29, 2024 at 10:08 AM Michel Niyoyita <micou12@gmail.com> wrote:
I am running ceph pacific , version 16 , ubuntu 20 OS , deployed using ceph-ansible.
Michel
On Mon, Jan 29, 2024 at 4:47 PM Josh Baergen <jbaergen@digitalocean.com> wrote:
Make sure you're on a fairly recent version of Ceph before doing this, though.
Josh
On Mon, Jan 29, 2024 at 5:05 AM Janne Johansson <icepic.dz@gmail.com> wrote:
Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita <micou12@gmail.com
:
Thank you Frank ,
All disks are HDDs . Would like to know if I can increase the number
of PGs
live in production without a negative impact to the cluster. if yes which commands to use .
Yes. "ceph osd pool set <poolname> pg_num <number larger than before>" where the number usually should be a power of two that leads to a number of PGs per OSD between 100-200.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (6)
-
E Taka
-
Frank Schilder
-
Janne Johansson
-
Josh Baergen
-
Michel Niyoyita
-
Wesley Dillingham