Hi, all I am suffering from slow backfilling and recovering. I increased to "max_backfills" = 64, "recovery_max_actives" = 64. I know it's unnessary to increase more in my ceph cluster. When I checked pg query, I found the number of "backfills_in_flight"&"recovering" is 4. I set "osd_recovery_max_single_start = 4" so I think "osd_recovery_max_single_start" affects to the number of "backfills_in_flight"&"recovering". I increased to "osd_recovery_max_single_start = 16" but the number of "backfills_in_flight"&"recovering" in pg query was not changed. I know "osd_recovery_max_single_start" runtime is true. Should I restart the osds to increase the number of "backfills_in_flight"&"recovering"? or Is there the other way?
Hi, What version of Ceph is this? Which scheduler is in use? mClock or WPQ? $ ceph config show osd.0 osd_op_queue If the OSD's scheduler is mClock then unless you set osd_mclock_override_recovery_settings to true (not recommended), changing osd_max_backfills and osd_recovery_max_active will have no impact. mClock requires using profiles. Read this [1]. Regards, Frédéric. [1] https://docs.ceph.com/en/reef/rados/configuration/mclock-config-ref/?#recove... ----- Le 31 Jan 25, à 8:25, Jaemin Joo jm7.joo@gmail.com a écrit :
Hi, all
I am suffering from slow backfilling and recovering. I increased to "max_backfills" = 64, "recovery_max_actives" = 64. I know it's unnessary to increase more in my ceph cluster. When I checked pg query, I found the number of "backfills_in_flight"&"recovering" is 4. I set "osd_recovery_max_single_start = 4" so I think "osd_recovery_max_single_start" affects to the number of "backfills_in_flight"&"recovering". I increased to "osd_recovery_max_single_start = 16" but the number of "backfills_in_flight"&"recovering" in pg query was not changed. I know "osd_recovery_max_single_start" runtime is true. Should I restart the osds to increase the number of "backfills_in_flight"&"recovering"? or Is there the other way? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thank you for your response quickly. My cluster version is 18.2.1 and the OSD's scheduler is mClick. I set osd_mclock_override_recovery_settings to true when I was changing osd_max_backfills & osd_recovery_max_active. Do I set osd_mclock_override_recovery_settings to true when I change osd_recovery_max_single_start? Could you explain more what osd_recovery_max_single_start is? I wonder whether it affects the number of "backfills_in_flight" & "recovering" in pg query.
Thank you for your response quickly. My cluster version is 18.2.1 and the OSD's scheduler is mClick. I set osd_mclock_override_recovery_settings to true when I was changing osd_max_backfills & osd_recovery_max_active. Do I set osd_mclock_override_recovery_settings to true when I change osd_recovery_max_single_start? Could you explain more what osd_recovery_max_single_start is? I wonder whether it affects the number of "backfills_in_flight" & "recovering" in pg query. 2025년 1월 31일 (금) 오후 5:05, Frédéric Nass <frederic.nass@univ-lorraine.fr>님이 작성:
Hi,
What version of Ceph is this? Which scheduler is in use? mClock or WPQ?
$ ceph config show osd.0 osd_op_queue
If the OSD's scheduler is mClock then unless you set osd_mclock_override_recovery_settings to true (not recommended), changing osd_max_backfills and osd_recovery_max_active will have no impact. mClock requires using profiles. Read this [1].
Regards, Frédéric.
[1] https://docs.ceph.com/en/reef/rados/configuration/mclock-config-ref/?#recove...
----- Le 31 Jan 25, à 8:25, Jaemin Joo jm7.joo@gmail.com a écrit :
Hi, all
I am suffering from slow backfilling and recovering. I increased to "max_backfills" = 64, "recovery_max_actives" = 64. I know it's unnessary to increase more in my ceph cluster. When I checked pg query, I found the number of "backfills_in_flight"&"recovering" is 4. I set "osd_recovery_max_single_start = 4" so I think "osd_recovery_max_single_start" affects to the number of "backfills_in_flight"&"recovering". I increased to "osd_recovery_max_single_start = 16" but the number of "backfills_in_flight"&"recovering" in pg query was not changed. I know "osd_recovery_max_single_start" runtime is true. Should I restart the osds to increase the number of "backfills_in_flight"&"recovering"? or Is there the other way? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I'm not sure about the effect of osd_mclock_override_recovery_settings over osd_recovery_max_active, at least the documentation doesn't mention any links between the two. One would have to go through the code to check this. Generally speaking, I would advise using either mClock with profiles or WPQ with historical options. Mixing the two makes things more unpredictable. Regards, Frédéric. ----- Le 31 Jan 25, à 9:51, Jaemin Joo <jm7.joo@gmail.com> a écrit :
Thank you for your response quickly.
My cluster version is 18.2.1 and the OSD's scheduler is mClick. I set osd_mclock_override_recovery_settings to true when I was changing osd_max_backfills & osd_recovery_max_active. Do I set osd_mclock_override_recovery_settings to true when I change osd_recovery_max_single_start? Could you explain more what osd_recovery_max_single_start is? I wonder whether it affects the number of "backfills_in_flight" & "recovering" in pg query.
2025년 1월 31일 (금) 오후 5:05, Frédéric Nass < [ mailto:frederic.nass@univ-lorraine.fr | frederic.nass@univ-lorraine.fr ] >님이 작성:
Hi,
What version of Ceph is this? Which scheduler is in use? mClock or WPQ?
$ ceph config show osd.0 osd_op_queue
If the OSD's scheduler is mClock then unless you set osd_mclock_override_recovery_settings to true (not recommended), changing osd_max_backfills and osd_recovery_max_active will have no impact. mClock requires using profiles. Read this [1].
Regards, Frédéric.
[1] [ https://docs.ceph.com/en/reef/rados/configuration/mclock-config-ref/?#recove... | https://docs.ceph.com/en/reef/rados/configuration/mclock-config-ref/?#recove... ]
----- Le 31 Jan 25, à 8:25, Jaemin Joo [ mailto:jm7.joo@gmail.com | jm7.joo@gmail.com ] a écrit :
Hi, all
I am suffering from slow backfilling and recovering. I increased to "max_backfills" = 64, "recovery_max_actives" = 64. I know it's unnessary to increase more in my ceph cluster. When I checked pg query, I found the number of "backfills_in_flight"&"recovering" is 4. I set "osd_recovery_max_single_start = 4" so I think "osd_recovery_max_single_start" affects to the number of "backfills_in_flight"&"recovering". I increased to "osd_recovery_max_single_start = 16" but the number of "backfills_in_flight"&"recovering" in pg query was not changed. I know "osd_recovery_max_single_start" runtime is true. Should I restart the osds to increase the number of "backfills_in_flight"&"recovering"? or Is there the other way? _______________________________________________ ceph-users mailing list -- [ mailto:ceph-users@ceph.io | ceph-users@ceph.io ] To unsubscribe send an email to [ mailto:ceph-users-leave@ceph.io | ceph-users-leave@ceph.io ]
Hi Jaemin, If you set osd_mclock_override_recovery_settings to true e.g. ~ # ceph config set osd osd_mclock_override_recovery_settings true and then increase max backfills like ~ # ceph config set osd osd_max_backfills 5 It will instantly increase backfills. I suggest starting with a low number of max backfills and monitor how your cluster behaves before increasing further. Important: When your backfill and recovery is done lower max backfill again. ~ # ceph config set osd osd_max_backfills 2 and the the overwrite back to false ~ # ceph config set osd osd_mclock_override_recovery_settings false Cheers Stephan Am Fr., 31. Jan. 2025 um 10:14 Uhr schrieb Frédéric Nass < frederic.nass@univ-lorraine.fr>:
I'm not sure about the effect of osd_mclock_override_recovery_settings over osd_recovery_max_active, at least the documentation doesn't mention any links between the two. One would have to go through the code to check this.
Generally speaking, I would advise using either mClock with profiles or WPQ with historical options. Mixing the two makes things more unpredictable.
Regards, Frédéric.
----- Le 31 Jan 25, à 9:51, Jaemin Joo <jm7.joo@gmail.com> a écrit :
Thank you for your response quickly.
My cluster version is 18.2.1 and the OSD's scheduler is mClick. I set osd_mclock_override_recovery_settings to true when I was changing osd_max_backfills & osd_recovery_max_active. Do I set osd_mclock_override_recovery_settings to true when I change osd_recovery_max_single_start? Could you explain more what osd_recovery_max_single_start is? I wonder whether it affects the number of "backfills_in_flight" & "recovering" in pg query.
2025년 1월 31일 (금) 오후 5:05, Frédéric Nass < [ mailto:frederic.nass@univ-lorraine.fr | frederic.nass@univ-lorraine.fr ] >님이 작성:
Hi,
What version of Ceph is this? Which scheduler is in use? mClock or WPQ?
$ ceph config show osd.0 osd_op_queue
If the OSD's scheduler is mClock then unless you set osd_mclock_override_recovery_settings to true (not recommended), changing osd_max_backfills and osd_recovery_max_active will have no impact. mClock requires using profiles. Read this [1].
Regards, Frédéric.
[1] [
https://docs.ceph.com/en/reef/rados/configuration/mclock-config-ref/?#recove...
|
https://docs.ceph.com/en/reef/rados/configuration/mclock-config-ref/?#recove...
]
----- Le 31 Jan 25, à 8:25, Jaemin Joo [ mailto:jm7.joo@gmail.com | jm7.joo@gmail.com ] a écrit :
Hi, all
I am suffering from slow backfilling and recovering. I increased to "max_backfills" = 64, "recovery_max_actives" = 64. I know it's unnessary to increase more in my ceph cluster. When I checked pg query, I found the number of "backfills_in_flight"&"recovering" is 4. I set "osd_recovery_max_single_start = 4" so I think "osd_recovery_max_single_start" affects to the number of "backfills_in_flight"&"recovering". I increased to "osd_recovery_max_single_start = 16" but the number of "backfills_in_flight"&"recovering" in pg query was not changed. I know "osd_recovery_max_single_start" runtime is true. Should I restart the osds to increase the number of "backfills_in_flight"&"recovering"? or Is there the other way? _______________________________________________ ceph-users mailing list -- [ mailto:ceph-users@ceph.io | ceph-users@ceph.io ] To unsubscribe send an email to [ mailto:ceph-users-leave@ceph.io | ceph-users-leave@ceph.io ]
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Frédéric Nass
-
jaemin joo
-
Jaemin Joo
-
Stephan Hohn