On 24.05.2024 21:07, Mazzystr wrote:
I did the obnoxious task of updating ceph.conf and restarting all my osds.
ceph --admin-daemon /var/run/ceph/ceph-osd.*.asok config get osd_op_queue { "osd_op_queue": "wpq" }
I have some spare memory on my target host/osd and increased the target memory of that OSD to 10 Gb and restarted. No effect observed. In fact mem usage on the host is stable so I don't think the change took effect even with updating ceph.conf, restart and a direct asok config set. target memory value is confirmed to be set via asok config get
Nothing has helped. I still cannot break the 21 MiB/s barrier.
Does anyone have any more ideas?
For recovery you can adjust the following. osd_max_backfills default is 1, in my system I get the best performance with 3 and wpq. The following I have not adjusted myself, but you can try. osd_recovery_max_active is default to 3. osd_recovery_op_priority is default to 3, a lower number increases the priority for recovery. All of them can be runtime adjusted. -- Kai Stian Olstad
Well this was an interesting journey through the bowels of Ceph. I have about 6 hours into tweaking every setting imaginable just to circle back to my basic configuration and 2G memory target per osd. I was never able to exceed 22 Mib/Sec recovery time during that journey. I did end up fixing the issue and now I see the following - io: recovery: 129 MiB/s, 33 objects/s This is normal for my measly cluster. I like micro ceph clusters. I have a lot of them. :) What was the fix? Adding another disc to the recovery process! I was recovering to one disc now I'm recovering to two. I have three total that need to be recovered. Somehow that one disc was completely swamped. I was unable to see it in htop, atop, iostat. Disc business was 6% max. My config is back to mclock scheduler, profile high_recovery_ops, and backfills of 256. Thank you everyone that took the time to review and contribute. Hopefully this provides some modern information for the next person that has slow recovery. /Chris C On Fri, May 24, 2024 at 1:43 PM Kai Stian Olstad <ceph+list@olstad.com> wrote:
On 24.05.2024 21:07, Mazzystr wrote:
I did the obnoxious task of updating ceph.conf and restarting all my osds.
ceph --admin-daemon /var/run/ceph/ceph-osd.*.asok config get osd_op_queue { "osd_op_queue": "wpq" }
I have some spare memory on my target host/osd and increased the target memory of that OSD to 10 Gb and restarted. No effect observed. In fact mem usage on the host is stable so I don't think the change took effect even with updating ceph.conf, restart and a direct asok config set. target memory value is confirmed to be set via asok config get
Nothing has helped. I still cannot break the 21 MiB/s barrier.
Does anyone have any more ideas?
For recovery you can adjust the following.
osd_max_backfills default is 1, in my system I get the best performance with 3 and wpq.
The following I have not adjusted myself, but you can try. osd_recovery_max_active is default to 3. osd_recovery_op_priority is default to 3, a lower number increases the priority for recovery.
All of them can be runtime adjusted.
-- Kai Stian Olstad
Hi! Could you please elaborate what you meant by "adding another disc to the recovery process"? /Z On Sat, 25 May 2024, 22:49 Mazzystr, <mazzystr@gmail.com> wrote:
Well this was an interesting journey through the bowels of Ceph. I have about 6 hours into tweaking every setting imaginable just to circle back to my basic configuration and 2G memory target per osd. I was never able to exceed 22 Mib/Sec recovery time during that journey.
I did end up fixing the issue and now I see the following -
io: recovery: 129 MiB/s, 33 objects/s
This is normal for my measly cluster. I like micro ceph clusters. I have a lot of them. :)
What was the fix? Adding another disc to the recovery process! I was recovering to one disc now I'm recovering to two. I have three total that need to be recovered. Somehow that one disc was completely swamped. I was unable to see it in htop, atop, iostat. Disc business was 6% max.
My config is back to mclock scheduler, profile high_recovery_ops, and backfills of 256.
Thank you everyone that took the time to review and contribute. Hopefully this provides some modern information for the next person that has slow recovery.
/Chris C
On Fri, May 24, 2024 at 1:43 PM Kai Stian Olstad <ceph+list@olstad.com> wrote:
On 24.05.2024 21:07, Mazzystr wrote:
I did the obnoxious task of updating ceph.conf and restarting all my osds.
ceph --admin-daemon /var/run/ceph/ceph-osd.*.asok config get osd_op_queue { "osd_op_queue": "wpq" }
I have some spare memory on my target host/osd and increased the target memory of that OSD to 10 Gb and restarted. No effect observed. In fact mem usage on the host is stable so I don't think the change took effect even with updating ceph.conf, restart and a direct asok config set. target memory value is confirmed to be set via asok config get
Nothing has helped. I still cannot break the 21 MiB/s barrier.
Does anyone have any more ideas?
For recovery you can adjust the following.
osd_max_backfills default is 1, in my system I get the best performance with 3 and wpq.
The following I have not adjusted myself, but you can try. osd_recovery_max_active is default to 3. osd_recovery_op_priority is default to 3, a lower number increases the priority for recovery.
All of them can be runtime adjusted.
-- Kai Stian Olstad
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I can't explain the problem. I have to recover three discs that are hdds. I figured on just replacing one to give the full recovery capacity of the cluster to that one disc. I was never able to achieve a higher recovery rate than about 22 MiB/sec so I just added the other two discs. Recovery bounced up to 129 MiB/sec for a while. Then things settled at 50 MiB/sec. I kept tinkering to try to get back to 120 and now things are back to 23 MiB/Sec again. This is very irritating. Cpu usage is minimal in the single digit %. Mem is right on target per target setting in ceph.conf. Disc's and network appear to be 20% utilized. I'm not a normal Ceph user. I don't care about client access at all. The mclock assumptions are wrong for me. I want my data to be replicated correctly as fast as possible. How do I open up the floodgates for maximum recovery performance? On Sat, May 25, 2024 at 8:13 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Hi!
Could you please elaborate what you meant by "adding another disc to the recovery process"?
/Z
On Sat, 25 May 2024, 22:49 Mazzystr, <mazzystr@gmail.com> wrote:
Well this was an interesting journey through the bowels of Ceph. I have about 6 hours into tweaking every setting imaginable just to circle back to my basic configuration and 2G memory target per osd. I was never able to exceed 22 Mib/Sec recovery time during that journey.
I did end up fixing the issue and now I see the following -
io: recovery: 129 MiB/s, 33 objects/s
This is normal for my measly cluster. I like micro ceph clusters. I have a lot of them. :)
What was the fix? Adding another disc to the recovery process! I was recovering to one disc now I'm recovering to two. I have three total that need to be recovered. Somehow that one disc was completely swamped. I was unable to see it in htop, atop, iostat. Disc business was 6% max.
My config is back to mclock scheduler, profile high_recovery_ops, and backfills of 256.
Thank you everyone that took the time to review and contribute. Hopefully this provides some modern information for the next person that has slow recovery.
/Chris C
On Fri, May 24, 2024 at 1:43 PM Kai Stian Olstad <ceph+list@olstad.com> wrote:
On 24.05.2024 21:07, Mazzystr wrote:
I did the obnoxious task of updating ceph.conf and restarting all my osds.
ceph --admin-daemon /var/run/ceph/ceph-osd.*.asok config get osd_op_queue { "osd_op_queue": "wpq" }
I have some spare memory on my target host/osd and increased the target memory of that OSD to 10 Gb and restarted. No effect observed. In fact mem usage on the host is stable so I don't think the change took effect even with updating ceph.conf, restart and a direct asok config set. target memory value is confirmed to be set via asok config get
Nothing has helped. I still cannot break the 21 MiB/s barrier.
Does anyone have any more ideas?
For recovery you can adjust the following.
osd_max_backfills default is 1, in my system I get the best performance with 3 and wpq.
The following I have not adjusted myself, but you can try. osd_recovery_max_active is default to 3. osd_recovery_op_priority is default to 3, a lower number increases the priority for recovery.
All of them can be runtime adjusted.
-- Kai Stian Olstad
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Isn't this just the limit of one HDD or the other HDD's for providing the data? Don't forget, recovery will drop even more for the last few objects. At least I noticed this when replacing a drive in my (little) cluster. Kind regards, Sake
Op 26-05-2024 09:36 CEST schreef Mazzystr <mazzystr@gmail.com>:
I can't explain the problem. I have to recover three discs that are hdds. I figured on just replacing one to give the full recovery capacity of the cluster to that one disc. I was never able to achieve a higher recovery rate than about 22 MiB/sec so I just added the other two discs. Recovery bounced up to 129 MiB/sec for a while. Then things settled at 50 MiB/sec. I kept tinkering to try to get back to 120 and now things are back to 23 MiB/Sec again. This is very irritating.
Cpu usage is minimal in the single digit %. Mem is right on target per target setting in ceph.conf. Disc's and network appear to be 20% utilized.
I'm not a normal Ceph user. I don't care about client access at all. The mclock assumptions are wrong for me. I want my data to be replicated correctly as fast as possible.
How do I open up the floodgates for maximum recovery performance?
On Sat, May 25, 2024 at 8:13 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Hi!
Could you please elaborate what you meant by "adding another disc to the recovery process"?
/Z
On Sat, 25 May 2024, 22:49 Mazzystr, <mazzystr@gmail.com> wrote:
Well this was an interesting journey through the bowels of Ceph. I have about 6 hours into tweaking every setting imaginable just to circle back to my basic configuration and 2G memory target per osd. I was never able to exceed 22 Mib/Sec recovery time during that journey.
I did end up fixing the issue and now I see the following -
io: recovery: 129 MiB/s, 33 objects/s
This is normal for my measly cluster. I like micro ceph clusters. I have a lot of them. :)
What was the fix? Adding another disc to the recovery process! I was recovering to one disc now I'm recovering to two. I have three total that need to be recovered. Somehow that one disc was completely swamped. I was unable to see it in htop, atop, iostat. Disc business was 6% max.
My config is back to mclock scheduler, profile high_recovery_ops, and backfills of 256.
Thank you everyone that took the time to review and contribute. Hopefully this provides some modern information for the next person that has slow recovery.
/Chris C
On Fri, May 24, 2024 at 1:43 PM Kai Stian Olstad <ceph+list@olstad.com> wrote:
On 24.05.2024 21:07, Mazzystr wrote:
I did the obnoxious task of updating ceph.conf and restarting all my osds.
ceph --admin-daemon /var/run/ceph/ceph-osd.*.asok config get osd_op_queue { "osd_op_queue": "wpq" }
I have some spare memory on my target host/osd and increased the target memory of that OSD to 10 Gb and restarted. No effect observed. In fact mem usage on the host is stable so I don't think the change took effect even with updating ceph.conf, restart and a direct asok config set. target memory value is confirmed to be set via asok config get
Nothing has helped. I still cannot break the 21 MiB/s barrier.
Does anyone have any more ideas?
For recovery you can adjust the following.
osd_max_backfills default is 1, in my system I get the best performance with 3 and wpq.
The following I have not adjusted myself, but you can try. osd_recovery_max_active is default to 3. osd_recovery_op_priority is default to 3, a lower number increases the priority for recovery.
All of them can be runtime adjusted.
-- Kai Stian Olstad
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
What about drives IOPS ? Hdd tops at an average 150, you can use iostat -xmt to get these values (also last column show disk utilization which is very usefull) On Sun, May 26, 2024, 09:37 Mazzystr <mazzystr@gmail.com> wrote:
I can't explain the problem. I have to recover three discs that are hdds. I figured on just replacing one to give the full recovery capacity of the cluster to that one disc. I was never able to achieve a higher recovery rate than about 22 MiB/sec so I just added the other two discs. Recovery bounced up to 129 MiB/sec for a while. Then things settled at 50 MiB/sec. I kept tinkering to try to get back to 120 and now things are back to 23 MiB/Sec again. This is very irritating.
Cpu usage is minimal in the single digit %. Mem is right on target per target setting in ceph.conf. Disc's and network appear to be 20% utilized.
I'm not a normal Ceph user. I don't care about client access at all. The mclock assumptions are wrong for me. I want my data to be replicated correctly as fast as possible.
How do I open up the floodgates for maximum recovery performance?
On Sat, May 25, 2024 at 8:13 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Hi!
Could you please elaborate what you meant by "adding another disc to the recovery process"?
/Z
On Sat, 25 May 2024, 22:49 Mazzystr, <mazzystr@gmail.com> wrote:
Well this was an interesting journey through the bowels of Ceph. I have about 6 hours into tweaking every setting imaginable just to circle back to my basic configuration and 2G memory target per osd. I was never able to exceed 22 Mib/Sec recovery time during that journey.
I did end up fixing the issue and now I see the following -
io: recovery: 129 MiB/s, 33 objects/s
This is normal for my measly cluster. I like micro ceph clusters. I have a lot of them. :)
What was the fix? Adding another disc to the recovery process! I was recovering to one disc now I'm recovering to two. I have three total that need to be recovered. Somehow that one disc was completely swamped. I was unable to see it in htop, atop, iostat. Disc business was 6% max.
My config is back to mclock scheduler, profile high_recovery_ops, and backfills of 256.
Thank you everyone that took the time to review and contribute. Hopefully this provides some modern information for the next person that has slow recovery.
/Chris C
On Fri, May 24, 2024 at 1:43 PM Kai Stian Olstad <ceph+list@olstad.com> wrote:
On 24.05.2024 21:07, Mazzystr wrote:
I did the obnoxious task of updating ceph.conf and restarting all my osds.
ceph --admin-daemon /var/run/ceph/ceph-osd.*.asok config get osd_op_queue { "osd_op_queue": "wpq" }
I have some spare memory on my target host/osd and increased the target memory of that OSD to 10 Gb and restarted. No effect observed. In fact mem usage on the host is stable so I don't think the change took effect even with updating ceph.conf, restart and a direct asok config set. target memory value is confirmed to be set via asok config get
Nothing has helped. I still cannot break the 21 MiB/s barrier.
Does anyone have any more ideas?
For recovery you can adjust the following.
osd_max_backfills default is 1, in my system I get the best performance with 3 and wpq.
The following I have not adjusted myself, but you can try. osd_recovery_max_active is default to 3. osd_recovery_op_priority is default to 3, a lower number increases the priority for recovery.
All of them can be runtime adjusted.
-- Kai Stian Olstad
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Also osd_max_backfills and osd_recovery_max_active can plays a role, but I wonder if they still has effect with the new mpq feature. On Sun, May 26, 2024, 09:37 Mazzystr <mazzystr@gmail.com> wrote:
I can't explain the problem. I have to recover three discs that are hdds. I figured on just replacing one to give the full recovery capacity of the cluster to that one disc. I was never able to achieve a higher recovery rate than about 22 MiB/sec so I just added the other two discs. Recovery bounced up to 129 MiB/sec for a while. Then things settled at 50 MiB/sec. I kept tinkering to try to get back to 120 and now things are back to 23 MiB/Sec again. This is very irritating.
Cpu usage is minimal in the single digit %. Mem is right on target per target setting in ceph.conf. Disc's and network appear to be 20% utilized.
I'm not a normal Ceph user. I don't care about client access at all. The mclock assumptions are wrong for me. I want my data to be replicated correctly as fast as possible.
How do I open up the floodgates for maximum recovery performance?
On Sat, May 25, 2024 at 8:13 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Hi!
Could you please elaborate what you meant by "adding another disc to the recovery process"?
/Z
On Sat, 25 May 2024, 22:49 Mazzystr, <mazzystr@gmail.com> wrote:
Well this was an interesting journey through the bowels of Ceph. I have about 6 hours into tweaking every setting imaginable just to circle back to my basic configuration and 2G memory target per osd. I was never able to exceed 22 Mib/Sec recovery time during that journey.
I did end up fixing the issue and now I see the following -
io: recovery: 129 MiB/s, 33 objects/s
This is normal for my measly cluster. I like micro ceph clusters. I have a lot of them. :)
What was the fix? Adding another disc to the recovery process! I was recovering to one disc now I'm recovering to two. I have three total that need to be recovered. Somehow that one disc was completely swamped. I was unable to see it in htop, atop, iostat. Disc business was 6% max.
My config is back to mclock scheduler, profile high_recovery_ops, and backfills of 256.
Thank you everyone that took the time to review and contribute. Hopefully this provides some modern information for the next person that has slow recovery.
/Chris C
On Fri, May 24, 2024 at 1:43 PM Kai Stian Olstad <ceph+list@olstad.com> wrote:
On 24.05.2024 21:07, Mazzystr wrote:
I did the obnoxious task of updating ceph.conf and restarting all my osds.
ceph --admin-daemon /var/run/ceph/ceph-osd.*.asok config get osd_op_queue { "osd_op_queue": "wpq" }
I have some spare memory on my target host/osd and increased the target memory of that OSD to 10 Gb and restarted. No effect observed. In fact mem usage on the host is stable so I don't think the change took effect even with updating ceph.conf, restart and a direct asok config set. target memory value is confirmed to be set via asok config get
Nothing has helped. I still cannot break the 21 MiB/s barrier.
Does anyone have any more ideas?
For recovery you can adjust the following.
osd_max_backfills default is 1, in my system I get the best performance with 3 and wpq.
The following I have not adjusted myself, but you can try. osd_recovery_max_active is default to 3. osd_recovery_op_priority is default to 3, a lower number increases the priority for recovery.
All of them can be runtime adjusted.
-- Kai Stian Olstad
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
With mClock, osd_max_backfills and osd_recovery_max_active can be modified at runtime after setting osd_mclock_override_recovery_settings to true. See the docs <https://docs.ceph.com/en/latest/rados/configuration/mclock-config-ref/#steps-to-modify-mclock-max-backfills-recovery-limits> for more information. There's no change in the behavior when recovery/backfill limits are modified with mClock enabled. I suspect when you added a new osd, the recovery traffic you observed could just be related to the backfill operation trying to move data due to PGs mapped to the new osd. -Sridhar
I suspect my initial spike in performance was pg's balancing between the three osd of the one host. host load is very low, under 1. hdd iops on the three discs hover around 80 +/- 5. atop shows about 20% business. Gig-ethernet shows about 20% utilized according to atop. I find it extremely hard to believe .2 gig-e can swamp three hdds. I give up. Recovery performance is what it is. Next week when this operation completes I have two more osd recovery operations where the osds are located on two different hosts. It'll be interesting to /C On Mon, May 27, 2024 at 1:17 AM Sridhar Seshasayee <sseshasa@redhat.com> wrote:
With mClock, osd_max_backfills and osd_recovery_max_active can be modified at runtime after setting osd_mclock_override_recovery_settings to true. See the docs <https://docs.ceph.com/en/latest/rados/configuration/mclock-config-ref/#steps-to-modify-mclock-max-backfills-recovery-limits> for more information.
There's no change in the behavior when recovery/backfill limits are modified with mClock enabled.
I suspect when you added a new osd, the recovery traffic you observed could just be related to the backfill operation trying to move data due to PGs mapped to the new osd.
-Sridhar
Each simultaneously. These are SATA3 with 128mb cache drives. The bus is 6 gb/s. I expect usage to be in the 90+% range not the 50% range. On Mon, May 27, 2024 at 5:37 PM Anthony D'Atri <aad@dreamsnake.net> wrote:
hdd iops on the three discs hover around 80 +/- 5.
Each or total? I wouldn’t expect much more than 80 per drive.
Each simultaneously. These are SATA3 with 128mb cache drives.
Turn off the cache.
The bus is 6 gb/s. I expect usage to be in the 90+% range not the 50% range.
"usage" as measured how?
On Mon, May 27, 2024 at 5:37 PM Anthony D'Atri <aad@dreamsnake.net> wrote:
hdd iops on the three discs hover around 80 +/- 5.
Each or total? I wouldn’t expect much more than 80 per drive.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (8)
-
Anthony D'Atri
-
Anthony D'Atri
-
Cedric
-
Kai Stian Olstad
-
Mazzystr
-
Sake Ceph
-
Sridhar Seshasayee
-
Zakhar Kirpichenko