Do not use SSDs with (small) SLC cache
Hi all, digging around debugging, why our (small: 10 Hosts/~60 OSDs) cluster is so slow even while recovering I found out one of our key issues are some SSDs with SLC cache (in our case Samsung SSD 870 EVO) - which we just recycled from other use cases in the hope to speed up our mainly hdd based cluster. We know it's a little bit random which objects get accelerated when not used as cache. However the opposite was the case. These type's ssds are only fast when operating in their SLC cache, which is only several Gigabytes in a multi-TB ssd [1]. When doing a big write or a backfill onto these SSDs we got really low IO-rates (around 10 MB/s even with 4M-objects). But it got even worse. Disclaimer: This is my view as a user, maybe a more technically involved person is able to correct me. Cause seems to be the mclock-scheduler which measures the iops an osd is able to do. As in the blog measured [2], this is usually a good thing as there is done some profiling and queing is done different. But in our case the osd_mclock_max_capacity_iops_ssd for most of the corresponding osds was very low. But not for everyone. I assume that it depends when mclock-scheduler measured the iops capacity. That led to a broken scheduling where backfills were at low speed and the ssd itself had nearly no disk usage because it was operating in it's cache again and could work faster. That issue could be solved by switching back to wpq scheduler for the affected SSDs. This scheduler seems to just queue up ios without throttling because of maximum iops reached. Now we see a still bad IO situation because of the slow SSDs but at least they are operating at their maximum (having typical settings like osd_recovery_max_active and osd_recovery_sleep* tuned). We are going to replace the SSDs to hopefully more consistent performing ones (even if their peak performance would be not as good). I hope this may help somebody in the future when being stuck in low performance recoverys. Refs: [1] https://www.tomshardware.com/reviews/samsung-870-evo-sata-ssd-review-the-bes... [2] https://ceph.io/en/news/blog/2022/mclock-vs-wpq-testing-with-background-ops-... Happy Storing! Michael Wodniok -- Michael Wodniok M.Sc. WorNet AG Bürgermeister-Graf-Ring 28 82538 Geretsried Simply42 und SecuMail sind Marken der WorNet AG. http://www.wor.net/ Handelsregister Amtsgericht München (HRB 129882) Vorstand: Christian Eich Aufsichtsratsvorsitzender: Dirk Steinkopf
Dear Michael, I don't have an explanation for your problem unfortunately, but I just wondered that you experience a drop in performance, that this SSD shouldn't have. Your SSDs drives (Samsung 870 EVO) should not get slower on large writes. You can verify this on the post you've attached [1] or here [3]. I am curious if replacing them with other disks will improve it. [3] https://www.anandtech.com/show/16480/the-samsung-870-evo-ssd-1tb-4tb-review/... Best Ken On 21.02.23 08:53, Michael Wodniok wrote:
Hi all,
digging around debugging, why our (small: 10 Hosts/~60 OSDs) cluster is so slow even while recovering I found out one of our key issues are some SSDs with SLC cache (in our case Samsung SSD 870 EVO) - which we just recycled from other use cases in the hope to speed up our mainly hdd based cluster. We know it's a little bit random which objects get accelerated when not used as cache.
However the opposite was the case. These type's ssds are only fast when operating in their SLC cache, which is only several Gigabytes in a multi-TB ssd [1]. When doing a big write or a backfill onto these SSDs we got really low IO-rates (around 10 MB/s even with 4M-objects).
But it got even worse. Disclaimer: This is my view as a user, maybe a more technically involved person is able to correct me. Cause seems to be the mclock-scheduler which measures the iops an osd is able to do. As in the blog measured [2], this is usually a good thing as there is done some profiling and queing is done different. But in our case the osd_mclock_max_capacity_iops_ssd for most of the corresponding osds was very low. But not for everyone. I assume that it depends when mclock-scheduler measured the iops capacity. That led to a broken scheduling where backfills were at low speed and the ssd itself had nearly no disk usage because it was operating in it's cache again and could work faster. That issue could be solved by switching back to wpq scheduler for the affected SSDs. This scheduler seems to just queue up ios without throttling because of maximum iops reached. Now we see a still bad IO situation because of the slow SSDs but at least they are operating at their maximum (having typical settings like osd_recovery_max_active and osd_recovery_sleep* tuned).
We are going to replace the SSDs to hopefully more consistent performing ones (even if their peak performance would be not as good).
I hope this may help somebody in the future when being stuck in low performance recoverys.
Refs:
[1] https://www.tomshardware.com/reviews/samsung-870-evo-sata-ssd-review-the-bes... [2] https://ceph.io/en/news/blog/2022/mclock-vs-wpq-testing-with-background-ops-...
Happy Storing! Michael Wodniok
--
Michael Wodniok M.Sc. WorNet AG Bürgermeister-Graf-Ring 28 82538 Geretsried
Simply42 und SecuMail sind Marken der WorNet AG. http://www.wor.net/
Handelsregister Amtsgericht München (HRB 129882) Vorstand: Christian Eich Aufsichtsratsvorsitzender: Dirk Steinkopf
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Ken, thank you for your hint - any input is appreciated. Please note that Ceph does highly random IO (especially when having small object sizes), AnandTech also states: "Some of our other tests have shown a few signs that the 870 EVO's write performance can drop when the SLC cache runs out, but this straightforward sequential write pass over the entire drive doesn't reveal any such behavior. The 870 EVO's sequential write performance is extremely consistent, even on the second write pass." [3] So this kind of cache handling is very interesting under the hood, because it seems the Samsung SSD 870 Evo is able to handle sequential io at nearly SATA line speed. However the random behaviour seems to be inconsistent - maybe I'm able to run a big fio on the ssd once I have one of these on my desk, but I can't promise I'll have time to. More interesting: We also have some Kingston SEDC450. Kingston even promises constant write speed [4] - even when specifying lower performance. At least those drives did not fell into my eye (but I did not examine the situation specially). [4] https://www.kingston.com/en/ssd/dc450-data-center-solid-state-drive Best regards, Michael -----Ursprüngliche Nachricht----- Von: mailing-lists <mailing-lists@indane.de> Gesendet: Dienstag, 21. Februar 2023 10:21 An: ceph-users@ceph.io Betreff: [ceph-users] Re: Do not use SSDs with (small) SLC cache Dear Michael, I don't have an explanation for your problem unfortunately, but I just wondered that you experience a drop in performance, that this SSD shouldn't have. Your SSDs drives (Samsung 870 EVO) should not get slower on large writes. You can verify this on the post you've attached [1] or here [3]. I am curious if replacing them with other disks will improve it. [3] https://www.anandtech.com/show/16480/the-samsung-870-evo-ssd-1tb-4tb-review/... Best Ken
What fio test would indicate this behaviour up front? I guess something like this, but with a duration larger than this disk cache? [randwrite-4k-seq] stonewall bs=4k rw=randwrite fsync=1
thank you for your hint - any input is appreciated. Please note that Ceph does highly random IO (especially when having small object sizes), AnandTech also states:
"Some of our other tests have shown a few signs that the 870 EVO's write performance can drop when the SLC cache runs out, but this straightforward sequential write pass over the entire drive doesn't reveal any such behavior. The 870 EVO's sequential write performance is extremely consistent, even on the second write pass." [3]
So this kind of cache handling is very interesting under the hood, because it seems the Samsung SSD 870 Evo is able to handle sequential io at nearly SATA line speed. However the random behaviour seems to be inconsistent - maybe I'm able to run a big fio on the ssd once I have one of these on my desk, but I can't promise I'll have time to.
More interesting: We also have some Kingston SEDC450. Kingston even promises constant write speed [4] - even when specifying lower performance. At least those drives did not fell into my eye (but I did not examine the situation specially).
[4] https://www.kingston.com/en/ssd/dc450-data-center-solid-state-drive
Best regards, Michael
-----Ursprüngliche Nachricht----- Von: mailing-lists <mailing-lists@indane.de> Gesendet: Dienstag, 21. Februar 2023 10:21 An: ceph-users@ceph.io Betreff: [ceph-users] Re: Do not use SSDs with (small) SLC cache
Dear Michael,
I don't have an explanation for your problem unfortunately, but I just wondered that you experience a drop in performance, that this SSD shouldn't have. Your SSDs drives (Samsung 870 EVO) should not get slower on large writes. You can verify this on the post you've attached [1] or here [3].
I am curious if replacing them with other disks will improve it.
[3] https://www.anandtech.com/show/16480/the-samsung-870-evo-ssd-1tb-4tb- review/4
Best
Ken
Hi Marc, I would try something the near your example, yes. You could add "size=<yourdisksize>" for testing until once written the whole disk (by io, not by all available cells) independent of time. If the result is either high deviation or the results are far from the specification it's very likely there is a SLC cache in use. Best regards, Michael -----Ursprüngliche Nachricht----- Von: Marc <Marc@f1-outsourcing.eu> Gesendet: Dienstag, 21. Februar 2023 11:27 An: Michael Wodniok <wodniok@wor.net>; ceph-users@ceph.io; Phil Regnauld <pr@x0.dk> Betreff: RE: [ceph-users] Re: Do not use SSDs with (small) SLC cache What fio test would indicate this behaviour up front? I guess something like this, but with a duration larger than this disk cache? [randwrite-4k-seq] stonewall bs=4k rw=randwrite fsync=1
thank you for your hint - any input is appreciated. Please note that Ceph does highly random IO (especially when having small object sizes), AnandTech also states:
"Some of our other tests have shown a few signs that the 870 EVO's write performance can drop when the SLC cache runs out, but this straightforward sequential write pass over the entire drive doesn't reveal any such behavior. The 870 EVO's sequential write performance is extremely consistent, even on the second write pass." [3]
So this kind of cache handling is very interesting under the hood, because it seems the Samsung SSD 870 Evo is able to handle sequential io at nearly SATA line speed. However the random behaviour seems to be inconsistent - maybe I'm able to run a big fio on the ssd once I have one of these on my desk, but I can't promise I'll have time to.
More interesting: We also have some Kingston SEDC450. Kingston even promises constant write speed [4] - even when specifying lower performance. At least those drives did not fell into my eye (but I did not examine the situation specially).
[4] https://www.kingston.com/en/ssd/dc450-data-center-solid-state-drive
Best regards, Michael
-----Ursprüngliche Nachricht----- Von: mailing-lists <mailing-lists@indane.de> Gesendet: Dienstag, 21. Februar 2023 10:21 An: ceph-users@ceph.io Betreff: [ceph-users] Re: Do not use SSDs with (small) SLC cache
Dear Michael,
I don't have an explanation for your problem unfortunately, but I just wondered that you experience a drop in performance, that this SSD shouldn't have. Your SSDs drives (Samsung 870 EVO) should not get slower on large writes. You can verify this on the post you've attached [1] or here [3].
I am curious if replacing them with other disks will improve it.
[3] https://www.anandtech.com/show/16480/the-samsung-870-evo-ssd-1tb-4tb- review/4
Best
Ken
Michael Wodniok (wodniok) writes:
Hi all,
digging around debugging, why our (small: 10 Hosts/~60 OSDs) cluster is so slow even while recovering I found out one of our key issues are some SSDs with SLC cache (in our case Samsung SSD 870 EVO) - which we just recycled from other use cases in the hope to speed up our mainly hdd based cluster. We know it's a little bit random which objects get accelerated when not used as cache.
Somewhat off-topic, but... may I recommend *not* using the 870s ? There is an issue with 870s failing around 14-17 TB written if they're not running the latest firmware... I recently pulled 25 of them out of our storage because of this. https://twitter.com/regnauld/status/1502185863052152833 Cheers, Phil
Hi, I'm just writing to share some old knowledge, which is: never, ever use consumer ssd for ceph! see e.g. https://old.reddit.com/r/Proxmox/comments/izg6e5/questions_on_running_ceph_u... -- Mit freundlichen Grüßen / Regards Sven Kieske Systementwickler / systems engineer Mittwald CM Service GmbH & Co. KG Königsberger Straße 4-6 32339 Espelkamp Tel.: 05772 / 293-900 Fax: 05772 / 293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer, Florian Jürgens St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen Informationen zur Datenverarbeitung im Rahmen unserer Geschäftstätigkeit gemäß Art. 13-14 DSGVO sind unter www.mittwald.de/ds abrufbar.
Hey I have seen that kind of behavior in the past and we manage to flash the firmware to increase the cache size, which will kill the drive a little bit faster so only to use in lab environments. I'm unaware if the Samsung magician software can do that. btw, a few evo's 9xx are listed by Samsung to receive upgrades or are available for replacement because of firmware issues, I don't know the exact list but the Samsung site for sure will have more information. Cheers! On Tue, Feb 21, 2023 at 10:23 AM Sven Kieske <S.Kieske@mittwald.de> wrote:
Hi,
I'm just writing to share some old knowledge, which is:
never, ever use consumer ssd for ceph!
see e.g. https://old.reddit.com/r/Proxmox/comments/izg6e5/questions_on_running_ceph_u... -- Mit freundlichen Grüßen / Regards
Sven Kieske Systementwickler / systems engineer
Mittwald CM Service GmbH & Co. KG Königsberger Straße 4-6 32339 Espelkamp
Tel.: 05772 / 293-900 Fax: 05772 / 293-333
Geschäftsführer: Robert Meyer, Florian Jürgens
St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen
Informationen zur Datenverarbeitung im Rahmen unserer Geschäftstätigkeit gemäß Art. 13-14 DSGVO sind unter www.mittwald.de/ds abrufbar.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.
A bit late to the game, but I'm not sure if it is your drives. I had a very similar issue to yours on enterprise drives (not that means much outside of support). What I was seeing is that a rebuild would kick off, PGs would instantly start to become laggy and then our clients (openstack rbd) would start getting hit by a slow-requests since it would start read locking the osd because of the expiring lease. This was felt by the client and was a production issue that cost the company. I spent weeks trying to tune the mClock profile, including the overall io cap, rebuild cap, and recovery cap (clients were unlimited the whole time). None of it really worked, so I switched to wpq. With that one configuration switch, all the problems went away with no real impact to rebuild time. To clarify I don't really care for fast rebuilds as long as the rebuild time was in a reasonable, but the client impact was just killing us. I also could never get this to trip unless it was during a recovery or rebuild. I could slam our cluster with 100k iops (random or sequential) from a bunch of different clients, which is about 50x our normal load (yeah, I know this cluster is massively over built in terms of performance), and there were zero issues. In our use case we have flagged mClock as unstable. Since we want this to work because the concept is awesome, we will retest at the 18.2. ________________________________ From: Michael Wodniok <wodniok@wor.net> Sent: Tuesday, February 21, 2023 12:53 AM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Do not use SSDs with (small) SLC cache Hi all, digging around debugging, why our (small: 10 Hosts/~60 OSDs) cluster is so slow even while recovering I found out one of our key issues are some SSDs with SLC cache (in our case Samsung SSD 870 EVO) - which we just recycled from other use cases in the hope to speed up our mainly hdd based cluster. We know it's a little bit random which objects get accelerated when not used as cache. However the opposite was the case. These type's ssds are only fast when operating in their SLC cache, which is only several Gigabytes in a multi-TB ssd [1]. When doing a big write or a backfill onto these SSDs we got really low IO-rates (around 10 MB/s even with 4M-objects). But it got even worse. Disclaimer: This is my view as a user, maybe a more technically involved person is able to correct me. Cause seems to be the mclock-scheduler which measures the iops an osd is able to do. As in the blog measured [2], this is usually a good thing as there is done some profiling and queing is done different. But in our case the osd_mclock_max_capacity_iops_ssd for most of the corresponding osds was very low. But not for everyone. I assume that it depends when mclock-scheduler measured the iops capacity. That led to a broken scheduling where backfills were at low speed and the ssd itself had nearly no disk usage because it was operating in it's cache again and could work faster. That issue could be solved by switching back to wpq scheduler for the affected SSDs. This scheduler seems to just queue up ios without throttling because of maximum iops reached. Now we see a still bad IO situation because of the slow SSDs but at least they are operating at their maximum (having typical settings like osd_recovery_max_active and osd_recovery_sleep* tuned). We are going to replace the SSDs to hopefully more consistent performing ones (even if their peak performance would be not as good). I hope this may help somebody in the future when being stuck in low performance recoverys. Refs: [1] https://www.tomshardware.com/reviews/samsung-870-evo-sata-ssd-review-the-bes... [2] https://ceph.io/en/news/blog/2022/mclock-vs-wpq-testing-with-background-ops-... Happy Storing! Michael Wodniok -- Michael Wodniok M.Sc. WorNet AG Bürgermeister-Graf-Ring 28 82538 Geretsried Simply42 und SecuMail sind Marken der WorNet AG. http://www.wor.net/ Handelsregister Amtsgericht München (HRB 129882) Vorstand: Christian Eich Aufsichtsratsvorsitzender: Dirk Steinkopf ________________________________ CONFIDENTIALITY NOTICE: This message is intended only for the use and review of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message solely to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify sender immediately by telephone or return email. Thank you.
participants (7)
-
Alvaro Soto
-
mailing-lists
-
Marc
-
Matthew Stroud
-
Michael Wodniok
-
Phil Regnauld
-
Sven Kieske