This isn't the first time I've seen drive cache cause problematic latency issues, and not always from the same manufacturer. Unfortunately it seems like you really have to test the drives you want to use before deploying them them to make sure you don't run into issues. Mark On 6/24/20 6:36 AM, Stefan Priebe - Profihost AG wrote:

HI Ben,

yes we have the same issues and switched to seagate for those reasons.

you can fix at least a big part of it by disabling the write cache of those drives - generally speaking it seems the toshiba firmware is broken.

I was not able to find a newer one.

Greets, Stefan

Am 24.06.20 um 09:43 schrieb Benoît Knecht:

...

Hi,

We have a Nautilus (14.2.9) Ceph cluster with two types of HDDs:

- TOSHIBA MG07ACA14TE [1] - HGST HUH721212ALE604 [2]

They're all bluestore OSDs with no separate DB+WAL and part of the same pool.

We noticed that while the HGST OSDs have a commit latency of about 15ms, the Toshiba OSDs hover around 150ms (these values come from the `ceph_osd_commit_latency_ms` metric in Prometheus).

On paper, it seems like those drives have very similar specs, so it's not clear to me why we're seeing such a large difference when it comes to commit latency.

Has anyone had any experience with those Toshiba drives? Or looking at the specs, do you spot anything suspicious?

And if you're running a Ceph cluster with various disk brands/models, have you ever noticed some of them standing out when looking at `ceph_osd_commit_latency_ms`?

Thanks in advance for your feedback.

Cheers,

-- Ben

[1]: https://toshiba.semicon-storage.com/content/dam/toshiba-ss/asia-pacific/docs... [2]: https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/pu... _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Thank you all for your answers, this was really helpful! Stefan Priebe wrote:

yes we have the same issues and switched to seagate for those reasons. you can fix at least a big part of it by disabling the write cache of those drives - generally speaking it seems the toshiba firmware is broken. I was not able to find a newer one.

Good to know that we're not alone :) I also looked for a newer firmware, to no avail. Igor Fedotov wrote:

Benoit, wondering what are the write cache settings in your case?

And do you see any difference after disabling it if any?

Write cache is enabled on all our OSDs (including the HGST drives that don't have a latency issue). To see if disabling write cache on the Toshiba drives would help, I turned it off on all 12 drives in one of our OSD nodes: ``` for disk in /dev/sd{a..l}; do hdparm -W0 $disk; done ``` and left it on in the remaining nodes. I used `rados bench write` to create some load on the cluster, and looked at ``` avg by (hostname) (ceph_osd_commit_latency_ms * on (ceph_daemon) group_left (hostname) ceph_osd_metadata) ``` in Prometheus. The hosts with write cache _enabled_ had a commit latency around 145ms, while the host with write cache _disabled_ had a commit latency around 25ms. So it definitely helps! Mark Nelson wrote:

This isn't the first time I've seen drive cache cause problematic latency issues, and not always from the same manufacturer. Unfortunately it seems like you really have to test the drives you want to use before deploying them them to make sure you don't run into issues.

That's very true! Data sheets and even public benchmarks can be quite deceiving, and two hard drives that seem to have similar performance profiles can perform very differently within a Ceph cluster. Lesson learned. Cheers, -- Ben

Hi Igor, Igor Fedotov wrote:

for the sake of completeness one more experiment please if possible:

turn off write cache for HGST drives and measure commit latency once again.

I just did the same experiment with HGST drives, and disabling the write cache on those drives brought the latency down from about 7.5ms to about 4ms. So it seems disabling the write cache across the board would be advisable in our case. Is it recommended in general, or specifically when the DB+WAL is on the same hard drive? Stefan, Mark, are you disabling the write cache on your HDDs by default? Cheers, -- Ben

Has anyone ever encountered a drive with a write cache that actually *helped*? I haven't. As in: would it be a good idea for the OSD to just disable the write cache on startup? Worst case it doesn't do anything, best case it improves latency. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Wed, Jun 24, 2020 at 3:49 PM Frank R <frankaritchie@gmail.com> wrote:

fyi, there is an interesting note on disabling the write cache here:

https://yourcmc.ru/wiki/index.php?title=Ceph_performance&mobileaction=toggle_view_desktop#Drive_cache_is_slowing_you_down

On Wed, Jun 24, 2020 at 9:45 AM Benoît Knecht <bknecht@protonmail.ch> wrote:

...

Hi Igor,

Igor Fedotov wrote:

...

for the sake of completeness one more experiment please if possible:

turn off write cache for HGST drives and measure commit latency once

again.

...

I just did the same experiment with HGST drives, and disabling the write

cache

...

on those drives brought the latency down from about 7.5ms to about 4ms.

So it seems disabling the write cache across the board would be advisable in our case. Is it recommended in general, or specifically when the DB+WAL is on the same hard drive?

Stefan, Mark, are you disabling the write cache on your HDDs by default?

Cheers,

-- Ben _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Well, what I was saying was "does it hurt to unconditionally run hdparm -W 0 on all disks?" Which disk would suffer from this? I haven't seen any disk where this would be a bad idea Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Wed, Jun 24, 2020 at 5:35 PM Frank Schilder <frans@dtu.dk> wrote:

Yes, non-volatile write cache helps as described in the wiki. When you disable write cache with hdparm, it actually only disables volatile write cache. That's why SSDs with power loss protection are recommended for ceph.

A SAS/SATA SSD without any write cache will perform poorly no matter what.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Paul Emmerich <paul.emmerich@croit.io> Sent: 24 June 2020 17:30:51 To: Frank R Cc: Benoît Knecht; s.priebe@profihost.ag; ceph-users@ceph.io Subject: [ceph-users] Re: High ceph_osd_commit_latency_ms on Toshiba MG07ACA14TE HDDs

Has anyone ever encountered a drive with a write cache that actually *helped*? I haven't.

As in: would it be a good idea for the OSD to just disable the write cache on startup? Worst case it doesn't do anything, best case it improves latency.

Paul

-- Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90

On Wed, Jun 24, 2020 at 3:49 PM Frank R <frankaritchie@gmail.com> wrote:

...

fyi, there is an interesting note on disabling the write cache here:

https://yourcmc.ru/wiki/index.php?title=Ceph_performance&mobileaction=toggle_view_desktop#Drive_cache_is_slowing_you_down

...

On Wed, Jun 24, 2020 at 9:45 AM Benoît Knecht <bknecht@protonmail.ch> wrote:

...

Hi Igor,

Igor Fedotov wrote:

...

for the sake of completeness one more experiment please if possible:

turn off write cache for HGST drives and measure commit latency once

again.

...

I just did the same experiment with HGST drives, and disabling the

write

...

cache

...

on those drives brought the latency down from about 7.5ms to about 4ms.

So it seems disabling the write cache across the board would be advisable in our case. Is it recommended in general, or specifically when the DB+WAL is on the same hard drive?

Stefan, Mark, are you disabling the write cache on your HDDs by default?

Cheers,

-- Ben _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

I run the corresponding smartctl command on every drive just before OSD daemon start.

How/where did you do this?

Sorry for the spam, but I need to add this disclaimer: Although it is documented as safe to disable volatile write cache on a disk in use, I would probably not do it. The required cache flush might be erroneous in the firmware. Therefore, the method I use will not necessarily apply to OSD set-ups with WAL/DB partitions, multiple OSDs per disk and other set-ups where several daemons share the same drive. Here, more logic seems warranted. This also means that OSD daemons can probably not just do it without checking if a drive is currently in use or not. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 24 June 2020 18:00:19 To: Marc Roos; paul.emmerich Cc: bknecht; ceph-users; s.priebe Subject: [ceph-users] Re: High ceph_osd_commit_latency_ms on Toshiba MG07ACA14TE HDDs I use the stable ceph/daemon containers and introduced my own startup script for the container entrypoint. On the action "disk activate", it does a smartctl on the device argument before executing entrypoint.sh. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Marc Roos <M.Roos@f1-outsourcing.eu> Sent: 24 June 2020 17:55:35 To: Frank Schilder; paul.emmerich Cc: bknecht; ceph-users; s.priebe Subject: RE: [ceph-users] Re: High ceph_osd_commit_latency_ms on Toshiba MG07ACA14TE HDDs

I run the corresponding smartctl command on every drive just before OSD daemon start.

How/where did you do this? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

I can remember reading this before. I was hoping you maybe had some setup with systemd scripts or maybe udev.

Yeah, doing this on boot up would be ideal. I was looking really hard into tuned and other services that claimed can do it, but required plugins or other stuff did/does not exist and documentation is close to non-existent. After spending a couple of days I gave up and went with the simple script-command version. If you come across something that allows easy configuration of this at boot-time, please let me know. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Marc Roos <M.Roos@f1-outsourcing.eu> Sent: 24 June 2020 18:08:49 To: Frank Schilder; paul.emmerich Cc: bknecht; ceph-users; s.priebe Subject: RE: [ceph-users] Re: High ceph_osd_commit_latency_ms on Toshiba MG07ACA14TE HDDs

Sorry for the spam, but I need to add this disclaimer:

Although it is documented as safe to disable volatile write cache on a disk in use, I would probably not do it. The required cache flush might be erroneous in the firmware.

I can remember reading this before. I was hoping you maybe had some setup with systemd scripts or maybe udev.

a udev rule may be easier On Wed, Jun 24, 2020 at 1:17 PM <DHilsbos@performair.com> wrote:

All;

This conversation has been fascinating.

I'm throwing my hat in the ring, though I know almost nothing about systemd...

Completely non-portable, but... Couldn't you write a script to issue the necessary commands to the desired drives, then create a system unit that calls it before OSD initialization?

Thank you,

Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International, Inc. DHilsbos@PerformAir.com www.PerformAir.com

-----Original Message----- From: Frank Schilder [mailto:frans@dtu.dk] Sent: Wednesday, June 24, 2020 9:15 AM To: Marc Roos; paul.emmerich Cc: bknecht; ceph-users; s.priebe Subject: [ceph-users] Re: High ceph_osd_commit_latency_ms on Toshiba MG07ACA14TE HDDs

...

I can remember reading this before. I was hoping you maybe had some setup with systemd scripts or maybe udev.

Yeah, doing this on boot up would be ideal. I was looking really hard into tuned and other services that claimed can do it, but required plugins or other stuff did/does not exist and documentation is close to non-existent.

After spending a couple of days I gave up and went with the simple script-command version.

If you come across something that allows easy configuration of this at boot-time, please let me know.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Marc Roos <M.Roos@f1-outsourcing.eu> Sent: 24 June 2020 18:08:49 To: Frank Schilder; paul.emmerich Cc: bknecht; ceph-users; s.priebe Subject: RE: [ceph-users] Re: High ceph_osd_commit_latency_ms on Toshiba MG07ACA14TE HDDs

...

Sorry for the spam, but I need to add this disclaimer:

...

Although it is documented as safe to disable volatile write cache on a disk in use, I would probably not do it. The required cache flush might be erroneous in the firmware.

I can remember reading this before. I was hoping you maybe had some setup with systemd scripts or maybe udev.

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

...

I can remember reading this before. I was hoping you maybe had some setup with systemd scripts or maybe udev.

Yeah, doing this on boot up would be ideal. I was looking really hard into tuned and other services that claimed can do it, but required plugins or other stuff did/does not exist and documentation is close to non-existent.

After spending a couple of days I gave up and went with the simple script-command version.

If you come across something that allows easy configuration of this at boot-time, please let me know.

FWIW, doing this at boot-time only doesn’t address drives that are added/replaced without a reboot. One could simply do so for them manually before deployment, but I thought I should mention it.

I'm throwing my hat in the ring, though I know almost nothing about systemd…

In certain ways you’re fortunate ;)

Couldn't you write a script to issue the necessary commands to the desired drives, then create a system unit that calls it before OSD initialization?

The systemd unit file I think accepts an ExecStartPre entry for a command to run before firing up the daemon, eg. /var/lib/ceph/ceph-osd-prestart.sh

a udev rule may be easier

I’m going to pretend that you did *not* just drop the U-bomb in polite company ;)

Hi, https://yourcmc.ru/wiki/Ceph_performance author here %) Disabling write cache is REALLY bad for SSDs without capacitors [consumer SSDs], also it's bad for HDDs with firmwares that don't have this bug-o-feature. The bug is really common though. I have no idea where it comes from, but it's really common. When you "disable" the write cache you actually "enable" the non-volatile write cache on those drives. Seagate EXOS drives also behave like that... It seems most EXOS drives have an SSD cache even though it's not mentioned in specs. And it gets enabled when you do hdparm -W 0. In theory hdparm -W 0 may hurt linear write performance even on those HDDs, though.

Well, what I was saying was "does it hurt to unconditionally run hdparm -W 0 on all disks?"

Which disk would suffer from this? I haven't seen any disk where this would be a bad idea

Paul

Hi all,

I did a quick test with wcache off[1]. And have the impression the simple rados bench of 2 minutes performed a bit worse on my slow hdd's.

This probably depends on whether or not the drive actually has non-volatile write cache. I noticed that from many vendors you can buy the seemingly exact same drive for a difference of something like 20$. My best bet is, that the slightly more expensive ones have functioning power loss protection hardware that passed the quality test and is disabled in the cheaper drives (probably among other things). Going for the cheapest version all the time can have its price. For the disks we are using, my impression is that disabling volatile write cache actually adds the volatile cache capacity to the non-volatile write cache. The disks start consuming more power, but also perform better with ceph. For our HDDs I have never seen a degradation, fortunately - or one could say that maybe they are so crappy that it couldn't get any worse :). In case our vendor reads this, this was a practical joke :) The main question here is, do you want to risk data loss on power loss? Ceph is extremely sensitive to data that was acknowledged as "on disk" by the firmware to disappear after power outage. This is different to journaled file systems like ext4, which manage to roll back to an earlier consistent version. One looses data but the fs is not damaged. Xfs has still problems with that though. With ceph you can loose entire pools without a viable recovery option as was described earlier in this thread.

Couldn't we just set (uncomment) write_cache = off in /etc/hdparm.conf?

I was pondering with that. The problem is, that on Centos systems it seems to be ignored, in general it does not apply to SAS drives, for example, and that it has no working way of configuring which drives to exclude. For example, while for data disks for ceph we have certain minimum requirements, like functioning power loss protection, for an OS boot drive I really don't care. Power outages on cheap drives that loose writes has not been a problem since ext4. A few log entries or contents of swap - who cares. Here, performance is more important than data security on power loss. I would require a configurable option that works in the same way for all types of protocols, SATA, SAS, NVMe disks, you name it. At time of writing, I don't know of any. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Marc Roos <M.Roos@f1-outsourcing.eu> Sent: 25 June 2020 00:01:51 To: paul.emmerich; vitalif Cc: bknecht; ceph-users; s.priebe Subject: [ceph-users] Re: High ceph_osd_commit_latency_ms on Toshiba MG07ACA14TE HDDs I did a quick test with wcache off[1]. And have the impression the simple rados bench of 2 minutes performed a bit worse on my slow hdd's. [1] IFS=$'\n' && for line in `mount | grep 'osd/ceph'| awk '{print $1" "$3}'| sed -e 's/1 / /' -e 's#/var/lib/ceph/osd/ceph-##'`;do IFS=' ' arr=($line); service ceph-osd@${arr[1]} stop && smartctl -s wcache,off ${arr[0]} && service ceph-osd@${arr[1]} start ;done -----Original Message----- To: Paul Emmerich Cc: Benoît Knecht; s.priebe@profihost.ag; ceph-users@ceph.io Subject: [ceph-users] Re: High ceph_osd_commit_latency_ms on Toshiba MG07ACA14TE HDDs Hi, https://yourcmc.ru/wiki/Ceph_performance author here %) Disabling write cache is REALLY bad for SSDs without capacitors [consumer SSDs], also it's bad for HDDs with firmwares that don't have this bug-o-feature. The bug is really common though. I have no idea where it comes from, but it's really common. When you "disable" the write cache you actually "enable" the non-volatile write cache on those drives. Seagate EXOS drives also behave like that... It seems most EXOS drives have an SSD cache even though it's not mentioned in specs. And it gets enabled when you do hdparm -W 0. In theory hdparm -W 0 may hurt linear write performance even on those HDDs, though.

Well, what I was saying was "does it hurt to unconditionally run hdparm -W 0 on all disks?"

Which disk would suffer from this? I haven't seen any disk where this would be a bad idea

Paul

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Just throwing my hat in here with a small bit of anecdotal experience. In the early days of experimenting with ceph, I had 24x 8T disk, all behind RAID controllers as R0 vd's with no BBU (so controller cache is WT, default value), and pdcache (disk write cache) enabled (default value). We had a lightning strike at our previous data center that killed power, and we ended up losing the entire ceph pool (not prod), due mostly in part to the pdcache setting. We then did an exhaustive failure test following that, further isolating the pdcache as the culprit, and not the controllers write cache. The controllers now have BBU's to further prevent issues, but WB cache with the BBU did not yield issues, only pdcache. So, all of this to say, in my experience, the on-disk write cache was a huge liability for losing writes. This was also in the filestore days, and most of our issues were with XFS, but the point remains. Write cache can be a consistency killer, and I recommend disabling where possible. Reed

On Jun 24, 2020, at 10:30 AM, Paul Emmerich <paul.emmerich@croit.io> wrote:

Has anyone ever encountered a drive with a write cache that actually *helped*? I haven't.

As in: would it be a good idea for the OSD to just disable the write cache on startup? Worst case it doesn't do anything, best case it improves latency.

Paul

-- Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90

On Wed, Jun 24, 2020 at 3:49 PM Frank R <frankaritchie@gmail.com> wrote:

...

fyi, there is an interesting note on disabling the write cache here:

https://yourcmc.ru/wiki/index.php?title=Ceph_performance&mobileaction=toggle_view_desktop#Drive_cache_is_slowing_you_down

On Wed, Jun 24, 2020 at 9:45 AM Benoît Knecht <bknecht@protonmail.ch> wrote:

...

Hi Igor,

Igor Fedotov wrote:

...

for the sake of completeness one more experiment please if possible:

turn off write cache for HGST drives and measure commit latency once

again.

...

I just did the same experiment with HGST drives, and disabling the write

cache

...

on those drives brought the latency down from about 7.5ms to about 4ms.

So it seems disabling the write cache across the board would be advisable in our case. Is it recommended in general, or specifically when the DB+WAL is on the same hard drive?

Stefan, Mark, are you disabling the write cache on your HDDs by default?

Cheers,

-- Ben _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io