Strange hardware behavior
Hi! I understand that this question is not quite for this mailing list, but nonetheless, experts who may be encountered this have gathered here. I have 24 servers, and on each, after six months of work, the following began to happen: [root@S-26-5-1-2 cph]# uname -a Linux S-26-5-1-2 5.2.11-1.el7.elrepo.x86_64 #1 SMP Thu Aug 29 08:10:52 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 3.76334 s, 279 MB/s [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 4.54834 s, 231 MB/s sdc - SSD disk. sdd - HDD. It can be seen that ssd works somehow slowly, and hdd - too quickly. Reboot - nothing changes. And only poweroff/poweron cycle change behavior to normal: [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 3.24042 s, 324 MB/s [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 13.7709 s, 76.1 MB/s Absoluteli nothing in system and ceph log (this servers used for OSD) about that. Perhaps someone has encountered similar behavior? WBR, Fyodor.
Hi Fyodor Whats the model of SSD? Regards -----Mensaje original----- De: Fyodor Ustinov <ufm@ufm.su> Enviado el: martes, 3 de septiembre de 2019 13:13 Para: ceph-users <ceph-users@ceph.io> Asunto: [ceph-users] Strange hardware behavior Hi! I understand that this question is not quite for this mailing list, but nonetheless, experts who may be encountered this have gathered here. I have 24 servers, and on each, after six months of work, the following began to happen: [root@S-26-5-1-2 cph]# uname -a Linux S-26-5-1-2 5.2.11-1.el7.elrepo.x86_64 #1 SMP Thu Aug 29 08:10:52 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 3.76334 s, 279 MB/s [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 4.54834 s, 231 MB/s sdc - SSD disk. sdd - HDD. It can be seen that ssd works somehow slowly, and hdd - too quickly. Reboot - nothing changes. And only poweroff/poweron cycle change behavior to normal: [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 3.24042 s, 324 MB/s [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 13.7709 s, 76.1 MB/s Absoluteli nothing in system and ceph log (this servers used for OSD) about that. Perhaps someone has encountered similar behavior? WBR, Fyodor. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi! Micron_1100_MTFD But not only SSD "too slowly". And HDD - "too quickly".
Hi Fyodor
Whats the model of SSD?
Regards
-----Mensaje original----- De: Fyodor Ustinov <ufm@ufm.su> Enviado el: martes, 3 de septiembre de 2019 13:13 Para: ceph-users <ceph-users@ceph.io> Asunto: [ceph-users] Strange hardware behavior
Hi!
I understand that this question is not quite for this mailing list, but nonetheless, experts who may be encountered this have gathered here.
I have 24 servers, and on each, after six months of work, the following began to happen:
[root@S-26-5-1-2 cph]# uname -a Linux S-26-5-1-2 5.2.11-1.el7.elrepo.x86_64 #1 SMP Thu Aug 29 08:10:52 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 3.76334 s, 279 MB/s
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 4.54834 s, 231 MB/s
sdc - SSD disk. sdd - HDD.
It can be seen that ssd works somehow slowly, and hdd - too quickly.
Reboot - nothing changes.
And only poweroff/poweron cycle change behavior to normal:
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 3.24042 s, 324 MB/s
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 13.7709 s, 76.1 MB/s
Absoluteli nothing in system and ceph log (this servers used for OSD) about that.
Perhaps someone has encountered similar behavior?
WBR, Fyodor. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yes indeed very funny case, are you sure sdd/sdc etc are not being reconnected(renumbered) to different drives because of some bus reset or other failure? Or maybe some udev rule is messing things up? -----Original Message----- From: Fyodor Ustinov [mailto:ufm@ufm.su] Sent: dinsdag 3 september 2019 14:06 To: EDH - Manuel Rios Fernandez Cc: ceph-users Subject: [ceph-users] Re: Strange hardware behavior Hi! Micron_1100_MTFD But not only SSD "too slowly". And HDD - "too quickly".
Hi Fyodor
Whats the model of SSD?
Regards
-----Mensaje original----- De: Fyodor Ustinov <ufm@ufm.su> Enviado el: martes, 3 de septiembre de 2019 13:13 Para: ceph-users <ceph-users@ceph.io> Asunto: [ceph-users] Strange hardware behavior
Hi!
I understand that this question is not quite for this mailing list, but nonetheless, experts who may be encountered this have gathered here.
I have 24 servers, and on each, after six months of work, the following began to happen:
[root@S-26-5-1-2 cph]# uname -a Linux S-26-5-1-2 5.2.11-1.el7.elrepo.x86_64 #1 SMP Thu Aug 29 08:10:52
EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 3.76334 s, 279 MB/s
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 4.54834 s, 231 MB/s
sdc - SSD disk. sdd - HDD.
It can be seen that ssd works somehow slowly, and hdd - too quickly.
Reboot - nothing changes.
And only poweroff/poweron cycle change behavior to normal:
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 3.24042 s, 324 MB/s
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 13.7709 s, 76.1 MB/s
Absoluteli nothing in system and ceph log (this servers used for OSD) about that.
Perhaps someone has encountered similar behavior?
WBR, Fyodor. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi! Absolutely. In reality, everything happened even longer: 1. I see such strange behavior 2. Reboot 3. Nothing changes 4. I upgrade kernel from 4.20.7 to 5.2.11 and reboot 5. Nothing changes 6. power off/power on 7. Everything starts to work as it should. And in fact, paragraphs 2 to 5 are optional. Power cycle enough.
Yes indeed very funny case, are you sure sdd/sdc etc are not being reconnected(renumbered) to different drives because of some bus reset or other failure? Or maybe some udev rule is messing things up?
-----Original Message----- From: Fyodor Ustinov [mailto:ufm@ufm.su] Sent: dinsdag 3 september 2019 14:06 To: EDH - Manuel Rios Fernandez Cc: ceph-users Subject: [ceph-users] Re: Strange hardware behavior
Hi!
Micron_1100_MTFD
But not only SSD "too slowly". And HDD - "too quickly".
Hi Fyodor
Whats the model of SSD?
Regards
-----Mensaje original----- De: Fyodor Ustinov <ufm@ufm.su> Enviado el: martes, 3 de septiembre de 2019 13:13 Para: ceph-users <ceph-users@ceph.io> Asunto: [ceph-users] Strange hardware behavior
Hi!
I understand that this question is not quite for this mailing list, but nonetheless, experts who may be encountered this have gathered here.
I have 24 servers, and on each, after six months of work, the following began to happen:
[root@S-26-5-1-2 cph]# uname -a Linux S-26-5-1-2 5.2.11-1.el7.elrepo.x86_64 #1 SMP Thu Aug 29 08:10:52
EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 3.76334 s, 279 MB/s
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 4.54834 s, 231 MB/s
sdc - SSD disk. sdd - HDD.
It can be seen that ssd works somehow slowly, and hdd - too quickly.
Reboot - nothing changes.
And only poweroff/poweron cycle change behavior to normal:
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 3.24042 s, 324 MB/s
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 13.7709 s, 76.1 MB/s
Absoluteli nothing in system and ceph log (this servers used for OSD) about that.
Perhaps someone has encountered similar behavior?
WBR, Fyodor. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Please never use dd for disk benchmarks. Use fio. For linear write: fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -iodepth=32 -rw=write -runtime=60 -filename=/dev/sdX
Hi! In this case, using dd is quite acceptable. ----- Original Message -----
From: vitalif@yourcmc.ru To: "Fyodor Ustinov" <ufm@ufm.su> Cc: "EDH - Manuel Rios Fernandez" <mriosfer@easydatahost.com>, "ceph-users" <ceph-users@ceph.io> Sent: Tuesday, 3 September, 2019 15:18:23 Subject: Re: [ceph-users] Re: Strange hardware behavior
Please never use dd for disk benchmarks.
Use fio. For linear write:
fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -iodepth=32 -rw=write -runtime=60 -filename=/dev/sdX
Still, please don't :) At least drop_caches before testing.
Hi!
In this case, using dd is quite acceptable.
----- Original Message -----
From: vitalif@yourcmc.ru To: "Fyodor Ustinov" <ufm@ufm.su> Cc: "EDH - Manuel Rios Fernandez" <mriosfer@easydatahost.com>, "ceph-users" <ceph-users@ceph.io> Sent: Tuesday, 3 September, 2019 15:18:23 Subject: Re: [ceph-users] Re: Strange hardware behavior
Please never use dd for disk benchmarks.
Use fio. For linear write:
fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -iodepth=32 -rw=write -runtime=60 -filename=/dev/sdX
-- With best regards, Vitaliy Filippov
Hi! "oflag=sync" or in some cases "direct,sync" avoid any cache usage. ----- Original Message -----
From: "Vitaliy Filippov" <vitalif@yourcmc.ru> To: "Fyodor Ustinov" <ufm@ufm.su> Cc: "EDH - Manuel Rios Fernandez" <mriosfer@easydatahost.com>, "ceph-users" <ceph-users@ceph.io> Sent: Wednesday, 4 September, 2019 13:38:44 Subject: Re: [ceph-users] Re: Strange hardware behavior
Still, please don't :)
At least drop_caches before testing.
Hi!
In this case, using dd is quite acceptable.
----- Original Message -----
From: vitalif@yourcmc.ru To: "Fyodor Ustinov" <ufm@ufm.su> Cc: "EDH - Manuel Rios Fernandez" <mriosfer@easydatahost.com>, "ceph-users" <ceph-users@ceph.io> Sent: Tuesday, 3 September, 2019 15:18:23 Subject: Re: [ceph-users] Re: Strange hardware behavior
Please never use dd for disk benchmarks.
Use fio. For linear write:
fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -iodepth=32 -rw=write -runtime=60 -filename=/dev/sdX
-- With best regards, Vitaliy Filippov
I understand, but I still don't trust dd and I still insist that you retest it with fio when you observe strange behaviour again :-) Either it's MAGIC or "one of the turtles is bullshitting", then you just need to find that turtle. -- With best regards, Vitaliy Filippov
Hey, are these drives connected to a RAID controller with a write cache? I've seen lots of weird behaviors with them. You said the problem persists when rebooting but not when power cycling which would reinforce a hardware component being the culprit in this case. Greetings Fabian Am Dienstag, den 03.09.2019, 14:13 +0300 schrieb Fyodor Ustinov:
Hi!
I understand that this question is not quite for this mailing list, but nonetheless, experts who may be encountered this have gathered here.
I have 24 servers, and on each, after six months of work, the following began to happen:
[root@S-26-5-1-2 cph]# uname -a Linux S-26-5-1-2 5.2.11-1.el7.elrepo.x86_64 #1 SMP Thu Aug 29 08:10:52 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 3.76334 s, 279 MB/s
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 4.54834 s, 231 MB/s
sdc - SSD disk. sdd - HDD.
It can be seen that ssd works somehow slowly, and hdd - too quickly.
Reboot - nothing changes.
And only poweroff/poweron cycle change behavior to normal:
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 3.24042 s, 324 MB/s
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 13.7709 s, 76.1 MB/s
Absoluteli nothing in system and ceph log (this servers used for OSD) about that.
Perhaps someone has encountered similar behavior?
WBR, Fyodor. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
That was my thought as well. It would be interesting to see the results of a much longer write test (say 100GB). Mark On 9/3/19 9:40 AM, Fabian Niepelt wrote:
Hey,
are these drives connected to a RAID controller with a write cache? I've seen lots of weird behaviors with them. You said the problem persists when rebooting but not when power cycling which would reinforce a hardware component being the culprit in this case.
Greetings Fabian
Am Dienstag, den 03.09.2019, 14:13 +0300 schrieb Fyodor Ustinov:
Hi!
I understand that this question is not quite for this mailing list, but nonetheless, experts who may be encountered this have gathered here.
I have 24 servers, and on each, after six months of work, the following began to happen:
[root@S-26-5-1-2 cph]# uname -a Linux S-26-5-1-2 5.2.11-1.el7.elrepo.x86_64 #1 SMP Thu Aug 29 08:10:52 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 3.76334 s, 279 MB/s
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 4.54834 s, 231 MB/s
sdc - SSD disk. sdd - HDD.
It can be seen that ssd works somehow slowly, and hdd - too quickly.
Reboot - nothing changes.
And only poweroff/poweron cycle change behavior to normal:
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 3.24042 s, 324 MB/s
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 13.7709 s, 76.1 MB/s
Absoluteli nothing in system and ceph log (this servers used for OSD) about that.
Perhaps someone has encountered similar behavior?
WBR, Fyodor. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi! No, this servers not have RAID controllers and all disks directly connected to motherboard. [root@S-26-6-2-3 cph]# dd if=/dev/zero of=/dev/sde bs=1M count=100000 oflag=sync status=progress 104686682112 bytes (105 GB) copied, 443.051024 s, 236 MB/s I think here is one of three options: 1. HDD actually does not write anything, but simply throws data away. But then it would be caught by the first deep scrub 2. Toshiba (all HDD in the cluster is TOSHIBA MG06ACA1) has learned to write to disk at high speed, but hides it from everyone. Due to an error in the firmware of the hard drive, the secret became apparent. That would be a better option. :) 3. The most likely option is that due to an error in Linux or the motherboard, one of the timers that Linux uses for timing starts to work incorrectly. I am talking about this with a Supermicro representative. ----- Original Message -----
From: "Mark Nelson" <mnelson@redhat.com> To: ceph-users@ceph.io Sent: Tuesday, 3 September, 2019 17:48:18 Subject: [ceph-users] Re: Strange hardware behavior
That was my thought as well. It would be interesting to see the results of a much longer write test (say 100GB).
Mark
On 9/3/19 9:40 AM, Fabian Niepelt wrote:
Hey,
are these drives connected to a RAID controller with a write cache? I've seen lots of weird behaviors with them. You said the problem persists when rebooting but not when power cycling which would reinforce a hardware component being the culprit in this case.
Greetings Fabian
Am Dienstag, den 03.09.2019, 14:13 +0300 schrieb Fyodor Ustinov:
Hi!
I understand that this question is not quite for this mailing list, but nonetheless, experts who may be encountered this have gathered here.
I have 24 servers, and on each, after six months of work, the following began to happen:
[root@S-26-5-1-2 cph]# uname -a Linux S-26-5-1-2 5.2.11-1.el7.elrepo.x86_64 #1 SMP Thu Aug 29 08:10:52 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 3.76334 s, 279 MB/s
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 4.54834 s, 231 MB/s
sdc - SSD disk. sdd - HDD.
It can be seen that ssd works somehow slowly, and hdd - too quickly.
Reboot - nothing changes.
And only poweroff/poweron cycle change behavior to normal:
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 3.24042 s, 324 MB/s
[root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 13.7709 s, 76.1 MB/s
Absoluteli nothing in system and ceph log (this servers used for OSD) about that.
Perhaps someone has encountered similar behavior?
WBR, Fyodor. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (8)
-
EDH - Manuel Rios Fernandez
-
Fabian Niepelt
-
Fyodor Ustinov
-
Marc Roos
-
Mark Nelson
-
vitalif＠yourcmc.ru
-
Vitaliy Filippov
-
Виталий Филиппов