Hi! Micron_1100_MTFD But not only SSD "too slowly". And HDD - "too quickly". > Hi Fyodor > > Whats the model of SSD? > > Regards > > > -----Mensaje original----- > De: Fyodor Ustinov <ufm(a)ufm.su> > Enviado el: martes, 3 de septiembre de 2019 13:13 > Para: ceph-users <ceph-users(a)ceph.io> > Asunto: [ceph-users] Strange hardware behavior > > Hi! > > I understand that this question is not quite for this mailing list, but > nonetheless, experts who may be encountered this have gathered here. > > I have 24 servers, and on each, after six months of work, the following > began to happen: > > [root@S-26-5-1-2 cph]# uname -a > Linux S-26-5-1-2 5.2.11-1.el7.elrepo.x86_64 #1 SMP Thu Aug 29 08:10:52 EDT > 2019 x86_64 x86_64 x86_64 GNU/Linux > > [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 > oflag=sync > 1048576000 bytes (1.0 GB) copied, 3.76334 s, 279 MB/s > > [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 > oflag=sync > 1048576000 bytes (1.0 GB) copied, 4.54834 s, 231 MB/s > > sdc - SSD disk. sdd - HDD. > > It can be seen that ssd works somehow slowly, and hdd - too quickly. > > Reboot - nothing changes. > > And only poweroff/poweron cycle change behavior to normal: > > [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 > oflag=sync > 1048576000 bytes (1.0 GB) copied, 3.24042 s, 324 MB/s > > [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 > oflag=sync > 1048576000 bytes (1.0 GB) copied, 13.7709 s, 76.1 MB/s > > Absoluteli nothing in system and ceph log (this servers used for OSD) about > that. > > Perhaps someone has encountered similar behavior? > > WBR, > Fyodor. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email > to ceph-users-leave(a)ceph.io

Hi! Absolutely. In reality, everything happened even longer: 1. I see such strange behavior 2. Reboot 3. Nothing changes 4. I upgrade kernel from 4.20.7 to 5.2.11 and reboot 5. Nothing changes 6. power off/power on 7. Everything starts to work as it should. And in fact, paragraphs 2 to 5 are optional. Power cycle enough. > Yes indeed very funny case, are you sure sdd/sdc etc are not being > reconnected(renumbered) to different drives because of some bus reset or > other failure? Or maybe some udev rule is messing things up? > > > > -----Original Message----- > From: Fyodor Ustinov [mailto:ufm@ufm.su] > Sent: dinsdag 3 september 2019 14:06 > To: EDH - Manuel Rios Fernandez > Cc: ceph-users > Subject: [ceph-users] Re: Strange hardware behavior > > Hi! > > Micron_1100_MTFD > > But not only SSD "too slowly". And HDD - "too quickly". > >> Hi Fyodor >> >> Whats the model of SSD? >> >> Regards >> >> >> -----Mensaje original----- >> De: Fyodor Ustinov <ufm(a)ufm.su> >> Enviado el: martes, 3 de septiembre de 2019 13:13 >> Para: ceph-users <ceph-users(a)ceph.io> >> Asunto: [ceph-users] Strange hardware behavior >> >> Hi! >> >> I understand that this question is not quite for this mailing list, >> but nonetheless, experts who may be encountered this have gathered > here. >> >> I have 24 servers, and on each, after six months of work, the >> following began to happen: >> >> [root@S-26-5-1-2 cph]# uname -a >> Linux S-26-5-1-2 5.2.11-1.el7.elrepo.x86_64 #1 SMP Thu Aug 29 08:10:52 > >> EDT >> 2019 x86_64 x86_64 x86_64 GNU/Linux >> >> [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 >> oflag=sync 1048576000 bytes (1.0 GB) copied, 3.76334 s, 279 MB/s >> >> [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 >> oflag=sync 1048576000 bytes (1.0 GB) copied, 4.54834 s, 231 MB/s >> >> sdc - SSD disk. sdd - HDD. >> >> It can be seen that ssd works somehow slowly, and hdd - too quickly. >> >> Reboot - nothing changes. >> >> And only poweroff/poweron cycle change behavior to normal: >> >> [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 >> oflag=sync 1048576000 bytes (1.0 GB) copied, 3.24042 s, 324 MB/s >> >> [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 >> oflag=sync 1048576000 bytes (1.0 GB) copied, 13.7709 s, 76.1 MB/s >> >> Absoluteli nothing in system and ceph log (this servers used for OSD) >> about that. >> >> Perhaps someone has encountered similar behavior? >> >> WBR, >> Fyodor. >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an >> email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io

Hi! In this case, using dd is quite acceptable. ----- Original Message ----- > Please never use dd for disk benchmarks. > > Use fio. For linear write: > > fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M > -iodepth=32 -rw=write -runtime=60 -filename=/dev/sdX

Hi! "oflag=sync" or in some cases "direct,sync" avoid any cache usage. ----- Original Message ----- > Still, please don't :) > > At least drop_caches before testing. > >> Hi! >> >> In this case, using dd is quite acceptable. >> >> ----- Original Message ----- >>> From: vitalif(a)yourcmc.ru >>> To: "Fyodor Ustinov" <ufm(a)ufm.su> >>> Cc: "EDH - Manuel Rios Fernandez" <mriosfer(a)easydatahost.com>om>, >>> "ceph-users" <ceph-users(a)ceph.io> >>> Sent: Tuesday, 3 September, 2019 15:18:23 >>> Subject: Re: [ceph-users] Re: Strange hardware behavior >> >>> Please never use dd for disk benchmarks. >>> >>> Use fio. For linear write: >>> >>> fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M >>> -iodepth=32 -rw=write -runtime=60 -filename=/dev/sdX > > > -- > With best regards, > Vitaliy Filippov

Hi! No, this servers not have RAID controllers and all disks directly connected to motherboard. [root@S-26-6-2-3 cph]# dd if=/dev/zero of=/dev/sde bs=1M count=100000 oflag=sync status=progress 104686682112 bytes (105 GB) copied, 443.051024 s, 236 MB/s I think here is one of three options: 1. HDD actually does not write anything, but simply throws data away. But then it would be caught by the first deep scrub 2. Toshiba (all HDD in the cluster is TOSHIBA MG06ACA1) has learned to write to disk at high speed, but hides it from everyone. Due to an error in the firmware of the hard drive, the secret became apparent. That would be a better option. :) 3. The most likely option is that due to an error in Linux or the motherboard, one of the timers that Linux uses for timing starts to work incorrectly. I am talking about this with a Supermicro representative. ----- Original Message ----- > That was my thought as well.  It would be interesting to see the results > of a much longer write test (say 100GB). > > > Mark > > > On 9/3/19 9:40 AM, Fabian Niepelt wrote: >> Hey, >> >> are these drives connected to a RAID controller with a write cache? I've seen >> lots of weird behaviors with them. You said the problem persists when rebooting >> but not when power cycling which would reinforce a hardware component being the >> culprit in this case. >> >> Greetings >> Fabian >> >> Am Dienstag, den 03.09.2019, 14:13 +0300 schrieb Fyodor Ustinov: >>> Hi! >>> >>> I understand that this question is not quite for this mailing list, but >>> nonetheless, experts who may be encountered this have gathered here. >>> >>> I have 24 servers, and on each, after six months of work, the following began >>> to happen: >>> >>> [root@S-26-5-1-2 cph]# uname -a >>> Linux S-26-5-1-2 5.2.11-1.el7.elrepo.x86_64 #1 SMP Thu Aug 29 08:10:52 EDT >>> 2019 x86_64 x86_64 x86_64 GNU/Linux >>> >>> [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 oflag=sync >>> 1048576000 bytes (1.0 GB) copied, 3.76334 s, 279 MB/s >>> >>> [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 oflag=sync >>> 1048576000 bytes (1.0 GB) copied, 4.54834 s, 231 MB/s >>> >>> sdc - SSD disk. sdd - HDD. >>> >>> It can be seen that ssd works somehow slowly, and hdd - too quickly. >>> >>> Reboot - nothing changes. >>> >>> And only poweroff/poweron cycle change behavior to normal: >>> >>> [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 oflag=sync >>> 1048576000 bytes (1.0 GB) copied, 3.24042 s, 324 MB/s >>> >>> [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 oflag=sync >>> 1048576000 bytes (1.0 GB) copied, 13.7709 s, 76.1 MB/s >>> >>> Absoluteli nothing in system and ceph log (this servers used for OSD) about >>> that. >>> >>> Perhaps someone has encountered similar behavior? >>> >>> WBR, >>> Fyodor. >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io