Agreed and you may see some bays run hot when the cabling is not optimal behind the bay, reseating and re routing the bay cabling if possible may help. -----Original Message----- From: Anthony D'Atri <anthony.datri@gmail.com> Sent: Tuesday, October 15, 2024 1:48 PM To: Mat Young <mat.young@scaleflux.com> Cc: Tim Sauerbein <sauerbein@icloud.com>; ceph-users <ceph-users@ceph.io> Subject: Re: [ceph-users] SLOW_OPS problems [External: Do not click links or open attachments without verifying the sender, always login to your account directly.] Oh yeah that’s really high for a drive. Do other drives in the same / other chassis show the same temps, or is this an outlier? With Dell chassis, for example, I’ve often had to increase the iDRAC fan speed offset to get the drive temps below 40C
On Oct 15, 2024, at 1:36 PM, Mat Young <mat.young@scaleflux.com> wrote:
Looking at the smartlog seems to show 63C current temp with 53C as worst case which doesn’t make a lot of sense. Could they drive be thermally throttling?
Rgds
mat
From: Tim Sauerbein <sauerbein@icloud.com> Sent: Tuesday, October 15, 2024 11:21 AM To: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] Re: SLOW_OPS problems
[External: Do not click links or open attachments without verifying the sender, always login to your account directly.] Sorry, forgot to mention: I did a secure erase on the drive yesterday, added it to the OSD again with the same result of slow ops a few hours later. > On 15 Oct 2024, at 16:07, Tim Sauerbein <sauerbein@icloud.com<mailto:sauerbein@icloud.com>> wrote: > >> NkdkJdXPPEBannerStart Be Careful With This Message From (Tim Sauerbein <sauerbein@icloud.com>)<https://godaddy.cloud-protect.net/email-detail s/?k=k1&payload=53616c7465645f5f15db257bb6f7446517f83bae6d9f7752b417bd 2af5cd0f818688bd2dbc09ea4d63ff123db727aac7a26336b0ac38030b4d5f6578b60f 00f399cecc7c56cdf55ccf24a1fbdf14dd1574c17a4c300de8705b37b4ef25d11fe41c e0f9fdb7e8228a29a4207e9be5cfb2fd78296edffa23c172a5cf397b0ff766a2178829 7658e61718f53d2a445d8056650a2d047cf31eeb08d1ff50f3ec7363971db9f2a6809e 803c3678894306df1b57d1d2463235136b2beacce4e62ccdca0169afeef74aea8b0af6 16a0d7c44fc6c0b69d24c4211ba1c98a19ac640aafae16cb463be63f93ad06a2c67696 d8bcba> Learn More<https://godaddy.cloud-protect.net/email-details/?k=k1&payload=536 16c7465645f5f15db257bb6f7446517f83bae6d9f7752b417bd2af5cd0f818688bd2db c09ea4d63ff123db727aac7a26336b0ac38030b4d5f6578b60f00f399cecc7c56cdf55 ccf24a1fbdf14dd1574c17a4c300de8705b37b4ef25d11fe41ce0f9fdb7e8228a29a42 07e9be5cfb2fd78296edffa23c172a5cf397b0ff766a21788297658e61718f53d2a445 d8056650a2d047cf31eeb08d1ff50f3ec7363971db9f2a6809e803c3678894306df1b5 7d1d2463235136b2beacce4e62ccdca0169afeef74aea8b0af616a0d7c44fc6c0b69d2 4c4211ba1c98a19ac640aafae16cb463be63f93ad06a2c67696d8bcba> Potential Impersonation The sender's identity could not be verified and someone may be impersonating the sender. Take caution when interacting with this message.
NkdkJdXPPEBannerEnd
Sorry, forgot to mention:
I did a secure erase on the drive yesterday, added it to the OSD again with the same result of slow ops a few hours later.
On 15 Oct 2024, at 16:07, Tim Sauerbein <sauerbein@icloud.com<mailto:sauerbein@icloud.com>> wrote:
On 14 Oct 2024, at 16:01, Anthony D'Atri <aad@dreamsnake.net<mailto:aad@dreamsnake.net>> wrote:
Remind me, have you sent me a full `smartctl -a` output for this drive?
See here, looks good though: https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_ sauerbein_6423231adb954d28c8c82a8422256355&d=DwIGaQ&c=euGZstcaTDllvim EN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=bDvyvkr2bb4BZTlvrJx55ZzKgQTyuyI1pgJpjo 3ippU&m=4BzydYt8R0yplK8QvqowJz2GaV9Lnf7dg9Xos6bDeNBYcAyuL4faGr3ma7qRj CKc&s=VKfARgTFDlE0uDcv7zs4xjZ6--dLJqws6_O877VPbyw&e=
If there’s a firmware update available, updating it with a subsequent secure-erase could plausibly recover it.
I don't think there is a firmware update publicly available. Other disks of same model and same firmware run without issues in my cluster btw.
On 14 Oct 2024, at 15:56, Mark Nelson <https://urldefense.proofpoint.com/v2/url?u=http-3A__mark.nelson-40clyso.com&d=DwIGaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=bDvyvkr2bb4BZTlvrJx55ZzKgQTyuyI1pgJpjo3ippU&m=4BzydYt8R0yplK8QvqowJz2GaV9Lnf7dg9Xos6bDeNBYcAyuL4faGr3ma7qRjCKc&s=59RCI_vd1KgnEwQGO9-paAVPPm3884F9Oq_hMho-S94&e=> wrote:
I've seen similar issues before where smart showed no failures but the drive performed terribly. You can try trimming the drive or even doing a secure format to see if it helps, but at least in the case I recall it was an issue with the drive itself.
I think that the disk is just faulty too. Do you have any idea of a test to run on the SSD to prove that independent of Ceph?
Thanks,
Tim
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
----------
This email has been scanned for spam and viruses. Visit the following link to report this email as spam:
https://godaddy.cloud-protect.net/app/report_spam.php?mod_id=11&mod_op tion=logitem&report=1&type=easyspam&k=k1&payload=53616c7465645f5f99858 a10fb5d88069d46e0a76bffcf608852edf1729c8dcab288411ea1dcf982c558f4d477a db88e6d258294685049d6531aa93e1b8c603e461cc0af17dbd4010838586edf64d1e3d 67b6155f1c08399ef0fbc70bf62abaf9580c00d306da64edb207e0b95ba9646aff218c 3a846470698aff34e3499a4dc272c128392c11d8a608369425815e393a3329a6a2a60a 1d16dad9704ea168caf _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
---------- This email has been scanned for spam and viruses. Visit the following link to report this email as spam: https://godaddy.cloud-protect.net/app/report_spam.php?mod_id=11&mod_option=logitem&report=1&type=easyspam&k=k1&payload=53616c7465645f5f08d8140a2e039990a3d9cd5ed578f9f6efa11f08fb971724916154b6336fa63a939c80b209cbef3c467d81bce4e1001de2128f88deff6d20da7e71ea745478bb4f2073cb6b656f85f1d8ff0fadab1544c1cc7e06ba007f3a80ff407b613a346d8be93804ed1053a5716624ea1ca5fc23c9aed410e3245fdd5ee523d381cf6d170adf55edaa55d3752836d6f71ebab9a1904e885b54827ea6