Dear list, I have a small cluster (Reef 18.2.4) with 7 hosts and 3-4 OSDs each (960GB/1.92TB mixed Intel D3-S4610, Samsung SM883, PM897 SSDs): cluster: id: ecff3ce8-539b-443e-a492-da428f4aa9e9 health: HEALTH_OK services: mon: 5 daemons, quorum titan,mangan,kalium,argon,chromium (age 2w) mgr: mangan(active, since 2w), standbys: titan, argon osd: 22 osds: 22 up (since 2w), 22 in (since 3M) data: pools: 2 pools, 513 pgs objects: 2.76M objects, 7.0 TiB usage: 16 TiB used, 15 TiB / 31 TiB avail pgs: 513 active+clean On that cluster RBD volumes for virtual machines are stored. For a couple of months now the cluster reports slow ops for some OSDs and some PGs as laggy. This happens once or twice a day, sometimes more and sometimes not at all for a few days, at completely random times, independent of when snapshots are deleted and trimmed and independent of the I/O load or load on the hosts. After about 30 seconds, during which the write speed goes to zero on the VMs, everything returns to normal. I cannot reproduce the slow ops manually by creating write load on the cluster. Even writing continuously with 300-400 MB/s full speed for 20 minutes does not create any problems. See attached log file for an example of a typical occurrence. I have also measured write load on the disks during the problems with iostat which just shows how writes stall, see also attached. The OSDs with slow ops are completely random, any of the disks would show up once in a while. Current config (I've tried optimising snaptrim and scrub which didn't help): # ceph config dump WHO MASK LEVEL OPTION VALUE RO global advanced auth_client_required cephx * global advanced auth_cluster_required cephx * global advanced auth_service_required cephx * global advanced bdev_async_discard true global advanced bdev_enable_discard true global advanced public_network 10.0.4.0/24 * mon advanced auth_allow_insecure_global_id_reclaim false mgr advanced mgr/balancer/active true mgr advanced mgr/balancer/mode upmap mgr unknown mgr/pg_autoscaler/autoscale_profile scale-up * osd basic osd_memory_target 4294967296 osd advanced osd_pg_max_concurrent_snap_trims 1 osd advanced osd_scrub_begin_hour 23 osd advanced osd_scrub_end_hour 4 osd advanced osd_scrub_sleep 1.000000 osd advanced osd_snap_trim_priority 1 osd advanced osd_snap_trim_sleep 2.000000 osd.0 basic osd_mclock_max_capacity_iops_ssd 29199.674019 osd.1 basic osd_mclock_max_capacity_iops_ssd 31554.530141 osd.10 basic osd_mclock_max_capacity_iops_ssd 25949.821194 osd.11 basic osd_mclock_max_capacity_iops_ssd 26300.596265 osd.12 basic osd_mclock_max_capacity_iops_ssd 25167.331294 osd.13 basic osd_mclock_max_capacity_iops_ssd 21606.610828 osd.14 basic osd_mclock_max_capacity_iops_ssd 27894.095121 osd.15 basic osd_mclock_max_capacity_iops_ssd 25929.047047 osd.16 basic osd_mclock_max_capacity_iops_ssd 15423.600235 osd.17 basic osd_mclock_max_capacity_iops_ssd 25097.493934 osd.18 basic osd_mclock_max_capacity_iops_ssd 25966.188007 osd.19 basic osd_mclock_max_capacity_iops_ssd 23628.746459 osd.2 basic osd_mclock_max_capacity_iops_ssd 32157.280832 osd.20 basic osd_mclock_max_capacity_iops_ssd 22722.682745 osd.3 basic osd_mclock_max_capacity_iops_ssd 33951.086556 osd.4 basic osd_mclock_max_capacity_iops_ssd 22736.907664 osd.5 basic osd_mclock_max_capacity_iops_ssd 21916.777510 osd.6 basic osd_mclock_max_capacity_iops_ssd 29984.954749 osd.7 basic osd_mclock_max_capacity_iops_ssd 26757.965797 osd.8 basic osd_mclock_max_capacity_iops_ssd 22738.921429 osd.9 basic osd_mclock_max_capacity_iops_ssd 24635.156413 Any help would be much appreciated! Thanks, Tim
Hi Tim, do you see the behaviour across all devices or does it only affect one type/manufacturer? Joachim www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Tim Sauerbein <sauerbein@icloud.com> schrieb am So., 29. Sept. 2024, 23:32:
Dear list,
I have a small cluster (Reef 18.2.4) with 7 hosts and 3-4 OSDs each (960GB/1.92TB mixed Intel D3-S4610, Samsung SM883, PM897 SSDs):
cluster: id: ecff3ce8-539b-443e-a492-da428f4aa9e9 health: HEALTH_OK
services: mon: 5 daemons, quorum titan,mangan,kalium,argon,chromium (age 2w) mgr: mangan(active, since 2w), standbys: titan, argon osd: 22 osds: 22 up (since 2w), 22 in (since 3M)
data: pools: 2 pools, 513 pgs objects: 2.76M objects, 7.0 TiB usage: 16 TiB used, 15 TiB / 31 TiB avail pgs: 513 active+clean
On that cluster RBD volumes for virtual machines are stored.
For a couple of months now the cluster reports slow ops for some OSDs and some PGs as laggy. This happens once or twice a day, sometimes more and sometimes not at all for a few days, at completely random times, independent of when snapshots are deleted and trimmed and independent of the I/O load or load on the hosts.
After about 30 seconds, during which the write speed goes to zero on the VMs, everything returns to normal. I cannot reproduce the slow ops manually by creating write load on the cluster. Even writing continuously with 300-400 MB/s full speed for 20 minutes does not create any problems.
See attached log file for an example of a typical occurrence. I have also measured write load on the disks during the problems with iostat which just shows how writes stall, see also attached.
The OSDs with slow ops are completely random, any of the disks would show up once in a while.
Current config (I've tried optimising snaptrim and scrub which didn't help):
# ceph config dump WHO MASK LEVEL OPTION VALUE RO global advanced auth_client_required cephx * global advanced auth_cluster_required cephx * global advanced auth_service_required cephx * global advanced bdev_async_discard true global advanced bdev_enable_discard true global advanced public_network 10.0.4.0/24 * mon advanced auth_allow_insecure_global_id_reclaim false mgr advanced mgr/balancer/active true mgr advanced mgr/balancer/mode upmap mgr unknown mgr/pg_autoscaler/autoscale_profile scale-up * osd basic osd_memory_target 4294967296 osd advanced osd_pg_max_concurrent_snap_trims 1 osd advanced osd_scrub_begin_hour 23 osd advanced osd_scrub_end_hour 4 osd advanced osd_scrub_sleep 1.000000 osd advanced osd_snap_trim_priority 1 osd advanced osd_snap_trim_sleep 2.000000 osd.0 basic osd_mclock_max_capacity_iops_ssd 29199.674019 osd.1 basic osd_mclock_max_capacity_iops_ssd 31554.530141 osd.10 basic osd_mclock_max_capacity_iops_ssd 25949.821194 osd.11 basic osd_mclock_max_capacity_iops_ssd 26300.596265 osd.12 basic osd_mclock_max_capacity_iops_ssd 25167.331294 osd.13 basic osd_mclock_max_capacity_iops_ssd 21606.610828 osd.14 basic osd_mclock_max_capacity_iops_ssd 27894.095121 osd.15 basic osd_mclock_max_capacity_iops_ssd 25929.047047 osd.16 basic osd_mclock_max_capacity_iops_ssd 15423.600235 osd.17 basic osd_mclock_max_capacity_iops_ssd 25097.493934 osd.18 basic osd_mclock_max_capacity_iops_ssd 25966.188007 osd.19 basic osd_mclock_max_capacity_iops_ssd 23628.746459 osd.2 basic osd_mclock_max_capacity_iops_ssd 32157.280832 osd.20 basic osd_mclock_max_capacity_iops_ssd 22722.682745 osd.3 basic osd_mclock_max_capacity_iops_ssd 33951.086556 osd.4 basic osd_mclock_max_capacity_iops_ssd 22736.907664 osd.5 basic osd_mclock_max_capacity_iops_ssd 21916.777510 osd.6 basic osd_mclock_max_capacity_iops_ssd 29984.954749 osd.7 basic osd_mclock_max_capacity_iops_ssd 26757.965797 osd.8 basic osd_mclock_max_capacity_iops_ssd 22738.921429 osd.9 basic osd_mclock_max_capacity_iops_ssd 24635.156413
Any help would be much appreciated!
Thanks, Tim
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 30 Sep 2024, at 06:23, Joachim Kraftmayer <joachim.kraftmayer@clyso.com> wrote:
do you see the behaviour across all devices or does it only affect one type/manufacturer?
All devices are affected equally, every time one or two random ODSs report slow ops. So I don't think the SSDs are to blame. Thanks, Tim
Remember that slow ops are a top of the iceberg thing, you only see ones that crest above 30s
On Sep 30, 2024, at 6:06 AM, Tim Sauerbein <sauerbein@icloud.com> wrote:
On 30 Sep 2024, at 06:23, Joachim Kraftmayer <joachim.kraftmayer@clyso.com> wrote:
do you see the behaviour across all devices or does it only affect one type/manufacturer?
All devices are affected equally, every time one or two random ODSs report slow ops. So I don't think the SSDs are to blame.
Thanks, Tim _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for the replies everyone!
On 30 Sep 2024, at 13:10, Anthony D'Atri <aad@dreamsnake.net> wrote:
Remember that slow ops are a top of the iceberg thing, you only see ones that crest above 30s
So far metrics of the hosted VMs show no other I/O slowdown except when these hiccups occur.
On 30 Sep 2024, at 13:35, Igor Fedotov <igor.fedotov@croit.io> wrote:
there is no log attached to your post, you better share it via some other means.
BTW - what log did you mean - monitor or OSD one?
It would be nice to have logs for a couple of OSDs suffering from slow ops, preferably relevant to two different cases.
Sorry, the attachments have apparently been stripped. See here for one incident (they all look the same but I can share more if relevant) monitor log, affected osd logs, iostat log: https://gist.github.com/sauerbein/5a485a6d2546475912709743e3cfbf4b Let me know if you need any other logs to analyse!
On 30 Sep 2024, at 14:34, Alexander Schreiber <als@thangorodrim.ch> wrote:
One cause for "slow ops" I discovered are networking issues. I had slow ops across my entire cluster (interconnected with 10G). Turns out the switch was bad an achieved < 10 MBit/s on one of the 10G links. Replaced the switch, tested the links again - got full 10G connectivity and the slow ops disappeared.
Thanks for the idea. The hosts are connected to two switches with fail-over bonding, normally communicating via the same switch. I will move them all over to the second switch to rule out a switch issue. Best regards, Tim
My point is that you may have more 10-30s delays that aren’t surfaced.
On Sep 30, 2024, at 10:17 AM, Tim Sauerbein <sauerbein@icloud.com> wrote:
Thanks for the replies everyone!
On 30 Sep 2024, at 13:10, Anthony D'Atri <aad@dreamsnake.net> wrote:
Remember that slow ops are a top of the iceberg thing, you only see ones that crest above 30s
So far metrics of the hosted VMs show no other I/O slowdown except when these hiccups occur.
On 30 Sep 2024, at 13:35, Igor Fedotov <igor.fedotov@croit.io> wrote:
there is no log attached to your post, you better share it via some other means.
BTW - what log did you mean - monitor or OSD one?
It would be nice to have logs for a couple of OSDs suffering from slow ops, preferably relevant to two different cases.
Sorry, the attachments have apparently been stripped. See here for one incident (they all look the same but I can share more if relevant) monitor log, affected osd logs, iostat log:
https://gist.github.com/sauerbein/5a485a6d2546475912709743e3cfbf4b
Let me know if you need any other logs to analyse!
On 30 Sep 2024, at 14:34, Alexander Schreiber <als@thangorodrim.ch> wrote:
One cause for "slow ops" I discovered are networking issues. I had slow ops across my entire cluster (interconnected with 10G). Turns out the switch was bad an achieved < 10 MBit/s on one of the 10G links. Replaced the switch, tested the links again - got full 10G connectivity and the slow ops disappeared.
Thanks for the idea. The hosts are connected to two switches with fail-over bonding, normally communicating via the same switch. I will move them all over to the second switch to rule out a switch issue.
Best regards, Tim _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Tim, first of all - given the provided logs - all the slow operastions are stuck in 'waiting for sub ops' state. Which apparently means that reported OSDs aren't suffering from local issues but stuck on replication operations to their peer OSDs. From my experince even a single "faulty" osd could cause such issues to multiple other daemons. And the way to troubleshoot is to find out what are the actual culprit OSD(s). To do that one might try to use the following approach: 1. When (or shortly after) the issue is happening - run 'ceph daemon osd.N dump_historic_ops' (or even 'dump_ops_in_flight') command against OSDs reporting slow operations. 2. From the above reports choose operations with extraordinary high duration, e.g. > 5 seconds and learn PG ids they've been run against, e.g. PG = 1.a in the following sample: "description": "osd_op(client.24184.0:23 >>>>1.a<<<<< 1:54253539:::benchmark_data_coalmon_70932_object22:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 3. For affected PG(s) learn which OSDs are backing specific it. E.g. by running ceph pg map <pgid> 4. If different PGs from the above step use specific OSD which is common to all (the majority) of them - higly likely it's a good candidate for additional investigation - partcularly relevant OSD logs inspection. Thanks, Igor On 9/30/2024 5:14 PM, Tim Sauerbein wrote:
Thanks for the replies everyone!
On 30 Sep 2024, at 13:10, Anthony D'Atri <aad@dreamsnake.net> wrote:
Remember that slow ops are a top of the iceberg thing, you only see ones that crest above 30s So far metrics of the hosted VMs show no other I/O slowdown except when these hiccups occur.
On 30 Sep 2024, at 13:35, Igor Fedotov <igor.fedotov@croit.io> wrote:
there is no log attached to your post, you better share it via some other means.
BTW - what log did you mean - monitor or OSD one?
It would be nice to have logs for a couple of OSDs suffering from slow ops, preferably relevant to two different cases.
Sorry, the attachments have apparently been stripped. See here for one incident (they all look the same but I can share more if relevant) monitor log, affected osd logs, iostat log:
https://gist.github.com/sauerbein/5a485a6d2546475912709743e3cfbf4b
Let me know if you need any other logs to analyse!
On 30 Sep 2024, at 14:34, Alexander Schreiber <als@thangorodrim.ch> wrote:
One cause for "slow ops" I discovered are networking issues. I had slow ops across my entire cluster (interconnected with 10G). Turns out the switch was bad an achieved < 10 MBit/s on one of the 10G links. Replaced the switch, tested the links again - got full 10G connectivity and the slow ops disappeared. Thanks for the idea. The hosts are connected to two switches with fail-over bonding, normally communicating via the same switch. I will move them all over to the second switch to rule out a switch issue.
Best regards, Tim _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Igor Fedotov Ceph Lead Developer Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
Hi Igor, Thanks for the valuable advice! I just wanted to provide feedback that it was indeed one single OSD causing the issues which I could triangulate as you said. After removing this OSD, the slow ops haven't occurred anymore. Best regards, Tim
On 1 Oct 2024, at 12:42, Igor Fedotov <igor.fedotov@croit.io> wrote:
Hi Tim,
first of all - given the provided logs - all the slow operastions are stuck in 'waiting for sub ops' state.
Which apparently means that reported OSDs aren't suffering from local issues but stuck on replication operations to their peer OSDs.
From my experince even a single "faulty" osd could cause such issues to multiple other daemons. And the way to troubleshoot is to find out what are the actual culprit OSD(s).
To do that one might try to use the following approach:
1. When (or shortly after) the issue is happening - run 'ceph daemon osd.N dump_historic_ops' (or even 'dump_ops_in_flight') command against OSDs reporting slow operations.
2. From the above reports choose operations with extraordinary high duration, e.g. > 5 seconds and learn PG ids they've been run against, e.g. PG = 1.a in the following sample:
"description": "osd_op(client.24184.0:23 >>>>1.a<<<<< 1:54253539:::benchmark_data_coalmon_70932_object22:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)",
3. For affected PG(s) learn which OSDs are backing specific it. E.g. by running ceph pg map <pgid>
4. If different PGs from the above step use specific OSD which is common to all (the majority) of them - higly likely it's a good candidate for additional investigation - partcularly relevant OSD logs inspection.
Thanks,
Igor
Hi Tim, thanks for the feedback, highly appreciated. Out of curiosity - have you found out what was the problem with that OSD? Some hardware issues? Regards, Igor On 10/14/2024 11:58 AM, Tim Sauerbein wrote:
Hi Igor,
Thanks for the valuable advice! I just wanted to provide feedback that it was indeed one single OSD causing the issues which I could triangulate as you said. After removing this OSD, the slow ops haven't occurred anymore.
Best regards, Tim
On 1 Oct 2024, at 12:42, Igor Fedotov <igor.fedotov@croit.io> wrote:
Hi Tim,
first of all - given the provided logs - all the slow operastions are stuck in 'waiting for sub ops' state.
Which apparently means that reported OSDs aren't suffering from local issues but stuck on replication operations to their peer OSDs.
From my experince even a single "faulty" osd could cause such issues to multiple other daemons. And the way to troubleshoot is to find out what are the actual culprit OSD(s).
To do that one might try to use the following approach:
1. When (or shortly after) the issue is happening - run 'ceph daemon osd.N dump_historic_ops' (or even 'dump_ops_in_flight') command against OSDs reporting slow operations.
2. From the above reports choose operations with extraordinary high duration, e.g. > 5 seconds and learn PG ids they've been run against, e.g. PG = 1.a in the following sample:
"description": "osd_op(client.24184.0:23 >>>>1.a<<<<< 1:54253539:::benchmark_data_coalmon_70932_object22:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)",
3. For affected PG(s) learn which OSDs are backing specific it. E.g. by running ceph pg map <pgid>
4. If different PGs from the above step use specific OSD which is common to all (the majority) of them - higly likely it's a good candidate for additional investigation - partcularly relevant OSD logs inspection.
Thanks,
Igor
-- Igor Fedotov Ceph Lead Developer Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
On 14 Oct 2024, at 10:12, Igor Fedotov <igor.fedotov@croit.io> wrote:
Out of curiosity - have you found out what was the problem with that OSD? Some hardware issues?
I guess the SSD is faulty, even though it doesn't show any issues in SMART. I will replace it next week to bring the OSD back online and will report if the issue reappears, which would mean something else is the cause. Thanks, Jan
On 10/14/24 05:05, Tim Sauerbein wrote:
On 14 Oct 2024, at 10:12, Igor Fedotov <igor.fedotov@croit.io> wrote:
Out of curiosity - have you found out what was the problem with that OSD? Some hardware issues? I guess the SSD is faulty, even though it doesn't show any issues in SMART. I will replace it next week to bring the OSD back online and will report if the issue reappears, which would mean something else is the cause. I've seen similar issues before where smart showed no failures but the drive performed terribly. You can try trimming the drive or even doing a secure format to see if it helps, but at least in the case I recall it was an issue with the drive itself.
-- Best Regards, Mark Nelson Head of Research and Development Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w: https://clyso.com | e: mark.nelson@clyso.com We are hiring: https://www.clyso.com/jobs/
Out of curiosity - have you found out what was the problem with that OSD? Some hardware issues? I guess the SSD is faulty, even though it doesn't show any issues in SMART. I will replace it next week to bring the OSD back online and will report if the issue reappears, which would mean something else is the cause. I've seen similar issues before where smart showed no failures but the drive performed terribly. You can try trimming the drive or even doing a secure format to see if it helps, but at least in the case I recall it was an issue with the drive itself.
Remind me, have you sent me a full `smartctl -a` output for this drive? If there’s a firmware update available, updating it with a subsequent secure-erase could plausibly recover it.
On 14 Oct 2024, at 16:01, Anthony D'Atri <aad@dreamsnake.net> wrote:
Remind me, have you sent me a full `smartctl -a` output for this drive?
See here, looks good though: https://gist.github.com/sauerbein/6423231adb954d28c8c82a8422256355
If there’s a firmware update available, updating it with a subsequent secure-erase could plausibly recover it.
I don't think there is a firmware update publicly available. Other disks of same model and same firmware run without issues in my cluster btw.
On 14 Oct 2024, at 15:56, Mark Nelson <mark.nelson@clyso.com> wrote:
I've seen similar issues before where smart showed no failures but the drive performed terribly. You can try trimming the drive or even doing a secure format to see if it helps, but at least in the case I recall it was an issue with the drive itself.
I think that the disk is just faulty too. Do you have any idea of a test to run on the SSD to prove that independent of Ceph? Thanks, Tim
Sorry, forgot to mention: I did a secure erase on the drive yesterday, added it to the OSD again with the same result of slow ops a few hours later.
On 15 Oct 2024, at 16:07, Tim Sauerbein <sauerbein@icloud.com> wrote:
On 14 Oct 2024, at 16:01, Anthony D'Atri <aad@dreamsnake.net> wrote:
Remind me, have you sent me a full `smartctl -a` output for this drive?
See here, looks good though: https://gist.github.com/sauerbein/6423231adb954d28c8c82a8422256355
If there’s a firmware update available, updating it with a subsequent secure-erase could plausibly recover it.
I don't think there is a firmware update publicly available. Other disks of same model and same firmware run without issues in my cluster btw.
On 14 Oct 2024, at 15:56, Mark Nelson <mark.nelson@clyso.com> wrote:
I've seen similar issues before where smart showed no failures but the drive performed terribly. You can try trimming the drive or even doing a secure format to see if it helps, but at least in the case I recall it was an issue with the drive itself.
I think that the disk is just faulty too. Do you have any idea of a test to run on the SSD to prove that independent of Ceph?
Thanks, Tim
Looking at the smartlog seems to show 63C current temp with 53C as worst case which doesn’t make a lot of sense. Could they drive be thermally throttling? Rgds mat From: Tim Sauerbein <sauerbein@icloud.com> Sent: Tuesday, October 15, 2024 11:21 AM To: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] Re: SLOW_OPS problems [External: Do not click links or open attachments without verifying the sender, always login to your account directly.] Sorry, forgot to mention: I did a secure erase on the drive yesterday, added it to the OSD again with the same result of slow ops a few hours later. > On 15 Oct 2024, at 16:07, Tim Sauerbein <sauerbein@icloud.com<mailto:sauerbein@icloud.com>> wrote: > >> NkdkJdXPPEBannerStart Be Careful With This Message From (Tim Sauerbein <sauerbein@icloud.com>)<https://godaddy.cloud-protect.net/email-details/?k=k1&payload=53616c7465645f5f15db257bb6f7446517f83bae6d9f7752b417bd2af5cd0f818688bd2dbc09ea4d63ff123db727aac7a26336b0ac38030b4d5f6578b60f00f399cecc7c56cdf55ccf24a1fbdf14dd1574c17a4c300de8705b37b4ef25d11fe41ce0f9fdb7e8228a29a4207e9be5cfb2fd78296edffa23c172a5cf397b0ff766a21788297658e61718f53d2a445d8056650a2d047cf31eeb08d1ff50f3ec7363971db9f2a6809e803c3678894306df1b57d1d2463235136b2beacce4e62ccdca0169afeef74aea8b0af616a0d7c44fc6c0b69d24c4211ba1c98a19ac640aafae16cb463be63f93ad06a2c67696d8bcba> Learn More<https://godaddy.cloud-protect.net/email-details/?k=k1&payload=53616c7465645f5f15db257bb6f7446517f83bae6d9f7752b417bd2af5cd0f818688bd2dbc09ea4d63ff123db727aac7a26336b0ac38030b4d5f6578b60f00f399cecc7c56cdf55ccf24a1fbdf14dd1574c17a4c300de8705b37b4ef25d11fe41ce0f9fdb7e8228a29a4207e9be5cfb2fd78296edffa23c172a5cf397b0ff766a21788297658e61718f53d2a445d8056650a2d047cf31eeb08d1ff50f3ec7363971db9f2a6809e803c3678894306df1b57d1d2463235136b2beacce4e62ccdca0169afeef74aea8b0af616a0d7c44fc6c0b69d24c4211ba1c98a19ac640aafae16cb463be63f93ad06a2c67696d8bcba> Potential Impersonation The sender's identity could not be verified and someone may be impersonating the sender. Take caution when interacting with this message. NkdkJdXPPEBannerEnd Sorry, forgot to mention: I did a secure erase on the drive yesterday, added it to the OSD again with the same result of slow ops a few hours later.
On 15 Oct 2024, at 16:07, Tim Sauerbein <sauerbein@icloud.com<mailto:sauerbein@icloud.com>> wrote:
On 14 Oct 2024, at 16:01, Anthony D'Atri <aad@dreamsnake.net<mailto:aad@dreamsnake.net>> wrote:
Remind me, have you sent me a full `smartctl -a` output for this drive?
If there’s a firmware update available, updating it with a subsequent secure-erase could plausibly recover it.
I don't think there is a firmware update publicly available. Other disks of same model and same firmware run without issues in my cluster btw.
On 14 Oct 2024, at 15:56, Mark Nelson <https://urldefense.proofpoint.com/v2/url?u=http-3A__mark.nelson-40clyso.com&d=DwIGaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=bDvyvkr2bb4BZTlvrJx55ZzKgQTyuyI1pgJpjo3ippU&m=4BzydYt8R0yplK8QvqowJz2GaV9Lnf7dg9Xos6bDeNBYcAyuL4faGr3ma7qRjCKc&s=59RCI_vd1KgnEwQGO9-paAVPPm3884F9Oq_hMho-S94&e=> wrote:
I've seen similar issues before where smart showed no failures but the drive performed terribly. You can try trimming the drive or even doing a secure format to see if it helps, but at least in the case I recall it was an issue with the drive itself.
I think that the disk is just faulty too. Do you have any idea of a test to run on the SSD to prove that independent of Ceph?
Thanks,
Tim
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> ---------- This email has been scanned for spam and viruses. Visit the following link to report this email as spam: https://godaddy.cloud-protect.net/app/report_spam.php?mod_id=11&mod_option=logitem&report=1&type=easyspam&k=k1&payload=53616c7465645f5f99858a10fb5d88069d46e0a76bffcf608852edf1729c8dcab288411ea1dcf982c558f4d477adb88e6d258294685049d6531aa93e1b8c603e461cc0af17dbd4010838586edf64d1e3d67b6155f1c08399ef0fbc70bf62abaf9580c00d306da64edb207e0b95ba9646aff218c3a846470698aff34e3499a4dc272c128392c11d8a608369425815e393a3329a6a2a60a1d16dad9704ea168caf
Oh yeah that’s really high for a drive. Do other drives in the same / other chassis show the same temps, or is this an outlier? With Dell chassis, for example, I’ve often had to increase the iDRAC fan speed offset to get the drive temps below 40C
On Oct 15, 2024, at 1:36 PM, Mat Young <mat.young@scaleflux.com> wrote:
Looking at the smartlog seems to show 63C current temp with 53C as worst case which doesn’t make a lot of sense. Could they drive be thermally throttling?
Rgds
mat
From: Tim Sauerbein <sauerbein@icloud.com> Sent: Tuesday, October 15, 2024 11:21 AM To: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] Re: SLOW_OPS problems
[External: Do not click links or open attachments without verifying the sender, always login to your account directly.] Sorry, forgot to mention: I did a secure erase on the drive yesterday, added it to the OSD again with the same result of slow ops a few hours later. > On 15 Oct 2024, at 16:07, Tim Sauerbein <sauerbein@icloud.com<mailto:sauerbein@icloud.com>> wrote: > >> NkdkJdXPPEBannerStart Be Careful With This Message From (Tim Sauerbein <sauerbein@icloud.com>)<https://godaddy.cloud-protect.net/email-details/?k=k1&payload=53616c7465645f5f15db257bb6f7446517f83bae6d9f7752b417bd2af5cd0f818688bd2dbc09ea4d63ff123db727aac7a26336b0ac38030b4d5f6578b60f00f399cecc7c56cdf55ccf24a1fbdf14dd1574c17a4c300de8705b37b4ef25d11fe41ce0f9fdb7e8228a29a4207e9be5cfb2fd78296edffa23c172a5cf397b0ff766a21788297658e61718f53d2a445d8056650a2d047cf31eeb08d1ff50f3ec7363971db9f2a6809e803c3678894306df1b57d1d2463235136b2beacce4e62ccdca0169afeef74aea8b0af616a0d7c44fc6c0b69d24c4211ba1c98a19ac640aafae16cb463be63f93ad06a2c67696d8bcba> Learn More<https://godaddy.cloud-protect.net/email-details/?k=k1&payload=53616c7465645f5f15db257bb6f7446517f83bae6d9f7752b417bd2af5cd0f818688bd2dbc09ea4d63ff123db727aac7a26336b0ac38030b4d5f6578b60f00f399cecc7c56cdf55ccf24a1fbdf14dd1574c17a4c300de8705b37b4ef25d11fe41ce0f9fdb7e8228a29a4207e9be5cfb2fd78296edffa23c172a5cf397b0ff766a21788297658e61718f53d2a445d8056650a2d047cf31eeb08d1ff50f3ec7363971db9f2a6809e803c3678894306df1b57d1d2463235136b2beacce4e62ccdca0169afeef74aea8b0af616a0d7c44fc6c0b69d24c4211ba1c98a19ac640aafae16cb463be63f93ad06a2c67696d8bcba> Potential Impersonation The sender's identity could not be verified and someone may be impersonating the sender. Take caution when interacting with this message.
NkdkJdXPPEBannerEnd
Sorry, forgot to mention:
I did a secure erase on the drive yesterday, added it to the OSD again with the same result of slow ops a few hours later.
On 15 Oct 2024, at 16:07, Tim Sauerbein <sauerbein@icloud.com<mailto:sauerbein@icloud.com>> wrote:
On 14 Oct 2024, at 16:01, Anthony D'Atri <aad@dreamsnake.net<mailto:aad@dreamsnake.net>> wrote:
Remind me, have you sent me a full `smartctl -a` output for this drive?
If there’s a firmware update available, updating it with a subsequent secure-erase could plausibly recover it.
I don't think there is a firmware update publicly available. Other disks of same model and same firmware run without issues in my cluster btw.
On 14 Oct 2024, at 15:56, Mark Nelson <https://urldefense.proofpoint.com/v2/url?u=http-3A__mark.nelson-40clyso.com&d=DwIGaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=bDvyvkr2bb4BZTlvrJx55ZzKgQTyuyI1pgJpjo3ippU&m=4BzydYt8R0yplK8QvqowJz2GaV9Lnf7dg9Xos6bDeNBYcAyuL4faGr3ma7qRjCKc&s=59RCI_vd1KgnEwQGO9-paAVPPm3884F9Oq_hMho-S94&e=> wrote:
I've seen similar issues before where smart showed no failures but the drive performed terribly. You can try trimming the drive or even doing a secure format to see if it helps, but at least in the case I recall it was an issue with the drive itself.
I think that the disk is just faulty too. Do you have any idea of a test to run on the SSD to prove that independent of Ceph?
Thanks,
Tim
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
----------
This email has been scanned for spam and viruses. Visit the following link to report this email as spam:
https://godaddy.cloud-protect.net/app/report_spam.php?mod_id=11&mod_option=logitem&report=1&type=easyspam&k=k1&payload=53616c7465645f5f99858a10fb5d88069d46e0a76bffcf608852edf1729c8dcab288411ea1dcf982c558f4d477adb88e6d258294685049d6531aa93e1b8c603e461cc0af17dbd4010838586edf64d1e3d67b6155f1c08399ef0fbc70bf62abaf9580c00d306da64edb207e0b95ba9646aff218c3a846470698aff34e3499a4dc272c128392c11d8a608369425815e393a3329a6a2a60a1d16dad9704ea168caf _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Agreed and you may see some bays run hot when the cabling is not optimal behind the bay, reseating and re routing the bay cabling if possible may help. -----Original Message----- From: Anthony D'Atri <anthony.datri@gmail.com> Sent: Tuesday, October 15, 2024 1:48 PM To: Mat Young <mat.young@scaleflux.com> Cc: Tim Sauerbein <sauerbein@icloud.com>; ceph-users <ceph-users@ceph.io> Subject: Re: [ceph-users] SLOW_OPS problems [External: Do not click links or open attachments without verifying the sender, always login to your account directly.] Oh yeah that’s really high for a drive. Do other drives in the same / other chassis show the same temps, or is this an outlier? With Dell chassis, for example, I’ve often had to increase the iDRAC fan speed offset to get the drive temps below 40C
On Oct 15, 2024, at 1:36 PM, Mat Young <mat.young@scaleflux.com> wrote:
Looking at the smartlog seems to show 63C current temp with 53C as worst case which doesn’t make a lot of sense. Could they drive be thermally throttling?
Rgds
mat
From: Tim Sauerbein <sauerbein@icloud.com> Sent: Tuesday, October 15, 2024 11:21 AM To: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] Re: SLOW_OPS problems
[External: Do not click links or open attachments without verifying the sender, always login to your account directly.] Sorry, forgot to mention: I did a secure erase on the drive yesterday, added it to the OSD again with the same result of slow ops a few hours later. > On 15 Oct 2024, at 16:07, Tim Sauerbein <sauerbein@icloud.com<mailto:sauerbein@icloud.com>> wrote: > >> NkdkJdXPPEBannerStart Be Careful With This Message From (Tim Sauerbein <sauerbein@icloud.com>)<https://godaddy.cloud-protect.net/email-detail s/?k=k1&payload=53616c7465645f5f15db257bb6f7446517f83bae6d9f7752b417bd 2af5cd0f818688bd2dbc09ea4d63ff123db727aac7a26336b0ac38030b4d5f6578b60f 00f399cecc7c56cdf55ccf24a1fbdf14dd1574c17a4c300de8705b37b4ef25d11fe41c e0f9fdb7e8228a29a4207e9be5cfb2fd78296edffa23c172a5cf397b0ff766a2178829 7658e61718f53d2a445d8056650a2d047cf31eeb08d1ff50f3ec7363971db9f2a6809e 803c3678894306df1b57d1d2463235136b2beacce4e62ccdca0169afeef74aea8b0af6 16a0d7c44fc6c0b69d24c4211ba1c98a19ac640aafae16cb463be63f93ad06a2c67696 d8bcba> Learn More<https://godaddy.cloud-protect.net/email-details/?k=k1&payload=536 16c7465645f5f15db257bb6f7446517f83bae6d9f7752b417bd2af5cd0f818688bd2db c09ea4d63ff123db727aac7a26336b0ac38030b4d5f6578b60f00f399cecc7c56cdf55 ccf24a1fbdf14dd1574c17a4c300de8705b37b4ef25d11fe41ce0f9fdb7e8228a29a42 07e9be5cfb2fd78296edffa23c172a5cf397b0ff766a21788297658e61718f53d2a445 d8056650a2d047cf31eeb08d1ff50f3ec7363971db9f2a6809e803c3678894306df1b5 7d1d2463235136b2beacce4e62ccdca0169afeef74aea8b0af616a0d7c44fc6c0b69d2 4c4211ba1c98a19ac640aafae16cb463be63f93ad06a2c67696d8bcba> Potential Impersonation The sender's identity could not be verified and someone may be impersonating the sender. Take caution when interacting with this message.
NkdkJdXPPEBannerEnd
Sorry, forgot to mention:
I did a secure erase on the drive yesterday, added it to the OSD again with the same result of slow ops a few hours later.
On 15 Oct 2024, at 16:07, Tim Sauerbein <sauerbein@icloud.com<mailto:sauerbein@icloud.com>> wrote:
On 14 Oct 2024, at 16:01, Anthony D'Atri <aad@dreamsnake.net<mailto:aad@dreamsnake.net>> wrote:
Remind me, have you sent me a full `smartctl -a` output for this drive?
See here, looks good though: https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_ sauerbein_6423231adb954d28c8c82a8422256355&d=DwIGaQ&c=euGZstcaTDllvim EN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=bDvyvkr2bb4BZTlvrJx55ZzKgQTyuyI1pgJpjo 3ippU&m=4BzydYt8R0yplK8QvqowJz2GaV9Lnf7dg9Xos6bDeNBYcAyuL4faGr3ma7qRj CKc&s=VKfARgTFDlE0uDcv7zs4xjZ6--dLJqws6_O877VPbyw&e=
If there’s a firmware update available, updating it with a subsequent secure-erase could plausibly recover it.
I don't think there is a firmware update publicly available. Other disks of same model and same firmware run without issues in my cluster btw.
On 14 Oct 2024, at 15:56, Mark Nelson <https://urldefense.proofpoint.com/v2/url?u=http-3A__mark.nelson-40clyso.com&d=DwIGaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=bDvyvkr2bb4BZTlvrJx55ZzKgQTyuyI1pgJpjo3ippU&m=4BzydYt8R0yplK8QvqowJz2GaV9Lnf7dg9Xos6bDeNBYcAyuL4faGr3ma7qRjCKc&s=59RCI_vd1KgnEwQGO9-paAVPPm3884F9Oq_hMho-S94&e=> wrote:
I've seen similar issues before where smart showed no failures but the drive performed terribly. You can try trimming the drive or even doing a secure format to see if it helps, but at least in the case I recall it was an issue with the drive itself.
I think that the disk is just faulty too. Do you have any idea of a test to run on the SSD to prove that independent of Ceph?
Thanks,
Tim
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
----------
This email has been scanned for spam and viruses. Visit the following link to report this email as spam:
https://godaddy.cloud-protect.net/app/report_spam.php?mod_id=11&mod_op tion=logitem&report=1&type=easyspam&k=k1&payload=53616c7465645f5f99858 a10fb5d88069d46e0a76bffcf608852edf1729c8dcab288411ea1dcf982c558f4d477a db88e6d258294685049d6531aa93e1b8c603e461cc0af17dbd4010838586edf64d1e3d 67b6155f1c08399ef0fbc70bf62abaf9580c00d306da64edb207e0b95ba9646aff218c 3a846470698aff34e3499a4dc272c128392c11d8a608369425815e393a3329a6a2a60a 1d16dad9704ea168caf _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
---------- This email has been scanned for spam and viruses. Visit the following link to report this email as spam: https://godaddy.cloud-protect.net/app/report_spam.php?mod_id=11&mod_option=logitem&report=1&type=easyspam&k=k1&payload=53616c7465645f5f08d8140a2e039990a3d9cd5ed578f9f6efa11f08fb971724916154b6336fa63a939c80b209cbef3c467d81bce4e1001de2128f88deff6d20da7e71ea745478bb4f2073cb6b656f85f1d8ff0fadab1544c1cc7e06ba007f3a80ff407b613a346d8be93804ed1053a5716624ea1ca5fc23c9aed410e3245fdd5ee523d381cf6d170adf55edaa55d3752836d6f71ebab9a1904e885b54827ea6
I've got a M.2 drive that will hit 94C on the surface of the drive as seen with my thermal camera if it doesn't have active cooling on it. :D FWIW, I don't recall seeing thermal throttling in the 60C range in the past. We've seen it at higher temps though. Mark On 10/15/24 12:48, Anthony D'Atri wrote:
Oh yeah that’s really high for a drive.
Do other drives in the same / other chassis show the same temps, or is this an outlier?
With Dell chassis, for example, I’ve often had to increase the iDRAC fan speed offset to get the drive temps below 40C
On Oct 15, 2024, at 1:36 PM, Mat Young <mat.young@scaleflux.com> wrote:
Looking at the smartlog seems to show 63C current temp with 53C as worst case which doesn’t make a lot of sense. Could they drive be thermally throttling?
Rgds
mat
From: Tim Sauerbein <sauerbein@icloud.com> Sent: Tuesday, October 15, 2024 11:21 AM To: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] Re: SLOW_OPS problems
[External: Do not click links or open attachments without verifying the sender, always login to your account directly.] Sorry, forgot to mention: I did a secure erase on the drive yesterday, added it to the OSD again with the same result of slow ops a few hours later. > On 15 Oct 2024, at 16:07, Tim Sauerbein <sauerbein@icloud.com<mailto:sauerbein@icloud.com>> wrote: > >> NkdkJdXPPEBannerStart Be Careful With This Message From (Tim Sauerbein <sauerbein@icloud.com>)<https://godaddy.cloud-protect.net/email-details/?k=k1&payload=53616c7465645f5f15db257bb6f7446517f83bae6d9f7752b417bd2af5cd0f818688bd2dbc09ea4d63ff123db727aac7a26336b0ac38030b4d5f6578b60f00f399cecc7c56cdf55ccf24a1fbdf14dd1574c17a4c300de8705b37b4ef25d11fe41ce0f9fdb7e8228a29a4207e9be5cfb2fd78296edffa23c172a5cf397b0ff766a21788297658e61718f53d2a445d8056650a2d047cf31eeb08d1ff50f3ec7363971db9f2a6809e803c3678894306df1b57d1d2463235136b2beacce4e62ccdca0169afeef74aea8b0af616a0d7c44fc6c0b69d24c4211ba1c98a19ac640aafae16cb463be63f93ad06a2c67696d8bcba> Learn More<https://godaddy.cloud-protect.net/email-details/?k=k1&payload=53616c7465645f5f15db257bb6f7446517f83bae6d9f7752b417bd2af5cd0f818688bd2dbc09ea4d63ff123db727aac7a26336b0ac38030b4d5f6578b60f00f399cecc7c56cdf55ccf24a1fbdf14dd1574c17a4c300de8705b37b4ef25d11fe41ce0f9fdb7e8228a29a4207e9be5cfb2fd78296edffa23c172a5cf397b0ff766a21788297658e61718f53d2a445d8056650a2d047cf31eeb08d1ff50f3ec7363971db9f2a6809e803c3678894306df1b57d1d2463235136b2beacce4e62ccdca0169afeef74aea8b0af616a0d7c44fc6c0b69d24c4211ba1c98a19ac640aafae16cb463be63f93ad06a2c67696d8bcba> Potential Impersonation The sender's identity could not be verified and someone may be impersonating the sender. Take caution when interacting with this message.
NkdkJdXPPEBannerEnd
Sorry, forgot to mention:
I did a secure erase on the drive yesterday, added it to the OSD again with the same result of slow ops a few hours later.
On 15 Oct 2024, at 16:07, Tim Sauerbein <sauerbein@icloud.com<mailto:sauerbein@icloud.com>> wrote:
On 14 Oct 2024, at 16:01, Anthony D'Atri <aad@dreamsnake.net<mailto:aad@dreamsnake.net>> wrote: Remind me, have you sent me a full `smartctl -a` output for this drive? See here, looks good though: https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_sauerbein_6423231adb954d28c8c82a8422256355&d=DwIGaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=bDvyvkr2bb4BZTlvrJx55ZzKgQTyuyI1pgJpjo3ippU&m=4BzydYt8R0yplK8QvqowJz2GaV9Lnf7dg9Xos6bDeNBYcAyuL4faGr3ma7qRjCKc&s=VKfARgTFDlE0uDcv7zs4xjZ6--dLJqws6_O877VPbyw&e= If there’s a firmware update available, updating it with a subsequent secure-erase could plausibly recover it. I don't think there is a firmware update publicly available. Other disks of same model and same firmware run without issues in my cluster btw. On 14 Oct 2024, at 15:56, Mark Nelson <https://urldefense.proofpoint.com/v2/url?u=http-3A__mark.nelson-40clyso.com&d=DwIGaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=bDvyvkr2bb4BZTlvrJx55ZzKgQTyuyI1pgJpjo3ippU&m=4BzydYt8R0yplK8QvqowJz2GaV9Lnf7dg9Xos6bDeNBYcAyuL4faGr3ma7qRjCKc&s=59RCI_vd1KgnEwQGO9-paAVPPm3884F9Oq_hMho-S94&e=> wrote: I've seen similar issues before where smart showed no failures but the drive performed terribly. You can try trimming the drive or even doing a secure format to see if it helps, but at least in the case I recall it was an issue with the drive itself. I think that the disk is just faulty too. Do you have any idea of a test to run on the SSD to prove that independent of Ceph? Thanks, Tim
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
----------
This email has been scanned for spam and viruses. Visit the following link to report this email as spam:
https://godaddy.cloud-protect.net/app/report_spam.php?mod_id=11&mod_option=logitem&report=1&type=easyspam&k=k1&payload=53616c7465645f5f99858a10fb5d88069d46e0a76bffcf608852edf1729c8dcab288411ea1dcf982c558f4d477adb88e6d258294685049d6531aa93e1b8c603e461cc0af17dbd4010838586edf64d1e3d67b6155f1c08399ef0fbc70bf62abaf9580c00d306da64edb207e0b95ba9646aff218c3a846470698aff34e3499a4dc272c128392c11d8a608369425815e393a3329a6a2a60a1d16dad9704ea168caf _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Best Regards, Mark Nelson Head of Research and Development Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w: https://clyso.com | e: mark.nelson@clyso.com We are hiring: https://www.clyso.com/jobs/
On Tue, Oct 15, 2024 at 05:36:15PM +0000, Mat Young wrote:
Looking at the smartlog seems to show 63C current temp with 53C as worst case which doesn’t make a lot of sense. Could they drive be thermally throttling?
That is the normalized value, shouldn't the value in RAW_VALUE be used instead? There the current is 37 and min is 28 and high 42. ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 190 Airflow_Temperature_Cel 0x0032 063 058 000 Old_age Always - 37 194 Temperature_Celsius 0x0022 063 058 000 Old_age Always - 37 (Min/Max 28/42) -- Kai Stian Olstad
On 15 Oct 2024, at 18:57, Kai Stian Olstad <ceph+list@olstad.com> wrote:
On Tue, Oct 15, 2024 at 05:36:15PM +0000, Mat Young wrote:
Looking at the smartlog seems to show 63C current temp with 53C as worst case which doesn’t make a lot of sense. Could they drive be thermally throttling?
That is the normalized value, shouldn't the value in RAW_VALUE be used instead? There the current is 37 and min is 28 and high 42.
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 190 Airflow_Temperature_Cel 0x0032 063 058 000 Old_age Always - 37 194 Temperature_Celsius 0x0022 063 058 000 Old_age Always - 37 (Min/Max 28/42)
I'm quite sure the raw value is the actual degrees celsius. I also checked a couple of SSDs in my other servers and they are all in the low / middle 30 degrees. This should be okay and not lead to thermal throttling. I will report back when I've - done some I/O tests on the SSD to see if I can reproduce the issue without Ceph - replaced the SSD with a spare one of the same model to see if the problem reappears Thanks all! Tim
On Mon, Sep 30, 2024 at 11:04:30AM +0100, Tim Sauerbein wrote:
On 30 Sep 2024, at 06:23, Joachim Kraftmayer <joachim.kraftmayer@clyso.com> wrote:
do you see the behaviour across all devices or does it only affect one type/manufacturer?
All devices are affected equally, every time one or two random ODSs report slow ops. So I don't think the SSDs are to blame.
One cause for "slow ops" I discovered are networking issues. I had slow ops across my entire cluster (interconnected with 10G). Turns out the switch was bad an achieved < 10 MBit/s on one of the 10G links. Replaced the switch, tested the links again - got full 10G connectivity and the slow ops disappeared. _This_ one was easy to test for: just run iperf3 across your links. Of course, there are plenty of other possible causes .. Kind regards, Alex. -- "Opportunity is missed by most people because it is dressed in overalls and looks like work." -- Thomas A. Edison
Hi Tim, there is no log attached to your post, you better share it via some other means. BTW - what log did you mean - monitor or OSD one? It would be nice to have logs for a couple of OSDs suffering from slow ops, preferably relevant to two different cases. Thanks, Igor On 9/29/2024 3:29 PM, Tim Sauerbein wrote:
ite speed
-- Igor Fedotov Ceph Lead Developer Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
participants (9)
-
Alexander Schreiber
-
Anthony D'Atri
-
Anthony D'Atri
-
Igor Fedotov
-
Joachim Kraftmayer
-
Kai Stian Olstad
-
Mark Nelson
-
Mat Young
-
Tim Sauerbein