Hi,
around the same time
sounds like deep-scrubbing. Did you verify if those OSDs from the mentioned pool were scrubbed around that time? The primary OSDs for the PGs would log that. Is that pool that heavily used? Or could there be one failing disk? Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@agoda.com>:
Hi,
For some reason daily I receive a slow ops which affect the rgw.log pool the most (which is pretty small, 32pg / 57k objects / 9.5GB data)
Could be the issue related to rocksdb tasks?
Some log lines: ... 2024-06-18T14:17:43.849+0700 7fd8002ea700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fd8002ea700' had timed out after 15 2024-06-18T14:17:50.865+0700 7fd81a31e700 -1 osd.461 472771 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.3349483990.0:462716190 22.11 22:8deed178:::meta.log.08e85e9f-c16e-43f0-b88d-362f3b7ced2d.15:head [call log.list in=69b] snapc 0=[] ondisk+read+known_if_redirected e472771) 2024-06-18T14:17:50.865+0700 7fd81a31e700 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'dc.rgw.log' : 1 ]) 2024-06-18T14:19:36.040+0700 7fd81e5b4700 1 heartbeat_map is_healthy 'OSD::osd_op_tp ... 2024-06-18T14:19:51.908+0700 7fd81a31e700 -1 osd.461 472771 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.3371539359.0:5095510 22.11 22:8df1b9e1:::datalog.sync-status.shard.61c9d940-fde4-4bed-9389-edc8d7741817.111: head [call lock.lock in=64b] snapc 0=[] ondisk+write+known_if_redirected e472771) 2024-06-18T14:19:51.908+0700 7fd81a31e700 0 log_channel(cluster) log [WRN] : 2 slow requests (by type [ 'delayed' : 2 ] most affected pool [ 'dc.rgw.log' : 2 ]) 2024-06-18T14:19:52.899+0700 7fd81a31e700 -1 osd.461 472771 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.3371539359.0:5095510 22.11 22:8df1b9e1:::datalog.sync-status.shard.61c9d940-fde4-4bed-9389-edc8d7741817.111:
I found this thread: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/FR5V466HBXGR... but I think it is not relevant for me.
We use only SSDs without separated rocksdb so we don' have spillover. CPU is not overloaded (80% idle) however it's interesting that in vmstat the "r" is pretty big which indicates waiting list for cpu:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 12 0 0 6205288 329803136 15633844 0 0 2149 1894 0 0 5 3 92 0 0 17 0 0 6156780 329827840 15637252 0 0 125584 250416 646098 1965051 10 8 82 0 0 21 0 0 6154636 329849504 15636112 0 0 99320 245024 493324 1682377 7 6 87 0 0 16 0 0 6144256 329869664 15636924 0 0 87484 301968 623345 1993374 8 7 84 0 0 19 0 0 6057012 329890080 15637932 0 0 160444 303664 549194 1820562 8 6 85 0 0
Any idea what this could indicate on octopus 15.2.17 with ubuntu 20.04?
ty
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io