Hello, We are experiencing slowdowns on one of our radosgw clusters. We restart the radosgw daemons every 2 hours and things start getting slow after an hour and a half. The avg get/put latencies go from 20ms/400ms to 1s/5s+ according to the metrics. When I stop traffic to one of the radosgw daemon by setting it to DRAIN in HAProxy and HAProxy reports 0 sessions to the daemon, I still see the objecter_requests still going up and down 15 min later. linger_ops seems to stay at a constant 10 the whole time.
[root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | length' 46 [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | length' 211 [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | length' 427 [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | length' 0 [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | length' 26 [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | length' 190
The requests say they are "call rgw.bucket_list in=234b" and mostly reference ".dir.5e9bc383-f7bd-4fd1-b607-1e563bfe0011.886814949.1.N" where N is 1-149 (the shard count for the bucket).
{ "tid": 2583119, "pg": "8.579482b0", "osd": 208, "object_id": ".dir.5e9bc383-f7bd-4fd1-b607-1e563bfe0011.886814949.1.124", "object_locator": "@8", "target_object_id": ".dir.5e9bc383-f7bd-4fd1-b607-1e563bfe0011.886814949.1.124", "target_object_locator": "@8", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "1796975.496664s", "age": 0.350012616, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "1970-01-01T00:00:00.000000+0000", "osd_ops": [ "call rgw.bucket_list in=234b" ] },
I don't think it should be other rgw processes because we have this daemon's ceph.conf set to disable the other threads.
rgw enable gc threads = false rgw enable lc threads = false rgw dynamic resharding = false
When I restart the service while it is still in a DRAINED state in HAProxy, checking the objecter_requests yields 0 even a few minutes after it has been up.
[root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | length' 0 [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | length' 0 [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | length' 0 [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | length' 0 [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | length' 0 [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | length' 0
Any thoughts on why these ops appear to be stuck/recurring until restarting the daemon? I think this is related to our performance issues but I don't know what the fix is. The rgws are 18.2.4 running as containers in Podman on Debian 11. Our other clusters do not exhibit this behavior. Thanks! Reid