radosgw new zonegroup hammers master with metadata sync
Hi, yesterday I added a new zonegroup and it looks like it seems to cycle over the same requests over and over again. In the log of the main zone I see these requests: 2023-06-20T09:48:37.979+0000 7f8941fb3700 1 beast: 0x7f8a602f3700: fd00:2380:0:24::136 - - [2023-06-20T09:48:37.979941+0000] "GET /admin/log?type=metadata&id=62&period=e8fc96f1-ae86-4dc1-b432-470b0772fded&max-entries=100&&rgwx-zonegroup=b39392eb-75f8-47f0-b4f3-7d3882930b26 HTTP/1.1" 200 44 - - - Only thing that changes is the &id. We have two other zonegroups that are configured identical (ceph.conf and period) and these don;t seem to spam the main rgw. root@host:~# radosgw-admin sync status realm 5d6f2ea4-b84a-459b-bce2-bccac338b3ef (main) zonegroup b39392eb-75f8-47f0-b4f3-7d3882930b26 (dc3) zone 96f5eca9-425b-4194-a152-86e310e91ddb (dc3) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master root@host:~# radosgw-admin period get { "id": "e8fc96f1-ae86-4dc1-b432-470b0772fded", "epoch": 92, "predecessor_uuid": "5349ac85-3d6d-4088-993f-7a1d4be3835a", "sync_status": [ "", ... "" ], "period_map": { "id": "e8fc96f1-ae86-4dc1-b432-470b0772fded", "zonegroups": [ { "id": "b39392eb-75f8-47f0-b4f3-7d3882930b26", "name": "dc3", "api_name": "dc3", "is_master": "false", "endpoints": [ ], "hostnames": [ ], "hostnames_s3website": [ ], "master_zone": "96f5eca9-425b-4194-a152-86e310e91ddb", "zones": [ { "id": "96f5eca9-425b-4194-a152-86e310e91ddb", "name": "dc3", "endpoints": [ ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "5d6f2ea4-b84a-459b-bce2-bccac338b3ef", "sync_policy": { "groups": [] } }, ... -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
hi Boris, we've been investigating reports of excessive polling from metadata sync. i just opened https://tracker.ceph.com/issues/61743 to track this. restarting the secondary zone radosgws should help as a temporary workaround On Tue, Jun 20, 2023 at 5:57 AM Boris Behrens <bb@kervyn.de> wrote:
Hi, yesterday I added a new zonegroup and it looks like it seems to cycle over the same requests over and over again.
In the log of the main zone I see these requests: 2023-06-20T09:48:37.979+0000 7f8941fb3700 1 beast: 0x7f8a602f3700: fd00:2380:0:24::136 - - [2023-06-20T09:48:37.979941+0000] "GET /admin/log?type=metadata&id=62&period=e8fc96f1-ae86-4dc1-b432-470b0772fded&max-entries=100&&rgwx-zonegroup=b39392eb-75f8-47f0-b4f3-7d3882930b26 HTTP/1.1" 200 44 - - -
Only thing that changes is the &id.
We have two other zonegroups that are configured identical (ceph.conf and period) and these don;t seem to spam the main rgw.
root@host:~# radosgw-admin sync status realm 5d6f2ea4-b84a-459b-bce2-bccac338b3ef (main) zonegroup b39392eb-75f8-47f0-b4f3-7d3882930b26 (dc3) zone 96f5eca9-425b-4194-a152-86e310e91ddb (dc3) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master
root@host:~# radosgw-admin period get { "id": "e8fc96f1-ae86-4dc1-b432-470b0772fded", "epoch": 92, "predecessor_uuid": "5349ac85-3d6d-4088-993f-7a1d4be3835a", "sync_status": [ "", ... "" ], "period_map": { "id": "e8fc96f1-ae86-4dc1-b432-470b0772fded", "zonegroups": [ { "id": "b39392eb-75f8-47f0-b4f3-7d3882930b26", "name": "dc3", "api_name": "dc3", "is_master": "false", "endpoints": [ ], "hostnames": [ ], "hostnames_s3website": [ ], "master_zone": "96f5eca9-425b-4194-a152-86e310e91ddb", "zones": [ { "id": "96f5eca9-425b-4194-a152-86e310e91ddb", "name": "dc3", "endpoints": [ ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "5d6f2ea4-b84a-459b-bce2-bccac338b3ef", "sync_policy": { "groups": [] } }, ...
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Casey, already did restart all RGW instances. Only helped for 2 minutes. We now stopped the new site. I will remove and recreate it later. As twi other sites don't have the problem I currently think I made a mistake in the process. Mit freundlichen Grüßen - Boris Behrens
Am 20.06.2023 um 18:30 schrieb Casey Bodley <cbodley@redhat.com>:
hi Boris,
we've been investigating reports of excessive polling from metadata sync. i just opened https://tracker.ceph.com/issues/61743 to track this. restarting the secondary zone radosgws should help as a temporary workaround
On Tue, Jun 20, 2023 at 5:57 AM Boris Behrens <bb@kervyn.de> wrote:
Hi, yesterday I added a new zonegroup and it looks like it seems to cycle over the same requests over and over again.
In the log of the main zone I see these requests: 2023-06-20T09:48:37.979+0000 7f8941fb3700 1 beast: 0x7f8a602f3700: fd00:2380:0:24::136 - - [2023-06-20T09:48:37.979941+0000] "GET /admin/log?type=metadata&id=62&period=e8fc96f1-ae86-4dc1-b432-470b0772fded&max-entries=100&&rgwx-zonegroup=b39392eb-75f8-47f0-b4f3-7d3882930b26 HTTP/1.1" 200 44 - - -
Only thing that changes is the &id.
We have two other zonegroups that are configured identical (ceph.conf and period) and these don;t seem to spam the main rgw.
root@host:~# radosgw-admin sync status realm 5d6f2ea4-b84a-459b-bce2-bccac338b3ef (main) zonegroup b39392eb-75f8-47f0-b4f3-7d3882930b26 (dc3) zone 96f5eca9-425b-4194-a152-86e310e91ddb (dc3) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master
root@host:~# radosgw-admin period get { "id": "e8fc96f1-ae86-4dc1-b432-470b0772fded", "epoch": 92, "predecessor_uuid": "5349ac85-3d6d-4088-993f-7a1d4be3835a", "sync_status": [ "", ... "" ], "period_map": { "id": "e8fc96f1-ae86-4dc1-b432-470b0772fded", "zonegroups": [ { "id": "b39392eb-75f8-47f0-b4f3-7d3882930b26", "name": "dc3", "api_name": "dc3", "is_master": "false", "endpoints": [ ], "hostnames": [ ], "hostnames_s3website": [ ], "master_zone": "96f5eca9-425b-4194-a152-86e310e91ddb", "zones": [ { "id": "96f5eca9-425b-4194-a152-86e310e91ddb", "name": "dc3", "endpoints": [ ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "5d6f2ea4-b84a-459b-bce2-bccac338b3ef", "sync_policy": { "groups": [] } }, ...
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I recreated the site and the problem still persists. I've upped the logging and saw this for a lot of buckets (i've stopped the debug log after some seconds). 2023-06-20T23:32:29.365+0000 7fcaab7fe700 20 get_system_obj_state: rctx=0x7fcaab7f9320 obj=dc3.rgw.meta:root:s3bucket-fra2 state=0x7fcba05ac0a0 s->prefetch_data=0 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache get: name=dc3.rgw.meta+root+s3bucket-fra2 : miss 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache put: name=dc3.rgw.meta+root+s3bucket-fra2 info.flags=0x6 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 adding dc3.rgw.meta+root+s3bucket-fra2 to cache LRU end 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache get: name=dc3.rgw.meta+root+s3bucket-fra2 : type miss (requested=0x1, cached=0x6) 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache put: name=dc3.rgw.meta+root+s3bucket-fra2 info.flags=0x1 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 moving dc3.rgw.meta+root+s3bucket-fra2 to cache LRU end 2023-06-20T23:32:29.365+0000 7fcaab7fe700 20 get_system_obj_state: rctx=0x7fcaab7f9320 obj=dc3.rgw.meta:root:.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 state=0x7fcba43ce0a0 s->prefetch_data=0 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache get: name=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 : miss 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache put: name=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 info.flags=0x16 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 adding dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 to cache LRU end 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache get: name=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 : type miss (requested=0x13, cached=0x16) 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache put: name=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 info.flags=0x13 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 moving dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 to cache LRU end 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 chain_cache_entry: cache_locator=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 Am Di., 20. Juni 2023 um 19:29 Uhr schrieb Boris <bb@kervyn.de>:
Hi Casey, already did restart all RGW instances. Only helped for 2 minutes. We now stopped the new site.
I will remove and recreate it later. As twi other sites don't have the problem I currently think I made a mistake in the process.
Mit freundlichen Grüßen - Boris Behrens
Am 20.06.2023 um 18:30 schrieb Casey Bodley <cbodley@redhat.com>:
hi Boris,
we've been investigating reports of excessive polling from metadata sync. i just opened https://tracker.ceph.com/issues/61743 to track this. restarting the secondary zone radosgws should help as a temporary workaround
On Tue, Jun 20, 2023 at 5:57 AM Boris Behrens <bb@kervyn.de> wrote:
Hi, yesterday I added a new zonegroup and it looks like it seems to cycle over the same requests over and over again.
In the log of the main zone I see these requests: 2023-06-20T09:48:37.979+0000 7f8941fb3700 1 beast: 0x7f8a602f3700: fd00:2380:0:24::136 - - [2023-06-20T09:48:37.979941+0000] "GET
/admin/log?type=metadata&id=62&period=e8fc96f1-ae86-4dc1-b432-470b0772fded&max-entries=100&&rgwx-zonegroup=b39392eb-75f8-47f0-b4f3-7d3882930b26
HTTP/1.1" 200 44 - - -
Only thing that changes is the &id.
We have two other zonegroups that are configured identical (ceph.conf and period) and these don;t seem to spam the main rgw.
root@host:~# radosgw-admin sync status realm 5d6f2ea4-b84a-459b-bce2-bccac338b3ef (main) zonegroup b39392eb-75f8-47f0-b4f3-7d3882930b26 (dc3) zone 96f5eca9-425b-4194-a152-86e310e91ddb (dc3) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master
root@host:~# radosgw-admin period get { "id": "e8fc96f1-ae86-4dc1-b432-470b0772fded", "epoch": 92, "predecessor_uuid": "5349ac85-3d6d-4088-993f-7a1d4be3835a", "sync_status": [ "", ... "" ], "period_map": { "id": "e8fc96f1-ae86-4dc1-b432-470b0772fded", "zonegroups": [ { "id": "b39392eb-75f8-47f0-b4f3-7d3882930b26", "name": "dc3", "api_name": "dc3", "is_master": "false", "endpoints": [ ], "hostnames": [ ], "hostnames_s3website": [ ], "master_zone": "96f5eca9-425b-4194-a152-86e310e91ddb", "zones": [ { "id": "96f5eca9-425b-4194-a152-86e310e91ddb", "name": "dc3", "endpoints": [ ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "5d6f2ea4-b84a-459b-bce2-bccac338b3ef", "sync_policy": { "groups": [] } }, ...
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
I've update the dc3 site from octopus to pacific and the problem is still there. I find it very weird that in only happens from one single zonegroup to the master and not from the other two. Am Mi., 21. Juni 2023 um 01:59 Uhr schrieb Boris Behrens <bb@kervyn.de>:
I recreated the site and the problem still persists.
I've upped the logging and saw this for a lot of buckets (i've stopped the debug log after some seconds). 2023-06-20T23:32:29.365+0000 7fcaab7fe700 20 get_system_obj_state: rctx=0x7fcaab7f9320 obj=dc3.rgw.meta:root:s3bucket-fra2 state=0x7fcba05ac0a0 s->prefetch_data=0 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache get: name=dc3.rgw.meta+root+s3bucket-fra2 : miss 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache put: name=dc3.rgw.meta+root+s3bucket-fra2 info.flags=0x6 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 adding dc3.rgw.meta+root+s3bucket-fra2 to cache LRU end 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache get: name=dc3.rgw.meta+root+s3bucket-fra2 : type miss (requested=0x1, cached=0x6) 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache put: name=dc3.rgw.meta+root+s3bucket-fra2 info.flags=0x1 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 moving dc3.rgw.meta+root+s3bucket-fra2 to cache LRU end 2023-06-20T23:32:29.365+0000 7fcaab7fe700 20 get_system_obj_state: rctx=0x7fcaab7f9320 obj=dc3.rgw.meta:root:.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 state=0x7fcba43ce0a0 s->prefetch_data=0 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache get: name=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 : miss 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache put: name=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 info.flags=0x16 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 adding dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 to cache LRU end 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache get: name=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 : type miss (requested=0x13, cached=0x16) 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache put: name=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 info.flags=0x13 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 moving dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 to cache LRU end 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 chain_cache_entry: cache_locator=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29
Am Di., 20. Juni 2023 um 19:29 Uhr schrieb Boris <bb@kervyn.de>:
Hi Casey, already did restart all RGW instances. Only helped for 2 minutes. We now stopped the new site.
I will remove and recreate it later. As twi other sites don't have the problem I currently think I made a mistake in the process.
Mit freundlichen Grüßen - Boris Behrens
Am 20.06.2023 um 18:30 schrieb Casey Bodley <cbodley@redhat.com>:
hi Boris,
we've been investigating reports of excessive polling from metadata sync. i just opened https://tracker.ceph.com/issues/61743 to track this. restarting the secondary zone radosgws should help as a temporary workaround
On Tue, Jun 20, 2023 at 5:57 AM Boris Behrens <bb@kervyn.de> wrote:
Hi, yesterday I added a new zonegroup and it looks like it seems to cycle over the same requests over and over again.
In the log of the main zone I see these requests: 2023-06-20T09:48:37.979+0000 7f8941fb3700 1 beast: 0x7f8a602f3700: fd00:2380:0:24::136 - - [2023-06-20T09:48:37.979941+0000] "GET
/admin/log?type=metadata&id=62&period=e8fc96f1-ae86-4dc1-b432-470b0772fded&max-entries=100&&rgwx-zonegroup=b39392eb-75f8-47f0-b4f3-7d3882930b26
HTTP/1.1" 200 44 - - -
Only thing that changes is the &id.
We have two other zonegroups that are configured identical (ceph.conf and period) and these don;t seem to spam the main rgw.
root@host:~# radosgw-admin sync status realm 5d6f2ea4-b84a-459b-bce2-bccac338b3ef (main) zonegroup b39392eb-75f8-47f0-b4f3-7d3882930b26 (dc3) zone 96f5eca9-425b-4194-a152-86e310e91ddb (dc3) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master
root@host:~# radosgw-admin period get { "id": "e8fc96f1-ae86-4dc1-b432-470b0772fded", "epoch": 92, "predecessor_uuid": "5349ac85-3d6d-4088-993f-7a1d4be3835a", "sync_status": [ "", ... "" ], "period_map": { "id": "e8fc96f1-ae86-4dc1-b432-470b0772fded", "zonegroups": [ { "id": "b39392eb-75f8-47f0-b4f3-7d3882930b26", "name": "dc3", "api_name": "dc3", "is_master": "false", "endpoints": [ ], "hostnames": [ ], "hostnames_s3website": [ ], "master_zone": "96f5eca9-425b-4194-a152-86e310e91ddb", "zones": [ { "id": "96f5eca9-425b-4194-a152-86e310e91ddb", "name": "dc3", "endpoints": [ ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "5d6f2ea4-b84a-459b-bce2-bccac338b3ef", "sync_policy": { "groups": [] } }, ...
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
Are there any ideas how to work with this? We disabled the logging so we do not run our of diskspace, but the rgw daemon still requires A LOT of cpu because of this. Am Mi., 21. Juni 2023 um 10:45 Uhr schrieb Boris Behrens <bb@kervyn.de>:
I've update the dc3 site from octopus to pacific and the problem is still there. I find it very weird that in only happens from one single zonegroup to the master and not from the other two.
Am Mi., 21. Juni 2023 um 01:59 Uhr schrieb Boris Behrens <bb@kervyn.de>:
I recreated the site and the problem still persists.
I've upped the logging and saw this for a lot of buckets (i've stopped the debug log after some seconds). 2023-06-20T23:32:29.365+0000 7fcaab7fe700 20 get_system_obj_state: rctx=0x7fcaab7f9320 obj=dc3.rgw.meta:root:s3bucket-fra2 state=0x7fcba05ac0a0 s->prefetch_data=0 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache get: name=dc3.rgw.meta+root+s3bucket-fra2 : miss 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache put: name=dc3.rgw.meta+root+s3bucket-fra2 info.flags=0x6 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 adding dc3.rgw.meta+root+s3bucket-fra2 to cache LRU end 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache get: name=dc3.rgw.meta+root+s3bucket-fra2 : type miss (requested=0x1, cached=0x6) 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache put: name=dc3.rgw.meta+root+s3bucket-fra2 info.flags=0x1 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 moving dc3.rgw.meta+root+s3bucket-fra2 to cache LRU end 2023-06-20T23:32:29.365+0000 7fcaab7fe700 20 get_system_obj_state: rctx=0x7fcaab7f9320 obj=dc3.rgw.meta:root:.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 state=0x7fcba43ce0a0 s->prefetch_data=0 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache get: name=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 : miss 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache put: name=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 info.flags=0x16 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 adding dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 to cache LRU end 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache get: name=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 : type miss (requested=0x13, cached=0x16) 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 cache put: name=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 info.flags=0x13 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 moving dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29 to cache LRU end 2023-06-20T23:32:29.365+0000 7fcaab7fe700 10 chain_cache_entry: cache_locator=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29
Am Di., 20. Juni 2023 um 19:29 Uhr schrieb Boris <bb@kervyn.de>:
Hi Casey, already did restart all RGW instances. Only helped for 2 minutes. We now stopped the new site.
I will remove and recreate it later. As twi other sites don't have the problem I currently think I made a mistake in the process.
Mit freundlichen Grüßen - Boris Behrens
Am 20.06.2023 um 18:30 schrieb Casey Bodley <cbodley@redhat.com>:
hi Boris,
we've been investigating reports of excessive polling from metadata sync. i just opened https://tracker.ceph.com/issues/61743 to track this. restarting the secondary zone radosgws should help as a temporary workaround
On Tue, Jun 20, 2023 at 5:57 AM Boris Behrens <bb@kervyn.de> wrote:
Hi, yesterday I added a new zonegroup and it looks like it seems to cycle over the same requests over and over again.
In the log of the main zone I see these requests: 2023-06-20T09:48:37.979+0000 7f8941fb3700 1 beast: 0x7f8a602f3700: fd00:2380:0:24::136 - - [2023-06-20T09:48:37.979941+0000] "GET
/admin/log?type=metadata&id=62&period=e8fc96f1-ae86-4dc1-b432-470b0772fded&max-entries=100&&rgwx-zonegroup=b39392eb-75f8-47f0-b4f3-7d3882930b26
HTTP/1.1" 200 44 - - -
Only thing that changes is the &id.
We have two other zonegroups that are configured identical (ceph.conf and period) and these don;t seem to spam the main rgw.
root@host:~# radosgw-admin sync status realm 5d6f2ea4-b84a-459b-bce2-bccac338b3ef (main) zonegroup b39392eb-75f8-47f0-b4f3-7d3882930b26 (dc3) zone 96f5eca9-425b-4194-a152-86e310e91ddb (dc3) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master
root@host:~# radosgw-admin period get { "id": "e8fc96f1-ae86-4dc1-b432-470b0772fded", "epoch": 92, "predecessor_uuid": "5349ac85-3d6d-4088-993f-7a1d4be3835a", "sync_status": [ "", ... "" ], "period_map": { "id": "e8fc96f1-ae86-4dc1-b432-470b0772fded", "zonegroups": [ { "id": "b39392eb-75f8-47f0-b4f3-7d3882930b26", "name": "dc3", "api_name": "dc3", "is_master": "false", "endpoints": [ ], "hostnames": [ ], "hostnames_s3website": [ ], "master_zone": "96f5eca9-425b-4194-a152-86e310e91ddb", "zones": [ { "id": "96f5eca9-425b-4194-a152-86e310e91ddb", "name": "dc3", "endpoints": [ ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "5d6f2ea4-b84a-459b-bce2-bccac338b3ef", "sync_policy": { "groups": [] } }, ...
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
participants (3)
-
Boris
-
Boris Behrens
-
Casey Bodley