Hi, according to [1] those are non-fatal errors:

These are the same non-fatal errors from above, where 404 errors from other zones are converted to -ENOENT. The RGWBackoffControlCR will continue to poll these objects for changes. These ERROR messages are unnecessarily spammy though, so I'd be in favor of removing them.

Can you show the sync status? Have you tried a full sync? Regards, Eugen [1] https://www.spinics.net/lists/ceph-users/msg53477.html Zitat von Huy Nguyen <viplanghe6@gmail.com>:

Hi, Currently I'm testing replication ability between 2 zones in the same cluster. But only metadata is synced, not the data. I checked endpoints, system_key,... all good. If anyone have any idea, please guide me to resolve this situation. Thanks

The radosgw shows this log on both side, primary and secondary:

rgw rados thread: ERROR: failed to run sync ... RGW-SYNC:data:sync:shard[54]: ERROR: failed to read remote data log info: ret=-5 RGW-SYNC:data:sync:shard[90]: ERROR: failed to read remote data log info: ret=-5 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -5 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -5

Here is my configure (it might be long):

zonegroup hn which contains hn-1 and hn-2, each zone have its own endpoint and radosgw instance on two different node: { "id": "36c29ef1-332b-4d6e-a341-ecc9cc2ef281", "epoch": 4, "predecessor_uuid": "e1981515-7355-4029-b26c-6be3de943be0", "sync_status": [], "period_map": { "id": "36c29ef1-332b-4d6e-a341-ecc9cc2ef281", "zonegroups": [ { "id": "9ced5b3b-5322-4ec9-862c-521bfff85c91", "name": "hn", "api_name": "hn", "is_master": true, "endpoints": [ "http://10.3.55.66:8080" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "2760e65c-b9d1-415a-929e-f5de2098d5cf", "zones": [ { "id": "0c5aac15-c899-46aa-bff9-353bf17f9602", "name": "hn-2", "endpoints": [ "http://10.3.53.107:8080" ], "log_meta": false, "log_data": true, "bucket_index_max_shards": 11, "read_only": false, "tier_type": "", "sync_from_all": true, "sync_from": [], "redirect_zone": "", "supported_features": [ "compress-encrypted", "resharding" ] }, { "id": "2760e65c-b9d1-415a-929e-f5de2098d5cf", "name": "hn-1", "endpoints": [ "http://10.3.55.66:8080" ], "log_meta": false, "log_data": true, "bucket_index_max_shards": 11, "read_only": false, "tier_type": "", "sync_from_all": true, "sync_from": [], "redirect_zone": "", "supported_features": [ "compress-encrypted", "resharding" ] } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "cb7337a4-5b38-41de-9763-ef7e22a79e17", "sync_policy": { "groups": [] }, "enabled_features": [ "resharding" ] } ], "short_zone_ids": [ { "key": "0c5aac15-c899-46aa-bff9-353bf17f9602", "val": 3534260942 }, { "key": "2760e65c-b9d1-415a-929e-f5de2098d5cf", "val": 1656529937 } ] }, "master_zonegroup": "9ced5b3b-5322-4ec9-862c-521bfff85c91", "master_zone": "2760e65c-b9d1-415a-929e-f5de2098d5cf", "period_config": { "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, ...ratelimit... }, "realm_id": "cb7337a4-5b38-41de-9763-ef7e22a79e17", "realm_name": "ss-realm", "realm_epoch": 2 }

zone hn-1:

{ "id": "2760e65c-b9d1-415a-929e-f5de2098d5cf", "name": "hn-1", "domain_root": "hn-1.rgw.meta:root", "control_pool": "hn-1.rgw.control", "gc_pool": "hn-1.rgw.log:gc", "lc_pool": "hn-1.rgw.log:lc", "log_pool": "hn-1.rgw.log", "intent_log_pool": "hn-1.rgw.log:intent", "usage_log_pool": "hn-1.rgw.log:usage", "roles_pool": "hn-1.rgw.meta:roles", "reshard_pool": "hn-1.rgw.log:reshard", "user_keys_pool": "hn-1.rgw.meta:users.keys", "user_email_pool": "hn-1.rgw.meta:users.email", "user_swift_pool": "hn-1.rgw.meta:users.swift", "user_uid_pool": "hn-1.rgw.meta:users.uid", "otp_pool": "hn-1.rgw.otp", "system_key": { "access_key": "LIW8M2OTOCOHA41OJUGO", "secret_key": "KxD3fIWrW5GMpaY1ynCp10j2ccCP3fkfCcgyNBDT" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "hn-1.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "hn-1.rgw.buckets.data" } }, "data_extra_pool": "hn-1.rgw.buckets.non-ec", "index_type": 0, "inline_data": true } } ], "realm_id": "cb7337a4-5b38-41de-9763-ef7e22a79e17", "notif_pool": "hn-1.rgw.log:notif" }

zone hn-2:

{ "id": "0c5aac15-c899-46aa-bff9-353bf17f9602", "name": "hn-2", "domain_root": "hn-2.rgw.meta:root", "control_pool": "hn-2.rgw.control", "gc_pool": "hn-2.rgw.log:gc", "lc_pool": "hn-2.rgw.log:lc", "log_pool": "hn-2.rgw.log", "intent_log_pool": "hn-2.rgw.log:intent", "usage_log_pool": "hn-2.rgw.log:usage", "roles_pool": "hn-2.rgw.meta:roles", "reshard_pool": "hn-2.rgw.log:reshard", "user_keys_pool": "hn-2.rgw.meta:users.keys", "user_email_pool": "hn-2.rgw.meta:users.email", "user_swift_pool": "hn-2.rgw.meta:users.swift", "user_uid_pool": "hn-2.rgw.meta:users.uid", "otp_pool": "hn-2.rgw.otp", "system_key": { "access_key": "LIW8M2OTOCOHA41OJUGO", "secret_key": "KxD3fIWrW5GMpaY1ynCp10j2ccCP3fkfCcgyNBDT" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "hn-2.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "hn-2.rgw.buckets.data" } }, "data_extra_pool": "hn-2.rgw.buckets.non-ec", "index_type": 0, "inline_data": true } } ], "realm_id": "cb7337a4-5b38-41de-9763-ef7e22a79e17", "notif_pool": "hn-2.rgw.log:notif" } _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io