Hello. I've a RGW bucket (versioning=on). And there was objects like this: radosgw-admin object stat --bucket=xdir --object=f5492238-50cb-4bc2-93fa-424869018946 { "name": "f5492238-50cb-4bc2-93fa-424869018946", "size": 0, "tag": "", "attrs": { "user.rgw.manifest": "", "user.rgw.olh.idtag": "5rs3x0qh152tn0j865k8ybo9xqy92qjn", "user.rgw.olh.info": "\u0001\u0001�", "user.rgw.olh.pending.00000000607c87b5pgo03tvm3sqt23i9": "\u0001\u0001\u0008", "user.rgw.olh.pending.00000000607c87b5pyv13ugk3fadvxw7": "\u0001\u0001\u0008", "user.rgw.olh.pending.00000000607c87b5qic02n0e54zsjkax": "\u0001\u0001\u0008", "user.rgw.olh.ver": "3" } } I'm not sure but I suspect that these objects maybe leftover from unfinished multisite sync. I've removed the zone and made it master (because I have to for something else) and I've created a newbucket with (versioning=off) and copied all objects from old rgw bucket to a new rgw bucket with "rclone" cmd: "rclone copy --files-from "object.list" old:bucket new:bucket --no-traverse -vv --progress --fast-list --no-check-dest --no-update-modtime" Config: [bucket] type = s3 provider = Ceph env_auth = false acl = private bucket = access_key_id = secret_access_key = endpoint = After the copy I've checked every object via "radosgw-admin object stat object=$i" and there was still these problematic objects. I've tried to overwrite these objects from the Backup-bucket at different cluster (these objects are fine) and I tried to be sure everything written as expected via object stat again and these pending object was gone and everything was ok. All of them was OK! After a while I started to see this warning in radosgw.log when the client GET or HEAD the objects. 2021-04-19 11:37:50.230 7f2d917eb700 1 ====== starting new request req=0x55a44414a710 ===== 2021-04-19 11:37:50.230 7f2d917eb700 0 WARNING: couldn't find acl header for object, generating default 2021-04-19 11:37:50.230 7f2d917eb700 1 ====== req done req=0x55a44414a710 op status=0 http_status=200 latency=0s ====== 2021-04-19 11:37:50.230 7f2d917eb700 1 beast: 0x55a44414a710: 10.10.10.1 - - [2021-04-19 11:37:50.0.230489s] "HEAD /xdir/f5492238-50cb-4bc2-93fa-424869018946 HTTP/1.1" 200 0 - "aws-sdk-java/1.11.638 Linux/3.10.0-1160.11.1.el7.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.281-b09 java/1.8.0_281 groovy/2.5.6 vendor/Oracle_Corporation" - These objects are problematic objects and somehow the overwritten objects is gone and these objects left after the WARNING. (versioning is still =off --> multisite = off --> the cluster alone in zonegroup and its masterzone) But I've checked every overwritten object via "radosgw-admin object stat object=$i" and there was all OK!!! How the hell they're gone and others came back? HOW? What should I do? Maybe "object rm" "log trim" and re-write again?