Hi,
Were at 14.2.2
We just found a broken bucket index, trying to repair with the common
commands ]# radosgw-admin bucket check --check-objects fix finish
instantly, but bucket should have near 60-70TB info.
[root@CEPH-MON01 home]# radosgw-admin bucket check --check-objects --bucket
BUCKETNAME --debug_rgw=10
2019-09-05 20:08:13.908 7fdc8db37580 10 cannot find current period
zonegroup using local zonegroup
2019-09-05 20:08:13.910 7fdc8db37580 10 Cannot find current period zone
using local zone
2019-09-05 20:08:13.933 7fdc8db37580 2 all 8 watchers are set, enabling
cache
2019-09-05 20:08:13.942 7fdc56ffd700 2
RGWDataChangesLog::ChangesRenewThread: start
2019-09-05 20:08:13.948 7fdc8db37580 10 cache get:
name=default.rgw.data.root++BUCKETNAME : miss
2019-09-05 20:08:13.950 7fdc8db37580 10 cache put:
name=default.rgw.data.root++BUCKETNAME info.flags=0x16
2019-09-05 20:08:13.950 7fdc8db37580 10 adding
default.rgw.data.root++BUCKETNAME to cache LRU end
2019-09-05 20:08:13.950 7fdc8db37580 10 updating xattr:
name=ceph.objclass.version bl.length()=42
2019-09-05 20:08:13.950 7fdc8db37580 10 cache get:
name=default.rgw.data.root++BUCKETNAME : type miss (requested=0x11,
cached=0x16)
2019-09-05 20:08:13.950 7fdc8db37580 10 cache put:
name=default.rgw.data.root++BUCKETNAME info.flags=0x11
2019-09-05 20:08:13.950 7fdc8db37580 10 moving
default.rgw.data.root++BUCKETNAME to cache LRU end
2019-09-05 20:08:13.950 7fdc8db37580 10 cache get:
name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-
fdc16f590a82.16313306.1 : miss
2019-09-05 20:08:13.951 7fdc8db37580 10 cache put:
name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-
fdc16f590a82.16313306.1 info.flags=0x16
2019-09-05 20:08:13.951 7fdc8db37580 10 adding
default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-fdc16
f590a82.16313306.1 to cache LRU end
2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr:
name=ceph.objclass.version bl.length()=42
2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=user.rgw.acl
bl.length()=145
2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=user.rgw.lc
bl.length()=467
2019-09-05 20:08:13.951 7fdc8db37580 10 cache get:
name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-
fdc16f590a82.16313306.1 : type miss (requested=0x13, cached=0x16)
2019-09-05 20:08:13.951 7fdc8db37580 10 cache put:
name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-
fdc16f590a82.16313306.1 info.flags=0x13
2019-09-05 20:08:13.951 7fdc8db37580 10 moving
default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-fdc16
f590a82.16313306.1 to cache LRU end
2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr:
name=ceph.objclass.version bl.length()=42
2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=user.rgw.acl
bl.length()=145
2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=user.rgw.lc
bl.length()=467
2019-09-05 20:08:13.951 7fdc8db37580 10 chain_cache_entry:
cache_locator=default.rgw.data.root++BUCKETNAME
2019-09-05 20:08:13.951 7fdc8db37580 10 chain_cache_entry:
cache_locator=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4
fe0-bbe4-fdc16f590a82.16313306.1
2019-09-05 20:08:13.951 7fdc8db37580 10 cache get:
name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-
fdc16f590a82.16313306.1 : type miss (requested=0x16, cached=0x13)
2019-09-05 20:08:13.952 7fdc8db37580 10 cache put:
name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-
fdc16f590a82.16313306.1 info.flags=0x16
2019-09-05 20:08:13.952 7fdc8db37580 10 moving
default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-fdc16
f590a82.16313306.1 to cache LRU end
2019-09-05 20:08:13.952 7fdc8db37580 10 updating xattr:
name=ceph.objclass.version bl.length()=42
2019-09-05 20:08:13.952 7fdc8db37580 10 updating xattr: name=user.rgw.acl
bl.length()=145
2019-09-05 20:08:13.952 7fdc8db37580 10 updating xattr: name=user.rgw.lc
bl.length()=467
2019-09-05 20:08:13.952 7fdc8db37580 10 cache get:
name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-
fdc16f590a82.16313306.1 : hit (requested=0x11, cached=0x17)
2019-09-05 20:08:13.952 7fdc8db37580 10 cls_bucket_list_ordered
BUCKETNAME[48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3856921.7] start
_multipart_[] num_entries 1001
2019-09-05 20:08:13.960 7fdc8db37580 2 removed watcher, disabling cache
Any recomendations for forcé recreate bucket index?
Regards
Manuel
Show replies by date
Checking the shards, bucket got 64.
But shard 48efb8c3-693c-4fe0-bbe4-fdc16f590a82.16313306.1.1 seems missing.
Radosgw-admin fix wont recreate again, any recommendation? Maybe change
shard numbers to lower?
Regards
Manuel
De: EDH - Manuel Rios Fernandez <mriosfer(a)easydatahost.com>
Enviado el: jueves, 5 de septiembre de 2019 21:19
Para: 'ceph-users' <ceph-users(a)ceph.io>
Asunto: [ceph-users] RGW bucket check --check-objects -fix failed
Hi,
Were at 14.2.2
We just found a broken bucket index, trying to repair with the common
commands ]# radosgw-admin bucket check --check-objects fix finish
instantly, but bucket should have near 60-70TB info.
[root@CEPH-MON01 home]# radosgw-admin bucket check --check-objects --bucket
BUCKETNAME --debug_rgw=10
2019-09-05 20:08:13.908 7fdc8db37580 10 cannot find current period
zonegroup using local zonegroup
2019-09-05 20:08:13.910 7fdc8db37580 10 Cannot find current period zone
using local zone
2019-09-05 20:08:13.933 7fdc8db37580 2 all 8 watchers are set, enabling
cache
2019-09-05 20:08:13.942 7fdc56ffd700 2
RGWDataChangesLog::ChangesRenewThread: start
2019-09-05 20:08:13.948 7fdc8db37580 10 cache get:
name=default.rgw.data.root++BUCKETNAME : miss
2019-09-05 20:08:13.950 7fdc8db37580 10 cache put:
name=default.rgw.data.root++BUCKETNAME info.flags=0x16
2019-09-05 20:08:13.950 7fdc8db37580 10 adding
default.rgw.data.root++BUCKETNAME to cache LRU end
2019-09-05 20:08:13.950 7fdc8db37580 10 updating xattr:
name=ceph.objclass.version bl.length()=42
2019-09-05 20:08:13.950 7fdc8db37580 10 cache get:
name=default.rgw.data.root++BUCKETNAME : type miss (requested=0x11,
cached=0x16)
2019-09-05 20:08:13.950 7fdc8db37580 10 cache put:
name=default.rgw.data.root++BUCKETNAME info.flags=0x11
2019-09-05 20:08:13.950 7fdc8db37580 10 moving
default.rgw.data.root++BUCKETNAME to cache LRU end
2019-09-05 20:08:13.950 7fdc8db37580 10 cache get:
name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-
fdc16f590a82.16313306.1 : miss
2019-09-05 20:08:13.951 7fdc8db37580 10 cache put:
name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-
fdc16f590a82.16313306.1 info.flags=0x16
2019-09-05 20:08:13.951 7fdc8db37580 10 adding
default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-fdc16
f590a82.16313306.1 to cache LRU end
2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr:
name=ceph.objclass.version bl.length()=42
2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=user.rgw.acl
bl.length()=145
2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=user.rgw.lc
bl.length()=467
2019-09-05 20:08:13.951 7fdc8db37580 10 cache get:
name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-
fdc16f590a82.16313306.1 : type miss (requested=0x13, cached=0x16)
2019-09-05 20:08:13.951 7fdc8db37580 10 cache put:
name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-
fdc16f590a82.16313306.1 info.flags=0x13
2019-09-05 20:08:13.951 7fdc8db37580 10 moving
default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-fdc16
f590a82.16313306.1 to cache LRU end
2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr:
name=ceph.objclass.version bl.length()=42
2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=user.rgw.acl
bl.length()=145
2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=user.rgw.lc
bl.length()=467
2019-09-05 20:08:13.951 7fdc8db37580 10 chain_cache_entry:
cache_locator=default.rgw.data.root++BUCKETNAME
2019-09-05 20:08:13.951 7fdc8db37580 10 chain_cache_entry:
cache_locator=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4
fe0-bbe4-fdc16f590a82.16313306.1
2019-09-05 20:08:13.951 7fdc8db37580 10 cache get:
name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-
fdc16f590a82.16313306.1 : type miss (requested=0x16, cached=0x13)
2019-09-05 20:08:13.952 7fdc8db37580 10 cache put:
name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-
fdc16f590a82.16313306.1 info.flags=0x16
2019-09-05 20:08:13.952 7fdc8db37580 10 moving
default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-fdc16
f590a82.16313306.1 to cache LRU end
2019-09-05 20:08:13.952 7fdc8db37580 10 updating xattr:
name=ceph.objclass.version bl.length()=42
2019-09-05 20:08:13.952 7fdc8db37580 10 updating xattr: name=user.rgw.acl
bl.length()=145
2019-09-05 20:08:13.952 7fdc8db37580 10 updating xattr: name=user.rgw.lc
bl.length()=467
2019-09-05 20:08:13.952 7fdc8db37580 10 cache get:
name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-
fdc16f590a82.16313306.1 : hit (requested=0x11, cached=0x17)
2019-09-05 20:08:13.952 7fdc8db37580 10 cls_bucket_list_ordered
BUCKETNAME[48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3856921.7] start
_multipart_[] num_entries 1001
2019-09-05 20:08:13.960 7fdc8db37580 2 removed watcher, disabling cache
Any recomendations for forcé recreate bucket index?
Regards
Manuel