Hi,
I recently attempted to run the ‘rgw-orphan-list’ tool against our cluster (octopus
15.2.7) to identify any orphans and noticed that the 'radosgw-admin bucket radoslist’
command appeared to be stuck in a loop.
I saw in the 'radosgw-admin-XXXXXX.intermediate’ output file the same sequence of
objects looping repeatedly, and the command would not progress.
There is a tracker item that appears to be for this issue
(
https://tracker.ceph.com/issues/47074), but I believe it is still unverified whether this
is a bug or not.
I ran the command 'radosgw-admin bucket radoslist --bucket=<bucketname>
--debug-rgw=20 > radoslist-out.txt 2> radoslist-err.txt’ in order to identify what
was happening as per the tracker, but I would appreciate some help in analysing the logs.
In the radoslist-err.txt file, I see entries similar to the following, which just repeat
until I cancel the command:
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++():
rule->part_size=0 rules.size()=1
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result:
ofs=345930 stripe_ofs=345930 part_ofs=0 rule->part_size=0
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++():
rule->part_size=0 rules.size()=1
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result:
ofs=360383 stripe_ofs=360383 part_ofs=0 rule->part_size=0
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++():
rule->part_size=0 rules.size()=1
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result:
ofs=302704 stripe_ofs=302704 part_ofs=0 rule->part_size=0
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++():
rule->part_size=0 rules.size()=1
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result:
ofs=381794 stripe_ofs=381794 part_ofs=0 rule->part_size=0
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++():
rule->part_size=0 rules.size()=1
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result:
ofs=334269 stripe_ofs=334269 part_ofs=0 rule->part_size=0
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++():
rule->part_size=0 rules.size()=1
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result:
ofs=378182 stripe_ofs=378182 part_ofs=0 rule->part_size=0
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++():
rule->part_size=0 rules.size()=1
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result:
ofs=325041 stripe_ofs=325041 part_ofs=0 rule->part_size=0
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++():
rule->part_size=0 rules.size()=1
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result:
ofs=295118 stripe_ofs=295118 part_ofs=0 rule->part_size=0
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++():
rule->part_size=0 rules.size()=1
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result:
ofs=315823 stripe_ofs=315823 part_ofs=0 rule->part_size=0
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++():
rule->part_size=0 rules.size()=1
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result:
ofs=346112 stripe_ofs=346112 part_ofs=0 rule->part_size=0
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++():
rule->part_size=0 rules.size()=1
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result:
ofs=303128 stripe_ofs=303128 part_ofs=0 rule->part_size=0
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++():
rule->part_size=0 rules.size()=1
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result:
ofs=364011 stripe_ofs=364011 part_ofs=0 rule->part_size=0
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++():
rule->part_size=0 rules.size()=1
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result:
ofs=331137 stripe_ofs=331137 part_ofs=0 rule->part_size=0
2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++():
rule->part_size=0 rules.size()=1
Just before these entries begin, I see the following:
2020-12-04T12:37:22.203+0100 7fc405f79080 20 RGWRados::Bucket::List::list_objects_ordered
INFO end of outer loop, truncated=1, count=4, attempt=9
2020-12-04T12:37:22.203+0100 7fc405f79080 20 RGWRadosList::do_incomplete_multipart
processing incomplete multipart entry RGWMultipartUploadEntry{
obj.key="_multipart_/halon_2_settings_2020-08-28.json.2~TD-WUPWhOicUKkz4NTxZcxuEl3b_IqC.meta"
mp=RGWMPObj:{
prefix="/halon_2_settings_2020-08-28.json.2~TD-WUPWhOicUKkz4NTxZcxuEl3b_IqC",
meta="/halon_2_settings_2020-08-28.json.2~TD-WUPWhOicUKkz4NTxZcxuEl3b_IqC.meta"
} }
2020-12-04T12:37:22.203+0100 7fc405f79080 20 RGWRadosList::do_incomplete_multipart
processing incomplete multipart entry RGWMultipartUploadEntry{
obj.key="_multipart_gl-events_0_2020-08-28.json.2~OQWHwayxwHwUMLteZWQZWRnsdgT5Cl2.meta"
mp=RGWMPObj:{
prefix="gl-events_0_2020-08-28.json.2~OQWHwayxwHwUMLteZWQZWRnsdgT5Cl2",
meta="gl-events_0_2020-08-28.json.2~OQWHwayxwHwUMLteZWQZWRnsdgT5Cl2.meta" } }
2020-12-04T12:37:22.203+0100 7fc405f79080 20 RGWRadosList::do_incomplete_multipart
processing incomplete multipart entry RGWMultipartUploadEntry{
obj.key="_multipart_gl-events_0_2020-08-28.json.2~zz-6rqcuHGmSPYTY2IRueMP5HIFOL6t.meta"
mp=RGWMPObj:{
prefix="gl-events_0_2020-08-28.json.2~zz-6rqcuHGmSPYTY2IRueMP5HIFOL6t",
meta="gl-events_0_2020-08-28.json.2~zz-6rqcuHGmSPYTY2IRueMP5HIFOL6t.meta" } }
2020-12-04T12:37:22.203+0100 7fc405f79080 20 RGWRadosList::do_incomplete_multipart
processing incomplete multipart entry RGWMultipartUploadEntry{
obj.key="_multipart_gl-events_1_2020-08-28.json.2~-WCk4AJE1k7or-KN6qTePxfIvHL1NLf.meta"
mp=RGWMPObj:{
prefix="gl-events_1_2020-08-28.json.2~-WCk4AJE1k7or-KN6qTePxfIvHL1NLf",
meta="gl-events_1_2020-08-28.json.2~-WCk4AJE1k7or-KN6qTePxfIvHL1NLf.meta" } }
Is anyone able to interpret these logs and see an explanation as to why the command
appears to be looping over the same objects?
Thanks,
James.