4 Apr
2021
4 Apr
'21
2:22 p.m.
Hi all,
I upgraded to the pacific version with cephadm. However, all our RGW daemons cannot start
anymore. any help is appreciated.
Here are the logs when starting RGW. I set debug_rados and debug_rgw to 20/20
systemd[1]: Started Ceph rgw.smil.b7-1.gpu006.twfefs for
e88d509a-f6fc-11ea-b25d-a0423f3ac864.
bash[9823]: WARNING: Error loading config file: .dockercfg: $HOME is not defined
bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 0 deferred set uid:gid to
167:167 (ceph:ceph)
bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 0 ceph version 16.2.0
(0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process radosgw, pid 7
bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 0 framework: beast
bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 0 framework conf key: port,
val: 7480
bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 1 radosgw_Main not setting
numa affinity
bash[9823]: debug 2021-04-04T13:01:03.735+0000 7ff7f742b700 20 reqs_thread_entry: start
bash[9823]: debug 2021-04-04T13:01:03.735+0000 7ff7f6c2a700 10 entry start
bash[9823]: debug 2021-04-04T13:01:03.739+0000 7ff80f172440 1 librados: starting msgr at
bash[9823]: debug 2021-04-04T13:01:03.739+0000 7ff80f172440 1 librados: starting
objecter
bash[9823]: debug 2021-04-04T13:01:03.739+0000 7ff80f172440 1 librados: setting wanted
keys
bash[9823]: debug 2021-04-04T13:01:03.739+0000 7ff80f172440 1 librados: calling monclient
init
bash[9823]: debug 2021-04-04T13:01:03.743+0000 7ff80f172440 1 librados: init done
bash[9823]: debug 2021-04-04T13:01:03.743+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.743+0000 7ff80f172440 10 librados: wait_for_osdmap
waiting
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: wait_for_osdmap
done waiting
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: read
oid=realms_names.smil nspace=
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=46
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: read
oid=realms.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace=
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=104
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 realm smil
6c92e1e6-c08b-4f4c-b211-7899893978fa
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: read
oid=realms.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace=
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=104
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: read
oid=periods.5528a87b-bca7-4936-b348-fcb8f213df20.latest_epoch nspace=
bash[9823]: debug 2021-04-04T13:01:03.751+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.751+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=10
bash[9823]: debug 2021-04-04T13:01:03.751+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.751+0000 7ff80f172440 10 librados: read
oid=periods.5528a87b-bca7-4936-b348-fcb8f213df20.1 nspace=
bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=877
bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 20 current period
5528a87b-bca7-4936-b348-fcb8f213df20
bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 10 librados: read
oid=converted nspace=
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: Objecter returned
from read r=-2
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados_obj.operate() r=-2
bl.length=0
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: read
oid=realms_names.smil nspace=
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=46
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: read
oid=realms.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace=
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=104
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: read
oid=default.region nspace=
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: Objecter returned
from read r=-2
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados_obj.operate() r=-2
bl.length=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
default.zonegroup.6c92e1e6-c08b-4f4c-b211-7899893978fa
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
zonegroup_info.c9c9b732-ec1b-474a-afd6-fe95b8a6ac20
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
default.realm
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
periods.bd67eb14-7081-49ab-bfe5-718b29257410.1
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
zone_info.ceb42b3a-cf4b-493d-94f9-beb7f3a76f99
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
realms.6c92e1e6-c08b-4f4c-b211-7899893978fa.control
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
periods.5528a87b-bca7-4936-b348-fcb8f213df20.1
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
zone_names.b7-1
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
periods.6c92e1e6-c08b-4f4c-b211-7899893978fa:staging
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
realms.6c92e1e6-c08b-4f4c-b211-7899893978fa
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
default.zone.6c92e1e6-c08b-4f4c-b211-7899893978fa
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
periods.bd67eb14-7081-49ab-bfe5-718b29257410.latest_epoch
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
periods.5528a87b-bca7-4936-b348-fcb8f213df20.latest_epoch
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
realms_names.smil
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
period_config.6c92e1e6-c08b-4f4c-b211-7899893978fa
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
zonegroups_names.default
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read
oid=zone_names.default nspace=
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned
from read r=-2
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=-2
bl.length=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read
oid=zonegroups_names.default nspace=
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=46
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read
oid=zonegroup_info.c9c9b732-ec1b-474a-afd6-fe95b8a6ac20 nspace=
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=403
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read
oid=region_map nspace=
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned
from read r=-2
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=-2
bl.length=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read
oid=realms_names.smil nspace=
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=46
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read
oid=realms.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace=
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=104
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read
oid=default.zonegroup.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace=
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=46
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read
oid=zonegroup_info.c9c9b732-ec1b-474a-afd6-fe95b8a6ac20 nspace=
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=403
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 period zonegroup init ret
0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 period zonegroup name
default
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 using current period
zonegroup default
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read
oid=zone_names.b7-1 nspace=
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=46
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read
oid=zone_info.ceb42b3a-cf4b-493d-94f9-beb7f3a76f99 nspace=
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=784
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 Cannot find current period
zone using local zone
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read
oid=zone_info.ceb42b3a-cf4b-493d-94f9-beb7f3a76f99 nspace=
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=784
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 zone b7-1 found
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: call
oid=bucket.sync-source-hints. nspace=
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned
from call r=-2
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=-2
bl.length=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: call
oid=bucket.sync-target-hints. nspace=
bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 10 librados: Objecter returned
from call r=-2
bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 20 rados_obj.operate() r=-2
bl.length=0
bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 20 started sync module
instance, tier type =
bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 20 started zone
id=ceb42b3a-cf4b-493d-94f9-beb7f3a76f99 (name=b7-1) with tier type =
bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 10 librados: create
oid=notify.0 nspace=
bash[9823]: debug 2021-04-04T13:01:04.215+0000 7ff80f172440 10 librados: Objecter returned
from create r=0
bash[9823]: debug 2021-04-04T13:01:04.219+0000 7ff80f172440 10 librados: create
oid=notify.1 nspace=
bash[9823]: debug 2021-04-04T13:01:04.255+0000 7ff80f172440 10 librados: Objecter returned
from create r=0
bash[9823]: debug 2021-04-04T13:01:04.255+0000 7ff80f172440 10 librados: create
oid=notify.2 nspace=
bash[9823]: debug 2021-04-04T13:01:04.307+0000 7ff80f172440 10 librados: Objecter returned
from create r=0
bash[9823]: debug 2021-04-04T13:01:04.307+0000 7ff80f172440 10 librados: create
oid=notify.3 nspace=
bash[9823]: debug 2021-04-04T13:01:04.367+0000 7ff80f172440 10 librados: Objecter returned
from create r=0
bash[9823]: debug 2021-04-04T13:01:04.367+0000 7ff80f172440 10 librados: create
oid=notify.4 nspace=
bash[9823]: debug 2021-04-04T13:01:04.423+0000 7ff80f172440 10 librados: Objecter returned
from create r=0
bash[9823]: debug 2021-04-04T13:01:04.423+0000 7ff80f172440 10 librados: create
oid=notify.5 nspace=
bash[9823]: debug 2021-04-04T13:01:04.431+0000 7ff80f172440 10 librados: Objecter returned
from create r=0
bash[9823]: debug 2021-04-04T13:01:04.431+0000 7ff80f172440 10 librados: create
oid=notify.6 nspace=
bash[9823]: debug 2021-04-04T13:01:04.495+0000 7ff80f172440 10 librados: Objecter returned
from create r=0
bash[9823]: debug 2021-04-04T13:01:04.495+0000 7ff80f172440 10 librados: create
oid=notify.7 nspace=
bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 10 librados: Objecter returned
from create r=0
bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=0
bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=1
bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=2
bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=3
bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=4
bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=5
bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 20 add_watcher() i=6
bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 20 add_watcher() i=7
bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 2 all 8 watchers are set,
enabling cache
bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: call
oid=data_log.0 nspace=
bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: Objecter returned
from call r=0
bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: call
oid=data_log.1 nspace=
bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: Objecter returned
from call r=-2
bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: call
oid=data_log.2 nspace=
bash[9823]: debug 2021-04-04T13:01:04.531+0000 7ff80f172440 10 librados: Objecter returned
from call r=-2
bash[9823]: debug 2021-04-04T13:01:04.531+0000 7ff80f172440 10 librados: call
oid=data_log.3 nspace=
bash[9823]: debug 2021-04-04T13:01:04.531+0000 7ff80f172440 10 librados: Objecter returned
from call r=-2
...
bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: call
oid=data_log.127 nspace=
bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: Objecter returned
from call r=-2
bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: call
oid=data_log.0 nspace=
bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: Objecter returned
from call r=-61
bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: call
oid=data_log.1 nspace=
bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: Objecter returned
from call r=-2
bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: call
oid=data_log.2 nspace=
bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: Objecter returned
from call r=-2
bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: call
oid=data_log.3 nspace=
bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: Objecter returned
from call r=-2
...
bash[9823]: debug 2021-04-04T13:01:04.959+0000 7ff80f172440 10 librados: call
oid=data_log.127 nspace=
bash[9823]: debug 2021-04-04T13:01:04.959+0000 7ff80f172440 10 librados: Objecter returned
from call r=-2
bash[9823]: debug 2021-04-04T13:01:04.959+0000 7ff80f172440 10 librados: call
oid=data_log.0 nspace=
bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 10 librados: Objecter returned
from call r=-5
bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 -1 static int
rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string,
std::unique_ptr<rgw::cls::fifo::FIFO>*, optional_yield,
std::optional<rados::cls::fifo::objv>,
std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925
create_meta failed: r=-5
bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 -1 int
RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*,
librados::v14_2_0::Rados*): Error when starting backend: Input/output error
bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 0 ERROR: failed to start
datalog_rados service ((5) Input/output error
bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 0 ERROR: failed to init
services (ret=(5) Input/output error)
bash[9823]: debug 2021-04-04T13:01:05.107+0000 7ff80f172440 10 librados: watch_flush
enter
bash[9823]: debug 2021-04-04T13:01:05.107+0000 7ff80f172440 10 librados: watch_flush exit
bash[9823]: debug 2021-04-04T13:01:05.107+0000 7ff80f172440 20 remove_watcher() i=0
bash[9823]: debug 2021-04-04T13:01:05.107+0000 7ff80f172440 2 removed watcher, disabling
cache
bash[9823]: debug 2021-04-04T13:01:05.131+0000 7ff80f172440 10 librados: watch_flush
enter
bash[9823]: debug 2021-04-04T13:01:05.131+0000 7ff80f172440 10 librados: watch_flush exit
bash[9823]: debug 2021-04-04T13:01:05.131+0000 7ff80f172440 20 remove_watcher() i=1
bash[9823]: debug 2021-04-04T13:01:05.159+0000 7ff80f172440 10 librados: watch_flush
enter
bash[9823]: debug 2021-04-04T13:01:05.159+0000 7ff80f172440 10 librados: watch_flush exit
...
bash[9823]: debug 2021-04-04T13:01:05.275+0000 7ff80f172440 20 remove_watcher() i=7
bash[9823]: debug 2021-04-04T13:01:05.275+0000 7ff80f172440 10 librados: watch_flush
enter
bash[9823]: debug 2021-04-04T13:01:05.275+0000 7ff80f172440 10 librados: watch_flush exit
bash[9823]: debug 2021-04-04T13:01:05.275+0000 7ff80f172440 1 librados: shutdown
bash[9823]: debug 2021-04-04T13:01:05.279+0000 7ff80f172440 -1 Couldn't init storage
provider (RADOS)
systemd[1]: ceph-e88d509a-f6fc-11ea-b25d-a0423f3ac864(a)rgw.smil.b7-1.gpu006.twfefs.service:
Main process exited, code=exited, status=5/NOTINSTALLED
systemd[1]: ceph-e88d509a-f6fc-11ea-b25d-a0423f3ac864(a)rgw.smil.b7-1.gpu006.twfefs.service:
Failed with result 'exit-code'.
systemd[1]: ceph-e88d509a-f6fc-11ea-b25d-a0423f3ac864(a)rgw.smil.b7-1.gpu006.twfefs.service:
Service hold-off time over, scheduling restart.
systemd[1]: ceph-e88d509a-f6fc-11ea-b25d-a0423f3ac864(a)rgw.smil.b7-1.gpu006.twfefs.service:
Scheduled restart job, restart counter is at 4.
systemd[1]: Stopped Ceph rgw.smil.b7-1.gpu006.twfefs for
e88d509a-f6fc-11ea-b25d-a0423f3ac864.
Weiwen Hu