4 Apr
2021
4 Apr
'21
2:22 p.m.
Hi all,
I upgraded to the pacific version with cephadm. However, all our RGW daemons cannot start
anymore. any help is appreciated.
Here are the logs when starting RGW. I set debug_rados and debug_rgw to 20/20
systemd[1]: Started Ceph rgw.smil.b7-1.gpu006.twfefs for
e88d509a-f6fc-11ea-b25d-a0423f3ac864.
bash[9823]: WARNING: Error loading config file: .dockercfg: $HOME is not defined
bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 0 deferred set uid:gid to
167:167 (ceph:ceph)
bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 0 ceph version 16.2.0
(0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process radosgw, pid 7
bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 0 framework: beast
bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 0 framework conf key: port,
val: 7480
bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 1 radosgw_Main not setting
numa affinity
bash[9823]: debug 2021-04-04T13:01:03.735+0000 7ff7f742b700 20 reqs_thread_entry: start
bash[9823]: debug 2021-04-04T13:01:03.735+0000 7ff7f6c2a700 10 entry start
bash[9823]: debug 2021-04-04T13:01:03.739+0000 7ff80f172440 1 librados: starting msgr at
bash[9823]: debug 2021-04-04T13:01:03.739+0000 7ff80f172440 1 librados: starting
objecter
bash[9823]: debug 2021-04-04T13:01:03.739+0000 7ff80f172440 1 librados: setting wanted
keys
bash[9823]: debug 2021-04-04T13:01:03.739+0000 7ff80f172440 1 librados: calling monclient
init
bash[9823]: debug 2021-04-04T13:01:03.743+0000 7ff80f172440 1 librados: init done
bash[9823]: debug 2021-04-04T13:01:03.743+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.743+0000 7ff80f172440 10 librados: wait_for_osdmap
waiting
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: wait_for_osdmap
done waiting
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: read
oid=realms_names.smil nspace=
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=46
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: read
oid=realms.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace=
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=104
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 realm smil
6c92e1e6-c08b-4f4c-b211-7899893978fa
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: read
oid=realms.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace=
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=104
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: read
oid=periods.5528a87b-bca7-4936-b348-fcb8f213df20.latest_epoch nspace=
bash[9823]: debug 2021-04-04T13:01:03.751+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.751+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=10
bash[9823]: debug 2021-04-04T13:01:03.751+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.751+0000 7ff80f172440 10 librados: read
oid=periods.5528a87b-bca7-4936-b348-fcb8f213df20.1 nspace=
bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=877
bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 20 current period
5528a87b-bca7-4936-b348-fcb8f213df20
bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 10 librados: read
oid=converted nspace=
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: Objecter returned
from read r=-2
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados_obj.operate() r=-2
bl.length=0
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: read
oid=realms_names.smil nspace=
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=46
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: read
oid=realms.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace=
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=104
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: read
oid=default.region nspace=
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: Objecter returned
from read r=-2
bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados_obj.operate() r=-2
bl.length=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
default.zonegroup.6c92e1e6-c08b-4f4c-b211-7899893978fa
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
zonegroup_info.c9c9b732-ec1b-474a-afd6-fe95b8a6ac20
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
default.realm
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
periods.bd67eb14-7081-49ab-bfe5-718b29257410.1
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
zone_info.ceb42b3a-cf4b-493d-94f9-beb7f3a76f99
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
realms.6c92e1e6-c08b-4f4c-b211-7899893978fa.control
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
periods.5528a87b-bca7-4936-b348-fcb8f213df20.1
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
zone_names.b7-1
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
periods.6c92e1e6-c08b-4f4c-b211-7899893978fa:staging
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
realms.6c92e1e6-c08b-4f4c-b211-7899893978fa
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
default.zone.6c92e1e6-c08b-4f4c-b211-7899893978fa
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
periods.bd67eb14-7081-49ab-bfe5-718b29257410.latest_epoch
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
periods.5528a87b-bca7-4936-b348-fcb8f213df20.latest_epoch
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
realms_names.smil
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
period_config.6c92e1e6-c08b-4f4c-b211-7899893978fa
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got
zonegroups_names.default
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read
oid=zone_names.default nspace=
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned
from read r=-2
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=-2
bl.length=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read
oid=zonegroups_names.default nspace=
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=46
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read
oid=zonegroup_info.c9c9b732-ec1b-474a-afd6-fe95b8a6ac20 nspace=
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=403
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read
oid=region_map nspace=
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned
from read r=-2
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=-2
bl.length=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read
oid=realms_names.smil nspace=
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=46
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read
oid=realms.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace=
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=104
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read
oid=default.zonegroup.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace=
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=46
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read
oid=zonegroup_info.c9c9b732-ec1b-474a-afd6-fe95b8a6ac20 nspace=
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=403
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 period zonegroup init ret
0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 period zonegroup name
default
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 using current period
zonegroup default
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read
oid=zone_names.b7-1 nspace=
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=46
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read
oid=zone_info.ceb42b3a-cf4b-493d-94f9-beb7f3a76f99 nspace=
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=784
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 Cannot find current period
zone using local zone
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read
oid=zone_info.ceb42b3a-cf4b-493d-94f9-beb7f3a76f99 nspace=
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned
from read r=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0
bl.length=784
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 zone b7-1 found
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: call
oid=bucket.sync-source-hints. nspace=
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned
from call r=-2
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=-2
bl.length=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0
bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: call
oid=bucket.sync-target-hints. nspace=
bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 10 librados: Objecter returned
from call r=-2
bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 20 rados_obj.operate() r=-2
bl.length=0
bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 20 started sync module
instance, tier type =
bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 20 started zone
id=ceb42b3a-cf4b-493d-94f9-beb7f3a76f99 (name=b7-1) with tier type =
bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 10 librados: create
oid=notify.0 nspace=
bash[9823]: debug 2021-04-04T13:01:04.215+0000 7ff80f172440 10 librados: Objecter returned
from create r=0
bash[9823]: debug 2021-04-04T13:01:04.219+0000 7ff80f172440 10 librados: create
oid=notify.1 nspace=
bash[9823]: debug 2021-04-04T13:01:04.255+0000 7ff80f172440 10 librados: Objecter returned
from create r=0
bash[9823]: debug 2021-04-04T13:01:04.255+0000 7ff80f172440 10 librados: create
oid=notify.2 nspace=
bash[9823]: debug 2021-04-04T13:01:04.307+0000 7ff80f172440 10 librados: Objecter returned
from create r=0
bash[9823]: debug 2021-04-04T13:01:04.307+0000 7ff80f172440 10 librados: create
oid=notify.3 nspace=
bash[9823]: debug 2021-04-04T13:01:04.367+0000 7ff80f172440 10 librados: Objecter returned
from create r=0
bash[9823]: debug 2021-04-04T13:01:04.367+0000 7ff80f172440 10 librados: create
oid=notify.4 nspace=
bash[9823]: debug 2021-04-04T13:01:04.423+0000 7ff80f172440 10 librados: Objecter returned
from create r=0
bash[9823]: debug 2021-04-04T13:01:04.423+0000 7ff80f172440 10 librados: create
oid=notify.5 nspace=
bash[9823]: debug 2021-04-04T13:01:04.431+0000 7ff80f172440 10 librados: Objecter returned
from create r=0
bash[9823]: debug 2021-04-04T13:01:04.431+0000 7ff80f172440 10 librados: create
oid=notify.6 nspace=
bash[9823]: debug 2021-04-04T13:01:04.495+0000 7ff80f172440 10 librados: Objecter returned
from create r=0
bash[9823]: debug 2021-04-04T13:01:04.495+0000 7ff80f172440 10 librados: create
oid=notify.7 nspace=
bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 10 librados: Objecter returned
from create r=0
bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=0
bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=1
bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=2
bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=3
bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=4
bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=5
bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 20 add_watcher() i=6
bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 20 add_watcher() i=7
bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 2 all 8 watchers are set,
enabling cache
bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: call
oid=data_log.0 nspace=
bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: Objecter returned
from call r=0
bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: call
oid=data_log.1 nspace=
bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: Objecter returned
from call r=-2
bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: call
oid=data_log.2 nspace=
bash[9823]: debug 2021-04-04T13:01:04.531+0000 7ff80f172440 10 librados: Objecter returned
from call r=-2
bash[9823]: debug 2021-04-04T13:01:04.531+0000 7ff80f172440 10 librados: call
oid=data_log.3 nspace=
bash[9823]: debug 2021-04-04T13:01:04.531+0000 7ff80f172440 10 librados: Objecter returned
from call r=-2
...
bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: call
oid=data_log.127 nspace=
bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: Objecter returned
from call r=-2
bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: call
oid=data_log.0 nspace=
bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: Objecter returned
from call r=-61
bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: call
oid=data_log.1 nspace=
bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: Objecter returned
from call r=-2
bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: call
oid=data_log.2 nspace=
bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: Objecter returned
from call r=-2
bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: call
oid=data_log.3 nspace=
bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: Objecter returned
from call r=-2
...
bash[9823]: debug 2021-04-04T13:01:04.959+0000 7ff80f172440 10 librados: call
oid=data_log.127 nspace=
bash[9823]: debug 2021-04-04T13:01:04.959+0000 7ff80f172440 10 librados: Objecter returned
from call r=-2
bash[9823]: debug 2021-04-04T13:01:04.959+0000 7ff80f172440 10 librados: call
oid=data_log.0 nspace=
bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 10 librados: Objecter returned
from call r=-5
bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 -1 static int
rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string,
std::unique_ptr<rgw::cls::fifo::FIFO>*, optional_yield,
std::optional<rados::cls::fifo::objv>,
std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925
create_meta failed: r=-5
bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 -1 int
RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*,
librados::v14_2_0::Rados*): Error when starting backend: Input/output error
bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 0 ERROR: failed to start
datalog_rados service ((5) Input/output error
bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 0 ERROR: failed to init
services (ret=(5) Input/output error)
bash[9823]: debug 2021-04-04T13:01:05.107+0000 7ff80f172440 10 librados: watch_flush
enter
bash[9823]: debug 2021-04-04T13:01:05.107+0000 7ff80f172440 10 librados: watch_flush exit
bash[9823]: debug 2021-04-04T13:01:05.107+0000 7ff80f172440 20 remove_watcher() i=0
bash[9823]: debug 2021-04-04T13:01:05.107+0000 7ff80f172440 2 removed watcher, disabling
cache
bash[9823]: debug 2021-04-04T13:01:05.131+0000 7ff80f172440 10 librados: watch_flush
enter
bash[9823]: debug 2021-04-04T13:01:05.131+0000 7ff80f172440 10 librados: watch_flush exit
bash[9823]: debug 2021-04-04T13:01:05.131+0000 7ff80f172440 20 remove_watcher() i=1
bash[9823]: debug 2021-04-04T13:01:05.159+0000 7ff80f172440 10 librados: watch_flush
enter
bash[9823]: debug 2021-04-04T13:01:05.159+0000 7ff80f172440 10 librados: watch_flush exit
...
bash[9823]: debug 2021-04-04T13:01:05.275+0000 7ff80f172440 20 remove_watcher() i=7
bash[9823]: debug 2021-04-04T13:01:05.275+0000 7ff80f172440 10 librados: watch_flush
enter
bash[9823]: debug 2021-04-04T13:01:05.275+0000 7ff80f172440 10 librados: watch_flush exit
bash[9823]: debug 2021-04-04T13:01:05.275+0000 7ff80f172440 1 librados: shutdown
bash[9823]: debug 2021-04-04T13:01:05.279+0000 7ff80f172440 -1 Couldn't init storage
provider (RADOS)
systemd[1]: ceph-e88d509a-f6fc-11ea-b25d-a0423f3ac864(a)rgw.smil.b7-1.gpu006.twfefs.service:
Main process exited, code=exited, status=5/NOTINSTALLED
systemd[1]: ceph-e88d509a-f6fc-11ea-b25d-a0423f3ac864(a)rgw.smil.b7-1.gpu006.twfefs.service:
Failed with result 'exit-code'.
systemd[1]: ceph-e88d509a-f6fc-11ea-b25d-a0423f3ac864(a)rgw.smil.b7-1.gpu006.twfefs.service:
Service hold-off time over, scheduling restart.
systemd[1]: ceph-e88d509a-f6fc-11ea-b25d-a0423f3ac864(a)rgw.smil.b7-1.gpu006.twfefs.service:
Scheduled restart job, restart counter is at 4.
systemd[1]: Stopped Ceph rgw.smil.b7-1.gpu006.twfefs for
e88d509a-f6fc-11ea-b25d-a0423f3ac864.
Weiwen Hu
5 Apr
5 Apr
6:58 p.m.
Hi,
Am 04.04.21 um 15:22 schrieb 胡 玮文:
bash[9823]: debug 2021-04-04T13:01:04.995+0000
7ff80f172440 -1 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx,
std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*, optional_yield,
std::optional<rados::cls::fifo::objv>,
std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925
create_meta failed: r=-5
bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 -1 int
RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*,
librados::v14_2_0::Rados*): Error when starting backend: Input/output error
bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 0 ERROR: failed to start
datalog_rados service ((5) Input/output error
bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 0 ERROR: failed to init
services (ret=(5) Input/output error)
I see the same issues on an upgraded cluster.
Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin
http://www.heinlein-support.de
Tel: 030 / 405051-43
Fax: 030 / 405051-19
Zwangsangaben lt. §35a GmbHG:
HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin
6 Apr
6 Apr
7:37 a.m.
Hello,
we see same problems. Deleting all the pools and redeploy rgw solved it on
that test cluster, however that is no solution for production ;)
systemd[1]: Started Ceph rados gateway.
radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 deferred set
uid:gid to 167:167 (ceph:ceph)
radosgw[7171]: failed to chown /dev/null: (30) Read-only file system
radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 ceph version
16.2.0-31-g5922b2b9c1 (5922b2b9c17f0877f84b0b3f2557ab72a628cbfe) pacific
(stable), process radosgw, pid 7171
radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework:
beast
radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework conf
key: ssl_port, val: 443
radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework conf
key: port, val: 80
radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework conf
key: ssl_certificate, val: /etc/ceph/rgwcert.pem
radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 1 radosgw_Main
not setting numa affinity
radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 static int
rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx,
std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*,
optional_yield, std::optional<rados::cls::fifo::objv>,
std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925
create_meta failed: r=-5
radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 static int
rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx,
std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*,
optional_yield, std::optional<rados::cls::fifo::objv>,
std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925
create_meta failed: r=-5
radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 int
RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*,
librados::v14_2_0::Rados*): Error when starting backend: Input/output error
radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 0 ERROR: failed
to start datalog_rados service ((5) Input/output error
radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 int
RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*,
librados::v14_2_0::Rados*): Error when starting backend: Input/output error
radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 0 ERROR: failed
to init services (ret=(5) Input/output error)
radosgw[7171]: 2021-04-04T14:37:51.700+0000 7fc6641efc00 -1 Couldn't init
storage provider (RADOS)
radosgw[7171]: 2021-04-04T14:37:51.700+0000 7fc6641efc00 -1 Couldn't init
storage provider (RADOS)
systemd[1]: ceph-radosgw(a)rgw.new-croit-host-C0DE01.service: Main process
exited, code=exited, status=5/NOTINSTALLED
systemd[1]: ceph-radosgw(a)rgw.new-croit-host-C0DE01.service: Unit entered
failed state.
systemd[1]: ceph-radosgw(a)rgw.new-croit-host-C0DE01.service: Failed with
result 'exit-code'.
--
Martin Verges
Managing director
Mobile: +49 174 9335695
E-Mail: martin.verges(a)croit.io
Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx
--
Martin Verges
Managing director
Mobile: +49 174 9335695
E-Mail: martin.verges(a)croit.io
Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx
On Mon, 5 Apr 2021 at 19:59, Robert Sander <r.sander(a)heinlein-support.de>
wrote:
Hi,
Am 04.04.21 um 15:22 schrieb 胡 玮文:
bash[9823]: debug 2021-04-04T13:01:04.995+0000
7ff80f172440 -1 static
int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx,
std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*,
optional_yield, std::optional<rados::cls::fifo::objv>,
std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925
create_meta failed: r=-5
bash[9823]: debug 2021-04-04T13:01:04.995+0000
7ff80f172440 -1 int
RGWDataChangesLog::start(const RGWZone*, const
RGWZoneParams&, RGWSI_Cls*,
librados::v14_2_0::Rados*): Error when starting backend: Input/output error
bash[9823]: debug 2021-04-04T13:01:04.995+0000
7ff80f172440 0 ERROR:
failed to start datalog_rados service ((5) Input/output
error
bash[9823]: debug 2021-04-04T13:01:04.995+0000
7ff80f172440 0 ERROR:
failed to init services (ret=(5) Input/output error)
I see the same issues on an upgraded cluster.
Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin
http://www.heinlein-support.de
Tel: 030 / 405051-43
Fax: 030 / 405051-19
Zwangsangaben lt. §35a GmbHG:
HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:53 p.m.
thanks for the details. this is a regression from changes to the
datalog storage for multisite - this -5 error is coming from the new
'fifo' backend. as a workaround, you can set the new
'rgw_data_log_backing' config variable back to 'omap'
Adam has fixes already merged to the pacific branch; be aware that the
first pacific point release will change the name of
'rgw_data_log_backing' to 'rgw_default_data_log_backing' and default
back to 'fifo'
On Tue, Apr 6, 2021 at 2:37 AM Martin Verges <martin.verges(a)croit.io> wrote:
Hello,
we see same problems. Deleting all the pools and redeploy rgw solved it on
that test cluster, however that is no solution for production ;)
systemd[1]: Started Ceph rados gateway.
radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 deferred set
uid:gid to 167:167 (ceph:ceph)
radosgw[7171]: failed to chown /dev/null: (30) Read-only file system
radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 ceph version
16.2.0-31-g5922b2b9c1 (5922b2b9c17f0877f84b0b3f2557ab72a628cbfe) pacific
(stable), process radosgw, pid 7171
radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework:
beast
radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework conf
key: ssl_port, val: 443
radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework conf
key: port, val: 80
radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework conf
key: ssl_certificate, val: /etc/ceph/rgwcert.pem
radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 1 radosgw_Main
not setting numa affinity
radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 static int
rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx,
std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*,
optional_yield, std::optional<rados::cls::fifo::objv>,
std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925
create_meta failed: r=-5
radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 static int
rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx,
std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*,
optional_yield, std::optional<rados::cls::fifo::objv>,
std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925
create_meta failed: r=-5
radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 int
RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*,
librados::v14_2_0::Rados*): Error when starting backend: Input/output error
radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 0 ERROR: failed
to start datalog_rados service ((5) Input/output error
radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 int
RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*,
librados::v14_2_0::Rados*): Error when starting backend: Input/output error
radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 0 ERROR: failed
to init services (ret=(5) Input/output error)
radosgw[7171]: 2021-04-04T14:37:51.700+0000 7fc6641efc00 -1 Couldn't init
storage provider (RADOS)
radosgw[7171]: 2021-04-04T14:37:51.700+0000 7fc6641efc00 -1 Couldn't init
storage provider (RADOS)
systemd[1]: ceph-radosgw(a)rgw.new-croit-host-C0DE01.service: Main process
exited, code=exited, status=5/NOTINSTALLED
systemd[1]: ceph-radosgw(a)rgw.new-croit-host-C0DE01.service: Unit entered
failed state.
systemd[1]: ceph-radosgw(a)rgw.new-croit-host-C0DE01.service: Failed with
result 'exit-code'.
--
Martin Verges
Managing director
Mobile: +49 174 9335695
E-Mail: martin.verges(a)croit.io
Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx
--
Martin Verges
Managing director
Mobile: +49 174 9335695
E-Mail: martin.verges(a)croit.io
Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx
On Mon, 5 Apr 2021 at 19:59, Robert Sander <r.sander(a)heinlein-support.de>
wrote:
Hi,
Am 04.04.21 um 15:22 schrieb 胡 玮文:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io bash[9823]: debug 2021-04-04T13:01:04.995+0000
7ff80f172440 -1 static
int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx,
std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*,
optional_yield, std::optional<rados::cls::fifo::objv>,
std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925
create_meta failed: r=-5
bash[9823]: debug 2021-04-04T13:01:04.995+0000
7ff80f172440 -1 int
RGWDataChangesLog::start(const RGWZone*, const
RGWZoneParams&, RGWSI_Cls*,
librados::v14_2_0::Rados*): Error when starting backend: Input/output error
bash[9823]: debug 2021-04-04T13:01:04.995+0000
7ff80f172440 0 ERROR:
failed to start datalog_rados service ((5) Input/output
error
bash[9823]: debug 2021-04-04T13:01:04.995+0000
7ff80f172440 0 ERROR:
failed to init services (ret=(5) Input/output error)
I see the same issues on an upgraded cluster.
Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin
http://www.heinlein-support.de
Tel: 030 / 405051-43
Fax: 030 / 405051-19
Zwangsangaben lt. §35a GmbHG:
HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
12 Apr
12 Apr
10:55 a.m.
Am 06.04.21 um 18:53 schrieb Casey Bodley:
thanks for the details. this is a regression from
changes to the
datalog storage for multisite - this -5 error is coming from the new
'fifo' backend. as a workaround, you can set the new
'rgw_data_log_backing' config variable back to 'omap'
Adam has fixes already merged to the pacific branch; be aware that the
first pacific point release will change the name of
'rgw_data_log_backing' to 'rgw_default_data_log_backing' and default
back to 'fifo'
So when you have a Ceph cluster with Rados-Gateways you should not
upgrade to Pacific currently.
Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin
http://www.heinlein-support.de
Tel: 030 / 405051-43
Fax: 030 / 405051-19
Zwangsangaben lt. §35a GmbHG:
HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin
1133
days inactive
1141
days old
4 comments
4 participants
participants (4)
-
Casey Bodley -
Martin Verges -
Robert Sander -
胡 玮文