RGW failed to start after upgrade to pacific
Hi all, I upgraded to the pacific version with cephadm. However, all our RGW daemons cannot start anymore. any help is appreciated. Here are the logs when starting RGW. I set debug_rados and debug_rgw to 20/20 systemd[1]: Started Ceph rgw.smil.b7-1.gpu006.twfefs for e88d509a-f6fc-11ea-b25d-a0423f3ac864. bash[9823]: WARNING: Error loading config file: .dockercfg: $HOME is not defined bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 0 deferred set uid:gid to 167:167 (ceph:ceph) bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process radosgw, pid 7 bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 0 framework: beast bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 0 framework conf key: port, val: 7480 bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 1 radosgw_Main not setting numa affinity bash[9823]: debug 2021-04-04T13:01:03.735+0000 7ff7f742b700 20 reqs_thread_entry: start bash[9823]: debug 2021-04-04T13:01:03.735+0000 7ff7f6c2a700 10 entry start bash[9823]: debug 2021-04-04T13:01:03.739+0000 7ff80f172440 1 librados: starting msgr at bash[9823]: debug 2021-04-04T13:01:03.739+0000 7ff80f172440 1 librados: starting objecter bash[9823]: debug 2021-04-04T13:01:03.739+0000 7ff80f172440 1 librados: setting wanted keys bash[9823]: debug 2021-04-04T13:01:03.739+0000 7ff80f172440 1 librados: calling monclient init bash[9823]: debug 2021-04-04T13:01:03.743+0000 7ff80f172440 1 librados: init done bash[9823]: debug 2021-04-04T13:01:03.743+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.743+0000 7ff80f172440 10 librados: wait_for_osdmap waiting bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: wait_for_osdmap done waiting bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: read oid=realms_names.smil nspace= bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=46 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: read oid=realms.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace= bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=104 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 realm smil 6c92e1e6-c08b-4f4c-b211-7899893978fa bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: read oid=realms.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace= bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=104 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: read oid=periods.5528a87b-bca7-4936-b348-fcb8f213df20.latest_epoch nspace= bash[9823]: debug 2021-04-04T13:01:03.751+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.751+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=10 bash[9823]: debug 2021-04-04T13:01:03.751+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.751+0000 7ff80f172440 10 librados: read oid=periods.5528a87b-bca7-4936-b348-fcb8f213df20.1 nspace= bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=877 bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 20 current period 5528a87b-bca7-4936-b348-fcb8f213df20 bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 10 librados: read oid=converted nspace= bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: Objecter returned from read r=-2 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados_obj.operate() r=-2 bl.length=0 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: read oid=realms_names.smil nspace= bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=46 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: read oid=realms.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace= bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=104 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: read oid=default.region nspace= bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: Objecter returned from read r=-2 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados_obj.operate() r=-2 bl.length=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got default.zonegroup.6c92e1e6-c08b-4f4c-b211-7899893978fa bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got zonegroup_info.c9c9b732-ec1b-474a-afd6-fe95b8a6ac20 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got default.realm bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got periods.bd67eb14-7081-49ab-bfe5-718b29257410.1 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got zone_info.ceb42b3a-cf4b-493d-94f9-beb7f3a76f99 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got realms.6c92e1e6-c08b-4f4c-b211-7899893978fa.control bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got periods.5528a87b-bca7-4936-b348-fcb8f213df20.1 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got zone_names.b7-1 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got periods.6c92e1e6-c08b-4f4c-b211-7899893978fa:staging bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got realms.6c92e1e6-c08b-4f4c-b211-7899893978fa bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got default.zone.6c92e1e6-c08b-4f4c-b211-7899893978fa bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got periods.bd67eb14-7081-49ab-bfe5-718b29257410.latest_epoch bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got periods.5528a87b-bca7-4936-b348-fcb8f213df20.latest_epoch bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got realms_names.smil bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got period_config.6c92e1e6-c08b-4f4c-b211-7899893978fa bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got zonegroups_names.default bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read oid=zone_names.default nspace= bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned from read r=-2 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=-2 bl.length=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read oid=zonegroups_names.default nspace= bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=46 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read oid=zonegroup_info.c9c9b732-ec1b-474a-afd6-fe95b8a6ac20 nspace= bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=403 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read oid=region_map nspace= bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned from read r=-2 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=-2 bl.length=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read oid=realms_names.smil nspace= bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=46 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read oid=realms.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace= bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=104 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read oid=default.zonegroup.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace= bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=46 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read oid=zonegroup_info.c9c9b732-ec1b-474a-afd6-fe95b8a6ac20 nspace= bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=403 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 period zonegroup init ret 0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 period zonegroup name default bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 using current period zonegroup default bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read oid=zone_names.b7-1 nspace= bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=46 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read oid=zone_info.ceb42b3a-cf4b-493d-94f9-beb7f3a76f99 nspace= bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=784 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 Cannot find current period zone using local zone bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read oid=zone_info.ceb42b3a-cf4b-493d-94f9-beb7f3a76f99 nspace= bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=784 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 zone b7-1 found bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: call oid=bucket.sync-source-hints. nspace= bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=-2 bl.length=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: call oid=bucket.sync-target-hints. nspace= bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 20 rados_obj.operate() r=-2 bl.length=0 bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 20 started sync module instance, tier type = bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 20 started zone id=ceb42b3a-cf4b-493d-94f9-beb7f3a76f99 (name=b7-1) with tier type = bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 10 librados: create oid=notify.0 nspace= bash[9823]: debug 2021-04-04T13:01:04.215+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.219+0000 7ff80f172440 10 librados: create oid=notify.1 nspace= bash[9823]: debug 2021-04-04T13:01:04.255+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.255+0000 7ff80f172440 10 librados: create oid=notify.2 nspace= bash[9823]: debug 2021-04-04T13:01:04.307+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.307+0000 7ff80f172440 10 librados: create oid=notify.3 nspace= bash[9823]: debug 2021-04-04T13:01:04.367+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.367+0000 7ff80f172440 10 librados: create oid=notify.4 nspace= bash[9823]: debug 2021-04-04T13:01:04.423+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.423+0000 7ff80f172440 10 librados: create oid=notify.5 nspace= bash[9823]: debug 2021-04-04T13:01:04.431+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.431+0000 7ff80f172440 10 librados: create oid=notify.6 nspace= bash[9823]: debug 2021-04-04T13:01:04.495+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.495+0000 7ff80f172440 10 librados: create oid=notify.7 nspace= bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=0 bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=1 bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=2 bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=3 bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=4 bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=5 bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 20 add_watcher() i=6 bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 20 add_watcher() i=7 bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 2 all 8 watchers are set, enabling cache bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: call oid=data_log.0 nspace= bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: Objecter returned from call r=0 bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: call oid=data_log.1 nspace= bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: call oid=data_log.2 nspace= bash[9823]: debug 2021-04-04T13:01:04.531+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:04.531+0000 7ff80f172440 10 librados: call oid=data_log.3 nspace= bash[9823]: debug 2021-04-04T13:01:04.531+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 ... bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: call oid=data_log.127 nspace= bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: call oid=data_log.0 nspace= bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: Objecter returned from call r=-61 bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: call oid=data_log.1 nspace= bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: call oid=data_log.2 nspace= bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: call oid=data_log.3 nspace= bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 ... bash[9823]: debug 2021-04-04T13:01:04.959+0000 7ff80f172440 10 librados: call oid=data_log.127 nspace= bash[9823]: debug 2021-04-04T13:01:04.959+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:04.959+0000 7ff80f172440 10 librados: call oid=data_log.0 nspace= bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 10 librados: Objecter returned from call r=-5 bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 -1 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*, optional_yield, std::optional<rados::cls::fifo::objv>, std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925 create_meta failed: r=-5 bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 -1 int RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*, librados::v14_2_0::Rados*): Error when starting backend: Input/output error bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 0 ERROR: failed to start datalog_rados service ((5) Input/output error bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 0 ERROR: failed to init services (ret=(5) Input/output error) bash[9823]: debug 2021-04-04T13:01:05.107+0000 7ff80f172440 10 librados: watch_flush enter bash[9823]: debug 2021-04-04T13:01:05.107+0000 7ff80f172440 10 librados: watch_flush exit bash[9823]: debug 2021-04-04T13:01:05.107+0000 7ff80f172440 20 remove_watcher() i=0 bash[9823]: debug 2021-04-04T13:01:05.107+0000 7ff80f172440 2 removed watcher, disabling cache bash[9823]: debug 2021-04-04T13:01:05.131+0000 7ff80f172440 10 librados: watch_flush enter bash[9823]: debug 2021-04-04T13:01:05.131+0000 7ff80f172440 10 librados: watch_flush exit bash[9823]: debug 2021-04-04T13:01:05.131+0000 7ff80f172440 20 remove_watcher() i=1 bash[9823]: debug 2021-04-04T13:01:05.159+0000 7ff80f172440 10 librados: watch_flush enter bash[9823]: debug 2021-04-04T13:01:05.159+0000 7ff80f172440 10 librados: watch_flush exit ... bash[9823]: debug 2021-04-04T13:01:05.275+0000 7ff80f172440 20 remove_watcher() i=7 bash[9823]: debug 2021-04-04T13:01:05.275+0000 7ff80f172440 10 librados: watch_flush enter bash[9823]: debug 2021-04-04T13:01:05.275+0000 7ff80f172440 10 librados: watch_flush exit bash[9823]: debug 2021-04-04T13:01:05.275+0000 7ff80f172440 1 librados: shutdown bash[9823]: debug 2021-04-04T13:01:05.279+0000 7ff80f172440 -1 Couldn't init storage provider (RADOS) systemd[1]: ceph-e88d509a-f6fc-11ea-b25d-a0423f3ac864@rgw.smil.b7-1.gpu006.twfefs.service: Main process exited, code=exited, status=5/NOTINSTALLED systemd[1]: ceph-e88d509a-f6fc-11ea-b25d-a0423f3ac864@rgw.smil.b7-1.gpu006.twfefs.service: Failed with result 'exit-code'. systemd[1]: ceph-e88d509a-f6fc-11ea-b25d-a0423f3ac864@rgw.smil.b7-1.gpu006.twfefs.service: Service hold-off time over, scheduling restart. systemd[1]: ceph-e88d509a-f6fc-11ea-b25d-a0423f3ac864@rgw.smil.b7-1.gpu006.twfefs.service: Scheduled restart job, restart counter is at 4. systemd[1]: Stopped Ceph rgw.smil.b7-1.gpu006.twfefs for e88d509a-f6fc-11ea-b25d-a0423f3ac864. Weiwen Hu
Hi, Am 04.04.21 um 15:22 schrieb 胡 玮文:
bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 -1 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*, optional_yield, std::optional<rados::cls::fifo::objv>, std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925 create_meta failed: r=-5 bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 -1 int RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*, librados::v14_2_0::Rados*): Error when starting backend: Input/output error bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 0 ERROR: failed to start datalog_rados service ((5) Input/output error bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 0 ERROR: failed to init services (ret=(5) Input/output error)
I see the same issues on an upgraded cluster. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
Hello, we see same problems. Deleting all the pools and redeploy rgw solved it on that test cluster, however that is no solution for production ;) systemd[1]: Started Ceph rados gateway. radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 deferred set uid:gid to 167:167 (ceph:ceph) radosgw[7171]: failed to chown /dev/null: (30) Read-only file system radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 ceph version 16.2.0-31-g5922b2b9c1 (5922b2b9c17f0877f84b0b3f2557ab72a628cbfe) pacific (stable), process radosgw, pid 7171 radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework: beast radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework conf key: ssl_port, val: 443 radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework conf key: port, val: 80 radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework conf key: ssl_certificate, val: /etc/ceph/rgwcert.pem radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 1 radosgw_Main not setting numa affinity radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*, optional_yield, std::optional<rados::cls::fifo::objv>, std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925 create_meta failed: r=-5 radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*, optional_yield, std::optional<rados::cls::fifo::objv>, std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925 create_meta failed: r=-5 radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 int RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*, librados::v14_2_0::Rados*): Error when starting backend: Input/output error radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 0 ERROR: failed to start datalog_rados service ((5) Input/output error radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 int RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*, librados::v14_2_0::Rados*): Error when starting backend: Input/output error radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 0 ERROR: failed to init services (ret=(5) Input/output error) radosgw[7171]: 2021-04-04T14:37:51.700+0000 7fc6641efc00 -1 Couldn't init storage provider (RADOS) radosgw[7171]: 2021-04-04T14:37:51.700+0000 7fc6641efc00 -1 Couldn't init storage provider (RADOS) systemd[1]: ceph-radosgw@rgw.new-croit-host-C0DE01.service: Main process exited, code=exited, status=5/NOTINSTALLED systemd[1]: ceph-radosgw@rgw.new-croit-host-C0DE01.service: Unit entered failed state. systemd[1]: ceph-radosgw@rgw.new-croit-host-C0DE01.service: Failed with result 'exit-code'. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx On Mon, 5 Apr 2021 at 19:59, Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,
Am 04.04.21 um 15:22 schrieb 胡 玮文:
bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 -1 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*, optional_yield, std::optional<rados::cls::fifo::objv>, std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925 create_meta failed: r=-5 bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 -1 int RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*, librados::v14_2_0::Rados*): Error when starting backend: Input/output error bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 0 ERROR: failed to start datalog_rados service ((5) Input/output error bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 0 ERROR: failed to init services (ret=(5) Input/output error)
I see the same issues on an upgraded cluster.
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
http://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
thanks for the details. this is a regression from changes to the datalog storage for multisite - this -5 error is coming from the new 'fifo' backend. as a workaround, you can set the new 'rgw_data_log_backing' config variable back to 'omap' Adam has fixes already merged to the pacific branch; be aware that the first pacific point release will change the name of 'rgw_data_log_backing' to 'rgw_default_data_log_backing' and default back to 'fifo' On Tue, Apr 6, 2021 at 2:37 AM Martin Verges <martin.verges@croit.io> wrote:
Hello,
we see same problems. Deleting all the pools and redeploy rgw solved it on that test cluster, however that is no solution for production ;)
systemd[1]: Started Ceph rados gateway. radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 deferred set uid:gid to 167:167 (ceph:ceph) radosgw[7171]: failed to chown /dev/null: (30) Read-only file system radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 ceph version 16.2.0-31-g5922b2b9c1 (5922b2b9c17f0877f84b0b3f2557ab72a628cbfe) pacific (stable), process radosgw, pid 7171 radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework: beast radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework conf key: ssl_port, val: 443 radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework conf key: port, val: 80 radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework conf key: ssl_certificate, val: /etc/ceph/rgwcert.pem radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 1 radosgw_Main not setting numa affinity radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*, optional_yield, std::optional<rados::cls::fifo::objv>, std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925 create_meta failed: r=-5 radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*, optional_yield, std::optional<rados::cls::fifo::objv>, std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925 create_meta failed: r=-5 radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 int RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*, librados::v14_2_0::Rados*): Error when starting backend: Input/output error radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 0 ERROR: failed to start datalog_rados service ((5) Input/output error radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 int RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*, librados::v14_2_0::Rados*): Error when starting backend: Input/output error radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 0 ERROR: failed to init services (ret=(5) Input/output error) radosgw[7171]: 2021-04-04T14:37:51.700+0000 7fc6641efc00 -1 Couldn't init storage provider (RADOS) radosgw[7171]: 2021-04-04T14:37:51.700+0000 7fc6641efc00 -1 Couldn't init storage provider (RADOS) systemd[1]: ceph-radosgw@rgw.new-croit-host-C0DE01.service: Main process exited, code=exited, status=5/NOTINSTALLED systemd[1]: ceph-radosgw@rgw.new-croit-host-C0DE01.service: Unit entered failed state. systemd[1]: ceph-radosgw@rgw.new-croit-host-C0DE01.service: Failed with result 'exit-code'.
-- Martin Verges Managing director
Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io YouTube: https://goo.gl/PGE1Bx
-- Martin Verges Managing director
Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io YouTube: https://goo.gl/PGE1Bx
On Mon, 5 Apr 2021 at 19:59, Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,
Am 04.04.21 um 15:22 schrieb 胡 玮文:
bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 -1 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*, optional_yield, std::optional<rados::cls::fifo::objv>, std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925 create_meta failed: r=-5 bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 -1 int RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*, librados::v14_2_0::Rados*): Error when starting backend: Input/output error bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 0 ERROR: failed to start datalog_rados service ((5) Input/output error bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 0 ERROR: failed to init services (ret=(5) Input/output error)
I see the same issues on an upgraded cluster.
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
http://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Am 06.04.21 um 18:53 schrieb Casey Bodley:
thanks for the details. this is a regression from changes to the datalog storage for multisite - this -5 error is coming from the new 'fifo' backend. as a workaround, you can set the new 'rgw_data_log_backing' config variable back to 'omap'
Adam has fixes already merged to the pacific branch; be aware that the first pacific point release will change the name of 'rgw_data_log_backing' to 'rgw_default_data_log_backing' and default back to 'fifo'
So when you have a Ceph cluster with Rados-Gateways you should not upgrade to Pacific currently. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
participants (4)
-
Casey Bodley
-
Martin Verges
-
Robert Sander
-
胡 玮文