Hi, I have been setting up a new cluster with a combination of cephadm and ceph orch. I have run in to a problem with rgw daemons that does not start. I have been following the documentation: https://docs.ceph.com/en/latest/cephadm/install/ - the RGW section ceph orch apply rgw ikea cn-dc9-1 --port=8080 --placement="3 itcnchn-bb4065 itcnchn-bb4066 itcnchn-bb4067" Then wait for everything to settle and then rgw's was expected to start on nodes itcnchn-bb4065 itcnchn-bb4066 itcnchn-bb4067 but that was not the case. ceph orch ps itcnchn-bb4065 NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID alertmanager.itcnchn-bb4065 itcnchn-bb4065 running (3w) 2m ago 4w 0.20.0 prom/alertmanager:v0.20.0 0881eb8f169f c595826b8867 crash.itcnchn-bb4065 itcnchn-bb4065 running (4w) 2m ago 4w 15.2.4 docker.io/ceph/ceph:v15 852b28cb10de 4598d37bf91c grafana.itcnchn-bb4065 itcnchn-bb4065 running (4w) 2m ago 4w 6.6.2 ceph/ceph-grafana:latest 87a51ecf0b1c ad8e16d12ef8 mgr.itcnchn-bb4065.jxtcer itcnchn-bb4065 running (4w) 2m ago 4w 15.2.4 docker.io/ceph/ceph:v15 852b28cb10de d9278a7e772c mon.itcnchn-bb4065 itcnchn-bb4065 running (4w) 2m ago 4w 15.2.4 docker.io/ceph/ceph:v15 852b28cb10de e0edd4a9038a prometheus.itcnchn-bb4065 itcnchn-bb4065 running (3w) 2m ago 4w 2.18.1 prom/prometheus:v2.18.1 de242295e225 1103b62e191b rgw.ikea.cn-dc9-1.itcnchn-bb4065.xpwict itcnchn-bb4065 error 2m ago 22h <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> I have logged into itcnchn-bb4065 and tried to start the rgw container my self, the command from (/var/lib/ceph/0aba4c3a-f735-11ea-af33-02423ee04865/rgw.ikea.cn-dc9-1.itcnchn-bb4065.xpwict/unit.run) /usr/bin/docker run --rm --net=host --ipc=host --name ceph-0aba4c3a-f735-11ea-af33-02423ee04865-rgw.ikea.cn-dc9-1.itcnchn-bb4065.xpwict -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e NODE_NAME=itcnchn-bb4065 -v /var/run/ceph/0aba4c3a-f735-11ea-af33-02423ee04865:/var/run/ceph:z -v /var/log/ceph/0aba4c3a-f735-11ea-af33-02423ee04865:/var/log/ceph:z -v /var/lib/ceph/0aba4c3a-f735-11ea-af33-02423ee04865/crash:/var/lib/ceph/crash:z -v /var/lib/ceph/0aba4c3a-f735-11ea-af33-02423ee04865/rgw.ikea.cn-dc9-1.itcnchn-bb4065.xpwict:/var/lib/ceph/radosgw/ceph-rgw.ikea.cn-dc9-1.itcnchn-bb4065.xpwict:z -v /var/lib/ceph/0aba4c3a-f735-11ea-af33-02423ee04865/rgw.ikea.cn-dc9-1.itcnchn-bb4065.xpwict/config:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/radosgw docker.io/ceph/ceph:v15 -n client.rgw.ikea.cn-dc9-1.itcnchn-bb4065.xpwict -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix="debug " debug 2020-10-14T06:55:54.835+0000 7f87075b1280 0 deferred set uid:gid to 167:167 (ceph:ceph) debug 2020-10-14T06:55:54.835+0000 7f87075b1280 0 ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable), process radosgw, pid 1 debug 2020-10-14T06:55:54.835+0000 7f87075b1280 0 framework: beast debug 2020-10-14T06:55:54.835+0000 7f87075b1280 0 framework conf key: port, val: 8080 debug 2020-10-14T06:55:54.835+0000 7f87075b1280 1 radosgw_Main not setting numa affinity debug 2020-10-14T06:55:54.875+0000 7f87075b1280 1 Cannot find zone id= (name=cn-dc9-1), switching to local zonegroup configuration debug 2020-10-14T06:55:54.875+0000 7f87075b1280 -1 Cannot find zone id= (name=cn-dc9-1) debug 2020-10-14T06:55:54.875+0000 7f87075b1280 0 ERROR: failed to start notify service ((22) Invalid argument debug 2020-10-14T06:55:54.875+0000 7f87075b1280 0 ERROR: failed to init services (ret=(22) Invalid argument) debug 2020-10-14T06:55:54.879+0000 7f87075b1280 -1 Couldn't init storage provider (RADOS) I am not really sure how to get past this point. Any pointers / help would be very appreciated. Thanks, - Karsten