Hi, I am trying to set up a new cluster with cephadm using a docker backend. The initial boot strap did not finish cleanly and it errored out waiting for the mon-ip, I used the command: cephadm bootstrap --mon-ip 192.168.0.1 With 192.168.0.1 being the ip address of this first host. I tried the command again but it failed as the new ceph node was actually running so it could not bind to the ports. After a bit of searching I was able to use "sudo cephadm shell —“ commands to change the username and password for the dashboard and login to it. I then used cephadm to add a new host with "sudo cephadm shell — ceph orch host add host2” Now in the inventory of the dashboard, and "ceph orch device ls” only devices on host2 are listed not host1. In the Cluster/Hosts section of the dashboard host1 has its root volume drive listed in devices, and host2 has the root volume drive and drive for the OSD listed. I successfully added an OSD with a drive on host2, trying the same command adjusted for host1 I get the following in the log: Dec 23 08:55:47 localhost systemd[1]: var-lib-docker-overlay2-91e9dffa86c333353dd6b445021c852d7ce8da6237d0d4d95909d68ef3d4fe23\x2dinit-merged.mount: Succeeded. Dec 23 08:55:47 localhost systemd[24638]: var-lib-docker-overlay2-91e9dffa86c333353dd6b445021c852d7ce8da6237d0d4d95909d68ef3d4fe23\x2dinit-merged.mount: Succeeded. Dec 23 08:55:47 localhost containerd[1470]: time="2020-12-23T08:55:47.369773808Z" level=info msg="shim containerd-shim started" address=/containerd-shim/80f876072532ebebdfef341a5c793654e27766f2d1708991a6f25599b24b6557.sock debug=false pid=28597 Dec 23 08:55:47 localhost bash[8745]: debug 2020-12-23T08:55:47.517+0000 ffff73d7a200 1 mon.host1@0(leader).osd e12 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 71303168 full_alloc: 71303168 kv_alloc: 876609536 Dec 23 08:55:47 localhost containerd[1470]: time="2020-12-23T08:55:47.621748606Z" level=info msg="shim reaped" id=69a786e4a61605c1e6eca5a6e0e5ed0900635a214b0f1c96a4f26ea7911a12ff Dec 23 08:55:47 localhost dockerd[2930]: time="2020-12-23T08:55:47.631479207Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete" Dec 23 08:55:47 localhost systemd[24638]: var-lib-docker-overlay2-91e9dffa86c333353dd6b445021c852d7ce8da6237d0d4d95909d68ef3d4fe23-merged.mount: Succeeded. Dec 23 08:55:47 localhost systemd[1]: var-lib-docker-overlay2-91e9dffa86c333353dd6b445021c852d7ce8da6237d0d4d95909d68ef3d4fe23-merged.mount: Succeeded. Dec 23 08:55:47 localhost systemd[24638]: var-lib-docker-overlay2-64bb135bc0cdab187566992dc9870068dee1430062e1a2b484381c19e03da895\x2dinit-merged.mount: Succeeded. Dec 23 08:55:47 localhost systemd[1]: var-lib-docker-overlay2-64bb135bc0cdab187566992dc9870068dee1430062e1a2b484381c19e03da895\x2dinit-merged.mount: Succeeded. Dec 23 08:55:47 localhost containerd[1470]: time="2020-12-23T08:55:47.972437378Z" level=info msg="shim containerd-shim started" address=/containerd-shim/4a61d63e1f46722ffa7a950c31145d167c5c69087d003e5928a6aa3a4831f031.sock debug=false pid=28659 Dec 23 08:55:48 localhost bash[8745]: cluster 2020-12-23T08:55:46.892633+0000 mgr.host1.kkssvi (mgr.24098) 24278 : cluster [DBG] pgmap v24212: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 931 GiB / 932 GiB avail Dec 23 08:55:48 localhost bash[8756]: debug 2020-12-23T08:55:48.889+0000 ffff93573700 0 log_channel(cluster) log [DBG] : pgmap v24213: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 931 GiB / 932 GiB avail Dec 23 08:55:49 localhost bash[8756]: debug 2020-12-23T08:55:49.085+0000 ffff9056f700 0 log_channel(audit) log [DBG] : from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "host1:/dev/nvme0n1", "target": ["mon-mgr", ""]}]: dispatch Dec 23 08:55:49 localhost bash[8745]: debug 2020-12-23T08:55:49.085+0000 ffff71575200 0 mon.host1@0(leader) e2 handle_command mon_command({"prefix": "osd tree", "states": ["destroyed"], "format": "json"} v 0) v1 Dec 23 08:55:49 localhost bash[8745]: debug 2020-12-23T08:55:49.085+0000 ffff71575200 0 log_channel(audit) log [DBG] : from='mgr.24098 192.168.0.1:0/2486989775' entity='mgr.host1.kkssvi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch Dec 23 08:55:49 localhost bash[8756]: debug 2020-12-23T08:55:49.089+0000 ffff8ed6d700 0 log_channel(cephadm) log [INF] : Found osd claims -> {} Dec 23 08:55:49 localhost bash[8756]: debug 2020-12-23T08:55:49.089+0000 ffff8ed6d700 0 log_channel(cephadm) log [INF] : Found osd claims for drivegroup None -> {} Dec 23 08:55:49 localhost containerd[1470]: time="2020-12-23T08:55:49.331868093Z" level=info msg="shim reaped" id=780a38dd49fce4a823c4c3d834abdd1cc17bbe0c0aa4f2dd7caeddf8dce1708e Dec 23 08:55:49 localhost dockerd[2930]: time="2020-12-23T08:55:49.341765820Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete" Dec 23 08:55:49 localhost systemd[24638]: var-lib-docker-overlay2-64bb135bc0cdab187566992dc9870068dee1430062e1a2b484381c19e03da895-merged.mount: Succeeded. Dec 23 08:55:49 localhost systemd[1]: var-lib-docker-overlay2-64bb135bc0cdab187566992dc9870068dee1430062e1a2b484381c19e03da895-merged.mount: Succeeded. Dec 23 08:55:49 localhost bash[8745]: audit 2020-12-23T08:55:49.091014+0000 mon.host1 (mon.0) 1093 : audit [DBG] from='mgr.24098 192.168.0.1:0/2486989775' entity='mgr.host1.kkssvi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch Dec 23 08:55:50 localhost bash[8745]: cluster 2020-12-23T08:55:48.893433+0000 mgr.host1.kkssvi (mgr.24098) 24279 : cluster [DBG] pgmap v24213: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 931 GiB / 932 GiB avail Dec 23 08:55:50 localhost bash[8745]: audit 2020-12-23T08:55:49.087597+0000 mgr.host1.kkssvi (mgr.24098) 24280 : audit [DBG] from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "host1:/dev/nvme0n1", "target": ["mon-mgr", ""]}]: dispatch Dec 23 08:55:50 localhost bash[8745]: cephadm 2020-12-23T08:55:49.093552+0000 mgr.host1.kkssvi (mgr.24098) 24281 : cephadm [INF] Found osd claims -> {} Dec 23 08:55:50 localhost bash[8745]: cephadm 2020-12-23T08:55:49.093933+0000 mgr.host1.kkssvi (mgr.24098) 24282 : cephadm [INF] Found osd claims for drivegroup None -> {} The other problem is logging is set to debug for both hosts, I tried "sudo cephadm shell -- ceph daemon mon.host1 config set mon_cluster_log_file_level info” which reports success but logging remains at debug level. If I try the same command with mon.host2 I get INFO:cephadm:Inferring fsid ae111111-1111-1111-1111-f1111a11111a INFO:cephadm:Inferring config /var/lib/ceph/ae147088-4486-11eb-9044-f1337a55707a/mon.host1/config INFO:cephadm:Using recent ceph image ceph/ceph:v15 admin_socket: exception getting command descriptions: [Errno 2] No such file or directory Which looks like it is trying to use the config for host1 on host2? Thanks, Duncan