The point I wanted to explore is the following: We have three nodes for the MON MGR and MDS roles. To bootstrap the cluster, we installed cephadm via apt only on the first node (M1). Then, we copied the ceph.pub key to the second node (M2) and the third node (M3). After that, we ran the following commands on M1: ceph orch host add m2 <IP> ceph orch host add m3 <IP> Note: cephadm is not installed on M2 and M3. Question: What happens if M1 fails completely? In this scenario, we would need the file: /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} to be able to launch the cephadm shell and recover the cluster. -- M Eugen Block via ceph-users <ceph-users@ceph.io> writes:

There’s no need to do that. Making it executable would be only required if you wanted to use that binary manually. The difference between the package and container version are negligible in this case. We still have cephadm version 16.2.13 installed on the hosts while the cluster is running 19.2.3. You’re fine with this minor difference between versions.

Zitat von Michel Le Cocq <lecocq@ipgp.fr>:

...

Hi, thanks for your answer.

I effectively have on my host : /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST}

But it's not executable ! I did a chmod +x ...

Then I can see that debian pkg version and the one in /var/lib are not the same ...

For Debian : cephadm version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable)

For Ceph in /var/lib ... cephadm version 18.2.8 (efac5a54607c13fa50d4822e50242b86e6e446df) reef (stable)

Is it a good idee to do on host :

# apt purge cephadm # chmod +x /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} # ln -s /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST} /root/cephadm

Thanks.

-- Michel

Eugen Block via ceph-users <ceph-users@ceph.io> writes:

...

Hi, responses inline...

Zitat von Michel Le Cocq via ceph-users <ceph-users@ceph.io>:

...

Actually, what confused me was this:

cephadm bootstrap --mon-ip <my-ip>

Among other things, this command: - Generates a new SSH key for the Ceph cluster and adds it to the /root/.ssh/authorized_keys file. - Saves a copy of the public key to the /etc/ceph/ceph.pub file.

You then need to add a node.

ssh-copy-id -f -i /etc/ceph/ceph.pub root@<new-host>

ceph orch host add <newhost> [<ip>] [<label1> ...]

I wondered where the private part of the SSH key /etc/ceph/ceph.pub was !

That's stored in the config-key db:

ceph config-key get mgr/cephadm/ssh_identity_key

...

-- M

Michel Le Cocq via ceph-users <ceph-users@ceph.io> writes:

...

Hi again.

I'm answering my own question.

It seems I read the wrong version of the doc ... sic!

It should be enough to set the “_admin” flag on all nodes:

and that we also need to install cephadm on the nodes we want to use for administration, right!?

Not necessarily, but it can be more convenient for debugging. The orchestrator stores a cephadm binary on each node to be able to manage them, it is stored in /var/lib/ceph/{FSID}/cephadm.{SOME_DIGEST}. That's used to remotely orchestrate the cluster/nodes etc. The original bootstrap node requires a cephadm binary either installed as a package or downloaded via curl as described in the docs to be able to bootstrap. After that you'll only need to execute cephadm commands locally on the nodes if you're debugging issues etc., otherwise the orchestrator will only use the file in /var/lib/ceph.

...

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io