Hi Eugen, Hi Frédéric, One after the other, please: Do you have the ceph-osd package installed on the servers? It's known
to be a potential issue when trying to use cephadm while ceph packages are installed (except for cephadm, of course, and maybe ceph-common).
Yes, ceph-osd is installed on all nodes. This must have happened automatically when setting them up. Is this wrong? Is it possible that there are still remainders from previous attempts
and not everything was properly cleaned up?
I've looked for remainders of old installations. In fact, there was one, a ruin, which remained from an unfinished bootstrap (because my whole cluster had broken down). I've removed them, but no use. Did you inspect the logs? /var/log/ceph/cephadm.log could show some hints. Yo. In fact, I found out about my issue in the logfile, but it doesn't say, *which *file can't be accessed due to lack of permission. And just one thought on the number of monitors: It's not useful to
have 4 MONs, an odd number is recommended. With 4 MONs you'd still lose quorum when two MONs go down, so it doesn't offer any more resiliency than 3 MONs.
Shall I reduce it to 3? And: In addition to Eugen's recommendations, could you also check for
conflicting 'ceph' user UIDs on the host and in containers (where the ceph UID is 167)? Just want to rule out this possibility [1].
I've seen user #167 several times in /var/lib/ceph and its subfolders without knowing who he in fact is. It's not user ceph. Regards Jacek