And: How exactly did you zap the drives? Did you reboot after zapping? What
usually works for me is 'cephadm ceph-volume lvm zap --destroy /dev/sdX' or via orchestrator 'ceph orch device zap {HOST} /dev/sdX'. Sometimes zapping isn't enough and I need to reboot (but usually VMs when I've been experimenting).
I followed this doc: https://docs.ceph.com/en/reef/ceph-volume/lvm/zap/ After the zap, I also performed a shred on the individual drives, as I had already run into Ceph not accepting the drives as "available", probably due to some tell-tale bit patterns remaining on the drives. After that, I rebooted the cluster. Am Mi., 22. Apr. 2026 um 18:01 Uhr schrieb Jacek Rużyczka < jacek.ruzyczka@googlemail.com>:
Hi Eugen, Hi Frédéric,
One after the other, please:
Do you have the ceph-osd package installed on the servers? It's known
to be a potential issue when trying to use cephadm while ceph packages are installed (except for cephadm, of course, and maybe ceph-common).
Yes, ceph-osd is installed on all nodes. This must have happened automatically when setting them up. Is this wrong?
Is it possible that there are still remainders from previous attempts
and not everything was properly cleaned up?
I've looked for remainders of old installations. In fact, there was one, a ruin, which remained from an unfinished bootstrap (because my whole cluster had broken down). I've removed them, but no use.
Did you inspect the logs? /var/log/ceph/cephadm.log could show some hints.
Yo. In fact, I found out about my issue in the logfile, but it doesn't say, *which *file can't be accessed due to lack of permission.
And just one thought on the number of monitors: It's not useful to
have 4 MONs, an odd number is recommended. With 4 MONs you'd still lose quorum when two MONs go down, so it doesn't offer any more resiliency than 3 MONs.
Shall I reduce it to 3?
And:
In addition to Eugen's recommendations, could you also check for
conflicting 'ceph' user UIDs on the host and in containers (where the ceph UID is 167)? Just want to rule out this possibility [1].
I've seen user #167 several times in /var/lib/ceph and its subfolders without knowing who he in fact is. It's not user ceph.
Regards Jacek