Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
Hi Eugen, Hi Frédéric, One after the other, please: Do you have the ceph-osd package installed on the servers? It's known
to be a potential issue when trying to use cephadm while ceph packages are installed (except for cephadm, of course, and maybe ceph-common).
Yes, ceph-osd is installed on all nodes. This must have happened automatically when setting them up. Is this wrong? Is it possible that there are still remainders from previous attempts
and not everything was properly cleaned up?
I've looked for remainders of old installations. In fact, there was one, a ruin, which remained from an unfinished bootstrap (because my whole cluster had broken down). I've removed them, but no use. Did you inspect the logs? /var/log/ceph/cephadm.log could show some hints. Yo. In fact, I found out about my issue in the logfile, but it doesn't say, *which *file can't be accessed due to lack of permission. And just one thought on the number of monitors: It's not useful to
have 4 MONs, an odd number is recommended. With 4 MONs you'd still lose quorum when two MONs go down, so it doesn't offer any more resiliency than 3 MONs.
Shall I reduce it to 3? And: In addition to Eugen's recommendations, could you also check for
conflicting 'ceph' user UIDs on the host and in containers (where the ceph UID is 167)? Just want to rule out this possibility [1].
I've seen user #167 several times in /var/lib/ceph and its subfolders without knowing who he in fact is. It's not user ceph. Regards Jacek
And: How exactly did you zap the drives? Did you reboot after zapping? What
usually works for me is 'cephadm ceph-volume lvm zap --destroy /dev/sdX' or via orchestrator 'ceph orch device zap {HOST} /dev/sdX'. Sometimes zapping isn't enough and I need to reboot (but usually VMs when I've been experimenting).
I followed this doc: https://docs.ceph.com/en/reef/ceph-volume/lvm/zap/ After the zap, I also performed a shred on the individual drives, as I had already run into Ceph not accepting the drives as "available", probably due to some tell-tale bit patterns remaining on the drives. After that, I rebooted the cluster. Am Mi., 22. Apr. 2026 um 18:01 Uhr schrieb Jacek Rużyczka < jacek.ruzyczka@googlemail.com>:
Hi Eugen, Hi Frédéric,
One after the other, please:
Do you have the ceph-osd package installed on the servers? It's known
to be a potential issue when trying to use cephadm while ceph packages are installed (except for cephadm, of course, and maybe ceph-common).
Yes, ceph-osd is installed on all nodes. This must have happened automatically when setting them up. Is this wrong?
Is it possible that there are still remainders from previous attempts
and not everything was properly cleaned up?
I've looked for remainders of old installations. In fact, there was one, a ruin, which remained from an unfinished bootstrap (because my whole cluster had broken down). I've removed them, but no use.
Did you inspect the logs? /var/log/ceph/cephadm.log could show some hints.
Yo. In fact, I found out about my issue in the logfile, but it doesn't say, *which *file can't be accessed due to lack of permission.
And just one thought on the number of monitors: It's not useful to
have 4 MONs, an odd number is recommended. With 4 MONs you'd still lose quorum when two MONs go down, so it doesn't offer any more resiliency than 3 MONs.
Shall I reduce it to 3?
And:
In addition to Eugen's recommendations, could you also check for
conflicting 'ceph' user UIDs on the host and in containers (where the ceph UID is 167)? Just want to rule out this possibility [1].
I've seen user #167 several times in /var/lib/ceph and its subfolders without knowing who he in fact is. It's not user ceph.
Regards Jacek
Hi, the zapping is correct then. I strongly recommend to remove all ceph-[mon,osd,mgr,mds] packages and leave only cephadm and docker installed. As I wrote, it can have an impact if you both cephadm and ceph packages installed, especially ceph-osd package, there have been several reports on this list. Here's the warning in the docs [0]:
When deploying new OSDs with cephadm, ensure that the ceph-osd package is not installed on the target host. If it is installed, conflicts may arise in the management and control of the OSD that may lead to errors or unexpected behavior.
Remove the ceph-osd package and reset directory permissions if necessary (or uninstall cephadm package as well, then reinstall it). As for the MONs: yes, I would recommend to reduce them to 3 although the fourth won't hurt you, it just doesn't provide any additional resiliency. What does the command 'id ceph' on the host show? Within a container it should show: [ceph: root@pacific /]# id ceph uid=167(ceph) gid=167(ceph) groups=167(ceph) Regards, Eugen [0] https://docs.ceph.com/en/latest/cephadm/services/osd/ Zitat von Jacek Rużyczka <jacek.ruzyczka@googlemail.com>:
And:
How exactly did you zap the drives? Did you reboot after zapping? What
usually works for me is 'cephadm ceph-volume lvm zap --destroy /dev/sdX' or via orchestrator 'ceph orch device zap {HOST} /dev/sdX'. Sometimes zapping isn't enough and I need to reboot (but usually VMs when I've been experimenting).
I followed this doc: https://docs.ceph.com/en/reef/ceph-volume/lvm/zap/
After the zap, I also performed a shred on the individual drives, as I had already run into Ceph not accepting the drives as "available", probably due to some tell-tale bit patterns remaining on the drives. After that, I rebooted the cluster.
Am Mi., 22. Apr. 2026 um 18:01 Uhr schrieb Jacek Rużyczka < jacek.ruzyczka@googlemail.com>:
Hi Eugen, Hi Frédéric,
One after the other, please:
Do you have the ceph-osd package installed on the servers? It's known
to be a potential issue when trying to use cephadm while ceph packages are installed (except for cephadm, of course, and maybe ceph-common).
Yes, ceph-osd is installed on all nodes. This must have happened automatically when setting them up. Is this wrong?
Is it possible that there are still remainders from previous attempts
and not everything was properly cleaned up?
I've looked for remainders of old installations. In fact, there was one, a ruin, which remained from an unfinished bootstrap (because my whole cluster had broken down). I've removed them, but no use.
Did you inspect the logs? /var/log/ceph/cephadm.log could show some hints.
Yo. In fact, I found out about my issue in the logfile, but it doesn't say, *which *file can't be accessed due to lack of permission.
And just one thought on the number of monitors: It's not useful to
have 4 MONs, an odd number is recommended. With 4 MONs you'd still lose quorum when two MONs go down, so it doesn't offer any more resiliency than 3 MONs.
Shall I reduce it to 3?
And:
In addition to Eugen's recommendations, could you also check for
conflicting 'ceph' user UIDs on the host and in containers (where the ceph UID is 167)? Just want to rule out this possibility [1].
I've seen user #167 several times in /var/lib/ceph and its subfolders without knowing who he in fact is. It's not user ceph.
Regards Jacek
participants (2)
-
Eugen Block
-
Jacek Rużyczka