Hi Jacek, Hi Eugen, In addition to Eugen's recommendations, could you also check for conflicting 'ceph' user UIDs on the host and in containers (where the ceph UID is 167)? Just want to rule out this possibility [1]. Best, Frédéric. [1] https://tracker.ceph.com/issues/63319 -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Check our online Config Diff Tool, it's great! https://analyzer.clyso.com/#/analyzer/config-diff https://clyso.com | frederic.nass@clyso.com Le mar. 21 avr. 2026 à 22:20, Eugen Block via ceph-users <ceph-users@ceph.io> a écrit :

Hi,

do you have the ceph-osd package installed on the servers? It's known to be a potential issue when trying to use cephadm while ceph packages are installed (except for cephadm, of course, and maybe ceph-common).

Is it possible that there are still remainders from previous attempts and not everything was properly cleaned up?

The host directory /var/lib/ceph/osd is not relevant for cephadm, it maps /var/lib/ceph/{FSID}/osd/ceph-{OSD_ID} into the container to /var/lib/ceph/osd/ceph-{OSD_ID}. So the permissions should be okay if you didn't modify them within the container.

How exactly did you zap the drives? Did you reboot after zapping? What usually works for me is 'cephadm ceph-volume lvm zap --destroy /dev/sdX' or via orchestrator 'ceph orch device zap {HOST} /dev/sdX'. Sometimes zapping isn't enough and I need to reboot (but usually VMs when I've been experimenting).

Did you inspect the logs? /var/log/ceph/cephadm.log could show some hints.

What's the output of 'ceph orch device ls'?

And just one thought on the number of monitors: It's not useful to have 4 MONs, an odd number is recommended. With 4 MONs you'd still lose quorum when two MONs go down, so it doesn't offer any more resiliency than 3 MONs.

Regards, Eugen

Zitat von Jacek Rużyczka via ceph-users <ceph-users@ceph.io>:

...

Hi,

I've got the following issue when trying to build the OSD store on a 4-node Ceph cluster:

mixtile@blade3n1:~$ sudo ceph orch apply osd --all-available-devices Scheduled osd.all-available-devices update...

Unfortunately, I *always *get this error message in the cephadm log:

2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 3 --monmap /var/lib/ceph/osd/ceph-3/activate.monmap --keyfile - --osdspec-affinity all-available-devices --osd-data /var/lib/ceph/osd/ceph-3/ --osd-uuid 3a8350dd-023a-49ae-8e4e-86b51082aec2 --setuser ceph --setgroup ceph 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: 2026-04-21T14:54:16.656+0000 ffffba0b0040 -1 bluestore(/var/lib/ceph/osd/ceph-3//block) No valid bdev label found 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: 2026-04-21T14:54:16.962+0000 ffffba0b0040 -1 bluestore(/var/lib/ceph/osd/ceph-3/) _read_fsid unparsable uuid 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 bdev(0xaaaae79c7400 /var/lib/ceph/osd/ceph-3//block) open open got: (13) Permission denied 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 bdev(0xaaaae79c7400 /var/lib/ceph/osd/ceph-3//block) open open got: (13) Permission denied 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied 2026-04-21 16:54:29,563 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-3/: (13) Permission denied 2026-04-21 16:54:29,563 ffff8c5b9720 INFO /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes

I've checked the permissions of /var/lib/ceph/osd on all nodes, but can't find a mistake there:

mixtile@blade3n1:~$ sudo ls -al /var/lib/ceph/osd total 8 drwxrwxrwx 2 ceph ceph 4096 Apr 21 15:57 . drwxr-x--- 14 ceph ceph 4096 Apr 21 14:54 ..

Isn't ceph the correct user here? I've also added user ceph to group disk, so that the drive /dev/nvme0n1 can be accessed directly by ceph, but no use:

mixtile@blade3n1:~$ ls -al /dev/nvme0n1 brw-rw---- 1 root disk 259, 0 Apr 21 17:05 /dev/nvme0n1

I've got four active hosts in my cluster:

mixtile@blade3n1:~$ sudo ceph orch host ls HOST ADDR LABELS STATUS blade3n1 10.20.0.11 _admin blade3n2 10.20.0.12 blade3n3 10.20.0.13 blade3n4 10.20.0.14 4 hosts in cluster

At Reddit, they suggest zapping all drives. I've done that, but no use.

Of course, the cluster can't work in such a situation:

mixtile@blade3n1:~$ sudo ceph -s cluster: id: 8aad3073-39a1-11f1-bf6e-f2704a1efa9b health: HEALTH_WARN Failed to apply 1 service(s): osd.all-available-devices 1 MDSs report slow metadata IOs Reduced data availability: 2 pgs inactive OSD count 1 < osd_pool_default_size 3

services: mon: 4 daemons, quorum blade3n1,blade3n2,blade3n3,blade3n4 (age 3h) mgr: blade3n2.cnqdje(active, since 3h), standbys: blade3n1.rrlwwv mds: 1/1 daemons up, 1 standby osd: 1 osds: 0 up, 0 in (since 7m)

data: volumes: 1/1 healthy pools: 2 pools, 2 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 100.000% pgs unknown 2 unknown

progress: Global Recovery Event (0s) [............................]

What's wrong here? I've got Ceph 19.2.3 Squid (stable) on Ubuntu 24.04.4 LTS (aarch64). Thank you.

Regards Jacek Rużyczka _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io