- ceph-users - lists.ceph.io

Re: Upgrade problem from Octopus to Quincy
by Eugen Block 22 Apr '26

22 Apr '26

Hi, I'm glad you could resolve the issue. Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > On 15/04/26 12:29, Iztok Gregori via ceph-users wrote: >> On 15/04/26 12:08, Eugen Block via ceph-users wrote: >>> Yeah, looks like it. I just compared the output with a fresh >>> Tentacle install, I only enabled log-to-file during bootstrap, so >>> log_to_file is true in my cluster as well. There's no >>> leveldb_log_to_ceph_log option anymore since the default has been >>> rocksdb for a while now. >> >> I put 'log_to_journald' to true and journalctl started to log. A >> lot (with debug=20) :-) >> >> Now I'm ready to debug the joining problem. I will keep you posted. >> > > Just an update, the restart of the Octopus daemon didn't had an > effect, one the new monitors were started they still stayed in the > "electing" phase causing the 'ceph ...' commands to hang. Not a lot > in the logs, or nothing that I was able to point out. > > After that we decided to stop the Octopus daemon and ran the cluster > with the remaining 2 Quincy ones. This did the job. > > Once the Octopus monitor wasn't running any more the new Quincy > monitors rapidly joined the cluster. And now we are able to continue > our upgrade path to Squid/Tentacle and cephadm. > > I'm not 100% sure of what was wrong, it seems that in an not > completely upgraded Quincy monitor cluster is not possible to add > new monitors. Once the monitors were all the same version, it > started to work again. > > > Thanks a lot for your help! > > Cheers > Iztok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
by Jacek Rużyczka 22 Apr '26

22 Apr '26

Hi Eugen, Hi Frédéric, One after the other, please: Do you have the ceph-osd package installed on the servers? It's known > to be a potential issue when trying to use cephadm while ceph packages > are installed (except for cephadm, of course, and maybe ceph-common). Yes, ceph-osd is installed on all nodes. This must have happened automatically when setting them up. Is this wrong? Is it possible that there are still remainders from previous attempts > and not everything was properly cleaned up? I've looked for remainders of old installations. In fact, there was one, a ruin, which remained from an unfinished bootstrap (because my whole cluster had broken down). I've removed them, but no use. Did you inspect the logs? /var/log/ceph/cephadm.log could show some hints. Yo. In fact, I found out about my issue in the logfile, but it doesn't say, *which *file can't be accessed due to lack of permission. And just one thought on the number of monitors: It's not useful to > have 4 MONs, an odd number is recommended. With 4 MONs you'd still > lose quorum when two MONs go down, so it doesn't offer any more > resiliency than 3 MONs. Shall I reduce it to 3? And: In addition to Eugen's recommendations, could you also check for > conflicting 'ceph' user UIDs on the host and in containers (where the ceph > UID is 167)? Just want to rule out this possibility [1]. I've seen user #167 several times in /var/lib/ceph and its subfolders without knowing who he in fact is. It's not user ceph. Regards Jacek

2 2

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
by Frédéric Nass 22 Apr '26

22 Apr '26

Hi Jacek, Hi Eugen, In addition to Eugen's recommendations, could you also check for conflicting 'ceph' user UIDs on the host and in containers (where the ceph UID is 167)? Just want to rule out this possibility [1]. Best, Frédéric. [1] https://tracker.ceph.com/issues/63319 -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Check our online Config Diff Tool, it's great! https://analyzer.clyso.com/#/analyzer/config-diff https://clyso.com | frederic.nass(a)clyso.com Le mar. 21 avr. 2026 à 22:20, Eugen Block via ceph-users <ceph-users(a)ceph.io> a écrit : > Hi, > > do you have the ceph-osd package installed on the servers? It's known > to be a potential issue when trying to use cephadm while ceph packages > are installed (except for cephadm, of course, and maybe ceph-common). > > Is it possible that there are still remainders from previous attempts > and not everything was properly cleaned up? > > The host directory /var/lib/ceph/osd is not relevant for cephadm, it > maps /var/lib/ceph/{FSID}/osd/ceph-{OSD_ID} into the container to > /var/lib/ceph/osd/ceph-{OSD_ID}. So the permissions should be okay if > you didn't modify them within the container. > > How exactly did you zap the drives? Did you reboot after zapping? What > usually works for me is 'cephadm ceph-volume lvm zap --destroy > /dev/sdX' or via orchestrator 'ceph orch device zap {HOST} /dev/sdX'. > Sometimes zapping isn't enough and I need to reboot (but usually VMs > when I've been experimenting). > > Did you inspect the logs? /var/log/ceph/cephadm.log could show some hints. > > What's the output of 'ceph orch device ls'? > > And just one thought on the number of monitors: It's not useful to > have 4 MONs, an odd number is recommended. With 4 MONs you'd still > lose quorum when two MONs go down, so it doesn't offer any more > resiliency than 3 MONs. > > Regards, > Eugen > > Zitat von Jacek Rużyczka via ceph-users <ceph-users(a)ceph.io>: > > > Hi, > > > > I've got the following issue when trying to build the OSD store on a > 4-node > > Ceph cluster: > > > > mixtile@blade3n1:~$ sudo ceph orch apply osd --all-available-devices > > Scheduled osd.all-available-devices update... > > > > Unfortunately, I *always *get this error message in the cephadm log: > > > > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr Running > > command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore > > --mkfs -i 3 --monmap /var/lib/ceph/osd/ceph-3/activate.monmap --keyfile - > > --osdspec-affinity all-available-devices --osd-data > > /var/lib/ceph/osd/ceph-3/ --osd-uuid 3a8350dd-023a-49ae-8e4e-86b51082aec2 > > --setuser ceph --setgroup ceph > > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr > stderr: > > 2026-04-21T14:54:16.656+0000 ffffba0b0040 -1 > > bluestore(/var/lib/ceph/osd/ceph-3//block) No valid bdev label found > > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr > stderr: > > 2026-04-21T14:54:16.962+0000 ffffba0b0040 -1 > > bluestore(/var/lib/ceph/osd/ceph-3/) _read_fsid unparsable uuid > > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr > stderr: > > 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 bdev(0xaaaae79c7400 > > /var/lib/ceph/osd/ceph-3//block) open open got: (13) Permission denied > > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr > stderr: > > 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 bdev(0xaaaae79c7400 > > /var/lib/ceph/osd/ceph-3//block) open open got: (13) Permission denied > > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr > stderr: > > 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 OSD::mkfs: ObjectStore::mkfs > > failed with error (13) Permission denied > > 2026-04-21 16:54:29,563 ffff8c5b9720 INFO /usr/bin/docker: stderr > stderr: > > 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 ** ERROR: error creating > > empty object store in /var/lib/ceph/osd/ceph-3/: (13) Permission denied > > 2026-04-21 16:54:29,563 ffff8c5b9720 INFO /usr/bin/docker: stderr --> Was > > unable to complete a new OSD, will rollback changes > > > > I've checked the permissions of /var/lib/ceph/osd on all nodes, but can't > > find a mistake there: > > > > mixtile@blade3n1:~$ sudo ls -al /var/lib/ceph/osd > > total 8 > > drwxrwxrwx 2 ceph ceph 4096 Apr 21 15:57 . > > drwxr-x--- 14 ceph ceph 4096 Apr 21 14:54 .. > > > > Isn't ceph the correct user here? I've also added user ceph to group > disk, > > so that the drive /dev/nvme0n1 can be accessed directly by ceph, but no > use: > > > > mixtile@blade3n1:~$ ls -al /dev/nvme0n1 > > brw-rw---- 1 root disk 259, 0 Apr 21 17:05 /dev/nvme0n1 > > > > I've got four active hosts in my cluster: > > > > mixtile@blade3n1:~$ sudo ceph orch host ls > > HOST ADDR LABELS STATUS > > blade3n1 10.20.0.11 _admin > > blade3n2 10.20.0.12 > > blade3n3 10.20.0.13 > > blade3n4 10.20.0.14 > > 4 hosts in cluster > > > > At Reddit, they suggest zapping all drives. I've done that, but no use. > > > > Of course, the cluster can't work in such a situation: > > > > mixtile@blade3n1:~$ sudo ceph -s > > cluster: > > id: 8aad3073-39a1-11f1-bf6e-f2704a1efa9b > > health: HEALTH_WARN > > Failed to apply 1 service(s): osd.all-available-devices > > 1 MDSs report slow metadata IOs > > Reduced data availability: 2 pgs inactive > > OSD count 1 < osd_pool_default_size 3 > > > > services: > > mon: 4 daemons, quorum blade3n1,blade3n2,blade3n3,blade3n4 (age 3h) > > mgr: blade3n2.cnqdje(active, since 3h), standbys: blade3n1.rrlwwv > > mds: 1/1 daemons up, 1 standby > > osd: 1 osds: 0 up, 0 in (since 7m) > > > > data: > > volumes: 1/1 healthy > > pools: 2 pools, 2 pgs > > objects: 0 objects, 0 B > > usage: 0 B used, 0 B / 0 B avail > > pgs: 100.000% pgs unknown > > 2 unknown > > > > progress: > > Global Recovery Event (0s) > > [............................] > > > > What's wrong here? I've got Ceph 19.2.3 Squid (stable) on Ubuntu 24.04.4 > > LTS (aarch64). Thank you. > > > > Regards > > Jacek Rużyczka > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Upgrade problem from Octopus to Quincy
by Iztok Gregori 22 Apr '26

22 Apr '26

On 15/04/26 12:29, Iztok Gregori via ceph-users wrote: > On 15/04/26 12:08, Eugen Block via ceph-users wrote: >> Yeah, looks like it. I just compared the output with a fresh Tentacle >> install, I only enabled log-to-file during bootstrap, so log_to_file >> is true in my cluster as well. There's no leveldb_log_to_ceph_log >> option anymore since the default has been rocksdb for a while now. > > I put 'log_to_journald' to true and journalctl started to log. A lot > (with debug=20) :-) > > Now I'm ready to debug the joining problem. I will keep you posted. > Just an update, the restart of the Octopus daemon didn't had an effect, one the new monitors were started they still stayed in the "electing" phase causing the 'ceph ...' commands to hang. Not a lot in the logs, or nothing that I was able to point out. After that we decided to stop the Octopus daemon and ran the cluster with the remaining 2 Quincy ones. This did the job. Once the Octopus monitor wasn't running any more the new Quincy monitors rapidly joined the cluster. And now we are able to continue our upgrade path to Squid/Tentacle and cephadm. I'm not 100% sure of what was wrong, it seems that in an not completely upgraded Quincy monitor cluster is not possible to add new monitors. Once the monitors were all the same version, it started to work again. Thanks a lot for your help! Cheers Iztok

1 0

CephFS Snapshot Mirroring - "File exists already an active peer" Error (EEXIST) Despite Empty
by Adiga, Anantha 22 Apr '26

22 Apr '26

Hi, This is issue with CephFS snapshot mirroring setup between two clusters and would appreciate your assistance. Environment: * Primary Cluster: Ceph 15.2.13/16.2.5 (mixed versions) [cid:image001.png@01DCD199.2933E2E0] * Backup Cluster: Ceph 17.2.7/18.2.7 (mixed versions) [cid:image002.png@01DCD199.2933E2E0] * Filesystem: cephfs * Issue: Unable to import peer bootstrap token Problem Description: When attempting to import a peer bootstrap token using: ceph fs snapshot mirror peer_bootstrap import cephfs <token> I receive the error: Error EEXIST: already an active peer mgr.server reply reply (17) File exists already an active peer However, ceph fs snapshot mirror peer_list cephfs returns an empty list [], indicating no peers are configured. Steps Performed: Initial Setup: 1. Enabled mirroring module: ceph mgr module enable mirroring 2. Deployed cephfs-mirror service: ceph orch apply cephfs-mirror 3. Enabled mirroring on filesystem: ceph fs snapshot mirror enable cephfs 4. Created remote user and bootstrap token on backup cluster 5. Verified mirroring daemons are running: ceph fs snapshot mirror daemon status Troubleshooting Attempts: 1. Reset filesystem mirroring: ceph fs snapshot mirror disable cephfs ceph fs snapshot mirror enable cephfs 2. Module reset: ceph mgr module disable mirroring ceph mgr module enable mirroring 3. Service recreation: ceph orch rm cephfs-mirror ceph orch apply cephfs-mirror 4. Checked for stale configuration: o ceph config-key ls | grep mirror shows: mgr/cephadm/spec.cephfs-mirror o ceph fs snapshot mirror dirmap cephfs returns empty o ceph fs snapshot mirror status cephfs shows no active mirroring Current Status: * Mirroring daemons: Running and visible in daemon status * Peer list: Empty [] * Filesystem mirroring: Enabled * Error persists: EEXIST despite no visible peers Diagnostic Information: * MGR logs show: mgr.server reply reply (17) File exists already an active peer * No connectivity issues between clusters (ports 6789, 3300, 6800+ tested) * Both clusters are healthy with no other issues Regards, Anantha

1 0

Re: squid 19.2.4 QE validation status
by Laura Flores 21 Apr '26

21 Apr '26

upgrade/quincy-x and upgrade/squid-x are approved: https://tracker.ceph.com/projects/rados/wiki/SQUID#v1924-httpstrackercephco… The upgrade/squid-p2p tests need to be rerun with https://github.com/ceph/ceph/pull/68526 included. Thanks, Laura Laura Flores Software Engineer; Ceph Storage 1 708 738 8804 Mobile She/Her/Hers IBM ________________________________ From: Afreen via ceph-users <ceph-users(a)ceph.io> Sent: Tuesday, April 21, 2026 12:55 PM To: Yuri Weinstein <yweinste(a)redhat.com>; Redouane Kachach <rkachach(a)redhat.com> Cc: dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> Subject: [EXTERNAL] [ceph-users] Re: squid 19.2.4 QE validation status * Dashboard approved.* On Tue, Apr 14, 2026 at 1:00 AM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > Details of this release are summarized here: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… > > Release Notes - TBD > LRC upgrade - N/A > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Redouane & Guillaume > rbd: Ilya > krbd: Ilya > > upgrade/squid-x: Laura, Patrick > upgrade/reef-x: Laura, Patrick > upgrade/telemetry-upgrade: Laura, Patrick > upgrade/squid-p2p: Laura, Patrick > > upgrade-clients:client-upgrade-reef-squid-squid: PASSED > upgrade-clients:client-upgrade-quincy-squid-squid: PASSED > > perf-basic: PASSED > crimson-rados: N/A > powercycle: PASSED > > TIA > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
by Eugen Block 21 Apr '26

21 Apr '26

Hi, do you have the ceph-osd package installed on the servers? It's known to be a potential issue when trying to use cephadm while ceph packages are installed (except for cephadm, of course, and maybe ceph-common). Is it possible that there are still remainders from previous attempts and not everything was properly cleaned up? The host directory /var/lib/ceph/osd is not relevant for cephadm, it maps /var/lib/ceph/{FSID}/osd/ceph-{OSD_ID} into the container to /var/lib/ceph/osd/ceph-{OSD_ID}. So the permissions should be okay if you didn't modify them within the container. How exactly did you zap the drives? Did you reboot after zapping? What usually works for me is 'cephadm ceph-volume lvm zap --destroy /dev/sdX' or via orchestrator 'ceph orch device zap {HOST} /dev/sdX'. Sometimes zapping isn't enough and I need to reboot (but usually VMs when I've been experimenting). Did you inspect the logs? /var/log/ceph/cephadm.log could show some hints. What's the output of 'ceph orch device ls'? And just one thought on the number of monitors: It's not useful to have 4 MONs, an odd number is recommended. With 4 MONs you'd still lose quorum when two MONs go down, so it doesn't offer any more resiliency than 3 MONs. Regards, Eugen Zitat von Jacek Rużyczka via ceph-users <ceph-users(a)ceph.io>: > Hi, > > I've got the following issue when trying to build the OSD store on a 4-node > Ceph cluster: > > mixtile@blade3n1:~$ sudo ceph orch apply osd --all-available-devices > Scheduled osd.all-available-devices update... > > Unfortunately, I *always *get this error message in the cephadm log: > > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr Running > command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore > --mkfs -i 3 --monmap /var/lib/ceph/osd/ceph-3/activate.monmap --keyfile - > --osdspec-affinity all-available-devices --osd-data > /var/lib/ceph/osd/ceph-3/ --osd-uuid 3a8350dd-023a-49ae-8e4e-86b51082aec2 > --setuser ceph --setgroup ceph > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: > 2026-04-21T14:54:16.656+0000 ffffba0b0040 -1 > bluestore(/var/lib/ceph/osd/ceph-3//block) No valid bdev label found > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: > 2026-04-21T14:54:16.962+0000 ffffba0b0040 -1 > bluestore(/var/lib/ceph/osd/ceph-3/) _read_fsid unparsable uuid > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: > 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 bdev(0xaaaae79c7400 > /var/lib/ceph/osd/ceph-3//block) open open got: (13) Permission denied > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: > 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 bdev(0xaaaae79c7400 > /var/lib/ceph/osd/ceph-3//block) open open got: (13) Permission denied > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: > 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 OSD::mkfs: ObjectStore::mkfs > failed with error (13) Permission denied > 2026-04-21 16:54:29,563 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: > 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 ** ERROR: error creating > empty object store in /var/lib/ceph/osd/ceph-3/: (13) Permission denied > 2026-04-21 16:54:29,563 ffff8c5b9720 INFO /usr/bin/docker: stderr --> Was > unable to complete a new OSD, will rollback changes > > I've checked the permissions of /var/lib/ceph/osd on all nodes, but can't > find a mistake there: > > mixtile@blade3n1:~$ sudo ls -al /var/lib/ceph/osd > total 8 > drwxrwxrwx 2 ceph ceph 4096 Apr 21 15:57 . > drwxr-x--- 14 ceph ceph 4096 Apr 21 14:54 .. > > Isn't ceph the correct user here? I've also added user ceph to group disk, > so that the drive /dev/nvme0n1 can be accessed directly by ceph, but no use: > > mixtile@blade3n1:~$ ls -al /dev/nvme0n1 > brw-rw---- 1 root disk 259, 0 Apr 21 17:05 /dev/nvme0n1 > > I've got four active hosts in my cluster: > > mixtile@blade3n1:~$ sudo ceph orch host ls > HOST ADDR LABELS STATUS > blade3n1 10.20.0.11 _admin > blade3n2 10.20.0.12 > blade3n3 10.20.0.13 > blade3n4 10.20.0.14 > 4 hosts in cluster > > At Reddit, they suggest zapping all drives. I've done that, but no use. > > Of course, the cluster can't work in such a situation: > > mixtile@blade3n1:~$ sudo ceph -s > cluster: > id: 8aad3073-39a1-11f1-bf6e-f2704a1efa9b > health: HEALTH_WARN > Failed to apply 1 service(s): osd.all-available-devices > 1 MDSs report slow metadata IOs > Reduced data availability: 2 pgs inactive > OSD count 1 < osd_pool_default_size 3 > > services: > mon: 4 daemons, quorum blade3n1,blade3n2,blade3n3,blade3n4 (age 3h) > mgr: blade3n2.cnqdje(active, since 3h), standbys: blade3n1.rrlwwv > mds: 1/1 daemons up, 1 standby > osd: 1 osds: 0 up, 0 in (since 7m) > > data: > volumes: 1/1 healthy > pools: 2 pools, 2 pgs > objects: 0 objects, 0 B > usage: 0 B used, 0 B / 0 B avail > pgs: 100.000% pgs unknown > 2 unknown > > progress: > Global Recovery Event (0s) > [............................] > > What's wrong here? I've got Ceph 19.2.3 Squid (stable) on Ubuntu 24.04.4 > LTS (aarch64). Thank you. > > Regards > Jacek Rużyczka > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: squid 19.2.4 QE validation status
by Afreen 21 Apr '26

21 Apr '26

* Dashboard approved.* On Tue, Apr 14, 2026 at 1:00 AM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > Details of this release are summarized here: > > https://tracker.ceph.com/issues/75029#note-5 > > Release Notes - TBD > LRC upgrade - N/A > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Redouane & Guillaume > rbd: Ilya > krbd: Ilya > > upgrade/squid-x: Laura, Patrick > upgrade/reef-x: Laura, Patrick > upgrade/telemetry-upgrade: Laura, Patrick > upgrade/squid-p2p: Laura, Patrick > > upgrade-clients:client-upgrade-reef-squid-squid: PASSED > upgrade-clients:client-upgrade-quincy-squid-squid: PASSED > > perf-basic: PASSED > crimson-rados: N/A > powercycle: PASSED > > TIA >

1 0

ceph orch apply osd error: creating empty object store: (13) Permission denied
by Jacek Rużyczka 21 Apr '26

21 Apr '26

Hi, I've got the following issue when trying to build the OSD store on a 4-node Ceph cluster: mixtile@blade3n1:~$ sudo ceph orch apply osd --all-available-devices Scheduled osd.all-available-devices update... Unfortunately, I *always *get this error message in the cephadm log: 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 3 --monmap /var/lib/ceph/osd/ceph-3/activate.monmap --keyfile - --osdspec-affinity all-available-devices --osd-data /var/lib/ceph/osd/ceph-3/ --osd-uuid 3a8350dd-023a-49ae-8e4e-86b51082aec2 --setuser ceph --setgroup ceph 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: 2026-04-21T14:54:16.656+0000 ffffba0b0040 -1 bluestore(/var/lib/ceph/osd/ceph-3//block) No valid bdev label found 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: 2026-04-21T14:54:16.962+0000 ffffba0b0040 -1 bluestore(/var/lib/ceph/osd/ceph-3/) _read_fsid unparsable uuid 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 bdev(0xaaaae79c7400 /var/lib/ceph/osd/ceph-3//block) open open got: (13) Permission denied 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 bdev(0xaaaae79c7400 /var/lib/ceph/osd/ceph-3//block) open open got: (13) Permission denied 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied 2026-04-21 16:54:29,563 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-3/: (13) Permission denied 2026-04-21 16:54:29,563 ffff8c5b9720 INFO /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes I've checked the permissions of /var/lib/ceph/osd on all nodes, but can't find a mistake there: mixtile@blade3n1:~$ sudo ls -al /var/lib/ceph/osd total 8 drwxrwxrwx 2 ceph ceph 4096 Apr 21 15:57 . drwxr-x--- 14 ceph ceph 4096 Apr 21 14:54 .. Isn't ceph the correct user here? I've also added user ceph to group disk, so that the drive /dev/nvme0n1 can be accessed directly by ceph, but no use: mixtile@blade3n1:~$ ls -al /dev/nvme0n1 brw-rw---- 1 root disk 259, 0 Apr 21 17:05 /dev/nvme0n1 I've got four active hosts in my cluster: mixtile@blade3n1:~$ sudo ceph orch host ls HOST ADDR LABELS STATUS blade3n1 10.20.0.11 _admin blade3n2 10.20.0.12 blade3n3 10.20.0.13 blade3n4 10.20.0.14 4 hosts in cluster At Reddit, they suggest zapping all drives. I've done that, but no use. Of course, the cluster can't work in such a situation: mixtile@blade3n1:~$ sudo ceph -s cluster: id: 8aad3073-39a1-11f1-bf6e-f2704a1efa9b health: HEALTH_WARN Failed to apply 1 service(s): osd.all-available-devices 1 MDSs report slow metadata IOs Reduced data availability: 2 pgs inactive OSD count 1 < osd_pool_default_size 3 services: mon: 4 daemons, quorum blade3n1,blade3n2,blade3n3,blade3n4 (age 3h) mgr: blade3n2.cnqdje(active, since 3h), standbys: blade3n1.rrlwwv mds: 1/1 daemons up, 1 standby osd: 1 osds: 0 up, 0 in (since 7m) data: volumes: 1/1 healthy pools: 2 pools, 2 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 100.000% pgs unknown 2 unknown progress: Global Recovery Event (0s) [............................] What's wrong here? I've got Ceph 19.2.3 Squid (stable) on Ubuntu 24.04.4 LTS (aarch64). Thank you. Regards Jacek Rużyczka

1 0