Hello everyone, I'm trying to add an OSD node to my current cluster. I created an lvm volume for this node to use for OSD. My current Ceph version is 14.2.6 and it runs on an RHEL 7 OS. However, I got error when trying to activate the node. I'm confused with the output. I tried to see what really happened, but I don't know how to proceed to pinpoint the issues. I appreciate so much for any help to clear the confusion. * Why did the "mon getmap" command send the result to /dev/stderr? I saw the monmap file having value and it looked like it got the monmap, is there any way to check if the monmap having problem? * What caused "_read_fsid unparsable uuid"? Sincerely, Hai [xxx@xxx.com@xxx430 ~]$ sudo ceph-volume --cluster aap-storage lvm create --data /dev/kubernetes/ceph-osd Running command: /bin/ceph-authtool --gen-print-key Running command: /bin/ceph --cluster aap-storage --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/aap-storage.keyring -i - osd new c84f74b3-9a3e-4cd9-a4ce-4c1819a56f90 Running command: /bin/ceph-authtool --gen-print-key Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/aap-storage-5 Running command: /sbin/restorecon /var/lib/ceph/osd/aap-storage-5 Running command: /bin/chown -h ceph:ceph /dev/kubernetes/ceph-osd Running command: /bin/chown -R ceph:ceph /dev/dm-9 Running command: /bin/ln -s /dev/kubernetes/ceph-osd /var/lib/ceph/osd/aap-storage-5/block Running command: /bin/ceph --cluster aap-storage --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/aap-storage.keyring mon getmap -o /var/lib/ceph/osd/aap-storage-5/activate.monmap stderr: got monmap epoch 8 Running command: /bin/ceph-authtool /var/lib/ceph/osd/aap-storage-5/keyring --create-keyring --name osd.5 --add-key AQB2RAhgnkcLIBAAdcdR5N4YKzSJmfoA6G6XvA== stdout: creating /var/lib/ceph/osd/aap-storage-5/keyring added entity osd.5 auth(key=AQB2RAhgnkcLIBAAdcdR5N4YKzSJmfoA6G6XvA==) Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/aap-storage-5/keyring Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/aap-storage-5/ Running command: /bin/ceph-osd --cluster aap-storage --osd-objectstore bluestore --mkfs -i 5 --monmap /var/lib/ceph/osd/aap-storage-5/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/aap-storage-5/ --osd-uuid c84f74b3-9a3e-4cd9-a4ce-4c1819a56f90 --setuser ceph --setgroup ceph stderr: 2021-01-20 15:55:52.319 7f0970d3ba80 -1 bluestore(/var/lib/ceph/osd/aap-storage-5/) _read_fsid unparsable uuid --> ceph-volume lvm prepare successful for: kubernetes/ceph-osd Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/aap-storage-5 Running command: /bin/ceph-bluestore-tool --cluster=aap-storage prime-osd-dir --dev /dev/kubernetes/ceph-osd --path /var/lib/ceph/osd/aap-storage-5 --no-mon-config Running command: /bin/ln -snf /dev/kubernetes/ceph-osd /var/lib/ceph/osd/aap-storage-5/block Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/aap-storage-5/block Running command: /bin/chown -R ceph:ceph /dev/dm-9 Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/aap-storage-5 Running command: /bin/systemctl enable ceph-volume@lvm-5-c84f74b3-9a3e-4cd9-a4ce-4c1819a56f90 stderr: Created symlink from /etc/systemd/system/multi-user.target.wants/ceph-volume(a)lvm-5-c84f74b3-9a3e-4cd9-a4ce-4c1819a56f90.service to /usr/lib/systemd/system/ceph-volume@.service. Running command: /bin/systemctl enable --runtime ceph-osd@5 stderr: Created symlink from /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)5.service to /usr/lib/systemd/system/ceph-osd@.service. Running command: /bin/systemctl start ceph-osd@5 stderr: Job for ceph-osd(a)5.service failed because the control process exited with error code. See "systemctl status ceph-osd(a)5.service&quot; and "journalctl -xe" for details. --> Was unable to complete a new OSD, will rollback changes Running command: /bin/ceph --cluster aap-storage --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/aap-storage.keyring osd purge-new osd.5 --yes-i-really-mean-it stderr: purged osd.5 --> RuntimeError: command returned non-zero exit status: 1 So I break the procedure into steps, and proceed them one by one. I saw the problem happen with the bluestore. [xxx@xxx.com@xxx430 ~]$ sudo ceph-osd -c /etc/ceph/${CLUSTER_NAME}.conf -k /etc/ceph/${CLUSTER_NAME}.client.admin.keyring -i $ID --mkfs \ 2021-01-21 16:59:14.907 7f724e882a80 -1 bluestore(/var/lib/ceph/osd/aap-storage-5//block) _read_bdev_label failed to open /var/lib/ceph/osd/aap-storage-5//block: (13) Permission denied 2021-01-21 16:59:14.908 7f724e882a80 -1 bluestore(/var/lib/ceph/osd/aap-storage-5//block) _read_bdev_label failed to open /var/lib/ceph/osd/aap-storage-5//block: (13) Permission denied 2021-01-21 16:59:14.908 7f724e882a80 -1 bluestore(/var/lib/ceph/osd/aap-storage-5/) _read_fsid unparsable uuid 2021-01-21 16:59:14.908 7f724e882a80 -1 bluestore(/var/lib/ceph/osd/aap-storage-5/) _setup_block_symlink_or_file failed to open block file: (13) Permission denied 2021-01-21 16:59:14.908 7f724e882a80 -1 bluestore(/var/lib/ceph/osd/aap-storage-5/) mkfs failed, (13) Permission denied 2021-01-21 16:59:14.908 7f724e882a80 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied 2021-01-21 16:59:14.908 7f724e882a80 -1 ** ERROR: error creating empty object store in /var/lib/ceph/osd/aap-storage-5/: (13) Permission denied Because of the permission problem, so I avoid using the ceph user for the ceph service, however, the problem still persisted. [xxx@xxx.com@xxx430 ~]$ sudo ceph-osd -c /etc/ceph/${CLUSTER_NAME}.conf -k /etc/ceph/${CLUSTER_NAME}.client.admin.keyring -i $ID --mkfs --monmap /var/lib/ceph/osd/${CLUSTER_NAME}-$ID/activate.monmap --osd-uuid $UUID --no-mon-config --osd-data /var/lib/ceph/osd/${CLUSTER_NAME}-$ID/ 2021-01-21 17:00:08.583 7f3acc546a80 -1 bluestore(/var/lib/ceph/osd/aap-storage-5/) _read_fsid unparsable uuid -- KPMG IT Service GmbH Sitz/Registergericht: Berlin/Amtsgericht Charlottenburg, HRB 87521 B Geschäftsführer: Hans-Christian Schwieger, Helmar Symmank Aufsichtsratsvorsitzender: WP StB Klaus Becker Allgemeine Informationen zur Datenverarbeitung im Rahmen unserer allgemeinen Geschäftstätigkeit sowie im Mandatsverhältnis gemäß EU Datenschutz-Grundverordnung sind hier <https://home.kpmg.com/content/dam/kpmg/de/pdf/Themen/2018/datenschutzinformationen-fuer-betroffene-kpmg-it-service-gmbh.pdf> abrufbar. Die Information in dieser E-Mail ist vertraulich und kann dem Berufsgeheimnis unterliegen. Sie ist ausschließlich für den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Sollten Sie nicht der für diese E-Mail bestimmte Adressat sein, ist Ihnen jede Veröffentlichung, Vervielfältigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Maßnahmen im Vertrauen auf erlangte Information untersagt. In dieser E-Mail enthaltene Meinungen oder Empfehlungen unterliegen den Bedingungen des jeweiligen Mandatsverhältnisses mit dem Adressaten. The information in this e-mail is confidential and may be legally privileged. It is intended solely for the addressee. Access to this e-mail by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Any opinions or advice contained in this e-mail are subject to the terms and conditions expressed in the governing KPMG client engagement letter.