Jan, Unfortunately I'm under immense pressure right now to get some form of Ceph into production, so it's going to be Luminous for now, or maybe a live upgrade to Nautilus without recreating the OSDs (if that's possible). The good news is that in the next couple months I expect to add more hardware that should be nearly identical.  I will gladly give it a go at that time and see if I can recreate.  (Or, if I manage to thoroughly crash my current fledgling cluster, I'll give it another go on one node while I'm up all night recovering.) If you could tell me where to look I'd gladly read some code and see if I can find anything that way.  Or if there's any sort of design document describing the deep internals I'd be glad to scan it to see if I've hit a corner case of some sort.  Actually, I'd be interested in reading those documents anyway if I could. Thanks. -Dave Dave Hall On 1/28/2020 3:05 AM, Jan Fajerski wrote:

On Mon, Jan 27, 2020 at 03:23:55PM -0500, Dave Hall wrote:

...

All,

I've just spent a significant amount of time unsuccessfully chasing the  _read_fsid unparsable uuid error on Debian 10 / Natilus 14.2.6. Since this is a brand new cluster, last night I gave up and moved back to Debian 9 / Luminous 12.2.11.  In both cases I'm using the packages from Debian Backports with ceph-ansible as my deployment tool. Note that above I said 'the _read_fsid unparsable uuid' error. I've searched around a bit and found some previously reported issues, but I did not see any conclusive resolutions.

I would like to get to Nautilus as quickly as possible, so I'd gladly provide additional information to help track down the cause of this symptom.  I can confirm that, looking at the ceph-volume.log on the OSD host I see no difference between the ceph-volume lvm batch command generated by the ceph-ansible versions associated with these two Ceph releases:

ceph-volume --cluster ceph lvm batch --bluestore --yes --block-db-size 133358734540 /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/nvme0n1

Note that I'm using --block-db-size to divide my NVMe into 12 segments as I have 4 empty drive bays on my OSD servers that I may eventually be able to fill.

My OSD hardware is:

Disk /dev/nvme0n1: 1.5 TiB, 1600321314816 bytes, 3125627568 sectors Disk /dev/sdc: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors Disk /dev/sdd: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors Disk /dev/sde: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors Disk /dev/sdf: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors Disk /dev/sdg: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors Disk /dev/sdh: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors Disk /dev/sdi: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors Disk /dev/sdj: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors

I'd send the output of ceph-volume inventory on Luminous, but I'm getting  -->: KeyError: 'human_readable_size'.

Please let me know if I can provide any further information. Mind re-running you ceph-volume command with debug output enabled: CEPH_VOLUME_DEBUG=true ceph-volume --cluster ceph lvm batch --bluestore ...

Ideally you could also openen a bug report here https://tracker.ceph.com/projects/ceph-volume/issues/new

Thanks!

...

Thanks.

-Dave

-- Dave Hall Binghamton University

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

On Tue, Jan 28, 2020 at 08:03:35PM +0100, bauen1 wrote:

Hi,

I've run into the same issue while testing:

ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)

debian bullseye

Ceph was installed using ceph-ansible on a vm from the repo http://download.ceph.com/debian-nautilus

The output of `sudo sh -c 'CEPH_VOLUME_DEBUG=true ceph-volume --cluster test lvm batch --bluestore /dev/vdb'` has been attached. Thx, I opened https://tracker.ceph.com/issues/43868. This looks like a bluestore/osd issue to me, though it might end up being ceph-volumes fault.

Also worth noting might be that '/var/lib/ceph/osd/test-0/fsid' is empty (but I don't know too much about the internals)

- bauen1

On 1/28/20 4:54 PM, Dave Hall wrote:

...

Jan,

Unfortunately I'm under immense pressure right now to get some form of Ceph into production, so it's going to be Luminous for now, or maybe a live upgrade to Nautilus without recreating the OSDs (if that's possible).

The good news is that in the next couple months I expect to add more hardware that should be nearly identical.  I will gladly give it a go at that time and see if I can recreate.  (Or, if I manage to thoroughly crash my current fledgling cluster, I'll give it another go on one node while I'm up all night recovering.)

If you could tell me where to look I'd gladly read some code and see if I can find anything that way.  Or if there's any sort of design document describing the deep internals I'd be glad to scan it to see if I've hit a corner case of some sort.  Actually, I'd be interested in reading those documents anyway if I could.

Thanks.

-Dave

Dave Hall

On 1/28/2020 3:05 AM, Jan Fajerski wrote:

...

On Mon, Jan 27, 2020 at 03:23:55PM -0500, Dave Hall wrote:

...

All,

I've just spent a significant amount of time unsuccessfully chasing the  _read_fsid unparsable uuid error on Debian 10 / Natilus 14.2.6. Since this is a brand new cluster, last night I gave up and moved back to Debian 9 / Luminous 12.2.11.  In both cases I'm using the packages from Debian Backports with ceph-ansible as my deployment tool. Note that above I said 'the _read_fsid unparsable uuid' error. I've searched around a bit and found some previously reported issues, but I did not see any conclusive resolutions.

I would like to get to Nautilus as quickly as possible, so I'd gladly provide additional information to help track down the cause of this symptom.  I can confirm that, looking at the ceph-volume.log on the OSD host I see no difference between the ceph-volume lvm batch command generated by the ceph-ansible versions associated with these two Ceph releases:

   ceph-volume --cluster ceph lvm batch --bluestore --yes    --block-db-size 133358734540 /dev/sdc /dev/sdd /dev/sde /dev/sdf    /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/nvme0n1

Note that I'm using --block-db-size to divide my NVMe into 12 segments as I have 4 empty drive bays on my OSD servers that I may eventually be able to fill.

My OSD hardware is:

   Disk /dev/nvme0n1: 1.5 TiB, 1600321314816 bytes, 3125627568 sectors    Disk /dev/sdc: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors    Disk /dev/sdd: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors    Disk /dev/sde: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors    Disk /dev/sdf: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors    Disk /dev/sdg: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors    Disk /dev/sdh: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors    Disk /dev/sdi: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors    Disk /dev/sdj: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors

I'd send the output of ceph-volume inventory on Luminous, but I'm getting  -->: KeyError: 'human_readable_size'.

Please let me know if I can provide any further information. Mind re-running you ceph-volume command with  debug output enabled: CEPH_VOLUME_DEBUG=true ceph-volume --cluster ceph lvm batch --bluestore ...

Ideally you could also openen a bug report here https://tracker.ceph.com/projects/ceph-volume/issues/new

Thanks!

...

Thanks.

-Dave

-- Dave Hall Binghamton University

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

sysadmin@ceph-test:~$ sudo setenforce 0 sysadmin@ceph-test:~$ sudo sh -c 'CEPH_VOLUME_DEBUG=true ceph-volume --cluster test lvm batch --bluestore /dev/vdb'

Total OSDs: 1

Type Path LV Size % of device ---------------------------------------------------------------------------------------------------- [data] /dev/vdb 63.00 GB 100.0% --> The above OSDs would be created if the operation continues --> do you want to proceed? (yes/no) yes Running command: /usr/sbin/vgcreate -s 1G --force --yes ceph-1cc81d7c-a153-462a-8080-ec3d217c7180 /dev/vdb stdout: Physical volume "/dev/vdb" successfully created. stdout: Volume group "ceph-1cc81d7c-a153-462a-8080-ec3d217c7180" successfully created Running command: /usr/sbin/lvcreate --yes -l 63 -n osd-data-bbd7752f-fad9-41d5-bbbe-e6fd512bcf8e ceph-1cc81d7c-a153-462a-8080-ec3d217c7180 stdout: Wiping ceph_bluestore signature on /dev/ceph-1cc81d7c-a153-462a-8080-ec3d217c7180/osd-data-bbd7752f-fad9-41d5-bbbe-e6fd512bcf8e. stdout: Logical volume "osd-data-bbd7752f-fad9-41d5-bbbe-e6fd512bcf8e" created. Running command: /bin/ceph-authtool --gen-print-key Running command: /bin/ceph --cluster test --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/test.keyring -i - osd new e3ebb6e0-82c8-4088-a6bd-abd729a575bb Running command: /bin/ceph-authtool --gen-print-key Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/test-0 Running command: /usr/sbin/restorecon /var/lib/ceph/osd/test-0 Running command: /bin/chown -h ceph:ceph /dev/ceph-1cc81d7c-a153-462a-8080-ec3d217c7180/osd-data-bbd7752f-fad9-41d5-bbbe-e6fd512bcf8e Running command: /bin/chown -R ceph:ceph /dev/dm-1 Running command: /bin/ln -s /dev/ceph-1cc81d7c-a153-462a-8080-ec3d217c7180/osd-data-bbd7752f-fad9-41d5-bbbe-e6fd512bcf8e /var/lib/ceph/osd/test-0/block Running command: /bin/ceph --cluster test --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/test.keyring mon getmap -o /var/lib/ceph/osd/test-0/activate.monmap stderr: got monmap epoch 1 Running command: /bin/ceph-authtool /var/lib/ceph/osd/test-0/keyring --create-keyring --name osd.0 --add-key AQAcgzBeTlc5BxAApXJgwyoRAHtrL9kk1tbs9w== stdout: creating /var/lib/ceph/osd/test-0/keyring stdout: added entity osd.0 auth(key=AQAcgzBeTlc5BxAApXJgwyoRAHtrL9kk1tbs9w==) Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/test-0/keyring Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/test-0/ Running command: /bin/ceph-osd --cluster test --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/test-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/test-0/ --osd-uuid e3ebb6e0-82c8-4088-a6bd-abd729a575bb --setuser ceph --setgroup ceph stderr: 2020-01-28 18:53:20.438 7f17de7b3c00 -1 bluestore(/var/lib/ceph/osd/test-0/) _read_fsid unparsable uuid stderr: terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_get> >' stderr: what(): boost::bad_get: failed value get using boost::get stderr: *** Caught signal (Aborted) ** stderr: in thread 7f17de7b3c00 thread_name:ceph-osd stderr: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable) stderr: 1: (()+0x13520) [0x7f17dee75520] stderr: 2: (gsignal()+0x141) [0x7f17de93b081] stderr: 3: (abort()+0x121) [0x7f17de926535] stderr: 4: (()+0x9a643) [0x7f17decba643] stderr: 5: (()+0xa5fd6) [0x7f17decc5fd6] stderr: 6: (()+0xa6041) [0x7f17decc6041] stderr: 7: (()+0xa6295) [0x7f17decc6295] stderr: 8: (()+0x49a92c) [0x56027edc792c] stderr: 9: (Option::size_t const md_config_t::get_val<Option::size_t>(ConfigValues const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x51) [0x56027eedeea1] stderr: 10: (BlueStore::_set_cache_sizes()+0x174) [0x56027f3fba44] stderr: 11: (BlueStore::_open_bdev(bool)+0x1c5) [0x56027f3fe845] stderr: 12: (BlueStore::mkfs()+0x6e0) [0x56027f484620] stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x1b3) [0x56027eef9b23] stderr: 14: (main()+0x1821) [0x56027eea68d1] stderr: 15: (__libc_start_main()+0xeb) [0x7f17de927bbb] stderr: 16: (_start()+0x2a) [0x56027eed903a] stderr: 2020-01-28 18:53:20.486 7f17de7b3c00 -1 *** Caught signal (Aborted) ** stderr: in thread 7f17de7b3c00 thread_name:ceph-osd stderr: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable) stderr: 1: (()+0x13520) [0x7f17dee75520] stderr: 2: (gsignal()+0x141) [0x7f17de93b081] stderr: 3: (abort()+0x121) [0x7f17de926535] stderr: 4: (()+0x9a643) [0x7f17decba643] stderr: 5: (()+0xa5fd6) [0x7f17decc5fd6] stderr: 6: (()+0xa6041) [0x7f17decc6041] stderr: 7: (()+0xa6295) [0x7f17decc6295] stderr: 8: (()+0x49a92c) [0x56027edc792c] stderr: 9: (Option::size_t const md_config_t::get_val<Option::size_t>(ConfigValues const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x51) [0x56027eedeea1] stderr: 10: (BlueStore::_set_cache_sizes()+0x174) [0x56027f3fba44] stderr: 11: (BlueStore::_open_bdev(bool)+0x1c5) [0x56027f3fe845] stderr: 12: (BlueStore::mkfs()+0x6e0) [0x56027f484620] stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x1b3) [0x56027eef9b23] stderr: 14: (main()+0x1821) [0x56027eea68d1] stderr: 15: (__libc_start_main()+0xeb) [0x7f17de927bbb] stderr: 16: (_start()+0x2a) [0x56027eed903a] stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. stderr: -5> 2020-01-28 18:53:20.438 7f17de7b3c00 -1 bluestore(/var/lib/ceph/osd/test-0/) _read_fsid unparsable uuid stderr: 0> 2020-01-28 18:53:20.486 7f17de7b3c00 -1 *** Caught signal (Aborted) ** stderr: in thread 7f17de7b3c00 thread_name:ceph-osd stderr: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable) stderr: 1: (()+0x13520) [0x7f17dee75520] stderr: 2: (gsignal()+0x141) [0x7f17de93b081] stderr: 3: (abort()+0x121) [0x7f17de926535] stderr: 4: (()+0x9a643) [0x7f17decba643] stderr: 5: (()+0xa5fd6) [0x7f17decc5fd6] stderr: 6: (()+0xa6041) [0x7f17decc6041] stderr: 7: (()+0xa6295) [0x7f17decc6295] stderr: 8: (()+0x49a92c) [0x56027edc792c] stderr: 9: (Option::size_t const md_config_t::get_val<Option::size_t>(ConfigValues const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x51) [0x56027eedeea1] stderr: 10: (BlueStore::_set_cache_sizes()+0x174) [0x56027f3fba44] stderr: 11: (BlueStore::_open_bdev(bool)+0x1c5) [0x56027f3fe845] stderr: 12: (BlueStore::mkfs()+0x6e0) [0x56027f484620] stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x1b3) [0x56027eef9b23] stderr: 14: (main()+0x1821) [0x56027eea68d1] stderr: 15: (__libc_start_main()+0xeb) [0x7f17de927bbb] stderr: 16: (_start()+0x2a) [0x56027eed903a] stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. stderr: -5> 2020-01-28 18:53:20.438 7f17de7b3c00 -1 bluestore(/var/lib/ceph/osd/test-0/) _read_fsid unparsable uuid stderr: 0> 2020-01-28 18:53:20.486 7f17de7b3c00 -1 *** Caught signal (Aborted) ** stderr: in thread 7f17de7b3c00 thread_name:ceph-osd stderr: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable) stderr: 1: (()+0x13520) [0x7f17dee75520] stderr: 2: (gsignal()+0x141) [0x7f17de93b081] stderr: 3: (abort()+0x121) [0x7f17de926535] stderr: 4: (()+0x9a643) [0x7f17decba643] stderr: 5: (()+0xa5fd6) [0x7f17decc5fd6] stderr: 6: (()+0xa6041) [0x7f17decc6041] stderr: 7: (()+0xa6295) [0x7f17decc6295] stderr: 8: (()+0x49a92c) [0x56027edc792c] stderr: 9: (Option::size_t const md_config_t::get_val<Option::size_t>(ConfigValues const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x51) [0x56027eedeea1] stderr: 10: (BlueStore::_set_cache_sizes()+0x174) [0x56027f3fba44] stderr: 11: (BlueStore::_open_bdev(bool)+0x1c5) [0x56027f3fe845] stderr: 12: (BlueStore::mkfs()+0x6e0) [0x56027f484620] stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x1b3) [0x56027eef9b23] stderr: 14: (main()+0x1821) [0x56027eea68d1] stderr: 15: (__libc_start_main()+0xeb) [0x7f17de927bbb] stderr: 16: (_start()+0x2a) [0x56027eed903a] stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --> Was unable to complete a new OSD, will rollback changes Running command: /bin/ceph --cluster test --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/test.keyring osd purge-new osd.0 --yes-i-really-mean-it stderr: purged osd.0 Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 38, in __init__ self.main(self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 149, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 40, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 325, in main self.execute() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 288, in execute self.strategy.execute() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/strategies/bluestore.py", line 124, in execute Create(command).main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 69, in main self.create(args) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 26, in create prepare_step.safe_prepare(args) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 219, in safe_prepare self.prepare() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 320, in prepare osd_fsid, File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 119, in prepare_bluestore db=db File "/usr/lib/python3/dist-packages/ceph_volume/util/prepare.py", line 430, in osd_mkfs_bluestore raise RuntimeError('Command failed with exit code %s: %s' % (returncode, ' '.join(command))) RuntimeError: Command failed with exit code 250: /bin/ceph-osd --cluster test --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/test-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/test-0/ --osd-uuid e3ebb6e0-82c8-4088-a6bd-abd729a575bb --setuser ceph --setgroup ceph sysadmin@ceph-test:~$ sudo setenforce 1 sysadmin@ceph-test:~$

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

-- Jan Fajerski Senior Software Engineer Enterprise Storage SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer

Jan, In trying to recover my OSDs after the upgrade from Nautilus described earlier, I eventually managed to make things worse to the point where I'm going to scrub and fully reinstall. So I zapped all of the devices on one of my nodes and reproduced the ceph-volume lvm create error I mentioned earlier, using the procedure from https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/ to lay out the LVs and issue ceph-volume lvm create. As I was concerned that maybe it was a size thing, I only create a 4TB block LV for my first attempt, and the full 12TB drive for my second attempt. The output is: root@ceph01:~# ceph-volume lvm create --bluestore --data ceph-block-0/block-0 --block.db ceph-db-0/db-0 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 6441f236-8694-46b9-9c6a-bf82af89765d Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-24 --> Absolute path not found for executable: selinuxenabled --> Ensure $PATH environment variable contains common executable locations Running command: /bin/chown -h ceph:ceph /dev/ceph-block-0/block-0 Running command: /bin/chown -R ceph:ceph /dev/dm-0 Running command: /bin/ln -s /dev/ceph-block-0/block-0 /var/lib/ceph/osd/ceph-24/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-24/activate.monmap stderr: got monmap epoch 4 Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-24/keyring --create-keyring --name osd.24 --add-key AQAuMjJe5OGHBRAAP94+1E7CzV5Rv9HFj9WVqA== stdout: creating /var/lib/ceph/osd/ceph-24/keyring added entity osd.24 auth(key=AQAuMjJe5OGHBRAAP94+1E7CzV5Rv9HFj9WVqA==) Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-24/keyring Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-24/ Running command: /bin/chown -h ceph:ceph /dev/ceph-db-0/db-0 Running command: /bin/chown -R ceph:ceph /dev/dm-1 Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 24 --monmap /var/lib/ceph/osd/ceph-24/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-db-0/db-0 --osd-data /var/lib/ceph/osd/ceph-24/ --osd-uuid 6441f236-8694-46b9-9c6a-bf82af89765d --setuser ceph --setgroup ceph stderr: 2020-01-29 20:32:33.054 7ff4c24abc80 -1 bluestore(/var/lib/ceph/osd/ceph-24/) _read_fsid unparsable uuid stderr: terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_get>

' stderr: what(): boost::bad_get: failed value get using boost::get stderr: *** Caught signal (Aborted) ** stderr: in thread 7ff4c24abc80 thread_name:ceph-osd stderr: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable) stderr: 1: (()+0x12730) [0x7ff4c2f54730] stderr: 2: (gsignal()+0x10b) [0x7ff4c2a377bb] stderr: 3: (abort()+0x121) [0x7ff4c2a22535] stderr: 4: (()+0x8c983) [0x7ff4c2dea983] stderr: 5: (()+0x928c6) [0x7ff4c2df08c6] stderr: 6: (()+0x92901) [0x7ff4c2df0901] stderr: 7: (()+0x92b34) [0x7ff4c2df0b34] stderr: 8: (()+0x5a3f53) [0x564eed1c4f53] stderr: 9: (Option::size_t const md_config_t::get_val<Option::size_t>(ConfigValues const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x81) [0x564eed1cac91] stderr: 10: (BlueStore::_set_cache_sizes()+0x15a) [0x564eed645d8a] stderr: 11: (BlueStore::_open_bdev(bool)+0x173) [0x564eed648b23] stderr: 12: (BlueStore::mkfs()+0x42b) [0x564eed6adeab] stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0xd5) [0x564eed1e4bf5] stderr: 14: (main()+0x1796) [0x564eed191366] stderr: 15: (__libc_start_main()+0xeb) [0x7ff4c2a2409b] stderr: 16: (_start()+0x2a) [0x564eed1c4c6a] stderr: 2020-01-29 20:32:33.062 7ff4c24abc80 -1 *** Caught signal (Aborted) ** stderr: in thread 7ff4c24abc80 thread_name:ceph-osd stderr: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable) stderr: 1: (()+0x12730) [0x7ff4c2f54730] stderr: 2: (gsignal()+0x10b) [0x7ff4c2a377bb] stderr: 3: (abort()+0x121) [0x7ff4c2a22535] stderr: 4: (()+0x8c983) [0x7ff4c2dea983] stderr: 5: (()+0x928c6) [0x7ff4c2df08c6] stderr: 6: (()+0x92901) [0x7ff4c2df0901] stderr: 7: (()+0x92b34) [0x7ff4c2df0b34] stderr: 8: (()+0x5a3f53) [0x564eed1c4f53] stderr: 9: (Option::size_t const md_config_t::get_val<Option::size_t>(ConfigValues const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x81) [0x564eed1cac91] stderr: 10: (BlueStore::_set_cache_sizes()+0x15a) [0x564eed645d8a] stderr: 11: (BlueStore::_open_bdev(bool)+0x173) [0x564eed648b23] stderr: 12: (BlueStore::mkfs()+0x42b) [0x564eed6adeab] stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0xd5) [0x564eed1e4bf5] stderr: 14: (main()+0x1796) [0x564eed191366] stderr: 15: (__libc_start_main()+0xeb) [0x7ff4c2a2409b] stderr: 16: (_start()+0x2a) [0x564eed1c4c6a] stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. stderr: -5> 2020-01-29 20:32:33.054 7ff4c24abc80 -1 bluestore(/var/lib/ceph/osd/ceph-24/) _read_fsid unparsable uuid stderr: 0> 2020-01-29 20:32:33.062 7ff4c24abc80 -1 *** Caught signal (Aborted) ** stderr: in thread 7ff4c24abc80 thread_name:ceph-osd stderr: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable) stderr: 1: (()+0x12730) [0x7ff4c2f54730] stderr: 2: (gsignal()+0x10b) [0x7ff4c2a377bb] stderr: 3: (abort()+0x121) [0x7ff4c2a22535] stderr: 4: (()+0x8c983) [0x7ff4c2dea983] stderr: 5: (()+0x928c6) [0x7ff4c2df08c6] stderr: 6: (()+0x92901) [0x7ff4c2df0901] stderr: 7: (()+0x92b34) [0x7ff4c2df0b34] stderr: 8: (()+0x5a3f53) [0x564eed1c4f53] stderr: 9: (Option::size_t const md_config_t::get_val<Option::size_t>(ConfigValues const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x81) [0x564eed1cac91] stderr: 10: (BlueStore::_set_cache_sizes()+0x15a) [0x564eed645d8a] stderr: 11: (BlueStore::_open_bdev(bool)+0x173) [0x564eed648b23] stderr: 12: (BlueStore::mkfs()+0x42b) [0x564eed6adeab] stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0xd5) [0x564eed1e4bf5] stderr: 14: (main()+0x1796) [0x564eed191366] stderr: 15: (__libc_start_main()+0xeb) [0x7ff4c2a2409b] stderr: 16: (_start()+0x2a) [0x564eed1c4c6a] stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. stderr: -5> 2020-01-29 20:32:33.054 7ff4c24abc80 -1 bluestore(/var/lib/ceph/osd/ceph-24/) _read_fsid unparsable uuid stderr: 0> 2020-01-29 20:32:33.062 7ff4c24abc80 -1 *** Caught signal (Aborted) ** stderr: in thread 7ff4c24abc80 thread_name:ceph-osd stderr: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable) stderr: 1: (()+0x12730) [0x7ff4c2f54730] stderr: 2: (gsignal()+0x10b) [0x7ff4c2a377bb] stderr: 3: (abort()+0x121) [0x7ff4c2a22535] stderr: 4: (()+0x8c983) [0x7ff4c2dea983] stderr: 5: (()+0x928c6) [0x7ff4c2df08c6] stderr: 6: (()+0x92901) [0x7ff4c2df0901] stderr: 7: (()+0x92b34) [0x7ff4c2df0b34] stderr: 8: (()+0x5a3f53) [0x564eed1c4f53] stderr: 9: (Option::size_t const md_config_t::get_val<Option::size_t>(ConfigValues const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x81) [0x564eed1cac91] stderr: 10: (BlueStore::_set_cache_sizes()+0x15a) [0x564eed645d8a] stderr: 11: (BlueStore::_open_bdev(bool)+0x173) [0x564eed648b23] stderr: 12: (BlueStore::mkfs()+0x42b) [0x564eed6adeab] stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0xd5) [0x564eed1e4bf5] stderr: 14: (main()+0x1796) [0x564eed191366] stderr: 15: (__libc_start_main()+0xeb) [0x7ff4c2a2409b] stderr: 16: (_start()+0x2a) [0x564eed1c4c6a] stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --> Was unable to complete a new OSD, will rollback changes Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.24 --yes-i-really-mean-it stderr: purged osd.24 --> RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 24 --monmap /var/lib/ceph/osd/ceph-24/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-db-0/db-0 --osd-data /var/lib/ceph/osd/ceph-24/ --osd-uuid 6441f236-8694-46b9-9c6a-bf82af89765d --setuser ceph --setgroup ceph root@ceph01:~#

Dave Hall Binghamton Universitykdhall@binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office) On 1/29/2020 3:15 AM, Jan Fajerski wrote: On Tue, Jan 28, 2020 at 08:03:35PM +0100, bauen1 wrote: Hi, I've run into the same issue while testing: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable) debian bullseye Ceph was installed using ceph-ansible on a vm from the repo http://download.ceph.com/debian-nautilus The output of `sudo sh -c 'CEPH_VOLUME_DEBUG=true ceph-volume --cluster test lvm batch --bluestore /dev/vdb'` has been attached. Thx, I opened https://tracker.ceph.com/issues/43868. This looks like a bluestore/osd issue to me, though it might end up being ceph-volumes fault. Also worth noting might be that '/var/lib/ceph/osd/test-0/fsid' is empty (but I don't know too much about the internals) - bauen1 On 1/28/20 4:54 PM, Dave Hall wrote: Jan, Unfortunately I'm under immense pressure right now to get some form of Ceph into production, so it's going to be Luminous for now, or maybe a live upgrade to Nautilus without recreating the OSDs (if that's possible). The good news is that in the next couple months I expect to add more hardware that should be nearly identical. I will gladly give it a go at that time and see if I can recreate. (Or, if I manage to thoroughly crash my current fledgling cluster, I'll give it another go on one node while I'm up all night recovering.) If you could tell me where to look I'd gladly read some code and see if I can find anything that way. Or if there's any sort of design document describing the deep internals I'd be glad to scan it to see if I've hit a corner case of some sort. Actually, I'd be interested in reading those documents anyway if I could. Thanks. -Dave Dave Hall On 1/28/2020 3:05 AM, Jan Fajerski wrote: On Mon, Jan 27, 2020 at 03:23:55PM -0500, Dave Hall wrote: All, I've just spent a significant amount of time unsuccessfully chasing the _read_fsid unparsable uuid error on Debian 10 / Natilus 14.2.6. Since this is a brand new cluster, last night I gave up and moved back to Debian 9 / Luminous 12.2.11. In both cases I'm using the packages

from Debian Backports with ceph-ansible as my deployment tool.

' stderr: what(): boost::bad_get: failed value get using boost::get stderr: *** Caught signal (Aborted) ** stderr: in thread 7f17de7b3c00 thread_name:ceph-osd stderr: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable) stderr: 1: (()+0x13520) [0x7f17dee75520] stderr: 2: (gsignal()+0x141) [0x7f17de93b081] stderr: 3: (abort()+0x121) [0x7f17de926535] stderr: 4: (()+0x9a643) [0x7f17decba643] stderr: 5: (()+0xa5fd6) [0x7f17decc5fd6] stderr: 6: (()+0xa6041) [0x7f17decc6041] stderr: 7: (()+0xa6295) [0x7f17decc6295] stderr: 8: (()+0x49a92c) [0x56027edc792c] stderr: 9: (Option::size_t const md_config_t::get_val<Option::size_t>(ConfigValues const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x51) [0x56027eedeea1] stderr: 10: (BlueStore::_set_cache_sizes()+0x174) [0x56027f3fba44] stderr: 11: (BlueStore::_open_bdev(bool)+0x1c5) [0x56027f3fe845] stderr: 12: (BlueStore::mkfs()+0x6e0) [0x56027f484620] stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x1b3) [0x56027eef9b23] stderr: 14: (main()+0x1821) [0x56027eea68d1] stderr: 15: (__libc_start_main()+0xeb) [0x7f17de927bbb] stderr: 16: (_start()+0x2a) [0x56027eed903a] stderr: 2020-01-28 18:53:20.486 7f17de7b3c00 -1 *** Caught signal (Aborted) ** stderr: in thread 7f17de7b3c00 thread_name:ceph-osd stderr: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable) stderr: 1: (()+0x13520) [0x7f17dee75520] stderr: 2: (gsignal()+0x141) [0x7f17de93b081] stderr: 3: (abort()+0x121) [0x7f17de926535] stderr: 4: (()+0x9a643) [0x7f17decba643] stderr: 5: (()+0xa5fd6) [0x7f17decc5fd6] stderr: 6: (()+0xa6041) [0x7f17decc6041] stderr: 7: (()+0xa6295) [0x7f17decc6295] stderr: 8: (()+0x49a92c) [0x56027edc792c] stderr: 9: (Option::size_t const md_config_t::get_val<Option::size_t>(ConfigValues const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x51) [0x56027eedeea1] stderr: 10: (BlueStore::_set_cache_sizes()+0x174) [0x56027f3fba44] stderr: 11: (BlueStore::_open_bdev(bool)+0x1c5) [0x56027f3fe845] stderr: 12: (BlueStore::mkfs()+0x6e0) [0x56027f484620] stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x1b3) [0x56027eef9b23] stderr: 14: (main()+0x1821) [0x56027eea68d1] stderr: 15: (__libc_start_main()+0xeb) [0x7f17de927bbb] stderr: 16: (_start()+0x2a) [0x56027eed903a] stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. stderr: -5> 2020-01-28 18:53:20.438 7f17de7b3c00 -1 bluestore(/var/lib/ceph/osd/test-0/) _read_fsid unparsable uuid stderr: 0> 2020-01-28 18:53:20.486 7f17de7b3c00 -1 *** Caught signal (Aborted) ** stderr: in thread 7f17de7b3c00 thread_name:ceph-osd stderr: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable) stderr: 1: (()+0x13520) [0x7f17dee75520] stderr: 2: (gsignal()+0x141) [0x7f17de93b081] stderr: 3: (abort()+0x121) [0x7f17de926535] stderr: 4: (()+0x9a643) [0x7f17decba643] stderr: 5: (()+0xa5fd6) [0x7f17decc5fd6] stderr: 6: (()+0xa6041) [0x7f17decc6041] stderr: 7: (()+0xa6295) [0x7f17decc6295] stderr: 8: (()+0x49a92c) [0x56027edc792c] stderr: 9: (Option::size_t const md_config_t::get_val<Option::size_t>(ConfigValues const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x51) [0x56027eedeea1] stderr: 10: (BlueStore::_set_cache_sizes()+0x174) [0x56027f3fba44] stderr: 11: (BlueStore::_open_bdev(bool)+0x1c5) [0x56027f3fe845] stderr: 12: (BlueStore::mkfs()+0x6e0) [0x56027f484620] stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x1b3) [0x56027eef9b23] stderr: 14: (main()+0x1821) [0x56027eea68d1] stderr: 15: (__libc_start_main()+0xeb) [0x7f17de927bbb] stderr: 16: (_start()+0x2a) [0x56027eed903a] stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. stderr: -5> 2020-01-28 18:53:20.438 7f17de7b3c00 -1 bluestore(/var/lib/ceph/osd/test-0/) _read_fsid unparsable uuid stderr: 0> 2020-01-28 18:53:20.486 7f17de7b3c00 -1 *** Caught signal (Aborted) ** stderr: in thread 7f17de7b3c00 thread_name:ceph-osd stderr: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable) stderr: 1: (()+0x13520) [0x7f17dee75520] stderr: 2: (gsignal()+0x141) [0x7f17de93b081] stderr: 3: (abort()+0x121) [0x7f17de926535] stderr: 4: (()+0x9a643) [0x7f17decba643] stderr: 5: (()+0xa5fd6) [0x7f17decc5fd6] stderr: 6: (()+0xa6041) [0x7f17decc6041] stderr: 7: (()+0xa6295) [0x7f17decc6295] stderr: 8: (()+0x49a92c) [0x56027edc792c] stderr: 9: (Option::size_t const md_config_t::get_val<Option::size_t>(ConfigValues const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x51) [0x56027eedeea1] stderr: 10: (BlueStore::_set_cache_sizes()+0x174) [0x56027f3fba44] stderr: 11: (BlueStore::_open_bdev(bool)+0x1c5) [0x56027f3fe845] stderr: 12: (BlueStore::mkfs()+0x6e0) [0x56027f484620] stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x1b3) [0x56027eef9b23] stderr: 14: (main()+0x1821) [0x56027eea68d1] stderr: 15: (__libc_start_main()+0xeb) [0x7f17de927bbb] stderr: 16: (_start()+0x2a) [0x56027eed903a] stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --> Was unable to complete a new OSD, will rollback changes Running command: /bin/ceph --cluster test --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/test.keyring osd purge-new osd.0 --yes-i-really-mean-it stderr: purged osd.0 Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 38, in __init__ self.main(self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 149, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py",

Note that above I said 'the _read_fsid unparsable uuid' error. I've searched around a bit and found some previously reported issues, but I did not see any conclusive resolutions. I would like to get to Nautilus as quickly as possible, so I'd gladly provide additional information to help track down the cause of this symptom. I can confirm that, looking at the ceph-volume.log on the OSD host I see no difference between the ceph-volume lvm batch command generated by the ceph-ansible versions associated with these two Ceph releases: ceph-volume --cluster ceph lvm batch --bluestore --yes --block-db-size 133358734540 /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/nvme0n1 Note that I'm using --block-db-size to divide my NVMe into 12 segments as I have 4 empty drive bays on my OSD servers that I may eventually be able to fill. My OSD hardware is: Disk /dev/nvme0n1: 1.5 TiB, 1600321314816 bytes, 3125627568 sectors Disk /dev/sdc: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors Disk /dev/sdd: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors Disk /dev/sde: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors Disk /dev/sdf: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors Disk /dev/sdg: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors Disk /dev/sdh: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors Disk /dev/sdi: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors Disk /dev/sdj: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors I'd send the output of ceph-volume inventory on Luminous, but I'm getting -->: KeyError: 'human_readable_size'. Please let me know if I can provide any further information. Mind re-running you ceph-volume command with debug output enabled: CEPH_VOLUME_DEBUG=true ceph-volume --cluster ceph lvm batch --bluestore ... Ideally you could also openen a bug report herehttps://tracker.ceph.com/projects/ceph-volume/issues/new Thanks! Thanks. -Dave -- Dave Hall Binghamton University _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io sysadmin@ceph-test:~$ sudo setenforce 0 sysadmin@ceph-test:~$ sudo sh -c 'CEPH_VOLUME_DEBUG=true ceph-volume --cluster test lvm batch --bluestore /dev/vdb' Total OSDs: 1 Type Path LV Size % of device ---------------------------------------------------------------------------------------------------- [data] /dev/vdb 63.00 GB 100.0% --> The above OSDs would be created if the operation continues --> do you want to proceed? (yes/no) yes Running command: /usr/sbin/vgcreate -s 1G --force --yes ceph-1cc81d7c-a153-462a-8080-ec3d217c7180 /dev/vdb stdout: Physical volume "/dev/vdb" successfully created. stdout: Volume group "ceph-1cc81d7c-a153-462a-8080-ec3d217c7180" successfully created Running command: /usr/sbin/lvcreate --yes -l 63 -n osd-data-bbd7752f-fad9-41d5-bbbe-e6fd512bcf8e ceph-1cc81d7c-a153-462a-8080-ec3d217c7180 stdout: Wiping ceph_bluestore signature on /dev/ceph-1cc81d7c-a153-462a-8080-ec3d217c7180/osd-data-bbd7752f-fad9-41d5-bbbe-e6fd512bcf8e. stdout: Logical volume "osd-data-bbd7752f-fad9-41d5-bbbe-e6fd512bcf8e" created. Running command: /bin/ceph-authtool --gen-print-key Running command: /bin/ceph --cluster test --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/test.keyring -i - osd new e3ebb6e0-82c8-4088-a6bd-abd729a575bb Running command: /bin/ceph-authtool --gen-print-key Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/test-0 Running command: /usr/sbin/restorecon /var/lib/ceph/osd/test-0 Running command: /bin/chown -h ceph:ceph /dev/ceph-1cc81d7c-a153-462a-8080-ec3d217c7180/osd-data-bbd7752f-fad9-41d5-bbbe-e6fd512bcf8e Running command: /bin/chown -R ceph:ceph /dev/dm-1 Running command: /bin/ln -s /dev/ceph-1cc81d7c-a153-462a-8080-ec3d217c7180/osd-data-bbd7752f-fad9-41d5-bbbe-e6fd512bcf8e /var/lib/ceph/osd/test-0/block Running command: /bin/ceph --cluster test --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/test.keyring mon getmap -o /var/lib/ceph/osd/test-0/activate.monmap stderr: got monmap epoch 1 Running command: /bin/ceph-authtool /var/lib/ceph/osd/test-0/keyring --create-keyring --name osd.0 --add-key AQAcgzBeTlc5BxAApXJgwyoRAHtrL9kk1tbs9w== stdout: creating /var/lib/ceph/osd/test-0/keyring stdout: added entity osd.0 auth(key=AQAcgzBeTlc5BxAApXJgwyoRAHtrL9kk1tbs9w==) Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/test-0/keyring Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/test-0/ Running command: /bin/ceph-osd --cluster test --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/test-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/test-0/ --osd-uuid e3ebb6e0-82c8-4088-a6bd-abd729a575bb --setuser ceph --setgroup ceph stderr: 2020-01-28 18:53:20.438 7f17de7b3c00 -1 bluestore(/var/lib/ceph/osd/test-0/) _read_fsid unparsable uuid stderr: terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_get> line 40, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 325, in main self.execute() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line 288, in execute self.strategy.execute() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/strategies/bluestore.py", line 124, in execute Create(command).main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 69, in main self.create(args) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 26, in create prepare_step.safe_prepare(args) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 219, in safe_prepare self.prepare() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 320, in prepare osd_fsid, File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 119, in prepare_bluestore db=db File "/usr/lib/python3/dist-packages/ceph_volume/util/prepare.py", line 430, in osd_mkfs_bluestore raise RuntimeError('Command failed with exit code %s: %s' % (returncode, ' '.join(command))) RuntimeError: Command failed with exit code 250: /bin/ceph-osd --cluster test --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/test-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/test-0/ --osd-uuid e3ebb6e0-82c8-4088-a6bd-abd729a575bb --setuser ceph --setgroup ceph sysadmin@ceph-test:~$ sudo setenforce 1 sysadmin@ceph-test:~$ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io