On Tue, Jan 28, 2020 at 08:03:35PM +0100, bauen1 wrote:
Hi,
I've run into the same issue while testing:
ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9)
nautilus (stable)
debian bullseye
Ceph was installed using ceph-ansible on a vm from the repo
http://download.ceph.com/debian-nautilus
The output of `sudo sh -c 'CEPH_VOLUME_DEBUG=true ceph-volume
--cluster test lvm batch --bluestore /dev/vdb'` has been attached.
Thx, I opened
https://tracker.ceph.com/issues/43868.
This looks like a bluestore/osd issue to me, though it might end up being
ceph-volumes fault.
Also worth noting might be that '/var/lib/ceph/osd/test-0/fsid' is
empty (but I don't know too much about the internals)
- bauen1
On 1/28/20 4:54 PM, Dave Hall wrote:
>Jan,
>
>Unfortunately I'm under immense pressure right now to get some form
>of Ceph into production, so it's going to be Luminous for now, or
>maybe a live upgrade to Nautilus without recreating the OSDs (if
>that's possible).
>
>The good news is that in the next couple months I expect to add more
>hardware that should be nearly identical. I will gladly give it a
>go at that time and see if I can recreate. (Or, if I manage to
>thoroughly crash my current fledgling cluster, I'll give it another
>go on one node while I'm up all night recovering.)
>
>If you could tell me where to look I'd gladly read some code and see
>if I can find anything that way. Or if there's any sort of design
>document describing the deep internals I'd be glad to scan it to see
>if I've hit a corner case of some sort. Actually, I'd be interested
>in reading those documents anyway if I could.
>
>Thanks.
>
>-Dave
>
>Dave Hall
>
>On 1/28/2020 3:05 AM, Jan Fajerski wrote:
>>On Mon, Jan 27, 2020 at 03:23:55PM -0500, Dave Hall wrote:
>>>All,
>>>
>>>I've just spent a significant amount of time unsuccessfully chasing
>>>the _read_fsid unparsable uuid error on Debian 10 / Natilus 14.2.6.
>>>Since this is a brand new cluster, last night I gave up and moved back
>>>to Debian 9 / Luminous 12.2.11. In both cases I'm using the packages
>>>from Debian Backports with ceph-ansible as my deployment tool.
>>>Note that above I said 'the _read_fsid unparsable uuid' error.
I've
>>>searched around a bit and found some previously reported issues, but I
>>>did not see any conclusive resolutions.
>>>
>>>I would like to get to Nautilus as quickly as possible, so I'd gladly
>>>provide additional information to help track down the cause of this
>>>symptom. I can confirm that, looking at the ceph-volume.log on the
>>>OSD host I see no difference between the ceph-volume lvm batch command
>>>generated by the ceph-ansible versions associated with these two Ceph
>>>releases:
>>>
>>> ceph-volume --cluster ceph lvm batch --bluestore --yes
>>> --block-db-size 133358734540 /dev/sdc /dev/sdd /dev/sde /dev/sdf
>>> /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/nvme0n1
>>>
>>>Note that I'm using --block-db-size to divide my NVMe into 12 segments
>>>as I have 4 empty drive bays on my OSD servers that I may eventually
>>>be able to fill.
>>>
>>>My OSD hardware is:
>>>
>>> Disk /dev/nvme0n1: 1.5 TiB, 1600321314816 bytes, 3125627568 sectors
>>> Disk /dev/sdc: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors
>>> Disk /dev/sdd: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors
>>> Disk /dev/sde: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors
>>> Disk /dev/sdf: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors
>>> Disk /dev/sdg: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors
>>> Disk /dev/sdh: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors
>>> Disk /dev/sdi: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors
>>> Disk /dev/sdj: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors
>>>
>>>I'd send the output of ceph-volume inventory on Luminous, but I'm
>>>getting -->: KeyError: 'human_readable_size'.
>>>
>>>Please let me know if I can provide any further information.
>>Mind re-running you ceph-volume command with debug output
>>enabled:
>>CEPH_VOLUME_DEBUG=true ceph-volume --cluster ceph lvm batch
>>--bluestore ...
>>
>>Ideally you could also openen a bug report here
>>https://tracker.ceph.com/projects/ceph-volume/issues/new
>>
>>Thanks!
>>>Thanks.
>>>
>>>-Dave
>>>
>>>--
>>>Dave Hall
>>>Binghamton University
>>>
>>>_______________________________________________
>>>ceph-users mailing list -- ceph-users(a)ceph.io
>>>To unsubscribe send an email to ceph-users-leave(a)ceph.io
>_______________________________________________
>ceph-users mailing list -- ceph-users(a)ceph.io
>To unsubscribe send an email to ceph-users-leave(a)ceph.io
sysadmin@ceph-test:~$ sudo setenforce 0
sysadmin@ceph-test:~$ sudo sh -c 'CEPH_VOLUME_DEBUG=true ceph-volume --cluster test
lvm batch --bluestore /dev/vdb'
Total OSDs: 1
Type Path LV Size
% of device
----------------------------------------------------------------------------------------------------
[data] /dev/vdb 63.00 GB
100.0%
--> The above OSDs would be created if the operation continues
--> do you want to proceed? (yes/no) yes
Running command: /usr/sbin/vgcreate -s 1G --force --yes
ceph-1cc81d7c-a153-462a-8080-ec3d217c7180 /dev/vdb
stdout: Physical volume "/dev/vdb" successfully created.
stdout: Volume group "ceph-1cc81d7c-a153-462a-8080-ec3d217c7180" successfully
created
Running command: /usr/sbin/lvcreate --yes -l 63 -n
osd-data-bbd7752f-fad9-41d5-bbbe-e6fd512bcf8e ceph-1cc81d7c-a153-462a-8080-ec3d217c7180
stdout: Wiping ceph_bluestore signature on
/dev/ceph-1cc81d7c-a153-462a-8080-ec3d217c7180/osd-data-bbd7752f-fad9-41d5-bbbe-e6fd512bcf8e.
stdout: Logical volume "osd-data-bbd7752f-fad9-41d5-bbbe-e6fd512bcf8e"
created.
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster test --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/test.keyring -i - osd new
e3ebb6e0-82c8-4088-a6bd-abd729a575bb
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/test-0
Running command: /usr/sbin/restorecon /var/lib/ceph/osd/test-0
Running command: /bin/chown -h ceph:ceph
/dev/ceph-1cc81d7c-a153-462a-8080-ec3d217c7180/osd-data-bbd7752f-fad9-41d5-bbbe-e6fd512bcf8e
Running command: /bin/chown -R ceph:ceph /dev/dm-1
Running command: /bin/ln -s
/dev/ceph-1cc81d7c-a153-462a-8080-ec3d217c7180/osd-data-bbd7752f-fad9-41d5-bbbe-e6fd512bcf8e
/var/lib/ceph/osd/test-0/block
Running command: /bin/ceph --cluster test --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/test.keyring mon getmap -o
/var/lib/ceph/osd/test-0/activate.monmap
stderr: got monmap epoch 1
Running command: /bin/ceph-authtool /var/lib/ceph/osd/test-0/keyring --create-keyring
--name osd.0 --add-key AQAcgzBeTlc5BxAApXJgwyoRAHtrL9kk1tbs9w==
stdout: creating /var/lib/ceph/osd/test-0/keyring
stdout: added entity osd.0 auth(key=AQAcgzBeTlc5BxAApXJgwyoRAHtrL9kk1tbs9w==)
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/test-0/keyring
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/test-0/
Running command: /bin/ceph-osd --cluster test --osd-objectstore bluestore --mkfs -i 0
--monmap /var/lib/ceph/osd/test-0/activate.monmap --keyfile - --osd-data
/var/lib/ceph/osd/test-0/ --osd-uuid e3ebb6e0-82c8-4088-a6bd-abd729a575bb --setuser ceph
--setgroup ceph
stderr: 2020-01-28 18:53:20.438 7f17de7b3c00 -1 bluestore(/var/lib/ceph/osd/test-0/)
_read_fsid unparsable uuid
stderr: terminate called after throwing an instance of
'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_get>
>'
stderr: what(): boost::bad_get: failed value get using boost::get
stderr: *** Caught signal (Aborted) **
stderr: in thread 7f17de7b3c00 thread_name:ceph-osd
stderr: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)
stderr: 1: (()+0x13520) [0x7f17dee75520]
stderr: 2: (gsignal()+0x141) [0x7f17de93b081]
stderr: 3: (abort()+0x121) [0x7f17de926535]
stderr: 4: (()+0x9a643) [0x7f17decba643]
stderr: 5: (()+0xa5fd6) [0x7f17decc5fd6]
stderr: 6: (()+0xa6041) [0x7f17decc6041]
stderr: 7: (()+0xa6295) [0x7f17decc6295]
stderr: 8: (()+0x49a92c) [0x56027edc792c]
stderr: 9: (Option::size_t const md_config_t::get_val<Option::size_t>(ConfigValues
const&, std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&) const+0x51) [0x56027eedeea1]
stderr: 10: (BlueStore::_set_cache_sizes()+0x174) [0x56027f3fba44]
stderr: 11: (BlueStore::_open_bdev(bool)+0x1c5) [0x56027f3fe845]
stderr: 12: (BlueStore::mkfs()+0x6e0) [0x56027f484620]
stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x1b3) [0x56027eef9b23]
stderr: 14: (main()+0x1821) [0x56027eea68d1]
stderr: 15: (__libc_start_main()+0xeb) [0x7f17de927bbb]
stderr: 16: (_start()+0x2a) [0x56027eed903a]
stderr: 2020-01-28 18:53:20.486 7f17de7b3c00 -1 *** Caught signal (Aborted) **
stderr: in thread 7f17de7b3c00 thread_name:ceph-osd
stderr: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)
stderr: 1: (()+0x13520) [0x7f17dee75520]
stderr: 2: (gsignal()+0x141) [0x7f17de93b081]
stderr: 3: (abort()+0x121) [0x7f17de926535]
stderr: 4: (()+0x9a643) [0x7f17decba643]
stderr: 5: (()+0xa5fd6) [0x7f17decc5fd6]
stderr: 6: (()+0xa6041) [0x7f17decc6041]
stderr: 7: (()+0xa6295) [0x7f17decc6295]
stderr: 8: (()+0x49a92c) [0x56027edc792c]
stderr: 9: (Option::size_t const md_config_t::get_val<Option::size_t>(ConfigValues
const&, std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&) const+0x51) [0x56027eedeea1]
stderr: 10: (BlueStore::_set_cache_sizes()+0x174) [0x56027f3fba44]
stderr: 11: (BlueStore::_open_bdev(bool)+0x1c5) [0x56027f3fe845]
stderr: 12: (BlueStore::mkfs()+0x6e0) [0x56027f484620]
stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x1b3) [0x56027eef9b23]
stderr: 14: (main()+0x1821) [0x56027eea68d1]
stderr: 15: (__libc_start_main()+0xeb) [0x7f17de927bbb]
stderr: 16: (_start()+0x2a) [0x56027eed903a]
stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
stderr: -5> 2020-01-28 18:53:20.438 7f17de7b3c00 -1
bluestore(/var/lib/ceph/osd/test-0/) _read_fsid unparsable uuid
stderr: 0> 2020-01-28 18:53:20.486 7f17de7b3c00 -1 *** Caught signal (Aborted) **
stderr: in thread 7f17de7b3c00 thread_name:ceph-osd
stderr: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)
stderr: 1: (()+0x13520) [0x7f17dee75520]
stderr: 2: (gsignal()+0x141) [0x7f17de93b081]
stderr: 3: (abort()+0x121) [0x7f17de926535]
stderr: 4: (()+0x9a643) [0x7f17decba643]
stderr: 5: (()+0xa5fd6) [0x7f17decc5fd6]
stderr: 6: (()+0xa6041) [0x7f17decc6041]
stderr: 7: (()+0xa6295) [0x7f17decc6295]
stderr: 8: (()+0x49a92c) [0x56027edc792c]
stderr: 9: (Option::size_t const md_config_t::get_val<Option::size_t>(ConfigValues
const&, std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&) const+0x51) [0x56027eedeea1]
stderr: 10: (BlueStore::_set_cache_sizes()+0x174) [0x56027f3fba44]
stderr: 11: (BlueStore::_open_bdev(bool)+0x1c5) [0x56027f3fe845]
stderr: 12: (BlueStore::mkfs()+0x6e0) [0x56027f484620]
stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x1b3) [0x56027eef9b23]
stderr: 14: (main()+0x1821) [0x56027eea68d1]
stderr: 15: (__libc_start_main()+0xeb) [0x7f17de927bbb]
stderr: 16: (_start()+0x2a) [0x56027eed903a]
stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
stderr: -5> 2020-01-28 18:53:20.438 7f17de7b3c00 -1
bluestore(/var/lib/ceph/osd/test-0/) _read_fsid unparsable uuid
stderr: 0> 2020-01-28 18:53:20.486 7f17de7b3c00 -1 *** Caught signal (Aborted) **
stderr: in thread 7f17de7b3c00 thread_name:ceph-osd
stderr: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)
stderr: 1: (()+0x13520) [0x7f17dee75520]
stderr: 2: (gsignal()+0x141) [0x7f17de93b081]
stderr: 3: (abort()+0x121) [0x7f17de926535]
stderr: 4: (()+0x9a643) [0x7f17decba643]
stderr: 5: (()+0xa5fd6) [0x7f17decc5fd6]
stderr: 6: (()+0xa6041) [0x7f17decc6041]
stderr: 7: (()+0xa6295) [0x7f17decc6295]
stderr: 8: (()+0x49a92c) [0x56027edc792c]
stderr: 9: (Option::size_t const md_config_t::get_val<Option::size_t>(ConfigValues
const&, std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&) const+0x51) [0x56027eedeea1]
stderr: 10: (BlueStore::_set_cache_sizes()+0x174) [0x56027f3fba44]
stderr: 11: (BlueStore::_open_bdev(bool)+0x1c5) [0x56027f3fe845]
stderr: 12: (BlueStore::mkfs()+0x6e0) [0x56027f484620]
stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x1b3) [0x56027eef9b23]
stderr: 14: (main()+0x1821) [0x56027eea68d1]
stderr: 15: (__libc_start_main()+0xeb) [0x7f17de927bbb]
stderr: 16: (_start()+0x2a) [0x56027eed903a]
stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
--> Was unable to complete a new OSD, will rollback changes
Running command: /bin/ceph --cluster test --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/test.keyring osd purge-new osd.0 --yes-i-really-mean-it
stderr: purged osd.0
Traceback (most recent call last):
File "/usr/sbin/ceph-volume", line 11, in <module>
load_entry_point('ceph-volume==1.0.0', 'console_scripts',
'ceph-volume')()
File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 38, in
__init__
self.main(self.argv)
File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in
newfunc
return f(*a, **kw)
File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 149, in main
terminal.dispatch(self.mapper, subcommand_args)
File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in
dispatch
instance.main()
File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line
40, in main
terminal.dispatch(self.mapper, self.argv)
File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in
dispatch
instance.main()
File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in
is_root
return func(*a, **kw)
File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line
325, in main
self.execute()
File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line
288, in execute
self.strategy.execute()
File
"/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/strategies/bluestore.py",
line 124, in execute
Create(command).main()
File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line
69, in main
self.create(args)
File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in
is_root
return func(*a, **kw)
File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line
26, in create
prepare_step.safe_prepare(args)
File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line
219, in safe_prepare
self.prepare()
File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in
is_root
return func(*a, **kw)
File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line
320, in prepare
osd_fsid,
File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line
119, in prepare_bluestore
db=db
File "/usr/lib/python3/dist-packages/ceph_volume/util/prepare.py", line 430,
in osd_mkfs_bluestore
raise RuntimeError('Command failed with exit code %s: %s' % (returncode, '
'.join(command)))
RuntimeError: Command failed with exit code 250: /bin/ceph-osd --cluster test
--osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/test-0/activate.monmap
--keyfile - --osd-data /var/lib/ceph/osd/test-0/ --osd-uuid
e3ebb6e0-82c8-4088-a6bd-abd729a575bb --setuser ceph --setgroup ceph
sysadmin@ceph-test:~$ sudo setenforce 1
sysadmin@ceph-test:~$
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Jan Fajerski
Senior Software Engineer Enterprise Storage
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Felix Imendörffer