resharding RocksDB after upgrade to Pacific breaks OSDs
Hi we upgraded our Ceph cluster from latest Octopus to Pacific 16.2.14 and then we followed the docs (https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#ro... <https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#rocksdb-sharding>) to reshard RocksDB on our OSDs. Despite resharding reports operation as successful, OSD fails to start. # ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-5/ --sharding="m(3) p(3,0-12) o(3,0-13)=block_cache={type=binned_lru} l p" reshard reshard success Oct 30 12:44:17 octopus2 ceph-osd[4521]: /build/ceph-16.2.14/src/kv/RocksDBStore.cc: 1223: FAILED ceph_assert(recreate_mode) Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x564047cb92b2] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 0> 2023-10-30T12:44:17.088+0000 7f4971ed2100 -1 *** Caught signal (Aborted) ** Oct 30 12:44:17 octopus2 ceph-osd[4521]: in thread 7f4971ed2100 thread_name:ceph-osd Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f4972921730] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: gsignal() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: abort() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x564047cb9303] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 11: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 12: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 13: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Oct 30 12:44:17 octopus2 ceph-osd[4521]: -1> 2023-10-30T12:44:17.084+0000 7f4971ed2100 -1 /build/ceph-16.2.14/src/kv/RocksDBStore.cc: In function 'int RocksDBStore::do_open(std::ostream&, bool, bool, const string&)' thread 7f4971ed2100 time 2023-10-30T12:44:17.087172+0000 I've submitted bug report here https://tracker.ceph.com/issues/63353 but may be community here have some ideas how to fix it unless it's really a bug. Thanks
Hi, this seems like a dangerous operation to me, I tried the same on two different virtual clusters, Reef and Pacific (all upgraded from previous releases). In Reef the reshard fails alltogether and the OSD fails to start, I had to recreate it. In Pacific the reshard reports a successful operation, but the OSD fails to start as well, with the same stack trace as yours. I wasn't aware of this resharding operation yet, but is it really safe? I don't have an idea how to fix, I just recreated the OSDs. Zitat von Denis Polom <denispolom@gmail.com>:
Hi
we upgraded our Ceph cluster from latest Octopus to Pacific 16.2.14 and then we followed the docs (https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#ro... <https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#rocksdb-sharding>) to reshard RocksDB on our OSDs.
Despite resharding reports operation as successful, OSD fails to start.
# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-5/ --sharding="m(3) p(3,0-12) o(3,0-13)=block_cache={type=binned_lru} l p" reshard reshard success
Oct 30 12:44:17 octopus2 ceph-osd[4521]: /build/ceph-16.2.14/src/kv/RocksDBStore.cc: 1223: FAILED ceph_assert(recreate_mode) Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x564047cb92b2] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 0> 2023-10-30T12:44:17.088+0000 7f4971ed2100 -1 *** Caught signal (Aborted) ** Oct 30 12:44:17 octopus2 ceph-osd[4521]: in thread 7f4971ed2100 thread_name:ceph-osd Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f4972921730] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: gsignal() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: abort() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x564047cb9303] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 11: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 12: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 13: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Oct 30 12:44:17 octopus2 ceph-osd[4521]: -1> 2023-10-30T12:44:17.084+0000 7f4971ed2100 -1 /build/ceph-16.2.14/src/kv/RocksDBStore.cc: In function 'int RocksDBStore::do_open(std::ostream&, bool, bool, const string&)' thread 7f4971ed2100 time 2023-10-30T12:44:17.087172+0000
I've submitted bug report here https://tracker.ceph.com/issues/63353 but may be community here have some ideas how to fix it unless it's really a bug.
Thanks _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, yes, exactly. I had to recreate OSD as well because daemon wasn't able to start. It's obviously a bug and should be fixed either in documentation or code. On 11/3/23 11:45, Eugen Block wrote:
Hi,
this seems like a dangerous operation to me, I tried the same on two different virtual clusters, Reef and Pacific (all upgraded from previous releases). In Reef the reshard fails alltogether and the OSD fails to start, I had to recreate it. In Pacific the reshard reports a successful operation, but the OSD fails to start as well, with the same stack trace as yours. I wasn't aware of this resharding operation yet, but is it really safe? I don't have an idea how to fix, I just recreated the OSDs.
Zitat von Denis Polom <denispolom@gmail.com>:
Hi
we upgraded our Ceph cluster from latest Octopus to Pacific 16.2.14 and then we followed the docs (https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#ro... <https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#rocksdb-sharding>) to reshard RocksDB on our OSDs.
Despite resharding reports operation as successful, OSD fails to start.
# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-5/ --sharding="m(3) p(3,0-12) o(3,0-13)=block_cache={type=binned_lru} l p" reshard reshard success
Oct 30 12:44:17 octopus2 ceph-osd[4521]: /build/ceph-16.2.14/src/kv/RocksDBStore.cc: 1223: FAILED ceph_assert(recreate_mode) Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x564047cb92b2] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 0> 2023-10-30T12:44:17.088+0000 7f4971ed2100 -1 *** Caught signal (Aborted) ** Oct 30 12:44:17 octopus2 ceph-osd[4521]: in thread 7f4971ed2100 thread_name:ceph-osd Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f4972921730] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: gsignal() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: abort() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x564047cb9303] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 11: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 12: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 13: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Oct 30 12:44:17 octopus2 ceph-osd[4521]: -1> 2023-10-30T12:44:17.084+0000 7f4971ed2100 -1 /build/ceph-16.2.14/src/kv/RocksDBStore.cc: In function 'int RocksDBStore::do_open(std::ostream&, bool, bool, const string&)' thread 7f4971ed2100 time 2023-10-30T12:44:17.087172+0000
I've submitted bug report here https://tracker.ceph.com/issues/63353 but may be community here have some ideas how to fix it unless it's really a bug.
Thanks _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
The ticket has been updated, but it's probably important enough to state on the list as well: The documentation is currently wrong in a way that running the command as documented will cause this corruption. The correct command to run is: ceph-bluestore-tool \ --path <data path> \ --sharding="m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P" \ reshard Josh On Fri, Nov 3, 2023 at 7:58 AM Denis Polom <denispolom@gmail.com> wrote:
Hi,
yes, exactly. I had to recreate OSD as well because daemon wasn't able to start.
It's obviously a bug and should be fixed either in documentation or code.
On 11/3/23 11:45, Eugen Block wrote:
Hi,
this seems like a dangerous operation to me, I tried the same on two different virtual clusters, Reef and Pacific (all upgraded from previous releases). In Reef the reshard fails alltogether and the OSD fails to start, I had to recreate it. In Pacific the reshard reports a successful operation, but the OSD fails to start as well, with the same stack trace as yours. I wasn't aware of this resharding operation yet, but is it really safe? I don't have an idea how to fix, I just recreated the OSDs.
Zitat von Denis Polom <denispolom@gmail.com>:
Hi
we upgraded our Ceph cluster from latest Octopus to Pacific 16.2.14 and then we followed the docs (https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#ro... <https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#rocksdb-sharding>) to reshard RocksDB on our OSDs.
Despite resharding reports operation as successful, OSD fails to start.
# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-5/ --sharding="m(3) p(3,0-12) o(3,0-13)=block_cache={type=binned_lru} l p" reshard reshard success
Oct 30 12:44:17 octopus2 ceph-osd[4521]: /build/ceph-16.2.14/src/kv/RocksDBStore.cc: 1223: FAILED ceph_assert(recreate_mode) Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x564047cb92b2] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 0> 2023-10-30T12:44:17.088+0000 7f4971ed2100 -1 *** Caught signal (Aborted) ** Oct 30 12:44:17 octopus2 ceph-osd[4521]: in thread 7f4971ed2100 thread_name:ceph-osd Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f4972921730] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: gsignal() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: abort() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x564047cb9303] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 11: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 12: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 13: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Oct 30 12:44:17 octopus2 ceph-osd[4521]: -1> 2023-10-30T12:44:17.084+0000 7f4971ed2100 -1 /build/ceph-16.2.14/src/kv/RocksDBStore.cc: In function 'int RocksDBStore::do_open(std::ostream&, bool, bool, const string&)' thread 7f4971ed2100 time 2023-10-30T12:44:17.087172+0000
I've submitted bug report here https://tracker.ceph.com/issues/63353 but may be community here have some ideas how to fix it unless it's really a bug.
Thanks _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yes, Josh beat me to it- this is an issue of incorrectly documenting the command. You can try the solution posted in the tracker issue. On Fri, Nov 3, 2023 at 10:43 AM Josh Baergen <jbaergen@digitalocean.com> wrote:
The ticket has been updated, but it's probably important enough to state on the list as well: The documentation is currently wrong in a way that running the command as documented will cause this corruption. The correct command to run is:
ceph-bluestore-tool \ --path <data path> \ --sharding="m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P" \ reshard
Josh
On Fri, Nov 3, 2023 at 7:58 AM Denis Polom <denispolom@gmail.com> wrote:
Hi,
yes, exactly. I had to recreate OSD as well because daemon wasn't able to start.
It's obviously a bug and should be fixed either in documentation or code.
On 11/3/23 11:45, Eugen Block wrote:
Hi,
this seems like a dangerous operation to me, I tried the same on two different virtual clusters, Reef and Pacific (all upgraded from previous releases). In Reef the reshard fails alltogether and the OSD fails to start, I had to recreate it. In Pacific the reshard reports a successful operation, but the OSD fails to start as well, with the same stack trace as yours. I wasn't aware of this resharding operation yet, but is it really safe? I don't have an idea how to fix, I just recreated the OSDs.
Zitat von Denis Polom <denispolom@gmail.com>:
Hi
we upgraded our Ceph cluster from latest Octopus to Pacific 16.2.14 and then we followed the docs (
< https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#ro... ) to reshard RocksDB on our OSDs.
Despite resharding reports operation as successful, OSD fails to start.
# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-5/ --sharding="m(3) p(3,0-12) o(3,0-13)=block_cache={type=binned_lru} l p" reshard reshard success
Oct 30 12:44:17 octopus2 ceph-osd[4521]: /build/ceph-16.2.14/src/kv/RocksDBStore.cc: 1223: FAILED ceph_assert(recreate_mode) Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x564047cb92b2] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 0> 2023-10-30T12:44:17.088+0000 7f4971ed2100 -1 *** Caught signal (Aborted) ** Oct 30 12:44:17 octopus2 ceph-osd[4521]: in thread 7f4971ed2100 thread_name:ceph-osd Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f4972921730] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: gsignal() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: abort() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x564047cb9303] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 11: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 12: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 13: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret
https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#ro... this.
Oct 30 12:44:17 octopus2 ceph-osd[4521]: -1> 2023-10-30T12:44:17.084+0000 7f4971ed2100 -1 /build/ceph-16.2.14/src/kv/RocksDBStore.cc: In function 'int RocksDBStore::do_open(std::ostream&, bool, bool, const string&)' thread 7f4971ed2100 time 2023-10-30T12:44:17.087172+0000
I've submitted bug report here https://tracker.ceph.com/issues/63353 but may be community here have some ideas how to fix it unless it's really a bug.
Thanks _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
If someone can point me at the errant docs locus I'll make it right.
On Nov 3, 2023, at 11:45, Laura Flores <lflores@redhat.com> wrote:
Yes, Josh beat me to it- this is an issue of incorrectly documenting the command. You can try the solution posted in the tracker issue.
On Fri, Nov 3, 2023 at 10:43 AM Josh Baergen <jbaergen@digitalocean.com> wrote:
The ticket has been updated, but it's probably important enough to state on the list as well: The documentation is currently wrong in a way that running the command as documented will cause this corruption. The correct command to run is:
ceph-bluestore-tool \ --path <data path> \ --sharding="m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P" \ reshard
Josh
On Fri, Nov 3, 2023 at 7:58 AM Denis Polom <denispolom@gmail.com> wrote:
Hi,
yes, exactly. I had to recreate OSD as well because daemon wasn't able to start.
It's obviously a bug and should be fixed either in documentation or code.
On 11/3/23 11:45, Eugen Block wrote:
Hi,
this seems like a dangerous operation to me, I tried the same on two different virtual clusters, Reef and Pacific (all upgraded from previous releases). In Reef the reshard fails alltogether and the OSD fails to start, I had to recreate it. In Pacific the reshard reports a successful operation, but the OSD fails to start as well, with the same stack trace as yours. I wasn't aware of this resharding operation yet, but is it really safe? I don't have an idea how to fix, I just recreated the OSDs.
Zitat von Denis Polom <denispolom@gmail.com>:
Hi
we upgraded our Ceph cluster from latest Octopus to Pacific 16.2.14 and then we followed the docs (
< https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#ro... ) to reshard RocksDB on our OSDs.
Despite resharding reports operation as successful, OSD fails to start.
# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-5/ --sharding="m(3) p(3,0-12) o(3,0-13)=block_cache={type=binned_lru} l p" reshard reshard success
Oct 30 12:44:17 octopus2 ceph-osd[4521]: /build/ceph-16.2.14/src/kv/RocksDBStore.cc: 1223: FAILED ceph_assert(recreate_mode) Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x564047cb92b2] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 0> 2023-10-30T12:44:17.088+0000 7f4971ed2100 -1 *** Caught signal (Aborted) ** Oct 30 12:44:17 octopus2 ceph-osd[4521]: in thread 7f4971ed2100 thread_name:ceph-osd Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f4972921730] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: gsignal() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: abort() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x564047cb9303] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 11: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 12: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 13: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret
https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#ro... this.
Oct 30 12:44:17 octopus2 ceph-osd[4521]: -1> 2023-10-30T12:44:17.084+0000 7f4971ed2100 -1 /build/ceph-16.2.14/src/kv/RocksDBStore.cc: In function 'int RocksDBStore::do_open(std::ostream&, bool, bool, const string&)' thread 7f4971ed2100 time 2023-10-30T12:44:17.087172+0000
I've submitted bug report here https://tracker.ceph.com/issues/63353 but may be community here have some ideas how to fix it unless it's really a bug.
Thanks _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I notice the documentation has been updated to put the L and P at the end of the --sharding option in uppercase, but the O(3,0-13) option is still lowercase: https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#ro... - Nelson On 11/3/23 11:13, Anthony D'Atri wrote:
If someone can point me at the errant docs locus I'll make it right.
On Nov 3, 2023, at 11:45, Laura Flores <lflores@redhat.com> wrote:
Yes, Josh beat me to it- this is an issue of incorrectly documenting the command. You can try the solution posted in the tracker issue.
On Fri, Nov 3, 2023 at 10:43 AM Josh Baergen <jbaergen@digitalocean.com> wrote:
The ticket has been updated, but it's probably important enough to state on the list as well: The documentation is currently wrong in a way that running the command as documented will cause this corruption. The correct command to run is:
ceph-bluestore-tool \ --path <data path> \ --sharding="m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P" \ reshard
Josh
Hi,
yes, exactly. I had to recreate OSD as well because daemon wasn't able to start.
It's obviously a bug and should be fixed either in documentation or code.
On 11/3/23 11:45, Eugen Block wrote:
Hi,
this seems like a dangerous operation to me, I tried the same on two different virtual clusters, Reef and Pacific (all upgraded from previous releases). In Reef the reshard fails alltogether and the OSD fails to start, I had to recreate it. In Pacific the reshard reports a successful operation, but the OSD fails to start as well, with the same stack trace as yours. I wasn't aware of this resharding operation yet, but is it really safe? I don't have an idea how to fix, I just recreated the OSDs.
Zitat von Denis Polom <denispolom@gmail.com>:
Hi
we upgraded our Ceph cluster from latest Octopus to Pacific 16.2.14 and then we followed the docs ( https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#ro... < https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#ro... ) to reshard RocksDB on our OSDs.
Despite resharding reports operation as successful, OSD fails to start. # ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-5/ --sharding="m(3) p(3,0-12) o(3,0-13)=block_cache={type=binned_lru} l p" reshard reshard success
Oct 30 12:44:17 octopus2 ceph-osd[4521]: /build/ceph-16.2.14/src/kv/RocksDBStore.cc: 1223: FAILED ceph_assert(recreate_mode) Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x564047cb92b2] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 0> 2023-10-30T12:44:17.088+0000 7f4971ed2100 -1 *** Caught signal (Aborted) ** Oct 30 12:44:17 octopus2 ceph-osd[4521]: in thread 7f4971ed2100 thread_name:ceph-osd Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f4972921730] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: gsignal() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: abort() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x564047cb9303] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 11: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 12: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 13: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret
On Fri, Nov 3, 2023 at 7:58 AM Denis Polom <denispolom@gmail.com> wrote: this.
Oct 30 12:44:17 octopus2 ceph-osd[4521]: -1> 2023-10-30T12:44:17.084+0000 7f4971ed2100 -1 /build/ceph-16.2.14/src/kv/RocksDBStore.cc: In function 'int RocksDBStore::do_open(std::ostream&, bool, bool, const string&)' thread 7f4971ed2100 time 2023-10-30T12:44:17.087172+0000
I've submitted bug report here https://tracker.ceph.com/issues/63353 but may be community here have some ideas how to fix it unless it's really a bug.
Thanks _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Nelson Hicks Information Technology SOCKET (573) 817-0000 ext. 210 nelsonh@socket.net
nm, Adam beat me to it.
On Nov 3, 2023, at 11:40, Josh Baergen <jbaergen@digitalocean.com> wrote:
The ticket has been updated, but it's probably important enough to state on the list as well: The documentation is currently wrong in a way that running the command as documented will cause this corruption. The correct command to run is:
ceph-bluestore-tool \ --path <data path> \ --sharding="m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P" \ reshard
Josh
On Fri, Nov 3, 2023 at 7:58 AM Denis Polom <denispolom@gmail.com> wrote:
Hi,
yes, exactly. I had to recreate OSD as well because daemon wasn't able to start.
It's obviously a bug and should be fixed either in documentation or code.
On 11/3/23 11:45, Eugen Block wrote:
Hi,
this seems like a dangerous operation to me, I tried the same on two different virtual clusters, Reef and Pacific (all upgraded from previous releases). In Reef the reshard fails alltogether and the OSD fails to start, I had to recreate it. In Pacific the reshard reports a successful operation, but the OSD fails to start as well, with the same stack trace as yours. I wasn't aware of this resharding operation yet, but is it really safe? I don't have an idea how to fix, I just recreated the OSDs.
Zitat von Denis Polom <denispolom@gmail.com>:
Hi
we upgraded our Ceph cluster from latest Octopus to Pacific 16.2.14 and then we followed the docs (https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#ro... <https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#rocksdb-sharding>) to reshard RocksDB on our OSDs.
Despite resharding reports operation as successful, OSD fails to start.
# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-5/ --sharding="m(3) p(3,0-12) o(3,0-13)=block_cache={type=binned_lru} l p" reshard reshard success
Oct 30 12:44:17 octopus2 ceph-osd[4521]: /build/ceph-16.2.14/src/kv/RocksDBStore.cc: 1223: FAILED ceph_assert(recreate_mode) Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x564047cb92b2] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 0> 2023-10-30T12:44:17.088+0000 7f4971ed2100 -1 *** Caught signal (Aborted) ** Oct 30 12:44:17 octopus2 ceph-osd[4521]: in thread 7f4971ed2100 thread_name:ceph-osd Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f4972921730] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: gsignal() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: abort() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x564047cb9303] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 11: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 12: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 13: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Oct 30 12:44:17 octopus2 ceph-osd[4521]: -1> 2023-10-30T12:44:17.084+0000 7f4971ed2100 -1 /build/ceph-16.2.14/src/kv/RocksDBStore.cc: In function 'int RocksDBStore::do_open(std::ostream&, bool, bool, const string&)' thread 7f4971ed2100 time 2023-10-30T12:44:17.087172+0000
I've submitted bug report here https://tracker.ceph.com/issues/63353 but may be community here have some ideas how to fix it unless it's really a bug.
Thanks _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks guys, this works! On 11/3/23 17:15, Anthony D'Atri wrote:
nm, Adam beat me to it.
On Nov 3, 2023, at 11:40, Josh Baergen <jbaergen@digitalocean.com> wrote:
The ticket has been updated, but it's probably important enough to state on the list as well: The documentation is currently wrong in a way that running the command as documented will cause this corruption. The correct command to run is:
ceph-bluestore-tool \ --path <data path> \ --sharding="m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P" \ reshard
Josh
On Fri, Nov 3, 2023 at 7:58 AM Denis Polom <denispolom@gmail.com> wrote:
Hi,
yes, exactly. I had to recreate OSD as well because daemon wasn't able to start.
It's obviously a bug and should be fixed either in documentation or code.
On 11/3/23 11:45, Eugen Block wrote:
Hi,
this seems like a dangerous operation to me, I tried the same on two different virtual clusters, Reef and Pacific (all upgraded from previous releases). In Reef the reshard fails alltogether and the OSD fails to start, I had to recreate it. In Pacific the reshard reports a successful operation, but the OSD fails to start as well, with the same stack trace as yours. I wasn't aware of this resharding operation yet, but is it really safe? I don't have an idea how to fix, I just recreated the OSDs.
Zitat von Denis Polom <denispolom@gmail.com>:
Hi
we upgraded our Ceph cluster from latest Octopus to Pacific 16.2.14 and then we followed the docs (https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#ro... <https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#rocksdb-sharding>) to reshard RocksDB on our OSDs.
Despite resharding reports operation as successful, OSD fails to start.
# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-5/ --sharding="m(3) p(3,0-12) o(3,0-13)=block_cache={type=binned_lru} l p" reshard reshard success
Oct 30 12:44:17 octopus2 ceph-osd[4521]: /build/ceph-16.2.14/src/kv/RocksDBStore.cc: 1223: FAILED ceph_assert(recreate_mode) Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x564047cb92b2] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 0> 2023-10-30T12:44:17.088+0000 7f4971ed2100 -1 *** Caught signal (Aborted) ** Oct 30 12:44:17 octopus2 ceph-osd[4521]: in thread 7f4971ed2100 thread_name:ceph-osd Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f4972921730] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: gsignal() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: abort() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x564047cb9303] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 11: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 12: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 13: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Oct 30 12:44:17 octopus2 ceph-osd[4521]: -1> 2023-10-30T12:44:17.084+0000 7f4971ed2100 -1 /build/ceph-16.2.14/src/kv/RocksDBStore.cc: In function 'int RocksDBStore::do_open(std::ostream&, bool, bool, const string&)' thread 7f4971ed2100 time 2023-10-30T12:44:17.087172+0000
I've submitted bug report here https://tracker.ceph.com/issues/63353 but may be community here have some ideas how to fix it unless it's really a bug.
Thanks _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (7)
-
Anthony D'Atri
-
Anthony D'Atri
-
Denis Polom
-
Eugen Block
-
Josh Baergen
-
Laura Flores
-
Nelson Hicks