Pacific bluestore_volume_selection_policy
After ~3 uneventful weeks after upgrading from 15.2.17 to 16.2.14 I’ve started seeing OSD crashes with "cur >= fnode.size” and "cur >= p.length”, which seems to be resolved in the next point release for pacific later this month, but until then, I’d love to keep the OSDs from flapping.
$ for crash in $(ceph crash ls | grep osd | awk '{print $1}') ; do ceph crash info $crash | egrep "(assert_condition|crash_id)" ; done "assert_condition": "cur >= fnode.size", "crash_id": "2024-01-03T09:07:55.698213Z_348af2d3-d4a7-4c27-9f71-70e6dc7c1af7", "assert_condition": "cur >= p.length", "crash_id": "2024-01-03T14:21:55.794692Z_4557c416-ffca-4165-aa91-d63698d41454", "assert_condition": "cur >= fnode.size", "crash_id": "2024-01-03T22:53:43.010010Z_15dc2b2a-30fb-4355-84b9-2f9560f08ea7", "assert_condition": "cur >= p.length", "crash_id": "2024-01-04T02:34:34.408976Z_2954a2c2-25d2-478e-92ad-d79c42d3ba43", "assert_condition": "cur2 >= p.length", "crash_id": "2024-01-04T21:57:07.100877Z_12f89c2c-4209-4f5a-b243-f0445ba629d2", "assert_condition": "cur >= p.length", "crash_id": "2024-01-05T00:35:08.561753Z_a189d967-ab02-4c61-bf68-1229222fd259", "assert_condition": "cur >= fnode.size", "crash_id": "2024-01-05T04:11:48.625086Z_a598cbaf-2c4f-4824-9939-1271eeba13ea", "assert_condition": "cur >= p.length", "crash_id": "2024-01-05T13:49:34.911210Z_953e38b9-8ae4-4cfe-8f22-d4b7cdf65cea", "assert_condition": "cur >= p.length", "crash_id": "2024-01-05T13:54:25.732770Z_4924b1c0-309c-4471-8c5d-c3aaea49166c", "assert_condition": "cur >= p.length", "crash_id": "2024-01-05T16:35:16.485416Z_0bca3d2a-2451-4275-a049-a65c58c1aff1”,
As noted in https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/YNJ35HXN4HX... <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/YNJ35HXN4HXF4XWB6IOZ2RKXX7EQCEIY/>
You can apparently work around the issue by setting 'bluestore_volume_selection_policy' config parameter to rocksdb_original.
However, after trying to set that parameter with `ceph config set osd.$osd bluestore_volume_selection_policy rocksdb_original` it doesn’t appear to set?
$ ceph config show-with-defaults osd.0 | grep bluestore_volume_selection_policy bluestore_volume_selection_policy use_some_extra
$ ceph config set osd.0 bluestore_volume_selection_policy rocksdb_original $ ceph config show osd.0 | grep bluestore_volume_selection_policy bluestore_volume_selection_policy use_some_extra default mom
This, I assume, should reflect the new setting, however it still shows the default “use_some_extra” value. But then this seems to imply that the config is set?
$ ceph config dump | grep bluestore_volume_selection_policy osd.0 dev bluestore_volume_selection_policy rocksdb_original * [snip] osd.9 dev bluestore_volume_selection_policy rocksdb_original *
Does this need to be set in ceph.conf or is there another setting that also needs to be set? Even after bouncing the OSD daemon, `ceph config show` still reports “use_some_extra" Appreciate any help they can offer to point me towards to bridge the gap between now and the next point release. Thanks, Reed
Hi, I just did the same in my lab environment and the config got applied to the daemon after a restart: pacific:~ # ceph tell osd.0 config show | grep bluestore_volume_selection_policy "bluestore_volume_selection_policy": "rocksdb_original", This is also a (tiny single-node) cluster running 16.2.14. Maybe you have some typo or something while doing the loop? Have you tried to set it for one OSD only and see if it starts with the config set? Zitat von Reed Dier <reed.dier@focusvq.com>:
After ~3 uneventful weeks after upgrading from 15.2.17 to 16.2.14 I’ve started seeing OSD crashes with "cur >= fnode.size” and "cur >= p.length”, which seems to be resolved in the next point release for pacific later this month, but until then, I’d love to keep the OSDs from flapping.
$ for crash in $(ceph crash ls | grep osd | awk '{print $1}') ; do ceph crash info $crash | egrep "(assert_condition|crash_id)" ; done "assert_condition": "cur >= fnode.size", "crash_id": "2024-01-03T09:07:55.698213Z_348af2d3-d4a7-4c27-9f71-70e6dc7c1af7", "assert_condition": "cur >= p.length", "crash_id": "2024-01-03T14:21:55.794692Z_4557c416-ffca-4165-aa91-d63698d41454", "assert_condition": "cur >= fnode.size", "crash_id": "2024-01-03T22:53:43.010010Z_15dc2b2a-30fb-4355-84b9-2f9560f08ea7", "assert_condition": "cur >= p.length", "crash_id": "2024-01-04T02:34:34.408976Z_2954a2c2-25d2-478e-92ad-d79c42d3ba43", "assert_condition": "cur2 >= p.length", "crash_id": "2024-01-04T21:57:07.100877Z_12f89c2c-4209-4f5a-b243-f0445ba629d2", "assert_condition": "cur >= p.length", "crash_id": "2024-01-05T00:35:08.561753Z_a189d967-ab02-4c61-bf68-1229222fd259", "assert_condition": "cur >= fnode.size", "crash_id": "2024-01-05T04:11:48.625086Z_a598cbaf-2c4f-4824-9939-1271eeba13ea", "assert_condition": "cur >= p.length", "crash_id": "2024-01-05T13:49:34.911210Z_953e38b9-8ae4-4cfe-8f22-d4b7cdf65cea", "assert_condition": "cur >= p.length", "crash_id": "2024-01-05T13:54:25.732770Z_4924b1c0-309c-4471-8c5d-c3aaea49166c", "assert_condition": "cur >= p.length", "crash_id": "2024-01-05T16:35:16.485416Z_0bca3d2a-2451-4275-a049-a65c58c1aff1”,
As noted in https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/YNJ35HXN4HX... <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/YNJ35HXN4HXF4XWB6IOZ2RKXX7EQCEIY/>
You can apparently work around the issue by setting 'bluestore_volume_selection_policy' config parameter to rocksdb_original.
However, after trying to set that parameter with `ceph config set osd.$osd bluestore_volume_selection_policy rocksdb_original` it doesn’t appear to set?
$ ceph config show-with-defaults osd.0 | grep bluestore_volume_selection_policy bluestore_volume_selection_policy use_some_extra
$ ceph config set osd.0 bluestore_volume_selection_policy rocksdb_original $ ceph config show osd.0 | grep bluestore_volume_selection_policy bluestore_volume_selection_policy use_some_extra default mom
This, I assume, should reflect the new setting, however it still shows the default “use_some_extra” value.
But then this seems to imply that the config is set?
$ ceph config dump | grep bluestore_volume_selection_policy osd.0 dev bluestore_volume_selection_policy rocksdb_original * [snip] osd.9 dev bluestore_volume_selection_policy rocksdb_original *
Does this need to be set in ceph.conf or is there another setting that also needs to be set? Even after bouncing the OSD daemon, `ceph config show` still reports “use_some_extra"
Appreciate any help they can offer to point me towards to bridge the gap between now and the next point release.
Thanks, Reed _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I ended up setting it in ceph.conf which appears to have worked (as far as I can tell).
[osd] bluestore_volume_selection_policy = rocksdb_original
$ ceph config show osd.0 | grep bluestore_volume_selection_policy bluestore_volume_selection_policy rocksdb_original file (mon[rocksdb_original])
So far so good… Reed
On Jan 8, 2024, at 2:04 AM, Eugen Block <eblock@nde.ag> wrote:
Hi,
I just did the same in my lab environment and the config got applied to the daemon after a restart:
pacific:~ # ceph tell osd.0 config show | grep bluestore_volume_selection_policy "bluestore_volume_selection_policy": "rocksdb_original",
This is also a (tiny single-node) cluster running 16.2.14. Maybe you have some typo or something while doing the loop? Have you tried to set it for one OSD only and see if it starts with the config set?
Zitat von Reed Dier <reed.dier@focusvq.com <mailto:reed.dier@focusvq.com>>:
After ~3 uneventful weeks after upgrading from 15.2.17 to 16.2.14 I’ve started seeing OSD crashes with "cur >= fnode.size” and "cur >= p.length”, which seems to be resolved in the next point release for pacific later this month, but until then, I’d love to keep the OSDs from flapping.
$ for crash in $(ceph crash ls | grep osd | awk '{print $1}') ; do ceph crash info $crash | egrep "(assert_condition|crash_id)" ; done "assert_condition": "cur >= fnode.size", "crash_id": "2024-01-03T09:07:55.698213Z_348af2d3-d4a7-4c27-9f71-70e6dc7c1af7", "assert_condition": "cur >= p.length", "crash_id": "2024-01-03T14:21:55.794692Z_4557c416-ffca-4165-aa91-d63698d41454", "assert_condition": "cur >= fnode.size", "crash_id": "2024-01-03T22:53:43.010010Z_15dc2b2a-30fb-4355-84b9-2f9560f08ea7", "assert_condition": "cur >= p.length", "crash_id": "2024-01-04T02:34:34.408976Z_2954a2c2-25d2-478e-92ad-d79c42d3ba43", "assert_condition": "cur2 >= p.length", "crash_id": "2024-01-04T21:57:07.100877Z_12f89c2c-4209-4f5a-b243-f0445ba629d2", "assert_condition": "cur >= p.length", "crash_id": "2024-01-05T00:35:08.561753Z_a189d967-ab02-4c61-bf68-1229222fd259", "assert_condition": "cur >= fnode.size", "crash_id": "2024-01-05T04:11:48.625086Z_a598cbaf-2c4f-4824-9939-1271eeba13ea", "assert_condition": "cur >= p.length", "crash_id": "2024-01-05T13:49:34.911210Z_953e38b9-8ae4-4cfe-8f22-d4b7cdf65cea", "assert_condition": "cur >= p.length", "crash_id": "2024-01-05T13:54:25.732770Z_4924b1c0-309c-4471-8c5d-c3aaea49166c", "assert_condition": "cur >= p.length", "crash_id": "2024-01-05T16:35:16.485416Z_0bca3d2a-2451-4275-a049-a65c58c1aff1”,
As noted in https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/YNJ35HXN4HX... <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/YNJ35HXN4HXF4XWB6IOZ2RKXX7EQCEIY/> <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/YNJ35HXN4HX... <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/YNJ35HXN4HXF4XWB6IOZ2RKXX7EQCEIY/>>
You can apparently work around the issue by setting 'bluestore_volume_selection_policy' config parameter to rocksdb_original.
However, after trying to set that parameter with `ceph config set osd.$osd bluestore_volume_selection_policy rocksdb_original` it doesn’t appear to set?
$ ceph config show-with-defaults osd.0 | grep bluestore_volume_selection_policy bluestore_volume_selection_policy use_some_extra
$ ceph config set osd.0 bluestore_volume_selection_policy rocksdb_original $ ceph config show osd.0 | grep bluestore_volume_selection_policy bluestore_volume_selection_policy use_some_extra default mom
This, I assume, should reflect the new setting, however it still shows the default “use_some_extra” value.
But then this seems to imply that the config is set?
$ ceph config dump | grep bluestore_volume_selection_policy osd.0 dev bluestore_volume_selection_policy rocksdb_original * [snip] osd.9 dev bluestore_volume_selection_policy rocksdb_original *
Does this need to be set in ceph.conf or is there another setting that also needs to be set? Even after bouncing the OSD daemon, `ceph config show` still reports “use_some_extra"
Appreciate any help they can offer to point me towards to bridge the gap between now and the next point release.
Thanks, Reed _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
Well, sadly, that setting doesn’t seem to resolve the issue. I set the value in ceph.conf for the OSDs with small WAL/DB devices that keep running into the issue,
$ ceph tell osd.12 config show | grep bluestore_volume_selection_policy "bluestore_volume_selection_policy": "rocksdb_original",
$ ceph crash info 2024-01-10T16:39:05.925534Z_f0c57ca3-b7e6-4511-b7ae-5834541d6c67 | egrep "(assert_condition|entity_name)" "assert_condition": "cur >= p.length", "entity_name": "osd.12",
So, I guess that configuration item doesn’t in fact prevent the crash as was purported. Looks like I may need to fast track moving to quincy… Reed
On Jan 8, 2024, at 9:47 AM, Reed Dier <reed.dier@focusvq.com> wrote:
I ended up setting it in ceph.conf which appears to have worked (as far as I can tell).
[osd] bluestore_volume_selection_policy = rocksdb_original
$ ceph config show osd.0 | grep bluestore_volume_selection_policy bluestore_volume_selection_policy rocksdb_original file (mon[rocksdb_original])
So far so good…
Reed
On Jan 8, 2024, at 2:04 AM, Eugen Block <eblock@nde.ag <mailto:eblock@nde.ag>> wrote:
Hi,
I just did the same in my lab environment and the config got applied to the daemon after a restart:
pacific:~ # ceph tell osd.0 config show | grep bluestore_volume_selection_policy "bluestore_volume_selection_policy": "rocksdb_original",
This is also a (tiny single-node) cluster running 16.2.14. Maybe you have some typo or something while doing the loop? Have you tried to set it for one OSD only and see if it starts with the config set?
Zitat von Reed Dier <reed.dier@focusvq.com <mailto:reed.dier@focusvq.com>>:
After ~3 uneventful weeks after upgrading from 15.2.17 to 16.2.14 I’ve started seeing OSD crashes with "cur >= fnode.size” and "cur >= p.length”, which seems to be resolved in the next point release for pacific later this month, but until then, I’d love to keep the OSDs from flapping.
$ for crash in $(ceph crash ls | grep osd | awk '{print $1}') ; do ceph crash info $crash | egrep "(assert_condition|crash_id)" ; done "assert_condition": "cur >= fnode.size", "crash_id": "2024-01-03T09:07:55.698213Z_348af2d3-d4a7-4c27-9f71-70e6dc7c1af7", "assert_condition": "cur >= p.length", "crash_id": "2024-01-03T14:21:55.794692Z_4557c416-ffca-4165-aa91-d63698d41454", "assert_condition": "cur >= fnode.size", "crash_id": "2024-01-03T22:53:43.010010Z_15dc2b2a-30fb-4355-84b9-2f9560f08ea7", "assert_condition": "cur >= p.length", "crash_id": "2024-01-04T02:34:34.408976Z_2954a2c2-25d2-478e-92ad-d79c42d3ba43", "assert_condition": "cur2 >= p.length", "crash_id": "2024-01-04T21:57:07.100877Z_12f89c2c-4209-4f5a-b243-f0445ba629d2", "assert_condition": "cur >= p.length", "crash_id": "2024-01-05T00:35:08.561753Z_a189d967-ab02-4c61-bf68-1229222fd259", "assert_condition": "cur >= fnode.size", "crash_id": "2024-01-05T04:11:48.625086Z_a598cbaf-2c4f-4824-9939-1271eeba13ea", "assert_condition": "cur >= p.length", "crash_id": "2024-01-05T13:49:34.911210Z_953e38b9-8ae4-4cfe-8f22-d4b7cdf65cea", "assert_condition": "cur >= p.length", "crash_id": "2024-01-05T13:54:25.732770Z_4924b1c0-309c-4471-8c5d-c3aaea49166c", "assert_condition": "cur >= p.length", "crash_id": "2024-01-05T16:35:16.485416Z_0bca3d2a-2451-4275-a049-a65c58c1aff1”,
As noted in https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/YNJ35HXN4HX... <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/YNJ35HXN4HXF4XWB6IOZ2RKXX7EQCEIY/> <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/YNJ35HXN4HX... <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/YNJ35HXN4HXF4XWB6IOZ2RKXX7EQCEIY/>>
You can apparently work around the issue by setting 'bluestore_volume_selection_policy' config parameter to rocksdb_original.
However, after trying to set that parameter with `ceph config set osd.$osd bluestore_volume_selection_policy rocksdb_original` it doesn’t appear to set?
$ ceph config show-with-defaults osd.0 | grep bluestore_volume_selection_policy bluestore_volume_selection_policy use_some_extra
$ ceph config set osd.0 bluestore_volume_selection_policy rocksdb_original $ ceph config show osd.0 | grep bluestore_volume_selection_policy bluestore_volume_selection_policy use_some_extra default mom
This, I assume, should reflect the new setting, however it still shows the default “use_some_extra” value.
But then this seems to imply that the config is set?
$ ceph config dump | grep bluestore_volume_selection_policy osd.0 dev bluestore_volume_selection_policy rocksdb_original * [snip] osd.9 dev bluestore_volume_selection_policy rocksdb_original *
Does this need to be set in ceph.conf or is there another setting that also needs to be set? Even after bouncing the OSD daemon, `ceph config show` still reports “use_some_extra"
Appreciate any help they can offer to point me towards to bridge the gap between now and the next point release.
Thanks, Reed _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
Hi Reed, it looks to me like your settings aren't effective. You might want to check OSD log rather than crash info and see the assertion's backtrace. Does it mention RocksDBBlueFSVolumeSelector as the one in https://tracker.ceph.com/issues/53906: ceph version 17.0.0-10229-g7e035110 (7e035110784fba02ba81944e444be9a36932c6a3) quincy (dev) 1: /lib64/libpthread.so.0(+0x12c20) [0x7f2beb318c20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x56347eb33bec] 5: /usr/bin/ceph-osd(+0x5d5daf) [0x56347eb33daf] 6: (RocksDBBlueFSVolumeSelector::add_usage(void*, bluefs_fnode_t const&)+0) [0x56347f1f7d00] 7: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x735) [0x56347f295b45] If so - then there is still a mess with proper parameter changes. Thanks Igor On 10/01/2024 20:13, Reed Dier wrote:
Well, sadly, that setting doesn’t seem to resolve the issue.
I set the value in ceph.conf for the OSDs with small WAL/DB devices that keep running into the issue,
$ ceph tell osd.12 config show | grep bluestore_volume_selection_policy "bluestore_volume_selection_policy": "rocksdb_original", $ ceph crash info 2024-01-10T16:39:05.925534Z_f0c57ca3-b7e6-4511-b7ae-5834541d6c67 | egrep "(assert_condition|entity_name)" "assert_condition": "cur >= p.length", "entity_name": "osd.12",
So, I guess that configuration item doesn’t in fact prevent the crash as was purported. Looks like I may need to fast track moving to quincy…
Reed
On Jan 8, 2024, at 9:47 AM, Reed Dier<reed.dier@focusvq.com> wrote:
I ended up setting it in ceph.conf which appears to have worked (as far as I can tell).
[osd] bluestore_volume_selection_policy = rocksdb_original $ ceph config show osd.0 | grep bluestore_volume_selection_policy bluestore_volume_selection_policy rocksdb_original file (mon[rocksdb_original]) So far so good…
Reed
On Jan 8, 2024, at 2:04 AM, Eugen Block <eblock@nde.ag <mailto:eblock@nde.ag>> wrote:
Hi,
I just did the same in my lab environment and the config got applied to the daemon after a restart:
pacific:~ # ceph tell osd.0 config show | grep bluestore_volume_selection_policy "bluestore_volume_selection_policy": "rocksdb_original",
This is also a (tiny single-node) cluster running 16.2.14. Maybe you have some typo or something while doing the loop? Have you tried to set it for one OSD only and see if it starts with the config set?
Zitat von Reed Dier <reed.dier@focusvq.com <mailto:reed.dier@focusvq.com>>:
After ~3 uneventful weeks after upgrading from 15.2.17 to 16.2.14 I’ve started seeing OSD crashes with "cur >= fnode.size” and "cur >= p.length”, which seems to be resolved in the next point release for pacific later this month, but until then, I’d love to keep the OSDs from flapping.
$ for crash in $(ceph crash ls | grep osd | awk '{print $1}') ; do ceph crash info $crash | egrep "(assert_condition|crash_id)" ; done "assert_condition": "cur >= fnode.size", "crash_id": "2024-01-03T09:07:55.698213Z_348af2d3-d4a7-4c27-9f71-70e6dc7c1af7", "assert_condition": "cur >= p.length", "crash_id": "2024-01-03T14:21:55.794692Z_4557c416-ffca-4165-aa91-d63698d41454", "assert_condition": "cur >= fnode.size", "crash_id": "2024-01-03T22:53:43.010010Z_15dc2b2a-30fb-4355-84b9-2f9560f08ea7", "assert_condition": "cur >= p.length", "crash_id": "2024-01-04T02:34:34.408976Z_2954a2c2-25d2-478e-92ad-d79c42d3ba43", "assert_condition": "cur2 >= p.length", "crash_id": "2024-01-04T21:57:07.100877Z_12f89c2c-4209-4f5a-b243-f0445ba629d2", "assert_condition": "cur >= p.length", "crash_id": "2024-01-05T00:35:08.561753Z_a189d967-ab02-4c61-bf68-1229222fd259", "assert_condition": "cur >= fnode.size", "crash_id": "2024-01-05T04:11:48.625086Z_a598cbaf-2c4f-4824-9939-1271eeba13ea", "assert_condition": "cur >= p.length", "crash_id": "2024-01-05T13:49:34.911210Z_953e38b9-8ae4-4cfe-8f22-d4b7cdf65cea", "assert_condition": "cur >= p.length", "crash_id": "2024-01-05T13:54:25.732770Z_4924b1c0-309c-4471-8c5d-c3aaea49166c", "assert_condition": "cur >= p.length", "crash_id": "2024-01-05T16:35:16.485416Z_0bca3d2a-2451-4275-a049-a65c58c1aff1”, As noted inhttps://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/YNJ35HXN4HX... <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/YNJ35HXN4HXF4XWB6IOZ2RKXX7EQCEIY/> <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/YNJ35HXN4HX... <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/YNJ35HXN4HXF4XWB6IOZ2RKXX7EQCEIY/>>
You can apparently work around the issue by setting 'bluestore_volume_selection_policy' config parameter to rocksdb_original. However, after trying to set that parameter with `ceph config set osd.$osd bluestore_volume_selection_policy rocksdb_original` it doesn’t appear to set?
$ ceph config show-with-defaults osd.0 | grep bluestore_volume_selection_policy bluestore_volume_selection_policy use_some_extra $ ceph config set osd.0 bluestore_volume_selection_policy rocksdb_original $ ceph config show osd.0 | grep bluestore_volume_selection_policy bluestore_volume_selection_policy use_some_extra default mom This, I assume, should reflect the new setting, however it still shows the default “use_some_extra” value.
But then this seems to imply that the config is set?
$ ceph config dump | grep bluestore_volume_selection_policy osd.0 dev bluestore_volume_selection_policy rocksdb_original * [snip] osd.9 dev bluestore_volume_selection_policy rocksdb_original * Does this need to be set in ceph.conf or is there another setting that also needs to be set? Even after bouncing the OSD daemon, `ceph config show` still reports “use_some_extra"
Appreciate any help they can offer to point me towards to bridge the gap between now and the next point release.
Thanks, Reed _______________________________________________ ceph-users mailing list --ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email toceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email toceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
Hi Igor, That’s correct (shown below). Would it be helpful for me to add logs/uploaded crash UUID’s to 53906 <https://tracker.ceph.com/issues/53906>, 53907 <https://tracker.ceph.com/issues/53907>, 54209 <https://tracker.ceph.com/issues/54209>, 62928 <https://tracker.ceph.com/issues/62928>, 63110 <https://tracker.ceph.com/issues/63110>, 63161 <https://tracker.ceph.com/issues/63161>, 63352 <https://tracker.ceph.com/issues/63352>? Or maybe open a new tracker to track that the parameter change isn’t being properly persisted or whatever appears to be happening? Thanks, Reed
/build/ceph-16.2.14/src/os/bluestore/BlueStore.h: 3870: FAILED ceph_assert(cur >= p.length)
ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x55d51970a987] 2: /usr/bin/ceph-osd(+0xad3b8f) [0x55d51970ab8f] 3: (RocksDBBlueFSVolumeSelector::sub_usage(void*, bluefs_fnode_t const&)+0x112) [0x55d519e040f2] 4: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x69d) [0x55d519ea0fad] 5: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xaa) [0x55d519ea14ea] 6: (BlueFS::fsync(BlueFS::FileWriter*)+0x7d) [0x55d519ec61ed] 7: (BlueRocksWritableFile::Sync()+0x19) [0x55d519ed5a59] 8: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x52) [0x55d51a3e37ce] 9: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x216) [0x55d51a5eddac] 10: (rocksdb::WritableFileWriter::Sync(bool)+0x17b) [0x55d51a5ed785] 11: (rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0x39a) [0x55d51a441bf8] 12: (rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x135e) [0x55d51a43d96c] 13: (rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x5d) [0x55d51a43c56f] 14: (RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x85) [0x55d51a388635] 15: (RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x9b) [0x55d51a38904b] 16: (BlueStore::_kv_sync_thread()+0x22bc) [0x55d519e016dc] 17: (BlueStore::KVSyncThread::entry()+0x11) [0x55d519e2de71] 18: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f490cf23609] 19: clone()
0> 2024-01-10T11:39:05.922-0500 7f48f978d700 -1 *** Caught signal (Aborted) ** in thread 7f48f978d700 thread_name:bstore_kv_sync
ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f490cf2f420] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1ad) [0x55d51970a9e2] 5: /usr/bin/ceph-osd(+0xad3b8f) [0x55d51970ab8f] 6: (RocksDBBlueFSVolumeSelector::sub_usage(void*, bluefs_fnode_t const&)+0x112) [0x55d519e040f2] 7: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x69d) [0x55d519ea0fad] 8: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xaa) [0x55d519ea14ea] 9: (BlueFS::fsync(BlueFS::FileWriter*)+0x7d) [0x55d519ec61ed] 10: (BlueRocksWritableFile::Sync()+0x19) [0x55d519ed5a59] 11: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x52) [0x55d51a3e37ce] 12: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x216) [0x55d51a5eddac] 13: (rocksdb::WritableFileWriter::Sync(bool)+0x17b) [0x55d51a5ed785] 14: (rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0x39a) [0x55d51a441bf8] 15: (rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x135e) [0x55d51a43d96c] 16: (rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x5d) [0x55d51a43c56f] 17: (RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x85) [0x55d51a388635] 18: (RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x9b) [0x55d51a38904b] 19: (BlueStore::_kv_sync_thread()+0x22bc) [0x55d519e016dc] 20: (BlueStore::KVSyncThread::entry()+0x11) [0x55d519e2de71] 21: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f490cf23609] 22: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
On Jan 10, 2024, at 12:06 PM, Igor Fedotov <igor.fedotov@croit.io> wrote:
Hi Reed,
it looks to me like your settings aren't effective. You might want to check OSD log rather than crash info and see the assertion's backtrace.
Does it mention RocksDBBlueFSVolumeSelector as the one in https://tracker.ceph.com/issues/53906 <https://tracker.ceph.com/issues/53906>:
ceph version 17.0.0-10229-g7e035110 (7e035110784fba02ba81944e444be9a36932c6a3) quincy (dev) 1: /lib64/libpthread.so.0(+0x12c20) [0x7f2beb318c20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x56347eb33bec] 5: /usr/bin/ceph-osd(+0x5d5daf) [0x56347eb33daf] 6: (RocksDBBlueFSVolumeSelector::add_usage(void*, bluefs_fnode_t const&)+0) [0x56347f1f7d00] 7: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x735) [0x56347f295b45]
If so - then there is still a mess with proper parameter changes.
Thanks Igor
On 10/01/2024 20:13, Reed Dier wrote:
Well, sadly, that setting doesn’t seem to resolve the issue.
I set the value in ceph.conf for the OSDs with small WAL/DB devices that keep running into the issue,
$ ceph tell osd.12 config show | grep bluestore_volume_selection_policy "bluestore_volume_selection_policy": "rocksdb_original", $ ceph crash info 2024-01-10T16:39:05.925534Z_f0c57ca3-b7e6-4511-b7ae-5834541d6c67 | egrep "(assert_condition|entity_name)" "assert_condition": "cur >= p.length", "entity_name": "osd.12",
So, I guess that configuration item doesn’t in fact prevent the crash as was purported. Looks like I may need to fast track moving to quincy…
Reed _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
Hi Reed, no much sense to attach the logs to the mentioned tickets - the problem with the assertion is well-known and has been already fixed. Your current issue is weird config update behavior which prevents from applying the work around. Feel free to open ticket about that but I don't think it's an efficient way - IIUC the problem isn't common and likely caused by something specific to your setup. Which rather means the fix wouldn't appear soon enough. Unfortunately that's not my area of expertise either so I'm of little help here as well. Nevertheless If I troubleshoot this config update issue I'd start the investigation by trying different parameters/daemons/hosts. Are you able to tune any parameter at all? Is it doable at different host or OSD? Not to mention that you might just try to restart monitors first ;) Thanks, Igor On 10/01/2024 21:38, Reed Dier wrote:
Hi Igor,
That’s correct (shown below). Would it be helpful for me to add logs/uploaded crash UUID’s to 53906 <https://tracker.ceph.com/issues/53906>, 53907 <https://tracker.ceph.com/issues/53907>, 54209 <https://tracker.ceph.com/issues/54209>, 62928 <https://tracker.ceph.com/issues/62928>, 63110 <https://tracker.ceph.com/issues/63110>, 63161 <https://tracker.ceph.com/issues/63161>, 63352 <https://tracker.ceph.com/issues/63352>? Or maybe open a new tracker to track that the parameter change isn’t being properly persisted or whatever appears to be happening?
Thanks, Reed
/build/ceph-16.2.14/src/os/bluestore/BlueStore.h: 3870: FAILED ceph_assert(cur >= p.length)
ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x55d51970a987] 2: /usr/bin/ceph-osd(+0xad3b8f) [0x55d51970ab8f] 3: (RocksDBBlueFSVolumeSelector::sub_usage(void*, bluefs_fnode_t const&)+0x112) [0x55d519e040f2] 4: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x69d) [0x55d519ea0fad] 5: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xaa) [0x55d519ea14ea] 6: (BlueFS::fsync(BlueFS::FileWriter*)+0x7d) [0x55d519ec61ed] 7: (BlueRocksWritableFile::Sync()+0x19) [0x55d519ed5a59] 8: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x52) [0x55d51a3e37ce] 9: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x216) [0x55d51a5eddac] 10: (rocksdb::WritableFileWriter::Sync(bool)+0x17b) [0x55d51a5ed785] 11: (rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0x39a) [0x55d51a441bf8] 12: (rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x135e) [0x55d51a43d96c] 13: (rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x5d) [0x55d51a43c56f] 14: (RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x85) [0x55d51a388635] 15: (RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x9b) [0x55d51a38904b] 16: (BlueStore::_kv_sync_thread()+0x22bc) [0x55d519e016dc] 17: (BlueStore::KVSyncThread::entry()+0x11) [0x55d519e2de71] 18: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f490cf23609] 19: clone()
0> 2024-01-10T11:39:05.922-0500 7f48f978d700 -1 *** Caught signal (Aborted) ** in thread 7f48f978d700 thread_name:bstore_kv_sync
ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f490cf2f420] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1ad) [0x55d51970a9e2] 5: /usr/bin/ceph-osd(+0xad3b8f) [0x55d51970ab8f] 6: (RocksDBBlueFSVolumeSelector::sub_usage(void*, bluefs_fnode_t const&)+0x112) [0x55d519e040f2] 7: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x69d) [0x55d519ea0fad] 8: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xaa) [0x55d519ea14ea] 9: (BlueFS::fsync(BlueFS::FileWriter*)+0x7d) [0x55d519ec61ed] 10: (BlueRocksWritableFile::Sync()+0x19) [0x55d519ed5a59] 11: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x52) [0x55d51a3e37ce] 12: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x216) [0x55d51a5eddac] 13: (rocksdb::WritableFileWriter::Sync(bool)+0x17b) [0x55d51a5ed785] 14: (rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0x39a) [0x55d51a441bf8] 15: (rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x135e) [0x55d51a43d96c] 16: (rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x5d) [0x55d51a43c56f] 17: (RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x85) [0x55d51a388635] 18: (RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x9b) [0x55d51a38904b] 19: (BlueStore::_kv_sync_thread()+0x22bc) [0x55d519e016dc] 20: (BlueStore::KVSyncThread::entry()+0x11) [0x55d519e2de71] 21: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f490cf23609] 22: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
On Jan 10, 2024, at 12:06 PM, Igor Fedotov <igor.fedotov@croit.io> wrote:
Hi Reed,
it looks to me like your settings aren't effective. You might want to check OSD log rather than crash info and see the assertion's backtrace.
Does it mention RocksDBBlueFSVolumeSelector as the one in https://tracker.ceph.com/issues/53906:
ceph version 17.0.0-10229-g7e035110 (7e035110784fba02ba81944e444be9a36932c6a3) quincy (dev) 1: /lib64/libpthread.so.0(+0x12c20) [0x7f2beb318c20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x56347eb33bec] 5: /usr/bin/ceph-osd(+0x5d5daf) [0x56347eb33daf] 6: (RocksDBBlueFSVolumeSelector::add_usage(void*, bluefs_fnode_t const&)+0) [0x56347f1f7d00] 7: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x735) [0x56347f295b45]
If so - then there is still a mess with proper parameter changes.
Thanks Igor
On 10/01/2024 20:13, Reed Dier wrote:
Well, sadly, that setting doesn’t seem to resolve the issue.
I set the value in ceph.conf for the OSDs with small WAL/DB devices that keep running into the issue,
$ ceph tell osd.12 config show | grep bluestore_volume_selection_policy "bluestore_volume_selection_policy": "rocksdb_original", $ ceph crash info 2024-01-10T16:39:05.925534Z_f0c57ca3-b7e6-4511-b7ae-5834541d6c67 | egrep "(assert_condition|entity_name)" "assert_condition": "cur >= p.length", "entity_name": "osd.12", So, I guess that configuration item doesn’t in fact prevent the crash as was purported. Looks like I may need to fast track moving to quincy…
Reed _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
participants (3)
-
Eugen Block
-
Igor Fedotov
-
Reed Dier