Hello, yesterday we upgraded a mimic cluster to v14.2.10, everything was running and ok. There was this new warning, 2 pool(s) have non-power-of-two pg_num and to get a HEALTH_OK state until we can expand this pools, i found this config option to suppress the warning: ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false which resulted in a crash of 40 osd processes (about 60% of the cluster). no restart possible, always the same crash. 2020-06-30 21:13:56.179 7fd2b7708c00 -1 osd.30 385679 log_to_monitors {default=true} *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] 2020-06-30 21:13:56.199 7fd2a5813700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -1547> 2020-06-30 21:13:51.171 7fd2b7708c00 -1 missing 'type' file, inferring filestore from current/ dir -738> 2020-06-30 21:13:56.179 7fd2b7708c00 -1 osd.30 385679 log_to_monitors {default=true} 0> 2020-06-30 21:13:56.199 7fd2a5813700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -1547> 2020-06-30 21:13:51.171 7fd2b7708c00 -1 missing 'type' file, inferring filestore from current/ dir -738> 2020-06-30 21:13:56.179 7fd2b7708c00 -1 osd.30 385679 log_to_monitors {default=true} 0> 2020-06-30 21:13:56.199 7fd2a5813700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. This is a mixed cluster of ubuntu xenial and bionic, it happens on both. It look's like, it happens when the new monmap arrived at the osd. The only fix i was able to come up with, downgrade ceph-osd to v14.2.9. Should i open a bug report? Regards Markus
Hi Markus, Yes, I think you should open a bug tracker with more from a crashing osd log file (e.g. all the -1> -2> etc. lines before the crash) and also from the mon leader if possible. Something strange is that the mon_warn_on_pool_pg_num_not_power_of_two feature is also present in v14.2.9 (it was added in v14.2.8). Which version did you upgrade from? Perhaps setting it to false was the trigger, but the crash is somewhere else in the OSD changes in v14.2.10. Cheers, Dan On Wed, Jul 1, 2020 at 9:09 AM Markus Binz <mbinz@solnet.ch> wrote:
Hello,
yesterday we upgraded a mimic cluster to v14.2.10, everything was running and ok.
There was this new warning, 2 pool(s) have non-power-of-two pg_num and to get a HEALTH_OK state until we can expand this pools, i found this config option to suppress the warning:
ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false which resulted in a crash of 40 osd processes (about 60% of the cluster).
no restart possible, always the same crash.
2020-06-30 21:13:56.179 7fd2b7708c00 -1 osd.30 385679 log_to_monitors {default=true} *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] 2020-06-30 21:13:56.199 7fd2a5813700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore
ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
-1547> 2020-06-30 21:13:51.171 7fd2b7708c00 -1 missing 'type' file, inferring filestore from current/ dir -738> 2020-06-30 21:13:56.179 7fd2b7708c00 -1 osd.30 385679 log_to_monitors {default=true} 0> 2020-06-30 21:13:56.199 7fd2a5813700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore
ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
-1547> 2020-06-30 21:13:51.171 7fd2b7708c00 -1 missing 'type' file, inferring filestore from current/ dir -738> 2020-06-30 21:13:56.179 7fd2b7708c00 -1 osd.30 385679 log_to_monitors {default=true} 0> 2020-06-30 21:13:56.199 7fd2a5813700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore
ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
This is a mixed cluster of ubuntu xenial and bionic, it happens on both.
It look's like, it happens when the new monmap arrived at the osd.
The only fix i was able to come up with, downgrade ceph-osd to v14.2.9.
Should i open a bug report?
Regards
Markus _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Markus, Did you make any progress with this? (Selfishly pinging to better understand whether the 14.2.9 to 14.2.10 upgrade is safe or not) Cheers, Dan On Wed, Jul 1, 2020 at 9:30 AM Dan van der Ster <dan@vanderster.com> wrote:
Hi Markus,
Yes, I think you should open a bug tracker with more from a crashing osd log file (e.g. all the -1> -2> etc. lines before the crash) and also from the mon leader if possible.
Something strange is that the mon_warn_on_pool_pg_num_not_power_of_two feature is also present in v14.2.9 (it was added in v14.2.8). Which version did you upgrade from? Perhaps setting it to false was the trigger, but the crash is somewhere else in the OSD changes in v14.2.10.
Cheers, Dan
On Wed, Jul 1, 2020 at 9:09 AM Markus Binz <mbinz@solnet.ch> wrote:
Hello,
yesterday we upgraded a mimic cluster to v14.2.10, everything was running and ok.
There was this new warning, 2 pool(s) have non-power-of-two pg_num and to get a HEALTH_OK state until we can expand this pools, i found this config option to suppress the warning:
ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false which resulted in a crash of 40 osd processes (about 60% of the cluster).
no restart possible, always the same crash.
2020-06-30 21:13:56.179 7fd2b7708c00 -1 osd.30 385679 log_to_monitors {default=true} *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] 2020-06-30 21:13:56.199 7fd2a5813700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore
ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
-1547> 2020-06-30 21:13:51.171 7fd2b7708c00 -1 missing 'type' file, inferring filestore from current/ dir -738> 2020-06-30 21:13:56.179 7fd2b7708c00 -1 osd.30 385679 log_to_monitors {default=true} 0> 2020-06-30 21:13:56.199 7fd2a5813700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore
ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
-1547> 2020-06-30 21:13:51.171 7fd2b7708c00 -1 missing 'type' file, inferring filestore from current/ dir -738> 2020-06-30 21:13:56.179 7fd2b7708c00 -1 osd.30 385679 log_to_monitors {default=true} 0> 2020-06-30 21:13:56.199 7fd2a5813700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore
ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
This is a mixed cluster of ubuntu xenial and bionic, it happens on both.
It look's like, it happens when the new monmap arrived at the osd.
The only fix i was able to come up with, downgrade ceph-osd to v14.2.9.
Should i open a bug report?
Regards
Markus _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, i just uploaded one of 196 crash reports. https://tracker.ceph.com/issues/46443 I tried to debug it myself... but it took to much time. Markus On 06.07.20 11:00, Dan van der Ster wrote:
Hi Markus,
Did you make any progress with this? (Selfishly pinging to better understand whether the 14.2.9 to 14.2.10 upgrade is safe or not)
Cheers, Dan
On Wed, Jul 1, 2020 at 9:30 AM Dan van der Ster <dan@vanderster.com> wrote:
Hi Markus,
Yes, I think you should open a bug tracker with more from a crashing osd log file (e.g. all the -1> -2> etc. lines before the crash) and also from the mon leader if possible.
Something strange is that the mon_warn_on_pool_pg_num_not_power_of_two feature is also present in v14.2.9 (it was added in v14.2.8). Which version did you upgrade from? Perhaps setting it to false was the trigger, but the crash is somewhere else in the OSD changes in v14.2.10.
Cheers, Dan
On Wed, Jul 1, 2020 at 9:09 AM Markus Binz <mbinz@solnet.ch> wrote:
Hello,
yesterday we upgraded a mimic cluster to v14.2.10, everything was running and ok.
There was this new warning, 2 pool(s) have non-power-of-two pg_num and to get a HEALTH_OK state until we can expand this pools, i found this config option to suppress the warning:
ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false which resulted in a crash of 40 osd processes (about 60% of the cluster).
no restart possible, always the same crash.
2020-06-30 21:13:56.179 7fd2b7708c00 -1 osd.30 385679 log_to_monitors {default=true} *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] 2020-06-30 21:13:56.199 7fd2a5813700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore
ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
-1547> 2020-06-30 21:13:51.171 7fd2b7708c00 -1 missing 'type' file, inferring filestore from current/ dir -738> 2020-06-30 21:13:56.179 7fd2b7708c00 -1 osd.30 385679 log_to_monitors {default=true} 0> 2020-06-30 21:13:56.199 7fd2a5813700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore
ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
-1547> 2020-06-30 21:13:51.171 7fd2b7708c00 -1 missing 'type' file, inferring filestore from current/ dir -738> 2020-06-30 21:13:56.179 7fd2b7708c00 -1 osd.30 385679 log_to_monitors {default=true} 0> 2020-06-30 21:13:56.199 7fd2a5813700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore
ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
This is a mixed cluster of ubuntu xenial and bionic, it happens on both.
It look's like, it happens when the new monmap arrived at the osd.
The only fix i was able to come up with, downgrade ceph-osd to v14.2.9.
Should i open a bug report?
Regards
Markus _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Markus Binz, mbinz@solnet.ch, MB44-RIPE, PGPKEY-ABC5F050 SolNet, Internet Solution Provider Phone: +41 32 517 6223 Fax: +41 32 685 9613
Looks like the problem starts here: -7145> 2020-06-30 21:27:08.626 7fcb54b0d700 2 osd.30 385679 got incremental 385680 but failed to encode full with correct crc; requesting -7139> 2020-06-30 21:27:08.626 7fcb54b0d700 0 log_channel(cluster) log [WRN] : failed to encode map e385680 with expected crc then eventually there's a crash in _committed_osd_maps. commit fa842716b6dc3b2077e296d388c646f1605568b0 touched the osdmap code in 14.2.10 so I wonder if there's a bug in there. Otherwise, the question I ask everyone with osdmap issues these days: are you using bluestore compression and lz4? Cheers, Dan On Fri, Jul 10, 2020 at 9:45 AM Markus Binz <mbinz@solnet.ch> wrote:
Hi,
i just uploaded one of 196 crash reports.
https://tracker.ceph.com/issues/46443
I tried to debug it myself... but it took to much time.
Markus
On 06.07.20 11:00, Dan van der Ster wrote:
Hi Markus,
Did you make any progress with this? (Selfishly pinging to better understand whether the 14.2.9 to 14.2.10 upgrade is safe or not)
Cheers, Dan
On Wed, Jul 1, 2020 at 9:30 AM Dan van der Ster <dan@vanderster.com> wrote:
Hi Markus,
Yes, I think you should open a bug tracker with more from a crashing osd log file (e.g. all the -1> -2> etc. lines before the crash) and also from the mon leader if possible.
Something strange is that the mon_warn_on_pool_pg_num_not_power_of_two feature is also present in v14.2.9 (it was added in v14.2.8). Which version did you upgrade from? Perhaps setting it to false was the trigger, but the crash is somewhere else in the OSD changes in v14.2.10.
Cheers, Dan
On Wed, Jul 1, 2020 at 9:09 AM Markus Binz <mbinz@solnet.ch> wrote:
Hello,
yesterday we upgraded a mimic cluster to v14.2.10, everything was running and ok.
There was this new warning, 2 pool(s) have non-power-of-two pg_num and to get a HEALTH_OK state until we can expand this pools, i found this config option to suppress the warning:
ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false which resulted in a crash of 40 osd processes (about 60% of the cluster).
no restart possible, always the same crash.
2020-06-30 21:13:56.179 7fd2b7708c00 -1 osd.30 385679 log_to_monitors {default=true} *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] 2020-06-30 21:13:56.199 7fd2a5813700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore
ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
-1547> 2020-06-30 21:13:51.171 7fd2b7708c00 -1 missing 'type' file, inferring filestore from current/ dir -738> 2020-06-30 21:13:56.179 7fd2b7708c00 -1 osd.30 385679 log_to_monitors {default=true} 0> 2020-06-30 21:13:56.199 7fd2a5813700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore
ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
-1547> 2020-06-30 21:13:51.171 7fd2b7708c00 -1 missing 'type' file, inferring filestore from current/ dir -738> 2020-06-30 21:13:56.179 7fd2b7708c00 -1 osd.30 385679 log_to_monitors {default=true} 0> 2020-06-30 21:13:56.199 7fd2a5813700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore
ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
This is a mixed cluster of ubuntu xenial and bionic, it happens on both.
It look's like, it happens when the new monmap arrived at the osd.
The only fix i was able to come up with, downgrade ceph-osd to v14.2.9.
Should i open a bug report?
Regards
Markus _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Markus Binz, mbinz@solnet.ch, MB44-RIPE, PGPKEY-ABC5F050 SolNet, Internet Solution Provider Phone: +41 32 517 6223 Fax: +41 32 685 9613
On Fr, 2020-07-10 at 23:52 +0200, Dan van der Ster wrote:
Otherwise, the question I ask everyone with osdmap issues these days: are you using bluestore compression and lz4?
Hi, first time on this list, so hi everybody! We saw these crashes with 14.2.9 and upgraded today to 14.2.10. No crashes so far since the upgrade (but it's running for only a few hours now). To answer the question: no, we don't run bluestore compression. But to add some more questioning: In the bug report/fix https://tracker.ceph.com/issues/46443 it is mentioned that this is a bug in the linux kernels monotonic timer implementation. However it is not mentioned if this was ever reported to the upstream linux kernel, or if there is a fix available for the kernel itself. Has anyone information regarding a bug report against the kernel? Did someone very this bug against the latest upstream kernel? If this is not the case, we will try to reproduce with upstream and possibly report it as a bug, because I think it is important to fix this upstream (if it is a problem with non distro kernels). -- Mit freundlichen Grüßen / Regards Sven Kieske Systementwickler Mittwald CM Service GmbH & Co. KG Königsberger Straße 4-6 32339 Espelkamp Tel.: 05772 / 293-900 Fax: 05772 / 293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer, Florian Jürgens St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen Informationen zur Datenverarbeitung im Rahmen unserer Geschäftstätigkeit gemäß Art. 13-14 DSGVO sind unter www.mittwald.de/ds abrufbar.
On Mon, Jul 13, 2020 at 6:37 PM Sven Kieske <S.Kieske@mittwald.de> wrote:
On Fr, 2020-07-10 at 23:52 +0200, Dan van der Ster wrote:
Otherwise, the question I ask everyone with osdmap issues these days: are you using bluestore compression and lz4?
Hi,
first time on this list, so hi everybody!
Hi Sven, welcome!
We saw these crashes with 14.2.9 and upgraded today to 14.2.10. No crashes so far since the upgrade (but it's running for only a few hours now).
To answer the question: no, we don't run bluestore compression.
But to add some more questioning:
In the bug report/fix https://tracker.ceph.com/issues/46443 it is mentioned that this is a bug in the linux kernels monotonic timer implementation.
I don't quite follow where you found a connection with the kernel... are you sure you have the same problem as Markus? Cheers, Dan
However it is not mentioned if this was ever reported to the upstream linux kernel, or if there is a fix available for the kernel itself.
Has anyone information regarding a bug report against the kernel? Did someone very this bug against the latest upstream kernel?
If this is not the case, we will try to reproduce with upstream and possibly report it as a bug, because I think it is important to fix this upstream (if it is a problem with non distro kernels).
-- Mit freundlichen Grüßen / Regards
Sven Kieske Systementwickler
Mittwald CM Service GmbH & Co. KG Königsberger Straße 4-6 32339 Espelkamp
Tel.: 05772 / 293-900 Fax: 05772 / 293-333
Geschäftsführer: Robert Meyer, Florian Jürgens
St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen
Informationen zur Datenverarbeitung im Rahmen unserer Geschäftstätigkeit gemäß Art. 13-14 DSGVO sind unter www.mittwald.de/ds abrufbar.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Di, 2020-07-14 at 08:57 +0200, Dan van der Ster wrote:
I don't quite follow where you found a connection with the kernel... are you sure you have the same problem as Markus?
Cheers, Dan
I'm sorry, I didn't read careful enough through the bug reports and mixed them up. In fact, I was talking about these: https://tracker.ceph.com/issues/43365 https://tracker.ceph.com/issues/44078 which seem not to be reported upstream. Sorry for derailing the thread. So, our crashes where not subject to the same root cause as mentioned by Markus. -- Mit freundlichen Grüßen / Regards Sven Kieske Systementwickler Mittwald CM Service GmbH & Co. KG Königsberger Straße 4-6 32339 Espelkamp Tel.: 05772 / 293-900 Fax: 05772 / 293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer, Florian Jürgens St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen Informationen zur Datenverarbeitung im Rahmen unserer Geschäftstätigkeit gemäß Art. 13-14 DSGVO sind unter www.mittwald.de/ds abrufbar.
participants (3)
-
Dan van der Ster
-
Markus Binz
-
Sven Kieske