[ceph-users] [v15.2.11] radosgw / RGW crash at start, Segmentation Fault

7 May 2021

      Hello,

Since I upgrade to Ceph Octopus v15.2.11, on Ubuntu 18.04,
Radosgw crash straight at start.

On Two clusters, one Lab, and some test on a production cluster, shows 
the same crash for radosgw.

As I don't find any similar bug in the Tracker, neither in this mailing 
list... Am I alone ?

The logs are :

May 07 14:04:59 fidcl-mrs4-sto-sds-07 systemd[1]: Started Ceph rados 
gateway.
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: *** Caught signal 
(Segmentation fault) **
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  in thread 
7ff556655140 thread_name:radosgw
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  ceph version 
15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable)
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  1: (()+0x3f040) 
[0x7ff554e19040]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  2: 
(std::locale::operator=(std::locale const&)+0x28) [0x55aadc0ade88]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  3: 
(std::ios_base::imbue(std::locale const&)+0x2e) [0x55aadc14ec3e]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  4: 
(std::basic_ios<char, std::char_traits<char> >::imbue(std::locale 
const&)+0x44) [0x55aadc0f5c54]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  5: 
(std::basic_ostream<char, std::char_traits<char> >& 
boost::asio::ip::operator<< <char, std::char_trait
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  6: (()+0x40ce0a) 
[0x7ff555847e0a]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  7: 
(radosgw_Main(int, char const**)+0x3430) [0x7ff5559c8430]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  8: 
(__libc_start_main()+0xe7) [0x7ff554dfbbf7]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  9: 
(_start()+0x2a) [0x55aadc0a836a]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]: 
2021-05-07T14:05:00.309+0200 7ff556655140 -1 *** Caught signal 
(Segmentation fault) **
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  in thread 
7ff556655140 thread_name:radosgw
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  ceph version 
15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable)
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  1: (()+0x3f040) 
[0x7ff554e19040]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  2: 
(std::locale::operator=(std::locale const&)+0x28) [0x55aadc0ade88]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  3: 
(std::ios_base::imbue(std::locale const&)+0x2e) [0x55aadc14ec3e]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  4: 
(std::basic_ios<char, std::char_traits<char> >::imbue(std::locale 
const&)+0x44) [0x55aadc0f5c54]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  5: 
(std::basic_ostream<char, std::char_traits<char> >& 
boost::asio::ip::operator<< <char, std::char_trait
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  6: (()+0x40ce0a) 
[0x7ff555847e0a]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  7: 
(radosgw_Main(int, char const**)+0x3430) [0x7ff5559c8430]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  8: 
(__libc_start_main()+0xe7) [0x7ff554dfbbf7]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  9: 
(_start()+0x2a) [0x55aadc0a836a]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  NOTE: a copy of 
the executable, or `objdump -rdS <executable>` is needed to interpret 
this.
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:      0> 
2021-05-07T14:05:00.309+0200 7ff556655140 -1 *** Caught signal 
(Segmentation fault) **
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  in thread 
7ff556655140 thread_name:radosgw
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  ceph version 
15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable)
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  1: (()+0x3f040) 
[0x7ff554e19040]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  2: 
(std::locale::operator=(std::locale const&)+0x28) [0x55aadc0ade88]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  3: 
(std::ios_base::imbue(std::locale const&)+0x2e) [0x55aadc14ec3e]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  4: 
(std::basic_ios<char, std::char_traits<char> >::imbue(std::locale 
const&)+0x44) [0x55aadc0f5c54]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  5: 
(std::basic_ostream<char, std::char_traits<char> >& 
boost::asio::ip::operator<< <char, std::char_trait
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  6: (()+0x40ce0a) 
[0x7ff555847e0a]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  7: 
(radosgw_Main(int, char const**)+0x3430) [0x7ff5559c8430]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  8: 
(__libc_start_main()+0xe7) [0x7ff554dfbbf7]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  9: 
(_start()+0x2a) [0x55aadc0a836a]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  NOTE: a copy of 
the executable, or `objdump -rdS <executable>` is needed to interpret 
this.
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:     -3> 
2021-05-07T14:05:00.309+0200 7ff556655140 -1 *** Caught signal 
(Segmentation fault) **
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  in thread 
7ff556655140 thread_name:radosgw
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  ceph version 
15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable)
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  1: (()+0x3f040) 
[0x7ff554e19040]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  2: 
(std::locale::operator=(std::locale const&)+0x28) [0x55aadc0ade88]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  3: 
(std::ios_base::imbue(std::locale const&)+0x2e) [0x55aadc14ec3e]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  4: 
(std::basic_ios<char, std::char_traits<char> >::imbue(std::locale 
const&)+0x44) [0x55aadc0f5c54]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  5: 
(std::basic_ostream<char, std::char_traits<char> >& 
boost::asio::ip::operator<< <char, std::char_trait
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  6: (()+0x40ce0a) 
[0x7ff555847e0a]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  7: 
(radosgw_Main(int, char const**)+0x3430) [0x7ff5559c8430]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  8: 
(__libc_start_main()+0xe7) [0x7ff554dfbbf7]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  9: 
(_start()+0x2a) [0x55aadc0a836a]
May 07 14:05:00 fidcl-mrs4-sto-sds-07 radosgw[155303]:  NOTE: a copy of 
the executable, or `objdump -rdS <executable>` is needed to interpret 
this.
May 07 14:05:00 fidcl-mrs4-sto-sds-07 systemd[1]: 
ceph-radosgw@rgw.fidcl-mrs4-sto-sds-07.rgw0.service: Main process 
exited, code=killed, status=11/SEGV
May 07 14:05:00 fidcl-mrs4-sto-sds-07 systemd[1]: 
ceph-radosgw@rgw.fidcl-mrs4-sto-sds-07.rgw0.service: Failed with result 
'signal'.
May 07 14:05:00 fidcl-mrs4-sto-sds-07 systemd[1]: 
ceph-radosgw@rgw.fidcl-mrs4-sto-sds-07.rgw0.service: Service hold-off 
time over, scheduling restart.
May 07 14:05:00 fidcl-mrs4-sto-sds-07 systemd[1]: 
ceph-radosgw@rgw.fidcl-mrs4-sto-sds-07.rgw0.service: Scheduled restart 
job, restart counter is at 653.
May 07 14:05:00 fidcl-mrs4-sto-sds-07 systemd[1]: Stopped Ceph rados 
gateway.
May 07 14:05:00 fidcl-mrs4-sto-sds-07 systemd[1]: 
ceph-radosgw@rgw.fidcl-mrs4-sto-sds-07.rgw0.service: Start request 
repeated too quickly.
May 07 14:05:00 fidcl-mrs4-sto-sds-07 systemd[1]: 
ceph-radosgw@rgw.fidcl-mrs4-sto-sds-07.rgw0.service: Failed with result 
'signal'.
May 07 14:05:00 fidcl-mrs4-sto-sds-07 systemd[1]: Failed to start Ceph 
rados gateway.

[ceph-users] [v15.2.11] radosgw / RGW crash at start, Segmentation Fault

Gilles Mocellin