On Fri, Aug 25, 2023 at 3:21 AM Ilya Dryomov <idryomov(a)gmail.com> wrote:
On Fri, Aug 25, 2023 at 9:41 AM Prashant Dhange
<pdhange(a)redhat.com>
wrote:
Hi Ilya,
G'day.
We were seeing the rgw bucket creation failures if application is not
enabled for the rgw control pool and ceph status was not reporting
the warning message "x pool(s) do not have an application enabled
(POOL_APP_NOT_ENABLED)".
Hi Prashant,
Could RGW be improved to emit a better log message in this case?
We also observed the RGW daemon crash in the
absence of application
was not enabled for the pool. There was no way to know the reason
behind RGW bucket creation failure. This issue has been raised on
BZ#2029585.
I assume the crash is the following:
debug -5> 2022-08-10T12:10:55.410+0000 7f6b90b27700 10
monclient: get_auth_request con 0x5652391ac000 auth_method 0
debug -4> 2022-08-10T12:10:55.532+0000 7f6ba64b2440 0 rgw
main: ERROR: notify_obj.operate() returned r=-1
debug -3> 2022-08-10T12:10:55.532+0000 7f6ba64b2440 -1 ERROR:
failed to initialize watch: (1) Operation not permitted
debug -2> 2022-08-10T12:10:55.532+0000 7f6ba64b2440 0 rgw
main: ERROR: failed to start notify service ((1) Operation not
permitted
debug -1> 2022-08-10T12:10:55.532+0000 7f6ba64b2440 0 rgw
main: ERROR: failed to init services (ret=(1) Operation not permitted)
debug 0> 2022-08-10T12:10:55.539+0000 7f6ba64b2440 -1 ***
Caught signal (Segmentation fault) **
in thread 7f6ba64b2440 thread_name:radosgw
ceph version 16.2.7-98.el8cp
(b20d33c3b301e005bed203d3cad7245da3549f80) pacific (stable)
1: /lib64/libpthread.so.0(+0x12c20) [0x7f6b9ab19c20]
2: /lib64/librados.so.2(+0xada95) [0x7f6ba4ecaa95]
3: /lib64/librados.so.2(+0x9dfd8) [0x7f6ba4ebafd8]
4: (RGWSI_Notify::unwatch(RGWSI_RADOS::Obj&, unsigned long)+0x2e)
[0x7f6ba5cac99e]
5: (RGWSI_Notify::finalize_watch()+0x40) [0x7f6ba5cad290]
6: (RGWSI_Notify::shutdown()+0x22) [0x7f6ba5cad302]
7: (RGWServices_Def::shutdown()+0x4e) [0x7f6ba57abcde]
8: (RGWServices_Def::~RGWServices_Def()+0x12) [0x7f6ba57abd62]
9: (RGWRados::~RGWRados()+0x80) [0x7f6ba5b8e990]
10: (RGWStoreManager::init_storage_provider(DoutPrefixProvider
const*, ceph::common::CephContext*, bool, bool, bool, bool, bool,
bool, bool)+0x137) [0x7f6ba5b8d277]
11: (radosgw_Main(int, char const**)+0x154b) [0x7f6ba574a33b]
12: __libc_start_main()
13: _start()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
It's not in RGW per se, but could be caused by RGW passing an invalid
pointer librados. Was this reported to the RGW team?
Yes, this was the RGW crash. We had this reported in the tracker#54719