27 Jan
2021
27 Jan
'21
8:41 p.m.
Anyone running octopus (v15)? Can you please share your experience of
radosgw-admin performance?
A simple 'radosgw-admin user list' took 11 minutes; if I use a v13.2.4
radosgw-admin, it can be finished in a few seconds.
This sounds like a performance regression to me. I've already filed a bug
report (https://tracker.ceph.com/issues/48983) but so far no feedback yet.
On Mon, Jan 25, 2021 at 10:06 AM Youzhong Yang <youzhong(a)gmail.com> wrote:
I upgraded our ceph cluster (6 bare metal nodes, 3 rgw
VMs) from v13.2.4
to v15.2.8. The mon, mgr, mds and osd daemons were all upgraded
successfully, everything looked good.
After the radosgw was upgraded, they refused to work, the log messages are
at the end of this e-mail.
Here are the things I tried:
1. I moved aside the pools for the rgw service, started from scratch
(creating realm, zonegroup, zone, users), but when I tried to run
'radosgw-admin user create ...', it appeared to be stuck and never
returned, other command like 'radosgw-admin period update --commit' also
got stuck.
2. I rolled back radosgw to the old version v13.2.4, then everything works
great again.
What am I missing here? Is there anything extra that needs to be done for
rgw after upgrading from mimic to octopus?
Please kindly help. Thanks.
---------------------
2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 deferred set uid:gid to
64045:64045 (ceph:ceph)
2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 ceph version 15.2.8
(bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable), process
radosgw, pid 898
2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 framework: civetweb
2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 framework conf key: port,
val: 80
2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 framework conf key:
num_threads, val: 1024
2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 framework conf key:
request_timeout_ms, val: 50000
2021-01-24T09:24:10.192-0500 7f638f79f9c0 1 radosgw_Main not setting numa
affinity
2021-01-24T09:29:10.195-0500 7f638cbcd700 -1 Initialization timeout,
failed to initialize
2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 deferred set uid:gid to
64045:64045 (ceph:ceph)
2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 ceph version 15.2.8
(bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable), process
radosgw, pid 1541
2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 framework: civetweb
2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 framework conf key: port,
val: 80
2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 framework conf key:
num_threads, val: 1024
2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 framework conf key:
request_timeout_ms, val: 50000
2021-01-24T09:29:10.367-0500 7f4c213ba9c0 1 radosgw_Main not setting numa
affinity
2021-01-24T09:29:25.883-0500 7f4c213ba9c0 1 robust_notify: If at first
you don't succeed: (110) Connection timed out
2021-01-24T09:29:25.883-0500 7f4c213ba9c0 0 ERROR: failed to distribute
cache for coredumps.rgw.log:meta.history
2021-01-24T09:32:27.754-0500 7fcdac2bf9c0 0 deferred set uid:gid to
64045:64045 (ceph:ceph)
2021-01-24T09:32:27.754-0500 7fcdac2bf9c0 0 ceph version 15.2.8
(bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable), process
radosgw, pid 978
2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 0 framework: civetweb
2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 0 framework conf key: port,
val: 80
2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 0 framework conf key:
num_threads, val: 1024
2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 0 framework conf key:
request_timeout_ms, val: 50000
2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 1 radosgw_Main not setting numa
affinity
2021-01-24T09:32:44.719-0500 7fcdac2bf9c0 1 robust_notify: If at first
you don't succeed: (110) Connection timed out
2021-01-24T09:32:44.719-0500 7fcdac2bf9c0 0 ERROR: failed to distribute
cache for coredumps.rgw.log:meta.history