Anyone running octopus (v15)? Can you please share your experience of radosgw-admin performance? A simple 'radosgw-admin user list' took 11 minutes; if I use a v13.2.4 radosgw-admin, it can be finished in a few seconds. This sounds like a performance regression to me. I've already filed a bug report (https://tracker.ceph.com/issues/48983) but so far no feedback yet. On Mon, Jan 25, 2021 at 10:06 AM Youzhong Yang <youzhong@gmail.com> wrote:

I upgraded our ceph cluster (6 bare metal nodes, 3 rgw VMs) from v13.2.4 to v15.2.8. The mon, mgr, mds and osd daemons were all upgraded successfully, everything looked good.

After the radosgw was upgraded, they refused to work, the log messages are at the end of this e-mail.

Here are the things I tried:

1. I moved aside the pools for the rgw service, started from scratch (creating realm, zonegroup, zone, users), but when I tried to run 'radosgw-admin user create ...', it appeared to be stuck and never returned, other command like 'radosgw-admin period update --commit' also got stuck.

2. I rolled back radosgw to the old version v13.2.4, then everything works great again.

What am I missing here? Is there anything extra that needs to be done for rgw after upgrading from mimic to octopus?

Please kindly help. Thanks.

--------------------- 2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 deferred set uid:gid to 64045:64045 (ceph:ceph) 2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable), process radosgw, pid 898 2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 framework: civetweb 2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 framework conf key: port, val: 80 2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 framework conf key: num_threads, val: 1024 2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 framework conf key: request_timeout_ms, val: 50000 2021-01-24T09:24:10.192-0500 7f638f79f9c0 1 radosgw_Main not setting numa affinity 2021-01-24T09:29:10.195-0500 7f638cbcd700 -1 Initialization timeout, failed to initialize 2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 deferred set uid:gid to 64045:64045 (ceph:ceph) 2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable), process radosgw, pid 1541 2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 framework: civetweb 2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 framework conf key: port, val: 80 2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 framework conf key: num_threads, val: 1024 2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 framework conf key: request_timeout_ms, val: 50000 2021-01-24T09:29:10.367-0500 7f4c213ba9c0 1 radosgw_Main not setting numa affinity 2021-01-24T09:29:25.883-0500 7f4c213ba9c0 1 robust_notify: If at first you don't succeed: (110) Connection timed out 2021-01-24T09:29:25.883-0500 7f4c213ba9c0 0 ERROR: failed to distribute cache for coredumps.rgw.log:meta.history 2021-01-24T09:32:27.754-0500 7fcdac2bf9c0 0 deferred set uid:gid to 64045:64045 (ceph:ceph) 2021-01-24T09:32:27.754-0500 7fcdac2bf9c0 0 ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable), process radosgw, pid 978 2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 0 framework: civetweb 2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 0 framework conf key: port, val: 80 2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 0 framework conf key: num_threads, val: 1024 2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 0 framework conf key: request_timeout_ms, val: 50000 2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 1 radosgw_Main not setting numa affinity 2021-01-24T09:32:44.719-0500 7fcdac2bf9c0 1 robust_notify: If at first you don't succeed: (110) Connection timed out 2021-01-24T09:32:44.719-0500 7fcdac2bf9c0 0 ERROR: failed to distribute cache for coredumps.rgw.log:meta.history