25 Jan
2021
25 Jan
'21
6:06 p.m.
I upgraded our ceph cluster (6 bare metal nodes, 3 rgw VMs) from v13.2.4 to
v15.2.8. The mon, mgr, mds and osd daemons were all upgraded successfully,
everything looked good.
After the radosgw was upgraded, they refused to work, the log messages are
at the end of this e-mail.
Here are the things I tried:
1. I moved aside the pools for the rgw service, started from scratch
(creating realm, zonegroup, zone, users), but when I tried to run
'radosgw-admin user create ...', it appeared to be stuck and never
returned, other command like 'radosgw-admin period update --commit' also
got stuck.
2. I rolled back radosgw to the old version v13.2.4, then everything works
great again.
What am I missing here? Is there anything extra that needs to be done for
rgw after upgrading from mimic to octopus?
Please kindly help. Thanks.
---------------------
2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 deferred set uid:gid to
64045:64045 (ceph:ceph)
2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 ceph version 15.2.8
(bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable), process
radosgw, pid 898
2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 framework: civetweb
2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 framework conf key: port, val:
80
2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 framework conf key:
num_threads, val: 1024
2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 framework conf key:
request_timeout_ms, val: 50000
2021-01-24T09:24:10.192-0500 7f638f79f9c0 1 radosgw_Main not setting numa
affinity
2021-01-24T09:29:10.195-0500 7f638cbcd700 -1 Initialization timeout, failed
to initialize
2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 deferred set uid:gid to
64045:64045 (ceph:ceph)
2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 ceph version 15.2.8
(bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable), process
radosgw, pid 1541
2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 framework: civetweb
2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 framework conf key: port, val:
80
2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 framework conf key:
num_threads, val: 1024
2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 framework conf key:
request_timeout_ms, val: 50000
2021-01-24T09:29:10.367-0500 7f4c213ba9c0 1 radosgw_Main not setting numa
affinity
2021-01-24T09:29:25.883-0500 7f4c213ba9c0 1 robust_notify: If at first you
don't succeed: (110) Connection timed out
2021-01-24T09:29:25.883-0500 7f4c213ba9c0 0 ERROR: failed to distribute
cache for coredumps.rgw.log:meta.history
2021-01-24T09:32:27.754-0500 7fcdac2bf9c0 0 deferred set uid:gid to
64045:64045 (ceph:ceph)
2021-01-24T09:32:27.754-0500 7fcdac2bf9c0 0 ceph version 15.2.8
(bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable), process
radosgw, pid 978
2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 0 framework: civetweb
2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 0 framework conf key: port, val:
80
2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 0 framework conf key:
num_threads, val: 1024
2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 0 framework conf key:
request_timeout_ms, val: 50000
2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 1 radosgw_Main not setting numa
affinity
2021-01-24T09:32:44.719-0500 7fcdac2bf9c0 1 robust_notify: If at first you
don't succeed: (110) Connection timed out
2021-01-24T09:32:44.719-0500 7fcdac2bf9c0 0 ERROR: failed to distribute
cache for coredumps.rgw.log:meta.history