27 Apr
2023
27 Apr
'23
3:38 p.m.
Good morning,
After upgrading from Octopus (15.2.17) to Pacific (16.2.12) two days
ago, I'm noticing that the MGR daemons keep failing over to standby and
then back every 24hrs. Watching the output of 'ceph orch ps' I can see
that the memory consumption of the mgr is steadily growing until it
becomes unresponsive.
When the mgr becomes unresponsive, tasks such as RESTful calls start to
fail, and the standby eventually takes over after ~20 minutes. I've
included a log of memory consumption (in 10 minute intervals) at the end
of this message. While the cluster recovers during this issue, the loss
of usage data during the outage, and the fact its occurring is
problematic. Any assistance would be appreciated.
Note, this is a cluster that has been upgraded from an original jewel
based ceph using filestore, through bluestore conversion, container
conversion, and now to Pacific. The data below shows memory use with
three mgr modules enabled: cephadm, restful, iostat. By disabling
iostat, I can reduce the rate of memory consumption increasing to about
200MB/hr.
Thanks
Gary.
Wed Apr 26 12:50:08 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(3h) 7m ago 3h 1099M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(16h) 7m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 13:00:09 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(3h) 6m ago 3h 1161M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(16h) 6m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 13:10:10 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(3h) 6m ago 3h 1230M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(16h) 6m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 13:20:11 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(4h) 6m ago 4h 1250M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(16h) 6m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 13:30:12 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(4h) 5m ago 4h 1318M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(16h) 5m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 13:40:13 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(4h) 5m ago 4h 1379M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(17h) 5m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 13:50:13 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(4h) 5m ago 4h 1442M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(17h) 5m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 14:00:14 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(4h) 4m ago 4h 1498M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(17h) 4m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 14:10:15 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(4h) 4m ago 4h 1554M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(17h) 4m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 14:20:16 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(5h) 4m ago 5h 1617M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(17h) 3m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 14:30:17 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(5h) 3m ago 5h 1677M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(17h) 3m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 14:40:18 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(5h) 3m ago 5h 1735M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(18h) 3m ago 18h 383M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 14:50:19 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(5h) 2m ago 5h 1806M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(18h) 2m ago 18h 383M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 15:00:20 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(5h) 2m ago 5h 1873M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(18h) 2m ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 15:10:21 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(5h) 2m ago 5h 1920M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(18h) 2m ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 15:20:21 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(6h) 114s ago 6h 1981M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(18h) 113s ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 15:30:22 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(6h) 95s ago 6h 2048M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(18h) 95s ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 15:40:23 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(6h) 80s ago 6h 2121M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(19h) 80s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 15:50:24 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(6h) 53s ago 6h 2170M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(19h) 53s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 16:00:25 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(6h) 35s ago 6h 2238M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(19h) 34s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 16:10:26 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(6h) 19s ago 6h 2266M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(19h) 19s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 16:20:27 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(7h) 10m ago 7h 2266M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(19h) 10m ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 16:30:27 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(7h) 9m ago 7h 2321M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(19h) 9m ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 16:40:28 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(7h) 9m ago 7h 2382M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(20h) 9m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 16:50:29 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(7h) 9m ago 7h 2449M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(20h) 9m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 17:00:30 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(7h) 8m ago 7h 2505M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(20h) 8m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 17:10:31 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(7h) 8m ago 7h 2562M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(20h) 8m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 17:20:32 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(8h) 8m ago 8h 2628M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(20h) 8m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 17:30:33 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(8h) 7m ago 8h 2692M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(20h) 7m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 17:40:34 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(8h) 7m ago 8h 2761M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(21h) 7m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 17:50:34 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(8h) 7m ago 8h 2811M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(21h) 7m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 18:00:35 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(8h) 6m ago 8h 2873M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(21h) 6m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 18:10:36 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(8h) 6m ago 8h 2923M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(21h) 6m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 18:20:37 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(9h) 6m ago 9h 2995M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(21h) 6m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 18:30:38 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(9h) 5m ago 9h 3055M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(21h) 5m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 18:40:39 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(9h) 5m ago 9h 3110M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(22h) 5m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 18:50:40 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(9h) 5m ago 9h 3166M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(22h) 5m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 19:00:41 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(9h) 4m ago 9h 3244M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(22h) 4m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 19:10:42 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(9h) 4m ago 9h 3289M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(22h) 4m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 19:20:42 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(10h) 4m ago 10h 3368M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(22h) 4m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 19:30:43 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(10h) 3m ago 10h 3437M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(22h) 3m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 19:40:44 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(10h) 3m ago 10h 3496M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(23h) 3m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 19:50:45 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(10h) 3m ago 10h 3546M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(23h) 3m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 20:00:46 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(10h) 2m ago 10h 3627M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(23h) 2m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 20:10:47 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(10h) 2m ago 10h 3637M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(23h) 2m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 20:20:48 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(11h) 92s ago 11h 492M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(23h) 91s ago 23h 418M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 20:31:55 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(11h) 51s ago 11h 373M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(23h) 51s ago 23h 520M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 20:41:56 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(11h) 32s ago 11h 374M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(0h) 32s ago 0h 585M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 20:51:57 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(11h) 17s ago 11h 376M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(0h) 17s ago 0h 647M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 21:01:58 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(11h) 10m ago 11h 376M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(0h) 10m ago 0h 647M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 21:11:58 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(11h) 9m ago 11h 378M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(0h) 9m ago 0h 698M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 21:21:59 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(12h) 9m ago 12h 379M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(0h) 9m ago 0h 758M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 21:32:00 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(12h) 9m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(0h) 9m ago 0h 825M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 21:42:01 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(12h) 8m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(1h) 8m ago 1h 895M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 21:52:02 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(12h) 8m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(1h) 8m ago 1h 948M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 22:02:03 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(12h) 8m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(1h) 8m ago 1h 1010M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 22:12:04 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(12h) 7m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(1h) 7m ago 1h 1071M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 22:22:05 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(13h) 7m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(1h) 7m ago 1h 1132M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 22:32:06 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(13h) 7m ago 13h 380M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(1h) 7m ago 1h 1183M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 22:42:06 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(13h) 6m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(2h) 6m ago 2h 1244M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 22:52:07 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(13h) 6m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(2h) 6m ago 2h 1311M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 23:02:08 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(13h) 6m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(2h) 6m ago 2h 1380M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 23:12:09 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(13h) 5m ago 13h 380M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(2h) 5m ago 2h 1441M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 23:22:10 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(14h) 5m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(2h) 5m ago 2h 1508M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 23:32:11 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(14h) 5m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(2h) 5m ago 2h 1541M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 23:42:12 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(14h) 4m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(3h) 4m ago 3h 1610M - 16.2.12 921c4e969fff 6fed006895e8
Wed Apr 26 23:52:13 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(14h) 4m ago 14h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(3h) 4m ago 3h 1674M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 00:02:14 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(14h) 4m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(3h) 4m ago 3h 1725M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 00:12:14 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(14h) 3m ago 14h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(3h) 3m ago 3h 1794M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 00:22:15 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(15h) 3m ago 15h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(3h) 3m ago 3h 1844M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 00:32:16 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(15h) 3m ago 15h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(3h) 3m ago 3h 1917M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 00:42:17 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(15h) 2m ago 15h 380M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(4h) 2m ago 4h 2002M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 00:52:18 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(15h) 2m ago 15h 380M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(4h) 2m ago 4h 2056M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 01:02:19 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(15h) 2m ago 15h 380M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(4h) 2m ago 4h 2113M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 01:12:20 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(15h) 92s ago 15h 380M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(4h) 91s ago 4h 2173M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 01:22:21 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(16h) 73s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(4h) 72s ago 4h 2201M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 01:32:22 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(16h) 57s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(4h) 57s ago 4h 2264M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 01:42:22 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(16h) 29s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(5h) 29s ago 5h 2324M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 01:52:23 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(16h) 10s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(5h) 10s ago 5h 2387M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 02:02:24 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(16h) 10m ago 16h 380M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(5h) 10m ago 5h 2387M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 02:12:25 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(16h) 9m ago 16h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(5h) 9m ago 5h 2442M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 02:22:26 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(17h) 9m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(5h) 9m ago 5h 2502M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 02:32:27 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(17h) 9m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(5h) 9m ago 5h 2552M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 02:42:28 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(17h) 8m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(6h) 8m ago 6h 2614M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 02:52:28 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(17h) 8m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(6h) 8m ago 6h 2681M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 03:02:29 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(17h) 8m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(6h) 8m ago 6h 2748M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 03:12:30 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(17h) 7m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(6h) 7m ago 6h 2810M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 03:22:31 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(18h) 7m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(6h) 7m ago 6h 2867M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 03:32:32 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(18h) 7m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(6h) 7m ago 6h 2916M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 03:42:33 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(18h) 6m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(7h) 6m ago 7h 3024M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 03:52:34 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(18h) 6m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(7h) 6m ago 7h 3070M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 04:02:35 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(18h) 6m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(7h) 6m ago 7h 3127M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 04:12:36 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(18h) 5m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(7h) 5m ago 7h 3183M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 04:22:36 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(19h) 5m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(7h) 5m ago 7h 3215M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 04:32:37 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(19h) 4m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(7h) 4m ago 7h 3290M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 04:42:38 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(19h) 4m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(8h) 4m ago 8h 3342M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 04:52:39 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(19h) 4m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(8h) 4m ago 8h 3392M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 05:02:40 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(19h) 3m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(8h) 3m ago 8h 3451M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 05:12:41 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(19h) 3m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(8h) 3m ago 8h 3516M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 05:22:42 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(20h) 3m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(8h) 3m ago 8h 3566M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 05:32:43 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(20h) 2m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(8h) 2m ago 8h 3632M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 05:42:44 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(20h) 2m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(9h) 2m ago 9h 3696M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 05:52:44 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(20h) 2m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(9h) 2m ago 9h 3760M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 06:02:45 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(20h) 115s ago 20h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(9h) 114s ago 9h 3813M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 06:12:46 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(20h) 99s ago 20h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(9h) 99s ago 9h 3869M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 06:22:47 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(21h) 72s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(9h) 72s ago 9h 3923M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 06:32:48 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(21h) 54s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(9h) 53s ago 9h 3990M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 06:42:49 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(21h) 38s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(10h) 38s ago 10h 4040M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 06:52:50 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(21h) 11s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(10h) 10s ago 10h 4097M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 07:02:51 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(21h) 10m ago 21h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(10h) 10m ago 10h 4097M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 07:12:51 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(21h) 9m ago 21h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(10h) 9m ago 10h 4157M - 16.2.12 921c4e969fff 6fed006895e8
Thu Apr 27 07:22:52 EDT 2023
mgr.controller04.chhvsy controller04.mydomain running
(22h) 9m ago 22h 381M - 16.2.12 921c4e969fff e898a9606028
mgr.storage02.jnvnrm storage02.mydomain *:8003 running
(10h) 9m ago 10h 4213M - 16.2.12 921c4e969fff 6fed006895e8
--
Gary Molenkamp Science Technology Services
Systems Administrator University of Western Ontario
molenkam(a)uwo.ca http://sts.sci.uwo.ca
(519) 661-2111 x86882 (519) 661-3566
2 May
2 May
3:39 p.m.
To follow up on this issue, I saw the additional comments on
https://tracker.ceph.com/issues/59580 regarding mgr caps.
By setting the mgr user caps back to the default, I was able to reduce
the memory leak from several 100MB/h to just a few MB/hr.
As the other commenter had posted, in order for zabbix to access OSD
data via RESTful, the mgr caps were set to:
ceph auth caps mgr.controller04.lvhgea mon 'allow *' osd 'allow *'
mds 'allow *'
Gary
On 2023-04-27 08:38, Gary Molenkamp wrote:
Good morning,
After upgrading from Octopus (15.2.17) to Pacific (16.2.12) two days
ago, I'm noticing that the MGR daemons keep failing over to standby
and then back every 24hrs. Watching the output of 'ceph orch ps' I
can see that the memory consumption of the mgr is steadily growing
until it becomes unresponsive.
When the mgr becomes unresponsive, tasks such as RESTful calls start
to fail, and the standby eventually takes over after ~20 minutes. I've
included a log of memory consumption (in 10 minute intervals) at the
end of this message. While the cluster recovers during this issue, the
loss of usage data during the outage, and the fact its occurring is
problematic. Any assistance would be appreciated.
Note, this is a cluster that has been upgraded from an original jewel
based ceph using filestore, through bluestore conversion, container
conversion, and now to Pacific. The data below shows memory use
with three mgr modules enabled: cephadm, restful, iostat. By
disabling iostat, I can reduce the rate of memory consumption
increasing to about 200MB/hr.
Thanks
Gary.
--
Gary Molenkamp Science Technology Services
Systems Administrator University of Western Ontario
molenkam(a)uwo.ca http://sts.sci.uwo.ca
(519) 661-2111 x86882 (519) 661-3566
383
days inactive
388
days old
1 comments
1 participants
participants (1)
-
Gary Molenkamp