I've met this issue when try to upgrade octopus 15.2.17 to 16.2.13 last night. Upgrade
process failed at mgr module phase after the new MGR version become to active state. I
tried to enable debug `ceph config set mgr mgr/cephadm/log_to_cluster_level debug
` and I saw the message like @xadhoom76 about config_key "registry_credentials"
I guessed the root cause because of this line
`json.loads(str(self.mgr.get_store('registry_credentials'))))` and the key_store
had a wrong value. Then I got the empty value when run this command "ceph config-key
dump | grep 'registry_credentials'" and the same for "ceph config-key
get mgr/cephadm/registry_credentials"
By check `cephadm` source i see the value should be a json format like that
ceph config-key set mgr/cephadm/registry_credentials '{"url":
"registry.local:5000", "username": "user-deployer",
"password": "xxxyyyyzzzzzz"}'
After set this key and `ceph mgr fail` to reload , my cluster issue was gone .