Module 'devicehealth' has failed
Hello everybody! Running 19.2.0 faced an issued still cannot struggle. And this is Module 'devicehealth' has failed: Expecting value: line 1 column 2378 (char 2377) in MGR log I see Mar 04 12:48:07 node2.ec.mts ceph-mgr[3821449]: Traceback (most recent call last): File "/usr/share/ceph/mgr/devicehealth/module.py", line 399, in serve self._do_serve() File "/usr/share/ceph/mgr/mgr_module.py", line 549, in check return func(self, *args, **kwargs) File "/usr/share/ceph/mgr/devicehealth/module.py", line 355, in _do_serve if self.db_ready() and self.enable_monitoring: File "/usr/share/ceph/mgr/mgr_module.py", line 1286, in db_ready return self.db is not None File "/usr/share/ceph/mgr/mgr_module.py", line 1298, in db self._db = self.open_db() File "/usr/share/ceph/mgr/mgr_module.py", line 1265, in open_db if not self.pool_exists(self.MGR_POOL_NAME): File "/usr/share/ceph/mgr/mgr_module.py", line 1106, in pool_exists pools = [p['pool_name'] for p in self.get('osd_map')['pools']] File "/usr/share/ceph/mgr/mgr_module.py", line 1416, in get obj = json.loads(obj) File "/lib64/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/lib64/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/lib64/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 2378 (char 2377) Unfortunately, I do now know how to turn devicehealth off as it is forbidden using standard commands line ceph device monitoring off. Any hint is appreciated. Thank in advance, guys!
Do you have a pool named ".mgr"? Zitat von Alex from North <service.plant@ya.ru>:
Hello everybody!
Running 19.2.0 faced an issued still cannot struggle. And this is Module 'devicehealth' has failed: Expecting value: line 1 column 2378 (char 2377)
in MGR log I see
Mar 04 12:48:07 node2.ec.mts ceph-mgr[3821449]: Traceback (most recent call last): File "/usr/share/ceph/mgr/devicehealth/module.py", line 399, in serve self._do_serve() File "/usr/share/ceph/mgr/mgr_module.py", line 549, in check return func(self, *args, **kwargs) File "/usr/share/ceph/mgr/devicehealth/module.py", line 355, in _do_serve if self.db_ready() and self.enable_monitoring: File "/usr/share/ceph/mgr/mgr_module.py", line 1286, in db_ready return self.db is not None File "/usr/share/ceph/mgr/mgr_module.py", line 1298, in db self._db = self.open_db() File "/usr/share/ceph/mgr/mgr_module.py", line 1265, in open_db if not self.pool_exists(self.MGR_POOL_NAME): File "/usr/share/ceph/mgr/mgr_module.py", line 1106, in pool_exists pools = [p['pool_name'] for p in self.get('osd_map')['pools']] File "/usr/share/ceph/mgr/mgr_module.py", line 1416, in get obj = json.loads(obj) File "/lib64/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/lib64/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/lib64/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 2378 (char 2377)
Unfortunately, I do now know how to turn devicehealth off as it is forbidden using standard commands line ceph device monitoring off.
Any hint is appreciated. Thank in advance, guys! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
And do you also have the device_health_metrics pool? During one of the upgrades to Quincy or so the older device_health_metrics should have been renamed. But on one customer cluster I found that both were still there, although that didn't cause any trouble. I don't really fully grasp yet what exactly goes wrong here, but maybe you have some log context as well, not only the stack trace? Zitat von Alex from North <service.plant@ya.ru>:
yes, I do
.mgr 10 1 769 KiB 2 2.3 MiB 0 4.7 PiB _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I will provide you any info you need, just gimme a sign. My starter post was related to 19.2.0. Now I downgraded (full reinstall as this is completely new cluster I wanna run) to 18.2.4 and the same story Mar 06 09:37:41 node1.ec.mts conmon[10588]: failed to collect metrics: Mar 06 09:37:41 node1.ec.mts conmon[10588]: Traceback (most recent call last): Mar 06 09:37:41 node1.ec.mts conmon[10588]: File "/usr/share/ceph/mgr/prometheus/module.py", line 514, in collect Mar 06 09:37:41 node1.ec.mts conmon[10588]: data = self.mod.collect() Mar 06 09:37:41 node1.ec.mts conmon[10588]: File "/usr/share/ceph/mgr/mgr_util.py", line 862, in wrapper Mar 06 09:37:41 node1.ec.mts conmon[10588]: result = f(*args, **kwargs) Mar 06 09:37:41 node1.ec.mts conmon[10588]: File "/usr/share/ceph/mgr/prometheus/module.py", line 1719, in collect Mar 06 09:37:41 node1.ec.mts conmon[10588]: self.get_metadata_and_osd_status() Mar 06 09:37:41 node1.ec.mts conmon[10588]: File "/usr/share/ceph/mgr/mgr_util.py", line 862, in wrapper Mar 06 09:37:41 node1.ec.mts conmon[10588]: result = f(*args, **kwargs) Mar 06 09:37:41 node1.ec.mts conmon[10588]: File "/usr/share/ceph/mgr/prometheus/module.py", line 1138, in get_metadata_and_osd_status Mar 06 09:37:41 node1.ec.mts conmon[10588]: osd_map = self.get('osd_map') Mar 06 09:37:41 node1.ec.mts conmon[10588]: File "/usr/share/ceph/mgr/mgr_module.py", line 1401, in get Mar 06 09:37:41 node1.ec.mts conmon[10588]: obj = json.loads(obj) Mar 06 09:37:41 node1.ec.mts conmon[10588]: File "/lib64/python3.9/json/__init__.py", line 346, in loads Mar 06 09:37:41 node1.ec.mts conmon[10588]: return _default_decoder.decode(s) Mar 06 09:37:41 node1.ec.mts conmon[10588]: File "/lib64/python3.9/json/decoder.py", line 337, in decode Mar 06 09:37:41 node1.ec.mts conmon[10588]: obj, end = self.raw_decode(s, idx=_w(s, 0).end()) Mar 06 09:37:41 node1.ec.mts conmon[10588]: File "/lib64/python3.9/json/decoder.py", line 355, in raw_decode Mar 06 09:37:41 node1.ec.mts conmon[10588]: raise JSONDecodeError("Expecting value", s, err.value) from None Mar 06 09:37:41 node1.ec.mts conmon[10588]: json.decoder.JSONDecodeError: Expecting value: line 1 column 2311 (char 2310) A bit more info, probably it helps somehow. This is a cluster out of 6 nodes by 116 OSD each (696 OSD total). When it was 5 nodes - no error, when 6th appeared - error sprang up. Maybe high OSD denstity give the error?
a bit more details. Now I've notices that ceph health detail signals to me that [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.node1.ec.all_disks osd.node1.ec.all_disks: Expecting value: line 1 column 2311 (char 2310) Okay, I checked my spec but do not see anything suspicious. Spec itself is here https://pastebin.com/9apchT02 Would be very grateful for the help!
Hm, is it really necessary to configure all the device paths manually? I'd recommend to use rotational flags to distinguish between OSD and DB/WAL devices. Can you give it a try with a simpler spec file? Something like: service_type: osd service_id: node1.ec.all_disks service_name: osd.node1.ec.all_disks placement: hosts: - node1.ec spec: data_devices: rotational: 1 db_devices: rotational: 0 Zitat von Alex from North <service.plant@ya.ru>:
a bit more details. Now I've notices that ceph health detail signals to me that
[WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.node1.ec.all_disks osd.node1.ec.all_disks: Expecting value: line 1 column 2311 (char 2310)
Okay, I checked my spec but do not see anything suspicious. Spec itself is here https://pastebin.com/9apchT02
Would be very grateful for the help! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I did. It says more or less the same Mar 06 10:44:05 node1.ec.mts conmon[10588]: 2025-03-06T10:44:05.769+0000 7faca5624640 -1 log_channel(cephadm) log [ERR] : Failed to apply osd.node1.ec.mts_all_disks spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd Mar 06 10:44:05 node1.ec.mts conmon[10588]: service_id: node1.ec.all_disks Mar 06 10:44:05 node1.ec.mts conmon[10588]: service_name: osd.node1.ec.all_disks Mar 06 10:44:05 node1.ec.mts conmon[10588]: placement: Mar 06 10:44:05 node1.ec.mts conmon[10588]: hosts: Mar 06 10:44:05 node1.ec.mts conmon[10588]: - node1.ec.mts Mar 06 10:44:05 node1.ec.mts conmon[10588]: spec: Mar 06 10:44:05 node1.ec.mts conmon[10588]: data_devices: Mar 06 10:44:05 node1.ec.mts conmon[10588]: rotational: 1 Mar 06 10:44:05 node1.ec.mts conmon[10588]: db_devices: Mar 06 10:44:05 node1.ec.mts conmon[10588]: rotational: 0 Mar 06 10:44:05 node1.ec.mts conmon[10588]: filter_logic: AND Mar 06 10:44:05 node1.ec.mts conmon[10588]: objectstore: bluestore Mar 06 10:44:05 node1.ec.mts conmon[10588]: ''')): Expecting value: line 1 column 2338 (char 2337) What line it is talking about?
I don't have a good idea right now, I literally took the same spec file as yours and it works fine for me in a tiny lab cluster. Maybe someone else has a good idea. Zitat von Alex from North <service.plant@ya.ru>:
I did. It says more or less the same
Mar 06 10:44:05 node1.ec.mts conmon[10588]: 2025-03-06T10:44:05.769+0000 7faca5624640 -1 log_channel(cephadm) log [ERR] : Failed to apply osd.node1.ec.mts_all_disks spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd Mar 06 10:44:05 node1.ec.mts conmon[10588]: service_id: node1.ec.all_disks Mar 06 10:44:05 node1.ec.mts conmon[10588]: service_name: osd.node1.ec.all_disks Mar 06 10:44:05 node1.ec.mts conmon[10588]: placement: Mar 06 10:44:05 node1.ec.mts conmon[10588]: hosts: Mar 06 10:44:05 node1.ec.mts conmon[10588]: - node1.ec.mts Mar 06 10:44:05 node1.ec.mts conmon[10588]: spec: Mar 06 10:44:05 node1.ec.mts conmon[10588]: data_devices: Mar 06 10:44:05 node1.ec.mts conmon[10588]: rotational: 1 Mar 06 10:44:05 node1.ec.mts conmon[10588]: db_devices: Mar 06 10:44:05 node1.ec.mts conmon[10588]: rotational: 0 Mar 06 10:44:05 node1.ec.mts conmon[10588]: filter_logic: AND Mar 06 10:44:05 node1.ec.mts conmon[10588]: objectstore: bluestore Mar 06 10:44:05 node1.ec.mts conmon[10588]: ''')): Expecting value: line 1 column 2338 (char 2337)
What line it is talking about? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for the help, buddy! I really appreciate it! Will try to wait. Maybe someone else jumps in.
participants (2)
-
Alex from North
-
Eugen Block