- ceph-users - lists.ceph.io

Re: [ceph-user][ceph-ansible] Quincy: Module 'devicehealth' has failed: unknown operation
by Frédéric Nass 02 Feb '26

02 Feb '26

Hi Danish, You seem to be hitting this bug [1] that was fixed in Quincy v17.2.8 by [2]. If you're using packages (.rpm,.deb), you could upgrade to v17.2.9 that has the fix. Do not use v17.2.8 due to a critical BlueStore bug. If you're using cephadm and containers, you'll have to upgrade to Reef as v17.2.9 had no container images. Best regards, Frédéric. [1] https://tracker.ceph.com/issues/55606 [2] https://github.com/ceph/ceph/pull/52461 -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Squishing Squids - A Ceph Compression Guide <https://www.eventbrite.com/e/squishing-squids-a-ceph-compression-guide-tick…>, February 25th, 9am PST. https://clyso.com | frederic.nass(a)clyso.com Le sam. 31 janv. 2026 à 09:09, Danish Khan via ceph-users < ceph-users(a)ceph.io> a écrit : > Dear Team, > > I am keep getting below error after every few days: > > Error : > Module 'devicehealth' has failed: unknown operation > > Actual error from MGR log: > > 2026-01-30T05:50:59.338+0530 7f314da26640 0 [devicehealth ERROR root] > Caught fatal database error: unknown operation > 2026-01-30T05:50:59.338+0530 7f314da26640 -1 log_channel(cluster) log [ERR] > : Unhandled exception from module 'devicehealth' while running on > mgr.controller3: unknown operation > 2026-01-30T05:50:59.338+0530 7f314da26640 -1 devicehealth.serve: > 2026-01-30T05:50:59.338+0530 7f314da26640 -1 Traceback (most recent call > last): > File "/usr/share/ceph/mgr/devicehealth/module.py", line 408, in serve > self._do_serve() > File "/usr/share/ceph/mgr/mgr_module.py", line 513, in check > return func(self, *args, **kwargs) > File "/usr/share/ceph/mgr/devicehealth/module.py", line 399, in _do_serve > self.scrape_all() > File "/usr/share/ceph/mgr/devicehealth/module.py", line 449, in > scrape_all > self.put_device_metrics(device, data) > File "/usr/share/ceph/mgr/devicehealth/module.py", line 525, in > put_device_metrics > self._create_device(devid) > File "/usr/share/ceph/mgr/devicehealth/module.py", line 511, in > _create_device > cursor = self.db.execute(SQL, (devid,)) > sqlite3.InternalError: unknown operation > > 2026-01-30T05:50:59.366+0530 7f3155335640 0 log_channel(cluster) log [DBG] > : pgmap v116150: 6977 pgs: 92 peering, 40 stale+active+clean, 191 > active+clean+laggy, 30 active+undersized, 1026 active+undersized+degraded, > 5598 active+clean; 48 TiB data, 153 TiB used, 1.2 PiB / 1.4 PiB avail; 44 > MiB/s wr, 1.00k op/s; 8043806/154077270 objects degraded (5.221%); 4.4 > GiB/s, 8 keys/s, 530 objects/s recovering > > And I get one warning for mgr daemon crash but all MGR Service > actually doesn't get failed or restarted automatically. > > I guess it is something related to sqlite3 which checks ceph > health internally. > > Although this is not causing any issue and error disappears after > restarting ceph-mgr service or doing a failover of MGR to another > controller. > > Does anyone know how we can resolve this issue permanently? > > Similar issue: > https://github.com/rook/rook/issues/12349 > > Regards, > Danish > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Experiences with Tentacle 20.2.0
by Chris Palmer 02 Feb '26

02 Feb '26

Thanks Wannes. Those are the caveats around EC optimisation that I alluded to below. Since we don't have any EC on this cluster that should not be relevant... Regards, Chris On 02/02/2026 09:30, Wannes Smet via ceph-users wrote: > This story jumps to mind: https://docs.clyso.com/blog/2026/01/13/tentacle-v20.2.0-ec-bug/ > ________________________________ > From: Chris Palmer via ceph-users <ceph-users(a)ceph.io> > Sent: Monday, February 2, 2026 10:16 > To: ceph-users <ceph-users(a)ceph.io> > Subject: [ceph-users] Experiences with Tentacle 20.2.0 > > I'd like to know whether anyone has any good or bad experiences with Tentacle 20. 2. 0 in a production environment. I have the opportunity to upgrade a small production cluster from 19. 2. 3 to 20. 2. 0. It uses Cephadm/RBD/CephFS/RGW with R3 only > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > I'd like to know whether anyone has any good or bad experiences with > Tentacle 20.2.0 in a production environment. > > I have the opportunity to upgrade a small production cluster from 19.2.3 > to 20.2.0. It uses Cephadm/RBD/CephFS/RGW with R3 only (no EC, so the > known caveats around EC optimisations won't be relevant). > > My experiences trying it in a lab test environment have been positive. > But it would be good to know if anyone is encountering any problems with > it in real life. > > Thanks, Chris > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Experiences with Tentacle 20.2.0
by Wannes Smet 02 Feb '26

02 Feb '26

This story jumps to mind: https://docs.clyso.com/blog/2026/01/13/tentacle-v20.2.0-ec-bug/ ________________________________ From: Chris Palmer via ceph-users <ceph-users(a)ceph.io> Sent: Monday, February 2, 2026 10:16 To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] Experiences with Tentacle 20.2.0 I'd like to know whether anyone has any good or bad experiences with Tentacle 20. 2. 0 in a production environment. I have the opportunity to upgrade a small production cluster from 19. 2. 3 to 20. 2. 0. It uses Cephadm/RBD/CephFS/RGW with R3 only ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I'd like to know whether anyone has any good or bad experiences with Tentacle 20.2.0 in a production environment. I have the opportunity to upgrade a small production cluster from 19.2.3 to 20.2.0. It uses Cephadm/RBD/CephFS/RGW with R3 only (no EC, so the known caveats around EC optimisations won't be relevant). My experiences trying it in a lab test environment have been positive. But it would be good to know if anyone is encountering any problems with it in real life. Thanks, Chris _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Experiences with Tentacle 20.2.0
by Chris Palmer 02 Feb '26

02 Feb '26

I'd like to know whether anyone has any good or bad experiences with Tentacle 20.2.0 in a production environment. I have the opportunity to upgrade a small production cluster from 19.2.3 to 20.2.0. It uses Cephadm/RBD/CephFS/RGW with R3 only (no EC, so the known caveats around EC optimisations won't be relevant). My experiences trying it in a lab test environment have been positive. But it would be good to know if anyone is encountering any problems with it in real life. Thanks, Chris

1 0

Re: Urgent help needed.
by gagan tiwari 31 Jan '26

31 Jan '26

Hi Boris, Its 7 osds nodes out of which 2 are acting as mon also. 1 mgr + mon node and 2 dedicated mon nodes Thanks, Gagan On Sat, Jan 31, 2026 at 6:45 PM Boris via ceph-users <ceph-users(a)ceph.io> wrote: > Hi Gagan, > do I understand correctly that you have dedicated 5 Mon, 1 Mgr and 7 OSD > nodes? 13 nodes i total? > > And you only need to wipe 3 Mon and the mgr nodes? > > If this is the case, you will not have problems with the data on the OSDs. > You can just move the mgr to one of the good mon nodes and remove the mon > nodes from quorum. > > The cluster will be fine. You might need to adjust the ceph.conf on the > clients, to not ise the mons you removed from the quorum. > > Alternatively you can just wipe and reinstall the systems. Ceph should > handle one missing node fine. > > You also rarely need a dedicated node for mgr. you can run them together > with the mons. They will failover of the acting mgr is not available > anymore. > > Cheers > Boris > > > Am 31.01.2026 um 13:02 schrieb gagan tiwari via ceph-users < > ceph-users(a)ceph.io>: > > > > HI Guys, > > Our ceph cluster seems to be compromised by rootkit > > attack. > > > > It has 5 mon including one mgr and 7 osd nodes. > > > > We will need to reformat and reinstall OS on ceph mgr node and 2 ceph mon > > nodes. > > > > 2 mon nodes and 7 osd nodes are safe. > > > > Ceph health status is showing ok. And the cluster is still working but > very > > slow. > > > > Please let me know how to proceed to have minimal disruption to the users > > and without any data loss > > > > Thanks, > > Gagan > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: Urgent help needed.
by William Edwards 31 Jan '26

31 Jan '26

William David Edwards > Op 31 jan 2026 om 13:02 heeft gagan tiwari via ceph-users <ceph-users(a)ceph.io> het volgende geschreven: > > HI Guys, > Our ceph cluster seems to be compromised by rootkit > attack. > How did you determine that? > It has 5 mon including one mgr and 7 osd nodes. > > We will need to reformat and reinstall OS on ceph mgr node and 2 ceph mon > nodes. > > 2 mon nodes and 7 osd nodes are safe. > > Ceph health status is showing ok. And the cluster is still working but very > slow. Why slow? > > Please let me know how to proceed to have minimal disruption to the users > and without any data loss What have you tried? What’s your plan so far? > > Thanks, > Gagan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Urgent help needed.
by Boris 31 Jan '26

31 Jan '26

Hi Gagan, do I understand correctly that you have dedicated 5 Mon, 1 Mgr and 7 OSD nodes? 13 nodes i total? And you only need to wipe 3 Mon and the mgr nodes? If this is the case, you will not have problems with the data on the OSDs. You can just move the mgr to one of the good mon nodes and remove the mon nodes from quorum. The cluster will be fine. You might need to adjust the ceph.conf on the clients, to not ise the mons you removed from the quorum. Alternatively you can just wipe and reinstall the systems. Ceph should handle one missing node fine. You also rarely need a dedicated node for mgr. you can run them together with the mons. They will failover of the acting mgr is not available anymore. Cheers Boris > Am 31.01.2026 um 13:02 schrieb gagan tiwari via ceph-users <ceph-users(a)ceph.io>: > > HI Guys, > Our ceph cluster seems to be compromised by rootkit > attack. > > It has 5 mon including one mgr and 7 osd nodes. > > We will need to reformat and reinstall OS on ceph mgr node and 2 ceph mon > nodes. > > 2 mon nodes and 7 osd nodes are safe. > > Ceph health status is showing ok. And the cluster is still working but very > slow. > > Please let me know how to proceed to have minimal disruption to the users > and without any data loss > > Thanks, > Gagan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Urgent help needed.
by gagan tiwari 31 Jan '26

31 Jan '26

HI Guys, Our ceph cluster seems to be compromised by rootkit attack. It has 5 mon including one mgr and 7 osd nodes. We will need to reformat and reinstall OS on ceph mgr node and 2 ceph mon nodes. 2 mon nodes and 7 osd nodes are safe. Ceph health status is showing ok. And the cluster is still working but very slow. Please let me know how to proceed to have minimal disruption to the users and without any data loss Thanks, Gagan

1 0

[ceph-user][ceph-ansible] Quincy: Module 'devicehealth' has failed: unknown operation
by Danish Khan 31 Jan '26

31 Jan '26

Dear Team, I am keep getting below error after every few days: Error : Module 'devicehealth' has failed: unknown operation Actual error from MGR log: 2026-01-30T05:50:59.338+0530 7f314da26640 0 [devicehealth ERROR root] Caught fatal database error: unknown operation 2026-01-30T05:50:59.338+0530 7f314da26640 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'devicehealth' while running on mgr.controller3: unknown operation 2026-01-30T05:50:59.338+0530 7f314da26640 -1 devicehealth.serve: 2026-01-30T05:50:59.338+0530 7f314da26640 -1 Traceback (most recent call last): File "/usr/share/ceph/mgr/devicehealth/module.py", line 408, in serve self._do_serve() File "/usr/share/ceph/mgr/mgr_module.py", line 513, in check return func(self, *args, **kwargs) File "/usr/share/ceph/mgr/devicehealth/module.py", line 399, in _do_serve self.scrape_all() File "/usr/share/ceph/mgr/devicehealth/module.py", line 449, in scrape_all self.put_device_metrics(device, data) File "/usr/share/ceph/mgr/devicehealth/module.py", line 525, in put_device_metrics self._create_device(devid) File "/usr/share/ceph/mgr/devicehealth/module.py", line 511, in _create_device cursor = self.db.execute(SQL, (devid,)) sqlite3.InternalError: unknown operation 2026-01-30T05:50:59.366+0530 7f3155335640 0 log_channel(cluster) log [DBG] : pgmap v116150: 6977 pgs: 92 peering, 40 stale+active+clean, 191 active+clean+laggy, 30 active+undersized, 1026 active+undersized+degraded, 5598 active+clean; 48 TiB data, 153 TiB used, 1.2 PiB / 1.4 PiB avail; 44 MiB/s wr, 1.00k op/s; 8043806/154077270 objects degraded (5.221%); 4.4 GiB/s, 8 keys/s, 530 objects/s recovering And I get one warning for mgr daemon crash but all MGR Service actually doesn't get failed or restarted automatically. I guess it is something related to sqlite3 which checks ceph health internally. Although this is not causing any issue and error disappears after restarting ceph-mgr service or doing a failover of MGR to another controller. Does anyone know how we can resolve this issue permanently? Similar issue: https://github.com/rook/rook/issues/12349 Regards, Danish

1 0

Ceph Community Updates for January
by Anthony Middleton 30 Jan '26

30 Jan '26

Hey Ceph Friends, How is it still January? It feels like this month has lasted eight weeks, but we keep making our way through it, so congrats to all of us! Here are some updates and events happening in the Ceph ecosystem. - A reminder that the Ceph Days Raleigh CFP is still open! Submit your proposal before February 16: https://docs.google.com/forms/d/e/1FAIpQLSfHUyLahd9EFAIJCF0niveMhszt_F5lgTU… - Check out the recording of this month's Ceph User + Dev Meetup. This month's session ran longer than expected, so we will continue this conversation and cover the remaining agenda topics in April. - Watch the recording here: https://youtu.be/r4c2SMtsVLI. <https://youtu.be/r4c2SMtsVLI> - We held our first Ceph Tech Talk of 2026 on "Running Teuthology Outside of the Sepia Lab." Here is the recording: https://youtu.be/6ovdJ79AqbM - *If you would like to lead a Tech Talk in the upcoming months, please submit your proposal here: https://airtable.com/apphc2dbSP8GuCdor/shrm3t1Fwue5YfAlf. <https://airtable.com/apphc2dbSP8GuCdor/shrm3t1Fwue5YfAlf>* That's all for now. If you have questions or concerns about anything Ceph-related, feel free to reach out to me via email at anthony(a)amicmid.com or on Slack <https://join.slack.com/t/ceph-storage/shared_invite/zt-3jlvf8f6e-45tyKGpqkk…>. Enjoy your weekend! Best, -- Anthony Middleton Community Manager, Ceph Foundation community-manager(a)ceph.io LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0