- ceph-users - lists.ceph.io

Module 'devicehealth' has failed
by Alex from North 06 Mar '25

06 Mar '25

Hello everybody! Running 19.2.0 faced an issued still cannot struggle. And this is Module 'devicehealth' has failed: Expecting value: line 1 column 2378 (char 2377) in MGR log I see Mar 04 12:48:07 node2.ec.mts ceph-mgr[3821449]: Traceback (most recent call last): File "/usr/share/ceph/mgr/devicehealth/module.py", line 399, in serve self._do_serve() File "/usr/share/ceph/mgr/mgr_module.py", line 549, in check return func(self, *args, **kwargs) File "/usr/share/ceph/mgr/devicehealth/module.py", line 355, in _do_serve if self.db_ready() and self.enable_monitoring: File "/usr/share/ceph/mgr/mgr_module.py", line 1286, in db_ready return self.db is not None File "/usr/share/ceph/mgr/mgr_module.py", line 1298, in db self._db = self.open_db() File "/usr/share/ceph/mgr/mgr_module.py", line 1265, in open_db if not self.pool_exists(self.MGR_POOL_NAME): File "/usr/share/ceph/mgr/mgr_module.py", line 1106, in pool_exists pools = [p['pool_name'] for p in self.get('osd_map')['pools']] File "/usr/share/ceph/mgr/mgr_module.py", line 1416, in get obj = json.loads(obj) File "/lib64/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/lib64/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/lib64/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 2378 (char 2377) Unfortunately, I do now know how to turn devicehealth off as it is forbidden using standard commands line ceph device monitoring off. Any hint is appreciated. Thank in advance, guys!

2 9

Guidance on Ceph Squid v19.20 Production Deployment – Best Practices and Requirements
by Altrel Fero 06 Mar '25

06 Mar '25

Dear Ceph Community, We need guidance on the best practices for deployment. Specifically, we would like to know: What are the recommended methods for Ceph deploying in Kubernetes? What are the best practices for a production deployment? What network configurations are recommended for a stable and efficient deployment? Is Ceph Squid v19.20 recommended for production use, or should we consider an alternative version? What are the recommended hardware and resource requirements for this setup? Looking forward to your valuable suggestions and insights Best regards, P.S. Altrel Fero ZyBiSys Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

1 0

Error Removing Zone from Zonegroup in Multisite Setup
by Mahnoosh Shahidi 06 Mar '25

06 Mar '25

Hi everyone, We have a Ceph multisite setup with a zonegroup containing three zones: one master and two secondary zones, one of which is designated as an archive. After recently upgrading to Ceph 19.2.1, I attempted to remove the archive zone from the zonegroup following the steps in the Ceph documentation. I ran radosgw-admin zonegroup remove --rgw-zonegroup=<ZONE_GROUP_NAME> --rgw-zone=<ZONE_NAME> --rgw-realm=<REALM_NAME> from the cluster containing the archive zone, followed by radosgw-admin period update --commit --rgw-zonegroup=<ZONE_GROUP_NAME> --rgw-zone=<ZONE_NAME> --rgw-realm=<REALM_NAME> However, the period update --commit command returns the error: ERROR: zonegroup <zonegroup_id> does not contain zone id <zone_id> The zone appears correctly in the zonegroup before the removal, and I expected the archive zone to be successfully removed with the period update committing without errors. Is there an additional step required, or could this be a bug related to Ceph 19.2.1? Any guidance or assistance would be appreciated. Thank you! Best regards, Mahnoosh

3 3

Re: Free space
by quaglio＠bol.com.br 06 Mar '25

06 Mar '25

1 0

mgr module 'orchestrator' is not enabled/loaded
by Alex from North 06 Mar '25

06 Mar '25

Hi all! Faced sad situation I don't know where to dig to. That is why I am here again, hoping for hints. Situation: all the "orch" commands aint available and give error in mgr log mgr.server reply reply (95) Operation not supported Module 'orchestrator' is not enabled/loaded (required by command 'orch ls'): use `ceph mgr module enable orchestrator` to enable it Mar 04 09:03:22 node2.ec.mts conmon[1238298]: 2025-03-04T09:03:22.842+0000 7f0659aed640 -1 mgr.server reply reply (95) Operation not supported Module 'orchestrator' is not enabled/loaded (required by command 'orch ls'): use `ceph mgr module enable orchestrator` to enable it Of course I've tried to enable it as suggested, but it is already enable (always-on). Tried to fail/systemd restart - all for nothing. As I understood, orchestrator is responsible for listening as well on port 8765 to give Prometheus needed info. So, I also have "devicehealth" error, as nothing is on port 8765 where Prom tries to knock. Thinking that maybe redeploy would help and having no possibility to use orchestrator, I tried to deploy new manager manually - the same story, orch is not available. Maybe orchestrator module in a crash loop? Where to check logs? There is nothing in /var/log/ceph/cephadm.log and nothing in mgr log. I am really stuck. Any hint appreciated. Thanks in advance.

3 3

Unintuitive (buggy?) CephFS behaviour when dealing with pool_namespace layout attribute
by Florian Haas 06 Mar '25

06 Mar '25

Hello everyone, I'm seeing some behaviour in CephFS that strikes me as unexpected, and I wonder if others have thoughts about it. Consider this scenario: * Ceph Reef (18.2.4) deployed with Cephadm running on Ubuntu Jammy, CephFS client is running kernel 5.15.0-133-generic. * CephFS is mounted to /mnt, using a CephX identity that has rwp permissions. * CephFS is using a single data pool named cephfs_data. * In the CephFS root directory, there is a subdirectory named "restricted". That subdirectory currently has a single file, "test2.bin", of 8 MiB. There is no other data in the data pool. I can verify this with the following rados command: rados -n client.cephfs -p cephfs_data ls 10000000003.00000001 10000000003.00000000 Now I run this, from the root of my mounted CephFS: cd /mnt setfattr -n ceph.dir.layout.pool_namespace -v restricted /mnt/restricted I realise this is naughty, because there is currently a file named /mnt/restricted/test2.bin, and I'm not supposed to set layout attributes on a non-empty directory. However, the command does succeed, and I am able to read back the ceph.dir.layout xattr: getfattr -n ceph.dir.layout restricted # file: restricted ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data pool_namespace=restricted" If at this stage I try to read RADOS objects in the "restricted" namespace, I get an empty list: rados -p cephfs_data -N restricted ls (empty output) Now I move my test file out of the restricted subdirectory and back in: mv /mnt/restricted/test2.bin /mnt/ sync mv /mnt/test2.bin /mnt/restricted/ sync No change: rados -p cephfs_data -N restricted ls (empty output) Next, I try moving the file out, and *copying* it back in: mv /mnt/restricted/test2.bin /mnt/ sync cp /mnt/test2.bin /mnt/restricted/ sync Now, I do see objects in the "restricted" namespace: rados -p cephfs_data -N restricted ls 100000001fd.00000001 100000001fd.00000000 Also, if I create a *new* file in the "restricted" subdirectory, then its RADOS objects do end up in the correct namespace: dd if=/dev/urandom of=/mnt/restricted/test3.bin count=3 bs=4M rados -p cephfs_data -N restricted ls 100000001fe.00000000 100000001fe.00000002 100000001fe.00000001 100000001fd.00000001 100000001fd.00000000 By contrast, here's what happens to a non-empty *file,* when I try to set ceph.file.layout: dd if=/dev/urandom of=/mnt/test4.bin count=4 bs=4M setfattr -n ceph.file.layout.pool_namespace -v restricted test4.bin setfattr: test4.bin: Directory not empty So that fails, somewhat expectedly (although the error message is odd). In summary, there are two things that confuse me here: 1. Why does setting ceph.dir.layout.pool_namespace on a non-empty directory succeed, when setting ceph.file.layout.pool_namespace on a non-empty file fails (and even confusingly with a "Directory not empty" message)? 2. Considering that setting ceph.dir.layout.pool_namespace on a non-empty directory does succeed, why does mv'ing a file to a directory with a different pool_namespace, and then mv'ing it back, not result in its RADOS objects moving to the other namespace? So I'm curious: is this a bug (or two), or have I been misunderstanding what's actually the expected behaviour? Cheers, Florian

4 5

Mixed cluster with AMD64 and ARM64 possible?
by Robert Sander 06 Mar '25

06 Mar '25

Hi, just a quick question: Is a mixed cluster with nodes running AMD64 and ARM64 CPUs possible? Is the cephadm orchestrator able to manage such a cluster? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

2 1

cephfs healthy but mounting it some data cannot be accessed
by xadhoom76＠gmail.com 06 Mar '25

06 Mar '25

Hi, we have access and we can mount cephfs volume but for some reason data in it can be listes but when we try to copy some files from it , some can be and some hangs Is there a way to fix it or skip the files we cannot access ? Best regards

1 0

upgrading ceph without orchestrator
by xadhoom76＠gmail.com 05 Mar '25

05 Mar '25

Is it possible to upgrade mgr without using ceph orch ?

2 1

Upgrade: 5 pgs have unknown state; cannot draw any conclusions
by xadhoom76＠gmail.com 04 Mar '25

04 Mar '25

Hi, we have a cluster in which we have lost the osds containg the 5 pgs. How to proceed to get cluster back working ? of course we will lose data..... We cannot upgrade or downgrade with unknown state. Best regards

1 0