- ceph-users - lists.ceph.io

Re: 500 - Internal Server Error ceph RGW Dashboard
by Nizamudeen A 04 Dec '25

04 Dec '25

Hi Vivien, the full_user_id is a key that exists in the latest rgw code (tentacle) but not present in the squid. https://github.com/ceph/ceph/pull/55451 But in the mgr code, the changes are made to the API and they exists in your cluster now but as the rgw is older the API cannot find the key which is why its raising the error here. So I think once you update the rgw, this should go away. Regards, Nizam On Wed, Dec 3, 2025 at 3:29 PM GLE, Vivien via ceph-users < ceph-users(a)ceph.io> wrote: > Hi, > > > I'm trying S3 on my cluster with a RGW in VM. My cluster is currently in > v20.2.0 and RGW is in v19.2.1. S3 is working fine the only problem is that > if I try to search (via RGW VM) for anything user relative (bucket or user > management) on the object ceph dashboard this search leads to 500 - > internal server error with those logs in the mgr > > > 2025-12-03T09:47:06.545+0000 700619b22640 0 [dashboard INFO request] > [::ffff:172.16.112.15:60786] [GET] [500] [0.155s] [admin] [513.0B] > /api/rgw/user/dashboard > 2025-12-03T09:47:06.585+0000 70061a323640 0 [dashboard ERROR exception] > Internal Server Error > Traceback (most recent call last): > File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in > dashboard_exception_handler > return handler(*args, **kwargs) > File "/lib/python3.9/site-packages/cherrypy/_cpdispatch.py", line 54, in > __call__ > return self.callable(*self.args, **self.kwargs) > File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", > line 264, in inner > ret = func(*args, **kwargs) > File "/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py", > line 193, in wrapper > return func(*vpath, **params) > File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 929, in get > return self._get(uid, daemon_name=daemon_name, stats=stats) > File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 941, in > _get > result['uid'] = result['full_user_id'] > KeyError: 'full_user_id' > > I found a workaround with deploying a container RGW via cephadm v20.2.0 > and using it to manage my bucket and user. > > Does radosgw needs to be at the same version as the ceph cluster ? Or is > it a bug ? > > Thanks > > Vivien > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- Nizamudeen A Sr. Software Engineer - IBM Partner Engineer IBM and Red Hat Ceph Storage Red Hat <https://www.redhat.com/> <https://www.redhat.com/>

1 0

Overriding service templates
by Stolte, Felix 04 Dec '25

04 Dec '25

Hi guys, the ganesha.conf file renedered by the nfs service is missing some parameters i would like to use. Is it possible to override the jinja2 template used for rendering the file with my own? Greetings Felix ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellv. Vorsitzende), Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

1 0

[Tentacle 20.2.0]: Size mismatch errors after enabling EC optimization
by David Walter 03 Dec '25

03 Dec '25

Hi, I've recently upgraded from squid 19.2.3 to tentacle 20.2.0 on our 10 Alma 9.6 machines and first everything was good. When I enabled the EC optimizations via ceph osd pool set submit_ec_hdd allow_ec_optimizations true Errors started to appear, basically after each scrubbing, like these: HEALTH_ERR 88909311 scrub errors; Possible data damage: 117 pgs inconsistent [ERR] OSD_SCRUB_ERRORS: 88909311 scrub errors [ERR] PG_DAMAGED: Possible data damage: 117 pgs inconsistent and I was unable repairing them. Some errors disappear after restarting all osd daemons but they come back after another scrubbing. When running "rados list-inconsistent-obj <PG>" on some PGs I get errors like these: {"osd":62,"primary":false,"shard":4,"errors":["size_mismatch_info","obj_size_info_mismatch"],"size":700416,"object_info":{"oid":{"oid":"1000793c33a.000000dd","key":"","snapid":-2,"hash":2213519396,"max":0,"pool":4,"namespace":""},"version":"43143'3253925","prior_version":"0'0","last_reqid":"client.1822828.0:160762343","user_version":3253925,"size":4194304,"mtime":"2025-11-10T22:13:15.257031-0500","local_mtime":"2025-11-10T22:14:44.294560-0500","lost":0,"flags":["dirty","data_digest"],"truncate_seq":0,"truncate_size":0,"data_digest":"0x7e22aafe","omap_digest":"0xffffffff","expected_object_size":0,"expected_write_size":0,"alloc_hint_flags":0,"manifest":{"type":0},"watchers":{}}},{"osd":65,"primary":false,"shard":3,"errors":[],"size":700416} So it looks like there is some size mismatch. I don't have any indication that data is lost so far. But the cluster is under pressure and daemons crash, so it's in a critical state. OSDs crash with errors like [ERR] : 4.cc shard 63(5) soid 4:33394f7f:::10001e60e03.00000000:head : candidate size 311296 info size 307200 mismatch [ERR] : 4.cc shard 63(5) soid 4:33394f7f:::10001e60e03.00000000:head : candidate size 311296 info size 307200 mismatch ... again, there is some size mismatch. I'm pretty sure this is the same error than what was mentioned in this thread: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/F3JYHK4INKS… But there has been no reply yet. Does anyone have an idea what can be done to solve this? Or what I can try to get more information? We don't have copies of most of the data and it would be bad if we loose some. Best regards and many thanks for your help, David

1 0

Re: Updating cephadm-managed Tentacle cluster from RC to production
by Marek Szuba 03 Dec '25

03 Dec '25

On 2025-12-03 15:57, Marek Szuba via ceph-users wrote: > Well, things have got interesting now. The above appears to have worked > for 4 out of 5 mgr daemons - but now most "ceph orch" commands give me > the error "Error ENOENT: Module not found". I can also see many > cephadm-related "module not found" error in system logs. Progress! Having carefully searched through all the logs, I discovered the module was unable to load due to a spurious field in certificate JSON. A quick look at the relevant bit of Python code then allowed me to identify the field "editable", present in all six cert/key config keys, as the culprit. Having carefully edited and reset the offending JSON, I was then able to successfully run 'ceph orch upgrade start --ceph-version 20.2.0' (which redid the mgr and mon upgrade - that was intentional, in case the manual redeployment with new image missed something). This has just successfully completed. @Michael: I did see that bit of the documentation, I just had no clue which config keys exactly to examine and what to look for in them. Fortunately it turned out that some of the operations resulting in "Module not found" also crashed the active mgr (interestingly enough, it wasn't all such operations - Ceph only noted two mgr crashes), and I was able to find the traceback in crash metadata. -- MS

1 0

Re: Updating cephadm-managed Tentacle cluster from RC to production
by Michael Worsham 03 Dec '25

03 Dec '25

Check this from the Ceph documentation on it... https://docs.ceph.com/en/latest/cephadm/upgrade/ Scroll down to the "Module Not Found" section. -- M ________________________________ From: Marek Szuba via ceph-users <ceph-users(a)ceph.io> Sent: Wednesday, December 3, 2025 10:51:57 AM To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Updating cephadm-managed Tentacle cluster from RC to production This is an external email. Please take care when clicking links or opening attachments. When in doubt, check with the Help Desk or Security. On 2025-12-03 14:06, Michael Worsham via ceph-users wrote: > ceph orch daemon redeploy <daemon_name> --image quay.io/ceph/ceph:v20.2.0 Well, things have got interesting now. The above appears to have worked for 4 out of 5 mgr daemons - but now most "ceph orch" commands give me the error "Error ENOENT: Module not found". I can also see many cephadm-related "module not found" error in system logs. Update a few minutes later: have updated the last mgr + all the mons to 20.2.0 by manually updating the container image in their respective unit.run files. Now all ten daemons are on stable Tentacle and the Ceph cluster seems happy - but cephadm still doesn't work. -- MS _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

1 0

Re: [dashboard] bug when adding a label to a host?
by Afreen Misbah 03 Dec '25

03 Dec '25

Hi, Thanks for reporting , you can track it here https://tracker.ceph.com/issues/74078. On Wed, Dec 3, 2025 at 9:55 PM Robert Sander via ceph-users < ceph-users(a)ceph.io> wrote: > Hi, > > Am 03.12.25 um 5:22 PM schrieb Bailey Allison via ceph-users: > > > > Double check the labels in the terminal. I thought I had the same issue > > when adding a label but it was more of a visual thing in the UI, all of > > the labels were still there. > > I had the cephadm log running while performing the editing in the > dashboard. And this told me that it just removed two existing labels. > > I did not add an existing label to the host but created a completely new > one. This empties the list of assigned labels somehow. The UI for that > dialogue really does not look very good. The UX was better in the older > dashboard. > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Afreen

1 0

Re: [dashboard] bug when adding a label to a host?
by Robert Sander 03 Dec '25

03 Dec '25

Hi, Am 03.12.25 um 5:22 PM schrieb Bailey Allison via ceph-users: > > Double check the labels in the terminal. I thought I had the same issue > when adding a label but it was more of a visual thing in the UI, all of > the labels were still there. I had the cephadm log running while performing the editing in the dashboard. And this told me that it just removed two existing labels. I did not add an existing label to the host but created a completely new one. This empties the list of assigned labels somehow. The UI for that dialogue really does not look very good. The UX was better in the older dashboard. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: [dashboard] bug when adding a label to a host?
by Bailey Allison 03 Dec '25

03 Dec '25

Hi Robert, Double check the labels in the terminal. I thought I had the same issue when adding a label but it was more of a visual thing in the UI, all of the labels were still there. Regards, Bailey Allison Service Team Lead 45Drives, Ltd. 866-594-7199 x868 On 2025-12-03 12:13, Robert Sander via ceph-users wrote: > Hi, > > I have a test cluster running on 20.2.0 with the cephadm orchestrator. > > The host edit form in the dashboard has been changed. > > When I try to add a new label the existing labels get removed. > > Is this a known issue? > > Regards

1 0

[dashboard] bug when adding a label to a host?
by Robert Sander 03 Dec '25

03 Dec '25

Hi, I have a test cluster running on 20.2.0 with the cephadm orchestrator. The host edit form in the dashboard has been changed. When I try to add a new label the existing labels get removed. Is this a known issue? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: Updating cephadm-managed Tentacle cluster from RC to production
by Marek Szuba 03 Dec '25

03 Dec '25

On 2025-12-03 14:06, Michael Worsham via ceph-users wrote: > ceph orch daemon redeploy <daemon_name> --image quay.io/ceph/ceph:v20.2.0 Well, things have got interesting now. The above appears to have worked for 4 out of 5 mgr daemons - but now most "ceph orch" commands give me the error "Error ENOENT: Module not found". I can also see many cephadm-related "module not found" error in system logs. Update a few minutes later: have updated the last mgr + all the mons to 20.2.0 by manually updating the container image in their respective unit.run files. Now all ten daemons are on stable Tentacle and the Ceph cluster seems happy - but cephadm still doesn't work. -- MS

1 0