- ceph-users - lists.ceph.io

Re: On your post: "[Tentacle 20.2.0]: Inconsistent pg's after enabling ec optimisation flag"
by Reto Gysi 04 Dec '25

04 Dec '25

Hi David Sorry I forgot to answer your mail. I both wanted to do some tests first and then also got the flu. The current situation on my small "cluster" is: - After disabling any load on the affected pool , after enough times ceph pg repair and node restarts the inconsistent pg errors/warnings disappeared and the cluster came back into a clean state. - However as soon I started some qemu virtual machines (both a linux and a windows vm) I got crashing osds again and inconsistent pg's. I thought I saw it was mentioned that ec_optimizations would require any client library modifications. But maybe that is not the case for qemu? I haven't tested that yet. - writing to an ext4 fs on an RBD image mounted directly via mount -t ceph on one of the cluster nodes didn't lead to nearl immediate /to very soon osd crashes and inconsistent pg's. - As for now I copied most of the images back to an erasure coded pool without ec_optimization support enabled and there I can run the VM images again with qemu without osd crashing and inconsistent pgs. Regading your questions: - What EC scheme do you use? We are using 6+2 The erasure coded pool I've enabled ec_optimization on used jerasure 3+2 with lz4 compression. - Do you use cephadm? Yes, the cluster has been deployed from the beginning with cephadm - Do you run on via kubernetes? No it's running on baremetal with debian 13 as OS (debian 12 at start of year) - What is the operating system that you are using? Debian 13. The libs on the debian base system are from 18.2.7+ds-1 - What was the version you updated from? I've upgraded from 19.2.3 to 20.2.0. I've ran the cluster without any issues (asides from some dashboard bugs) for a week, before I tried to enable allow_ec_optimization flag on one of the erasure coded pools - What version did you use to create the pool? The original pool was created under 17.2.6 (quincy) back in 2023 - The effected pool, is it hard drives? Yes the pool is hard drives Regards Reto Am Do., 4. Dez. 2025 um 15:21 Uhr schrieb David Walter <david.walter(a)cern.ch >: > Hi Reto, > > I would be very thankful for a reply since it seems we have to solve > this on our own. > Do you have any new insights into the problem? > > Things that could help are: > > - What EC scheme do you use? We are using 6+2 > - Do you use cephadm? > - Do you run on via kubernetes? > - What is the operating system that you are using? > - What was the version you updated from? > - What version did you use to create the pool? > - The effected pool, is it hard drives? > > Best, > > David > > On 12/1/25 07:59, David Walter wrote: > > Hi Reto, > > > > I've seen your post on the ceph users list, unfortunately I was not > > able to reply on the thread or post something myself. > > > > I have the same problem. I was upgrading from ceph squid 19.2.3 to > > ceph tentacle 20.2.0 and all was good first. > > But when I enabled ec optimizations I get scrubbing errors like this: > > > > HEALTH_ERR 88909311 scrub errors; Possible data damage: 117 pgs > > inconsistent > > [ERR] OSD_SCRUB_ERRORS: 88909311 scrub errors > > [ERR] PG_DAMAGED: Possible data damage: 117 pgs inconsistent > > > > and I was unable repairing them. Some errors disappear after > > restarting all osd daemons but they come back after another scrubbing. > > I don't have any indication that data is lost so far. But the cluster > > is under pressure and daemons crash, so it's in a critical state. > > > > When running "rados list-inconsistent-obj <PG>" one some PGs I get > > errors like these: > > > > > {"osd":62,"primary":false,"shard":4,"errors":["size_mismatch_info","obj_size_info_mismatch"],"size":700416,"object_info":{"oid":{"oid":"1000793c33a.000000dd","key":"","snapid":-2,"hash":2213519396,"max":0,"pool":4,"namespace":""},"version":"43143'3253925","prior_version":"0'0","last_reqid":"client.1822828.0:160762343","user_version":3253925,"size":4194304,"mtime":"2025-11-10T22:13:15.257031-0500","local_mtime":"2025-11-10T22:14:44.294560-0500","lost":0,"flags":["dirty","data_digest"],"truncate_seq":0,"truncate_size":0,"data_digest":"0x7e22aafe","omap_digest":"0xffffffff","expected_object_size":0,"expected_write_size":0,"alloc_hint_flags":0,"manifest":{"type":0},"watchers":{}}},{"osd":65,"primary":false,"shard":3,"errors":[],"size":700416} > > > > > > > Did you find out more about this issue or how do you plan to move > > forward? > > > > Best, > > > > David > > >

1 0

Re: [ceph-users] Re: Stuck v2 notification migration
by Lázár György 04 Dec '25

04 Dec '25

Thank you so much! This worked. This issue kept me busy for about a week, since the rados objects backing the old pubsub framework are not really documented. I'm impressed that I got a highly technical and correct answer so quickly. Since other people might be using multisite the same way we did (with creating the same topic on both sites), this might be worth mentioning in the docs somewhere. Regards, György Lázár Csütörtök, December 04, 2025 15:30 CET dátummal, Casey Bodley via ceph-users <ceph-users(a)ceph.io> ezt írta: > (cc Yuval and Ali) > > the problem is that v1 topics did not support multisite, so i'm > guessing you had to recreate those same topics on each zone. the v2 > migration code can only run on the primary zone, so multisite can't > automatically reconcile those v1 topics on your secondary zone > > > WARNING: topic migration in process. please try again later > > you're seeing this warning on the secondary zone because the v1 topics > still exist there. that's also why "radosgw-admin topic list" is only > showing the v1 topics > > all v1 topics were stored in a single rados object per tenant, named > "pubsub.{tenant}" in the site2.rgw.log pool. if you're not using > tenants, that would just be "pubsub." > > since the v2 topic metadata has already replicated, it should be safe > to delete any of those v1 rados objects manually to complete that v2 > migration: > > 1. first make a local backup: > > $ rados -p site2.rgw.log get pubsub. pubsub.bak > > 2. delete the rados object: > > $ rados -p site2.rgw.log rm pubsub. > > 3. verify that v2 topics are now visible to radosgw-admin: > > $ radosgw-admin topic list > > 4. restart radosgw instances that are still caching the v1 topic metadata > > repeat steps 1 and 2 for other tenant names as needed. if necessary, > you can restore the v1 metadata with: > > $ rados -p site2.rgw.log put pubsub. pubsub.bak > > > On Thu, Dec 4, 2025 at 6:08 AM Lázár György via ceph-users > <ceph-users(a)ceph.io> wrote: > > > > Hello, > > I have an object multisite setup. I'm on squid 19.2.3 (upgraded, not a greenfield deployment - both clusters are on the same builds), so I upgraded to notification_v2 as well. However, while the primary zone converted the notifications and topics to v2 easily, the secondary zone seems to have issues with this (topics and notifications are not synced to the secondary, but the rados objects are synced, shown via rados -p site2.rgw.log -N notif ls. The radosgw-admin topic list command also gives back the v1 format of the json output on the secondary zone, while having the v2 json output on the primary). What's (almost) certain is, that one of the clusters (the primary) wants to use v2, and the other the v1 notification style, and mixing these two causes issues (topics randomly breaking, etc). Also, since 19.2.3, in theory, if I create a topic on the secondary zone, the operation should be forwarded to the primary zone. While this happens, the topic creation still fails with an 503 error, while the topic still gets created. The rgw on the secondary site after this half-failed operation also spits out logs like: > > rgw debug 2025-11-26T11:51:14.241+0000 7ffa54948640 1 req 12442569609481506669 0.000000000s sns:pubsub_topic_create ERROR: topic not found > > rgw debug 2025-11-26T11:51:14.254+0000 7ffa2e8fc640 1 req 12442569609481506669 0.013000983s sns:pubsub_topic_create WARNING: topic migration in process. please try again later > > I tried to create another multisite setup (same clusters, different realm), and there the notification v2 scheme works without an issue. I think if I would delete the secondary zone and re-create it (including the pools), it would fix the issue, but since that operation has a relatively large blast radius (especially if something goes wrong, but there are some services that use only the secondary zone, so they should be redirected beforehand), I'm a bit hesitant to do it (before the christmas holidays). I also tried searching for these error messages on google, with no luck. I also tried searching for these error messages in the RGW source code, and still no luck. I can find them in the compiled RGW binary at least. > > > > Any help would be greatly appreciated! I would be also very happy if I wouldn't have to nuke the secondary zone. > > > > Redards, > > György Lázár > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: bucket/user usage metrics (size, objects count)
by Konstantin Shalygin 04 Dec '25

04 Dec '25

Hi, https://github.com/blemmenes/radosgw_usage_exporter k Sent from my iPhone > On 4 Dec 2025, at 17:47, Mindaugas Mikučionis via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > Running squid. What is best way to expose metrics per bucket/user usage > (size per bucket, object count per bucket, size per user, object count per > user) ? > As i see ceph-exporter exposes only per bucket/user ops counter metrics. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [ceph-users] bucket/user usage metrics (size, objects count)
by Lázár György 04 Dec '25

04 Dec '25

Hi! Not sure about "the best way", but we are running this (with a few custom modifications): https://github.com/blemmenes/radosgw_usage_exporter Regards, György Lázár Csütörtök, December 04, 2025 15:47 CET dátummal, Mindaugas Mikučionis via ceph-users <ceph-users(a)ceph.io> ezt írta: > Hi, > > Running squid. What is best way to expose metrics per bucket/user usage > (size per bucket, object count per bucket, size per user, object count per > user) ? > As i see ceph-exporter exposes only per bucket/user ops counter metrics. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

bucket/user usage metrics (size, objects count)
by Mindaugas Mikučionis 04 Dec '25

04 Dec '25

Hi, Running squid. What is best way to expose metrics per bucket/user usage (size per bucket, object count per bucket, size per user, object count per user) ? As i see ceph-exporter exposes only per bucket/user ops counter metrics.

1 0

Re: Updating cephadm-managed Tentacle cluster from RC to production
by Michael Worsham 04 Dec '25

04 Dec '25

So I take it that you're all good now? -- Michael ________________________________ From: Marek Szuba via ceph-users <ceph-users(a)ceph.io> Sent: Wednesday, December 3, 2025 12:01 PM To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Updating cephadm-managed Tentacle cluster from RC to production This is an external email. Please take care when clicking links or opening attachments. When in doubt, check with the Help Desk or Security. On 2025-12-03 15:57, Marek Szuba via ceph-users wrote: > Well, things have got interesting now. The above appears to have worked > for 4 out of 5 mgr daemons - but now most "ceph orch" commands give me > the error "Error ENOENT: Module not found". I can also see many > cephadm-related "module not found" error in system logs. Progress! Having carefully searched through all the logs, I discovered the module was unable to load due to a spurious field in certificate JSON. A quick look at the relevant bit of Python code then allowed me to identify the field "editable", present in all six cert/key config keys, as the culprit. Having carefully edited and reset the offending JSON, I was then able to successfully run 'ceph orch upgrade start --ceph-version 20.2.0' (which redid the mgr and mon upgrade - that was intentional, in case the manual redeployment with new image missed something). This has just successfully completed. @Michael: I did see that bit of the documentation, I just had no clue which config keys exactly to examine and what to look for in them. Fortunately it turned out that some of the operations resulting in "Module not found" also crashed the active mgr (interestingly enough, it wasn't all such operations - Ceph only noted two mgr crashes), and I was able to find the traceback in crash metadata. -- MS _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

1 0

Re: Stuck v2 notification migration
by Casey Bodley 04 Dec '25

04 Dec '25

(cc Yuval and Ali) the problem is that v1 topics did not support multisite, so i'm guessing you had to recreate those same topics on each zone. the v2 migration code can only run on the primary zone, so multisite can't automatically reconcile those v1 topics on your secondary zone > WARNING: topic migration in process. please try again later you're seeing this warning on the secondary zone because the v1 topics still exist there. that's also why "radosgw-admin topic list" is only showing the v1 topics all v1 topics were stored in a single rados object per tenant, named "pubsub.{tenant}" in the site2.rgw.log pool. if you're not using tenants, that would just be "pubsub." since the v2 topic metadata has already replicated, it should be safe to delete any of those v1 rados objects manually to complete that v2 migration: 1. first make a local backup: $ rados -p site2.rgw.log get pubsub. pubsub.bak 2. delete the rados object: $ rados -p site2.rgw.log rm pubsub. 3. verify that v2 topics are now visible to radosgw-admin: $ radosgw-admin topic list 4. restart radosgw instances that are still caching the v1 topic metadata repeat steps 1 and 2 for other tenant names as needed. if necessary, you can restore the v1 metadata with: $ rados -p site2.rgw.log put pubsub. pubsub.bak On Thu, Dec 4, 2025 at 6:08 AM Lázár György via ceph-users <ceph-users(a)ceph.io> wrote: > > Hello, > I have an object multisite setup. I'm on squid 19.2.3 (upgraded, not a greenfield deployment - both clusters are on the same builds), so I upgraded to notification_v2 as well. However, while the primary zone converted the notifications and topics to v2 easily, the secondary zone seems to have issues with this (topics and notifications are not synced to the secondary, but the rados objects are synced, shown via rados -p site2.rgw.log -N notif ls. The radosgw-admin topic list command also gives back the v1 format of the json output on the secondary zone, while having the v2 json output on the primary). What's (almost) certain is, that one of the clusters (the primary) wants to use v2, and the other the v1 notification style, and mixing these two causes issues (topics randomly breaking, etc). Also, since 19.2.3, in theory, if I create a topic on the secondary zone, the operation should be forwarded to the primary zone. While this happens, the topic creation still fails with an 503 error, while the topic still gets created. The rgw on the secondary site after this half-failed operation also spits out logs like: > rgw debug 2025-11-26T11:51:14.241+0000 7ffa54948640 1 req 12442569609481506669 0.000000000s sns:pubsub_topic_create ERROR: topic not found > rgw debug 2025-11-26T11:51:14.254+0000 7ffa2e8fc640 1 req 12442569609481506669 0.013000983s sns:pubsub_topic_create WARNING: topic migration in process. please try again later > I tried to create another multisite setup (same clusters, different realm), and there the notification v2 scheme works without an issue. I think if I would delete the secondary zone and re-create it (including the pools), it would fix the issue, but since that operation has a relatively large blast radius (especially if something goes wrong, but there are some services that use only the secondary zone, so they should be redirected beforehand), I'm a bit hesitant to do it (before the christmas holidays). I also tried searching for these error messages on google, with no luck. I also tried searching for these error messages in the RGW source code, and still no luck. I can find them in the compiled RGW binary at least. > > Any help would be greatly appreciated! I would be also very happy if I wouldn't have to nuke the secondary zone. > > Redards, > György Lázár > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Stuck v2 notification migration
by Lázár György 04 Dec '25

04 Dec '25

Hello, I have an object multisite setup. I'm on squid 19.2.3 (upgraded, not a greenfield deployment - both clusters are on the same builds), so I upgraded to notification_v2 as well. However, while the primary zone converted the notifications and topics to v2 easily, the secondary zone seems to have issues with this (topics and notifications are not synced to the secondary, but the rados objects are synced, shown via rados -p site2.rgw.log -N notif ls. The radosgw-admin topic list command also gives back the v1 format of the json output on the secondary zone, while having the v2 json output on the primary). What's (almost) certain is, that one of the clusters (the primary) wants to use v2, and the other the v1 notification style, and mixing these two causes issues (topics randomly breaking, etc). Also, since 19.2.3, in theory, if I create a topic on the secondary zone, the operation should be forwarded to the primary zone. While this happens, the topic creation still fails with an 503 error, while the topic still gets created. The rgw on the secondary site after this half-failed operation also spits out logs like: rgw debug 2025-11-26T11:51:14.241+0000 7ffa54948640 1 req 12442569609481506669 0.000000000s sns:pubsub_topic_create ERROR: topic not found rgw debug 2025-11-26T11:51:14.254+0000 7ffa2e8fc640 1 req 12442569609481506669 0.013000983s sns:pubsub_topic_create WARNING: topic migration in process. please try again later I tried to create another multisite setup (same clusters, different realm), and there the notification v2 scheme works without an issue. I think if I would delete the secondary zone and re-create it (including the pools), it would fix the issue, but since that operation has a relatively large blast radius (especially if something goes wrong, but there are some services that use only the secondary zone, so they should be redirected beforehand), I'm a bit hesitant to do it (before the christmas holidays). I also tried searching for these error messages on google, with no luck. I also tried searching for these error messages in the RGW source code, and still no luck. I can find them in the compiled RGW binary at least. Any help would be greatly appreciated! I would be also very happy if I wouldn't have to nuke the secondary zone. Redards, György Lázár

1 0

Re: 500 - Internal Server Error ceph RGW Dashboard
by Nizamudeen A 04 Dec '25

04 Dec '25

Hi Vivien, the full_user_id is a key that exists in the latest rgw code (tentacle) but not present in the squid. https://github.com/ceph/ceph/pull/55451 But in the mgr code, the changes are made to the API and they exists in your cluster now but as the rgw is older the API cannot find the key which is why its raising the error here. So I think once you update the rgw, this should go away. Regards, Nizam On Wed, Dec 3, 2025 at 3:29 PM GLE, Vivien via ceph-users < ceph-users(a)ceph.io> wrote: > Hi, > > > I'm trying S3 on my cluster with a RGW in VM. My cluster is currently in > v20.2.0 and RGW is in v19.2.1. S3 is working fine the only problem is that > if I try to search (via RGW VM) for anything user relative (bucket or user > management) on the object ceph dashboard this search leads to 500 - > internal server error with those logs in the mgr > > > 2025-12-03T09:47:06.545+0000 700619b22640 0 [dashboard INFO request] > [::ffff:172.16.112.15:60786] [GET] [500] [0.155s] [admin] [513.0B] > /api/rgw/user/dashboard > 2025-12-03T09:47:06.585+0000 70061a323640 0 [dashboard ERROR exception] > Internal Server Error > Traceback (most recent call last): > File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in > dashboard_exception_handler > return handler(*args, **kwargs) > File "/lib/python3.9/site-packages/cherrypy/_cpdispatch.py", line 54, in > __call__ > return self.callable(*self.args, **self.kwargs) > File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", > line 264, in inner > ret = func(*args, **kwargs) > File "/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py", > line 193, in wrapper > return func(*vpath, **params) > File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 929, in get > return self._get(uid, daemon_name=daemon_name, stats=stats) > File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 941, in > _get > result['uid'] = result['full_user_id'] > KeyError: 'full_user_id' > > I found a workaround with deploying a container RGW via cephadm v20.2.0 > and using it to manage my bucket and user. > > Does radosgw needs to be at the same version as the ceph cluster ? Or is > it a bug ? > > Thanks > > Vivien > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- Nizamudeen A Sr. Software Engineer - IBM Partner Engineer IBM and Red Hat Ceph Storage Red Hat <https://www.redhat.com/> <https://www.redhat.com/>

1 0

Overriding service templates
by Stolte, Felix 04 Dec '25

04 Dec '25

Hi guys, the ganesha.conf file renedered by the nfs service is missing some parameters i would like to use. Is it possible to override the jinja2 template used for rendering the file with my own? Greetings Felix ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellv. Vorsitzende), Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

1 0