- ceph-users - lists.ceph.io

ceph mon disable_stretch_mode...no valid command found!
by Maximilian Kaya 05 Dec '25

05 Dec '25

Hi, due to ceph reef documentation I need to run only this command but I am unable to run this to exit the stretched cluster: ceph mon disable_stretch_mode distributed_cluster_rule --yes-i-really-mean-it ... no valid command found; 10 closest matches: ... Error EINVAL: invalid command … what do I am missing ? I have ceph 18.2.7 with 5 mons and I would like to extend datacenter by adding dc3 with same number of racks,hosts and osds to get back to repsize 3 using crush rule distributed_cluster_rule: rule distributed_cluster_rule { id 3 type replicated step take default step choose firstn 0 type datacenter step chooseleaf firstn 1 type rack step emit } .. #ceph mon dump epoch 87 fsid 583647c2-4603-11ef-9941-b360866a782f min_mon_release 18 (reef) election_strategy: 1 stretch_mode_enabled 1 tiebreaker_mon mon3 disallowed_leaders mon3 0: [v2:172.16.1.1:3300/0,v1:172.16.1.1:6789/0] mon.mon1; crush_location {datacenter=dc1} 1: [v2:172.16.1.2:3300/0,v1:172.16.1.2:6789/0] mon.mon2; crush_location {datacenter=dc2} 2: [v2:172.16.1.3:3300/0,v1:172.16.1.3:6789/0] mon.mon3; crush_location {datacenter=quorum_room} 3: [v2:172.16.1.4:3300/0,v1:172.16.1.4:6789/0] mon.mon4; crush_location {datacenter=dc2} 4: [v2:172.16.1.5:3300/0,v1:172.16.1.5:6789/0] mon.mon5; crush_location {datacenter=dc1} is there any other way to exit stretched cluster? any help would be appreciated. BR Max

1 0

Re: reef 18.2.8 QE validation status
by Travis Nielsen 04 Dec '25

04 Dec '25

The latest Reef tests are passing for Rook. Travis On Mon, Dec 1, 2025 at 7:18 AM Ilya Dryomov via ceph-users < ceph-users(a)ceph.io> wrote: > On Sat, Nov 29, 2025 at 6:08 PM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > > > > Rerun scheduled for krbd > > It's still the same. I'd suggest keeping an eye on the run before > "admitting" it for a possible approval -- something that is completely > red with not a single job reaching the point where the actual test > starts clearly can't be approved. > > Thanks, > > Ilya > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: On your post: "[Tentacle 20.2.0]: Inconsistent pg's after enabling ec optimisation flag"
by Reto Gysi 04 Dec '25

04 Dec '25

Hi David Sorry I forgot to answer your mail. I both wanted to do some tests first and then also got the flu. The current situation on my small "cluster" is: - After disabling any load on the affected pool , after enough times ceph pg repair and node restarts the inconsistent pg errors/warnings disappeared and the cluster came back into a clean state. - However as soon I started some qemu virtual machines (both a linux and a windows vm) I got crashing osds again and inconsistent pg's. I thought I saw it was mentioned that ec_optimizations would require any client library modifications. But maybe that is not the case for qemu? I haven't tested that yet. - writing to an ext4 fs on an RBD image mounted directly via mount -t ceph on one of the cluster nodes didn't lead to nearl immediate /to very soon osd crashes and inconsistent pg's. - As for now I copied most of the images back to an erasure coded pool without ec_optimization support enabled and there I can run the VM images again with qemu without osd crashing and inconsistent pgs. Regading your questions: - What EC scheme do you use? We are using 6+2 The erasure coded pool I've enabled ec_optimization on used jerasure 3+2 with lz4 compression. - Do you use cephadm? Yes, the cluster has been deployed from the beginning with cephadm - Do you run on via kubernetes? No it's running on baremetal with debian 13 as OS (debian 12 at start of year) - What is the operating system that you are using? Debian 13. The libs on the debian base system are from 18.2.7+ds-1 - What was the version you updated from? I've upgraded from 19.2.3 to 20.2.0. I've ran the cluster without any issues (asides from some dashboard bugs) for a week, before I tried to enable allow_ec_optimization flag on one of the erasure coded pools - What version did you use to create the pool? The original pool was created under 17.2.6 (quincy) back in 2023 - The effected pool, is it hard drives? Yes the pool is hard drives Regards Reto Am Do., 4. Dez. 2025 um 15:21 Uhr schrieb David Walter <david.walter(a)cern.ch >: > Hi Reto, > > I would be very thankful for a reply since it seems we have to solve > this on our own. > Do you have any new insights into the problem? > > Things that could help are: > > - What EC scheme do you use? We are using 6+2 > - Do you use cephadm? > - Do you run on via kubernetes? > - What is the operating system that you are using? > - What was the version you updated from? > - What version did you use to create the pool? > - The effected pool, is it hard drives? > > Best, > > David > > On 12/1/25 07:59, David Walter wrote: > > Hi Reto, > > > > I've seen your post on the ceph users list, unfortunately I was not > > able to reply on the thread or post something myself. > > > > I have the same problem. I was upgrading from ceph squid 19.2.3 to > > ceph tentacle 20.2.0 and all was good first. > > But when I enabled ec optimizations I get scrubbing errors like this: > > > > HEALTH_ERR 88909311 scrub errors; Possible data damage: 117 pgs > > inconsistent > > [ERR] OSD_SCRUB_ERRORS: 88909311 scrub errors > > [ERR] PG_DAMAGED: Possible data damage: 117 pgs inconsistent > > > > and I was unable repairing them. Some errors disappear after > > restarting all osd daemons but they come back after another scrubbing. > > I don't have any indication that data is lost so far. But the cluster > > is under pressure and daemons crash, so it's in a critical state. > > > > When running "rados list-inconsistent-obj <PG>" one some PGs I get > > errors like these: > > > > > {"osd":62,"primary":false,"shard":4,"errors":["size_mismatch_info","obj_size_info_mismatch"],"size":700416,"object_info":{"oid":{"oid":"1000793c33a.000000dd","key":"","snapid":-2,"hash":2213519396,"max":0,"pool":4,"namespace":""},"version":"43143'3253925","prior_version":"0'0","last_reqid":"client.1822828.0:160762343","user_version":3253925,"size":4194304,"mtime":"2025-11-10T22:13:15.257031-0500","local_mtime":"2025-11-10T22:14:44.294560-0500","lost":0,"flags":["dirty","data_digest"],"truncate_seq":0,"truncate_size":0,"data_digest":"0x7e22aafe","omap_digest":"0xffffffff","expected_object_size":0,"expected_write_size":0,"alloc_hint_flags":0,"manifest":{"type":0},"watchers":{}}},{"osd":65,"primary":false,"shard":3,"errors":[],"size":700416} > > > > > > > Did you find out more about this issue or how do you plan to move > > forward? > > > > Best, > > > > David > > >

1 0

Re: [ceph-users] Re: Stuck v2 notification migration
by Lázár György 04 Dec '25

04 Dec '25

Thank you so much! This worked. This issue kept me busy for about a week, since the rados objects backing the old pubsub framework are not really documented. I'm impressed that I got a highly technical and correct answer so quickly. Since other people might be using multisite the same way we did (with creating the same topic on both sites), this might be worth mentioning in the docs somewhere. Regards, György Lázár Csütörtök, December 04, 2025 15:30 CET dátummal, Casey Bodley via ceph-users <ceph-users(a)ceph.io> ezt írta: > (cc Yuval and Ali) > > the problem is that v1 topics did not support multisite, so i'm > guessing you had to recreate those same topics on each zone. the v2 > migration code can only run on the primary zone, so multisite can't > automatically reconcile those v1 topics on your secondary zone > > > WARNING: topic migration in process. please try again later > > you're seeing this warning on the secondary zone because the v1 topics > still exist there. that's also why "radosgw-admin topic list" is only > showing the v1 topics > > all v1 topics were stored in a single rados object per tenant, named > "pubsub.{tenant}" in the site2.rgw.log pool. if you're not using > tenants, that would just be "pubsub." > > since the v2 topic metadata has already replicated, it should be safe > to delete any of those v1 rados objects manually to complete that v2 > migration: > > 1. first make a local backup: > > $ rados -p site2.rgw.log get pubsub. pubsub.bak > > 2. delete the rados object: > > $ rados -p site2.rgw.log rm pubsub. > > 3. verify that v2 topics are now visible to radosgw-admin: > > $ radosgw-admin topic list > > 4. restart radosgw instances that are still caching the v1 topic metadata > > repeat steps 1 and 2 for other tenant names as needed. if necessary, > you can restore the v1 metadata with: > > $ rados -p site2.rgw.log put pubsub. pubsub.bak > > > On Thu, Dec 4, 2025 at 6:08 AM Lázár György via ceph-users > <ceph-users(a)ceph.io> wrote: > > > > Hello, > > I have an object multisite setup. I'm on squid 19.2.3 (upgraded, not a greenfield deployment - both clusters are on the same builds), so I upgraded to notification_v2 as well. However, while the primary zone converted the notifications and topics to v2 easily, the secondary zone seems to have issues with this (topics and notifications are not synced to the secondary, but the rados objects are synced, shown via rados -p site2.rgw.log -N notif ls. The radosgw-admin topic list command also gives back the v1 format of the json output on the secondary zone, while having the v2 json output on the primary). What's (almost) certain is, that one of the clusters (the primary) wants to use v2, and the other the v1 notification style, and mixing these two causes issues (topics randomly breaking, etc). Also, since 19.2.3, in theory, if I create a topic on the secondary zone, the operation should be forwarded to the primary zone. While this happens, the topic creation still fails with an 503 error, while the topic still gets created. The rgw on the secondary site after this half-failed operation also spits out logs like: > > rgw debug 2025-11-26T11:51:14.241+0000 7ffa54948640 1 req 12442569609481506669 0.000000000s sns:pubsub_topic_create ERROR: topic not found > > rgw debug 2025-11-26T11:51:14.254+0000 7ffa2e8fc640 1 req 12442569609481506669 0.013000983s sns:pubsub_topic_create WARNING: topic migration in process. please try again later > > I tried to create another multisite setup (same clusters, different realm), and there the notification v2 scheme works without an issue. I think if I would delete the secondary zone and re-create it (including the pools), it would fix the issue, but since that operation has a relatively large blast radius (especially if something goes wrong, but there are some services that use only the secondary zone, so they should be redirected beforehand), I'm a bit hesitant to do it (before the christmas holidays). I also tried searching for these error messages on google, with no luck. I also tried searching for these error messages in the RGW source code, and still no luck. I can find them in the compiled RGW binary at least. > > > > Any help would be greatly appreciated! I would be also very happy if I wouldn't have to nuke the secondary zone. > > > > Redards, > > György Lázár > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: bucket/user usage metrics (size, objects count)
by Konstantin Shalygin 04 Dec '25

04 Dec '25

Hi, https://github.com/blemmenes/radosgw_usage_exporter k Sent from my iPhone > On 4 Dec 2025, at 17:47, Mindaugas Mikučionis via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > Running squid. What is best way to expose metrics per bucket/user usage > (size per bucket, object count per bucket, size per user, object count per > user) ? > As i see ceph-exporter exposes only per bucket/user ops counter metrics. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [ceph-users] bucket/user usage metrics (size, objects count)
by Lázár György 04 Dec '25

04 Dec '25

Hi! Not sure about "the best way", but we are running this (with a few custom modifications): https://github.com/blemmenes/radosgw_usage_exporter Regards, György Lázár Csütörtök, December 04, 2025 15:47 CET dátummal, Mindaugas Mikučionis via ceph-users <ceph-users(a)ceph.io> ezt írta: > Hi, > > Running squid. What is best way to expose metrics per bucket/user usage > (size per bucket, object count per bucket, size per user, object count per > user) ? > As i see ceph-exporter exposes only per bucket/user ops counter metrics. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

bucket/user usage metrics (size, objects count)
by Mindaugas Mikučionis 04 Dec '25

04 Dec '25

Hi, Running squid. What is best way to expose metrics per bucket/user usage (size per bucket, object count per bucket, size per user, object count per user) ? As i see ceph-exporter exposes only per bucket/user ops counter metrics.

1 0

Re: Updating cephadm-managed Tentacle cluster from RC to production
by Michael Worsham 04 Dec '25

04 Dec '25

So I take it that you're all good now? -- Michael ________________________________ From: Marek Szuba via ceph-users <ceph-users(a)ceph.io> Sent: Wednesday, December 3, 2025 12:01 PM To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Updating cephadm-managed Tentacle cluster from RC to production This is an external email. Please take care when clicking links or opening attachments. When in doubt, check with the Help Desk or Security. On 2025-12-03 15:57, Marek Szuba via ceph-users wrote: > Well, things have got interesting now. The above appears to have worked > for 4 out of 5 mgr daemons - but now most "ceph orch" commands give me > the error "Error ENOENT: Module not found". I can also see many > cephadm-related "module not found" error in system logs. Progress! Having carefully searched through all the logs, I discovered the module was unable to load due to a spurious field in certificate JSON. A quick look at the relevant bit of Python code then allowed me to identify the field "editable", present in all six cert/key config keys, as the culprit. Having carefully edited and reset the offending JSON, I was then able to successfully run 'ceph orch upgrade start --ceph-version 20.2.0' (which redid the mgr and mon upgrade - that was intentional, in case the manual redeployment with new image missed something). This has just successfully completed. @Michael: I did see that bit of the documentation, I just had no clue which config keys exactly to examine and what to look for in them. Fortunately it turned out that some of the operations resulting in "Module not found" also crashed the active mgr (interestingly enough, it wasn't all such operations - Ceph only noted two mgr crashes), and I was able to find the traceback in crash metadata. -- MS _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

1 0

Re: Stuck v2 notification migration
by Casey Bodley 04 Dec '25

04 Dec '25

(cc Yuval and Ali) the problem is that v1 topics did not support multisite, so i'm guessing you had to recreate those same topics on each zone. the v2 migration code can only run on the primary zone, so multisite can't automatically reconcile those v1 topics on your secondary zone > WARNING: topic migration in process. please try again later you're seeing this warning on the secondary zone because the v1 topics still exist there. that's also why "radosgw-admin topic list" is only showing the v1 topics all v1 topics were stored in a single rados object per tenant, named "pubsub.{tenant}" in the site2.rgw.log pool. if you're not using tenants, that would just be "pubsub." since the v2 topic metadata has already replicated, it should be safe to delete any of those v1 rados objects manually to complete that v2 migration: 1. first make a local backup: $ rados -p site2.rgw.log get pubsub. pubsub.bak 2. delete the rados object: $ rados -p site2.rgw.log rm pubsub. 3. verify that v2 topics are now visible to radosgw-admin: $ radosgw-admin topic list 4. restart radosgw instances that are still caching the v1 topic metadata repeat steps 1 and 2 for other tenant names as needed. if necessary, you can restore the v1 metadata with: $ rados -p site2.rgw.log put pubsub. pubsub.bak On Thu, Dec 4, 2025 at 6:08 AM Lázár György via ceph-users <ceph-users(a)ceph.io> wrote: > > Hello, > I have an object multisite setup. I'm on squid 19.2.3 (upgraded, not a greenfield deployment - both clusters are on the same builds), so I upgraded to notification_v2 as well. However, while the primary zone converted the notifications and topics to v2 easily, the secondary zone seems to have issues with this (topics and notifications are not synced to the secondary, but the rados objects are synced, shown via rados -p site2.rgw.log -N notif ls. The radosgw-admin topic list command also gives back the v1 format of the json output on the secondary zone, while having the v2 json output on the primary). What's (almost) certain is, that one of the clusters (the primary) wants to use v2, and the other the v1 notification style, and mixing these two causes issues (topics randomly breaking, etc). Also, since 19.2.3, in theory, if I create a topic on the secondary zone, the operation should be forwarded to the primary zone. While this happens, the topic creation still fails with an 503 error, while the topic still gets created. The rgw on the secondary site after this half-failed operation also spits out logs like: > rgw debug 2025-11-26T11:51:14.241+0000 7ffa54948640 1 req 12442569609481506669 0.000000000s sns:pubsub_topic_create ERROR: topic not found > rgw debug 2025-11-26T11:51:14.254+0000 7ffa2e8fc640 1 req 12442569609481506669 0.013000983s sns:pubsub_topic_create WARNING: topic migration in process. please try again later > I tried to create another multisite setup (same clusters, different realm), and there the notification v2 scheme works without an issue. I think if I would delete the secondary zone and re-create it (including the pools), it would fix the issue, but since that operation has a relatively large blast radius (especially if something goes wrong, but there are some services that use only the secondary zone, so they should be redirected beforehand), I'm a bit hesitant to do it (before the christmas holidays). I also tried searching for these error messages on google, with no luck. I also tried searching for these error messages in the RGW source code, and still no luck. I can find them in the compiled RGW binary at least. > > Any help would be greatly appreciated! I would be also very happy if I wouldn't have to nuke the secondary zone. > > Redards, > György Lázár > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Stuck v2 notification migration
by Lázár György 04 Dec '25

04 Dec '25

Hello, I have an object multisite setup. I'm on squid 19.2.3 (upgraded, not a greenfield deployment - both clusters are on the same builds), so I upgraded to notification_v2 as well. However, while the primary zone converted the notifications and topics to v2 easily, the secondary zone seems to have issues with this (topics and notifications are not synced to the secondary, but the rados objects are synced, shown via rados -p site2.rgw.log -N notif ls. The radosgw-admin topic list command also gives back the v1 format of the json output on the secondary zone, while having the v2 json output on the primary). What's (almost) certain is, that one of the clusters (the primary) wants to use v2, and the other the v1 notification style, and mixing these two causes issues (topics randomly breaking, etc). Also, since 19.2.3, in theory, if I create a topic on the secondary zone, the operation should be forwarded to the primary zone. While this happens, the topic creation still fails with an 503 error, while the topic still gets created. The rgw on the secondary site after this half-failed operation also spits out logs like: rgw debug 2025-11-26T11:51:14.241+0000 7ffa54948640 1 req 12442569609481506669 0.000000000s sns:pubsub_topic_create ERROR: topic not found rgw debug 2025-11-26T11:51:14.254+0000 7ffa2e8fc640 1 req 12442569609481506669 0.013000983s sns:pubsub_topic_create WARNING: topic migration in process. please try again later I tried to create another multisite setup (same clusters, different realm), and there the notification v2 scheme works without an issue. I think if I would delete the secondary zone and re-create it (including the pools), it would fix the issue, but since that operation has a relatively large blast radius (especially if something goes wrong, but there are some services that use only the secondary zone, so they should be redirected beforehand), I'm a bit hesitant to do it (before the christmas holidays). I also tried searching for these error messages on google, with no luck. I also tried searching for these error messages in the RGW source code, and still no luck. I can find them in the compiled RGW binary at least. Any help would be greatly appreciated! I would be also very happy if I wouldn't have to nuke the secondary zone. Redards, György Lázár

1 0