- ceph-users - lists.ceph.io

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Eugen Block 03 Feb '26

03 Feb '26

So the cluster is currently recovering and then you wanted to remove another OSD? I'm just trying to get the full picture here. Let's assume the cluster is backfilling because you removed one OSD (and it's not done yet), then you decide to remove another one. If we assume a replicated pool (size 3, min_size 2), this could cause inactive PGs until backfill has finished. Depending on the failure domain (host?) you need to be careful when removing OSDs from multiple failure domains at the same time. That's why some background here would help to better understand what really happens. And does it happen every time you replace a disk? Does it always impact the same PGs? Does the cluster report inactive PGs for more than a few seconds? I was asking for RGW logs during that outage, not the "normal" behaviour (starting new request), so some error logs. The reshardlogs messages are also nothing unusual. Regarding the replacement procedure, it looks like you don't use the orchestrator, right? Am Di., 3. Feb. 2026 um 10:30 Uhr schrieb Rok Jaklič via ceph-users < ceph-users(a)ceph.io>: > ceph -s is in recovery state, moving around 200-300MB/s of data. > > [root@ctplmon1 ~]# ceph osd ok-to-stop 239 > > {"ok_to_stop":true,"osds":[239],"num_ok_pgs":94,"num_not_ok_pgs":0,"ok_become_degraded":["9.e","9.40","9.4a","9.4e","9.69","9.7a","9.87","9.bc","9.c5"," > 9.cf > > ","9.dd","9.128","9.172","9.1b6","9.1c7","9.227","9.244","9.246","9.24e","9.2b3","9.2d3","9.2ee","9.2f1","9.31c","9.335","9.361","9.364","9.38b","9.3b0","9.3d9","9.408","9.41b","9.483","9.49a","9.4c1","9.4dc","9.4ea","9.4fd","9.553","9.554","9.55b","9.57a","9.586","9.5c7","9.61a","9.627","9.68b","9.69d","9.711","9.71d","9.737","9.784","10.12","10.82","10.18c","10.1c7","10.290","10.310","10.390","10.434","10.4aa","10.4e1","10.525","10.558","10.57f","10.5e2","10.5e7","10.62f","10.736","10.842","10.8d9","10.904","10.964","10.977","10.9eb","10.a1a","10.a40","10.b80","10.bbf","10.be2","10.c7c","10.c9b","10.cb0","10.cb3","10.d7a","10.dc8","10.dcf","10.dde","10.e5b","10.e6d","10.e78","10.ece","10.ef1","10.f60"]} > > rgw logs is usually something like: > 2026-02-03T09:48:32.050+0100 7f34ff13a640 1 ====== starting new request > req=0x7f34b8f0c640 ===== > 2026-02-03T09:48:35.901+0100 7f34cb0d2640 1 ====== starting new request > req=0x7f34b8e8b640 ===== > 2026-02-03T09:49:41.408+0100 7f34e00fc640 1 ====== starting new request > req=0x7f34b8d08640 ===== > 2026-02-03T09:49:46.267+0100 7f35511de640 1 ====== starting new request > req=0x7f34b8c87640 ===== > > while I can see also: > 2026-02-03T09:49:50.912+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000000 to be held by another RGW process; > skipping for now > 2026-02-03T09:49:50.919+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000002 to be held by another RGW process; > skipping for now > 2026-02-03T09:49:50.926+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000004 to be held by another RGW process; > skipping for now > 2026-02-03T09:49:50.934+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000006 to be held by another RGW process; > skipping for now > 2026-02-03T09:49:50.953+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000008 to be held by another RGW process; > skipping for now > 2026-02-03T09:49:50.960+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000010 to be held by another RGW process; > skipping for now > 2026-02-03T09:49:50.968+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000012 to be held by another RGW process; > skipping for now > 2026-02-03T09:49:50.975+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000014 to be held by another RGW process; > skipping for now > > but do not know if it is related. > > ... > > Should I use some different strategy to "replace" failed disks? > > Right now I am doing something like: > ceph osd crush reweight osd.{osdid} 4 3 2 1 ... in steps > > while ! ceph osd safe-to-destroy osd.{osdid} ; do sleep 10 ; done > > systemctl stop ceph-osd@{id} > ceph osd down osd.{id} > ceph osd destroy {id} --yes-i-really-mean-it > > ceph-volume lvm prepare --osd-id {id} --data /dev/sdX > ceph-volume lvm list > ceph-volume lvm activate {id} {osd fsid} > > > > On Tue, Feb 3, 2026 at 10:21 AM Eugen Block <eblock82(a)gmail.com> wrote: > > > Hi, > > > > what does 'ceph osd ok-to-stop <OSD_ID>' show? What do the rgw logs > > reveal? What's the 'ceph status' during that time? > > > > Am Di., 3. Feb. 2026 um 10:18 Uhr schrieb Rok Jaklič via ceph-users < > > ceph-users(a)ceph.io>: > > > >> Hi, > >> > >> we currently have 240 OSDs in the cluster, but when I set the reweight > of > >> just one failed OSD to 0 (from 5.45798), radosgw stops working for > >> some reason. > >> > >> If I do "gentle reweight" in steps of 1 for example (from 5.4 to 4.4), > it > >> works ok. > >> > >> Any ideas why? > >> > >> Running on reef ceph version 18.2.7. > >> > >> Kind regards, > >> Rok > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

[squid] buckets not in bucket stats in multisite setup when create in a non master zonegroup
by Boris 03 Feb '26

03 Feb '26

Hi, we just encountered a strange problem: When we run radosgw-admin bucket stats on the master zonegroup we get some of these errors error getting bucket stats bucket=rregergf ret=-2 This is the debug_rgw=20 output: error getting bucket stats bucket=rregergf ret=2026-02-03T00:50:32.347+0000 722007981a80 20 cls_bucket_head: CLSRGWIssueGetDirHeader() returned -2 -2 failure: (2) No such file or directory: But the bucketname is listed in a bucket list command. After some digging we found that the bucket was created after the upgrade to squid in a non master zonegroup. Doing the radosgw-admin bucket stats command in any zonegroup, except the master, yields results. We checked, and it is every bucket created after the upgrade, that was not created in the master zonegroup. We only sync metadata between the zonegrouops, and I just ran a fresh metadata sync on all zonegroups. Any ideas what this could be?

1 1

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Eugen Block 03 Feb '26

03 Feb '26

Hi, what does 'ceph osd ok-to-stop <OSD_ID>' show? What do the rgw logs reveal? What's the 'ceph status' during that time? Am Di., 3. Feb. 2026 um 10:18 Uhr schrieb Rok Jaklič via ceph-users < ceph-users(a)ceph.io>: > Hi, > > we currently have 240 OSDs in the cluster, but when I set the reweight of > just one failed OSD to 0 (from 5.45798), radosgw stops working for > some reason. > > If I do "gentle reweight" in steps of 1 for example (from 5.4 to 4.4), it > works ok. > > Any ideas why? > > Running on reef ceph version 18.2.7. > > Kind regards, > Rok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 2

radosgw (s3) stops when setting weight to 0 on one OSD
by Rok Jaklič 03 Feb '26

03 Feb '26

Hi, we currently have 240 OSDs in the cluster, but when I set the reweight of just one failed OSD to 0 (from 5.45798), radosgw stops working for some reason. If I do "gentle reweight" in steps of 1 for example (from 5.4 to 4.4), it works ok. Any ideas why? Running on reef ceph version 18.2.7. Kind regards, Rok

1 0

Re: upmap-read balancer with CephFS kernel clients
by Eugen Block 03 Feb '26

03 Feb '26

Ah, forgot to mention that our compute and control nodes (OpenStack) also mount the CephFS for live migration purposes (Nova) and image to volume conversion (Cinder), and they apparently also report their feature set as "luminous". Am Di., 3. Feb. 2026 um 08:15 Uhr schrieb Eugen Block via ceph-users < ceph-users(a)ceph.io>: > Hi Janek, > > this 'ceph features' output that you're probably referring to (or maybe > 'ceph tell mon.X sessions') comes up every now and then, for example [0], > [1], [2]. I also still don't have a definitive answer, unfortunately. But > from my understanding those are compatibility features that don't > necessarily reflect the actual ceph version. For example, I have a test > cluster with tentacle: > > tentacle:~ # ceph versions > { > "mon": { > "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) > tentacle (stable - RelWithDebInfo)": 1 > }, > "mgr": { > "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) > tentacle (stable - RelWithDebInfo)": 2 > }, > "osd": { > "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) > tentacle (stable - RelWithDebInfo)": 3 > }, > "mds": { > "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) > tentacle (stable - RelWithDebInfo)": 1 > }, > "overall": { > "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) > tentacle (stable - RelWithDebInfo)": 7 > } > } > > but 'ceph features' shows squid: > > tentacle:~ # ceph features > { > "mon": [ > { > "features": "0x3f07fffffffdffff", > "release": "squid", > "num": 1 > } > ], > "mds": [ > { > "features": "0x3f07fffffffdffff", > "release": "squid", > "num": 1 > } > ], > "osd": [ > { > "features": "0x3f07fffffffdffff", > "release": "squid", > "num": 3 > } > ], > ... > > And this cluster also has "luminous" clients connected although I enforced > "squid" with 'ceph osd set-require-min-compat-client squid'. In our > production cluster I haven't enforced that, it is still running with > luminous as min-compat-client because we actually do have a couple of older > clients and don't want to risk anything, but we're going to have to deal > with them at some point. > > Anyway, I'm not recommending to enforce the require-min-compat-client > update, I'd rather suggest to set up a test cluster with a couple of your > typical clients maybe, and then see if they successfully connect. I did the > same in my lab virtual environment that resembles our prod. cluster > (OpenStack with Ceph), there I enforced "reef" as min-compat-client and new > mounts from Ubuntu machines (24.04, kernel 6.8.0) with Ceph 19.2.1 packages > still report luminous on a CephFS mount. But the mount works, so I assume > that everything is okay in the lab. But I'm still not feeling confident > enough to recommend you to update it. ;-) > Hopefully someone else chimes in and contradicts or confirms my > statement(s). > > Regards, > Eugen > > [0] > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/A4IWXINDBM6… > [1] > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RVND3Y3Z7FD… > [2] > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/WSADGYBR7PL… > > Am Mi., 28. Jan. 2026 um 09:16 Uhr schrieb Janek Bevendorff via ceph-users > < > ceph-users(a)ceph.io>: > > > I just noticed that this applies to RBD clients as well. How is this > > supposed to work? Can you use only radosgw and ceph-fuse if you want to > > use upmap-read? It seems strange to me that the most central Ceph > > systems don't work with these new features. > > > > > > On 26/01/2026 18:55, Janek Bevendorff via ceph-users wrote: > > > Hi, > > > > > > Is there a way to use the new upmap-read balancer mode when I have > > > CephFS kernel clients connected? Our cluster runs Squid and all > > > connected clients are on kernel 6.11-6.14, so they should be new > > > enough. However, they identify themselves as luminous (I could not > > > find clear documentation about this, but this seems to be hard-coded > > > and by design), so I cannot run ceph osd set-require-min-compat-client > > > reef, which is required for the balancer module to work. > > > > > > I have 500 clients connected and migrating them all to FUSE would be a > > > headache. Is there a way around that? Some of my pools have extremely > > > high read balance scores and we're also struggling with performance > > > issues as a result. > > > > > > Janek > > > > > > > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > -- > > Bauhaus-Universität Weimar > > Bauhausstr. 9a, R308 > > 99423 Weimar, Germany > > > > Phone: +49 3643 58 3577 > > www.webis.de > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: upmap-read balancer with CephFS kernel clients
by Eugen Block 03 Feb '26

03 Feb '26

Hi Janek, this 'ceph features' output that you're probably referring to (or maybe 'ceph tell mon.X sessions') comes up every now and then, for example [0], [1], [2]. I also still don't have a definitive answer, unfortunately. But from my understanding those are compatibility features that don't necessarily reflect the actual ceph version. For example, I have a test cluster with tentacle: tentacle:~ # ceph versions { "mon": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 1 }, "mgr": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 2 }, "osd": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 3 }, "mds": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 1 }, "overall": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 7 } } but 'ceph features' shows squid: tentacle:~ # ceph features { "mon": [ { "features": "0x3f07fffffffdffff", "release": "squid", "num": 1 } ], "mds": [ { "features": "0x3f07fffffffdffff", "release": "squid", "num": 1 } ], "osd": [ { "features": "0x3f07fffffffdffff", "release": "squid", "num": 3 } ], ... And this cluster also has "luminous" clients connected although I enforced "squid" with 'ceph osd set-require-min-compat-client squid'. In our production cluster I haven't enforced that, it is still running with luminous as min-compat-client because we actually do have a couple of older clients and don't want to risk anything, but we're going to have to deal with them at some point. Anyway, I'm not recommending to enforce the require-min-compat-client update, I'd rather suggest to set up a test cluster with a couple of your typical clients maybe, and then see if they successfully connect. I did the same in my lab virtual environment that resembles our prod. cluster (OpenStack with Ceph), there I enforced "reef" as min-compat-client and new mounts from Ubuntu machines (24.04, kernel 6.8.0) with Ceph 19.2.1 packages still report luminous on a CephFS mount. But the mount works, so I assume that everything is okay in the lab. But I'm still not feeling confident enough to recommend you to update it. ;-) Hopefully someone else chimes in and contradicts or confirms my statement(s). Regards, Eugen [0] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/A4IWXINDBM6… [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RVND3Y3Z7FD… [2] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/WSADGYBR7PL… Am Mi., 28. Jan. 2026 um 09:16 Uhr schrieb Janek Bevendorff via ceph-users < ceph-users(a)ceph.io>: > I just noticed that this applies to RBD clients as well. How is this > supposed to work? Can you use only radosgw and ceph-fuse if you want to > use upmap-read? It seems strange to me that the most central Ceph > systems don't work with these new features. > > > On 26/01/2026 18:55, Janek Bevendorff via ceph-users wrote: > > Hi, > > > > Is there a way to use the new upmap-read balancer mode when I have > > CephFS kernel clients connected? Our cluster runs Squid and all > > connected clients are on kernel 6.11-6.14, so they should be new > > enough. However, they identify themselves as luminous (I could not > > find clear documentation about this, but this seems to be hard-coded > > and by design), so I cannot run ceph osd set-require-min-compat-client > > reef, which is required for the balancer module to work. > > > > I have 500 clients connected and migrating them all to FUSE would be a > > headache. Is there a way around that? Some of my pools have extremely > > high read balance scores and we're also struggling with performance > > issues as a result. > > > > Janek > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > Bauhaus-Universität Weimar > Bauhausstr. 9a, R308 > 99423 Weimar, Germany > > Phone: +49 3643 58 3577 > www.webis.de > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

CSC minutes 2026-02-02
by Yaarit Hatuka 03 Feb '26

03 Feb '26

- [Dan] Ganesha caretaker: what is involved, JoshH (Rubin) may be able to help. - https://github.com/ceph/ceph/pull/66098 <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ceph_ceph_p…> - David G - happy to help with CI/github actions - we need mapping of ceph versions to ganesha versions - current test coverage: qa/suites/fs/nfs/ - [jmundack] - Reminder - CSC Charter Vote!!. - Link to charter - https://docs.google.com/document/d/1Jekr58LYSppznIZOZ3c136WhRYCISfSA9sH_ySJ… <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_docume…> - Check if your email is accurate/up to date at https://docs.google.com/spreadsheets/d/1eQKJnCYUTgfqOlG7x695sWDNw6t422JdyPW… <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_spread…> - Please look for an e-mail with subject "Vote: Approve Ceph Technical Charter" (the link is specific to a voter) - Current status: - total members - 37 - Request to be removed - 1 - emails bounced - 2 - voted - 25! - to vote - 9 more! - Election extended to 2/11 - Q from Greg - Did our existing code of conduct get approved along the way for this process, or are we going to need to deal with that? Section 4(b) - rocky 10 work in progress - most components are testing the new changes - next we need to focus on ubuntu to allow backport to tentacle and squid (to allow upgrade path) - tentacle 20.2.1 - started QA validation - reef's next point release to be reviewed and released this week

1 0

Help recovering OSD: rocksdb: submit_common error: Corruption: block checksum mismatch
by Theo Cabrerizo Diem 03 Feb '26

03 Feb '26

Hello all, I've been using ceph 19.2.2 on alpine in a small cluster: 3 mons + 4 OSDs (12Tb each). Due a planning mistake the OSDs have been running on 2 nodes only, which I was working to get spread to 4 nodes. My luck not being the best, I had a power loss which made 2 OSDs (one on each node) fail to start, which of course led to 23% of the PGs not being available. One of the OSDs (id 0) seems to be gone for good due hardware fault, so I'm putting efforts to fix the other OSD (id 2). When I attempt to use 'ceph-bluestore-tool fsck ...' on 19.2.2 it core dumps almost immediately. So I've tried to use it from the official v20 container. Running "ceph-bluestore-tool fsck --path /var/lib/ceph/osd" (where activate is exposing the osd) outputs: 2026-02-02T18:12:18.895+0000 7f0c543eac00 -1 bluestore(/var/lib/ceph/osd) fsck error: free extent 0x1714c521000~978b26df000 intersects allocatedblocks fsck status: remaining 1 error(s) and warning(s) # And running it with repair instead: 2026-02-02T22:55:53.404+0000 7f8203afdc00 -1 bluestore(/var/lib/ceph/osd) fsck error: free extent 0x1714c521000~978b26df000 intersects allocatedblocks 2026-02-02T22:55:54.799+0000 7f8203afdc00 -1 rocksdb: submit_common error: Corruption: block checksum mismatch: stored = 0, computed = 293241624, type = 4 in db/170024.sst offset 16265037 size 455 code = Rocksdb transaction: MergeCF( prefix = b key = 0x000001714D100000 value size = 16) MergeCF( prefix = b key = 0x000001714D100000 value size = 16) MergeCF( prefix = b key = 0x000001714D100000 value size = 16) <..... several times the same message...> MergeCF( prefix = b key = 0x000001714D100000 value size = 16) MergeCF( prefix = b key = 0x000001714D100000 value size = 16) /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: In function 'unsigned int BlueStoreRepairer::apply(KeyValueDB*)' thread 7f8203afdc00 time 2026-02-02T22:55:55.024481+0000 /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: 19514: FAILED ceph_assert(ok) ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12f) [0x7f8204b6b8f8] 2: ceph-bluestore-tool(+0x19cccd) [0x561c022a7ccd] 3: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x6277) [0x561c02416d37] 4: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x1cf) [0x561c024104bf] 5: main() 6: /lib64/libc.so.6(+0x2a610) [0x7f8203f63610] 7: __libc_start_main() 8: _start() 2026-02-02T22:55:55.047+0000 7f8203afdc00 -1 /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: In function 'unsigned int BlueStoreRepairer::apply(KeyValueDB*)' thread 7f8203afdc00 time 2026-02-02T22:55:55.024481+0000 /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: 19514: FAILED ceph_assert(ok) ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12f) [0x7f8204b6b8f8] 2: ceph-bluestore-tool(+0x19cccd) [0x561c022a7ccd] 3: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x6277) [0x561c02416d37] 4: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x1cf) [0x561c024104bf] 5: main() 6: /lib64/libc.so.6(+0x2a610) [0x7f8203f63610] 7: __libc_start_main() 8: _start() *** Caught signal (Aborted) ** in thread 7f8203afdc00 thread_name:ceph-bluestore- ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) 1: /lib64/libc.so.6(+0x3fc30) [0x7f8203f78c30] 2: /lib64/libc.so.6(+0x8d03c) [0x7f8203fc603c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18e) [0x7f8204b6b957] 6: ceph-bluestore-tool(+0x19cccd) [0x561c022a7ccd] 7: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x6277) [0x561c02416d37] 8: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x1cf) [0x561c024104bf] 9: main() 10: /lib64/libc.so.6(+0x2a610) [0x7f8203f63610] 11: __libc_start_main() 12: _start() 2026-02-02T22:55:55.082+0000 7f8203afdc00 -1 *** Caught signal (Aborted) ** in thread 7f8203afdc00 thread_name:ceph-bluestore- ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) 1: /lib64/libc.so.6(+0x3fc30) [0x7f8203f78c30] 2: /lib64/libc.so.6(+0x8d03c) [0x7f8203fc603c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18e) [0x7f8204b6b957] 6: ceph-bluestore-tool(+0x19cccd) [0x561c022a7ccd] 7: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x6277) [0x561c02416d37] 8: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x1cf) [0x561c024104bf] 9: main() 10: /lib64/libc.so.6(+0x2a610) [0x7f8203f63610] 11: __libc_start_main() 12: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -499> 2026-02-02T22:55:53.404+0000 7f8203afdc00 -1 bluestore(/var/lib/ceph/osd) fsck error: free extent 0x1714c521000~978b26df000 intersects allocated blocks -498> 2026-02-02T22:55:54.799+0000 7f8203afdc00 -1 rocksdb: submit_common error: Corruption: block checksum mismatch: stored = 0, computed = 293241624, type = 4 in db/170024.sst offset 16265037 size 455 code = ☻ Rocksdb transaction: Trying to spin up OSD throws this output: <... snipped for brevity ..> -27> 2026-01-24T01:50:57.764+0000 7eff73af5ed0 0 osd.2 2691 crush map has features 432629308056666112 was 8705, adjusting msgr requires for mons -26> 2026-01-24T01:50:57.764+0000 7eff73af5ed0 0 osd.2 2691 crush map has features 3314933069573799936, adjusting msgr requires for osds -25> 2026-01-24T01:50:57.764+0000 7eff73af5ed0 1 osd.2 2691 check_osdmap_features require_osd_release unknown -> squid -24> 2026-01-24T01:51:02.717+0000 7eff72019b30 5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.133333 -23> 2026-01-24T01:51:02.717+0000 7eff72019b30 5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.0240964 -22> 2026-01-24T01:51:02.717+0000 7eff72019b30 5 bluestore.MempoolThread(0x7eff7397ebe0) _resize_shards cache_size: 6496138035 kv_alloc: 2785017856 kv_used: 6271162 kv_onode_alloc: 503316480 kv_onode_used: 115387532 meta_alloc: 2717908992 meta_used: 57075 data_alloc: 436207616 data_used: 20480 -21> 2026-01-24T01:51:05.010+0000 7eff725b7b30 3 rocksdb: [db/db_impl/db_impl_compaction_flush.cc:3496] Compaction error: Corruption: block checksum mismatch: stored = 0, computed = 293241624, type = 4 in db/170024.sst offset 16265037 size 455 -20> 2026-01-24T01:51:05.010+0000 7eff725b7b30 3 rocksdb: [db/error_handler.cc:397] Background IO error Corruption: block checksum mismatch:stored = 0, computed = 293241624, type = 4 in db/170024.sst offset 16265037 size 455 -19> 2026-01-24T01:51:05.010+0000 7eff725b7b30 4 rocksdb: [db/error_handler.cc:285] ErrorHandler: Set regular background error -18> 2026-01-24T01:51:05.010+0000 7eff725b7b30 4 rocksdb: (Original Log Time 2026/01/24-01:51:05.011167) [db/compaction/compaction_job.cc:865] [default] compacted to: files[8 3 0 0 0 0 0] max score 0.01, MB/sec: 16.4 rd, 2.2 wr, level 1, files in(8, 1) out(1 +0 blob) MB in(0.0, 114.7 +0.0 blob) out(15.5 +0.0 blob), read-write-amplify(13694.2) write-amplify(1631.5) Corruption: block checksum mismatch: stored = 0, computed = 293241624, type = 4 in db/170024.sst offset 16265037 size 455, records in: 20619364, records dropped: 17423737 output_compression: LZ4 -17> 2026-01-24T01:51:05.010+0000 7eff725b7b30 4 rocksdb: (Original Log Time 2026/01/24-01:51:05.011226) EVENT_LOG_v1 {"time_micros": 1769219465011193, "job": 3, "event": "compaction_finished", "compaction_time_micros": 7334898, "compaction_time_cpu_micros": 7009405, "output_level": 1, "num_output_files": 1, "total_output_size": 16264458, "num_input_records": 20619364, "num_output_records": 3195627, "num_subcompactions": 1, "output_compression": "LZ4", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [8, 3, 0, 0, 0, 0, 0]} -16> 2026-01-24T01:51:05.010+0000 7eff725b7b30 2 rocksdb: [db/db_impl/db_impl_compaction_flush.cc:2986] Waiting after background compaction error: Corruption: block checksum mismatch: stored = 0, computed = 293241624, type = 4 in db/170024.sst offset 16265037 size 455,Accumulated background error counts: 1 -15> 2026-01-24T01:51:06.032+0000 7eff725b7b30 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file db/170263.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 -14> 2026-01-24T01:51:06.032+0000 7eff725b7b30 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1769219466032857, "job": 3, "event": "table_file_deletion", "file_number": 170263} -13> 2026-01-24T01:51:07.746+0000 7eff72019b30 5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.117647 -12> 2026-01-24T01:51:07.746+0000 7eff72019b30 5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.0243902 -11> 2026-01-24T01:51:07.746+0000 7eff72019b30 5 bluestore.MempoolThread(0x7eff7397ebe0) _resize_shards cache_size: 6496138035 kv_alloc: 2751463424 kv_used: 3155996 kv_onode_alloc: 570425344 kv_onode_used: 183341998 meta_alloc: 2684354560 meta_used: 57075 data_alloc: 436207616 data_used: 20480 -10> 2026-01-24T01:51:12.768+0000 7eff72019b30 5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.111111 -9> 2026-01-24T01:51:12.768+0000 7eff72019b30 5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.0246914 -8> 2026-01-24T01:51:12.768+0000 7eff72019b30 5 bluestore.MempoolThread(0x7eff7397ebe0) _resize_shards cache_size: 6496138035 kv_alloc: 2717908992 kv_used: 3155996 kv_onode_alloc: 603979776 kv_onode_used: 226502426 meta_alloc: 2650800128 meta_used: 57075 data_alloc: 436207616 data_used: 20480 -7> 2026-01-24T01:51:15.044+0000 7eff73af5ed0 0 osd.2 2691 load_pgs -6> 2026-01-24T01:51:15.108+0000 7eff73af5ed0 5 osd.2 pg_epoch: 2691 pg[6.c(unlocked)] enter Initial -5> 2026-01-24T01:51:15.108+0000 7eff73af5ed0 5 osd.2 pg_epoch: 2691 pg[6.c( empty local-lis/les=2690/2691 n=0 ec=196/142 lis/c=2690/2690 les/c/f=2691/2691/0 sis=2690) [1,3]/[1,3,2] r=2 lpr=0 crt=0'0 mlcod 0'0 unknown mbc={}] exit Initial 0.000424 0 0.000000 -4> 2026-01-24T01:51:15.108+0000 7eff73af5ed0 5 osd.2 pg_epoch: 2691 pg[6.c( empty local-lis/les=2690/2691 n=0 ec=196/142 lis/c=2690/2690 les/c/f=2691/2691/0 sis=2690) [1,3]/[1,3,2] r=2 lpr=0 crt=0'0 mlcod 0'0 unknown mbc={}] enter Reset -3> 2026-01-24T01:51:15.108+0000 7eff72065b30 5 bluestore(/var/lib/ceph/osd) _kv_sync_thread utilization: idle 17.420808792s of 17.420808792s, submitted: 0 -2> 2026-01-24T01:51:15.108+0000 7eff72065b30 -1 rocksdb: submit_common error: Corruption: block checksum mismatch: stored = 0, computed = 293241624, type = 4 in db/170024.sst offset 16265037 size 455 code = Rocksdb transaction: PutCF( prefix = O key = 0x7F800000000000000630000000'!!='0xFFFFFFFFFFFFFFFEFFFFFFFFFFFFFFFF6F value size = 34) PutCF( prefix = S key = 'nid_max' value size = 8) PutCF( prefix = S key = 'blobid_max' value size = 8) -1> 2026-01-24T01:51:15.108+0000 7eff72065b30 -1 /home/buildozer/aports/community/ceph19/src/ceph-19.2.2/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_txc_apply_kv(TransContext*, bool)' thread 7eff72065b30 time 2026-01-24T01:51:15.109433+0000 /home/buildozer/aports/community/ceph19/src/ceph-19.2.2/src/os/bluestore/BlueStore.cc: 14045: FAILED ceph_assert(r == 0) ceph version Development (no_version) squid (stable) 0> 2026-01-24T01:51:15.109+0000 7eff72065b30 -1 *** Caught signal (Aborted) ** in thread 7eff72065b30 thread_name: Any help getting the OSD back running would be greatly appreciated. There's only a handful of folders from cephfs I would like to recover from the system which weren't backed up. The system was far from full (OSDs were 30% full). Of course I can post full logs if needed, just let me know if I should attach them in an email in the mailing list or if preferable to use something like pastebin or another way. Many thanks, Theo

1 0

Re: CephFS “rookfs” marked damaged after cephfs-journal-tool reset on Rook/Kubernetes (Reef 18.2.2)
by Eugen Block 03 Feb '26

03 Feb '26

Hi, you need to fix the stale PGs first. Do they belong to one of the CephFS pools? If that is the case your journal issue most likely roots in those stale PGs. Anth All via ceph-users <ceph-users(a)ceph.io> schrieb am Mo. 2. Feb. 2026 um 19:58: > Hi all, > I’m running a Ceph cluster managed by Rook on Kubernetes, and my CephFS > metadata/journal appears to be in a bad state. I’d like to get advice > before I attempt any destructive metadata repair operations. > Below is a summary of the situation. > *Environment* > > - Orchestrator: Rook/Ceph on Kubernetes (namespace: rook-ceph) > - Ceph version: 18.2.2 (Reef, stable) for mon/mgr/osd/mds > - Filesystem: > - Name: rookfs > - One metadata pool > - One data pool > > > *Cluster health* > ceph status shows: > > - Health: HEALTH_WARN > - Warnings: > - 1 MDS reports slow metadata IOs > - 1 MDS reports slow requests > - Reduced data availability: some PGs in stale state > - A large number of daemons have recently crashed > > MDS section reports: > > - 1/1 MDS daemons up, 1 hot standby > > > CephFS state and MDS status > ceph fs dump for rookfs shows > > - Filesystem rookfs is marked damaged. > - max_mds = 1. > - in set is empty. > - up set is {0=<mds_id>}. > - Flags mention allow_standby_replay. > > > So, the filesystem is marked damaged in the fsmap, while one MDS is still > up:active on rank 0. > ceph tell mds.* status confirms that the MDS for rookfs is: > > - state: up:active > - fs_name: rookfs > - whoami: 0 > > > > I ran the following commands on the CephFS journal: > > 1. Journal reset: > > bash > cephfs-journal-tool --rank=rookfs:0 journal reset > > This completed and indicated a new journal start offset. > 2. Journal inspection: > > bash > cephfs-journal-tool --rank=rookfs:0 journal inspect > > Output : > - Bad entry start ptr (...) at certain offsets > - Overall journal integrity: DAMAGED > - Corrupt regions reported, including a range up to ffffffffffffffff > > So, even after the reset, cephfs-journal-tool reports the journal as > DAMAGED with > corrupt regions. > Listing the metadata pool shows at least the mds_snaptable object, so the > metadata pool is not empty. > > > *Current behaviour* > > - ceph fs status is sometimes very slow or appears to hang. > - Ceph health reports: > - “MDSs report slow metadata IOs” > - “MDSs report slow requests” > - Stale PGs in the cluster > - The filesystem rookfs is marked damaged in ceph fs dump, but the > MDS is still up:active on rank 0. > > Any guidance or best practices for handling this kind of journal corruption > and damaged filesystem in a Rook/Kubernetes setup would be greatly > appreciated, including precautions you would strongly recommend before > running the heavy-repair commands. > Best regards, > Anthony > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

CephFS “rookfs” marked damaged after cephfs-journal-tool reset on Rook/Kubernetes (Reef 18.2.2)
by Anth All 03 Feb '26

03 Feb '26

Hi all, I’m running a Ceph cluster managed by Rook on Kubernetes, and my CephFS metadata/journal appears to be in a bad state. I’d like to get advice before I attempt any destructive metadata repair operations. Below is a summary of the situation. *Environment* - Orchestrator: Rook/Ceph on Kubernetes (namespace: rook-ceph) - Ceph version: 18.2.2 (Reef, stable) for mon/mgr/osd/mds - Filesystem: - Name: rookfs - One metadata pool - One data pool *Cluster health* ceph status shows: - Health: HEALTH_WARN - Warnings: - 1 MDS reports slow metadata IOs - 1 MDS reports slow requests - Reduced data availability: some PGs in stale state - A large number of daemons have recently crashed MDS section reports: - 1/1 MDS daemons up, 1 hot standby CephFS state and MDS status ceph fs dump for rookfs shows - Filesystem rookfs is marked damaged. - max_mds = 1. - in set is empty. - up set is {0=<mds_id>}. - Flags mention allow_standby_replay. So, the filesystem is marked damaged in the fsmap, while one MDS is still up:active on rank 0. ceph tell mds.* status confirms that the MDS for rookfs is: - state: up:active - fs_name: rookfs - whoami: 0 I ran the following commands on the CephFS journal: 1. Journal reset: bash cephfs-journal-tool --rank=rookfs:0 journal reset This completed and indicated a new journal start offset. 2. Journal inspection: bash cephfs-journal-tool --rank=rookfs:0 journal inspect Output : - Bad entry start ptr (...) at certain offsets - Overall journal integrity: DAMAGED - Corrupt regions reported, including a range up to ffffffffffffffff So, even after the reset, cephfs-journal-tool reports the journal as DAMAGED with corrupt regions. Listing the metadata pool shows at least the mds_snaptable object, so the metadata pool is not empty. *Current behaviour* - ceph fs status is sometimes very slow or appears to hang. - Ceph health reports: - “MDSs report slow metadata IOs” - “MDSs report slow requests” - Stale PGs in the cluster - The filesystem rookfs is marked damaged in ceph fs dump, but the MDS is still up:active on rank 0. Any guidance or best practices for handling this kind of journal corruption and damaged filesystem in a Rook/Kubernetes setup would be greatly appreciated, including precautions you would strongly recommend before running the heavy-repair commands. Best regards, Anthony

1 0