- ceph-users - lists.ceph.io

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Eugen Block 03 Feb '26

03 Feb '26

Hi, what does 'ceph osd ok-to-stop <OSD_ID>' show? What do the rgw logs reveal? What's the 'ceph status' during that time? Am Di., 3. Feb. 2026 um 10:18 Uhr schrieb Rok Jaklič via ceph-users < ceph-users(a)ceph.io>: > Hi, > > we currently have 240 OSDs in the cluster, but when I set the reweight of > just one failed OSD to 0 (from 5.45798), radosgw stops working for > some reason. > > If I do "gentle reweight" in steps of 1 for example (from 5.4 to 4.4), it > works ok. > > Any ideas why? > > Running on reef ceph version 18.2.7. > > Kind regards, > Rok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 2

radosgw (s3) stops when setting weight to 0 on one OSD
by Rok Jaklič 03 Feb '26

03 Feb '26

Hi, we currently have 240 OSDs in the cluster, but when I set the reweight of just one failed OSD to 0 (from 5.45798), radosgw stops working for some reason. If I do "gentle reweight" in steps of 1 for example (from 5.4 to 4.4), it works ok. Any ideas why? Running on reef ceph version 18.2.7. Kind regards, Rok

1 0

Re: upmap-read balancer with CephFS kernel clients
by Eugen Block 03 Feb '26

03 Feb '26

Ah, forgot to mention that our compute and control nodes (OpenStack) also mount the CephFS for live migration purposes (Nova) and image to volume conversion (Cinder), and they apparently also report their feature set as "luminous". Am Di., 3. Feb. 2026 um 08:15 Uhr schrieb Eugen Block via ceph-users < ceph-users(a)ceph.io>: > Hi Janek, > > this 'ceph features' output that you're probably referring to (or maybe > 'ceph tell mon.X sessions') comes up every now and then, for example [0], > [1], [2]. I also still don't have a definitive answer, unfortunately. But > from my understanding those are compatibility features that don't > necessarily reflect the actual ceph version. For example, I have a test > cluster with tentacle: > > tentacle:~ # ceph versions > { > "mon": { > "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) > tentacle (stable - RelWithDebInfo)": 1 > }, > "mgr": { > "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) > tentacle (stable - RelWithDebInfo)": 2 > }, > "osd": { > "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) > tentacle (stable - RelWithDebInfo)": 3 > }, > "mds": { > "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) > tentacle (stable - RelWithDebInfo)": 1 > }, > "overall": { > "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) > tentacle (stable - RelWithDebInfo)": 7 > } > } > > but 'ceph features' shows squid: > > tentacle:~ # ceph features > { > "mon": [ > { > "features": "0x3f07fffffffdffff", > "release": "squid", > "num": 1 > } > ], > "mds": [ > { > "features": "0x3f07fffffffdffff", > "release": "squid", > "num": 1 > } > ], > "osd": [ > { > "features": "0x3f07fffffffdffff", > "release": "squid", > "num": 3 > } > ], > ... > > And this cluster also has "luminous" clients connected although I enforced > "squid" with 'ceph osd set-require-min-compat-client squid'. In our > production cluster I haven't enforced that, it is still running with > luminous as min-compat-client because we actually do have a couple of older > clients and don't want to risk anything, but we're going to have to deal > with them at some point. > > Anyway, I'm not recommending to enforce the require-min-compat-client > update, I'd rather suggest to set up a test cluster with a couple of your > typical clients maybe, and then see if they successfully connect. I did the > same in my lab virtual environment that resembles our prod. cluster > (OpenStack with Ceph), there I enforced "reef" as min-compat-client and new > mounts from Ubuntu machines (24.04, kernel 6.8.0) with Ceph 19.2.1 packages > still report luminous on a CephFS mount. But the mount works, so I assume > that everything is okay in the lab. But I'm still not feeling confident > enough to recommend you to update it. ;-) > Hopefully someone else chimes in and contradicts or confirms my > statement(s). > > Regards, > Eugen > > [0] > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/A4IWXINDBM6… > [1] > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RVND3Y3Z7FD… > [2] > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/WSADGYBR7PL… > > Am Mi., 28. Jan. 2026 um 09:16 Uhr schrieb Janek Bevendorff via ceph-users > < > ceph-users(a)ceph.io>: > > > I just noticed that this applies to RBD clients as well. How is this > > supposed to work? Can you use only radosgw and ceph-fuse if you want to > > use upmap-read? It seems strange to me that the most central Ceph > > systems don't work with these new features. > > > > > > On 26/01/2026 18:55, Janek Bevendorff via ceph-users wrote: > > > Hi, > > > > > > Is there a way to use the new upmap-read balancer mode when I have > > > CephFS kernel clients connected? Our cluster runs Squid and all > > > connected clients are on kernel 6.11-6.14, so they should be new > > > enough. However, they identify themselves as luminous (I could not > > > find clear documentation about this, but this seems to be hard-coded > > > and by design), so I cannot run ceph osd set-require-min-compat-client > > > reef, which is required for the balancer module to work. > > > > > > I have 500 clients connected and migrating them all to FUSE would be a > > > headache. Is there a way around that? Some of my pools have extremely > > > high read balance scores and we're also struggling with performance > > > issues as a result. > > > > > > Janek > > > > > > > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > -- > > Bauhaus-Universität Weimar > > Bauhausstr. 9a, R308 > > 99423 Weimar, Germany > > > > Phone: +49 3643 58 3577 > > www.webis.de > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: upmap-read balancer with CephFS kernel clients
by Eugen Block 03 Feb '26

03 Feb '26

Hi Janek, this 'ceph features' output that you're probably referring to (or maybe 'ceph tell mon.X sessions') comes up every now and then, for example [0], [1], [2]. I also still don't have a definitive answer, unfortunately. But from my understanding those are compatibility features that don't necessarily reflect the actual ceph version. For example, I have a test cluster with tentacle: tentacle:~ # ceph versions { "mon": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 1 }, "mgr": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 2 }, "osd": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 3 }, "mds": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 1 }, "overall": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 7 } } but 'ceph features' shows squid: tentacle:~ # ceph features { "mon": [ { "features": "0x3f07fffffffdffff", "release": "squid", "num": 1 } ], "mds": [ { "features": "0x3f07fffffffdffff", "release": "squid", "num": 1 } ], "osd": [ { "features": "0x3f07fffffffdffff", "release": "squid", "num": 3 } ], ... And this cluster also has "luminous" clients connected although I enforced "squid" with 'ceph osd set-require-min-compat-client squid'. In our production cluster I haven't enforced that, it is still running with luminous as min-compat-client because we actually do have a couple of older clients and don't want to risk anything, but we're going to have to deal with them at some point. Anyway, I'm not recommending to enforce the require-min-compat-client update, I'd rather suggest to set up a test cluster with a couple of your typical clients maybe, and then see if they successfully connect. I did the same in my lab virtual environment that resembles our prod. cluster (OpenStack with Ceph), there I enforced "reef" as min-compat-client and new mounts from Ubuntu machines (24.04, kernel 6.8.0) with Ceph 19.2.1 packages still report luminous on a CephFS mount. But the mount works, so I assume that everything is okay in the lab. But I'm still not feeling confident enough to recommend you to update it. ;-) Hopefully someone else chimes in and contradicts or confirms my statement(s). Regards, Eugen [0] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/A4IWXINDBM6… [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RVND3Y3Z7FD… [2] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/WSADGYBR7PL… Am Mi., 28. Jan. 2026 um 09:16 Uhr schrieb Janek Bevendorff via ceph-users < ceph-users(a)ceph.io>: > I just noticed that this applies to RBD clients as well. How is this > supposed to work? Can you use only radosgw and ceph-fuse if you want to > use upmap-read? It seems strange to me that the most central Ceph > systems don't work with these new features. > > > On 26/01/2026 18:55, Janek Bevendorff via ceph-users wrote: > > Hi, > > > > Is there a way to use the new upmap-read balancer mode when I have > > CephFS kernel clients connected? Our cluster runs Squid and all > > connected clients are on kernel 6.11-6.14, so they should be new > > enough. However, they identify themselves as luminous (I could not > > find clear documentation about this, but this seems to be hard-coded > > and by design), so I cannot run ceph osd set-require-min-compat-client > > reef, which is required for the balancer module to work. > > > > I have 500 clients connected and migrating them all to FUSE would be a > > headache. Is there a way around that? Some of my pools have extremely > > high read balance scores and we're also struggling with performance > > issues as a result. > > > > Janek > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > Bauhaus-Universität Weimar > Bauhausstr. 9a, R308 > 99423 Weimar, Germany > > Phone: +49 3643 58 3577 > www.webis.de > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

CSC minutes 2026-02-02
by Yaarit Hatuka 03 Feb '26

03 Feb '26

- [Dan] Ganesha caretaker: what is involved, JoshH (Rubin) may be able to help. - https://github.com/ceph/ceph/pull/66098 <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ceph_ceph_p…> - David G - happy to help with CI/github actions - we need mapping of ceph versions to ganesha versions - current test coverage: qa/suites/fs/nfs/ - [jmundack] - Reminder - CSC Charter Vote!!. - Link to charter - https://docs.google.com/document/d/1Jekr58LYSppznIZOZ3c136WhRYCISfSA9sH_ySJ… <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_docume…> - Check if your email is accurate/up to date at https://docs.google.com/spreadsheets/d/1eQKJnCYUTgfqOlG7x695sWDNw6t422JdyPW… <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_spread…> - Please look for an e-mail with subject "Vote: Approve Ceph Technical Charter" (the link is specific to a voter) - Current status: - total members - 37 - Request to be removed - 1 - emails bounced - 2 - voted - 25! - to vote - 9 more! - Election extended to 2/11 - Q from Greg - Did our existing code of conduct get approved along the way for this process, or are we going to need to deal with that? Section 4(b) - rocky 10 work in progress - most components are testing the new changes - next we need to focus on ubuntu to allow backport to tentacle and squid (to allow upgrade path) - tentacle 20.2.1 - started QA validation - reef's next point release to be reviewed and released this week

1 0

Help recovering OSD: rocksdb: submit_common error: Corruption: block checksum mismatch
by Theo Cabrerizo Diem 02 Feb '26

02 Feb '26

Hello all, I've been using ceph 19.2.2 on alpine in a small cluster: 3 mons + 4 OSDs (12Tb each). Due a planning mistake the OSDs have been running on 2 nodes only, which I was working to get spread to 4 nodes. My luck not being the best, I had a power loss which made 2 OSDs (one on each node) fail to start, which of course led to 23% of the PGs not being available. One of the OSDs (id 0) seems to be gone for good due hardware fault, so I'm putting efforts to fix the other OSD (id 2). When I attempt to use 'ceph-bluestore-tool fsck ...' on 19.2.2 it core dumps almost immediately. So I've tried to use it from the official v20 container. Running "ceph-bluestore-tool fsck --path /var/lib/ceph/osd" (where activate is exposing the osd) outputs: 2026-02-02T18:12:18.895+0000 7f0c543eac00 -1 bluestore(/var/lib/ceph/osd) fsck error: free extent 0x1714c521000~978b26df000 intersects allocatedblocks fsck status: remaining 1 error(s) and warning(s) # And running it with repair instead: 2026-02-02T22:55:53.404+0000 7f8203afdc00 -1 bluestore(/var/lib/ceph/osd) fsck error: free extent 0x1714c521000~978b26df000 intersects allocatedblocks 2026-02-02T22:55:54.799+0000 7f8203afdc00 -1 rocksdb: submit_common error: Corruption: block checksum mismatch: stored = 0, computed = 293241624, type = 4 in db/170024.sst offset 16265037 size 455 code = Rocksdb transaction: MergeCF( prefix = b key = 0x000001714D100000 value size = 16) MergeCF( prefix = b key = 0x000001714D100000 value size = 16) MergeCF( prefix = b key = 0x000001714D100000 value size = 16) <..... several times the same message...> MergeCF( prefix = b key = 0x000001714D100000 value size = 16) MergeCF( prefix = b key = 0x000001714D100000 value size = 16) /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: In function 'unsigned int BlueStoreRepairer::apply(KeyValueDB*)' thread 7f8203afdc00 time 2026-02-02T22:55:55.024481+0000 /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: 19514: FAILED ceph_assert(ok) ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12f) [0x7f8204b6b8f8] 2: ceph-bluestore-tool(+0x19cccd) [0x561c022a7ccd] 3: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x6277) [0x561c02416d37] 4: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x1cf) [0x561c024104bf] 5: main() 6: /lib64/libc.so.6(+0x2a610) [0x7f8203f63610] 7: __libc_start_main() 8: _start() 2026-02-02T22:55:55.047+0000 7f8203afdc00 -1 /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: In function 'unsigned int BlueStoreRepairer::apply(KeyValueDB*)' thread 7f8203afdc00 time 2026-02-02T22:55:55.024481+0000 /ceph/rpmbuild/BUILD/ceph-20.2.0/src/os/bluestore/BlueStore.cc: 19514: FAILED ceph_assert(ok) ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12f) [0x7f8204b6b8f8] 2: ceph-bluestore-tool(+0x19cccd) [0x561c022a7ccd] 3: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x6277) [0x561c02416d37] 4: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x1cf) [0x561c024104bf] 5: main() 6: /lib64/libc.so.6(+0x2a610) [0x7f8203f63610] 7: __libc_start_main() 8: _start() *** Caught signal (Aborted) ** in thread 7f8203afdc00 thread_name:ceph-bluestore- ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) 1: /lib64/libc.so.6(+0x3fc30) [0x7f8203f78c30] 2: /lib64/libc.so.6(+0x8d03c) [0x7f8203fc603c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18e) [0x7f8204b6b957] 6: ceph-bluestore-tool(+0x19cccd) [0x561c022a7ccd] 7: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x6277) [0x561c02416d37] 8: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x1cf) [0x561c024104bf] 9: main() 10: /lib64/libc.so.6(+0x2a610) [0x7f8203f63610] 11: __libc_start_main() 12: _start() 2026-02-02T22:55:55.082+0000 7f8203afdc00 -1 *** Caught signal (Aborted) ** in thread 7f8203afdc00 thread_name:ceph-bluestore- ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) 1: /lib64/libc.so.6(+0x3fc30) [0x7f8203f78c30] 2: /lib64/libc.so.6(+0x8d03c) [0x7f8203fc603c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18e) [0x7f8204b6b957] 6: ceph-bluestore-tool(+0x19cccd) [0x561c022a7ccd] 7: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x6277) [0x561c02416d37] 8: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x1cf) [0x561c024104bf] 9: main() 10: /lib64/libc.so.6(+0x2a610) [0x7f8203f63610] 11: __libc_start_main() 12: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -499> 2026-02-02T22:55:53.404+0000 7f8203afdc00 -1 bluestore(/var/lib/ceph/osd) fsck error: free extent 0x1714c521000~978b26df000 intersects allocated blocks -498> 2026-02-02T22:55:54.799+0000 7f8203afdc00 -1 rocksdb: submit_common error: Corruption: block checksum mismatch: stored = 0, computed = 293241624, type = 4 in db/170024.sst offset 16265037 size 455 code = ☻ Rocksdb transaction: Trying to spin up OSD throws this output: <... snipped for brevity ..> -27> 2026-01-24T01:50:57.764+0000 7eff73af5ed0 0 osd.2 2691 crush map has features 432629308056666112 was 8705, adjusting msgr requires for mons -26> 2026-01-24T01:50:57.764+0000 7eff73af5ed0 0 osd.2 2691 crush map has features 3314933069573799936, adjusting msgr requires for osds -25> 2026-01-24T01:50:57.764+0000 7eff73af5ed0 1 osd.2 2691 check_osdmap_features require_osd_release unknown -> squid -24> 2026-01-24T01:51:02.717+0000 7eff72019b30 5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.133333 -23> 2026-01-24T01:51:02.717+0000 7eff72019b30 5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.0240964 -22> 2026-01-24T01:51:02.717+0000 7eff72019b30 5 bluestore.MempoolThread(0x7eff7397ebe0) _resize_shards cache_size: 6496138035 kv_alloc: 2785017856 kv_used: 6271162 kv_onode_alloc: 503316480 kv_onode_used: 115387532 meta_alloc: 2717908992 meta_used: 57075 data_alloc: 436207616 data_used: 20480 -21> 2026-01-24T01:51:05.010+0000 7eff725b7b30 3 rocksdb: [db/db_impl/db_impl_compaction_flush.cc:3496] Compaction error: Corruption: block checksum mismatch: stored = 0, computed = 293241624, type = 4 in db/170024.sst offset 16265037 size 455 -20> 2026-01-24T01:51:05.010+0000 7eff725b7b30 3 rocksdb: [db/error_handler.cc:397] Background IO error Corruption: block checksum mismatch:stored = 0, computed = 293241624, type = 4 in db/170024.sst offset 16265037 size 455 -19> 2026-01-24T01:51:05.010+0000 7eff725b7b30 4 rocksdb: [db/error_handler.cc:285] ErrorHandler: Set regular background error -18> 2026-01-24T01:51:05.010+0000 7eff725b7b30 4 rocksdb: (Original Log Time 2026/01/24-01:51:05.011167) [db/compaction/compaction_job.cc:865] [default] compacted to: files[8 3 0 0 0 0 0] max score 0.01, MB/sec: 16.4 rd, 2.2 wr, level 1, files in(8, 1) out(1 +0 blob) MB in(0.0, 114.7 +0.0 blob) out(15.5 +0.0 blob), read-write-amplify(13694.2) write-amplify(1631.5) Corruption: block checksum mismatch: stored = 0, computed = 293241624, type = 4 in db/170024.sst offset 16265037 size 455, records in: 20619364, records dropped: 17423737 output_compression: LZ4 -17> 2026-01-24T01:51:05.010+0000 7eff725b7b30 4 rocksdb: (Original Log Time 2026/01/24-01:51:05.011226) EVENT_LOG_v1 {"time_micros": 1769219465011193, "job": 3, "event": "compaction_finished", "compaction_time_micros": 7334898, "compaction_time_cpu_micros": 7009405, "output_level": 1, "num_output_files": 1, "total_output_size": 16264458, "num_input_records": 20619364, "num_output_records": 3195627, "num_subcompactions": 1, "output_compression": "LZ4", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [8, 3, 0, 0, 0, 0, 0]} -16> 2026-01-24T01:51:05.010+0000 7eff725b7b30 2 rocksdb: [db/db_impl/db_impl_compaction_flush.cc:2986] Waiting after background compaction error: Corruption: block checksum mismatch: stored = 0, computed = 293241624, type = 4 in db/170024.sst offset 16265037 size 455,Accumulated background error counts: 1 -15> 2026-01-24T01:51:06.032+0000 7eff725b7b30 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file db/170263.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 -14> 2026-01-24T01:51:06.032+0000 7eff725b7b30 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1769219466032857, "job": 3, "event": "table_file_deletion", "file_number": 170263} -13> 2026-01-24T01:51:07.746+0000 7eff72019b30 5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.117647 -12> 2026-01-24T01:51:07.746+0000 7eff72019b30 5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.0243902 -11> 2026-01-24T01:51:07.746+0000 7eff72019b30 5 bluestore.MempoolThread(0x7eff7397ebe0) _resize_shards cache_size: 6496138035 kv_alloc: 2751463424 kv_used: 3155996 kv_onode_alloc: 570425344 kv_onode_used: 183341998 meta_alloc: 2684354560 meta_used: 57075 data_alloc: 436207616 data_used: 20480 -10> 2026-01-24T01:51:12.768+0000 7eff72019b30 5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.111111 -9> 2026-01-24T01:51:12.768+0000 7eff72019b30 5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.0246914 -8> 2026-01-24T01:51:12.768+0000 7eff72019b30 5 bluestore.MempoolThread(0x7eff7397ebe0) _resize_shards cache_size: 6496138035 kv_alloc: 2717908992 kv_used: 3155996 kv_onode_alloc: 603979776 kv_onode_used: 226502426 meta_alloc: 2650800128 meta_used: 57075 data_alloc: 436207616 data_used: 20480 -7> 2026-01-24T01:51:15.044+0000 7eff73af5ed0 0 osd.2 2691 load_pgs -6> 2026-01-24T01:51:15.108+0000 7eff73af5ed0 5 osd.2 pg_epoch: 2691 pg[6.c(unlocked)] enter Initial -5> 2026-01-24T01:51:15.108+0000 7eff73af5ed0 5 osd.2 pg_epoch: 2691 pg[6.c( empty local-lis/les=2690/2691 n=0 ec=196/142 lis/c=2690/2690 les/c/f=2691/2691/0 sis=2690) [1,3]/[1,3,2] r=2 lpr=0 crt=0'0 mlcod 0'0 unknown mbc={}] exit Initial 0.000424 0 0.000000 -4> 2026-01-24T01:51:15.108+0000 7eff73af5ed0 5 osd.2 pg_epoch: 2691 pg[6.c( empty local-lis/les=2690/2691 n=0 ec=196/142 lis/c=2690/2690 les/c/f=2691/2691/0 sis=2690) [1,3]/[1,3,2] r=2 lpr=0 crt=0'0 mlcod 0'0 unknown mbc={}] enter Reset -3> 2026-01-24T01:51:15.108+0000 7eff72065b30 5 bluestore(/var/lib/ceph/osd) _kv_sync_thread utilization: idle 17.420808792s of 17.420808792s, submitted: 0 -2> 2026-01-24T01:51:15.108+0000 7eff72065b30 -1 rocksdb: submit_common error: Corruption: block checksum mismatch: stored = 0, computed = 293241624, type = 4 in db/170024.sst offset 16265037 size 455 code = Rocksdb transaction: PutCF( prefix = O key = 0x7F800000000000000630000000'!!='0xFFFFFFFFFFFFFFFEFFFFFFFFFFFFFFFF6F value size = 34) PutCF( prefix = S key = 'nid_max' value size = 8) PutCF( prefix = S key = 'blobid_max' value size = 8) -1> 2026-01-24T01:51:15.108+0000 7eff72065b30 -1 /home/buildozer/aports/community/ceph19/src/ceph-19.2.2/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_txc_apply_kv(TransContext*, bool)' thread 7eff72065b30 time 2026-01-24T01:51:15.109433+0000 /home/buildozer/aports/community/ceph19/src/ceph-19.2.2/src/os/bluestore/BlueStore.cc: 14045: FAILED ceph_assert(r == 0) ceph version Development (no_version) squid (stable) 0> 2026-01-24T01:51:15.109+0000 7eff72065b30 -1 *** Caught signal (Aborted) ** in thread 7eff72065b30 thread_name: Any help getting the OSD back running would be greatly appreciated. There's only a handful of folders from cephfs I would like to recover from the system which weren't backed up. The system was far from full (OSDs were 30% full). Of course I can post full logs if needed, just let me know if I should attach them in an email in the mailing list or if preferable to use something like pastebin or another way. Many thanks, Theo

1 0

Re: CephFS “rookfs” marked damaged after cephfs-journal-tool reset on Rook/Kubernetes (Reef 18.2.2)
by Eugen Block 02 Feb '26

02 Feb '26

Hi, you need to fix the stale PGs first. Do they belong to one of the CephFS pools? If that is the case your journal issue most likely roots in those stale PGs. Anth All via ceph-users <ceph-users(a)ceph.io> schrieb am Mo. 2. Feb. 2026 um 19:58: > Hi all, > I’m running a Ceph cluster managed by Rook on Kubernetes, and my CephFS > metadata/journal appears to be in a bad state. I’d like to get advice > before I attempt any destructive metadata repair operations. > Below is a summary of the situation. > *Environment* > > - Orchestrator: Rook/Ceph on Kubernetes (namespace: rook-ceph) > - Ceph version: 18.2.2 (Reef, stable) for mon/mgr/osd/mds > - Filesystem: > - Name: rookfs > - One metadata pool > - One data pool > > > *Cluster health* > ceph status shows: > > - Health: HEALTH_WARN > - Warnings: > - 1 MDS reports slow metadata IOs > - 1 MDS reports slow requests > - Reduced data availability: some PGs in stale state > - A large number of daemons have recently crashed > > MDS section reports: > > - 1/1 MDS daemons up, 1 hot standby > > > CephFS state and MDS status > ceph fs dump for rookfs shows > > - Filesystem rookfs is marked damaged. > - max_mds = 1. > - in set is empty. > - up set is {0=<mds_id>}. > - Flags mention allow_standby_replay. > > > So, the filesystem is marked damaged in the fsmap, while one MDS is still > up:active on rank 0. > ceph tell mds.* status confirms that the MDS for rookfs is: > > - state: up:active > - fs_name: rookfs > - whoami: 0 > > > > I ran the following commands on the CephFS journal: > > 1. Journal reset: > > bash > cephfs-journal-tool --rank=rookfs:0 journal reset > > This completed and indicated a new journal start offset. > 2. Journal inspection: > > bash > cephfs-journal-tool --rank=rookfs:0 journal inspect > > Output : > - Bad entry start ptr (...) at certain offsets > - Overall journal integrity: DAMAGED > - Corrupt regions reported, including a range up to ffffffffffffffff > > So, even after the reset, cephfs-journal-tool reports the journal as > DAMAGED with > corrupt regions. > Listing the metadata pool shows at least the mds_snaptable object, so the > metadata pool is not empty. > > > *Current behaviour* > > - ceph fs status is sometimes very slow or appears to hang. > - Ceph health reports: > - “MDSs report slow metadata IOs” > - “MDSs report slow requests” > - Stale PGs in the cluster > - The filesystem rookfs is marked damaged in ceph fs dump, but the > MDS is still up:active on rank 0. > > Any guidance or best practices for handling this kind of journal corruption > and damaged filesystem in a Rook/Kubernetes setup would be greatly > appreciated, including precautions you would strongly recommend before > running the heavy-repair commands. > Best regards, > Anthony > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

CephFS “rookfs” marked damaged after cephfs-journal-tool reset on Rook/Kubernetes (Reef 18.2.2)
by Anth All 02 Feb '26

02 Feb '26

Hi all, I’m running a Ceph cluster managed by Rook on Kubernetes, and my CephFS metadata/journal appears to be in a bad state. I’d like to get advice before I attempt any destructive metadata repair operations. Below is a summary of the situation. *Environment* - Orchestrator: Rook/Ceph on Kubernetes (namespace: rook-ceph) - Ceph version: 18.2.2 (Reef, stable) for mon/mgr/osd/mds - Filesystem: - Name: rookfs - One metadata pool - One data pool *Cluster health* ceph status shows: - Health: HEALTH_WARN - Warnings: - 1 MDS reports slow metadata IOs - 1 MDS reports slow requests - Reduced data availability: some PGs in stale state - A large number of daemons have recently crashed MDS section reports: - 1/1 MDS daemons up, 1 hot standby CephFS state and MDS status ceph fs dump for rookfs shows - Filesystem rookfs is marked damaged. - max_mds = 1. - in set is empty. - up set is {0=<mds_id>}. - Flags mention allow_standby_replay. So, the filesystem is marked damaged in the fsmap, while one MDS is still up:active on rank 0. ceph tell mds.* status confirms that the MDS for rookfs is: - state: up:active - fs_name: rookfs - whoami: 0 I ran the following commands on the CephFS journal: 1. Journal reset: bash cephfs-journal-tool --rank=rookfs:0 journal reset This completed and indicated a new journal start offset. 2. Journal inspection: bash cephfs-journal-tool --rank=rookfs:0 journal inspect Output : - Bad entry start ptr (...) at certain offsets - Overall journal integrity: DAMAGED - Corrupt regions reported, including a range up to ffffffffffffffff So, even after the reset, cephfs-journal-tool reports the journal as DAMAGED with corrupt regions. Listing the metadata pool shows at least the mds_snaptable object, so the metadata pool is not empty. *Current behaviour* - ceph fs status is sometimes very slow or appears to hang. - Ceph health reports: - “MDSs report slow metadata IOs” - “MDSs report slow requests” - Stale PGs in the cluster - The filesystem rookfs is marked damaged in ceph fs dump, but the MDS is still up:active on rank 0. Any guidance or best practices for handling this kind of journal corruption and damaged filesystem in a Rook/Kubernetes setup would be greatly appreciated, including precautions you would strongly recommend before running the heavy-repair commands. Best regards, Anthony

1 0

Re: [ceph-user][ceph-ansible] Quincy: Module 'devicehealth' has failed: unknown operation
by Frédéric Nass 02 Feb '26

02 Feb '26

Hi Danish, You seem to be hitting this bug [1] that was fixed in Quincy v17.2.8 by [2]. If you're using packages (.rpm,.deb), you could upgrade to v17.2.9 that has the fix. Do not use v17.2.8 due to a critical BlueStore bug. If you're using cephadm and containers, you'll have to upgrade to Reef as v17.2.9 had no container images. Best regards, Frédéric. [1] https://tracker.ceph.com/issues/55606 [2] https://github.com/ceph/ceph/pull/52461 -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Squishing Squids - A Ceph Compression Guide <https://www.eventbrite.com/e/squishing-squids-a-ceph-compression-guide-tick…>, February 25th, 9am PST. https://clyso.com | frederic.nass(a)clyso.com Le sam. 31 janv. 2026 à 09:09, Danish Khan via ceph-users < ceph-users(a)ceph.io> a écrit : > Dear Team, > > I am keep getting below error after every few days: > > Error : > Module 'devicehealth' has failed: unknown operation > > Actual error from MGR log: > > 2026-01-30T05:50:59.338+0530 7f314da26640 0 [devicehealth ERROR root] > Caught fatal database error: unknown operation > 2026-01-30T05:50:59.338+0530 7f314da26640 -1 log_channel(cluster) log [ERR] > : Unhandled exception from module 'devicehealth' while running on > mgr.controller3: unknown operation > 2026-01-30T05:50:59.338+0530 7f314da26640 -1 devicehealth.serve: > 2026-01-30T05:50:59.338+0530 7f314da26640 -1 Traceback (most recent call > last): > File "/usr/share/ceph/mgr/devicehealth/module.py", line 408, in serve > self._do_serve() > File "/usr/share/ceph/mgr/mgr_module.py", line 513, in check > return func(self, *args, **kwargs) > File "/usr/share/ceph/mgr/devicehealth/module.py", line 399, in _do_serve > self.scrape_all() > File "/usr/share/ceph/mgr/devicehealth/module.py", line 449, in > scrape_all > self.put_device_metrics(device, data) > File "/usr/share/ceph/mgr/devicehealth/module.py", line 525, in > put_device_metrics > self._create_device(devid) > File "/usr/share/ceph/mgr/devicehealth/module.py", line 511, in > _create_device > cursor = self.db.execute(SQL, (devid,)) > sqlite3.InternalError: unknown operation > > 2026-01-30T05:50:59.366+0530 7f3155335640 0 log_channel(cluster) log [DBG] > : pgmap v116150: 6977 pgs: 92 peering, 40 stale+active+clean, 191 > active+clean+laggy, 30 active+undersized, 1026 active+undersized+degraded, > 5598 active+clean; 48 TiB data, 153 TiB used, 1.2 PiB / 1.4 PiB avail; 44 > MiB/s wr, 1.00k op/s; 8043806/154077270 objects degraded (5.221%); 4.4 > GiB/s, 8 keys/s, 530 objects/s recovering > > And I get one warning for mgr daemon crash but all MGR Service > actually doesn't get failed or restarted automatically. > > I guess it is something related to sqlite3 which checks ceph > health internally. > > Although this is not causing any issue and error disappears after > restarting ceph-mgr service or doing a failover of MGR to another > controller. > > Does anyone know how we can resolve this issue permanently? > > Similar issue: > https://github.com/rook/rook/issues/12349 > > Regards, > Danish > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Experiences with Tentacle 20.2.0
by Chris Palmer 02 Feb '26

02 Feb '26

Thanks Wannes. Those are the caveats around EC optimisation that I alluded to below. Since we don't have any EC on this cluster that should not be relevant... Regards, Chris On 02/02/2026 09:30, Wannes Smet via ceph-users wrote: > This story jumps to mind: https://docs.clyso.com/blog/2026/01/13/tentacle-v20.2.0-ec-bug/ > ________________________________ > From: Chris Palmer via ceph-users <ceph-users(a)ceph.io> > Sent: Monday, February 2, 2026 10:16 > To: ceph-users <ceph-users(a)ceph.io> > Subject: [ceph-users] Experiences with Tentacle 20.2.0 > > I'd like to know whether anyone has any good or bad experiences with Tentacle 20. 2. 0 in a production environment. I have the opportunity to upgrade a small production cluster from 19. 2. 3 to 20. 2. 0. It uses Cephadm/RBD/CephFS/RGW with R3 only > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > I'd like to know whether anyone has any good or bad experiences with > Tentacle 20.2.0 in a production environment. > > I have the opportunity to upgrade a small production cluster from 19.2.3 > to 20.2.0. It uses Cephadm/RBD/CephFS/RGW with R3 only (no EC, so the > known caveats around EC optimisations won't be relevant). > > My experiences trying it in a lab test environment have been positive. > But it would be good to know if anyone is encountering any problems with > it in real life. > > Thanks, Chris > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0