- ceph-users - lists.ceph.io

Re: Help needed: multiple OSDs refuse to restart (and pg in down state)
by Massimo Sgaravatto 20 Sep '26

20 Sep '26

Hi Anthony, all I am running ceph squid 19.2.3 I copied the relevant part of the log (from the start till the crash) for a OSD (osd.206) in https://cernbox.cern.ch/s/56sdGTQsotbN2XE So as far as I can understand it crashes because of: 2026-09-16T21:16:06.859+0200 7f5431ee1640 4 rocksdb: [db/compaction/compaction_job.cc:1588] [O-2] [JOB 9] Generated table #148643: 151504 keys, 67943642 bytes, temperature: kUnknown 2026-09-16T21:16:06.859+0200 7f5431ee1640 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1789586166860678, "cf_name": "O-2", "job": 9, "event": "table_file_creation", "file_number": 148643, "file_size": 67943642, "file_checksum": "", "f\ ile_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 67111852, "index_size": 1538071, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_v\ alue_is_delta_encoded": 1, "filter_size": 378821, "raw_key_size": 12956410, "raw_average_key_size": 85, "raw_value_size": 67640956, "raw_average_value_size": 446, "num_data_blocks": 18954, "num_entries": 151504, "num_filter_entrie\ s": 151504, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "O-2", "column_family_id": 9, "comparator": "leveldb\ .BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "LZ4", "compression_options": "window_bits=-14; level=32767; strategy=0; \ max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1778598719, "oldest_key_time": 0, "file_creation_time": 1789586166, "slow_compression_estimated_data_size":\ 0, "fast_compression_estimated_data_size": 0, "db_id": "1103608a-90ce-4939-82c0-de96ff245167", "db_session_id": "FO9J42Q67M9AK232DQXE", "orig_file_number": 148643, "seqno_to_time_mapping": "N/A"}} 2026-09-16T21:16:07.421+0200 7f541fcaf640 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.3/rpm/el9/BUILD/ceph-19.2.3/src/o\ s/bluestore/bluestore_types.cc: In function 'bool bluestore_blob_use_tracker_t::put(uint32_t, uint32_t, PExtentVector*)' thread 7f541fcaf640 time 2026-09-16T21:16:07.418902+0200 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.3/rpm/el9/BUILD/ceph-19.2.3/src/os/bluestore/bluestore_types.cc: 511: FAILED c\ eph_assert(diff <= bytes_per_au[pos]) ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x113) [0x560cc2d4c85d] 2: /usr/bin/ceph-osd(+0x401a14) [0x560cc2d4ca14] 3: /usr/bin/ceph-osd(+0x3f2930) [0x560cc2d3d930] 4: (BlueStore::Blob::put_ref(BlueStore::Collection*, unsigned int, unsigned int, std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)5, bluestore_pextent_t> >*)+0xaa) [0x560cc32bb2fa] 5: (BlueStore::OldExtent::create(boost::intrusive_ptr<BlueStore::Collection>, unsigned int, unsigned int, unsigned int, boost::intrusive_ptr<BlueStore::Blob>&)+0x11d) [0x560cc32bf47d] 6: (BlueStore::ExtentMap::punch_hole(boost::intrusive_ptr<BlueStore::Collection>&, unsigned long, unsigned long, boost::intrusive::list<BlueStore::OldExtent, boost::intrusive::member_hook<BlueStore::OldExtent, boost::intrusive::l\ ist_member_hook<>, &BlueStore::OldExtent::old_extent_item> >*)+0x451) [0x560cc32cbe21] 7: (BlueStore::_do_truncate(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&, unsigned long, std::set<BlueStore::SharedBlob*, std::less<BlueStore::SharedBlob*>, std::\ allocator<BlueStore::SharedBlob*> >*)+0x205) [0x560cc33483f5] 8: (BlueStore::_do_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&)+0xc1) [0x560cc334d781] 9: (BlueStore::_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&)+0x7f) [0x560cc334f0af] 10: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x12a6) [0x560cc3334206] 11: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x2f3) [0\ x560cc3335123] 12: /usr/bin/ceph-osd(+0x5239a8) [0x560cc2e6e9a8] 13: (OSD::dispatch_context(PeeringCtx&, PG*, std::shared_ptr<OSDMap const>, ThreadPool::TPHandle*)+0x115) [0x560cc2edf495] 14: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x2bc) [0x560cc2eea72c] 15: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x560cc31359b1] 16: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xcd0) [0x560cc2f04b70] 17: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x560cc34238aa] 18: /usr/bin/ceph-osd(+0xad8e64) [0x560cc3423e64] 19: /lib64/libc.so.6(+0x8b2ea) [0x7f543ce8b2ea] 20: /lib64/libc.so.6(+0x1103d0) [0x7f543cf103d0] 2026-09-16T21:16:07.426+0200 7f541fcaf640 -1 *** Caught signal (Aborted) ** in thread 7f541fcaf640 thread_name:tp_osd_tp ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) 1: /lib64/libc.so.6(+0x3fc30) [0x7f543ce3fc30] 2: /lib64/libc.so.6(+0x8d02c) [0x7f543ce8d02c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x169) [0x560cc2d4c8b3] 6: /usr/bin/ceph-osd(+0x401a14) [0x560cc2d4ca14] 7: /usr/bin/ceph-osd(+0x3f2930) [0x560cc2d3d930] 8: (BlueStore::Blob::put_ref(BlueStore::Collection*, unsigned int, unsigned int, std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)5, bluestore_pextent_t> >*)+0xaa) [0x560cc32bb2fa] 9: (BlueStore::OldExtent::create(boost::intrusive_ptr<BlueStore::Collection>, unsigned int, unsigned int, unsigned int, boost::intrusive_ptr<BlueStore::Blob>&)+0x11d) [0x560cc32bf47d] 10: (BlueStore::ExtentMap::punch_hole(boost::intrusive_ptr<BlueStore::Collection>&, unsigned long, unsigned long, boost::intrusive::list<BlueStore::OldExtent, boost::intrusive::member_hook<BlueStore::OldExtent, boost::intrusive::\ list_member_hook<>, &BlueStore::OldExtent::old_extent_item> >*)+0x451) [0x560cc32cbe21] 11: (BlueStore::_do_truncate(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&, unsigned long, std::set<BlueStore::SharedBlob*, std::less<BlueStore::SharedBlob*>, std:\ :allocator<BlueStore::SharedBlob*> >*)+0x205) [0x560cc33483f5] 12: (BlueStore::_do_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&)+0xc1) [0x560cc334d781] 13: (BlueStore::_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&)+0x7f) [0x560cc334f0af] 14: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x12a6) [0x560cc3334206] 15: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x2f3) [0\ x560cc3335123] 16: /usr/bin/ceph-osd(+0x5239a8) [0x560cc2e6e9a8] 17: (OSD::dispatch_context(PeeringCtx&, PG*, std::shared_ptr<OSDMap const>, ThreadPool::TPHandle*)+0x115) [0x560cc2edf495] 18: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x2bc) [0x560cc2eea72c] 19: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x560cc31359b1] 20: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xcd0) [0x560cc2f04b70] 21: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x560cc34238aa] 22: /usr/bin/ceph-osd(+0xad8e64) [0x560cc3423e64] 23: /lib64/libc.so.6(+0x8b2ea) [0x7f543ce8b2ea] 24: /lib64/libc.so.6(+0x1103d0) [0x7f543cf103d0] Thanks, Massimo On Thu, Sep 17, 2026 at 4:44 AM Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > Start with what they log when they try to start. > > And what release you’re running. > > > On Sep 16, 2026, at 4:47 PM, Massimo Sgaravatto <ceph-users(a)ceph.io> > wrote: > > > > Dear all > > Today we had a problem in our ceph cluster: basically for some (still > > unknown) reasons, 5 OSDs (206, 233, 250, 257, 277) crashed and now they > > refuse to start. > > > > > > The main problem is that 1 pg is in down+remapped state because it is > > waiting for 3 of these OSDs [*] > > > > > > Doing a ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-xxx --op fsck, > I > > see [**] that the problem is with 2 objects > > (rbd_data.44.f397cc4c21bbfb.000000000000319b > > and rbd_data.44.f397cc4c21bbfb.0000000000000e66 > > A "--op repair" is not able to fix the problem > > > > > > I am not sure how to proceed now > > > > Is the only option removing the problematic object from each OSD, doing > > e.g.: > > > > > > ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-206 --pgid > 43.1b0 > > > '{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":7151\ > > 31231,"max":0,"pool":43,"namespace":"","shardid":3}' remove > > > > ? > > > > Then the OSDs should hopefully restart, right ? > > > > Or are there better options ? > > > > Your help will be really appreciated ! > > > > > > Thanks, Massimo > > > > [*] > > ceph pg 43.1b0 query > > ... > > ... > > > > "blocked": "peering is blocked due to down osds", > > "down_osds_we_would_probe": [ > > 206, > > 233, > > 250 > > ], > > "peering_blocked_by": [ > > { > > "osd": 206, > > "current_lost_at": 0, > > "comment": "starting or marking this osd lost may let > > us proceed" > > }, > > { > > "osd": 233, > > "current_lost_at": 0, > > "comment": "starting or marking this osd lost may let > > us proceed" > > }, > > { > > "osd": 250, > > "current_lost_at": 0, > > "comment": "starting or marking this osd lost may let > > us proceed" > > } > > ] > > > > [**] > > > > [root@ceph-osd-17 ~]# ceph-bluestore-tool --path > /var/lib/ceph/osd/ceph-206 > > --op fsck > > 2026-09-16T21:26:23.609+0200 7fb5aa3ddac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-206) fsck error: > > 3#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a lextent at 0x6c000~4000 spans a shard boundary > > 2026-09-16T21:26:23.609+0200 7fb5aa3ddac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-206) fsck error: > > 3#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a lextent at 0x6d000 overlaps with the previous, which ends at > 0x70000 > > 2026-09-16T21:26:23.609+0200 7fb5aa3ddac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-206) fsck error: > > 3#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a blob Blob(0x5605e5760b60 spanning 1 > > > blob([!~4000,0xa211d395000~1000,!~3000,0x23b97906000~1000,0x23b97226000~3000] > > llen=0xc000 csum+shared crc32c/0x1000/48\ > > ) use_tracker(0xc*0x1000 0x[0,0,0,0,1000,0,0,0,1000,1000,1000,1000]) > > SharedBlob(0x5606c99ebfc0 sbid 0x37ae23b)) doesn't match expected ref_map > > use_tracker(0xc*0x\ > > 1000 0x[0,0,0,0,1000,0,0,0,1000,2000,2000,2000]) > > fsck status: remaining 3 error(s) and warning(s) > > > > root@ceph-osd-17 ~]# ceph-objectstore-tool --data-path > > /var/lib/ceph/osd/ceph-206 --op list | grep > > "rbd_data.44.f397cc4c21bbfb.000000000000319b" > > > ["43.1b0s3",{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":4210196912,"max":0,"pool":43,"namespace":"","generation":44630682,"s\ > > hard_id":3,"max":0}] > > > > > > ---------------------------------------- > > ---------------------------------------- > > > > [root@ceph-osd-18 ~]# ceph-bluestore-tool --path > /var/lib/ceph/osd/ceph-233 > > --op fsck > > 2026-09-16T21:37:09.598+0200 7f8ee4c49ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-233) fsck error: > > 4#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a lextent at 0x6c000~4000 spans a shard boundary > > 2026-09-16T21:37:09.598+0200 7f8ee4c49ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-233) fsck error: > > 4#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a lextent at 0x6d000 overlaps with the previous, which ends at > 0x70000 > > 2026-09-16T21:37:09.598+0200 7f8ee4c49ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-233) fsck error: > > 4#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a blob Blob(0x5557c8bef2b0 spanning 1 > > > blob([!~4000,0x2ce88f4c000~1000,!~3000,0x2ce89560000~1000,0x2ce88a80000~3000] > > llen=0xc000 csum+shared crc32c/0x1000/48\ > > ) use_tracker(0xc*0x1000 0x[0,0,0,0,1000,0,0,0,1000,1000,1000,1000]) > > SharedBlob(0x5557fff8b1a0 sbid 0x323170b)) doesn't match expected ref_map > > use_tracker(0xc*0x\ > > 1000 0x[0,0,0,0,1000,0,0,0,1000,2000,2000,2000]) > > fsck status: remaining 3 error(s) and warning(s) > > > > [root@ceph-osd-18 ~]# ceph-objectstore-tool --data-path > > /var/lib/ceph/osd/ceph-233 --op list | grep > > "rbd_data.44.f397cc4c21bbfb.000000000000319b" > > > ["43.1b0s4",{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":4210196912,"max":0,"pool":43,"namespace":"","generation":44630682,"s\ > > hard_id":4,"max":0}] > > [root@ceph-osd-18 ~]# > > > > > > ---------------------------------------- > > > > > > [root@ceph-osd-19 ~]# ceph-bluestore-tool --path > /var/lib/ceph/osd/ceph-250 > > --op fsck > > 2026-09-16T21:59:02.401+0200 7fa2e8795ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-250) fsck error: > > 0#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a lextent at 0x6c000~4000 spans a shard boundary > > 2026-09-16T21:59:02.401+0200 7fa2e8795ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-250) fsck error: > > 0#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a lextent at 0x6d000 overlaps with the previous, which ends at > 0x70000 > > 2026-09-16T21:59:02.401+0200 7fa2e8795ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-250) fsck error: > > 0#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a blob Blob(0x556a29cc4820 spanning 1 > > > blob([!~4000,0x464866d9000~1000,!~3000,0x4648731f000~1000,0x46485e09000~3000] > > llen=0xc000 csum+shared crc32c/0x1000/48\ > > ) use_tracker(0xc*0x1000 0x[0,0,0,0,1000,0,0,0,1000,1000,1000,1000]) > > SharedBlob(0x556a1fb84e40 sbid 0x98f249)) doesn't match expected ref_map > > use_tracker(0xc*0x1\ > > 000 0x[0,0,0,0,1000,0,0,0,1000,2000,2000,2000]) > > fsck status: remaining 3 error(s) and warning(s) > > > > [root@ceph-osd-19 ~]# ceph-objectstore-tool --data-path > > /var/lib/ceph/osd/ceph-250 --op list | grep > > "rbd_data.44.f397cc4c21bbfb.000000000000319b" > > > ["43.1b0s0",{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":4210196912,"max":0,"pool":43,"namespace":"","generation":44630682,"s\ > > hard_id":0,"max":0}] > > ---------------------------------------- > > > > [root@ceph-osd-19 ~]# ceph-bluestore-tool --path > /var/lib/ceph/osd/ceph-257 > > --op fsck > > 2026-09-16T22:06:09.627+0200 7fded8db6ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-257) fsck error: > > 0#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ > > dbe3c7 lextent at 0x6a000~6000 spans a shard boundary > > 2026-09-16T22:06:09.627+0200 7fded8db6ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-257) fsck error: > > 0#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ > > dbe3c7 lextent at 0x6e000 overlaps with the previous, which ends at > 0x70000 > > 2026-09-16T22:06:09.627+0200 7fded8db6ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-257) fsck error: > > 0#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ > > dbe3c7 blob Blob(0x561e442541a0 spanning 2 > > blob([!~4000,0x19c87d9b000~1000,0x46e6508e000~3000,0x46e64693000~2000] > > llen=0xa000 csum+shared crc32c/0x1000/40) use_t\ > > racker(0xa*0x1000 0x[0,0,0,0,1000,1000,1000,1000,1000,1000]) > > SharedBlob(0x561e440f7620 sbid 0x3e6b92e)) doesn't match expected ref_map > > use_tracker(0xa*0x1000 0x[\ > > 0,0,0,0,1000,1000,1000,1000,2000,2000]) > > fsck status: remaining 3 error(s) and warning(s) > > [root@ceph-osd-19 ~]# > > > > [root@ceph-osd-19 ~]# ceph-objectstore-tool --data-path > > /var/lib/ceph/osd/ceph-257 --op list | grep > > "rbd_data.44.f397cc4c21bbfb.0000000000000e66" > > > ["43.70s0",{"oid":"rbd_data.44.f397cc4c21bbfb.0000000000000e66","key":"","snapid":-2,"hash":2802327152,"max":0,"pool":43,"namespace":"","generation":47965127,"sh\ > > ard_id":0,"max":0}] > > > > > > > > ---------------------------------------- > > > > ---------------------------------------- > > > > [root@ceph-osd-20 ~]# ceph-bluestore-tool --path > /var/lib/ceph/osd/ceph-277 > > --op fsck > > 2026-09-16T22:08:04.530+0200 7f35041c8ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-277) fsck error: > > 3#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ > > dbe3c7 lextent at 0x6a000~6000 spans a shard boundary > > 2026-09-16T22:08:04.530+0200 7f35041c8ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-277) fsck error: > > 3#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ > > dbe3c7 lextent at 0x6e000 overlaps with the previous, which ends at > 0x70000 > > 2026-09-16T22:08:04.530+0200 7f35041c8ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-277) fsck error: > > 3#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ > > dbe3c7 blob Blob(0x55ead1a808f0 spanning 2 > > blob([!~3000,0x1754e9e3000~1000,0x1754eb48000~3000,0x1754e809000~2000] > > llen=0x9000 csum crc32c/0x1000/36) use_tracker(\ > > 0x9*0x1000 0x[0,0,0,1000,1000,1000,1000,1000,1000]) (shared_blob=NULL)) > > doesn't match expected ref_map use_tracker(0x9*0x1000 > > 0x[0,0,0,1000,1000,1000,1000,2000,2\ > > 000]) > > fsck status: remaining 3 error(s) and warning(s) > > [root@ceph-osd-20 ~]# > > > > > > root@ceph-osd-20 ~]# ceph-objectstore-tool --data-path > > /var/lib/ceph/osd/ceph-277 --op list | grep > > "rbd_data.44.f397cc4c21bbfb.0000000000000e66" > > > ["43.70s3",{"oid":"rbd_data.44.f397cc4c21bbfb.0000000000000e66","key":"","snapid":-2,"hash":2802327152,"max":0,"pool":43,"namespace":"","generation":47965127,"sh\ > > ard_id":3,"max":0}] > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

6 11

Rotating cephx client keys in openstack VMs
by Bartosz Rabiega 19 Sep '26

19 Sep '26

Hello, I'm researching a way to rotate cephx client keys to aes256k 100% online for openstack VMs (qemu) with ceph rbd volumes attached (librbd). The problem ----------- Unfortunately I don't see any reasonable way to do it. Obviously I might be missing something (or a lot of something) Usually libvirt configures VM disk with following auth: <disk ...> <auth username='cephxuser'> <secret type='ceph' uuid='id-of-libvirt-secret'/> </auth> </disk> The cephx key is stored in a libvirt secret, it is read when VM starts and then stored in RAM for a VM lifetime (librbd AFAIK). If we rotate cephxuser key the living VMs will eventually stop working - when either auth_mon_ticket_ttl is exhausted or some connection to mon needs to be re-established librbd will try to authenticate with old key and fail. Potential solutions ------------------- A - rotate key, and distribute it as the same libvirt secret: - running VMs will eventually stop working - hard reboot or live migration shall fix them - stopped (SHUTOFF) VMs will be OK when started - new VMs and new attachments will be OK B - use new user and new key - new libvirt secret - running VMs will be OK, as long as we keep old user and key, live migration will probably fix the VM and use new user and new key - stopped (SHUTOFF) VMs will be OK when started, but will fail when old user/key is removed. username and libvirt secret uuid is kept in nova database and nothing will update it - new VMs and new attachments will be OK C - use the same user and have 2 keys for it - old and new. Eventually remove old one - running VMs will be OK, they will switch to new key during live migration - stopped (SHUTOFF) VMs will be OK, libvirt secret points to the new key so when VM is started it will use it. User is the same so no DB update is required - new VMs and new attachments will be OK A - not acceptable, breaks running VMs B - very difficult, requires a lot of VM migration, requires a lot of work to fix SHELVED/SHUTOFF VMs (no tooling for that AFAIK?) C - looks the best but not possible, ceph doesn't allow multiple keys per single user Summary ------- If my understanding is correct the effort to rotate the keys for openstack is tremendous and requires nova/cinder DB modification to fix SHUTOFF vms. Probably SHELVED vms will have the same problem. If you reached this point, thanks for reading :) Any ideas or comments? Bartosz Rabiega

2 1

We dodged a bullet
by Stefan Kooman 18 Sep '26

18 Sep '26

Hi, Yeah, so that is a bit of a clickbait title, but I do think it's true. In the hope this might be useful for someone, or some cluster, I decided to write this down. So what happened? First some context: subject at hand is a Ceph pacific cluster that has been running fine for over nine years (Went through 12 -> 13 -> 14 -> 15 -> 16 with many point releases in between). All OSDs were recreated in 16.2.11 to make use of the RocksDB column families without the need to do resharding (which would have taken forever with the amount of OMAP we have). This cluster has been provisioned with a home build Ansible playbook based on the manual procedure [1], and is basically an automation of all these manual steps. For OSDs we used the "short form" variant [2]. The first issue we encountered was when we added new storage nodes / OSDs to the cluster. The deployment of the first storage node took longer than expected (PCIe NVMe devices that were in use for OS were not correctly filtered and turned up as eligible for OSD creation). After we fixed that we could succesfully proceed. However, the first OSD that was added hit the "out" timer (we did not have noout flag set) and was set "out" by the cluster. We issued the "ceph osd in osd.$id" command to get the OSD back in. The state for all new OSDs was "down" and "in" apart from this OSD, which was in state "down" and "out". This immediately resulted in a assert of the monitors (num_down_in_osds <= num_in_osds). Different monitors crashed in quick succession but the quorum was never lost, and three mons (out of five) were always online. It also did not impact production traffic as now data was stored on them yet. The "ceph osd in osd.$id" that was hanging (due to crashing monitors) was aborted and shortly after that the monitors stopped crashing. The storage node got rebooted and the OSDs were all "up" and "in", apart from one. But this time we could successfully set the osd "in" ("up" and "in" state). We tried to reproduce this in a (virtual) test cluster that is as close a match to production but did not manage to reproduce. We figured this might have been a race condition of some kind (while the command was issued new OSDs might also have been added). The storage nodes were rebalanced and the new nodes were now also taking production workload. Somewhat later we decided to test a change (disable bypass workqueue flags of the encrypted (dmcrypt) devices) and shutdown the OSDs. Immediately after that several mons hit the same assert again. Although the OSDs had gracefully shut itself down ... because of the asserting mons no new osdmaps were created and the peer OSDs were not informed that there peers were down. Thus all involved OSDs had to figure this out themselves which they directly did ... but of course this did result in a lot of slow OSD operations. The OSDs were quickly started again and only until all OSDs were "up" and "in" again the monitors stopped crashing. The monitors did not loose quorum although we were of course afraid that they would. As it turned out this was likely not a race condition we started to look into this further. A search query on Ceph's issue tracker [3] resulted in a lot of hits. Turned out this issues has been hit by clusters provisioned by ROOK, manual procedure (ours) and potentially others and that the issue had been seen on clusters from 16.x, 17.x as well as 19.x and 20.x. This note gives a good summary [4]. It was fixed by this PR [5] and backported to Squid and Tentacle. But the reason *why* we hit this issue was not clear to us. Until we saw this report [6] wherein a user reported that their cluster had a lot of their OSDs with a "NEW" bit and when they would set an OSD "out" and "in" this bit would disappear and so did their issues (not hitting the assert anymore). We dumped the osdmap of the cluster and found out that *all* of our OSDs had the "NEW" bit active apart from one ... the one osd that had been through the OSD "out" / "in" state before. We only expected a "NEW" bit to be enabled for truely new OSDs, not OSDs that have been in the cluster for many years already. We decided not to wait until we would hit a scenario like the following: - one or more OSDs go down (broken or otherwise unable to start again) - have the monitors hit the assert - unable to get out of this situation easily as at that point you cannot issue any ceph commands as the monitors are asserting and the OSDs that went down cannot be brought back up. - hit a lot of slow ops in the meantime until the monitors would set the OSDs down ... or out eventually by itself (down out timer) which could have taken a long time in our case - have the risk of the monitors loosing the quorum and experience unavailability for an extended period of time. We backported the fix to the OSDMap.cc and created our own packages in case we would need them. After that we went ahead and set every OSD "out" and "in" to get rid of the NEW bit and that worked fine. The problem did not trigger anymore, as expected. Today I dived a bit deeper in the OSDMap.cc code responsible for this behavior ... turns out that there is a quicker way to fix this. Relevant code snippet: for (const auto &weight : inc.new_weight) { set_weight(weight.first, weight.second); // if we are marking in, clear the AUTOOUT and NEW bits, and clear // xinfo old_weight. if (weight.second) { osd_state[weight.first] &= ~(CEPH_OSD_AUTOOUT | CEPH_OSD_NEW); osd_xinfo[weight.first].old_weight = 0; } } If you perform a osd reweight, like "ceph osd reweight osd.0 1" it will remove the NEW bit. This is a way quicker process than "out" / "in". We added a reweight step to get rid of the "NEW" bit in our playbook if we would need to provision / replace OSDs again in the future. The reason why we hit this issue now, and not ever before, with the NEW bit OSDs in them and the bug still present is still unclear. There must be some other condition(s) that we hit this time and not before. I've checked our Cephadm based clusters and those don't have any NEW bits present. I'm not sure what steps Cephadm executes under the hood when it creates new OSDs but I guess this involves a step were it performs a reweight. For clusters out there that have been manually provisioned in the past, or created with Ansible playbooks that follow a similar procedure there might be OSDs present with this NEW bit that sooner or later have the risk of hitting this assert (until they run a version in which this is fixed). I advise you to check your clusters and get rid of these NEW bits because AFAIK their only purpose is to have this bit set when they are indeed, new, and they might harm your cluster(s) in the future. Questions we still have: - What other condition(s) are needed to trigger this assert - Would the manual install procedure need to include this reweight step, or is there another way to have the NEW bit cleared? Gr. Stefan [1]: https://docs.ceph.com/en/latest/install/manual-deployment [2]: https://docs.ceph.com/en/latest/install/manual-deployment/#short-form [3]: https://tracker.ceph.com/search?page=3&q=ceph_assert%28num_down_in_osds+%3C… [4]: https://tracker.ceph.com/issues/70869#note-9 [5]: https://github.com/ceph/ceph/pull/64009 [6]: https://tracker.ceph.com/issues/52535#note-12

1 0

Grafana, Prometheus, client certificates and mgmt-gateway in 20.2.3
by Robert Sander 18 Sep '26

18 Sep '26

Hi, we stumbled across an issue that is introduced with the new mgmt-gateway service. As soon as it is active the Prometheus service is configured to require client certificates for its API. It looks like the Grafana service is configured with client certificates according to the new service spec but it does not seem to use them. The result is no metrics in the Grafana dashboards. Can anyone confirm this behaviour? After removing the mgmt-gateway service and redeploying all monitoring services everything works again. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

2 9

Re: Performance issue after tentacle upgrade
by Moti 18 Sep '26

18 Sep '26

Hi Oguzhan, Thanks for your reply I did upgrade to 20.2.3 but the problem is still there. Latency reported by RGW matches what prober is reporting, so the latency jump is visible even if I unwrap latency from RGW beast logs. I tried investigating perf dump output per osd/rgw > host > cluster; also the hostoric_ops dump, but I can't spot a visible bottleneck, the latency is consistently high for HEAD requests and despite being 5x bigger than before, it's still some milliseconds that gets lost between other heavier ops latencies + sadly I don't have a 'before' perf dump result and comparing it to a healthy reef cluster is not giving any clues since the clusters differ entirely. Some piece of clue that I don't know how to interpret: when I re-index or re-create the target bucket, HEAD latency bounces back to its before-upgrade value! in my case it stayed there for an entire week before again peaking to 5x, the increase happens suddenly and not gradually, I found no informative logs around when the jump happens, and bucket is not growing in size so ..should I assume some sort of index corruption is happening? Regards On Tue, 11 Aug 2026, 17:42 Oguzhan Ozmen (BLOOMBERG/ 120 PARK) via ceph-users, <ceph-users(a)ceph.io> wrote: > Hey Moti, > > Zitat von Moti via ceph-users <ceph-users(a)ceph.io>: > > I recently upgraded from 18.2.4 to 20.2.1 > > > > I did notice this issue: > > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/C5BXQ26KD3… > > but that's not much similar to our case since we don't see any jump in > CPU or memory usage. > > Although the symptoms don't look the same, I'd recommend upgrading to > v20.2.3 to get the fix for this issue and the other fixes that went in: > https://docs.ceph.com/en/latest/releases/tentacle/#v20-2-3-tentacle > > After v20.2.3 upgrade, if you still see the same issue, then I'd suggest > collecting "perf" output per process per component (rgw, osd mainly) to > further analyze the bottleneck. > > You can also quickly check rgw logs to see what RGW reports as the latency > for those PUT/GET/HEAD requests whether it matches the latency you see on > your end (your wkld generator); especially for HEAD where no data streaming > happens. RGW logs "start" and "end" of a request. This is to rule out > whether bottleneck is outside of ceph (e.g., load-balancers). > > Cheers, > Oguzhan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

September Ceph Science User Group Virtual Meeting
by Enrico Bocchi 18 Sep '26

18 Sep '26

Dear all, We will be having a Ceph science/research/big cluster call next Tuesday, September 22nd, at 14:00 UTC. This time, it is my turn to tell a horror story about how an SSD firmware bug nearly killed our CephFS. The Ceph Science Virtual Meeting is an informal open call for community members primarily from hpc/htc/research/big cluster environments, though anyone is welcome, where we discuss whatever is on our minds regarding Ceph: Updates, outages, features, maintenance, ... There is no single set presenter for the length of the meeting, but it is somewhat moderated to keep the conversation flowing. If you wish to discuss something specific, please add it to the Pad linked below. If you are interested in giving a short talk during one of the next meetings, please reach out to me. Event Pad URL: https://pad.ceph.com/p/Ceph_Science_User_Group_20260922 Main Pad for the Ceph Science User Group: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Join the meeting at: https://meet.jit.si/ceph-science-wg Meetings will be recorded and posted to the Ceph Youtube channel. Virtual event details: September 22nd, 2026 14:00 UTC See the Ceph Community Calendar (https://ceph.io/en/community/meetups/) for the series information and for the meeting time in your timezone. See you there! Enrico

1 0

19.2.6 crash daemon - aes256k not supported?
by Andreas Eriksson 18 Sep '26

18 Sep '26

Hi, When finishing up an upgrade to 19.2.6 I rotated keys for the crash daemon. # ceph auth rotate --key-type=aes256k client.crash.$myhost # ceph orch reconfig crash.$myhost # ceph orch daemon restart crash.$myhost But it seems it's not supported. I get this in the daemon logs: Sep 18 10:21:24 myhost bash[3327369]: INFO:ceph-crash:pinging cluster to exercise our key Sep 18 10:21:24 myhost bash[3327369]: 2026-09-18T08:21:24.374+0000 766f4adb1640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] and I support [1] Sep 18 10:21:24 myhost bash[3327369]: 2026-09-18T08:21:24.374+0000 766f4a5b0640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] and I support [1] Sep 18 10:21:24 myhost bash[3327369]: 2026-09-18T08:21:24.374+0000 766f49daf640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] and I support [1] Sep 18 10:21:24 myhost bash[3327369]: 2026-09-18T08:21:24.374+0000 766f4c83b640 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication Sep 18 10:21:24 myhost bash[3327369]: [errno 13] RADOS permission denied (error connecting to the cluster) Sep 18 10:21:24 myhost bash[3327369]: INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s

1 0

Massive mon slow-ops when restarting mon daemons
by Alex Walender 17 Sep '26

17 Sep '26

Hey there! I’m currently in the process of upgrading the production cluster from v18.2.8 -> v19.2.6 via „ceph orch upgrade“. I tested this upgrade in our staging environment and everything went great. On our production environment I encounter some worrisome issues with the mon daemons. The cluster itself: 5x MON Daemons 5x MGR Daemons 1500x OSD - 2 larger Pools with each 16k in PG. … so a somewhat larger cluster. The orchestrator started re-deploying the MGR, which went fine. During the MON redeployment, the cluster went into warning with messages like „34846 slow ops. mon.mon1 has slow ops“. The counter climbed rapidly. Only way to stop this was to restart the MON in question. Sometimes I had to restart the MON multiple times. Afterwards the cluster is healthy again….until I restart any MON. The symptoms return until restarting fixes it again. I tried to dump the mon_ops_in_flight, but the command just hangs (probably because of the large error count). Filtering the journal for the issues: ``` journalctl -g "oldest is" --since "3 hours ago" -u ceph-d8a6f365-99de-42f2-b32b-c9ea11df945c(a)mon.partnach <mailto:ceph-d8a6f365-99de-42f2-b32b-c9ea11df945c@mon.partnach> Sep 14 09:45:05 partnach conmon[1735]: 2026-09-14T07:45:05.331+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18286 slow ops, oldest is log(1000 entries from seq 2 at 2026-08-26T04:51:47.819315+0000) Sep 14 09:45:10 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 18505 slow ops, oldest is log(1000 entries from seq 4003 at 2026-09-11T12:25:52.136679+0000) Sep 14 09:45:10 partnach conmon[1735]: 2026-09-14T07:45:10.827+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18505 slow ops, oldest is log(1000 entries from seq 4003 at 2026-09-11T12:25:52.136679+0000) Sep 14 09:45:15 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 18671 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-07T16:47:58.529125+0000) Sep 14 09:45:15 partnach conmon[1735]: 2026-09-14T07:45:15.924+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18671 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-07T16:47:58.529125+0000) Sep 14 09:45:21 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 18814 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-10T20:12:26.565558+0000) Sep 14 09:45:21 partnach conmon[1735]: 2026-09-14T07:45:21.224+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18814 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-10T20:12:26.565558+0000) Sep 14 09:45:26 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19042 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-10T14:13:26.625340+0000) Sep 14 09:45:26 partnach conmon[1735]: 2026-09-14T07:45:26.700+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19042 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-10T14:13:26.625340+0000) Sep 14 09:45:32 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19280 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-13T08:23:50.558481+0000) Sep 14 09:45:32 partnach conmon[1735]: 2026-09-14T07:45:32.432+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19280 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-13T08:23:50.558481+0000) Sep 14 09:45:37 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19433 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-11T20:28:14.104877+0000) Sep 14 09:45:37 partnach conmon[1735]: 2026-09-14T07:45:37.568+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19433 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-11T20:28:14.104877+0000) Sep 14 09:45:43 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19632 slow ops, oldest is log(1000 entries from seq 2 at 2026-09-10T08:47:13.259138+0000) Sep 14 09:45:43 partnach conmon[1735]: 2026-09-14T07:45:43.292+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19632 slow ops, oldest is log(1000 entries from seq 2 at 2026-09-10T08:47:13.259138+0000) ``` I’m not quite sure what the lock is here. The mon tries to store some old cluster log entries from various dates? Checked the underlying network fabric, CPU load, clock skew, compacted the rocksdb manually on each mon ect. All is fine. I also set "ceph config set global mon_cluster_log_level info“ **before** the 10th of September. Luckily the upgrade has finished with the MON daemons and is now re-deploying all OSDs. Does anyone know whats going on? Some kind of old hanging Log-Entries? Best Regards, Alex Walender --------------------------- M.Sc Alex Walender Institut für Bio- und Geowissenschaften IBG 5 - Computergestützte Metagenomik / de.NBI Cloud Site Bielefeld Büro : Universität Bielefeld (UHG), N7-101 Tel. : +49-521-106-2907 Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellv. Vorsitzende), Prof. Dr. Ir. Pieter Jansens

2 1

Help needed: multiple OSDs refuse to restart (and pg in down state)
by Massimo Sgaravatto 16 Sep '26

16 Sep '26

Dear all Today we had a problem in our ceph cluster: basically for some (still unknown) reasons, 5 OSDs (206, 233, 250, 257, 277) crashed and now they refuse to start. The main problem is that 1 pg is in down+remapped state because it is waiting for 3 of these OSDs [*] Doing a ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-xxx --op fsck, I see [**] that the problem is with 2 objects (rbd_data.44.f397cc4c21bbfb.000000000000319b and rbd_data.44.f397cc4c21bbfb.0000000000000e66 A "--op repair" is not able to fix the problem I am not sure how to proceed now Is the only option removing the problematic object from each OSD, doing e.g.: ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-206 --pgid 43.1b0 '{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":7151\ 31231,"max":0,"pool":43,"namespace":"","shardid":3}' remove ? Then the OSDs should hopefully restart, right ? Or are there better options ? Your help will be really appreciated ! Thanks, Massimo [*] ceph pg 43.1b0 query ... ... "blocked": "peering is blocked due to down osds", "down_osds_we_would_probe": [ 206, 233, 250 ], "peering_blocked_by": [ { "osd": 206, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 233, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 250, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" } ] [**] [root@ceph-osd-17 ~]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-206 --op fsck 2026-09-16T21:26:23.609+0200 7fb5aa3ddac0 -1 bluestore(/var/lib/ceph/osd/ceph-206) fsck error: 3#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a lextent at 0x6c000~4000 spans a shard boundary 2026-09-16T21:26:23.609+0200 7fb5aa3ddac0 -1 bluestore(/var/lib/ceph/osd/ceph-206) fsck error: 3#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a lextent at 0x6d000 overlaps with the previous, which ends at 0x70000 2026-09-16T21:26:23.609+0200 7fb5aa3ddac0 -1 bluestore(/var/lib/ceph/osd/ceph-206) fsck error: 3#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a blob Blob(0x5605e5760b60 spanning 1 blob([!~4000,0xa211d395000~1000,!~3000,0x23b97906000~1000,0x23b97226000~3000] llen=0xc000 csum+shared crc32c/0x1000/48\ ) use_tracker(0xc*0x1000 0x[0,0,0,0,1000,0,0,0,1000,1000,1000,1000]) SharedBlob(0x5606c99ebfc0 sbid 0x37ae23b)) doesn't match expected ref_map use_tracker(0xc*0x\ 1000 0x[0,0,0,0,1000,0,0,0,1000,2000,2000,2000]) fsck status: remaining 3 error(s) and warning(s) root@ceph-osd-17 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-206 --op list | grep "rbd_data.44.f397cc4c21bbfb.000000000000319b" ["43.1b0s3",{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":4210196912,"max":0,"pool":43,"namespace":"","generation":44630682,"s\ hard_id":3,"max":0}] ---------------------------------------- ---------------------------------------- [root@ceph-osd-18 ~]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-233 --op fsck 2026-09-16T21:37:09.598+0200 7f8ee4c49ac0 -1 bluestore(/var/lib/ceph/osd/ceph-233) fsck error: 4#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a lextent at 0x6c000~4000 spans a shard boundary 2026-09-16T21:37:09.598+0200 7f8ee4c49ac0 -1 bluestore(/var/lib/ceph/osd/ceph-233) fsck error: 4#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a lextent at 0x6d000 overlaps with the previous, which ends at 0x70000 2026-09-16T21:37:09.598+0200 7f8ee4c49ac0 -1 bluestore(/var/lib/ceph/osd/ceph-233) fsck error: 4#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a blob Blob(0x5557c8bef2b0 spanning 1 blob([!~4000,0x2ce88f4c000~1000,!~3000,0x2ce89560000~1000,0x2ce88a80000~3000] llen=0xc000 csum+shared crc32c/0x1000/48\ ) use_tracker(0xc*0x1000 0x[0,0,0,0,1000,0,0,0,1000,1000,1000,1000]) SharedBlob(0x5557fff8b1a0 sbid 0x323170b)) doesn't match expected ref_map use_tracker(0xc*0x\ 1000 0x[0,0,0,0,1000,0,0,0,1000,2000,2000,2000]) fsck status: remaining 3 error(s) and warning(s) [root@ceph-osd-18 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-233 --op list | grep "rbd_data.44.f397cc4c21bbfb.000000000000319b" ["43.1b0s4",{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":4210196912,"max":0,"pool":43,"namespace":"","generation":44630682,"s\ hard_id":4,"max":0}] [root@ceph-osd-18 ~]# ---------------------------------------- [root@ceph-osd-19 ~]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-250 --op fsck 2026-09-16T21:59:02.401+0200 7fa2e8795ac0 -1 bluestore(/var/lib/ceph/osd/ceph-250) fsck error: 0#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a lextent at 0x6c000~4000 spans a shard boundary 2026-09-16T21:59:02.401+0200 7fa2e8795ac0 -1 bluestore(/var/lib/ceph/osd/ceph-250) fsck error: 0#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a lextent at 0x6d000 overlaps with the previous, which ends at 0x70000 2026-09-16T21:59:02.401+0200 7fa2e8795ac0 -1 bluestore(/var/lib/ceph/osd/ceph-250) fsck error: 0#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a blob Blob(0x556a29cc4820 spanning 1 blob([!~4000,0x464866d9000~1000,!~3000,0x4648731f000~1000,0x46485e09000~3000] llen=0xc000 csum+shared crc32c/0x1000/48\ ) use_tracker(0xc*0x1000 0x[0,0,0,0,1000,0,0,0,1000,1000,1000,1000]) SharedBlob(0x556a1fb84e40 sbid 0x98f249)) doesn't match expected ref_map use_tracker(0xc*0x1\ 000 0x[0,0,0,0,1000,0,0,0,1000,2000,2000,2000]) fsck status: remaining 3 error(s) and warning(s) [root@ceph-osd-19 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-250 --op list | grep "rbd_data.44.f397cc4c21bbfb.000000000000319b" ["43.1b0s0",{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":4210196912,"max":0,"pool":43,"namespace":"","generation":44630682,"s\ hard_id":0,"max":0}] ---------------------------------------- [root@ceph-osd-19 ~]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-257 --op fsck 2026-09-16T22:06:09.627+0200 7fded8db6ac0 -1 bluestore(/var/lib/ceph/osd/ceph-257) fsck error: 0#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ dbe3c7 lextent at 0x6a000~6000 spans a shard boundary 2026-09-16T22:06:09.627+0200 7fded8db6ac0 -1 bluestore(/var/lib/ceph/osd/ceph-257) fsck error: 0#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ dbe3c7 lextent at 0x6e000 overlaps with the previous, which ends at 0x70000 2026-09-16T22:06:09.627+0200 7fded8db6ac0 -1 bluestore(/var/lib/ceph/osd/ceph-257) fsck error: 0#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ dbe3c7 blob Blob(0x561e442541a0 spanning 2 blob([!~4000,0x19c87d9b000~1000,0x46e6508e000~3000,0x46e64693000~2000] llen=0xa000 csum+shared crc32c/0x1000/40) use_t\ racker(0xa*0x1000 0x[0,0,0,0,1000,1000,1000,1000,1000,1000]) SharedBlob(0x561e440f7620 sbid 0x3e6b92e)) doesn't match expected ref_map use_tracker(0xa*0x1000 0x[\ 0,0,0,0,1000,1000,1000,1000,2000,2000]) fsck status: remaining 3 error(s) and warning(s) [root@ceph-osd-19 ~]# [root@ceph-osd-19 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-257 --op list | grep "rbd_data.44.f397cc4c21bbfb.0000000000000e66" ["43.70s0",{"oid":"rbd_data.44.f397cc4c21bbfb.0000000000000e66","key":"","snapid":-2,"hash":2802327152,"max":0,"pool":43,"namespace":"","generation":47965127,"sh\ ard_id":0,"max":0}] ---------------------------------------- ---------------------------------------- [root@ceph-osd-20 ~]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-277 --op fsck 2026-09-16T22:08:04.530+0200 7f35041c8ac0 -1 bluestore(/var/lib/ceph/osd/ceph-277) fsck error: 3#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ dbe3c7 lextent at 0x6a000~6000 spans a shard boundary 2026-09-16T22:08:04.530+0200 7f35041c8ac0 -1 bluestore(/var/lib/ceph/osd/ceph-277) fsck error: 3#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ dbe3c7 lextent at 0x6e000 overlaps with the previous, which ends at 0x70000 2026-09-16T22:08:04.530+0200 7f35041c8ac0 -1 bluestore(/var/lib/ceph/osd/ceph-277) fsck error: 3#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ dbe3c7 blob Blob(0x55ead1a808f0 spanning 2 blob([!~3000,0x1754e9e3000~1000,0x1754eb48000~3000,0x1754e809000~2000] llen=0x9000 csum crc32c/0x1000/36) use_tracker(\ 0x9*0x1000 0x[0,0,0,1000,1000,1000,1000,1000,1000]) (shared_blob=NULL)) doesn't match expected ref_map use_tracker(0x9*0x1000 0x[0,0,0,1000,1000,1000,1000,2000,2\ 000]) fsck status: remaining 3 error(s) and warning(s) [root@ceph-osd-20 ~]# root@ceph-osd-20 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-277 --op list | grep "rbd_data.44.f397cc4c21bbfb.0000000000000e66" ["43.70s3",{"oid":"rbd_data.44.f397cc4c21bbfb.0000000000000e66","key":"","snapid":-2,"hash":2802327152,"max":0,"pool":43,"namespace":"","generation":47965127,"sh\ ard_id":3,"max":0}]

1 0

Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
by Frank Ederveen 16 Sep '26

16 Sep '26

> If you upgrade to 20.2.4 the key rotation for the Ceph services will > be done by the orchestrator automatically. But only for the Ceph > services, no client keyrings will be touched by cephadm. So you can > safely upgrade and fix the CVEs but can still use clients with the aes > format keys. I have upgraded two cephadm managed test clusters from 20.2.1 to 20.2.4 with limited success on both. It allows new and old keys, and the default is to hand out new keys, but all of my clients (osd, rgw, admin, bootstrap, crash) are still using the insecure key. I ran the upgrade again with cephadm which did restart all the OSD, but the issue remains. All previous upgrades have gone very easily, this one.. not so much. Manually rotating and copying keys around is not an option for clusters with > 1k osd. Is there going to be a new release soon, or some scripting which fixes things? health: HEALTH_WARN 87 auth client entities with insecure key types Monitors are configured to allow auth using insecure key types Monitors are configured to allow creation of insecure key types ... [WRN] AUTH_INSECURE_CLIENT_KEY_TYPE: 87 auth client entities with insecure key types entity client.admin using insecure key type: aes entity client.bootstrap-mds using insecure key type: aes entity client.bootstrap-mgr using insecure key type: aes entity client.bootstrap-osd using insecure key type: aes entity client.bootstrap-rbd using insecure key type: aes entity client.bootstrap-rbd-mirror using insecure key type: aes entity client.bootstrap-rgw using insecure key type: aes entity client.crash.node1-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node2-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node3-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node4-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node5-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node6-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node7-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node8-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.osd-lockbox.02f209c1-ade4-492a-8a65-22fd21a71365 using insecure key type: aes entity client.osd-lockbox.0cae9ed5-2976-4047-9628-f90d53b3e577 using insecure key type: aes entity client.osd-lockbox.1656cd37-7cd9-4ade-a396-5d973c866711 using insecure key type: aes entity client.osd-lockbox.1d17a243-d4a5-4f7d-9aeb-88e14dc45812 using insecure key type: aes ... Thanks, Frank ________________________________ From: Eugen Block <ceph-users(a)ceph.io> Sent: 31 August 2026 07:44:50 To: ceph-users(a)ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4 If you upgrade to 20.2.4 the key rotation for the Ceph services will be done by the orchestrator automatically. But only for the Ceph services, no client keyrings will be touched by cephadm. So you can safely upgrade and fix the CVEs but can still use clients with the aes format keys. Zitat von Brent Kennedy <ceph-users(a)ceph.io>: > Question, sort of related. If upgrading from squid to tentacle, > will it automatically try to rotate keys are part of the cluster > upgrade or is that still a manual process? We would like to avoid > the key rotation for now as we have some proxmox nodes using ceph > storage in some of the clusters. > > Regards, > -Brent > > Existing Clusters: > US Production(HDD): Squid 19.2.3 Cephadm with 11 osd servers, 5 > mons, 4 rgw, 2 iscsigw, 2 mds > UK Production(HDD): Squid 19.2.3 Cephadm with 20 osd servers, 5 > mons, 4 rgw, 2 iscsigw, 2 mds > US Production(SSD): Squid 19.2.3 Cephadm with 6 osd servers, 5 mons, > 4 rgw, 2 mds > UK Production(SSD): Squid 19.2.3 cephadm with 6 osd servers, 5 mons, > 4 rgw, 2 mds > > > > -----Original Message----- > From: Patrick Donnelly <ceph-users(a)ceph.io> > Sent: Saturday, August 29, 2026 10:38 AM > To: Saif Mohammad <samdto987(a)gmail.com> > Cc: ceph-users(a)ceph.io > Subject: [ceph-users] Re: Clarification on Client Key Rotation > During Upgrade to 19.2.6/20.2.4 > > Hello, > > On Fri, Aug 28, 2026 at 10:20 PM Saif Mohammad <ceph-users(a)ceph.io> wrote: >> >> Hi, >> >> While upgrading to Squid 19.2.6 or Tentacle 20.2.4, since these >> versions introduce the secure key type, I have a clarification >> regarding client key rotation. >> For the client.admin key, we need to rotate the key and then >> import/update the keyring. >> For the other client keys, do we only need to rotate the key and >> verify that the key has changed? Is there no need to import/update >> the keyring for the other clients, since there is no local keyring >> available? > > Step (9) of [1] says: > > "Then copy and import the key to each machine using that client.$ID > credential." > > [1] > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-… > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Red Hat Partner Engineer > IBM, Inc. > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 4