- ceph-users - lists.ceph.io

19.2.6 crash daemon - aes256k not supported?
by Andreas Eriksson 21 Sep '26

21 Sep '26

Hi, When finishing up an upgrade to 19.2.6 I rotated keys for the crash daemon. # ceph auth rotate --key-type=aes256k client.crash.$myhost # ceph orch reconfig crash.$myhost # ceph orch daemon restart crash.$myhost But it seems it's not supported. I get this in the daemon logs: Sep 18 10:21:24 myhost bash[3327369]: INFO:ceph-crash:pinging cluster to exercise our key Sep 18 10:21:24 myhost bash[3327369]: 2026-09-18T08:21:24.374+0000 766f4adb1640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] and I support [1] Sep 18 10:21:24 myhost bash[3327369]: 2026-09-18T08:21:24.374+0000 766f4a5b0640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] and I support [1] Sep 18 10:21:24 myhost bash[3327369]: 2026-09-18T08:21:24.374+0000 766f49daf640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] and I support [1] Sep 18 10:21:24 myhost bash[3327369]: 2026-09-18T08:21:24.374+0000 766f4c83b640 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication Sep 18 10:21:24 myhost bash[3327369]: [errno 13] RADOS permission denied (error connecting to the cluster) Sep 18 10:21:24 myhost bash[3327369]: INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s

2 1

Re: Help needed: multiple OSDs refuse to restart (and pg in down state)
by Massimo Sgaravatto 20 Sep '26

20 Sep '26

Hi Anthony, all I am running ceph squid 19.2.3 I copied the relevant part of the log (from the start till the crash) for a OSD (osd.206) in https://cernbox.cern.ch/s/56sdGTQsotbN2XE So as far as I can understand it crashes because of: 2026-09-16T21:16:06.859+0200 7f5431ee1640 4 rocksdb: [db/compaction/compaction_job.cc:1588] [O-2] [JOB 9] Generated table #148643: 151504 keys, 67943642 bytes, temperature: kUnknown 2026-09-16T21:16:06.859+0200 7f5431ee1640 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1789586166860678, "cf_name": "O-2", "job": 9, "event": "table_file_creation", "file_number": 148643, "file_size": 67943642, "file_checksum": "", "f\ ile_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 67111852, "index_size": 1538071, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_v\ alue_is_delta_encoded": 1, "filter_size": 378821, "raw_key_size": 12956410, "raw_average_key_size": 85, "raw_value_size": 67640956, "raw_average_value_size": 446, "num_data_blocks": 18954, "num_entries": 151504, "num_filter_entrie\ s": 151504, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "O-2", "column_family_id": 9, "comparator": "leveldb\ .BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "LZ4", "compression_options": "window_bits=-14; level=32767; strategy=0; \ max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1778598719, "oldest_key_time": 0, "file_creation_time": 1789586166, "slow_compression_estimated_data_size":\ 0, "fast_compression_estimated_data_size": 0, "db_id": "1103608a-90ce-4939-82c0-de96ff245167", "db_session_id": "FO9J42Q67M9AK232DQXE", "orig_file_number": 148643, "seqno_to_time_mapping": "N/A"}} 2026-09-16T21:16:07.421+0200 7f541fcaf640 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.3/rpm/el9/BUILD/ceph-19.2.3/src/o\ s/bluestore/bluestore_types.cc: In function 'bool bluestore_blob_use_tracker_t::put(uint32_t, uint32_t, PExtentVector*)' thread 7f541fcaf640 time 2026-09-16T21:16:07.418902+0200 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.3/rpm/el9/BUILD/ceph-19.2.3/src/os/bluestore/bluestore_types.cc: 511: FAILED c\ eph_assert(diff <= bytes_per_au[pos]) ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x113) [0x560cc2d4c85d] 2: /usr/bin/ceph-osd(+0x401a14) [0x560cc2d4ca14] 3: /usr/bin/ceph-osd(+0x3f2930) [0x560cc2d3d930] 4: (BlueStore::Blob::put_ref(BlueStore::Collection*, unsigned int, unsigned int, std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)5, bluestore_pextent_t> >*)+0xaa) [0x560cc32bb2fa] 5: (BlueStore::OldExtent::create(boost::intrusive_ptr<BlueStore::Collection>, unsigned int, unsigned int, unsigned int, boost::intrusive_ptr<BlueStore::Blob>&)+0x11d) [0x560cc32bf47d] 6: (BlueStore::ExtentMap::punch_hole(boost::intrusive_ptr<BlueStore::Collection>&, unsigned long, unsigned long, boost::intrusive::list<BlueStore::OldExtent, boost::intrusive::member_hook<BlueStore::OldExtent, boost::intrusive::l\ ist_member_hook<>, &BlueStore::OldExtent::old_extent_item> >*)+0x451) [0x560cc32cbe21] 7: (BlueStore::_do_truncate(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&, unsigned long, std::set<BlueStore::SharedBlob*, std::less<BlueStore::SharedBlob*>, std::\ allocator<BlueStore::SharedBlob*> >*)+0x205) [0x560cc33483f5] 8: (BlueStore::_do_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&)+0xc1) [0x560cc334d781] 9: (BlueStore::_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&)+0x7f) [0x560cc334f0af] 10: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x12a6) [0x560cc3334206] 11: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x2f3) [0\ x560cc3335123] 12: /usr/bin/ceph-osd(+0x5239a8) [0x560cc2e6e9a8] 13: (OSD::dispatch_context(PeeringCtx&, PG*, std::shared_ptr<OSDMap const>, ThreadPool::TPHandle*)+0x115) [0x560cc2edf495] 14: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x2bc) [0x560cc2eea72c] 15: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x560cc31359b1] 16: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xcd0) [0x560cc2f04b70] 17: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x560cc34238aa] 18: /usr/bin/ceph-osd(+0xad8e64) [0x560cc3423e64] 19: /lib64/libc.so.6(+0x8b2ea) [0x7f543ce8b2ea] 20: /lib64/libc.so.6(+0x1103d0) [0x7f543cf103d0] 2026-09-16T21:16:07.426+0200 7f541fcaf640 -1 *** Caught signal (Aborted) ** in thread 7f541fcaf640 thread_name:tp_osd_tp ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) 1: /lib64/libc.so.6(+0x3fc30) [0x7f543ce3fc30] 2: /lib64/libc.so.6(+0x8d02c) [0x7f543ce8d02c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x169) [0x560cc2d4c8b3] 6: /usr/bin/ceph-osd(+0x401a14) [0x560cc2d4ca14] 7: /usr/bin/ceph-osd(+0x3f2930) [0x560cc2d3d930] 8: (BlueStore::Blob::put_ref(BlueStore::Collection*, unsigned int, unsigned int, std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)5, bluestore_pextent_t> >*)+0xaa) [0x560cc32bb2fa] 9: (BlueStore::OldExtent::create(boost::intrusive_ptr<BlueStore::Collection>, unsigned int, unsigned int, unsigned int, boost::intrusive_ptr<BlueStore::Blob>&)+0x11d) [0x560cc32bf47d] 10: (BlueStore::ExtentMap::punch_hole(boost::intrusive_ptr<BlueStore::Collection>&, unsigned long, unsigned long, boost::intrusive::list<BlueStore::OldExtent, boost::intrusive::member_hook<BlueStore::OldExtent, boost::intrusive::\ list_member_hook<>, &BlueStore::OldExtent::old_extent_item> >*)+0x451) [0x560cc32cbe21] 11: (BlueStore::_do_truncate(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&, unsigned long, std::set<BlueStore::SharedBlob*, std::less<BlueStore::SharedBlob*>, std:\ :allocator<BlueStore::SharedBlob*> >*)+0x205) [0x560cc33483f5] 12: (BlueStore::_do_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&)+0xc1) [0x560cc334d781] 13: (BlueStore::_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&)+0x7f) [0x560cc334f0af] 14: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x12a6) [0x560cc3334206] 15: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x2f3) [0\ x560cc3335123] 16: /usr/bin/ceph-osd(+0x5239a8) [0x560cc2e6e9a8] 17: (OSD::dispatch_context(PeeringCtx&, PG*, std::shared_ptr<OSDMap const>, ThreadPool::TPHandle*)+0x115) [0x560cc2edf495] 18: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x2bc) [0x560cc2eea72c] 19: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x560cc31359b1] 20: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xcd0) [0x560cc2f04b70] 21: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x560cc34238aa] 22: /usr/bin/ceph-osd(+0xad8e64) [0x560cc3423e64] 23: /lib64/libc.so.6(+0x8b2ea) [0x7f543ce8b2ea] 24: /lib64/libc.so.6(+0x1103d0) [0x7f543cf103d0] Thanks, Massimo On Thu, Sep 17, 2026 at 4:44 AM Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > Start with what they log when they try to start. > > And what release you’re running. > > > On Sep 16, 2026, at 4:47 PM, Massimo Sgaravatto <ceph-users(a)ceph.io> > wrote: > > > > Dear all > > Today we had a problem in our ceph cluster: basically for some (still > > unknown) reasons, 5 OSDs (206, 233, 250, 257, 277) crashed and now they > > refuse to start. > > > > > > The main problem is that 1 pg is in down+remapped state because it is > > waiting for 3 of these OSDs [*] > > > > > > Doing a ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-xxx --op fsck, > I > > see [**] that the problem is with 2 objects > > (rbd_data.44.f397cc4c21bbfb.000000000000319b > > and rbd_data.44.f397cc4c21bbfb.0000000000000e66 > > A "--op repair" is not able to fix the problem > > > > > > I am not sure how to proceed now > > > > Is the only option removing the problematic object from each OSD, doing > > e.g.: > > > > > > ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-206 --pgid > 43.1b0 > > > '{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":7151\ > > 31231,"max":0,"pool":43,"namespace":"","shardid":3}' remove > > > > ? > > > > Then the OSDs should hopefully restart, right ? > > > > Or are there better options ? > > > > Your help will be really appreciated ! > > > > > > Thanks, Massimo > > > > [*] > > ceph pg 43.1b0 query > > ... > > ... > > > > "blocked": "peering is blocked due to down osds", > > "down_osds_we_would_probe": [ > > 206, > > 233, > > 250 > > ], > > "peering_blocked_by": [ > > { > > "osd": 206, > > "current_lost_at": 0, > > "comment": "starting or marking this osd lost may let > > us proceed" > > }, > > { > > "osd": 233, > > "current_lost_at": 0, > > "comment": "starting or marking this osd lost may let > > us proceed" > > }, > > { > > "osd": 250, > > "current_lost_at": 0, > > "comment": "starting or marking this osd lost may let > > us proceed" > > } > > ] > > > > [**] > > > > [root@ceph-osd-17 ~]# ceph-bluestore-tool --path > /var/lib/ceph/osd/ceph-206 > > --op fsck > > 2026-09-16T21:26:23.609+0200 7fb5aa3ddac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-206) fsck error: > > 3#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a lextent at 0x6c000~4000 spans a shard boundary > > 2026-09-16T21:26:23.609+0200 7fb5aa3ddac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-206) fsck error: > > 3#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a lextent at 0x6d000 overlaps with the previous, which ends at > 0x70000 > > 2026-09-16T21:26:23.609+0200 7fb5aa3ddac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-206) fsck error: > > 3#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a blob Blob(0x5605e5760b60 spanning 1 > > > blob([!~4000,0xa211d395000~1000,!~3000,0x23b97906000~1000,0x23b97226000~3000] > > llen=0xc000 csum+shared crc32c/0x1000/48\ > > ) use_tracker(0xc*0x1000 0x[0,0,0,0,1000,0,0,0,1000,1000,1000,1000]) > > SharedBlob(0x5606c99ebfc0 sbid 0x37ae23b)) doesn't match expected ref_map > > use_tracker(0xc*0x\ > > 1000 0x[0,0,0,0,1000,0,0,0,1000,2000,2000,2000]) > > fsck status: remaining 3 error(s) and warning(s) > > > > root@ceph-osd-17 ~]# ceph-objectstore-tool --data-path > > /var/lib/ceph/osd/ceph-206 --op list | grep > > "rbd_data.44.f397cc4c21bbfb.000000000000319b" > > > ["43.1b0s3",{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":4210196912,"max":0,"pool":43,"namespace":"","generation":44630682,"s\ > > hard_id":3,"max":0}] > > > > > > ---------------------------------------- > > ---------------------------------------- > > > > [root@ceph-osd-18 ~]# ceph-bluestore-tool --path > /var/lib/ceph/osd/ceph-233 > > --op fsck > > 2026-09-16T21:37:09.598+0200 7f8ee4c49ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-233) fsck error: > > 4#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a lextent at 0x6c000~4000 spans a shard boundary > > 2026-09-16T21:37:09.598+0200 7f8ee4c49ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-233) fsck error: > > 4#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a lextent at 0x6d000 overlaps with the previous, which ends at > 0x70000 > > 2026-09-16T21:37:09.598+0200 7f8ee4c49ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-233) fsck error: > > 4#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a blob Blob(0x5557c8bef2b0 spanning 1 > > > blob([!~4000,0x2ce88f4c000~1000,!~3000,0x2ce89560000~1000,0x2ce88a80000~3000] > > llen=0xc000 csum+shared crc32c/0x1000/48\ > > ) use_tracker(0xc*0x1000 0x[0,0,0,0,1000,0,0,0,1000,1000,1000,1000]) > > SharedBlob(0x5557fff8b1a0 sbid 0x323170b)) doesn't match expected ref_map > > use_tracker(0xc*0x\ > > 1000 0x[0,0,0,0,1000,0,0,0,1000,2000,2000,2000]) > > fsck status: remaining 3 error(s) and warning(s) > > > > [root@ceph-osd-18 ~]# ceph-objectstore-tool --data-path > > /var/lib/ceph/osd/ceph-233 --op list | grep > > "rbd_data.44.f397cc4c21bbfb.000000000000319b" > > > ["43.1b0s4",{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":4210196912,"max":0,"pool":43,"namespace":"","generation":44630682,"s\ > > hard_id":4,"max":0}] > > [root@ceph-osd-18 ~]# > > > > > > ---------------------------------------- > > > > > > [root@ceph-osd-19 ~]# ceph-bluestore-tool --path > /var/lib/ceph/osd/ceph-250 > > --op fsck > > 2026-09-16T21:59:02.401+0200 7fa2e8795ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-250) fsck error: > > 0#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a lextent at 0x6c000~4000 spans a shard boundary > > 2026-09-16T21:59:02.401+0200 7fa2e8795ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-250) fsck error: > > 0#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a lextent at 0x6d000 overlaps with the previous, which ends at > 0x70000 > > 2026-09-16T21:59:02.401+0200 7fa2e8795ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-250) fsck error: > > 0#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a blob Blob(0x556a29cc4820 spanning 1 > > > blob([!~4000,0x464866d9000~1000,!~3000,0x4648731f000~1000,0x46485e09000~3000] > > llen=0xc000 csum+shared crc32c/0x1000/48\ > > ) use_tracker(0xc*0x1000 0x[0,0,0,0,1000,0,0,0,1000,1000,1000,1000]) > > SharedBlob(0x556a1fb84e40 sbid 0x98f249)) doesn't match expected ref_map > > use_tracker(0xc*0x1\ > > 000 0x[0,0,0,0,1000,0,0,0,1000,2000,2000,2000]) > > fsck status: remaining 3 error(s) and warning(s) > > > > [root@ceph-osd-19 ~]# ceph-objectstore-tool --data-path > > /var/lib/ceph/osd/ceph-250 --op list | grep > > "rbd_data.44.f397cc4c21bbfb.000000000000319b" > > > ["43.1b0s0",{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":4210196912,"max":0,"pool":43,"namespace":"","generation":44630682,"s\ > > hard_id":0,"max":0}] > > ---------------------------------------- > > > > [root@ceph-osd-19 ~]# ceph-bluestore-tool --path > /var/lib/ceph/osd/ceph-257 > > --op fsck > > 2026-09-16T22:06:09.627+0200 7fded8db6ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-257) fsck error: > > 0#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ > > dbe3c7 lextent at 0x6a000~6000 spans a shard boundary > > 2026-09-16T22:06:09.627+0200 7fded8db6ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-257) fsck error: > > 0#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ > > dbe3c7 lextent at 0x6e000 overlaps with the previous, which ends at > 0x70000 > > 2026-09-16T22:06:09.627+0200 7fded8db6ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-257) fsck error: > > 0#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ > > dbe3c7 blob Blob(0x561e442541a0 spanning 2 > > blob([!~4000,0x19c87d9b000~1000,0x46e6508e000~3000,0x46e64693000~2000] > > llen=0xa000 csum+shared crc32c/0x1000/40) use_t\ > > racker(0xa*0x1000 0x[0,0,0,0,1000,1000,1000,1000,1000,1000]) > > SharedBlob(0x561e440f7620 sbid 0x3e6b92e)) doesn't match expected ref_map > > use_tracker(0xa*0x1000 0x[\ > > 0,0,0,0,1000,1000,1000,1000,2000,2000]) > > fsck status: remaining 3 error(s) and warning(s) > > [root@ceph-osd-19 ~]# > > > > [root@ceph-osd-19 ~]# ceph-objectstore-tool --data-path > > /var/lib/ceph/osd/ceph-257 --op list | grep > > "rbd_data.44.f397cc4c21bbfb.0000000000000e66" > > > ["43.70s0",{"oid":"rbd_data.44.f397cc4c21bbfb.0000000000000e66","key":"","snapid":-2,"hash":2802327152,"max":0,"pool":43,"namespace":"","generation":47965127,"sh\ > > ard_id":0,"max":0}] > > > > > > > > ---------------------------------------- > > > > ---------------------------------------- > > > > [root@ceph-osd-20 ~]# ceph-bluestore-tool --path > /var/lib/ceph/osd/ceph-277 > > --op fsck > > 2026-09-16T22:08:04.530+0200 7f35041c8ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-277) fsck error: > > 3#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ > > dbe3c7 lextent at 0x6a000~6000 spans a shard boundary > > 2026-09-16T22:08:04.530+0200 7f35041c8ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-277) fsck error: > > 3#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ > > dbe3c7 lextent at 0x6e000 overlaps with the previous, which ends at > 0x70000 > > 2026-09-16T22:08:04.530+0200 7f35041c8ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-277) fsck error: > > 3#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ > > dbe3c7 blob Blob(0x55ead1a808f0 spanning 2 > > blob([!~3000,0x1754e9e3000~1000,0x1754eb48000~3000,0x1754e809000~2000] > > llen=0x9000 csum crc32c/0x1000/36) use_tracker(\ > > 0x9*0x1000 0x[0,0,0,1000,1000,1000,1000,1000,1000]) (shared_blob=NULL)) > > doesn't match expected ref_map use_tracker(0x9*0x1000 > > 0x[0,0,0,1000,1000,1000,1000,2000,2\ > > 000]) > > fsck status: remaining 3 error(s) and warning(s) > > [root@ceph-osd-20 ~]# > > > > > > root@ceph-osd-20 ~]# ceph-objectstore-tool --data-path > > /var/lib/ceph/osd/ceph-277 --op list | grep > > "rbd_data.44.f397cc4c21bbfb.0000000000000e66" > > > ["43.70s3",{"oid":"rbd_data.44.f397cc4c21bbfb.0000000000000e66","key":"","snapid":-2,"hash":2802327152,"max":0,"pool":43,"namespace":"","generation":47965127,"sh\ > > ard_id":3,"max":0}] > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

6 12

September Ceph Science User Group Virtual Meeting
by Enrico Bocchi 18 Sep '26

18 Sep '26

Dear all, We will be having a Ceph science/research/big cluster call next Tuesday, September 22nd, at 14:00 UTC. This time, it is my turn to tell a horror story about how an SSD firmware bug nearly killed our CephFS. The Ceph Science Virtual Meeting is an informal open call for community members primarily from hpc/htc/research/big cluster environments, though anyone is welcome, where we discuss whatever is on our minds regarding Ceph: Updates, outages, features, maintenance, ... There is no single set presenter for the length of the meeting, but it is somewhat moderated to keep the conversation flowing. If you wish to discuss something specific, please add it to the Pad linked below. If you are interested in giving a short talk during one of the next meetings, please reach out to me. Event Pad URL: https://pad.ceph.com/p/Ceph_Science_User_Group_20260922 Main Pad for the Ceph Science User Group: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Join the meeting at: https://meet.jit.si/ceph-science-wg Meetings will be recorded and posted to the Ceph Youtube channel. Virtual event details: September 22nd, 2026 14:00 UTC See the Ceph Community Calendar (https://ceph.io/en/community/meetups/) for the series information and for the meeting time in your timezone. See you there! Enrico

1 0

Help needed: multiple OSDs refuse to restart (and pg in down state)
by Massimo Sgaravatto 16 Sep '26

16 Sep '26

Dear all Today we had a problem in our ceph cluster: basically for some (still unknown) reasons, 5 OSDs (206, 233, 250, 257, 277) crashed and now they refuse to start. The main problem is that 1 pg is in down+remapped state because it is waiting for 3 of these OSDs [*] Doing a ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-xxx --op fsck, I see [**] that the problem is with 2 objects (rbd_data.44.f397cc4c21bbfb.000000000000319b and rbd_data.44.f397cc4c21bbfb.0000000000000e66 A "--op repair" is not able to fix the problem I am not sure how to proceed now Is the only option removing the problematic object from each OSD, doing e.g.: ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-206 --pgid 43.1b0 '{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":7151\ 31231,"max":0,"pool":43,"namespace":"","shardid":3}' remove ? Then the OSDs should hopefully restart, right ? Or are there better options ? Your help will be really appreciated ! Thanks, Massimo [*] ceph pg 43.1b0 query ... ... "blocked": "peering is blocked due to down osds", "down_osds_we_would_probe": [ 206, 233, 250 ], "peering_blocked_by": [ { "osd": 206, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 233, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 250, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" } ] [**] [root@ceph-osd-17 ~]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-206 --op fsck 2026-09-16T21:26:23.609+0200 7fb5aa3ddac0 -1 bluestore(/var/lib/ceph/osd/ceph-206) fsck error: 3#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a lextent at 0x6c000~4000 spans a shard boundary 2026-09-16T21:26:23.609+0200 7fb5aa3ddac0 -1 bluestore(/var/lib/ceph/osd/ceph-206) fsck error: 3#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a lextent at 0x6d000 overlaps with the previous, which ends at 0x70000 2026-09-16T21:26:23.609+0200 7fb5aa3ddac0 -1 bluestore(/var/lib/ceph/osd/ceph-206) fsck error: 3#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a blob Blob(0x5605e5760b60 spanning 1 blob([!~4000,0xa211d395000~1000,!~3000,0x23b97906000~1000,0x23b97226000~3000] llen=0xc000 csum+shared crc32c/0x1000/48\ ) use_tracker(0xc*0x1000 0x[0,0,0,0,1000,0,0,0,1000,1000,1000,1000]) SharedBlob(0x5606c99ebfc0 sbid 0x37ae23b)) doesn't match expected ref_map use_tracker(0xc*0x\ 1000 0x[0,0,0,0,1000,0,0,0,1000,2000,2000,2000]) fsck status: remaining 3 error(s) and warning(s) root@ceph-osd-17 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-206 --op list | grep "rbd_data.44.f397cc4c21bbfb.000000000000319b" ["43.1b0s3",{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":4210196912,"max":0,"pool":43,"namespace":"","generation":44630682,"s\ hard_id":3,"max":0}] ---------------------------------------- ---------------------------------------- [root@ceph-osd-18 ~]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-233 --op fsck 2026-09-16T21:37:09.598+0200 7f8ee4c49ac0 -1 bluestore(/var/lib/ceph/osd/ceph-233) fsck error: 4#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a lextent at 0x6c000~4000 spans a shard boundary 2026-09-16T21:37:09.598+0200 7f8ee4c49ac0 -1 bluestore(/var/lib/ceph/osd/ceph-233) fsck error: 4#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a lextent at 0x6d000 overlaps with the previous, which ends at 0x70000 2026-09-16T21:37:09.598+0200 7f8ee4c49ac0 -1 bluestore(/var/lib/ceph/osd/ceph-233) fsck error: 4#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a blob Blob(0x5557c8bef2b0 spanning 1 blob([!~4000,0x2ce88f4c000~1000,!~3000,0x2ce89560000~1000,0x2ce88a80000~3000] llen=0xc000 csum+shared crc32c/0x1000/48\ ) use_tracker(0xc*0x1000 0x[0,0,0,0,1000,0,0,0,1000,1000,1000,1000]) SharedBlob(0x5557fff8b1a0 sbid 0x323170b)) doesn't match expected ref_map use_tracker(0xc*0x\ 1000 0x[0,0,0,0,1000,0,0,0,1000,2000,2000,2000]) fsck status: remaining 3 error(s) and warning(s) [root@ceph-osd-18 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-233 --op list | grep "rbd_data.44.f397cc4c21bbfb.000000000000319b" ["43.1b0s4",{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":4210196912,"max":0,"pool":43,"namespace":"","generation":44630682,"s\ hard_id":4,"max":0}] [root@ceph-osd-18 ~]# ---------------------------------------- [root@ceph-osd-19 ~]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-250 --op fsck 2026-09-16T21:59:02.401+0200 7fa2e8795ac0 -1 bluestore(/var/lib/ceph/osd/ceph-250) fsck error: 0#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a lextent at 0x6c000~4000 spans a shard boundary 2026-09-16T21:59:02.401+0200 7fa2e8795ac0 -1 bluestore(/var/lib/ceph/osd/ceph-250) fsck error: 0#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a lextent at 0x6d000 overlaps with the previous, which ends at 0x70000 2026-09-16T21:59:02.401+0200 7fa2e8795ac0 -1 bluestore(/var/lib/ceph/osd/ceph-250) fsck error: 0#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a blob Blob(0x556a29cc4820 spanning 1 blob([!~4000,0x464866d9000~1000,!~3000,0x4648731f000~1000,0x46485e09000~3000] llen=0xc000 csum+shared crc32c/0x1000/48\ ) use_tracker(0xc*0x1000 0x[0,0,0,0,1000,0,0,0,1000,1000,1000,1000]) SharedBlob(0x556a1fb84e40 sbid 0x98f249)) doesn't match expected ref_map use_tracker(0xc*0x1\ 000 0x[0,0,0,0,1000,0,0,0,1000,2000,2000,2000]) fsck status: remaining 3 error(s) and warning(s) [root@ceph-osd-19 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-250 --op list | grep "rbd_data.44.f397cc4c21bbfb.000000000000319b" ["43.1b0s0",{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":4210196912,"max":0,"pool":43,"namespace":"","generation":44630682,"s\ hard_id":0,"max":0}] ---------------------------------------- [root@ceph-osd-19 ~]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-257 --op fsck 2026-09-16T22:06:09.627+0200 7fded8db6ac0 -1 bluestore(/var/lib/ceph/osd/ceph-257) fsck error: 0#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ dbe3c7 lextent at 0x6a000~6000 spans a shard boundary 2026-09-16T22:06:09.627+0200 7fded8db6ac0 -1 bluestore(/var/lib/ceph/osd/ceph-257) fsck error: 0#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ dbe3c7 lextent at 0x6e000 overlaps with the previous, which ends at 0x70000 2026-09-16T22:06:09.627+0200 7fded8db6ac0 -1 bluestore(/var/lib/ceph/osd/ceph-257) fsck error: 0#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ dbe3c7 blob Blob(0x561e442541a0 spanning 2 blob([!~4000,0x19c87d9b000~1000,0x46e6508e000~3000,0x46e64693000~2000] llen=0xa000 csum+shared crc32c/0x1000/40) use_t\ racker(0xa*0x1000 0x[0,0,0,0,1000,1000,1000,1000,1000,1000]) SharedBlob(0x561e440f7620 sbid 0x3e6b92e)) doesn't match expected ref_map use_tracker(0xa*0x1000 0x[\ 0,0,0,0,1000,1000,1000,1000,2000,2000]) fsck status: remaining 3 error(s) and warning(s) [root@ceph-osd-19 ~]# [root@ceph-osd-19 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-257 --op list | grep "rbd_data.44.f397cc4c21bbfb.0000000000000e66" ["43.70s0",{"oid":"rbd_data.44.f397cc4c21bbfb.0000000000000e66","key":"","snapid":-2,"hash":2802327152,"max":0,"pool":43,"namespace":"","generation":47965127,"sh\ ard_id":0,"max":0}] ---------------------------------------- ---------------------------------------- [root@ceph-osd-20 ~]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-277 --op fsck 2026-09-16T22:08:04.530+0200 7f35041c8ac0 -1 bluestore(/var/lib/ceph/osd/ceph-277) fsck error: 3#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ dbe3c7 lextent at 0x6a000~6000 spans a shard boundary 2026-09-16T22:08:04.530+0200 7f35041c8ac0 -1 bluestore(/var/lib/ceph/osd/ceph-277) fsck error: 3#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ dbe3c7 lextent at 0x6e000 overlaps with the previous, which ends at 0x70000 2026-09-16T22:08:04.530+0200 7f35041c8ac0 -1 bluestore(/var/lib/ceph/osd/ceph-277) fsck error: 3#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ dbe3c7 blob Blob(0x55ead1a808f0 spanning 2 blob([!~3000,0x1754e9e3000~1000,0x1754eb48000~3000,0x1754e809000~2000] llen=0x9000 csum crc32c/0x1000/36) use_tracker(\ 0x9*0x1000 0x[0,0,0,1000,1000,1000,1000,1000,1000]) (shared_blob=NULL)) doesn't match expected ref_map use_tracker(0x9*0x1000 0x[0,0,0,1000,1000,1000,1000,2000,2\ 000]) fsck status: remaining 3 error(s) and warning(s) [root@ceph-osd-20 ~]# root@ceph-osd-20 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-277 --op list | grep "rbd_data.44.f397cc4c21bbfb.0000000000000e66" ["43.70s3",{"oid":"rbd_data.44.f397cc4c21bbfb.0000000000000e66","key":"","snapid":-2,"hash":2802327152,"max":0,"pool":43,"namespace":"","generation":47965127,"sh\ ard_id":3,"max":0}]

1 0

User Experience upgrading Hyperconverged CEPH Quincy on Proxmox 8.4 to Proxmox 9.x and CEPH tentacle.
by Tino Todino 15 Sep '26

15 Sep '26

Hi folks, We have a 7 node Hyperconverged CEPH Quincy cluster with 3 OSD's per host. So small fry really 🙂 We are upgrading our backbone from 10 to 25Gbe and as part of the process I'd like to upgrade the nodes from Promxox 8.4 to the latest version 9.2 (iirc). As part of this, CEPH needs to be upgraded. I was wondering if anyone has done this before and what the experience was like please? So, gotcha's, learning points, must do's, never do's etc. If no-one has tried this, I guess I'll be the one updating the mailing list with the findings, but that won't be until November most likely. 🙂 Thanks Tino Coastsense Ltd This E-mail is intended solely for the person or organisation to which it is addressed. It may contain privileged or confidential information and, if you are not the intended recipient, you must not copy, distribute or take any action in reliance upon it. Any views or opinions presented are solely those of the author and do not necessarily represent those of Marlan Maritime Technologies Ltd. If you have received this E-mail in error, please notify us as soon as possible and delete it from your computer. Marlan Maritime Technologies Ltd Registered in England & Wales 323 Mariners House, Norfolk Street, Liverpool. L1 0BG Company No. 08492427.

3 2

Experience with SEDs in Ceph clusters
by Serhat Rıfat Demircan 15 Sep '26

15 Sep '26

Hi all, Is anyone using Self-Encrypting Drives (SEDs) in production Ceph clusters? I’d be interested to hear about your experience, especially any operational issues, compatibility problems, performance impact, or unexpected behavior you have encountered. Also, if there are any Ceph-specific considerations or recommendations when using SEDs, I’d appreciate your feedback. Thanks, Serhat

2 2

Activity Sent From Your Own Account
by ceph-users-owner＠ceph.io 14 Sep '26

14 Sep '26

Your message to the ceph-users mailing-list was rejected for the following reasons: The message is not from a list member The original message as received by Mailman is attached.

1 0

Re: Tentacle 20.2.4: OSDs crash in PGLog::merge_log after monitor was rebuilt with --mkfs
by Eugen Block 14 Sep '26

14 Sep '26

Please don't drop the list off your responses. I'm not entirely sure yet what to make of the epochs. I'm thinking about mon_min_osdmap_epochs (default 500) and similar parameters. But I'm heading out on vacation now, hopefully someone else will chime in. Good luck! Eugen Zitat von Martin Konold <martin.konold(a)konsec.com>: > Hi Eugen, > > thanks for the quick reply -- one clarification: the monitor was > already rebuilt with --mkfs yesterday evening, before I posted today. > ceph-pve-07.bak is the leftover original store, not something I > am proposing to move. So the question is whether restoring it is > the right way back, not whether to abandon the current one. > >> Did you verify that all service keyrings were successfully >> rotated after the upgrade? > > No, and that turned out to be the core of the auth problem. The > rotation touched the mon DB but not the bluestore osd_key labels: > > osd.2 mon DB AgBf7qJq... label/tmpfs AgA8cKJq... > osd.5 mon DB AgBg7qJq... label/tmpfs AgA8cKJq... > > osd.0, osd.3 and osd.9 are still in that state -- they run fine > on existing sessions but hold pre-rotation keys in their labels, > so they would not come back after a restart. set-label-key > refuses to write while the OSD is up, so I cannot fix that > pre-emptively. > > Caps were inconsistent too: osd.0 and osd.3 have > "caps mon = allow rwx" (the pre-profile form), osd.1 and osd.9 > were missing "caps mgr" entirely. > >> You could have allowed the insecure cipher again > > I did, that was the first thing after finding the monmap policy: > > ceph mon set auth_allowed_ciphers aes,aes256k > > That fixed authentication -- osd.2 and osd.5 now get through and > reach peering. client.admin was fine throughout, so the command > went through without trouble. > > What remains is the PGLog assert, and it is not an auth problem. > Checking the timestamps: osd.5 first hit the assert at 21:04:21, > while its monclient log at 21:04:15 shows a healthy session > (wait_auth_rotating done, uptodate secrets). The auth failures > only start at 21:24. So the crashes came first; the cipher > lockout landed on top and obscured them. > > The assert shows an empty authoritative log against a populated > local one: > > merge_log log((573038'28581091,573038'28581091], > crt=573038'28581091) from osd.3 > into log((573040'28581121,573042'28581681], > crt=573042'28581681) > > Also worth noting: at 21:04 osd.2 had to re-add itself to the > CRUSH map ("osd crush create-or-move id 2 weight 7.2774 > host=pve-08"), which suggests the rebuilt store did not know it. > My working assumption is that the incomplete CRUSH map during > that window let writes be acked against acting sets that later > could not reconcile. > > So the open question stands: with the original store sitting at > osdmap 573036 and the divergence starting at 573038, is a > restore the right approach, and how should the ~500 epoch gap to > the OSDs be handled? > > root@pve-07:~# ceph osd tree > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > -1 43.66434 root default > -61 14.55478 host pve-07 > 0 nvme 7.27739 osd.0 up 1.00000 1.00000 > 3 nvme 7.27739 osd.3 up 1.00000 1.00000 > -66 14.55478 host pve-08 > 2 nvme 7.27739 osd.2 down 1.00000 1.00000 > 5 nvme 7.27739 osd.5 down 1.00000 1.00000 > -71 14.55478 host pve-09 > 1 nvme 7.27739 osd.1 down 1.00000 1.00000 > 9 nvme 7.27739 osd.9 up 1.00000 1.00000 > > > Kind Regards, > --martin > > Am Freitag, September 11, 2026 10:56 CEST, schrieb Eugen Block > <ceph-users(a)ceph.io>: > >> Hi, >> >> that seems a little drastic to move the mon store. Did you verify that >> all service keyrings (mgr,mon,osd,mds) were successfully rotated after >> the upgrade? >> >> You could have allowed the insecure cipher again to see if that >> resolves the issue and finish the upgrade (if steps were missing). >> >> ceph mon set auth_allowed_ciphers aes,aes256k >> >> I assume you rotated the admin keyring as well so changing >> auth_allowed_ciphers would be possible? >> >> Zitat von Martin Konold <ceph-users(a)ceph.io>: >> >> > Hi, >> > >> > Yesterday we got an update via Proxmox. The initial update went well >> > but hell broke loose when I tried to dissallow the old aes >> > authentification in order top get rid of a warning. >> > >> > I'm looking for guidance before taking any further destructive >> > action on a 3-node Proxmox cluster (6 OSDs, one replicated pool >> > 3/2, ~2.5 TiB). No users on it, no time pressure. >> > >> > Timeline >> > -------- >> > 10.09 10:59 monmap epoch 81 shows an inconsistent cipher policy: >> > auth_allowed_ciphers aes256k >> > auth_preferred_cipher aes >> > New keys are created as aes but aes is not allowed >> > to authenticate. Any OSD restarted after this point >> > was locked out. >> > 10.09 15:20 last good osdmap in the original mon store: epoch >> > 573036, all 6 OSDs up/in. >> > 10.09 ~16:32 original mon store moved aside (now ceph-pve-07.bak, >> > 72 MB, 26k keys, intact). >> > 10.09 20:38 monitor rebuilt with monmaptool/--mkfs. New store has >> > 56 auth keys vs 128 in the original, 28 mon_config_key >> > vs 1162, and caps like "mon = allow rwx" on osd.0/3 -- >> > clearly a reconstruction. >> > 10.09 21:04 osd.2 and osd.5 crash in PGLog::merge_log. Auth was >> > still working at this point (wait_auth_rotating done, >> > and osd.2 had to re-add itself via "osd crush >> > create-or-move"). >> > 10.09 21:24 auth failures start on top of the crashes (handle_auth >> > _bad_method). Resolved since: mon key was rotated >> > without updating the bluestore osd_key label. >> > >> > Current state >> > ------------- >> > 3 of 6 OSDs (0, 3, 9) are up and running on pre-existing cephx >> > sessions. osd.1, osd.2, osd.5 crash during peering: >> > >> > ./src/osd/PGLog.cc: 401: FAILED ceph_assert( >> > log.head >= olog.tail && olog.head >= log.tail) >> > PeeringState::Stray::react(MLogRec const&) -> merge_log >> > >> > The merge_log lines show an *empty* authoritative log being sent >> > against a populated local one: >> > >> > merge_log log((573038'28581091,573038'28581091], >> > crt=573038'28581091) from osd.3 >> > into log((573040'28581121,573042'28581681], >> > crt=573042'28581681) >> > >> > So the surviving OSDs are behind, and the crashing ones hold log >> > entries from epochs 573038-573042 that the primary has no record >> > of. That range starts two epochs after the last osdmap in the >> > original store. >> > >> > original mon store max osdmap 573036 >> > divergent pg log entries 573038-573042 >> > current osdmap (after restarts) 573538 >> > >> > ceph -s (before I stopped the crashing OSDs): 278/708728 objects >> > unfound, 40 pgs recovery_unfound, 11 incomplete, 11 down. >> > >> > Questions >> > --------- >> > 1. Is restoring the original mon store the right approach here? >> > It is only 2 epochs behind where the divergence starts, which >> > seems like the best possible starting point. >> > 2. How should the 502-epoch gap be handled -- the OSDs have >> > osdmaps far newer than the restored monitor would have? >> > 3. Is the assert itself worth a tracker issue? An empty >> > authoritative log against a populated local one looks like a >> > case that should fall back to backfill rather than assert. >> > >> > What I have NOT done: no mark_unfound_lost, no min_size 1, no >> > ceph-objectstore-tool remove. All three mon stores >> > are backed up. The original store is untouched. >> > >> > One more constraint: the bluestore osd_key labels on the three >> > running OSDs (0, 3, 9) are stale -- they hold pre-rotation keys. >> > They run fine now but would not come back after a restart, and >> > set-label-key refuses to write while the OSD is up. So any plan >> > involving an OSD restart has to fix the label in the same stop/ >> > start window. >> > >> > Happy to provide full logs with debug_osd=20, ceph pg query >> > output, or monstore dumps. >> > >> > Thanks, >> > --martin >> > >> > -- >> > martin.konold(a)konsec.com >> > Phone: +49 711 21 72 74-90 >> > KONSEC GmbH -⁠ make things real >> > Amtsgericht Stuttgart, HRB 23690 >> > Geschäftsführer: Andreas Mack >> > Im Köller 3, 70794 Filderstadt, Germany >> > >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > martin.konold(a)konsec.com > Phone: +49 711 21 72 74-90 > KONSEC GmbH -⁠ make things real > Amtsgericht Stuttgart, HRB 23690 > Geschäftsführer: Andreas Mack > Im Köller 3, 70794 Filderstadt, Germany

2 1

Tentacle: Emergency Allowed Ciphers
by Eugen Block 11 Sep '26

11 Sep '26

Hi, I was trying to use the mon_auth_emergency_allowed_ciphers setting to rescue a test cluster as described in the docs [0]. It's a cephadm managed cluster where I set auth_allowed_ciphers to aes256k before rotating the admin keyring. Then I added mon_auth_emergency_allowed_ciphers = true to all mon's config files: reef1:~ # vi /var/lib/ceph/{FSID}/mon.reef1/config and restarted the mon services. According to the config dump the setting is active (for all 3 daemons): [ceph: root@reef2 /]# ceph daemon mon.reef2 config show | grep cipher "mon_auth_emergency_allowed_ciphers": "true", The daemon logs this during startup but I'm not sure if it's relevant right now: Sep 11 10:25:08 reef2 ceph-{FSID}-mon-reef2[88175]: 2026-09-11T08:25:08.611+0000 7f0a8765bd80 -1 mon.reef2@-1(???) e6 init: invalid cipher: true I can't use the unrotated admin keyring although the docs state that I should be able to: > This will allow your existing client.admin or other administrative > key to authenticate as normal. What am I missing? Thanks. Eugen [0] https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h…

3 4

CFPs Reminder
by Anthony Middleton 11 Sep '26

11 Sep '26

Happy Friday, Ceph friends! Just a reminder that the CFPs for Ceph Days Zürich <https://docs.google.com/forms/d/e/1FAIpQLSfUAulFnfdrHesAi92Y2Dkkm7whEXQBel1…> and Ceph Meetup Melbourne <https://docs.google.com/forms/d/e/1FAIpQLSdDgJGT1AyhJFaXPrjvx0X9lYSasU1Iju8…> are still open. Please let me know if you have any questions, and I can connect you with the event organizers. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0