September 2026 - ceph-users

Re: Help needed: multiple OSDs refuse to restart (and pg in down state)
by Massimo Sgaravatto 20 Sep '26

20 Sep '26

Hi Anthony, all I am running ceph squid 19.2.3 I copied the relevant part of the log (from the start till the crash) for a OSD (osd.206) in https://cernbox.cern.ch/s/56sdGTQsotbN2XE So as far as I can understand it crashes because of: 2026-09-16T21:16:06.859+0200 7f5431ee1640 4 rocksdb: [db/compaction/compaction_job.cc:1588] [O-2] [JOB 9] Generated table #148643: 151504 keys, 67943642 bytes, temperature: kUnknown 2026-09-16T21:16:06.859+0200 7f5431ee1640 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1789586166860678, "cf_name": "O-2", "job": 9, "event": "table_file_creation", "file_number": 148643, "file_size": 67943642, "file_checksum": "", "f\ ile_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 67111852, "index_size": 1538071, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_v\ alue_is_delta_encoded": 1, "filter_size": 378821, "raw_key_size": 12956410, "raw_average_key_size": 85, "raw_value_size": 67640956, "raw_average_value_size": 446, "num_data_blocks": 18954, "num_entries": 151504, "num_filter_entrie\ s": 151504, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "O-2", "column_family_id": 9, "comparator": "leveldb\ .BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "LZ4", "compression_options": "window_bits=-14; level=32767; strategy=0; \ max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1778598719, "oldest_key_time": 0, "file_creation_time": 1789586166, "slow_compression_estimated_data_size":\ 0, "fast_compression_estimated_data_size": 0, "db_id": "1103608a-90ce-4939-82c0-de96ff245167", "db_session_id": "FO9J42Q67M9AK232DQXE", "orig_file_number": 148643, "seqno_to_time_mapping": "N/A"}} 2026-09-16T21:16:07.421+0200 7f541fcaf640 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.3/rpm/el9/BUILD/ceph-19.2.3/src/o\ s/bluestore/bluestore_types.cc: In function 'bool bluestore_blob_use_tracker_t::put(uint32_t, uint32_t, PExtentVector*)' thread 7f541fcaf640 time 2026-09-16T21:16:07.418902+0200 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.3/rpm/el9/BUILD/ceph-19.2.3/src/os/bluestore/bluestore_types.cc: 511: FAILED c\ eph_assert(diff <= bytes_per_au[pos]) ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x113) [0x560cc2d4c85d] 2: /usr/bin/ceph-osd(+0x401a14) [0x560cc2d4ca14] 3: /usr/bin/ceph-osd(+0x3f2930) [0x560cc2d3d930] 4: (BlueStore::Blob::put_ref(BlueStore::Collection*, unsigned int, unsigned int, std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)5, bluestore_pextent_t> >*)+0xaa) [0x560cc32bb2fa] 5: (BlueStore::OldExtent::create(boost::intrusive_ptr<BlueStore::Collection>, unsigned int, unsigned int, unsigned int, boost::intrusive_ptr<BlueStore::Blob>&)+0x11d) [0x560cc32bf47d] 6: (BlueStore::ExtentMap::punch_hole(boost::intrusive_ptr<BlueStore::Collection>&, unsigned long, unsigned long, boost::intrusive::list<BlueStore::OldExtent, boost::intrusive::member_hook<BlueStore::OldExtent, boost::intrusive::l\ ist_member_hook<>, &BlueStore::OldExtent::old_extent_item> >*)+0x451) [0x560cc32cbe21] 7: (BlueStore::_do_truncate(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&, unsigned long, std::set<BlueStore::SharedBlob*, std::less<BlueStore::SharedBlob*>, std::\ allocator<BlueStore::SharedBlob*> >*)+0x205) [0x560cc33483f5] 8: (BlueStore::_do_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&)+0xc1) [0x560cc334d781] 9: (BlueStore::_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&)+0x7f) [0x560cc334f0af] 10: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x12a6) [0x560cc3334206] 11: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x2f3) [0\ x560cc3335123] 12: /usr/bin/ceph-osd(+0x5239a8) [0x560cc2e6e9a8] 13: (OSD::dispatch_context(PeeringCtx&, PG*, std::shared_ptr<OSDMap const>, ThreadPool::TPHandle*)+0x115) [0x560cc2edf495] 14: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x2bc) [0x560cc2eea72c] 15: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x560cc31359b1] 16: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xcd0) [0x560cc2f04b70] 17: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x560cc34238aa] 18: /usr/bin/ceph-osd(+0xad8e64) [0x560cc3423e64] 19: /lib64/libc.so.6(+0x8b2ea) [0x7f543ce8b2ea] 20: /lib64/libc.so.6(+0x1103d0) [0x7f543cf103d0] 2026-09-16T21:16:07.426+0200 7f541fcaf640 -1 *** Caught signal (Aborted) ** in thread 7f541fcaf640 thread_name:tp_osd_tp ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) 1: /lib64/libc.so.6(+0x3fc30) [0x7f543ce3fc30] 2: /lib64/libc.so.6(+0x8d02c) [0x7f543ce8d02c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x169) [0x560cc2d4c8b3] 6: /usr/bin/ceph-osd(+0x401a14) [0x560cc2d4ca14] 7: /usr/bin/ceph-osd(+0x3f2930) [0x560cc2d3d930] 8: (BlueStore::Blob::put_ref(BlueStore::Collection*, unsigned int, unsigned int, std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)5, bluestore_pextent_t> >*)+0xaa) [0x560cc32bb2fa] 9: (BlueStore::OldExtent::create(boost::intrusive_ptr<BlueStore::Collection>, unsigned int, unsigned int, unsigned int, boost::intrusive_ptr<BlueStore::Blob>&)+0x11d) [0x560cc32bf47d] 10: (BlueStore::ExtentMap::punch_hole(boost::intrusive_ptr<BlueStore::Collection>&, unsigned long, unsigned long, boost::intrusive::list<BlueStore::OldExtent, boost::intrusive::member_hook<BlueStore::OldExtent, boost::intrusive::\ list_member_hook<>, &BlueStore::OldExtent::old_extent_item> >*)+0x451) [0x560cc32cbe21] 11: (BlueStore::_do_truncate(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&, unsigned long, std::set<BlueStore::SharedBlob*, std::less<BlueStore::SharedBlob*>, std:\ :allocator<BlueStore::SharedBlob*> >*)+0x205) [0x560cc33483f5] 12: (BlueStore::_do_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&)+0xc1) [0x560cc334d781] 13: (BlueStore::_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&)+0x7f) [0x560cc334f0af] 14: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x12a6) [0x560cc3334206] 15: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x2f3) [0\ x560cc3335123] 16: /usr/bin/ceph-osd(+0x5239a8) [0x560cc2e6e9a8] 17: (OSD::dispatch_context(PeeringCtx&, PG*, std::shared_ptr<OSDMap const>, ThreadPool::TPHandle*)+0x115) [0x560cc2edf495] 18: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x2bc) [0x560cc2eea72c] 19: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x560cc31359b1] 20: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xcd0) [0x560cc2f04b70] 21: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x560cc34238aa] 22: /usr/bin/ceph-osd(+0xad8e64) [0x560cc3423e64] 23: /lib64/libc.so.6(+0x8b2ea) [0x7f543ce8b2ea] 24: /lib64/libc.so.6(+0x1103d0) [0x7f543cf103d0] Thanks, Massimo On Thu, Sep 17, 2026 at 4:44 AM Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > Start with what they log when they try to start. > > And what release you’re running. > > > On Sep 16, 2026, at 4:47 PM, Massimo Sgaravatto <ceph-users(a)ceph.io> > wrote: > > > > Dear all > > Today we had a problem in our ceph cluster: basically for some (still > > unknown) reasons, 5 OSDs (206, 233, 250, 257, 277) crashed and now they > > refuse to start. > > > > > > The main problem is that 1 pg is in down+remapped state because it is > > waiting for 3 of these OSDs [*] > > > > > > Doing a ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-xxx --op fsck, > I > > see [**] that the problem is with 2 objects > > (rbd_data.44.f397cc4c21bbfb.000000000000319b > > and rbd_data.44.f397cc4c21bbfb.0000000000000e66 > > A "--op repair" is not able to fix the problem > > > > > > I am not sure how to proceed now > > > > Is the only option removing the problematic object from each OSD, doing > > e.g.: > > > > > > ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-206 --pgid > 43.1b0 > > > '{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":7151\ > > 31231,"max":0,"pool":43,"namespace":"","shardid":3}' remove > > > > ? > > > > Then the OSDs should hopefully restart, right ? > > > > Or are there better options ? > > > > Your help will be really appreciated ! > > > > > > Thanks, Massimo > > > > [*] > > ceph pg 43.1b0 query > > ... > > ... > > > > "blocked": "peering is blocked due to down osds", > > "down_osds_we_would_probe": [ > > 206, > > 233, > > 250 > > ], > > "peering_blocked_by": [ > > { > > "osd": 206, > > "current_lost_at": 0, > > "comment": "starting or marking this osd lost may let > > us proceed" > > }, > > { > > "osd": 233, > > "current_lost_at": 0, > > "comment": "starting or marking this osd lost may let > > us proceed" > > }, > > { > > "osd": 250, > > "current_lost_at": 0, > > "comment": "starting or marking this osd lost may let > > us proceed" > > } > > ] > > > > [**] > > > > [root@ceph-osd-17 ~]# ceph-bluestore-tool --path > /var/lib/ceph/osd/ceph-206 > > --op fsck > > 2026-09-16T21:26:23.609+0200 7fb5aa3ddac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-206) fsck error: > > 3#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a lextent at 0x6c000~4000 spans a shard boundary > > 2026-09-16T21:26:23.609+0200 7fb5aa3ddac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-206) fsck error: > > 3#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a lextent at 0x6d000 overlaps with the previous, which ends at > 0x70000 > > 2026-09-16T21:26:23.609+0200 7fb5aa3ddac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-206) fsck error: > > 3#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a blob Blob(0x5605e5760b60 spanning 1 > > > blob([!~4000,0xa211d395000~1000,!~3000,0x23b97906000~1000,0x23b97226000~3000] > > llen=0xc000 csum+shared crc32c/0x1000/48\ > > ) use_tracker(0xc*0x1000 0x[0,0,0,0,1000,0,0,0,1000,1000,1000,1000]) > > SharedBlob(0x5606c99ebfc0 sbid 0x37ae23b)) doesn't match expected ref_map > > use_tracker(0xc*0x\ > > 1000 0x[0,0,0,0,1000,0,0,0,1000,2000,2000,2000]) > > fsck status: remaining 3 error(s) and warning(s) > > > > root@ceph-osd-17 ~]# ceph-objectstore-tool --data-path > > /var/lib/ceph/osd/ceph-206 --op list | grep > > "rbd_data.44.f397cc4c21bbfb.000000000000319b" > > > ["43.1b0s3",{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":4210196912,"max":0,"pool":43,"namespace":"","generation":44630682,"s\ > > hard_id":3,"max":0}] > > > > > > ---------------------------------------- > > ---------------------------------------- > > > > [root@ceph-osd-18 ~]# ceph-bluestore-tool --path > /var/lib/ceph/osd/ceph-233 > > --op fsck > > 2026-09-16T21:37:09.598+0200 7f8ee4c49ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-233) fsck error: > > 4#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a lextent at 0x6c000~4000 spans a shard boundary > > 2026-09-16T21:37:09.598+0200 7f8ee4c49ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-233) fsck error: > > 4#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a lextent at 0x6d000 overlaps with the previous, which ends at > 0x70000 > > 2026-09-16T21:37:09.598+0200 7f8ee4c49ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-233) fsck error: > > 4#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a blob Blob(0x5557c8bef2b0 spanning 1 > > > blob([!~4000,0x2ce88f4c000~1000,!~3000,0x2ce89560000~1000,0x2ce88a80000~3000] > > llen=0xc000 csum+shared crc32c/0x1000/48\ > > ) use_tracker(0xc*0x1000 0x[0,0,0,0,1000,0,0,0,1000,1000,1000,1000]) > > SharedBlob(0x5557fff8b1a0 sbid 0x323170b)) doesn't match expected ref_map > > use_tracker(0xc*0x\ > > 1000 0x[0,0,0,0,1000,0,0,0,1000,2000,2000,2000]) > > fsck status: remaining 3 error(s) and warning(s) > > > > [root@ceph-osd-18 ~]# ceph-objectstore-tool --data-path > > /var/lib/ceph/osd/ceph-233 --op list | grep > > "rbd_data.44.f397cc4c21bbfb.000000000000319b" > > > ["43.1b0s4",{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":4210196912,"max":0,"pool":43,"namespace":"","generation":44630682,"s\ > > hard_id":4,"max":0}] > > [root@ceph-osd-18 ~]# > > > > > > ---------------------------------------- > > > > > > [root@ceph-osd-19 ~]# ceph-bluestore-tool --path > /var/lib/ceph/osd/ceph-250 > > --op fsck > > 2026-09-16T21:59:02.401+0200 7fa2e8795ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-250) fsck error: > > 0#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a lextent at 0x6c000~4000 spans a shard boundary > > 2026-09-16T21:59:02.401+0200 7fa2e8795ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-250) fsck error: > > 0#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a lextent at 0x6d000 overlaps with the previous, which ends at > 0x70000 > > 2026-09-16T21:59:02.401+0200 7fa2e8795ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-250) fsck error: > > 0#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ > > a9029a blob Blob(0x556a29cc4820 spanning 1 > > > blob([!~4000,0x464866d9000~1000,!~3000,0x4648731f000~1000,0x46485e09000~3000] > > llen=0xc000 csum+shared crc32c/0x1000/48\ > > ) use_tracker(0xc*0x1000 0x[0,0,0,0,1000,0,0,0,1000,1000,1000,1000]) > > SharedBlob(0x556a1fb84e40 sbid 0x98f249)) doesn't match expected ref_map > > use_tracker(0xc*0x1\ > > 000 0x[0,0,0,0,1000,0,0,0,1000,2000,2000,2000]) > > fsck status: remaining 3 error(s) and warning(s) > > > > [root@ceph-osd-19 ~]# ceph-objectstore-tool --data-path > > /var/lib/ceph/osd/ceph-250 --op list | grep > > "rbd_data.44.f397cc4c21bbfb.000000000000319b" > > > ["43.1b0s0",{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":4210196912,"max":0,"pool":43,"namespace":"","generation":44630682,"s\ > > hard_id":0,"max":0}] > > ---------------------------------------- > > > > [root@ceph-osd-19 ~]# ceph-bluestore-tool --path > /var/lib/ceph/osd/ceph-257 > > --op fsck > > 2026-09-16T22:06:09.627+0200 7fded8db6ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-257) fsck error: > > 0#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ > > dbe3c7 lextent at 0x6a000~6000 spans a shard boundary > > 2026-09-16T22:06:09.627+0200 7fded8db6ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-257) fsck error: > > 0#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ > > dbe3c7 lextent at 0x6e000 overlaps with the previous, which ends at > 0x70000 > > 2026-09-16T22:06:09.627+0200 7fded8db6ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-257) fsck error: > > 0#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ > > dbe3c7 blob Blob(0x561e442541a0 spanning 2 > > blob([!~4000,0x19c87d9b000~1000,0x46e6508e000~3000,0x46e64693000~2000] > > llen=0xa000 csum+shared crc32c/0x1000/40) use_t\ > > racker(0xa*0x1000 0x[0,0,0,0,1000,1000,1000,1000,1000,1000]) > > SharedBlob(0x561e440f7620 sbid 0x3e6b92e)) doesn't match expected ref_map > > use_tracker(0xa*0x1000 0x[\ > > 0,0,0,0,1000,1000,1000,1000,2000,2000]) > > fsck status: remaining 3 error(s) and warning(s) > > [root@ceph-osd-19 ~]# > > > > [root@ceph-osd-19 ~]# ceph-objectstore-tool --data-path > > /var/lib/ceph/osd/ceph-257 --op list | grep > > "rbd_data.44.f397cc4c21bbfb.0000000000000e66" > > > ["43.70s0",{"oid":"rbd_data.44.f397cc4c21bbfb.0000000000000e66","key":"","snapid":-2,"hash":2802327152,"max":0,"pool":43,"namespace":"","generation":47965127,"sh\ > > ard_id":0,"max":0}] > > > > > > > > ---------------------------------------- > > > > ---------------------------------------- > > > > [root@ceph-osd-20 ~]# ceph-bluestore-tool --path > /var/lib/ceph/osd/ceph-277 > > --op fsck > > 2026-09-16T22:08:04.530+0200 7f35041c8ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-277) fsck error: > > 3#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ > > dbe3c7 lextent at 0x6a000~6000 spans a shard boundary > > 2026-09-16T22:08:04.530+0200 7f35041c8ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-277) fsck error: > > 3#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ > > dbe3c7 lextent at 0x6e000 overlaps with the previous, which ends at > 0x70000 > > 2026-09-16T22:08:04.530+0200 7f35041c8ac0 -1 > > bluestore(/var/lib/ceph/osd/ceph-277) fsck error: > > 3#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ > > dbe3c7 blob Blob(0x55ead1a808f0 spanning 2 > > blob([!~3000,0x1754e9e3000~1000,0x1754eb48000~3000,0x1754e809000~2000] > > llen=0x9000 csum crc32c/0x1000/36) use_tracker(\ > > 0x9*0x1000 0x[0,0,0,1000,1000,1000,1000,1000,1000]) (shared_blob=NULL)) > > doesn't match expected ref_map use_tracker(0x9*0x1000 > > 0x[0,0,0,1000,1000,1000,1000,2000,2\ > > 000]) > > fsck status: remaining 3 error(s) and warning(s) > > [root@ceph-osd-20 ~]# > > > > > > root@ceph-osd-20 ~]# ceph-objectstore-tool --data-path > > /var/lib/ceph/osd/ceph-277 --op list | grep > > "rbd_data.44.f397cc4c21bbfb.0000000000000e66" > > > ["43.70s3",{"oid":"rbd_data.44.f397cc4c21bbfb.0000000000000e66","key":"","snapid":-2,"hash":2802327152,"max":0,"pool":43,"namespace":"","generation":47965127,"sh\ > > ard_id":3,"max":0}] > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

6 12

Rotating cephx client keys in openstack VMs
by Bartosz Rabiega 19 Sep '26

19 Sep '26

Hello, I'm researching a way to rotate cephx client keys to aes256k 100% online for openstack VMs (qemu) with ceph rbd volumes attached (librbd). The problem ----------- Unfortunately I don't see any reasonable way to do it. Obviously I might be missing something (or a lot of something) Usually libvirt configures VM disk with following auth: <disk ...> <auth username='cephxuser'> <secret type='ceph' uuid='id-of-libvirt-secret'/> </auth> </disk> The cephx key is stored in a libvirt secret, it is read when VM starts and then stored in RAM for a VM lifetime (librbd AFAIK). If we rotate cephxuser key the living VMs will eventually stop working - when either auth_mon_ticket_ttl is exhausted or some connection to mon needs to be re-established librbd will try to authenticate with old key and fail. Potential solutions ------------------- A - rotate key, and distribute it as the same libvirt secret: - running VMs will eventually stop working - hard reboot or live migration shall fix them - stopped (SHUTOFF) VMs will be OK when started - new VMs and new attachments will be OK B - use new user and new key - new libvirt secret - running VMs will be OK, as long as we keep old user and key, live migration will probably fix the VM and use new user and new key - stopped (SHUTOFF) VMs will be OK when started, but will fail when old user/key is removed. username and libvirt secret uuid is kept in nova database and nothing will update it - new VMs and new attachments will be OK C - use the same user and have 2 keys for it - old and new. Eventually remove old one - running VMs will be OK, they will switch to new key during live migration - stopped (SHUTOFF) VMs will be OK, libvirt secret points to the new key so when VM is started it will use it. User is the same so no DB update is required - new VMs and new attachments will be OK A - not acceptable, breaks running VMs B - very difficult, requires a lot of VM migration, requires a lot of work to fix SHELVED/SHUTOFF VMs (no tooling for that AFAIK?) C - looks the best but not possible, ceph doesn't allow multiple keys per single user Summary ------- If my understanding is correct the effort to rotate the keys for openstack is tremendous and requires nova/cinder DB modification to fix SHUTOFF vms. Probably SHELVED vms will have the same problem. If you reached this point, thanks for reading :) Any ideas or comments? Bartosz Rabiega

2 1

We dodged a bullet
by Stefan Kooman 18 Sep '26

18 Sep '26

Hi, Yeah, so that is a bit of a clickbait title, but I do think it's true. In the hope this might be useful for someone, or some cluster, I decided to write this down. So what happened? First some context: subject at hand is a Ceph pacific cluster that has been running fine for over nine years (Went through 12 -> 13 -> 14 -> 15 -> 16 with many point releases in between). All OSDs were recreated in 16.2.11 to make use of the RocksDB column families without the need to do resharding (which would have taken forever with the amount of OMAP we have). This cluster has been provisioned with a home build Ansible playbook based on the manual procedure [1], and is basically an automation of all these manual steps. For OSDs we used the "short form" variant [2]. The first issue we encountered was when we added new storage nodes / OSDs to the cluster. The deployment of the first storage node took longer than expected (PCIe NVMe devices that were in use for OS were not correctly filtered and turned up as eligible for OSD creation). After we fixed that we could succesfully proceed. However, the first OSD that was added hit the "out" timer (we did not have noout flag set) and was set "out" by the cluster. We issued the "ceph osd in osd.$id" command to get the OSD back in. The state for all new OSDs was "down" and "in" apart from this OSD, which was in state "down" and "out". This immediately resulted in a assert of the monitors (num_down_in_osds <= num_in_osds). Different monitors crashed in quick succession but the quorum was never lost, and three mons (out of five) were always online. It also did not impact production traffic as now data was stored on them yet. The "ceph osd in osd.$id" that was hanging (due to crashing monitors) was aborted and shortly after that the monitors stopped crashing. The storage node got rebooted and the OSDs were all "up" and "in", apart from one. But this time we could successfully set the osd "in" ("up" and "in" state). We tried to reproduce this in a (virtual) test cluster that is as close a match to production but did not manage to reproduce. We figured this might have been a race condition of some kind (while the command was issued new OSDs might also have been added). The storage nodes were rebalanced and the new nodes were now also taking production workload. Somewhat later we decided to test a change (disable bypass workqueue flags of the encrypted (dmcrypt) devices) and shutdown the OSDs. Immediately after that several mons hit the same assert again. Although the OSDs had gracefully shut itself down ... because of the asserting mons no new osdmaps were created and the peer OSDs were not informed that there peers were down. Thus all involved OSDs had to figure this out themselves which they directly did ... but of course this did result in a lot of slow OSD operations. The OSDs were quickly started again and only until all OSDs were "up" and "in" again the monitors stopped crashing. The monitors did not loose quorum although we were of course afraid that they would. As it turned out this was likely not a race condition we started to look into this further. A search query on Ceph's issue tracker [3] resulted in a lot of hits. Turned out this issues has been hit by clusters provisioned by ROOK, manual procedure (ours) and potentially others and that the issue had been seen on clusters from 16.x, 17.x as well as 19.x and 20.x. This note gives a good summary [4]. It was fixed by this PR [5] and backported to Squid and Tentacle. But the reason *why* we hit this issue was not clear to us. Until we saw this report [6] wherein a user reported that their cluster had a lot of their OSDs with a "NEW" bit and when they would set an OSD "out" and "in" this bit would disappear and so did their issues (not hitting the assert anymore). We dumped the osdmap of the cluster and found out that *all* of our OSDs had the "NEW" bit active apart from one ... the one osd that had been through the OSD "out" / "in" state before. We only expected a "NEW" bit to be enabled for truely new OSDs, not OSDs that have been in the cluster for many years already. We decided not to wait until we would hit a scenario like the following: - one or more OSDs go down (broken or otherwise unable to start again) - have the monitors hit the assert - unable to get out of this situation easily as at that point you cannot issue any ceph commands as the monitors are asserting and the OSDs that went down cannot be brought back up. - hit a lot of slow ops in the meantime until the monitors would set the OSDs down ... or out eventually by itself (down out timer) which could have taken a long time in our case - have the risk of the monitors loosing the quorum and experience unavailability for an extended period of time. We backported the fix to the OSDMap.cc and created our own packages in case we would need them. After that we went ahead and set every OSD "out" and "in" to get rid of the NEW bit and that worked fine. The problem did not trigger anymore, as expected. Today I dived a bit deeper in the OSDMap.cc code responsible for this behavior ... turns out that there is a quicker way to fix this. Relevant code snippet: for (const auto &weight : inc.new_weight) { set_weight(weight.first, weight.second); // if we are marking in, clear the AUTOOUT and NEW bits, and clear // xinfo old_weight. if (weight.second) { osd_state[weight.first] &= ~(CEPH_OSD_AUTOOUT | CEPH_OSD_NEW); osd_xinfo[weight.first].old_weight = 0; } } If you perform a osd reweight, like "ceph osd reweight osd.0 1" it will remove the NEW bit. This is a way quicker process than "out" / "in". We added a reweight step to get rid of the "NEW" bit in our playbook if we would need to provision / replace OSDs again in the future. The reason why we hit this issue now, and not ever before, with the NEW bit OSDs in them and the bug still present is still unclear. There must be some other condition(s) that we hit this time and not before. I've checked our Cephadm based clusters and those don't have any NEW bits present. I'm not sure what steps Cephadm executes under the hood when it creates new OSDs but I guess this involves a step were it performs a reweight. For clusters out there that have been manually provisioned in the past, or created with Ansible playbooks that follow a similar procedure there might be OSDs present with this NEW bit that sooner or later have the risk of hitting this assert (until they run a version in which this is fixed). I advise you to check your clusters and get rid of these NEW bits because AFAIK their only purpose is to have this bit set when they are indeed, new, and they might harm your cluster(s) in the future. Questions we still have: - What other condition(s) are needed to trigger this assert - Would the manual install procedure need to include this reweight step, or is there another way to have the NEW bit cleared? Gr. Stefan [1]: https://docs.ceph.com/en/latest/install/manual-deployment [2]: https://docs.ceph.com/en/latest/install/manual-deployment/#short-form [3]: https://tracker.ceph.com/search?page=3&q=ceph_assert%28num_down_in_osds+%3C… [4]: https://tracker.ceph.com/issues/70869#note-9 [5]: https://github.com/ceph/ceph/pull/64009 [6]: https://tracker.ceph.com/issues/52535#note-12

1 0

Grafana, Prometheus, client certificates and mgmt-gateway in 20.2.3
by Robert Sander 18 Sep '26

18 Sep '26

Hi, we stumbled across an issue that is introduced with the new mgmt-gateway service. As soon as it is active the Prometheus service is configured to require client certificates for its API. It looks like the Grafana service is configured with client certificates according to the new service spec but it does not seem to use them. The result is no metrics in the Grafana dashboards. Can anyone confirm this behaviour? After removing the mgmt-gateway service and redeploying all monitoring services everything works again. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

2 9

Re: Performance issue after tentacle upgrade
by Moti 18 Sep '26

18 Sep '26

Hi Oguzhan, Thanks for your reply I did upgrade to 20.2.3 but the problem is still there. Latency reported by RGW matches what prober is reporting, so the latency jump is visible even if I unwrap latency from RGW beast logs. I tried investigating perf dump output per osd/rgw > host > cluster; also the hostoric_ops dump, but I can't spot a visible bottleneck, the latency is consistently high for HEAD requests and despite being 5x bigger than before, it's still some milliseconds that gets lost between other heavier ops latencies + sadly I don't have a 'before' perf dump result and comparing it to a healthy reef cluster is not giving any clues since the clusters differ entirely. Some piece of clue that I don't know how to interpret: when I re-index or re-create the target bucket, HEAD latency bounces back to its before-upgrade value! in my case it stayed there for an entire week before again peaking to 5x, the increase happens suddenly and not gradually, I found no informative logs around when the jump happens, and bucket is not growing in size so ..should I assume some sort of index corruption is happening? Regards On Tue, 11 Aug 2026, 17:42 Oguzhan Ozmen (BLOOMBERG/ 120 PARK) via ceph-users, <ceph-users(a)ceph.io> wrote: > Hey Moti, > > Zitat von Moti via ceph-users <ceph-users(a)ceph.io>: > > I recently upgraded from 18.2.4 to 20.2.1 > > > > I did notice this issue: > > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/C5BXQ26KD3… > > but that's not much similar to our case since we don't see any jump in > CPU or memory usage. > > Although the symptoms don't look the same, I'd recommend upgrading to > v20.2.3 to get the fix for this issue and the other fixes that went in: > https://docs.ceph.com/en/latest/releases/tentacle/#v20-2-3-tentacle > > After v20.2.3 upgrade, if you still see the same issue, then I'd suggest > collecting "perf" output per process per component (rgw, osd mainly) to > further analyze the bottleneck. > > You can also quickly check rgw logs to see what RGW reports as the latency > for those PUT/GET/HEAD requests whether it matches the latency you see on > your end (your wkld generator); especially for HEAD where no data streaming > happens. RGW logs "start" and "end" of a request. This is to rule out > whether bottleneck is outside of ceph (e.g., load-balancers). > > Cheers, > Oguzhan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

September Ceph Science User Group Virtual Meeting
by Enrico Bocchi 18 Sep '26

18 Sep '26

Dear all, We will be having a Ceph science/research/big cluster call next Tuesday, September 22nd, at 14:00 UTC. This time, it is my turn to tell a horror story about how an SSD firmware bug nearly killed our CephFS. The Ceph Science Virtual Meeting is an informal open call for community members primarily from hpc/htc/research/big cluster environments, though anyone is welcome, where we discuss whatever is on our minds regarding Ceph: Updates, outages, features, maintenance, ... There is no single set presenter for the length of the meeting, but it is somewhat moderated to keep the conversation flowing. If you wish to discuss something specific, please add it to the Pad linked below. If you are interested in giving a short talk during one of the next meetings, please reach out to me. Event Pad URL: https://pad.ceph.com/p/Ceph_Science_User_Group_20260922 Main Pad for the Ceph Science User Group: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Join the meeting at: https://meet.jit.si/ceph-science-wg Meetings will be recorded and posted to the Ceph Youtube channel. Virtual event details: September 22nd, 2026 14:00 UTC See the Ceph Community Calendar (https://ceph.io/en/community/meetups/) for the series information and for the meeting time in your timezone. See you there! Enrico

1 0

19.2.6 crash daemon - aes256k not supported?
by Andreas Eriksson 18 Sep '26

18 Sep '26

Hi, When finishing up an upgrade to 19.2.6 I rotated keys for the crash daemon. # ceph auth rotate --key-type=aes256k client.crash.$myhost # ceph orch reconfig crash.$myhost # ceph orch daemon restart crash.$myhost But it seems it's not supported. I get this in the daemon logs: Sep 18 10:21:24 myhost bash[3327369]: INFO:ceph-crash:pinging cluster to exercise our key Sep 18 10:21:24 myhost bash[3327369]: 2026-09-18T08:21:24.374+0000 766f4adb1640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] and I support [1] Sep 18 10:21:24 myhost bash[3327369]: 2026-09-18T08:21:24.374+0000 766f4a5b0640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] and I support [1] Sep 18 10:21:24 myhost bash[3327369]: 2026-09-18T08:21:24.374+0000 766f49daf640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] and I support [1] Sep 18 10:21:24 myhost bash[3327369]: 2026-09-18T08:21:24.374+0000 766f4c83b640 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication Sep 18 10:21:24 myhost bash[3327369]: [errno 13] RADOS permission denied (error connecting to the cluster) Sep 18 10:21:24 myhost bash[3327369]: INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s

1 0

Massive mon slow-ops when restarting mon daemons
by Alex Walender 17 Sep '26

17 Sep '26

Hey there! I’m currently in the process of upgrading the production cluster from v18.2.8 -> v19.2.6 via „ceph orch upgrade“. I tested this upgrade in our staging environment and everything went great. On our production environment I encounter some worrisome issues with the mon daemons. The cluster itself: 5x MON Daemons 5x MGR Daemons 1500x OSD - 2 larger Pools with each 16k in PG. … so a somewhat larger cluster. The orchestrator started re-deploying the MGR, which went fine. During the MON redeployment, the cluster went into warning with messages like „34846 slow ops. mon.mon1 has slow ops“. The counter climbed rapidly. Only way to stop this was to restart the MON in question. Sometimes I had to restart the MON multiple times. Afterwards the cluster is healthy again….until I restart any MON. The symptoms return until restarting fixes it again. I tried to dump the mon_ops_in_flight, but the command just hangs (probably because of the large error count). Filtering the journal for the issues: ``` journalctl -g "oldest is" --since "3 hours ago" -u ceph-d8a6f365-99de-42f2-b32b-c9ea11df945c(a)mon.partnach <mailto:ceph-d8a6f365-99de-42f2-b32b-c9ea11df945c@mon.partnach> Sep 14 09:45:05 partnach conmon[1735]: 2026-09-14T07:45:05.331+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18286 slow ops, oldest is log(1000 entries from seq 2 at 2026-08-26T04:51:47.819315+0000) Sep 14 09:45:10 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 18505 slow ops, oldest is log(1000 entries from seq 4003 at 2026-09-11T12:25:52.136679+0000) Sep 14 09:45:10 partnach conmon[1735]: 2026-09-14T07:45:10.827+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18505 slow ops, oldest is log(1000 entries from seq 4003 at 2026-09-11T12:25:52.136679+0000) Sep 14 09:45:15 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 18671 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-07T16:47:58.529125+0000) Sep 14 09:45:15 partnach conmon[1735]: 2026-09-14T07:45:15.924+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18671 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-07T16:47:58.529125+0000) Sep 14 09:45:21 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 18814 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-10T20:12:26.565558+0000) Sep 14 09:45:21 partnach conmon[1735]: 2026-09-14T07:45:21.224+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18814 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-10T20:12:26.565558+0000) Sep 14 09:45:26 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19042 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-10T14:13:26.625340+0000) Sep 14 09:45:26 partnach conmon[1735]: 2026-09-14T07:45:26.700+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19042 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-10T14:13:26.625340+0000) Sep 14 09:45:32 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19280 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-13T08:23:50.558481+0000) Sep 14 09:45:32 partnach conmon[1735]: 2026-09-14T07:45:32.432+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19280 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-13T08:23:50.558481+0000) Sep 14 09:45:37 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19433 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-11T20:28:14.104877+0000) Sep 14 09:45:37 partnach conmon[1735]: 2026-09-14T07:45:37.568+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19433 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-11T20:28:14.104877+0000) Sep 14 09:45:43 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19632 slow ops, oldest is log(1000 entries from seq 2 at 2026-09-10T08:47:13.259138+0000) Sep 14 09:45:43 partnach conmon[1735]: 2026-09-14T07:45:43.292+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19632 slow ops, oldest is log(1000 entries from seq 2 at 2026-09-10T08:47:13.259138+0000) ``` I’m not quite sure what the lock is here. The mon tries to store some old cluster log entries from various dates? Checked the underlying network fabric, CPU load, clock skew, compacted the rocksdb manually on each mon ect. All is fine. I also set "ceph config set global mon_cluster_log_level info“ **before** the 10th of September. Luckily the upgrade has finished with the MON daemons and is now re-deploying all OSDs. Does anyone know whats going on? Some kind of old hanging Log-Entries? Best Regards, Alex Walender --------------------------- M.Sc Alex Walender Institut für Bio- und Geowissenschaften IBG 5 - Computergestützte Metagenomik / de.NBI Cloud Site Bielefeld Büro : Universität Bielefeld (UHG), N7-101 Tel. : +49-521-106-2907 Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellv. Vorsitzende), Prof. Dr. Ir. Pieter Jansens

2 1

Help needed: multiple OSDs refuse to restart (and pg in down state)
by Massimo Sgaravatto 16 Sep '26

16 Sep '26

Dear all Today we had a problem in our ceph cluster: basically for some (still unknown) reasons, 5 OSDs (206, 233, 250, 257, 277) crashed and now they refuse to start. The main problem is that 1 pg is in down+remapped state because it is waiting for 3 of these OSDs [*] Doing a ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-xxx --op fsck, I see [**] that the problem is with 2 objects (rbd_data.44.f397cc4c21bbfb.000000000000319b and rbd_data.44.f397cc4c21bbfb.0000000000000e66 A "--op repair" is not able to fix the problem I am not sure how to proceed now Is the only option removing the problematic object from each OSD, doing e.g.: ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-206 --pgid 43.1b0 '{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":7151\ 31231,"max":0,"pool":43,"namespace":"","shardid":3}' remove ? Then the OSDs should hopefully restart, right ? Or are there better options ? Your help will be really appreciated ! Thanks, Massimo [*] ceph pg 43.1b0 query ... ... "blocked": "peering is blocked due to down osds", "down_osds_we_would_probe": [ 206, 233, 250 ], "peering_blocked_by": [ { "osd": 206, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 233, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 250, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" } ] [**] [root@ceph-osd-17 ~]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-206 --op fsck 2026-09-16T21:26:23.609+0200 7fb5aa3ddac0 -1 bluestore(/var/lib/ceph/osd/ceph-206) fsck error: 3#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a lextent at 0x6c000~4000 spans a shard boundary 2026-09-16T21:26:23.609+0200 7fb5aa3ddac0 -1 bluestore(/var/lib/ceph/osd/ceph-206) fsck error: 3#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a lextent at 0x6d000 overlaps with the previous, which ends at 0x70000 2026-09-16T21:26:23.609+0200 7fb5aa3ddac0 -1 bluestore(/var/lib/ceph/osd/ceph-206) fsck error: 3#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a blob Blob(0x5605e5760b60 spanning 1 blob([!~4000,0xa211d395000~1000,!~3000,0x23b97906000~1000,0x23b97226000~3000] llen=0xc000 csum+shared crc32c/0x1000/48\ ) use_tracker(0xc*0x1000 0x[0,0,0,0,1000,0,0,0,1000,1000,1000,1000]) SharedBlob(0x5606c99ebfc0 sbid 0x37ae23b)) doesn't match expected ref_map use_tracker(0xc*0x\ 1000 0x[0,0,0,0,1000,0,0,0,1000,2000,2000,2000]) fsck status: remaining 3 error(s) and warning(s) root@ceph-osd-17 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-206 --op list | grep "rbd_data.44.f397cc4c21bbfb.000000000000319b" ["43.1b0s3",{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":4210196912,"max":0,"pool":43,"namespace":"","generation":44630682,"s\ hard_id":3,"max":0}] ---------------------------------------- ---------------------------------------- [root@ceph-osd-18 ~]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-233 --op fsck 2026-09-16T21:37:09.598+0200 7f8ee4c49ac0 -1 bluestore(/var/lib/ceph/osd/ceph-233) fsck error: 4#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a lextent at 0x6c000~4000 spans a shard boundary 2026-09-16T21:37:09.598+0200 7f8ee4c49ac0 -1 bluestore(/var/lib/ceph/osd/ceph-233) fsck error: 4#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a lextent at 0x6d000 overlaps with the previous, which ends at 0x70000 2026-09-16T21:37:09.598+0200 7f8ee4c49ac0 -1 bluestore(/var/lib/ceph/osd/ceph-233) fsck error: 4#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a blob Blob(0x5557c8bef2b0 spanning 1 blob([!~4000,0x2ce88f4c000~1000,!~3000,0x2ce89560000~1000,0x2ce88a80000~3000] llen=0xc000 csum+shared crc32c/0x1000/48\ ) use_tracker(0xc*0x1000 0x[0,0,0,0,1000,0,0,0,1000,1000,1000,1000]) SharedBlob(0x5557fff8b1a0 sbid 0x323170b)) doesn't match expected ref_map use_tracker(0xc*0x\ 1000 0x[0,0,0,0,1000,0,0,0,1000,2000,2000,2000]) fsck status: remaining 3 error(s) and warning(s) [root@ceph-osd-18 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-233 --op list | grep "rbd_data.44.f397cc4c21bbfb.000000000000319b" ["43.1b0s4",{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":4210196912,"max":0,"pool":43,"namespace":"","generation":44630682,"s\ hard_id":4,"max":0}] [root@ceph-osd-18 ~]# ---------------------------------------- [root@ceph-osd-19 ~]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-250 --op fsck 2026-09-16T21:59:02.401+0200 7fa2e8795ac0 -1 bluestore(/var/lib/ceph/osd/ceph-250) fsck error: 0#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a lextent at 0x6c000~4000 spans a shard boundary 2026-09-16T21:59:02.401+0200 7fa2e8795ac0 -1 bluestore(/var/lib/ceph/osd/ceph-250) fsck error: 0#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a lextent at 0x6d000 overlaps with the previous, which ends at 0x70000 2026-09-16T21:59:02.401+0200 7fa2e8795ac0 -1 bluestore(/var/lib/ceph/osd/ceph-250) fsck error: 0#43:0d814f5f:::rbd_data.44.f397cc4c21bbfb.000000000000319b:head#2\ a9029a blob Blob(0x556a29cc4820 spanning 1 blob([!~4000,0x464866d9000~1000,!~3000,0x4648731f000~1000,0x46485e09000~3000] llen=0xc000 csum+shared crc32c/0x1000/48\ ) use_tracker(0xc*0x1000 0x[0,0,0,0,1000,0,0,0,1000,1000,1000,1000]) SharedBlob(0x556a1fb84e40 sbid 0x98f249)) doesn't match expected ref_map use_tracker(0xc*0x1\ 000 0x[0,0,0,0,1000,0,0,0,1000,2000,2000,2000]) fsck status: remaining 3 error(s) and warning(s) [root@ceph-osd-19 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-250 --op list | grep "rbd_data.44.f397cc4c21bbfb.000000000000319b" ["43.1b0s0",{"oid":"rbd_data.44.f397cc4c21bbfb.000000000000319b","key":"","snapid":-2,"hash":4210196912,"max":0,"pool":43,"namespace":"","generation":44630682,"s\ hard_id":0,"max":0}] ---------------------------------------- [root@ceph-osd-19 ~]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-257 --op fsck 2026-09-16T22:06:09.627+0200 7fded8db6ac0 -1 bluestore(/var/lib/ceph/osd/ceph-257) fsck error: 0#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ dbe3c7 lextent at 0x6a000~6000 spans a shard boundary 2026-09-16T22:06:09.627+0200 7fded8db6ac0 -1 bluestore(/var/lib/ceph/osd/ceph-257) fsck error: 0#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ dbe3c7 lextent at 0x6e000 overlaps with the previous, which ends at 0x70000 2026-09-16T22:06:09.627+0200 7fded8db6ac0 -1 bluestore(/var/lib/ceph/osd/ceph-257) fsck error: 0#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ dbe3c7 blob Blob(0x561e442541a0 spanning 2 blob([!~4000,0x19c87d9b000~1000,0x46e6508e000~3000,0x46e64693000~2000] llen=0xa000 csum+shared crc32c/0x1000/40) use_t\ racker(0xa*0x1000 0x[0,0,0,0,1000,1000,1000,1000,1000,1000]) SharedBlob(0x561e440f7620 sbid 0x3e6b92e)) doesn't match expected ref_map use_tracker(0xa*0x1000 0x[\ 0,0,0,0,1000,1000,1000,1000,2000,2000]) fsck status: remaining 3 error(s) and warning(s) [root@ceph-osd-19 ~]# [root@ceph-osd-19 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-257 --op list | grep "rbd_data.44.f397cc4c21bbfb.0000000000000e66" ["43.70s0",{"oid":"rbd_data.44.f397cc4c21bbfb.0000000000000e66","key":"","snapid":-2,"hash":2802327152,"max":0,"pool":43,"namespace":"","generation":47965127,"sh\ ard_id":0,"max":0}] ---------------------------------------- ---------------------------------------- [root@ceph-osd-20 ~]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-277 --op fsck 2026-09-16T22:08:04.530+0200 7f35041c8ac0 -1 bluestore(/var/lib/ceph/osd/ceph-277) fsck error: 3#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ dbe3c7 lextent at 0x6a000~6000 spans a shard boundary 2026-09-16T22:08:04.530+0200 7f35041c8ac0 -1 bluestore(/var/lib/ceph/osd/ceph-277) fsck error: 3#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ dbe3c7 lextent at 0x6e000 overlaps with the previous, which ends at 0x70000 2026-09-16T22:08:04.530+0200 7f35041c8ac0 -1 bluestore(/var/lib/ceph/osd/ceph-277) fsck error: 3#43:0e7810e5:::rbd_data.44.f397cc4c21bbfb.0000000000000e66:head#2\ dbe3c7 blob Blob(0x55ead1a808f0 spanning 2 blob([!~3000,0x1754e9e3000~1000,0x1754eb48000~3000,0x1754e809000~2000] llen=0x9000 csum crc32c/0x1000/36) use_tracker(\ 0x9*0x1000 0x[0,0,0,1000,1000,1000,1000,1000,1000]) (shared_blob=NULL)) doesn't match expected ref_map use_tracker(0x9*0x1000 0x[0,0,0,1000,1000,1000,1000,2000,2\ 000]) fsck status: remaining 3 error(s) and warning(s) [root@ceph-osd-20 ~]# root@ceph-osd-20 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-277 --op list | grep "rbd_data.44.f397cc4c21bbfb.0000000000000e66" ["43.70s3",{"oid":"rbd_data.44.f397cc4c21bbfb.0000000000000e66","key":"","snapid":-2,"hash":2802327152,"max":0,"pool":43,"namespace":"","generation":47965127,"sh\ ard_id":3,"max":0}]

1 0

Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
by Frank Ederveen 16 Sep '26

16 Sep '26

> If you upgrade to 20.2.4 the key rotation for the Ceph services will > be done by the orchestrator automatically. But only for the Ceph > services, no client keyrings will be touched by cephadm. So you can > safely upgrade and fix the CVEs but can still use clients with the aes > format keys. I have upgraded two cephadm managed test clusters from 20.2.1 to 20.2.4 with limited success on both. It allows new and old keys, and the default is to hand out new keys, but all of my clients (osd, rgw, admin, bootstrap, crash) are still using the insecure key. I ran the upgrade again with cephadm which did restart all the OSD, but the issue remains. All previous upgrades have gone very easily, this one.. not so much. Manually rotating and copying keys around is not an option for clusters with > 1k osd. Is there going to be a new release soon, or some scripting which fixes things? health: HEALTH_WARN 87 auth client entities with insecure key types Monitors are configured to allow auth using insecure key types Monitors are configured to allow creation of insecure key types ... [WRN] AUTH_INSECURE_CLIENT_KEY_TYPE: 87 auth client entities with insecure key types entity client.admin using insecure key type: aes entity client.bootstrap-mds using insecure key type: aes entity client.bootstrap-mgr using insecure key type: aes entity client.bootstrap-osd using insecure key type: aes entity client.bootstrap-rbd using insecure key type: aes entity client.bootstrap-rbd-mirror using insecure key type: aes entity client.bootstrap-rgw using insecure key type: aes entity client.crash.node1-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node2-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node3-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node4-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node5-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node6-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node7-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node8-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.osd-lockbox.02f209c1-ade4-492a-8a65-22fd21a71365 using insecure key type: aes entity client.osd-lockbox.0cae9ed5-2976-4047-9628-f90d53b3e577 using insecure key type: aes entity client.osd-lockbox.1656cd37-7cd9-4ade-a396-5d973c866711 using insecure key type: aes entity client.osd-lockbox.1d17a243-d4a5-4f7d-9aeb-88e14dc45812 using insecure key type: aes ... Thanks, Frank ________________________________ From: Eugen Block <ceph-users(a)ceph.io> Sent: 31 August 2026 07:44:50 To: ceph-users(a)ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4 If you upgrade to 20.2.4 the key rotation for the Ceph services will be done by the orchestrator automatically. But only for the Ceph services, no client keyrings will be touched by cephadm. So you can safely upgrade and fix the CVEs but can still use clients with the aes format keys. Zitat von Brent Kennedy <ceph-users(a)ceph.io>: > Question, sort of related. If upgrading from squid to tentacle, > will it automatically try to rotate keys are part of the cluster > upgrade or is that still a manual process? We would like to avoid > the key rotation for now as we have some proxmox nodes using ceph > storage in some of the clusters. > > Regards, > -Brent > > Existing Clusters: > US Production(HDD): Squid 19.2.3 Cephadm with 11 osd servers, 5 > mons, 4 rgw, 2 iscsigw, 2 mds > UK Production(HDD): Squid 19.2.3 Cephadm with 20 osd servers, 5 > mons, 4 rgw, 2 iscsigw, 2 mds > US Production(SSD): Squid 19.2.3 Cephadm with 6 osd servers, 5 mons, > 4 rgw, 2 mds > UK Production(SSD): Squid 19.2.3 cephadm with 6 osd servers, 5 mons, > 4 rgw, 2 mds > > > > -----Original Message----- > From: Patrick Donnelly <ceph-users(a)ceph.io> > Sent: Saturday, August 29, 2026 10:38 AM > To: Saif Mohammad <samdto987(a)gmail.com> > Cc: ceph-users(a)ceph.io > Subject: [ceph-users] Re: Clarification on Client Key Rotation > During Upgrade to 19.2.6/20.2.4 > > Hello, > > On Fri, Aug 28, 2026 at 10:20 PM Saif Mohammad <ceph-users(a)ceph.io> wrote: >> >> Hi, >> >> While upgrading to Squid 19.2.6 or Tentacle 20.2.4, since these >> versions introduce the secure key type, I have a clarification >> regarding client key rotation. >> For the client.admin key, we need to rotate the key and then >> import/update the keyring. >> For the other client keys, do we only need to rotate the key and >> verify that the key has changed? Is there no need to import/update >> the keyring for the other clients, since there is no local keyring >> available? > > Step (9) of [1] says: > > "Then copy and import the key to each machine using that client.$ID > credential." > > [1] > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-… > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Red Hat Partner Engineer > IBM, Inc. > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 4

User Experience upgrading Hyperconverged CEPH Quincy on Proxmox 8.4 to Proxmox 9.x and CEPH tentacle.
by Tino Todino 15 Sep '26

15 Sep '26

Hi folks, We have a 7 node Hyperconverged CEPH Quincy cluster with 3 OSD's per host. So small fry really 🙂 We are upgrading our backbone from 10 to 25Gbe and as part of the process I'd like to upgrade the nodes from Promxox 8.4 to the latest version 9.2 (iirc). As part of this, CEPH needs to be upgraded. I was wondering if anyone has done this before and what the experience was like please? So, gotcha's, learning points, must do's, never do's etc. If no-one has tried this, I guess I'll be the one updating the mailing list with the findings, but that won't be until November most likely. 🙂 Thanks Tino Coastsense Ltd This E-mail is intended solely for the person or organisation to which it is addressed. It may contain privileged or confidential information and, if you are not the intended recipient, you must not copy, distribute or take any action in reliance upon it. Any views or opinions presented are solely those of the author and do not necessarily represent those of Marlan Maritime Technologies Ltd. If you have received this E-mail in error, please notify us as soon as possible and delete it from your computer. Marlan Maritime Technologies Ltd Registered in England & Wales 323 Mariners House, Norfolk Street, Liverpool. L1 0BG Company No. 08492427.

3 2

Experience with SEDs in Ceph clusters
by Serhat Rıfat Demircan 15 Sep '26

15 Sep '26

Hi all, Is anyone using Self-Encrypting Drives (SEDs) in production Ceph clusters? I’d be interested to hear about your experience, especially any operational issues, compatibility problems, performance impact, or unexpected behavior you have encountered. Also, if there are any Ceph-specific considerations or recommendations when using SEDs, I’d appreciate your feedback. Thanks, Serhat

2 2

Activity Sent From Your Own Account
by ceph-users-owner＠ceph.io 14 Sep '26

14 Sep '26

Your message to the ceph-users mailing-list was rejected for the following reasons: The message is not from a list member The original message as received by Mailman is attached.

1 0

Re: Tentacle 20.2.4: OSDs crash in PGLog::merge_log after monitor was rebuilt with --mkfs
by Eugen Block 14 Sep '26

14 Sep '26

Please don't drop the list off your responses. I'm not entirely sure yet what to make of the epochs. I'm thinking about mon_min_osdmap_epochs (default 500) and similar parameters. But I'm heading out on vacation now, hopefully someone else will chime in. Good luck! Eugen Zitat von Martin Konold <martin.konold(a)konsec.com>: > Hi Eugen, > > thanks for the quick reply -- one clarification: the monitor was > already rebuilt with --mkfs yesterday evening, before I posted today. > ceph-pve-07.bak is the leftover original store, not something I > am proposing to move. So the question is whether restoring it is > the right way back, not whether to abandon the current one. > >> Did you verify that all service keyrings were successfully >> rotated after the upgrade? > > No, and that turned out to be the core of the auth problem. The > rotation touched the mon DB but not the bluestore osd_key labels: > > osd.2 mon DB AgBf7qJq... label/tmpfs AgA8cKJq... > osd.5 mon DB AgBg7qJq... label/tmpfs AgA8cKJq... > > osd.0, osd.3 and osd.9 are still in that state -- they run fine > on existing sessions but hold pre-rotation keys in their labels, > so they would not come back after a restart. set-label-key > refuses to write while the OSD is up, so I cannot fix that > pre-emptively. > > Caps were inconsistent too: osd.0 and osd.3 have > "caps mon = allow rwx" (the pre-profile form), osd.1 and osd.9 > were missing "caps mgr" entirely. > >> You could have allowed the insecure cipher again > > I did, that was the first thing after finding the monmap policy: > > ceph mon set auth_allowed_ciphers aes,aes256k > > That fixed authentication -- osd.2 and osd.5 now get through and > reach peering. client.admin was fine throughout, so the command > went through without trouble. > > What remains is the PGLog assert, and it is not an auth problem. > Checking the timestamps: osd.5 first hit the assert at 21:04:21, > while its monclient log at 21:04:15 shows a healthy session > (wait_auth_rotating done, uptodate secrets). The auth failures > only start at 21:24. So the crashes came first; the cipher > lockout landed on top and obscured them. > > The assert shows an empty authoritative log against a populated > local one: > > merge_log log((573038'28581091,573038'28581091], > crt=573038'28581091) from osd.3 > into log((573040'28581121,573042'28581681], > crt=573042'28581681) > > Also worth noting: at 21:04 osd.2 had to re-add itself to the > CRUSH map ("osd crush create-or-move id 2 weight 7.2774 > host=pve-08"), which suggests the rebuilt store did not know it. > My working assumption is that the incomplete CRUSH map during > that window let writes be acked against acting sets that later > could not reconcile. > > So the open question stands: with the original store sitting at > osdmap 573036 and the divergence starting at 573038, is a > restore the right approach, and how should the ~500 epoch gap to > the OSDs be handled? > > root@pve-07:~# ceph osd tree > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > -1 43.66434 root default > -61 14.55478 host pve-07 > 0 nvme 7.27739 osd.0 up 1.00000 1.00000 > 3 nvme 7.27739 osd.3 up 1.00000 1.00000 > -66 14.55478 host pve-08 > 2 nvme 7.27739 osd.2 down 1.00000 1.00000 > 5 nvme 7.27739 osd.5 down 1.00000 1.00000 > -71 14.55478 host pve-09 > 1 nvme 7.27739 osd.1 down 1.00000 1.00000 > 9 nvme 7.27739 osd.9 up 1.00000 1.00000 > > > Kind Regards, > --martin > > Am Freitag, September 11, 2026 10:56 CEST, schrieb Eugen Block > <ceph-users(a)ceph.io>: > >> Hi, >> >> that seems a little drastic to move the mon store. Did you verify that >> all service keyrings (mgr,mon,osd,mds) were successfully rotated after >> the upgrade? >> >> You could have allowed the insecure cipher again to see if that >> resolves the issue and finish the upgrade (if steps were missing). >> >> ceph mon set auth_allowed_ciphers aes,aes256k >> >> I assume you rotated the admin keyring as well so changing >> auth_allowed_ciphers would be possible? >> >> Zitat von Martin Konold <ceph-users(a)ceph.io>: >> >> > Hi, >> > >> > Yesterday we got an update via Proxmox. The initial update went well >> > but hell broke loose when I tried to dissallow the old aes >> > authentification in order top get rid of a warning. >> > >> > I'm looking for guidance before taking any further destructive >> > action on a 3-node Proxmox cluster (6 OSDs, one replicated pool >> > 3/2, ~2.5 TiB). No users on it, no time pressure. >> > >> > Timeline >> > -------- >> > 10.09 10:59 monmap epoch 81 shows an inconsistent cipher policy: >> > auth_allowed_ciphers aes256k >> > auth_preferred_cipher aes >> > New keys are created as aes but aes is not allowed >> > to authenticate. Any OSD restarted after this point >> > was locked out. >> > 10.09 15:20 last good osdmap in the original mon store: epoch >> > 573036, all 6 OSDs up/in. >> > 10.09 ~16:32 original mon store moved aside (now ceph-pve-07.bak, >> > 72 MB, 26k keys, intact). >> > 10.09 20:38 monitor rebuilt with monmaptool/--mkfs. New store has >> > 56 auth keys vs 128 in the original, 28 mon_config_key >> > vs 1162, and caps like "mon = allow rwx" on osd.0/3 -- >> > clearly a reconstruction. >> > 10.09 21:04 osd.2 and osd.5 crash in PGLog::merge_log. Auth was >> > still working at this point (wait_auth_rotating done, >> > and osd.2 had to re-add itself via "osd crush >> > create-or-move"). >> > 10.09 21:24 auth failures start on top of the crashes (handle_auth >> > _bad_method). Resolved since: mon key was rotated >> > without updating the bluestore osd_key label. >> > >> > Current state >> > ------------- >> > 3 of 6 OSDs (0, 3, 9) are up and running on pre-existing cephx >> > sessions. osd.1, osd.2, osd.5 crash during peering: >> > >> > ./src/osd/PGLog.cc: 401: FAILED ceph_assert( >> > log.head >= olog.tail && olog.head >= log.tail) >> > PeeringState::Stray::react(MLogRec const&) -> merge_log >> > >> > The merge_log lines show an *empty* authoritative log being sent >> > against a populated local one: >> > >> > merge_log log((573038'28581091,573038'28581091], >> > crt=573038'28581091) from osd.3 >> > into log((573040'28581121,573042'28581681], >> > crt=573042'28581681) >> > >> > So the surviving OSDs are behind, and the crashing ones hold log >> > entries from epochs 573038-573042 that the primary has no record >> > of. That range starts two epochs after the last osdmap in the >> > original store. >> > >> > original mon store max osdmap 573036 >> > divergent pg log entries 573038-573042 >> > current osdmap (after restarts) 573538 >> > >> > ceph -s (before I stopped the crashing OSDs): 278/708728 objects >> > unfound, 40 pgs recovery_unfound, 11 incomplete, 11 down. >> > >> > Questions >> > --------- >> > 1. Is restoring the original mon store the right approach here? >> > It is only 2 epochs behind where the divergence starts, which >> > seems like the best possible starting point. >> > 2. How should the 502-epoch gap be handled -- the OSDs have >> > osdmaps far newer than the restored monitor would have? >> > 3. Is the assert itself worth a tracker issue? An empty >> > authoritative log against a populated local one looks like a >> > case that should fall back to backfill rather than assert. >> > >> > What I have NOT done: no mark_unfound_lost, no min_size 1, no >> > ceph-objectstore-tool remove. All three mon stores >> > are backed up. The original store is untouched. >> > >> > One more constraint: the bluestore osd_key labels on the three >> > running OSDs (0, 3, 9) are stale -- they hold pre-rotation keys. >> > They run fine now but would not come back after a restart, and >> > set-label-key refuses to write while the OSD is up. So any plan >> > involving an OSD restart has to fix the label in the same stop/ >> > start window. >> > >> > Happy to provide full logs with debug_osd=20, ceph pg query >> > output, or monstore dumps. >> > >> > Thanks, >> > --martin >> > >> > -- >> > martin.konold(a)konsec.com >> > Phone: +49 711 21 72 74-90 >> > KONSEC GmbH -⁠ make things real >> > Amtsgericht Stuttgart, HRB 23690 >> > Geschäftsführer: Andreas Mack >> > Im Köller 3, 70794 Filderstadt, Germany >> > >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > martin.konold(a)konsec.com > Phone: +49 711 21 72 74-90 > KONSEC GmbH -⁠ make things real > Amtsgericht Stuttgart, HRB 23690 > Geschäftsführer: Andreas Mack > Im Köller 3, 70794 Filderstadt, Germany

2 1

Tentacle: Emergency Allowed Ciphers
by Eugen Block 11 Sep '26

11 Sep '26

Hi, I was trying to use the mon_auth_emergency_allowed_ciphers setting to rescue a test cluster as described in the docs [0]. It's a cephadm managed cluster where I set auth_allowed_ciphers to aes256k before rotating the admin keyring. Then I added mon_auth_emergency_allowed_ciphers = true to all mon's config files: reef1:~ # vi /var/lib/ceph/{FSID}/mon.reef1/config and restarted the mon services. According to the config dump the setting is active (for all 3 daemons): [ceph: root@reef2 /]# ceph daemon mon.reef2 config show | grep cipher "mon_auth_emergency_allowed_ciphers": "true", The daemon logs this during startup but I'm not sure if it's relevant right now: Sep 11 10:25:08 reef2 ceph-{FSID}-mon-reef2[88175]: 2026-09-11T08:25:08.611+0000 7f0a8765bd80 -1 mon.reef2@-1(???) e6 init: invalid cipher: true I can't use the unrotated admin keyring although the docs state that I should be able to: > This will allow your existing client.admin or other administrative > key to authenticate as normal. What am I missing? Thanks. Eugen [0] https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h…

3 4

CFPs Reminder
by Anthony Middleton 11 Sep '26

11 Sep '26

Happy Friday, Ceph friends! Just a reminder that the CFPs for Ceph Days Zürich <https://docs.google.com/forms/d/e/1FAIpQLSfUAulFnfdrHesAi92Y2Dkkm7whEXQBel1…> and Ceph Meetup Melbourne <https://docs.google.com/forms/d/e/1FAIpQLSdDgJGT1AyhJFaXPrjvx0X9lYSasU1Iju8…> are still open. Please let me know if you have any questions, and I can connect you with the event organizers. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Tentacle 20.2.4: OSDs crash in PGLog::merge_log after monitor was rebuilt with --mkfs
by Martin Konold 11 Sep '26

11 Sep '26

Hi, Yesterday we got an update via Proxmox. The initial update went well but hell broke loose when I tried to dissallow the old aes authentification in order top get rid of a warning. I'm looking for guidance before taking any further destructive action on a 3-node Proxmox cluster (6 OSDs, one replicated pool 3/2, ~2.5 TiB). No users on it, no time pressure. Timeline -------- 10.09 10:59 monmap epoch 81 shows an inconsistent cipher policy: auth_allowed_ciphers aes256k auth_preferred_cipher aes New keys are created as aes but aes is not allowed to authenticate. Any OSD restarted after this point was locked out. 10.09 15:20 last good osdmap in the original mon store: epoch 573036, all 6 OSDs up/in. 10.09 ~16:32 original mon store moved aside (now ceph-pve-07.bak, 72 MB, 26k keys, intact). 10.09 20:38 monitor rebuilt with monmaptool/--mkfs. New store has 56 auth keys vs 128 in the original, 28 mon_config_key vs 1162, and caps like "mon = allow rwx" on osd.0/3 -- clearly a reconstruction. 10.09 21:04 osd.2 and osd.5 crash in PGLog::merge_log. Auth was still working at this point (wait_auth_rotating done, and osd.2 had to re-add itself via "osd crush create-or-move"). 10.09 21:24 auth failures start on top of the crashes (handle_auth _bad_method). Resolved since: mon key was rotated without updating the bluestore osd_key label. Current state ------------- 3 of 6 OSDs (0, 3, 9) are up and running on pre-existing cephx sessions. osd.1, osd.2, osd.5 crash during peering: ./src/osd/PGLog.cc: 401: FAILED ceph_assert( log.head >= olog.tail && olog.head >= log.tail) PeeringState::Stray::react(MLogRec const&) -> merge_log The merge_log lines show an *empty* authoritative log being sent against a populated local one: merge_log log((573038'28581091,573038'28581091], crt=573038'28581091) from osd.3 into log((573040'28581121,573042'28581681], crt=573042'28581681) So the surviving OSDs are behind, and the crashing ones hold log entries from epochs 573038-573042 that the primary has no record of. That range starts two epochs after the last osdmap in the original store. original mon store max osdmap 573036 divergent pg log entries 573038-573042 current osdmap (after restarts) 573538 ceph -s (before I stopped the crashing OSDs): 278/708728 objects unfound, 40 pgs recovery_unfound, 11 incomplete, 11 down. Questions --------- 1. Is restoring the original mon store the right approach here? It is only 2 epochs behind where the divergence starts, which seems like the best possible starting point. 2. How should the 502-epoch gap be handled -- the OSDs have osdmaps far newer than the restored monitor would have? 3. Is the assert itself worth a tracker issue? An empty authoritative log against a populated local one looks like a case that should fall back to backfill rather than assert. What I have NOT done: no mark_unfound_lost, no min_size 1, no ceph-objectstore-tool remove. All three mon stores are backed up. The original store is untouched. One more constraint: the bluestore osd_key labels on the three running OSDs (0, 3, 9) are stale -- they hold pre-rotation keys. They run fine now but would not come back after a restart, and set-label-key refuses to write while the OSD is up. So any plan involving an OSD restart has to fix the label in the same stop/ start window. Happy to provide full logs with debug_osd=20, ceph pg query output, or monstore dumps. Thanks, --martin -- martin.konold(a)konsec.com Phone: +49 711 21 72 74-90 KONSEC GmbH -⁠ make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany

2 1

ceph squid nfs export create uses old keyring format
by Meekhof, Benjeman 10 Sep '26

10 Sep '26

We recently upgraded from 19.2.3 to 19.2.6. Cluster is deployed with cephadm and I used ceph orch to do the upgrade. I've encountered a small issue with new aes256 keyrings and the 'ceph nfs export create' command. When I disable insecure keyring creation ('ceph config set mon mon_auth_allow_insecure_key false') the nfs export create command fails: ceph nfs export create cephfs --cluster-id clusterid --pseudo-path /test --fsname fsname --path=/test Error EINVAL: Expecting value: line 1 column 1 (char 0) When I set allow insecure key to true and try the command again it creates the export but with an insecure key that generates health warnings on the cluster. The export works fine as far as mounting it and seeing the data I expect. thanks, Ben

2 1

whoX
by ivax.ai 09 Sep '26

09 Sep '26

whoX por ivaX.ai whoX ( https://ivax.ai ) ***************************************************************** Logo incluido. Plazo de entrega 25 días después de aprobación diseño. ***************************************************************** ****************************** +34 938 424 303 x(a)ivax.ai ****************************** AVISO LEGAL COMPLETO DE PRIVACIDAD Y PROTECCIÓN DE DATOS PERSONALES (LGPD) 1. Identificación del Controlador de Datos: De conformidad con las disposiciones de la Ley General de Protección de Datos Personales de Brasil (Ley Nº 13.709/2018 - LGPD), se informa al titular de los datos que la entidad responsable del tratamiento de su información personal es Worxfly SL, con domicilio enc/ Velázquez, 27, 28001, Madrid, en adelante denominada el "Controlador". 2. Datos Recopilados y Base Legal del Tratamiento: Para el envío de este boletín informativo (newsletter), comunicaciones operativas, novedades, contenidos educativos y campañas de marketing directo, el Controlador trata datos personales limitados a: su nombre completo, dirección de correo electrónico. La base legal que legitima el tratamiento de estos datos es el Consentimiento libre, informado e inequívoco del titular (Art. 7, inciso I de la LGPD), otorgado de forma voluntaria al suscribirse a nuestros canales de comunicación, o bien el Interés Legítimo del Controlador (Art. 7, inciso IX de la LGPD) para el desarrollo y promoción de nuestras actividades, siempre respetando las expectativas legítimas del usuario. 3. Finalidad del Tratamiento y Principios de la LGPD: En estricto cumplimiento del principio de finalidad y necesidad (Art. 6, I y III de la LGPD), sus datos serán utilizados única y exclusivamente para: (i) Gestionar su suscripción y remitir el newsletter periódico; (ii) Personalizar el contenido enviado en función de sus interacciones legítimas; (iii) Analizar métricas de apertura y clics para optimizar la calidad de los envíos. Nos comprometemos a no utilizar sus datos para finalidades distintas a las aquí informadas sin su autorización previa. 4. Compartición y Transferencia Internacional de Datos: El Controlador no comercializa, cede ni comparte sus datos personales con terceros con fines lucrativos. No obstante, para la ejecución técnica de los envíos, sus datos son alojados e integrados en plataformas de automatización de email marketing subcontratadas (Operadores de Datos). Dichos proveedores pueden almacenar la información en servidores ubicados fuera del territorio nacional (como en los EE.UU. o la Unión Europea), configurando una Transferencia Internacional de Datos (Art. 33 de la LGPD). El Controlador garantiza que utiliza proveedores que cumplen con los más estrictos estándares de seguridad y que están alineados con cláusulas contractuales contractuales tipo para la protección de datos. 5. Período de Retención de la Información: Sus datos personales se conservarán en nuestra base de datos de forma indefinida mientras se mantenga activa su suscripción y persista el interés mutuo. Se procederá al bloqueo o eliminación definitiva de sus datos cuando: (a) El titular revoque el consentimiento manifestado; (b) Los datos dejen de ser necesarios para las finalidades que motivaron su recogida; o (c) Por mandato legal o requerimiento de la Autoridad Nacional de Protección de Datos (ANPD). 6. Derechos del Titular (Art. 18 de la LGPD): Le recordamos que usted, como titular de los datos, tiene derecho a ejercer en cualquier momento y de forma gratuita ante el Controlador los siguientes derechos: * Confirmación de la existencia del tratamiento de sus datos. * Acceso completo a los datos personales almacenados. * Corrección de datos incompletos, inexactos o desactualizados. * Anonimización, bloqueo o eliminación de datos innecesarios o tratados en desconformidad con la ley. * Portabilidad de sus datos a otro proveedor de servicios. * Eliminación de los datos personales tratados bajo el amparo de su consentimiento. * Información sobre las entidades públicas y privadas con las que el controlador realizó un uso compartido de datos. * Información sobre la posibilidad de no proporcionar el consentimiento y las consecuencias de dicha negativa. * Revocación del consentimiento otorgado previamente. En cumplimiento de lo dispuesto en la Protección de Datos, para el ejercicio de sus derechos de acceso, rectificación, cancelación y oposición al tratamiento de sus datos personales, contenidos en nuestras condiciones de protección de datos, solamente tiene que responder a este e-mail indicando su email en el asunto, o bien a través del envío de un correo ordinario a la dirección: INCLUIR DIRECCIÓN. Si desea darse de baja también puede hacer clic aquí ( https://boletines.ivax.ai/unsubscribe/53pdzm2/click?token=q0xuqxy4 ) . FULL LEGAL NOTICE ON PRIVACY AND PERSONAL DATA PROTECTION (LGPD) 1. Identification of the Data Controller: In accordance with the provisions of the Brazilian General Data Protection Law (Law No. 13.709/2018 - LGPD), the data subject is informed that the entity responsible for processing their personal information is Worxfly SL, with registered office at Calle Velázquez, 27, 28001, Madrid, hereinafter referred to as the "Controller". 2. Data Collected and Legal Basis for Processing: For the purpose of sending this newsletter, operational communications, news, educational content, and direct marketing campaigns, the Controller processes personal data limited to: your full name and email address. The legal basis for processing this data is the free, informed, and unambiguous consent of the data subject (Article 7, paragraph I of the LGPD), given voluntarily when subscribing to our communication channels, or the legitimate interest of the data controller (Article 7, paragraph IX of the LGPD) for the development and promotion of our activities, always respecting the user's legitimate expectations. 3. Purpose of Processing and LGPD Principles: In strict compliance with the principle of purpose limitation and necessity (Article 6, paragraphs I and III of the LGPD), your data will be used solely and exclusively for: (i) Managing your subscription and sending the periodic newsletter; (ii) Personalizing the content sent based on your legitimate interactions; (iii) Analyzing open and click-through rates to optimize the quality of our mailings. We commit to not using your data for purposes other than those stated here without your prior authorization. 4. Sharing and International Transfer of Data: The Controller does not sell, transfer, or share your personal data with third parties for profit. However, for the technical execution of mailings, your data is hosted and integrated into subcontracted email marketing automation platforms (Data Operators). These providers may store the information on servers located outside of Spain (such as in the USA or the European Union), constituting an International Data Transfer (Art. 33 of the LGPD). The Controller guarantees that it uses providers that comply with the strictest security standards and are aligned with standard contractual clauses for data protection. 5. Data Retention Period: Your personal data will be kept in our database indefinitely as long as your subscription remains active and mutual interest persists. Your data will be blocked or permanently deleted when: (a) You revoke your consent; (b) The data is no longer necessary for the purposes for which it was collected; or (c) By legal mandate or request of the National Data Protection Authority (ANPD). 6. Data Subject Rights (Art. 18 of the LGPD): We remind you that you, as the data subject, have the right to exercise the following rights at any time and free of charge with the Controller: Confirmation of the existence of the processing of your data. Full access to the personal data stored. Correction of incomplete, inaccurate, or outdated data. Anonymization, blocking, or deletion of unnecessary data or data processed unlawfully. Portability of your data to another service provider. Deletion of personal data processed based on your consent. Information about the public and private entities with which the controller has shared data. Information about the possibility of not providing consent and the consequences of such refusal. Revocation of previously granted consent. In compliance with data protection regulations, to exercise your rights of access, rectification, erasure, and objection to the processing of your personal data, as outlined in our data protection policy, simply reply to this email with your email address in the subject line, or send a letter to the following address: [INSERT ADDRESS]. You can also unsubscribe by clicking aquí ( https://boletines.ivax.ai/unsubscribe/53pdzm2/click?token=q0xuqxy4 ) . MENTIONS LÉGALES COMPLÈTES RELATIVES À LA PROTECTION DES DONNÉES PERSONNELLES (LGPD) 1. Identification du responsable du traitement : Conformément aux dispositions de la Loi générale brésilienne sur la protection des données (Loi n° 13.709/2018 – LGPD), la personne concernée est informée que l’entité responsable du traitement de ses données personnelles est Worxfly SL, dont le siège social est situé Calle Velázquez, 27, 28001 Madrid, ci-après dénommée le « Responsable du traitement ». 2. Données collectées et base juridique du traitement : Aux fins de l’envoi de cette newsletter, de communications opérationnelles, d’actualités, de contenus pédagogiques et de campagnes de marketing direct, le Responsable du traitement traite les données personnelles suivantes : vos nom et prénom ainsi que votre adresse électronique. Le traitement de ces données est fondé sur le consentement libre, éclairé et univoque de la personne concernée (article 7, paragraphe 1 de la LGPD), donné volontairement lors de l'inscription à nos communications, ou sur l'intérêt légitime du responsable du traitement (article 7, paragraphe IX de la LGPD) pour le développement et la promotion de ses activités, dans le respect des attentes légitimes de l'utilisateur. 3. Finalité du traitement et principes de la LGPD : Conformément au principe de limitation des finalités et de nécessité (article 6, paragraphes 1 et 3 de la LGPD), vos données seront utilisées exclusivement pour : (i) gérer votre abonnement et vous envoyer la newsletter ; (ii) personnaliser le contenu envoyé en fonction de vos interactions légitimes ; (iii) analyser les taux d'ouverture et de clics afin d'optimiser la qualité de nos communications. Nous nous engageons à ne pas utiliser vos données à d'autres fins que celles mentionnées ici sans votre autorisation préalable. 4. Partage et transfert international des données : Le responsable du traitement ne vend, ne transfère ni ne partage vos données personnelles avec des tiers à des fins lucratives. Toutefois, pour l'exécution technique des envois, vos données sont hébergées et intégrées à des plateformes d'automatisation marketing par courriel sous-traitées (Opérateurs de données). Ces prestataires peuvent stocker les informations sur des serveurs situés hors d'Espagne (notamment aux États-Unis ou dans l'Union européenne), ce qui constitue un transfert international de données (article 33 de la LGPD). Le Responsable du traitement garantit qu'il fait appel à des prestataires respectant les normes de sécurité les plus strictes et utilisant des clauses contractuelles types en matière de protection des données. 5. Durée de conservation des données : Vos données personnelles seront conservées dans notre base de données indéfiniment, tant que votre abonnement reste actif et que l'intérêt mutuel persiste. Vos données seront bloquées ou supprimées définitivement dans les cas suivants : (a) vous retirez votre consentement ; (b) les données ne sont plus nécessaires aux fins pour lesquelles elles ont été collectées ; ou (c) sur instruction légale ou à la demande de l'Autorité nationale de protection des données (ANPD). 6. Droits de la personne concernée (Art. 18 de la LGPD) : Nous vous rappelons que vous disposez, en tant que personne concernée, du droit d’exercer à tout moment et gratuitement les droits suivants auprès du responsable du traitement : Confirmation de l’existence du traitement de vos données. Accès complet aux données personnelles conservées. Rectification des données incomplètes, inexactes ou obsolètes. Anonymisation, blocage ou suppression des données inutiles ou traitées illégalement. Portabilité de vos données vers un autre prestataire de services. Suppression des données personnelles traitées sur la base de votre consentement. Informations sur les organismes publics et privés avec lesquels le responsable du traitement a partagé des données. Informations sur la possibilité de ne pas donner son consentement et les conséquences d’un tel refus. Retrait d’un consentement précédemment accordé. Conformément à la réglementation sur la protection des données, pour exercer vos droits d'accès, de rectification, d'effacement et d'opposition au traitement de vos données personnelles, tels que décrits dans notre politique de protection des données, il vous suffit de répondre à ce courriel en indiquant votre adresse électronique dans l'objet, ou d'envoyer un courrier à l'adresse suivante : [INSÉRER L'ADRESSE]. Vous pouvez également vous désabonner en cliquant aquí ( https://boletines.ivax.ai/unsubscribe/53pdzm2/click?token=q0xuqxy4 ) .

1 0

Re: Cephadm staggered upgrade --limit parameter not always applied
by Andreas Eriksson 09 Sep '26

09 Sep '26

I'm doing further tests in a small cluster. Wondering a bit about the rotation of keys, since when I later upgrade the RGW service the mon,mgr and osds are restarted due to key rotation as well. I'm wondering if this is intentional? 3 nodes running mon/mgr/monitoring, 3 OSD nodes with additional RGW daemons. Starting at v17.2.7, upgrading all but the RGWs to v19.2.6 # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.6 --daemon_types mgr,mon,osd,crash Upgrade RGWs # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.6 --services rgw.vps-lab1 On 8/26/26 5:37 PM, Eugen Block wrote: > I started to conduct some tests and document in which version the > limits worked. I’m not done yet, I’ll update the respective tracker > issue when I get there. > To perform the staggered update you could just redeploy rgw daemons > with a new image after all preceding upgrades have finished (mgr, mon, > crash, osd, mds) with ceph orch daemon redeploy rgw.<some-rgw> <image>. > > I don’t have the exact syntax at hand right now, but it should be > quite similar. > > > Zitat von Eugen Block <eblock(a)nde.ag>: > >> Hi, >> >> I haven't found anything explaining this yet, but I also haven't >> looked for a while. I just retried limiting RGW upgrade (from Reef to >> Tentacle): >> >> reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v20.2.4 >> --daemon-types rgw --limit 1 >> >> But the limit is still ignored: >> >> 2026-08-26T10:15:23.709+0000 7f8c09d01640 0 [cephadm INFO >> cephadm.upgrade] Upgrade: Updating rgw.nde.reef1.pmauyp (1/1) >> 2026-08-26T10:15:23.709+0000 7f8c09d01640 0 log_channel(cephadm) log >> [INF] : Upgrade: Updating rgw.nde.reef1.pmauyp (1/1) >> 2026-08-26T10:15:26.554+0000 7f8c09d01640 0 [cephadm INFO >> cephadm.upgrade] Upgrade: Updating rgw.nde.reef2.oqrpma (1/1) >> 2026-08-26T10:15:26.555+0000 7f8c09d01640 0 log_channel(cephadm) log >> [INF] : Upgrade: Updating rgw.nde.reef2.oqrpma (1/1) >> 2026-08-26T10:15:29.133+0000 7f8c09d01640 0 [cephadm INFO >> cephadm.upgrade] Upgrade: Updating rgw.nde.reef3.boujxu (1/1) >> 2026-08-26T10:15:29.133+0000 7f8c09d01640 0 log_channel(cephadm) log >> [INF] : Upgrade: Updating rgw.nde.reef3.boujxu (1/1) >> >> All three RGW daemons have been upgraded automatically. I'm pretty >> sure that it worked at some point with RGWs, in my notes from one of >> our customer upgrades where this worked: >> >> ceph orch upgrade start --image registry:5000/ceph/ceph:v18.2.5 >> --daemon-types rgw --limit 1 >> >> I can't tell when it stopped working properly. >> >> Regards, >> Eugen >> >> >> Zitat von Andreas Eriksson <norrland(a)nullbyte.se>: >> >>> Hi, >>> >>> Were you able to find anything more about this? I'm looking at >>> upgrading a bunch of clusters from previous major versions, but do >>> not want to upgrade all RGW services. And also limit the number of >>> osds at the time that are being upgraded, to keep the recovery >>> traffic lower for example. >>> >>> Br, >>> >>> On 3/17/26 3:35 PM, Eugen Block via ceph-users wrote: >>>> Adding the dev list >>>> >>>> Zitat von Eugen Block <eblock(a)nde.ag>: >>>> >>>>> Hi, >>>>> >>>>> I did a quick search on tracker but couldn't find anything >>>>> related. A customer reported this, and I can confirm the behaviour >>>>> on a lab cluster. I usually perform a staggered upgrade with >>>>> --daemon-types and --limit, but not for all daemon types. So I >>>>> haven't stumbled across this yet myself, but our customer did (I >>>>> can reproduce with --daemon-types as well). They upgraded from >>>>> latest Reef to latest Squid and reported that despite providing >>>>> the --limit parameter, the mons were all upgraded. So I tried to >>>>> reproduce, and the behaviour is not really clear to me, I'll try >>>>> to clarify. >>>>> >>>>> # Start with MGRs >>>>> >>>>> reef1:~ # ceph orch upgrade start --image >>>>> quay.io/ceph/ceph:v19.2.3 --services mgr --limit 1 >>>>> >>>>> Upgrading MGRs with limit works, but it doesn't reflect in the MGR >>>>> log. Usually, I expect a line like this: >>>>> >>>>> ...[cephadm INFO root] Hit upgrade limit of 1. Stopping upgrade >>>>> >>>>> But there is no such line in the logs. Then I upgrade the rest of >>>>> the MGRs. >>>>> >>>>> # Continue with MONs >>>>> >>>>> reef1:~ # ceph orch upgrade start --image >>>>> quay.io/ceph/ceph:v19.2.3 --services mon --limit 1 >>>>> >>>>> This gets even weirder, the orchestrator upgrades 2 out of 3 MONs. >>>>> And again, no such line in the log (Hit upgrade limit). What I >>>>> noticed was a MGR respawn after the first MON had been upgraded >>>>> successfully. Maybe some state of the upgrade progress gets lost >>>>> during the respawn? I then upgraded the remaining MON. Then >>>>> ceph-crash is upgraded successfully. >>>>> >>>>> # Upgrade OSD >>>>> >>>>> reef1:~ # ceph orch upgrade start --image >>>>> quay.io/ceph/ceph:v19.2.3 --services osd.osd.standalone --limit 1 >>>>> >>>>> And this is the first service that actually reports the upgrade >>>>> limit: >>>>> >>>>> 2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 [cephadm INFO >>>>> cephadm.upgrade] Upgrade: Updating osd.0 (1/1) >>>>> 2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 log_channel(cephadm) >>>>> log [INF] : Upgrade: Updating osd.0 (1/1) >>>>> 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 [cephadm INFO root] >>>>> Hit upgrade limit of 1. Stopping upgrade >>>>> 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 log_channel(cephadm) >>>>> log [INF] : Hit upgrade limit of 1. Stopping upgrade >>>>> 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 [cephadm INFO >>>>> cephadm.upgrade] Upgrade: Setting container_image for all nvmeof >>>>> 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 log_channel(cephadm) >>>>> log [INF] : Upgrade: Setting container_image for all nvmeof >>>>> 2026-03-05T15:47:47.492+0000 7f5bfdc73640 0 [cephadm INFO >>>>> cephadm.upgrade] Upgrade: Finalizing container_image settings >>>>> 2026-03-05T15:47:47.493+0000 7f5bfdc73640 0 log_channel(cephadm) >>>>> log [INF] : Upgrade: Finalizing container_image settings >>>>> 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 [cephadm INFO >>>>> cephadm.upgrade] Upgrade: Complete! >>>>> 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 log_channel(cephadm) >>>>> log [INF] : Upgrade: Complete! >>>>> >>>>> >>>>> This is really irritating and inconsistent: If the orchestrator >>>>> does honor --limit with other services than OSD, why isn't that >>>>> visible in the logs? And what's with the MONs? Why 2 out of 3? >>>>> >>>>> Any pointers appreciated! Not sure which Ceph versions might be >>>>> affected by this, I'll try out a couple more upgrade paths. >>>>> >>>>> Thanks, >>>>> Eugen >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Replacing an OSD in Squid
by Patrick Begou 08 Sep '26

08 Sep '26

Hi I'm replacing the HDD of osd.2 in my squid cluster following the provided documentation: https://docs.ceph.com/en/squid/rados/operations/add-or-rm-osds/ The OSD was yet out and the daemon down. The osd.2 is destroyed: cephosddestroy2--yes-i-really-mean-it I've stopped the node and replaced the HDD. Ceph (graphic board) see the HDD as available. But I'm unable to go further with the ceph-volume commands: # ceph-volume lvm zap /dev/sda --> RuntimeError: /dev/mapper/osvg-rootvol not found. # ceph-volume lvm prepare --osd-id 2 --data /dev/sda --> RuntimeError: /dev/mapper/osvg-rootvol not found. Something wrong in my cluster or a step is missing in the documentation ? Thank Patrick

2 10

Trouble With Attaching OSDs From A Previous Cluster To A New One
by Jacek Rużyczka 07 Sep '26

07 Sep '26

Hi folks, I've now been struggling for five weeks with my broken Ceph cluster (4 nodes on Ubuntu 24, containerised with Docker) and can't find a solution, not even with help from Google's AI. Here is what happened: 1. The cluster FS was no longer accessible, and even ceph -s no longer worked (error 110). The MON count had gone down to 2, and an hour later, all 4 OSD daemons were also gone. The corresponding containers had also disappeared (docker ps -a no longer displayed them) together with their logs. 2. A week later, one of the nodes got frozen. Accessing its serial console returned a bunch of low-level errors you can read in this forum thread if you wish: https://community.mixtile.com/t/node-3-unreachable-100-cpu-load/941/6 3. It turned out that some essential files on the nodes (like ceph.conf or keyrings) had simply disappeared. Further research revealed trouble with the internal flash storage (eMMC) of the nodes. So, the data on them was apparently corrupt, what forced me to ressort to "safer" eMMC settings, re-flash them, and re-install Ceph (with the FSID of the old cluster, of course). After that, 4 MONs and 2 MGRs worked, but: 4. Re-integrating the existing OSDs (on the NVMe SSDs, not on the internal eMMCs) did not work: Eugen's script https://heiterbiswolkig.blogs.nde.ag/2026/06/28/cephadm-recover-mon-store-u… ended me up with a "No such file or directory" error. Google's AI made a bunch of suggestions, which did not help either (mostly due to "Permission denied" errors). It also told me to shut down one of the MONs, extract a monmap, and inject it into the new cluster. But the "permission denied" errors persisted for at least two weeks until we could get rid of them. 5. Two OSDs could get revivif <https://dict.leo.org/englisch-deutsch/revivify>ied, but ceph -s marked them as "in", but "down" whatever we did. Extracting the keys from the OSDs and adding them to the config of the "new" cluster didn't help. So I got stuck in a dead end. 6. Now, Google's AI wants me to extract the raw data (which is hopefully still complete) from the drives for whatever purpose and store it on another computer, but I'm afraid that doing this won't help me either. So I need some reliable expert advice in order to bring my cluster back to life. What shall I do here? Are the files in my cluster maybe also corrupted by the abovementioned incident? Thank you. Regards Jacek

1 0

Mount Client/Superblock Sharing Broken Bugfix
by Benjamin Huth 07 Sep '26

07 Sep '26

Hey folks, I opened a bug report on the ceph issue tracker a couple months ago detailing how the mount sharing for the cephfs kernel client was broken and how it was impacting me as a talos user with lots of pods sharing the same, readonly cephfs mounts. The bug report is here: https://tracker.ceph.com/issues/78440 I noticed some work being done in approximately the same area by Xiubo Li from Clyso, and was hoping someone could piggyback my fix into what he's doing: https://github.com/ceph/ceph-client/commit/b9d944f1c8c2044d08609fdc238d27df… My patch looks like this and is compatible with his new SB_ACTIVE flag: fs/ceph/super.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/ceph/super.c b/fs/ceph/super.c index 72935f665f11..13a8f246f1a0 100644 --- a/fs/ceph/super.c +++ b/fs/ceph/super.c @@ -1265,7 +1265,8 @@ static int ceph_compare_super(struct super_block *sb, struct fs_context *fc) doutc(cl, "fsid doesn't match\n"); return 0; } - if (fc->sb_flags != (sb->s_flags & ~SB_BORN)) { + if (fc->sb_flags != (sb->s_flags & + ~(SB_BORN | SB_NODIRATIME | SB_NOATIME | SB_ACTIVE))) { doutc(cl, "flags differ\n"); return 0; } base-commit: dc173b37415e8f738fc4de477490056b479ddc9f I've been running this patch on my talos cluster and it does correctly resolve pods not sharing the same mount and inode cache. Thanks!

1 0

Authentication looks broken
by Sake 06 Sep '26

06 Sep '26

Hi all! We upgraded a staging environment from 19.2.4 to 19.2.6 and rotated all of the keys without issues for the cluster. But for the Cephfs clients we noticed something strange. We created new keys for the Cephfs (kernel) clients so they could switch-over with just a remount. All went fine and thus we removed the old keys. Still everything kept working. Only then we rebooted one of the nodes to apply a new kernel. But it wasn't accessible after a couple of minutes, so we checked the console output and there where authentication errors for mds and mon. With a reset the node got online again, but without it's cephfs mount. When investigating the issue, we forgot to adjust the client name in the fstab. It was using the old name instead of the new name. So it looks like when mounting a cephfs filesystem, the backend isn't checking if the combination of client name and key is correct. Just if either is correct, you can mount the filesystem. Also, why isn't the umount failing after some time when authentication fails and then reboots the machine? Are others hitting this issue also? I'm going to check this on another system later on, but I thought I share this observation. System info Ceph is deployed via Cephadm Clients are running on RHEL 9 Using kernel mounts Installed ceph-common 19.2.6 on clients Kind regards, Sake

1 0

Unable to connect to filesystem Tentacle 20.2.4
by Johan 05 Sep '26

05 Sep '26

I have a cephadm managed cluster, Tentacle 20.2.4 and Debian 12 & 13. On the Debian 12 Bookworm servers I can connect to the cluster. id: <ceph-id> health: HEALTH_OK (muted: AUTH_INSECURE_CLIENT_KEY_TYPE(13d AUTH_INSECURE_KEYS_ALLOWED(13d)) services: mon: 3 daemons, quorum lagenhet4,lagenhet3,lagenhet5 (age 117m) [leader: lagenhet4] mgr: lagenhet3.rbkplm(active, since 2h), standbys: lagenhet4.xontzc mds: 1/1 daemons up, 1 standby osd: 8 osds: 8 up (since 2h), 8 in (since 12d) ... Worth noting is that my osd-disks are encrypted and those keys are not rotated. Keys for the crash services are not rotated either (a number of bootstrap-users are left too, but read in a earlier conversation that those should be safe to delete). One client that I previously have been using to connect to a cephfs have had the key rotated, but I can't connect to the filesystem. If the keyfile is written as before [client.fsuser] key = <key> or [client.fsuser] key = <key> caps mds = "allow rw fsname=filesystem" caps mon = "allow r fsname=filesystem" caps osd = "allow rw tag cephfs data=filesystem" and only replace the key I get the error adding ceph secret key to kernel failed: Unknown error 524 couldn't append secret option: -524 when I try to mount the filesystem. If I only have the key itself in the keyfile I get mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorised To test if I have proper authorisation I created a new user sudo ceph fs authorize newuser client.newuser / rw | sudo tee /etc/ceph/ceph.client.newuser.keyring and I still get the same errors. Looking at the keys in the dashboard it shows the same keys as I'm using. I tested to create a new filesystem (information regarding previous filesystems are removed) # ceph osd pool create dummy_data pool 'dummy_data' created # ceph osd pool create dummy_meta pool 'dummy_meta' created # ceph fs new dummyfs dummy_meta dummy_data Pool 'dummy_data' (id '21') has pg autoscale mode 'on' but is not marked as bulk. Consider setting the flag by running # ceph osd pool set dummy_data bulk true new fs with metadata pool 22 and data pool 21 # ceph fs ls name: dummyfs, metadata pool: dummy_meta, data pools: [dummy_data ] # sudo ceph orch apply mds dummyfs Scheduled mds.dummyfs update... # ceph mds stat <deleted>:1 dummyfs:1 {<deleted>:0=<deleted>.lagenhet4.kvscuw=up:active,dummyfs:0=dummyfs.lagenhet5.jpqcdl=up:active} 2 up:standby # ceph fs authorize dummyfs client.dummyuser / rw | tee /etc/ceph/ceph.client.dummyuser.keyring [client.dummyuser] key = AgBvWpxqEJPpOiAAjr6ziBqhdz23pCcU/2pCeY2ZGHNVKV4pAt13jCeKxr0= caps mds = "allow rw fsname=dummyfs" caps mon = "allow r fsname=dummyfs" caps osd = "allow rw tag cephfs data=dummyfs" # mount -t ceph dummyuser@.dummyfs=/ /datastorage/dummyfs adding ceph secret key to kernel failed: Unknown error 524 couldn't append secret option: -524 I manually change keyring-file to include only the key. # mount -t ceph dummyuser@.dummyfs=/ /datastorage/dummyfs mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized This procedure to create a filesystem has worked before. I also tested this procedure on my main cluster, Tentacle 20.2.1, and there was no problem and didn't have to edit keyring-file. Anyone have any idea what could be wrong? /Johan

2 1

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Johan 05 Sep '26

05 Sep '26

Sorry for the delayed answer, I have been setting up a new server to try handle the situation. Im using cephadm. The conainers are running fine as far as I can see. Checking both my two clusters I realise ceph-common was not updated since Reef. It hasn't been any problem until now, easy to forget when you have to manually update the release in apt source-file. I now have a server running Debian 12 Bookworm in the cluster (and updated ceph-common) and I can without problem connect to the cluster. id: <ceph-id> health: HEALTH_OK (muted: AUTH_INSECURE_CLIENT_KEY_TYPE(13d AUTH_INSECURE_KEYS_ALLOWED(13d)) services: mon: 3 daemons, quorum lagenhet4,lagenhet3,lagenhet5 (age 117m) [leader: lagenhet4] mgr: lagenhet3.rbkplm(active, since 2h), standbys: lagenhet4.xontzc mds: 1/1 daemons up, 1 standby osd: 8 osds: 8 up (since 2h), 8 in (since 12d) ... I have observed some other problems that I will address in another mail. /Johan Den 2026-08-26 kl. 08:44, skrev Eugen Block: > Hi, > > according to the "OS Recommendations" [0] Ceph Tentacle is not available > for Debian 13. Umbrella will be built for Trixie. > Are you using cephadm managed clusters or package based? With cephadm > you're not that strictly tied to the installed package versions. > > Regards, > Eugen > > [0] https://docs.ceph.com/en/latest/start/os-recommendations/ > > Zitat von Johan <ceph-users(a)ceph.io>: > >> Ok, >> >> on Debian 12 it was not a problem upgrading to ceph 20.2.4 using apt >> and https://download.ceph.com/debian-tentacle/ >> >> With Debian 13 which I have at my troublesome cluster I get several >> dependency-issues. >> ceph-common is dependent on libicu72 which doesn't have any candidates >> python3-cephfs (and others) is dependent on python3 < 3.12 but >> 3.13.5-1 will be installed >> etc >> >> Is this because I perhaps hadn't upgraded ceph-common and others >> properly before upgrading Debian from 12 to 13? >> >> /Johan >> >> Den 2026-08-24 kl. 11:06, skrev Johan: >>> Im not a professional computer tech but have two ceph clusters >>> (cephadm, Trixie, Debian 12/13). >>> >>> I upgraded the least sensitive one using the dashboard. When I >>> upgrade the admin client I now get >>> auth: error parsing file /etc/ceph/ceph.client.admin.keyring: error >>> setting modifier for [client.admin] type=key val=<key>: Malformed >>> input [buffer:3] >>> auth: failed to load /etc/ceph/ceph.client.admin.keyring: (5) >>> Input/ output error >>> >>> I get the same respons when using a ordinary user to connect the >>> filesystem. >>> >>> I found this issue describing the same error >>> https://github.com/rook/rook/issues/18240 >>> >>> As they, in above issue, discussed which version/image of the toolbox >>> was used I checked my version of ceph at the command line. I get >>> ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef >>> (stable) >>> I would guess this is the problem. >>> >>> How do I get to run up-to-date version on the command line? >>> >>> /Johan >>> >>> >>> Den 2026-08-19 kl. 18:09, skrev Patrick Donnelly: >>>> Today, August 19th, 2026, the Ceph project has released Tentacle >>>> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >>>> components. >>>> >>>> This is the sixth backport release in the Squid series. >>>> >>>> This is the fourth backport release in the Tentacle series. >>>> >>>> Additional details will be posted on the Ceph website/blog: >>>> https://ceph.io/en/news/blog/ >>>> >>>> We strongly recommend that all Ceph operators upgrade to one of these >>>> releases as soon as possible. >>>> >>>> >>>> ## Critical Upgrade Steps >>>> >>>> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >>>> This is the first time Ceph has introduced a new key type for CephX >>>> credentials. Therefore, a new procedure exists for upgrading and >>>> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >>>> cluster, the operator should familiarize themselves with the procedure >>>> documented at: >>>> >>>> https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/ >>>> index.html#upgrading-and-rotating-cephx-keys >>>> >>>> These steps are for package-based deployments of Ceph. In other >>>> sections, the document also includes new information about key >>>> rotation procedures, monitor authentication controls, and emergency >>>> recovery procedures. >>>> >>>> Deployments using cephadm will automate the process except for client >>>> keys. You may notice that Cephadm spends more time than normal on the >>>> upgrade after updating all daemon images. This is due to a new process >>>> rotating the OSD and MDS keys. Please be patient. There are plans to >>>> improve this in the future. >>>> >>>> Deployments using Rook will also automate rotation of some client keys >>>> with some exclusions. Look to the Rook project’s announcement for more >>>> details. >>>> >>>> Client and kernel upgrades are recommended to support aes256k. Note >>>> that upstream Linux kernel client support began in kernel 7.0 and has >>>> been backported to CentOS Stream 9 and 10. Check with your >>>> distribution vendor for backported key support before rotating client >>>> keys used by the kernel. >>>> >>>> As part of upgrading a Ceph cluster, six new health warnings and >>>> errors will be generated. This is normal. As part of the process to >>>> upgrade/rotate entity keys, you will address the warnings and errors >>>> as you go. These are documented beginning here: >>>> >>>> https://docs.ceph.com/en/latest/rados/operations/health-checks/ >>>> index.html#auth-insecure-keys-creatable >>>> >>>> The CephX documentation has been brought up to date and vetted for >>>> accuracy. Those changes are live here: >>>> >>>> https://docs.ceph.com/en/latest/architecture/index.html#high- >>>> availability-authentication >>>> >>>> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >>>> requests with `host` and `x-amz-` headers not included in the signed >>>> subset. Unfortunately, the REST client used in multisite was >>>> generating such improperly signed requests. >>>> >>>> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >>>> option to true before you begin to upgrade. After all clusters are >>>> upgraded, set the option to ``false`` again. >>>> >>>> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >>>> secrets stored in the Monitor config-key store will be forthcoming. In >>>> the meantime, operators of cephadm-managed clusters are strongly >>>> encouraged to rotate the cephadm SSH key, as an established process >>>> exists: >>>> >>>> See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh- >>>> configuration >>>> >>>> Rotating other secrets is advised but the process is not yet >>>> established. Operators should assess their cluster's potential >>>> exposure to determine if immediate Monitor config-key secret rotation >>>> is required for their environment. >>>> >>>> >>>> ## CVE Fixes >>>> >>>> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >>>> of AES-CBC. >>>> >>>> See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 >>>> >>>> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >>>> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >>>> in improper verification of a cryptographic signature. >>>> >>>> See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 >>>> >>>> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >>>> subscription handler. >>>> >>>> See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 >>>> >>>> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >>>> cryptographic signatures in RGW's SigV4 verifier. >>>> >>>> See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 >>>> >>>> ## Other Media >>>> >>>> A Cephalocon 2025 talk covers some of the details surrounding the new >>>> CephX key upgrade process. >>>> >>>> https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and- >>>> cephx- keys-patrick-donnelly-ibm-yehuda-sadeh-weinraub-ubiquiti >>>> >>>> ## Getting Ceph >>>> ------------ >>>> * Source at https://github.com/ceph/ceph >>>> * v19.2.6 Tarball at https://download.ceph.com/tarballs/ >>>> ceph-19.2.6.tar.gz >>>> * v20.2.4 Tarball at https://download.ceph.com/tarballs/ >>>> ceph-20.2.4.tar.gz >>>> * Containers at https://quay.io/repository/ceph/ceph >>>> * For packages, see https://docs.ceph.com/en/latest/install/get- >>>> packages/ >>>> * For upgrade with cephadm, see: >>>> https://docs.ceph.com/en/latest/cephadm/upgrade/ >>>> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >>>> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >>>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Issue with encrypted OSDs when upgrading from version 19.2.4 to 19.2.6 using cephadm
by Eugen Block 04 Sep '26

04 Sep '26

I did a lot of tests with a virtual test cluster in the past two weeks with encrypted OSDs and I haven't encountered that yet. I would check for ceph-osd and ceph-volume package on the affected hosts. Or for remainders from previous cephadm installations, anything not properly cleaned up maybe? What do the OSDs log when they try to start up? It sounds like some stale systemd units or something, hard to tell. Have you rebooted the hosts in question? On one of my lab clusters I noticed issues with systemd when I re-bootstrap a cluster without cephadm multiple times. I have a bash script that works just fine on a freshly rebooted system, but after wiping everything and starting over, at some point the script doesn't work anymore, especially when it comes to create OSDs. But this might be unrelated, just mentioning it. Zitat von Eugeniusz Pokora <ceph-users(a)ceph.io>: > When upgrading Ceph cluster from version 19.2.4 to 19.2.6 using > cephadm, an issue occurred with encrypted OSDs. > > 2026-09-02 10:53:28,494 7f24e08141c0 DEBUG Determined image: > 'quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089' > 2026-09-02 10:53:28,501 7f24e08141c0 INFO Reconfig daemon osd.36 ... > 2026-09-02 10:53:29,885 7f24e08141c0 INFO Stopping osd.36 to update > osd_key bluestore label > 2026-09-02 10:53:31,670 7f24e08141c0 INFO Opening osd.36 with crypsetup > 2026-09-02 10:53:33,955 7f24e08141c0 INFO Non-zero exit code 5 from > /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host > --entrypoint /tmp/cryptsetup_action.sh --privileged --group-add=disk > --init -e > CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089 -e NODE_NAME=k08r06cph03 -v /dev:/dev -v /tmp/cephadm-osd-36-rotate-helper.sh:/tmp/cryptsetup_action.sh > quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089 > 2026-09-02 10:53:33,956 7f24e08141c0 INFO /usr/bin/podman: stderr > Cannot use device > /dev/ceph-d1270182-40c9-49c8-9a52-f29d0475aeba/osd-block-80ae3d20-07df-4db5-96b6-2ad575a592ad which is in use (already mapped or > mounted). > 2026-09-02 10:53:33,956 7f24e08141c0 ERROR ERROR: Got error rotating > osd keyring while using cryptsetup tool > Out: > Err:Cannot use device > /dev/ceph-d1270182-40c9-49c8-9a52-f29d0475aeba/osd-block-80ae3d20-07df-4db5-96b6-2ad575a592ad which is in use (already mapped or > mounted). > > The issue did not occur on all hosts in the cluster (fortunately, it > was a test env), but it affected all OSDs on affected host. After > error occurred OSDs remained in a down state and had to be up > manually. > After several attempts, the update was successfully completed, but > we had to rotate the OSD keyrings ourselves. > A few hours latter when monmap was changed and cephadm scheduled a > reconfig this problem happened again, on 2 out of 4 nodes of the > cluster all osds goes down due to cephadm reconfig and this same > error: "Cannot use device... which is in use (already mapped or > mounted)". > > Has anyone encountered similar problems? > > Regards > Eugeniusz P

1 0

Re: Upgrade 19.2.6 to 20.2.4 changes auth_cipher config
by Patrick Donnelly 04 Sep '26

04 Sep '26

On Fri, Sep 4, 2026 at 3:06 AM Eugen Block <ceph-users(a)ceph.io> wrote: > > Oops, I hit send a bit too early. > > I wanted to ask if this is by design or if it's worth a bug report. It looks like a bug. Please make a ticket. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 1

19.2.7 QE validation status
by Yuri Weinstein 04 Sep '26

04 Sep '26

Details of this release are summarized here: https://tracker.ceph.com/issues/78483#note-7 Release Notes (drafts) - https://github.com/ceph/ceph/pull/71539 https://github.com/ceph/ceph.io/pull/1198 LRC/SRC upgrade - TBD (Patrick,Dan / The Core Team) PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER! Seeking approvals/reviews for: smoke: Laura rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson, Eric fs: Venky orch/ceph-volume: Redouane & Guillaume rbd: Ilya krbd: Ilya upgrade/quincy-x: PASSED upgrade/reef-x: PASSED upgrade/telemetry-upgrade: PASSED upgrade/squid-p2p: Laura, Patrick upgrade-clients:client-upgrade-reef-squid-squid: FAILED (suspect https://tracker.ceph.com/issues/79948) upgrade-clients:client-upgrade-quincy-squid-squid: PASSED perf-basic: PASSED powercycle: PASSED TIA

1 0

Maintenance Notification: Mailing List Migration 9/4
by David Galloway 04 Sep '26

04 Sep '26

Hello, Just a quick heads up that the mailing lists will be migrating back into the Sepia lab on 9/4 at 10am Eastern time. Aside from no longer needing to pay AWS for the service, additional benefits we’ll see from this migration include: * Upgrade from mailman 3.3.1 to 3.3.10. This version includes a task runner that will automatically clean up stuck list subscriptions. This was the main cause for the Web UI hanging and going down over the past few years. * Postgres DB migration from 9.6 to highly available Crunchy DB based on Postgres 16 * Lab-wide SMTP relay * Modern anti-spam stack (postscreen, rspamd) * In-house DKIM signing (mail-tester.com rates us a 8.9/10 score) * Mailing alias (vs. an entire list) support * Proper reverse DNS (helps our spam score) The migration itself shouldn’t take more than 30 minutes. It’s already been tested and everything that can be staged ahead of time is already ready. -- David Galloway Ceph Engineering Labs – Infrastructure Architect david.galloway(a)ibm.com IBM

1 3

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Eugen Block 04 Sep '26

04 Sep '26

I wonder if the upgrade actually finished then. What's the output of 'ceph orch upgrade status'? Mask sensitive data if necessary. Is it paused? If it is, is there a reason for the pause in the output? You could try resuming it: ceph orch upgrade resume You could redeploy a single daemon and see if that works: ceph orch daemon redeploy mgr.XXX quay.io/ceph/ceph:19.2.6 Or restart the upgrade again in case it was incomplete. Note that this will again rotate all service keyrings (no client keyrings will be touched). Regards, Eugen Zitat von Stig Telfer <ceph-users(a)ceph.io>: > I hit the same issue in the docs. > > Following the upgrade from 19.2.5 to 19.2.6, I still see MDS, OSD > and MGR services failing the health check > AUTH_INSECURE_SERVICE_KEY_TYPE. It looks like Cephadm hit an > unexpected issue and never completed the job of key rotation. > > Is there a way to trigger Cephadm to attempt this again? There's a > lot of daemons to manually process otherwise. > > Thanks, > Stig > >> On 20 Aug 2026, at 14:47, Adam King <ceph-users(a)ceph.io> wrote: >> >> That command got removed for the time being as it used an old way of doing >> the key rotation that wasn't built to work with the changes for the cve and >> seemed to be broken for a number of daemon types. Since the name would make >> people want to use it it was decided to just temporarily disables it so we >> can fix it and reintroduce it in a new version as an easier way to rotate >> the keys. For any keys tied to a cephadm deployed daemon that aren't >> automatically rotated (cephadm handles mgr, mon, osd, and mds keyrings >> during upgrade) it's a matter of `ceph auth rotate <entity-name> --key-type >> aes256k` then `ceph orch daemon redeploy <daemon-name>`. For anyone using >> nfs exports managed through the nfs module the export's keyrings also need >> to be rotated. `ceph auth rotate <export-keyring-entity> --key-type >> aes256k`, `ceph nfs export info <nfs-cluster-name> <pseudo-path> > >> export_file`, `ceph nfs export apply <nfs-cluster-name> -i export_file`. >> Between rotating the keyrings for the nfs cluster/daemons and rotating the >> keyring for the individual export(s) those export(s) are unavailable iirc. >> >> On Thu, Aug 20, 2026 at 7:11 AM Pedro Macedo <ceph-users(a)ceph.io> wrote: >> >>> >>> On 2026-08-20 12:57 PM, Maximilian Stinsky-Damke wrote: >>>> Hi, >>>> >>>> I upgraded a test cluster to Ceph Squid 19.2.6 via cephadm. The >>> following cephadm managed client keys were not rotated automatically: >>>> >>>> client.bootstrap-* (bootstrap-rgw, bootstrap-osd, ...) >>>> client.ceph-exporter. >>>> client.crash. >>>> client.osd-lockbox.. >>>> client.rgw. >>>> >>>> The Squid docs mention ceph orch daemon rotate-key <name>, but in 19.2.* >>> this command does not exist. >>> >>> Also seeing this on the 20.2.4 image via cephadm - rotate-key is missing >>> there, but present on 20.2.3. >>> >>>> >>> https://docs.ceph.com/en/squid/cephadm/services/#deploying-a-daemon-on-a-ho… >>>> >>> https://github.com/ceph/ceph/blob/v19.2.6/src/pybind/mgr/cephadm/module.py#… >>>> >>>> What is the correct way to rotate these client keys via cephadm as part >>> of the cve fix? >>>> >>>> Thanks for any guidance. >>>> Best Regards >>>> Max >>>> >>>> ________________________________ >>>> From: Patrick Donnelly <ceph-users(a)ceph.io> >>>> Sent: 19 August 2026 18:08 >>>> To: ceph-users <ceph-users(a)ceph.io> >>>> Subject: [ceph-users] [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 >>> released >>>> >>>> Warning: This email is from an unusual correspondent. >>>> Warning: Make sure this is someone you trust. >>>> >>>> Today, August 19th, 2026, the Ceph project has released Tentacle >>>> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >>>> components. >>>> >>>> This is the sixth backport release in the Squid series. >>>> >>>> This is the fourth backport release in the Tentacle series. >>>> >>>> Additional details will be posted on the Ceph website/blog: >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fceph.io%2Fen%2Fnews%2Fblog%2F… >>>> >>>> We strongly recommend that all Ceph operators upgrade to one of these >>>> releases as soon as possible. >>>> >>>> >>>> ## Critical Upgrade Steps >>>> >>>> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >>>> This is the first time Ceph has introduced a new key type for CephX >>>> credentials. Therefore, a new procedure exists for upgrading and >>>> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >>>> cluster, the operator should familiarize themselves with the procedure >>>> documented at: >>>> >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> >>>> These steps are for package-based deployments of Ceph. In other >>>> sections, the document also includes new information about key >>>> rotation procedures, monitor authentication controls, and emergency >>>> recovery procedures. >>>> >>>> Deployments using cephadm will automate the process except for client >>>> keys. You may notice that Cephadm spends more time than normal on the >>>> upgrade after updating all daemon images. This is due to a new process >>>> rotating the OSD and MDS keys. Please be patient. There are plans to >>>> improve this in the future. >>>> >>>> Deployments using Rook will also automate rotation of some client keys >>>> with some exclusions. Look to the Rook project’s announcement for more >>>> details. >>>> >>>> Client and kernel upgrades are recommended to support aes256k. Note >>>> that upstream Linux kernel client support began in kernel 7.0 and has >>>> been backported to CentOS Stream 9 and 10. Check with your >>>> distribution vendor for backported key support before rotating client >>>> keys used by the kernel. >>>> >>>> As part of upgrading a Ceph cluster, six new health warnings and >>>> errors will be generated. This is normal. As part of the process to >>>> upgrade/rotate entity keys, you will address the warnings and errors >>>> as you go. These are documented beginning here: >>>> >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> >>>> The CephX documentation has been brought up to date and vetted for >>>> accuracy. Those changes are live here: >>>> >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> >>>> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >>>> requests with `host` and `x-amz-` headers not included in the signed >>>> subset. Unfortunately, the REST client used in multisite was >>>> generating such improperly signed requests. >>>> >>>> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >>>> option to true before you begin to upgrade. After all clusters are >>>> upgraded, set the option to ``false`` again. >>>> >>>> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >>>> secrets stored in the Monitor config-key store will be forthcoming. In >>>> the meantime, operators of cephadm-managed clusters are strongly >>>> encouraged to rotate the cephadm SSH key, as an established process >>>> exists: >>>> >>>> See: >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> >>>> Rotating other secrets is advised but the process is not yet >>>> established. Operators should assess their cluster's potential >>>> exposure to determine if immediate Monitor config-key secret rotation >>>> is required for their environment. >>>> >>>> >>>> ## CVE Fixes >>>> >>>> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >>>> of AES-CBC. >>>> >>>> See: >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> >>>> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >>>> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >>>> in improper verification of a cryptographic signature. >>>> >>>> See: >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> >>>> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >>>> subscription handler. >>>> >>>> See: >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> >>>> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >>>> cryptographic signatures in RGW's SigV4 verifier. >>>> >>>> See: >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> >>>> ## Other Media >>>> >>>> A Cephalocon 2025 talk covers some of the details surrounding the new >>>> CephX key upgrade process. >>>> >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fcephalocon2025.sched.com%2Fev… >>>> >>>> ## Getting Ceph >>>> ------------ >>>> * Source at >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fgithub.com%2Fceph%2Fceph&e=50… >>>> * v19.2.6 Tarball at >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… >>>> * v20.2.4 Tarball at >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… >>>> * Containers at >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fquay.io%2Frepository%2Fceph%2… >>>> * For packages, see >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> * For upgrade with cephadm, see: >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >>>> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >>>> >>>> -- >>>> Patrick Donnelly, Ph.D. >>>> He / Him / His >>>> Red Hat Partner Engineer >>>> IBM, Inc. >>>> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>>> -- >>>> This message has been checked by Libraesva ESG and is found to be clean. >>>> Follow this link to report as spam/bad: >>>> https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/report-as-bad >>>> Follow this link to blocklist sender: >>>> https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/blocklist >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Issue with encrypted OSDs when upgrading from version 19.2.4 to 19.2.6 using cephadm
by Eugeniusz Pokora 04 Sep '26

04 Sep '26

When upgrading Ceph cluster from version 19.2.4 to 19.2.6 using cephadm, an issue occurred with encrypted OSDs. 2026-09-02 10:53:28,494 7f24e08141c0 DEBUG Determined image: 'quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089' 2026-09-02 10:53:28,501 7f24e08141c0 INFO Reconfig daemon osd.36 ... 2026-09-02 10:53:29,885 7f24e08141c0 INFO Stopping osd.36 to update osd_key bluestore label 2026-09-02 10:53:31,670 7f24e08141c0 INFO Opening osd.36 with crypsetup 2026-09-02 10:53:33,955 7f24e08141c0 INFO Non-zero exit code 5 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /tmp/cryptsetup_action.sh --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089 -e NODE_NAME=k08r06cph03 -v /dev:/dev -v /tmp/cephadm-osd-36-rotate-helper.sh:/tmp/cryptsetup_action.sh quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089 2026-09-02 10:53:33,956 7f24e08141c0 INFO /usr/bin/podman: stderr Cannot use device /dev/ceph-d1270182-40c9-49c8-9a52-f29d0475aeba/osd-block-80ae3d20-07df-4db5-96b6-2ad575a592ad which is in use (already mapped or mounted). 2026-09-02 10:53:33,956 7f24e08141c0 ERROR ERROR: Got error rotating osd keyring while using cryptsetup tool Out: Err:Cannot use device /dev/ceph-d1270182-40c9-49c8-9a52-f29d0475aeba/osd-block-80ae3d20-07df-4db5-96b6-2ad575a592ad which is in use (already mapped or mounted). The issue did not occur on all hosts in the cluster (fortunately, it was a test env), but it affected all OSDs on affected host. After error occurred OSDs remained in a down state and had to be up manually. After several attempts, the update was successfully completed, but we had to rotate the OSD keyrings ourselves. A few hours latter when monmap was changed and cephadm scheduled a reconfig this problem happened again, on 2 out of 4 nodes of the cluster all osds goes down due to cephadm reconfig and this same error: "Cannot use device... which is in use (already mapped or mounted)". Has anyone encountered similar problems? Regards Eugeniusz P

1 0

Issue with encrypted OSDs when upgrading from version 19.2.4 to 19.2.6 using cephadm
by Eugeniusz Pokora 04 Sep '26

04 Sep '26

1 0

Upgrade 19.2.6 to 20.2.4 changes auth_cipher config
by Eugen Block 04 Sep '26

04 Sep '26

Hi, I just tested the upgrade from a single-node mini cluster (only mgr and mon daemons) bootstrapped with 19.2.6 to 20.2.4. After the bootstrap I have no warnings (except that I don't have any OSDs), this is the excerpt from the mon dump: min_mon_release 19 (squid) election_strategy: 1 0: [v2:192.168.124.6:3300/0,v1:192.168.124.6:6789/0] mon.tentacle auth_epoch 0 auth_service_cipher aes256k auth_allowed_ciphers aes256k auth_preferred_cipher aes256k After the upgrade to 20.2.4 the mon dump looks like this (look at auth_allowed_ciphers): min_mon_release 20 (tentacle) election_strategy: 1 0: [v2:192.168.124.6:3300/0,v1:192.168.124.6:6789/0] mon.tentacle auth_epoch 0 auth_service_cipher aes256k auth_allowed_ciphers aes, aes256k auth_preferred_cipher aes256k Obviously, the cluster now warns about the MONs allowing insecure keys: > Monitors are configured to allow auth using insecure key types > Monitors are configured to allow creation of insecure key types

1 1

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Stig Telfer 03 Sep '26

03 Sep '26

I hit the same issue in the docs. Following the upgrade from 19.2.5 to 19.2.6, I still see MDS, OSD and MGR services failing the health check AUTH_INSECURE_SERVICE_KEY_TYPE. It looks like Cephadm hit an unexpected issue and never completed the job of key rotation. Is there a way to trigger Cephadm to attempt this again? There's a lot of daemons to manually process otherwise. Thanks, Stig > On 20 Aug 2026, at 14:47, Adam King <ceph-users(a)ceph.io> wrote: > > That command got removed for the time being as it used an old way of doing > the key rotation that wasn't built to work with the changes for the cve and > seemed to be broken for a number of daemon types. Since the name would make > people want to use it it was decided to just temporarily disables it so we > can fix it and reintroduce it in a new version as an easier way to rotate > the keys. For any keys tied to a cephadm deployed daemon that aren't > automatically rotated (cephadm handles mgr, mon, osd, and mds keyrings > during upgrade) it's a matter of `ceph auth rotate <entity-name> --key-type > aes256k` then `ceph orch daemon redeploy <daemon-name>`. For anyone using > nfs exports managed through the nfs module the export's keyrings also need > to be rotated. `ceph auth rotate <export-keyring-entity> --key-type > aes256k`, `ceph nfs export info <nfs-cluster-name> <pseudo-path> > > export_file`, `ceph nfs export apply <nfs-cluster-name> -i export_file`. > Between rotating the keyrings for the nfs cluster/daemons and rotating the > keyring for the individual export(s) those export(s) are unavailable iirc. > > On Thu, Aug 20, 2026 at 7:11 AM Pedro Macedo <ceph-users(a)ceph.io> wrote: > >> >> On 2026-08-20 12:57 PM, Maximilian Stinsky-Damke wrote: >>> Hi, >>> >>> I upgraded a test cluster to Ceph Squid 19.2.6 via cephadm. The >> following cephadm managed client keys were not rotated automatically: >>> >>> client.bootstrap-* (bootstrap-rgw, bootstrap-osd, ...) >>> client.ceph-exporter. >>> client.crash. >>> client.osd-lockbox.. >>> client.rgw. >>> >>> The Squid docs mention ceph orch daemon rotate-key <name>, but in 19.2.* >> this command does not exist. >> >> Also seeing this on the 20.2.4 image via cephadm - rotate-key is missing >> there, but present on 20.2.3. >> >>> >> https://docs.ceph.com/en/squid/cephadm/services/#deploying-a-daemon-on-a-ho… >>> >> https://github.com/ceph/ceph/blob/v19.2.6/src/pybind/mgr/cephadm/module.py#… >>> >>> What is the correct way to rotate these client keys via cephadm as part >> of the cve fix? >>> >>> Thanks for any guidance. >>> Best Regards >>> Max >>> >>> ________________________________ >>> From: Patrick Donnelly <ceph-users(a)ceph.io> >>> Sent: 19 August 2026 18:08 >>> To: ceph-users <ceph-users(a)ceph.io> >>> Subject: [ceph-users] [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 >> released >>> >>> Warning: This email is from an unusual correspondent. >>> Warning: Make sure this is someone you trust. >>> >>> Today, August 19th, 2026, the Ceph project has released Tentacle >>> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >>> components. >>> >>> This is the sixth backport release in the Squid series. >>> >>> This is the fourth backport release in the Tentacle series. >>> >>> Additional details will be posted on the Ceph website/blog: >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fceph.io%2Fen%2Fnews%2Fblog%2F… >>> >>> We strongly recommend that all Ceph operators upgrade to one of these >>> releases as soon as possible. >>> >>> >>> ## Critical Upgrade Steps >>> >>> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >>> This is the first time Ceph has introduced a new key type for CephX >>> credentials. Therefore, a new procedure exists for upgrading and >>> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >>> cluster, the operator should familiarize themselves with the procedure >>> documented at: >>> >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> >>> These steps are for package-based deployments of Ceph. In other >>> sections, the document also includes new information about key >>> rotation procedures, monitor authentication controls, and emergency >>> recovery procedures. >>> >>> Deployments using cephadm will automate the process except for client >>> keys. You may notice that Cephadm spends more time than normal on the >>> upgrade after updating all daemon images. This is due to a new process >>> rotating the OSD and MDS keys. Please be patient. There are plans to >>> improve this in the future. >>> >>> Deployments using Rook will also automate rotation of some client keys >>> with some exclusions. Look to the Rook project’s announcement for more >>> details. >>> >>> Client and kernel upgrades are recommended to support aes256k. Note >>> that upstream Linux kernel client support began in kernel 7.0 and has >>> been backported to CentOS Stream 9 and 10. Check with your >>> distribution vendor for backported key support before rotating client >>> keys used by the kernel. >>> >>> As part of upgrading a Ceph cluster, six new health warnings and >>> errors will be generated. This is normal. As part of the process to >>> upgrade/rotate entity keys, you will address the warnings and errors >>> as you go. These are documented beginning here: >>> >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> >>> The CephX documentation has been brought up to date and vetted for >>> accuracy. Those changes are live here: >>> >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> >>> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >>> requests with `host` and `x-amz-` headers not included in the signed >>> subset. Unfortunately, the REST client used in multisite was >>> generating such improperly signed requests. >>> >>> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >>> option to true before you begin to upgrade. After all clusters are >>> upgraded, set the option to ``false`` again. >>> >>> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >>> secrets stored in the Monitor config-key store will be forthcoming. In >>> the meantime, operators of cephadm-managed clusters are strongly >>> encouraged to rotate the cephadm SSH key, as an established process >>> exists: >>> >>> See: >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> >>> Rotating other secrets is advised but the process is not yet >>> established. Operators should assess their cluster's potential >>> exposure to determine if immediate Monitor config-key secret rotation >>> is required for their environment. >>> >>> >>> ## CVE Fixes >>> >>> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >>> of AES-CBC. >>> >>> See: >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> >>> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >>> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >>> in improper verification of a cryptographic signature. >>> >>> See: >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> >>> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >>> subscription handler. >>> >>> See: >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> >>> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >>> cryptographic signatures in RGW's SigV4 verifier. >>> >>> See: >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> >>> ## Other Media >>> >>> A Cephalocon 2025 talk covers some of the details surrounding the new >>> CephX key upgrade process. >>> >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fcephalocon2025.sched.com%2Fev… >>> >>> ## Getting Ceph >>> ------------ >>> * Source at >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fgithub.com%2Fceph%2Fceph&e=50… >>> * v19.2.6 Tarball at >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… >>> * v20.2.4 Tarball at >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… >>> * Containers at >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fquay.io%2Frepository%2Fceph%2… >>> * For packages, see >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> * For upgrade with cephadm, see: >>> >> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >>> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >>> >>> -- >>> Patrick Donnelly, Ph.D. >>> He / Him / His >>> Red Hat Partner Engineer >>> IBM, Inc. >>> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> -- >>> This message has been checked by Libraesva ESG and is found to be clean. >>> Follow this link to report as spam/bad: >>> https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/report-as-bad >>> Follow this link to blocklist sender: >>> https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/blocklist >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Ceph Days Chicago Call For Presentations
by Anthony Middleton 03 Sep '26

03 Sep '26

Hello, Ceph friends! I hope this week is going well. I'm sending this email on behalf of the Ceph Days Chicago <https://ceph.io/en/community/events/2026/ceph-days-chicago/> organizers. We are pleased to announce that Ceph Days Chicago 2026 is accepting submissions for presentations and talks. This is an opportunity to share your experience, insights, and contributions with the Ceph community. We are seeking presentations that cover: - Project updates and recent improvements - Ceph deployment and operational experience - Use cases and real-world applications - Technical architecture and design discussions - Roadmap and future directions Important Dates - Call for Papers closes: *Thursday, October 15, 2026* - Event date: *November 16, 2026* - Location: *University of Chicago, 1201 E 60th St, Chicago, IL 60637* Submission Details To submit your proposal, please visit: https://forms.gle/vdcHycWuWTEyDbxz8 The Ceph Days Chicago team welcomes submissions from community members at all experience levels. For questions about your submission or event details, please contact ceph-days-chicago(a)ceph.io. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

umbrella 21.1.1 RC2 QE validation status
by Yuri Weinstein 03 Sep '26

03 Sep '26

Details of this release are summarized here: https://tracker.ceph.com/issues/78413 Release Notes - email version for RCs https://github.com/ceph/ceph/pull/70240#issuecomment-5232852937 @Neha Ojha <nojha(a)redhat.com> @Patrick Donnelly <pdonnell(a)redhat.com> LRC upgrade - TBD PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER! Seeking approvals/reviews for: smoke: PASSED rados: Kamoltat, Radek, Nitzan, Afreen (dashboard), Travis (rook) rgw: Adam Emerson fs: Venky orch/ceph-volume: Redouane, Guillaume rbd: Ilya krbd: Ilya nvmeof: PASSED upgrade: Kamoltat upgrade-clients:client-upgrade-tentacle-umbrella-umbrella: PASSED upgrade-clients:client-upgrade-squid-umbrella-umbrella: PASSED perf-basic: PASSED crimson-rados: Matan powercycle: Brad (same as before) TIA

1 1

Release notes update: don't (try to) upgrade from v19.2.6 to Tentacle < 20.2.4
by Eugen Block 03 Sep '26

03 Sep '26

Hi, I know it might be unlikely, but not all users are (yet) subscribed to this list or for some reason can't catch up. I feel like the docs should clearly state that you can't upgrade from 19.2.6 to Tentacle < 20.2.4. I'm almost sure that there will be users tying to do exactly that just because 20 > 19. :-) Regards, Eugen

1 0

Orch stops refreshing daemon list
by Chris Palmer 02 Sep '26

02 Sep '26

I have a problem following a 20.2.3 -> 20.2.4 upgrade. This has happened once on a virtualised test cluster, and now on a small production cluster. After it happened on test, I rolled all nodes back to snapshots and tried again, and it worked, so I assumed finger trouble and proceeded to the production cluster. But now it is stuck... - The upgrade itself is completely successful. Repeated "ceph orch ps --refresh" looks good and shows the "refreshed" value returning to zero each time. - I then rotate the SSH keys, and everything at first appears ok. "check-host" works between every pair of hosts in each direction. - But... "ceph orch ps --refresh" has now stopped refreshing. And any attempts to restart or redeploy a daemon all say it is scheduled but nothing happens. I've tried failing MGRs, rebooting a node then failing back to the MGR on it. There's nothing odd that I can spot in the MGR logs. After the reboot, "ceph orch ps --refresh" still shows the OLD container IDs, and an ever-increasing last-refresh time. I've also rotated the client.admin key on the production cluster without any problems. I don't think that is causing this orch problem, as the test cluster originally failed before client.admin had been rotated. Can anyone suggest how I may troubleshoot this? Regards, Chris

1 0

Re: Tentacle v20.2.4 packages build for Ubuntu Noble (24.04)
by Patrick Donnelly 02 Sep '26

02 Sep '26

On Wed, Sep 2, 2026 at 7:07 AM Eugen Block <ceph-users(a)ceph.io> wrote: > > Hi Patrick, > > thanks for the efforts to bring 20.2.4 to Noble! > > I just installed those packages on a virtual machine that I use for > testing. I can create a cluster successfully (package based), but the > MGR ends up in a respawn loop, reporting these error messages: > > ---snip--- > Sep 02 10:27:10 tentacle-ubuntu24 ceph-mgr[3289]: Failed to import > NVMeoFClient and related components: cannot import name 'NVMeoFClient' > from 'dashboard.services.nvmeof_client' > (/usr/share/ceph/mgr/dashboard/services/nvmeof_client.py) > Sep 02 10:27:10 tentacle-ubuntu24 ceph-mgr[3289]: ERROR:root:Module > 'xmltodict' is not installed. > ---snip--- > > The xmltodict error can easily be fixed by installing > python3-xmltodict, but it probably should be a dependency then when > installing the ceph-mgr package, I assume? Correct. That is an example of a change that will need backported for v20.2.5. > The NVMeoFClient message doesn't seem to be critical, at least I can't > notice anything obvious right now, but I also have no nvme service > deployed or anything. The dashboard also works fine. > > Should I create tracker for the xmltodict error? No need, I'll take care of it. Thank you for testing and the report! -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: Grafana certificate in version 19
by Redouane Kachach 02 Sep '26

02 Sep '26

I hope this helps: https://github.com/ceph/ceph/pull/66348#issuecomment-3641268398 Best, Redouane. On Fri, Aug 7, 2026 at 3:51 PM Chris Palmer via ceph-users < ceph-users(a)ceph.io> wrote: > I never got to the bottom of this, on either squid or tentacle. I have > an externally-generated cert/key. As a workaround I do this after each > upgrade (most recently to 20.2.3): > > |cp ~/my_grafana_cert.key /var/lib/ceph/`ceph fsid`/grafana.`hostname > -s`/etc/grafana/certs/cert_key cp ~/||my_grafana_cert||.crt > /var/lib/ceph/`ceph fsid`/grafana.||`hostname > -s`||/etc/grafana/certs/cert_file ceph orch daemon restart > grafana.||`hostname -s`||--force| > > It would be nice to know what the right way is. > > Regards, Chris > > > On 07/08/2026 13:51, Robert Sander via ceph-users wrote: > > Hi, > > > > how do I set the grafana certificate and key in Ceph 19 that is used on > > port 3000? > > > > The documentation on > > > https://docs.ceph.com/en/squid/cephadm/services/monitoring/#configuring-ssl… > > states: > > > > # ceph config-key set mgr/cephadm/{hostname}/grafana_key -i $PWD/key.pem > > # ceph config-key set mgr/cephadm/{hostname}/grafana_crt -i > > $PWD/certificate.pem > > # ceph orch reconfig grafana > > > > But after doing that grafana still serves a self-signed certificate with > > the common name containing the hostname of the first MON (which is not > > correct). > > > > Digging into "ceph config-key dump" shows two entries named > > mgr/cephadm/cert_store.cert.grafana_cert and > > mgr/cephadm/cert_store.key.grafana_key > > which seem to cause this. > > > > AFAIK the certmgr is only available in Ceph 20. But the docs for this > > version > > look the same: > > > https://docs.ceph.com/en/tentacle/cephadm/services/monitoring/#configuring-… > > > > Only the docs for Ceph 21 got an update and mention certmgr: > > > https://docs.ceph.com/en/umbrella/cephadm/services/monitoring/#configuring-… > > > > > > The orchestrator in Ceph 19 seems to have something called cert-store: > > > > # ceph orch cert-store > > no valid command found; 4 closest matches: > > orch cert-store cert ls [--format > > {plain|json|json-pretty|yaml|xml-pretty|xml}] > > orch cert-store key ls [--format > > {plain|json|json-pretty|yaml|xml-pretty|xml}] > > orch cert-store get cert <entity> [--service_name <value>] [--hostname > > <value>] [--no-exception-when-missing] > > orch cert-store get key <entity> [--service_name <value>] [--hostname > > <value>] [--no-exception-when-missing] > > Error EINVAL: invalid command > > > > But there is no help available. > > "ceph orch cert-store --help" does not return anything specific. > > Searching for "cert-store" in the Ceph 19 docs also does not return > > anything. > > > > Regards > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: Tentacle v20.2.4 packages build for Ubuntu Noble (24.04)
by Eugen Block 02 Sep '26

02 Sep '26

Hi Patrick, thanks for the efforts to bring 20.2.4 to Noble! I just installed those packages on a virtual machine that I use for testing. I can create a cluster successfully (package based), but the MGR ends up in a respawn loop, reporting these error messages: ---snip--- Sep 02 10:27:10 tentacle-ubuntu24 ceph-mgr[3289]: Failed to import NVMeoFClient and related components: cannot import name 'NVMeoFClient' from 'dashboard.services.nvmeof_client' (/usr/share/ceph/mgr/dashboard/services/nvmeof_client.py) Sep 02 10:27:10 tentacle-ubuntu24 ceph-mgr[3289]: ERROR:root:Module 'xmltodict' is not installed. ---snip--- The xmltodict error can easily be fixed by installing python3-xmltodict, but it probably should be a dependency then when installing the ceph-mgr package, I assume? The NVMeoFClient message doesn't seem to be critical, at least I can't notice anything obvious right now, but I also have no nvme service deployed or anything. The dashboard also works fine. Should I create tracker for the xmltodict error? Thanks! Eugen Zitat von Patrick Donnelly <ceph-users(a)ceph.io>: > Hi everyone, > > Due to an internal miscommunication, packages were inadvertently built > for Ubuntu Noble in Tentacle v20.2.1 [1] and v20.2.2 [2]. Following > those releases, I had published a roadmap matrix of distribution > support [3] which marked Noble support as "Upcoming" for Tentacle, as > testing had not yet been configured for it. Subsequently, the build > team used that roadmap as the authoritative source of support, so > Noble was not built or packaged for v20.2.3 or v20.2.4. > > We recognize this puts community clusters that were using Noble > packages for v20.2.1 and v20.2.2 in a difficult position due to the > CVE fixes. So we've generated signed Ubuntu 24.04 packages for v20.2.4 > (not v20.2.3). Please understand that these packages have not been > tested in any capacity beyond basic validation that they install. > > As I've mentioned in other threads on the ceph-users list, I expect > v20.2.5 [4] to have full and tested support for Ubuntu Noble. > > [1] https://download.ceph.com/debian-20.2.1/dists/noble/ > [2] https://download.ceph.com/debian-20.2.2/dists/noble/ > [3] https://docs.ceph.com/en/latest/start/os-recommendations/ > [4] https://tracker.ceph.com/issues/79030 > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Red Hat Partner Engineer > IBM, Inc. > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Tentacle v20.2.4 packages build for Ubuntu Noble (24.04)
by Patrick Donnelly 01 Sep '26

01 Sep '26

Hi everyone, Due to an internal miscommunication, packages were inadvertently built for Ubuntu Noble in Tentacle v20.2.1 [1] and v20.2.2 [2]. Following those releases, I had published a roadmap matrix of distribution support [3] which marked Noble support as "Upcoming" for Tentacle, as testing had not yet been configured for it. Subsequently, the build team used that roadmap as the authoritative source of support, so Noble was not built or packaged for v20.2.3 or v20.2.4. We recognize this puts community clusters that were using Noble packages for v20.2.1 and v20.2.2 in a difficult position due to the CVE fixes. So we've generated signed Ubuntu 24.04 packages for v20.2.4 (not v20.2.3). Please understand that these packages have not been tested in any capacity beyond basic validation that they install. As I've mentioned in other threads on the ceph-users list, I expect v20.2.5 [4] to have full and tested support for Ubuntu Noble. [1] https://download.ceph.com/debian-20.2.1/dists/noble/ [2] https://download.ceph.com/debian-20.2.2/dists/noble/ [3] https://docs.ceph.com/en/latest/start/os-recommendations/ [4] https://tracker.ceph.com/issues/79030 -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: Using CephFS for encrypted Kubernetes volumes
by Eugen Block 01 Sep '26

01 Sep '26

If you're suspecting ceph-csi to be the issue here, I recommend to create an issue here (if you haven't already): https://github.com/ceph/ceph-csi/ Zitat von Stefan Möding via ceph-users <ceph-users(a)ceph.io>: > Hi, > > thanks for your feedback! > > Eugen Block via ceph-users <ceph-users(a)ceph.io> writes: > >> that sounds a bit impractical to create a new pool for each tenant >> regarding PG and pool management (adding and possibly removing data >> pools from the CephFS), depending on how many tenants you have or plan >> to have. > > In our setup each K8S cluster would be on tenant. So the pools would be > on the subvolumegroup level and we expect to go ahead with less than ten > pools for the foreseeable future. > > >> What I just tested was to manually encrypt a directory in a test >> cluster with fscrypt, that seems to work as intended. But I don't have >> anything with k8s to test it at the moment. > > From what I have tested, the problem seems to be located in ceph-csi > where the encryption is initially set up. The code tries to allocates > an exclusive lock on an object representing the volume to prevent a race > condition with multiple clients sharing the volume. The relevant code > looks like this: > > ioctx, err := volOptions.GetConnection().GetIoctx(volOptions.MetadataPool) > if err != nil { > log.ErrorLog(ctx, "Failed to create ioctx: %s", err) > return err > } > lock := iolock.NewLock(ioctx, objectUUID, lockName, lockCookie, > lockDesc, lockDuration) > err = lock.LockExclusive(ctx) > if err != nil { > log.ErrorLog(ctx, "failed to create lock for volume ID %s: %v", volID, err) > return err > } > > The error message on the ceph/csi pods clearly shows that the > LockExclusive call fails with “Operation not permitted”. I failed to > find the place in the relevant code, where the namespace for this > operation is set, so it looks like the default namespace is used. But > the capabilities only allow a specific namespace and so this could be > the root cause. > > I tried to use the following code to create an object in the metadata > pool and lock it. I used the same user with the same capabilities: > > % rados --id=k8s-test --pool=cephfs.meta --namespace=k8s-test put > foobar /dev/null > % rados --id=k8s-test --pool=cephfs.meta --namespace=k8s-test lock > get foobar lock1 --lock-cookie=cookie --lock-duration=60 > --lock-type=exclusive > % rados --id=k8s-test --pool=cephfs.meta --namespace=k8s-test lock > info foobar lock1 > {"name":"lock1","type":"exclusive","tag":"","lockers":[{"name":"client.2690590","cookie":"cookie","description":"","expiration":"2026-07-31T08:21:36.190835+0200","addr":"172.21.70.10:0/2292729948"}]} > > So the capabilities seem to be sufficient for the lock operation. But > leaving out the namespace when creating the lock will show the exact > same error message that the ceph-csi shows: > > % rados --id=k8s-test --pool=cephfs.meta --namespace=k8s-test put > foobar /dev/null > % rados --id=k8s-test --pool=cephfs.meta lock get foobar lock1 > --lock-cookie=cookie --lock-duration=60 --lock-type=exclusive > ERROR: failed locking: (1) Operation not permitted > error 1: (1) Operation not permitted > > -- > Stefan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0