Consistent OSD crashes for ceph 17.2.5 which is causing osd up and down - ceph-users

27 Dec 2023

We are running rook-ceph deployed as a operator in kubernetes with rook
version 1.10.8 and ceph 17.2.5.

Its working fine but we are seeing frequent OSD daemon crash in 3-4 days
and restarts without any problem also we are seeing flapping osds i.e osd
up down.

Recently daemon crash happened for 2 OSDs at same time on different nodes
with below error in crash info :

-305> 2023-12-17T14:50:14.413+0000 7f53b5f91700 -1 *** Caught signal
(Aborted) **
 in thread 7f53b5f91700 thread_name:tp_osd_tp
 ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)
 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f53d93ddcf0]
 2: gsignal()
 3: abort()
 4: /lib64/libc.so.6(+0x21d79) [0x7f53d8025d79]
 5: /lib64/libc.so.6(+0x47456) [0x7f53d804b456]
 6: (MOSDRepOp::encode_payload(unsigned long)+0x2d0) [0x55acc0f81730]
 7: (Message::encode(unsigned long, int, bool)+0x2e) [0x55acc140ec2e]
 8: (ProtocolV2::send_message(Message*)+0x25e) [0x55acc16a5aae]
 9: (AsyncConnection::send_message(Message*)+0x18e) [0x55acc167dc4e]
 10: (OSDService::send_message_osd_cluster(int, Message*, unsigned
int)+0x2bd) [0x55acc0b4b11d]
 11: (ReplicatedBackend::issue_op(hobject_t const&, eversion_t const&,
unsigned long, osd_reqid_t, eversion_t, eversion_t, hobject_t,
hobject_t, std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t>
...
  const&,
std::optional<pg_hit_set_history_t>&, ReplicatedBackend::InProgressOp*,
ceph::os::Transaction&)+0x6c8)
[0x55acc0f69368]
 12: (ReplicatedBackend::submit_transaction(hobject_t const&,
object_stat_sum_t const&, eversion_t const&,
std::unique_ptr<PGTransaction, std::default_delete<PGTransaction>
>&&,
eversion_t const&, eversion_t const&, std::vector<pg_log_entry_t,
std::allocator<pg_log_entry_t> >&&,
std::optional<pg_hit_set_history_t>&, Context*, unsigned long,
osd_reqid_t, boost::intrusive_ptr<OpRequest>)+0x5e7) [0x55acc0f6c907]
 13: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*,
PrimaryLogPG::OpContext*)+0x50d) [0x55acc0c92ebd]
 14: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xd25)
[0x55acc0cf0295]
 15: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x288d)
[0x55acc0cf78fd]
 16: (OSD::dequeue_op(boost::intrusive_ptr<PG>,
boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1c0)
[0x55acc0b56900]
 17: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*,
boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x6d)
[0x55acc0e552ad]
 18: (OSD::ShardedOpWQ::_process(unsigned int,
ceph::heartbeat_handle_d*)+0x115f) [0x55acc0b69dbf]
 19: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435)
[0x55acc12c78c5]
 20: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55acc12c9fe4]
 21: /lib64/libpthread.so.0(+0x81ca) [0x7f53d93d31ca]
 22: clone()

It has also has below errors before the crash:
scrub-queue::*remove_from_osd_queue* removing pg[2.4f0] failed. State was:
unregistering

Please help to troubleshoot the issue and fix it

Already posted on ceph tracker but no reply there since 3-4 days