Hello all! I'm got repeatedlyOSD crashes for 3 of OSD which stracktrace: ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable) 1: (()+0x911e70) [0x564d0067fe70] 2: (()+0xf5d0) [0x7f1272dad5d0] 3: (gsignal()+0x37) [0x7f1271dce2c7] 4: (abort()+0x148) [0x7f1271dcf9b8] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x242) [0x7f12762252b2] 6: (()+0x25a337) [0x7f1276225337] 7: (PrimaryLogPG::hit_set_trim(std::unique_ptr<PrimaryLogPG::OpContext, std::default_delete<PrimaryLogPG::OpContext> >&, unsigned int)+0x930) [0x564d002ab480] 8: (PrimaryLogPG::hit_set_persist()+0xa0c) [0x564d002afafc] 9: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x2989) [0x564d002c5f09] 10: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xc99) [0x564d002cac09] 11: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1b7) [0x564d00124c87] 12: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x564d0039d8c2] 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x592) [0x564d00144ae2] 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x3d3) [0x7f127622aec3] 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x7f127622bab0] 16: (()+0x7dd5) [0x7f1272da5dd5] 17: (clone()+0x6d) [0x7f1271e95f6d] I think this happens after some network troubles and one PG was marked recovery_unfound, to solve this problem this command was executed: ceph pg 2.f8 mark_unfound_lost revert Also last lines in debug output before crash are related to this PG: -10001> 2020-04-23 14:16:18.790 7f534915a700 10 osd.12 pg_epoch: 10476 pg[2.f8( v 8673'30651498 (6953'30648450,8673'30651498] local-lis/les=10475/10476 n=14 ec=66/66 lis/c 10475/10427 les/c/f 10476/10428/201 10475/10475/10473) [12,13,17] r=0 lpr=10475 pi=[10427,10475)/2 crt=8673'30651498 lcod 0'0 mlcod 0'0 active mbc={}] get_object_context: obc NOT found in cache: 2:1f000000:.ceph-internal::hit_set_2.f8_archive_2020-04-22 02%3a57%3a10.496532Z_2020-04-22 03%3a57%3a11.211949Z:head -10001> 2020-04-23 14:16:18.790 7f534915a700 10 osd.12 pg_epoch: 10476 pg[2.f8( v 8673'30651498 (6953'30648450,8673'30651498] local-lis/les=10475/10476 n=14 ec=66/66 lis/c 10475/10427 les/c/f 10476/10428/201 10475/10475/10473) [12,13,17] r=0 lpr=10475 pi=[10427,10475)/2 crt=8673'30651498 lcod 0'0 mlcod 0'0 active mbc={}] get_object_context: no obc for soid 2:1f000000:.ceph-internal::hit_set_2.f8_archive_2020-04-22 02%3a57%3a10.496532Z_2020-04-22 03%3a57%3a11.211949Z:head and !can_create I'm evicted must of PG's from cache pool, and currently we have only two rados objects in this PG: # rados --pgid 2.f8 ls rbd_data.10d4416b8b4567.0000000000001dfb rbd_header.1db946b8b4567 I tried to remove one of them: rados -p vms-cache rm rbd_header.1db946b8b4567 But command not completed for last ~8 hours, because repeatedly OSD crashes. Does we have any ways to solve this problem without data loss? -- MATPOCKuH