[ceph-users] Nautilus cluster damaged + crashing OSDs

20 Apr 2020

      Hi,

one of our customers had his Ceph cluster crashed due to a power or network outage (they still try to figure out what happened).

The cluster is very unhealthy but recovering:

# ceph -s
  cluster:
    id:     1c95ca5d-948b-4113-9246-14761cb9a82a
    health: HEALTH_ERR
            1 filesystem is degraded
            1 mds daemon damaged
            1 osds down
            1 pools have many more objects per pg than average
            1/115117480 objects unfound (0.000%)
            Reduced data availability: 71 pgs inactive, 53 pgs down, 18 pgs peering, 27 pgs stale
            Possible data damage: 1 pg recovery_unfound
            Degraded data redundancy: 7303464/230234960 objects degraded (3.172%), 693 pgs degraded, 945 pgs undersized
            14 daemons have recently crashed

  services:
    mon: 3 daemons, quorum maslxlabstore01,maslxlabstore02,maslxlabstore04 (age 64m)
    mgr: maslxlabstore01(active, since 69m), standbys: maslxlabstore03, maslxlabstore02, maslxlabstore04
    mds: cephfs:2/3 {0=maslxlabstore03=up:resolve,1=maslxlabstore01=up:resolve} 2 up:standby, 1 damaged
    osd: 140 osds: 130 up (since 4m), 131 in (since 4m); 847 remapped pgs
    rgw: 4 daemons active (maslxlabstore01.rgw0, maslxlabstore02.rgw0, maslxlabstore03.rgw0, maslxlabstore04.rgw0)

  data:
    pools:   6 pools, 8328 pgs
    objects: 115.12M objects, 218 TiB
    usage:   425 TiB used, 290 TiB / 715 TiB avail
    pgs:     0.853% pgs not active
             7303464/230234960 objects degraded (3.172%)
             13486/230234960 objects misplaced (0.006%)
             1/115117480 objects unfound (0.000%)
             7311 active+clean
             338  active+undersized+degraded+remapped+backfill_wait
             255  active+undersized+degraded+remapped+backfilling
             215  active+undersized+remapped+backfilling
             99   active+undersized+degraded
             44   down
             37   active+undersized+remapped+backfill_wait
             13   stale+peering
             9    stale+down
             5    stale+remapped+peering
             1    active+recovery_unfound+undersized+degraded+remapped
             1    active+clean+remapped

  io:
    client:   168 B/s rd, 0 B/s wr, 0 op/s rd, 0 op/s wr
    recovery: 1.9 GiB/s, 15 keys/s, 948 objects/s

The MDS cluster is unable to start because one of them is damaged.

10 of the OSDs do not start. They crash very early in the boot process:

2020-04-20 16:26:14.935 7f818ec8cc00  0 set uid:gid to 64045:64045 (ceph:ceph)
2020-04-20 16:26:14.935 7f818ec8cc00  0 ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable), process ceph-osd, pid 69463
2020-04-20 16:26:14.935 7f818ec8cc00  0 pidfile_write: ignore empty --pid-file
2020-04-20 16:26:15.503 7f818ec8cc00  0 starting osd.42 osd_data /var/lib/ceph/osd/ceph-42 /var/lib/ceph/osd/ceph-42/journal
2020-04-20 16:26:15.523 7f818ec8cc00  0 load: jerasure load: lrc load: isa 
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option compaction_readahead_size = 2MB
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option compaction_style = kCompactionStyleLevel
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option compaction_threads = 32
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option compression = kNoCompression
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option flusher_threads = 8
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option level0_file_num_compaction_trigger = 8
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option level0_slowdown_writes_trigger = 32
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option level0_stop_writes_trigger = 64
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option max_background_compactions = 31
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option max_bytes_for_level_base = 536870912
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option max_bytes_for_level_multiplier = 8
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option max_write_buffer_number = 32
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option min_write_buffer_number_to_merge = 2
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option recycle_log_file_num = 32
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option target_file_size_base = 67108864
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option write_buffer_size = 67108864
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option compaction_readahead_size = 2MB
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option compaction_style = kCompactionStyleLevel
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option compaction_threads = 32
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option compression = kNoCompression
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option flusher_threads = 8
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option level0_file_num_compaction_trigger = 8
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option level0_slowdown_writes_trigger = 32
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option level0_stop_writes_trigger = 64
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option max_background_compactions = 31
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option max_bytes_for_level_base = 536870912
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option max_bytes_for_level_multiplier = 8
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option max_write_buffer_number = 32
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option min_write_buffer_number_to_merge = 2
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option recycle_log_file_num = 32
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option target_file_size_base = 67108864
2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option write_buffer_size = 67108864
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option compaction_readahead_size = 2MB
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option compaction_style = kCompactionStyleLevel
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option compaction_threads = 32
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option compression = kNoCompression
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option flusher_threads = 8
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option level0_file_num_compaction_trigger = 8
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option level0_slowdown_writes_trigger = 32
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option level0_stop_writes_trigger = 64
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option max_background_compactions = 31
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option max_bytes_for_level_base = 536870912
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option max_bytes_for_level_multiplier = 8
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option max_write_buffer_number = 32
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option min_write_buffer_number_to_merge = 2
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option recycle_log_file_num = 32
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option target_file_size_base = 67108864
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option write_buffer_size = 67108864
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option compaction_readahead_size = 2MB
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option compaction_style = kCompactionStyleLevel
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option compaction_threads = 32
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option compression = kNoCompression
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option flusher_threads = 8
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option level0_file_num_compaction_trigger = 8
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option level0_slowdown_writes_trigger = 32
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option level0_stop_writes_trigger = 64
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option max_background_compactions = 31
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option max_bytes_for_level_base = 536870912
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option max_bytes_for_level_multiplier = 8
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option max_write_buffer_number = 32
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option min_write_buffer_number_to_merge = 2
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option recycle_log_file_num = 32
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option target_file_size_base = 67108864
2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option write_buffer_size = 67108864
2020-04-20 16:26:18.131 7f818ec8cc00  0 _get_class not permitted to load lua
2020-04-20 16:26:18.131 7f818ec8cc00  0 _get_class not permitted to load kvs
2020-04-20 16:26:18.131 7f818ec8cc00  0 _get_class not permitted to load sdk
2020-04-20 16:26:18.131 7f818ec8cc00  0 <cls> /build/ceph-14.2.9/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs
2020-04-20 16:26:18.131 7f818ec8cc00  0 <cls> /build/ceph-14.2.9/src/cls/hello/cls_hello.cc:296: loading cls_hello
2020-04-20 16:26:18.131 7f818ec8cc00  0 osd.42 6008 crush map has features 288514051259236352, adjusting msgr requires for clients
2020-04-20 16:26:18.131 7f818ec8cc00  0 osd.42 6008 crush map has features 288514051259236352 was 8705, adjusting msgr requires for mons
2020-04-20 16:26:18.131 7f818ec8cc00  0 osd.42 6008 crush map has features 3314933000852226048, adjusting msgr requires for osds
2020-04-20 16:26:22.023 7f818ec8cc00  0 osd.42 6008 load_pgs
2020-04-20 16:26:22.499 7f818ec8cc00  0 osd.42 6008 load_pgs opened 109 pgs
2020-04-20 16:26:22.499 7f818ec8cc00  0 osd.42 6008 using weightedpriority op queue with priority op cut off at 64.
2020-04-20 16:26:22.499 7f818ec8cc00 -1 osd.42 6008 log_to_monitors {default=true}
2020-04-20 16:26:22.511 7f818ec8cc00  0 osd.42 6008 done with init, starting boot process
2020-04-20 16:26:23.883 7f815331c700 -1 /build/ceph-14.2.9/src/osd/PGLog.cc: In function 'void PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)' thread 7f815331c700 time 2020-04-20 16:26:23.884183
/build/ceph-14.2.9/src/osd/PGLog.cc: 368: FAILED ceph_assert(log.head >= olog.tail && olog.head >= log.tail)

 ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x564fbd9349d2]
 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x564fbd934bad]
 3: (PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0x1cc0) [0x564fbdaff930]
 4: (PG::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_shard_t)+0x64) [0x564fbda4eca4]
 5: (PG::proc_master_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_missing_set<false>&, pg_shard_t)+0x97) [0x564fbda7fe47]
 6: (PG::RecoveryState::GetLog::react(PG::RecoveryState::GotLog const&)+0xa6) [0x564fbda9d4f6]
 7: (boost::statechart::simple_state<PG::RecoveryState::GetLog, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x191) [0x564fbdaf0e21]
 8: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>::process_queued_events()+0xb3) [0x564fbdabdfc3]
 9: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x87) [0x564fbdabe227]
 10: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x122) [0x564fbdaada12]
 11: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b4) [0x564fbd9e1f54]
 12: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x50) [0x564fbdc710c0]
 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbf5) [0x564fbd9d5995]
 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x564fbdfdb8cc]
 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x564fbdfdea90]
 16: (()+0x76db) [0x7f818c8666db]
 17: (clone()+0x3f) [0x7f818b60688f]

2020-04-20 16:26:23.887 7f815331c700 -1 *** Caught signal (Aborted) **
 in thread 7f815331c700 thread_name:tp_osd_tp

 ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)
 1: (()+0x12890) [0x7f818c871890]
 2: (gsignal()+0xc7) [0x7f818b523e97]
 3: (abort()+0x141) [0x7f818b525801]
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x564fbd934a23]
 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x564fbd934bad]
 6: (PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0x1cc0) [0x564fbdaff930]
 7: (PG::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_shard_t)+0x64) [0x564fbda4eca4]
 8: (PG::proc_master_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_missing_set<false>&, pg_shard_t)+0x97) [0x564fbda7fe47]
 9: (PG::RecoveryState::GetLog::react(PG::RecoveryState::GotLog const&)+0xa6) [0x564fbda9d4f6]
 10: (boost::statechart::simple_state<PG::RecoveryState::GetLog, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x191) [0x564fbdaf0e21]
 11: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>::process_queued_events()+0xb3) [0x564fbdabdfc3]
 12: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x87) [0x564fbdabe227]
 13: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x122) [0x564fbdaada12]
 14: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b4) [0x564fbd9e1f54]
 15: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x50) [0x564fbdc710c0]
 16: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbf5) [0x564fbd9d5995]
 17: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x564fbdfdb8cc]
 18: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x564fbdfdea90]
 19: (()+0x76db) [0x7f818c8666db]
 20: (clone()+0x3f) [0x7f818b60688f]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
  -105> 2020-04-20 16:26:14.923 7f818ec8cc00  5 asok(0x564fc8472000) register_command assert hook 0x564fc83ae510
  -104> 2020-04-20 16:26:14.923 7f818ec8cc00  5 asok(0x564fc8472000) register_command abort hook 0x564fc83ae510
  -103> 2020-04-20 16:26:14.923 7f818ec8cc00  5 asok(0x564fc8472000) register_command perfcounters_dump hook 0x564fc83ae510
  -102> 2020-04-20 16:26:14.923 7f818ec8cc00  5 asok(0x564fc8472000) register_command 1 hook 0x564fc83ae510
  -101> 2020-04-20 16:26:14.923 7f818ec8cc00  5 asok(0x564fc8472000) register_command perf dump hook 0x564fc83ae510
  -100> 2020-04-20 16:26:14.923 7f818ec8cc00  5 asok(0x564fc8472000) register_command perfcounters_schema hook 0x564fc83ae510
   -99> 2020-04-20 16:26:14.923 7f818ec8cc00  5 asok(0x564fc8472000) register_command perf histogram dump hook 0x564fc83ae510
   -98> 2020-04-20 16:26:14.923 7f818ec8cc00  5 asok(0x564fc8472000) register_command 2 hook 0x564fc83ae510
   -97> 2020-04-20 16:26:14.923 7f818ec8cc00  5 asok(0x564fc8472000) register_command perf schema hook 0x564fc83ae510
   -96> 2020-04-20 16:26:14.923 7f818ec8cc00  5 asok(0x564fc8472000) register_command perf histogram schema hook 0x564fc83ae510
   -95> 2020-04-20 16:26:14.923 7f818ec8cc00  5 asok(0x564fc8472000) register_command perf reset hook 0x564fc83ae510
   -94> 2020-04-20 16:26:14.923 7f818ec8cc00  5 asok(0x564fc8472000) register_command config show hook 0x564fc83ae510
   -93> 2020-04-20 16:26:14.923 7f818ec8cc00  5 asok(0x564fc8472000) register_command config help hook 0x564fc83ae510
   -92> 2020-04-20 16:26:14.923 7f818ec8cc00  5 asok(0x564fc8472000) register_command config set hook 0x564fc83ae510
   -91> 2020-04-20 16:26:14.923 7f818ec8cc00  5 asok(0x564fc8472000) register_command config unset hook 0x564fc83ae510
   -90> 2020-04-20 16:26:14.923 7f818ec8cc00  5 asok(0x564fc8472000) register_command config get hook 0x564fc83ae510
   -89> 2020-04-20 16:26:14.923 7f818ec8cc00  5 asok(0x564fc8472000) register_command config diff hook 0x564fc83ae510
   -88> 2020-04-20 16:26:14.923 7f818ec8cc00  5 asok(0x564fc8472000) register_command config diff get hook 0x564fc83ae510
   -87> 2020-04-20 16:26:14.923 7f818ec8cc00  5 asok(0x564fc8472000) register_command log flush hook 0x564fc83ae510
   -86> 2020-04-20 16:26:14.923 7f818ec8cc00  5 asok(0x564fc8472000) register_command log dump hook 0x564fc83ae510
   -85> 2020-04-20 16:26:14.923 7f818ec8cc00  5 asok(0x564fc8472000) register_command log reopen hook 0x564fc83ae510
   -84> 2020-04-20 16:26:14.923 7f818ec8cc00  5 asok(0x564fc8472000) register_command dump_mempools hook 0x564fc9076068
   -83> 2020-04-20 16:26:14.935 7f818ec8cc00  0 set uid:gid to 64045:64045 (ceph:ceph)
   -82> 2020-04-20 16:26:14.935 7f818ec8cc00  0 ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable), process ceph-osd, pid 69463
   -81> 2020-04-20 16:26:14.935 7f818ec8cc00  0 pidfile_write: ignore empty --pid-file
   -80> 2020-04-20 16:26:15.503 7f818ec8cc00  0 starting osd.42 osd_data /var/lib/ceph/osd/ceph-42 /var/lib/ceph/osd/ceph-42/journal
   -79> 2020-04-20 16:26:15.523 7f818ec8cc00  0 load: jerasure load: lrc load: isa 
   -78> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option compaction_readahead_size = 2MB
   -77> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option compaction_style = kCompactionStyleLevel
   -76> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option compaction_threads = 32
   -75> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option compression = kNoCompression
   -74> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option flusher_threads = 8
   -73> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option level0_file_num_compaction_trigger = 8
   -72> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option level0_slowdown_writes_trigger = 32
   -71> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option level0_stop_writes_trigger = 64
   -70> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option max_background_compactions = 31
   -69> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option max_bytes_for_level_base = 536870912
   -68> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option max_bytes_for_level_multiplier = 8
   -67> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option max_write_buffer_number = 32
   -66> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option min_write_buffer_number_to_merge = 2
   -65> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option recycle_log_file_num = 32
   -64> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option target_file_size_base = 67108864
   -63> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option write_buffer_size = 67108864
   -62> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option compaction_readahead_size = 2MB
   -61> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option compaction_style = kCompactionStyleLevel
   -60> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option compaction_threads = 32
   -59> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option compression = kNoCompression
   -58> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option flusher_threads = 8
   -57> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option level0_file_num_compaction_trigger = 8
   -56> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option level0_slowdown_writes_trigger = 32
   -55> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option level0_stop_writes_trigger = 64
   -54> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option max_background_compactions = 31
   -53> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option max_bytes_for_level_base = 536870912
   -52> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option max_bytes_for_level_multiplier = 8
   -51> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option max_write_buffer_number = 32
   -50> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option min_write_buffer_number_to_merge = 2
   -49> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option recycle_log_file_num = 32
   -48> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option target_file_size_base = 67108864
   -47> 2020-04-20 16:26:16.339 7f818ec8cc00  0  set rocksdb option write_buffer_size = 67108864
   -46> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option compaction_readahead_size = 2MB
   -45> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option compaction_style = kCompactionStyleLevel
   -44> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option compaction_threads = 32
   -43> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option compression = kNoCompression
   -42> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option flusher_threads = 8
   -41> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option level0_file_num_compaction_trigger = 8
   -40> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option level0_slowdown_writes_trigger = 32
   -39> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option level0_stop_writes_trigger = 64
   -38> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option max_background_compactions = 31
   -37> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option max_bytes_for_level_base = 536870912
   -36> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option max_bytes_for_level_multiplier = 8
   -35> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option max_write_buffer_number = 32
   -34> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option min_write_buffer_number_to_merge = 2
   -33> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option recycle_log_file_num = 32
   -32> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option target_file_size_base = 67108864
   -31> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option write_buffer_size = 67108864
   -30> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option compaction_readahead_size = 2MB
   -29> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option compaction_style = kCompactionStyleLevel
   -28> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option compaction_threads = 32
   -27> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option compression = kNoCompression
   -26> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option flusher_threads = 8
   -25> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option level0_file_num_compaction_trigger = 8
   -24> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option level0_slowdown_writes_trigger = 32
   -23> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option level0_stop_writes_trigger = 64
   -22> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option max_background_compactions = 31
   -21> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option max_bytes_for_level_base = 536870912
   -20> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option max_bytes_for_level_multiplier = 8
   -19> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option max_write_buffer_number = 32
   -18> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option min_write_buffer_number_to_merge = 2
   -17> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option recycle_log_file_num = 32
   -16> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option target_file_size_base = 67108864
   -15> 2020-04-20 16:26:17.731 7f818ec8cc00  0  set rocksdb option write_buffer_size = 67108864
   -14> 2020-04-20 16:26:18.131 7f818ec8cc00  0 _get_class not permitted to load lua
   -13> 2020-04-20 16:26:18.131 7f818ec8cc00  0 _get_class not permitted to load kvs
   -12> 2020-04-20 16:26:18.131 7f818ec8cc00  0 _get_class not permitted to load sdk
   -11> 2020-04-20 16:26:18.131 7f818ec8cc00  0 <cls> /build/ceph-14.2.9/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs
   -10> 2020-04-20 16:26:18.131 7f818ec8cc00  0 <cls> /build/ceph-14.2.9/src/cls/hello/cls_hello.cc:296: loading cls_hello
    -9> 2020-04-20 16:26:18.131 7f818ec8cc00  0 osd.42 6008 crush map has features 288514051259236352, adjusting msgr requires for clients
    -8> 2020-04-20 16:26:18.131 7f818ec8cc00  0 osd.42 6008 crush map has features 288514051259236352 was 8705, adjusting msgr requires for mons
    -7> 2020-04-20 16:26:18.131 7f818ec8cc00  0 osd.42 6008 crush map has features 3314933000852226048, adjusting msgr requires for osds
    -6> 2020-04-20 16:26:22.023 7f818ec8cc00  0 osd.42 6008 load_pgs
    -5> 2020-04-20 16:26:22.499 7f818ec8cc00  0 osd.42 6008 load_pgs opened 109 pgs
    -4> 2020-04-20 16:26:22.499 7f818ec8cc00  0 osd.42 6008 using weightedpriority op queue with priority op cut off at 64.
    -3> 2020-04-20 16:26:22.499 7f818ec8cc00 -1 osd.42 6008 log_to_monitors {default=true}
    -2> 2020-04-20 16:26:22.511 7f818ec8cc00  0 osd.42 6008 done with init, starting boot process
    -1> 2020-04-20 16:26:23.883 7f815331c700 -1 /build/ceph-14.2.9/src/osd/PGLog.cc: In function 'void PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)' thread 7f815331c700 time 2020-04-20 16:26:23.884183
/build/ceph-14.2.9/src/osd/PGLog.cc: 368: FAILED ceph_assert(log.head >= olog.tail && olog.head >= log.tail)

 ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x564fbd9349d2]
 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x564fbd934bad]
 3: (PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0x1cc0) [0x564fbdaff930]
 4: (PG::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_shard_t)+0x64) [0x564fbda4eca4]
 5: (PG::proc_master_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_missing_set<false>&, pg_shard_t)+0x97) [0x564fbda7fe47]
 6: (PG::RecoveryState::GetLog::react(PG::RecoveryState::GotLog const&)+0xa6) [0x564fbda9d4f6]
 7: (boost::statechart::simple_state<PG::RecoveryState::GetLog, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x191) [0x564fbdaf0e21]
 8: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>::process_queued_events()+0xb3) [0x564fbdabdfc3]
 9: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x87) [0x564fbdabe227]
 10: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x122) [0x564fbdaada12]
 11: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b4) [0x564fbd9e1f54]
 12: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x50) [0x564fbdc710c0]
 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbf5) [0x564fbd9d5995]
 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x564fbdfdb8cc]
 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x564fbdfdea90]
 16: (()+0x76db) [0x7f818c8666db]
 17: (clone()+0x3f) [0x7f818b60688f]

     0> 2020-04-20 16:26:23.887 7f815331c700 -1 *** Caught signal (Aborted) **
 in thread 7f815331c700 thread_name:tp_osd_tp

 ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)
 1: (()+0x12890) [0x7f818c871890]
 2: (gsignal()+0xc7) [0x7f818b523e97]
 3: (abort()+0x141) [0x7f818b525801]
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x564fbd934a23]
 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x564fbd934bad]
 6: (PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0x1cc0) [0x564fbdaff930]
 7: (PG::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_shard_t)+0x64) [0x564fbda4eca4]
 8: (PG::proc_master_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_missing_set<false>&, pg_shard_t)+0x97) [0x564fbda7fe47]
 9: (PG::RecoveryState::GetLog::react(PG::RecoveryState::GotLog const&)+0xa6) [0x564fbda9d4f6]
 10: (boost::statechart::simple_state<PG::RecoveryState::GetLog, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x191) [0x564fbdaf0e21]
 11: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>::process_queued_events()+0xb3) [0x564fbdabdfc3]
 12: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x87) [0x564fbdabe227]
 13: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x122) [0x564fbdaada12]
 14: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b4) [0x564fbd9e1f54]
 15: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x50) [0x564fbdc710c0]
 16: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbf5) [0x564fbd9d5995]
 17: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x564fbdfdb8cc]
 18: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x564fbdfdea90]
 19: (()+0x76db) [0x7f818c8666db]
 20: (clone()+0x3f) [0x7f818b60688f]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 0 none
   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   0/ 0 mds
   0/ 0 mds_balancer
   0/ 0 mds_locker
   0/ 0 mds_log
   0/ 0 mds_log_expire
   0/ 0 mds_migrator
   0/ 0 buffer
   0/ 0 timer
   0/ 0 filer
   0/ 0 striper
   0/ 0 objecter
   0/ 0 rados
   0/ 0 rbd
   0/ 0 rbd_mirror
   0/ 0 rbd_replay
   0/ 0 journaler
   0/ 0 objectcacher
   0/ 0 client
   0/ 0 osd
   0/ 0 optracker
   0/ 0 objclass
   0/ 0 filestore
   0/ 0 journal
   0/ 0 ms
   0/ 0 mon
   0/ 0 monc
   0/ 0 paxos
   0/ 0 tp
   0/ 0 auth
   0/ 0 crypto
   0/ 0 finisher
   0/ 0 reserver
   0/ 0 heartbeatmap
   0/ 0 perfcounter
   0/ 0 rgw
   1/ 5 rgw_sync
   0/ 0 civetweb
   0/ 0 javaclient
   0/ 0 asok
   0/ 0 throttle
   0/ 0 refs
   0/ 0 xio
   0/ 0 compressor
   0/ 0 bluestore
   0/ 0 bluefs
   0/ 0 bdev
   0/ 0 kstore
   0/ 0 rocksdb
   0/ 0 leveldb
   0/ 0 memdb
   0/ 0 kinetic
   0/ 0 fuse
   0/ 0 mgr
   0/ 0 mgrc
   0/ 0 dpdk
   0/ 0 eventtrace
   1/ 5 prioritycache
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.42.log
--- end dump of recent events ---

It would be nice if anybody could give me a hint on where to look further.

Regards
-- 
Robert Sander
Heinlein Support GmbH
Schwedter Str. 8/9b, 10119 Berlin

http://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Zwangsangaben lt. §35a GmbHG: 
HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin