[ceph-users] Issues upgrading to 16.2.1

Hello Tried cephadm upgrade form 16.2.0 to 16.2.1 Managers were updated first then process halted on first monitor being upgraded. The monitor fails to start: root@host3:/var/lib/ceph/c8ee2878-9d54-11eb-bbca-1c34da4b9fb6/mon.host3# /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-mon --privileged --group-add=disk --init --name ceph-c8ee2878-9d54-11eb-bbca-1c34da4b9fb6-mon.host3 -e CONTAINER_IMAGE=ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a -e NODE_NAME=host3 -e CEPH_USE_RANDOM_NONCE=1 -v /var/run/ceph/c8ee2878-9d54-11eb-bbca-1c34da4b9fb6:/var/run/ceph:z -v /var/log/ceph/c8ee2878-9d54-11eb-bbca-1c34da4b9fb6:/var/log/ceph:z -v /var/lib/ceph/c8ee2878-9d54-11eb-bbca-1c34da4b9fb6/crash:/var/lib/ceph/crash:z -v /var/lib/ceph/c8ee2878-9d54-11eb-bbca-1c34da4b9fb6/mon.host3:/var/lib/ceph/mon/ceph-host3:z -v /var/lib/ceph/c8ee2878-9d54-11eb-bbca-1c34da4b9fb6/mon.host3/config:/etc/ceph/ceph.conf:z -v /dev:/dev -v /run/udev:/run/udev ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a -n mon.host3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true '--default-log-stderr-prefix=debug ' --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true debug 2021-04-20T14:06:19.437+0000 7f11e080a700  0 set uid:gid to 167:167 (ceph:ceph) debug 2021-04-20T14:06:19.437+0000 7f11e080a700  0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-mon, pid 7 debug 2021-04-20T14:06:19.437+0000 7f11e080a700  0 pidfile_write: ignore empty --pid-file debug 2021-04-20T14:06:19.441+0000 7f11e080a700  0 load: jerasure load: lrc load: isa debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: RocksDB version: 6.8.1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Git sha rocksdb_build_git_sha:@0@ debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Compile date Mar 30 2021 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: DB SUMMARY debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: CURRENT file:  CURRENT debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: IDENTITY file:  IDENTITY debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: MANIFEST file:  MANIFEST-000152 size: 221 Bytes debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: SST files in /var/lib/ceph/mon/ceph-host3/store.db dir, Total Num: 2, files: 000137.sst 000139.sst debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-host3/store.db: 000153.log size: 0 ; debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                         Options.error_if_exists: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                       Options.create_if_missing: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                         Options.paranoid_checks: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                                     Options.env: 0x5641244cf1c0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                                      Options.fs: Posix File System debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                                Options.info_log: 0x564126753220 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                Options.max_file_opening_threads: 16 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                              Options.statistics: (nil) debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                               Options.use_fsync: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                       Options.max_log_file_size: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                  Options.max_manifest_file_size: 1073741824 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                   Options.log_file_time_to_roll: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                       Options.keep_log_file_num: 1000 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                    Options.recycle_log_file_num: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                         Options.allow_fallocate: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                        Options.allow_mmap_reads: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                       Options.allow_mmap_writes: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                        Options.use_direct_reads: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:          Options.create_missing_column_families: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                              Options.db_log_dir: debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                                 Options.wal_dir: /var/lib/ceph/mon/ceph-host3/store.db debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                Options.table_cache_numshardbits: 6 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                      Options.max_subcompactions: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                  Options.max_background_flushes: -1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                         Options.WAL_ttl_seconds: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                       Options.WAL_size_limit_MB: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:             Options.manifest_preallocation_size: 4194304 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                     Options.is_fd_close_on_exec: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                   Options.advise_random_on_open: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                    Options.db_write_buffer_size: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                    Options.write_buffer_manager: 0x5641267e9e90 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:         Options.access_hint_on_compaction_start: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:           Options.random_access_max_buffer_size: 1048576 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                      Options.use_adaptive_mutex: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                            Options.rate_limiter: (nil) debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                       Options.wal_recovery_mode: 2 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                  Options.enable_thread_tracking: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                  Options.enable_pipelined_write: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                  Options.unordered_write: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:         Options.allow_concurrent_memtable_write: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:             Options.write_thread_max_yield_usec: 100 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:            Options.write_thread_slow_yield_usec: 3 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                               Options.row_cache: None debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                              Options.wal_filter: None debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:             Options.avoid_flush_during_recovery: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:             Options.allow_ingest_behind: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:             Options.preserve_deletes: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:             Options.two_write_queues: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:             Options.manual_wal_flush: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:             Options.atomic_flush: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:             Options.avoid_unnecessary_blocking_io: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                 Options.persist_stats_to_disk: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                 Options.write_dbid_to_manifest: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                 Options.log_readahead_size: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                 Options.sst_file_checksum_func: Unknown debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:             Options.max_background_jobs: 2 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:             Options.max_background_compactions: -1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:             Options.avoid_flush_during_shutdown: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:           Options.writable_file_max_buffer_size: 1048576 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:             Options.delayed_write_rate : 16777216 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:             Options.max_total_wal_size: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:             Options.delete_obsolete_files_period_micros: 21600000000 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                   Options.stats_dump_period_sec: 600 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                 Options.stats_persist_period_sec: 600 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                 Options.stats_history_buffer_size: 1048576 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                          Options.max_open_files: -1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                          Options.bytes_per_sync: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                      Options.wal_bytes_per_sync: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                   Options.strict_bytes_per_sync: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.compaction_readahead_size: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Compression algorithms supported: debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: kZSTDNotFinalCompression supported: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: kZSTD supported: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: kXpressCompression supported: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: kLZ4HCCompression supported: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: kLZ4Compression supported: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: kBZip2Compression supported: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: kZlibCompression supported: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: kSnappyCompression supported: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Fast CRC32 supported: Supported on x86 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-host3/store.db/MANIFEST-000152 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:               Options.comparator: leveldb.BytewiseComparator debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:           Options.merge_operator: debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.compaction_filter: None debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.compaction_filter_factory: None debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:         Options.memtable_factory: SkipListFactory debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:            Options.table_factory: BlockBasedTable debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:            table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5641266f60d8)   cache_index_and_filter_blocks: 1   cache_index_and_filter_blocks_with_high_priority: 0   pin_l0_filter_and_index_blocks_in_cache: 0   pin_top_level_index_and_filter: 1   index_type: 0   data_block_index_type: 0   index_shortening: 1   data_block_hash_table_util_ratio: 0.750000   hash_index_allow_collision: 1   checksum: 1   no_block_cache: 0   block_cache: 0x56412672ef10   block_cache_name: BinnedLRUCache   block_cache_options:     capacity : 536870912     num_shard_bits : 4     strict_capacity_limit : 0     high_pri_pool_ratio: 0.000   block_cache_compressed: (nil)   persistent_cache: (nil)   block_size: 4096   block_size_deviation: 10   block_restart_interval: 16   index_block_restart_interval: 1   metadata_block_size: 4096   partition_filters: 0   use_delta_encoding: 1   filter_policy: rocksdb.BuiltinBloomFilter   whole_key_filtering: 1   verify_compression: 0   read_amp_bytes_per_bit: 0   format_version: 2   enable_index_compression: 1   block_align: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.write_buffer_size: 33554432 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.max_write_buffer_number: 2 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:          Options.compression: NoCompression debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                  Options.bottommost_compression: Disabled debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.prefix_extractor: nullptr debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:             Options.num_levels: 7 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.min_write_buffer_number_to_merge: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.bottommost_compression_opts.level: 32767 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.bottommost_compression_opts.strategy: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.bottommost_compression_opts.enabled: false debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:            Options.compression_opts.window_bits: -14 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                  Options.compression_opts.level: 32767 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:               Options.compression_opts.strategy: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:         Options.compression_opts.max_dict_bytes: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:         Options.compression_opts.zstd_max_train_bytes: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                  Options.compression_opts.enabled: false debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.level0_file_num_compaction_trigger: 4 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:              Options.level0_stop_writes_trigger: 36 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                   Options.target_file_size_base: 67108864 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:             Options.target_file_size_multiplier: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                Options.max_bytes_for_level_base: 268435456 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:          Options.max_bytes_for_level_multiplier: 10.000000 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.max_sequential_skip_in_iterations: 8 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                    Options.max_compaction_bytes: 1677721600 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                        Options.arena_block_size: 4194304 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                Options.disable_auto_compactions: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                        Options.compaction_style: kCompactionStyleLevel debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                          Options.compaction_pri: kMinOverlappingRatio debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.compaction_options_universal.size_ratio: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                   Options.table_properties_collectors: debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                   Options.inplace_update_support: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                 Options.inplace_update_num_locks: 10000 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:               Options.memtable_prefix_bloom_size_ratio: 0.000000 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:               Options.memtable_whole_key_filtering: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb: Options.memtable_huge_page_size: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                           Options.bloom_locality: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                    Options.max_successive_merges: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                Options.optimize_filters_for_hits: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                Options.paranoid_file_checks: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                Options.force_consistency_checks: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                Options.report_bg_io_stats: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:                               Options.ttl: 2592000 debug 2021-04-20T14:06:19.441+0000 7f11e080a700  4 rocksdb:          Options.periodic_compaction_seconds: 0 debug 2021-04-20T14:06:19.445+0000 7f11e080a700  4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-host3/store.db/MANIFEST-000152 succeeded,manifest_file_number is 152, next_file_number is 154, last_sequence is 73904, log_number is 151,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 debug 2021-04-20T14:06:19.445+0000 7f11e080a700  4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 151 debug 2021-04-20T14:06:19.445+0000 7f11e080a700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618927579447606, "job": 1, "event": "recovery_started", "log_files": [153]} debug 2021-04-20T14:06:19.445+0000 7f11e080a700  4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #153 mode 2 debug 2021-04-20T14:06:19.445+0000 7f11e080a700  4 rocksdb: [version_set.cc:3826] Creating manifest 155 debug 2021-04-20T14:06:19.445+0000 7f11e080a700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618927579449426, "job": 1, "event": "recovery_finished"} debug 2021-04-20T14:06:19.445+0000 7f11e080a700  4 rocksdb: DB pointer 0x5641267f3800 debug 2021-04-20T14:06:19.449+0000 7f11c7624700  4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- debug 2021-04-20T14:06:19.449+0000 7f11c7624700  4 rocksdb: [db_impl/db_impl.cc:851] ** DB Stats ** Uptime(secs): 0.0 total, 0.0 interval Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------   L0      1/0    3.10 MB   0.2      0.0     0.0      0.0 0.0      0.0       0.0   0.0      0.0      0.0 0.00              0.00         0    0.000       0      0   L6      1/0   38.24 MB   0.0      0.0     0.0      0.0 0.0      0.0       0.0   0.0      0.0      0.0 0.00              0.00         0    0.000       0      0  Sum      2/0   41.34 MB   0.0      0.0     0.0      0.0 0.0      0.0       0.0   0.0      0.0      0.0 0.00              0.00         0    0.000       0      0  Int      0/0    0.00 KB   0.0      0.0     0.0      0.0 0.0      0.0       0.0   0.0      0.0      0.0 0.00              0.00         0    0.000       0      0 ** Compaction Stats [default] ** Priority    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** ** Compaction Stats [default] ** Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------   L0      1/0    3.10 MB   0.2      0.0     0.0      0.0 0.0      0.0       0.0   0.0      0.0      0.0 0.00              0.00         0    0.000       0      0   L6      1/0   38.24 MB   0.0      0.0     0.0      0.0 0.0      0.0       0.0   0.0      0.0      0.0 0.00              0.00         0    0.000       0      0  Sum      2/0   41.34 MB   0.0      0.0     0.0      0.0 0.0      0.0       0.0   0.0      0.0      0.0 0.00              0.00         0    0.000       0      0  Int      0/0    0.00 KB   0.0      0.0     0.0      0.0 0.0      0.0       0.0   0.0      0.0      0.0 0.00              0.00         0    0.000       0      0 ** Compaction Stats [default] ** Priority    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** debug 2021-04-20T14:06:19.449+0000 7f11e080a700  0 starting mon.host3 rank 2 at public addrs [v2:108.61.24.82:3300/0,v1:108.61.24.82:6789/0] at bind addrs [v2:108.61.24.82:3300/0,v1:108.61.24.82:6789/0] mon_data /var/lib/ceph/mon/ceph-host3 fsid c8ee2878-9d54-11eb-bbca-1c34da4b9fb6 debug 2021-04-20T14:06:19.449+0000 7f11e080a700  1 mon.host3@-1(???) e4 preinit fsid c8ee2878-9d54-11eb-bbca-1c34da4b9fb6 debug 2021-04-20T14:06:19.449+0000 7f11e080a700  0 mon.host3@-1(???).mds e1 new map debug 2021-04-20T14:06:19.449+0000 7f11e080a700  0 mon.host3@-1(???).mds e1 print_map e1 enable_multiple, ever_enabled_multiple: 0,0 compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} legacy client fscid: -1 No filesystems configured debug 2021-04-20T14:06:19.449+0000 7f11e080a700  0 mon.host3@-1(???).osd e293 crush map has features 3314933000852226048, adjusting msgr requires debug 2021-04-20T14:06:19.449+0000 7f11e080a700  0 mon.host3@-1(???).osd e293 crush map has features 288514051259236352, adjusting msgr requires debug 2021-04-20T14:06:19.449+0000 7f11e080a700  0 mon.host3@-1(???).osd e293 crush map has features 288514051259236352, adjusting msgr requires debug 2021-04-20T14:06:19.449+0000 7f11e080a700  0 mon.host3@-1(???).osd e293 crush map has features 288514051259236352, adjusting msgr requires audit 2021-04-14T20:10:31.225267+0000 mon.host1 (mon.0) 6360 : audit [DBG] from='client.? 108.61.24.80:0/2572567093' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch debug 2021-04-20T14:06:19.449+0000 7f11e080a700  1 mon.host3@-1(???).paxosservice(auth 1..86) refresh upgraded, format 0 -> 3 debug 2021-04-20T14:06:19.457+0000 7f11e080a700  0 mon.host3@-1(probing) e4  my rank is now 2 (was -1) debug 2021-04-20T14:06:19.661+0000 7f11c9e29700  0 mon.host3@2(probing) e5  removed from monmap, suicide.