[ceph-users] Re: After hardware failure tried to recover ceph and followed instructions for recovery using OSDS

Try to deploy the new osd so the PGs can recover as far as possible, wait for recovery. Try to bring the third MON back online while PGs recover. Then we'll see how far you get. Zitat von Manolis Daramas <mdaramas@motivian.com>:

It has now removed it successfully.

ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 1.65987 root default -3 0.29289 host node01 0 hdd 0.14650 osd.0 up 1.00000 1.00000 4 hdd 0.04880 osd.4 up 1.00000 1.00000 8 hdd 0.04880 osd.8 up 1.00000 1.00000 10 hdd 0.04880 osd.10 up 1.00000 1.00000 -5 0.68349 host node02 2 hdd 0.14650 osd.2 up 1.00000 1.00000 5 hdd 0.04880 osd.5 up 1.00000 1.00000 7 hdd 0.04880 osd.7 up 1.00000 1.00000 9 hdd 0.04880 osd.9 up 1.00000 1.00000 12 hdd 0.39059 osd.12 up 1.00000 1.00000 -7 0.68349 host node03 1 hdd 0.14650 osd.1 up 1.00000 1.00000 3 hdd 0.04880 osd.3 up 1.00000 1.00000 6 hdd 0.04880 osd.6 up 1.00000 1.00000 11 hdd 0.04880 osd.11 up 1.00000 1.00000 13 hdd 0.39059 osd.13 up 1.00000 1.00000

What are the next steps now ?

Thanks,

Manolis Daramas

-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Tuesday, December 5, 2023 4:21 PM To: Manolis Daramas <mdaramas@motivian.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: After hardware failure tried to recover ceph and followed instructions for recovery using OSDS

And how about this?

osd crush rm osd.14

Zitat von Manolis Daramas <mdaramas@motivian.com>:

...

Hi Eugen,

I think that it cannot even be removed with ceph osd purge command, see below:

root@node01:~# ceph osd purge 14 --force --yes-i-really-mean-it osd.14 does not exist

root@node01:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 2.05046 root default -3 0.68349 host node01 0 hdd 0.14650 osd.0 up 1.00000 1.00000 4 hdd 0.04880 osd.4 up 1.00000 1.00000 8 hdd 0.04880 osd.8 up 1.00000 1.00000 10 hdd 0.04880 osd.10 up 1.00000 1.00000 14 hdd 0.39059 osd.14 DNE 0 -5 0.68349 host node02 2 hdd 0.14650 osd.2 up 1.00000 1.00000 5 hdd 0.04880 osd.5 up 1.00000 1.00000 7 hdd 0.04880 osd.7 up 1.00000 1.00000 9 hdd 0.04880 osd.9 up 1.00000 1.00000 12 hdd 0.39059 osd.12 up 1.00000 1.00000 -7 0.68349 host node03 1 hdd 0.14650 osd.1 up 1.00000 1.00000 3 hdd 0.04880 osd.3 up 1.00000 1.00000 6 hdd 0.04880 osd.6 up 1.00000 1.00000 11 hdd 0.04880 osd.11 up 1.00000 1.00000 13 hdd 0.39059 osd.13 up 1.00000 1.00000

The output of journalctl with --no-pager option is below:

-- Logs begin at Thu 2023-11-23 06:16:17 EET, end at Tue 2023-12-05 16:01:49 EET. -- Dec 05 15:23:16 node01 systemd[1]: Started Ceph mon.node01 for be4304e4-b0d5-11ec-8c6a-2965d4229f37. Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 set uid:gid to 167:167 (ceph:ceph) Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable), process ceph-mon, pid 7 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 pidfile_write: ignore empty --pid-file Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.667+0000 7fab0e25bb80 0 load: jerasure load: lrc Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: RocksDB version: 6.15.5 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compile date Oct 25 2023 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB SUMMARY Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB Session ID: 69YYUY1IJK4BA9QQDDK8 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: CURRENT file: CURRENT Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: IDENTITY file: IDENTITY Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: MANIFEST file: MANIFEST-000226 size: 308 Bytes Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-node01/store.db dir, Total Num: 3, files: 000091.sst 000094.sst 000100.sst Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-node01/store.db: 000227.log size: 0 ; Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.error_if_exists: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.create_if_missing: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.paranoid_checks: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.env: 0x55be1e3aea80 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.fs: Posix File System Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.info_log: 0x55be20918360 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_file_opening_threads: 16 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.statistics: (nil) Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.use_fsync: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_log_file_size: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_manifest_file_size: 1073741824 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.log_file_time_to_roll: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.keep_log_file_num: 1000 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.recycle_log_file_num: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.allow_fallocate: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.allow_mmap_reads: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.allow_mmap_writes: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.use_direct_reads: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.create_missing_column_families: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.db_log_dir: Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-node01/store.db Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.table_cache_numshardbits: 6 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.WAL_ttl_seconds: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.WAL_size_limit_MB: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.manifest_preallocation_size: 4194304 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.is_fd_close_on_exec: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.advise_random_on_open: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.db_write_buffer_size: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.write_buffer_manager: 0x55be2168c990 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.access_hint_on_compaction_start: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.use_adaptive_mutex: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.rate_limiter: (nil) Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.wal_recovery_mode: 2 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.enable_thread_tracking: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.enable_pipelined_write: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.unordered_write: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.write_thread_max_yield_usec: 100 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.row_cache: None Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.wal_filter: None Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.avoid_flush_during_recovery: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.allow_ingest_behind: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.preserve_deletes: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.two_write_queues: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.manual_wal_flush: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.atomic_flush: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.persist_stats_to_disk: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.write_dbid_to_manifest: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.log_readahead_size: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.file_checksum_gen_factory: Unknown Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.best_efforts_recovery: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.allow_data_in_errors: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.db_host_id: __hostname__ Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_background_jobs: 2 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_background_compactions: -1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_subcompactions: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.delayed_write_rate : 16777216 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_total_wal_size: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.stats_dump_period_sec: 600 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.stats_persist_period_sec: 600 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.stats_history_buffer_size: 1048576 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_open_files: -1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bytes_per_sync: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.wal_bytes_per_sync: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.strict_bytes_per_sync: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_readahead_size: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_background_flushes: -1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compression algorithms supported: Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kZSTDNotFinalCompression supported: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kZSTD supported: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kXpressCompression supported: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kLZ4HCCompression supported: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kZlibCompression supported: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kSnappyCompression supported: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kLZ4Compression supported: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kBZip2Compression supported: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Fast CRC32 supported: Supported on x86 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.merge_operator: Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_filter: None Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_filter_factory: None Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.sst_partitioner_factory: None Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.memtable_factory: SkipListFactory Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.table_factory: BlockBasedTable Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55be2096afb0) Dec 05 15:23:17 node01 bash[1969]: cache_index_and_filter_blocks: 1 Dec 05 15:23:17 node01 bash[1969]: cache_index_and_filter_blocks_with_high_priority: 0 Dec 05 15:23:17 node01 bash[1969]: pin_l0_filter_and_index_blocks_in_cache: 0 Dec 05 15:23:17 node01 bash[1969]: pin_top_level_index_and_filter: 1 Dec 05 15:23:17 node01 bash[1969]: index_type: 0 Dec 05 15:23:17 node01 bash[1969]: data_block_index_type: 0 Dec 05 15:23:17 node01 bash[1969]: index_shortening: 1 Dec 05 15:23:17 node01 bash[1969]: data_block_hash_table_util_ratio: 0.750000 Dec 05 15:23:17 node01 bash[1969]: hash_index_allow_collision: 1 Dec 05 15:23:17 node01 bash[1969]: checksum: 1 Dec 05 15:23:17 node01 bash[1969]: no_block_cache: 0 Dec 05 15:23:17 node01 bash[1969]: block_cache: 0x55be2099d090 Dec 05 15:23:17 node01 bash[1969]: block_cache_name: BinnedLRUCache Dec 05 15:23:17 node01 bash[1969]: block_cache_options: Dec 05 15:23:17 node01 bash[1969]: capacity : 536870912 Dec 05 15:23:17 node01 bash[1969]: num_shard_bits : 4 Dec 05 15:23:17 node01 bash[1969]: strict_capacity_limit : 0 Dec 05 15:23:17 node01 bash[1969]: high_pri_pool_ratio: 0.000 Dec 05 15:23:17 node01 bash[1969]: block_cache_compressed: (nil) Dec 05 15:23:17 node01 bash[1969]: persistent_cache: (nil) Dec 05 15:23:17 node01 bash[1969]: block_size: 4096 Dec 05 15:23:17 node01 bash[1969]: block_size_deviation: 10 Dec 05 15:23:17 node01 bash[1969]: block_restart_interval: 16 Dec 05 15:23:17 node01 bash[1969]: index_block_restart_interval: 1 Dec 05 15:23:17 node01 bash[1969]: metadata_block_size: 4096 Dec 05 15:23:17 node01 bash[1969]: partition_filters: 0 Dec 05 15:23:17 node01 bash[1969]: use_delta_encoding: 1 Dec 05 15:23:17 node01 bash[1969]: filter_policy: rocksdb.BuiltinBloomFilter Dec 05 15:23:17 node01 bash[1969]: whole_key_filtering: 1 Dec 05 15:23:17 node01 bash[1969]: verify_compression: 0 Dec 05 15:23:17 node01 bash[1969]: read_amp_bytes_per_bit: 0 Dec 05 15:23:17 node01 bash[1969]: format_version: 4 Dec 05 15:23:17 node01 bash[1969]: enable_index_compression: 1 Dec 05 15:23:17 node01 bash[1969]: block_align: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.write_buffer_size: 33554432 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_write_buffer_number: 2 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression: NoCompression Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression: Disabled Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.prefix_extractor: nullptr Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.num_levels: 7 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression_opts.enabled: false Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression_opts.window_bits: -14 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression_opts.level: 32767 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression_opts.strategy: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression_opts.parallel_threads: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression_opts.enabled: false Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.level0_stop_writes_trigger: 36 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.target_file_size_base: 67108864 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.target_file_size_multiplier: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_compaction_bytes: 1677721600 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.arena_block_size: 4194304 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.disable_auto_compactions: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.table_properties_collectors: Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.inplace_update_support: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.inplace_update_num_locks: 10000 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.memtable_whole_key_filtering: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.memtable_huge_page_size: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bloom_locality: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_successive_merges: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.optimize_filters_for_hits: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.paranoid_file_checks: Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.force_consistency_checks: 1 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.report_bg_io_stats: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.ttl: 2592000 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.periodic_compaction_seconds: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.enable_blob_files: false Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.min_blob_size: 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.blob_file_size: 268435456 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.blob_compression_type: NoCompression Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.enable_blob_garbage_collection: false Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:3457] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 succeeded,manifest_file_number is 226, next_file_number is 228, last_sequence is 21201, log_number is 223,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 223 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:4082] Creating manifest 230 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:3457] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1701782597693048, "job": 1, "event": "recovery_started", "wal_files": [227]} Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #227 mode 2 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:4082] Creating manifest 231 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:3457] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1701782597693989, "job": 1, "event": "recovery_finished"} Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-node01/store.db/000227.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55be20931180 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: DB pointer 0x55be20a06000 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: [db/db_impl/db_impl.cc:903] Dec 05 15:23:17 node01 bash[1969]: ** DB Stats ** Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval Dec 05 15:23:17 node01 bash[1969]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Dec 05 15:23:17 node01 bash[1969]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Dec 05 15:23:17 node01 bash[1969]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Dec 05 15:23:17 node01 bash[1969]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Dec 05 15:23:17 node01 bash[1969]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Dec 05 15:23:17 node01 bash[1969]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Dec 05 15:23:17 node01 bash[1969]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** Dec 05 15:23:17 node01 bash[1969]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Dec 05 15:23:17 node01 bash[1969]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, interval 0.000 Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, interval 0.000 Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative 0, interval 0 Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, interval 0 Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By Level [default] ** Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Dec 05 15:23:17 node01 bash[1969]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** Dec 05 15:23:17 node01 bash[1969]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Dec 05 15:23:17 node01 bash[1969]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, interval 0.000 Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, interval 0.000 Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative 0, interval 0 Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, interval 0 Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By Level [default] ** Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.719+0000 7fab0e25bb80 0 starting mon.node01 rank 0 at public addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] at bind addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] mon_data /var/lib/ceph/mon/ceph-node01 fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.723+0000 7fab0e25bb80 1 mon.node01@-1(???) e0 preinit fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37 Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd e9898 crush map has features 3314933000854323200, adjusting msgr requires Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd e9898 crush map has features 432629239337189376, adjusting msgr requires Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd e9898 crush map has features 432629239337189376, adjusting msgr requires Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd e9898 crush map has features 432629239337189376, adjusting msgr requires Dec 05 15:23:17 node01 bash[1969]: terminate called after throwing an instance of 'std::invalid_argument' Dec 05 15:23:17 node01 bash[1969]: what(): stoull Dec 05 15:23:17 node01 bash[1969]: *** Caught signal (Aborted) ** Dec 05 15:23:17 node01 bash[1969]: in thread 7fab0e25bb80 thread_name:ceph-mon Dec 05 15:23:17 node01 bash[1969]: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) Dec 05 15:23:17 node01 bash[1969]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fab0b6c8cf0] Dec 05 15:23:17 node01 bash[1969]: 2: gsignal() Dec 05 15:23:17 node01 bash[1969]: 3: abort() Dec 05 15:23:17 node01 bash[1969]: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7fab0acda09b] Dec 05 15:23:17 node01 bash[1969]: 5: /lib64/libstdc++.so.6(+0x9654c) [0x7fab0ace054c] Dec 05 15:23:17 node01 bash[1969]: 6: /lib64/libstdc++.so.6(+0x965a7) [0x7fab0ace05a7] Dec 05 15:23:17 node01 bash[1969]: 7: /lib64/libstdc++.so.6(+0x96808) [0x7fab0ace0808] Dec 05 15:23:17 node01 bash[1969]: 8: /lib64/libstdc++.so.6(+0x91ff9) [0x7fab0acdbff9] Dec 05 15:23:17 node01 bash[1969]: 9: (LogMonitor::log_external_backlog()+0xe70) [0x55be1dc27190] Dec 05 15:23:17 node01 bash[1969]: 10: (LogMonitor::update_from_paxos(bool*)+0x54) [0x55be1dc27334] Dec 05 15:23:17 node01 bash[1969]: 11: (Monitor::refresh_from_paxos(bool*)+0x104) [0x55be1db9ae74] Dec 05 15:23:17 node01 bash[1969]: 12: (Monitor::preinit()+0x95d) [0x55be1dbc93ad] Dec 05 15:23:17 node01 bash[1969]: 13: main() Dec 05 15:23:17 node01 bash[1969]: 14: __libc_start_main() Dec 05 15:23:17 node01 bash[1969]: 15: _start() Dec 05 15:23:17 node01 bash[1969]: debug 2023-12-05T13:23:17.735+0000 7fab0e25bb80 -1 *** Caught signal (Aborted) ** Dec 05 15:23:17 node01 bash[1969]: in thread 7fab0e25bb80 thread_name:ceph-mon Dec 05 15:23:17 node01 bash[1969]: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) Dec 05 15:23:17 node01 bash[1969]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fab0b6c8cf0] Dec 05 15:23:17 node01 bash[1969]: 2: gsignal() Dec 05 15:23:17 node01 bash[1969]: 3: abort() Dec 05 15:23:17 node01 bash[1969]: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7fab0acda09b] Dec 05 15:23:17 node01 bash[1969]: 5: /lib64/libstdc++.so.6(+0x9654c) [0x7fab0ace054c] Dec 05 15:23:17 node01 bash[1969]: 6: /lib64/libstdc++.so.6(+0x965a7) [0x7fab0ace05a7] Dec 05 15:23:17 node01 bash[1969]: 7: /lib64/libstdc++.so.6(+0x96808) [0x7fab0ace0808] Dec 05 15:23:17 node01 bash[1969]: 8: /lib64/libstdc++.so.6(+0x91ff9) [0x7fab0acdbff9] Dec 05 15:23:17 node01 bash[1969]: 9: (LogMonitor::log_external_backlog()+0xe70) [0x55be1dc27190] Dec 05 15:23:17 node01 bash[1969]: 10: (LogMonitor::update_from_paxos(bool*)+0x54) [0x55be1dc27334] Dec 05 15:23:17 node01 bash[1969]: 11: (Monitor::refresh_from_paxos(bool*)+0x104) [0x55be1db9ae74] Dec 05 15:23:17 node01 bash[1969]: 12: (Monitor::preinit()+0x95d) [0x55be1dbc93ad] Dec 05 15:23:17 node01 bash[1969]: 13: main() Dec 05 15:23:17 node01 bash[1969]: 14: __libc_start_main() Dec 05 15:23:17 node01 bash[1969]: 15: _start() Dec 05 15:23:17 node01 bash[1969]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Dec 05 15:23:17 node01 bash[1969]: --- begin dump of recent events --- Dec 05 15:23:17 node01 bash[1969]: debug -290> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command assert hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -289> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command abort hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -288> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command leak_some_memory hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -287> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command perfcounters_dump hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -286> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command 1 hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -285> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command perf dump hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -284> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command perfcounters_schema hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -283> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command perf histogram dump hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -282> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command 2 hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -281> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command perf schema hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -280> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command perf histogram schema hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -279> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command perf reset hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -278> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command config show hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -277> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command config help hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -276> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command config set hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -275> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command config unset hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -274> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command config get hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -273> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command config diff hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -272> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command config diff get hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -271> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command injectargs hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -270> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command log flush hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -269> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command log dump hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -268> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command log reopen hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -267> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command dump_mempools hook 0x55be2168a068 Dec 05 15:23:17 node01 bash[1969]: debug -266> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 set uid:gid to 167:167 (ceph:ceph) Dec 05 15:23:17 node01 bash[1969]: debug -265> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable), process ceph-mon, pid 7 Dec 05 15:23:17 node01 bash[1969]: debug -264> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 pidfile_write: ignore empty --pid-file Dec 05 15:23:17 node01 bash[1969]: debug -263> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) init /var/run/ceph/ceph-mon.node01.asok Dec 05 15:23:17 node01 bash[1969]: debug -262> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) bind_and_listen /var/run/ceph/ceph-mon.node01.asok Dec 05 15:23:17 node01 bash[1969]: debug -261> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command 0 hook 0x55be2096b030 Dec 05 15:23:17 node01 bash[1969]: debug -260> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command version hook 0x55be2096b030 Dec 05 15:23:17 node01 bash[1969]: debug -259> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command git_version hook 0x55be2096b030 Dec 05 15:23:17 node01 bash[1969]: debug -258> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command help hook 0x55be208de5b0 Dec 05 15:23:17 node01 bash[1969]: debug -257> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command get_command_descriptions hook 0x55be208de5a0 Dec 05 15:23:17 node01 bash[1969]: debug -256> 2023-12-05T13:23:17.639+0000 7fab076ba700 5 asok(0x55be20bc0000) entry start Dec 05 15:23:17 node01 bash[1969]: debug -255> 2023-12-05T13:23:17.667+0000 7fab0e25bb80 0 load: jerasure load: lrc Dec 05 15:23:17 node01 bash[1969]: debug -254> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option level_compaction_dynamic_level_bytes = true Dec 05 15:23:17 node01 bash[1969]: debug -253> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option write_buffer_size = 33554432 Dec 05 15:23:17 node01 bash[1969]: debug -252> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option compression = kNoCompression Dec 05 15:23:17 node01 bash[1969]: debug -251> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option level_compaction_dynamic_level_bytes = true Dec 05 15:23:17 node01 bash[1969]: debug -250> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option write_buffer_size = 33554432 Dec 05 15:23:17 node01 bash[1969]: debug -249> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option compression = kNoCompression Dec 05 15:23:17 node01 bash[1969]: debug -248> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 5 rocksdb: verify_sharding column families from rocksdb: [default] Dec 05 15:23:17 node01 bash[1969]: debug -247> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: RocksDB version: 6.15.5 Dec 05 15:23:17 node01 bash[1969]: debug -246> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ Dec 05 15:23:17 node01 bash[1969]: debug -245> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compile date Oct 25 2023 Dec 05 15:23:17 node01 bash[1969]: debug -244> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB SUMMARY Dec 05 15:23:17 node01 bash[1969]: debug -243> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB Session ID: 69YYUY1IJK4BA9QQDDK8 Dec 05 15:23:17 node01 bash[1969]: debug -242> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: CURRENT file: CURRENT Dec 05 15:23:17 node01 bash[1969]: debug -241> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: IDENTITY file: IDENTITY Dec 05 15:23:17 node01 bash[1969]: debug -240> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: MANIFEST file: MANIFEST-000226 size: 308 Bytes Dec 05 15:23:17 node01 bash[1969]: debug -239> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-node01/store.db dir, Total Num: 3, files: 000091.sst 000094.sst 000100.sst Dec 05 15:23:17 node01 bash[1969]: debug -238> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-node01/store.db: 000227.log size: 0 ; Dec 05 15:23:17 node01 bash[1969]: debug -237> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.error_if_exists: 0 Dec 05 15:23:17 node01 bash[1969]: debug -236> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.create_if_missing: 0 Dec 05 15:23:17 node01 bash[1969]: debug -235> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.paranoid_checks: 1 Dec 05 15:23:17 node01 bash[1969]: debug -234> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 Dec 05 15:23:17 node01 bash[1969]: debug -233> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.env: 0x55be1e3aea80 Dec 05 15:23:17 node01 bash[1969]: debug -232> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.fs: Posix File System Dec 05 15:23:17 node01 bash[1969]: debug -231> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.info_log: 0x55be20918360 Dec 05 15:23:17 node01 bash[1969]: debug -230> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_file_opening_threads: 16 Dec 05 15:23:17 node01 bash[1969]: debug -229> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.statistics: (nil) Dec 05 15:23:17 node01 bash[1969]: debug -228> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.use_fsync: 0 Dec 05 15:23:17 node01 bash[1969]: debug -227> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_log_file_size: 0 Dec 05 15:23:17 node01 bash[1969]: debug -226> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_manifest_file_size: 1073741824 Dec 05 15:23:17 node01 bash[1969]: debug -225> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.log_file_time_to_roll: 0 Dec 05 15:23:17 node01 bash[1969]: debug -224> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.keep_log_file_num: 1000 Dec 05 15:23:17 node01 bash[1969]: debug -223> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.recycle_log_file_num: 0 Dec 05 15:23:17 node01 bash[1969]: debug -222> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.allow_fallocate: 1 Dec 05 15:23:17 node01 bash[1969]: debug -221> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.allow_mmap_reads: 0 Dec 05 15:23:17 node01 bash[1969]: debug -220> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.allow_mmap_writes: 0 Dec 05 15:23:17 node01 bash[1969]: debug -219> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.use_direct_reads: 0 Dec 05 15:23:17 node01 bash[1969]: debug -218> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 Dec 05 15:23:17 node01 bash[1969]: debug -217> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.create_missing_column_families: 0 Dec 05 15:23:17 node01 bash[1969]: debug -216> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.db_log_dir: Dec 05 15:23:17 node01 bash[1969]: debug -215> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-node01/store.db Dec 05 15:23:17 node01 bash[1969]: debug -214> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.table_cache_numshardbits: 6 Dec 05 15:23:17 node01 bash[1969]: debug -213> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.WAL_ttl_seconds: 0 Dec 05 15:23:17 node01 bash[1969]: debug -212> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.WAL_size_limit_MB: 0 Dec 05 15:23:17 node01 bash[1969]: debug -211> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 Dec 05 15:23:17 node01 bash[1969]: debug -210> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.manifest_preallocation_size: 4194304 Dec 05 15:23:17 node01 bash[1969]: debug -209> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.is_fd_close_on_exec: 1 Dec 05 15:23:17 node01 bash[1969]: debug -208> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.advise_random_on_open: 1 Dec 05 15:23:17 node01 bash[1969]: debug -207> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.db_write_buffer_size: 0 Dec 05 15:23:17 node01 bash[1969]: debug -206> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.write_buffer_manager: 0x55be2168c990 Dec 05 15:23:17 node01 bash[1969]: debug -205> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.access_hint_on_compaction_start: 1 Dec 05 15:23:17 node01 bash[1969]: debug -204> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 Dec 05 15:23:17 node01 bash[1969]: debug -203> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 Dec 05 15:23:17 node01 bash[1969]: debug -202> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.use_adaptive_mutex: 0 Dec 05 15:23:17 node01 bash[1969]: debug -201> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.rate_limiter: (nil) Dec 05 15:23:17 node01 bash[1969]: debug -200> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 Dec 05 15:23:17 node01 bash[1969]: debug -199> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.wal_recovery_mode: 2 Dec 05 15:23:17 node01 bash[1969]: debug -198> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.enable_thread_tracking: 0 Dec 05 15:23:17 node01 bash[1969]: debug -197> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.enable_pipelined_write: 0 Dec 05 15:23:17 node01 bash[1969]: debug -196> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.unordered_write: 0 Dec 05 15:23:17 node01 bash[1969]: debug -195> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 Dec 05 15:23:17 node01 bash[1969]: debug -194> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 Dec 05 15:23:17 node01 bash[1969]: debug -193> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.write_thread_max_yield_usec: 100 Dec 05 15:23:17 node01 bash[1969]: debug -192> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 Dec 05 15:23:17 node01 bash[1969]: debug -191> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.row_cache: None Dec 05 15:23:17 node01 bash[1969]: debug -190> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.wal_filter: None Dec 05 15:23:17 node01 bash[1969]: debug -189> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.avoid_flush_during_recovery: 0 Dec 05 15:23:17 node01 bash[1969]: debug -188> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.allow_ingest_behind: 0 Dec 05 15:23:17 node01 bash[1969]: debug -187> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.preserve_deletes: 0 Dec 05 15:23:17 node01 bash[1969]: debug -186> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.two_write_queues: 0 Dec 05 15:23:17 node01 bash[1969]: debug -185> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.manual_wal_flush: 0 Dec 05 15:23:17 node01 bash[1969]: debug -184> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.atomic_flush: 0 Dec 05 15:23:17 node01 bash[1969]: debug -183> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 Dec 05 15:23:17 node01 bash[1969]: debug -182> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.persist_stats_to_disk: 0 Dec 05 15:23:17 node01 bash[1969]: debug -181> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.write_dbid_to_manifest: 0 Dec 05 15:23:17 node01 bash[1969]: debug -180> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.log_readahead_size: 0 Dec 05 15:23:17 node01 bash[1969]: debug -179> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.file_checksum_gen_factory: Unknown Dec 05 15:23:17 node01 bash[1969]: debug -178> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.best_efforts_recovery: 0 Dec 05 15:23:17 node01 bash[1969]: debug -177> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 Dec 05 15:23:17 node01 bash[1969]: debug -176> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 Dec 05 15:23:17 node01 bash[1969]: debug -175> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.allow_data_in_errors: 0 Dec 05 15:23:17 node01 bash[1969]: debug -174> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.db_host_id: __hostname__ Dec 05 15:23:17 node01 bash[1969]: debug -173> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_background_jobs: 2 Dec 05 15:23:17 node01 bash[1969]: debug -172> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_background_compactions: -1 Dec 05 15:23:17 node01 bash[1969]: debug -171> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_subcompactions: 1 Dec 05 15:23:17 node01 bash[1969]: debug -170> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 Dec 05 15:23:17 node01 bash[1969]: debug -169> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 Dec 05 15:23:17 node01 bash[1969]: debug -168> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.delayed_write_rate : 16777216 Dec 05 15:23:17 node01 bash[1969]: debug -167> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_total_wal_size: 0 Dec 05 15:23:17 node01 bash[1969]: debug -166> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 Dec 05 15:23:17 node01 bash[1969]: debug -165> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.stats_dump_period_sec: 600 Dec 05 15:23:17 node01 bash[1969]: debug -164> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.stats_persist_period_sec: 600 Dec 05 15:23:17 node01 bash[1969]: debug -163> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.stats_history_buffer_size: 1048576 Dec 05 15:23:17 node01 bash[1969]: debug -162> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_open_files: -1 Dec 05 15:23:17 node01 bash[1969]: debug -161> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bytes_per_sync: 0 Dec 05 15:23:17 node01 bash[1969]: debug -160> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.wal_bytes_per_sync: 0 Dec 05 15:23:17 node01 bash[1969]: debug -159> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.strict_bytes_per_sync: 0 Dec 05 15:23:17 node01 bash[1969]: debug -158> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_readahead_size: 0 Dec 05 15:23:17 node01 bash[1969]: debug -157> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_background_flushes: -1 Dec 05 15:23:17 node01 bash[1969]: debug -156> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compression algorithms supported: Dec 05 15:23:17 node01 bash[1969]: debug -155> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kZSTDNotFinalCompression supported: 0 Dec 05 15:23:17 node01 bash[1969]: debug -154> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kZSTD supported: 0 Dec 05 15:23:17 node01 bash[1969]: debug -153> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kXpressCompression supported: 0 Dec 05 15:23:17 node01 bash[1969]: debug -152> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kLZ4HCCompression supported: 1 Dec 05 15:23:17 node01 bash[1969]: debug -151> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kZlibCompression supported: 1 Dec 05 15:23:17 node01 bash[1969]: debug -150> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kSnappyCompression supported: 1 Dec 05 15:23:17 node01 bash[1969]: debug -149> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kLZ4Compression supported: 1 Dec 05 15:23:17 node01 bash[1969]: debug -148> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kBZip2Compression supported: 0 Dec 05 15:23:17 node01 bash[1969]: debug -147> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Fast CRC32 supported: Supported on x86 Dec 05 15:23:17 node01 bash[1969]: debug -146> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 Dec 05 15:23:17 node01 bash[1969]: debug -145> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: Dec 05 15:23:17 node01 bash[1969]: debug -144> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator Dec 05 15:23:17 node01 bash[1969]: debug -143> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.merge_operator: Dec 05 15:23:17 node01 bash[1969]: debug -142> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_filter: None Dec 05 15:23:17 node01 bash[1969]: debug -141> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_filter_factory: None Dec 05 15:23:17 node01 bash[1969]: debug -140> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.sst_partitioner_factory: None Dec 05 15:23:17 node01 bash[1969]: debug -139> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.memtable_factory: SkipListFactory Dec 05 15:23:17 node01 bash[1969]: debug -138> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.table_factory: BlockBasedTable Dec 05 15:23:17 node01 bash[1969]: debug -137> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55be2096afb0) Dec 05 15:23:17 node01 bash[1969]: cache_index_and_filter_blocks: 1 Dec 05 15:23:17 node01 bash[1969]: cache_index_and_filter_blocks_with_high_priority: 0 Dec 05 15:23:17 node01 bash[1969]: pin_l0_filter_and_index_blocks_in_cache: 0 Dec 05 15:23:17 node01 bash[1969]: pin_top_level_index_and_filter: 1 Dec 05 15:23:17 node01 bash[1969]: index_type: 0 Dec 05 15:23:17 node01 bash[1969]: data_block_index_type: 0 Dec 05 15:23:17 node01 bash[1969]: index_shortening: 1 Dec 05 15:23:17 node01 bash[1969]: data_block_hash_table_util_ratio: 0.750000 Dec 05 15:23:17 node01 bash[1969]: hash_index_allow_collision: 1 Dec 05 15:23:17 node01 bash[1969]: checksum: 1 Dec 05 15:23:17 node01 bash[1969]: no_block_cache: 0 Dec 05 15:23:17 node01 bash[1969]: block_cache: 0x55be2099d090 Dec 05 15:23:17 node01 bash[1969]: block_cache_name: BinnedLRUCache Dec 05 15:23:17 node01 bash[1969]: block_cache_options: Dec 05 15:23:17 node01 bash[1969]: capacity : 536870912 Dec 05 15:23:17 node01 bash[1969]: num_shard_bits : 4 Dec 05 15:23:17 node01 bash[1969]: strict_capacity_limit : 0 Dec 05 15:23:17 node01 bash[1969]: high_pri_pool_ratio: 0.000 Dec 05 15:23:17 node01 bash[1969]: block_cache_compressed: (nil) Dec 05 15:23:17 node01 bash[1969]: persistent_cache: (nil) Dec 05 15:23:17 node01 bash[1969]: block_size: 4096 Dec 05 15:23:17 node01 bash[1969]: block_size_deviation: 10 Dec 05 15:23:17 node01 bash[1969]: block_restart_interval: 16 Dec 05 15:23:17 node01 bash[1969]: index_block_restart_interval: 1 Dec 05 15:23:17 node01 bash[1969]: metadata_block_size: 4096 Dec 05 15:23:17 node01 bash[1969]: partition_filters: 0 Dec 05 15:23:17 node01 bash[1969]: use_delta_encoding: 1 Dec 05 15:23:17 node01 bash[1969]: filter_policy: rocksdb.BuiltinBloomFilter Dec 05 15:23:17 node01 bash[1969]: whole_key_filtering: 1 Dec 05 15:23:17 node01 bash[1969]: verify_compression: 0 Dec 05 15:23:17 node01 bash[1969]: read_amp_bytes_per_bit: 0 Dec 05 15:23:17 node01 bash[1969]: format_version: 4 Dec 05 15:23:17 node01 bash[1969]: enable_index_compression: 1 Dec 05 15:23:17 node01 bash[1969]: block_align: 0 Dec 05 15:23:17 node01 bash[1969]: debug -136> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.write_buffer_size: 33554432 Dec 05 15:23:17 node01 bash[1969]: debug -135> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_write_buffer_number: 2 Dec 05 15:23:17 node01 bash[1969]: debug -134> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression: NoCompression Dec 05 15:23:17 node01 bash[1969]: debug -133> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression: Disabled Dec 05 15:23:17 node01 bash[1969]: debug -132> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.prefix_extractor: nullptr Dec 05 15:23:17 node01 bash[1969]: debug -131> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr Dec 05 15:23:17 node01 bash[1969]: debug -130> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.num_levels: 7 Dec 05 15:23:17 node01 bash[1969]: debug -129> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 Dec 05 15:23:17 node01 bash[1969]: debug -128> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 Dec 05 15:23:17 node01 bash[1969]: debug -127> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 Dec 05 15:23:17 node01 bash[1969]: debug -126> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 Dec 05 15:23:17 node01 bash[1969]: debug -125> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 Dec 05 15:23:17 node01 bash[1969]: debug -124> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 Dec 05 15:23:17 node01 bash[1969]: debug -123> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 Dec 05 15:23:17 node01 bash[1969]: debug -122> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 Dec 05 15:23:17 node01 bash[1969]: debug -121> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 Dec 05 15:23:17 node01 bash[1969]: debug -120> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression_opts.enabled: false Dec 05 15:23:17 node01 bash[1969]: debug -119> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression_opts.window_bits: -14 Dec 05 15:23:17 node01 bash[1969]: debug -118> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression_opts.level: 32767 Dec 05 15:23:17 node01 bash[1969]: debug -117> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression_opts.strategy: 0 Dec 05 15:23:17 node01 bash[1969]: debug -116> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 Dec 05 15:23:17 node01 bash[1969]: debug -115> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 Dec 05 15:23:17 node01 bash[1969]: debug -114> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression_opts.parallel_threads: 1 Dec 05 15:23:17 node01 bash[1969]: debug -113> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression_opts.enabled: false Dec 05 15:23:17 node01 bash[1969]: debug -112> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 Dec 05 15:23:17 node01 bash[1969]: debug -111> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 Dec 05 15:23:17 node01 bash[1969]: debug -110> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.level0_stop_writes_trigger: 36 Dec 05 15:23:17 node01 bash[1969]: debug -109> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.target_file_size_base: 67108864 Dec 05 15:23:17 node01 bash[1969]: debug -108> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.target_file_size_multiplier: 1 Dec 05 15:23:17 node01 bash[1969]: debug -107> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 Dec 05 15:23:17 node01 bash[1969]: debug -106> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 Dec 05 15:23:17 node01 bash[1969]: debug -105> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 Dec 05 15:23:17 node01 bash[1969]: debug -104> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 Dec 05 15:23:17 node01 bash[1969]: debug -103> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 Dec 05 15:23:17 node01 bash[1969]: debug -102> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 Dec 05 15:23:17 node01 bash[1969]: debug -101> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 Dec 05 15:23:17 node01 bash[1969]: debug -100> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 Dec 05 15:23:17 node01 bash[1969]: debug -99> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 Dec 05 15:23:17 node01 bash[1969]: debug -98> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 Dec 05 15:23:17 node01 bash[1969]: debug -97> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 Dec 05 15:23:17 node01 bash[1969]: debug -96> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_compaction_bytes: 1677721600 Dec 05 15:23:17 node01 bash[1969]: debug -95> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.arena_block_size: 4194304 Dec 05 15:23:17 node01 bash[1969]: debug -94> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 Dec 05 15:23:17 node01 bash[1969]: debug -93> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 Dec 05 15:23:17 node01 bash[1969]: debug -92> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 Dec 05 15:23:17 node01 bash[1969]: debug -91> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.disable_auto_compactions: 0 Dec 05 15:23:17 node01 bash[1969]: debug -90> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel Dec 05 15:23:17 node01 bash[1969]: debug -89> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio Dec 05 15:23:17 node01 bash[1969]: debug -88> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 Dec 05 15:23:17 node01 bash[1969]: debug -87> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 Dec 05 15:23:17 node01 bash[1969]: debug -86> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 Dec 05 15:23:17 node01 bash[1969]: debug -85> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 Dec 05 15:23:17 node01 bash[1969]: debug -84> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 Dec 05 15:23:17 node01 bash[1969]: debug -83> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize Dec 05 15:23:17 node01 bash[1969]: debug -82> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 Dec 05 15:23:17 node01 bash[1969]: debug -81> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 Dec 05 15:23:17 node01 bash[1969]: debug -80> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.table_properties_collectors: Dec 05 15:23:17 node01 bash[1969]: debug -79> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.inplace_update_support: 0 Dec 05 15:23:17 node01 bash[1969]: debug -78> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.inplace_update_num_locks: 10000 Dec 05 15:23:17 node01 bash[1969]: debug -77> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 Dec 05 15:23:17 node01 bash[1969]: debug -76> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.memtable_whole_key_filtering: 0 Dec 05 15:23:17 node01 bash[1969]: debug -75> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.memtable_huge_page_size: 0 Dec 05 15:23:17 node01 bash[1969]: debug -74> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bloom_locality: 0 Dec 05 15:23:17 node01 bash[1969]: debug -73> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_successive_merges: 0 Dec 05 15:23:17 node01 bash[1969]: debug -72> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.optimize_filters_for_hits: 0 Dec 05 15:23:17 node01 bash[1969]: debug -71> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.paranoid_file_checks: 0 Dec 05 15:23:17 node01 bash[1969]: debug -70> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.force_consistency_checks: 1 Dec 05 15:23:17 node01 bash[1969]: debug -69> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.report_bg_io_stats: 0 Dec 05 15:23:17 node01 bash[1969]: debug -68> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.ttl: 2592000 Dec 05 15:23:17 node01 bash[1969]: debug -67> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.periodic_compaction_seconds: 0 Dec 05 15:23:17 node01 bash[1969]: debug -66> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.enable_blob_files: false Dec 05 15:23:17 node01 bash[1969]: debug -65> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.min_blob_size: 0 Dec 05 15:23:17 node01 bash[1969]: debug -64> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.blob_file_size: 268435456 Dec 05 15:23:17 node01 bash[1969]: debug -63> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.blob_compression_type: NoCompression Dec 05 15:23:17 node01 bash[1969]: debug -62> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.enable_blob_garbage_collection: false Dec 05 15:23:17 node01 bash[1969]: debug -61> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 Dec 05 15:23:17 node01 bash[1969]: debug -60> 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:3457] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. Dec 05 15:23:17 node01 bash[1969]: debug -59> 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 succeeded,manifest_file_number is 226, next_file_number is 228, last_sequence is 21201, log_number is 223,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 Dec 05 15:23:17 node01 bash[1969]: debug -58> 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 223 Dec 05 15:23:17 node01 bash[1969]: debug -57> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:4082] Creating manifest 230 Dec 05 15:23:17 node01 bash[1969]: debug -56> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:3457] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. Dec 05 15:23:17 node01 bash[1969]: debug -55> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1701782597693048, "job": 1, "event": "recovery_started", "wal_files": [227]} Dec 05 15:23:17 node01 bash[1969]: debug -54> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #227 mode 2 Dec 05 15:23:17 node01 bash[1969]: debug -53> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:4082] Creating manifest 231 Dec 05 15:23:17 node01 bash[1969]: debug -52> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:3457] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. Dec 05 15:23:17 node01 bash[1969]: debug -51> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1701782597693989, "job": 1, "event": "recovery_finished"} Dec 05 15:23:17 node01 bash[1969]: debug -50> 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-node01/store.db/000227.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 Dec 05 15:23:17 node01 bash[1969]: debug -49> 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55be20931180 Dec 05 15:23:17 node01 bash[1969]: debug -48> 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: DB pointer 0x55be20a06000 Dec 05 15:23:17 node01 bash[1969]: debug -47> 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- Dec 05 15:23:17 node01 bash[1969]: debug -46> 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: [db/db_impl/db_impl.cc:903] Dec 05 15:23:17 node01 bash[1969]: ** DB Stats ** Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval Dec 05 15:23:17 node01 bash[1969]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Dec 05 15:23:17 node01 bash[1969]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Dec 05 15:23:17 node01 bash[1969]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Dec 05 15:23:17 node01 bash[1969]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Dec 05 15:23:17 node01 bash[1969]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Dec 05 15:23:17 node01 bash[1969]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Dec 05 15:23:17 node01 bash[1969]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** Dec 05 15:23:17 node01 bash[1969]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Dec 05 15:23:17 node01 bash[1969]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, interval 0.000 Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, interval 0.000 Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative 0, interval 0 Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, interval 0 Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By Level [default] ** Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Dec 05 15:23:17 node01 bash[1969]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** Dec 05 15:23:17 node01 bash[1969]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Dec 05 15:23:17 node01 bash[1969]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, interval 0.000 Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, interval 0.000 Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative 0, interval 0 Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, interval 0 Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By Level [default] ** Dec 05 15:23:17 node01 bash[1969]: debug -45> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding auth protocol: cephx Dec 05 15:23:17 node01 bash[1969]: debug -44> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding auth protocol: cephx Dec 05 15:23:17 node01 bash[1969]: debug -43> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding auth protocol: cephx Dec 05 15:23:17 node01 bash[1969]: debug -42> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding auth protocol: none Dec 05 15:23:17 node01 bash[1969]: debug -41> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -40> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -39> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -38> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -37> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -36> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -35> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -34> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -33> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -32> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -31> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -30> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -29> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-node01/keyring Dec 05 15:23:17 node01 bash[1969]: debug -28> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 0 starting mon.node01 rank 0 at public addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] at bind addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] mon_data /var/lib/ceph/mon/ceph-node01 fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37 Dec 05 15:23:17 node01 bash[1969]: debug -27> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding auth protocol: cephx Dec 05 15:23:17 node01 bash[1969]: debug -26> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding auth protocol: cephx Dec 05 15:23:17 node01 bash[1969]: debug -25> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding auth protocol: cephx Dec 05 15:23:17 node01 bash[1969]: debug -24> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding auth protocol: none Dec 05 15:23:17 node01 bash[1969]: debug -23> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -22> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -21> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -20> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -19> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -18> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -17> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -16> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -15> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -14> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -13> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -12> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -11> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-node01/keyring Dec 05 15:23:17 node01 bash[1969]: debug -10> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 adding auth protocol: cephx Dec 05 15:23:17 node01 bash[1969]: debug -9> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 adding auth protocol: cephx Dec 05 15:23:17 node01 bash[1969]: debug -8> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) Dec 05 15:23:17 node01 bash[1969]: debug -7> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 10 log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) Dec 05 15:23:17 node01 bash[1969]: debug -6> 2023-12-05T13:23:17.723+0000 7fab0e25bb80 1 mon.node01@-1(???) e0 preinit fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37 Dec 05 15:23:17 node01 bash[1969]: debug -5> 2023-12-05T13:23:17.723+0000 7fab0e25bb80 5 mon.node01@-1(???).mds e0 Unable to load 'last_metadata' Dec 05 15:23:17 node01 bash[1969]: debug -4> 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd e9898 crush map has features 3314933000854323200, adjusting msgr requires Dec 05 15:23:17 node01 bash[1969]: debug -3> 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd e9898 crush map has features 432629239337189376, adjusting msgr requires Dec 05 15:23:17 node01 bash[1969]: debug -2> 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd e9898 crush map has features 432629239337189376, adjusting msgr requires Dec 05 15:23:17 node01 bash[1969]: debug -1> 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd e9898 crush map has features 432629239337189376, adjusting msgr requires Dec 05 15:23:17 node01 bash[1969]: debug 0> 2023-12-05T13:23:17.735+0000 7fab0e25bb80 -1 *** Caught signal (Aborted) ** Dec 05 15:23:17 node01 bash[1969]: in thread 7fab0e25bb80 thread_name:ceph-mon Dec 05 15:23:17 node01 bash[1969]: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) Dec 05 15:23:17 node01 bash[1969]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fab0b6c8cf0] Dec 05 15:23:17 node01 bash[1969]: 2: gsignal() Dec 05 15:23:17 node01 bash[1969]: 3: abort() Dec 05 15:23:17 node01 bash[1969]: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7fab0acda09b] Dec 05 15:23:17 node01 bash[1969]: 5: /lib64/libstdc++.so.6(+0x9654c) [0x7fab0ace054c] Dec 05 15:23:17 node01 bash[1969]: 6: /lib64/libstdc++.so.6(+0x965a7) [0x7fab0ace05a7] Dec 05 15:23:17 node01 bash[1969]: 7: /lib64/libstdc++.so.6(+0x96808) [0x7fab0ace0808] Dec 05 15:23:17 node01 bash[1969]: 8: /lib64/libstdc++.so.6(+0x91ff9) [0x7fab0acdbff9] Dec 05 15:23:17 node01 bash[1969]: 9: (LogMonitor::log_external_backlog()+0xe70) [0x55be1dc27190] Dec 05 15:23:17 node01 bash[1969]: 10: (LogMonitor::update_from_paxos(bool*)+0x54) [0x55be1dc27334] Dec 05 15:23:17 node01 bash[1969]: 11: (Monitor::refresh_from_paxos(bool*)+0x104) [0x55be1db9ae74] Dec 05 15:23:17 node01 bash[1969]: 12: (Monitor::preinit()+0x95d) [0x55be1dbc93ad] Dec 05 15:23:17 node01 bash[1969]: 13: main() Dec 05 15:23:17 node01 bash[1969]: 14: __libc_start_main() Dec 05 15:23:17 node01 bash[1969]: 15: _start() Dec 05 15:23:17 node01 bash[1969]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Dec 05 15:23:17 node01 bash[1969]: --- logging levels --- Dec 05 15:23:17 node01 bash[1969]: 0/ 5 none Dec 05 15:23:17 node01 bash[1969]: 0/ 1 lockdep Dec 05 15:23:17 node01 bash[1969]: 0/ 1 context Dec 05 15:23:17 node01 bash[1969]: 1/ 1 crush Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_balancer Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_locker Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_log Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_log_expire Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_migrator Dec 05 15:23:17 node01 bash[1969]: 0/ 1 buffer Dec 05 15:23:17 node01 bash[1969]: 0/ 1 timer Dec 05 15:23:17 node01 bash[1969]: 0/ 1 filer Dec 05 15:23:17 node01 bash[1969]: 0/ 1 striper Dec 05 15:23:17 node01 bash[1969]: 0/ 1 objecter Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rados Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_mirror Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_replay Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_pwl Dec 05 15:23:17 node01 bash[1969]: 0/ 5 journaler Dec 05 15:23:17 node01 bash[1969]: 0/ 5 objectcacher Dec 05 15:23:17 node01 bash[1969]: 0/ 5 immutable_obj_cache Dec 05 15:23:17 node01 bash[1969]: 0/ 5 client Dec 05 15:23:17 node01 bash[1969]: 1/ 5 osd Dec 05 15:23:17 node01 bash[1969]: 0/ 5 optracker Dec 05 15:23:17 node01 bash[1969]: 0/ 5 objclass Dec 05 15:23:17 node01 bash[1969]: 1/ 3 filestore Dec 05 15:23:17 node01 bash[1969]: 1/ 3 journal Dec 05 15:23:17 node01 bash[1969]: 0/ 0 ms Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mon Dec 05 15:23:17 node01 bash[1969]: 0/10 monc Dec 05 15:23:17 node01 bash[1969]: 1/ 5 paxos Dec 05 15:23:17 node01 bash[1969]: 0/ 5 tp Dec 05 15:23:17 node01 bash[1969]: 1/ 5 auth Dec 05 15:23:17 node01 bash[1969]: 1/ 5 crypto Dec 05 15:23:17 node01 bash[1969]: 1/ 1 finisher Dec 05 15:23:17 node01 bash[1969]: 1/ 1 reserver Dec 05 15:23:17 node01 bash[1969]: 1/ 5 heartbeatmap Dec 05 15:23:17 node01 bash[1969]: 1/ 5 perfcounter Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw_sync Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw_datacache Dec 05 15:23:17 node01 bash[1969]: 1/10 civetweb Dec 05 15:23:17 node01 bash[1969]: 1/ 5 javaclient Dec 05 15:23:17 node01 bash[1969]: 1/ 5 asok Dec 05 15:23:17 node01 bash[1969]: 1/ 1 throttle Dec 05 15:23:17 node01 bash[1969]: 0/ 0 refs Dec 05 15:23:17 node01 bash[1969]: 1/ 5 compressor Dec 05 15:23:17 node01 bash[1969]: 1/ 5 bluestore Dec 05 15:23:17 node01 bash[1969]: 1/ 5 bluefs Dec 05 15:23:17 node01 bash[1969]: 1/ 3 bdev Dec 05 15:23:17 node01 bash[1969]: 1/ 5 kstore Dec 05 15:23:17 node01 bash[1969]: 4/ 5 rocksdb Dec 05 15:23:17 node01 bash[1969]: 4/ 5 leveldb Dec 05 15:23:17 node01 bash[1969]: 4/ 5 memdb Dec 05 15:23:17 node01 bash[1969]: 1/ 5 fuse Dec 05 15:23:17 node01 bash[1969]: 2/ 5 mgr Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mgrc Dec 05 15:23:17 node01 bash[1969]: 1/ 5 dpdk Dec 05 15:23:17 node01 bash[1969]: 1/ 5 eventtrace Dec 05 15:23:17 node01 bash[1969]: 1/ 5 prioritycache Dec 05 15:23:17 node01 bash[1969]: 0/ 5 test Dec 05 15:23:17 node01 bash[1969]: 0/ 5 cephfs_mirror Dec 05 15:23:17 node01 bash[1969]: 0/ 5 cephsqlite Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_onode Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_odata Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_omap Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_tm Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_cleaner Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_lba Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_cache Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_journal Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_device Dec 05 15:23:17 node01 bash[1969]: 0/ 5 alienstore Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mclock Dec 05 15:23:17 node01 bash[1969]: 1/ 5 ceph_exporter Dec 05 15:23:17 node01 bash[1969]: -2/-2 (syslog threshold) Dec 05 15:23:17 node01 bash[1969]: 99/99 (stderr threshold) Dec 05 15:23:17 node01 bash[1969]: --- pthread ID / name mapping for recent threads --- Dec 05 15:23:17 node01 bash[1969]: 7faafe4ad700 / ceph-mon Dec 05 15:23:17 node01 bash[1969]: 7fab076ba700 / admin_socket Dec 05 15:23:17 node01 bash[1969]: 7fab0e25bb80 / ceph-mon Dec 05 15:23:17 node01 bash[1969]: max_recent 10000 Dec 05 15:23:17 node01 bash[1969]: max_new 10000 Dec 05 15:23:17 node01 bash[1969]: log_file Dec 05 15:23:17 node01 bash[1969]: --- end dump of recent events --- Dec 05 15:23:17 node01 bash[1969]: --- begin dump of recent events --- Dec 05 15:23:17 node01 bash[1969]: debug -290> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command assert hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -289> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command abort hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -288> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command leak_some_memory hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -287> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command perfcounters_dump hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -286> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command 1 hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -285> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command perf dump hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -284> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command perfcounters_schema hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -283> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command perf histogram dump hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -282> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command 2 hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -281> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command perf schema hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -280> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command perf histogram schema hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -279> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command perf reset hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -278> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command config show hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -277> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command config help hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -276> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command config set hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -275> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command config unset hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -274> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command config get hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -273> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command config diff hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -272> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command config diff get hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -271> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command injectargs hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -270> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command log flush hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -269> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command log dump hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -268> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command log reopen hook 0x55be208de8b0 Dec 05 15:23:17 node01 bash[1969]: debug -267> 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command dump_mempools hook 0x55be2168a068 Dec 05 15:23:17 node01 bash[1969]: debug -266> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 set uid:gid to 167:167 (ceph:ceph) Dec 05 15:23:17 node01 bash[1969]: debug -265> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable), process ceph-mon, pid 7 Dec 05 15:23:17 node01 bash[1969]: debug -264> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 pidfile_write: ignore empty --pid-file Dec 05 15:23:17 node01 bash[1969]: debug -263> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) init /var/run/ceph/ceph-mon.node01.asok Dec 05 15:23:17 node01 bash[1969]: debug -262> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) bind_and_listen /var/run/ceph/ceph-mon.node01.asok Dec 05 15:23:17 node01 bash[1969]: debug -261> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command 0 hook 0x55be2096b030 Dec 05 15:23:17 node01 bash[1969]: debug -260> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command version hook 0x55be2096b030 Dec 05 15:23:17 node01 bash[1969]: debug -259> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command git_version hook 0x55be2096b030 Dec 05 15:23:17 node01 bash[1969]: debug -258> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command help hook 0x55be208de5b0 Dec 05 15:23:17 node01 bash[1969]: debug -257> 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) register_command get_command_descriptions hook 0x55be208de5a0 Dec 05 15:23:17 node01 bash[1969]: debug -256> 2023-12-05T13:23:17.639+0000 7fab076ba700 5 asok(0x55be20bc0000) entry start Dec 05 15:23:17 node01 bash[1969]: debug -255> 2023-12-05T13:23:17.667+0000 7fab0e25bb80 0 load: jerasure load: lrc Dec 05 15:23:17 node01 bash[1969]: debug -254> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option level_compaction_dynamic_level_bytes = true Dec 05 15:23:17 node01 bash[1969]: debug -253> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option write_buffer_size = 33554432 Dec 05 15:23:17 node01 bash[1969]: debug -252> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option compression = kNoCompression Dec 05 15:23:17 node01 bash[1969]: debug -251> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option level_compaction_dynamic_level_bytes = true Dec 05 15:23:17 node01 bash[1969]: debug -250> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option write_buffer_size = 33554432 Dec 05 15:23:17 node01 bash[1969]: debug -249> 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option compression = kNoCompression Dec 05 15:23:17 node01 bash[1969]: debug -248> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 5 rocksdb: verify_sharding column families from rocksdb: [default] Dec 05 15:23:17 node01 bash[1969]: debug -247> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: RocksDB version: 6.15.5 Dec 05 15:23:17 node01 bash[1969]: debug -246> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ Dec 05 15:23:17 node01 bash[1969]: debug -245> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compile date Oct 25 2023 Dec 05 15:23:17 node01 bash[1969]: debug -244> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB SUMMARY Dec 05 15:23:17 node01 bash[1969]: debug -243> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB Session ID: 69YYUY1IJK4BA9QQDDK8 Dec 05 15:23:17 node01 bash[1969]: debug -242> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: CURRENT file: CURRENT Dec 05 15:23:17 node01 bash[1969]: debug -241> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: IDENTITY file: IDENTITY Dec 05 15:23:17 node01 bash[1969]: debug -240> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: MANIFEST file: MANIFEST-000226 size: 308 Bytes Dec 05 15:23:17 node01 bash[1969]: debug -239> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-node01/store.db dir, Total Num: 3, files: 000091.sst 000094.sst 000100.sst Dec 05 15:23:17 node01 bash[1969]: debug -238> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-node01/store.db: 000227.log size: 0 ; Dec 05 15:23:17 node01 bash[1969]: debug -237> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.error_if_exists: 0 Dec 05 15:23:17 node01 bash[1969]: debug -236> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.create_if_missing: 0 Dec 05 15:23:17 node01 bash[1969]: debug -235> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.paranoid_checks: 1 Dec 05 15:23:17 node01 bash[1969]: debug -234> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 Dec 05 15:23:17 node01 bash[1969]: debug -233> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.env: 0x55be1e3aea80 Dec 05 15:23:17 node01 bash[1969]: debug -232> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.fs: Posix File System Dec 05 15:23:17 node01 bash[1969]: debug -231> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.info_log: 0x55be20918360 Dec 05 15:23:17 node01 bash[1969]: debug -230> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_file_opening_threads: 16 Dec 05 15:23:17 node01 bash[1969]: debug -229> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.statistics: (nil) Dec 05 15:23:17 node01 bash[1969]: debug -228> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.use_fsync: 0 Dec 05 15:23:17 node01 bash[1969]: debug -227> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_log_file_size: 0 Dec 05 15:23:17 node01 bash[1969]: debug -226> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_manifest_file_size: 1073741824 Dec 05 15:23:17 node01 bash[1969]: debug -225> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.log_file_time_to_roll: 0 Dec 05 15:23:17 node01 bash[1969]: debug -224> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.keep_log_file_num: 1000 Dec 05 15:23:17 node01 bash[1969]: debug -223> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.recycle_log_file_num: 0 Dec 05 15:23:17 node01 bash[1969]: debug -222> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.allow_fallocate: 1 Dec 05 15:23:17 node01 bash[1969]: debug -221> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.allow_mmap_reads: 0 Dec 05 15:23:17 node01 bash[1969]: debug -220> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.allow_mmap_writes: 0 Dec 05 15:23:17 node01 bash[1969]: debug -219> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.use_direct_reads: 0 Dec 05 15:23:17 node01 bash[1969]: debug -218> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 Dec 05 15:23:17 node01 bash[1969]: debug -217> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.create_missing_column_families: 0 Dec 05 15:23:17 node01 bash[1969]: debug -216> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.db_log_dir: Dec 05 15:23:17 node01 bash[1969]: debug -215> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-node01/store.db Dec 05 15:23:17 node01 bash[1969]: debug -214> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.table_cache_numshardbits: 6 Dec 05 15:23:17 node01 bash[1969]: debug -213> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.WAL_ttl_seconds: 0 Dec 05 15:23:17 node01 bash[1969]: debug -212> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.WAL_size_limit_MB: 0 Dec 05 15:23:17 node01 bash[1969]: debug -211> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 Dec 05 15:23:17 node01 bash[1969]: debug -210> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.manifest_preallocation_size: 4194304 Dec 05 15:23:17 node01 bash[1969]: debug -209> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.is_fd_close_on_exec: 1 Dec 05 15:23:17 node01 bash[1969]: debug -208> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.advise_random_on_open: 1 Dec 05 15:23:17 node01 bash[1969]: debug -207> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.db_write_buffer_size: 0 Dec 05 15:23:17 node01 bash[1969]: debug -206> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.write_buffer_manager: 0x55be2168c990 Dec 05 15:23:17 node01 bash[1969]: debug -205> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.access_hint_on_compaction_start: 1 Dec 05 15:23:17 node01 bash[1969]: debug -204> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 Dec 05 15:23:17 node01 bash[1969]: debug -203> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 Dec 05 15:23:17 node01 bash[1969]: debug -202> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.use_adaptive_mutex: 0 Dec 05 15:23:17 node01 bash[1969]: debug -201> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.rate_limiter: (nil) Dec 05 15:23:17 node01 bash[1969]: debug -200> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 Dec 05 15:23:17 node01 bash[1969]: debug -199> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.wal_recovery_mode: 2 Dec 05 15:23:17 node01 bash[1969]: debug -198> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.enable_thread_tracking: 0 Dec 05 15:23:17 node01 bash[1969]: debug -197> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.enable_pipelined_write: 0 Dec 05 15:23:17 node01 bash[1969]: debug -196> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.unordered_write: 0 Dec 05 15:23:17 node01 bash[1969]: debug -195> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 Dec 05 15:23:17 node01 bash[1969]: debug -194> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 Dec 05 15:23:17 node01 bash[1969]: debug -193> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.write_thread_max_yield_usec: 100 Dec 05 15:23:17 node01 bash[1969]: debug -192> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 Dec 05 15:23:17 node01 bash[1969]: debug -191> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.row_cache: None Dec 05 15:23:17 node01 bash[1969]: debug -190> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.wal_filter: None Dec 05 15:23:17 node01 bash[1969]: debug -189> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.avoid_flush_during_recovery: 0 Dec 05 15:23:17 node01 bash[1969]: debug -188> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.allow_ingest_behind: 0 Dec 05 15:23:17 node01 bash[1969]: debug -187> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.preserve_deletes: 0 Dec 05 15:23:17 node01 bash[1969]: debug -186> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.two_write_queues: 0 Dec 05 15:23:17 node01 bash[1969]: debug -185> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.manual_wal_flush: 0 Dec 05 15:23:17 node01 bash[1969]: debug -184> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.atomic_flush: 0 Dec 05 15:23:17 node01 bash[1969]: debug -183> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 Dec 05 15:23:17 node01 bash[1969]: debug -182> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.persist_stats_to_disk: 0 Dec 05 15:23:17 node01 bash[1969]: debug -181> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.write_dbid_to_manifest: 0 Dec 05 15:23:17 node01 bash[1969]: debug -180> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.log_readahead_size: 0 Dec 05 15:23:17 node01 bash[1969]: debug -179> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.file_checksum_gen_factory: Unknown Dec 05 15:23:17 node01 bash[1969]: debug -178> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.best_efforts_recovery: 0 Dec 05 15:23:17 node01 bash[1969]: debug -177> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 Dec 05 15:23:17 node01 bash[1969]: debug -176> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 Dec 05 15:23:17 node01 bash[1969]: debug -175> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.allow_data_in_errors: 0 Dec 05 15:23:17 node01 bash[1969]: debug -174> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.db_host_id: __hostname__ Dec 05 15:23:17 node01 bash[1969]: debug -173> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_background_jobs: 2 Dec 05 15:23:17 node01 bash[1969]: debug -172> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_background_compactions: -1 Dec 05 15:23:17 node01 bash[1969]: debug -171> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_subcompactions: 1 Dec 05 15:23:17 node01 bash[1969]: debug -170> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 Dec 05 15:23:17 node01 bash[1969]: debug -169> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 Dec 05 15:23:17 node01 bash[1969]: debug -168> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.delayed_write_rate : 16777216 Dec 05 15:23:17 node01 bash[1969]: debug -167> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_total_wal_size: 0 Dec 05 15:23:17 node01 bash[1969]: debug -166> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 Dec 05 15:23:17 node01 bash[1969]: debug -165> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.stats_dump_period_sec: 600 Dec 05 15:23:17 node01 bash[1969]: debug -164> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.stats_persist_period_sec: 600 Dec 05 15:23:17 node01 bash[1969]: debug -163> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.stats_history_buffer_size: 1048576 Dec 05 15:23:17 node01 bash[1969]: debug -162> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_open_files: -1 Dec 05 15:23:17 node01 bash[1969]: debug -161> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bytes_per_sync: 0 Dec 05 15:23:17 node01 bash[1969]: debug -160> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.wal_bytes_per_sync: 0 Dec 05 15:23:17 node01 bash[1969]: debug -159> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.strict_bytes_per_sync: 0 Dec 05 15:23:17 node01 bash[1969]: debug -158> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_readahead_size: 0 Dec 05 15:23:17 node01 bash[1969]: debug -157> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_background_flushes: -1 Dec 05 15:23:17 node01 bash[1969]: debug -156> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compression algorithms supported: Dec 05 15:23:17 node01 bash[1969]: debug -155> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kZSTDNotFinalCompression supported: 0 Dec 05 15:23:17 node01 bash[1969]: debug -154> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kZSTD supported: 0 Dec 05 15:23:17 node01 bash[1969]: debug -153> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kXpressCompression supported: 0 Dec 05 15:23:17 node01 bash[1969]: debug -152> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kLZ4HCCompression supported: 1 Dec 05 15:23:17 node01 bash[1969]: debug -151> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kZlibCompression supported: 1 Dec 05 15:23:17 node01 bash[1969]: debug -150> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kSnappyCompression supported: 1 Dec 05 15:23:17 node01 bash[1969]: debug -149> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kLZ4Compression supported: 1 Dec 05 15:23:17 node01 bash[1969]: debug -148> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kBZip2Compression supported: 0 Dec 05 15:23:17 node01 bash[1969]: debug -147> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Fast CRC32 supported: Supported on x86 Dec 05 15:23:17 node01 bash[1969]: debug -146> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 Dec 05 15:23:17 node01 bash[1969]: debug -145> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: Dec 05 15:23:17 node01 bash[1969]: debug -144> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator Dec 05 15:23:17 node01 bash[1969]: debug -143> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.merge_operator: Dec 05 15:23:17 node01 bash[1969]: debug -142> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_filter: None Dec 05 15:23:17 node01 bash[1969]: debug -141> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_filter_factory: None Dec 05 15:23:17 node01 bash[1969]: debug -140> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.sst_partitioner_factory: None Dec 05 15:23:17 node01 bash[1969]: debug -139> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.memtable_factory: SkipListFactory Dec 05 15:23:17 node01 bash[1969]: debug -138> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.table_factory: BlockBasedTable Dec 05 15:23:17 node01 bash[1969]: debug -137> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55be2096afb0) Dec 05 15:23:17 node01 bash[1969]: cache_index_and_filter_blocks: 1 Dec 05 15:23:17 node01 bash[1969]: cache_index_and_filter_blocks_with_high_priority: 0 Dec 05 15:23:17 node01 bash[1969]: pin_l0_filter_and_index_blocks_in_cache: 0 Dec 05 15:23:17 node01 bash[1969]: pin_top_level_index_and_filter: 1 Dec 05 15:23:17 node01 bash[1969]: index_type: 0 Dec 05 15:23:17 node01 bash[1969]: data_block_index_type: 0 Dec 05 15:23:17 node01 bash[1969]: index_shortening: 1 Dec 05 15:23:17 node01 bash[1969]: data_block_hash_table_util_ratio: 0.750000 Dec 05 15:23:17 node01 bash[1969]: hash_index_allow_collision: 1 Dec 05 15:23:17 node01 bash[1969]: checksum: 1 Dec 05 15:23:17 node01 bash[1969]: no_block_cache: 0 Dec 05 15:23:17 node01 bash[1969]: block_cache: 0x55be2099d090 Dec 05 15:23:17 node01 bash[1969]: block_cache_name: BinnedLRUCache Dec 05 15:23:17 node01 bash[1969]: block_cache_options: Dec 05 15:23:17 node01 bash[1969]: capacity : 536870912 Dec 05 15:23:17 node01 bash[1969]: num_shard_bits : 4 Dec 05 15:23:17 node01 bash[1969]: strict_capacity_limit : 0 Dec 05 15:23:17 node01 bash[1969]: high_pri_pool_ratio: 0.000 Dec 05 15:23:17 node01 bash[1969]: block_cache_compressed: (nil) Dec 05 15:23:17 node01 bash[1969]: persistent_cache: (nil) Dec 05 15:23:17 node01 bash[1969]: block_size: 4096 Dec 05 15:23:17 node01 bash[1969]: block_size_deviation: 10 Dec 05 15:23:17 node01 bash[1969]: block_restart_interval: 16 Dec 05 15:23:17 node01 bash[1969]: index_block_restart_interval: 1 Dec 05 15:23:17 node01 bash[1969]: metadata_block_size: 4096 Dec 05 15:23:17 node01 bash[1969]: partition_filters: 0 Dec 05 15:23:17 node01 bash[1969]: use_delta_encoding: 1 Dec 05 15:23:17 node01 bash[1969]: filter_policy: rocksdb.BuiltinBloomFilter Dec 05 15:23:17 node01 bash[1969]: whole_key_filtering: 1 Dec 05 15:23:17 node01 bash[1969]: verify_compression: 0 Dec 05 15:23:17 node01 bash[1969]: read_amp_bytes_per_bit: 0 Dec 05 15:23:17 node01 bash[1969]: format_version: 4 Dec 05 15:23:17 node01 bash[1969]: enable_index_compression: 1 Dec 05 15:23:17 node01 bash[1969]: block_align: 0 Dec 05 15:23:17 node01 bash[1969]: debug -136> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.write_buffer_size: 33554432 Dec 05 15:23:17 node01 bash[1969]: debug -135> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_write_buffer_number: 2 Dec 05 15:23:17 node01 bash[1969]: debug -134> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression: NoCompression Dec 05 15:23:17 node01 bash[1969]: debug -133> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression: Disabled Dec 05 15:23:17 node01 bash[1969]: debug -132> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.prefix_extractor: nullptr Dec 05 15:23:17 node01 bash[1969]: debug -131> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr Dec 05 15:23:17 node01 bash[1969]: debug -130> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.num_levels: 7 Dec 05 15:23:17 node01 bash[1969]: debug -129> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 Dec 05 15:23:17 node01 bash[1969]: debug -128> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 Dec 05 15:23:17 node01 bash[1969]: debug -127> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 Dec 05 15:23:17 node01 bash[1969]: debug -126> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 Dec 05 15:23:17 node01 bash[1969]: debug -125> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 Dec 05 15:23:17 node01 bash[1969]: debug -124> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 Dec 05 15:23:17 node01 bash[1969]: debug -123> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 Dec 05 15:23:17 node01 bash[1969]: debug -122> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 Dec 05 15:23:17 node01 bash[1969]: debug -121> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 Dec 05 15:23:17 node01 bash[1969]: debug -120> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bottommost_compression_opts.enabled: false Dec 05 15:23:17 node01 bash[1969]: debug -119> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression_opts.window_bits: -14 Dec 05 15:23:17 node01 bash[1969]: debug -118> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression_opts.level: 32767 Dec 05 15:23:17 node01 bash[1969]: debug -117> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression_opts.strategy: 0 Dec 05 15:23:17 node01 bash[1969]: debug -116> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 Dec 05 15:23:17 node01 bash[1969]: debug -115> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 Dec 05 15:23:17 node01 bash[1969]: debug -114> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression_opts.parallel_threads: 1 Dec 05 15:23:17 node01 bash[1969]: debug -113> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compression_opts.enabled: false Dec 05 15:23:17 node01 bash[1969]: debug -112> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 Dec 05 15:23:17 node01 bash[1969]: debug -111> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 Dec 05 15:23:17 node01 bash[1969]: debug -110> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.level0_stop_writes_trigger: 36 Dec 05 15:23:17 node01 bash[1969]: debug -109> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.target_file_size_base: 67108864 Dec 05 15:23:17 node01 bash[1969]: debug -108> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.target_file_size_multiplier: 1 Dec 05 15:23:17 node01 bash[1969]: debug -107> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 Dec 05 15:23:17 node01 bash[1969]: debug -106> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 Dec 05 15:23:17 node01 bash[1969]: debug -105> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 Dec 05 15:23:17 node01 bash[1969]: debug -104> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 Dec 05 15:23:17 node01 bash[1969]: debug -103> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 Dec 05 15:23:17 node01 bash[1969]: debug -102> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 Dec 05 15:23:17 node01 bash[1969]: debug -101> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 Dec 05 15:23:17 node01 bash[1969]: debug -100> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 Dec 05 15:23:17 node01 bash[1969]: debug -99> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 Dec 05 15:23:17 node01 bash[1969]: debug -98> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 Dec 05 15:23:17 node01 bash[1969]: debug -97> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 Dec 05 15:23:17 node01 bash[1969]: debug -96> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_compaction_bytes: 1677721600 Dec 05 15:23:17 node01 bash[1969]: debug -95> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.arena_block_size: 4194304 Dec 05 15:23:17 node01 bash[1969]: debug -94> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 Dec 05 15:23:17 node01 bash[1969]: debug -93> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 Dec 05 15:23:17 node01 bash[1969]: debug -92> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 Dec 05 15:23:17 node01 bash[1969]: debug -91> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.disable_auto_compactions: 0 Dec 05 15:23:17 node01 bash[1969]: debug -90> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel Dec 05 15:23:17 node01 bash[1969]: debug -89> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio Dec 05 15:23:17 node01 bash[1969]: debug -88> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 Dec 05 15:23:17 node01 bash[1969]: debug -87> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 Dec 05 15:23:17 node01 bash[1969]: debug -86> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 Dec 05 15:23:17 node01 bash[1969]: debug -85> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 Dec 05 15:23:17 node01 bash[1969]: debug -84> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 Dec 05 15:23:17 node01 bash[1969]: debug -83> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize Dec 05 15:23:17 node01 bash[1969]: debug -82> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 Dec 05 15:23:17 node01 bash[1969]: debug -81> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 Dec 05 15:23:17 node01 bash[1969]: debug -80> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.table_properties_collectors: Dec 05 15:23:17 node01 bash[1969]: debug -79> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.inplace_update_support: 0 Dec 05 15:23:17 node01 bash[1969]: debug -78> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.inplace_update_num_locks: 10000 Dec 05 15:23:17 node01 bash[1969]: debug -77> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 Dec 05 15:23:17 node01 bash[1969]: debug -76> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.memtable_whole_key_filtering: 0 Dec 05 15:23:17 node01 bash[1969]: debug -75> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.memtable_huge_page_size: 0 Dec 05 15:23:17 node01 bash[1969]: debug -74> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.bloom_locality: 0 Dec 05 15:23:17 node01 bash[1969]: debug -73> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.max_successive_merges: 0 Dec 05 15:23:17 node01 bash[1969]: debug -72> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.optimize_filters_for_hits: 0 Dec 05 15:23:17 node01 bash[1969]: debug -71> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.paranoid_file_checks: 0 Dec 05 15:23:17 node01 bash[1969]: debug -70> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.force_consistency_checks: 1 Dec 05 15:23:17 node01 bash[1969]: debug -69> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.report_bg_io_stats: 0 Dec 05 15:23:17 node01 bash[1969]: debug -68> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.ttl: 2592000 Dec 05 15:23:17 node01 bash[1969]: debug -67> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.periodic_compaction_seconds: 0 Dec 05 15:23:17 node01 bash[1969]: debug -66> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.enable_blob_files: false Dec 05 15:23:17 node01 bash[1969]: debug -65> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.min_blob_size: 0 Dec 05 15:23:17 node01 bash[1969]: debug -64> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.blob_file_size: 268435456 Dec 05 15:23:17 node01 bash[1969]: debug -63> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.blob_compression_type: NoCompression Dec 05 15:23:17 node01 bash[1969]: debug -62> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.enable_blob_garbage_collection: false Dec 05 15:23:17 node01 bash[1969]: debug -61> 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 Dec 05 15:23:17 node01 bash[1969]: debug -60> 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:3457] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. Dec 05 15:23:17 node01 bash[1969]: debug -59> 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 succeeded,manifest_file_number is 226, next_file_number is 228, last_sequence is 21201, log_number is 223,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 Dec 05 15:23:17 node01 bash[1969]: debug -58> 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 223 Dec 05 15:23:17 node01 bash[1969]: debug -57> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:4082] Creating manifest 230 Dec 05 15:23:17 node01 bash[1969]: debug -56> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:3457] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. Dec 05 15:23:17 node01 bash[1969]: debug -55> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1701782597693048, "job": 1, "event": "recovery_started", "wal_files": [227]} Dec 05 15:23:17 node01 bash[1969]: debug -54> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #227 mode 2 Dec 05 15:23:17 node01 bash[1969]: debug -53> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:4082] Creating manifest 231 Dec 05 15:23:17 node01 bash[1969]: debug -52> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: [db/version_set.cc:3457] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. Dec 05 15:23:17 node01 bash[1969]: debug -51> 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1701782597693989, "job": 1, "event": "recovery_finished"} Dec 05 15:23:17 node01 bash[1969]: debug -50> 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-node01/store.db/000227.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 Dec 05 15:23:17 node01 bash[1969]: debug -49> 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55be20931180 Dec 05 15:23:17 node01 bash[1969]: debug -48> 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: DB pointer 0x55be20a06000 Dec 05 15:23:17 node01 bash[1969]: debug -47> 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- Dec 05 15:23:17 node01 bash[1969]: debug -46> 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: [db/db_impl/db_impl.cc:903] Dec 05 15:23:17 node01 bash[1969]: ** DB Stats ** Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval Dec 05 15:23:17 node01 bash[1969]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Dec 05 15:23:17 node01 bash[1969]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Dec 05 15:23:17 node01 bash[1969]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Dec 05 15:23:17 node01 bash[1969]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Dec 05 15:23:17 node01 bash[1969]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Dec 05 15:23:17 node01 bash[1969]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Dec 05 15:23:17 node01 bash[1969]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** Dec 05 15:23:17 node01 bash[1969]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Dec 05 15:23:17 node01 bash[1969]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, interval 0.000 Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, interval 0.000 Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative 0, interval 0 Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, interval 0 Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By Level [default] ** Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Dec 05 15:23:17 node01 bash[1969]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** Dec 05 15:23:17 node01 bash[1969]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Dec 05 15:23:17 node01 bash[1969]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, interval 0.000 Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, interval 0.000 Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative 0, interval 0 Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, interval 0 Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By Level [default] ** Dec 05 15:23:17 node01 bash[1969]: debug -45> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding auth protocol: cephx Dec 05 15:23:17 node01 bash[1969]: debug -44> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding auth protocol: cephx Dec 05 15:23:17 node01 bash[1969]: debug -43> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding auth protocol: cephx Dec 05 15:23:17 node01 bash[1969]: debug -42> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding auth protocol: none Dec 05 15:23:17 node01 bash[1969]: debug -41> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -40> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -39> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -38> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -37> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -36> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -35> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -34> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -33> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -32> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -31> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -30> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167a140) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -29> 2023-12-05T13:23:17.715+0000 7fab0e25bb80 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-node01/keyring Dec 05 15:23:17 node01 bash[1969]: debug -28> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 0 starting mon.node01 rank 0 at public addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] at bind addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] mon_data /var/lib/ceph/mon/ceph-node01 fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37 Dec 05 15:23:17 node01 bash[1969]: debug -27> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding auth protocol: cephx Dec 05 15:23:17 node01 bash[1969]: debug -26> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding auth protocol: cephx Dec 05 15:23:17 node01 bash[1969]: debug -25> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding auth protocol: cephx Dec 05 15:23:17 node01 bash[1969]: debug -24> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding auth protocol: none Dec 05 15:23:17 node01 bash[1969]: debug -23> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -22> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -21> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -20> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -19> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -18> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -17> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -16> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -15> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -14> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -13> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: crc Dec 05 15:23:17 node01 bash[1969]: debug -12> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 AuthRegistry(0x55be2167aa40) adding con mode: secure Dec 05 15:23:17 node01 bash[1969]: debug -11> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-node01/keyring Dec 05 15:23:17 node01 bash[1969]: debug -10> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 adding auth protocol: cephx Dec 05 15:23:17 node01 bash[1969]: debug -9> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 adding auth protocol: cephx Dec 05 15:23:17 node01 bash[1969]: debug -8> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) Dec 05 15:23:17 node01 bash[1969]: debug -7> 2023-12-05T13:23:17.719+0000 7fab0e25bb80 10 log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) Dec 05 15:23:17 node01 bash[1969]: debug -6> 2023-12-05T13:23:17.723+0000 7fab0e25bb80 1 mon.node01@-1(???) e0 preinit fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37 Dec 05 15:23:17 node01 bash[1969]: debug -5> 2023-12-05T13:23:17.723+0000 7fab0e25bb80 5 mon.node01@-1(???).mds e0 Unable to load 'last_metadata' Dec 05 15:23:17 node01 bash[1969]: debug -4> 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd e9898 crush map has features 3314933000854323200, adjusting msgr requires Dec 05 15:23:17 node01 bash[1969]: debug -3> 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd e9898 crush map has features 432629239337189376, adjusting msgr requires Dec 05 15:23:17 node01 bash[1969]: debug -2> 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd e9898 crush map has features 432629239337189376, adjusting msgr requires Dec 05 15:23:17 node01 bash[1969]: debug -1> 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01@-1(???).osd e9898 crush map has features 432629239337189376, adjusting msgr requires Dec 05 15:23:17 node01 bash[1969]: debug 0> 2023-12-05T13:23:17.735+0000 7fab0e25bb80 -1 *** Caught signal (Aborted) ** Dec 05 15:23:17 node01 bash[1969]: in thread 7fab0e25bb80 thread_name:ceph-mon Dec 05 15:23:17 node01 bash[1969]: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) Dec 05 15:23:17 node01 bash[1969]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fab0b6c8cf0] Dec 05 15:23:17 node01 bash[1969]: 2: gsignal() Dec 05 15:23:17 node01 bash[1969]: 3: abort() Dec 05 15:23:17 node01 bash[1969]: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7fab0acda09b] Dec 05 15:23:17 node01 bash[1969]: 5: /lib64/libstdc++.so.6(+0x9654c) [0x7fab0ace054c] Dec 05 15:23:17 node01 bash[1969]: 6: /lib64/libstdc++.so.6(+0x965a7) [0x7fab0ace05a7] Dec 05 15:23:17 node01 bash[1969]: 7: /lib64/libstdc++.so.6(+0x96808) [0x7fab0ace0808] Dec 05 15:23:17 node01 bash[1969]: 8: /lib64/libstdc++.so.6(+0x91ff9) [0x7fab0acdbff9] Dec 05 15:23:17 node01 bash[1969]: 9: (LogMonitor::log_external_backlog()+0xe70) [0x55be1dc27190] Dec 05 15:23:17 node01 bash[1969]: 10: (LogMonitor::update_from_paxos(bool*)+0x54) [0x55be1dc27334] Dec 05 15:23:17 node01 bash[1969]: 11: (Monitor::refresh_from_paxos(bool*)+0x104) [0x55be1db9ae74] Dec 05 15:23:17 node01 bash[1969]: 12: (Monitor::preinit()+0x95d) [0x55be1dbc93ad] Dec 05 15:23:17 node01 bash[1969]: 13: main() Dec 05 15:23:17 node01 bash[1969]: 14: __libc_start_main() Dec 05 15:23:17 node01 bash[1969]: 15: _start() Dec 05 15:23:17 node01 bash[1969]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Dec 05 15:23:17 node01 bash[1969]: --- logging levels --- Dec 05 15:23:17 node01 bash[1969]: 0/ 5 none Dec 05 15:23:17 node01 bash[1969]: 0/ 1 lockdep Dec 05 15:23:17 node01 bash[1969]: 0/ 1 context Dec 05 15:23:17 node01 bash[1969]: 1/ 1 crush Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_balancer Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_locker Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_log Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_log_expire Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_migrator Dec 05 15:23:17 node01 bash[1969]: 0/ 1 buffer Dec 05 15:23:17 node01 bash[1969]: 0/ 1 timer Dec 05 15:23:17 node01 bash[1969]: 0/ 1 filer Dec 05 15:23:17 node01 bash[1969]: 0/ 1 striper Dec 05 15:23:17 node01 bash[1969]: 0/ 1 objecter Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rados Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_mirror Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_replay Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_pwl Dec 05 15:23:17 node01 bash[1969]: 0/ 5 journaler Dec 05 15:23:17 node01 bash[1969]: 0/ 5 objectcacher Dec 05 15:23:17 node01 bash[1969]: 0/ 5 immutable_obj_cache Dec 05 15:23:17 node01 bash[1969]: 0/ 5 client Dec 05 15:23:17 node01 bash[1969]: 1/ 5 osd Dec 05 15:23:17 node01 bash[1969]: 0/ 5 optracker Dec 05 15:23:17 node01 bash[1969]: 0/ 5 objclass Dec 05 15:23:17 node01 bash[1969]: 1/ 3 filestore Dec 05 15:23:17 node01 bash[1969]: 1/ 3 journal Dec 05 15:23:17 node01 bash[1969]: 0/ 0 ms Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mon Dec 05 15:23:17 node01 bash[1969]: 0/10 monc Dec 05 15:23:17 node01 bash[1969]: 1/ 5 paxos Dec 05 15:23:17 node01 bash[1969]: 0/ 5 tp Dec 05 15:23:17 node01 bash[1969]: 1/ 5 auth Dec 05 15:23:17 node01 bash[1969]: 1/ 5 crypto Dec 05 15:23:17 node01 bash[1969]: 1/ 1 finisher Dec 05 15:23:17 node01 bash[1969]: 1/ 1 reserver Dec 05 15:23:17 node01 bash[1969]: 1/ 5 heartbeatmap Dec 05 15:23:17 node01 bash[1969]: 1/ 5 perfcounter Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw_sync Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw_datacache Dec 05 15:23:17 node01 bash[1969]: 1/10 civetweb Dec 05 15:23:17 node01 bash[1969]: 1/ 5 javaclient Dec 05 15:23:17 node01 bash[1969]: 1/ 5 asok Dec 05 15:23:17 node01 bash[1969]: 1/ 1 throttle Dec 05 15:23:17 node01 bash[1969]: 0/ 0 refs Dec 05 15:23:17 node01 bash[1969]: 1/ 5 compressor Dec 05 15:23:17 node01 bash[1969]: 1/ 5 bluestore Dec 05 15:23:17 node01 bash[1969]: 1/ 5 bluefs Dec 05 15:23:17 node01 bash[1969]: 1/ 3 bdev Dec 05 15:23:17 node01 bash[1969]: 1/ 5 kstore Dec 05 15:23:17 node01 bash[1969]: 4/ 5 rocksdb Dec 05 15:23:17 node01 bash[1969]: 4/ 5 leveldb Dec 05 15:23:17 node01 bash[1969]: 4/ 5 memdb Dec 05 15:23:17 node01 bash[1969]: 1/ 5 fuse Dec 05 15:23:17 node01 bash[1969]: 2/ 5 mgr Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mgrc Dec 05 15:23:17 node01 bash[1969]: 1/ 5 dpdk Dec 05 15:23:17 node01 bash[1969]: 1/ 5 eventtrace Dec 05 15:23:17 node01 bash[1969]: 1/ 5 prioritycache Dec 05 15:23:17 node01 bash[1969]: 0/ 5 test Dec 05 15:23:17 node01 bash[1969]: 0/ 5 cephfs_mirror Dec 05 15:23:17 node01 bash[1969]: 0/ 5 cephsqlite Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_onode Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_odata Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_omap Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_tm Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_cleaner Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_lba Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_cache Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_journal Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_device Dec 05 15:23:17 node01 bash[1969]: 0/ 5 alienstore Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mclock Dec 05 15:23:17 node01 bash[1969]: 1/ 5 ceph_exporter Dec 05 15:23:17 node01 bash[1969]: -2/-2 (syslog threshold) Dec 05 15:23:17 node01 bash[1969]: 99/99 (stderr threshold) Dec 05 15:23:17 node01 bash[1969]: --- pthread ID / name mapping for recent threads --- Dec 05 15:23:17 node01 bash[1969]: 7faafe4ad700 / ceph-mon Dec 05 15:23:17 node01 bash[1969]: 7fab076ba700 / admin_socket Dec 05 15:23:17 node01 bash[1969]: 7fab0e25bb80 / ceph-mon Dec 05 15:23:17 node01 bash[1969]: max_recent 10000 Dec 05 15:23:17 node01 bash[1969]: max_new 10000 Dec 05 15:23:17 node01 bash[1969]: log_file /var/lib/ceph/crash/2023-12-05T13:23:17.739483Z_174a5e49-22df-4c2f-be14-4e88cc65b2f2/log Dec 05 15:23:17 node01 bash[1969]: --- end dump of recent events --- Dec 05 15:23:17 node01 systemd[1]: ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37@mon.node01.service: Main process exited, code=exited, status=134/n/a Dec 05 15:23:18 node01 systemd[1]: ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37@mon.node01.service: Failed with result 'exit-code'. Dec 05 15:23:29 node01 systemd[1]: ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37@mon.node01.service: Scheduled restart job, restart counter is at 1. Dec 05 15:23:29 node01 systemd[1]: Stopped Ceph mon.node01 for be4304e4-b0d5-11ec-8c6a-2965d4229f37. Dec 05 15:23:29 node01 systemd[1]: Started Ceph mon.node01 for be4304e4-b0d5-11ec-8c6a-2965d4229f37. Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.299+0000 7fd6a1f76b80 0 set uid:gid to 167:167 (ceph:ceph) Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.299+0000 7fd6a1f76b80 0 ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable), process ceph-mon, pid 7 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.299+0000 7fd6a1f76b80 0 pidfile_write: ignore empty --pid-file Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 0 load: jerasure load: lrc Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: RocksDB version: 6.15.5 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Compile date Oct 25 2023 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: DB SUMMARY Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: DB Session ID: OQS561SAY9M8P46I122B Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: CURRENT file: CURRENT Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: IDENTITY file: IDENTITY Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: MANIFEST file: MANIFEST-000231 size: 308 Bytes Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-node01/store.db dir, Total Num: 3, files: 000091.sst 000094.sst 000100.sst Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-node01/store.db: 000232.log size: 0 ; Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.error_if_exists: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.create_if_missing: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.paranoid_checks: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.env: 0x562241625a80 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.fs: Posix File System Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.info_log: 0x562243b9a360 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_file_opening_threads: 16 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.statistics: (nil) Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.use_fsync: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_log_file_size: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_manifest_file_size: 1073741824 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.log_file_time_to_roll: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.keep_log_file_num: 1000 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.recycle_log_file_num: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.allow_fallocate: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.allow_mmap_reads: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.allow_mmap_writes: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.use_direct_reads: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.create_missing_column_families: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.db_log_dir: Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-node01/store.db Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.table_cache_numshardbits: 6 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.WAL_ttl_seconds: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.WAL_size_limit_MB: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.manifest_preallocation_size: 4194304 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.is_fd_close_on_exec: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.advise_random_on_open: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.db_write_buffer_size: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.write_buffer_manager: 0x56224490e990 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.access_hint_on_compaction_start: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.use_adaptive_mutex: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.rate_limiter: (nil) Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.wal_recovery_mode: 2 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.enable_thread_tracking: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.enable_pipelined_write: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.unordered_write: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.write_thread_max_yield_usec: 100 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.row_cache: None Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.wal_filter: None Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.avoid_flush_during_recovery: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.allow_ingest_behind: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.preserve_deletes: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.two_write_queues: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.manual_wal_flush: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.atomic_flush: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.persist_stats_to_disk: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.write_dbid_to_manifest: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.log_readahead_size: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.file_checksum_gen_factory: Unknown Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.best_efforts_recovery: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.allow_data_in_errors: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.db_host_id: __hostname__ Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_background_jobs: 2 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_background_compactions: -1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_subcompactions: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.delayed_write_rate : 16777216 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_total_wal_size: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.stats_dump_period_sec: 600 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.stats_persist_period_sec: 600 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.stats_history_buffer_size: 1048576 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_open_files: -1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.bytes_per_sync: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.wal_bytes_per_sync: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.strict_bytes_per_sync: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.compaction_readahead_size: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_background_flushes: -1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Compression algorithms supported: Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: kZSTDNotFinalCompression supported: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: kZSTD supported: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: kXpressCompression supported: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: kLZ4HCCompression supported: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: kZlibCompression supported: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: kSnappyCompression supported: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: kLZ4Compression supported: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: kBZip2Compression supported: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Fast CRC32 supported: Supported on x86 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000231 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.merge_operator: Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.compaction_filter: None Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.compaction_filter_factory: None Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.sst_partitioner_factory: None Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.memtable_factory: SkipListFactory Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.table_factory: BlockBasedTable Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562243becfb0) Dec 05 15:23:29 node01 bash[6375]: cache_index_and_filter_blocks: 1 Dec 05 15:23:29 node01 bash[6375]: cache_index_and_filter_blocks_with_high_priority: 0 Dec 05 15:23:29 node01 bash[6375]: pin_l0_filter_and_index_blocks_in_cache: 0 Dec 05 15:23:29 node01 bash[6375]: pin_top_level_index_and_filter: 1 Dec 05 15:23:29 node01 bash[6375]: index_type: 0 Dec 05 15:23:29 node01 bash[6375]: data_block_index_type: 0 Dec 05 15:23:29 node01 bash[6375]: index_shortening: 1 Dec 05 15:23:29 node01 bash[6375]: data_block_hash_table_util_ratio: 0.750000 Dec 05 15:23:29 node01 bash[6375]: hash_index_allow_collision: 1 Dec 05 15:23:29 node01 bash[6375]: checksum: 1 Dec 05 15:23:29 node01 bash[6375]: no_block_cache: 0 Dec 05 15:23:29 node01 bash[6375]: block_cache: 0x562243c1f090 Dec 05 15:23:29 node01 bash[6375]: block_cache_name: BinnedLRUCache Dec 05 15:23:29 node01 bash[6375]: block_cache_options: Dec 05 15:23:29 node01 bash[6375]: capacity : 536870912 Dec 05 15:23:29 node01 bash[6375]: num_shard_bits : 4 Dec 05 15:23:29 node01 bash[6375]: strict_capacity_limit : 0 Dec 05 15:23:29 node01 bash[6375]: high_pri_pool_ratio: 0.000 Dec 05 15:23:29 node01 bash[6375]: block_cache_compressed: (nil) Dec 05 15:23:29 node01 bash[6375]: persistent_cache: (nil) Dec 05 15:23:29 node01 bash[6375]: block_size: 4096 Dec 05 15:23:29 node01 bash[6375]: block_size_deviation: 10 Dec 05 15:23:29 node01 bash[6375]: block_restart_interval: 16 Dec 05 15:23:29 node01 bash[6375]: index_block_restart_interval: 1 Dec 05 15:23:29 node01 bash[6375]: metadata_block_size: 4096 Dec 05 15:23:29 node01 bash[6375]: partition_filters: 0 Dec 05 15:23:29 node01 bash[6375]: use_delta_encoding: 1 Dec 05 15:23:29 node01 bash[6375]: filter_policy: rocksdb.BuiltinBloomFilter Dec 05 15:23:29 node01 bash[6375]: whole_key_filtering: 1 Dec 05 15:23:29 node01 bash[6375]: verify_compression: 0 Dec 05 15:23:29 node01 bash[6375]: read_amp_bytes_per_bit: 0 Dec 05 15:23:29 node01 bash[6375]: format_version: 4 Dec 05 15:23:29 node01 bash[6375]: enable_index_compression: 1 Dec 05 15:23:29 node01 bash[6375]: block_align: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.write_buffer_size: 33554432 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_write_buffer_number: 2 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.compression: NoCompression Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.bottommost_compression: Disabled Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.prefix_extractor: nullptr Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.num_levels: 7 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.bottommost_compression_opts.enabled: false Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.compression_opts.window_bits: -14 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.compression_opts.level: 32767 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.compression_opts.strategy: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.compression_opts.parallel_threads: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.compression_opts.enabled: false Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.level0_stop_writes_trigger: 36 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.target_file_size_base: 67108864 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.target_file_size_multiplier: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 Dec 05 15:23:29 node01 bash[6375]: debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Options.max_ [Text gekürzt]