November 2023 - ceph-users

Space reclaim doesn't happening in nautilus RBD pool

by Szabo, Istvan (Agoda)

Hi, Is there any config on Ceph that block/not perform space reclaim? I test on one pool which has only one image 1.8 TiB in used. rbd $p du im/root warning: fast-diff map is not enabled for root. operation may be slow. NAME PROVISIONED USED root 2.2 TiB 1.8 TiB I already removed all snaphots and now pool has only one image alone. I run both fstrim over the filesystem (XFS) and try rbd sparsify im/root (don't know what it is exactly but it mentions to reclaim something) It still shows the pool used 6.9 TiB which totally not make sense right? It should be up to 3.6 (1.8 * 2) according to its replica? POOLS: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR im 19 32 3.5 TiB 918.34k 6.9 TiB 4.80 69 TiB N/A 10 TiB 918.34k 0 B 0 B I think now some of others pool have this issue too, we do clean up a lot but seems space not reclaimed. I estimate more than 50 TiB should be able to reclaim, actual usage of this cluster much less than current reported number. Thank you for your help. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 months

2
4
0 0

CephFS pool not releasing space after data deletion

by Kuhring, Mathias

Dear Ceph users, Our CephFS is not releasing/freeing up space after deleting hundreds of terabytes of data. By now, this drives us in a "nearfull" osd/pool situation and thus throttles IO. We are on ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable). Recently, we moved a bunch of data to a new pool with better EC. This was done by adding a new EC pool to the FS. Then assigning the FS root to the new EC pool via the directory layout xattr (so all new data is written to the new pool). And finally copying old data to new folders. I swapped the data as follows to remain the old directory structures. I also made snapshots for validation purposes. So basically: cp -r mymount/mydata/ mymount/new/ # this creates copy on new pool mkdir mymount/mydata/.snap/tovalidate mkdir mymount/new/mydata/.snap/tovalidate mv mymount/mydata/ mymount/old/ mv mymount/new/mydata mymount/ I could see the increase of data in the new pool as expected (ceph df). I compared the snapshots with hashdeep to make sure the new data is alright. Then I went ahead deleting the old data, basically: rmdir mymount/old/mydata/.snap/* # this also included a bunch of other older snapshots rm -r mymount/old/mydata At first we had a bunch of PGs with snaptrim/snaptrim_wait. But they are done for quite some time now. And now, already two weeks later the size of the old pool still hasn't really decreased. I'm still waiting for around 500 TB to be released (and much more is planned). I honestly have no clue, where to go from here. From my point of view (i.e. the CephFS mount), the data is gone. I also never hard/soft-linked it anywhere. This doesn't seem to be a regular issue. At least I couldn't find anything related or resolved in the docs or user list, yet. If anybody has an idea how to resolve this, I would highly appreciate it. Best Wishes, Mathias

5 months

3
4
0 0

After hardware failure tried to recover ceph and followed instructions for recovery using OSDS

by Manolis Daramas

Hello everyone, We had a recent power failure on a server which hosts a 3-node ceph cluster (with Ubuntu 20.04 and Ceph version 17.2.7) and we think that we may have lost some of our data if not all of them. We have followed the instructions on https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-mon/#re… but with no luck. We have kept a backup of store.db folder on all 3 nodes prior the below steps. We have stopped ceph.target on all 3 nodes. We have run the first part of the script and we have altered it according to our configuration: ms=/root/mon-store mkdir $ms hosts="node01 node02 node03" # collect the cluster map from stopped OSDs for host in $hosts; do rsync -avz $ms/. root@$host:$ms.remote rm -rf $ms ssh root@$host <<EOF for osd in /var/lib/ceph/be4304e4-b0d5-11ec-8c6a-2965d4229f37/osd*; do ceph-objectstore-tool --data-path \$osd --no-mon-config --op update-mon-db --mon-store-path $ms.remote done EOF rsync -avz root@$host:$ms.remote/. $ms done and the results were: for node01 sd.0 : 0 osdmaps trimmed, 673 osdmaps added. osd.10 : 9225 osdmaps trimmed, 0 osdmaps added. Mount failed with '(5) Input/output error' osd.4 : 0 osdmaps trimmed, 0 osdmaps added. osd.8 : 0 osdmaps trimmed, 0 osdmaps added. receiving incremental file list created directory /root/mon-store ./ kv_backend store.db/ store.db/000008.sst store.db/000014.sst store.db/000020.sst store.db/000022.log store.db/CURRENT store.db/IDENTITY store.db/LOCK store.db/MANIFEST-000021 store.db/OPTIONS-000018 store.db/OPTIONS-000024 sent 248 bytes received 286,474 bytes 191,148.00 bytes/sec total size is 7,869,025 speedup is 27.44 sending incremental file list created directory /root/mon-store.remote ./ kv_backend store.db/ store.db/000008.sst store.db/000014.sst store.db/000020.sst store.db/000022.log store.db/CURRENT store.db/IDENTITY store.db/LOCK store.db/MANIFEST-000021 store.db/OPTIONS-000018 store.db/OPTIONS-000024 sent 286,478 bytes received 285 bytes 191,175.33 bytes/sec total size is 7,869,025 speedup is 27.44 for node02 osd.12 : 0 osdmaps trimmed, 0 osdmaps added. osd.2 : 0 osdmaps trimmed, 0 osdmaps added. osd.5 : 0 osdmaps trimmed, 0 osdmaps added. osd.7 : 0 osdmaps trimmed, 0 osdmaps added. osd.9 : 0 osdmaps trimmed, 0 osdmaps added. receiving incremental file list created directory /root/mon-store ./ kv_backend store.db/ store.db/000008.sst store.db/000014.sst store.db/000020.sst store.db/000026.sst store.db/000032.sst store.db/000038.sst store.db/000044.sst store.db/000050.sst store.db/000052.log store.db/CURRENT store.db/IDENTITY store.db/LOCK store.db/MANIFEST-000051 store.db/OPTIONS-000048 store.db/OPTIONS-000054 sent 343 bytes received 291,082 bytes 194,283.33 bytes/sec total size is 7,875,746 speedup is 27.02 sending incremental file list created directory /root/mon-store.remote ./ kv_backend store.db/ store.db/000008.sst store.db/000014.sst store.db/000020.sst store.db/000026.sst store.db/000032.sst store.db/000038.sst store.db/000044.sst store.db/000050.sst store.db/000052.log store.db/CURRENT store.db/IDENTITY store.db/LOCK store.db/MANIFEST-000051 store.db/OPTIONS-000048 store.db/OPTIONS-000054 sent 291,078 bytes received 380 bytes 582,916.00 bytes/sec total size is 7,875,746 speedup is 27.02 for node03 osd.1 : 0 osdmaps trimmed, 0 osdmaps added. osd.11 : 0 osdmaps trimmed, 0 osdmaps added. osd.13 : 0 osdmaps trimmed, 0 osdmaps added. osd.3 : 0 osdmaps trimmed, 0 osdmaps added. osd.6 : 0 osdmaps trimmed, 0 osdmaps added. receiving incremental file list created directory /root/mon-store ./ kv_backend store.db/ store.db/000008.sst store.db/000014.sst store.db/000020.sst store.db/000026.sst store.db/000032.sst store.db/000038.sst store.db/000044.sst store.db/000050.sst store.db/000056.sst store.db/000062.sst store.db/000068.sst store.db/000074.sst store.db/000080.sst store.db/000082.log store.db/CURRENT store.db/IDENTITY store.db/LOCK store.db/MANIFEST-000081 store.db/OPTIONS-000078 store.db/OPTIONS-000084 sent 438 bytes received 295,659 bytes 592,194.00 bytes/sec total size is 7,882,477 speedup is 26.62 Then we have run the (in order to rebuild the monstore DB and fix it): ceph-monstore-tool /root/mon-store rebuild -- --keyring /etc/ceph/ceph.client.admin.keyring --mon-ids node01 node02 node03 and the output is below: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: RocksDB version: 6.15.5 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Compile date Oct 25 2023 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: DB SUMMARY 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: DB Session ID: OS2T69IQ02SU5OKHBI40 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: CURRENT file: CURRENT 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: IDENTITY file: IDENTITY 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: MANIFEST file: MANIFEST-000081 size: 1083 Bytes 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: SST files in /root/mon-store/store.db dir, Total Num: 13, files: 000008.sst 000014.sst 000020.sst 000026.sst 000032.sst 000038.sst 000044.sst 000050.sst 000056.sst 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Write Ahead Log file in /root/mon-store/store.db: 000082.log size: 244 ; 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.error_if_exists: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.create_if_missing: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.paranoid_checks: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.env: 0x56017c8d1c20 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.fs: Posix File System 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.info_log: 0x56017d4c3860 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.statistics: (nil) 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.use_fsync: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_log_file_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_fallocate: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.use_direct_reads: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.create_missing_column_families: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.db_log_dir: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.wal_dir: /root/mon-store/store.db 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.advise_random_on_open: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.write_buffer_manager: 0x56017d1f6a20 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.rate_limiter: (nil) 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.unordered_write: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.row_cache: None 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.wal_filter: None 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.preserve_deletes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.two_write_queues: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.manual_wal_flush: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.atomic_flush: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.log_readahead_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.best_efforts_recovery: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_data_in_errors: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.db_host_id: __hostname__ 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_background_jobs: 2 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_background_compactions: -1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_subcompactions: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_total_wal_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_open_files: -1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bytes_per_sync: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_background_flushes: -1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Compression algorithms supported: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kZSTD supported: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kXpressCompression supported: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kLZ4Compression supported: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kBZip2Compression supported: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kZlibCompression supported: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kSnappyCompression supported: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /root/mon-store/store.db/MANIFEST-000081 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.merge_operator: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_filter: None 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_filter_factory: None 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.sst_partitioner_factory: None 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56017d234f80) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 0 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 index_shortening: 1 data_block_hash_table_util_ratio: 0.750000 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x56017d22f610 block_cache_name: BinnedLRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 4 enable_index_compression: 1 block_align: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression: NoCompression 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.num_levels: 7 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.level: 32767 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.parallel_threads: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.enabled: false 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.arena_block_size: 4194304 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.table_properties_collectors: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.inplace_update_support: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bloom_locality: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_successive_merges: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.force_consistency_checks: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.ttl: 2592000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.enable_blob_files: false 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.min_blob_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.blob_file_size: 268435456 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.blob_compression_type: NoCompression 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.enable_blob_garbage_collection: false 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-11-17T12:26:24.156+0200 7f482b393600 4 rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/root/mon-store/store.db/MANIFEST-000081 succeeded,manifest_file_number is 81, next_file_number is 83, last_sequence is 21183, log_number is 77,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-17T12:26:24.156+0200 7f482b393600 4 rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 77 2023-11-17T12:26:24.156+0200 7f482b393600 4 rocksdb: [db/version_set.cc:4082] Creating manifest 85 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784162798, "job": 1, "event": "recovery_started", "wal_files": [82]} 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #82 mode 2 2023-11-17T12:26:24.160+0200 7f482b393600 3 rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784163944, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 86, "file_size": 1266, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 238, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 72, "raw_average_key_size": 24, "raw_value_size": 148, "raw_average_value_size": 49, "num_data_blocks": 1, "num_entries": 3, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700216784, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "53025a24-2059-43e1-a0f7-a87a28e33d38", "db_session_id": "OS2T69IQ02SU5OKHBI40"}} 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: [db/version_set.cc:4082] Creating manifest 87 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784166273, "job": 1, "event": "recovery_finished"} 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: [db/column_family.cc:983] [default] Increasing compaction threads because we have 14 level-0 files 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000082.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.164+0200 7f482b393600 4 rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x56017d230700 2023-11-17T12:26:24.164+0200 7f482b393600 4 rocksdb: DB pointer 0x56017df56000 adding auth for 'client.admin': auth(key=AQCsdUViHYjTGBAAf7/1KYZjb0h3x3EOywqbbQ==) with caps({mds=allow *,mgr=allow *,mon=allow *,osd=allow *}) 2023-11-17T12:26:24.164+0200 7f482a349700 4 rocksdb: [db/compaction/compaction_job.cc:1881] [default] [JOB 3] Compacting 14@0 files to L6, score 3.50 2023-11-17T12:26:24.164+0200 7f482a349700 4 rocksdb: [db/compaction/compaction_job.cc:1887] [default] Compaction start summary: Base version 3 Base level 0, inputs: [86(1266B) 80(1266B) 74(1267B) 68(1267B) 62(1266B) 56(1265B) 50(1265B) 44(1265B) 38(1265B) 32(1266B) 26(1265B) 20(1265B) 14(283KB) 8(7387KB)] 2023-11-17T12:26:24.164+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784169200, "job": 3, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [86, 80, 74, 68, 62, 56, 50, 44, 38, 32, 26, 20, 14, 8], "score": 3.5, "input_data_size": 7870219} 2023-11-17T12:26:24.164+0200 7f4822339700 4 rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2023-11-17T12:26:24.164+0200 7f4822339700 4 rocksdb: [db/db_impl/db_impl.cc:903] ** DB Stats ** Uptime(secs): 0.0 total, 0.0 interval Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Sum 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Sum 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.10 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** 2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb: [db/compaction/compaction_job.cc:1516] [default] [JOB 3] Generated table #91: 1366 keys, 7566988 bytes 2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784213586, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 91, "file_size": 7566988, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 7541895, "index_size": 20610, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3525, "raw_key_size": 29308, "raw_average_key_size": 21, "raw_value_size": 7503048, "raw_average_value_size": 5492, "num_data_blocks": 764, "num_entries": 1366, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700216681, "oldest_key_time": 0, "file_creation_time": 1700216784, "db_id": "53025a24-2059-43e1-a0f7-a87a28e33d38", "db_session_id": "OS2T69IQ02SU5OKHBI40"}} 2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb: [db/compaction/compaction_job.cc:1594] [default] [JOB 3] Compacted 14@0 files to L6 => 7566988 bytes 2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb: [db/version_set.cc:3457] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: (Original Log Time 2023/11/17-12:26:24.215298) [db/compaction/compaction_job.cc:812] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 177.1 rd, 170.3 wr, level 6, files in(14, 0) out(1) MB in(7.5, 0.0) out(7.2), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 19842, records dropped: 18476 output_compression: NoCompression 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: (Original Log Time 2023/11/17-12:26:24.215314) EVENT_LOG_v1 {"time_micros": 1700216784215306, "job": 3, "event": "compaction_finished", "compaction_time_micros": 44437, "compaction_time_cpu_micros": 40923, "output_level": 6, "num_output_files": 1, "total_output_size": 7566988, "num_input_records": 19842, "num_output_records": 1366, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000086.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215520, "job": 3, "event": "table_file_deletion", "file_number": 86} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000080.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215570, "job": 3, "event": "table_file_deletion", "file_number": 80} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000074.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215603, "job": 3, "event": "table_file_deletion", "file_number": 74} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000068.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215641, "job": 3, "event": "table_file_deletion", "file_number": 68} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000062.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215672, "job": 3, "event": "table_file_deletion", "file_number": 62} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000056.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215708, "job": 3, "event": "table_file_deletion", "file_number": 56} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000050.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215739, "job": 3, "event": "table_file_deletion", "file_number": 50} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000044.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215772, "job": 3, "event": "table_file_deletion", "file_number": 44} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000038.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215804, "job": 3, "event": "table_file_deletion", "file_number": 38} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000032.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215831, "job": 3, "event": "table_file_deletion", "file_number": 32} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000026.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215858, "job": 3, "event": "table_file_deletion", "file_number": 26} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000020.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215888, "job": 3, "event": "table_file_deletion", "file_number": 20} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000014.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215952, "job": 3, "event": "table_file_deletion", "file_number": 14} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000008.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784216804, "job": 3, "event": "table_file_deletion", "file_number": 8} update_mkfs generating seed initial monmap epoch 0 fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37 last_changed 2023-11-17T12:26:24.222814+0200 created 2023-11-17T12:26:24.222814+0200 min_mon_release 0 (unknown) election_strategy: 1 0: [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] mon.node01 1: [v2:10.40.99.12:3300/0,v1:10.40.99.12:6789/0] mon.node02 2: [v2:10.40.99.13:3300/0,v1:10.40.99.13:6789/0] mon.node03 2023-11-17T12:26:24.220+0200 7f482b393600 4 rocksdb: [db/db_impl/db_impl.cc:446] Shutdown: canceling all background work 2023-11-17T12:26:24.220+0200 7f482b393600 4 rocksdb: [db/db_impl/db_impl.cc:625] Shutdown complete Then we copied the /root/mon-store/store.db folder across on all 3 nodes and tried to start ceph.target service again. The output on node01 is below: d31781fa6b4c quay.io/ceph/ceph "/usr/bin/ceph-mds -..." 55 minutes ago Up 55 minutes ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mds-storage-node01-cjrvjc e385c32651d2 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-10 904f522c4cb5 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-0 033edf99a98e quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-4 70344a6e87a0 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-8 905b782aedcf quay.io/prometheus/prometheus:v2.43.0 "/bin/prometheus --c..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-prometheus-node01 ff191654eb3e quay.io/prometheus/node-exporter:v1.5.0 "/bin/node_exporter ..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-node-exporter-node01 459c46f4bdb7 quay.io/ceph/ceph "/usr/bin/ceph-mgr -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mgr-node01-xlciyx cacfe8abcbbf quay.io/ceph/ceph "/usr/bin/ceph-crash..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-crash-node01 e216ef2af166 quay.io/prometheus/alertmanager:v0.25.0 "/bin/alertmanager -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-alertmanager-node01 d3238b2285d1 quay.io/ceph/ceph-grafana:9.4.7 "/bin/sh -c 'grafana..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-grafana-node01 The output on node02 is below: 2aec62685dee quay.io/ceph/ceph "/usr/bin/ceph-mds -..." 54 minutes ago Up 54 minutes ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mds-storage-node02-lyudbp 249b04f32f8c quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-5 a2c96f56b517 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-2 87496d374a29 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-12 55fe47765917 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-9 76171e25dbde quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-7 220472e8c1bf quay.io/ceph/ceph "/usr/bin/ceph-mgr -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mgr-node02-gudauu 0c783e73e543 quay.io/prometheus/node-exporter:v1.5.0 "/bin/node_exporter ..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-node-exporter-node02 4e638003fa2e quay.io/ceph/ceph "/usr/bin/ceph-crash..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-crash-node02 42719d5cfdbf quay.io/ceph/ceph "/usr/bin/ceph-mon -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mon-node02 The output on node03 is below: 7e5879dce643 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-11 d53996ff33b9 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-3 e1ac5a8b87d3 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-1 f4cda871218d quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-13 969e670dc47c quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-6 a49e91a7bb8e quay.io/prometheus/node-exporter:v1.5.0 "/bin/node_exporter ..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-node-exporter-node03 835c3893a3f4 quay.io/ceph/ceph "/usr/bin/ceph-crash..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-crash-node03 bfa6f5b989ea quay.io/ceph/ceph "/usr/bin/ceph-mon -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mon-node03 # ceph -s (output below): cluster: id: be4304e4-b0d5-11ec-8c6a-2965d4229f37 health: HEALTH_ERR 20 stray daemon(s) not managed by cephadm 3 stray host(s) with 20 daemon(s) not managed by cephadm 1/3 mons down, quorum node02,node03 1/523510 objects unfound (0.000%) 3 nearfull osd(s) 1 osds exist in the crush map but not in the osdmap Low space hindering backfill (add storage if this doesn't resolve itself): 20 pgs backfill_toofull Possible data damage: 1 pg recovery_unfound Degraded data redundancy: 74666/1570530 objects degraded (4.754%), 21 pgs degraded, 21 pgs undersized 3 pool(s) nearfull services: mon: 3 daemons, quorum node02,node03 (age 2d), out of quorum: node01 mgr: node01.xlciyx(active, since 2d), standbys: node02.gudauu osd: 14 osds: 14 up (since 2d), 14 in (since 3d); 21 remapped pgs data: pools: 3 pools, 161 pgs objects: 523.51k objects, 299 GiB usage: 1014 GiB used, 836 GiB / 1.8 TiB avail pgs: 74666/1570530 objects degraded (4.754%) 1/523510 objects unfound (0.000%) 140 active+clean 20 active+undersized+degraded+remapped+backfill_toofull 1. active+recovery_unfound+undersized+degraded+remapped # ceph fs ls (output below): No filesystems enabled It looks like that we have a problem with the orchestrator now (we've lost cephadm orchestrator) and we also cannot see the filesystem. May you please assist since we are not able to mount the filesystem ? Thank you, Manolis Daramas Under the General Data Protection Regulation (GDPR) (EU) 2016/679, Motivian as Data Controller has a legal duty to protect any information collected from you via email. Information contained in this email and any attachments may be privileged or confidential and intended for the exclusive use of the original recipient. If you have received this email by mistake, please advise the sender immediately and delete the email, including emptying your deleted email box. Information included in this email is reserved to named addressee's eyes only. You may not share this message or any of its attachments to anyone. Please note that as the recipient, it is your responsibility to check the email for malicious software. Motivian puts the security of the client at a high priority. Therefore, we have put efforts into ensuring that the message is error and virus-free. Unfortunately, full security of the email cannot be ensured as, despite our efforts, the data included in emails could be infected, intercepted, or corrupted. Therefore, the recipient should check the email for threats with proper software, as the sender does not accept liability for any damage inflicted by viewing the content of this email.

5 months

2
5
0 0

MDS stuck in up:rejoin

by Eric Tittley

Hi all, For about a week our CephFS has experienced issues with its MDS. Currently the MDS is stuck in "up:rejoin" Issues become apparent when simple commands like "mv foo bar/" hung. I unmounted CephFS offline on the clients, evicted those remaining, and then issued ceph config set mds.0 mds_wipe_sessions true ceph config set mds.1 mds_wipe_sessions true which allowed me to delete the hung requests. I've lost the exact commands I used, but something like rados -p cephfs_metadata ls | grep mds rados rm -p cephfs_metadata mds0_openfiles.0 etc This allowed the MDS to get to "up:rejoin" where it has been stuck ever since which is getting on five days. # ceph mds stat cephfs:1/1 {0=cephfs.ceph00.uvlkrw=up:rejoin} 2 up:standby root@ceph00:/var/log/ceph/a614303a-5eb5-11ed-b492-011f01e12c9a# ceph -s cluster: id: a614303a-5eb5-11ed-b492-011f01e12c9a health: HEALTH_WARN 1 filesystem is degraded 1 pgs not deep-scrubbed in time 2 pool(s) do not have an application enabled 1 daemons have recently crashed services: mon: 3 daemons, quorum ceph00,ceph01,ceph02 (age 57m) mgr: ceph01.lvdgyr(active, since 2h), standbys: ceph00.gpwpgs mds: 1/1 daemons up, 2 standby osd: 91 osds: 90 up (since 78m), 90 in (since 112m) data: volumes: 0/1 healthy, 1 recovering pools: 5 pools, 1539 pgs objects: 138.83M objects, 485 TiB usage: 971 TiB used, 348 TiB / 1.3 PiB avail pgs: 1527 active+clean 12 active+clean+scrubbing+deep io: client: 3.1 MiB/s rd, 3.16k op/s rd, 0 op/s wr # ceph --version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) I've tried failing the MDS so it switches. Rebooted a couple of times. I've added more OSDs to the metadata pool and took one out as I thought it might be a bad metadata OSD (The "recently crashed" daemon). The error logs are full of (prefix to all are: Nov 27 14:02:44 ceph00 bash[2145]: debug 2023-11-27T14:02:44.619+0000 7f74e845e700 1 -- [v2:192.168.1.128:6800/2157301677,v1:192.168.1.128:6801/2157301677] --> [v2:192.168.1.133:6896/4289132926,v1:192.168.1.133:6897/4289132926] ) crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_message enqueueing message m=0x559be00adc00 type=42 osd_op(mds.0.36244:8142873 3.ff 3:ff5b34d6:::1.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e32465) v8 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).write_message sending message m=0x559be00adc00 seq=8142643 osd_op(mds.0.36244:8142873 3.ff 3:ff5b34d6:::1.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e32465) v8 crc :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message got 154 + 0 + 30 byte message. envelope type=43 src osd.89 off 0 crc :-1 s=READ_MESSAGE_COMPLETE pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message received message m=0x559be01f4480 seq=8142643 from=osd.89 type=43 osd_op_reply(8142873 1.00000000 [getxattr (30) out=30b] v0'0 uv560123 ondisk = 0) v8 osd_op_reply(8142873 1.00000000 [getxattr (30) out=30b] v0'0 uv560123 ondisk = 0) v8 ==== 154+0+30 (crc 0 0 0) 0x559be01f4480 con 0x559be00ad800 osd_op(unknown.0.36244:8142874 3.ff 3:ff5b34d6:::1.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e32465) v8 -- 0x559be2caec00 con 0x559be00ad800 Repeating multiple times a second (and filling /var) Prior to taking one of the cephfs_metadata OSDs offline, these came from communications from ceph00 to the node hosting the suspected bad OSD. Now they are between ceph00 and the host of the replacement metadata OSD. Does anyone have any suggestion on how to get the MDS to switch from "up:rejoin" to "up:active"? Is there any way to debug this, to determine what issue really is? I'm unable to interpret the debug log. Cheers, Eric ________________________________________________________ Dr Eric Tittley Research Computing Officer www.roe.ac.uk/~ert<http://www.roe.ac.uk/~ert> Institute for Astronomy Royal Observatory, Edinburgh The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

5 months

2
5
0 0

mds slow request with “failed to authpin, subtree is being exported"

by zxcs

HI, Experts, we are using cephfs with 16.2.* with multi active mds, and recently, we have two nodes mount with ceph-fuse due to the old os system. and one nodes run a python script with `glob.glob(path)`, and another client doing `cp` operation on the same path. then we see some log about `mds slow request`, and logs complain “failed to authpin, subtree is being exported" then need to restart mds, our question is, does there any dead lock? how can we avoid this and how to fix it without restart mds(it will influence other users) ? Thanks a ton! xz

5 months

6
11
0 0

the image used size becomes 0 after export/import with snapshot

by Tony Liu

Hi, I have an image with a snapshot and some changes after snapshot. ``` $ rbd du backup/f0408e1e-06b6-437b-a2b5-70e3751d0a26 NAME PROVISIONED USED f0408e1e-06b6-437b-a2b5-70e3751d0a26@snapshot-eb085877-7557-4620-9c01-c5587b857029 10 GiB 2.4 GiB f0408e1e-06b6-437b-a2b5-70e3751d0a26 10 GiB 2.4 GiB <TOTAL> 10 GiB 4.8 GiB ``` If there is no changes after snapshot, the image line will show 0 used. I did export and import. ``` $ rbd export --export-format 2 backup/f0408e1e-06b6-437b-a2b5-70e3751d0a26 - | rbd import --export-format 2 - backup/test Exporting image: 100% complete...done. Importing image: 100% complete...done. ``` When check the imported image, the image line shows 0 used. ``` $ rbd du backup/test NAME PROVISIONED USED test@snapshot-eb085877-7557-4620-9c01-c5587b857029 10 GiB 2.4 GiB test 10 GiB 0 B <TOTAL> 10 GiB 2.4 GiB ``` Any clues how that happened? I'd expect the same du as the source. I tried another quick test. It works fine. ``` $ rbd create backup/test-src --size 10G $ sudo rbd map backup/test-src /dev/rbd0 $ echo "hello" | sudo tee /dev/rbd0 hello $ rbd du backup/test-src NAME PROVISIONED USED test-src 10 GiB 4 MiB $ rbd snap create backup/test-src@snap-1 Creating snap: 100% complete...done. $ rbd du backup/test-src NAME PROVISIONED USED test-src@snap-1 10 GiB 4 MiB test-src 10 GiB 0 B <TOTAL> 10 GiB 4 MiB $ echo "world" | sudo tee /dev/rbd0 world $ rbd du backup/test-src NAME PROVISIONED USED test-src@snap-1 10 GiB 4 MiB test-src 10 GiB 4 MiB <TOTAL> 10 GiB 8 MiB $ rbd export --export-format 2 backup/test-src - | rbd import --export-format 2 - backup/test-dst Exporting image: 100% complete...done. Importing image: 100% complete...done. $ rbd du backup/test-dst NAME PROVISIONED USED test-dst@snap-1 10 GiB 4 MiB test-dst 10 GiB 4 MiB <TOTAL> 10 GiB 8 MiB ``` Thanks! Tony

5 months

2
2
0 0

Ceph/daemon container lvm tools don’t work

by Gaël THEROND

Is there anyone using containerized CEPH over CentOS Stream 9 Hosts already? I think there is a pretty big issue in here if CEPH images are built over CentOS but never tested against it.

5 months, 1 week

1
0
0 0

Recommended architecture

by Francisco Arencibia Quesada

Hello again guys, Can you recommend me a book that explains best practices with Ceph, for example is it okay to have mon,mgr, osd in the same virtual machine, what is the recommended architecture according to your experience? Because by default is doing this: Cluster Ceph | +----------------------------+----------------------------+ | | | |10.0.0.52 |10.0.0.194 |10.0.0.229 +-----------+-----------+ +-----------+-----------+ +-----------+-----------+ |[node01.jotelulu.space]| [node02.jotelulu.space] |[node03.jotelulu.space]| | OSD +----+ OSD +----+ OSD | | Monitor Daemon | | Monitor Daemon | Monitor Daemon | | Manager Daemon | |Manager Daemon(standby) | | | +-----------------------+ +-----------------------+ +-----------------------+ -- Regards *Francisco Arencibia Quesada.* *DevOps Engineer*

5 months, 1 week

3
2
0 0

Public/private network

by Albert Shih

Hi everyone. Status : Installing a ceph cluster Version : 17.2.7 Quincy OS : Debian 11. Each of my server got two ip address. One public and one private. When I'm trying to deploy my cluster with on a server server1 (the hostname) with cephadm bootstrap --mon-id hostname --mon-ip IP_PRIVATE --cluster-network PRIVATE_SUB I end up with private network for ceph config get mon public_network So I try to change it with ceph config set mon public_network PUBLIC_SUB still with lsof -i |grep -i listen I got ceph-mgr 31427 ceph 49u IPv4 119937 0t0 TCP server1-ceph.private.:7150 (LISTEN) node_expo 31572 nobody 3u IPv6 65495 0t0 TCP *:9100 (LISTEN) alertmana 31573 nobody 3u IPv6 21377 0t0 TCP *:9094 (LISTEN) alertmana 31573 nobody 8u IPv6 136298 0t0 TCP *:9093 (LISTEN) prometheu 31757 nobody 7u IPv6 109680 0t0 TCP *:9095 (LISTEN) grafana 31758 node-exporter 11u IPv6 100726 0t0 TCP *:3000 (LISTEN) ceph-mon 31850 ceph 27u IPv4 139664 0t0 TCP server1-ceph.private.:3300 (LISTEN) ceph-mon 31850 ceph 28u IPv4 139665 0t0 TCP server1-ceph.private.:6789 (LISTEN) So the ceph-mon listen on the private interface. Is this something normal ? Because according to https://access.redhat.com/documentation/fr-fr/red_hat_ceph_storage/5/html/c… only the OSD should listen on private network. Is they are anyway to configure booth public_network and private_network with cephadm bootstrap ? Regards. -- Albert SHIH 🦫 🐸 France Heure locale/Local time: jeu. 30 nov. 2023 18:27:08 CET

5 months, 1 week

2
1
0 0

MDS_DAMAGE in 17.2.7 / Cannot delete affected files

by Sebastian Knust

Hi, After updating from 17.2.6 to 17.2.7 with cephadm, our cluster went into MDS_DAMAGE state. We had some prior issues with faulty kernel clients not releasing capabilities, therefore the update might just be a coincidence. `ceph tell mds.cephfs:0 damage ls` lists 56 affected files all with these general details: { "damage_type": "dentry", "id": 123456, "ino": 1234567890, "frag": "*", "dname": "some-filename.ext", "snap_id": "head", "path": "/full/path/to/file" } The behaviour upon trying to access file information in the (Kernel mounted) filesystem is a bit inconsistent. Generally, the first `stat` call seems to result in "Input/output error", the next call provides all `stat` data as expected from an undamaged file. The file can be read with `cat` with full and correct content (verified with backup) once the stat call succeeds. Scrubbing the affected subdirectories with `ceph tell mds.cephfs:0 scrub start /path/to/dir/ recursive,repair,force` does not fix the issue. Trying to delete the file results in an "Input/output error". If the stat calls beforehand succeeded, this also crashes the active MDS with these messages in the system journal: > Nov 24 14:21:15 iceph-18.servernet ceph-mds[1946861]: mds.0.cache.den(0x10012271195 DisplaySettings.json) newly corrupt dentry to be committed: [dentry #0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json [1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x10012271197 state=1073741824 | inodepin=1 0x56413e1e2780] > Nov 24 14:21:15 iceph-18.servernet ceph-mds[1946861]: log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json [1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x10012271197 state=1073741824 | inodepin=1 0x56413e1e2780] > Nov 24 14:21:15 iceph-18.servernet ceph-eafd0514-3644-11eb-bc6a-3cecef2330fa-mds-cephfs-iceph-18-ujfqnd[1946838]: 2023-11-24T13:21:15.654+0000 7f3fdcde0700 -1 mds.0.cache.den(0x10012271195 DisplaySettings.json) newly corrupt dentry to be committed: [dentry #0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json [1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x1001> > Nov 24 14:21:15 iceph-18.servernet ceph-eafd0514-3644-11eb-bc6a-3cecef2330fa-mds-cephfs-iceph-18-ujfqnd[1946838]: 2023-11-24T13:21:15.654+0000 7f3fdcde0700 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json [1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x10012> > Nov 24 14:21:15 iceph-18.servernet ceph-eafd0514-3644-11eb-bc6a-3cecef2330fa-mds-cephfs-iceph-18-ujfqnd[1946838]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f3fdcde0700 time 2023-11-24T13:21:15.655088+0000 > Nov 24 14:21:15 iceph-18.servernet ceph-mds[1946861]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f3fdcde0700 time 2023-11-24T13:21:15.655088+0000 > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDSRank.cc: 937: ceph_abort_msg("abort() called") > > ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) > 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xd7) [0x7f3fe5a1cb03] > 2: (MDSRank::abort(std::basic_string_view<char, std::char_traits<char> >)+0x7d) [0x5640f2e6fa2d] > 3: (CDentry::check_corruption(bool)+0x740) [0x5640f30e4820] > 4: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x5640f2f41877] > 5: (EOpen::add_clean_inode(CInode*)+0x121) [0x5640f2f49fc1] > 6: (Locker::adjust_cap_wanted(Capability*, int, int)+0x426) [0x5640f305e036] > 7: (Locker::process_request_cap_release(boost::intrusive_ptr<MDRequestImpl>&, client_t, ceph_mds_request_release const&, std::basic_string_view<char, std::char_traits<char> >)+0x599) [0x5640f307f7e9] > 8: (Server::handle_client_request(boost::intrusive_ptr<MClientRequest const> const&)+0xc06) [0x5640f2f2a7c6] > 9: (Server::dispatch(boost::intrusive_ptr<Message const> const&)+0x13c) [0x5640f2f2ef6c] > 10: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x5db) [0x5640f2e7727b] > 11: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x5c) [0x5640f2e778bc] > 12: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1bf) [0x5640f2e60c2f] > 13: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x478) [0x7f3fe5c97ed8] > 14: (DispatchQueue::entry()+0x50f) [0x7f3fe5c9531f] > 15: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3fe5d5f381] > 16: /lib64/libpthread.so.0(+0x81ca) [0x7f3fe4a0b1ca] > 17: clone() Deleting the file with cephfs-shell also does give Input/output error (5). Does anyone have an idea on how to proceed here? I am perfectly fine with loosing the affected files, they can all be easily restored from backup. Cheers Sebastian

5 months, 1 week

3
5
0 0

2024

2023

2022

2021

2020

2019

ceph-users November 2023