December 2023 - ceph-users

Space reclaim doesn't happening in nautilus RBD pool

by Szabo, Istvan (Agoda)

Hi, Is there any config on Ceph that block/not perform space reclaim? I test on one pool which has only one image 1.8 TiB in used. rbd $p du im/root warning: fast-diff map is not enabled for root. operation may be slow. NAME PROVISIONED USED root 2.2 TiB 1.8 TiB I already removed all snaphots and now pool has only one image alone. I run both fstrim over the filesystem (XFS) and try rbd sparsify im/root (don't know what it is exactly but it mentions to reclaim something) It still shows the pool used 6.9 TiB which totally not make sense right? It should be up to 3.6 (1.8 * 2) according to its replica? POOLS: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR im 19 32 3.5 TiB 918.34k 6.9 TiB 4.80 69 TiB N/A 10 TiB 918.34k 0 B 0 B I think now some of others pool have this issue too, we do clean up a lot but seems space not reclaimed. I estimate more than 50 TiB should be able to reclaim, actual usage of this cluster much less than current reported number. Thank you for your help. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 months, 2 weeks

2
4
0 0

Ceph 16.2.14: osd crash, bdev() _aio_thread got r=-1 ((1) Operation not permitted)

by Zakhar Kirpichenko

Hi, One of our 16.2.14 cluster OSDs crashed again because of the dreaded https://tracker.ceph.com/issues/53906 bug. Usually an OSD, which crashed because of this bug, restarts within seconds and continues normal operation. This time it failed to restart and kept crashing: "assert_condition": "abort", "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14/rpm/el8/BUILD/ceph-16.2.14/src/blk/kernel/KernelDevice.cc", "assert_func": "void KernelDevice::_aio_thread()", "assert_line": 604, "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14/rpm/el8/BUILD/ceph-16.2.14/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f08520e2700 time 2023-12-03T04:00:36.689614+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14/rpm/el8/BUILD/ceph-16.2.14/src/blk/kernel/KernelDevice.cc: 604: ceph_abort_msg(\"Unexpected IO error. This may suggest HW issue. Please check your dmesg!\")\n", "assert_thread_name": "bstore_aio", "backtrace": [ "/lib64/libpthread.so.0(+0x12cf0) [0x7f085e308cf0]", "gsignal()", "abort()", "(ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x55f01d9494cb]", "(KernelDevice::_aio_thread()+0x1285) [0x55f01e4b5c15]", "(KernelDevice::AioCompletionThread::entry()+0x11) [0x55f01e4c0ee1]", "/lib64/libpthread.so.0(+0x81ca) [0x7f085e2fe1ca]", "clone()" ], There was nothing in dmesg though and the block device looked healthy. I took the OSD down, ran a long SMART test on its block drive, ran a read test on the drive and found no issues. I tried restarting the OSD again and found in its debug that it failed because of an "2023-12-03T04:00:36.686+0000 7f08520e2700 -1 bdev(0x55f02a28a400 /var/lib/ceph/osd/ceph-56/block) _aio_thread got r=-1 ((1) Operation not permitted)" error: https://pastebin.com/gDat6rfk I remember hitting this previously: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/GYL72G3F4P…, and this time a host reboot completely resolved the issue. It would be good to understand what has triggered this condition and how it can be resolved without rebooting the whole host. I would very much appreciate any suggestions. Best regards, Zakhar

5 months, 2 weeks

3
6
0 0

CephFS pool not releasing space after data deletion

by Kuhring, Mathias

Dear Ceph users, Our CephFS is not releasing/freeing up space after deleting hundreds of terabytes of data. By now, this drives us in a "nearfull" osd/pool situation and thus throttles IO. We are on ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable). Recently, we moved a bunch of data to a new pool with better EC. This was done by adding a new EC pool to the FS. Then assigning the FS root to the new EC pool via the directory layout xattr (so all new data is written to the new pool). And finally copying old data to new folders. I swapped the data as follows to remain the old directory structures. I also made snapshots for validation purposes. So basically: cp -r mymount/mydata/ mymount/new/ # this creates copy on new pool mkdir mymount/mydata/.snap/tovalidate mkdir mymount/new/mydata/.snap/tovalidate mv mymount/mydata/ mymount/old/ mv mymount/new/mydata mymount/ I could see the increase of data in the new pool as expected (ceph df). I compared the snapshots with hashdeep to make sure the new data is alright. Then I went ahead deleting the old data, basically: rmdir mymount/old/mydata/.snap/* # this also included a bunch of other older snapshots rm -r mymount/old/mydata At first we had a bunch of PGs with snaptrim/snaptrim_wait. But they are done for quite some time now. And now, already two weeks later the size of the old pool still hasn't really decreased. I'm still waiting for around 500 TB to be released (and much more is planned). I honestly have no clue, where to go from here. From my point of view (i.e. the CephFS mount), the data is gone. I also never hard/soft-linked it anywhere. This doesn't seem to be a regular issue. At least I couldn't find anything related or resolved in the docs or user list, yet. If anybody has an idea how to resolve this, I would highly appreciate it. Best Wishes, Mathias

5 months, 2 weeks

3
4
0 0

After hardware failure tried to recover ceph and followed instructions for recovery using OSDS

by Manolis Daramas

Hello everyone, We had a recent power failure on a server which hosts a 3-node ceph cluster (with Ubuntu 20.04 and Ceph version 17.2.7) and we think that we may have lost some of our data if not all of them. We have followed the instructions on https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-mon/#re… but with no luck. We have kept a backup of store.db folder on all 3 nodes prior the below steps. We have stopped ceph.target on all 3 nodes. We have run the first part of the script and we have altered it according to our configuration: ms=/root/mon-store mkdir $ms hosts="node01 node02 node03" # collect the cluster map from stopped OSDs for host in $hosts; do rsync -avz $ms/. root@$host:$ms.remote rm -rf $ms ssh root@$host <<EOF for osd in /var/lib/ceph/be4304e4-b0d5-11ec-8c6a-2965d4229f37/osd*; do ceph-objectstore-tool --data-path \$osd --no-mon-config --op update-mon-db --mon-store-path $ms.remote done EOF rsync -avz root@$host:$ms.remote/. $ms done and the results were: for node01 sd.0 : 0 osdmaps trimmed, 673 osdmaps added. osd.10 : 9225 osdmaps trimmed, 0 osdmaps added. Mount failed with '(5) Input/output error' osd.4 : 0 osdmaps trimmed, 0 osdmaps added. osd.8 : 0 osdmaps trimmed, 0 osdmaps added. receiving incremental file list created directory /root/mon-store ./ kv_backend store.db/ store.db/000008.sst store.db/000014.sst store.db/000020.sst store.db/000022.log store.db/CURRENT store.db/IDENTITY store.db/LOCK store.db/MANIFEST-000021 store.db/OPTIONS-000018 store.db/OPTIONS-000024 sent 248 bytes received 286,474 bytes 191,148.00 bytes/sec total size is 7,869,025 speedup is 27.44 sending incremental file list created directory /root/mon-store.remote ./ kv_backend store.db/ store.db/000008.sst store.db/000014.sst store.db/000020.sst store.db/000022.log store.db/CURRENT store.db/IDENTITY store.db/LOCK store.db/MANIFEST-000021 store.db/OPTIONS-000018 store.db/OPTIONS-000024 sent 286,478 bytes received 285 bytes 191,175.33 bytes/sec total size is 7,869,025 speedup is 27.44 for node02 osd.12 : 0 osdmaps trimmed, 0 osdmaps added. osd.2 : 0 osdmaps trimmed, 0 osdmaps added. osd.5 : 0 osdmaps trimmed, 0 osdmaps added. osd.7 : 0 osdmaps trimmed, 0 osdmaps added. osd.9 : 0 osdmaps trimmed, 0 osdmaps added. receiving incremental file list created directory /root/mon-store ./ kv_backend store.db/ store.db/000008.sst store.db/000014.sst store.db/000020.sst store.db/000026.sst store.db/000032.sst store.db/000038.sst store.db/000044.sst store.db/000050.sst store.db/000052.log store.db/CURRENT store.db/IDENTITY store.db/LOCK store.db/MANIFEST-000051 store.db/OPTIONS-000048 store.db/OPTIONS-000054 sent 343 bytes received 291,082 bytes 194,283.33 bytes/sec total size is 7,875,746 speedup is 27.02 sending incremental file list created directory /root/mon-store.remote ./ kv_backend store.db/ store.db/000008.sst store.db/000014.sst store.db/000020.sst store.db/000026.sst store.db/000032.sst store.db/000038.sst store.db/000044.sst store.db/000050.sst store.db/000052.log store.db/CURRENT store.db/IDENTITY store.db/LOCK store.db/MANIFEST-000051 store.db/OPTIONS-000048 store.db/OPTIONS-000054 sent 291,078 bytes received 380 bytes 582,916.00 bytes/sec total size is 7,875,746 speedup is 27.02 for node03 osd.1 : 0 osdmaps trimmed, 0 osdmaps added. osd.11 : 0 osdmaps trimmed, 0 osdmaps added. osd.13 : 0 osdmaps trimmed, 0 osdmaps added. osd.3 : 0 osdmaps trimmed, 0 osdmaps added. osd.6 : 0 osdmaps trimmed, 0 osdmaps added. receiving incremental file list created directory /root/mon-store ./ kv_backend store.db/ store.db/000008.sst store.db/000014.sst store.db/000020.sst store.db/000026.sst store.db/000032.sst store.db/000038.sst store.db/000044.sst store.db/000050.sst store.db/000056.sst store.db/000062.sst store.db/000068.sst store.db/000074.sst store.db/000080.sst store.db/000082.log store.db/CURRENT store.db/IDENTITY store.db/LOCK store.db/MANIFEST-000081 store.db/OPTIONS-000078 store.db/OPTIONS-000084 sent 438 bytes received 295,659 bytes 592,194.00 bytes/sec total size is 7,882,477 speedup is 26.62 Then we have run the (in order to rebuild the monstore DB and fix it): ceph-monstore-tool /root/mon-store rebuild -- --keyring /etc/ceph/ceph.client.admin.keyring --mon-ids node01 node02 node03 and the output is below: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: RocksDB version: 6.15.5 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Compile date Oct 25 2023 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: DB SUMMARY 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: DB Session ID: OS2T69IQ02SU5OKHBI40 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: CURRENT file: CURRENT 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: IDENTITY file: IDENTITY 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: MANIFEST file: MANIFEST-000081 size: 1083 Bytes 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: SST files in /root/mon-store/store.db dir, Total Num: 13, files: 000008.sst 000014.sst 000020.sst 000026.sst 000032.sst 000038.sst 000044.sst 000050.sst 000056.sst 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Write Ahead Log file in /root/mon-store/store.db: 000082.log size: 244 ; 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.error_if_exists: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.create_if_missing: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.paranoid_checks: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.env: 0x56017c8d1c20 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.fs: Posix File System 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.info_log: 0x56017d4c3860 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.statistics: (nil) 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.use_fsync: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_log_file_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_fallocate: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.use_direct_reads: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.create_missing_column_families: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.db_log_dir: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.wal_dir: /root/mon-store/store.db 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.advise_random_on_open: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.write_buffer_manager: 0x56017d1f6a20 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.rate_limiter: (nil) 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.unordered_write: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.row_cache: None 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.wal_filter: None 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.preserve_deletes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.two_write_queues: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.manual_wal_flush: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.atomic_flush: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.log_readahead_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.best_efforts_recovery: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_data_in_errors: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.db_host_id: __hostname__ 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_background_jobs: 2 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_background_compactions: -1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_subcompactions: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_total_wal_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_open_files: -1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bytes_per_sync: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_background_flushes: -1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Compression algorithms supported: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kZSTD supported: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kXpressCompression supported: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kLZ4Compression supported: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kBZip2Compression supported: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kZlibCompression supported: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kSnappyCompression supported: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /root/mon-store/store.db/MANIFEST-000081 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.merge_operator: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_filter: None 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_filter_factory: None 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.sst_partitioner_factory: None 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56017d234f80) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 0 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 index_shortening: 1 data_block_hash_table_util_ratio: 0.750000 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x56017d22f610 block_cache_name: BinnedLRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 4 enable_index_compression: 1 block_align: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression: NoCompression 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.num_levels: 7 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.level: 32767 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.parallel_threads: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.enabled: false 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.arena_block_size: 4194304 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.table_properties_collectors: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.inplace_update_support: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bloom_locality: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_successive_merges: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.force_consistency_checks: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.ttl: 2592000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.enable_blob_files: false 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.min_blob_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.blob_file_size: 268435456 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.blob_compression_type: NoCompression 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.enable_blob_garbage_collection: false 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-11-17T12:26:24.156+0200 7f482b393600 4 rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/root/mon-store/store.db/MANIFEST-000081 succeeded,manifest_file_number is 81, next_file_number is 83, last_sequence is 21183, log_number is 77,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-17T12:26:24.156+0200 7f482b393600 4 rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 77 2023-11-17T12:26:24.156+0200 7f482b393600 4 rocksdb: [db/version_set.cc:4082] Creating manifest 85 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784162798, "job": 1, "event": "recovery_started", "wal_files": [82]} 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #82 mode 2 2023-11-17T12:26:24.160+0200 7f482b393600 3 rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784163944, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 86, "file_size": 1266, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 238, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 72, "raw_average_key_size": 24, "raw_value_size": 148, "raw_average_value_size": 49, "num_data_blocks": 1, "num_entries": 3, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700216784, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "53025a24-2059-43e1-a0f7-a87a28e33d38", "db_session_id": "OS2T69IQ02SU5OKHBI40"}} 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: [db/version_set.cc:4082] Creating manifest 87 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784166273, "job": 1, "event": "recovery_finished"} 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: [db/column_family.cc:983] [default] Increasing compaction threads because we have 14 level-0 files 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000082.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.164+0200 7f482b393600 4 rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x56017d230700 2023-11-17T12:26:24.164+0200 7f482b393600 4 rocksdb: DB pointer 0x56017df56000 adding auth for 'client.admin': auth(key=AQCsdUViHYjTGBAAf7/1KYZjb0h3x3EOywqbbQ==) with caps({mds=allow *,mgr=allow *,mon=allow *,osd=allow *}) 2023-11-17T12:26:24.164+0200 7f482a349700 4 rocksdb: [db/compaction/compaction_job.cc:1881] [default] [JOB 3] Compacting 14@0 files to L6, score 3.50 2023-11-17T12:26:24.164+0200 7f482a349700 4 rocksdb: [db/compaction/compaction_job.cc:1887] [default] Compaction start summary: Base version 3 Base level 0, inputs: [86(1266B) 80(1266B) 74(1267B) 68(1267B) 62(1266B) 56(1265B) 50(1265B) 44(1265B) 38(1265B) 32(1266B) 26(1265B) 20(1265B) 14(283KB) 8(7387KB)] 2023-11-17T12:26:24.164+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784169200, "job": 3, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [86, 80, 74, 68, 62, 56, 50, 44, 38, 32, 26, 20, 14, 8], "score": 3.5, "input_data_size": 7870219} 2023-11-17T12:26:24.164+0200 7f4822339700 4 rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2023-11-17T12:26:24.164+0200 7f4822339700 4 rocksdb: [db/db_impl/db_impl.cc:903] ** DB Stats ** Uptime(secs): 0.0 total, 0.0 interval Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Sum 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Sum 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.10 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** 2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb: [db/compaction/compaction_job.cc:1516] [default] [JOB 3] Generated table #91: 1366 keys, 7566988 bytes 2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784213586, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 91, "file_size": 7566988, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 7541895, "index_size": 20610, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3525, "raw_key_size": 29308, "raw_average_key_size": 21, "raw_value_size": 7503048, "raw_average_value_size": 5492, "num_data_blocks": 764, "num_entries": 1366, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700216681, "oldest_key_time": 0, "file_creation_time": 1700216784, "db_id": "53025a24-2059-43e1-a0f7-a87a28e33d38", "db_session_id": "OS2T69IQ02SU5OKHBI40"}} 2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb: [db/compaction/compaction_job.cc:1594] [default] [JOB 3] Compacted 14@0 files to L6 => 7566988 bytes 2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb: [db/version_set.cc:3457] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: (Original Log Time 2023/11/17-12:26:24.215298) [db/compaction/compaction_job.cc:812] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 177.1 rd, 170.3 wr, level 6, files in(14, 0) out(1) MB in(7.5, 0.0) out(7.2), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 19842, records dropped: 18476 output_compression: NoCompression 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: (Original Log Time 2023/11/17-12:26:24.215314) EVENT_LOG_v1 {"time_micros": 1700216784215306, "job": 3, "event": "compaction_finished", "compaction_time_micros": 44437, "compaction_time_cpu_micros": 40923, "output_level": 6, "num_output_files": 1, "total_output_size": 7566988, "num_input_records": 19842, "num_output_records": 1366, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000086.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215520, "job": 3, "event": "table_file_deletion", "file_number": 86} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000080.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215570, "job": 3, "event": "table_file_deletion", "file_number": 80} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000074.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215603, "job": 3, "event": "table_file_deletion", "file_number": 74} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000068.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215641, "job": 3, "event": "table_file_deletion", "file_number": 68} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000062.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215672, "job": 3, "event": "table_file_deletion", "file_number": 62} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000056.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215708, "job": 3, "event": "table_file_deletion", "file_number": 56} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000050.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215739, "job": 3, "event": "table_file_deletion", "file_number": 50} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000044.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215772, "job": 3, "event": "table_file_deletion", "file_number": 44} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000038.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215804, "job": 3, "event": "table_file_deletion", "file_number": 38} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000032.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215831, "job": 3, "event": "table_file_deletion", "file_number": 32} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000026.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215858, "job": 3, "event": "table_file_deletion", "file_number": 26} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000020.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215888, "job": 3, "event": "table_file_deletion", "file_number": 20} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000014.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215952, "job": 3, "event": "table_file_deletion", "file_number": 14} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000008.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784216804, "job": 3, "event": "table_file_deletion", "file_number": 8} update_mkfs generating seed initial monmap epoch 0 fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37 last_changed 2023-11-17T12:26:24.222814+0200 created 2023-11-17T12:26:24.222814+0200 min_mon_release 0 (unknown) election_strategy: 1 0: [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] mon.node01 1: [v2:10.40.99.12:3300/0,v1:10.40.99.12:6789/0] mon.node02 2: [v2:10.40.99.13:3300/0,v1:10.40.99.13:6789/0] mon.node03 2023-11-17T12:26:24.220+0200 7f482b393600 4 rocksdb: [db/db_impl/db_impl.cc:446] Shutdown: canceling all background work 2023-11-17T12:26:24.220+0200 7f482b393600 4 rocksdb: [db/db_impl/db_impl.cc:625] Shutdown complete Then we copied the /root/mon-store/store.db folder across on all 3 nodes and tried to start ceph.target service again. The output on node01 is below: d31781fa6b4c quay.io/ceph/ceph "/usr/bin/ceph-mds -..." 55 minutes ago Up 55 minutes ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mds-storage-node01-cjrvjc e385c32651d2 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-10 904f522c4cb5 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-0 033edf99a98e quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-4 70344a6e87a0 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-8 905b782aedcf quay.io/prometheus/prometheus:v2.43.0 "/bin/prometheus --c..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-prometheus-node01 ff191654eb3e quay.io/prometheus/node-exporter:v1.5.0 "/bin/node_exporter ..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-node-exporter-node01 459c46f4bdb7 quay.io/ceph/ceph "/usr/bin/ceph-mgr -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mgr-node01-xlciyx cacfe8abcbbf quay.io/ceph/ceph "/usr/bin/ceph-crash..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-crash-node01 e216ef2af166 quay.io/prometheus/alertmanager:v0.25.0 "/bin/alertmanager -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-alertmanager-node01 d3238b2285d1 quay.io/ceph/ceph-grafana:9.4.7 "/bin/sh -c 'grafana..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-grafana-node01 The output on node02 is below: 2aec62685dee quay.io/ceph/ceph "/usr/bin/ceph-mds -..." 54 minutes ago Up 54 minutes ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mds-storage-node02-lyudbp 249b04f32f8c quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-5 a2c96f56b517 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-2 87496d374a29 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-12 55fe47765917 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-9 76171e25dbde quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-7 220472e8c1bf quay.io/ceph/ceph "/usr/bin/ceph-mgr -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mgr-node02-gudauu 0c783e73e543 quay.io/prometheus/node-exporter:v1.5.0 "/bin/node_exporter ..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-node-exporter-node02 4e638003fa2e quay.io/ceph/ceph "/usr/bin/ceph-crash..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-crash-node02 42719d5cfdbf quay.io/ceph/ceph "/usr/bin/ceph-mon -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mon-node02 The output on node03 is below: 7e5879dce643 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-11 d53996ff33b9 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-3 e1ac5a8b87d3 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-1 f4cda871218d quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-13 969e670dc47c quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-6 a49e91a7bb8e quay.io/prometheus/node-exporter:v1.5.0 "/bin/node_exporter ..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-node-exporter-node03 835c3893a3f4 quay.io/ceph/ceph "/usr/bin/ceph-crash..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-crash-node03 bfa6f5b989ea quay.io/ceph/ceph "/usr/bin/ceph-mon -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mon-node03 # ceph -s (output below): cluster: id: be4304e4-b0d5-11ec-8c6a-2965d4229f37 health: HEALTH_ERR 20 stray daemon(s) not managed by cephadm 3 stray host(s) with 20 daemon(s) not managed by cephadm 1/3 mons down, quorum node02,node03 1/523510 objects unfound (0.000%) 3 nearfull osd(s) 1 osds exist in the crush map but not in the osdmap Low space hindering backfill (add storage if this doesn't resolve itself): 20 pgs backfill_toofull Possible data damage: 1 pg recovery_unfound Degraded data redundancy: 74666/1570530 objects degraded (4.754%), 21 pgs degraded, 21 pgs undersized 3 pool(s) nearfull services: mon: 3 daemons, quorum node02,node03 (age 2d), out of quorum: node01 mgr: node01.xlciyx(active, since 2d), standbys: node02.gudauu osd: 14 osds: 14 up (since 2d), 14 in (since 3d); 21 remapped pgs data: pools: 3 pools, 161 pgs objects: 523.51k objects, 299 GiB usage: 1014 GiB used, 836 GiB / 1.8 TiB avail pgs: 74666/1570530 objects degraded (4.754%) 1/523510 objects unfound (0.000%) 140 active+clean 20 active+undersized+degraded+remapped+backfill_toofull 1. active+recovery_unfound+undersized+degraded+remapped # ceph fs ls (output below): No filesystems enabled It looks like that we have a problem with the orchestrator now (we've lost cephadm orchestrator) and we also cannot see the filesystem. May you please assist since we are not able to mount the filesystem ? Thank you, Manolis Daramas Under the General Data Protection Regulation (GDPR) (EU) 2016/679, Motivian as Data Controller has a legal duty to protect any information collected from you via email. Information contained in this email and any attachments may be privileged or confidential and intended for the exclusive use of the original recipient. If you have received this email by mistake, please advise the sender immediately and delete the email, including emptying your deleted email box. Information included in this email is reserved to named addressee's eyes only. You may not share this message or any of its attachments to anyone. Please note that as the recipient, it is your responsibility to check the email for malicious software. Motivian puts the security of the client at a high priority. Therefore, we have put efforts into ensuring that the message is error and virus-free. Unfortunately, full security of the email cannot be ensured as, despite our efforts, the data included in emails could be infected, intercepted, or corrupted. Therefore, the recipient should check the email for threats with proper software, as the sender does not accept liability for any damage inflicted by viewing the content of this email.

5 months, 2 weeks

2
5
0 0

MDS stuck in up:rejoin

by Eric Tittley

Hi all, For about a week our CephFS has experienced issues with its MDS. Currently the MDS is stuck in "up:rejoin" Issues become apparent when simple commands like "mv foo bar/" hung. I unmounted CephFS offline on the clients, evicted those remaining, and then issued ceph config set mds.0 mds_wipe_sessions true ceph config set mds.1 mds_wipe_sessions true which allowed me to delete the hung requests. I've lost the exact commands I used, but something like rados -p cephfs_metadata ls | grep mds rados rm -p cephfs_metadata mds0_openfiles.0 etc This allowed the MDS to get to "up:rejoin" where it has been stuck ever since which is getting on five days. # ceph mds stat cephfs:1/1 {0=cephfs.ceph00.uvlkrw=up:rejoin} 2 up:standby root@ceph00:/var/log/ceph/a614303a-5eb5-11ed-b492-011f01e12c9a# ceph -s cluster: id: a614303a-5eb5-11ed-b492-011f01e12c9a health: HEALTH_WARN 1 filesystem is degraded 1 pgs not deep-scrubbed in time 2 pool(s) do not have an application enabled 1 daemons have recently crashed services: mon: 3 daemons, quorum ceph00,ceph01,ceph02 (age 57m) mgr: ceph01.lvdgyr(active, since 2h), standbys: ceph00.gpwpgs mds: 1/1 daemons up, 2 standby osd: 91 osds: 90 up (since 78m), 90 in (since 112m) data: volumes: 0/1 healthy, 1 recovering pools: 5 pools, 1539 pgs objects: 138.83M objects, 485 TiB usage: 971 TiB used, 348 TiB / 1.3 PiB avail pgs: 1527 active+clean 12 active+clean+scrubbing+deep io: client: 3.1 MiB/s rd, 3.16k op/s rd, 0 op/s wr # ceph --version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) I've tried failing the MDS so it switches. Rebooted a couple of times. I've added more OSDs to the metadata pool and took one out as I thought it might be a bad metadata OSD (The "recently crashed" daemon). The error logs are full of (prefix to all are: Nov 27 14:02:44 ceph00 bash[2145]: debug 2023-11-27T14:02:44.619+0000 7f74e845e700 1 -- [v2:192.168.1.128:6800/2157301677,v1:192.168.1.128:6801/2157301677] --> [v2:192.168.1.133:6896/4289132926,v1:192.168.1.133:6897/4289132926] ) crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_message enqueueing message m=0x559be00adc00 type=42 osd_op(mds.0.36244:8142873 3.ff 3:ff5b34d6:::1.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e32465) v8 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).write_message sending message m=0x559be00adc00 seq=8142643 osd_op(mds.0.36244:8142873 3.ff 3:ff5b34d6:::1.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e32465) v8 crc :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message got 154 + 0 + 30 byte message. envelope type=43 src osd.89 off 0 crc :-1 s=READ_MESSAGE_COMPLETE pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message received message m=0x559be01f4480 seq=8142643 from=osd.89 type=43 osd_op_reply(8142873 1.00000000 [getxattr (30) out=30b] v0'0 uv560123 ondisk = 0) v8 osd_op_reply(8142873 1.00000000 [getxattr (30) out=30b] v0'0 uv560123 ondisk = 0) v8 ==== 154+0+30 (crc 0 0 0) 0x559be01f4480 con 0x559be00ad800 osd_op(unknown.0.36244:8142874 3.ff 3:ff5b34d6:::1.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e32465) v8 -- 0x559be2caec00 con 0x559be00ad800 Repeating multiple times a second (and filling /var) Prior to taking one of the cephfs_metadata OSDs offline, these came from communications from ceph00 to the node hosting the suspected bad OSD. Now they are between ceph00 and the host of the replacement metadata OSD. Does anyone have any suggestion on how to get the MDS to switch from "up:rejoin" to "up:active"? Is there any way to debug this, to determine what issue really is? I'm unable to interpret the debug log. Cheers, Eric ________________________________________________________ Dr Eric Tittley Research Computing Officer www.roe.ac.uk/~ert<http://www.roe.ac.uk/~ert> Institute for Astronomy Royal Observatory, Edinburgh The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

5 months, 2 weeks

2
5
0 0

Re: ceph-users Digest, Vol 114, Issue 14

by duluxoz

Hi Zitat, I'm confused - doesn't k4 m2 mean that you can loose any 2 out of the 6 osds? Cheers Dulux-Oz On 05/12/2023 20:02, ceph-users-request(a)ceph.io wrote: > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. Re: EC Profiles & DR (David Rivera) > 2. Re: EC Profiles & DR (duluxoz) > 3. Re: EC Profiles & DR (Eugen Block) > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s

5 months, 2 weeks

3
2
0 0

mds slow request with “failed to authpin, subtree is being exported"

by zxcs

HI, Experts, we are using cephfs with 16.2.* with multi active mds, and recently, we have two nodes mount with ceph-fuse due to the old os system. and one nodes run a python script with `glob.glob(path)`, and another client doing `cp` operation on the same path. then we see some log about `mds slow request`, and logs complain “failed to authpin, subtree is being exported" then need to restart mds, our question is, does there any dead lock? how can we avoid this and how to fix it without restart mds(it will influence other users) ? Thanks a ton! xz

5 months, 2 weeks

6
11
0 0

the image used size becomes 0 after export/import with snapshot

by Tony Liu

Hi, I have an image with a snapshot and some changes after snapshot. ``` $ rbd du backup/f0408e1e-06b6-437b-a2b5-70e3751d0a26 NAME PROVISIONED USED f0408e1e-06b6-437b-a2b5-70e3751d0a26@snapshot-eb085877-7557-4620-9c01-c5587b857029 10 GiB 2.4 GiB f0408e1e-06b6-437b-a2b5-70e3751d0a26 10 GiB 2.4 GiB <TOTAL> 10 GiB 4.8 GiB ``` If there is no changes after snapshot, the image line will show 0 used. I did export and import. ``` $ rbd export --export-format 2 backup/f0408e1e-06b6-437b-a2b5-70e3751d0a26 - | rbd import --export-format 2 - backup/test Exporting image: 100% complete...done. Importing image: 100% complete...done. ``` When check the imported image, the image line shows 0 used. ``` $ rbd du backup/test NAME PROVISIONED USED test@snapshot-eb085877-7557-4620-9c01-c5587b857029 10 GiB 2.4 GiB test 10 GiB 0 B <TOTAL> 10 GiB 2.4 GiB ``` Any clues how that happened? I'd expect the same du as the source. I tried another quick test. It works fine. ``` $ rbd create backup/test-src --size 10G $ sudo rbd map backup/test-src /dev/rbd0 $ echo "hello" | sudo tee /dev/rbd0 hello $ rbd du backup/test-src NAME PROVISIONED USED test-src 10 GiB 4 MiB $ rbd snap create backup/test-src@snap-1 Creating snap: 100% complete...done. $ rbd du backup/test-src NAME PROVISIONED USED test-src@snap-1 10 GiB 4 MiB test-src 10 GiB 0 B <TOTAL> 10 GiB 4 MiB $ echo "world" | sudo tee /dev/rbd0 world $ rbd du backup/test-src NAME PROVISIONED USED test-src@snap-1 10 GiB 4 MiB test-src 10 GiB 4 MiB <TOTAL> 10 GiB 8 MiB $ rbd export --export-format 2 backup/test-src - | rbd import --export-format 2 - backup/test-dst Exporting image: 100% complete...done. Importing image: 100% complete...done. $ rbd du backup/test-dst NAME PROVISIONED USED test-dst@snap-1 10 GiB 4 MiB test-dst 10 GiB 4 MiB <TOTAL> 10 GiB 8 MiB ``` Thanks! Tony

5 months, 2 weeks

2
2
0 0

Libvirt and Ceph: libvirtd tries to open random RBD images

by Jayanth Reddy

Hello Users, We're using libvirt with KVM and the orchestrator is Cloudstack. I raised the issue already at Cloudstack at https://github.com/apache/cloudstack/issues/8211 but appears to be at libvirtd. Did the same in libvirt ML at https://lists.libvirt.org/archives/list/users@lists.libvirt.org/thread/SA2I… but I'm now here looking for answers. Below is our environment & issue description: Ceph: v17.2.0 Pool: replicated Number of block images in this pool: more than 1250 # virsh pool-info c15508c7-5c2c-317f-aa2e-29f307771415 Name: c15508c7-5c2c-317f-aa2e-29f307771415 UUID: c15508c7-5c2c-317f-aa2e-29f307771415 State: running Persistent: no Autostart: no Capacity: 1.25 PiB Allocation: 489.52 TiB Available: 787.36 TiB # kvm --version QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.27) Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers # libvirtd --version libvirtd (libvirt) 6.0.0 It appears that one of our Cloudstack KVM clusters having 8 hosts is having the issue. We have HCI on these 8 hosts and there are around 700+ VMs running. But strange enough, there are these logs like below on hosts. Oct 25 13:38:11 hv-01 libvirtd[9464]: failed to open the RBD image '087bb114-448a-41d2-9f5d-6865b62eed15': No such file or directory Oct 25 20:35:22 hv-01 libvirtd[9464]: failed to open the RBD image 'ccc1168a-5ffa-4b6d-a953-8e0ac788ebc5': No such file or directory Oct 26 09:48:33 hv-01 libvirtd[9464]: failed to open the RBD image 'a3fe82f8-afc9-4604-b55e-91b676514a18': No such file or directory We've got DNS servers on which there is an`A` record resolving to all the IPv4 Addresses of 5 monitors and there have not been any issues with the DNS resolution. But the issue of "failed to open the RBD image 'ccc1168a-5ffa-4b6d-a953-8e0ac788ebc5': No such file or directory" gets more weird because the VM that is making use of that RBD image lets say "087bb114-448a-41d2-9f5d-6865b62eed15" is running on an altogether different host like "hv-06". On further inspection of that specific Virtual Machine, it has been running on that host "hv-06" for more than 4 months or so. Fortunately, the Virtual Machine has no issues and has been running since then. There are absolutely no issues with any of the Virtual Machines because of these warnings. From libvirtd mailing lists, one of the community members helped me understand that libvirt only tries to get the info of the images and doesn't open for reading or writing. All hosts where there is libvirtd tries doing the same. We manually did "virsh pool-refresh" which CloudStack itself takes care of at regular intervals and the warning messages still appear. Please help me find the cause and let me know if further information is needed. Thanks, Jayanth Reddy

5 months, 2 weeks

2
2
0 0

Nov/Dec Ceph Science Virtual User Group

by Kevin Hrpcek

Hey All, So I got busy and failed at getting an email out with a couple days notice for last week so let's meet up this week! We will be having a Ceph science/research/big cluster call on Wednesday December 6th. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research/big cluster environments (though anyone is welcome) where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. NOTE: The change to using Jitsi for the meeting. We are no longer using the bluejeans meeting links. The ceph calendar event does not yet reflect this and has the wrong day as well. Pad URL: https://pad.ceph.com/p/Ceph_Science_User_Group_20231206 Virtual event details: December 6th, 2023 15:00 UTC 4pm Central European 9am Central US Description: Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://meet.jit.si/ceph-science-wg Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS/TROPICS Space Science & Engineering Center University of Wisconsin-Madison

5 months, 2 weeks

1
0
0 0

2024

2023

2022

2021

2020

2019

ceph-users December 2023