- ceph-users - lists.ceph.io

New Ceph cluster- having issue with one monitor
by Robert W. Eckert 21 Apr '21

21 Apr '21

Hi, I have pieced together some pcs which I had been using to run a windows DFS cluster. the 3 servers all have 3 4Tb Hard Drives and 1 2Tb SSD, but they have different CPUs All of them are running RHEL8, and have 2.5 Gbps NICs in them. The install was with cephadm, and the ceph processes are all running under podman. Two of the servers - which are on old hardware, one a Core I5 2500, the other a J5005 embedded cpu are very stable, and have had no real issues. The other server an AMD 3600 on an X570 motherboard runs great, but the monitor goes down at least once a day. I have a script I run to copy from one of the other servers when it goes down, but I would prefer to just have it run. I don't know if it is tied together or not, but I also see OSD_SCRUB_ERRORS and PG_DAMAGED. The load on the cluster is light-I have some backups from my pcs running to it, a copy of all my photos, 3 private Minecraft servers running in podman containers, and an ark survival server also running in a podman container with a mount pointing to the ceph file system. Last night, I stopped the Minecraft and ark containers before I went to bed, and this morning, everything was clean. This afternoon, with no load on the filesystem, the monitor went down, and I have 8 scrub errors. I have not been able to find where the ceph monitor logs are going, I thought under either /var/log/ceph or /var/log/ceph/fe3a7cb0-69ca-11eb-8d45-c86000d08867 But the log files there are 0 bytes. Attached is the output from ceph-report. What can I try to improve logging for troubleshooting or get this one monitor stable? Thanks, Rob

1 0

MDS replay takes forever and cephfs is down
by Flemming Frandsen 21 Apr '21

21 Apr '21

I tried restarting an MDS server using: systemctl restart ceph-mds(a)ardmore.service This caused the standby server to enter replay state and the fs started hanging for several minutes. In a slight panic I restarted the other mds server, which was replaced by the standby server and it almost immediately entered resolve state. fs dump shows a seq number counting upwards very slowly for the replay'ing MDS server, I have no idea how far it needs to count: # ceph fs dump dumped fsmap epoch 1030314 e1030314 enable_multiple, ever_enabled_multiple: 0,0 compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in sep arate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} legacy client fscid: 1 Filesystem 'cephfs' (1) fs_name cephfs epoch 1030314 flags 12 created 2019-09-09 13:08:26.830927 modified 2021-04-21 14:04:14.672440 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 min_compat_client -1 (unspecified) last_failure 0 last_failure_osd_epoch 13610 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in sep arate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 2 in 0,1 up {0=10398946,1=10404857} failed damaged stopped data_pools [1] metadata_pool 2 inline_data disabled balancer standby_count_wanted 1 [mds.dalmore{0:10398946} state up:replay seq 215 addr [v2: 10.0.37.222:6800/2681188441,v1:10.0.37.222:6801/2681188441]] [mds.cragganmore{1:10404857} state up:resolve seq 201 addr [v2: 10.0.37.221:6800/871249119,v1:10.0.37.221:6801/871249119]] Standby daemons: [mds.ardmore{-1:10408652} state up:standby seq 2 addr [v2: 10.0.37.223:6800/4096598841,v1:10.0.37.223:6801/4096598841]] Earlier today we added a new OSD host with 12 new OSDs and backfilling is proceeding as expected: cluster: id: e2007417-a346-4af7-8aa9-4ce8f0d73661 health: HEALTH_WARN 1 filesystem is degraded 1 MDSs behind on trimming services: mon: 3 daemons, quorum cragganmore,dalmore,ardmore (age 5w) mgr: ardmore(active, since 2w), standbys: dalmore, cragganmore mds: cephfs:2/2 {0=dalmore=up:replay,1=cragganmore=up:resolve} 1 up:standby osd: 69 osds: 69 up (since 102m), 69 in (since 102m); 443 remapped pgs rgw: 9 daemons active (ardmore.rgw0, ardmore.rgw1, ardmore.rgw2, cragganmore.rgw0, cragganmore.rgw1, cragganmore.rgw2, dalmore.rgw0, dalmore.rgw1, dalmore.rgw2) task status: scrub status: mds.cragganmore: idle mds.dalmore: idle data: pools: 13 pools, 1440 pgs objects: 50.57M objects, 9.0 TiB usage: 34 TiB used, 37 TiB / 71 TiB avail pgs: 30195420/151707033 objects misplaced (19.904%) 997 active+clean 431 active+remapped+backfill_wait 12 active+remapped+backfilling io: client: 65 MiB/s rd, 206 KiB/s wr, 17 op/s rd, 8 op/s wr recovery: 5.5 MiB/s, 23 objects/s progress: Rebalancing after osd.62 marked in [======================........] Rebalancing after osd.67 marked in [===========...................] Rebalancing after osd.68 marked in [============..................] Rebalancing after osd.64 marked in [=====================.........] Rebalancing after osd.60 marked in [====================..........] Rebalancing after osd.66 marked in [=============.................] Rebalancing after osd.63 marked in [=====================.........] Rebalancing after osd.61 marked in [======================........] Rebalancing after osd.59 marked in [======================........] Rebalancing after osd.58 marked in [========================......] Rebalancing after osd.57 marked in [===========================...] Rebalancing after osd.65 marked in [==================............] It seems we're running a mix of versions: ceph versions { "mon": { "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.19 (bb796b9b5bab9463106022eef406373182465d11) nautilus (stable)": 3 }, "osd": { "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 57, "ceph version 14.2.20 (36274af6eb7f2a5055f2d53ad448f2694e9046a0) nautilus (stable)": 12 }, "mds": { "ceph version 14.2.19 (bb796b9b5bab9463106022eef406373182465d11) nautilus (stable)": 3 }, "rgw": { "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 9 }, "overall": { "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 69, "ceph version 14.2.19 (bb796b9b5bab9463106022eef406373182465d11) nautilus (stable)": 6, "ceph version 14.2.20 (36274af6eb7f2a5055f2d53ad448f2694e9046a0) nautilus (stable)": 12 } } Any hints will be greatly appreciated. -- Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/

2 2

Getting `InvalidInput` when trying to create a notification topic with Kafka endpoint
by Szabo, Istvan (Agoda) 21 Apr '21

21 Apr '21

Hi Ceph Users, Here is the latest request I tried but still not working curl -v -H 'Date: Tue, 20 Apr 2021 16:05:47 +0000' -H 'Authorization: AWS <accessid>:<signature>' -L -H 'content-type: application/x-www-form-urlencoded' -k -X POST https://servername -d Action=CreateTopic&Name=test-ceph-event-replication&Attributes.entry.8.key=push-endpoint&Attributes.entry.8.value=kafka://<username>:<password>@servername2:9093&Attributes.entry.5.key=use-ssl&Attributes.entry.5.value=true And the response I get is still Invalid Input <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidInput</Code><RequestId>tx000000000000007993081-00607efbdd-1c7e96b-hkg</RequestId><HostId>1c7e96b-hkg-data</HostId></Error> Can someone please help with this? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Swift Stat Timeout
by Dylan Griff 21 Apr '21

21 Apr '21

Hey folks! We have a user with ~1900 buckets in our RGW service and running this stat command results in a timeout for them: swift -A https://<URL>:443/auth/1.0 -U <UID> -K <KEY> stat Running the same command, but specifiying one of their buckets, returns promptly. Running the command for a different user with minimal buckets returns promptly as well. Turning up debug logging to 20 for rgw resulted in a great deal of logs showing: 20 reading from default.rgw.meta:root:.bucket.meta.<BUCKET-ID> 20 get_system_obj_state: rctx=0x559b32a6b570 obj=default.rgw.meta:root:.bucket.meta.<BUCKET-ID> state=0x559b32c37e40 s->prefetch_data=0 10 cache get: name=default.rgw.meta+root+.bucket.meta.<BUCKET-ID> : hit (requested=0x16, cached=0x17) 20 get_system_obj_state: s->obj_tag was set empty 10 cache get: name=default.rgw.meta+root+.bucket.meta.<BUCKET-ID> : hit (requested=0x11, cached=0x17) Which looks like to me it is iterating getting the state of all their stuff. My question: is ~1900 an unreasonable amount of buckets such that we should expect to see this full account 'stat' command timeout? Or should I be expecting it to return promptly still? Thanks! Cheers, Dylan -- Dylan Griff Senior System Administrator CLE D063 RCS - Systems - University of Victoria

1 2

HBA vs caching Raid controller
by Nico Schottelius 21 Apr '21

21 Apr '21

Good evening, I've to tackle an old, probably recurring topic: HBAs vs. Raid controllers. While generally speaking many people in the ceph field recommend to go with HBAs, it seems in our infrastructure the only server we phased in with an HBA vs. raid controller is actually doing worse in terms of latency. For the background: we have many Perc H800+MD1200 [1] systems running with 10TB HDDs (raid0, read ahead, writeback cache). One server has LSI SAS3008 [0] instead of the Perc H800, which comes with 512MB RAM + BBU. On most servers latencies are around 4-12ms (average 6ms), on the system with the LSI controller we see 20-60ms (average 30ms) latency. Now, my question is, are we doing some inherently wrong with the SAS3008 or does in fact the cache help to possible reduce seek time? We were considering to move more towards LSI HBAs to reduce maintenance effort, however if we have a factor of 5 in latency between the two different systems, it might be better to stay on the H800 path for disks. Any input/experiences appreciated. Best regards, Nico [0] 05:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02) Subsystem: Dell 12Gbps HBA Kernel driver in use: mpt3sas Kernel modules: mpt3sas [1] 08:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 05) Subsystem: Dell PERC H800 Adapter Kernel driver in use: megaraid_sas Kernel modules: megaraid_sas -- Sustainable and modern Infrastructures by ungleich.ch

5 12

ceph orch upgrade fails when pulling container image
by Robert Sander 21 Apr '21

21 Apr '21

Hi, # docker pull ceph/ceph:v16.2.1 Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit How do I update a Ceph cluster in this situation? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 2

Single OSD crash/restarting during scrub operation on specific PG
by Mark Johnson 21 Apr '21

21 Apr '21

We've recently recovered from a bit of a disaster where we had some power outages (combination of data centre power maintenance and us not having our redundant power supplies connected to the correct redundant power circuits - lesson learnt). We ended up with one OSD that wouldn't start - seems like perhaps filesystem corruption as an FSCK found and fixed a couple of errors but the OSD still wouldn't start so we ended up marking it lost and letting data backfill to other OSDs. Ended up with a handful of 'incomplete' or 'incomplete/down' pgs which was causing radosgw to stop accepting connections. Found a useful blog post from somebody that got us to a point of using the ceph-objectstore-tool to determine the correct remaining copy and mark pgs as complete. Backfill operations then wrote out pgs to different OSDs and the cluster returned to a health OK state, and radosgw started working normally. At least, that's what I thought. Now, I'm occasionally seeing one OSD crashing every now and then, sometimes after a few hours, sometimes after only 10 minutes. It always starts itself up again and the queued up backfills cancel and the cluster returns to OK until the next time. It's always the same OSD and going through the logs just now, it seems to always occur when performing a scrub operation on the same pg (although, I haven't checked every single instance to be completely sure). We're running Jewel (yes, I know it's old but we can't upgrade). Here's the last couple of lines from the OSD log when it crashes on two different occasions - I've used the hex code from the "Caught signal" line to reference events that match that same code in both instances. Looks roughly the same on both occasions in that it's always the same pg, however the last object shown in the log prior to the crash always seems to be different. -91> 2021-04-21 02:37:32.118290 7fed046e6700 5 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: false, divergent_priors: 0, writeout_from: 3110'3174946, trimmed: -90> 2021-04-21 02:37:32.118380 7fed046e6700 5 -- op tracker -- seq: 2219, time: 2021-04-21 02:37:32.118379, event: commit_queued_for_journal_write, op: osd_repop(client.3095191.0:19172420 30.65 30:a78d321d:::71b98f4a-2ef6-466a-bb1d-eb477b317c78.604098.18432929_1149756827.ogg:head v 3110'3174946) -1> 2021-04-21 02:37:32.748831 7fed046e6700 5 -- op tracker -- seq: 2221, time: 2021-04-21 02:37:32.748830, event: reached_pg, op: replica scrub(pg: 30.65,from:0'0,to:2923'3171906,epoch:3110,start:30:a63a08df:::71b98f4a-2ef6-466a-bb1d-eb477b317c78.577500.22382834__shadow_.cqXthfu1litKEyNZ53I_voGLwuhonVX_1:0,end:30:a63a13fd:::71b98f4a-2ef6-466a-bb1d-eb477b317c78.604098.26065941_1234799607.gsm:0,chunky:1,deep:1,seed:4294967295,version:6) 0> 2021-04-21 02:37:32.797826 7fed046e6700 -1 os/filestore/FileStore.cc: In function 'int FileStore::lfn_find(const ghobject_t&, const Index&, IndexedPath*)' thread 7fed046e6700 time 2021-04-21 02:37:32.790356 2021-04-21 02:37:32.859265 7fed046e6700 -1 *** Caught signal (Aborted) ** in thread 7fed046e6700 thread_name:tp_osd_tp 0> 2021-04-21 02:37:32.859265 7fed046e6700 -1 *** Caught signal (Aborted) ** in thread 7fed046e6700 thread_name:tp_osd_tp -17> 2021-04-21 03:55:09.090430 7f43382c7700 5 -- op tracker -- seq: 1596, time: 2021-04-21 03:55:09.090430, event: done, op: replica scrub(pg: 30.65,from:0'0,to:2979'3174652,epoch:3122,start:30:a639eb4a:::71b98f4a-2ef6-466a-bb1d-eb477b317c78.577500.17157485_1132337117.ogg:0,end:30:a639f7f1:::71b98f4a-2ef6-466a-bb1d-eb477b317c78.604098.21768594_1188151257.ogg:0,chunky:1,deep:1,seed:4294967295,version:6) -5> 2021-04-21 03:55:09.777503 7f43382c7700 5 -- op tracker -- seq: 1598, time: 2021-04-21 03:55:09.777476, event: reached_pg, op: replica scrub(pg: 30.65,from:0'0,to:2929'3172006,epoch:3122,start:30:a63a047c:::71b98f4a-2ef6-466a-bb1d-eb477b317c78.577500.18649542__shadow_.tKOWzKIibnLhX3Bu32FiiuG0FH1lIl4_1:0,end:30:a63a0ea6:::71b98f4a-2ef6-466a-bb1d-eb477b317c78.577500.14411965_1101425097.gsm:0,chunky:1,deep:1,seed:4294967295,version:6) 0> 2021-04-21 03:55:10.089217 7f43382c7700 -1 os/filestore/FileStore.cc: In function 'int FileStore::lfn_find(const ghobject_t&, const Index&, IndexedPath*)' thread 7f43382c7700 time 2021-04-21 03:55:10.081373 2021-04-21 03:55:10.157208 7f43382c7700 -1 *** Caught signal (Aborted) ** in thread 7f43382c7700 thread_name:tp_osd_tp 0> 2021-04-21 03:55:10.157208 7f43382c7700 -1 *** Caught signal (Aborted) ** in thread 7f43382c7700 thread_name:tp_osd_tp Any ideas what to do next? Regards, Mark Johnson

1 0

Issues upgrading to 16.2.1
by Radoslav Milanov 20 Apr '21

20 Apr '21

Hello Tried cephadm upgrade form 16.2.0 to 16.2.1 Managers were updated first then process halted on first monitor being upgraded. The monitor fails to start: root@host3:/var/lib/ceph/c8ee2878-9d54-11eb-bbca-1c34da4b9fb6/mon.host3# /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-mon --privileged --group-add=disk --init --name ceph-c8ee2878-9d54-11eb-bbca-1c34da4b9fb6-mon.host3 -e CONTAINER_IMAGE=ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a -e NODE_NAME=host3 -e CEPH_USE_RANDOM_NONCE=1 -v /var/run/ceph/c8ee2878-9d54-11eb-bbca-1c34da4b9fb6:/var/run/ceph:z -v /var/log/ceph/c8ee2878-9d54-11eb-bbca-1c34da4b9fb6:/var/log/ceph:z -v /var/lib/ceph/c8ee2878-9d54-11eb-bbca-1c34da4b9fb6/crash:/var/lib/ceph/crash:z -v /var/lib/ceph/c8ee2878-9d54-11eb-bbca-1c34da4b9fb6/mon.host3:/var/lib/ceph/mon/ceph-host3:z -v /var/lib/ceph/c8ee2878-9d54-11eb-bbca-1c34da4b9fb6/mon.host3/config:/etc/ceph/ceph.conf:z -v /dev:/dev -v /run/udev:/run/udev ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a -n mon.host3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true '--default-log-stderr-prefix=debug ' --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true debug 2021-04-20T14:06:19.437+0000 7f11e080a700 0 set uid:gid to 167:167 (ceph:ceph) debug 2021-04-20T14:06:19.437+0000 7f11e080a700 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-mon, pid 7 debug 2021-04-20T14:06:19.437+0000 7f11e080a700 0 pidfile_write: ignore empty --pid-file debug 2021-04-20T14:06:19.441+0000 7f11e080a700 0 load: jerasure load: lrc load: isa debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: RocksDB version: 6.8.1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Compile date Mar 30 2021 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: DB SUMMARY debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: CURRENT file: CURRENT debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: IDENTITY file: IDENTITY debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: MANIFEST file: MANIFEST-000152 size: 221 Bytes debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-host3/store.db dir, Total Num: 2, files: 000137.sst 000139.sst debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-host3/store.db: 000153.log size: 0 ; debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.error_if_exists: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.create_if_missing: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.paranoid_checks: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.env: 0x5641244cf1c0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.fs: Posix File System debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.info_log: 0x564126753220 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_file_opening_threads: 16 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.statistics: (nil) debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.use_fsync: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_log_file_size: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_manifest_file_size: 1073741824 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.log_file_time_to_roll: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.keep_log_file_num: 1000 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.recycle_log_file_num: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.allow_fallocate: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.allow_mmap_reads: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.allow_mmap_writes: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.use_direct_reads: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.create_missing_column_families: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.db_log_dir: debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-host3/store.db debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.table_cache_numshardbits: 6 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_subcompactions: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_background_flushes: -1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.WAL_ttl_seconds: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.WAL_size_limit_MB: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.manifest_preallocation_size: 4194304 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.is_fd_close_on_exec: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.advise_random_on_open: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.db_write_buffer_size: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.write_buffer_manager: 0x5641267e9e90 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.access_hint_on_compaction_start: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.use_adaptive_mutex: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.rate_limiter: (nil) debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.wal_recovery_mode: 2 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.enable_thread_tracking: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.enable_pipelined_write: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.unordered_write: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.write_thread_max_yield_usec: 100 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.row_cache: None debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.wal_filter: None debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.avoid_flush_during_recovery: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.allow_ingest_behind: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.preserve_deletes: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.two_write_queues: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.manual_wal_flush: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.atomic_flush: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.persist_stats_to_disk: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.write_dbid_to_manifest: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.log_readahead_size: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.sst_file_checksum_func: Unknown debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_background_jobs: 2 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_background_compactions: -1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.delayed_write_rate : 16777216 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_total_wal_size: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.stats_dump_period_sec: 600 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.stats_persist_period_sec: 600 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.stats_history_buffer_size: 1048576 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_open_files: -1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.bytes_per_sync: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.wal_bytes_per_sync: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.strict_bytes_per_sync: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_readahead_size: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Compression algorithms supported: debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: kZSTDNotFinalCompression supported: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: kZSTD supported: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: kXpressCompression supported: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: kLZ4HCCompression supported: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: kLZ4Compression supported: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: kBZip2Compression supported: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: kZlibCompression supported: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: kSnappyCompression supported: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Fast CRC32 supported: Supported on x86 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-host3/store.db/MANIFEST-000152 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.merge_operator: debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_filter: None debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_filter_factory: None debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.memtable_factory: SkipListFactory debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.table_factory: BlockBasedTable debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5641266f60d8) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 0 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 index_shortening: 1 data_block_hash_table_util_ratio: 0.750000 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x56412672ef10 block_cache_name: BinnedLRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.write_buffer_size: 33554432 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_write_buffer_number: 2 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compression: NoCompression debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.bottommost_compression: Disabled debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.prefix_extractor: nullptr debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.num_levels: 7 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.bottommost_compression_opts.enabled: false debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compression_opts.window_bits: -14 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compression_opts.level: 32767 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compression_opts.strategy: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compression_opts.enabled: false debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.level0_stop_writes_trigger: 36 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.target_file_size_base: 67108864 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.target_file_size_multiplier: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_compaction_bytes: 1677721600 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.arena_block_size: 4194304 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.disable_auto_compactions: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.table_properties_collectors: debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.inplace_update_support: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.inplace_update_num_locks: 10000 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.memtable_whole_key_filtering: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.memtable_huge_page_size: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.bloom_locality: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_successive_merges: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.optimize_filters_for_hits: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.paranoid_file_checks: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.force_consistency_checks: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.report_bg_io_stats: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.ttl: 2592000 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.periodic_compaction_seconds: 0 debug 2021-04-20T14:06:19.445+0000 7f11e080a700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-host3/store.db/MANIFEST-000152 succeeded,manifest_file_number is 152, next_file_number is 154, last_sequence is 73904, log_number is 151,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 debug 2021-04-20T14:06:19.445+0000 7f11e080a700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 151 debug 2021-04-20T14:06:19.445+0000 7f11e080a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618927579447606, "job": 1, "event": "recovery_started", "log_files": [153]} debug 2021-04-20T14:06:19.445+0000 7f11e080a700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #153 mode 2 debug 2021-04-20T14:06:19.445+0000 7f11e080a700 4 rocksdb: [version_set.cc:3826] Creating manifest 155 debug 2021-04-20T14:06:19.445+0000 7f11e080a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618927579449426, "job": 1, "event": "recovery_finished"} debug 2021-04-20T14:06:19.445+0000 7f11e080a700 4 rocksdb: DB pointer 0x5641267f3800 debug 2021-04-20T14:06:19.449+0000 7f11c7624700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- debug 2021-04-20T14:06:19.449+0000 7f11c7624700 4 rocksdb: [db_impl/db_impl.cc:851] ** DB Stats ** Uptime(secs): 0.0 total, 0.0 interval Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 1/0 3.10 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 L6 1/0 38.24 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 2/0 41.34 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 1/0 3.10 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 L6 1/0 38.24 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 2/0 41.34 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** debug 2021-04-20T14:06:19.449+0000 7f11e080a700 0 starting mon.host3 rank 2 at public addrs [v2:108.61.24.82:3300/0,v1:108.61.24.82:6789/0] at bind addrs [v2:108.61.24.82:3300/0,v1:108.61.24.82:6789/0] mon_data /var/lib/ceph/mon/ceph-host3 fsid c8ee2878-9d54-11eb-bbca-1c34da4b9fb6 debug 2021-04-20T14:06:19.449+0000 7f11e080a700 1 mon.host3@-1(???) e4 preinit fsid c8ee2878-9d54-11eb-bbca-1c34da4b9fb6 debug 2021-04-20T14:06:19.449+0000 7f11e080a700 0 mon.host3(a)-1(???).mds e1 new map debug 2021-04-20T14:06:19.449+0000 7f11e080a700 0 mon.host3(a)-1(???).mds e1 print_map e1 enable_multiple, ever_enabled_multiple: 0,0 compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} legacy client fscid: -1 No filesystems configured debug 2021-04-20T14:06:19.449+0000 7f11e080a700 0 mon.host3(a)-1(???).osd e293 crush map has features 3314933000852226048, adjusting msgr requires debug 2021-04-20T14:06:19.449+0000 7f11e080a700 0 mon.host3(a)-1(???).osd e293 crush map has features 288514051259236352, adjusting msgr requires debug 2021-04-20T14:06:19.449+0000 7f11e080a700 0 mon.host3(a)-1(???).osd e293 crush map has features 288514051259236352, adjusting msgr requires debug 2021-04-20T14:06:19.449+0000 7f11e080a700 0 mon.host3(a)-1(???).osd e293 crush map has features 288514051259236352, adjusting msgr requires audit 2021-04-14T20:10:31.225267+0000 mon.host1 (mon.0) 6360 : audit [DBG] from='client.? 108.61.24.80:0/2572567093' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch debug 2021-04-20T14:06:19.449+0000 7f11e080a700 1 mon.host3(a)-1(???).paxosservice(auth 1..86) refresh upgraded, format 0 -> 3 debug 2021-04-20T14:06:19.457+0000 7f11e080a700 0 mon.host3@-1(probing) e4 my rank is now 2 (was -1) debug 2021-04-20T14:06:19.661+0000 7f11c9e29700 0 mon.host3@2(probing) e5 removed from monmap, suicide.

1 0

v14.2.20 Nautilus released
by David Galloway 20 Apr '21

20 Apr '21

This is the 20th bugfix release in the Nautilus stable series. It addresses a security vulnerability in the Ceph authentication framework. We recommend users to update to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v14-2-20-nautilus-released Security Fixes -------------- * This release includes a security fix that ensures the global_id value (a numeric value that should be unique for every authenticated client or daemon in the cluster) is reclaimed after a network disconnect or ticket renewal in a secure fashion. Two new health alerts may appear during the upgrade indicating that there are clients or daemons that are not yet patched with the appropriate fix. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.20.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 36274af6eb7f2a5055f2d53ad448f2694e9046a0

4 4

BlueFS spillover detected (Nautilus 14.2.16)
by by morphin 20 Apr '21

20 Apr '21

Hello. I'm trying to fix a wrong cluster deployment (Nautilus 14.2.16) Cluster usage is %40 EC pool with RGW Every node has: 20 x OSD = TOSHIBA MG08SCA16TEY 16.0TB 2 x DB = NVME PM1725b 1.6TB (linux mdadm raid1) NVME usage always goes around %90-99. With "iostat -xdh 1" r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util Device 168.00 3619.00 7.2M 367.7M 0.00 90510.00 0.0% 96.2% 1.10 9.21 22.86 43.8k 104.0k 0.25 96.0% nvme0c0n1 19.00 3670.00 1.7M 373.5M 0.00 90510.00 0.0% 96.1% 0.26 29.61 95.99 89.7k 104.2k 0.27 98.0% nvme1c1n1 The problem is: BLUEFS_SPILLOVER BlueFS spillover detected on 120 OSD(s) osd.194 spilled over 42 GiB metadata from 'db' device (39 GiB used of 50 GiB) to slow device osd.195 spilled over 34 GiB metadata from 'db' device (40 GiB used of 50 GiB) to slow device osd.196 spilled over 28 GiB metadata from 'db' device (40 GiB used of 50 GiB) to slow device osd.197 spilled over 25 GiB metadata from 'db' device (41 GiB used of 50 GiB) to slow device osd.198 spilled over 30 GiB metadata from 'db' device (41 GiB used of 50 GiB) to slow device Block and wal size: bluestore_block_db_size = 53687091200 bluestore_block_wal_size = 0 nvme0n1 259:2 0 1.5T 0 disk └─md0 9:0 0 1.5T 0 raid1 ├─md0p1 259:4 0 50G 0 md ├─md0p2 259:5 0 50G 0 md +n └─md0p20 259:22 0 50G 0 md How can I change the level up to 500MB --> 5GB --> 50GB ?

2 4