December 2023 - ceph-users

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Problem while upgrade 17.2.6 to 17.2.7
by Jean-Marc FONTANA 21 Feb '25

21 Feb '25

Hello everyone, We operate two clusters that we installed with ceph-deploy in Nautilus version on Debian 10. We use them for external S3 storage (owncloud) and rbd disk images.We had them upgraded to Octopus and Pacific versions on Debian 11 and recently converted them to cephadm and upgraded to Quincy (17.2.6). As we now have the orchestrator, we tried updating to 17.2.7 using the command# ceph orch upgrade start --image quay.io/ceph/ceph:v17.2.7 Everything went well, both clusters work perfectly for our use, except that the Rados gateway configuration is no longer accessible from the dashboard with the following error messageError connecting to Object Gateway: RGW REST API failed request with status code 404. We tried a few solutions found on the internet (reset rgw credentials, restart rgw adnd mgr, reenable dashboard, ...), unsuccessfully. Does somebody have an idea ? Best regards, Jean-Marc Fontana

5 9

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

multifs and snapshots
by Dmitry Melekhov 12 Nov '24

12 Nov '24

Hello! There is note here https://docs.ceph.com/en/reef/dev/cephfs-snapshots/ about multiply filesystems and snapshots: If each FS gets its own pool things probably work, but this isn’t tested and may not be true. Is it still untested ? Thank you!

3 4

Help needed to configure erasure coding LRC plugin
by Michel Jouvin 16 Oct '24

16 Oct '24

Hi, As discussed in another thread (Crushmap rule for multi-datacenter erasure coding), I'm trying to create an EC pool spanning 3 datacenters (datacenters are present in the crushmap), with the objective to be resilient to 1 DC down, at least keeping the readonly access to the pool and if possible the read-write access, and have a storage efficiency better than 3 replica (let say a storage overhead <= 2). In the discussion, somebody mentioned LRC plugin as a possible jerasure alternative to implement this without tweaking the crushmap rule to implement the 2-step OSD allocation. I looked at the documentation (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/) but I have some questions if someone has experience/expertise with this LRC plugin. I tried to create a rule for using 5 OSDs per datacenter (15 in total), with 3 (9 in total) being data chunks and others being coding chunks. For this, based of my understanding of examples, I used k=9, m=3, l=4. Is it right? Is this configuration equivalent, in terms of redundancy, to a jerasure configuration with k=9, m=6? The resulting rule, which looks correct to me, is: -------- { "rule_id": 6, "rule_name": "test_lrc_2", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 15, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -4, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 5, "type": "host" }, { "op": "emit" } ] } ------------ Unfortunately, it doesn't work as expected: a pool created with this rule ends up with its pages active+undersize, which is unexpected for me. Looking at 'ceph health detail` output, I see for each page something like: pg 52.14 is stuck undersized for 27m, current state active+undersized, last acting [90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647] For each PG, there is 3 '2147483647' entries and I guess it is the reason of the problem. What are these entries about? Clearly it is not OSD entries... Looks like a negative number, -1, which in terms of crushmap ID is the crushmap root (named "default" in our configuration). Any trivial mistake I would have made? Thanks in advance for any help or for sharing any successful configuration? Best regards, Michel

4 24

Not all Bucket Shards being used
by Christian Kugler 01 Oct '24

01 Oct '24

Hi, I have trouble with large OMAP files in a cluster in the RGW index pool. Some background information about the cluster: There is CephFS and RBD usage on the main cluster but for this issue I think only S3 is interesting. There is one realm, one zonegroup with two zones which have a bidirectional sync set up. Since this does not allow for autoresharding we have to do it by hand in this cluster – looking forward to Reef! From the logs: cluster 2023-07-17T22:59:03.018722+0000 osd.75 (osd.75) 623978 : cluster [WRN] Large omap object found. Object: 34:bcec3016:::.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5:head PG: 34.680c373d (34.5) Key count: 962091 Size (bytes): 277963182 The offending bucket looks like this: # radosgw-admin bucket stats \ | jq '.[] | select(.marker =="3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9") |"\(.num_shards) \(.usage["rgw.main"].num_objects)"' -r 131 9463833 Last week the number of objects was about 12 million. Which is why I reshareded the offending bucket twice, I think. Once to 129 and the second time to 131 because I wanted some leeway (or lieway? scnr, Sage). Unfortunately, even after a week the objects were still to big (the log line above is quite recent), so I looked into it again. # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.0 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.1 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.2 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.3 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.4 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.6 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.7 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.8 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.9 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.10 # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V \ |xargs -IOMAP sh -c \ 'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l' 1013854 1011007 1012287 1011232 1013565 998262 1012777 1012713 1012230 1010690 997111 Apparently, only 11 shards are in use. This would explain why the "Key usage" (from the log line) is about ten times higher than I would expect. How can I deal with this issue? One thing I could try to fix this would be to reshard to a lower number, but I am not sure if there are any risks associated with "downsharding". After that I could reshard to something like 97. Or I could directly "downshard" to 97. Also, the second zone has a similar problem, but as the error messsage lets me know, this would be a bad idea. Will it just take more time until the sharding is transferred to the seconds zone? Best, Christian Kugler

2 5

Fixing BlueFS spillover (pacific 16.2.14)
by Chris Dunlop 09 Jul '24

09 Jul '24

Hi, tl;dr why are my osds still spilling? I've recently upgraded to 16.2.14 from 16.2.9 and started receiving bluefs spillover warnings (due to the "fix spillover alert" per the 16.2.14 release notes). E.g. from 'ceph health detail', the warning on one of these (there are a few): osd.76 spilled over 128 KiB metadata from 'db' device (56 GiB used of 60 GiB) to slow device This is a 15T HDD with only a 60G SSD for the db so it's not surprising it spilled as it's way below the recommendation for rbd usage at db size 1-2% of the storage size. There was some spare space on the db ssd so I increased the size of the db LV up over 400G and did an bluefs-bdev-expand. However, days later, I'm still getting the spillover warning for that osd, including after running a manual compact: # ceph tell osd.76 compact See attached perf-dump-76 for the perf dump output: # cephadm enter --name 'osd.76' ceph daemon 'osd.76' perf dump" | jq -r '.bluefs' In particular, if my understanding is correct, that's telling me the db available size is 487G (i.e. the LV expand worked), of which it's using 59G, and there's 128K spilled to the slow device: "db_total_bytes": 512309059584, # 487G "db_used_bytes": 63470305280, # 59G "slow_used_bytes": 131072, # 128K A "bluefs stats" also says the db is using 128K of slow storage (although perhaps it's getting the info from the same place as the perf dump?): # ceph tell osd.76 bluefs stats 1 : device size 0x7747ffe000 : using 0xea6200000(59 GiB) 2 : device size 0xe8d7fc00000 : using 0x6554d689000(6.3 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 8.8 MiB 1 WAL 0 B 2.5 GiB 0 B 0 B 0 B 751 MiB 8 DB 0 B 56 GiB 128 KiB 0 B 0 B 50 GiB 842 SLOW 0 B 0 B 0 B 0 B 0 B 0 B 0 TOTAL 0 B 58 GiB 128 KiB 0 B 0 B 0 B 850 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 3.9 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 71 GiB 282 MiB 0 B 0 B 62 GiB SLOW 0 B 0 B 0 B 0 B 0 B 0 B TOTAL 0 B 74 GiB 282 MiB 0 B 0 B 0 B >> SIZE << 0 B 453 GiB 14 TiB I had a look at the "DUMPING STATS" output in the logs bug I don't know how to interpret it. I did try calculating the total of the sizes on the "Sum" lines but that comes to 100G so I don't know what that all means. See attached log-stats-76. I also tried "ceph-kvstore-tool bluestore-kv ... stats": $ { cephadm unit --fsid $clusterid --name osd.76 stop cephadm shell --fsid $clusterid --name osd.76 -- ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-76 stats cephadm unit --fsid $clusterid --name osd.76 start } Output attached as bluestore-kv-stats-76. I can't see anything interesting in there, although again I don't really know how to interpret it. So... why is this osd db still spilling onto slow storage, and how do I fix things so it's no longer using the slow storage? And a bonus issue... on another osd that hasn't yet been resized (i.e. again with a grossly undersized 60G db on SSD with a 15T HDD) I'm also getting a spillover warning. The "bluefs stats" seems to be saying the db is NOT currently spilling (i.e. "0 B" the DB/SLOW position in the matrix), but there's "something" currently using 59G on the slow device: $ ceph tell osd.85 bluefs stats 1 : device size 0xeffffe000 : using 0x3a3900000(15 GiB) 2 : device size 0xe8d7fc00000 : using 0x7aea7434000(7.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 7.4 MiB 1 WAL 0 B 564 MiB 0 B 0 B 0 B 132 MiB 2 DB 0 B 11 GiB 0 B 0 B 0 B 8.1 GiB 177 SLOW 0 B 3.0 GiB 59 GiB 0 B 0 B 56 GiB 898 TOTAL 0 B 13 GiB 59 GiB 0 B 0 B 0 B 1072 MAXIMUMS: LOG 0 B 24 MiB 0 B 0 B 0 B 20 MiB WAL 0 B 2.8 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 22 GiB 448 KiB 0 B 0 B 18 GiB SLOW 0 B 3.3 GiB 62 GiB 0 B 0 B 62 GiB TOTAL 0 B 27 GiB 62 GiB 0 B 0 B 0 B >> SIZE << 0 B 57 GiB 14 TiB Is there anywhere that describes how to interpret this output, and specifically, what stuff is going into the SLOW row? Seemingly there's 898 "files" there, but not LOG, WAL or DB files - so what are they? Cheers, Chris

4 7

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

Re: MDS crashes to damaged metadata
by Patrick Donnelly 10 Jun '24

10 Jun '24

On Thu, Dec 15, 2022 at 9:32 AM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): > > 2022-12-08T13:05:48.919+0100 7f440afec700 -1 /build/ceph-16.2.10/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f440afec700 time 2022-12-08T13:05:48.921223+0100 > /build/ceph-16.2.10/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE) > > 12 Minutes lates the unlink_local error crashes appeared again. This time with a new file. During debugging we noticed a MTU mismatch between MDS (1500) and client (9000) with cephfs kernel mount. The client is also creating the snapshots via mkdir in the .snap directory. > > We disabled snapshot creation for now, but really need this feature. I uploaded the mds logs of the first crash along with the information above to https://tracker.ceph.com/issues/38452 > > I would greatly appreciate it, if you could answer me the following question: > > Is the Bug related to our MTU Mismatch? We fixed the MTU Issue going back to 1500 on all nodes in the ceph public network on the weekend also. I doubt it. > If you need a debug level 20 log of the ScatterLock for further analysis, i could schedule snapshots at the end of our workdays and increase the debug level 5 Minutes arround snap shot creation. This would be very helpful! -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 4

16.2.14: [progress WARNING root] complete: ev {UUID} does not exist
by Zakhar Kirpichenko 03 May '24

03 May '24

Hi, Mgr of my cluster logs this every few seconds: [progress WARNING root] complete: ev 7de5bb74-790b-4fda-8838-e4af4af18c62 does not exist [progress WARNING root] complete: ev fff93fce-b630-4141-81ee-19e7a3e61483 does not exist [progress WARNING root] complete: ev a02f6966-5b9f-49e8-89c4-b4fb8e6f4423 does not exist [progress WARNING root] complete: ev 8d318560-ff1a-477f-9386-43f6b51080bf does not exist [progress WARNING root] complete: ev ff3740a9-6434-470a-808f-a2762fb542a0 does not exist [progress WARNING root] complete: ev 7d0589f1-545e-4970-867b-8482ce48d7f0 does not exist [progress WARNING root] complete: ev 78d57e43-5be5-43f0-8b1a-cdc60e410892 does not exist I would appreciate an advice on what these warnings mean and how they can be resolved. Best regards, Zakhar

4 7

Re: Recoveries without any misplaced objects?
by Hector Martin 24 Apr '24

24 Apr '24

On 29/05/2023 20.55, Anthony D'Atri wrote: > Check the uptime for the OSDs in question I restarted all my OSDs within the past 10 days or so. Maybe OSD restarts are somehow breaking these stats? > >> On May 29, 2023, at 6:44 AM, Hector Martin <marcan(a)marcan.st> wrote: >> >> Hi, >> >> I'm watching a cluster finish a bunch of backfilling, and I noticed that >> quite often PGs end up with zero misplaced objects, even though they are >> still backfilling. >> >> Right now the cluster is down to 6 backfilling PGs: >> >> data: >> volumes: 1/1 healthy >> pools: 6 pools, 268 pgs >> objects: 18.79M objects, 29 TiB >> usage: 49 TiB used, 25 TiB / 75 TiB avail >> pgs: 262 active+clean >> 6 active+remapped+backfilling >> >> But there are no misplaced objects, and the misplaced column in `ceph pg >> dump` is zero for all PGs. >> >> If I do a `ceph pg dump_json`, I can see `num_objects_recovered` >> increasing for these PGs... but the misplaced count is still 0. >> >> Is there something else that would cause recoveries/backfills other than >> misplaced objects? Or perhaps there is a bug somewhere causing the >> misplaced object count to be misreported as 0 sometimes? >> >> # ceph -v >> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy >> (stable) >> >> - Hector >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > - Hector

3 3

Status of IPv4 / IPv6 dual stack?
by Robert Sander 23 Apr '24

23 Apr '24

Hi, as the documentation sends mixed signals in https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#ipv… "Note Binding to IPv4 is enabled by default, so if you just add the option to bind to IPv6 you’ll actually put yourself into dual stack mode." and https://docs.ceph.com/en/latest/rados/configuration/msgr2/#address-formats "Note The ability to bind to multiple ports has paved the way for dual-stack IPv4 and IPv6 support. That said, dual-stack operation is not yet supported as of Quincy v17.2.0." just the quick questions: Is a dual stacked networking with IPv4 and IPv6 now supported or not? From which version on is it considered stable? Are OSDs now able to register themselves with two IP addresses in the cluster map? MONs too? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

6 6

Ceph 16.2.x mon compactions, disk writes
by Zakhar Kirpichenko 17 Apr '24

17 Apr '24

Hi, Monitors in our 16.2.14 cluster appear to quite often run "manual compaction" tasks: debug 2023-10-09T09:30:53.888+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696843853892760, "job": 64225, "event": "flush_started", "num_memtables": 1, "num_entries": 715, "num_deletes": 251, "total_data_size": 3870352, "memory_usage": 3886744, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:30:53.904+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:30:53.910204) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:30:53.911004) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:32:08.956+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696843928961390, "job": 64228, "event": "flush_started", "num_memtables": 1, "num_entries": 1580, "num_deletes": 502, "total_data_size": 8404605, "memory_usage": 8465840, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:32:08.972+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:32:08.977739) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'logm .. 'logm; will stop at (end) debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:32:08.978512) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'logm .. 'logm; will stop at (end) debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.028+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696844009033151, "job": 64231, "event": "flush_started", "num_memtables": 1, "num_entries": 1430, "num_deletes": 251, "total_data_size": 8975535, "memory_usage": 9035920, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:33:29.044+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:33:29.049585) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:33:29.050355) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'paxos .. 'paxos; will stop at (end) I have removed a lot of interim log messages to save space. During each compaction the monitor process writes approximately 500-600 MB of data to disk over a short period of time. These writes add up to tens of gigabytes per hour and hundreds of gigabytes per day. Monitor rocksdb and compaction options are default: "mon_compact_on_bootstrap": "false", "mon_compact_on_start": "false", "mon_compact_on_trim": "true", "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", Is this expected behavior? Is this something I can adjust in order to extend the system storage life? Best regards, Zakhar

5 32

Setting S3 bucket policies with multi-tenants
by Thomas Bennett 15 Apr '24

15 Apr '24

Hi, I'm running Ceph Quincy (17.2.6) with a rados-gateway. I have muti tenants, for example: - Tenant1$manager - Tenant1$readwrite I would like to set a policy on a bucket (backups for example) owned by *Tenant1$manager* to allow *Tenant1$readwrite* access to that bucket. I can't find any documentation that discusses this scenario. Does anyone know how to specify the Principle and Resource section of a policy.json file? Or any other configuration that I might be missing? I've tried some variations on Principal and Resource including and excluding tenant information, but not no luck yet. For example: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": {"AWS": ["arn:aws:iam:::user/*Tenant1$readwrite*"]}, "Action": ["s3:ListBucket","s3:GetObject", ,"s3:PutObject"], "Resource": [ "arn:aws:s3:::*Tenant1/backups*" ] }] } I'm using s3cmd for testing, so: s3cmd --config s3cfg.manager setpolicy policy.json s3://backups/ Returns: s3://backups/: Policy updated And then testing: s3cmd --config s3cfg.readwrite ls s3://backups/ ERROR: Access to bucket 'backups' was denied ERROR: S3 error: 403 (AccessDenied) Thanks, Tom

4 4

A couple OSDs not starting after host reboot
by Alison Peisker 11 Apr '24

11 Apr '24

Hi all, We rebooted all the nodes in our 17.2.5 cluster after performing kernel updates, but 2 of the OSDs on different nodes are not coming back up. This is a production cluster using cephadm. The error message from the OSD log is ceph-osd[87340]: ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-665: (2) No such file or directory The error message from ceph-volume is 2023-08-23T16:12:43.452-0500 7f0cad968600 2 bluestore(/dev/mapper/ceph--febad5a5--ba44--41aa--a39e--b9897f757752-osd--block--87e548f4--b9b5--4ed8--aca8--de703a341a50) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input We tried restarting the daemons and rebooting the node again, but still see the same error. Has anyone experienced this issue before? How do we fix this? Thanks, Alison

9 13

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

Why you might want packages not containers for Ceph deployments
by Matthew Vernon 25 Mar '24

25 Mar '24

Hi, In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0]. I'd like to suggest, briefly, that: * containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes. For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice. Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages): * you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice) We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part. Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems! I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder". I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :) Regards, and apologies for the wall of text, Matthew [0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

26 65

MDS subtree pinning
by Sake Ceph 15 Mar '24

15 Mar '24

Hi! As I'm reading through the documentation about subtree pinning, I was wondering if the following is possible. We've got the following directory structure. / /app1 /app2 /app3 /app4 Can I pin /app1 to MDS rank 0 and 1, the directory /app2 to rank 2 and finally /app3 and /app4 to rank 3? I would like to load balance the subfolders of /app1 to 2 (or 3) MDS servers. Best regards, Sake

2 2

Upgrading from Pacific to Quincy fails with "Unexpected error"
by Reza Bakhshayeshi 13 Mar '24

13 Mar '24

Hi all, I have a problem regarding upgrading Ceph cluster from Pacific to Quincy version with cephadm. I have successfully upgraded the cluster to the latest Pacific (16.2.11). But when I run the following command to upgrade the cluster to 17.2.5, After upgrading 3/4 mgrs, the upgrade process stops with "Unexpected error". (everything is on a private network) ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v17.2.5 I also tried the 17.2.4 version. cephadm fails to check the hosts' status and marks them as offline: cephadm 2023-04-06T10:19:59.998510+0000 mgr.host9.arhpnd (mgr.4516356) 5782 : cephadm [DBG] host host4 (x.x.x.x) failed check cephadm 2023-04-06T10:19:59.998553+0000 mgr.host9.arhpnd (mgr.4516356) 5783 : cephadm [DBG] Host "host4" marked as offline. Skipping daemon refresh cephadm 2023-04-06T10:19:59.998581+0000 mgr.host9.arhpnd (mgr.4516356) 5784 : cephadm [DBG] Host "host4" marked as offline. Skipping gather facts refresh cephadm 2023-04-06T10:19:59.998609+0000 mgr.host9.arhpnd (mgr.4516356) 5785 : cephadm [DBG] Host "host4" marked as offline. Skipping network refresh cephadm 2023-04-06T10:19:59.998633+0000 mgr.host9.arhpnd (mgr.4516356) 5786 : cephadm [DBG] Host "host4" marked as offline. Skipping device refresh cephadm 2023-04-06T10:19:59.998659+0000 mgr.host9.arhpnd (mgr.4516356) 5787 : cephadm [DBG] Host "host4" marked as offline. Skipping osdspec preview refresh cephadm 2023-04-06T10:19:59.998682+0000 mgr.host9.arhpnd (mgr.4516356) 5788 : cephadm [DBG] Host "host4" marked as offline. Skipping autotune cluster 2023-04-06T10:20:00.000151+0000 mon.host8 (mon.0) 158587 : cluster [ERR] Health detail: HEALTH_ERR 9 hosts fail cephadm check; Upgrade: failed due to an unexpected exception cluster 2023-04-06T10:20:00.000191+0000 mon.host8 (mon.0) 158588 : cluster [ERR] [WRN] CEPHADM_HOST_CHECK_FAILED: 9 hosts fail cephadm check cluster 2023-04-06T10:20:00.000202+0000 mon.host8 (mon.0) 158589 : cluster [ERR] host host7 (x.x.x.x) failed check: Unable to reach remote host host7. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000213+0000 mon.host8 (mon.0) 158590 : cluster [ERR] host host2 (x.x.x.x) failed check: Unable to reach remote host host2. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000220+0000 mon.host8 (mon.0) 158591 : cluster [ERR] host host8 (x.x.x.x) failed check: Unable to reach remote host host8. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000228+0000 mon.host8 (mon.0) 158592 : cluster [ERR] host host4 (x.x.x.x) failed check: Unable to reach remote host host4. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000240+0000 mon.host8 (mon.0) 158593 : cluster [ERR] host host3 (x.x.x.x) failed check: Unable to reach remote host host3. Process exited with non-zero exit status 3 and here are some outputs of the commands: [root@host8 ~]# ceph -s cluster: id: xxx health: HEALTH_ERR 9 hosts fail cephadm check Upgrade: failed due to an unexpected exception services: mon: 5 daemons, quorum host8,host1,host7,host2,host9 (age 2w) mgr: host9.arhpnd(active, since 105m), standbys: host8.jowfih, host1.warjsr, host2.qyavjj mds: 1/1 daemons up, 3 standby osd: 37 osds: 37 up (since 8h), 37 in (since 3w) data: io: client: progress: Upgrade to 17.2.5 (0s) [............................] [root@host8 ~]# ceph orch upgrade status { "target_image": "my-private-repo/quay-io/ceph/ceph@sha256 :34c763383e3323c6bb35f3f2229af9f466518d9db926111277f5e27ed543c427", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "3/59 daemons upgraded", "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception", "is_paused": true } [root@host8 ~]# ceph cephadm check-host host7 check-host failed: Host 'host7' not found. Use 'ceph orch host ls' to see all managed hosts. [root@host8 ~]# ceph versions { "mon": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 1, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 }, "osd": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 37 }, "mds": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 4 }, "overall": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 47, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 } } The strange thing is I can rollback the cluster status by failing to not-upgraded mgr like this: ceph mgr fail ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v16.2.11 Would you happen to have any idea about this? Best regards, Reza

4 9

Disable signature url in ceph rgw
by marc＠singer.services 07 Mar '24

07 Mar '24

Hi Ceph users We are using Ceph Pacific (16) in this specific deployment. In our use case we do not want our users to be able to generate signature v4 URLs because they bypass the policies that we set on buckets (e.g IP restrictions). Currently we have a sidecar reverse proxy running that filters requests with signature URL specific request parameters. This is obviously not very efficient and we are looking to replace this somehow in the future. 1. Is there an option in RGW to disable this signed URLs (e.g returning status 403)? 2. If not is this planned or would it make sense to add it as a configuration option? 3. Or is the behaviour of not respecting bucket policies in RGW with signature v4 URLs a bug and they should be actually applied? Thanks you for your help and let me know if you have any questions Marc Singer

4 4

FS down - mds degraded
by Sake Ceph 23 Feb '24

23 Feb '24

Starting a new thread, forgot subject in the previous. So our FS down. Got the following error, what can I do? # ceph health detail HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged [WRN] FS_DEGRADED: 1 filesystem is degraded fs atlassian/prod is degraded [ERR] MDS_DAMAGE: 1 mds daemon damaged fs atlassian-prod mds.1 is damaged # ceph fs get atlassian-prod Filesystem 'atlassian-prod' (2) fs_name atlassian-prod epoch 43440 flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay created 2023-05-10T08:45:46.911064+0000 modified 2023-12-21T06:47:19.291154+0000 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 required_client_features {} last_failure 0 last_failure_osd_epoch 29480 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 3 in 0,1,2 up {0=1073573,2=1073583} failed damaged 1 stopped data_pools [5] metadata_pool 4 inline_data disabled balancer standby_count_wanted 1 [mds.atlassian-prod.pwsoel13142.egsdfl{0:1073573} state up:resolve seq 573 join_fscid=2 addr [v2:10.233.127.22:6800/61692284,v1:10.233.127.22:6801/61692284] compat {c=[1],r=[1],i=[7ff]}] [mds.atlassian-prod.pwsoel13143.qlvypn{2:1073583} state up:resolve seq 571 join_fscid=2 addr [v2:10.233.127.18:6800/3627858294,v1:10.233.127.18:6801/3627858294] compat {c=[1],r=[1],i=[7ff]}] Best regards, Sake

4 7

easy way to find out the number of allocated objects for a RBD image
by Tony Liu 29 Jan '24

29 Jan '24

Hi, Other than get all objects of the pool and filter by image ID, is there any easier way to get the number of allocated objects for a RBD image? What I really want to know is the actual usage of an image. An allocated object could be used partially, but that's fine, no need to be 100% accurate. To get the object count and times object size, that should be sufficient. "rbd export" exports actual used data, but to get the actual usage by exporting the image seems too much. This brings up another question, is there any way to know the export size before running it? Thanks! Tony

4 4

Re: Ceph OSD reported Slow operations
by V A Prabha 28 Jan '24

28 Jan '24

Hi Eugen Please find the details below root@meghdootctr1:/var/log/ceph# ceph -s cluster: id: c59da971-57d1-43bd-b2b7-865d392412a5 health: HEALTH_WARN nodeep-scrub flag(s) set 544 pgs not deep-scrubbed in time services: mon: 3 daemons, quorum meghdootctr1,meghdootctr2,meghdootctr3 (age 5d) mgr: meghdootctr1(active, since 5d), standbys: meghdootctr2, meghdootctr3 mds: 3 up:standby osd: 36 osds: 36 up (since 34h), 36 in (since 34h) flags nodeep-scrub data: pools: 2 pools, 544 pgs objects: 10.14M objects, 39 TiB usage: 116 TiB used, 63 TiB / 179 TiB avail pgs: 544 active+clean io: client: 24 MiB/s rd, 16 MiB/s wr, 2.02k op/s rd, 907 op/s wr Ceph Versions: root@meghdootctr1:/var/log/ceph# ceph --version ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) Ceph df -h https://pastebin.com/1ffucyJg Ceph OSD performance dump https://pastebin.com/1R6YQksE Ceph tell osd.XX bench (Out of 36 osds only 8 OSDs give High IOPS value of 250 +. Out of that 4 OSDs are from HP 3PAR and 4 OSDS from DELL EMC. We are using only 4 OSDs from HP3 par and it is working fine without any latency and iops issues from the beginning but the remaining 32 OSDs are from DELL EMC in which 4 OSDs are much better than the remaining 28 OSDs) https://pastebin.com/CixaQmBi Please help me to identify if the issue is with the DELL EMC Storage, Ceph configuration parameter tuning or the Overload in the cloud setup On November 1, 2023 at 9:48 PM Eugen Block <eblock(a)nde.ag> wrote: > Hi, > > for starters please add more cluster details like 'ceph status', 'ceph > versions', 'ceph osd df tree'. Increasing the to 10G was the right > thing to do, you don't get far with 1G with real cluster load. How are > the OSDs configured (HDD only, SSD only or HDD with rocksdb on SSD)? > How is the disk utilization? > > Regards, > Eugen > > Zitat von prabhav(a)cdac.in: > > > In a production setup of 36 OSDs( SAS disks) totalling 180 TB > > allocated to a single Ceph Cluster with 3 monitors and 3 managers. > > There were 830 volumes and VMs created in Openstack with Ceph as a > > backend. On Sep 21, users reported slowness in accessing the VMs. > > Analysing the logs lead us to problem with SAS , Network congestion > > and Ceph configuration( as all default values were used). We updated > > the Network from 1Gbps to 10Gbps for public and cluster networking. > > There was no change. > > The ceph benchmark performance showed that 28 OSDs out of 36 OSDs > > reported very low IOPS of 30 to 50 while the remaining showed 300+ > > IOPS. > > We gradually started reducing the load on the ceph cluster and now > > the volumes count is 650. Now the slow operations has gradually > > reduced but I am aware that this is not the solution. > > Ceph configuration is updated with increasing the > > osd_journal_size to 10 GB, > > osd_max_backfills = 1 > > osd_recovery_max_active = 1 > > osd_recovery_op_priority = 1 > > bluestore_cache_trim_max_skip_pinned=10000 > > > > After one month, now we faced another issue with Mgr daemon stopped > > in all 3 quorums and 16 OSDs went down. From the > > ceph-mon,ceph-mgr.log could not get the reason. Please guide me as > > its a production setup > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294 ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------

4 15

Ceph 16.2.14: ceph-mgr getting oom-killed
by Zakhar Kirpichenko 25 Jan '24

25 Jan '24

Hi, I'm facing a rather new issue with our Ceph cluster: from time to time ceph-mgr on one of the two mgr nodes gets oom-killed after consuming over 100 GB RAM: [Nov21 15:02] tp_osd_tp invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 [ +0.000010] oom_kill_process.cold+0xb/0x10 [ +0.000002] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ +0.000008] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=504d37b566d9fd442d45904a00584b4f61c93c5d49dc59eb1c948b3d1c096907,mems_allowed=0-1,global_oom,task_memcg=/docker/3826be8f9115479117ddb8b721ca57585b2bdd58a27c7ed7b38e8d83eb795957,task=ceph-mgr,pid=3941610,uid=167 [ +0.000697] Out of memory: Killed process 3941610 (ceph-mgr) total-vm:146986656kB, anon-rss:125340436kB, file-rss:0kB, shmem-rss:0kB, UID:167 pgtables:260356kB oom_score_adj:0 [ +6.509769] oom_reaper: reaped process 3941610 (ceph-mgr), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB The cluster is stable and operating normally, there's nothing unusual going on before, during or after the kill, thus it's unclear what causes the mgr to balloon, use all RAM and get killed. Systemd logs aren't very helpful: they just show normal mgr operations until it fails to allocate memory and gets killed: https://pastebin.com/MLyw9iVi The mgr experienced this issue several times in the last 2 months, and the events don't appear to correlate with any other events in the cluster because basically nothing else happened at around those times. How can I investigate this and figure out what's causing the mgr to consume all memory and get killed? I would very much appreciate any advice! Best regards, Zakhar

6 21

Upgrading nautilus / centos7 to octopus / ubuntu 20.04. - Suggestions and hints?
by Götz Reinicke 17 Jan '24

17 Jan '24

Hi, As I’v read and thought a lot about the migration as this is a bigger project, I was wondering if anyone has done that already and might share some notes or playbooks, because in all readings there where some parts missing or miss understandable to me. I do have some different approaches in mind, so may be you have some suggestions or hints. a) upgrade nautilus on centos 7 with the few missing features like dashboard and prometheus. After that migrate one node after an other to ubuntu 20.04 with octopus and than upgrade ceph to the recent stable version. b) migrate one node after an other to ubuntu 18.04 with nautilus and then upgrade to octupus and after that to ubuntu 20.04. or c) upgrade one node after an other to ubuntu 20.04 with octopus and join it to the cluster until all nodes are upgraded. For test I tried c) with a mon node, but adding that to the cluster fails with some failed state, still probing for the other mons. (I dont have the right log at hand right now.) So my questions are: a) What would be the best (most stable) migration path and b) is it in general possible to add a new octopus mon (not upgraded one) to a nautilus cluster, where the other mons are still on nautilus? I hope my thoughts and questions are understandable :) Thanks for any hint and suggestion. Best . Götz

7 11

RGW - user created bucket with name of already created bucket
by Ondřej Kukla 15 Jan '24

15 Jan '24

Hello, I would like to share a quite worrying experience I’ve just found on one of my production clusters. User successfully created a bucket with name of a bucket that already exists! He is not bucket owner - the original user is, but he is able to see it when he does ListBuckets over s3 api. (Both accounts are able to do it now - only the original owner is able to interact with it) This bucket is also counted to the new users usage stats. Has anyone noticed this before? This cluster is running on Quincy - 17.2.6. Is there a way to detach the bucket from the new owner so he doesn’t have a bucket that doesn’t belong to him? Regards, Ondrej

2 3

RGW rate-limiting or anti-hammering for (external) auth requests // Anti-DoS measures
by Christian Rohmann 12 Jan '24

12 Jan '24

Hey Ceph-Users, RGW does have options [1] to rate limit ops or bandwidth per bucket or user. But those only come into play when the request is authenticated. I'd like to also protect the authentication subsystem from malicious or invalid requests. So in case e.g. some EC2 credentials are not valid (anymore) and clients start hammering the RGW with those requests, I'd like to make it cheap to deal with those requests. Especially in case some external authentication like OpenStack Keystone [2] is used, valid access tokens are cached within the RGW. But requests with invalid credentials end up being sent at full rate to the external API [3] as there is no negative caching. And even if there was, that would only limit the external auth requests for the same set of invalid credentials, but it would surely reduce the load in that case: Since the HTTP request is blocking .... > [...] > 2023-12-18T15:25:55.861+0000 7fec91dbb640 20 sending request to > https://keystone.example.com/v3/s3tokens > 2023-12-18T15:25:55.861+0000 7fec91dbb640 20 register_request > mgr=0x561a407ae0c0 req_data->id=778, curl_handle=0x7fedaccb36e0 > 2023-12-18T15:25:55.861+0000 7fec91dbb640 20 WARNING: blocking http > request > 2023-12-18T15:25:55.861+0000 7fede37fe640 20 link_request > req_data=0x561a40a418b0 req_data->id=778, curl_handle=0x7fedaccb36e0 > [...] this does not only stress the external authentication API (keystone in this case), but also blocks RGW threads for the duration of the external call. I am currently looking into using the capabilities of HAProxy to rate limit requests based on their resulting http-response [4]. So in essence to rate-limit or tarpit clients that "produce" a high number of 403 "InvalidAccessKeyId" responses. To have less collateral it might make sense to limit based on the presented credentials themselves. But this would require to extract and track HTTP headers or URL parameters (presigned URLs) [5] and to put them into tables. * What are your thoughts on the matter? * What kind of measures did you put in place? * Does it make sense to extend RGWs capabilities to deal with those cases itself? ** adding negative caching ** rate limits on concurrent external authentication requests (or is there a pool of connections for those requests?) Regards Christian [1] https://docs.ceph.com/en/latest/radosgw/admin/#rate-limit-management [2] https://docs.ceph.com/en/latest/radosgw/keystone/#integrating-with-openstac… [3] https://github.com/ceph/ceph/blob/86bb77eb9633bfd002e73b5e58b863bc2d0df594/… [4] https://www.haproxy.com/documentation/haproxy-configuration-manual/latest/#… [5] https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-reque…

2 3

Is there any way to merge an rbd image's full backup and a diff?
by Satoru Takeuchi 12 Jan '24

12 Jan '24

Hi, I'm developing RBD images' backup system. In my case, a backup data must be stored at least two weeks. To meet this requirement, I'd like to take backups as follows: 1. Take a full backup by rbd export first. 2. Take a differencial backups everyday. 3. Merge the full backup and the oldest (taken two weeks ago) diff. As a result of evaluation, I confirmed there is no problem in step 1 and 2. However, I found that step 3 couldn't be accomplished by `rbd merge-diff <full backup> <diff>` because `rbd merge-diff` only accepts a diff as a first parameter. Is there any way to merge a full backup and a diff? Thanks, Satoru

2 5

How to configure something like osd_deep_scrub_min_interval?
by Frank Schilder 09 Jan '24

09 Jan '24

Hi folks, I am fighting a bit with odd deep-scrub behavior on HDDs and discovered a likely cause of why the distribution of last_deep_scrub_stamps is so weird. I wrote a small script to extract a histogram of scrubs by "days not scrubbed" (more precisely, intervals not scrubbed; see code) to find out how (deep-) scrub times are distributed. Output below. What I expected is along the lines that HDD-OSDs try to scrub every 1-3 days, while they try to deep-scrub every 7-14 days. In other words, OSDs that have been deep-scrubbed within the last 7 days would *never* be in scrubbing+deep state. However, what I see is completely different. There seems to be no distinction between scrub- and deep-scrub start times. This is really unexpected as nobody would try to deep-scrub HDDs every day. Weekly to bi-weekly is normal, specifically for large drives. Is there a way to configure something like osd_deep_scrub_min_interval (no, I don't want to run cron jobs for scrubbing yet)? In the output below, I would like to be able to configure a minimum period of 1-2 weeks before the next deep-scrub happens. How can I do that? The observed behavior is very unusual for RAID systems (if its not a bug in the report script). With this behavior its not surprising that people complain about "not deep-scrubbed in time" messages and too high deep-scrub IO load when such a large percentage of OSDs is needlessly deep-scrubbed after 1-6 days again already. Sample output: # scrub-report dumped pgs Scrub report: 4121 PGs not scrubbed since 1 intervals (6h) 3831 PGs not scrubbed since 2 intervals (6h) 4012 PGs not scrubbed since 3 intervals (6h) 3986 PGs not scrubbed since 4 intervals (6h) 2998 PGs not scrubbed since 5 intervals (6h) 1488 PGs not scrubbed since 6 intervals (6h) 909 PGs not scrubbed since 7 intervals (6h) 771 PGs not scrubbed since 8 intervals (6h) 582 PGs not scrubbed since 9 intervals (6h) 2 scrubbing 431 PGs not scrubbed since 10 intervals (6h) 333 PGs not scrubbed since 11 intervals (6h) 1 scrubbing 265 PGs not scrubbed since 12 intervals (6h) 195 PGs not scrubbed since 13 intervals (6h) 116 PGs not scrubbed since 14 intervals (6h) 78 PGs not scrubbed since 15 intervals (6h) 1 scrubbing 72 PGs not scrubbed since 16 intervals (6h) 37 PGs not scrubbed since 17 intervals (6h) 5 PGs not scrubbed since 18 intervals (6h) 14.237* 19.5cd* 19.12cc* 19.1233* 14.40e* 33 PGs not scrubbed since 20 intervals (6h) 23 PGs not scrubbed since 21 intervals (6h) 16 PGs not scrubbed since 22 intervals (6h) 12 PGs not scrubbed since 23 intervals (6h) 8 PGs not scrubbed since 24 intervals (6h) 2 PGs not scrubbed since 25 intervals (6h) 19.eef* 19.bb3* 4 PGs not scrubbed since 26 intervals (6h) 19.b4c* 19.10b8* 19.f13* 14.1ed* 5 PGs not scrubbed since 27 intervals (6h) 19.43f* 19.231* 19.1dbe* 19.1788* 19.16c0* 6 PGs not scrubbed since 28 intervals (6h) 2 PGs not scrubbed since 30 intervals (6h) 19.10f6* 14.9d* 3 PGs not scrubbed since 31 intervals (6h) 19.1322* 19.1318* 8.a* 1 PGs not scrubbed since 32 intervals (6h) 19.133f* 1 PGs not scrubbed since 33 intervals (6h) 19.1103* 3 PGs not scrubbed since 36 intervals (6h) 19.19cc* 19.12f4* 19.248* 1 PGs not scrubbed since 39 intervals (6h) 19.1984* 1 PGs not scrubbed since 41 intervals (6h) 14.449* 1 PGs not scrubbed since 44 intervals (6h) 19.179f* Deep-scrub report: 3723 PGs not deep-scrubbed since 1 intervals (24h) 4621 PGs not deep-scrubbed since 2 intervals (24h) 8 scrubbing+deep 3588 PGs not deep-scrubbed since 3 intervals (24h) 8 scrubbing+deep 2929 PGs not deep-scrubbed since 4 intervals (24h) 3 scrubbing+deep 1705 PGs not deep-scrubbed since 5 intervals (24h) 4 scrubbing+deep 1904 PGs not deep-scrubbed since 6 intervals (24h) 5 scrubbing+deep 1540 PGs not deep-scrubbed since 7 intervals (24h) 7 scrubbing+deep 1304 PGs not deep-scrubbed since 8 intervals (24h) 7 scrubbing+deep 923 PGs not deep-scrubbed since 9 intervals (24h) 5 scrubbing+deep 557 PGs not deep-scrubbed since 10 intervals (24h) 7 scrubbing+deep 501 PGs not deep-scrubbed since 11 intervals (24h) 2 scrubbing+deep 363 PGs not deep-scrubbed since 12 intervals (24h) 2 scrubbing+deep 377 PGs not deep-scrubbed since 13 intervals (24h) 1 scrubbing+deep 383 PGs not deep-scrubbed since 14 intervals (24h) 2 scrubbing+deep 252 PGs not deep-scrubbed since 15 intervals (24h) 2 scrubbing+deep 116 PGs not deep-scrubbed since 16 intervals (24h) 5 scrubbing+deep 47 PGs not deep-scrubbed since 17 intervals (24h) 2 scrubbing+deep 10 PGs not deep-scrubbed since 18 intervals (24h) 2 PGs not deep-scrubbed since 19 intervals (24h) 19.1c6c* 19.a01* 1 PGs not deep-scrubbed since 20 intervals (24h) 14.1ed* 2 PGs not deep-scrubbed since 21 intervals (24h) 19.1322* 19.10f6* 1 PGs not deep-scrubbed since 23 intervals (24h) 19.19cc* 1 PGs not deep-scrubbed since 24 intervals (24h) 19.179f* PGs marked with a * are on busy OSDs and not eligible for scrubbing. The script (pasted here because attaching doesn't work): # cat bin/scrub-report #!/bin/bash # Compute last scrub interval count. Scrub interval 6h, deep-scrub interval 24h. # Print how many PGs have not been (deep-)scrubbed since #intervals. ceph -f json pg dump pgs 2>&1 > /root/.cache/ceph/pgs_dump.json echo "" T0="$(date +%s)" scrub_info="$(jq --arg T0 "$T0" -rc '.pg_stats[] | [ .pgid, (.last_scrub_stamp[:19]+"Z" | (($T0|tonumber) - fromdateiso8601)/(60*60*6)|ceil), (.last_deep_scrub_stamp[:19]+"Z" | (($T0|tonumber) - fromdateiso8601)/(60*60*24)|ceil), .state, (.acting | join(" ")) ] | @tsv ' /root/.cache/ceph/pgs_dump.json)" # less <<<"$scrub_info" # 1 2 3 4 5..NF # pg_id scrub-ints deep-scrub-ints status acting[] awk <<<"$scrub_info" '{ for(i=5; i<=NF; ++i) pg_osds[$1]=pg_osds[$1] " " $i if($4 == "active+clean") { si_mx=si_mx<$2 ? $2 : si_mx dsi_mx=dsi_mx<$3 ? $3 : dsi_mx pg_sn[$2]++ pg_sn_ids[$2]=pg_sn_ids[$2] " " $1 pg_dsn[$3]++ pg_dsn_ids[$3]=pg_dsn_ids[$3] " " $1 } else if($4 ~ /scrubbing\+deep/) { deep_scrubbing[$3]++ for(i=5; i<=NF; ++i) osd[$i]="busy" } else if($4 ~ /scrubbing/) { scrubbing[$2]++ for(i=5; i<=NF; ++i) osd[$i]="busy" } else { unclean[$2]++ unclean_d[$3]++ si_mx=si_mx<$2 ? $2 : si_mx dsi_mx=dsi_mx<$3 ? $3 : dsi_mx pg_sn[$2]++ pg_sn_ids[$2]=pg_sn_ids[$2] " " $1 pg_dsn[$3]++ pg_dsn_ids[$3]=pg_dsn_ids[$3] " " $1 for(i=5; i<=NF; ++i) osd[$i]="busy" } } END { print "Scrub report:" for(si=1; si<=si_mx; ++si) { if(pg_sn[si]==0 && scrubbing[si]==0 && unclean[si]==0) continue; printf("%7d PGs not scrubbed since %2d intervals (6h)", pg_sn[si], si) if(scrubbing[si]) printf(" %d scrubbing", scrubbing[si]) if(unclean[si]) printf(" %d unclean", unclean[si]) if(pg_sn[si]<=5) { split(pg_sn_ids[si], pgs) osds_busy=0 for(pg in pgs) { split(pg_osds[pgs[pg]], osds) for(o in osds) if(osd[osds[o]]=="busy") osds_busy=1 if(osds_busy) printf(" %s*", pgs[pg]) if(!osds_busy) printf(" %s", pgs[pg]) } } printf("\n") } print "" print "Deep-scrub report:" for(dsi=1; dsi<=dsi_mx; ++dsi) { if(pg_dsn[dsi]==0 && deep_scrubbing[dsi]==0 && unclean_d[dsi]==0) continue; printf("%7d PGs not deep-scrubbed since %2d intervals (24h)", pg_dsn[dsi], dsi) if(deep_scrubbing[dsi]) printf(" %d scrubbing+deep", deep_scrubbing[dsi]) if(unclean_d[dsi]) printf(" %d unclean", unclean_d[dsi]) if(pg_dsn[dsi]<=5) { split(pg_dsn_ids[dsi], pgs) osds_busy=0 for(pg in pgs) { split(pg_osds[pgs[pg]], osds) for(o in osds) if(osd[osds[o]]=="busy") osds_busy=1 if(osds_busy) printf(" %s*", pgs[pg]) if(!osds_busy) printf(" %s", pgs[pg]) } } printf("\n") } print "" print "PGs marked with a * are on busy OSDs and not eligible for scrubbing." } ' Don't forget the last "'" when copy-pasting. Thanks for any pointers. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 6

Stuck in upgrade process to reef
by Jan Marek 08 Jan '24

08 Jan '24

Hello, I've problem: my ceph cluster (3x mon nodes, 6x osd nodes, every osd node have 12 rotational disk and one NVMe device for bluestore DB). CEPH is installed by ceph orchestrator and have bluefs storage on osd. I've started process upgrade from version 17.2.6 to 18.2.1 by invocating: ceph orch upgrade start --ceph-version 18.2.1 After upgrade of mon and mgr processes orchestrator tried to upgrade the first OSD node, but they are falling down. I've stop the process of upgrade, but I have 1 osd node completely down. After upgrade I've got some error messages and I've found /var/lib/ceph/crashxxxx directories, I attach to this message files, which I've found here. Please, can you advice, what now I can do? It seems, that rocksdb is even non-compatible or corrupted :-( Thanks in advance. Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html

2 8

Unable to find Refresh Interval Option in Ceph Dashboard (Ceph v18.2.1 "reef")- Seeking Assistance
by Alam Mohammad 04 Jan '24

04 Jan '24

Hello, We've been using Ceph for managing our storage infrastructure, and we recently upgraded to the latest version (Ceph v18.2.1 "reef"). However, We've noticed that the "refresh interval" option seems to be missing in the dashboard, and we are facing challenges with monitoring our cluster in real-time. In the earlier version of the Ceph dashboard, there was a useful "refresh interval" option that allowed us to customize the update frequency of the dashboard. This was particularly handy for monitoring changes and responding promptly. However, after the upgrade to Ceph v18.2.1 "reef", We can't seem to find this option anywhere in the dashboard. Additionally, we observed an automatic refresh occurring at every 25 seconds. Seeking guidance locating and tuning the refresh interval settings in the latest version of Ceph to potentially reduce this interval. We've explored the dashboard settings thoroughly and reviewed the release notes for Ceph v18.2.1 "reef", but we couldn't find any mention of the removal of the "refresh interval" option. Any guidance or insights would be greatly appreciated! Thanks, Mohammad Saif Ceph Enthusiast

3 2

Re: Ceph newbee questions
by Anthony D'Atri 03 Jan '24

03 Jan '24

>> >> You can do that for a PoC, but that's a bad idea for any production workload. You'd want at least three nodes with OSDs to use the default RF=3 replication. You can do RF=2, but at the peril of your mortal data. > > I'm not sure I agree - I think size=2, min_size=2 is no worse than > RAID1 for data security. size=2, min_size=2 *is* RAID1. Except that you become unavailable if a single drive is unavailable. >> That isn't even the main risk as I understand it. Of course a double > failure is going to be a problem with size=2, or traditional RAID1, > and I think anybody choosing this configuration accepts this risk. We see people often enough who don’t know that. I’ve seen double failures. ymmv. > As I understand it, the reason min_size=1 is a trap has nothing to do > with double failures per se. It’s one of the concerns. > > The issue is that Ceph OSDs are somewhat prone to flapping during > recovery (OOM, etc). So even if the disk is fine, an OSD can go down > for a short time. If you have size=2, min=1 configured, then when > this happens the PG will become degraded and will continue operating > on the other OSD, and the flapping OSD becomes stale. Then when it > comes back up it recovers. The problem is that if the other OSD has a > permanent failure (disk crash/etc) while the first OSD is flapping, > now you have no good OSDs, because when the flapping OSD comes back up > it is stale, and its PGs have no peer. Indeed, arguably that’s an overlapping failure. I’ve seen this too, and have a pg query to demonstrate it. > I suspect there are ways to re-activate it, though this will result in potential data > inconsistency since writes were allowed to the cluster and will then > get rolled back. Yep. > With only two OSDs I'm guessing that would be the > main impact (well, depending on journaling behavior/etc), but if you > have more OSDs than that then you could have situations where one file > is getting rolled back, and some other file isn't, and so on. But you’d have a voting majority. > > With min_size=2 you're fairly safe from flapping because there will > always be two replicas that have the most recent version of every PG, > and so you can still tolerate a permanent failure of one of them. Exactly. > > size=2, min=2 doesn't suffer this failure mode, because anytime there > is flapping the PG goes inactive and no writes can be made, so when > the other OSD comes back up there is nothing to recover. Of course > this results in IO blocks and downtime, which is obviously > undesirable, but it is likely a more recoverable state than > inconsistent writes. Agreed, the difference between availability and durability. Depends what’s important to you. > > Apologies if I've gotten any of that wrong, but my understanding is > that it is these sorts of failure modes that cause min_size=1 to be a > trap. This isn't the sort of thing that typically happens in a RAID1 > config, or at least that admins don't think about. It’s both.

2 4

ValueError: invalid literal for int() with base 10: '443 ssl_certificate=c
by Владимир Клеусов 02 Jan '24

02 Jan '24

Hi, When trying to log in to rgb via the dashboard, an error appears in the logs ValueError: invalid literal for int() with base 10: '443 ssl_certificate=config://rgw/cert/rgw.test' RGW with SSL If rgw is without ssl, everything works fine ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable) Please tell me how to solve this?

2 2

ceph-iscsi on RL9
by duluxoz 02 Jan '24

02 Jan '24

Hi All, Just successfully(?) completed a "live" update of the first node of a Ceph Quincy cluster from RL8 to RL9. Everything "seems" to be working - EXCEPT the iSCSI Gateway on that box. During the update the ceph-iscsi package was removed (ie `ceph-iscsi-3.6-2.g97f5b02.el8.noarch.rpm` - this is the latest package available from the Ceph Repos). So, obviously, I reinstalled the package. However, `dnf` is throwing errors (unsurprisingly, as that package is an el8 package and this box is now running el9): that package requires python 3.6 and el9 runs with python 3.8 (I believe). So my question(s) is: Can I simply "downgrade" python to 3.6, or is there an el9-compatible version of `ceph-iscsi` somewhere, and/or is there some process I need to follow to get the iSCSI Gateway back up and running? Some further info: The next step in my "happy-happy-fun-time-holiday-ICT-maintenance" was to upgrade the current Ceph Cluster to use `cephadm` and to go from Ceph-Quincy to Ceph-Reef - is this my ultimate upgrade path to get the iSCSI G/W back? BTW the Ceph Cluster is used *only* to provide iSCSI LUNS to an oVirt (KVM) Cluster front-end. Because it is the holidays I can take the entire network down (ie shutdown all the VMs) to facilitate this update process, which also means that I can use some other way (ie a non-iSCSI way - I think) to connect the Ceph SAN Cluster to the oVirt VM-Hosting Cluster - if *this* is the solution (ie no iSCSI) does someone have any experience in running oVirt off of Ceph in a non-iSCSI way - and could you be so kind as to provide some pointers/documentation/help? And before anyone says it, let me: "I broke, now I own it" :-) Thanks in advance, and everyone have a Merry Christmas, Heavenly Hanukkah, Quality Kwanzaa, Really-good (upcoming) Ramadan, and/or a Happy Holidays. Cheers Dulux-Oz

2 2

What is the maximum number of Rados gateway objects in one cluster using the bucket index and in one bucket?
by steve jung 02 Jan '24

02 Jan '24

Hello. We are using Ceph storage to test whether we can run the service by uploading and saving more than 40 billion files. So I'd like to check the contents below. 1) Maximum number of Rados gateway objects that can be stored in one cluster using the bucket index 2) Maximum number of Rados gateway objects that can be stored in one bucket Although we have referred to the limitations on the number of Rados gateway objects mentioned in existing documents, it seems theoretically unlimited If you have operated the number of files at the level we think in actual services or products, we would appreciate it if you could share them. Below are related documents and related settings values. > Related documents - https://documentation.suse.com/ses/5.5/html/ses-all/cha-ceph-gw.html - https://www.ibm.com/docs/en/storage-ceph/6?topic=resharding-limitations-buc… - https://docs.ceph.com/en/latest/dev/radosgw/bucket_index/ > Related config - rgw_dynamic_resharding: true - rgw_max_objs_per_shard: 100000 - rgw_max_dynamic_shards : 65521

2 1

mds generates slow request: peer_request, how to deal with it?
by David Yang 02 Jan '24

02 Jan '24

I hope this message finds you well. I have a cephfs cluster with 3 active mds, and use 3-node samba to export through the kernel. Currently, there are 2 node mds experiencing slow requests. We have tried restarting the mds. After a few hours, the replay log status became active. But the slow request reappears. The slow request does not seem to come from the client, but from the request of the mds node. Looking forward to your prompt response. HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests mds.osd44(mds.0): 2 slow requests are blocked > 30 secs mds.osd43(mds.1): 2 slow requests are blocked > 30 secs [WRN] MDS_TRIM: 2 MDSs behind on trimming mds.osd44(mds.0): Behind on trimming (18642/1024) max_segments: 1024, num_segments: 18642 mds.osd43(mds.1): Behind on trimming (976612/1024) max_segments: 1024, num_segments: 976612 mds.0 { "ops": [ { "description": "peer_request:mds.1:1", "initiated_at": "2023-12-31T11:19:38.679925+0800", "age": 4358.8009461359998, "duration": 4358.8009636369998, "type_data": { "flag_point": "dispatched", "reqid": "mds.1:1", "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2023-12-31T11:19:38.679925+0800", "event": "initiated" }, { "time": "2023-12-31T11:19:38.679925+0800", "event": "throttled" }, { "time": "2023-12-31T11:19:38.679925+0800", "event": "header_read" }, { "time": "2023-12-31T11:19:38.679936+0800", "event": "all_read" }, { "time": "2023-12-31T11:19:38.679940+0800", "event": "dispatched" } ] } }, { "description": "peer_request:mds.1:2", "initiated_at": "2023-12-31T11:19:38.679938+0800", "age": 4358.8009326969996, "duration": 4358.8009763549999, "type_data": { "flag_point": "dispatched", "reqid": "mds.1:2", "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2023-12-31T11:19:38.679938+0800", "event": "initiated" }, { "time": "2023-12-31T11:19:38.679938+0800", "event": "throttled" }, { "time": "2023-12-31T11:19:38.679938+0800", "event": "header_read" }, { "time": "2023-12-31T11:19:38.679941+0800", "event": "all_read" }, { "time": "2023-12-31T11:19:38.679991+0800", "event": "dispatched" } ] } } ], "complaint_time": 30, "num_blocked_ops": 2 } mds.1 { "ops": [ { "description": "internal op exportdir:mds.1:1", "initiated_at": "2023-12-31T11:19:34.416451+0800", "age": 4384.38814198, "duration": 4384.3881617610004, "type_data": { "flag_point": "failed to wrlock, waiting", "reqid": "mds.1:1", "op_type": "internal_op", "internal_op": 5377, "op_name": "exportdir", "events": [ { "time": "2023-12-31T11:19:34.416451+0800", "event": "initiated" }, { "time": "2023-12-31T11:19:34.416451+0800", "event": "throttled" }, { "time": "2023-12-31T11:19:34.416451+0800", "event": "header_read" }, { "time": "2023-12-31T11:19:34.416451+0800", "event": "all_read" }, { "time": "2023-12-31T11:19:34.416451+0800", "event": "dispatched" }, { "time": "2023-12-31T11:19:38.679923+0800", "event": "requesting remote authpins" }, { "time": "2023-12-31T11:19:38.693981+0800", "event": "failed to wrlock, waiting" } ] } }, { "description": "internal op exportdir:mds.1:2", "initiated_at": "2023-12-31T11:19:34.416482+0800", "age": 4384.3881117999999, "duration": 4384.3881714600002, "type_data": { "flag_point": "failed to wrlock, waiting", "reqid": "mds.1:2", "op_type": "internal_op", "internal_op": 5377, "op_name": "exportdir", "events": [ { "time": "2023-12-31T11:19:34.416482+0800", "event": "initiated" }, { "time": "2023-12-31T11:19:34.416482+0800", "event": "throttled" }, { "time": "2023-12-31T11:19:34.416482+0800", "event": "header_read" }, { "time": "2023-12-31T11:19:34.416482+0800", "event": "all_read" }, { "time": "2023-12-31T11:19:34.416482+0800", "event": "dispatched" }, { "time": "2023-12-31T11:19:38.679929+0800", "event": "requesting remote authpins" }, { "time": "2023-12-31T11:19:38.693995+0800", "event": "failed to wrlock, waiting" } ] } } ], "complaint_time": 30, "num_blocked_ops": 2 } I can't find any other solution other than restarting the mds service with slow requests. Currently, the backlog of mds logs in the metadata pool exceeds 4TB. Best regards,

2 2

cephadm - podman vs docker
by Murilo Morais 31 Dec '23

31 Dec '23

Good morning everybody! Guys, are there any differences or limitations when using Docker instead of Podman? Context: I have a cluster with Debian 11 running Podman (3.0.1), but the iSCSI service, when restarted, the "tcmu-runner" binary is in "Z State" and the "rbd-target-api" script enters "D State" and never dies, causing the service not to start until I perform a reboot. On machines that use distributions based on Red Hat with podman 4+ this behavior does not happen. I don't want to use a repository that I don't know about just to update podman. I haven't tested it with Debian 12 yet, as we experienced some problems with bootstrap, so we decided to use Debian 11. I'm thinking about testing with Docker but I don't know what the difference is between both solutions in the CEPH context.

2 1

About slow query of Block-Images
by Phong Tran Thanh 29 Dec '23

29 Dec '23

Hi community, When I list images of rbd in ceph dashboard, Block->Images, list image is too slow to view, how can i get it faster. I am using ceph reef version 18.2.1 Thanks to the community. *Tran Thanh Phong* Email: tranphong079(a)gmail.com Skype: tranphong079

1 0

Re: RGW requests piling up
by Gauvain Pocentek 29 Dec '23

29 Dec '23

I'd like to say that it was something smart but it was a bit of luck. I logged in on a hypervisor (we run OSDs and OpenStack hypervisors on the same hosts) to deal with another issue, and while checking the system I noticed that one of the OSDs was using a lot more CPU than the others. It made me think that the increased IOPS could put a strain on some of the OSDs without impacting the whole cluster so I decided to increate pg_num to spread the operations to more OSDs, and it did the trick. The qlen metric went back to something similar to what we had before the problems started. We're going to look into adding CPU/RAM monitoring for all the OSDs next. Gauvain On Fri, Dec 22, 2023 at 2:58 PM Drew Weaver <drew.weaver(a)thenap.com> wrote: > Can you say how you determined that this was a problem? > > -----Original Message----- > From: Gauvain Pocentek <gauvainpocentek(a)gmail.com> > Sent: Friday, December 22, 2023 8:09 AM > To: ceph-users(a)ceph.io > Subject: [ceph-users] Re: RGW requests piling up > > Hi again, > > It turns out that our rados cluster wasn't that happy, the rgw index pool > wasn't able to handle the load. Scaling the PG number helped (256 to 512), > and the RGW is back to a normal behaviour. > > There is still a huge number of read IOPS on the index, and we'll try to > figure out what's happening there. > > Gauvain > > On Thu, Dec 21, 2023 at 1:40 PM Gauvain Pocentek < > gauvainpocentek(a)gmail.com> > wrote: > > > Hello Ceph users, > > > > We've been having an issue with RGW for a couple days and we would > > appreciate some help, ideas, or guidance to figure out the issue. > > > > We run a multi-site setup which has been working pretty fine so far. > > We don't actually have data replication enabled yet, only metadata > > replication. On the master region we've started to see requests piling > > up in the rgw process, leading to very slow operations and failures > > all other the place (clients timeout before getting responses from > > rgw). The workaround for now is to restart the rgw containers regularly. > > > > We've made a mistake and forcefully deleted a bucket on a secondary > > zone, this might be the trigger but we are not sure. > > > > Other symptoms include: > > > > * Increased memory usage of the RGW processes (we bumped the container > > limits from 4G to 48G to cater for that) > > * Lots of read IOPS on the index pool (4 or 5 times more compared to > > what we were seeing before) > > * The prometheus ceph_rgw_qlen and ceph_rgw_qactive metrics (number of > > active requests) seem to show that the number of concurrent requests > > increases with time, although we don't see more requests coming in on > > the load-balancer side. > > > > The current thought is that the RGW process doesn't close the requests > > properly, or that some requests just hang. After a restart of the > > process things look OK but the situation turns bad fairly quickly > > (after 1 hour we start to see many timeouts). > > > > The rados cluster seems completely healthy, it is also used for rbd > > volumes, and we haven't seen any degradation there. > > > > Has anyone experienced that kind of issue? Anything we should be > > looking at? > > > > Thanks for your help! > > > > Gauvain > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io >

1 1

Fwd: ceph-dashboard odd behavior when visiting through haproxy
by Demian Romeijn 28 Dec '23

28 Dec '23

Hi ceph-users, I'm not sure if this mail got send correctly, my colleague seems to not have received it. Either way. We've managed to replicate this issue from a local http_check test. The ceph-mgr seems to go down with every first visit, and works perfectly fine right after a couple re-visits. Has anyone else seen this issue before? Thanks in advance, - Demian From: Demian Romeijn <dromeijn(a)tuxis.nl> To: <ceph-users(a)ceph.io> Sent: 12/22/2023 2:14 PM Subject: ceph-dashboard odd behavior when visiting through haproxy I'm currently trying to setup a ceph-dashboard using the official documentation on how to do so. I've managed to log-in by just visiting the URL & port, and by visting it through haproxy. However using haproxy to visit the site results in odd behavior. At my first login, nothing loads on the page and eventually at ~5s it times me out, sending me back to the log-in screen. After logging back on to the dashboard, everything loads and functions as expected. I can refresh my browser as many times as I want and it still keeps on working. After some time, usually ~30 minutes or so of inactivity, the problem arises again. Haproxy tells us the server is down for about ~10 seconds, running a simple HTTP check results in the following aswell: CRITICAL - Socket timeout after 10 seconds. In the ceph-mgr logs there isn't any special error other than: [dashboard ERROR frontend.error] (https://*redacted*/#/login): Http failure response for https://*redacted*/ui-api/orchestrator/get_name: 401 OK None It seems as such the ceph dashboard is "overloaded", changing haproxy config (following the official ceph documentation on how to set it up) to do health-checks less often results in the problem happening less often. Anything I might've overlooked that could sort out the issue?

1 0

About lost disk with erasure code
by Phong Tran Thanh 28 Dec '23

28 Dec '23

Hi community, I am running ceph with block rbd with 6 nodes, erasure code 4+2 with min_size of pool is 4. When three osd is down, and an PG is state down, some pools is can't write data, suppose three osd can't start and pg stuck in down state, how i can delete or recreate pg to replace down pg or another way to allow pool to write/read data? Thanks for the community *Tran Thanh Phong* Email: tranphong079(a)gmail.com Skype: tranphong079

3 4

mds crashes with 18.2.1
by Andrej Filipčič 27 Dec '23

27 Dec '23

Hi, I just upgraded from 17.2.6 to 18.2.1 and have some issues with mds. mds started crashing with 2023-12-27T13:21:30.491+0100 7f717b5886c0 1 mds.f9sn015 Updating MDS map to version 2689280 from mon.5 2023-12-27T13:21:30.491+0100 7f717b5886c0 1 mds.0.2689276 handle_mds_map i am now mds.0.2689276 2023-12-27T13:21:30.491+0100 7f717b5886c0 1 mds.0.2689276 handle_mds_map state change up:clientreplay --> up:active 2023-12-27T13:21:30.491+0100 7f717b5886c0 1 mds.0.2689276 active_start 2023-12-27T13:21:30.524+0100 7f717b5886c0 1 mds.0.2689276 cluster recovered. 2023-12-27T13:21:30.551+0100 7f7176d7f6c0 -1 /var/tmp/portage/sys-cluster/ceph-18.2.1-r2/work/ceph-18.2.1/src/mds/Server.cc: In funct ion 'CInode* Server::prepare_new_inode(MDRequestRef&, CDir*, inodeno_t, unsigned int, const file_layout_t*)' thread 7f7176d7f6c0 time 2023-12-27T13:21:30.548697+0100 /var/tmp/portage/sys-cluster/ceph-18.2.1-r2/work/ceph-18.2.1/src/mds/Server.cc: 3441: FAILED ceph_assert(_inode->gid != (unsigned)-1) and I could not bring it back again. As a workaround I was able to start mds 17.2.6 and it somehow recovered. Then I started 18 mds again, which soon after startup finds this corruption: [ { "damage_type": "dentry", "id": 4247331390, "ino": 1, "frag": "*", "dname": "lost+found", "snap_id": "head", "path": "/lost+found" } ] There are few corrupted files in some other directories ( leftovers from several releases before I never managed to fix), and if I start mds scrub there, mds crashes again, maybe because of corrupted lost+found. If I try to remove lost+found, mds crashes again. Do you have any hint how to recover from this? Best regards, Andrej -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-477-3166 -------------------------------------------------------------

1 1

Consistent OSD crashes for ceph 17.2.5 which is causing osd up and down
by Akash Warkhade 27 Dec '23

27 Dec '23

We are running rook-ceph deployed as a operator in kubernetes with rook version 1.10.8 and ceph 17.2.5. Its working fine but we are seeing frequent OSD daemon crash in 3-4 days and restarts without any problem also we are seeing flapping osds i.e osd up down. Recently daemon crash happened for 2 OSDs at same time on different nodes with below error in crash info : -305> 2023-12-17T14:50:14.413+0000 7f53b5f91700 -1 *** Caught signal (Aborted) ** in thread 7f53b5f91700 thread_name:tp_osd_tp ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f53d93ddcf0] 2: gsignal() 3: abort() 4: /lib64/libc.so.6(+0x21d79) [0x7f53d8025d79] 5: /lib64/libc.so.6(+0x47456) [0x7f53d804b456] 6: (MOSDRepOp::encode_payload(unsigned long)+0x2d0) [0x55acc0f81730] 7: (Message::encode(unsigned long, int, bool)+0x2e) [0x55acc140ec2e] 8: (ProtocolV2::send_message(Message*)+0x25e) [0x55acc16a5aae] 9: (AsyncConnection::send_message(Message*)+0x18e) [0x55acc167dc4e] 10: (OSDService::send_message_osd_cluster(int, Message*, unsigned int)+0x2bd) [0x55acc0b4b11d] 11: (ReplicatedBackend::issue_op(hobject_t const&, eversion_t const&, unsigned long, osd_reqid_t, eversion_t, eversion_t, hobject_t, hobject_t, std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t>&, ReplicatedBackend::InProgressOp*, ceph::os::Transaction&)+0x6c8) [0x55acc0f69368] 12: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >&&, eversion_t const&, eversion_t const&, std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> >&&, std::optional<pg_hit_set_history_t>&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr<OpRequest>)+0x5e7) [0x55acc0f6c907] 13: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x50d) [0x55acc0c92ebd] 14: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xd25) [0x55acc0cf0295] 15: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x288d) [0x55acc0cf78fd] 16: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1c0) [0x55acc0b56900] 17: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x6d) [0x55acc0e552ad] 18: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55acc0b69dbf] 19: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55acc12c78c5] 20: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55acc12c9fe4] 21: /lib64/libpthread.so.0(+0x81ca) [0x7f53d93d31ca] 22: clone() It has also has below errors before the crash: scrub-queue::*remove_from_osd_queue* removing pg[2.4f0] failed. State was: unregistering Please help to troubleshoot the issue and fix it Already posted on ceph tracker but no reply there since 3-4 days

1 0

CephFS delayed deletion
by Miroslav Svoboda 26 Dec '23

26 Dec '23

Hi, how can I increase a files deletion speed? Every files was deleted from cephfs on my pool, but ceph df still show 50% usage of pool. I know about delayed deletion (https://docs.ceph.com/en/latest) but is there some way to little speed up this? I significantly increase mds_max_purge_ops and mds_max_purge_files, but this is not helped me. Thanks for response. Svoboda Miroslav

1 0

Building new cluster had a couple of questions
by Drew Weaver 26 Dec '23

26 Dec '23

Howdy, I am going to be replacing an old cluster pretty soon and I am looking for a few suggestions. #1 cephadm or ceph-ansible for management? #2 Since the whole... CentOS thing... what distro appears to be the most straightforward to use with Ceph? I was going to try and deploy it on Rocky 9. That is all I have. Thanks, -Drew

11 20

Reef v18.2.1: ceph osd pool autoscale-status gives empty output
by Jayanth Reddy 25 Dec '23

25 Dec '23

Hello Users, I deployed a new cluster with v18.2.1 but noticed that pg_num and pgp_num always remained 1 for the pools with autoscale turned on. Below is the env and the relevant information ceph> version ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) ceph> status cluster: id: 273c8410-a333-11ee-b3c2-9791c3098e2b health: HEALTH_WARN clock skew detected on mon.ec-rgw-s3 services: mon: 3 daemons, quorum ec-rgw-s1,ec-rgw-s2,ec-rgw-s3 (age 48m) mgr: ec-rgw-s1.icpgxx(active, since 69m), standbys: ec-rgw-s2.quzjfv osd: 3 osds: 3 up (since 29m), 3 in (since 49m) rgw: 1 daemon active (1 hosts, 1 zones) data: pools: 8 pools, 23 pgs objects: 1.85k objects, 6.0 GiB usage: 4.8 GiB used, 595 GiB / 600 GiB avail pgs: 23 active+clean ceph> osd pool get noautoscale noautoscale is off ceph> osd pool autoscale-status ceph> osd pool autoscale-status ceph> osd pool ls detail pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 3.00 pool 2 'default.rgw.buckets.data' erasure profile ec-21 size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode off last_change 111 lfor 0/0/55 flags hashpspool stripe_width 8192 compression_algorithm lz4 compression_mode force application rgw pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 34 flags hashpspool stripe_width 0 application rgw read_balance_score 3.00 pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 37 flags hashpspool stripe_width 0 application rgw read_balance_score 3.00 pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 39 flags hashpspool stripe_width 0 application rgw read_balance_score 3.00 pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 41 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 3.00 pool 7 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 44 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 3.00 pool 8 'default.rgw.buckets.non-ec' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 47 flags hashpspool stripe_width 0 application rgw read_balance_score 3.00 I'd manually changed for pool ID 2. Is this by any chance due to PR [1]? [1] https://github.com/ceph/ceph/pull/53658 Thanks, Jayanth

1 1

OSD is usable, but not shown in "ceph orch device ls"
by E Taka 23 Dec '23

23 Dec '23

Hello, in our cluster we have one node with SSD, which are used, but we cannot see it in "ceph orch device ls". Everything als looks OK. For better understanding, the diskname is /dev/sda, it's osd.138: ~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 1 7T 0 disk ~# wipefs /dev/sda DEVICE OFFSET TYPE UUID LABEL sda 0x0 ceph_bluestore ~# ceph osd tree -9 15.42809 host ceph06 138 ssd 6.98630 osd.138 up 1.00000 1.00000 The file ceph-osd.138.log does not look unusal to me. ceph-volume.log show that the SSD is found by the "lsblk" command of the volume processing. It is not possible to add the SSD by "# ceph orch daemon add osd ceph06:/dev/sda Error message in this case is a question asking if it is already used, even if the SSD is fully wiped via "wipefs -a" or by overwriting the entire disk with the dd command. But It is possible to add it to the cluster by using the option "--method raw". Do you have an idea what happened here and how can I debug this behaviour?

1 0

Ceph newbee questions
by Marcus 22 Dec '23

22 Dec '23

Hi all, I am all new with ceph and I come from gluster. We have had our eyes on ceph for several years and as the gluster project seems to slow down we now think it is time to start look into ceph. I have manually configured a ceph cluster with ceph fs on debian bookworm. What is the difference from installing with cephadm compared to manuall install, any benefits that you miss with manual install? There are also another couple of things that I can not figure out reading the documentation. Most of our files are small and from my understanding replication is then recomended, right? The plan is to set ceph up like this: 1 x "admin node" 2 x "storage nodes" The admin node will run mon, mgr and mds. The storage nodes will run mon, mgr, mds and 8x osd (8 disks). This works well to setup but I can not get my head around is how things are replicated over nodes and disks. In ceth.conf I set the folowing: osd pool default size = 2 osd pool default min size = 1 So the idea is that we always have 2 copies of the data. I do not seem to be able to figure out the replication when things starts to fail. If the admin node goes down, one of the data nodes will run the mon, mgr and mds. This will slow things down but will be fine until we have a new admin node in place again. (or if there is something I am missing here?) If just one data node goes down we will still not loose any data and that is fine until we have a new server. But what if one data node goes down and one disk of the other data node breaks, will I loose data then? Or how many disks can I loose before I loose data? This is what I can not get my head around, how to think when disaster strikes, how much hardware can I loose before I loose data? Or have I got it all wrong? Is it a bad idea with just 2 fileservers is more servers required? The second thing I have a problem with is snapshots. I manage to create snapshot in root with command: ceph fs subvolume snapshot create <vol_name> / <snap_name> But it fails if I try to create a shapshot in any other directory then in the root. Second of all if I try to create a snapshot from the client with: mkdir /mnt-ceph/.snap/my_snapshot I get the same error in all directories: Permission dened. I have not found any sollution to this, am I missing something here as well? Any config missing? Many thanks for your support!! Best regrads Marcus

3 2

ceph-dashboard odd behavior when visiting through haproxy
by Demian Romeijn 22 Dec '23

22 Dec '23

I'm currently trying to setup a ceph-dashboard using the official documentation on how to do so. I've managed to log-in by just visiting the URL & port, and by visting it through haproxy. However using haproxy to visit the site results in odd behavior. At my first login, nothing loads on the page and eventually at ~5s it times me out, sending me back to the log-in screen. After logging back on to the dashboard, everything loads and functions as expected. I can refresh my browser as many times as I want and it still keeps on working. After some time, usually ~30 minutes or so of inactivity, the problem arises again. Haproxy tells us the server is down for about ~10 seconds, running a simple HTTP check results in the following aswell: CRITICAL - Socket timeout after 10 seconds. In the ceph-mgr logs there isn't any special error other than: [dashboard ERROR frontend.error] (https://*redacted*/#/login): Http failure response for https://*redacted*/ui-api/orchestrator/get_name: 401 OK None It seems as such the ceph dashboard is "overloaded", changing haproxy config (following the official ceph documentation on how to set it up) to do health-checks less often results in the problem happening less often. Anything I might've overlooked that could sort out the issue?

1 0

RGW requests piling up
by Gauvain Pocentek 22 Dec '23

22 Dec '23

Hello Ceph users, We've been having an issue with RGW for a couple days and we would appreciate some help, ideas, or guidance to figure out the issue. We run a multi-site setup which has been working pretty fine so far. We don't actually have data replication enabled yet, only metadata replication. On the master region we've started to see requests piling up in the rgw process, leading to very slow operations and failures all other the place (clients timeout before getting responses from rgw). The workaround for now is to restart the rgw containers regularly. We've made a mistake and forcefully deleted a bucket on a secondary zone, this might be the trigger but we are not sure. Other symptoms include: * Increased memory usage of the RGW processes (we bumped the container limits from 4G to 48G to cater for that) * Lots of read IOPS on the index pool (4 or 5 times more compared to what we were seeing before) * The prometheus ceph_rgw_qlen and ceph_rgw_qactive metrics (number of active requests) seem to show that the number of concurrent requests increases with time, although we don't see more requests coming in on the load-balancer side. The current thought is that the RGW process doesn't close the requests properly, or that some requests just hang. After a restart of the process things look OK but the situation turns bad fairly quickly (after 1 hour we start to see many timeouts). The rados cluster seems completely healthy, it is also used for rbd volumes, and we haven't seen any degradation there. Has anyone experienced that kind of issue? Anything we should be looking at? Thanks for your help! Gauvain

1 1

Is there a way to find out which client uses which version of ceph?
by Simon Oosthoek 21 Dec '23

21 Dec '23

Hi, Our cluster is currently running quincy, and I want to set the minimal client version to luminous, to enable upmap balancer, but when I tried to, I got this: # ceph osd set-require-min-compat-client luminous Error EPERM: cannot set require_min_compat_client to luminous: 2 connected client(s) look like jewel (missing 0x800000000000000); add --yes-i-really-mean-it to do it anyway I think I know the most likely candidate (and I've asked them), but is there a way to find out, the way ceph seems to know? tnx /Simon -- I'm using my gmail.com address, because the gmail.com dmarc policy is "none", some mail servers will reject this (microsoft?) others will instead allow this when I send mail to a mailling list which has not yet been configured to send mail "on behalf of" the sender, but rather do a kind of "forward". The latter situation causes dkim/dmarc failures and the dmarc policy will be applied. see https://wiki.list.org/DEV/DMARC for more details

3 3

AssumeRoleWithWebIdentity with ServiceAccounts and IAM Roles
by Charlie Savage 21 Dec '23

21 Dec '23

Is it possible to configure Ceph so that STS AssumeRoleWithWebIdentity works with a Kubernetes serviceaccount token? My goal is that a pod running in a Kubernetes cluster can call AssumeRoleWithWebIdentity specifying an IAM role (previously created in Ceph) and the Kubernetes oicd service account token and get back a valid access key and secret. This would then be used to access objects in buckets hosted by Ceph object storage. This would allow our code to run unchanged between the cloud (S3) and on premise (Ceph providing object storage). Original AWS document is here - https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-… Minio implementation is here - https://min.io/docs/minio/kubernetes/upstream/developers/sts-for-operator.h… Kubernetes OIDC endpoints (Service account issuer discovery) discussed here - https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-… I have setup a Ceph role that specifies an oicd url pointing to Kubernetes API server and passes the service token. But I still need to enable STS in ceph I believe, and have ceph talk to Kubernetes oicd. Before continuing though I am wondering if this setup is supported? Thanks, Charlie

1 0

FS down
by Sake 21 Dec '23

21 Dec '23

1 0

v18.2.1 Reef released
by Yuri Weinstein 20 Dec '23

20 Dec '23

We're happy to announce the 1st backport release in the Reef series. This is the first backport release in the Reef series, and the first with Debian packages, for Debian Bookworm. We recommend all users update to this release. https://ceph.io/en/news/blog/2023/v18-2-1-reef-released/ Notable Changes --------------- * RGW: S3 multipart uploads using Server-Side Encryption now replicate correctly in multi-site. Previously, the replicas of such objects were corrupted on decryption. A new tool, ``radosgw-admin bucket resync encrypted multipart``, can be used to identify these original multipart uploads. The ``LastModified`` timestamp of any identified object is incremented by 1ns to cause peer zones to replicate it again. For multi-site deployments that make any use of Server-Side Encryption, we recommended running this command against every bucket in every zone after all zones have upgraded. * CEPHFS: MDS evicts clients which are not advancing their request tids which causes a large buildup of session metadata resulting in the MDS going read-only due to the RADOS operation exceeding the size threshold. `mds_session_metadata_threshold` config controls the maximum size that a (encoded) session metadata can grow. * RGW: New tools have been added to radosgw-admin for identifying and correcting issues with versioned bucket indexes. Historical bugs with the versioned bucket index transaction workflow made it possible for the index to accumulate extraneous "book-keeping" olh entries and plain placeholder entries. In some specific scenarios where clients made concurrent requests referencing the same object key, it was likely that a lot of extra index entries would accumulate. When a significant number of these entries are present in a single bucket index shard, they can cause high bucket listing latencies and lifecycle processing failures. To check whether a versioned bucket has unnecessary olh entries, users can now run ``radosgw-admin bucket check olh``. If the ``--fix`` flag is used, the extra entries will be safely removed. A distinct issue from the one described thus far, it is also possible that some versioned buckets are maintaining extra unlinked objects that are not listable from the S3/ Swift APIs. These extra objects are typically a result of PUT requests that exited abnormally, in the middle of a bucket index transaction - so the client would not have received a successful response. Bugs in prior releases made these unlinked objects easy to reproduce with any PUT request that was made on a bucket that was actively resharding. Besides the extra space that these hidden, unlinked objects consume, there can be another side effect in certain scenarios, caused by the nature of the failure mode that produced them, where a client of a bucket that was a victim of this bug may find the object associated with the key to7fe91d5d5842e04be3b4f514d6dd990c54b29c76 be in an inconsistent state. To check whether a versioned bucket has unlinked entries, users can now run ``radosgw-admin bucket check unlinked``. If the ``--fix`` flag is used, the unlinked objects will be safely removed. Finally, a third issue made it possible for versioned bucket index stats to be accounted inaccurately. The tooling for recalculating versioned bucket stats also had a bug, and was not previously capable of fixing these inaccuracies. This release resolves those issues and users can now expect that the existing ``radosgw-admin bucket check`` command will produce correct results. We recommend that users with versioned buckets, especially those that existed on prior releases, use these new tools to check whether their buckets are affected and to clean them up accordingly. * mgr/snap-schedule: For clusters with multiple CephFS file systems, all the snap-schedule commands now expect the '--fs' argument. * RADOS: A POOL_APP_NOT_ENABLED health warning will now be reported if the application is not enabled for the pool irrespective of whether the pool is in use or not. Always add ``application`` label to a pool to avoid reporting of POOL_APP_NOT_ENABLED health warning for that pool. The user might temporarilty mute this warning using ``ceph health mute POOL_APP_NOT_ENABLED``. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-18.2.1.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 7fe91d5d5842e04be3b4f514d6dd990c54b29c76

7 8

Re: cephadm file "/sbin/cephadm", line 10098 PK ^
by farhad kh 20 Dec '23

20 Dec '23

hi,thank you for guidance There is no ability to change the global image before launching,I need to download the images from the private registry during the initial setup. i used option --image but it not worked. # cephadm bootstrap --image rgistry.test/ceph/ceph:v18 --mon-ip 192.168.0.160 --initial-dashboard-password P@ssw0rd --dashboard-password-noupdate --allow-fqdn-hostname --ssh-user cephadmin usage: cephadm [-h] [--image IMAGE] [--docker] [--data-dir DATA_DIR] [--log-dir LOG_DIR] [--logrotate-dir LOGROTATE_DIR] [--sysctl-dir SYSCTL_DIR] [--unit-dir UNIT_DIR] [--verbose] [--timeout TIMEOUT] [--retry RETRY] [--env ENV] [--no-container-init] [--no-cgroups-split] {version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,zap-osds,unit,logs,bootstrap,deploy,_orch,check-host,prepare-host,add-repo,rm-repo,install,registry-login,gather-facts,host-maintenance,agent,disk-rescan} ... cephadm: error: unrecognized arguments: --image rgistry.test/ceph/ceph:v18 also i used cephadm registry-login and it show loged but when i bootstrap first node trying download image from quay registry. cephadm bootstrap --mon-ip 192.168.0.160 --registry-json /root/mylogin.json --initial-dashboard-password P@ssw0rd --dashboard-password-noupdate --allow-fqdn-hostname --ssh-user cephadmin Creating directory /etc/ceph for ceph.conf Verifying ssh connectivity using standard pubkey authentication ... Adding key to cephadmin@localhost authorized_keys... Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit chronyd.service is enabled and running Repeating the final host check... docker (/usr/bin/docker) is present systemctl is present lvcreate is present Unit chronyd.service is enabled and running Host looks OK Cluster fsid: 3c00e38c-9e2e-11ee-95cd-000c29e9f44e Verifying IP 192.168.0.160 port 3300 ... Verifying IP 192.168.0.160 port 6789 ... Mon IP `192.168.0.160` is in CIDR network `192.168.0.0/24` Mon IP `192.168.0.160` is in CIDR network `192.168.0.0/24` Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network Pulling custom registry login info from /root/mylogin.json. Logging into custom registry. Pulling container image quay.io/ceph/ceph:v18... befor i edited cephadm script but now file in coded and can't edited. and i don't know how can fix it :(((

3 2

CLT Meeting Minutes 2023-12-20
by Laura Flores 20 Dec '23

20 Dec '23

(Zac Dover) "make check" fails now even for some docs builds. For example: https://github.com/ceph/ceph/pull/54970, which is a simple edit of ReStructured Text in doc/radosgw/compression.rst. Greg Farnum and Dan Mick have already done preliminary investigation of this matter here: https://ceph-storage.slack.com/archives/C1HFJ4VTN/p1703048785756359. - Follow Slack thread for updates; we'll continue looking into it Still 38 PRs to scrub for 16.2.15: https://github.com/ceph/ceph/pulls?q=is%3Aopen+is%3Apr+milestone%3Apacific - Looking for PRs that are necessary in the release, as well as non-trivial PRs that have been open for a while -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

Logging control
by Tim Holloway 20 Dec '23

20 Dec '23

Ceph version is Pacific (16.2.14), upgraded from a sloppy Octopus. I ran afoul of all the best bugs in Octopus, and in the process switched on a lot of stuff better left alone, including some detailed debug logging. Now I can't turn it off. I am confidently informed by the documentation that the first step would be the command: ceph daemon osd.1 config show | less But instead of config information I get back: Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" Which seems to be kind of insane. Attempting to get daemon config info on a monitor on that machine gives: admin_socket: exception getting command descriptions: [Errno 2] No such file or directory Which doesn't help either. Anyone got an idea?

4 6

Re: Support of SNMP on CEPH ansible
by Lokendra Rathour 20 Dec '23

20 Dec '23

Hi Team, please help in the reference of the issue raised. Best Regards, Lokendra On Wed, Dec 13, 2023 at 2:33 PM Kushagr Gupta <kushagrguptasps.mun(a)gmail.com> wrote: > Hi Team, > > *Environment:* > We have deployed a ceph setup using ceph-ansible. > Ceph-version: 18.2.0 > OS: Almalinux 8.8 > We have a 3 node-setup. > > *Queries:* > > 1. Is SNMP supported for ceph-ansible?Is there some other way to setup > SNMP gateway for the ceph cluster? > 2. Do we have a procedure to set backend for ceph-orchestrator via > ceph-ansible? Which backend to use? > 3. Are there any CEPH MIB files which work independent of prometheus. > > > *Description:* > We are trying to perform SNMP monitoring for the ceph cluster using the > following link: > > 1. > https://docs.ceph.com/en/quincy/cephadm/services/snmp-gateway/#:~:text=Ceph's%20SNMP%20integration%20focuses%20on,a%20designated%20SNMP%20management%20platform > . > 2. > https://www.ibm.com/docs/en/storage-ceph/7?topic=traps-deploying-snmp-gatew… > > But when we try to follow the steps mentioned in the above link, we get > the following error when we try to run any "ceph orch" we get the following > error: > "Error ENOENT: No orchestrator configured (try `ceph orch set backend`)" > > After going through following links: > 1. > https://www.ibm.com/docs/en/storage-ceph/5?topic=operations-use-ceph-orches… > 2. > https://forum.proxmox.com/threads/ceph-mgr-orchestrator-enabled-but-showing… > 3. https://docs.ceph.com/en/latest/mgr/orchestrator_modules/ > I think since we have deployed the cluster using ceph-ansible, we can't > use the ceph-orch commands. > When we checked in the cluster, the following are the enabled modules: > " > [root@storagenode1 ~]# ceph mgr module ls > MODULE > balancer on (always on) > crash on (always on) > devicehealth on (always on) > orchestrator on (always on) > pg_autoscaler on (always on) > progress on (always on) > rbd_support on (always on) > status on (always on) > telemetry on (always on) > volumes on (always on) > alerts on > iostat on > nfs on > prometheus on > restful on > dashboard - > influx - > insights - > localpool - > mds_autoscaler - > mirroring - > osd_perf_query - > osd_support - > rgw - > selftest - > snap_schedule - > stats - > telegraf - > test_orchestrator - > zabbix - > [root@storagenode1 ~]# > " > As can be seen above, orchestrator is on. > > Also, We were exploring more about snmp and as per the file: > "/etc/prometheus/ceph/ceph_default_alerts.yml" on the ceph storage, the > OIDs in the file represents the OID for ceph components via prometheus. > For example: > for the following OID: 1.3.6.1.4.1.50495.1.2.1.2.1 > [root@storagenode3 ~]# snmpwalk -v 2c -c 209ijvfwer0df92jd -O e 10.0.1.36 > 1.3.6.1.4.1.50495.1.2.1.2.1 > CEPH-MIB::promHealthStatusError = No Such Object available on this agent > at this OID > [root@storagenode3 ~]# > > Kindly help us for the same. > > Thanks and regards, > Kushagra Gupta > -- ~ Lokendra skype: lokendrarathour

3 2

Re: Can not activate some OSDs after upgrade (bad crc on label)
by Huseyin Cotuk 20 Dec '23

20 Dec '23

Hello again, We understood that the issue arises from a hardware crash with the help of Dan van der Ster and Mykola from Clyso. After upgrading Ceph, we encountered an unexpected crash resulted with a reboot. After comparing the first blocks of running and failed OSDs, we found that HW crash caused a corruption on the first 23 bytes of block devices. First few bytes of the block device of a failed OSD contains: 00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000010: 0000 0000 0000 0a63 3965 6533 6566 362d .......c9ee3ef6- 00000020: 3733 6437 2d34 3032 392d 3963 6436 2d30 73d7-4029-9cd6-0 00000030: 3836 6363 3935 6432 6632 370a 0201 a901 86cc95d2f27..... 00000040: 0000 c9ee 3ef6 73d7 4029 9cd6 086c c95d ....>.s.@)...l.] 00000050: 2f27 0000 4002 4707 0000 8ceb 6665 c827 /'..@.G.....fe.' 00000060: 0409 0400 0000 6d61 696e 0d00 0000 0a00 ......main…… and a running OSD contains: 00000000: 626c 7565 7374 6f72 6520 626c 6f63 6b20 bluestore block 00000010: 6465 7669 6365 0a38 6637 3732 3532 312d device.8f772521- 00000020: 6535 3663 2d34 6135 622d 6239 3763 2d31 e56c-4a5b-b97c-1 00000030: 6233 3630 6439 6266 6135 340a 0201 a901 b360d9bfa54..... 00000040: 0000 8f77 2521 e56c 4a5b b97c 1b36 0d9b ...w%!.lJ[.|.6.. 00000050: fa54 0000 4002 4707 0000 c8eb 6665 cd4c .T..@.G.....fe.L 00000060: 6233 0400 0000 6d61 696e 0d00 0000 0a00 b3....main…… It turned out that the first 23 bytes of data is corrupted during HW crash. So we copied the first 23 bytes of this data from a running OSD with the following command: dd if=/dev/ceph-block-21/block-21 of=/root/header.21.dat bs=23 count=1 Then we copied the exact 23 bytes to the every failed OSD block device after backup and the problem is resolved. for i in {12..20} ; do dd if=/dev/ceph-block-$i/block-$i of=/root/backup.$i.1M bs=1M count=1 ; dd if=/root/header.21.dat of=/dev/ceph-block-$i/block-$i bs=23 count=1 ; done At the end of the day, it turned out that the lsiutil tool is not compatible with our kernel and caused the crash. The following link contains the detailed information. https://support.huawei.com/enterprise/en/knowledge/KB1000001578https://supp… I want to thank to Dan and Mykola from Clyso and appreciate their help. BR, Huseyin Cotuk hcotuk(a)gmail.com

2 1

No User + Dev Monthly Meetup this week - Happy Holidays!
by Laura Flores 19 Dec '23

19 Dec '23

Hi all, A quick reminder that the User + Dev Monthly Meetup that was scheduled for this week December 21 is cancelled due to the holidays. The User + Dev Monthly Meetup will resume in the new year on January 18. If you have a topic you'd like to present at an upcoming meetup, you're welcome to submit it here: https://docs.google.com/forms/d/e/1FAIpQLSdboBhxVoBZoaHm8xSmeBoemuXoV_rmh4v… Wishing everyone a happy holiday season! Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

Can not activate some OSDs after upgrade (bad crc on label)
by Huseyin Cotuk 19 Dec '23

19 Dec '23

Hello Cephers, I have two identical Ceph clusters with 32 OSDs each, running radosgw with EC. They were running Octopus on Ubuntu 20.04. On one of these clusters, I have upgraded OS to Ubuntu 22.04 and Ceph version is upgraded to Quincy 17.2.6. This cluster completed the process without any issue and it works as expected. On the second cluster, I followed the same procedure and upgraded the cluster. After upgrade 9 of 32 OSDs can not be activated. AFAIU, the label of these OSDs can not be read. ceph-volume lvm activate {osd.id} {osd_fsid} command fails as below: stderr: failed to read label for /dev/ceph-block-13/block-13: (5) Input/output error stderr: 2023-12-19T11:46:25.310+0300 7f088cd7ea80 -1 bluestore(/dev/ceph-block-13/block-13) _read_bdev_label bad crc on label, expected 2340927273 != actual 2067505886 All ceph-bluestore-tool and ceph-object-storetool commands fail with the same message, so I can not try repair, fsck or migrate. # ceph-bluestore-tool repair --deep yes --path /var/lib/ceph/osd/ceph-13/ failed to load os-type: (2) No such file or directory 2023-12-19T13:57:06.551+0300 7f39b1635a80 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label bad crc on label, expected 2340927273 != actual 2067505886 I also tried show label with bluestore-tool without success. # ceph-bluestore-tool show-label --dev /dev/ceph-block-13/block-13 unable to read label for /dev/ceph-block-13/block-13: (5) Input/output error 2023-12-19T14:01:19.668+0300 7fdcdd111a80 -1 bluestore(/dev/ceph-block-13/block-13) _read_bdev_label bad crc on label, expected 2340927273 != actual 2067505886 I can get the information including osd_fsif, block_uuid of all failed OSDs via ceph-volume lvm list like below. ====== osd.13 ====== [block] /dev/ceph-block-13/block-13 block device /dev/ceph-block-13/block-13 block uuid jFaTba-ln5r-muQd-7Ef9-3tWe-JwvO-qW9nqi cephx lockbox secret cluster fsid 4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f cluster name ceph crush device class None encrypted 0 osd fsid c9ee3ef6-73d7-4029-9cd6-086cc95d2f27 osd id 13 osdspec affinity type block vdo 0 devices /dev/mapper/mpathb All vgs and lvs look healthy. # lvdisplay ceph-block-13/block-13 --- Logical volume --- LV Path /dev/ceph-block-13/block-13 LV Name block-13 VG Name ceph-block-13 LV UUID jFaTba-ln5r-muQd-7Ef9-3tWe-JwvO-qW9nqi LV Write Access read/write LV Creation host, time ank-backup01, 2023-11-29 10:41:53 +0300 LV Status available # open 0 LV Size <7.28 TiB Current LE 1907721 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:33 This is a single node cluster running only radosgw. The environment is as follows: # ceph -v ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) # lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 22.04.3 LTS Release: 22.04 Codename: jammy # ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "osd_replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "choose_firstn", "num": 0, "type": "osd" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "default.rgw.buckets.data", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -1, "item_name": "default" }, { "op": "choose_indep", "num": 0, "type": "osd" }, { "op": "emit" } ] } ] ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 226.29962 root default -3 226.29962 host ank-backup01 0 hdd 7.29999 osd.0 up 1.00000 1.00000 1 hdd 7.29999 osd.1 up 1.00000 1.00000 2 hdd 7.29999 osd.2 up 1.00000 1.00000 3 hdd 7.29999 osd.3 up 1.00000 1.00000 4 hdd 7.29999 osd.4 up 1.00000 1.00000 5 hdd 7.29999 osd.5 up 1.00000 1.00000 6 hdd 7.29999 osd.6 up 1.00000 1.00000 7 hdd 7.29999 osd.7 up 1.00000 1.00000 8 hdd 7.29999 osd.8 up 1.00000 1.00000 9 hdd 7.29999 osd.9 up 1.00000 1.00000 10 hdd 7.29999 osd.10 up 1.00000 1.00000 11 hdd 7.29999 osd.11 up 1.00000 1.00000 12 hdd 7.29999 osd.12 down 0 1.00000 13 hdd 7.29999 osd.13 down 0 1.00000 14 hdd 7.29999 osd.14 down 0 1.00000 15 hdd 7.29999 osd.15 down 0 1.00000 16 hdd 7.29999 osd.16 down 0 1.00000 17 hdd 7.29999 osd.17 down 0 1.00000 18 hdd 7.29999 osd.18 down 0 1.00000 19 hdd 7.29999 osd.19 down 0 1.00000 20 hdd 7.29999 osd.20 down 0 1.00000 21 hdd 7.29999 osd.21 up 1.00000 1.00000 22 hdd 7.29999 osd.22 up 1.00000 1.00000 23 hdd 7.29999 osd.23 up 1.00000 1.00000 24 hdd 7.29999 osd.24 up 1.00000 1.00000 25 hdd 7.29999 osd.25 up 1.00000 1.00000 26 hdd 7.29999 osd.26 up 1.00000 1.00000 27 hdd 7.29999 osd.27 up 1.00000 1.00000 28 hdd 7.29999 osd.28 up 1.00000 1.00000 29 hdd 7.29999 osd.29 up 1.00000 1.00000 30 hdd 7.29999 osd.30 up 1.00000 1.00000 31 hdd 7.29999 osd.31 up 1.00000 1.00000 Does anybody have any idea why the labels of these OSDs can not be read? Any help would be appreciated. Best Regards, Huseyin Cotuk hcotuk(a)gmail.com

1 0

cephadm Adding OSD wal device on a new
by Adiga, Anantha 19 Dec '23

19 Dec '23

Hi, After adding a node to the cluster (3 nodes) with cephadm, how do I add OSDs with the same configuration on the other nodes ? The other nodes have 12 drives for data osd-block AND 2 drives for wal osd-wal. There are 6 LVs in each wal disk for the 12 data drives. I have added the ODS with ceph orch daemon add osd hostname:/dev/nvme0n1 How do I attach the wal devices to the OSDs? I have the WAL volumes created nvme3n1 259:5 0 349.3G 0 disk |-ceph--75d65cd1--91e4--4a8f--869b--e2a550f83104-osd--wal--dad0df4e--149a--4e80--b451--79f9b81838b8 253:12 0 58.2G 0 lvm |-ceph--75d65cd1--91e4--4a8f--869b--e2a550f83104-osd--wal--a5f2e93a--7bf0--4904--a233--3946b855c764 253:13 0 58.2G 0 lvm |-ceph--75d65cd1--91e4--4a8f--869b--e2a550f83104-osd--wal--cc949e1b--2560--4d38--bc27--558550881726 253:14 0 58.2G 0 lvm |-ceph--75d65cd1--91e4--4a8f--869b--e2a550f83104-osd--wal--8846f50e--7e92--4f66--a738--ce3a89650019 253:15 0 58.2G 0 lvm |-ceph--75d65cd1--91e4--4a8f--869b--e2a550f83104-osd--wal--6d646762--483a--40ca--8c51--ea54e0684a94 253:16 0 58.2G 0 lvm `-ceph--75d65cd1--91e4--4a8f--869b--e2a550f83104-osd--wal--74e58163--de1d--4062--a658--5b0356d43a87 253:17 0 58.2G 0 lvm How do I attach the wal volumes to the OSDs osd.36 nvme0n1 259:1 0 5.8T 0 disk `-ceph--3df7c5c3--c2c0--4498--9e17--2af79e448abc-osd--block--804b50ea--d44c--4cad--9177--8d722f737df9 253:0 0 5.8T 0 lvm Osd.37 nvme1n1 259:3 0 5.8T 0 disk `-ceph--30858acc--c48b--4a08--bb98--4c9b59112c59-osd--block--0a3a198b--66ec--4ed9--94da--fb171e190e38 253:1 0 5.8T 0 lvm nvme3n1 Thank you, Anantha

2 1

mgr finish mon failed to return metadata for mds
by Manolis Daramas 19 Dec '23

19 Dec '23

The current ceph version that we use is 17.2.7. We see in the Manager logs the below errors: 2 mgr.server handle_open ignoring open from mds.storage.node01.zjltbu v2:10.40.99.11:6800/1327026642; not ready for session (expect reconnect) 0 7faf43715700 1 mgr finish mon failed to return metadata for mds.storage.node01.zjltbu: (2) No such file or directory and in # ceph fs status Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1759, in _handle_command return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/status/module.py", line 109, in handle_fs_status assert metadata AssertionError Does anyone know what they are and what we can do to fix them ? Thanks, Manolis Under the General Data Protection Regulation (GDPR) (EU) 2016/679, Motivian as Data Controller has a legal duty to protect any information collected from you via email. Information contained in this email and any attachments may be privileged or confidential and intended for the exclusive use of the original recipient. If you have received this email by mistake, please advise the sender immediately and delete the email, including emptying your deleted email box. Information included in this email is reserved to named addressee's eyes only. You may not share this message or any of its attachments to anyone. Please note that as the recipient, it is your responsibility to check the email for malicious software. Motivian puts the security of the client at a high priority. Therefore, we have put efforts into ensuring that the message is error and virus-free. Unfortunately, full security of the email cannot be ensured as, despite our efforts, the data included in emails could be infected, intercepted, or corrupted. Therefore, the recipient should check the email for threats with proper software, as the sender does not accept liability for any damage inflicted by viewing the content of this email.

3 2

Terrible cephfs rmdir performance
by Paul Mezzanini 19 Dec '23

19 Dec '23

Long story short, we've got a lot of empty directories that I'm working on removing. While removing directories, using "perf top -g" we can watch the MDS daemon go to 100% cpu usage with "SnapRealm:: split_at" and "CInode::is_ancestor_of". It's this 2 year old bug that still is around. https://tracker.ceph.com/issues/53192 To help combat this, we've moved our snapshot schedule down the tree one level so the snaprealm is significantly smaller. Our luck with multiple active MDSs hasn't been great so we are still on a single MDS. To help split the load, I'm working on moving different workloads to different filesytems within ceph. A user can still fairly easily overwhelm the MDS's finisher thread and basically stop all cephfs io through that MDS. I'm hoping we can get some other people chiming in with "Me Too!" so there can be some traction behind fixing this. It's a longstanding bug so the version is less important, but we are on 17.2.7. Thoughts? -paul -- Paul Mezzanini Platform Engineer III Research Computing Rochester Institute of Technology “End users is a description, not a goal.”

2 1

MDS crashing repeatedly
by Thomas Widhalm 19 Dec '23

19 Dec '23

Hi, I have a 18.2.0 Ceph cluster and my MDS are now crashing repeatedly. After a few automatic restart, every MDS is removed and only one stays active. But it's flagged "laggy" and I can't even start a scrub on it. In the log I have this during crashes: Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 2023-12-13T14:54:02.721+0000 7f15ea108700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.0/rpm/el8/BUILD/ceph-18.2.0/src/mds/MDCache.cc: In function 'void MDCache::journal_cow_dentry(MutationImpl*, EMetaBlob*, CDentry*, snapid_t, CInode**, CDentry::linkage_t*)' thread 7f15ea108700 time 2023-12-13T14:54:02.720383+0000 Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.0/rpm/el8/BUILD/ceph-18.2.0/src/mds/MDCache.cc: 1638: FAILED ceph_assert(follows >= realm->get_newest_seq()) Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable) Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f15f5ef9dbb] Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2a8f81) [0x7f15f5ef9f81] Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 3: (MDCache::journal_cow_dentry(MutationImpl*, EMetaBlob*, CDentry*, snapid_t, CInode**, CDentry::linkage_t*)+0xae2) [0x55727bc0c672] Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 4: (MDCache::journal_dirty_inode(MutationImpl*, EMetaBlob*, CInode*, snapid_t)+0xc5) [0x55727bc0d0d5] Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 5: (Locker::scatter_writebehind(ScatterLock*)+0x5f6) [0x55727bce40f6] Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 6: (Locker::simple_sync(SimpleLock*, bool*)+0x388) [0x55727bceb908] Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 7: (Locker::scatter_nudge(ScatterLock*, MDSContext*, bool)+0x30d) [0x55727bcef25d] Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 8: (Locker::scatter_tick()+0x1e7) [0x55727bd0bc37] Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 9: (Locker::tick()+0xd) [0x55727bd0c0ed] Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 10: (MDSRankDispatcher::tick()+0x1ef) [0x55727bb08e9f] Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 11: (Context::complete(int)+0xd) [0x55727bade2cd] Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 12: (CommonSafeTimer<ceph::fair_mutex>::timer_thread()+0x16d) [0x7f15f5fea1cd] Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 13: (CommonSafeTimerThread<ceph::fair_mutex>::entry()+0x11) [0x7f15f5feb2a1] Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 14: /lib64/libpthread.so.0(+0x81ca) [0x7f15f4ca11ca] Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 15: clone() Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: *** Caught signal (Aborted) ** Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: in thread 7f15ea108700 thread_name:safe_timer Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable) Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f15f4cabcf0] Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 2: gsignal() Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 3: abort() Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f15f5ef9e15] Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a8f81) [0x7f15f5ef9f81] Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 6: (MDCache::journal_cow_dentry(MutationImpl*, EMetaBlob*, CDentry*, snapid_t, CInode**, CDentry::linkage_t*)+0xae2) [0x55727bc0c672] Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 7: (MDCache::journal_dirty_inode(MutationImpl*, EMetaBlob*, CInode*, snapid_t)+0xc5) [0x55727bc0d0d5] Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 8: (Locker::scatter_writebehind(ScatterLock*)+0x5f6) [0x55727bce40f6] Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 9: (Locker::simple_sync(SimpleLock*, bool*)+0x388) [0x55727bceb908] Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 10: (Locker::scatter_nudge(ScatterLock*, MDSContext*, bool)+0x30d) [0x55727bcef25d] Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 11: (Locker::scatter_tick()+0x1e7) [0x55727bd0bc37] Dec 13 15:54:02 ceph04 ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]: 12: (Locker::tick()+0xd) [0x55727bd0c0ed] I tried the following to get it back: ceph fs fail cephfs cephfs-data-scan cleanup --filesystem cephfs cephfs_data cephfs-journal-tool --rank cephfs:0 event recover_dentries list cephfs-table-tool cephfs:all reset session cephfs-journal-tool --rank cephfs:0 journal reset cephfs-data-scan scan_extents --worker_n 0 --worker_m 4 --filesystem cephfs cephfs_data cephfs-data-scan scan_inodes --worker_n 0 --worker_m 4 --filesystem cephfs cephfs_data cephfs-data-scan scan_links --filesystem cephfs ceph mds repaired 0 ceph fs set cephfs joinable true (with the data scan commands of course running on several systems simultanously) Unfortunately it didn't help at all. I'm quite sure that my undersized hardware is the root cause because problems with metadata already occured in the past (with 17.x) and it was always during times of higher load (e.g. taking Proxmox backups while deleting CephFS snapshots). I now have a strategy to lower the load and update the hardware. But still - I need my data back. Any ideas? Cheers, Thomas -- http://www.widhalm.or.at GnuPG : 6265BAE6 , A84CB603 Threema: H7AV7D33 Telegram, Signal: widhalmt(a)widhalm.or.at

2 1

cephadm file "/sbin/cephadm", line 10098 PK ^
by farhad kh 18 Dec '23

18 Dec '23

Hello, I downloaded cephadm from the link below. https://download.ceph.com/rpm-18.2.0/el8/noarch/ I change the address of the images to the address of my private registry, ``` DEFAULT_IMAGE = 'opkbhfpspsp0101.fns/ceph/ceph:v18' DEFAULT_IMAGE_IS_MAIN = False DEFAULT_IMAGE_RELEASE = 'reef' DEFAULT_PROMETHEUS_IMAGE = 'opkbhfpspsp0101.fns/ceph/prometheus:v2.43.0' DEFAULT_LOKI_IMAGE = 'opkbhfpspsp0101.fns/ceph/loki:2.4.0' DEFAULT_PROMTAIL_IMAGE = 'opkbhfpspsp0101.fns/ceph/promtail:2.4.0' DEFAULT_NODE_EXPORTER_IMAGE = 'opkbhfpspsp0101.fns/ceph/node-exporter:v1.5.0' DEFAULT_ALERT_MANAGER_IMAGE = 'opkbhfpspsp0101.fns/ceph/alertmanager:v0.25.0' DEFAULT_GRAFANA_IMAGE = 'opkbhfpspsp0101.fns/ceph/ceph-grafana:9.4.7' DEFAULT_HAPROXY_IMAGE = 'opkbhfpspsp0101.fns/ceph/haproxy:2.3' DEFAULT_KEEPALIVED_IMAGE = 'opkbhfpspsp0101.fns/ceph/keepalived:2.2.4' DEFAULT_SNMP_GATEWAY_IMAGE = 'opkbhfpspsp0101.fns/ceph/snmp-notifier:v1.2.1' DEFAULT_ELASTICSEARCH_IMAGE = 'opkbhfpspsp0101.fns/ceph/elasticsearch:6.8.23' DEFAULT_JAEGER_COLLECTOR_IMAGE = 'opkbhfpspsp0101.fns/ceph/jaeger-collector:1.29' DEFAULT_JAEGER_AGENT_IMAGE = 'opkbhfpspsp0101.fns/ceph/jaeger-agent:1.29' DEFAULT_JAEGER_QUERY_IMAGE = 'opkbhfpspsp0101.fns/ceph/jaeger-query:1.29' DEFAULT_REGISTRY = 'opkbhfpspsp0101.fns' # normalize unqualified digests to this ``` but I encounter this error ` File "/sbin/cephadm", line 10098 PK ^ `. Also, there is a wrong line at the beginning of cephadm file PK^C^D^T^@^@^@^@^@¥<9a>^CW<8e>^[º^×Ü^E^@×Ü^E^@^K^@^@^@__main__.py#!/usr/bin/python3

2 1

Ceph Cluster Deployment - Recommendation
by Amardeep Singh 18 Dec '23

18 Dec '23

Hi Everyone, We are in the process of planning a ceph cluster deployment for our data infrastructure. To provide you with a bit of context, we have deployed hardware across two data halls in our data center, and they are connected via a 10Gb interconnect. The hardware configuration for 4 x Ceph Cluster (2 x servers in both data halls) * 2 x AMD EPYC 7513 - 32 Cores / 64 Threads * 512GB RAM * 2 x 960 GB (OS DISKS) * 8x Micron 7450 PRO 7680GB NVMe - PCIe Gen4 * Intel X550-T2 - 10GbE Dual-Port RJ45 Server Adaptor Our primary usage is Object Gateway and will be running 4 x RGW Service. We are aiming to deploy using cephadm and utilize all nodes for MON/MGR/RGW and OSD's Given our limited experience with Ceph, we are reaching out to the knowledgeable members of this community for recommendations and best practices. We would greatly appreciate any insights or advice you can share regarding the following aspects: Cluster Topology: Considering our hardware setup with two data halls connected via a 10Gb interconnect, what would be the recommended cluster topology for optimal performance and fault tolerance? Best Practices for Deployment: Are there any recommended best practices for deploying Ceph in a similar environment? Any challenges we should be aware of? Thank you in advance for your time and assistance. Regards, Amar DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the sender. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation.

3 3

OSD has Rocksdb corruption that crashes ceph-bluestore-tool repair
by Malcolm Haak 18 Dec '23

18 Dec '23

Hello all, I had an OSD go offline due to UWE. When restarting the OSD service, to try and at least get it to drain cleanly of that data that wasn't damaged, the ceph-osd process would crash. I then attempted to repair it using ceph-bluestore-tool. I can run fsck and it will complete without issue, however when attempting to run repair it crashes in the exact same way that ceph-osd crashes. I'll attach the tail end of the output here: 2023-12-17T20:24:53.320+1000 7fdb7bf17740 -1 rocksdb: submit_common error: Corruption: block checksum mismatch: stored = 1106056583, computed = 657190205, type = 1 in db/020524.sst offset 21626321 size 4014 code = Rocksdb transaction: PutCF( prefix = S key = 'per_pool_omap' value size = 1) -442> 2023-12-17T20:24:53.386+1000 7fdb7bf17740 -1 /usr/src/debug/ceph/ceph-18.2.0/src/os/bluestore/BlueStore.cc: In function 'unsigned int BlueStoreRepairer::apply(KeyValueDB*)' thread 7fdb7bf17740 time 2023-12-17T20:24:53.341999+1000 /usr/src/debug/ceph/ceph-18.2.0/src/os/bluestore/BlueStore.cc: 17982: FAILED ceph_assert(ok) ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x136) [0x7fdb7b6502c9] 2: /usr/lib/ceph/libceph-common.so.2(+0x2504a4) [0x7fdb7b6504a4] 3: (BlueStoreRepairer::apply(KeyValueDB*)+0x5af) [0x559afb98cc7f] 4: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x45fc) [0x559afba2436c] 5: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x204) [0x559afba31014] 6: main() 7: /usr/lib/libc.so.6(+0x27cd0) [0x7fdb7ae45cd0] 8: __libc_start_main() 9: _start() -441> 2023-12-17T20:24:53.390+1000 7fdb7bf17740 -1 *** Caught signal (Aborted) ** in thread 7fdb7bf17740 thread_name:ceph-bluestore- ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable) 1: /usr/lib/libc.so.6(+0x3e710) [0x7fdb7ae5c710] 2: /usr/lib/libc.so.6(+0x8e83c) [0x7fdb7aeac83c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x191) [0x7fdb7b650324] 6: /usr/lib/ceph/libceph-common.so.2(+0x2504a4) [0x7fdb7b6504a4] 7: (BlueStoreRepairer::apply(KeyValueDB*)+0x5af) [0x559afb98cc7f] 8: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x45fc) [0x559afba2436c] 9: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x204) [0x559afba31014] 10: main() 11: /usr/lib/libc.so.6(+0x27cd0) [0x7fdb7ae45cd0] 12: __libc_start_main() 13: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. The reason I need to get this OSD functioning is I had two other OSD's fail causing a single PG to be in down state. The weird thing is, I got one of those back up without issue (ceph-osd crashed due to root filling and alert not sending) but the PG is still down. So I need to get this other one back up (or the data extracted) to get that PG back from down. Thanks in advance

2 1

Cephfs MDS tunning for deep-learning data-flow
by mhnx 16 Dec '23

16 Dec '23

Hello everyone! How are you doing? I wasn't around for two years but I'm back and working on a new development. I deployed 2x ceph cluster: 1- user_data:5x node [8x4TB Sata SSD, 2x 25Gbit network], 2- data-gen: 3x node [8x4TB Sata SSD, 2x 25Gbit network], note: hardware is not my choice and I know I have TRIM issue and also I couldn't use any PCI-E nvme for wal+db because 1u servers and no empty slots --------------------- At test phase everything was good, I reached 1GB/s for 18 clients at the same time. But when I migrate to production (60 GPU server client + 40 CPU server client) the speed issue begin because of the default parameters as usual and now I'm working on adaptation by debugging current data work flow I have and I'm researching how can I improve my environment. So far, I couldn't find useful guide or informations in one place and I just wanted to share my findings, benchmarks and ideas with the community and if I'm lucky enough, maybe I will get awesome recommendations from some old friends and enjoy get in touch after a while. :) Starting from here, I will only share technical information about my environment: 1- Cluster user_data: 5x node [8x4TB Sata SSD, 2x 25Gbit network] = Replication 2 - A: I only have 1 pool in this cluster and information is below: - ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 146 TiB 106 TiB 40 TiB 40 TiB 27.50 TOTAL 146 TiB 106 TiB 40 TiB 40 TiB 27.50 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 286 MiB 73 859 MiB 0 32 TiB cephfs.ud-data.meta 9 512 65 GiB 2.87M 131 GiB 0.13 48 TiB cephfs.ud-data.data 10 2048 23 TiB 95.34M 40 TiB 29.39 48 TiB - B: In this cluster, every user(50) has a subvolume and the quota is 1TB/for each users - C: In each subvolume, users has "home and data" directory. - D: home directory size 5-10GB and client uses it for docker home directory at each login - E: I'm also storing users personal or development data around 2TB/each user - F: I only have 1x active MDS server and 4x standby as below. - ceph fs status > ud-data - 84 clients > ======= > RANK STATE MDS ACTIVITY DNS INOS DIRS > CAPS > 0 active ud-data.ud-04.seggyv Reqs: 372 /s 4343k 4326k 69.7k > 2055k > POOL TYPE USED AVAIL > cephfs.ud-data.meta metadata 130G 47.5T > cephfs.ud-data.data data 39.5T 47.5T > STANDBY MDS > ud-data.ud-01.uatjle > ud-data.ud-02.xcoojt > ud-data.ud-05.rnhcfe > ud-data.ud-03.lhwkml > MDS version: ceph version 17.2.6 > (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) - What is my issue? 2023-12-15T21:07:47.175542+0000 mon.ud-01 [WRN] Health check failed: 1 clients failing to respond to cache pressure (MDS_CLIENT_RECALL) 2023-12-15T21:09:35.002112+0000 mon.ud-01 [INF] MDS health message cleared (mds.?): Client gpu-server-11 failing to respond to cache pressure 2023-12-15T21:09:35.391235+0000 mon.ud-01 [INF] Health check cleared: MDS_CLIENT_RECALL (was: 1 clients failing to respond to cache pressure) 2023-12-15T21:09:35.391304+0000 mon.ud-01 [INF] Cluster is now healthy 2023-12-15T21:10:00.000169+0000 mon.ud-01 [INF] overall HEALTH_OK For every read and write in client's trying to reach ceph MDS server and requests some data: issue 1: home data is around 5-10GB and users need all the time. I need to store it one time and prevent new requests. issue 2: users process generates new data by only reading some data one time and they write generated data one time. No need to cache this data at all. What I want to do ??? 1- I want to deploy 2x active MDS server for only "home" directory in each subvolume: - These 2x home MDS servers must send the data to client and cache in the client to reduce new requests even for simple "ls" command 2- I want to deploy 2x active MDS server for only "data" directory in each subvolume: - These 2x MDS servers must be configured to not hold any CACHE if it is not required constantly. The cache life time must be short and must be independent. - Constantly requested data from one client must be cached locally in that client to reduce requests and load on the MDS server. ------------------------------------------------------------ I believe you understand my data-flow and my needs. Let's talk what we can do about it. Note: I'm still researching and these are my finding and my plan so far. it is not completed, and this is the main reason why I'm writing this mail. ceph fs set $MYFS max_mds 4 mds_cache_memory_limit | default 4GiB --> 16GiB mds_cache_reservation | default 0.05 --> ?? mds_health_cache_threshold | default 1.5 --> ?? mds_cache_trim_threshold | default 256KiB --> ?? mds_cache_trim_decay_rate | default 1.0 --> ?? mds_cache_mid mds_decay_halflife mds_client_prealloc_inos mds_dirstat_min_interval mds_session_cache_liveness_magnitude mds_session_cache_liveness_decay_rate mds_max_caps_per_client mds_recall_max_caps mds_recall_max_decay_threshold mds_recall_max_decay_rate mds_recall_global_max_decay_threshold mds_session_cap_acquisition_throttle mds_session_cap_acquisition_decay_rate mds_session_max_caps_throttle_ratio mds_cap_acquisition_throttle_retry_request_timeout -Manually pinning directory trees to a particular rank As you can see, I'm at the beginning of this journey and I will be grateful if you can help me, share your knowledge, even I'm ready to help developers to use my system as a test bench to improve ceph as always! Best regards folks! - Özkan

1 1

rbd trash: snapshot id is protected from removal
by Eugen Block 15 Dec '23

15 Dec '23

Hi, I've been searching and trying things but to no avail yet. This is uncritical because it's a test cluster only, but I'd still like to have a solution in case this somehow will make it into our production clusters. It's an Openstack Victoria Cloud with Ceph backend. If one tries to remove a glance image (openstack image delete {UUID}' which usually has a protected snapshot it will fail to do so, but apparently the snapshot is actually moved to the trash namespace. And since it is protected, I can't remove it: storage01:~ # rbd -p images snap ls 278ffe2b-67a7-40d0-87b7-903f2fc9c3b4 --all SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 159 1a97db13-307e-4820-8dc2-8549e9ba1ad7 39 MiB Thu Dec 14 08:29:56 2023 trash (snap) storage01:~ # rbd snap rm --snap-id 159 images/278ffe2b-67a7-40d0-87b7-903f2fc9c3b4 rbd: snapshot id 159 is protected from removal. storage01:~ # rbd snap ls images/278ffe2b-67a7-40d0-87b7-903f2fc9c3b4 storage01:~ # This is a small image and only a test environment, but these orphans could potentially fill up lots of space. In a newer openstack version (I tried with Antelope) this doesn't seem to work like that anymore, so that's good. But how would I get rid of that trash snapshot in this cluster? Thanks! Eugen

2 2

Corrupted and inconsistent reads from CephFS on EC pool
by aschmitz 15 Dec '23

15 Dec '23

Hi everyone, I'm seeing different results from reading files, depending on which OSDs are running, including some incorrect reads with all OSDs running, in CephFS from a pool with erasure coding. I'm running Ceph 17.2.6. # More detail In particular, I have a relatively large backup of some files, combined with SHA-256 hashes of the files (which were verified when the backup was created, approximately 7 months ago). Verifying these hashes currently gives several errors, both in large and small files, but somewhat tilted towards larger files. Investigating which PGs stored the relevant files (using cephfs-data-scan pg_files) didn't show the problem to be isolated to one PG, but did show several PGs that contained OSD 15 as an active member. Taking OSD 15 offline leads to *better* reads (more files with correct SHA-256 hashes), but not completely correct reads. Further investigation implicated OSD 34 as another potential issue, but taking it offline also results in more correct files but not completely. Bringing the stopped OSDs (15 and 34) back online results in the earlier (incorrect) hashes when reading files, as might be expected, but this seems to demonstrate that the correct information (or at least more correct information) is still on the drives. The hashes I receive for a given corrupted file are consistent from read to read (including on different hosts, to avoid caching as an issue), but obviously sometimes change if I take an affected OSD offline. # Recent history I have Ceph configured with a deep scrub interval of approximately 30 days, and they have completed regularly with no issues identified. However, within the past two weeks I added two additional drives to the cluster, and rebalancing took about two weeks to complete: the placement groups I took notice of having issues were not deep scrubbed since the replacement completed, so it is possible something got corrupted during the rebalance. Neither OSD 15 nor 34 is a new drive, and as far as I have experienced (and Ceph's health indications have shown), all of the existing OSDs have behaved correctly up to this point. # Configuration I created an erasure coding profile for the pool in question using the following command: ceph osd erasure-code-profile set erasure_k4_m2 \ plugin=jerasure \ k=4 m=2 \ technique=blaum_roth \ crush-device-class=hdd And the following CRUSH rule is used for the pool: rule erasure_k4_m2_hdd_rule { id 3 type erasure min_size 4 max_size 6 step take default class hdd step choose indep 3 type host step chooseleaf indep 2 type osd step emit } # Questions 1. Does this behavior ring a bell to anyone? Is there something obvious I'm missing or should do? 2. Is deep scrubbing likely to help the situation? Hurt it? (Hopefully not hurt: I've prioritized deep scrubbing of the PGs on OSD 15 and 34, and will likely follow up with the rest of the pool.) 3. Is there a way to force "full reads" or otherwise to use all of the EC chunks (potentially in tandem with on-disk checksums) to identify the correct data, rather than a combination of the data from the primary OSDs? Thanks for any insights you might have, aschmitz

1 0

Etag change of a parent object
by Rok Jaklič 15 Dec '23

15 Dec '23

Hi, shouldn't etag of a "parent" object change when "child" objects are added on s3? Example: 1. I add an object to test bucket: "example/" - size 0 "example/" has an etag XYZ1 2. I add an object to test bucket: "example/test1.txt" - size 12 "example/test1.txt" has an etag XYZ2 "example/" has an etag XYZ1 ... should this change? I understand that object storage is not hierarchical by design and objects are "not connected" by some other means than the bucket name. Kind regards, Rok

4 3

Howto: 'one line patch' in deployed cluster?
by Harry G Coin 14 Dec '23

14 Dec '23

Is there a 'Howto' or 'workflow' to implement a one-line patch in a running cluster? With full understanding it will be gone on the next upgrade? Hopefully without having to set up an entire packaging/development environment? Thanks! To implement: * /Subject/: Re: Permanent KeyError: 'TYPE' ->17.2.7: return self.blkid_api['TYPE'] == 'part' * /From/: Sascha Lucas <ceph-users@xxxxxxxxx> Problem found: in my case this is caused by DRBD secondary block devices, which can not be read until promoted to primary. ceph_volume/util/disk.py runs in blkid(): $ blkid -c /dev/null -p /dev/drbd4 blkid: error: /dev/drbd4: Wrong medium type but does not care about its return code. A quick fix is to use the get() method to automatically fall back to None for non existing keys: --- a/ceph_volume/util/device.py 2023-11-10 07:00:01.552497107 +0000 +++ b/ceph_volume/util/device.py 2023-11-10 08:54:40.320718690 +0000 @@ -476,13 +476,13 @@ @property def is_partition(self): self.load_blkid_api() if self.disk_api: return self.disk_api['TYPE'] == 'part' elif self.blkid_api: - return self.blkid_api['TYPE'] == 'part' + return self.blkid_api.get('TYPE') == 'part' return False Don't know why this is triggered in 17.2.7.

1 0

Ceph orch made block_db too small, not accounting for multiple nvmes, how-to fix it?
by Mikael Öhman 14 Dec '23

14 Dec '23

Hi Our hosts have 3 NVMEs and 48 spinning drives each. We found that ceph orch made the default lvm size for the block_db 1/3 the total size of the NVMEs. I suspect that ceph only considered one of the NVMEs when determining the size, based the closely related issue; https://tracker.ceph.com/issues/54541 We started having some bluefs spillover events now, so I'm looking for a way to fix this. Best idea I have so far is to manually specify the "block_db_size" in the osd_spec, then just recreating the entire block_db. Though I'm not sure if that means we'll hit the same issue https://tracker.ceph.com/issues/54541 instead. There would also be a lot of data to move in order to do this to a total of 588 OSD's. Maybe there is a way to just maybe remove and re-add (bigger) block_db? I would appreciate any suggestions or tips. Best regards, Mikael

1 1

Pool Migration / Import/Export
by duluxoz 14 Dec '23

14 Dec '23

Hi All, I find myself in the position of having to change the k/m values on an ec-pool. I've discovered that I simply can't change the ec-profile, but have to create a "new ec-profile" and a "new ec-pool" using the new values, then migrate the "old ec-pool" to the new (see: https://ceph.io/en/news/blog/2015/ceph-pool-migration/ => Using Rados Export/Import). (Yes, a PITA, but it has to be done, and better doing it now when the data-size isn't that big - yet!) My only concern is that the "old ec-pool" is a `--data-pool` and part of an rbd image (ie the image was created with `rbd create --size 2T ec_rbd_pool/disk01 --data-pool ec_pool --image-feature journaling`). So my Q is: What are the "gotchas" (if any) with this, or is it simpler to backup the data (already done), destroy and recreate the rbd image from scratch, and restore the data to the re-created pool(s)/image? FTR: space isn't an issue (I've got plenty to play with) but I'm looking for the quickest way to do this as its a live (but little used) system, so I can take it down for a few hours (or more), but don't particular *want* to have it off-line for longer than absolutely necessary. Thanks in advance for any advice/help/warnings/etc :-) Cheers Dulux-Oz

2 1

Re: increasing number of (deep) scrubs
by Frank Schilder 13 Dec '23

13 Dec '23

Hi Dan, thanks for your answer. I don't have a problem with increasing osd_max_scrubs (=1 at the moment) as such. I would simply prefer a somewhat finer grained way of controlling scrubbing than just doubling or tripling it right away. Some more info. These 2 pools are data pools for a large FS. Unfortunately, we have a large percentage of small files, which is a pain for recovery and seemingly also for deep scrubbing. Our OSDs are about 25% used and I had to increase the warning interval already to 2 weeks. With all the warning grace parameters this means that we manage to deep scrub everything about every month. I need to plan for 75% utilisation and a 3 months period is a bit far on the risky side. Our data is to a large percentage cold data. Client reads will not do the check for us, we need to combat bit-rot pro-actively. The reasons I'm interested in parameters initiating more scrubs while also converting more scrubs into deep scrubs are, that 1) scrubs seem to complete very fast. I almost never catch a PG in state "scrubbing", I usually only see "deep scrubbing". 2) I suspect the low deep-scrub count is due to a low number of deep-scrubs scheduled and not due to conflicting per-OSD deep scrub reservations. With the OSD count we have and the distribution over 12 servers I would expect at least a peak of 50% OSDs being active in scrubbing instead of the 25% peak I'm seeing now. It ought to be possible to schedule more PGs for deep scrub than actually are. 3) Every OSD having only 1 deep scrub active seems to have no measurable impact on user IO. If I could just get more PGs scheduled with 1 deep scrub per OSD it would already help a lot. Once this is working, I can eventually increase osd_max_scrubs when the OSDs fill up. For now I would just like that (deep) scrub scheduling looks a bit harder and schedules more eligible PGs per time unit. If we can get deep scrubbing up to an average of 42PGs completing per hour with keeping osd_max_scrubs=1 to maintain current IO impact, we should be able to complete a deep scrub with 75% full OSDs in about 30 days. This is the current tail-time with 25% utilisation. I believe currently a deep scrub of a PG in these pools takes 2-3 hours. Its just a gut feeling from some repair and deep-scrub commands, I would need to check logs for more precise info. Increasing osd_max_scrubs would then be a further and not the only option to push for more deep scrubbing. My expectation would be that values of 2-3 are fine due to the increasingly higher percentage of cold data for which no interference with client IO will happen. Hope that makes sense and there is a way beyond bumping osd_max_scrubs to increase the number of scheduled and executed deep scrubs. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dvanders(a)gmail.com> Sent: 05 January 2023 15:36 To: Frank Schilder Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] increasing number of (deep) scrubs Hi Frank, What is your current osd_max_scrubs, and why don't you want to increase it? With 8+2, 8+3 pools each scrub is occupying the scrub slot on 10 or 11 OSDs, so at a minimum it could take 3-4x the amount of time to scrub the data than if those were replicated pools. If you want the scrub to complete in time, you need to increase the amount of scrub slots accordingly. On the other hand, IMHO the 1-week deadline for deep scrubs is often much too ambitious for large clusters -- increasing the scrub intervals is one solution, or I find it simpler to increase mon_warn_pg_not_scrubbed_ratio and mon_warn_pg_not_deep_scrubbed_ratio until you find a ratio that works for your cluster. Of course, all of this can impact detection of bit-rot, which anyway can be covered by client reads if most data is accessed periodically. But if the cluster is mostly idle or objects are generally not read, then it would be preferable to increase slots osd_max_scrubs. Cheers, Dan On Tue, Jan 3, 2023 at 2:30 AM Frank Schilder <frans(a)dtu.dk> wrote: > > Hi all, > > we are using 16T and 18T spinning drives as OSDs and I'm observing that they are not scrubbed as often as I would like. It looks like too few scrubs are scheduled for these large OSDs. My estimate is as follows: we have 852 spinning OSDs backing a 8+2 pool with 2024 and an 8+3 pool with 8192 PGs. On average I see something like 10PGs of pool 1 and 12 PGs of pool 2 (deep) scrubbing. This amounts to only 232 out of 852 OSDs scrubbing and seems to be due to a conservative rate of (deep) scrubs being scheduled. The PGs (dep) scrub fairly quickly. > > I would like to increase gently the number of scrubs scheduled for these drives and *not* the number of scrubs per OSD. I'm looking at parameters like: > > osd_scrub_backoff_ratio > osd_deep_scrub_randomize_ratio > > I'm wondering if lowering osd_scrub_backoff_ratio to 0.5 and, maybe, increasing osd_deep_scrub_randomize_ratio to 0.2 would have the desired effect? Are there other parameters to look at that allow gradual changes in the number of scrubs going on? > > Thanks a lot for your help! > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 3

cephfs read hang after cluster stuck, but need attach the process to continue
by zxcs 13 Dec '23

13 Dec '23

Hi, experts, we are using cephfs with 16.2.* with multi active mds, and recently we see a strange thing, we have some c++ code about read file from cephfs. the client code just call very base read(), and when the cluster hit mds has slow request, and later the cluster back to normal. the read hang. then we need to gdb attach -p `pid of c++ process`, just attach, do nothing, the code then continue running. Our question is why this happens? and config could tune or we need change out c++ read code? Thanks xz

1 0

Cephfs too many repaired copies on osds
by zxcs 13 Dec '23

13 Dec '23

Hi, Experts, we are using cephfs with 16.2.* with multi active mds, and recently we see an osd report “full object read crc *** != expected oxffffffff on ****:head” “missing primary copy of ***: will try to read copies on **” from `ceph -s`, could see OSD_TOO_MANY_REPAIRS: Too many repaired reads on ** OSDs. we don’t know how to fix this , could you are please help shed some light here? Thanks a ton! Thanks xz

2 2

mds.0.journaler.pq(ro) _finish_read got error -2
by Eugen Block 12 Dec '23

12 Dec '23

Hi, I'm trying to help someone with a broken CephFS. We managed to recover basic ceph functionality but the CephFS is still inaccessible (currently read-only). We went through the disaster recovery steps but to no avail. Here's a snippet from the startup logs: ---snip--- mds.0.41 Booting: 2: waiting for purge queue recovered mds.0.journaler.pq(ro) _finish_probe_end write_pos = 14797504512 (header had 14789452521). recovered. mds.0.purge_queue operator(): open complete mds.0.purge_queue operator(): recovering write_pos monclient: get_auth_request con 0x55c280bc5c00 auth_method 0 monclient: get_auth_request con 0x55c280ee0c00 auth_method 0 mds.0.journaler.pq(ro) _finish_read got error -2 mds.0.purge_queue _recover: Error -2 recovering write_pos mds.0.purge_queue _go_readonly: going readonly because internal IO failed: No such file or directory mds.0.journaler.pq(ro) set_readonly mds.0.41 unhandled write error (2) No such file or directory, force readonly... mds.0.cache force file system read-only force file system read-only ---snip--- I've added the dev mailing list, maybe someone can give some advice how to continue from here (we could try to recover with an empty metadata pool). Or is this FS lost? Thanks! Eugen

3 4

Announcing go-ceph v0.25.0
by Anoop C S 12 Dec '23

12 Dec '23

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.25.0 More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Anoop C S

1 0

MDS recovery with existing pools
by Eugen Block 12 Dec '23

12 Dec '23

Hi, following up on the previous thread (After hardware failure tried to recover ceph and followed instructions for recovery using OSDS), we were able to get ceph back into a healthy state (including the unfound object). Now the CephFS needs to be recovered and I'm having trouble to fully understand the docs [1] which the next steps would be. We ran the following which according to [1] sets the state to existing but failed: ceph fs new <fs_name> <metadata_pool> <data_pool> --force --recover But how to continue from here? Should we expect an active MDS at this point or not? Because the "ceph fs status" output still shows rank 0 as failed. We then tried: ceph fs set <fs_name> joinable true But apparently it was already joinable, nothing changed. Before doing anything (destructive) from the advanced options [2] I wanted to ask the community, how to get on from here. I pasted the mds logs at the bottom, I'm not really sure if the current state is expected or not. Apparently, the journal recovers but the purge_queue does not: mds.0.41 Booting: 2: waiting for purge queue recovered mds.0.journaler.pq(ro) _finish_probe_end write_pos = 14797504512 (header had 14789452521). recovered. mds.0.purge_queue operator(): open complete mds.0.purge_queue operator(): recovering write_pos monclient: get_auth_request con 0x55c280bc5c00 auth_method 0 monclient: get_auth_request con 0x55c280ee0c00 auth_method 0 mds.0.journaler.pq(ro) _finish_read got error -2 mds.0.purge_queue _recover: Error -2 recovering write_pos mds.0.purge_queue _go_readonly: going readonly because internal IO failed: No such file or directory mds.0.journaler.pq(ro) set_readonly mds.0.41 unhandled write error (2) No such file or directory, force readonly... mds.0.cache force file system read-only force file system read-only Is this expected because the "--recover" flag prevents an active MDS or not? Before running "ceph mds rmfailed ..." and/or "ceph fs reset <file system name>" with the --yes-i-really-mean-it flag I'd like to ask for your input. In which case should we run those commands? The docs are not really clear to me. Any input is highly appreciated! Thanks! Eugen [1] https://docs.ceph.com/en/latest/cephfs/recover-fs-after-mon-store-loss/ [2] https://docs.ceph.com/en/latest/cephfs/administration/#advanced-cephfs-admi… ---snip--- Dec 07 15:35:48 node02 bash[692598]: debug -90> 2023-12-07T13:35:47.730+0000 7f4cd855f700 1 mds.storage.node02.hemalk Updating MDS map to version 41 from mon.0 Dec 07 15:35:48 node02 bash[692598]: debug -89> 2023-12-07T13:35:47.730+0000 7f4cd855f700 4 mds.0.purge_queue operator(): data pool 3 not found in OSDMap Dec 07 15:35:48 node02 bash[692598]: debug -88> 2023-12-07T13:35:47.730+0000 7f4cd855f700 5 asok(0x55c27fe86000) register_command objecter_requests hook 0x55c27fe16310 Dec 07 15:35:48 node02 bash[692598]: debug -87> 2023-12-07T13:35:47.730+0000 7f4cd855f700 10 monclient: _renew_subs Dec 07 15:35:48 node02 bash[692598]: debug -86> 2023-12-07T13:35:47.730+0000 7f4cd855f700 10 monclient: _send_mon_message to mon.node02 at v2:10.40.99.12:3300/0 Dec 07 15:35:48 node02 bash[692598]: debug -85> 2023-12-07T13:35:47.730+0000 7f4cd855f700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) Dec 07 15:35:48 node02 bash[692598]: debug -84> 2023-12-07T13:35:47.730+0000 7f4cd855f700 4 mds.0.purge_queue operator(): data pool 3 not found in OSDMap Dec 07 15:35:48 node02 bash[692598]: debug -83> 2023-12-07T13:35:47.730+0000 7f4cd855f700 4 mds.0.0 apply_blocklist: killed 0, blocklisted sessions (0 blocklist entries, 0) Dec 07 15:35:48 node02 bash[692598]: debug -82> 2023-12-07T13:35:47.730+0000 7f4cd855f700 1 mds.0.41 handle_mds_map i am now mds.0.41 Dec 07 15:35:48 node02 bash[692598]: debug -81> 2023-12-07T13:35:47.734+0000 7f4cd855f700 1 mds.0.41 handle_mds_map state change up:standby --> up:replay Dec 07 15:35:48 node02 bash[692598]: debug -80> 2023-12-07T13:35:47.734+0000 7f4cd855f700 5 mds.beacon.storage.node02.hemalk set_want_state: up:standby -> up:replay Dec 07 15:35:48 node02 bash[692598]: debug -79> 2023-12-07T13:35:47.734+0000 7f4cd855f700 1 mds.0.41 replay_start Dec 07 15:35:48 node02 bash[692598]: debug -78> 2023-12-07T13:35:47.734+0000 7f4cd855f700 2 mds.0.41 Booting: 0: opening inotable Dec 07 15:35:48 node02 bash[692598]: debug -77> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: _send_mon_message to mon.node02 at v2:10.40.99.12:3300/0 Dec 07 15:35:48 node02 bash[692598]: debug -76> 2023-12-07T13:35:47.734+0000 7f4cd855f700 2 mds.0.41 Booting: 0: opening sessionmap Dec 07 15:35:48 node02 bash[692598]: debug -75> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: _send_mon_message to mon.node02 at v2:10.40.99.12:3300/0 Dec 07 15:35:48 node02 bash[692598]: debug -74> 2023-12-07T13:35:47.734+0000 7f4cd855f700 2 mds.0.41 Booting: 0: opening mds log Dec 07 15:35:48 node02 bash[692598]: debug -73> 2023-12-07T13:35:47.734+0000 7f4cd855f700 5 mds.0.log open discovering log bounds Dec 07 15:35:48 node02 bash[692598]: debug -72> 2023-12-07T13:35:47.734+0000 7f4cd855f700 2 mds.0.41 Booting: 0: opening purge queue (async) Dec 07 15:35:48 node02 bash[692598]: debug -71> 2023-12-07T13:35:47.734+0000 7f4cd855f700 4 mds.0.purge_queue open: opening Dec 07 15:35:48 node02 bash[692598]: debug -70> 2023-12-07T13:35:47.734+0000 7f4cd855f700 1 mds.0.journaler.pq(ro) recover start Dec 07 15:35:48 node02 bash[692598]: debug -69> 2023-12-07T13:35:47.734+0000 7f4cd855f700 1 mds.0.journaler.pq(ro) read_head Dec 07 15:35:48 node02 bash[692598]: debug -68> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: _send_mon_message to mon.node02 at v2:10.40.99.12:3300/0 Dec 07 15:35:48 node02 bash[692598]: debug -67> 2023-12-07T13:35:47.734+0000 7f4cd855f700 2 mds.0.41 Booting: 0: loading open file table (async) Dec 07 15:35:48 node02 bash[692598]: debug -66> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: _send_mon_message to mon.node02 at v2:10.40.99.12:3300/0 Dec 07 15:35:48 node02 bash[692598]: debug -65> 2023-12-07T13:35:47.734+0000 7f4cd855f700 2 mds.0.41 Booting: 0: opening snap table Dec 07 15:35:48 node02 bash[692598]: debug -64> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: _send_mon_message to mon.node02 at v2:10.40.99.12:3300/0 Dec 07 15:35:48 node02 bash[692598]: debug -63> 2023-12-07T13:35:47.734+0000 7f4cd1d52700 4 mds.0.journalpointer Reading journal pointer '400.00000000' Dec 07 15:35:48 node02 bash[692598]: debug -62> 2023-12-07T13:35:47.734+0000 7f4cd1d52700 10 monclient: _send_mon_message to mon.node02 at v2:10.40.99.12:3300/0 Dec 07 15:35:48 node02 bash[692598]: debug -61> 2023-12-07T13:35:47.734+0000 7f4cd4557700 2 mds.0.cache Memory usage: total 316452, rss 43088, heap 198940, baseline 198940, 0 / 0 inodes have caps, 0 caps, 0 caps per inode Dec 07 15:35:48 node02 bash[692598]: debug -60> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: _renew_subs Dec 07 15:35:48 node02 bash[692598]: debug -59> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: _send_mon_message to mon.node02 at v2:10.40.99.12:3300/0 Dec 07 15:35:48 node02 bash[692598]: debug -58> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: handle_get_version_reply finishing 1 version 10835 Dec 07 15:35:48 node02 bash[692598]: debug -57> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: handle_get_version_reply finishing 2 version 10835 Dec 07 15:35:48 node02 bash[692598]: debug -56> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: handle_get_version_reply finishing 3 version 10835 Dec 07 15:35:48 node02 bash[692598]: debug -55> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: handle_get_version_reply finishing 4 version 10835 Dec 07 15:35:48 node02 bash[692598]: debug -54> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: handle_get_version_reply finishing 5 version 10835 Dec 07 15:35:48 node02 bash[692598]: debug -53> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: handle_get_version_reply finishing 6 version 10835 Dec 07 15:35:48 node02 bash[692598]: debug -52> 2023-12-07T13:35:47.734+0000 7f4cdb565700 10 monclient: get_auth_request con 0x55c280bc5800 auth_method 0 Dec 07 15:35:48 node02 bash[692598]: debug -51> 2023-12-07T13:35:47.734+0000 7f4cdbd66700 10 monclient: get_auth_request con 0x55c280dc6800 auth_method 0 Dec 07 15:35:48 node02 bash[692598]: debug -50> 2023-12-07T13:35:47.734+0000 7f4cdad64700 10 monclient: get_auth_request con 0x55c280dc7800 auth_method 0 Dec 07 15:35:48 node02 bash[692598]: debug -49> 2023-12-07T13:35:47.734+0000 7f4cd3555700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 14789115904, expire 14789452521, write 14789452521, stream_format 1). probing for end of log (from 14789452521)... Dec 07 15:35:48 node02 bash[692598]: debug -48> 2023-12-07T13:35:47.734+0000 7f4cd3555700 1 mds.0.journaler.pq(ro) probing for end of the log Dec 07 15:35:48 node02 bash[692598]: debug -47> 2023-12-07T13:35:47.738+0000 7f4cd1d52700 1 mds.0.journaler.mdlog(ro) recover start Dec 07 15:35:48 node02 bash[692598]: debug -46> 2023-12-07T13:35:47.738+0000 7f4cd1d52700 1 mds.0.journaler.mdlog(ro) read_head Dec 07 15:35:48 node02 bash[692598]: debug -45> 2023-12-07T13:35:47.738+0000 7f4cd1d52700 4 mds.0.log Waiting for journal 0x200 to recover... Dec 07 15:35:48 node02 bash[692598]: debug -44> 2023-12-07T13:35:47.738+0000 7f4cdbd66700 10 monclient: get_auth_request con 0x55c280dc7c00 auth_method 0 Dec 07 15:35:48 node02 bash[692598]: debug -43> 2023-12-07T13:35:47.738+0000 7f4cd2553700 1 mds.0.journaler.mdlog(ro) _finish_read_head loghead(trim 1416940748800, expire 1416947000701, write 1417125359769, stream_format 1). probing for end of log (from 1417125359769)... Dec 07 15:35:48 node02 bash[692598]: debug -42> 2023-12-07T13:35:47.738+0000 7f4cd2553700 1 mds.0.journaler.mdlog(ro) probing for end of the log Dec 07 15:35:48 node02 bash[692598]: debug -41> 2023-12-07T13:35:47.738+0000 7f4cdb565700 10 monclient: get_auth_request con 0x55c280e2fc00 auth_method 0 Dec 07 15:35:48 node02 bash[692598]: debug -40> 2023-12-07T13:35:47.738+0000 7f4cdad64700 10 monclient: get_auth_request con 0x55c280ee0400 auth_method 0 Dec 07 15:35:48 node02 bash[692598]: debug -39> 2023-12-07T13:35:47.738+0000 7f4cd2553700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 1417129492480 (header had 1417125359769). recovered. Dec 07 15:35:48 node02 bash[692598]: debug -38> 2023-12-07T13:35:47.738+0000 7f4cd1d52700 4 mds.0.log Journal 0x200 recovered. Dec 07 15:35:48 node02 bash[692598]: debug -37> 2023-12-07T13:35:47.738+0000 7f4cd1d52700 4 mds.0.log Recovered journal 0x200 in format 1 Dec 07 15:35:48 node02 bash[692598]: debug -36> 2023-12-07T13:35:47.738+0000 7f4cd1d52700 2 mds.0.41 Booting: 1: loading/discovering base inodes Dec 07 15:35:48 node02 bash[692598]: debug -35> 2023-12-07T13:35:47.738+0000 7f4cd1d52700 0 mds.0.cache creating system inode with ino:0x100 Dec 07 15:35:48 node02 bash[692598]: debug -34> 2023-12-07T13:35:47.738+0000 7f4cd1d52700 0 mds.0.cache creating system inode with ino:0x1 Dec 07 15:35:48 node02 bash[692598]: debug -33> 2023-12-07T13:35:47.742+0000 7f4cdbd66700 10 monclient: get_auth_request con 0x55c280dc7400 auth_method 0 Dec 07 15:35:48 node02 bash[692598]: debug -32> 2023-12-07T13:35:47.742+0000 7f4cd2553700 2 mds.0.41 Booting: 2: replaying mds log Dec 07 15:35:48 node02 bash[692598]: debug -31> 2023-12-07T13:35:47.742+0000 7f4cd2553700 2 mds.0.41 Booting: 2: waiting for purge queue recovered Dec 07 15:35:48 node02 bash[692598]: debug -30> 2023-12-07T13:35:47.742+0000 7f4cd3555700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 14797504512 (header had 14789452521). recovered. Dec 07 15:35:48 node02 bash[692598]: debug -29> 2023-12-07T13:35:47.742+0000 7f4cd3555700 4 mds.0.purge_queue operator(): open complete Dec 07 15:35:48 node02 bash[692598]: debug -28> 2023-12-07T13:35:47.742+0000 7f4cd3555700 4 mds.0.purge_queue operator(): recovering write_pos Dec 07 15:35:48 node02 bash[692598]: debug -27> 2023-12-07T13:35:47.742+0000 7f4cdb565700 10 monclient: get_auth_request con 0x55c280bc5c00 auth_method 0 Dec 07 15:35:48 node02 bash[692598]: debug -26> 2023-12-07T13:35:47.742+0000 7f4cdad64700 10 monclient: get_auth_request con 0x55c280ee0c00 auth_method 0 Dec 07 15:35:48 node02 bash[692598]: debug -25> 2023-12-07T13:35:47.746+0000 7f4cd3555700 0 mds.0.journaler.pq(ro) _finish_read got error -2 Dec 07 15:35:48 node02 bash[692598]: debug -24> 2023-12-07T13:35:47.746+0000 7f4cd3555700 -1 mds.0.purge_queue _recover: Error -2 recovering write_pos Dec 07 15:35:48 node02 bash[692598]: debug -23> 2023-12-07T13:35:47.746+0000 7f4cd3555700 1 mds.0.purge_queue _go_readonly: going readonly because internal IO failed: No such file or directory Dec 07 15:35:48 node02 bash[692598]: debug -22> 2023-12-07T13:35:47.746+0000 7f4cd3555700 1 mds.0.journaler.pq(ro) set_readonly Dec 07 15:35:48 node02 bash[692598]: debug -21> 2023-12-07T13:35:47.746+0000 7f4cd3555700 -1 mds.0.41 unhandled write error (2) No such file or directory, force readonly... Dec 07 15:35:48 node02 bash[692598]: debug -20> 2023-12-07T13:35:47.746+0000 7f4cd3555700 1 mds.0.cache force file system read-only Dec 07 15:35:48 node02 bash[692598]: debug -19> 2023-12-07T13:35:47.746+0000 7f4cd3555700 0 log_channel(cluster) log [WRN] : force file system read-only ---snip---

2 5

Deleting files from lost+found in 18.2.0
by Thomas Widhalm 11 Dec '23

11 Dec '23

Hi, I saw in the changelogs that it should now finally be possible to delete files from lost+found with 18.x . I upgraded recently but still I can't delete or move files from there. I tried to change permissions but still every time I try it, I get "read only filesystem" but only when acting on files in lost+found. I searched the documentation but I can't find anything related. Is there a special trick or a flag I have to set? Cheers, Thomas -- http://www.widhalm.or.at GnuPG : 6265BAE6 , A84CB603 Threema: H7AV7D33 Telegram, Signal: widhalmt(a)widhalm.or.at

1 0

Ceph 17.2.7 to 18.2.0 issues
by pclark6063＠outlook.com 11 Dec '23

11 Dec '23

Hi All, Recently I upgraded my cluster from Quincy to Reef. Everything appeared to go smoothly and without any issues arising. I was forced to poweroff the cluster, performing the ususal procedures beforehand and everything appears to have come back fine. Every service reports green across the board except.... If i try to copy any files from a cephfs mountpoint whether kernel or fuse the actual copy will hang. ls/stat etc all work which indicates metadata appears fine but copying always hangs. I can copy objects direct using the rados toolset which indicates the underlying data exists. The system itself reports no errors and thinks its healthy. The entire cluster and cephfs clients are all Rocky9. Any advice would be much appreciatd. I'd find this easier to deal with if the cluster actually gave me an error....

3 4

reef 18.2.1 QE Validation status
by Yuri Weinstein 11 Dec '23

11 Dec '23

Details of this release are summarized here: https://tracker.ceph.com/issues/63443#note-1 Seeking approvals/reviews for: smoke - Laura, Radek, Prashant, Venky (POOL_APP_NOT_ENABLE failures) rados - Neha, Radek, Travis, Ernesto, Adam King rgw - Casey fs - Venky orch - Adam King rbd - Ilya krbd - Ilya upgrade/quincy-x (reef) - Laura PTL powercycle - Brad perf-basic - Laura, Prashant (POOL_APP_NOT_ENABLE failures) Please reply to this email with approval and/or trackers of known issues/PRs to address them. TIA YuriW

12 30

Osd full
by Mohamed LAMDAOUAR 11 Dec '23

11 Dec '23

Hello the team, We initially had a cluster of 3 machines with 4 osd on each machine, we added 4 machines in the cluster (each machine with 4 osd) We launched the balancing but it never finished, still in progress. But the big issue: we have an osd full and all the pools on this osd are read only. *ceph osd df *: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 20 hdd 9.09569 1.00000 9.1 TiB 580 GiB 576 GiB 1.2 GiB 3.1 GiB 8.5 TiB 6.23 0.32 169 up 21 hdd 9.09569 1.00000 9.1 TiB 1.5 TiB 1.5 TiB 252 MiB 7.7 GiB 7.6 TiB 16.08 0.82 247 up 22 hdd 9.09569 1.00000 9.1 TiB 671 GiB 667 GiB 204 MiB 4.1 GiB 8.4 TiB 7.21 0.37 136 up 23 hdd 9.09569 1.00000 9.1 TiB 665 GiB 660 GiB 270 MiB 4.5 GiB 8.4 TiB 7.14 0.37 124 up 0 hdd 9.09569 1.00000 9.1 TiB 1.2 TiB 1.2 TiB 87 MiB 6.0 GiB 7.9 TiB 13.30 0.68 230 up 1 hdd 9.09569 1.00000 9.1 TiB 1.3 TiB 1.3 TiB 347 MiB 6.6 GiB 7.8 TiB 14.01 0.72 153 up 2 hdd 9.09569 0.65009 9.1 TiB 1.8 TiB 1.8 TiB 443 MiB 7.3 GiB 7.3 TiB 20.00 1.03 147 up 3 hdd 9.09569 1.00000 9.1 TiB 617 GiB 611 GiB 220 MiB 5.8 GiB 8.5 TiB 6.62 0.34 101 up 4 hdd 9.09569 0.80005 9.1 TiB 2.0 TiB 2.0 TiB 293 MiB 8.2 GiB 7.1 TiB 22.12 1.13 137 up 5 hdd 9.09569 1.00000 9.1 TiB 857 GiB 852 GiB 157 MiB 4.9 GiB 8.3 TiB 9.20 0.47 155 up 6 hdd 9.09569 1.00000 9.1 TiB 580 GiB 575 GiB 678 MiB 4.4 GiB 8.5 TiB 6.23 0.32 114 up 7 hdd 9.09569 0.50000 9.1 TiB 7.7 TiB 7.7 TiB 103 MiB 16 GiB 1.4 TiB 85.03 4.36 201 up 24 hdd 9.09569 1.00000 9.1 TiB 1.2 TiB 1.2 TiB 133 MiB 6.2 GiB 7.9 TiB 13.11 0.67 225 up 25 hdd 9.09569 0.34999 9.1 TiB 8.3 TiB 8.2 TiB 101 MiB 17 GiB 860 GiB 90.77 4.66 159 up 26 hdd 9.09569 1.00000 9.1 TiB 665 GiB 661 GiB 292 MiB 3.8 GiB 8.4 TiB 7.14 0.37 107 up 27 hdd 9.09569 1.00000 9.1 TiB 427 GiB 423 GiB 241 MiB 3.4 GiB 8.7 TiB 4.58 0.24 103 up 8 hdd 9.09569 1.00000 9.1 TiB 845 GiB 839 GiB 831 MiB 5.9 GiB 8.3 TiB 9.07 0.47 163 up 9 hdd 9.09569 1.00000 9.1 TiB 727 GiB 722 GiB 162 MiB 4.8 GiB 8.4 TiB 7.80 0.40 169 up 10 hdd 9.09569 0.80005 9.1 TiB 1.9 TiB 1.9 TiB 742 MiB 7.5 GiB 7.2 TiB 21.01 1.08 136 up 11 hdd 9.09569 1.00000 9.1 TiB 733 GiB 727 GiB 498 MiB 5.2 GiB 8.4 TiB 7.87 0.40 163 up 12 hdd 9.09569 1.00000 9.1 TiB 892 GiB 886 GiB 318 MiB 5.6 GiB 8.2 TiB 9.58 0.49 254 up 13 hdd 9.09569 1.00000 9.1 TiB 759 GiB 755 GiB 37 MiB 4.0 GiB 8.4 TiB 8.15 0.42 134 up 14 hdd 9.09569 0.85004 9.1 TiB 2.3 TiB 2.3 TiB 245 MiB 7.7 GiB 6.8 TiB 24.96 1.28 142 up 15 hdd 9.09569 1.00000 9.1 TiB 7.3 TiB 7.3 TiB 435 MiB 16 GiB 1.8 TiB 80.17 4.11 213 up 16 hdd 9.09569 1.00000 9.1 TiB 784 GiB 781 GiB 104 MiB 3.6 GiB 8.3 TiB 8.42 0.43 247 up 17 hdd 9.09569 1.00000 9.1 TiB 861 GiB 856 GiB 269 MiB 5.1 GiB 8.3 TiB 9.25 0.47 102 up 18 hdd 9.09569 1.00000 9.1 TiB 1.9 TiB 1.9 TiB 962 MiB 8.2 GiB 7.2 TiB 21.15 1.09 283 up 19 hdd 9.09569 1.00000 9.1 TiB 893 GiB 888 GiB 291 MiB 4.6 GiB 8.2 TiB 9.59 0.49 148 up TOTAL 255 TiB 50 TiB 49 TiB 9.7 GiB 187 GiB 205 TiB 19.49 MIN/MAX VAR: 0.24/4.66 STDDEV: 19.63 *ceph health detail |grep -i wrn* [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set [WRN] OSD_NEARFULL: 2 nearfull osd(s) [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 16 pgs backfill_toofull [WRN] PG_NOT_DEEP_SCRUBBED: 1360 pgs not deep-scrubbed in time [WRN] PG_NOT_SCRUBBED: 53 pgs not scrubbed in time [WRN] POOL_NEARFULL: 36 pool(s) nearfull Thanks the team ;)

2 1

OSD CPU and write latency increase after upgrade from 15.2.16 to 17.2.6
by Tony Yao 11 Dec '23

11 Dec '23

Hi， Recently, I upgraded Ceph from 15.2.16 to 17.2.6, but I found that OSD CPU usage increased from 30% to 90% or more, and OSD subop_w_latency increased from 600us to 5ms. This is incredible. My hardware environment: 12 nodes x 12 NVMe (Intel P4510 4T) I tried to set the OSD configuration to the default value and saw no improvement either. Have you ever encountered this problem？ What might have gone wrong? Best regards, Tony Yao

1 1

ceph fs (meta) data inconsistent
by Frank Schilder 11 Dec '23

11 Dec '23

Dear fellow cephers, today we observed a somewhat worrisome inconsistency on our ceph fs. A file created on one host showed up as 0 length on all other hosts: [user1@host1 h2lib]$ ls -lh total 37M -rw-rw---- 1 user1 user1 12K Nov 1 11:59 dll_wrapper.py [user2@host2 h2lib]# ls -l total 34 -rw-rw----. 1 user1 user1 0 Nov 1 11:59 dll_wrapper.py [user1@host1 h2lib]$ cp dll_wrapper.py dll_wrapper.py.test [user1@host1 h2lib]$ ls -l total 37199 -rw-rw---- 1 user1 user1 11641 Nov 1 11:59 dll_wrapper.py -rw-rw---- 1 user1 user1 11641 Nov 1 13:10 dll_wrapper.py.test [user2@host2 h2lib]# ls -l total 45 -rw-rw----. 1 user1 user1 0 Nov 1 11:59 dll_wrapper.py -rw-rw----. 1 user1 user1 11641 Nov 1 13:10 dll_wrapper.py.test Executing a sync on all these hosts did not help. However, deleting the problematic file and replacing it with a copy seemed to work around the issue. We saw this with ceph kclients of different versions, it seems to be on the MDS side. How can this happen and how dangerous is it? ceph fs status (showing ceph version): # ceph fs status con-fs2 - 1662 clients ======= RANK STATE MDS ACTIVITY DNS INOS 0 active ceph-15 Reqs: 14 /s 2307k 2278k 1 active ceph-11 Reqs: 159 /s 4208k 4203k 2 active ceph-17 Reqs: 3 /s 4533k 4501k 3 active ceph-24 Reqs: 3 /s 4593k 4300k 4 active ceph-14 Reqs: 1 /s 4228k 4226k 5 active ceph-13 Reqs: 5 /s 1994k 1782k 6 active ceph-16 Reqs: 8 /s 5022k 4841k 7 active ceph-23 Reqs: 9 /s 4140k 4116k POOL TYPE USED AVAIL con-fs2-meta1 metadata 2177G 7085G con-fs2-meta2 data 0 7085G con-fs2-data data 1242T 4233T con-fs2-data-ec-ssd data 706G 22.1T con-fs2-data2 data 3409T 3848T STANDBY MDS ceph-10 ceph-08 ceph-09 ceph-12 MDS version: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) There is no health issue: # ceph status cluster: id: abc health: HEALTH_WARN 3 pgs not deep-scrubbed in time services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 9w) mgr: ceph-25(active, since 7w), standbys: ceph-26, ceph-01, ceph-03, ceph-02 mds: con-fs2:8 4 up:standby 8 up:active osd: 1284 osds: 1279 up (since 2d), 1279 in (since 5d) task status: data: pools: 14 pools, 25065 pgs objects: 2.20G objects, 3.9 PiB usage: 4.9 PiB used, 8.2 PiB / 13 PiB avail pgs: 25039 active+clean 26 active+clean+scrubbing+deep io: client: 799 MiB/s rd, 55 MiB/s wr, 3.12k op/s rd, 1.82k op/s wr The inconsistency seems undiagnosed, I couldn't find anything interesting in the cluster log. What should I look for and where? I moved the folder to another location for diagnosis. Unfortunately, I don't have 2 clients any more showing different numbers, I see a 0 length now everywhere for the moved folder. I'm pretty sure though that the file still is non-zero length. Thanks for any pointers. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 18

How to replace a disk with minimal impact on performance
by Michal Strnad 08 Dec '23

08 Dec '23

Hi guys! Based on our observation of the impact of the balancer on the performance of the entire cluster, we have drawn conclusions that we would like to discuss with you. - A newly created pool should be balanced before being handed over to the user. This, I believe, is quite evident. - When replacing a disk, it is advisable to exchange it directly for a new one. As soon as the OSD replacement occurs, the balancer should be invoked to realign any improperly placed PGs during the disk outage and disk recovery. Perhaps an even better method is to pause recovery and backfilling before removing the disk, remove the disk itself, promptly add a new one, and then resume recovery and backfilling. It's essential to per form all of this as quickly as possible (using a script). Ad. We are using a community balancer developed by Jonas Jelton because the built-in one does not meet our requirements. What are your thoughts on this? Michal

2 1

nfs export over RGW issue in Pacific
by Adiga, Anantha 07 Dec '23

07 Dec '23

Hi, oot@a001s016:~# cephadm version Using recent ceph image ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) root@a001s016:~# root@a001s016:~# cephadm shell Inferring fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 Inferring config /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/mon.a001s016/config Using recent ceph image ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 root@a001s016:~# ceph version ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) --------------------- But, Cephadm does not show "nfs export create rgw...." nfs export create cephfs <fsname> <clusterid> <binding> [--readonly] [<path>] nfs export rm <clusterid> <binding> nfs export delete <clusterid> <binding> nfs export ls <clusterid> [--detailed] nfs export get <clusterid> <binding> nfs export update --------------------- However, Ceph Dashboard allows to create the export see below: Access Type RW Cluster nfs-1 Daemons nfs-1.0.0.zp3110b001a0101.uckows, nfs-1.1.0.zp3110b001a0102.hhpebb, nfs-1.2.0.zp3110b001a0103.bbkpcb, nfs-1.3.0.zp3110b001a0104.zujkso NFS Protocol NFSv4 Object Gateway User admin Path buc-cluster-inventory Pseudo /rgwnfs_cluster_inventory Squash no_root_squash Storage Backend Object Gateway Transport TCP --------------------- While nfs export is created, pseudo "/rgwnfs_cluster_inventory", cephadm is not listing it # ceph nfs export ls nfs-1 [ "/cluster_inventory", "/oob_crashdump" ] # Anantha

2 2

Difficulty adding / using a non-default RGW placement target & storage class
by Anthony D'Atri 07 Dec '23

07 Dec '23

I'm having difficulty adding and using a non-default placement target & storage class and would appreciate insights. Am I going about this incorrectly? Rook does not yet have the ability to do this, so I'm adding it by hand. Following instructions on the net I added a second bucket pool, placement target, and storageclass, and created a user defaulting to the new pg/sc, but I get an error when trying to create a bucket: [rook@rook-ceph-tools-5ff8d58445-gkl5w .aws]$ s5cmd --endpoint-url http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc mb s3://foofoobars ERROR "mb s3://foofoobars": InvalidLocationConstraint: status code: 400, request id: tx0000057b71002881d48ca-0065495d54-1abe555-ceph-objectstore, host id: I found an article suggesting that the placement target and/or storage class should have the api_name prepended, so I tried setting either or both to "ceph-objectstore:HDD-EC" / "ceph-objectstore:GLACIER" with no success. I suspect that I'm missing something subtle -- or that Rook has provisioned these bits in an atypical fashion. Log entry: /var/log/ceph/ceph-client.rgw.ceph.objectstore.a.log-2023-11-06T21:40:36.543+0000 7f6573a9f700 1 ====== starting new request req=0x7f64818ba730 ===== /var/log/ceph/ceph-client.rgw.ceph.objectstore.a.log-2023-11-06T21:40:36.546+0000 7f6570a99700 0 req 6320538205097380042 0.003000009s s3:create_bucket could not find user default placement id HDD-EC/GLACIER within zonegroup /var/log/ceph/ceph-client.rgw.ceph.objectstore.a.log-2023-11-06T21:40:36.546+0000 7f6570a99700 1 ====== req done req=0x7f64818ba730 op status=-2208 http_status=400 latency=0.003000009s ====== /var/log/ceph/ceph-client.rgw.ceph.objectstore.a.log:2023-11-06T21:40:36.546+0000 7f6570a99700 1 beast: 0x7f64818ba730: 10.233.90.156 - aad [06/Nov/2023:21:40:36.543 +0000] "PUT /foofoobars HTTP/1.1" 400 266 - "aws-sdk-go/1.40.25 (go1.18.3; linux; amd64)" - latency=0.003000009s [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ ceph -v ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) Here's the second buckets pool, constrained to HDDs. AFAICT it can share the index and data_extra_pool created for the default / STANDARD pt/sc by Rook. I initially omitted ec_overwrites but enabled it after creation. pool 19 'ceph-objectstore.rgw.buckets.data' erasure profile ceph-objectstore.rgw.buckets.data_ecprofile size 6 min_size 5 crush_rule 10 object_hash rjenkins pg_num 8192 pgp_num 8192 autoscale_mode off last_change 165350 lfor 0/156300/165341 flags hashpspool,ec_overwrites stripe_width 16384 application rook-ceph-rgw pool 21 'ceph-objectstore.rgw.buckets.data.hdd' erasure profile ceph-objectstore.rgw.buckets.data_ecprofile_hdd size 6 min_size 5 crush_rule 11 object_hash rjenkins pg_num 8192 pgp_num 8192 autoscale_mode off last_change 167193 lfor 0/0/164453 flags hashpspool,ec_overwrites stripe_width 16384 application rook-ceph-rgw [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ radosgw-admin zonegroup get { "id": "d994155c-2a9c-4e37-ae30-64fd2934ff99", "name": "ceph-objectstore", "api_name": "ceph-objectstore", "is_master": "true", "endpoints": [ "http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc:80" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "72035401-a6d9-426b-8c89-9a17e268825f", "zones": [ { "id": "72035401-a6d9-426b-8c89-9a17e268825f", "name": "ceph-objectstore", "endpoints": [ "http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc:80" ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "HDD-EC", "tags": [], "storage_classes": [ "GLACIER" ] }, { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "51fb8875-31ac-40ef-ab21-0ffd4e229f15", "sync_policy": { "groups": [] } } [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ radosgw-admin zone get { "id": "72035401-a6d9-426b-8c89-9a17e268825f", "name": "ceph-objectstore", "domain_root": "ceph-objectstore.rgw.meta:root", "control_pool": "ceph-objectstore.rgw.control", "gc_pool": "ceph-objectstore.rgw.log:gc", "lc_pool": "ceph-objectstore.rgw.log:lc", "log_pool": "ceph-objectstore.rgw.log", "intent_log_pool": "ceph-objectstore.rgw.log:intent", "usage_log_pool": "ceph-objectstore.rgw.log:usage", "roles_pool": "ceph-objectstore.rgw.meta:roles", "reshard_pool": "ceph-objectstore.rgw.log:reshard", "user_keys_pool": "ceph-objectstore.rgw.meta:users.keys", "user_email_pool": "ceph-objectstore.rgw.meta:users.email", "user_swift_pool": "ceph-objectstore.rgw.meta:users.swift", "user_uid_pool": "ceph-objectstore.rgw.meta:users.uid", "otp_pool": "ceph-objectstore.rgw.otp", "system_key": { "access_key": "", "secret_key": "" }, "placement_pools": [ { "key": "HDD-EC", "val": { "index_pool": "ceph-objectstore.rgw.buckets.index", "storage_classes": { "GLACIER": { "data_pool": "ceph-objectstore.rgw.buckets.data.hdd" }, "STANDARD": {} # <------------- seems like this shouldn't be here? }, "data_extra_pool": "ceph-objectstore.rgw.buckets.non-ec", "index_type": 0 } }, { "key": "default-placement", "val": { "index_pool": "ceph-objectstore.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "ceph-objectstore.rgw.buckets.data" } }, "data_extra_pool": "ceph-objectstore.rgw.buckets.non-ec", "index_type": 0 } } ], "realm_id": "", "notif_pool": "ceph-objectstore.rgw.log:notif" } [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ radosgw-admin user info --uid=aad { "user_id": "aad", "display_name": "Anthony", "email": "", "suspended": 0, "max_buckets": 1000, "subusers": [], "keys": [ { "user": "aad", "access_key": "xxxx", "secret_key": "yyyyyy" } ], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "HDD-EC", "default_storage_class": "GLACIER", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw", "mfa_ids": [] }

2 1

Moving from ceph-ansible to cephadm and upgrading from pacific to octopus
by wodel youchi 07 Dec '23

07 Dec '23

Hi, I have an Openstack platform deployed with Yoga and ceph-ansible pacific on Rocky 8. Now I need to do an upgrade to Openstack zed with octopus on Rocky 9. This is the path of the upgrade I have traced - upgrade my nodes to Rocky 9 keeping Openstack yoga with ceph-ansible pacific. - convert ceph pacific from ceph-ansible to cephadm. - stop Openstack platform yoga - upgrade ceph pacific to octopus - upgrade Openstack yoga to zed. Any thoughts or guide lines to keep in mind and follow regarding ceph convertion and upgrade. Ps : on my ceph I have rbd, rgw and cephfs pools. Regards.

1 0

About delete old bucket lifecycle
by VÔ VI 07 Dec '23

07 Dec '23

Hi community, I am have multiple bucket was delete but lifecycle of bucket still exist, how i can delete it with radosgw-admin, because user can't access to bucket for delete lifecycle. User for this bucket does not exist. root@ceph:~# radosgw-admin lc list [ { "bucket": ":r30203:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.8", "started": "Wed, 06 Dec 2023 10:43:55 GMT", "status": "COMPLETE" }, { "bucket": ":r30304:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.13", "started": "Wed, 06 Dec 2023 10:43:54 GMT", "status": "COMPLETE" }, { "bucket": ":ec3204cam04:f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.1", "started": "Wed, 06 Dec 2023 10:44:30 GMT", "status": "COMPLETE" }, { "bucket": ":r30105:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.5", "started": "Wed, 06 Dec 2023 10:44:40 GMT", "status": "COMPLETE" }, { "bucket": ":r30303:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.14", "started": "Wed, 06 Dec 2023 10:44:40 GMT", "status": "COMPLETE" }, { "bucket": ":ec3201cam02:f3fec4b6-a248-4f3f-be75-b8055e61233a.56439.2", "started": "Wed, 06 Dec 2023 10:43:56 GMT", "status": "COMPLETE" }, Thanks to the community.

2 2

Assistance Needed with Ceph Cluster Slow Ops Issue
by Peter 06 Dec '23

06 Dec '23

Dear all, I am reaching out regarding an issue with our Ceph cluster that has been recurring every six hours. Upon investigating the problem using the "ceph daemon dump_historic_slow_ops" command, I observed that the issue appears to be related to slow operations, specifically getting stuck at "Waiting for RW Locks." The wait times often range from one to two seconds. Our cluster use SAS SSD disks from Samsung for the storage pool in question. While these disks are of high quality and should provide sufficient speed, the problem persists. The slow ops occurrence is consistent every six hours. I would greatly appreciate any insights or suggestions you may have to address and resolve this issue. If there are specific optimizations or configurations that could improve the situation, please advise. below are some output: root@lasas003:~# ceph -v ceph version 15.2.17 (542df8d06ef24dbddcf4994db16bcc4c89c9ec2d) octopus (stable) "events": [ { "event": "initiated", "time": "2023-12-06T08:34:18.501644-0800", "duration": 0 }, { "event": "throttled", "time": "2023-12-06T08:34:18.501644-0800", "duration": 3.067e-06 }, { "event": "header_read", "time": "2023-12-06T08:34:18.501647-0800", "duration": 3.5429999999999998e-06 }, { "event": "all_read", "time": "2023-12-06T08:34:18.501650-0800", "duration": 9.3399999999999997e-07 }, { "event": "dispatched", "time": "2023-12-06T08:34:18.501651-0800", "duration": 3.2830000000000002e-06 }, { "event": "queued_for_pg", "time": "2023-12-06T08:34:18.501654-0800", "duration": 1.3819939990000001 }, { "event": "reached_pg", "time": "2023-12-06T08:34:19.883648-0800", "duration": 5.7980000000000002e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:19.883654-0800", "duration": 4.2484711649999998 }, { "event": "reached_pg", "time": "2023-12-06T08:34:24.132125-0800", "duration": 1.0667e-05 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:24.132136-0800", "duration": 2.1593527840000002 }, { "event": "reached_pg", "time": "2023-12-06T08:34:26.291489-0800", "duration": 3.292e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:26.291492-0800", "duration": 0.43918164700000001 }, { "event": "reached_pg", "time": "2023-12-06T08:34:26.730674-0800", "duration": 5.1529999999999996e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:26.730679-0800", "duration": 1.0531516869999999 }, { "event": "reached_pg", "time": "2023-12-06T08:34:27.783831-0800", "duration": 5.1329999999999998e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:27.783836-0800", "duration": 1.232525088 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.016361-0800", "duration": 3.844e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.016365-0800", "duration": 0.0051385700000000003 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.021503-0800", "duration": 4.7600000000000002e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.021508-0800", "duration": 0.0092808779999999994 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.030789-0800", "duration": 4.0690000000000003e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.030793-0800", "duration": 0.55757725499999999 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.588370-0800", "duration": 5.5060000000000003e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.588376-0800", "duration": 0.0064168929999999999 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.594793-0800", "duration": 7.0690000000000004e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.594800-0800", "duration": 0.0026404089999999998 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.597440-0800", "duration": 3.3440000000000001e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.597444-0800", "duration": 0.0051126670000000004 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.602556-0800", "duration": 5.0200000000000002e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.602561-0800", "duration": 0.0040569960000000002 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.606618-0800", "duration": 5.0989999999999998e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.606623-0800", "duration": 0.0068874100000000001 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.613511-0800", "duration": 1.4636e-05 }, { "event": "started", "time": "2023-12-06T08:34:29.613525-0800", "duration": 0.00028943699999999997 }, { "event": "done", "time": "2023-12-06T08:34:29.613815-0800", "duration": 11.112171102 } Thank you in advance for your assistance.

2 3

EC Profiles & DR
by duluxoz 06 Dec '23

06 Dec '23

Hi All, Looking for some help/explanation around erasure code pools, etc. I set up a 3-node Ceph (Quincy) cluster with each box holding 7 OSDs (HDDs) and each box running Monitor, Manager, and iSCSI Gateway. For the record the cluster runs beautifully, without resource issues, etc. I created an Erasure Code Profile, etc: ~~~ ceph osd erasure-code-profile set my_ec_profile plugin=jerasure k=4 m=2 crush-failure-domain=osd ceph osd crush rule create-erasure my_ec_rule my_ec_profile ceph osd crush rule create-replicated my_replicated_rule default host ~~~ My Crush Map is: ~~~ # begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable chooseleaf_stable 1 tunable straw_calc_version 1 tunable allowed_bucket_algs 54 # devices device 0 osd.0 class hdd device 1 osd.1 class hdd device 2 osd.2 class hdd device 3 osd.3 class hdd device 4 osd.4 class hdd device 5 osd.5 class hdd device 6 osd.6 class hdd device 7 osd.7 class hdd device 8 osd.8 class hdd device 9 osd.9 class hdd device 10 osd.10 class hdd device 11 osd.11 class hdd device 12 osd.12 class hdd device 13 osd.13 class hdd device 14 osd.14 class hdd device 15 osd.15 class hdd device 16 osd.16 class hdd device 17 osd.17 class hdd device 18 osd.18 class hdd device 19 osd.19 class hdd device 20 osd.20 class hdd # types type 0 osd type 1 host type 2 chassis type 3 rack type 4 row type 5 pdu type 6 pod type 7 room type 8 datacenter type 9 zone type 10 region type 11 root # buckets host ceph_1 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.0 weight 5.34769 item osd.1 weight 5.45799 item osd.2 weight 5.45799 item osd.3 weight 5.45799 item osd.4 weight 5.45799 item osd.5 weight 5.45799 item osd.6 weight 5.45799 } host ceph_2 { id -5 # do not change unnecessarily id -6 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.7 weight 5.34769 item osd.8 weight 5.45799 item osd.9 weight 5.45799 item osd.10 weight 5.45799 item osd.11 weight 5.45799 item osd.12 weight 5.45799 item osd.13 weight 5.45799 } host ceph_3 { id -7 # do not change unnecessarily id -8 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.14 weight 5.34769 item osd.15 weight 5.45799 item osd.16 weight 5.45799 item osd.17 weight 5.45799 item osd.18 weight 5.45799 item osd.19 weight 5.45799 item osd.20 weight 5.45799 } root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily # weight 114.28693 alg straw2 hash 0 # rjenkins1 item ceph_1 weight 38.09564 item ceph_2 weight 38.09564 item ceph_3 weight 38.09564 } # rules rule replicated_rule { id 0 type replicated step take default step chooseleaf firstn 0 type host step emit } rule my_replicated_rule { id 1 type replicated step take default step chooseleaf firstn 0 type host step emit } rule my_ec_rule { id 2 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step choose indep 3 type host step chooseleaf indep 2 type osd step emit } # end crush map ~~~ Finally I create a pool: ~~~ ceph osd pool create my_pool 32 32 erasure my_ec_profile my_ec_rule ceph osd pool application enable my_meta_pool rbd rbd pool init my_meta_pool rbd pool init my_pool rbd create --size 16T my_pool/my_disk_1 --data-pool my_pool --image-feature journaling ~~~ So all this is to have some VMs (oVirt VMs, for the record) with automatic fall-over in the case of a Ceph Node loss - ie I was trying to "replicate" a 3-Disk RAID 5 array across the Ceph Nodes, so that I could loose a Node and still have a working set of VMs. However, I took one of the Ceph Nodes down (gracefully) for some maintenance the other day and I lost *all* the VMs (ie oVirt complained that there was no active pool). As soon as I brought the down node back up everything was good again. So my question is: What did I do wrong with my config? Sound I, for example, change the EC Profile to `k=2, m=1`, but how is that practically different from `k=4, m=2` - yes, the later spreads the pool over more disks, but it should still only put 2 disks on each node, shouldn't it? Thanks in advance Cheers Dulux-Oz

11 18

CLT Meeting minutes 2023-12-06
by Laura Flores 06 Dec '23

06 Dec '23

Closing the loop (blocked waiting for Neha's input): how are we using Gibba on a day-to-day basis? Is it only used for checking reef point releases? - To be discussed again next week, as Neha had a conflict [Nizam] http://old.ceph.com/pgcalc is not working anymore, is there any replacement for pgcalc page in the new ceph site? - Last attempt at a fix https://github.com/ceph/ceph.io/issues/265 - Nizam will take a look; the PR needs some CSS work - We may not even need this link due to the autoscaler, etc. - Perhaps add some kind of "banner" to the pgcalc page to make it clear that users should look to the autoscaler Update on 18.2.1 bluestore issue? - Fix has been raised; also not as serious as initially suspected - Fix is in testing -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

2 1

ceph df reports incorrect stats
by Frank Schilder 06 Dec '23

06 Dec '23

Dear fellow cephers, we got a problem with ceph df: ceph df reports incorrect USED. It would be great if someone could look at this, if a ceph operator doesn't discover this issue, they might run out of space without noticing. This has been reported before but didn't get much attention: https://www.spinics.net/lists/ceph-users/msg74602.html https://www.spinics.net/lists/ceph-users/msg74630.html The symptom: STORED=USED in output of ceph df. All reports I know of are for octopus clusters, but I suspect newer versions are affected as well. I don't have a reproducer yet (still lacking a test cluster). Here is a correct usage report: ==> logs/health_231203.log <== --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 13 PiB 7.8 PiB 4.8 PiB 4.8 PiB 38.29 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL con-fs2-data 14 2048 1.1 PiB 402.93M 1.2 PiB 20.95 3.7 PiB con-fs2-data2 19 8192 2.7 PiB 1.10G 3.4 PiB 42.78 3.3 PiB Here is an incorrect one: ==> logs/health_231204.log <== --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 13 PiB 7.8 PiB 4.8 PiB 4.8 PiB 38.06 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL con-fs2-data 14 2048 1.1 PiB 402.93M 1.1 PiB 18.82 3.6 PiB con-fs2-data2 19 8192 2.7 PiB 1.10G 2.7 PiB 37.09 3.3 PiB That the first report is correct and not the second is supported by the output of ceph osd df tree, showing a use of 4.6PB in alignment with the first output of ceph df. Note that the date of the ceph osd df tree output is identical to the date of the incorrect ceph df output, hence, ceph osd df tree is *not* affected by this issue: ==> ceph osd df tree 231204 <=== SIZE RAW USE DATA OMAP META AVAIL NAME 12 PiB 4.6 PiB 4.6 PiB 2.2 TiB 19 TiB 7.5 PiB datacenter ContainerSquare 0 B 0 B 0 B 0 B 0 B 0 B room CON-161-A 12 PiB 4.6 PiB 4.6 PiB 2.2 TiB 19 TiB 7.5 PiB room CON-161-A1 In our case, the problem showed up out of nowhere. Here the log snippet for the time window within which the flip happened (compare the lines for con-fs2-data?-pools): ==> logs/health_231203.log <== ceph status/df/pool stats/health detail at 16:30:03: cluster: health: HEALTH_OK services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 3M) mgr: ceph-25(active, since 2M), standbys: ceph-26, ceph-01, ceph-03, ceph-02 mds: con-fs2:8 4 up:standby 8 up:active osd: 1284 osds: 1279 up (since 14h), 1279 in (since 2w) task status: data: pools: 14 pools, 25065 pgs objects: 2.23G objects, 4.0 PiB usage: 5.0 PiB used, 8.1 PiB / 13 PiB avail pgs: 25035 active+clean 29 active+clean+scrubbing+deep 1 active+clean+scrubbing io: client: 215 MiB/s rd, 140 MiB/s wr, 2.34k op/s rd, 1.89k op/s wr --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED fs_meta 51 TiB 45 TiB 831 GiB 6.0 TiB 11.84 hdd 13 PiB 7.8 PiB 4.8 PiB 4.8 PiB 38.08 rbd_data 283 TiB 171 TiB 111 TiB 112 TiB 39.44 rbd_perf 42 TiB 22 TiB 20 TiB 20 TiB 48.60 TOTAL 13 PiB 8.1 PiB 4.9 PiB 5.0 PiB 38.04 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL sr-rbd-meta-one 1 128 13 GiB 16.57k 38 GiB 0.03 39 TiB sr-rbd-data-one 2 4096 121 TiB 32.06M 108 TiB 48.08 88 TiB sr-rbd-one-stretch 3 160 262 GiB 68.81k 573 GiB 0.48 39 TiB con-rbd-meta-hpc-one 7 50 12 KiB 45 372 KiB 0 9.2 TiB con-rbd-data-hpc-one 8 150 24 GiB 6.10k 24 GiB 0 3.6 PiB sr-rbd-data-one-hdd 11 1024 137 TiB 35.95M 193 TiB 46.57 166 TiB con-fs2-meta1 12 512 554 GiB 76.76M 2.2 TiB 7.26 6.9 TiB con-fs2-meta2 13 4096 0 B 574.23M 0 B 0 6.9 TiB con-fs2-data 14 2048 1.1 PiB 402.93M 1.2 PiB 21.09 3.6 PiB con-fs2-data-ec-ssd 17 256 700 GiB 7.27M 706 GiB 2.44 22 TiB ms-rbd-one 18 256 805 GiB 210.92k 1.4 TiB 1.18 39 TiB con-fs2-data2 19 8192 2.7 PiB 1.10G 3.4 PiB 42.96 3.3 PiB sr-rbd-data-one-perf 20 4096 6.8 TiB 1.81M 20 TiB 57.09 5.1 TiB device_health_metrics 21 1 1.4 GiB 1.11k 4.2 GiB 0 39 TiB ceph status/df/pool stats/health detail at 16:30:10: cluster: health: HEALTH_OK services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 3M) mgr: ceph-25(active, since 2M), standbys: ceph-26, ceph-01, ceph-03, ceph-02 mds: con-fs2:8 4 up:standby 8 up:active osd: 1284 osds: 1279 up (since 14h), 1279 in (since 2w) task status: data: pools: 14 pools, 25065 pgs objects: 2.23G objects, 4.0 PiB usage: 5.0 PiB used, 8.1 PiB / 13 PiB avail pgs: 25035 active+clean 29 active+clean+scrubbing+deep 1 active+clean+scrubbing io: client: 241 MiB/s rd, 174 MiB/s wr, 2.68k op/s rd, 2.34k op/s wr --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED fs_meta 51 TiB 45 TiB 830 GiB 6.0 TiB 11.84 hdd 13 PiB 7.8 PiB 4.8 PiB 4.8 PiB 38.08 rbd_data 283 TiB 171 TiB 111 TiB 112 TiB 39.44 rbd_perf 42 TiB 22 TiB 20 TiB 20 TiB 48.60 TOTAL 13 PiB 8.1 PiB 4.9 PiB 5.0 PiB 38.04 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL sr-rbd-meta-one 1 128 13 GiB 16.57k 13 GiB 0.01 39 TiB sr-rbd-data-one 2 4096 92 TiB 32.06M 92 TiB 44.11 88 TiB sr-rbd-one-stretch 3 160 222 GiB 68.81k 222 GiB 0.19 39 TiB con-rbd-meta-hpc-one 7 50 6.9 KiB 45 6.9 KiB 0 9.2 TiB con-rbd-data-hpc-one 8 150 23 GiB 6.10k 23 GiB 0 3.6 PiB sr-rbd-data-one-hdd 11 1024 135 TiB 35.95M 135 TiB 37.88 166 TiB con-fs2-meta1 12 512 367 GiB 76.76M 367 GiB 1.28 6.9 TiB con-fs2-meta2 13 4096 0 B 574.23M 0 B 0 6.9 TiB con-fs2-data 14 2048 1.1 PiB 402.93M 1.1 PiB 18.82 3.6 PiB con-fs2-data-ec-ssd 17 256 515 GiB 7.27M 515 GiB 1.79 22 TiB ms-rbd-one 18 256 579 GiB 210.92k 579 GiB 0.48 39 TiB con-fs2-data2 19 8192 2.7 PiB 1.10G 2.7 PiB 37.09 3.3 PiB sr-rbd-data-one-perf 20 4096 6.9 TiB 1.81M 6.9 TiB 31.29 5.1 TiB device_health_metrics 21 1 1.2 GiB 1.11k 1.2 GiB 0 39 TiB For us, the issue disappeared after taking some OSDs in a second root down. These OSDs were moved there for draining, we use a second crush root for this purpose. Here the log snippet with the time window within which the back-flip to correct reporting happened: ==> logs/health_231205.log <== ceph status/df/pool stats/health detail at 17:42:58: cluster: health: HEALTH_WARN 1 osds down 24 hosts (12 osds) down 1 root (12 osds) down services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 3M) mgr: ceph-25(active, since 2M), standbys: ceph-26, ceph-01, ceph-03, ceph-02 mds: con-fs2:8 4 up:standby 8 up:active osd: 1284 osds: 1267 up (since 19m), 1268 in (since 0.401448s) task status: data: pools: 14 pools, 25065 pgs objects: 2.23G objects, 4.0 PiB usage: 5.0 PiB used, 8.0 PiB / 13 PiB avail pgs: 25034 active+clean 31 active+clean+scrubbing+deep io: client: 118 MiB/s rd, 789 MiB/s wr, 1.75k op/s rd, 2.14k op/s wr --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED fs_meta 51 TiB 45 TiB 731 GiB 5.9 TiB 11.65 hdd 13 PiB 7.8 PiB 4.8 PiB 4.8 PiB 38.36 rbd_data 283 TiB 171 TiB 111 TiB 112 TiB 39.59 rbd_perf 42 TiB 22 TiB 20 TiB 20 TiB 48.19 TOTAL 13 PiB 8.0 PiB 4.9 PiB 5.0 PiB 38.32 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL sr-rbd-meta-one 1 128 14 GiB 16.94k 14 GiB 0.01 39 TiB sr-rbd-data-one 2 4096 93 TiB 32.32M 93 TiB 44.29 88 TiB sr-rbd-one-stretch 3 160 222 GiB 68.81k 222 GiB 0.19 39 TiB con-rbd-meta-hpc-one 7 50 6.9 KiB 45 6.9 KiB 0 9.2 TiB con-rbd-data-hpc-one 8 150 23 GiB 6.10k 23 GiB 0 3.6 PiB sr-rbd-data-one-hdd 11 1024 135 TiB 36.08M 135 TiB 38.00 165 TiB con-fs2-meta1 12 512 367 GiB 76.81M 367 GiB 1.28 6.9 TiB con-fs2-meta2 13 4096 0 B 572.65M 0 B 0 6.9 TiB con-fs2-data 14 2048 1.1 PiB 402.93M 1.1 PiB 18.83 3.6 PiB con-fs2-data-ec-ssd 17 256 515 GiB 7.27M 515 GiB 1.78 22 TiB ms-rbd-one 18 256 579 GiB 210.92k 579 GiB 0.48 39 TiB con-fs2-data2 19 8192 2.7 PiB 1.10G 2.7 PiB 37.16 3.3 PiB sr-rbd-data-one-perf 20 4096 6.9 TiB 1.81M 6.9 TiB 31.07 5.1 TiB device_health_metrics 21 1 1.2 GiB 1.11k 1.2 GiB 0 39 TiB ceph status/df/pool stats/health detail at 17:43:04: cluster: health: HEALTH_OK services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 3M) mgr: ceph-25(active, since 2M), standbys: ceph-26, ceph-01, ceph-03, ceph-02 mds: con-fs2:8 4 up:standby 8 up:active osd: 1284 osds: 1267 up (since 19m), 1267 in (since 6s) task status: data: pools: 14 pools, 25065 pgs objects: 2.23G objects, 4.0 PiB usage: 5.0 PiB used, 8.0 PiB / 13 PiB avail pgs: 25035 active+clean 30 active+clean+scrubbing+deep io: client: 151 MiB/s rd, 840 MiB/s wr, 2.13k op/s rd, 2.10k op/s wr --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED fs_meta 51 TiB 45 TiB 731 GiB 5.9 TiB 11.65 hdd 13 PiB 7.7 PiB 4.8 PiB 4.8 PiB 38.42 rbd_data 283 TiB 171 TiB 111 TiB 112 TiB 39.59 rbd_perf 42 TiB 22 TiB 20 TiB 20 TiB 48.19 TOTAL 13 PiB 8.0 PiB 4.9 PiB 5.0 PiB 38.37 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL sr-rbd-meta-one 1 128 14 GiB 16.94k 42 GiB 0.04 39 TiB sr-rbd-data-one 2 4096 122 TiB 32.32M 109 TiB 48.26 88 TiB sr-rbd-one-stretch 3 160 262 GiB 68.81k 573 GiB 0.48 39 TiB con-rbd-meta-hpc-one 7 50 11 KiB 45 368 KiB 0 9.2 TiB con-rbd-data-hpc-one 8 150 24 GiB 6.10k 24 GiB 0 3.6 PiB sr-rbd-data-one-hdd 11 1024 138 TiB 36.08M 193 TiB 46.69 165 TiB con-fs2-meta1 12 512 555 GiB 76.81M 2.2 TiB 7.26 6.9 TiB con-fs2-meta2 13 4096 0 B 572.65M 0 B 0 6.9 TiB con-fs2-data 14 2048 1.1 PiB 402.93M 1.2 PiB 21.09 3.6 PiB con-fs2-data-ec-ssd 17 256 700 GiB 7.27M 706 GiB 2.43 22 TiB ms-rbd-one 18 256 805 GiB 210.92k 1.4 TiB 1.18 39 TiB con-fs2-data2 19 8192 2.7 PiB 1.10G 3.4 PiB 43.01 3.3 PiB sr-rbd-data-one-perf 20 4096 6.8 TiB 1.81M 20 TiB 56.75 5.1 TiB device_health_metrics 21 1 1.4 GiB 1.11k 4.2 GiB 0 39 TiB This observation leads me to suspect that having multiple crush roots might be a reason for this observation. Our crush tree looks like this (OSDs removed), it has 3 different roots (BB, DTU and default): ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -78 106.92188 root BB -99 0 host bb-04 -102 0 host bb-05 -105 0 host bb-06 -325 0 host bb-06-old -108 0 host bb-07 -331 0 host bb-07-old -3 8.91016 host bb-08 -9 8.91016 host bb-09 -18 8.91016 host bb-10 -21 8.91016 host bb-11 -28 8.91016 host bb-12 -34 8.91016 host bb-13 -72 8.91016 host bb-14 -75 8.91016 host bb-15 -111 8.91016 host bb-16 -114 8.91016 host bb-17 -117 0 host bb-18 -142 0 host bb-19 -145 0 host bb-20 -241 0 host bb-21 -246 0 host bb-22 -251 8.91016 host bb-23 -256 8.91016 host bb-24 -151 0 host bb-office -40 14614.77832 root DTU -42 0 region Lyngby -41 14614.77832 region Risoe -50 12843.79590 datacenter ContainerSquare -56 0 room CON-161-A -57 12843.79590 room CON-161-A1 -11 1092.49060 host ceph-08 -13 1074.27673 host ceph-09 -23 1075.67920 host ceph-10 -15 1067.16492 host ceph-11 -25 1080.21912 host ceph-12 -83 1061.17480 host ceph-13 -85 1047.70276 host ceph-14 -87 1079.02820 host ceph-15 -136 1012.55048 host ceph-16 -139 1073.61475 host ceph-17 -261 1125.57202 host ceph-23 -262 1054.32227 host ceph-24 -148 885.49133 datacenter MultiSite -65 86.16304 host ceph-04 -67 101.50623 host ceph-05 -69 104.85805 host ceph-06 -71 96.39923 host ceph-07 -81 97.54230 host ceph-18 -94 98.48271 host ceph-19 -4 97.20181 host ceph-20 -64 99.77657 host ceph-21 -66 103.56137 host ceph-22 -49 885.49133 datacenter ServerRoom -55 885.49133 room SR-113 -65 86.16304 host ceph-04 -67 101.50623 host ceph-05 -69 104.85805 host ceph-06 -71 96.39923 host ceph-07 -81 97.54230 host ceph-18 -94 98.48271 host ceph-19 -4 97.20181 host ceph-20 -64 99.77657 host ceph-21 -66 103.56137 host ceph-22 -1 0 root default Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 1

Re: After hardware failure tried to recover ceph and followed instructions for recovery using OSDS
by Eugen Block 06 Dec '23

06 Dec '23

And how about this? osd crush rm osd.14 Zitat von Manolis Daramas <mdaramas(a)motivian.com>: > Hi Eugen, > > I think that it cannot even be removed with ceph osd purge command, > see below: > > root@node01:~# ceph osd purge 14 --force --yes-i-really-mean-it > osd.14 does not exist > > root@node01:~# ceph osd tree > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > -1 2.05046 root default > -3 0.68349 host node01 > 0 hdd 0.14650 osd.0 up 1.00000 1.00000 > 4 hdd 0.04880 osd.4 up 1.00000 1.00000 > 8 hdd 0.04880 osd.8 up 1.00000 1.00000 > 10 hdd 0.04880 osd.10 up 1.00000 1.00000 > 14 hdd 0.39059 osd.14 DNE 0 > -5 0.68349 host node02 > 2 hdd 0.14650 osd.2 up 1.00000 1.00000 > 5 hdd 0.04880 osd.5 up 1.00000 1.00000 > 7 hdd 0.04880 osd.7 up 1.00000 1.00000 > 9 hdd 0.04880 osd.9 up 1.00000 1.00000 > 12 hdd 0.39059 osd.12 up 1.00000 1.00000 > -7 0.68349 host node03 > 1 hdd 0.14650 osd.1 up 1.00000 1.00000 > 3 hdd 0.04880 osd.3 up 1.00000 1.00000 > 6 hdd 0.04880 osd.6 up 1.00000 1.00000 > 11 hdd 0.04880 osd.11 up 1.00000 1.00000 > 13 hdd 0.39059 osd.13 up 1.00000 1.00000 > > The output of journalctl with --no-pager option is below: > > -- Logs begin at Thu 2023-11-23 06:16:17 EET, end at Tue 2023-12-05 > 16:01:49 EET. -- > Dec 05 15:23:16 node01 systemd[1]: Started Ceph mon.node01 for > be4304e4-b0d5-11ec-8c6a-2965d4229f37. > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 set uid:gid to 167:167 > (ceph:ceph) > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable), process > ceph-mon, pid 7 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 pidfile_write: ignore > empty --pid-file > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.667+0000 7fab0e25bb80 0 load: jerasure load: lrc > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: RocksDB > version: 6.15.5 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Git sha > rocksdb_build_git_sha:@0@ > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compile date > Oct 25 2023 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB SUMMARY > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB Session ID: > 69YYUY1IJK4BA9QQDDK8 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: CURRENT file: > CURRENT > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: IDENTITY file: > IDENTITY > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: MANIFEST file: > MANIFEST-000226 size: 308 Bytes > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: SST files in > /var/lib/ceph/mon/ceph-node01/store.db dir, Total Num: 3, files: > 000091.sst 000094.sst 000100.sst > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Write Ahead > Log file in /var/lib/ceph/mon/ceph-node01/store.db: 000227.log size: > 0 ; > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.error_if_exists: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.create_if_missing: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.paranoid_checks: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.track_and_verify_wals_in_manifest: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.env: 0x55be1e3aea80 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.fs: Posix File System > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.info_log: 0x55be20918360 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_file_opening_threads: 16 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.statistics: (nil) > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_fsync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_log_file_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_manifest_file_size: 1073741824 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.log_file_time_to_roll: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.keep_log_file_num: 1000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.recycle_log_file_num: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_fallocate: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_mmap_reads: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_mmap_writes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_direct_reads: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_direct_io_for_flush_and_compaction: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.create_missing_column_families: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_log_dir: > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_dir: > /var/lib/ceph/mon/ceph-node01/store.db > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_cache_numshardbits: 6 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.WAL_ttl_seconds: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.WAL_size_limit_MB: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_batch_group_size_bytes: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.manifest_preallocation_size: 4194304 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.is_fd_close_on_exec: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.advise_random_on_open: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_write_buffer_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_buffer_manager: 0x55be2168c990 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.access_hint_on_compaction_start: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.new_table_reader_for_compaction_inputs: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.random_access_max_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_adaptive_mutex: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.rate_limiter: (nil) > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.sst_file_manager.rate_bytes_per_sec: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_recovery_mode: 2 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_thread_tracking: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_pipelined_write: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.unordered_write: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_concurrent_memtable_write: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_write_thread_adaptive_yield: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_thread_max_yield_usec: 100 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_thread_slow_yield_usec: 3 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.row_cache: None > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_filter: None > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_flush_during_recovery: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_ingest_behind: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.preserve_deletes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.two_write_queues: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.manual_wal_flush: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.atomic_flush: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_unnecessary_blocking_io: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.persist_stats_to_disk: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_dbid_to_manifest: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.log_readahead_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.file_checksum_gen_factory: Unknown > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.best_efforts_recovery: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bgerror_resume_count: 2147483647 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bgerror_resume_retry_interval: 1000000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_data_in_errors: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_host_id: __hostname__ > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_jobs: 2 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_compactions: -1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_subcompactions: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_flush_during_shutdown: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.writable_file_max_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.delayed_write_rate : 16777216 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_total_wal_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.delete_obsolete_files_period_micros: 21600000000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_dump_period_sec: 600 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_persist_period_sec: 600 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_history_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_open_files: -1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.strict_bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_readahead_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_flushes: -1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compression > algorithms supported: > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kZSTDNotFinalCompression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kZSTD > supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kXpressCompression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kLZ4HCCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kZlibCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kSnappyCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kLZ4Compression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kBZip2Compression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Fast CRC32 > supported: Supported on x86 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4724] Recovering from manifest file: > /var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > [db/column_family.cc:595] --------------- Options for column family > [default]: > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.comparator: leveldb.BytewiseComparator > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.merge_operator: > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_filter: None > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_filter_factory: None > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.sst_partitioner_factory: None > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_factory: SkipListFactory > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_factory: BlockBasedTable > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > table_factory options: flush_block_policy_factory: > FlushBlockBySizePolicyFactory (0x55be2096afb0) > Dec 05 15:23:17 node01 bash[1969]: cache_index_and_filter_blocks: 1 > Dec 05 15:23:17 node01 bash[1969]: > cache_index_and_filter_blocks_with_high_priority: 0 > Dec 05 15:23:17 node01 bash[1969]: > pin_l0_filter_and_index_blocks_in_cache: 0 > Dec 05 15:23:17 node01 bash[1969]: pin_top_level_index_and_filter: 1 > Dec 05 15:23:17 node01 bash[1969]: index_type: 0 > Dec 05 15:23:17 node01 bash[1969]: data_block_index_type: 0 > Dec 05 15:23:17 node01 bash[1969]: index_shortening: 1 > Dec 05 15:23:17 node01 bash[1969]: > data_block_hash_table_util_ratio: 0.750000 > Dec 05 15:23:17 node01 bash[1969]: hash_index_allow_collision: 1 > Dec 05 15:23:17 node01 bash[1969]: checksum: 1 > Dec 05 15:23:17 node01 bash[1969]: no_block_cache: 0 > Dec 05 15:23:17 node01 bash[1969]: block_cache: 0x55be2099d090 > Dec 05 15:23:17 node01 bash[1969]: block_cache_name: BinnedLRUCache > Dec 05 15:23:17 node01 bash[1969]: block_cache_options: > Dec 05 15:23:17 node01 bash[1969]: capacity : 536870912 > Dec 05 15:23:17 node01 bash[1969]: num_shard_bits : 4 > Dec 05 15:23:17 node01 bash[1969]: strict_capacity_limit : 0 > Dec 05 15:23:17 node01 bash[1969]: high_pri_pool_ratio: 0.000 > Dec 05 15:23:17 node01 bash[1969]: block_cache_compressed: (nil) > Dec 05 15:23:17 node01 bash[1969]: persistent_cache: (nil) > Dec 05 15:23:17 node01 bash[1969]: block_size: 4096 > Dec 05 15:23:17 node01 bash[1969]: block_size_deviation: 10 > Dec 05 15:23:17 node01 bash[1969]: block_restart_interval: 16 > Dec 05 15:23:17 node01 bash[1969]: index_block_restart_interval: 1 > Dec 05 15:23:17 node01 bash[1969]: metadata_block_size: 4096 > Dec 05 15:23:17 node01 bash[1969]: partition_filters: 0 > Dec 05 15:23:17 node01 bash[1969]: use_delta_encoding: 1 > Dec 05 15:23:17 node01 bash[1969]: filter_policy: > rocksdb.BuiltinBloomFilter > Dec 05 15:23:17 node01 bash[1969]: whole_key_filtering: 1 > Dec 05 15:23:17 node01 bash[1969]: verify_compression: 0 > Dec 05 15:23:17 node01 bash[1969]: read_amp_bytes_per_bit: 0 > Dec 05 15:23:17 node01 bash[1969]: format_version: 4 > Dec 05 15:23:17 node01 bash[1969]: enable_index_compression: 1 > Dec 05 15:23:17 node01 bash[1969]: block_align: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_buffer_size: 33554432 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_number: 2 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression: NoCompression > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression: Disabled > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.prefix_extractor: nullptr > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_insert_with_hint_prefix_extractor: nullptr > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.num_levels: 7 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.min_write_buffer_number_to_merge: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_number_to_maintain: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_size_to_maintain: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.window_bits: -14 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.level: 32767 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.strategy: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.max_dict_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.parallel_threads: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.enabled: false > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.window_bits: -14 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.level: 32767 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.strategy: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.max_dict_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.parallel_threads: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.enabled: false > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_file_num_compaction_trigger: 4 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_slowdown_writes_trigger: 20 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_stop_writes_trigger: 36 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.target_file_size_base: 67108864 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.target_file_size_multiplier: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_base: 268435456 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level_compaction_dynamic_level_bytes: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier: 10.000000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[0]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[1]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[2]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[3]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[4]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[5]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[6]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_sequential_skip_in_iterations: 8 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_compaction_bytes: 1677721600 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.arena_block_size: 4194304 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.soft_pending_compaction_bytes_limit: 68719476736 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.hard_pending_compaction_bytes_limit: 274877906944 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.rate_limit_delay_max_milliseconds: 100 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.disable_auto_compactions: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_style: kCompactionStyleLevel > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_pri: kMinOverlappingRatio > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.size_ratio: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.min_merge_width: 2 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.max_merge_width: 4294967295 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.max_size_amplification_percent: > 200 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.compression_size_percent: -1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.stop_style: > kCompactionStopStyleTotalSize > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_fifo.max_table_files_size: 1073741824 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_fifo.allow_compaction: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_properties_collectors: > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.inplace_update_support: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.inplace_update_num_locks: 10000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_prefix_bloom_size_ratio: 0.000000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_whole_key_filtering: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_huge_page_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bloom_locality: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_successive_merges: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.optimize_filters_for_hits: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.paranoid_file_checks: Dec 05 15:23:17 node01 bash[1969]: > debug 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.force_consistency_checks: 1 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.report_bg_io_stats: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.ttl: 2592000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.periodic_compaction_seconds: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_blob_files: false > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.min_blob_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_file_size: 268435456 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_compression_type: NoCompression > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_blob_garbage_collection: false > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_garbage_collection_age_cutoff: 0.250000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4764] Recovered from manifest > file:/var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > succeeded,manifest_file_number is 226, next_file_number is 228, > last_sequence is 21201, log_number is 223,prev_log_number is > 0,max_column_family is 0,min_log_number_to_keep is 0 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4779] Column family [default] (ID 0), log number > is 223 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4082] Creating manifest 230 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1701782597693048, "job": 1, "event": > "recovery_started", "wal_files": [227]} > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/db_impl/db_impl_open.cc:845] Recovering log #227 mode 2 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4082] Creating manifest 231 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1701782597693989, "job": 1, "event": > "recovery_finished"} > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > [file/delete_scheduler.cc:69] Deleted file > /var/lib/ceph/mon/ceph-node01/store.db/000227.log immediately, > rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > [db/db_impl/db_impl_open.cc:1700] SstFileManager instance > 0x55be20931180 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: DB pointer > 0x55be20a06000 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > [db/db_impl/db_impl.cc:903] > Dec 05 15:23:17 node01 bash[1969]: ** DB Stats ** > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Cumulative writes: 0 writes, 0 > keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, > 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Cumulative WAL: 0 writes, 0 > syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Cumulative stall: 00:00:0.000 > H:M:S, 0.0 percent > Dec 05 15:23:17 node01 bash[1969]: Interval writes: 0 writes, 0 > keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, > 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Interval WAL: 0 writes, 0 syncs, > 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Interval stall: 00:00:0.000 > H:M:S, 0.0 percent > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > level0_numfiles_with_compaction, 0 stop for > pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > memtable_compaction, 0 memtable_slowdown, interval 0 total count > Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > Level [default] ** > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > level0_numfiles_with_compaction, 0 stop for > pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > memtable_compaction, 0 memtable_slowdown, interval 0 total count > Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > Level [default] ** > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 0 starting mon.node01 > rank 0 at public addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] > at bind addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] mon_data > /var/lib/ceph/mon/ceph-node01 fsid > be4304e4-b0d5-11ec-8c6a-2965d4229f37 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.723+0000 7fab0e25bb80 1 mon.node01@-1(???) e0 > preinit fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37 > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 3314933000854323200, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: terminate called after throwing > an instance of 'std::invalid_argument' > Dec 05 15:23:17 node01 bash[1969]: what(): stoull > Dec 05 15:23:17 node01 bash[1969]: *** Caught signal (Aborted) ** > Dec 05 15:23:17 node01 bash[1969]: in thread 7fab0e25bb80 > thread_name:ceph-mon > Dec 05 15:23:17 node01 bash[1969]: ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) > Dec 05 15:23:17 node01 bash[1969]: 1: > /lib64/libpthread.so.0(+0x12cf0) [0x7fab0b6c8cf0] > Dec 05 15:23:17 node01 bash[1969]: 2: gsignal() > Dec 05 15:23:17 node01 bash[1969]: 3: abort() > Dec 05 15:23:17 node01 bash[1969]: 4: > /lib64/libstdc++.so.6(+0x9009b) [0x7fab0acda09b] > Dec 05 15:23:17 node01 bash[1969]: 5: > /lib64/libstdc++.so.6(+0x9654c) [0x7fab0ace054c] > Dec 05 15:23:17 node01 bash[1969]: 6: > /lib64/libstdc++.so.6(+0x965a7) [0x7fab0ace05a7] > Dec 05 15:23:17 node01 bash[1969]: 7: > /lib64/libstdc++.so.6(+0x96808) [0x7fab0ace0808] > Dec 05 15:23:17 node01 bash[1969]: 8: > /lib64/libstdc++.so.6(+0x91ff9) [0x7fab0acdbff9] > Dec 05 15:23:17 node01 bash[1969]: 9: > (LogMonitor::log_external_backlog()+0xe70) [0x55be1dc27190] > Dec 05 15:23:17 node01 bash[1969]: 10: > (LogMonitor::update_from_paxos(bool*)+0x54) [0x55be1dc27334] > Dec 05 15:23:17 node01 bash[1969]: 11: > (Monitor::refresh_from_paxos(bool*)+0x104) [0x55be1db9ae74] > Dec 05 15:23:17 node01 bash[1969]: 12: (Monitor::preinit()+0x95d) > [0x55be1dbc93ad] > Dec 05 15:23:17 node01 bash[1969]: 13: main() > Dec 05 15:23:17 node01 bash[1969]: 14: __libc_start_main() > Dec 05 15:23:17 node01 bash[1969]: 15: _start() > Dec 05 15:23:17 node01 bash[1969]: debug > 2023-12-05T13:23:17.735+0000 7fab0e25bb80 -1 *** Caught signal > (Aborted) ** > Dec 05 15:23:17 node01 bash[1969]: in thread 7fab0e25bb80 > thread_name:ceph-mon > Dec 05 15:23:17 node01 bash[1969]: ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) > Dec 05 15:23:17 node01 bash[1969]: 1: > /lib64/libpthread.so.0(+0x12cf0) [0x7fab0b6c8cf0] > Dec 05 15:23:17 node01 bash[1969]: 2: gsignal() > Dec 05 15:23:17 node01 bash[1969]: 3: abort() > Dec 05 15:23:17 node01 bash[1969]: 4: > /lib64/libstdc++.so.6(+0x9009b) [0x7fab0acda09b] > Dec 05 15:23:17 node01 bash[1969]: 5: > /lib64/libstdc++.so.6(+0x9654c) [0x7fab0ace054c] > Dec 05 15:23:17 node01 bash[1969]: 6: > /lib64/libstdc++.so.6(+0x965a7) [0x7fab0ace05a7] > Dec 05 15:23:17 node01 bash[1969]: 7: > /lib64/libstdc++.so.6(+0x96808) [0x7fab0ace0808] > Dec 05 15:23:17 node01 bash[1969]: 8: > /lib64/libstdc++.so.6(+0x91ff9) [0x7fab0acdbff9] > Dec 05 15:23:17 node01 bash[1969]: 9: > (LogMonitor::log_external_backlog()+0xe70) [0x55be1dc27190] > Dec 05 15:23:17 node01 bash[1969]: 10: > (LogMonitor::update_from_paxos(bool*)+0x54) [0x55be1dc27334] > Dec 05 15:23:17 node01 bash[1969]: 11: > (Monitor::refresh_from_paxos(bool*)+0x104) [0x55be1db9ae74] > Dec 05 15:23:17 node01 bash[1969]: 12: (Monitor::preinit()+0x95d) > [0x55be1dbc93ad] > Dec 05 15:23:17 node01 bash[1969]: 13: main() > Dec 05 15:23:17 node01 bash[1969]: 14: __libc_start_main() > Dec 05 15:23:17 node01 bash[1969]: 15: _start() > Dec 05 15:23:17 node01 bash[1969]: NOTE: a copy of the executable, > or `objdump -rdS <executable>` is needed to interpret this. > Dec 05 15:23:17 node01 bash[1969]: --- begin dump of recent events --- > Dec 05 15:23:17 node01 bash[1969]: debug -290> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command assert hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -289> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command abort hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -288> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command leak_some_memory hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -287> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perfcounters_dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -286> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command 1 hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -285> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -284> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perfcounters_schema hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -283> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf histogram dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -282> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command 2 hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -281> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf schema hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -280> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf histogram schema hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -279> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf reset hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -278> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config show hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -277> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config help hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -276> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config set hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -275> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config unset hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -274> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config get hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -273> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config diff hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -272> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config diff get hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -271> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command injectargs hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -270> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command log flush hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -269> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command log dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -268> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command log reopen hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -267> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command dump_mempools hook 0x55be2168a068 > Dec 05 15:23:17 node01 bash[1969]: debug -266> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 set uid:gid to 167:167 > (ceph:ceph) > Dec 05 15:23:17 node01 bash[1969]: debug -265> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable), process > ceph-mon, pid 7 > Dec 05 15:23:17 node01 bash[1969]: debug -264> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 pidfile_write: ignore > empty --pid-file > Dec 05 15:23:17 node01 bash[1969]: debug -263> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > init /var/run/ceph/ceph-mon.node01.asok > Dec 05 15:23:17 node01 bash[1969]: debug -262> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > bind_and_listen /var/run/ceph/ceph-mon.node01.asok > Dec 05 15:23:17 node01 bash[1969]: debug -261> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command 0 hook 0x55be2096b030 > Dec 05 15:23:17 node01 bash[1969]: debug -260> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command version hook 0x55be2096b030 > Dec 05 15:23:17 node01 bash[1969]: debug -259> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command git_version hook 0x55be2096b030 > Dec 05 15:23:17 node01 bash[1969]: debug -258> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command help hook 0x55be208de5b0 > Dec 05 15:23:17 node01 bash[1969]: debug -257> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command get_command_descriptions hook 0x55be208de5a0 > Dec 05 15:23:17 node01 bash[1969]: debug -256> > 2023-12-05T13:23:17.639+0000 7fab076ba700 5 asok(0x55be20bc0000) > entry start > Dec 05 15:23:17 node01 bash[1969]: debug -255> > 2023-12-05T13:23:17.667+0000 7fab0e25bb80 0 load: jerasure load: lrc > Dec 05 15:23:17 node01 bash[1969]: debug -254> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > level_compaction_dynamic_level_bytes = true > Dec 05 15:23:17 node01 bash[1969]: debug -253> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > write_buffer_size = 33554432 > Dec 05 15:23:17 node01 bash[1969]: debug -252> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > compression = kNoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -251> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > level_compaction_dynamic_level_bytes = true > Dec 05 15:23:17 node01 bash[1969]: debug -250> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > write_buffer_size = 33554432 > Dec 05 15:23:17 node01 bash[1969]: debug -249> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > compression = kNoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -248> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 5 rocksdb: > verify_sharding column families from rocksdb: [default] > Dec 05 15:23:17 node01 bash[1969]: debug -247> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: RocksDB > version: 6.15.5 > Dec 05 15:23:17 node01 bash[1969]: debug -246> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Git sha > rocksdb_build_git_sha:@0@ > Dec 05 15:23:17 node01 bash[1969]: debug -245> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compile date > Oct 25 2023 > Dec 05 15:23:17 node01 bash[1969]: debug -244> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB SUMMARY > Dec 05 15:23:17 node01 bash[1969]: debug -243> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB Session ID: > 69YYUY1IJK4BA9QQDDK8 > Dec 05 15:23:17 node01 bash[1969]: debug -242> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: CURRENT file: > CURRENT > Dec 05 15:23:17 node01 bash[1969]: debug -241> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: IDENTITY file: > IDENTITY > Dec 05 15:23:17 node01 bash[1969]: debug -240> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: MANIFEST file: > MANIFEST-000226 size: 308 Bytes > Dec 05 15:23:17 node01 bash[1969]: debug -239> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: SST files in > /var/lib/ceph/mon/ceph-node01/store.db dir, Total Num: 3, files: > 000091.sst 000094.sst 000100.sst > Dec 05 15:23:17 node01 bash[1969]: debug -238> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Write Ahead > Log file in /var/lib/ceph/mon/ceph-node01/store.db: 000227.log size: > 0 ; > Dec 05 15:23:17 node01 bash[1969]: debug -237> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.error_if_exists: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -236> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.create_if_missing: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -235> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.paranoid_checks: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -234> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.track_and_verify_wals_in_manifest: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -233> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.env: 0x55be1e3aea80 > Dec 05 15:23:17 node01 bash[1969]: debug -232> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.fs: Posix File System > Dec 05 15:23:17 node01 bash[1969]: debug -231> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.info_log: 0x55be20918360 > Dec 05 15:23:17 node01 bash[1969]: debug -230> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_file_opening_threads: 16 > Dec 05 15:23:17 node01 bash[1969]: debug -229> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.statistics: (nil) > Dec 05 15:23:17 node01 bash[1969]: debug -228> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_fsync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -227> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_log_file_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -226> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_manifest_file_size: 1073741824 > Dec 05 15:23:17 node01 bash[1969]: debug -225> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.log_file_time_to_roll: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -224> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.keep_log_file_num: 1000 > Dec 05 15:23:17 node01 bash[1969]: debug -223> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.recycle_log_file_num: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -222> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_fallocate: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -221> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_mmap_reads: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -220> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_mmap_writes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -219> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_direct_reads: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -218> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_direct_io_for_flush_and_compaction: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -217> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.create_missing_column_families: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -216> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_log_dir: > Dec 05 15:23:17 node01 bash[1969]: debug -215> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_dir: > /var/lib/ceph/mon/ceph-node01/store.db > Dec 05 15:23:17 node01 bash[1969]: debug -214> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_cache_numshardbits: 6 > Dec 05 15:23:17 node01 bash[1969]: debug -213> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.WAL_ttl_seconds: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -212> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.WAL_size_limit_MB: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -211> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_batch_group_size_bytes: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -210> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.manifest_preallocation_size: 4194304 > Dec 05 15:23:17 node01 bash[1969]: debug -209> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.is_fd_close_on_exec: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -208> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.advise_random_on_open: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -207> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_write_buffer_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -206> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_buffer_manager: 0x55be2168c990 > Dec 05 15:23:17 node01 bash[1969]: debug -205> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.access_hint_on_compaction_start: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -204> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.new_table_reader_for_compaction_inputs: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -203> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.random_access_max_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -202> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_adaptive_mutex: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -201> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.rate_limiter: (nil) > Dec 05 15:23:17 node01 bash[1969]: debug -200> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.sst_file_manager.rate_bytes_per_sec: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -199> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_recovery_mode: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -198> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_thread_tracking: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -197> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_pipelined_write: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -196> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.unordered_write: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -195> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_concurrent_memtable_write: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -194> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_write_thread_adaptive_yield: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -193> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_thread_max_yield_usec: 100 > Dec 05 15:23:17 node01 bash[1969]: debug -192> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_thread_slow_yield_usec: 3 > Dec 05 15:23:17 node01 bash[1969]: debug -191> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.row_cache: None > Dec 05 15:23:17 node01 bash[1969]: debug -190> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_filter: None > Dec 05 15:23:17 node01 bash[1969]: debug -189> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_flush_during_recovery: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -188> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_ingest_behind: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -187> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.preserve_deletes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -186> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.two_write_queues: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -185> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.manual_wal_flush: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -184> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.atomic_flush: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -183> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_unnecessary_blocking_io: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -182> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.persist_stats_to_disk: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -181> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_dbid_to_manifest: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -180> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.log_readahead_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -179> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.file_checksum_gen_factory: Unknown > Dec 05 15:23:17 node01 bash[1969]: debug -178> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.best_efforts_recovery: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -177> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bgerror_resume_count: 2147483647 > Dec 05 15:23:17 node01 bash[1969]: debug -176> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bgerror_resume_retry_interval: 1000000 > Dec 05 15:23:17 node01 bash[1969]: debug -175> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_data_in_errors: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -174> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_host_id: __hostname__ > Dec 05 15:23:17 node01 bash[1969]: debug -173> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_jobs: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -172> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_compactions: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -171> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_subcompactions: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -170> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_flush_during_shutdown: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -169> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.writable_file_max_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -168> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.delayed_write_rate : 16777216 > Dec 05 15:23:17 node01 bash[1969]: debug -167> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_total_wal_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -166> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.delete_obsolete_files_period_micros: 21600000000 > Dec 05 15:23:17 node01 bash[1969]: debug -165> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_dump_period_sec: 600 > Dec 05 15:23:17 node01 bash[1969]: debug -164> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_persist_period_sec: 600 > Dec 05 15:23:17 node01 bash[1969]: debug -163> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_history_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -162> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_open_files: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -161> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -160> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -159> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.strict_bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -158> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_readahead_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -157> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_flushes: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -156> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compression > algorithms supported: > Dec 05 15:23:17 node01 bash[1969]: debug -155> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kZSTDNotFinalCompression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -154> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kZSTD > supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -153> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kXpressCompression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -152> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kLZ4HCCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -151> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kZlibCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -150> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kSnappyCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -149> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kLZ4Compression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -148> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kBZip2Compression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -147> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Fast CRC32 > supported: Supported on x86 > Dec 05 15:23:17 node01 bash[1969]: debug -146> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4724] Recovering from manifest file: > /var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > Dec 05 15:23:17 node01 bash[1969]: debug -145> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > [db/column_family.cc:595] --------------- Options for column family > [default]: > Dec 05 15:23:17 node01 bash[1969]: debug -144> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.comparator: leveldb.BytewiseComparator > Dec 05 15:23:17 node01 bash[1969]: debug -143> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.merge_operator: > Dec 05 15:23:17 node01 bash[1969]: debug -142> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_filter: None > Dec 05 15:23:17 node01 bash[1969]: debug -141> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_filter_factory: None > Dec 05 15:23:17 node01 bash[1969]: debug -140> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.sst_partitioner_factory: None > Dec 05 15:23:17 node01 bash[1969]: debug -139> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_factory: SkipListFactory > Dec 05 15:23:17 node01 bash[1969]: debug -138> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_factory: BlockBasedTable > Dec 05 15:23:17 node01 bash[1969]: debug -137> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > table_factory options: flush_block_policy_factory: > FlushBlockBySizePolicyFactory (0x55be2096afb0) > Dec 05 15:23:17 node01 bash[1969]: cache_index_and_filter_blocks: 1 > Dec 05 15:23:17 node01 bash[1969]: > cache_index_and_filter_blocks_with_high_priority: 0 > Dec 05 15:23:17 node01 bash[1969]: > pin_l0_filter_and_index_blocks_in_cache: 0 > Dec 05 15:23:17 node01 bash[1969]: pin_top_level_index_and_filter: 1 > Dec 05 15:23:17 node01 bash[1969]: index_type: 0 > Dec 05 15:23:17 node01 bash[1969]: data_block_index_type: 0 > Dec 05 15:23:17 node01 bash[1969]: index_shortening: 1 > Dec 05 15:23:17 node01 bash[1969]: > data_block_hash_table_util_ratio: 0.750000 > Dec 05 15:23:17 node01 bash[1969]: hash_index_allow_collision: 1 > Dec 05 15:23:17 node01 bash[1969]: checksum: 1 > Dec 05 15:23:17 node01 bash[1969]: no_block_cache: 0 > Dec 05 15:23:17 node01 bash[1969]: block_cache: 0x55be2099d090 > Dec 05 15:23:17 node01 bash[1969]: block_cache_name: BinnedLRUCache > Dec 05 15:23:17 node01 bash[1969]: block_cache_options: > Dec 05 15:23:17 node01 bash[1969]: capacity : 536870912 > Dec 05 15:23:17 node01 bash[1969]: num_shard_bits : 4 > Dec 05 15:23:17 node01 bash[1969]: strict_capacity_limit : 0 > Dec 05 15:23:17 node01 bash[1969]: high_pri_pool_ratio: 0.000 > Dec 05 15:23:17 node01 bash[1969]: block_cache_compressed: (nil) > Dec 05 15:23:17 node01 bash[1969]: persistent_cache: (nil) > Dec 05 15:23:17 node01 bash[1969]: block_size: 4096 > Dec 05 15:23:17 node01 bash[1969]: block_size_deviation: 10 > Dec 05 15:23:17 node01 bash[1969]: block_restart_interval: 16 > Dec 05 15:23:17 node01 bash[1969]: index_block_restart_interval: 1 > Dec 05 15:23:17 node01 bash[1969]: metadata_block_size: 4096 > Dec 05 15:23:17 node01 bash[1969]: partition_filters: 0 > Dec 05 15:23:17 node01 bash[1969]: use_delta_encoding: 1 > Dec 05 15:23:17 node01 bash[1969]: filter_policy: > rocksdb.BuiltinBloomFilter > Dec 05 15:23:17 node01 bash[1969]: whole_key_filtering: 1 > Dec 05 15:23:17 node01 bash[1969]: verify_compression: 0 > Dec 05 15:23:17 node01 bash[1969]: read_amp_bytes_per_bit: 0 > Dec 05 15:23:17 node01 bash[1969]: format_version: 4 > Dec 05 15:23:17 node01 bash[1969]: enable_index_compression: 1 > Dec 05 15:23:17 node01 bash[1969]: block_align: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -136> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_buffer_size: 33554432 > Dec 05 15:23:17 node01 bash[1969]: debug -135> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_number: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -134> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression: NoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -133> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression: Disabled > Dec 05 15:23:17 node01 bash[1969]: debug -132> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.prefix_extractor: nullptr > Dec 05 15:23:17 node01 bash[1969]: debug -131> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_insert_with_hint_prefix_extractor: nullptr > Dec 05 15:23:17 node01 bash[1969]: debug -130> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.num_levels: 7 > Dec 05 15:23:17 node01 bash[1969]: debug -129> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.min_write_buffer_number_to_merge: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -128> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_number_to_maintain: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -127> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_size_to_maintain: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -126> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.window_bits: -14 > Dec 05 15:23:17 node01 bash[1969]: debug -125> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.level: 32767 > Dec 05 15:23:17 node01 bash[1969]: debug -124> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.strategy: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -123> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.max_dict_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -122> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -121> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.parallel_threads: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -120> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.enabled: false > Dec 05 15:23:17 node01 bash[1969]: debug -119> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.window_bits: -14 > Dec 05 15:23:17 node01 bash[1969]: debug -118> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.level: 32767 > Dec 05 15:23:17 node01 bash[1969]: debug -117> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.strategy: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -116> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.max_dict_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -115> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -114> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.parallel_threads: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -113> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.enabled: false > Dec 05 15:23:17 node01 bash[1969]: debug -112> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_file_num_compaction_trigger: 4 > Dec 05 15:23:17 node01 bash[1969]: debug -111> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_slowdown_writes_trigger: 20 > Dec 05 15:23:17 node01 bash[1969]: debug -110> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_stop_writes_trigger: 36 > Dec 05 15:23:17 node01 bash[1969]: debug -109> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.target_file_size_base: 67108864 > Dec 05 15:23:17 node01 bash[1969]: debug -108> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.target_file_size_multiplier: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -107> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_base: 268435456 > Dec 05 15:23:17 node01 bash[1969]: debug -106> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level_compaction_dynamic_level_bytes: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -105> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier: 10.000000 > Dec 05 15:23:17 node01 bash[1969]: debug -104> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[0]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -103> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[1]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -102> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[2]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -101> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[3]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -100> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[4]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -99> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[5]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -98> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[6]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -97> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_sequential_skip_in_iterations: 8 > Dec 05 15:23:17 node01 bash[1969]: debug -96> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_compaction_bytes: 1677721600 > Dec 05 15:23:17 node01 bash[1969]: debug -95> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.arena_block_size: 4194304 > Dec 05 15:23:17 node01 bash[1969]: debug -94> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.soft_pending_compaction_bytes_limit: 68719476736 > Dec 05 15:23:17 node01 bash[1969]: debug -93> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.hard_pending_compaction_bytes_limit: 274877906944 > Dec 05 15:23:17 node01 bash[1969]: debug -92> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.rate_limit_delay_max_milliseconds: 100 > Dec 05 15:23:17 node01 bash[1969]: debug -91> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.disable_auto_compactions: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -90> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_style: kCompactionStyleLevel > Dec 05 15:23:17 node01 bash[1969]: debug -89> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_pri: kMinOverlappingRatio > Dec 05 15:23:17 node01 bash[1969]: debug -88> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.size_ratio: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -87> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.min_merge_width: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -86> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.max_merge_width: 4294967295 > Dec 05 15:23:17 node01 bash[1969]: debug -85> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.max_size_amplification_percent: > 200 > Dec 05 15:23:17 node01 bash[1969]: debug -84> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.compression_size_percent: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -83> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.stop_style: > kCompactionStopStyleTotalSize > Dec 05 15:23:17 node01 bash[1969]: debug -82> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_fifo.max_table_files_size: 1073741824 > Dec 05 15:23:17 node01 bash[1969]: debug -81> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_fifo.allow_compaction: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -80> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_properties_collectors: > Dec 05 15:23:17 node01 bash[1969]: debug -79> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.inplace_update_support: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -78> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.inplace_update_num_locks: 10000 > Dec 05 15:23:17 node01 bash[1969]: debug -77> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_prefix_bloom_size_ratio: 0.000000 > Dec 05 15:23:17 node01 bash[1969]: debug -76> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_whole_key_filtering: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -75> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_huge_page_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -74> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bloom_locality: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -73> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_successive_merges: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -72> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.optimize_filters_for_hits: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -71> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.paranoid_file_checks: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -70> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.force_consistency_checks: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -69> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.report_bg_io_stats: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -68> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.ttl: 2592000 > Dec 05 15:23:17 node01 bash[1969]: debug -67> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.periodic_compaction_seconds: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -66> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_blob_files: false > Dec 05 15:23:17 node01 bash[1969]: debug -65> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.min_blob_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -64> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_file_size: 268435456 > Dec 05 15:23:17 node01 bash[1969]: debug -63> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_compression_type: NoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -62> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_blob_garbage_collection: false > Dec 05 15:23:17 node01 bash[1969]: debug -61> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_garbage_collection_age_cutoff: 0.250000 > Dec 05 15:23:17 node01 bash[1969]: debug -60> > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug -59> > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4764] Recovered from manifest > file:/var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > succeeded,manifest_file_number is 226, next_file_number is 228, > last_sequence is 21201, log_number is 223,prev_log_number is > 0,max_column_family is 0,min_log_number_to_keep is 0 > Dec 05 15:23:17 node01 bash[1969]: debug -58> > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4779] Column family [default] (ID 0), log number > is 223 > Dec 05 15:23:17 node01 bash[1969]: debug -57> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4082] Creating manifest 230 > Dec 05 15:23:17 node01 bash[1969]: debug -56> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug -55> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1701782597693048, "job": 1, "event": > "recovery_started", "wal_files": [227]} > Dec 05 15:23:17 node01 bash[1969]: debug -54> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/db_impl/db_impl_open.cc:845] Recovering log #227 mode 2 > Dec 05 15:23:17 node01 bash[1969]: debug -53> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4082] Creating manifest 231 > Dec 05 15:23:17 node01 bash[1969]: debug -52> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug -51> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1701782597693989, "job": 1, "event": > "recovery_finished"} > Dec 05 15:23:17 node01 bash[1969]: debug -50> > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > [file/delete_scheduler.cc:69] Deleted file > /var/lib/ceph/mon/ceph-node01/store.db/000227.log immediately, > rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 > Dec 05 15:23:17 node01 bash[1969]: debug -49> > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > [db/db_impl/db_impl_open.cc:1700] SstFileManager instance > 0x55be20931180 > Dec 05 15:23:17 node01 bash[1969]: debug -48> > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: DB pointer > 0x55be20a06000 > Dec 05 15:23:17 node01 bash[1969]: debug -47> > 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- > Dec 05 15:23:17 node01 bash[1969]: debug -46> > 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > [db/db_impl/db_impl.cc:903] > Dec 05 15:23:17 node01 bash[1969]: ** DB Stats ** > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Cumulative writes: 0 writes, 0 > keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, > 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Cumulative WAL: 0 writes, 0 > syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Cumulative stall: 00:00:0.000 > H:M:S, 0.0 percent > Dec 05 15:23:17 node01 bash[1969]: Interval writes: 0 writes, 0 > keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, > 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Interval WAL: 0 writes, 0 syncs, > 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Interval stall: 00:00:0.000 > H:M:S, 0.0 percent > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > level0_numfiles_with_compaction, 0 stop for > pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > memtable_compaction, 0 memtable_slowdown, interval 0 total count > Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > Level [default] ** > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > level0_numfiles_with_compaction, 0 stop for > pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > memtable_compaction, 0 memtable_slowdown, interval 0 total count > Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > Level [default] ** > Dec 05 15:23:17 node01 bash[1969]: debug -45> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -44> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -43> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -42> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: none > Dec 05 15:23:17 node01 bash[1969]: debug -41> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -40> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -39> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -38> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -37> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -36> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -35> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -34> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -33> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -32> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -31> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -30> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -29> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 2 auth: KeyRing::load: > loaded key file /var/lib/ceph/mon/ceph-node01/keyring > Dec 05 15:23:17 node01 bash[1969]: debug -28> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 0 starting mon.node01 > rank 0 at public addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] > at bind addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] mon_data > /var/lib/ceph/mon/ceph-node01 fsid > be4304e4-b0d5-11ec-8c6a-2965d4229f37 > Dec 05 15:23:17 node01 bash[1969]: debug -27> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -26> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -25> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -24> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: none > Dec 05 15:23:17 node01 bash[1969]: debug -23> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -22> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -21> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -20> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -19> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -18> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -17> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -16> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -15> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -14> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -13> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -12> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -11> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 2 auth: KeyRing::load: > loaded key file /var/lib/ceph/mon/ceph-node01/keyring > Dec 05 15:23:17 node01 bash[1969]: debug -10> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 adding auth protocol: > cephx > Dec 05 15:23:17 node01 bash[1969]: debug -9> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 adding auth protocol: > cephx > Dec 05 15:23:17 node01 bash[1969]: debug -8> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 10 log_channel(cluster) > update_config to_monitors: true to_syslog: false syslog_facility: > prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: > 12201) > Dec 05 15:23:17 node01 bash[1969]: debug -7> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 10 log_channel(audit) > update_config to_monitors: true to_syslog: false syslog_facility: > prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: > 12201) > Dec 05 15:23:17 node01 bash[1969]: debug -6> > 2023-12-05T13:23:17.723+0000 7fab0e25bb80 1 mon.node01@-1(???) e0 > preinit fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37 > Dec 05 15:23:17 node01 bash[1969]: debug -5> > 2023-12-05T13:23:17.723+0000 7fab0e25bb80 5 mon.node01(a)-1(???).mds > e0 Unable to load 'last_metadata' > Dec 05 15:23:17 node01 bash[1969]: debug -4> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 3314933000854323200, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug -3> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug -2> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug -1> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug 0> > 2023-12-05T13:23:17.735+0000 7fab0e25bb80 -1 *** Caught signal > (Aborted) ** > Dec 05 15:23:17 node01 bash[1969]: in thread 7fab0e25bb80 > thread_name:ceph-mon > Dec 05 15:23:17 node01 bash[1969]: ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) > Dec 05 15:23:17 node01 bash[1969]: 1: > /lib64/libpthread.so.0(+0x12cf0) [0x7fab0b6c8cf0] > Dec 05 15:23:17 node01 bash[1969]: 2: gsignal() > Dec 05 15:23:17 node01 bash[1969]: 3: abort() > Dec 05 15:23:17 node01 bash[1969]: 4: > /lib64/libstdc++.so.6(+0x9009b) [0x7fab0acda09b] > Dec 05 15:23:17 node01 bash[1969]: 5: > /lib64/libstdc++.so.6(+0x9654c) [0x7fab0ace054c] > Dec 05 15:23:17 node01 bash[1969]: 6: > /lib64/libstdc++.so.6(+0x965a7) [0x7fab0ace05a7] > Dec 05 15:23:17 node01 bash[1969]: 7: > /lib64/libstdc++.so.6(+0x96808) [0x7fab0ace0808] > Dec 05 15:23:17 node01 bash[1969]: 8: > /lib64/libstdc++.so.6(+0x91ff9) [0x7fab0acdbff9] > Dec 05 15:23:17 node01 bash[1969]: 9: > (LogMonitor::log_external_backlog()+0xe70) [0x55be1dc27190] > Dec 05 15:23:17 node01 bash[1969]: 10: > (LogMonitor::update_from_paxos(bool*)+0x54) [0x55be1dc27334] > Dec 05 15:23:17 node01 bash[1969]: 11: > (Monitor::refresh_from_paxos(bool*)+0x104) [0x55be1db9ae74] > Dec 05 15:23:17 node01 bash[1969]: 12: (Monitor::preinit()+0x95d) > [0x55be1dbc93ad] > Dec 05 15:23:17 node01 bash[1969]: 13: main() > Dec 05 15:23:17 node01 bash[1969]: 14: __libc_start_main() > Dec 05 15:23:17 node01 bash[1969]: 15: _start() > Dec 05 15:23:17 node01 bash[1969]: NOTE: a copy of the executable, > or `objdump -rdS <executable>` is needed to interpret this. > Dec 05 15:23:17 node01 bash[1969]: --- logging levels --- > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 none > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 lockdep > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 context > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 crush > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_balancer > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_locker > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_log > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_log_expire > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_migrator > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 buffer > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 timer > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 filer > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 striper > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 objecter > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rados > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_mirror > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_replay > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_pwl > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 journaler > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 objectcacher > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 immutable_obj_cache > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 client > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 osd > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 optracker > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 objclass > Dec 05 15:23:17 node01 bash[1969]: 1/ 3 filestore > Dec 05 15:23:17 node01 bash[1969]: 1/ 3 journal > Dec 05 15:23:17 node01 bash[1969]: 0/ 0 ms > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mon > Dec 05 15:23:17 node01 bash[1969]: 0/10 monc > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 paxos > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 tp > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 auth > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 crypto > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 finisher > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 reserver > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 heartbeatmap > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 perfcounter > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw_sync > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw_datacache > Dec 05 15:23:17 node01 bash[1969]: 1/10 civetweb > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 javaclient > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 asok > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 throttle > Dec 05 15:23:17 node01 bash[1969]: 0/ 0 refs > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 compressor > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 bluestore > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 bluefs > Dec 05 15:23:17 node01 bash[1969]: 1/ 3 bdev > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 kstore > Dec 05 15:23:17 node01 bash[1969]: 4/ 5 rocksdb > Dec 05 15:23:17 node01 bash[1969]: 4/ 5 leveldb > Dec 05 15:23:17 node01 bash[1969]: 4/ 5 memdb > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 fuse > Dec 05 15:23:17 node01 bash[1969]: 2/ 5 mgr > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mgrc > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 dpdk > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 eventtrace > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 prioritycache > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 test > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 cephfs_mirror > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 cephsqlite > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_onode > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_odata > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_omap > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_tm > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_cleaner > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_lba > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_cache > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_journal > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_device > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 alienstore > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mclock > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 ceph_exporter > Dec 05 15:23:17 node01 bash[1969]: -2/-2 (syslog threshold) > Dec 05 15:23:17 node01 bash[1969]: 99/99 (stderr threshold) > Dec 05 15:23:17 node01 bash[1969]: --- pthread ID / name mapping for > recent threads --- > Dec 05 15:23:17 node01 bash[1969]: 7faafe4ad700 / ceph-mon > Dec 05 15:23:17 node01 bash[1969]: 7fab076ba700 / admin_socket > Dec 05 15:23:17 node01 bash[1969]: 7fab0e25bb80 / ceph-mon > Dec 05 15:23:17 node01 bash[1969]: max_recent 10000 > Dec 05 15:23:17 node01 bash[1969]: max_new 10000 > Dec 05 15:23:17 node01 bash[1969]: log_file > Dec 05 15:23:17 node01 bash[1969]: --- end dump of recent events --- > Dec 05 15:23:17 node01 bash[1969]: --- begin dump of recent events --- > Dec 05 15:23:17 node01 bash[1969]: debug -290> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command assert hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -289> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command abort hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -288> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command leak_some_memory hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -287> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perfcounters_dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -286> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command 1 hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -285> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -284> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perfcounters_schema hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -283> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf histogram dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -282> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command 2 hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -281> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf schema hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -280> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf histogram schema hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -279> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command perf reset hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -278> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config show hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -277> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config help hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -276> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config set hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -275> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config unset hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -274> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config get hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -273> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config diff hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -272> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command config diff get hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -271> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command injectargs hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -270> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command log flush hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -269> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command log dump hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -268> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command log reopen hook 0x55be208de8b0 > Dec 05 15:23:17 node01 bash[1969]: debug -267> > 2023-12-05T13:23:17.623+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command dump_mempools hook 0x55be2168a068 > Dec 05 15:23:17 node01 bash[1969]: debug -266> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 set uid:gid to 167:167 > (ceph:ceph) > Dec 05 15:23:17 node01 bash[1969]: debug -265> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable), process > ceph-mon, pid 7 > Dec 05 15:23:17 node01 bash[1969]: debug -264> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 0 pidfile_write: ignore > empty --pid-file > Dec 05 15:23:17 node01 bash[1969]: debug -263> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > init /var/run/ceph/ceph-mon.node01.asok > Dec 05 15:23:17 node01 bash[1969]: debug -262> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > bind_and_listen /var/run/ceph/ceph-mon.node01.asok > Dec 05 15:23:17 node01 bash[1969]: debug -261> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command 0 hook 0x55be2096b030 > Dec 05 15:23:17 node01 bash[1969]: debug -260> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command version hook 0x55be2096b030 > Dec 05 15:23:17 node01 bash[1969]: debug -259> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command git_version hook 0x55be2096b030 > Dec 05 15:23:17 node01 bash[1969]: debug -258> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command help hook 0x55be208de5b0 > Dec 05 15:23:17 node01 bash[1969]: debug -257> > 2023-12-05T13:23:17.639+0000 7fab0e25bb80 5 asok(0x55be20bc0000) > register_command get_command_descriptions hook 0x55be208de5a0 > Dec 05 15:23:17 node01 bash[1969]: debug -256> > 2023-12-05T13:23:17.639+0000 7fab076ba700 5 asok(0x55be20bc0000) > entry start > Dec 05 15:23:17 node01 bash[1969]: debug -255> > 2023-12-05T13:23:17.667+0000 7fab0e25bb80 0 load: jerasure load: lrc > Dec 05 15:23:17 node01 bash[1969]: debug -254> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > level_compaction_dynamic_level_bytes = true > Dec 05 15:23:17 node01 bash[1969]: debug -253> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > write_buffer_size = 33554432 > Dec 05 15:23:17 node01 bash[1969]: debug -252> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > compression = kNoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -251> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > level_compaction_dynamic_level_bytes = true > Dec 05 15:23:17 node01 bash[1969]: debug -250> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > write_buffer_size = 33554432 > Dec 05 15:23:17 node01 bash[1969]: debug -249> > 2023-12-05T13:23:17.671+0000 7fab0e25bb80 1 set rocksdb option > compression = kNoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -248> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 5 rocksdb: > verify_sharding column families from rocksdb: [default] > Dec 05 15:23:17 node01 bash[1969]: debug -247> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: RocksDB > version: 6.15.5 > Dec 05 15:23:17 node01 bash[1969]: debug -246> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Git sha > rocksdb_build_git_sha:@0@ > Dec 05 15:23:17 node01 bash[1969]: debug -245> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compile date > Oct 25 2023 > Dec 05 15:23:17 node01 bash[1969]: debug -244> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB SUMMARY > Dec 05 15:23:17 node01 bash[1969]: debug -243> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: DB Session ID: > 69YYUY1IJK4BA9QQDDK8 > Dec 05 15:23:17 node01 bash[1969]: debug -242> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: CURRENT file: > CURRENT > Dec 05 15:23:17 node01 bash[1969]: debug -241> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: IDENTITY file: > IDENTITY > Dec 05 15:23:17 node01 bash[1969]: debug -240> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: MANIFEST file: > MANIFEST-000226 size: 308 Bytes > Dec 05 15:23:17 node01 bash[1969]: debug -239> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: SST files in > /var/lib/ceph/mon/ceph-node01/store.db dir, Total Num: 3, files: > 000091.sst 000094.sst 000100.sst > Dec 05 15:23:17 node01 bash[1969]: debug -238> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Write Ahead > Log file in /var/lib/ceph/mon/ceph-node01/store.db: 000227.log size: > 0 ; > Dec 05 15:23:17 node01 bash[1969]: debug -237> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.error_if_exists: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -236> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.create_if_missing: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -235> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.paranoid_checks: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -234> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.track_and_verify_wals_in_manifest: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -233> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.env: 0x55be1e3aea80 > Dec 05 15:23:17 node01 bash[1969]: debug -232> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.fs: Posix File System > Dec 05 15:23:17 node01 bash[1969]: debug -231> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.info_log: 0x55be20918360 > Dec 05 15:23:17 node01 bash[1969]: debug -230> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_file_opening_threads: 16 > Dec 05 15:23:17 node01 bash[1969]: debug -229> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.statistics: (nil) > Dec 05 15:23:17 node01 bash[1969]: debug -228> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_fsync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -227> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_log_file_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -226> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_manifest_file_size: 1073741824 > Dec 05 15:23:17 node01 bash[1969]: debug -225> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.log_file_time_to_roll: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -224> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.keep_log_file_num: 1000 > Dec 05 15:23:17 node01 bash[1969]: debug -223> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.recycle_log_file_num: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -222> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_fallocate: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -221> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_mmap_reads: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -220> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_mmap_writes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -219> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_direct_reads: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -218> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_direct_io_for_flush_and_compaction: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -217> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.create_missing_column_families: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -216> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_log_dir: > Dec 05 15:23:17 node01 bash[1969]: debug -215> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_dir: > /var/lib/ceph/mon/ceph-node01/store.db > Dec 05 15:23:17 node01 bash[1969]: debug -214> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_cache_numshardbits: 6 > Dec 05 15:23:17 node01 bash[1969]: debug -213> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.WAL_ttl_seconds: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -212> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.WAL_size_limit_MB: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -211> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_batch_group_size_bytes: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -210> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.manifest_preallocation_size: 4194304 > Dec 05 15:23:17 node01 bash[1969]: debug -209> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.is_fd_close_on_exec: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -208> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.advise_random_on_open: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -207> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_write_buffer_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -206> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_buffer_manager: 0x55be2168c990 > Dec 05 15:23:17 node01 bash[1969]: debug -205> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.access_hint_on_compaction_start: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -204> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.new_table_reader_for_compaction_inputs: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -203> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.random_access_max_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -202> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.use_adaptive_mutex: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -201> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.rate_limiter: (nil) > Dec 05 15:23:17 node01 bash[1969]: debug -200> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.sst_file_manager.rate_bytes_per_sec: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -199> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_recovery_mode: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -198> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_thread_tracking: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -197> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_pipelined_write: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -196> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.unordered_write: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -195> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_concurrent_memtable_write: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -194> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_write_thread_adaptive_yield: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -193> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_thread_max_yield_usec: 100 > Dec 05 15:23:17 node01 bash[1969]: debug -192> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_thread_slow_yield_usec: 3 > Dec 05 15:23:17 node01 bash[1969]: debug -191> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.row_cache: None > Dec 05 15:23:17 node01 bash[1969]: debug -190> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_filter: None > Dec 05 15:23:17 node01 bash[1969]: debug -189> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_flush_during_recovery: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -188> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_ingest_behind: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -187> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.preserve_deletes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -186> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.two_write_queues: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -185> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.manual_wal_flush: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -184> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.atomic_flush: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -183> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_unnecessary_blocking_io: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -182> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.persist_stats_to_disk: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -181> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_dbid_to_manifest: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -180> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.log_readahead_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -179> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.file_checksum_gen_factory: Unknown > Dec 05 15:23:17 node01 bash[1969]: debug -178> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.best_efforts_recovery: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -177> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bgerror_resume_count: 2147483647 > Dec 05 15:23:17 node01 bash[1969]: debug -176> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bgerror_resume_retry_interval: 1000000 > Dec 05 15:23:17 node01 bash[1969]: debug -175> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.allow_data_in_errors: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -174> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.db_host_id: __hostname__ > Dec 05 15:23:17 node01 bash[1969]: debug -173> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_jobs: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -172> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_compactions: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -171> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_subcompactions: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -170> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.avoid_flush_during_shutdown: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -169> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.writable_file_max_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -168> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.delayed_write_rate : 16777216 > Dec 05 15:23:17 node01 bash[1969]: debug -167> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_total_wal_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -166> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.delete_obsolete_files_period_micros: 21600000000 > Dec 05 15:23:17 node01 bash[1969]: debug -165> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_dump_period_sec: 600 > Dec 05 15:23:17 node01 bash[1969]: debug -164> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_persist_period_sec: 600 > Dec 05 15:23:17 node01 bash[1969]: debug -163> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.stats_history_buffer_size: 1048576 > Dec 05 15:23:17 node01 bash[1969]: debug -162> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_open_files: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -161> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -160> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.wal_bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -159> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.strict_bytes_per_sync: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -158> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_readahead_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -157> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_background_flushes: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -156> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Compression > algorithms supported: > Dec 05 15:23:17 node01 bash[1969]: debug -155> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kZSTDNotFinalCompression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -154> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: kZSTD > supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -153> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kXpressCompression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -152> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kLZ4HCCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -151> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kZlibCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -150> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kSnappyCompression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -149> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kLZ4Compression supported: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -148> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > kBZip2Compression supported: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -147> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: Fast CRC32 > supported: Supported on x86 > Dec 05 15:23:17 node01 bash[1969]: debug -146> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4724] Recovering from manifest file: > /var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > Dec 05 15:23:17 node01 bash[1969]: debug -145> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > [db/column_family.cc:595] --------------- Options for column family > [default]: > Dec 05 15:23:17 node01 bash[1969]: debug -144> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.comparator: leveldb.BytewiseComparator > Dec 05 15:23:17 node01 bash[1969]: debug -143> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.merge_operator: > Dec 05 15:23:17 node01 bash[1969]: debug -142> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_filter: None > Dec 05 15:23:17 node01 bash[1969]: debug -141> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_filter_factory: None > Dec 05 15:23:17 node01 bash[1969]: debug -140> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.sst_partitioner_factory: None > Dec 05 15:23:17 node01 bash[1969]: debug -139> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_factory: SkipListFactory > Dec 05 15:23:17 node01 bash[1969]: debug -138> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_factory: BlockBasedTable > Dec 05 15:23:17 node01 bash[1969]: debug -137> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > table_factory options: flush_block_policy_factory: > FlushBlockBySizePolicyFactory (0x55be2096afb0) > Dec 05 15:23:17 node01 bash[1969]: cache_index_and_filter_blocks: 1 > Dec 05 15:23:17 node01 bash[1969]: > cache_index_and_filter_blocks_with_high_priority: 0 > Dec 05 15:23:17 node01 bash[1969]: > pin_l0_filter_and_index_blocks_in_cache: 0 > Dec 05 15:23:17 node01 bash[1969]: pin_top_level_index_and_filter: 1 > Dec 05 15:23:17 node01 bash[1969]: index_type: 0 > Dec 05 15:23:17 node01 bash[1969]: data_block_index_type: 0 > Dec 05 15:23:17 node01 bash[1969]: index_shortening: 1 > Dec 05 15:23:17 node01 bash[1969]: > data_block_hash_table_util_ratio: 0.750000 > Dec 05 15:23:17 node01 bash[1969]: hash_index_allow_collision: 1 > Dec 05 15:23:17 node01 bash[1969]: checksum: 1 > Dec 05 15:23:17 node01 bash[1969]: no_block_cache: 0 > Dec 05 15:23:17 node01 bash[1969]: block_cache: 0x55be2099d090 > Dec 05 15:23:17 node01 bash[1969]: block_cache_name: BinnedLRUCache > Dec 05 15:23:17 node01 bash[1969]: block_cache_options: > Dec 05 15:23:17 node01 bash[1969]: capacity : 536870912 > Dec 05 15:23:17 node01 bash[1969]: num_shard_bits : 4 > Dec 05 15:23:17 node01 bash[1969]: strict_capacity_limit : 0 > Dec 05 15:23:17 node01 bash[1969]: high_pri_pool_ratio: 0.000 > Dec 05 15:23:17 node01 bash[1969]: block_cache_compressed: (nil) > Dec 05 15:23:17 node01 bash[1969]: persistent_cache: (nil) > Dec 05 15:23:17 node01 bash[1969]: block_size: 4096 > Dec 05 15:23:17 node01 bash[1969]: block_size_deviation: 10 > Dec 05 15:23:17 node01 bash[1969]: block_restart_interval: 16 > Dec 05 15:23:17 node01 bash[1969]: index_block_restart_interval: 1 > Dec 05 15:23:17 node01 bash[1969]: metadata_block_size: 4096 > Dec 05 15:23:17 node01 bash[1969]: partition_filters: 0 > Dec 05 15:23:17 node01 bash[1969]: use_delta_encoding: 1 > Dec 05 15:23:17 node01 bash[1969]: filter_policy: > rocksdb.BuiltinBloomFilter > Dec 05 15:23:17 node01 bash[1969]: whole_key_filtering: 1 > Dec 05 15:23:17 node01 bash[1969]: verify_compression: 0 > Dec 05 15:23:17 node01 bash[1969]: read_amp_bytes_per_bit: 0 > Dec 05 15:23:17 node01 bash[1969]: format_version: 4 > Dec 05 15:23:17 node01 bash[1969]: enable_index_compression: 1 > Dec 05 15:23:17 node01 bash[1969]: block_align: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -136> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.write_buffer_size: 33554432 > Dec 05 15:23:17 node01 bash[1969]: debug -135> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_number: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -134> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression: NoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -133> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression: Disabled > Dec 05 15:23:17 node01 bash[1969]: debug -132> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.prefix_extractor: nullptr > Dec 05 15:23:17 node01 bash[1969]: debug -131> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_insert_with_hint_prefix_extractor: nullptr > Dec 05 15:23:17 node01 bash[1969]: debug -130> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.num_levels: 7 > Dec 05 15:23:17 node01 bash[1969]: debug -129> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.min_write_buffer_number_to_merge: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -128> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_number_to_maintain: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -127> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_write_buffer_size_to_maintain: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -126> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.window_bits: -14 > Dec 05 15:23:17 node01 bash[1969]: debug -125> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.level: 32767 > Dec 05 15:23:17 node01 bash[1969]: debug -124> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.strategy: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -123> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.max_dict_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -122> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -121> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.parallel_threads: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -120> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bottommost_compression_opts.enabled: false > Dec 05 15:23:17 node01 bash[1969]: debug -119> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.window_bits: -14 > Dec 05 15:23:17 node01 bash[1969]: debug -118> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.level: 32767 > Dec 05 15:23:17 node01 bash[1969]: debug -117> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.strategy: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -116> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.max_dict_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -115> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -114> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.parallel_threads: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -113> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compression_opts.enabled: false > Dec 05 15:23:17 node01 bash[1969]: debug -112> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_file_num_compaction_trigger: 4 > Dec 05 15:23:17 node01 bash[1969]: debug -111> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_slowdown_writes_trigger: 20 > Dec 05 15:23:17 node01 bash[1969]: debug -110> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level0_stop_writes_trigger: 36 > Dec 05 15:23:17 node01 bash[1969]: debug -109> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.target_file_size_base: 67108864 > Dec 05 15:23:17 node01 bash[1969]: debug -108> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.target_file_size_multiplier: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -107> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_base: 268435456 > Dec 05 15:23:17 node01 bash[1969]: debug -106> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.level_compaction_dynamic_level_bytes: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -105> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier: 10.000000 > Dec 05 15:23:17 node01 bash[1969]: debug -104> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[0]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -103> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[1]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -102> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[2]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -101> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[3]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -100> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[4]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -99> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[5]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -98> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[6]: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -97> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_sequential_skip_in_iterations: 8 > Dec 05 15:23:17 node01 bash[1969]: debug -96> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_compaction_bytes: 1677721600 > Dec 05 15:23:17 node01 bash[1969]: debug -95> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.arena_block_size: 4194304 > Dec 05 15:23:17 node01 bash[1969]: debug -94> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.soft_pending_compaction_bytes_limit: 68719476736 > Dec 05 15:23:17 node01 bash[1969]: debug -93> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.hard_pending_compaction_bytes_limit: 274877906944 > Dec 05 15:23:17 node01 bash[1969]: debug -92> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.rate_limit_delay_max_milliseconds: 100 > Dec 05 15:23:17 node01 bash[1969]: debug -91> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.disable_auto_compactions: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -90> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_style: kCompactionStyleLevel > Dec 05 15:23:17 node01 bash[1969]: debug -89> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_pri: kMinOverlappingRatio > Dec 05 15:23:17 node01 bash[1969]: debug -88> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.size_ratio: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -87> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.min_merge_width: 2 > Dec 05 15:23:17 node01 bash[1969]: debug -86> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.max_merge_width: 4294967295 > Dec 05 15:23:17 node01 bash[1969]: debug -85> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.max_size_amplification_percent: > 200 > Dec 05 15:23:17 node01 bash[1969]: debug -84> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.compression_size_percent: -1 > Dec 05 15:23:17 node01 bash[1969]: debug -83> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_universal.stop_style: > kCompactionStopStyleTotalSize > Dec 05 15:23:17 node01 bash[1969]: debug -82> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_fifo.max_table_files_size: 1073741824 > Dec 05 15:23:17 node01 bash[1969]: debug -81> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.compaction_options_fifo.allow_compaction: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -80> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.table_properties_collectors: > Dec 05 15:23:17 node01 bash[1969]: debug -79> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.inplace_update_support: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -78> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.inplace_update_num_locks: 10000 > Dec 05 15:23:17 node01 bash[1969]: debug -77> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_prefix_bloom_size_ratio: 0.000000 > Dec 05 15:23:17 node01 bash[1969]: debug -76> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_whole_key_filtering: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -75> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.memtable_huge_page_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -74> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.bloom_locality: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -73> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.max_successive_merges: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -72> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.optimize_filters_for_hits: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -71> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.paranoid_file_checks: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -70> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.force_consistency_checks: 1 > Dec 05 15:23:17 node01 bash[1969]: debug -69> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.report_bg_io_stats: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -68> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.ttl: 2592000 > Dec 05 15:23:17 node01 bash[1969]: debug -67> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.periodic_compaction_seconds: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -66> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_blob_files: false > Dec 05 15:23:17 node01 bash[1969]: debug -65> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.min_blob_size: 0 > Dec 05 15:23:17 node01 bash[1969]: debug -64> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_file_size: 268435456 > Dec 05 15:23:17 node01 bash[1969]: debug -63> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_compression_type: NoCompression > Dec 05 15:23:17 node01 bash[1969]: debug -62> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.enable_blob_garbage_collection: false > Dec 05 15:23:17 node01 bash[1969]: debug -61> > 2023-12-05T13:23:17.679+0000 7fab0e25bb80 4 rocksdb: > Options.blob_garbage_collection_age_cutoff: 0.250000 > Dec 05 15:23:17 node01 bash[1969]: debug -60> > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug -59> > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4764] Recovered from manifest > file:/var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000226 > succeeded,manifest_file_number is 226, next_file_number is 228, > last_sequence is 21201, log_number is 223,prev_log_number is > 0,max_column_family is 0,min_log_number_to_keep is 0 > Dec 05 15:23:17 node01 bash[1969]: debug -58> > 2023-12-05T13:23:17.683+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4779] Column family [default] (ID 0), log number > is 223 > Dec 05 15:23:17 node01 bash[1969]: debug -57> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4082] Creating manifest 230 > Dec 05 15:23:17 node01 bash[1969]: debug -56> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug -55> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1701782597693048, "job": 1, "event": > "recovery_started", "wal_files": [227]} > Dec 05 15:23:17 node01 bash[1969]: debug -54> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/db_impl/db_impl_open.cc:845] Recovering log #227 mode 2 > Dec 05 15:23:17 node01 bash[1969]: debug -53> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:4082] Creating manifest 231 > Dec 05 15:23:17 node01 bash[1969]: debug -52> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: > [db/version_set.cc:3457] More existing levels in DB than needed. > max_bytes_for_level_multiplier may not be guaranteed. > Dec 05 15:23:17 node01 bash[1969]: debug -51> > 2023-12-05T13:23:17.687+0000 7fab0e25bb80 4 rocksdb: EVENT_LOG_v1 > {"time_micros": 1701782597693989, "job": 1, "event": > "recovery_finished"} > Dec 05 15:23:17 node01 bash[1969]: debug -50> > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > [file/delete_scheduler.cc:69] Deleted file > /var/lib/ceph/mon/ceph-node01/store.db/000227.log immediately, > rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 > Dec 05 15:23:17 node01 bash[1969]: debug -49> > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: > [db/db_impl/db_impl_open.cc:1700] SstFileManager instance > 0x55be20931180 > Dec 05 15:23:17 node01 bash[1969]: debug -48> > 2023-12-05T13:23:17.691+0000 7fab0e25bb80 4 rocksdb: DB pointer > 0x55be20a06000 > Dec 05 15:23:17 node01 bash[1969]: debug -47> > 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- > Dec 05 15:23:17 node01 bash[1969]: debug -46> > 2023-12-05T13:23:17.695+0000 7faafe4ad700 4 rocksdb: > [db/db_impl/db_impl.cc:903] > Dec 05 15:23:17 node01 bash[1969]: ** DB Stats ** > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Cumulative writes: 0 writes, 0 > keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, > 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Cumulative WAL: 0 writes, 0 > syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Cumulative stall: 00:00:0.000 > H:M:S, 0.0 percent > Dec 05 15:23:17 node01 bash[1969]: Interval writes: 0 writes, 0 > keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, > 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Interval WAL: 0 writes, 0 syncs, > 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s > Dec 05 15:23:17 node01 bash[1969]: Interval stall: 00:00:0.000 > H:M:S, 0.0 percent > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > level0_numfiles_with_compaction, 0 stop for > pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > memtable_compaction, 0 memtable_slowdown, interval 0 total count > Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > Level [default] ** > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Level Files Size Score > Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: L0 2/0 7.21 MB 0.5 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: L6 1/0 7.22 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Sum 3/0 14.43 MB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: Int 0/0 0.00 KB 0.0 > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.00 0.00 0 0.000 0 0 > Dec 05 15:23:17 node01 bash[1969]: ** Compaction Stats [default] ** > Dec 05 15:23:17 node01 bash[1969]: Priority Files Size > Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp > Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) > KeyIn KeyDrop > Dec 05 15:23:17 node01 bash[1969]: > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Dec 05 15:23:17 node01 bash[1969]: Uptime(secs): 0.0 total, 0.0 interval > Dec 05 15:23:17 node01 bash[1969]: Flush(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(GB): cumulative 0.000, > interval 0.000 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Total Files): cumulative > 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(L0 Files): cumulative 0, > interval 0 > Dec 05 15:23:17 node01 bash[1969]: AddFile(Keys): cumulative 0, interval 0 > Dec 05 15:23:17 node01 bash[1969]: Cumulative compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Interval compaction: 0.00 GB > write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds > Dec 05 15:23:17 node01 bash[1969]: Stalls(count): 0 level0_slowdown, > 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 > level0_numfiles_with_compaction, 0 stop for > pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > memtable_compaction, 0 memtable_slowdown, interval 0 total count > Dec 05 15:23:17 node01 bash[1969]: ** File Read Latency Histogram By > Level [default] ** > Dec 05 15:23:17 node01 bash[1969]: debug -45> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -44> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -43> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -42> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding auth protocol: none > Dec 05 15:23:17 node01 bash[1969]: debug -41> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -40> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -39> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -38> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -37> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -36> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -35> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -34> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -33> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -32> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -31> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -30> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167a140) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -29> > 2023-12-05T13:23:17.715+0000 7fab0e25bb80 2 auth: KeyRing::load: > loaded key file /var/lib/ceph/mon/ceph-node01/keyring > Dec 05 15:23:17 node01 bash[1969]: debug -28> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 0 starting mon.node01 > rank 0 at public addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] > at bind addrs [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] mon_data > /var/lib/ceph/mon/ceph-node01 fsid > be4304e4-b0d5-11ec-8c6a-2965d4229f37 > Dec 05 15:23:17 node01 bash[1969]: debug -27> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -26> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -25> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: cephx > Dec 05 15:23:17 node01 bash[1969]: debug -24> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding auth protocol: none > Dec 05 15:23:17 node01 bash[1969]: debug -23> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -22> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -21> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -20> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -19> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -18> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -17> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -16> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -15> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -14> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -13> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: crc > Dec 05 15:23:17 node01 bash[1969]: debug -12> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 > AuthRegistry(0x55be2167aa40) adding con mode: secure > Dec 05 15:23:17 node01 bash[1969]: debug -11> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 2 auth: KeyRing::load: > loaded key file /var/lib/ceph/mon/ceph-node01/keyring > Dec 05 15:23:17 node01 bash[1969]: debug -10> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 adding auth protocol: > cephx > Dec 05 15:23:17 node01 bash[1969]: debug -9> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 5 adding auth protocol: > cephx > Dec 05 15:23:17 node01 bash[1969]: debug -8> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 10 log_channel(cluster) > update_config to_monitors: true to_syslog: false syslog_facility: > prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: > 12201) > Dec 05 15:23:17 node01 bash[1969]: debug -7> > 2023-12-05T13:23:17.719+0000 7fab0e25bb80 10 log_channel(audit) > update_config to_monitors: true to_syslog: false syslog_facility: > prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: > 12201) > Dec 05 15:23:17 node01 bash[1969]: debug -6> > 2023-12-05T13:23:17.723+0000 7fab0e25bb80 1 mon.node01@-1(???) e0 > preinit fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37 > Dec 05 15:23:17 node01 bash[1969]: debug -5> > 2023-12-05T13:23:17.723+0000 7fab0e25bb80 5 mon.node01(a)-1(???).mds > e0 Unable to load 'last_metadata' > Dec 05 15:23:17 node01 bash[1969]: debug -4> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 3314933000854323200, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug -3> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug -2> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug -1> > 2023-12-05T13:23:17.731+0000 7fab0e25bb80 0 mon.node01(a)-1(???).osd > e9898 crush map has features 432629239337189376, adjusting msgr > requires > Dec 05 15:23:17 node01 bash[1969]: debug 0> > 2023-12-05T13:23:17.735+0000 7fab0e25bb80 -1 *** Caught signal > (Aborted) ** > Dec 05 15:23:17 node01 bash[1969]: in thread 7fab0e25bb80 > thread_name:ceph-mon > Dec 05 15:23:17 node01 bash[1969]: ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) > Dec 05 15:23:17 node01 bash[1969]: 1: > /lib64/libpthread.so.0(+0x12cf0) [0x7fab0b6c8cf0] > Dec 05 15:23:17 node01 bash[1969]: 2: gsignal() > Dec 05 15:23:17 node01 bash[1969]: 3: abort() > Dec 05 15:23:17 node01 bash[1969]: 4: > /lib64/libstdc++.so.6(+0x9009b) [0x7fab0acda09b] > Dec 05 15:23:17 node01 bash[1969]: 5: > /lib64/libstdc++.so.6(+0x9654c) [0x7fab0ace054c] > Dec 05 15:23:17 node01 bash[1969]: 6: > /lib64/libstdc++.so.6(+0x965a7) [0x7fab0ace05a7] > Dec 05 15:23:17 node01 bash[1969]: 7: > /lib64/libstdc++.so.6(+0x96808) [0x7fab0ace0808] > Dec 05 15:23:17 node01 bash[1969]: 8: > /lib64/libstdc++.so.6(+0x91ff9) [0x7fab0acdbff9] > Dec 05 15:23:17 node01 bash[1969]: 9: > (LogMonitor::log_external_backlog()+0xe70) [0x55be1dc27190] > Dec 05 15:23:17 node01 bash[1969]: 10: > (LogMonitor::update_from_paxos(bool*)+0x54) [0x55be1dc27334] > Dec 05 15:23:17 node01 bash[1969]: 11: > (Monitor::refresh_from_paxos(bool*)+0x104) [0x55be1db9ae74] > Dec 05 15:23:17 node01 bash[1969]: 12: (Monitor::preinit()+0x95d) > [0x55be1dbc93ad] > Dec 05 15:23:17 node01 bash[1969]: 13: main() > Dec 05 15:23:17 node01 bash[1969]: 14: __libc_start_main() > Dec 05 15:23:17 node01 bash[1969]: 15: _start() > Dec 05 15:23:17 node01 bash[1969]: NOTE: a copy of the executable, > or `objdump -rdS <executable>` is needed to interpret this. > Dec 05 15:23:17 node01 bash[1969]: --- logging levels --- > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 none > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 lockdep > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 context > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 crush > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_balancer > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_locker > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_log > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_log_expire > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mds_migrator > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 buffer > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 timer > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 filer > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 striper > Dec 05 15:23:17 node01 bash[1969]: 0/ 1 objecter > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rados > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_mirror > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_replay > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 rbd_pwl > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 journaler > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 objectcacher > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 immutable_obj_cache > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 client > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 osd > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 optracker > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 objclass > Dec 05 15:23:17 node01 bash[1969]: 1/ 3 filestore > Dec 05 15:23:17 node01 bash[1969]: 1/ 3 journal > Dec 05 15:23:17 node01 bash[1969]: 0/ 0 ms > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mon > Dec 05 15:23:17 node01 bash[1969]: 0/10 monc > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 paxos > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 tp > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 auth > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 crypto > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 finisher > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 reserver > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 heartbeatmap > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 perfcounter > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw_sync > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 rgw_datacache > Dec 05 15:23:17 node01 bash[1969]: 1/10 civetweb > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 javaclient > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 asok > Dec 05 15:23:17 node01 bash[1969]: 1/ 1 throttle > Dec 05 15:23:17 node01 bash[1969]: 0/ 0 refs > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 compressor > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 bluestore > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 bluefs > Dec 05 15:23:17 node01 bash[1969]: 1/ 3 bdev > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 kstore > Dec 05 15:23:17 node01 bash[1969]: 4/ 5 rocksdb > Dec 05 15:23:17 node01 bash[1969]: 4/ 5 leveldb > Dec 05 15:23:17 node01 bash[1969]: 4/ 5 memdb > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 fuse > Dec 05 15:23:17 node01 bash[1969]: 2/ 5 mgr > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mgrc > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 dpdk > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 eventtrace > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 prioritycache > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 test > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 cephfs_mirror > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 cephsqlite > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_onode > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_odata > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_omap > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_tm > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_cleaner > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_lba > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_cache > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_journal > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 seastore_device > Dec 05 15:23:17 node01 bash[1969]: 0/ 5 alienstore > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 mclock > Dec 05 15:23:17 node01 bash[1969]: 1/ 5 ceph_exporter > Dec 05 15:23:17 node01 bash[1969]: -2/-2 (syslog threshold) > Dec 05 15:23:17 node01 bash[1969]: 99/99 (stderr threshold) > Dec 05 15:23:17 node01 bash[1969]: --- pthread ID / name mapping for > recent threads --- > Dec 05 15:23:17 node01 bash[1969]: 7faafe4ad700 / ceph-mon > Dec 05 15:23:17 node01 bash[1969]: 7fab076ba700 / admin_socket > Dec 05 15:23:17 node01 bash[1969]: 7fab0e25bb80 / ceph-mon > Dec 05 15:23:17 node01 bash[1969]: max_recent 10000 > Dec 05 15:23:17 node01 bash[1969]: max_new 10000 > Dec 05 15:23:17 node01 bash[1969]: log_file > /var/lib/ceph/crash/2023-12-05T13:23:17.739483Z_174a5e49-22df-4c2f-be14-4e88cc65b2f2/log > Dec 05 15:23:17 node01 bash[1969]: --- end dump of recent events --- > Dec 05 15:23:17 node01 systemd[1]: > ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37(a)mon.node01.service: Main > process exited, code=exited, status=134/n/a > Dec 05 15:23:18 node01 systemd[1]: > ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37(a)mon.node01.service: Failed > with result 'exit-code'. > Dec 05 15:23:29 node01 systemd[1]: > ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37(a)mon.node01.service: > Scheduled restart job, restart counter is at 1. > Dec 05 15:23:29 node01 systemd[1]: Stopped Ceph mon.node01 for > be4304e4-b0d5-11ec-8c6a-2965d4229f37. > Dec 05 15:23:29 node01 systemd[1]: Started Ceph mon.node01 for > be4304e4-b0d5-11ec-8c6a-2965d4229f37. > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.299+0000 7fd6a1f76b80 0 set uid:gid to 167:167 > (ceph:ceph) > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.299+0000 7fd6a1f76b80 0 ceph version 17.2.7 > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable), process > ceph-mon, pid 7 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.299+0000 7fd6a1f76b80 0 pidfile_write: ignore > empty --pid-file > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 0 load: jerasure load: lrc > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: RocksDB > version: 6.15.5 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Git sha > rocksdb_build_git_sha:@0@ > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Compile date > Oct 25 2023 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: DB SUMMARY > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: DB Session ID: > OQS561SAY9M8P46I122B > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: CURRENT file: > CURRENT > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: IDENTITY file: > IDENTITY > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: MANIFEST file: > MANIFEST-000231 size: 308 Bytes > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: SST files in > /var/lib/ceph/mon/ceph-node01/store.db dir, Total Num: 3, files: > 000091.sst 000094.sst 000100.sst > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Write Ahead > Log file in /var/lib/ceph/mon/ceph-node01/store.db: 000232.log size: > 0 ; > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.error_if_exists: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.create_if_missing: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.paranoid_checks: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.track_and_verify_wals_in_manifest: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.env: 0x562241625a80 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.fs: Posix File System > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.info_log: 0x562243b9a360 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_file_opening_threads: 16 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.statistics: (nil) > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.use_fsync: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_log_file_size: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_manifest_file_size: 1073741824 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.log_file_time_to_roll: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.keep_log_file_num: 1000 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.recycle_log_file_num: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.allow_fallocate: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.allow_mmap_reads: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.allow_mmap_writes: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.use_direct_reads: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.use_direct_io_for_flush_and_compaction: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.create_missing_column_families: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.db_log_dir: > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.wal_dir: > /var/lib/ceph/mon/ceph-node01/store.db > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.table_cache_numshardbits: 6 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.WAL_ttl_seconds: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.WAL_size_limit_MB: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_write_batch_group_size_bytes: 1048576 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.manifest_preallocation_size: 4194304 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.is_fd_close_on_exec: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.advise_random_on_open: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.db_write_buffer_size: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.write_buffer_manager: 0x56224490e990 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.access_hint_on_compaction_start: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.new_table_reader_for_compaction_inputs: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.random_access_max_buffer_size: 1048576 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.use_adaptive_mutex: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.rate_limiter: (nil) > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.sst_file_manager.rate_bytes_per_sec: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.wal_recovery_mode: 2 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.enable_thread_tracking: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.enable_pipelined_write: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.unordered_write: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.allow_concurrent_memtable_write: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.enable_write_thread_adaptive_yield: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.write_thread_max_yield_usec: 100 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.write_thread_slow_yield_usec: 3 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.row_cache: None > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.wal_filter: None > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.avoid_flush_during_recovery: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.allow_ingest_behind: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.preserve_deletes: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.two_write_queues: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.manual_wal_flush: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.atomic_flush: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.avoid_unnecessary_blocking_io: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.persist_stats_to_disk: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.write_dbid_to_manifest: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.log_readahead_size: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.file_checksum_gen_factory: Unknown > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.best_efforts_recovery: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bgerror_resume_count: 2147483647 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bgerror_resume_retry_interval: 1000000 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.allow_data_in_errors: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.db_host_id: __hostname__ > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_background_jobs: 2 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_background_compactions: -1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_subcompactions: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.avoid_flush_during_shutdown: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.writable_file_max_buffer_size: 1048576 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.delayed_write_rate : 16777216 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_total_wal_size: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.delete_obsolete_files_period_micros: 21600000000 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.stats_dump_period_sec: 600 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.stats_persist_period_sec: 600 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.stats_history_buffer_size: 1048576 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_open_files: -1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bytes_per_sync: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.wal_bytes_per_sync: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.strict_bytes_per_sync: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_readahead_size: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_background_flushes: -1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Compression > algorithms supported: > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > kZSTDNotFinalCompression supported: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: kZSTD > supported: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > kXpressCompression supported: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > kLZ4HCCompression supported: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > kZlibCompression supported: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > kSnappyCompression supported: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > kLZ4Compression supported: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > kBZip2Compression supported: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: Fast CRC32 > supported: Supported on x86 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > [db/version_set.cc:4724] Recovering from manifest file: > /var/lib/ceph/mon/ceph-node01/store.db/MANIFEST-000231 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > [db/column_family.cc:595] --------------- Options for column family > [default]: > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.comparator: leveldb.BytewiseComparator > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.merge_operator: > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_filter: None > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_filter_factory: None > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.sst_partitioner_factory: None > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.memtable_factory: SkipListFactory > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.table_factory: BlockBasedTable > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > table_factory options: flush_block_policy_factory: > FlushBlockBySizePolicyFactory (0x562243becfb0) > Dec 05 15:23:29 node01 bash[6375]: cache_index_and_filter_blocks: 1 > Dec 05 15:23:29 node01 bash[6375]: > cache_index_and_filter_blocks_with_high_priority: 0 > Dec 05 15:23:29 node01 bash[6375]: > pin_l0_filter_and_index_blocks_in_cache: 0 > Dec 05 15:23:29 node01 bash[6375]: pin_top_level_index_and_filter: 1 > Dec 05 15:23:29 node01 bash[6375]: index_type: 0 > Dec 05 15:23:29 node01 bash[6375]: data_block_index_type: 0 > Dec 05 15:23:29 node01 bash[6375]: index_shortening: 1 > Dec 05 15:23:29 node01 bash[6375]: > data_block_hash_table_util_ratio: 0.750000 > Dec 05 15:23:29 node01 bash[6375]: hash_index_allow_collision: 1 > Dec 05 15:23:29 node01 bash[6375]: checksum: 1 > Dec 05 15:23:29 node01 bash[6375]: no_block_cache: 0 > Dec 05 15:23:29 node01 bash[6375]: block_cache: 0x562243c1f090 > Dec 05 15:23:29 node01 bash[6375]: block_cache_name: BinnedLRUCache > Dec 05 15:23:29 node01 bash[6375]: block_cache_options: > Dec 05 15:23:29 node01 bash[6375]: capacity : 536870912 > Dec 05 15:23:29 node01 bash[6375]: num_shard_bits : 4 > Dec 05 15:23:29 node01 bash[6375]: strict_capacity_limit : 0 > Dec 05 15:23:29 node01 bash[6375]: high_pri_pool_ratio: 0.000 > Dec 05 15:23:29 node01 bash[6375]: block_cache_compressed: (nil) > Dec 05 15:23:29 node01 bash[6375]: persistent_cache: (nil) > Dec 05 15:23:29 node01 bash[6375]: block_size: 4096 > Dec 05 15:23:29 node01 bash[6375]: block_size_deviation: 10 > Dec 05 15:23:29 node01 bash[6375]: block_restart_interval: 16 > Dec 05 15:23:29 node01 bash[6375]: index_block_restart_interval: 1 > Dec 05 15:23:29 node01 bash[6375]: metadata_block_size: 4096 > Dec 05 15:23:29 node01 bash[6375]: partition_filters: 0 > Dec 05 15:23:29 node01 bash[6375]: use_delta_encoding: 1 > Dec 05 15:23:29 node01 bash[6375]: filter_policy: > rocksdb.BuiltinBloomFilter > Dec 05 15:23:29 node01 bash[6375]: whole_key_filtering: 1 > Dec 05 15:23:29 node01 bash[6375]: verify_compression: 0 > Dec 05 15:23:29 node01 bash[6375]: read_amp_bytes_per_bit: 0 > Dec 05 15:23:29 node01 bash[6375]: format_version: 4 > Dec 05 15:23:29 node01 bash[6375]: enable_index_compression: 1 > Dec 05 15:23:29 node01 bash[6375]: block_align: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.write_buffer_size: 33554432 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_write_buffer_number: 2 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression: NoCompression > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression: Disabled > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.prefix_extractor: nullptr > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.memtable_insert_with_hint_prefix_extractor: nullptr > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.num_levels: 7 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.min_write_buffer_number_to_merge: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_write_buffer_number_to_maintain: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_write_buffer_size_to_maintain: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression_opts.window_bits: -14 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression_opts.level: 32767 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression_opts.strategy: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression_opts.max_dict_bytes: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression_opts.parallel_threads: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bottommost_compression_opts.enabled: false > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression_opts.window_bits: -14 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression_opts.level: 32767 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression_opts.strategy: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression_opts.max_dict_bytes: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression_opts.zstd_max_train_bytes: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression_opts.parallel_threads: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compression_opts.enabled: false > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.level0_file_num_compaction_trigger: 4 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.level0_slowdown_writes_trigger: 20 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.level0_stop_writes_trigger: 36 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.target_file_size_base: 67108864 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.target_file_size_multiplier: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_base: 268435456 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.level_compaction_dynamic_level_bytes: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier: 10.000000 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[0]: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[1]: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[2]: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[3]: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[4]: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[5]: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_bytes_for_level_multiplier_addtl[6]: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_sequential_skip_in_iterations: 8 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_compaction_bytes: 1677721600 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.arena_block_size: 4194304 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.soft_pending_compaction_bytes_limit: 68719476736 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.hard_pending_compaction_bytes_limit: 274877906944 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.rate_limit_delay_max_milliseconds: 100 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.disable_auto_compactions: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_style: kCompactionStyleLevel > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_pri: kMinOverlappingRatio > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_universal.size_ratio: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_universal.min_merge_width: 2 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_universal.max_merge_width: 4294967295 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_universal.max_size_amplification_percent: > 200 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_universal.compression_size_percent: -1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_universal.stop_style: > kCompactionStopStyleTotalSize > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_fifo.max_table_files_size: 1073741824 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.compaction_options_fifo.allow_compaction: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.table_properties_collectors: > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.inplace_update_support: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.inplace_update_num_locks: 10000 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.memtable_prefix_bloom_size_ratio: 0.000000 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.memtable_whole_key_filtering: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.memtable_huge_page_size: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.bloom_locality: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.max_successive_merges: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.optimize_filters_for_hits: 0 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.paranoid_file_checks: Dec 05 15:23:29 node01 bash[6375]: > debug 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.force_consistency_checks: 1 > Dec 05 15:23:29 node01 bash[6375]: debug > 2023-12-05T13:23:29.303+0000 7fd6a1f76b80 4 rocksdb: > Options.report_bg_io_stats: 0 > Dec 05 15:23:29 > [Text gekürzt]

3 6

Space reclaim doesn't happening in nautilus RBD pool
by Szabo, Istvan (Agoda) 06 Dec '23

06 Dec '23

Hi, Is there any config on Ceph that block/not perform space reclaim? I test on one pool which has only one image 1.8 TiB in used. rbd $p du im/root warning: fast-diff map is not enabled for root. operation may be slow. NAME PROVISIONED USED root 2.2 TiB 1.8 TiB I already removed all snaphots and now pool has only one image alone. I run both fstrim over the filesystem (XFS) and try rbd sparsify im/root (don't know what it is exactly but it mentions to reclaim something) It still shows the pool used 6.9 TiB which totally not make sense right? It should be up to 3.6 (1.8 * 2) according to its replica? POOLS: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR im 19 32 3.5 TiB 918.34k 6.9 TiB 4.80 69 TiB N/A 10 TiB 918.34k 0 B 0 B I think now some of others pool have this issue too, we do clean up a lot but seems space not reclaimed. I estimate more than 50 TiB should be able to reclaim, actual usage of this cluster much less than current reported number. Thank you for your help. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 4