November 2023 - ceph-users

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Problem while upgrade 17.2.6 to 17.2.7
by Jean-Marc FONTANA 21 Feb '25

21 Feb '25

Hello everyone, We operate two clusters that we installed with ceph-deploy in Nautilus version on Debian 10. We use them for external S3 storage (owncloud) and rbd disk images.We had them upgraded to Octopus and Pacific versions on Debian 11 and recently converted them to cephadm and upgraded to Quincy (17.2.6). As we now have the orchestrator, we tried updating to 17.2.7 using the command# ceph orch upgrade start --image quay.io/ceph/ceph:v17.2.7 Everything went well, both clusters work perfectly for our use, except that the Rados gateway configuration is no longer accessible from the dashboard with the following error messageError connecting to Object Gateway: RGW REST API failed request with status code 404. We tried a few solutions found on the internet (reset rgw credentials, restart rgw adnd mgr, reenable dashboard, ...), unsuccessfully. Does somebody have an idea ? Best regards, Jean-Marc Fontana

5 9

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

multifs and snapshots
by Dmitry Melekhov 12 Nov '24

12 Nov '24

Hello! There is note here https://docs.ceph.com/en/reef/dev/cephfs-snapshots/ about multiply filesystems and snapshots: If each FS gets its own pool things probably work, but this isn’t tested and may not be true. Is it still untested ? Thank you!

3 4

Help needed to configure erasure coding LRC plugin
by Michel Jouvin 16 Oct '24

16 Oct '24

Hi, As discussed in another thread (Crushmap rule for multi-datacenter erasure coding), I'm trying to create an EC pool spanning 3 datacenters (datacenters are present in the crushmap), with the objective to be resilient to 1 DC down, at least keeping the readonly access to the pool and if possible the read-write access, and have a storage efficiency better than 3 replica (let say a storage overhead <= 2). In the discussion, somebody mentioned LRC plugin as a possible jerasure alternative to implement this without tweaking the crushmap rule to implement the 2-step OSD allocation. I looked at the documentation (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/) but I have some questions if someone has experience/expertise with this LRC plugin. I tried to create a rule for using 5 OSDs per datacenter (15 in total), with 3 (9 in total) being data chunks and others being coding chunks. For this, based of my understanding of examples, I used k=9, m=3, l=4. Is it right? Is this configuration equivalent, in terms of redundancy, to a jerasure configuration with k=9, m=6? The resulting rule, which looks correct to me, is: -------- { "rule_id": 6, "rule_name": "test_lrc_2", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 15, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -4, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 5, "type": "host" }, { "op": "emit" } ] } ------------ Unfortunately, it doesn't work as expected: a pool created with this rule ends up with its pages active+undersize, which is unexpected for me. Looking at 'ceph health detail` output, I see for each page something like: pg 52.14 is stuck undersized for 27m, current state active+undersized, last acting [90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647] For each PG, there is 3 '2147483647' entries and I guess it is the reason of the problem. What are these entries about? Clearly it is not OSD entries... Looks like a negative number, -1, which in terms of crushmap ID is the crushmap root (named "default" in our configuration). Any trivial mistake I would have made? Thanks in advance for any help or for sharing any successful configuration? Best regards, Michel

4 24

Not all Bucket Shards being used
by Christian Kugler 01 Oct '24

01 Oct '24

Hi, I have trouble with large OMAP files in a cluster in the RGW index pool. Some background information about the cluster: There is CephFS and RBD usage on the main cluster but for this issue I think only S3 is interesting. There is one realm, one zonegroup with two zones which have a bidirectional sync set up. Since this does not allow for autoresharding we have to do it by hand in this cluster – looking forward to Reef! From the logs: cluster 2023-07-17T22:59:03.018722+0000 osd.75 (osd.75) 623978 : cluster [WRN] Large omap object found. Object: 34:bcec3016:::.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5:head PG: 34.680c373d (34.5) Key count: 962091 Size (bytes): 277963182 The offending bucket looks like this: # radosgw-admin bucket stats \ | jq '.[] | select(.marker =="3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9") |"\(.num_shards) \(.usage["rgw.main"].num_objects)"' -r 131 9463833 Last week the number of objects was about 12 million. Which is why I reshareded the offending bucket twice, I think. Once to 129 and the second time to 131 because I wanted some leeway (or lieway? scnr, Sage). Unfortunately, even after a week the objects were still to big (the log line above is quite recent), so I looked into it again. # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.0 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.1 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.2 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.3 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.4 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.6 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.7 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.8 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.9 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.10 # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V \ |xargs -IOMAP sh -c \ 'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l' 1013854 1011007 1012287 1011232 1013565 998262 1012777 1012713 1012230 1010690 997111 Apparently, only 11 shards are in use. This would explain why the "Key usage" (from the log line) is about ten times higher than I would expect. How can I deal with this issue? One thing I could try to fix this would be to reshard to a lower number, but I am not sure if there are any risks associated with "downsharding". After that I could reshard to something like 97. Or I could directly "downshard" to 97. Also, the second zone has a similar problem, but as the error messsage lets me know, this would be a bad idea. Will it just take more time until the sharding is transferred to the seconds zone? Best, Christian Kugler

2 5

Fixing BlueFS spillover (pacific 16.2.14)
by Chris Dunlop 09 Jul '24

09 Jul '24

Hi, tl;dr why are my osds still spilling? I've recently upgraded to 16.2.14 from 16.2.9 and started receiving bluefs spillover warnings (due to the "fix spillover alert" per the 16.2.14 release notes). E.g. from 'ceph health detail', the warning on one of these (there are a few): osd.76 spilled over 128 KiB metadata from 'db' device (56 GiB used of 60 GiB) to slow device This is a 15T HDD with only a 60G SSD for the db so it's not surprising it spilled as it's way below the recommendation for rbd usage at db size 1-2% of the storage size. There was some spare space on the db ssd so I increased the size of the db LV up over 400G and did an bluefs-bdev-expand. However, days later, I'm still getting the spillover warning for that osd, including after running a manual compact: # ceph tell osd.76 compact See attached perf-dump-76 for the perf dump output: # cephadm enter --name 'osd.76' ceph daemon 'osd.76' perf dump" | jq -r '.bluefs' In particular, if my understanding is correct, that's telling me the db available size is 487G (i.e. the LV expand worked), of which it's using 59G, and there's 128K spilled to the slow device: "db_total_bytes": 512309059584, # 487G "db_used_bytes": 63470305280, # 59G "slow_used_bytes": 131072, # 128K A "bluefs stats" also says the db is using 128K of slow storage (although perhaps it's getting the info from the same place as the perf dump?): # ceph tell osd.76 bluefs stats 1 : device size 0x7747ffe000 : using 0xea6200000(59 GiB) 2 : device size 0xe8d7fc00000 : using 0x6554d689000(6.3 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 8.8 MiB 1 WAL 0 B 2.5 GiB 0 B 0 B 0 B 751 MiB 8 DB 0 B 56 GiB 128 KiB 0 B 0 B 50 GiB 842 SLOW 0 B 0 B 0 B 0 B 0 B 0 B 0 TOTAL 0 B 58 GiB 128 KiB 0 B 0 B 0 B 850 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 3.9 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 71 GiB 282 MiB 0 B 0 B 62 GiB SLOW 0 B 0 B 0 B 0 B 0 B 0 B TOTAL 0 B 74 GiB 282 MiB 0 B 0 B 0 B >> SIZE << 0 B 453 GiB 14 TiB I had a look at the "DUMPING STATS" output in the logs bug I don't know how to interpret it. I did try calculating the total of the sizes on the "Sum" lines but that comes to 100G so I don't know what that all means. See attached log-stats-76. I also tried "ceph-kvstore-tool bluestore-kv ... stats": $ { cephadm unit --fsid $clusterid --name osd.76 stop cephadm shell --fsid $clusterid --name osd.76 -- ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-76 stats cephadm unit --fsid $clusterid --name osd.76 start } Output attached as bluestore-kv-stats-76. I can't see anything interesting in there, although again I don't really know how to interpret it. So... why is this osd db still spilling onto slow storage, and how do I fix things so it's no longer using the slow storage? And a bonus issue... on another osd that hasn't yet been resized (i.e. again with a grossly undersized 60G db on SSD with a 15T HDD) I'm also getting a spillover warning. The "bluefs stats" seems to be saying the db is NOT currently spilling (i.e. "0 B" the DB/SLOW position in the matrix), but there's "something" currently using 59G on the slow device: $ ceph tell osd.85 bluefs stats 1 : device size 0xeffffe000 : using 0x3a3900000(15 GiB) 2 : device size 0xe8d7fc00000 : using 0x7aea7434000(7.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 7.4 MiB 1 WAL 0 B 564 MiB 0 B 0 B 0 B 132 MiB 2 DB 0 B 11 GiB 0 B 0 B 0 B 8.1 GiB 177 SLOW 0 B 3.0 GiB 59 GiB 0 B 0 B 56 GiB 898 TOTAL 0 B 13 GiB 59 GiB 0 B 0 B 0 B 1072 MAXIMUMS: LOG 0 B 24 MiB 0 B 0 B 0 B 20 MiB WAL 0 B 2.8 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 22 GiB 448 KiB 0 B 0 B 18 GiB SLOW 0 B 3.3 GiB 62 GiB 0 B 0 B 62 GiB TOTAL 0 B 27 GiB 62 GiB 0 B 0 B 0 B >> SIZE << 0 B 57 GiB 14 TiB Is there anywhere that describes how to interpret this output, and specifically, what stuff is going into the SLOW row? Seemingly there's 898 "files" there, but not LOG, WAL or DB files - so what are they? Cheers, Chris

4 7

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

Re: MDS crashes to damaged metadata
by Patrick Donnelly 10 Jun '24

10 Jun '24

On Thu, Dec 15, 2022 at 9:32 AM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): > > 2022-12-08T13:05:48.919+0100 7f440afec700 -1 /build/ceph-16.2.10/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f440afec700 time 2022-12-08T13:05:48.921223+0100 > /build/ceph-16.2.10/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE) > > 12 Minutes lates the unlink_local error crashes appeared again. This time with a new file. During debugging we noticed a MTU mismatch between MDS (1500) and client (9000) with cephfs kernel mount. The client is also creating the snapshots via mkdir in the .snap directory. > > We disabled snapshot creation for now, but really need this feature. I uploaded the mds logs of the first crash along with the information above to https://tracker.ceph.com/issues/38452 > > I would greatly appreciate it, if you could answer me the following question: > > Is the Bug related to our MTU Mismatch? We fixed the MTU Issue going back to 1500 on all nodes in the ceph public network on the weekend also. I doubt it. > If you need a debug level 20 log of the ScatterLock for further analysis, i could schedule snapshots at the end of our workdays and increase the debug level 5 Minutes arround snap shot creation. This would be very helpful! -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 4

16.2.14: [progress WARNING root] complete: ev {UUID} does not exist
by Zakhar Kirpichenko 03 May '24

03 May '24

Hi, Mgr of my cluster logs this every few seconds: [progress WARNING root] complete: ev 7de5bb74-790b-4fda-8838-e4af4af18c62 does not exist [progress WARNING root] complete: ev fff93fce-b630-4141-81ee-19e7a3e61483 does not exist [progress WARNING root] complete: ev a02f6966-5b9f-49e8-89c4-b4fb8e6f4423 does not exist [progress WARNING root] complete: ev 8d318560-ff1a-477f-9386-43f6b51080bf does not exist [progress WARNING root] complete: ev ff3740a9-6434-470a-808f-a2762fb542a0 does not exist [progress WARNING root] complete: ev 7d0589f1-545e-4970-867b-8482ce48d7f0 does not exist [progress WARNING root] complete: ev 78d57e43-5be5-43f0-8b1a-cdc60e410892 does not exist I would appreciate an advice on what these warnings mean and how they can be resolved. Best regards, Zakhar

4 7

Re: Recoveries without any misplaced objects?
by Hector Martin 24 Apr '24

24 Apr '24

On 29/05/2023 20.55, Anthony D'Atri wrote: > Check the uptime for the OSDs in question I restarted all my OSDs within the past 10 days or so. Maybe OSD restarts are somehow breaking these stats? > >> On May 29, 2023, at 6:44 AM, Hector Martin <marcan(a)marcan.st> wrote: >> >> Hi, >> >> I'm watching a cluster finish a bunch of backfilling, and I noticed that >> quite often PGs end up with zero misplaced objects, even though they are >> still backfilling. >> >> Right now the cluster is down to 6 backfilling PGs: >> >> data: >> volumes: 1/1 healthy >> pools: 6 pools, 268 pgs >> objects: 18.79M objects, 29 TiB >> usage: 49 TiB used, 25 TiB / 75 TiB avail >> pgs: 262 active+clean >> 6 active+remapped+backfilling >> >> But there are no misplaced objects, and the misplaced column in `ceph pg >> dump` is zero for all PGs. >> >> If I do a `ceph pg dump_json`, I can see `num_objects_recovered` >> increasing for these PGs... but the misplaced count is still 0. >> >> Is there something else that would cause recoveries/backfills other than >> misplaced objects? Or perhaps there is a bug somewhere causing the >> misplaced object count to be misreported as 0 sometimes? >> >> # ceph -v >> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy >> (stable) >> >> - Hector >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > - Hector

3 3

Status of IPv4 / IPv6 dual stack?
by Robert Sander 23 Apr '24

23 Apr '24

Hi, as the documentation sends mixed signals in https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#ipv… "Note Binding to IPv4 is enabled by default, so if you just add the option to bind to IPv6 you’ll actually put yourself into dual stack mode." and https://docs.ceph.com/en/latest/rados/configuration/msgr2/#address-formats "Note The ability to bind to multiple ports has paved the way for dual-stack IPv4 and IPv6 support. That said, dual-stack operation is not yet supported as of Quincy v17.2.0." just the quick questions: Is a dual stacked networking with IPv4 and IPv6 now supported or not? From which version on is it considered stable? Are OSDs now able to register themselves with two IP addresses in the cluster map? MONs too? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

6 6

Ceph 16.2.x mon compactions, disk writes
by Zakhar Kirpichenko 17 Apr '24

17 Apr '24

Hi, Monitors in our 16.2.14 cluster appear to quite often run "manual compaction" tasks: debug 2023-10-09T09:30:53.888+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696843853892760, "job": 64225, "event": "flush_started", "num_memtables": 1, "num_entries": 715, "num_deletes": 251, "total_data_size": 3870352, "memory_usage": 3886744, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:30:53.904+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:30:53.910204) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:30:53.911004) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:32:08.956+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696843928961390, "job": 64228, "event": "flush_started", "num_memtables": 1, "num_entries": 1580, "num_deletes": 502, "total_data_size": 8404605, "memory_usage": 8465840, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:32:08.972+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:32:08.977739) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'logm .. 'logm; will stop at (end) debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:32:08.978512) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'logm .. 'logm; will stop at (end) debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.028+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696844009033151, "job": 64231, "event": "flush_started", "num_memtables": 1, "num_entries": 1430, "num_deletes": 251, "total_data_size": 8975535, "memory_usage": 9035920, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:33:29.044+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:33:29.049585) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:33:29.050355) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'paxos .. 'paxos; will stop at (end) I have removed a lot of interim log messages to save space. During each compaction the monitor process writes approximately 500-600 MB of data to disk over a short period of time. These writes add up to tens of gigabytes per hour and hundreds of gigabytes per day. Monitor rocksdb and compaction options are default: "mon_compact_on_bootstrap": "false", "mon_compact_on_start": "false", "mon_compact_on_trim": "true", "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", Is this expected behavior? Is this something I can adjust in order to extend the system storage life? Best regards, Zakhar

5 32

Setting S3 bucket policies with multi-tenants
by Thomas Bennett 15 Apr '24

15 Apr '24

Hi, I'm running Ceph Quincy (17.2.6) with a rados-gateway. I have muti tenants, for example: - Tenant1$manager - Tenant1$readwrite I would like to set a policy on a bucket (backups for example) owned by *Tenant1$manager* to allow *Tenant1$readwrite* access to that bucket. I can't find any documentation that discusses this scenario. Does anyone know how to specify the Principle and Resource section of a policy.json file? Or any other configuration that I might be missing? I've tried some variations on Principal and Resource including and excluding tenant information, but not no luck yet. For example: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": {"AWS": ["arn:aws:iam:::user/*Tenant1$readwrite*"]}, "Action": ["s3:ListBucket","s3:GetObject", ,"s3:PutObject"], "Resource": [ "arn:aws:s3:::*Tenant1/backups*" ] }] } I'm using s3cmd for testing, so: s3cmd --config s3cfg.manager setpolicy policy.json s3://backups/ Returns: s3://backups/: Policy updated And then testing: s3cmd --config s3cfg.readwrite ls s3://backups/ ERROR: Access to bucket 'backups' was denied ERROR: S3 error: 403 (AccessDenied) Thanks, Tom

4 4

A couple OSDs not starting after host reboot
by Alison Peisker 11 Apr '24

11 Apr '24

Hi all, We rebooted all the nodes in our 17.2.5 cluster after performing kernel updates, but 2 of the OSDs on different nodes are not coming back up. This is a production cluster using cephadm. The error message from the OSD log is ceph-osd[87340]: ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-665: (2) No such file or directory The error message from ceph-volume is 2023-08-23T16:12:43.452-0500 7f0cad968600 2 bluestore(/dev/mapper/ceph--febad5a5--ba44--41aa--a39e--b9897f757752-osd--block--87e548f4--b9b5--4ed8--aca8--de703a341a50) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input We tried restarting the daemons and rebooting the node again, but still see the same error. Has anyone experienced this issue before? How do we fix this? Thanks, Alison

9 13

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

Why you might want packages not containers for Ceph deployments
by Matthew Vernon 25 Mar '24

25 Mar '24

Hi, In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0]. I'd like to suggest, briefly, that: * containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes. For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice. Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages): * you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice) We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part. Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems! I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder". I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :) Regards, and apologies for the wall of text, Matthew [0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

26 65

Upgrading from Pacific to Quincy fails with "Unexpected error"
by Reza Bakhshayeshi 13 Mar '24

13 Mar '24

Hi all, I have a problem regarding upgrading Ceph cluster from Pacific to Quincy version with cephadm. I have successfully upgraded the cluster to the latest Pacific (16.2.11). But when I run the following command to upgrade the cluster to 17.2.5, After upgrading 3/4 mgrs, the upgrade process stops with "Unexpected error". (everything is on a private network) ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v17.2.5 I also tried the 17.2.4 version. cephadm fails to check the hosts' status and marks them as offline: cephadm 2023-04-06T10:19:59.998510+0000 mgr.host9.arhpnd (mgr.4516356) 5782 : cephadm [DBG] host host4 (x.x.x.x) failed check cephadm 2023-04-06T10:19:59.998553+0000 mgr.host9.arhpnd (mgr.4516356) 5783 : cephadm [DBG] Host "host4" marked as offline. Skipping daemon refresh cephadm 2023-04-06T10:19:59.998581+0000 mgr.host9.arhpnd (mgr.4516356) 5784 : cephadm [DBG] Host "host4" marked as offline. Skipping gather facts refresh cephadm 2023-04-06T10:19:59.998609+0000 mgr.host9.arhpnd (mgr.4516356) 5785 : cephadm [DBG] Host "host4" marked as offline. Skipping network refresh cephadm 2023-04-06T10:19:59.998633+0000 mgr.host9.arhpnd (mgr.4516356) 5786 : cephadm [DBG] Host "host4" marked as offline. Skipping device refresh cephadm 2023-04-06T10:19:59.998659+0000 mgr.host9.arhpnd (mgr.4516356) 5787 : cephadm [DBG] Host "host4" marked as offline. Skipping osdspec preview refresh cephadm 2023-04-06T10:19:59.998682+0000 mgr.host9.arhpnd (mgr.4516356) 5788 : cephadm [DBG] Host "host4" marked as offline. Skipping autotune cluster 2023-04-06T10:20:00.000151+0000 mon.host8 (mon.0) 158587 : cluster [ERR] Health detail: HEALTH_ERR 9 hosts fail cephadm check; Upgrade: failed due to an unexpected exception cluster 2023-04-06T10:20:00.000191+0000 mon.host8 (mon.0) 158588 : cluster [ERR] [WRN] CEPHADM_HOST_CHECK_FAILED: 9 hosts fail cephadm check cluster 2023-04-06T10:20:00.000202+0000 mon.host8 (mon.0) 158589 : cluster [ERR] host host7 (x.x.x.x) failed check: Unable to reach remote host host7. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000213+0000 mon.host8 (mon.0) 158590 : cluster [ERR] host host2 (x.x.x.x) failed check: Unable to reach remote host host2. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000220+0000 mon.host8 (mon.0) 158591 : cluster [ERR] host host8 (x.x.x.x) failed check: Unable to reach remote host host8. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000228+0000 mon.host8 (mon.0) 158592 : cluster [ERR] host host4 (x.x.x.x) failed check: Unable to reach remote host host4. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000240+0000 mon.host8 (mon.0) 158593 : cluster [ERR] host host3 (x.x.x.x) failed check: Unable to reach remote host host3. Process exited with non-zero exit status 3 and here are some outputs of the commands: [root@host8 ~]# ceph -s cluster: id: xxx health: HEALTH_ERR 9 hosts fail cephadm check Upgrade: failed due to an unexpected exception services: mon: 5 daemons, quorum host8,host1,host7,host2,host9 (age 2w) mgr: host9.arhpnd(active, since 105m), standbys: host8.jowfih, host1.warjsr, host2.qyavjj mds: 1/1 daemons up, 3 standby osd: 37 osds: 37 up (since 8h), 37 in (since 3w) data: io: client: progress: Upgrade to 17.2.5 (0s) [............................] [root@host8 ~]# ceph orch upgrade status { "target_image": "my-private-repo/quay-io/ceph/ceph@sha256 :34c763383e3323c6bb35f3f2229af9f466518d9db926111277f5e27ed543c427", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "3/59 daemons upgraded", "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception", "is_paused": true } [root@host8 ~]# ceph cephadm check-host host7 check-host failed: Host 'host7' not found. Use 'ceph orch host ls' to see all managed hosts. [root@host8 ~]# ceph versions { "mon": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 1, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 }, "osd": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 37 }, "mds": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 4 }, "overall": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 47, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 } } The strange thing is I can rollback the cluster status by failing to not-upgraded mgr like this: ceph mgr fail ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v16.2.11 Would you happen to have any idea about this? Best regards, Reza

4 9

easy way to find out the number of allocated objects for a RBD image
by Tony Liu 29 Jan '24

29 Jan '24

Hi, Other than get all objects of the pool and filter by image ID, is there any easier way to get the number of allocated objects for a RBD image? What I really want to know is the actual usage of an image. An allocated object could be used partially, but that's fine, no need to be 100% accurate. To get the object count and times object size, that should be sufficient. "rbd export" exports actual used data, but to get the actual usage by exporting the image seems too much. This brings up another question, is there any way to know the export size before running it? Thanks! Tony

4 4

Re: Ceph OSD reported Slow operations
by V A Prabha 28 Jan '24

28 Jan '24

Hi Eugen Please find the details below root@meghdootctr1:/var/log/ceph# ceph -s cluster: id: c59da971-57d1-43bd-b2b7-865d392412a5 health: HEALTH_WARN nodeep-scrub flag(s) set 544 pgs not deep-scrubbed in time services: mon: 3 daemons, quorum meghdootctr1,meghdootctr2,meghdootctr3 (age 5d) mgr: meghdootctr1(active, since 5d), standbys: meghdootctr2, meghdootctr3 mds: 3 up:standby osd: 36 osds: 36 up (since 34h), 36 in (since 34h) flags nodeep-scrub data: pools: 2 pools, 544 pgs objects: 10.14M objects, 39 TiB usage: 116 TiB used, 63 TiB / 179 TiB avail pgs: 544 active+clean io: client: 24 MiB/s rd, 16 MiB/s wr, 2.02k op/s rd, 907 op/s wr Ceph Versions: root@meghdootctr1:/var/log/ceph# ceph --version ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) Ceph df -h https://pastebin.com/1ffucyJg Ceph OSD performance dump https://pastebin.com/1R6YQksE Ceph tell osd.XX bench (Out of 36 osds only 8 OSDs give High IOPS value of 250 +. Out of that 4 OSDs are from HP 3PAR and 4 OSDS from DELL EMC. We are using only 4 OSDs from HP3 par and it is working fine without any latency and iops issues from the beginning but the remaining 32 OSDs are from DELL EMC in which 4 OSDs are much better than the remaining 28 OSDs) https://pastebin.com/CixaQmBi Please help me to identify if the issue is with the DELL EMC Storage, Ceph configuration parameter tuning or the Overload in the cloud setup On November 1, 2023 at 9:48 PM Eugen Block <eblock(a)nde.ag> wrote: > Hi, > > for starters please add more cluster details like 'ceph status', 'ceph > versions', 'ceph osd df tree'. Increasing the to 10G was the right > thing to do, you don't get far with 1G with real cluster load. How are > the OSDs configured (HDD only, SSD only or HDD with rocksdb on SSD)? > How is the disk utilization? > > Regards, > Eugen > > Zitat von prabhav(a)cdac.in: > > > In a production setup of 36 OSDs( SAS disks) totalling 180 TB > > allocated to a single Ceph Cluster with 3 monitors and 3 managers. > > There were 830 volumes and VMs created in Openstack with Ceph as a > > backend. On Sep 21, users reported slowness in accessing the VMs. > > Analysing the logs lead us to problem with SAS , Network congestion > > and Ceph configuration( as all default values were used). We updated > > the Network from 1Gbps to 10Gbps for public and cluster networking. > > There was no change. > > The ceph benchmark performance showed that 28 OSDs out of 36 OSDs > > reported very low IOPS of 30 to 50 while the remaining showed 300+ > > IOPS. > > We gradually started reducing the load on the ceph cluster and now > > the volumes count is 650. Now the slow operations has gradually > > reduced but I am aware that this is not the solution. > > Ceph configuration is updated with increasing the > > osd_journal_size to 10 GB, > > osd_max_backfills = 1 > > osd_recovery_max_active = 1 > > osd_recovery_op_priority = 1 > > bluestore_cache_trim_max_skip_pinned=10000 > > > > After one month, now we faced another issue with Mgr daemon stopped > > in all 3 quorums and 16 OSDs went down. From the > > ceph-mon,ceph-mgr.log could not get the reason. Please guide me as > > its a production setup > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294 ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------

4 15

Ceph 16.2.14: ceph-mgr getting oom-killed
by Zakhar Kirpichenko 25 Jan '24

25 Jan '24

Hi, I'm facing a rather new issue with our Ceph cluster: from time to time ceph-mgr on one of the two mgr nodes gets oom-killed after consuming over 100 GB RAM: [Nov21 15:02] tp_osd_tp invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 [ +0.000010] oom_kill_process.cold+0xb/0x10 [ +0.000002] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ +0.000008] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=504d37b566d9fd442d45904a00584b4f61c93c5d49dc59eb1c948b3d1c096907,mems_allowed=0-1,global_oom,task_memcg=/docker/3826be8f9115479117ddb8b721ca57585b2bdd58a27c7ed7b38e8d83eb795957,task=ceph-mgr,pid=3941610,uid=167 [ +0.000697] Out of memory: Killed process 3941610 (ceph-mgr) total-vm:146986656kB, anon-rss:125340436kB, file-rss:0kB, shmem-rss:0kB, UID:167 pgtables:260356kB oom_score_adj:0 [ +6.509769] oom_reaper: reaped process 3941610 (ceph-mgr), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB The cluster is stable and operating normally, there's nothing unusual going on before, during or after the kill, thus it's unclear what causes the mgr to balloon, use all RAM and get killed. Systemd logs aren't very helpful: they just show normal mgr operations until it fails to allocate memory and gets killed: https://pastebin.com/MLyw9iVi The mgr experienced this issue several times in the last 2 months, and the events don't appear to correlate with any other events in the cluster because basically nothing else happened at around those times. How can I investigate this and figure out what's causing the mgr to consume all memory and get killed? I would very much appreciate any advice! Best regards, Zakhar

6 21

Upgrading nautilus / centos7 to octopus / ubuntu 20.04. - Suggestions and hints?
by Götz Reinicke 17 Jan '24

17 Jan '24

Hi, As I’v read and thought a lot about the migration as this is a bigger project, I was wondering if anyone has done that already and might share some notes or playbooks, because in all readings there where some parts missing or miss understandable to me. I do have some different approaches in mind, so may be you have some suggestions or hints. a) upgrade nautilus on centos 7 with the few missing features like dashboard and prometheus. After that migrate one node after an other to ubuntu 20.04 with octopus and than upgrade ceph to the recent stable version. b) migrate one node after an other to ubuntu 18.04 with nautilus and then upgrade to octupus and after that to ubuntu 20.04. or c) upgrade one node after an other to ubuntu 20.04 with octopus and join it to the cluster until all nodes are upgraded. For test I tried c) with a mon node, but adding that to the cluster fails with some failed state, still probing for the other mons. (I dont have the right log at hand right now.) So my questions are: a) What would be the best (most stable) migration path and b) is it in general possible to add a new octopus mon (not upgraded one) to a nautilus cluster, where the other mons are still on nautilus? I hope my thoughts and questions are understandable :) Thanks for any hint and suggestion. Best . Götz

7 11

How to configure something like osd_deep_scrub_min_interval?
by Frank Schilder 09 Jan '24

09 Jan '24

Hi folks, I am fighting a bit with odd deep-scrub behavior on HDDs and discovered a likely cause of why the distribution of last_deep_scrub_stamps is so weird. I wrote a small script to extract a histogram of scrubs by "days not scrubbed" (more precisely, intervals not scrubbed; see code) to find out how (deep-) scrub times are distributed. Output below. What I expected is along the lines that HDD-OSDs try to scrub every 1-3 days, while they try to deep-scrub every 7-14 days. In other words, OSDs that have been deep-scrubbed within the last 7 days would *never* be in scrubbing+deep state. However, what I see is completely different. There seems to be no distinction between scrub- and deep-scrub start times. This is really unexpected as nobody would try to deep-scrub HDDs every day. Weekly to bi-weekly is normal, specifically for large drives. Is there a way to configure something like osd_deep_scrub_min_interval (no, I don't want to run cron jobs for scrubbing yet)? In the output below, I would like to be able to configure a minimum period of 1-2 weeks before the next deep-scrub happens. How can I do that? The observed behavior is very unusual for RAID systems (if its not a bug in the report script). With this behavior its not surprising that people complain about "not deep-scrubbed in time" messages and too high deep-scrub IO load when such a large percentage of OSDs is needlessly deep-scrubbed after 1-6 days again already. Sample output: # scrub-report dumped pgs Scrub report: 4121 PGs not scrubbed since 1 intervals (6h) 3831 PGs not scrubbed since 2 intervals (6h) 4012 PGs not scrubbed since 3 intervals (6h) 3986 PGs not scrubbed since 4 intervals (6h) 2998 PGs not scrubbed since 5 intervals (6h) 1488 PGs not scrubbed since 6 intervals (6h) 909 PGs not scrubbed since 7 intervals (6h) 771 PGs not scrubbed since 8 intervals (6h) 582 PGs not scrubbed since 9 intervals (6h) 2 scrubbing 431 PGs not scrubbed since 10 intervals (6h) 333 PGs not scrubbed since 11 intervals (6h) 1 scrubbing 265 PGs not scrubbed since 12 intervals (6h) 195 PGs not scrubbed since 13 intervals (6h) 116 PGs not scrubbed since 14 intervals (6h) 78 PGs not scrubbed since 15 intervals (6h) 1 scrubbing 72 PGs not scrubbed since 16 intervals (6h) 37 PGs not scrubbed since 17 intervals (6h) 5 PGs not scrubbed since 18 intervals (6h) 14.237* 19.5cd* 19.12cc* 19.1233* 14.40e* 33 PGs not scrubbed since 20 intervals (6h) 23 PGs not scrubbed since 21 intervals (6h) 16 PGs not scrubbed since 22 intervals (6h) 12 PGs not scrubbed since 23 intervals (6h) 8 PGs not scrubbed since 24 intervals (6h) 2 PGs not scrubbed since 25 intervals (6h) 19.eef* 19.bb3* 4 PGs not scrubbed since 26 intervals (6h) 19.b4c* 19.10b8* 19.f13* 14.1ed* 5 PGs not scrubbed since 27 intervals (6h) 19.43f* 19.231* 19.1dbe* 19.1788* 19.16c0* 6 PGs not scrubbed since 28 intervals (6h) 2 PGs not scrubbed since 30 intervals (6h) 19.10f6* 14.9d* 3 PGs not scrubbed since 31 intervals (6h) 19.1322* 19.1318* 8.a* 1 PGs not scrubbed since 32 intervals (6h) 19.133f* 1 PGs not scrubbed since 33 intervals (6h) 19.1103* 3 PGs not scrubbed since 36 intervals (6h) 19.19cc* 19.12f4* 19.248* 1 PGs not scrubbed since 39 intervals (6h) 19.1984* 1 PGs not scrubbed since 41 intervals (6h) 14.449* 1 PGs not scrubbed since 44 intervals (6h) 19.179f* Deep-scrub report: 3723 PGs not deep-scrubbed since 1 intervals (24h) 4621 PGs not deep-scrubbed since 2 intervals (24h) 8 scrubbing+deep 3588 PGs not deep-scrubbed since 3 intervals (24h) 8 scrubbing+deep 2929 PGs not deep-scrubbed since 4 intervals (24h) 3 scrubbing+deep 1705 PGs not deep-scrubbed since 5 intervals (24h) 4 scrubbing+deep 1904 PGs not deep-scrubbed since 6 intervals (24h) 5 scrubbing+deep 1540 PGs not deep-scrubbed since 7 intervals (24h) 7 scrubbing+deep 1304 PGs not deep-scrubbed since 8 intervals (24h) 7 scrubbing+deep 923 PGs not deep-scrubbed since 9 intervals (24h) 5 scrubbing+deep 557 PGs not deep-scrubbed since 10 intervals (24h) 7 scrubbing+deep 501 PGs not deep-scrubbed since 11 intervals (24h) 2 scrubbing+deep 363 PGs not deep-scrubbed since 12 intervals (24h) 2 scrubbing+deep 377 PGs not deep-scrubbed since 13 intervals (24h) 1 scrubbing+deep 383 PGs not deep-scrubbed since 14 intervals (24h) 2 scrubbing+deep 252 PGs not deep-scrubbed since 15 intervals (24h) 2 scrubbing+deep 116 PGs not deep-scrubbed since 16 intervals (24h) 5 scrubbing+deep 47 PGs not deep-scrubbed since 17 intervals (24h) 2 scrubbing+deep 10 PGs not deep-scrubbed since 18 intervals (24h) 2 PGs not deep-scrubbed since 19 intervals (24h) 19.1c6c* 19.a01* 1 PGs not deep-scrubbed since 20 intervals (24h) 14.1ed* 2 PGs not deep-scrubbed since 21 intervals (24h) 19.1322* 19.10f6* 1 PGs not deep-scrubbed since 23 intervals (24h) 19.19cc* 1 PGs not deep-scrubbed since 24 intervals (24h) 19.179f* PGs marked with a * are on busy OSDs and not eligible for scrubbing. The script (pasted here because attaching doesn't work): # cat bin/scrub-report #!/bin/bash # Compute last scrub interval count. Scrub interval 6h, deep-scrub interval 24h. # Print how many PGs have not been (deep-)scrubbed since #intervals. ceph -f json pg dump pgs 2>&1 > /root/.cache/ceph/pgs_dump.json echo "" T0="$(date +%s)" scrub_info="$(jq --arg T0 "$T0" -rc '.pg_stats[] | [ .pgid, (.last_scrub_stamp[:19]+"Z" | (($T0|tonumber) - fromdateiso8601)/(60*60*6)|ceil), (.last_deep_scrub_stamp[:19]+"Z" | (($T0|tonumber) - fromdateiso8601)/(60*60*24)|ceil), .state, (.acting | join(" ")) ] | @tsv ' /root/.cache/ceph/pgs_dump.json)" # less <<<"$scrub_info" # 1 2 3 4 5..NF # pg_id scrub-ints deep-scrub-ints status acting[] awk <<<"$scrub_info" '{ for(i=5; i<=NF; ++i) pg_osds[$1]=pg_osds[$1] " " $i if($4 == "active+clean") { si_mx=si_mx<$2 ? $2 : si_mx dsi_mx=dsi_mx<$3 ? $3 : dsi_mx pg_sn[$2]++ pg_sn_ids[$2]=pg_sn_ids[$2] " " $1 pg_dsn[$3]++ pg_dsn_ids[$3]=pg_dsn_ids[$3] " " $1 } else if($4 ~ /scrubbing\+deep/) { deep_scrubbing[$3]++ for(i=5; i<=NF; ++i) osd[$i]="busy" } else if($4 ~ /scrubbing/) { scrubbing[$2]++ for(i=5; i<=NF; ++i) osd[$i]="busy" } else { unclean[$2]++ unclean_d[$3]++ si_mx=si_mx<$2 ? $2 : si_mx dsi_mx=dsi_mx<$3 ? $3 : dsi_mx pg_sn[$2]++ pg_sn_ids[$2]=pg_sn_ids[$2] " " $1 pg_dsn[$3]++ pg_dsn_ids[$3]=pg_dsn_ids[$3] " " $1 for(i=5; i<=NF; ++i) osd[$i]="busy" } } END { print "Scrub report:" for(si=1; si<=si_mx; ++si) { if(pg_sn[si]==0 && scrubbing[si]==0 && unclean[si]==0) continue; printf("%7d PGs not scrubbed since %2d intervals (6h)", pg_sn[si], si) if(scrubbing[si]) printf(" %d scrubbing", scrubbing[si]) if(unclean[si]) printf(" %d unclean", unclean[si]) if(pg_sn[si]<=5) { split(pg_sn_ids[si], pgs) osds_busy=0 for(pg in pgs) { split(pg_osds[pgs[pg]], osds) for(o in osds) if(osd[osds[o]]=="busy") osds_busy=1 if(osds_busy) printf(" %s*", pgs[pg]) if(!osds_busy) printf(" %s", pgs[pg]) } } printf("\n") } print "" print "Deep-scrub report:" for(dsi=1; dsi<=dsi_mx; ++dsi) { if(pg_dsn[dsi]==0 && deep_scrubbing[dsi]==0 && unclean_d[dsi]==0) continue; printf("%7d PGs not deep-scrubbed since %2d intervals (24h)", pg_dsn[dsi], dsi) if(deep_scrubbing[dsi]) printf(" %d scrubbing+deep", deep_scrubbing[dsi]) if(unclean_d[dsi]) printf(" %d unclean", unclean_d[dsi]) if(pg_dsn[dsi]<=5) { split(pg_dsn_ids[dsi], pgs) osds_busy=0 for(pg in pgs) { split(pg_osds[pgs[pg]], osds) for(o in osds) if(osd[osds[o]]=="busy") osds_busy=1 if(osds_busy) printf(" %s*", pgs[pg]) if(!osds_busy) printf(" %s", pgs[pg]) } } printf("\n") } print "" print "PGs marked with a * are on busy OSDs and not eligible for scrubbing." } ' Don't forget the last "'" when copy-pasting. Thanks for any pointers. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 6

What is the maximum number of Rados gateway objects in one cluster using the bucket index and in one bucket?
by steve jung 02 Jan '24

02 Jan '24

Hello. We are using Ceph storage to test whether we can run the service by uploading and saving more than 40 billion files. So I'd like to check the contents below. 1) Maximum number of Rados gateway objects that can be stored in one cluster using the bucket index 2) Maximum number of Rados gateway objects that can be stored in one bucket Although we have referred to the limitations on the number of Rados gateway objects mentioned in existing documents, it seems theoretically unlimited If you have operated the number of files at the level we think in actual services or products, we would appreciate it if you could share them. Below are related documents and related settings values. > Related documents - https://documentation.suse.com/ses/5.5/html/ses-all/cha-ceph-gw.html - https://www.ibm.com/docs/en/storage-ceph/6?topic=resharding-limitations-buc… - https://docs.ceph.com/en/latest/dev/radosgw/bucket_index/ > Related config - rgw_dynamic_resharding: true - rgw_max_objs_per_shard: 100000 - rgw_max_dynamic_shards : 65521

2 1

Re: increasing number of (deep) scrubs
by Frank Schilder 13 Dec '23

13 Dec '23

Hi Dan, thanks for your answer. I don't have a problem with increasing osd_max_scrubs (=1 at the moment) as such. I would simply prefer a somewhat finer grained way of controlling scrubbing than just doubling or tripling it right away. Some more info. These 2 pools are data pools for a large FS. Unfortunately, we have a large percentage of small files, which is a pain for recovery and seemingly also for deep scrubbing. Our OSDs are about 25% used and I had to increase the warning interval already to 2 weeks. With all the warning grace parameters this means that we manage to deep scrub everything about every month. I need to plan for 75% utilisation and a 3 months period is a bit far on the risky side. Our data is to a large percentage cold data. Client reads will not do the check for us, we need to combat bit-rot pro-actively. The reasons I'm interested in parameters initiating more scrubs while also converting more scrubs into deep scrubs are, that 1) scrubs seem to complete very fast. I almost never catch a PG in state "scrubbing", I usually only see "deep scrubbing". 2) I suspect the low deep-scrub count is due to a low number of deep-scrubs scheduled and not due to conflicting per-OSD deep scrub reservations. With the OSD count we have and the distribution over 12 servers I would expect at least a peak of 50% OSDs being active in scrubbing instead of the 25% peak I'm seeing now. It ought to be possible to schedule more PGs for deep scrub than actually are. 3) Every OSD having only 1 deep scrub active seems to have no measurable impact on user IO. If I could just get more PGs scheduled with 1 deep scrub per OSD it would already help a lot. Once this is working, I can eventually increase osd_max_scrubs when the OSDs fill up. For now I would just like that (deep) scrub scheduling looks a bit harder and schedules more eligible PGs per time unit. If we can get deep scrubbing up to an average of 42PGs completing per hour with keeping osd_max_scrubs=1 to maintain current IO impact, we should be able to complete a deep scrub with 75% full OSDs in about 30 days. This is the current tail-time with 25% utilisation. I believe currently a deep scrub of a PG in these pools takes 2-3 hours. Its just a gut feeling from some repair and deep-scrub commands, I would need to check logs for more precise info. Increasing osd_max_scrubs would then be a further and not the only option to push for more deep scrubbing. My expectation would be that values of 2-3 are fine due to the increasingly higher percentage of cold data for which no interference with client IO will happen. Hope that makes sense and there is a way beyond bumping osd_max_scrubs to increase the number of scheduled and executed deep scrubs. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dvanders(a)gmail.com> Sent: 05 January 2023 15:36 To: Frank Schilder Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] increasing number of (deep) scrubs Hi Frank, What is your current osd_max_scrubs, and why don't you want to increase it? With 8+2, 8+3 pools each scrub is occupying the scrub slot on 10 or 11 OSDs, so at a minimum it could take 3-4x the amount of time to scrub the data than if those were replicated pools. If you want the scrub to complete in time, you need to increase the amount of scrub slots accordingly. On the other hand, IMHO the 1-week deadline for deep scrubs is often much too ambitious for large clusters -- increasing the scrub intervals is one solution, or I find it simpler to increase mon_warn_pg_not_scrubbed_ratio and mon_warn_pg_not_deep_scrubbed_ratio until you find a ratio that works for your cluster. Of course, all of this can impact detection of bit-rot, which anyway can be covered by client reads if most data is accessed periodically. But if the cluster is mostly idle or objects are generally not read, then it would be preferable to increase slots osd_max_scrubs. Cheers, Dan On Tue, Jan 3, 2023 at 2:30 AM Frank Schilder <frans(a)dtu.dk> wrote: > > Hi all, > > we are using 16T and 18T spinning drives as OSDs and I'm observing that they are not scrubbed as often as I would like. It looks like too few scrubs are scheduled for these large OSDs. My estimate is as follows: we have 852 spinning OSDs backing a 8+2 pool with 2024 and an 8+3 pool with 8192 PGs. On average I see something like 10PGs of pool 1 and 12 PGs of pool 2 (deep) scrubbing. This amounts to only 232 out of 852 OSDs scrubbing and seems to be due to a conservative rate of (deep) scrubs being scheduled. The PGs (dep) scrub fairly quickly. > > I would like to increase gently the number of scrubs scheduled for these drives and *not* the number of scrubs per OSD. I'm looking at parameters like: > > osd_scrub_backoff_ratio > osd_deep_scrub_randomize_ratio > > I'm wondering if lowering osd_scrub_backoff_ratio to 0.5 and, maybe, increasing osd_deep_scrub_randomize_ratio to 0.2 would have the desired effect? Are there other parameters to look at that allow gradual changes in the number of scrubs going on? > > Thanks a lot for your help! > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 3

reef 18.2.1 QE Validation status
by Yuri Weinstein 11 Dec '23

11 Dec '23

Details of this release are summarized here: https://tracker.ceph.com/issues/63443#note-1 Seeking approvals/reviews for: smoke - Laura, Radek, Prashant, Venky (POOL_APP_NOT_ENABLE failures) rados - Neha, Radek, Travis, Ernesto, Adam King rgw - Casey fs - Venky orch - Adam King rbd - Ilya krbd - Ilya upgrade/quincy-x (reef) - Laura PTL powercycle - Brad perf-basic - Laura, Prashant (POOL_APP_NOT_ENABLE failures) Please reply to this email with approval and/or trackers of known issues/PRs to address them. TIA YuriW

12 30

ceph fs (meta) data inconsistent
by Frank Schilder 11 Dec '23

11 Dec '23

Dear fellow cephers, today we observed a somewhat worrisome inconsistency on our ceph fs. A file created on one host showed up as 0 length on all other hosts: [user1@host1 h2lib]$ ls -lh total 37M -rw-rw---- 1 user1 user1 12K Nov 1 11:59 dll_wrapper.py [user2@host2 h2lib]# ls -l total 34 -rw-rw----. 1 user1 user1 0 Nov 1 11:59 dll_wrapper.py [user1@host1 h2lib]$ cp dll_wrapper.py dll_wrapper.py.test [user1@host1 h2lib]$ ls -l total 37199 -rw-rw---- 1 user1 user1 11641 Nov 1 11:59 dll_wrapper.py -rw-rw---- 1 user1 user1 11641 Nov 1 13:10 dll_wrapper.py.test [user2@host2 h2lib]# ls -l total 45 -rw-rw----. 1 user1 user1 0 Nov 1 11:59 dll_wrapper.py -rw-rw----. 1 user1 user1 11641 Nov 1 13:10 dll_wrapper.py.test Executing a sync on all these hosts did not help. However, deleting the problematic file and replacing it with a copy seemed to work around the issue. We saw this with ceph kclients of different versions, it seems to be on the MDS side. How can this happen and how dangerous is it? ceph fs status (showing ceph version): # ceph fs status con-fs2 - 1662 clients ======= RANK STATE MDS ACTIVITY DNS INOS 0 active ceph-15 Reqs: 14 /s 2307k 2278k 1 active ceph-11 Reqs: 159 /s 4208k 4203k 2 active ceph-17 Reqs: 3 /s 4533k 4501k 3 active ceph-24 Reqs: 3 /s 4593k 4300k 4 active ceph-14 Reqs: 1 /s 4228k 4226k 5 active ceph-13 Reqs: 5 /s 1994k 1782k 6 active ceph-16 Reqs: 8 /s 5022k 4841k 7 active ceph-23 Reqs: 9 /s 4140k 4116k POOL TYPE USED AVAIL con-fs2-meta1 metadata 2177G 7085G con-fs2-meta2 data 0 7085G con-fs2-data data 1242T 4233T con-fs2-data-ec-ssd data 706G 22.1T con-fs2-data2 data 3409T 3848T STANDBY MDS ceph-10 ceph-08 ceph-09 ceph-12 MDS version: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) There is no health issue: # ceph status cluster: id: abc health: HEALTH_WARN 3 pgs not deep-scrubbed in time services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 9w) mgr: ceph-25(active, since 7w), standbys: ceph-26, ceph-01, ceph-03, ceph-02 mds: con-fs2:8 4 up:standby 8 up:active osd: 1284 osds: 1279 up (since 2d), 1279 in (since 5d) task status: data: pools: 14 pools, 25065 pgs objects: 2.20G objects, 3.9 PiB usage: 4.9 PiB used, 8.2 PiB / 13 PiB avail pgs: 25039 active+clean 26 active+clean+scrubbing+deep io: client: 799 MiB/s rd, 55 MiB/s wr, 3.12k op/s rd, 1.82k op/s wr The inconsistency seems undiagnosed, I couldn't find anything interesting in the cluster log. What should I look for and where? I moved the folder to another location for diagnosis. Unfortunately, I don't have 2 clients any more showing different numbers, I see a 0 length now everywhere for the moved folder. I'm pretty sure though that the file still is non-zero length. Thanks for any pointers. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 18

Difficulty adding / using a non-default RGW placement target & storage class
by Anthony D'Atri 07 Dec '23

07 Dec '23

I'm having difficulty adding and using a non-default placement target & storage class and would appreciate insights. Am I going about this incorrectly? Rook does not yet have the ability to do this, so I'm adding it by hand. Following instructions on the net I added a second bucket pool, placement target, and storageclass, and created a user defaulting to the new pg/sc, but I get an error when trying to create a bucket: [rook@rook-ceph-tools-5ff8d58445-gkl5w .aws]$ s5cmd --endpoint-url http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc mb s3://foofoobars ERROR "mb s3://foofoobars": InvalidLocationConstraint: status code: 400, request id: tx0000057b71002881d48ca-0065495d54-1abe555-ceph-objectstore, host id: I found an article suggesting that the placement target and/or storage class should have the api_name prepended, so I tried setting either or both to "ceph-objectstore:HDD-EC" / "ceph-objectstore:GLACIER" with no success. I suspect that I'm missing something subtle -- or that Rook has provisioned these bits in an atypical fashion. Log entry: /var/log/ceph/ceph-client.rgw.ceph.objectstore.a.log-2023-11-06T21:40:36.543+0000 7f6573a9f700 1 ====== starting new request req=0x7f64818ba730 ===== /var/log/ceph/ceph-client.rgw.ceph.objectstore.a.log-2023-11-06T21:40:36.546+0000 7f6570a99700 0 req 6320538205097380042 0.003000009s s3:create_bucket could not find user default placement id HDD-EC/GLACIER within zonegroup /var/log/ceph/ceph-client.rgw.ceph.objectstore.a.log-2023-11-06T21:40:36.546+0000 7f6570a99700 1 ====== req done req=0x7f64818ba730 op status=-2208 http_status=400 latency=0.003000009s ====== /var/log/ceph/ceph-client.rgw.ceph.objectstore.a.log:2023-11-06T21:40:36.546+0000 7f6570a99700 1 beast: 0x7f64818ba730: 10.233.90.156 - aad [06/Nov/2023:21:40:36.543 +0000] "PUT /foofoobars HTTP/1.1" 400 266 - "aws-sdk-go/1.40.25 (go1.18.3; linux; amd64)" - latency=0.003000009s [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ ceph -v ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) Here's the second buckets pool, constrained to HDDs. AFAICT it can share the index and data_extra_pool created for the default / STANDARD pt/sc by Rook. I initially omitted ec_overwrites but enabled it after creation. pool 19 'ceph-objectstore.rgw.buckets.data' erasure profile ceph-objectstore.rgw.buckets.data_ecprofile size 6 min_size 5 crush_rule 10 object_hash rjenkins pg_num 8192 pgp_num 8192 autoscale_mode off last_change 165350 lfor 0/156300/165341 flags hashpspool,ec_overwrites stripe_width 16384 application rook-ceph-rgw pool 21 'ceph-objectstore.rgw.buckets.data.hdd' erasure profile ceph-objectstore.rgw.buckets.data_ecprofile_hdd size 6 min_size 5 crush_rule 11 object_hash rjenkins pg_num 8192 pgp_num 8192 autoscale_mode off last_change 167193 lfor 0/0/164453 flags hashpspool,ec_overwrites stripe_width 16384 application rook-ceph-rgw [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ radosgw-admin zonegroup get { "id": "d994155c-2a9c-4e37-ae30-64fd2934ff99", "name": "ceph-objectstore", "api_name": "ceph-objectstore", "is_master": "true", "endpoints": [ "http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc:80" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "72035401-a6d9-426b-8c89-9a17e268825f", "zones": [ { "id": "72035401-a6d9-426b-8c89-9a17e268825f", "name": "ceph-objectstore", "endpoints": [ "http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc:80" ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "HDD-EC", "tags": [], "storage_classes": [ "GLACIER" ] }, { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "51fb8875-31ac-40ef-ab21-0ffd4e229f15", "sync_policy": { "groups": [] } } [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ radosgw-admin zone get { "id": "72035401-a6d9-426b-8c89-9a17e268825f", "name": "ceph-objectstore", "domain_root": "ceph-objectstore.rgw.meta:root", "control_pool": "ceph-objectstore.rgw.control", "gc_pool": "ceph-objectstore.rgw.log:gc", "lc_pool": "ceph-objectstore.rgw.log:lc", "log_pool": "ceph-objectstore.rgw.log", "intent_log_pool": "ceph-objectstore.rgw.log:intent", "usage_log_pool": "ceph-objectstore.rgw.log:usage", "roles_pool": "ceph-objectstore.rgw.meta:roles", "reshard_pool": "ceph-objectstore.rgw.log:reshard", "user_keys_pool": "ceph-objectstore.rgw.meta:users.keys", "user_email_pool": "ceph-objectstore.rgw.meta:users.email", "user_swift_pool": "ceph-objectstore.rgw.meta:users.swift", "user_uid_pool": "ceph-objectstore.rgw.meta:users.uid", "otp_pool": "ceph-objectstore.rgw.otp", "system_key": { "access_key": "", "secret_key": "" }, "placement_pools": [ { "key": "HDD-EC", "val": { "index_pool": "ceph-objectstore.rgw.buckets.index", "storage_classes": { "GLACIER": { "data_pool": "ceph-objectstore.rgw.buckets.data.hdd" }, "STANDARD": {} # <------------- seems like this shouldn't be here? }, "data_extra_pool": "ceph-objectstore.rgw.buckets.non-ec", "index_type": 0 } }, { "key": "default-placement", "val": { "index_pool": "ceph-objectstore.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "ceph-objectstore.rgw.buckets.data" } }, "data_extra_pool": "ceph-objectstore.rgw.buckets.non-ec", "index_type": 0 } } ], "realm_id": "", "notif_pool": "ceph-objectstore.rgw.log:notif" } [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ radosgw-admin user info --uid=aad { "user_id": "aad", "display_name": "Anthony", "email": "", "suspended": 0, "max_buckets": 1000, "subusers": [], "keys": [ { "user": "aad", "access_key": "xxxx", "secret_key": "yyyyyy" } ], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "HDD-EC", "default_storage_class": "GLACIER", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw", "mfa_ids": [] }

2 1

Space reclaim doesn't happening in nautilus RBD pool
by Szabo, Istvan (Agoda) 06 Dec '23

06 Dec '23

Hi, Is there any config on Ceph that block/not perform space reclaim? I test on one pool which has only one image 1.8 TiB in used. rbd $p du im/root warning: fast-diff map is not enabled for root. operation may be slow. NAME PROVISIONED USED root 2.2 TiB 1.8 TiB I already removed all snaphots and now pool has only one image alone. I run both fstrim over the filesystem (XFS) and try rbd sparsify im/root (don't know what it is exactly but it mentions to reclaim something) It still shows the pool used 6.9 TiB which totally not make sense right? It should be up to 3.6 (1.8 * 2) according to its replica? POOLS: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR im 19 32 3.5 TiB 918.34k 6.9 TiB 4.80 69 TiB N/A 10 TiB 918.34k 0 B 0 B I think now some of others pool have this issue too, we do clean up a lot but seems space not reclaimed. I estimate more than 50 TiB should be able to reclaim, actual usage of this cluster much less than current reported number. Thank you for your help. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 4

CephFS pool not releasing space after data deletion
by Kuhring, Mathias 05 Dec '23

05 Dec '23

Dear Ceph users, Our CephFS is not releasing/freeing up space after deleting hundreds of terabytes of data. By now, this drives us in a "nearfull" osd/pool situation and thus throttles IO. We are on ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable). Recently, we moved a bunch of data to a new pool with better EC. This was done by adding a new EC pool to the FS. Then assigning the FS root to the new EC pool via the directory layout xattr (so all new data is written to the new pool). And finally copying old data to new folders. I swapped the data as follows to remain the old directory structures. I also made snapshots for validation purposes. So basically: cp -r mymount/mydata/ mymount/new/ # this creates copy on new pool mkdir mymount/mydata/.snap/tovalidate mkdir mymount/new/mydata/.snap/tovalidate mv mymount/mydata/ mymount/old/ mv mymount/new/mydata mymount/ I could see the increase of data in the new pool as expected (ceph df). I compared the snapshots with hashdeep to make sure the new data is alright. Then I went ahead deleting the old data, basically: rmdir mymount/old/mydata/.snap/* # this also included a bunch of other older snapshots rm -r mymount/old/mydata At first we had a bunch of PGs with snaptrim/snaptrim_wait. But they are done for quite some time now. And now, already two weeks later the size of the old pool still hasn't really decreased. I'm still waiting for around 500 TB to be released (and much more is planned). I honestly have no clue, where to go from here. From my point of view (i.e. the CephFS mount), the data is gone. I also never hard/soft-linked it anywhere. This doesn't seem to be a regular issue. At least I couldn't find anything related or resolved in the docs or user list, yet. If anybody has an idea how to resolve this, I would highly appreciate it. Best Wishes, Mathias

3 4

After hardware failure tried to recover ceph and followed instructions for recovery using OSDS
by Manolis Daramas 05 Dec '23

05 Dec '23

Hello everyone, We had a recent power failure on a server which hosts a 3-node ceph cluster (with Ubuntu 20.04 and Ceph version 17.2.7) and we think that we may have lost some of our data if not all of them. We have followed the instructions on https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-mon/#re… but with no luck. We have kept a backup of store.db folder on all 3 nodes prior the below steps. We have stopped ceph.target on all 3 nodes. We have run the first part of the script and we have altered it according to our configuration: ms=/root/mon-store mkdir $ms hosts="node01 node02 node03" # collect the cluster map from stopped OSDs for host in $hosts; do rsync -avz $ms/. root@$host:$ms.remote rm -rf $ms ssh root@$host <<EOF for osd in /var/lib/ceph/be4304e4-b0d5-11ec-8c6a-2965d4229f37/osd*; do ceph-objectstore-tool --data-path \$osd --no-mon-config --op update-mon-db --mon-store-path $ms.remote done EOF rsync -avz root@$host:$ms.remote/. $ms done and the results were: for node01 sd.0 : 0 osdmaps trimmed, 673 osdmaps added. osd.10 : 9225 osdmaps trimmed, 0 osdmaps added. Mount failed with '(5) Input/output error' osd.4 : 0 osdmaps trimmed, 0 osdmaps added. osd.8 : 0 osdmaps trimmed, 0 osdmaps added. receiving incremental file list created directory /root/mon-store ./ kv_backend store.db/ store.db/000008.sst store.db/000014.sst store.db/000020.sst store.db/000022.log store.db/CURRENT store.db/IDENTITY store.db/LOCK store.db/MANIFEST-000021 store.db/OPTIONS-000018 store.db/OPTIONS-000024 sent 248 bytes received 286,474 bytes 191,148.00 bytes/sec total size is 7,869,025 speedup is 27.44 sending incremental file list created directory /root/mon-store.remote ./ kv_backend store.db/ store.db/000008.sst store.db/000014.sst store.db/000020.sst store.db/000022.log store.db/CURRENT store.db/IDENTITY store.db/LOCK store.db/MANIFEST-000021 store.db/OPTIONS-000018 store.db/OPTIONS-000024 sent 286,478 bytes received 285 bytes 191,175.33 bytes/sec total size is 7,869,025 speedup is 27.44 for node02 osd.12 : 0 osdmaps trimmed, 0 osdmaps added. osd.2 : 0 osdmaps trimmed, 0 osdmaps added. osd.5 : 0 osdmaps trimmed, 0 osdmaps added. osd.7 : 0 osdmaps trimmed, 0 osdmaps added. osd.9 : 0 osdmaps trimmed, 0 osdmaps added. receiving incremental file list created directory /root/mon-store ./ kv_backend store.db/ store.db/000008.sst store.db/000014.sst store.db/000020.sst store.db/000026.sst store.db/000032.sst store.db/000038.sst store.db/000044.sst store.db/000050.sst store.db/000052.log store.db/CURRENT store.db/IDENTITY store.db/LOCK store.db/MANIFEST-000051 store.db/OPTIONS-000048 store.db/OPTIONS-000054 sent 343 bytes received 291,082 bytes 194,283.33 bytes/sec total size is 7,875,746 speedup is 27.02 sending incremental file list created directory /root/mon-store.remote ./ kv_backend store.db/ store.db/000008.sst store.db/000014.sst store.db/000020.sst store.db/000026.sst store.db/000032.sst store.db/000038.sst store.db/000044.sst store.db/000050.sst store.db/000052.log store.db/CURRENT store.db/IDENTITY store.db/LOCK store.db/MANIFEST-000051 store.db/OPTIONS-000048 store.db/OPTIONS-000054 sent 291,078 bytes received 380 bytes 582,916.00 bytes/sec total size is 7,875,746 speedup is 27.02 for node03 osd.1 : 0 osdmaps trimmed, 0 osdmaps added. osd.11 : 0 osdmaps trimmed, 0 osdmaps added. osd.13 : 0 osdmaps trimmed, 0 osdmaps added. osd.3 : 0 osdmaps trimmed, 0 osdmaps added. osd.6 : 0 osdmaps trimmed, 0 osdmaps added. receiving incremental file list created directory /root/mon-store ./ kv_backend store.db/ store.db/000008.sst store.db/000014.sst store.db/000020.sst store.db/000026.sst store.db/000032.sst store.db/000038.sst store.db/000044.sst store.db/000050.sst store.db/000056.sst store.db/000062.sst store.db/000068.sst store.db/000074.sst store.db/000080.sst store.db/000082.log store.db/CURRENT store.db/IDENTITY store.db/LOCK store.db/MANIFEST-000081 store.db/OPTIONS-000078 store.db/OPTIONS-000084 sent 438 bytes received 295,659 bytes 592,194.00 bytes/sec total size is 7,882,477 speedup is 26.62 Then we have run the (in order to rebuild the monstore DB and fix it): ceph-monstore-tool /root/mon-store rebuild -- --keyring /etc/ceph/ceph.client.admin.keyring --mon-ids node01 node02 node03 and the output is below: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: RocksDB version: 6.15.5 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Compile date Oct 25 2023 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: DB SUMMARY 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: DB Session ID: OS2T69IQ02SU5OKHBI40 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: CURRENT file: CURRENT 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: IDENTITY file: IDENTITY 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: MANIFEST file: MANIFEST-000081 size: 1083 Bytes 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: SST files in /root/mon-store/store.db dir, Total Num: 13, files: 000008.sst 000014.sst 000020.sst 000026.sst 000032.sst 000038.sst 000044.sst 000050.sst 000056.sst 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Write Ahead Log file in /root/mon-store/store.db: 000082.log size: 244 ; 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.error_if_exists: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.create_if_missing: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.paranoid_checks: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.env: 0x56017c8d1c20 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.fs: Posix File System 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.info_log: 0x56017d4c3860 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.statistics: (nil) 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.use_fsync: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_log_file_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_fallocate: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.use_direct_reads: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.create_missing_column_families: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.db_log_dir: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.wal_dir: /root/mon-store/store.db 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.advise_random_on_open: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.write_buffer_manager: 0x56017d1f6a20 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.rate_limiter: (nil) 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.unordered_write: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.row_cache: None 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.wal_filter: None 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.preserve_deletes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.two_write_queues: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.manual_wal_flush: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.atomic_flush: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.log_readahead_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.best_efforts_recovery: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.allow_data_in_errors: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.db_host_id: __hostname__ 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_background_jobs: 2 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_background_compactions: -1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_subcompactions: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_total_wal_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_open_files: -1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bytes_per_sync: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_background_flushes: -1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Compression algorithms supported: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kZSTD supported: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kXpressCompression supported: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kLZ4Compression supported: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kBZip2Compression supported: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kZlibCompression supported: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: kSnappyCompression supported: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /root/mon-store/store.db/MANIFEST-000081 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.merge_operator: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_filter: None 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_filter_factory: None 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.sst_partitioner_factory: None 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56017d234f80) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 0 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 index_shortening: 1 data_block_hash_table_util_ratio: 0.750000 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x56017d22f610 block_cache_name: BinnedLRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 4 enable_index_compression: 1 block_align: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression: NoCompression 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.num_levels: 7 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.level: 32767 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.parallel_threads: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compression_opts.enabled: false 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.arena_block_size: 4194304 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.table_properties_collectors: 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.inplace_update_support: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.bloom_locality: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.max_successive_merges: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.force_consistency_checks: 1 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.ttl: 2592000 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.enable_blob_files: false 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.min_blob_size: 0 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.blob_file_size: 268435456 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.blob_compression_type: NoCompression 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.enable_blob_garbage_collection: false 2023-11-17T12:26:24.152+0200 7f482b393600 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-11-17T12:26:24.156+0200 7f482b393600 4 rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/root/mon-store/store.db/MANIFEST-000081 succeeded,manifest_file_number is 81, next_file_number is 83, last_sequence is 21183, log_number is 77,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-17T12:26:24.156+0200 7f482b393600 4 rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 77 2023-11-17T12:26:24.156+0200 7f482b393600 4 rocksdb: [db/version_set.cc:4082] Creating manifest 85 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784162798, "job": 1, "event": "recovery_started", "wal_files": [82]} 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #82 mode 2 2023-11-17T12:26:24.160+0200 7f482b393600 3 rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784163944, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 86, "file_size": 1266, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 238, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 72, "raw_average_key_size": 24, "raw_value_size": 148, "raw_average_value_size": 49, "num_data_blocks": 1, "num_entries": 3, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700216784, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "53025a24-2059-43e1-a0f7-a87a28e33d38", "db_session_id": "OS2T69IQ02SU5OKHBI40"}} 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: [db/version_set.cc:4082] Creating manifest 87 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784166273, "job": 1, "event": "recovery_finished"} 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: [db/column_family.cc:983] [default] Increasing compaction threads because we have 14 level-0 files 2023-11-17T12:26:24.160+0200 7f482b393600 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000082.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.164+0200 7f482b393600 4 rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x56017d230700 2023-11-17T12:26:24.164+0200 7f482b393600 4 rocksdb: DB pointer 0x56017df56000 adding auth for 'client.admin': auth(key=AQCsdUViHYjTGBAAf7/1KYZjb0h3x3EOywqbbQ==) with caps({mds=allow *,mgr=allow *,mon=allow *,osd=allow *}) 2023-11-17T12:26:24.164+0200 7f482a349700 4 rocksdb: [db/compaction/compaction_job.cc:1881] [default] [JOB 3] Compacting 14@0 files to L6, score 3.50 2023-11-17T12:26:24.164+0200 7f482a349700 4 rocksdb: [db/compaction/compaction_job.cc:1887] [default] Compaction start summary: Base version 3 Base level 0, inputs: [86(1266B) 80(1266B) 74(1267B) 68(1267B) 62(1266B) 56(1265B) 50(1265B) 44(1265B) 38(1265B) 32(1266B) 26(1265B) 20(1265B) 14(283KB) 8(7387KB)] 2023-11-17T12:26:24.164+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784169200, "job": 3, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [86, 80, 74, 68, 62, 56, 50, 44, 38, 32, 26, 20, 14, 8], "score": 3.5, "input_data_size": 7870219} 2023-11-17T12:26:24.164+0200 7f4822339700 4 rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2023-11-17T12:26:24.164+0200 7f4822339700 4 rocksdb: [db/db_impl/db_impl.cc:903] ** DB Stats ** Uptime(secs): 0.0 total, 0.0 interval Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Sum 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Sum 14/14 7.51 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.1 0.00 0.00 1 0.001 0 0 Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.10 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** 2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb: [db/compaction/compaction_job.cc:1516] [default] [JOB 3] Generated table #91: 1366 keys, 7566988 bytes 2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784213586, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 91, "file_size": 7566988, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 7541895, "index_size": 20610, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3525, "raw_key_size": 29308, "raw_average_key_size": 21, "raw_value_size": 7503048, "raw_average_value_size": 5492, "num_data_blocks": 764, "num_entries": 1366, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700216681, "oldest_key_time": 0, "file_creation_time": 1700216784, "db_id": "53025a24-2059-43e1-a0f7-a87a28e33d38", "db_session_id": "OS2T69IQ02SU5OKHBI40"}} 2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb: [db/compaction/compaction_job.cc:1594] [default] [JOB 3] Compacted 14@0 files to L6 => 7566988 bytes 2023-11-17T12:26:24.208+0200 7f482a349700 4 rocksdb: [db/version_set.cc:3457] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: (Original Log Time 2023/11/17-12:26:24.215298) [db/compaction/compaction_job.cc:812] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 177.1 rd, 170.3 wr, level 6, files in(14, 0) out(1) MB in(7.5, 0.0) out(7.2), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 19842, records dropped: 18476 output_compression: NoCompression 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: (Original Log Time 2023/11/17-12:26:24.215314) EVENT_LOG_v1 {"time_micros": 1700216784215306, "job": 3, "event": "compaction_finished", "compaction_time_micros": 44437, "compaction_time_cpu_micros": 40923, "output_level": 6, "num_output_files": 1, "total_output_size": 7566988, "num_input_records": 19842, "num_output_records": 1366, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000086.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215520, "job": 3, "event": "table_file_deletion", "file_number": 86} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000080.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215570, "job": 3, "event": "table_file_deletion", "file_number": 80} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000074.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215603, "job": 3, "event": "table_file_deletion", "file_number": 74} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000068.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215641, "job": 3, "event": "table_file_deletion", "file_number": 68} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000062.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215672, "job": 3, "event": "table_file_deletion", "file_number": 62} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000056.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215708, "job": 3, "event": "table_file_deletion", "file_number": 56} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000050.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215739, "job": 3, "event": "table_file_deletion", "file_number": 50} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000044.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215772, "job": 3, "event": "table_file_deletion", "file_number": 44} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000038.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215804, "job": 3, "event": "table_file_deletion", "file_number": 38} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000032.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215831, "job": 3, "event": "table_file_deletion", "file_number": 32} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000026.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215858, "job": 3, "event": "table_file_deletion", "file_number": 26} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000020.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215888, "job": 3, "event": "table_file_deletion", "file_number": 20} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000014.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784215952, "job": 3, "event": "table_file_deletion", "file_number": 14} 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /root/mon-store/store.db/000008.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-17T12:26:24.212+0200 7f482a349700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700216784216804, "job": 3, "event": "table_file_deletion", "file_number": 8} update_mkfs generating seed initial monmap epoch 0 fsid be4304e4-b0d5-11ec-8c6a-2965d4229f37 last_changed 2023-11-17T12:26:24.222814+0200 created 2023-11-17T12:26:24.222814+0200 min_mon_release 0 (unknown) election_strategy: 1 0: [v2:10.40.99.11:3300/0,v1:10.40.99.11:6789/0] mon.node01 1: [v2:10.40.99.12:3300/0,v1:10.40.99.12:6789/0] mon.node02 2: [v2:10.40.99.13:3300/0,v1:10.40.99.13:6789/0] mon.node03 2023-11-17T12:26:24.220+0200 7f482b393600 4 rocksdb: [db/db_impl/db_impl.cc:446] Shutdown: canceling all background work 2023-11-17T12:26:24.220+0200 7f482b393600 4 rocksdb: [db/db_impl/db_impl.cc:625] Shutdown complete Then we copied the /root/mon-store/store.db folder across on all 3 nodes and tried to start ceph.target service again. The output on node01 is below: d31781fa6b4c quay.io/ceph/ceph "/usr/bin/ceph-mds -..." 55 minutes ago Up 55 minutes ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mds-storage-node01-cjrvjc e385c32651d2 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-10 904f522c4cb5 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-0 033edf99a98e quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-4 70344a6e87a0 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-8 905b782aedcf quay.io/prometheus/prometheus:v2.43.0 "/bin/prometheus --c..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-prometheus-node01 ff191654eb3e quay.io/prometheus/node-exporter:v1.5.0 "/bin/node_exporter ..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-node-exporter-node01 459c46f4bdb7 quay.io/ceph/ceph "/usr/bin/ceph-mgr -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mgr-node01-xlciyx cacfe8abcbbf quay.io/ceph/ceph "/usr/bin/ceph-crash..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-crash-node01 e216ef2af166 quay.io/prometheus/alertmanager:v0.25.0 "/bin/alertmanager -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-alertmanager-node01 d3238b2285d1 quay.io/ceph/ceph-grafana:9.4.7 "/bin/sh -c 'grafana..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-grafana-node01 The output on node02 is below: 2aec62685dee quay.io/ceph/ceph "/usr/bin/ceph-mds -..." 54 minutes ago Up 54 minutes ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mds-storage-node02-lyudbp 249b04f32f8c quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-5 a2c96f56b517 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-2 87496d374a29 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-12 55fe47765917 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-9 76171e25dbde quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-7 220472e8c1bf quay.io/ceph/ceph "/usr/bin/ceph-mgr -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mgr-node02-gudauu 0c783e73e543 quay.io/prometheus/node-exporter:v1.5.0 "/bin/node_exporter ..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-node-exporter-node02 4e638003fa2e quay.io/ceph/ceph "/usr/bin/ceph-crash..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-crash-node02 42719d5cfdbf quay.io/ceph/ceph "/usr/bin/ceph-mon -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mon-node02 The output on node03 is below: 7e5879dce643 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-11 d53996ff33b9 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-3 e1ac5a8b87d3 quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-1 f4cda871218d quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-13 969e670dc47c quay.io/ceph/ceph "/usr/bin/ceph-osd -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-osd-6 a49e91a7bb8e quay.io/prometheus/node-exporter:v1.5.0 "/bin/node_exporter ..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-node-exporter-node03 835c3893a3f4 quay.io/ceph/ceph "/usr/bin/ceph-crash..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-crash-node03 bfa6f5b989ea quay.io/ceph/ceph "/usr/bin/ceph-mon -..." 2 days ago Up 2 days ceph-be4304e4-b0d5-11ec-8c6a-2965d4229f37-mon-node03 # ceph -s (output below): cluster: id: be4304e4-b0d5-11ec-8c6a-2965d4229f37 health: HEALTH_ERR 20 stray daemon(s) not managed by cephadm 3 stray host(s) with 20 daemon(s) not managed by cephadm 1/3 mons down, quorum node02,node03 1/523510 objects unfound (0.000%) 3 nearfull osd(s) 1 osds exist in the crush map but not in the osdmap Low space hindering backfill (add storage if this doesn't resolve itself): 20 pgs backfill_toofull Possible data damage: 1 pg recovery_unfound Degraded data redundancy: 74666/1570530 objects degraded (4.754%), 21 pgs degraded, 21 pgs undersized 3 pool(s) nearfull services: mon: 3 daemons, quorum node02,node03 (age 2d), out of quorum: node01 mgr: node01.xlciyx(active, since 2d), standbys: node02.gudauu osd: 14 osds: 14 up (since 2d), 14 in (since 3d); 21 remapped pgs data: pools: 3 pools, 161 pgs objects: 523.51k objects, 299 GiB usage: 1014 GiB used, 836 GiB / 1.8 TiB avail pgs: 74666/1570530 objects degraded (4.754%) 1/523510 objects unfound (0.000%) 140 active+clean 20 active+undersized+degraded+remapped+backfill_toofull 1. active+recovery_unfound+undersized+degraded+remapped # ceph fs ls (output below): No filesystems enabled It looks like that we have a problem with the orchestrator now (we've lost cephadm orchestrator) and we also cannot see the filesystem. May you please assist since we are not able to mount the filesystem ? Thank you, Manolis Daramas Under the General Data Protection Regulation (GDPR) (EU) 2016/679, Motivian as Data Controller has a legal duty to protect any information collected from you via email. Information contained in this email and any attachments may be privileged or confidential and intended for the exclusive use of the original recipient. If you have received this email by mistake, please advise the sender immediately and delete the email, including emptying your deleted email box. Information included in this email is reserved to named addressee's eyes only. You may not share this message or any of its attachments to anyone. Please note that as the recipient, it is your responsibility to check the email for malicious software. Motivian puts the security of the client at a high priority. Therefore, we have put efforts into ensuring that the message is error and virus-free. Unfortunately, full security of the email cannot be ensured as, despite our efforts, the data included in emails could be infected, intercepted, or corrupted. Therefore, the recipient should check the email for threats with proper software, as the sender does not accept liability for any damage inflicted by viewing the content of this email.

2 5

MDS stuck in up:rejoin
by Eric Tittley 05 Dec '23

05 Dec '23

Hi all, For about a week our CephFS has experienced issues with its MDS. Currently the MDS is stuck in "up:rejoin" Issues become apparent when simple commands like "mv foo bar/" hung. I unmounted CephFS offline on the clients, evicted those remaining, and then issued ceph config set mds.0 mds_wipe_sessions true ceph config set mds.1 mds_wipe_sessions true which allowed me to delete the hung requests. I've lost the exact commands I used, but something like rados -p cephfs_metadata ls | grep mds rados rm -p cephfs_metadata mds0_openfiles.0 etc This allowed the MDS to get to "up:rejoin" where it has been stuck ever since which is getting on five days. # ceph mds stat cephfs:1/1 {0=cephfs.ceph00.uvlkrw=up:rejoin} 2 up:standby root@ceph00:/var/log/ceph/a614303a-5eb5-11ed-b492-011f01e12c9a# ceph -s cluster: id: a614303a-5eb5-11ed-b492-011f01e12c9a health: HEALTH_WARN 1 filesystem is degraded 1 pgs not deep-scrubbed in time 2 pool(s) do not have an application enabled 1 daemons have recently crashed services: mon: 3 daemons, quorum ceph00,ceph01,ceph02 (age 57m) mgr: ceph01.lvdgyr(active, since 2h), standbys: ceph00.gpwpgs mds: 1/1 daemons up, 2 standby osd: 91 osds: 90 up (since 78m), 90 in (since 112m) data: volumes: 0/1 healthy, 1 recovering pools: 5 pools, 1539 pgs objects: 138.83M objects, 485 TiB usage: 971 TiB used, 348 TiB / 1.3 PiB avail pgs: 1527 active+clean 12 active+clean+scrubbing+deep io: client: 3.1 MiB/s rd, 3.16k op/s rd, 0 op/s wr # ceph --version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) I've tried failing the MDS so it switches. Rebooted a couple of times. I've added more OSDs to the metadata pool and took one out as I thought it might be a bad metadata OSD (The "recently crashed" daemon). The error logs are full of (prefix to all are: Nov 27 14:02:44 ceph00 bash[2145]: debug 2023-11-27T14:02:44.619+0000 7f74e845e700 1 -- [v2:192.168.1.128:6800/2157301677,v1:192.168.1.128:6801/2157301677] --> [v2:192.168.1.133:6896/4289132926,v1:192.168.1.133:6897/4289132926] ) crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_message enqueueing message m=0x559be00adc00 type=42 osd_op(mds.0.36244:8142873 3.ff 3:ff5b34d6:::1.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e32465) v8 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).write_message sending message m=0x559be00adc00 seq=8142643 osd_op(mds.0.36244:8142873 3.ff 3:ff5b34d6:::1.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e32465) v8 crc :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message got 154 + 0 + 30 byte message. envelope type=43 src osd.89 off 0 crc :-1 s=READ_MESSAGE_COMPLETE pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message received message m=0x559be01f4480 seq=8142643 from=osd.89 type=43 osd_op_reply(8142873 1.00000000 [getxattr (30) out=30b] v0'0 uv560123 ondisk = 0) v8 osd_op_reply(8142873 1.00000000 [getxattr (30) out=30b] v0'0 uv560123 ondisk = 0) v8 ==== 154+0+30 (crc 0 0 0) 0x559be01f4480 con 0x559be00ad800 osd_op(unknown.0.36244:8142874 3.ff 3:ff5b34d6:::1.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e32465) v8 -- 0x559be2caec00 con 0x559be00ad800 Repeating multiple times a second (and filling /var) Prior to taking one of the cephfs_metadata OSDs offline, these came from communications from ceph00 to the node hosting the suspected bad OSD. Now they are between ceph00 and the host of the replacement metadata OSD. Does anyone have any suggestion on how to get the MDS to switch from "up:rejoin" to "up:active"? Is there any way to debug this, to determine what issue really is? I'm unable to interpret the debug log. Cheers, Eric ________________________________________________________ Dr Eric Tittley Research Computing Officer www.roe.ac.uk/~ert<http://www.roe.ac.uk/~ert> Institute for Astronomy Royal Observatory, Edinburgh The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

2 5

mds slow request with “failed to authpin, subtree is being exported"
by zxcs 05 Dec '23

05 Dec '23

HI, Experts, we are using cephfs with 16.2.* with multi active mds, and recently, we have two nodes mount with ceph-fuse due to the old os system. and one nodes run a python script with `glob.glob(path)`, and another client doing `cp` operation on the same path. then we see some log about `mds slow request`, and logs complain “failed to authpin, subtree is being exported" then need to restart mds, our question is, does there any dead lock? how can we avoid this and how to fix it without restart mds(it will influence other users) ? Thanks a ton! xz

6 11

the image used size becomes 0 after export/import with snapshot
by Tony Liu 05 Dec '23

05 Dec '23

Hi, I have an image with a snapshot and some changes after snapshot. ``` $ rbd du backup/f0408e1e-06b6-437b-a2b5-70e3751d0a26 NAME PROVISIONED USED f0408e1e-06b6-437b-a2b5-70e3751d0a26@snapshot-eb085877-7557-4620-9c01-c5587b857029 10 GiB 2.4 GiB f0408e1e-06b6-437b-a2b5-70e3751d0a26 10 GiB 2.4 GiB <TOTAL> 10 GiB 4.8 GiB ``` If there is no changes after snapshot, the image line will show 0 used. I did export and import. ``` $ rbd export --export-format 2 backup/f0408e1e-06b6-437b-a2b5-70e3751d0a26 - | rbd import --export-format 2 - backup/test Exporting image: 100% complete...done. Importing image: 100% complete...done. ``` When check the imported image, the image line shows 0 used. ``` $ rbd du backup/test NAME PROVISIONED USED test@snapshot-eb085877-7557-4620-9c01-c5587b857029 10 GiB 2.4 GiB test 10 GiB 0 B <TOTAL> 10 GiB 2.4 GiB ``` Any clues how that happened? I'd expect the same du as the source. I tried another quick test. It works fine. ``` $ rbd create backup/test-src --size 10G $ sudo rbd map backup/test-src /dev/rbd0 $ echo "hello" | sudo tee /dev/rbd0 hello $ rbd du backup/test-src NAME PROVISIONED USED test-src 10 GiB 4 MiB $ rbd snap create backup/test-src@snap-1 Creating snap: 100% complete...done. $ rbd du backup/test-src NAME PROVISIONED USED test-src@snap-1 10 GiB 4 MiB test-src 10 GiB 0 B <TOTAL> 10 GiB 4 MiB $ echo "world" | sudo tee /dev/rbd0 world $ rbd du backup/test-src NAME PROVISIONED USED test-src@snap-1 10 GiB 4 MiB test-src 10 GiB 4 MiB <TOTAL> 10 GiB 8 MiB $ rbd export --export-format 2 backup/test-src - | rbd import --export-format 2 - backup/test-dst Exporting image: 100% complete...done. Importing image: 100% complete...done. $ rbd du backup/test-dst NAME PROVISIONED USED test-dst@snap-1 10 GiB 4 MiB test-dst 10 GiB 4 MiB <TOTAL> 10 GiB 8 MiB ``` Thanks! Tony

2 2

Ceph/daemon container lvm tools don’t work
by Gaël THEROND 30 Nov '23

30 Nov '23

Is there anyone using containerized CEPH over CentOS Stream 9 Hosts already? I think there is a pretty big issue in here if CEPH images are built over CentOS but never tested against it.

1 0

Recommended architecture
by Francisco Arencibia Quesada 30 Nov '23

30 Nov '23

Hello again guys, Can you recommend me a book that explains best practices with Ceph, for example is it okay to have mon,mgr, osd in the same virtual machine, what is the recommended architecture according to your experience? Because by default is doing this: Cluster Ceph | +----------------------------+----------------------------+ | | | |10.0.0.52 |10.0.0.194 |10.0.0.229 +-----------+-----------+ +-----------+-----------+ +-----------+-----------+ |[node01.jotelulu.space]| [node02.jotelulu.space] |[node03.jotelulu.space]| | OSD +----+ OSD +----+ OSD | | Monitor Daemon | | Monitor Daemon | Monitor Daemon | | Manager Daemon | |Manager Daemon(standby) | | | +-----------------------+ +-----------------------+ +-----------------------+ -- Regards *Francisco Arencibia Quesada.* *DevOps Engineer*

3 2

Public/private network
by Albert Shih 30 Nov '23

30 Nov '23

Hi everyone. Status : Installing a ceph cluster Version : 17.2.7 Quincy OS : Debian 11. Each of my server got two ip address. One public and one private. When I'm trying to deploy my cluster with on a server server1 (the hostname) with cephadm bootstrap --mon-id hostname --mon-ip IP_PRIVATE --cluster-network PRIVATE_SUB I end up with private network for ceph config get mon public_network So I try to change it with ceph config set mon public_network PUBLIC_SUB still with lsof -i |grep -i listen I got ceph-mgr 31427 ceph 49u IPv4 119937 0t0 TCP server1-ceph.private.:7150 (LISTEN) node_expo 31572 nobody 3u IPv6 65495 0t0 TCP *:9100 (LISTEN) alertmana 31573 nobody 3u IPv6 21377 0t0 TCP *:9094 (LISTEN) alertmana 31573 nobody 8u IPv6 136298 0t0 TCP *:9093 (LISTEN) prometheu 31757 nobody 7u IPv6 109680 0t0 TCP *:9095 (LISTEN) grafana 31758 node-exporter 11u IPv6 100726 0t0 TCP *:3000 (LISTEN) ceph-mon 31850 ceph 27u IPv4 139664 0t0 TCP server1-ceph.private.:3300 (LISTEN) ceph-mon 31850 ceph 28u IPv4 139665 0t0 TCP server1-ceph.private.:6789 (LISTEN) So the ceph-mon listen on the private interface. Is this something normal ? Because according to https://access.redhat.com/documentation/fr-fr/red_hat_ceph_storage/5/html/c… only the OSD should listen on private network. Is they are anyway to configure booth public_network and private_network with cephadm bootstrap ? Regards. -- Albert SHIH 🦫 🐸 France Heure locale/Local time: jeu. 30 nov. 2023 18:27:08 CET

2 1

MDS_DAMAGE in 17.2.7 / Cannot delete affected files
by Sebastian Knust 30 Nov '23

30 Nov '23

Hi, After updating from 17.2.6 to 17.2.7 with cephadm, our cluster went into MDS_DAMAGE state. We had some prior issues with faulty kernel clients not releasing capabilities, therefore the update might just be a coincidence. `ceph tell mds.cephfs:0 damage ls` lists 56 affected files all with these general details: { "damage_type": "dentry", "id": 123456, "ino": 1234567890, "frag": "*", "dname": "some-filename.ext", "snap_id": "head", "path": "/full/path/to/file" } The behaviour upon trying to access file information in the (Kernel mounted) filesystem is a bit inconsistent. Generally, the first `stat` call seems to result in "Input/output error", the next call provides all `stat` data as expected from an undamaged file. The file can be read with `cat` with full and correct content (verified with backup) once the stat call succeeds. Scrubbing the affected subdirectories with `ceph tell mds.cephfs:0 scrub start /path/to/dir/ recursive,repair,force` does not fix the issue. Trying to delete the file results in an "Input/output error". If the stat calls beforehand succeeded, this also crashes the active MDS with these messages in the system journal: > Nov 24 14:21:15 iceph-18.servernet ceph-mds[1946861]: mds.0.cache.den(0x10012271195 DisplaySettings.json) newly corrupt dentry to be committed: [dentry #0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json [1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x10012271197 state=1073741824 | inodepin=1 0x56413e1e2780] > Nov 24 14:21:15 iceph-18.servernet ceph-mds[1946861]: log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json [1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x10012271197 state=1073741824 | inodepin=1 0x56413e1e2780] > Nov 24 14:21:15 iceph-18.servernet ceph-eafd0514-3644-11eb-bc6a-3cecef2330fa-mds-cephfs-iceph-18-ujfqnd[1946838]: 2023-11-24T13:21:15.654+0000 7f3fdcde0700 -1 mds.0.cache.den(0x10012271195 DisplaySettings.json) newly corrupt dentry to be committed: [dentry #0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json [1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x1001> > Nov 24 14:21:15 iceph-18.servernet ceph-eafd0514-3644-11eb-bc6a-3cecef2330fa-mds-cephfs-iceph-18-ujfqnd[1946838]: 2023-11-24T13:21:15.654+0000 7f3fdcde0700 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json [1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x10012> > Nov 24 14:21:15 iceph-18.servernet ceph-eafd0514-3644-11eb-bc6a-3cecef2330fa-mds-cephfs-iceph-18-ujfqnd[1946838]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f3fdcde0700 time 2023-11-24T13:21:15.655088+0000 > Nov 24 14:21:15 iceph-18.servernet ceph-mds[1946861]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f3fdcde0700 time 2023-11-24T13:21:15.655088+0000 > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDSRank.cc: 937: ceph_abort_msg("abort() called") > > ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) > 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xd7) [0x7f3fe5a1cb03] > 2: (MDSRank::abort(std::basic_string_view<char, std::char_traits<char> >)+0x7d) [0x5640f2e6fa2d] > 3: (CDentry::check_corruption(bool)+0x740) [0x5640f30e4820] > 4: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x5640f2f41877] > 5: (EOpen::add_clean_inode(CInode*)+0x121) [0x5640f2f49fc1] > 6: (Locker::adjust_cap_wanted(Capability*, int, int)+0x426) [0x5640f305e036] > 7: (Locker::process_request_cap_release(boost::intrusive_ptr<MDRequestImpl>&, client_t, ceph_mds_request_release const&, std::basic_string_view<char, std::char_traits<char> >)+0x599) [0x5640f307f7e9] > 8: (Server::handle_client_request(boost::intrusive_ptr<MClientRequest const> const&)+0xc06) [0x5640f2f2a7c6] > 9: (Server::dispatch(boost::intrusive_ptr<Message const> const&)+0x13c) [0x5640f2f2ef6c] > 10: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x5db) [0x5640f2e7727b] > 11: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x5c) [0x5640f2e778bc] > 12: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1bf) [0x5640f2e60c2f] > 13: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x478) [0x7f3fe5c97ed8] > 14: (DispatchQueue::entry()+0x50f) [0x7f3fe5c9531f] > 15: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3fe5d5f381] > 16: /lib64/libpthread.so.0(+0x81ca) [0x7f3fe4a0b1ca] > 17: clone() Deleting the file with cephfs-shell also does give Input/output error (5). Does anyone have an idea on how to proceed here? I am perfectly fine with loosing the affected files, they can all be easily restored from backup. Cheers Sebastian

3 5

rook-ceph RAW USE / DATA size difference reported after osd resize operation
by merp 30 Nov '23

30 Nov '23

Hi, I am set to resize OSDs in ceph cluster to extend overall cluster capacity, by adding 40GB to each of disk and noticed that after disk resize and OSD restart RAW USE size grows proportionally to new size, ex. by 20GB while DATA remains the same, which makes new space not readily available. Here is the osd output of cluster: kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 1 hdd 0.09769 1.00000 100 GiB 83 GiB 82 GiB 164 MiB 891 MiB 17 GiB 82.79 1.00 77 up 3 hdd 0.09769 1.00000 100 GiB 83 GiB 82 GiB 355 MiB 772 MiB 17 GiB 82.74 1.00 84 up 2 hdd 0.09769 1.00000 100 GiB 84 GiB 82 GiB 337 MiB 1.3 GiB 16 GiB 83.88 1.01 82 up 4 hdd 0.09769 1.00000 140 GiB 125 GiB 84 GiB 148 MiB 919 MiB 15 GiB 89.24 1.07 80 up 6 hdd 0.09769 1.00000 140 GiB 106 GiB 104 GiB 333 MiB 1015 MiB 34 GiB 75.47 0.91 107 up 7 hdd 0.09769 1.00000 140 GiB 118 GiB 97 GiB 351 MiB 1.2 GiB 22 GiB 84.48 1.02 101 up TOTAL 720 GiB 598 GiB 531 GiB 1.6 GiB 6.1 GiB 122 GiB 83.10 MIN/MAX VAR: 0.91/1.07 STDDEV: 4.06 The OSDs I managed to extend are 7, 6 and 4. Only OSD number 6 detected new size and did not inflate RAW USE, OSD 7 and 4 have RAW USE vs DATA gap. kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 720 GiB 122 GiB 598 GiB 598 GiB 83.10 TOTAL 720 GiB 122 GiB 598 GiB 598 GiB 83.10 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 449 KiB 2 1.3 MiB 0 16 GiB cephfs-metadata 2 16 832 MiB 245.62k 2.4 GiB 4.80 16 GiB cephfs-replicated 3 128 176 GiB 545.23k 530 GiB 91.63 16 GiB replicapool 4 32 19 B 2 12 KiB 0 16 GiB This reports nearly 600GB used, while it should be more like 530GB as cephfs-replicated pool is reporting its data usage. Any ideas why is this happening? Should I continue with extension of all OSDs to 140GB to see if that makes a difference? Br, merp.

1 0

error deploying ceph
by Francisco Arencibia Quesada 30 Nov '23

30 Nov '23

Hello guys, This situation is driving me crazy, I have tried to deploy a ceph cluster, in all ways possible, even with ansible and at some point it breaks. I'm using Ubuntu 22.0.4. This is one of the errors I'm having, some problem with ceph-exporter. Please could you help me, I have been dealing with this for like 5 days. Kind regards root@node1-ceph:~# cephadm bootstrap --mon-ip 10.0.0.52 Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit systemd-timesyncd.service is enabled and running Repeating the final host check... docker (/usr/bin/docker) is present systemctl is present lvcreate is present Unit systemd-timesyncd.service is enabled and running Host looks OK Cluster fsid: 4ce3a92a-8ddd-11ee-9b23-6341187f70c1 Verifying IP 10.0.0.52 port 3300 ... Verifying IP 10.0.0.52 port 6789 ... Mon IP `10.0.0.52` is in CIDR network `10.0.0.0/24` Mon IP `10.0.0.52` is in CIDR network `10.0.0.0/24` Mon IP `10.0.0.52` is in CIDR network `10.0.0.1/32` Mon IP `10.0.0.52` is in CIDR network `10.0.0.1/32` Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network Pulling container image quay.io/ceph/ceph:v17... Ceph version: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) Extracting ceph user uid/gid from container image... Creating initial keys... Creating initial monmap... Creating mon... Waiting for mon to start... Waiting for mon... mon is available Assimilating anything we can from ceph.conf... Generating new minimal ceph.conf... Restarting the monitor... Setting mon public_network to 10.0.0.1/32,10.0.0.0/24 Wrote config to /etc/ceph/ceph.conf Wrote keyring to /etc/ceph/ceph.client.admin.keyring Creating mgr... Verifying port 9283 ... Waiting for mgr to start... Waiting for mgr... mgr not available, waiting (1/15)... mgr not available, waiting (2/15)... mgr not available, waiting (3/15)... mgr not available, waiting (4/15)... mgr not available, waiting (5/15)... mgr is available Enabling cephadm module... Waiting for the mgr to restart... Waiting for mgr epoch 5... mgr epoch 5 is available Setting orchestrator backend to cephadm... Generating ssh key... Wrote public SSH key to /etc/ceph/ceph.pub Adding key to root@localhost authorized_keys... Adding host node1-ceph... Deploying mon service with default placement... Deploying mgr service with default placement... Deploying crash service with default placement... Deploying ceph-exporter service with default placement... Non-zero exit code 22 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v17 -e NODE_NAME=node1-ceph -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4ce3a92a-8ddd-11ee-9b23-6341187f70c1:/var/log/ceph:z -v /tmp/ceph-tmp6yz3vt5s:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfhd01qwu:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v17 orch apply ceph-exporter /usr/bin/ceph: stderr Error EINVAL: Usage: /usr/bin/ceph: stderr ceph orch apply -i <yaml spec> [--dry-run] /usr/bin/ceph: stderr ceph orch apply <service_type> [--placement=<placement_string>] [--unmanaged] /usr/bin/ceph: stderr Traceback (most recent call last): File "/usr/sbin/cephadm", line 9653, in <module> main() File "/usr/sbin/cephadm", line 9641, in main r = ctx.func(ctx) File "/usr/sbin/cephadm", line 2205, in _default_image return func(ctx) File "/usr/sbin/cephadm", line 5774, in command_bootstrap prepare_ssh(ctx, cli, wait_for_mgr_restart) File "/usr/sbin/cephadm", line 5275, in prepare_ssh cli(['orch', 'apply', t]) File "/usr/sbin/cephadm", line 5708, in cli return CephContainer( File "/usr/sbin/cephadm", line 4144, in run out, _, _ = call_throws(self.ctx, self.run_cmd(), File "/usr/sbin/cephadm", line 1853, in call_throws raise RuntimeError('Failed command: %s' % ' '.join(command)) RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v17 -e NODE_NAME=node1-ceph -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4ce3a92a-8ddd-11ee-9b23-6341187f70c1:/var/log/ceph:z -v /tmp/ceph-tmp6yz3vt5s:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfhd01qwu:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v17 orch apply ceph-exporter -- *Francisco Arencibia Quesada.* *DevOps Engineer*

3 6

Re: Where is a simple getting started guide for a very basic cluster?
by Leo28C 30 Nov '23

30 Nov '23

The system has been running for years and it would take me weeks to reconfigure all of its services properly. I'm proceeding with just the Debian nodes for now; worst case scenario I'll just have to rent a new server to act as a third node. I was really hoping to use the CentOS one since it's already there. Somebody else mentioned DRBD earlier so I've looked into that too, but in that case I ran into problems with the Debian systems instead because apparently only DRBD 9 and up supports more than two nodes, and the only Debian kernel module packages I've found are for DRDB 8. Ironically, CentOS does have a working kmod-drbd90 package available. On Wed, Nov 29, 2023 at 5:53 PM Anthony D'Atri <aad(a)dreamsnake.net> wrote: > Update it to 8 or Rocky 9? > > > On Nov 29, 2023, at 14:05, Leo28C <leo28c(a)gmail.com> wrote: > > > >> Why are you talking about version 14 now anyhow? > > > > One of my nodes is running CentOS 7 and the latest version I found for it > > is 14, unless there's a way to get 15 working on it that I don't know > about? > > > > On Wed, Nov 29, 2023 at 1:48 AM Robert Sander < > r.sander(a)heinlein-support.de> > > wrote: > > > >> On 11/28/23 17:50, Leo28C wrote: > >>> Problem is I don't have the cephadm command on every node. Do I need it > >>> on all nodes or just one of them? I tried installing it via curl but my > >>> ceph version is 14.2.22 which is not on the archive anymore so the curl > >>> command returns a 404 error html file. How do I get cephadm for 14.2? > >> > >> There is no cephadm for Ceph 14 as the orchestrator was first introduced > >> in version 15. > >> > >> Why are you talking about version 14 now anyhow? > >> As long as your nodes fulfill the requirements for cephadm you can > >> install the latest version of Ceph. > >> > >> PS: Please reply to the list. > >> > >> Regards > >> -- > >> Robert Sander > >> Heinlein Consulting GmbH > >> Schwedter Str. 8/9b, 10119 Berlin > >> > >> https://www.heinlein-support.de > >> > >> Tel: 030 / 405051-43 > >> Fax: 030 / 405051-19 > >> > >> Amtsgericht Berlin-Charlottenburg - HRB 220009 B > >> Geschäftsführer: Peer Heinlein - Sitz: Berlin > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Uploading file from admin to other users bucket in multi tenant mode
by Rok Jaklič 29 Nov '23

29 Nov '23

Hi, I have set following permission to admin user: radosgw-admin caps add --uid=admin --tenant=admin --caps="users=*;buckets=*" Now I would like to upload some object with admin user to some other user/tenant (tester1$tester1) to his bucket test1. Other user has uid tester1 and tenant tester1 and bucket test1 created. I've tried with python: headers = {'x-amz-meta-tenancy': 'tester1'} client.upload_file(file_path, bucket_name, object_name, ExtraArgs={'Metadata': headers}) But I get response: An error occurred (NoSuchBucket) when calling the PutObject operation: None Any ideas why I get this error even though bucket test1 for tester1$tester1 exists? Kind regards, Rok

1 0

Re: Where is a simple getting started guide for a very basic cluster?
by Robert Sander 29 Nov '23

29 Nov '23

On 11/28/23 17:50, Leo28C wrote: > Problem is I don't have the cephadm command on every node. Do I need it > on all nodes or just one of them? I tried installing it via curl but my > ceph version is 14.2.22 which is not on the archive anymore so the curl > command returns a 404 error html file. How do I get cephadm for 14.2? There is no cephadm for Ceph 14 as the orchestrator was first introduced in version 15. Why are you talking about version 14 now anyhow? As long as your nodes fulfill the requirements for cephadm you can install the latest version of Ceph. PS: Please reply to the list. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

2 1

Bucket/object create/update/delete notification
by Rok Jaklič 29 Nov '23

29 Nov '23

Hi, I would like to get info if the bucket or object got updated. I can get this info with a changed etag of an object, but not I cannot get etag from bucket, so I am looking at https://docs.ceph.com/en/latest/radosgw/notifications/ How do I create a topic and where do I send request with parameters? Anyone have an example (cli, python, ...) with topic creation where I can get notification to a http endpoint when an object got created in a bucket? Kind regards, Rok

2 2

Re: openstack Vm shutoff by itself
by AJ_ sunny 29 Nov '23

29 Nov '23

Hi Eugen, Please find the details below FROM HYPERVISOR SYSLOG:- Nov 29 07:07:46 kernel: [ 1171.392249] IPv6: ADDRCONF(NETDEV_CHANGE): qvoc178e343-d8: link becomes ready Nov 29 07:07:46 kernel: [ 1171.392460] IPv6: ADDRCONF(NETDEV_CHANGE): qvbc178e343-d8: link becomes ready Nov 29 07:07:46 kernel: [ 1171.397266] qbrc178e343-d8: port 1(qvbc178e343-d8) entered blocking state Nov 29 07:07:46kernel: [ 1171.397268] qbrc178e343-d8: port 1(qvbc178e343-d8) entered disabled state Nov 29 07:07:46 kernel: [ 1171.397414] device qvbc178e343-d8 entered promiscuous mode FROM DMESG LOG:- [Wed Nov 29 07:07:45 2023] qbrc178e343-d8: port 1(qvbc178e343-d8) entered disabled state [Wed Nov 29 07:07:45 2023] device qvbc178e343-d8 entered promiscuous mode [Wed Nov 29 07:07:45 2023] qbrc178e343-d8: port 1(qvbc178e343-d8) entered blocking state [Wed Nov 29 07:07:45 2023] qbrc178e343-d8: port 1(qvbc178e343-d8) entered forwarding state [Wed Nov 29 07:07:45 2023] device qvoc178e343-d8 entered promiscuous mode [Wed Nov 29 07:07:49 2023] qbrc178e343-d8: port 2(tapc178e343-d8) entered blocking state [Wed Nov 29 07:07:49 2023] qbrc178e343-d8: port 2(tapc178e343-d8) entered disabled state [Wed Nov 29 07:07:49 2023] device tapc178e343-d8 entered promiscuous mode [Wed Nov 29 07:07:49 2023] qbrc178e343-d8: port 2(tapc178e343-d8) entered blocking state [Wed Nov 29 07:07:49 2023] qbrc178e343-d8: port 2(tapc178e343-d8) entered forwarding state NOVA-COMPUTE LOG:- 2023-11-29 00:38:31.027 7 INFO nova.compute.manager [-] [instance: 1dbd1562-44c1-44b7-9d1e-97ac61716db3] VM Stopped (Lifecycle Event) 2023-11-29 00:38:31.115 7 INFO nova.compute.manager [req-cda5058f-c026-4586-a2b3-50f9727f1220 - - - - -] [instance: 1dbd1562-44c1-44b7-9d1e-97 ac61716db3] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power _state in the DB to match the hypervisor. 2023-11-29 00:38:34.045 7 INFO nova.compute.manager [-] [instance: 46b48b4e-3675-453c-8c87-f21f1b7fb86c] VM Stopped (Lifecycle Event) 2023-11-29 00:38:34.080 7 INFO nova.compute.manager [-] [instance: b3df30a6-de61-448e-8451-938309b20ab5] VM Stopped (Lifecycle Event) 2023-11-29 00:38:34.122 7 INFO nova.compute.manager [req-a6d0b47f-f50b-4bbc-a0d7-df49511ca4a7 - - - - -] [instance: 46b48b4e-3675-453c-8c87-f2 1f1b7fb86c] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power _state in the DB to match the hypervisor. 2023-11-29 00:38:34.164 7 INFO nova.compute.manager [req-7aab66e0-73f6-4ae1-960d-2f24dfba3131 - - - - -] [instance: b3df30a6-de61-448e-8451-93 8309b20ab5] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power _state in the DB to match the hypervisor. There are multiple virtual machine went down on different hypervisor And also os disk resides on ceph storage During this incident if directly restart the machine then we will get I/O error on console so first we have to rebuild the os-disk volume from ceph then we have restart the machine for proper functioning of VM I checked instance console logs also but nothing found suspicious Thanks & Regards Arihant Jain On Mon, Nov 27, 2023 at 7:49 PM Eugen Block <eblock(a)nde.ag> wrote: > I don't see how ceph could be the issue here. Do you have libvirt logs > and dmesg output from the hypervisor and something from the VM like > the relevant syslog excerpts (before it gets shut down)? Is only one > VM affected or all from the same hypervisor or several across > different hypervisors? > The nova-compute.log doesn't seem to be enough, but you could also > enable debug logs to see if it reveals more. > > Zitat von AJ_ sunny <jains8550(a)gmail.com>: > > > Hi team, > > > > After doing above changes I am still getting the issue in which machine > > continuously went shutdown > > > > In nova-compute logs I am getting only this footprint > > > > Logs:- > > 2023-10-16 08:48:10.971 7 WARNING nova.compute.manager > > [req-c7b731db-2b61-400e-917f-8645c9984696 f226d81a45dd46488fb2e19515 848 > > 316d215042914de190f5f9e1c8466bf0 default default] [instance: > > 4b04d3f1-1fbd-4b63-b693-a0ef316ecff3] Received unexpected - vent > > network-vif-plugged-f191f6c8-dff5-4c1b-94b3-8d91aa6ff5ac for instance > with > > vm_state active and task_state None. 2023-10-21 22:42:44.589 7 INFO > > nova.compute.manager [-] [instance: 4b04d3f1-1fbd-4b63-b693-a0ef316ecff3] > > VM Stopped (Lifecyc Event) > > > > 2023-10-21 22:42:44.683 7 INFO nova.compute.manager > > [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d -] [instance: 4b04d3f1- > > fbd-4b63-b693-a0ef316ecff3] During _sync_instance_power_state the DB > > power_state (1) does not match the vm_power_state from ti e hypervisor > (4). > > Updating power_state in the DB to match the hypervisor. > > > > 2023-10-21 22:42:44.811 7 WARNING nova.compute.manager > > [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d ----] [instance: 4b04d3f > > 1-1fbd-4b63-b693-a0ef316ecff3] Instance shutdown by itself. Calling the > > stop API. Current vm_state: active, current task_state : None, original > DB > > power_state: 1, current VM power_state: 4 2023-10-21 22:42:44.977 7 INFO > > nova.compute.manager [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d -] > > [instance: 4b04d3f1-1 > > > > fbd-4b63-b693-a0ef316ecff3] Instance is already powered off in the > > hypervisor when stop is called. > > > > > > And in this architecture we are using ceph is the backend storage for > > Nova,glance & cinder > > When machine auto goes down and if i try to start the machine it will go > in > > error i.e. in Vm console is show I/O ERROR during boot so first we need > to > > rebuild the volume from ceph side then I have to start the machine > > Rbd object-map rebuild<volume-id> > > Openstack server start <server-id> > > > > So this issue is showing two faces one from ceph side and another from > > nova-compute log > > can someone please help me out to fix out this issue asap > > > > Thanks & Regards > > Arihant Jain > > > > On Tue, 24 Oct, 2023, 4:56 pm , <smooney(a)redhat.com> wrote: > > > >> On Tue, 2023-10-24 at 10:11 +0530, AJ_ sunny wrote: > >> > Hi team, > >> > > >> > Vm is not shutting off by owner from inside its automatically went to > >> > shutdown i.e. libvirt lifecycle stop event triggering > >> > In my nova.conf configuration I am using ram_allocation_ratio = 1.5 > >> > And previously I tried to set in nova.conf > >> > Sync_power_state_interval = -1 but still facing the same problem > >> > OOM might be causing this issue > >> > Can you please give me some idea to fix this issue if OOM is the cause > >> the general answer is swap. > >> > >> nova should alwasy be deployed with swap even if you do not have over > >> commit enabled. > >> there are a few reason for this the first being python allocates memory > >> diffently if > >> any swap is aviable, even 1G is enough to have it not try to commit all > >> memory. so > >> when swap is aviable the nova/neutron agents will use much less resident > >> memeory even with > >> out usign any of the swap space. > >> > >> we have some docs about this downstream > >> > >> > https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/17… > >> > >> if you are being ultra conservative we recommend allocating (ram * > >> allocation ratio) in swap so in your case allcoate > >> 1.5 times your ram as swap. we woudl expect the actul useage of swap to > be > >> a small fraction of that however so we > >> also provide a formula for > >> > >> overcommit_ratio = NovaRAMAllocationRatio - 1 > >> Minimum swap size (MB) = (total_RAM * overcommit_ratio) + > RHEL_min_swap > >> Recommended swap size (MB) = total_RAM * (overcommit_ratio + > >> percentage_of_RAM_to_use_for_swap) > >> > >> so say your host had 64G of ram with an allocation ratio of 1.5 and a > min > >> swap percentaiong of 25% > >> the conserviver swap recommentation would be > >> > >> (64*(0.5+0.25)) + disto_min_swap > >> (64*0.75) + 4G = 52G of recommended swap > >> > >> if your wondering why we add a min swap precentage and disto min swap > its > >> basically to acocund for the > >> Qemu and host OS memory overhead as well as the memory used by the > >> nova/neutron agents and libvirt/ovs > >> > >> > >> if your not using memory over commit my general recommdation is if you > >> have less then 64G of ram allcoate 16G if you > >> have more then 256G of ram allocate 64G and you should be fine. when you > >> do use memofy over commit you must > >> have at least enouch swap to account for the qemu overhead of all > instance > >> + the over committed memory. > >> > >> > >> the other common cause of OOM errors is if you are using numa affinity > and > >> the guest dont request > >> hw:mem_page_size=<something> without setting a mem_page_size request we > >> dont do numa aware memory placement. the kernel > >> OOM system works > >> on a per numa node basis, numa affintiy does not supprot memory over > >> commit either so that is likly not your issue. > >> i jsut said i woudl mention it to cover all basis. > >> > >> regards > >> sean > >> > >> > >> > >> > > >> > > >> > Thanks & Regards > >> > Arihant Jain > >> > > >> > On Mon, 23 Oct, 2023, 11:29 pm , <smooney(a)redhat.com> wrote: > >> > > >> > > On Mon, 2023-10-23 at 13:19 -0400, Jonathan Proulx wrote: > >> > > > > >> > > > I've seen similar log traces with overcommitted memory when the > >> > > > hypervisor runs out of physical memory and OOM killer gets the VM > >> > > > process. > >> > > > > >> > > > This is an unusuall configuration (I think) but if the VM owner > >> claims > >> > > > they didn't power down the VM internally you might look at the > local > >> > > > hypevisor logs to see if the VM process crashed or was killed for > >> some > >> > > > other reason. > >> > > yep OOM events are one common causes fo this. > >> > > > >> > > nova is bacialy just saying "hay you said this vm should be active > but > >> its > >> > > not, im going to update the db to reflect > >> > > reality." you can turn that off with > >> > > > >> > > > >> > https://docs.openstack.org/nova/latest/configuration/config.html#workaround… > >> > > or > >> > > > >> > > > >> > https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.sy… > >> > > either disabel the sync via setign the interval to -1 > >> > > or disable haneling the virt lifecycle events. > >> > > > >> > > i would recommend the sync_power_state_interval approach but again > if > >> vms > >> > > are stopping > >> > > and you dont know why you likely should discover why rahter then > just > >> > > turning if the update of the nova db to reflect > >> > > the actual sate. > >> > > > >> > > > > >> > > > -Jon > >> > > > > >> > > > On Mon, Oct 23, 2023 at 02:02:26PM +0100, smooney(a)redhat.com > wrote: > >> > > > :On Mon, 2023-10-23 at 17:45 +0530, AJ_ sunny wrote: > >> > > > :> Hi team, > >> > > > :> > >> > > > :> I am using openstack kolla ansible on wallaby version and > >> currently I > >> > > am > >> > > > :> facing issue with virtual machine, vm is shutoff by itself and > and > >> > > from log > >> > > > :> it seems libvirt lifecycle stop event triggering again and > again > >> > > > :> > >> > > > :> Logs:- > >> > > > :> 2023-10-16 08:48:10.971 7 WARNING nova.compute.manager > >> > > > :> [req-c7b731db-2b61-400e-917f-8645c9984696 > >> f226d81a45dd46488fb2e19515 > >> > > 848 > >> > > > :> 316d215042914de190f5f9e1c8466bf0 default default] [instance: > >> > > > :> 4b04d3f1-1fbd-4b63-b693-a0ef316ecff3] Received unexpected - > vent > >> > > > :> network-vif-plugged-f191f6c8-dff5-4c1b-94b3-8d91aa6ff5ac for > >> instance > >> > > with > >> > > > :> vm_state active and task_state None. 2023-10-21 22:42:44.589 7 > >> INFO > >> > > > :> nova.compute.manager [-] [instance: > >> > > 4b04d3f1-1fbd-4b63-b693-a0ef316ecff3] > >> > > > :> VM Stopped (Lifecyc Event) > >> > > > :> > >> > > > :> 2023-10-21 22:42:44.683 7 INFO nova.compute.manager > >> > > > :> [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d -] [instance: > 4b04d3f1- > >> > > > :> fbd-4b63-b693-a0ef316ecff3] During _sync_instance_power_state > the > >> DB > >> > > > :> power_state (1) does not match the vm_power_state from ti e > >> > > hypervisor (4). > >> > > > :> Updating power_state in the DB to match the hypervisor. > >> > > > :> > >> > > > :> 2023-10-21 22:42:44.811 7 WARNING nova.compute.manager > >> > > > :> [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d ----] [instance: > 4b04d3f > >> > > > :> 1-1fbd-4b63-b693-a0ef316ecff3] Instance shutdown by itself. > >> Calling > >> > > the > >> > > > :> stop API. Current vm_state: active, current task_state : None, > >> > > original DB > >> > > > :> power_state: 1, current VM power_state: 4 2023-10-21 > 22:42:44.977 > >> 7 > >> > > INFO > >> > > > :> nova.compute.manager [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d > -] > >> > > > :> [instance: 4b04d3f1-1 > >> > > > :> > >> > > > :> fbd-4b63-b693-a0ef316ecff3] Instance is already powered off in > the > >> > > > :> hypervisor when stop is called. > >> > > > : > >> > > > :that sounds like the guest os shutdown the vm. > >> > > > :i.e. somethign in the guest ran sudo poweroff > >> > > > :then nova detected teh vm was stoped by the user and updated its > db > >> to > >> > > match > >> > > > :that. > >> > > > : > >> > > > :that is the expected beahvior wehn you have the power sync > enabled. > >> > > > :it is enabled by default. > >> > > > :> > >> > > > :> > >> > > > :> Thanks & Regards > >> > > > :> Arihant Jain > >> > > > :> +91 8299719369 > >> > > > : > >> > > > > >> > > > >> > > > >> > >> > > > >

1 0

Re: Best Practice for OSD Balancing
by Anthony D'Atri 29 Nov '23

29 Nov '23

>> >> 1) They’re client aka desktop SSDs, not “enterprise” >> 2) They’re a partition of a larger OSD shared with other purposes > > Yup. They're a mix of SATA SSDs and NVMes, but everything is > consumer-grade. They're only 10% full on average and I'm not > super-concerned with performance. If they did get full I'd allocate > more space for them. Performance is more than adequate for the very > light loads they have. Fair enough. We sometimes see people bringing a toothpick to a gun fight and expecting a different result, so I had to ask. Just keep an eye on their endurance burn. > > > It is interesting because Quincy had no issues with the autoscaler > with the exact same cluster config. It might be a Rook issue, or it > might just be because so many PGs are remapped. I'll take another > look at that once it reaches more of a steady state. > > In any case, if the balancer is designed more for equal-sized OSDs I > can always just play with reweights to balance things. Look into the JJ balancer, I’ve read good things about it. > > -- > Rich

1 1

Re: Best Practice for OSD Balancing
by Anthony D'Atri 29 Nov '23

29 Nov '23

>> Very small and/or non-uniform clusters can be corner cases for many things, especially if they don’t have enough PGs. What is your failure domain — host or OSD? > > Failure domain is host, Your host buckets do vary in weight by roughly a factor of two. They naturally will get PGs more or less relative to their aggregate CRUSH weight, and thus also the OSDs on each host. > and PG number should be fairly reasonable. Reason is in the eye of the beholder. I make the PG ratio for the cluster as a whole to be ~~90. I would definitely add more, that should help. >> Are your OSDs sized uniformly? Please send the output of the following commands: > > OSDs are definitely not uniform in size. This might be the issue with > the automation. > > You asked for it, but I do apologize for the wall of text that follows... You described a small cluster, so this is peanuts. >> `ceph osd tree` > > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > -1 131.65762 root default > -25 16.46977 host k8s1 > 14 hdd 5.45799 osd.14 up 0.90002 1.00000 > 19 hdd 10.91409 osd.19 up 1.00000 1.00000 > 22 ssd 0.09769 osd.22 up 1.00000 1.00000 > -13 25.56458 host k8s3 > 2 hdd 10.91409 osd.2 up 0.84998 1.00000 > 3 hdd 1.81940 osd.3 up 0.75002 1.00000 > 20 hdd 12.73340 osd.20 up 1.00000 1.00000 > 10 ssd 0.09769 osd.10 up 1.00000 1.00000 > -14 12.83107 host k8s4 > 0 hdd 10.91399 osd.0 up 1.00000 1.00000 > 5 hdd 1.81940 osd.5 up 1.00000 1.00000 > 11 ssd 0.09769 osd.11 up 1.00000 1.00000 > -2 14.65048 host k8s5 > 1 hdd 1.81940 osd.1 up 0.70001 1.00000 > 17 hdd 12.73340 osd.17 up 1.00000 1.00000 > 12 ssd 0.09769 osd.12 up 1.00000 1.00000 > -6 14.65048 host k8s6 > 4 hdd 1.81940 osd.4 up 0.75000 1.00000 > 16 hdd 12.73340 osd.16 up 0.95001 1.00000 > 13 ssd 0.09769 osd.13 up 1.00000 1.00000 > -3 23.74518 host k8s7 > 6 hdd 12.73340 osd.6 up 1.00000 1.00000 > 15 hdd 10.91409 osd.15 up 0.95001 1.00000 > 8 ssd 0.09769 osd.8 up 1.00000 1.00000 > -9 23.74606 host k8s8 > 7 hdd 14.55269 osd.7 up 1.00000 1.00000 > 18 hdd 9.09569 osd.18 up 1.00000 1.00000 > 9 ssd 0.09769 osd.9 up 1.00000 1.00000 Looks like one 100GB SSD OSD per host? This is AIUI the screaming minimum size for an OSD. With WAL, DB, cluster maps, and other overhead there doesn’t end up being much space left for payload data. On larger OSDs the overhead is much more into the noise floor. Given the side of these SSD OSDs, I suspect at least one of the following is true? 1) They’re client aka desktop SSDs, not “enterprise” 2) They’re a partition of a larger OSD shared with other purposes I suspect that this alone would be enough to frustrate the balancer, which AFAIK doesn’t take overhead into consideration. You might disable the balancer module, reset the reweights to 1.00, and try the JJ balancer but I dunno that it would be night vs day. > Note this cluster is in the middle of re-creating all the OSDs to > modify the OSD allocation size min_alloc_size? Were they created on an older Ceph release? Current defaults for [non]rotational media are both 4KB; they used to be 64KB but were changed with some churn …. around the Pacific / Octopus era IIRC. If you’re re-creating to minimize space amp, does that mean you’re running RGW with a significant fraction of small objects? With RBD — or CephFS with larger files — that isn’t so much an issue. > I have scrubbing disabled since I'm > basically rewriting just about everything in the cluster weekly right > now but normally that would be on. > > cluster: > id: ba455d73-116e-4f24-8a34-a45e3ba9f44c > health: HEALTH_WARN > noscrub,nodeep-scrub flag(s) set > 546 pgs not deep-scrubbed in time > 542 pgs not scrubbed in time > > services: > mon: 3 daemons, quorum e,f,g (age 7d) > mgr: a(active, since 7d) > mds: 1/1 daemons up, 1 hot standby > osd: 22 osds: 22 up (since 5h), 22 in (since 33h); 101 remapped pgs > flags noscrub,nodeep-scrub > rgw: 1 daemon active (1 hosts, 1 zones) > > data: > volumes: 1/1 healthy > pools: 13 pools, 617 pgs > objects: 9.36M objects, 33 TiB > usage: 67 TiB used, 65 TiB / 132 TiB avail > pgs: 1778936/21708668 objects misplaced (8.195%) > 516 active+clean > 100 active+remapped+backfill_wait > 1 active+remapped+backfilling > > io: > client: 371 KiB/s rd, 2.8 MiB/s wr, 2 op/s rd, 7 op/s wr > recovery: 25 MiB/s, 6 objects/s > > progress: > Global Recovery Event (7d) > [=======================.....] (remaining: 36h) > >> `ceph osd df` > > Note that these are not in a steady state right now. OSD 6 in > particular was just re-created and is repopulating. A few of the > reweights were set to deal with some gross issues in balance - when it > all settles down I plan to optimize them. > > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS > 14 hdd 5.45799 0.90002 5.5 TiB 3.0 TiB 3.0 TiB 2.0 MiB 11 GiB 2.4 TiB 55.51 1.09 72 up > 19 hdd 10.91409 1.00000 11 TiB 6.2 TiB 6.2 TiB 3.1 MiB 16 GiB 4.7 TiB 57.12 1.12 144 up Unless you were to carefully segregate larger and smaller HDDs into separate pools, right-sizing the PG could is tricky. 144 is okay, 72 is a bit low, upstream guidance notwithstanding. I would still bump some of your pg_nums a bit. > 22 ssd 0.09769 1.00000 100 GiB 2.4 GiB 1.8 GiB 167 MiB 504 MiB 98 GiB 2.43 0.05 32 up > 2 hdd 10.91409 0.84998 11 TiB 4.5 TiB 4.5 TiB 5.0 MiB 9.7 GiB 6.4 TiB 41.11 0.81 99 up > 3 hdd 1.81940 0.75002 1.8 TiB 1.0 TiB 1.0 TiB 2.3 MiB 3.8 GiB 818 GiB 56.11 1.10 21 up > 20 hdd 12.73340 1.00000 13 TiB 7.1 TiB 7.1 TiB 3.7 MiB 16 GiB 5.6 TiB 56.01 1.10 165 up > 10 ssd 0.09769 1.00000 100 GiB 1.3 GiB 299 MiB 185 MiB 835 MiB 99 GiB 1.29 0.03 38 up > 0 hdd 10.91399 1.00000 11 TiB 6.5 TiB 6.5 TiB 3.7 MiB 15 GiB 4.4 TiB 59.41 1.17 144 up > 5 hdd 1.81940 1.00000 1.8 TiB 845 GiB 842 GiB 1.7 MiB 3.3 GiB 1018 GiB 45.36 0.89 23 up > 11 ssd 0.09769 1.00000 100 GiB 3.1 GiB 1.3 GiB 157 MiB 1.6.GiB 97 GiB 3.09 0.06 33 up > 1 hdd 1.81940 0.70001 1.8 TiB 983 GiB 979 GiB 1.3 MiB 3.4. GiB 880 GiB 52.76 1.04 26 up > 17 hdd 12.73340 1.00000 13 TiB 7.3 TiB 7.2 TiB 3.6 MiB 15 GiB 5.5 TiB 56.95 1.12 159 up > 12 ssd 0.09769 1.00000 100 GiB 1.5 GiB 120 MiB 55 MiB 1.3 GiB 99 GiB 1.49 0.03 21 up > 4 hdd 1.81940 0.75000 1.8 TiB 1.0 TiB 1.0 TiB 2.5 MiB 3.0 GiB 820 GiB 55.98 1.10 24 up > 16 hdd 12.73340 0.95001 13 TiB 7.6 TiB 7.5 TiB 7.9 MiB 16 GiB 5.2 TiB 59.32 1.17 171 up > 13 ssd 0.09769 1.00000 100 GiB 2.4 GiB 528 MiB 196 MiB 1.7 GiB 98 GiB 2.38 0.05 33 up > 6 hdd 12.73340 1.00000 13 TiB 1.7 TiB 1.7 TiB 1.3 MiB 4.5 GiB 11 TiB 13.66 0.27 48 up > 15 hdd 10.91409 0.95001 11 TiB 6.5 TiB 6.5 TiB 5.2 MiB 13 GiB 4.4 TiB 59.42 1.17 155 up > 8 ssd 0.09769 1.00000 100 GiB 1.9 GiB 1.1 GiB 116 MiB 788 MiB 98 GiB 1.95 0.04 26 up > 7 hdd 14.55269 1.00000 15 TiB 7.8 TiB 7.7 TiB 3.9 MiB 16 GiB 6.8 TiB 53.32 1.05 172 up > 18 hdd 9.09569 1.00000 9.1 TiB 4.9 TiB 4.9 TiB 3.9 MiB 11 GiB 4.2 TiB 53.96 1.06 109 up > 9 ssd 0.09769 1.00000 100 GiB 2.2 GiB 391 MiB 264 MiB 1.6 GiB 98 GiB 2.25 0.04 40 up > TOTAL 132 TiB 67 TiB 67 TiB 1.2 GiB 164 GiB 65 TiB 50.82 > MIN/MAX VAR: 0.03/1.17 STDDEV: 29.78 > > >> `ceph osd dump | grep pool` > > pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 7 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on pg_num_max 32 pg_num_min 1 application mgr Check the CRUSH rule for this pool. On my clusters Rook creates it without specifying a device-class, but the other pools get rules that do specify a device class. By way of the shadow CRUSH topology, this sort of looks like multiple CRUSH roots to the pg_autoscaler, which is why you have no output from the status below. I added a bit to the docs earlier this year to call this out. Perhaps the Rook folks on the list might have thoughts about preventing this situation, I don’t recall if I created a github issue for it. That said, I’m personally not a fan of the pg autoscaler and tend to disable it. ymmv. Unless you enable the “bulk” option, it may well be that you have too few PGs for effective bin packing. Think about filling a 55 gal drum with beach balls vs with golf balls. So many pools for such a small cluster …. are you actively using CephFS, RBD, *and* RGW? If not, I’d suggest removing whatever you aren’t using and adjusting pg_num for the pools you are using. > pool 2 'myfs-metadata' replicated size 3 min_size 2 crush_rule 25 object_hash rjenkins pg_num 16 pgp_num 16 > pool 3 'myfs-replicated' replicated size 2 min_size 1 crush_rule 26 object_hash rjenkins pg_num 256 pgp_num 256 > pool 4 'pvc-generic-pool' replicated size 3 min_size 2 crush_rule 17 object_hash rjenkins pg_num 128 pgp_num 128 > pool 13 'myfs-eck2m2' erasure profile myfs-eck2m2_ecprofile size 4 min_size 3 crush_rule 8 pg_num 128 pgp_num 128 > pool 22 'my-store.rgw.otp' replicated size 3 min_size 2 crush_rule 24 pg_num 8 pgp_num 8 > pool 23 'my-store.rgw.buckets.index' replicated size 3 min_size 2 pg_num 8 pgp_num 8 > pool 24 'my-store.rgw.log' replicated size 3 min_size 2 crush_rule 23 pg_num 8 pgp_num 8 > pool 25 'my-store.rgw.control' replicated size 3 min_size 2 crush_rule 19 object_hash rjenkins pg_num 8 pgp_num 8 > pool 26 '.rgw.root' replicated size 3 min_size 2 crush_rule 18 pg_num 8 pgp_num 8 > pool 27 'my-store.rgw.buckets.non-ec' replicated size 3 min_size 2 pg_num 8 pgp_num 8 > pool 28 'my-store.rgw.meta' replicated size 3 min_size 2 pg_num 8 pgp_num 8 > pool 29 'my-store.rgw.buckets.data' erasure profile my-store.rgw.buckets.data_ecprofile size 4 min_size 3 pg_num 32 pgp_num 32 autoscale_mode on Is that a 2,2 or 3,1 profile? > >> `ceph balancer status` > > This does have normal output when the cluster isn't in the middle of recovery. > > { > "active": true, > "last_optimize_duration": "0:00:00.000107", > "last_optimize_started": "Tue Nov 28 22:11:56 2023", > "mode": "upmap", > "no_optimization_needed": true, > "optimize_result": "Too many objects (0.081907 > 0.050000) are > misplaced; try again later", > "plans": [] > } > >> `ceph osd pool autoscale-status` > > No output for this. I'm not sure why See above, I suspected this. > - this has given output in the > past. Might be due to being in the middle of recovery, or it might be > a Reef issue (I don't think I've looked at this since upgrading). In > any case, PG counts are in the osd dump, and I have the hdd storage > classes set to warn I think. > >> The balancer module can be confounded by certain complex topologies like multiple device classes and/or CRUSH roots. >> >> Since you’re using Rook, I wonder if you might be hitting something that I’ve seen myself; the above commands will tell the tale. > > Yeah, if it is designed for equally-sized OSDs then it isn't going to > work quite right for me. I do try to keep hosts reasonably balanced, > but not individual OSDs. Ceph is fantastic for flexibility, but it’s not above giving us enough rope to hang ourselves with. > > -- > Rich

2 1

Best Practice for OSD Balancing
by Rich Freeman 28 Nov '23

28 Nov '23

I'm fairly new to Ceph and running Rook on a fairly small cluster (half a dozen nodes, about 15 OSDs). I notice that OSD space use can vary quite a bit - upwards of 10-20%. In the documentation I see multiple ways of managing this, but no guidance on what the "correct" or best way to go about this is. As far as I can tell there is the balancer, manual manipulation of upmaps via the command line tools, and OSD reweight. The last two can be optimized with tools to calculate appropriate corrections. There is also the new read/active upmap (at least for non-EC pools), which is manually triggered. The balancer alone is leaving fairly wide deviations in space use, and at times during recovery this can become more significant. I've seen OSDs hit the 80% threshold and start impacting IO when the entire cluster is only 50-60% full during recovery. I've started using ceph osd reweight-by-utilization and that seems much more effective at balancing things, but this seems redundant with the balancer which I have turned on. What is generally considered the best practice for OSD balancing? -- Rich

3 3

Rook-Ceph OSD Deployment Error
by P Wagner-Beccard 28 Nov '23

28 Nov '23

Hi Mailing-Lister's, I am reaching out for assistance regarding a deployment issue I am facing with Ceph on a 4 node RKE2 cluster. We are attempting to deploy Ceph via the rook helm chart, but we are encountering an issue that apparently seems related to a known bug (https://tracker.ceph.com/issues/61597) During the OSD preparation phase, the deployment consistently fails with an IndexError: list index out of range. The logs indicate a problem occurs when configuring new Disks, specifically using /dev/dm-3 as a metadata device. It's important to note that /dev/dm-3 is an LVM on top of an mdadm RAID, which might or might not be contributing to this issue. (I swear, this setup worked already) Here is a snippet of the error from the deployment logs: > 2023-11-23 23:11:30.196913 D | exec: IndexError: list index out of range > 2023-11-23 23:11:30.236962 C | rookcmd: failed to configure devices: failed to initialize osd: failed ceph-volume report: exit status 1 https://paste.openstack.org/show/bileqRFKbolrBlTqszmC/ We have attempted different configurations, including specifying devices explicitly and using the useAllDevices: true option with a specified metadata device (/dev/dm-3 or the /dev/pv_md0/lv_md0 path). However, the issue persists across multiple configurations. tested configurations are as follows: Explicit device specification: ```yaml nodes: - name: "ceph01.maas" devices: - name: /dev/dm-1 - name: /dev/dm-2 - name: "sdb" config: metadataDevice: "/dev/dm-3" - name: "sdc" config: metadataDevice: "/dev/dm-3" ``` General device specification with metadata device: ```yaml storage: useAllNodes: true useAllDevices: true config: metadataDevice: /dev/dm-3 ``` I would greatly appreciate any insights or recommendations on how to proceed or work around this issue. Is there a halfway decent way to apply the fix or maybe a workaround that we can apply to successfully deploy Ceph in our environment? Kind regards,

2 2

How to speed up rgw lifecycle
by VÔ VI 28 Nov '23

28 Nov '23

Hi community, My ceph cluster is using s3 with three pools and obj/s approximately 4.5k obj/s and the rgw lifecycle delete per pool is only 60-70 objects/s How can I speed up the lc rgw process? 60 70 objects/s is too slow Thanks a lot

2 1

osdmaptool target & deviation calculation
by Robert Hish 28 Nov '23

28 Nov '23

Question about the osdmaptool deviation calculations; For instance, ----- osdmaptool omap --upmap output.txt --upmap-pool cephfs_data-rep3 --upmap-max 1000 --upmap-deviation 5 osdmaptool: osdmap file 'omap' writing upmap command output to: output.txt checking for upmap cleanups upmap, max-count 1000, max deviation 5 limiting to pools cephfs_data-rep3 ([30]) pools cephfs_data-rep3 prepared 0/1000 changes Unable to find further optimization, or distribution is already perfect ----- The evaluated pool is all-on-hdd, and the pool was created with PGS > number of hdd OSDs in the cluster. So each hdd OSD is being used at least once by this pool. Is it correct to assume that the osdmaptool is relying on the equations set at ceph-17.2.5/src/osd/OSDMap.cc:5143 5143 // This function calculates the 2 maps osd_deviation and deviation_osd which 5144 // hold the deviation between the current number of PGs which map to an OSD 5145 // and the optimal number. ... # pgs_per_weight # ceph-17.2.5/src/osd/OSDMap.cc:4806 4806 float pgs_per_weight = total_pgs / osd_weight_total; # target # ceph-17.2.5/src/osd/OSDMap.cc:5156 5156 float target = osd_weight.at(oid) * pgs_per_weight; # deviation # ceph-17.2.5/src/osd/OSDMap.cc:5157 5157 float deviation = (float)opgs.size() - target; And so for pgs_per_weight I calculate ceph -f json osd df | jq '[ .nodes[] | select (.device_class == "hdd") .pgs ] | add' divided by ceph -f json osd df | jq '[ .nodes[] | select (.device_class == "hdd") .crush_weight ] | add' (each hdd OSD in this cluster has identical weight) target = osd_weight.at(oid) * pgs_per_weight I calculate deviation for each osd deviation = opgs.size - target where, opgs.size = the number of PGs at an OSD. i.e. The value of $19 for each $1, in `ceph osd df hdd | awk '{ print $1 " " $19 }'` The result is many many OSDs with a deviation well above the upmap_max_deviation which is at default: 5 So I am wondering if I am miscalculating something, or if I'm not aware of further things the osdmaptool is considering when formulating upmap suggestions? -Robert

2 1

How balancer module balance data
by bryansoong21＠gmail.com 27 Nov '23

27 Nov '23

Hello, We are running a pacific 16.2.10 cluster and enabled the balancer module, here is the configuration: [root@ceph-1 ~]# ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.052548", "last_optimize_started": "Fri Nov 17 17:09:57 2023", "mode": "upmap", "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", "plans": [] } [root@ceph-1 ~]# ceph balancer eval current cluster score 0.017742 (lower is better) Here is the balancer configuration of upmap_max_deviation: # ceph config get mgr mgr/balancer/upmap_max_deviation 5 We have two different types of OSDS, one is 7681G and another is 3840G. When I checked our PG distribution on each type of OSD, I found the PG distribution is not evenly, for the 7681G OSDs, the OSD distribution varies from 136 to 158; while for the 3840G OSDs, it varies from 60 to 83, seems the upmap_max_deviation is almost +/- 10. So I just wondering if this is expected or do I need to change the upmap_max_deviation to a smaller value. Thanks for answering my question.

2 1

OSDs failing to start due to crc32 and osdmap error
by Denis Polom 27 Nov '23

27 Nov '23

Hi we have issue to start some OSDs on one node on our Ceph Quincy 17.2.7 cluster. Some OSDs on that node are running fine, but some failing to start. Looks like crc32 checksum error, and failing to get OSD map. I found a some discussions on that but nothing helped. I've also tried to insert current OSD map but that ends with error: # CEPH_ARGS="--bluestore-ignore-data-csum" ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-888/ --op set-osdmap --file osdmap osdmap (#-1:20684533:::osdmap.931991:0#) does not exist. Log is bellow Any ideas please? Thank you From log file: 2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to public interface 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2, device location [0x10000~1000], logical extent 0x0~1000, object #-1:7b3f43c4:::osd_superblock:0# 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0 failed to load OSD map for epoch 927580, got 0 bytes /build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time 2023-11-27T16:01:51.443522+0100 /build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret) ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x561ad07d2624] 2: ceph-osd(+0xc2e836) [0x561ad07d2836] 3: (OSD::init()+0x4026) [0x561ad08e5a86] 4: main() 5: __libc_start_main() 6: _start() *** Caught signal (Aborted) ** in thread 7f3f17aa13c0 thread_name:ceph-osd 2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1 /build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time 2023-11-27T16:01:51.443522+0100 /build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret) ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x561ad07d2624] 2: ceph-osd(+0xc2e836) [0x561ad07d2836] 3: (OSD::init()+0x4026) [0x561ad08e5a86] 4: main() 5: __libc_start_main() 6: _start() ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b7) [0x561ad07d268c] 5: ceph-osd(+0xc2e836) [0x561ad07d2836] 6: (OSD::init()+0x4026) [0x561ad08e5a86] 7: main() 8: __libc_start_main() 9: _start() 2023-11-27T16:01:51.447+0100 7f3f17aa13c0 -1 *** Caught signal (Aborted) ** in thread 7f3f17aa13c0 thread_name:ceph-osd ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b7) [0x561ad07d268c] 5: ceph-osd(+0xc2e836) [0x561ad07d2836] 6: (OSD::init()+0x4026) [0x561ad08e5a86] 7: main() 8: __libc_start_main() 9: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -558> 2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to public interface -5> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2, device location [0x10000~1000], logical extent 0x0~1000, object #-1:7b3f43c4:::osd_superblock:0# -2> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0 failed to load OSD map for epoch 927580, got 0 bytes -1> 2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1 /build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time 2023-11-27T16:01:51.443522+0100 /build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret) ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x561ad07d2624] 2: ceph-osd(+0xc2e836) [0x561ad07d2836] 3: (OSD::init()+0x4026) [0x561ad08e5a86] 4: main() 5: __libc_start_main() 6: _start() 0> 2023-11-27T16:01:51.447+0100 7f3f17aa13c0 -1 *** Caught signal (Aborted) ** in thread 7f3f17aa13c0 thread_name:ceph-osd ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b7) [0x561ad07d268c] 5: ceph-osd(+0xc2e836) [0x561ad07d2836] 6: (OSD::init()+0x4026) [0x561ad08e5a86] 7: main() 8: __libc_start_main() 9: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -562> 2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to public interface -9> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2, device location [0x10000~1000], logical extent 0x0~1000, object #-1:7b3f43c4:::osd_superblock:0# -6> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0 failed to load OSD map for epoch 927580, got 0 bytes -5> 2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1 /build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time 2023-11-27T16:01:51.443522+0100 /build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret) ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x561ad07d2624] 2: ceph-osd(+0xc2e836) [0x561ad07d2836] 3: (OSD::init()+0x4026) [0x561ad08e5a86] 4: main() 5: __libc_start_main() 6: _start() -4> 2023-11-27T16:01:51.447+0100 7f3f17aa13c0 -1 *** Caught signal (Aborted) ** in thread 7f3f17aa13c0 thread_name:ceph-osd ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b7) [0x561ad07d268c] 5: ceph-osd(+0xc2e836) [0x561ad07d2836] 6: (OSD::init()+0x4026) [0x561ad08e5a86] 7: main() 8: __libc_start_main() 9: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Aborted

3 8

understand "extent"
by Tony Liu 27 Nov '23

27 Nov '23

Hi, The context is RBD on bluestore. I did check extent on Wiki. I see "extent" when talking about snapshot and export/import. For example, when create a snapshot, we mark extents. When there is write to marked extents, we will make a copy. I also know that user data on block device maps to objects. How "extent" and "object" are related? Can I say extent is a set of continuous objects (with default tripe settings)? Thanks! Tony

2 1

About number of osd node can be failed with erasure code 3+2
by tranphong079＠gmail.com 27 Nov '23

27 Nov '23

Hi Groups, Recently I was setting up a ceph cluster with 10 nodes 144 osd, and I use S3 for it with pool erasure code EC3+2 on it. I have a question, how many osd nodes can fail with erasure code 3+2 with cluster working normal (read, write)? and can i choose better erasure code ec7+3, 8+2 etc..? With the erasure code algorithm, it only ensures no data loss, but does not guarantee that the cluster operates normally and does not block IO when osd nodes down. Is that right? Thanks to the community.

2 1

Issue with CephFS (mds stuck in clientreplay status) since upgrade to 18.2.0.
by Lo Re Giuseppe 27 Nov '23

27 Nov '23

Hi, We have upgraded one ceph cluster from 17.2.7 to 18.2.0. Since then we are having CephFS issues. For example this morning: “”” [root@naret-monitor01 ~]# ceph -s cluster: id: 63334166-d991-11eb-99de-40a6b72108d0 health: HEALTH_WARN 1 filesystem is degraded 3 clients failing to advance oldest client/flush tid 3 MDSs report slow requests 6 pgs not scrubbed in time 29 daemons have recently crashed … “”” The ceph orch, ceph crash and ceph fs status commands were hanging. After a “ceph mgr fail” those commands started to respond. Then I have noticed that there was one mds with most of the slow operations, “”” [WRN] MDS_SLOW_REQUEST: 3 MDSs report slow requests mds.cephfs.naret-monitor01.nuakzo(mds.0): 18 slow requests are blocked > 30 secs mds.cephfs.naret-monitor01.uvevbf(mds.1): 1683 slow requests are blocked > 30 secs mds.cephfs.naret-monitor02.exceuo(mds.2): 1 slow requests are blocked > 30 secs “”” Then I tried to restart it with “”” [root@naret-monitor01 ~]# ceph orch daemon restart mds.cephfs.naret-monitor01.uvevbf Scheduled to restart mds.cephfs.naret-monitor01.uvevbf on host 'naret-monitor01' “”” After the cephfs entered into this situation: “”” [root@naret-monitor01 ~]# ceph fs status cephfs - 198 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephfs.naret-monitor01.nuakzo Reqs: 0 /s 17.2k 16.2k 1892 14.3k 1 active cephfs.naret-monitor02.ztdghf Reqs: 0 /s 28.1k 10.3k 752 6881 2 clientreplay cephfs.naret-monitor02.exceuo 63.0k 6491 541 66 3 active cephfs.naret-monitor03.lqppte Reqs: 0 /s 16.7k 13.4k 8233 990 POOL TYPE USED AVAIL cephfs.cephfs.meta metadata 5888M 18.5T cephfs.cephfs.data data 119G 215T cephfs.cephfs.data.e_4_2 data 2289G 3241T cephfs.cephfs.data.e_8_3 data 9997G 470T STANDBY MDS cephfs.naret-monitor03.eflouf cephfs.naret-monitor01.uvevbf MDS version: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable) “”” The file system is totally unresponsive (we can mount it on client nodes but any operations like a simple ls hangs). During the night we had a lot of mds crashes, I can share the content. Does anybody have an idea on how to tackle this problem? Best, Giuseppe

3 3

Does cephfs ensure close-to-open consistency after enabling lazyio?
by Jianjun Zheng 27 Nov '23

27 Nov '23

2 1

import/export with --export-format 2
by Tony Liu 27 Nov '23

27 Nov '23

Hi, src-image is 1GB (provisioned size). I did the following 3 tests. 1. rbd export src-image - | rbd import - dst-image 2. rbd export --export-format 2 src-image - | rbd import --export-format 2 - dst-image 3. rbd export --export-format 2 src-image - | rbd import - dst-image With #1 and #2, dst-image size (rbd info) is the same as src-image, which is expected. With #3, dst-image size (rbd info) is close to used size (rbd du), not the provisioned size of src-image. I'm not sure if this image is actually useable when write into it. The questions is that, is #3 not supposed to be used at all? I checked doc, didn't see something like "--export-format 2 has to be used for importing the image which is exported with --export-format 2 option". Any comments? Thanks! Tony

2 1

Experience with deduplication
by Szabo, Istvan (Agoda) 27 Nov '23

27 Nov '23

Hi Developers, What is the status of the deduplication for objectsore? I see it under the dev area only since octopus even with the latest release. https://docs.ceph.com/en/octopus/dev/deduplication/ Is it something that can be used in production? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Ceph/daemon container lvm tools don’t work
by Gaël THEROND 27 Nov '23

27 Nov '23

Hi team, I’m experimenting a bit CentOS Stream 9 on our infrastructure as we’re migrating away from CentOS Stream 8. As our deployment model is an hyperconverged one, I have CEPH and OPENSTACK running on the same hosts (OSDs+NOVA/CINDER). That prohibits me to let CEPH nodes running on CentOS Stream 8. However, I’ve noticed an issue related to LVM, which is a bit annoying, when running CentOS Stream 8 based ceph/daemon container over a CentOS Stream 9 node. Indeed, when the ceph-volume lvm batch is processed and perform the lvm subcommand process call, the pvcreate do not persist anymore on the host and so consequently, when you reboot, all ceph related disks disappear as LVM can’t find their PVID or device references. Even a pvscan doesn’t find them. The only workaround that we found out right now is to pvcreate the host disks, pvremove them, dd them and then run the installation process again, from my understanding it kinda warmup the host lvm system and consequently let it keep up with the ceph-volume run later on. For additional information: We deploy the cluster using ceph-ansible stable-6.0 branch (pacific). The container come from quay.io and is 6.0.11 CentOS Stream 8 release. This release is working perfectly on a CentOS Stream 8 based host. One weird thing that we catched is that on a CentOS Stream 8 host using this CentOS Stream 8 based image, lvm/dm are creating new dm block devices named after ceph vg/lv where on a CentOS Stream 9 based host, lvm/dm create the new vg/lv as symlink to dm-xY dm block devices. And last question, is there any planned build of the ceph/daemon:6.0.11 or greater image based on CentOS Stream 9 ? Thanks a lot! PS: I did forget the [ceph-users] tag at first post and don’t know if it’s something important for the mailing distribution.

1 0

Where is a simple getting started guide for a very basic cluster?
by Leo28C 27 Nov '23

27 Nov '23

I'm pulling my hair trying to get a simple cluster going. I first tried Gluster but I have an old system that can't handle the latest version, so I resorted to Ceph. However, I can't get my cluster to work. I tried to find tutorials but everything uses tools on top of Ceph, whereas I'm trying to use it barebones. (I don't trust nfs or samba for this purpose) The three systems I'm trying to cluster together have a variety of Linux distros, and one of them is discontinued, so I can't use a lot of newer stuff. I need to make do with the commands `ceph`, `ceph-mon`, and `ceph-volume`. I do *not* have access to `ceph-disk` which I ran into a lot online. ChatGPT guided me up to a certain point before she started making stuff up, so I need to start from scratch. I just need to share a directory (or rather a block device in the case of Ceph) across these three systems that already have a secure connection to one another. Can somebody please help me out? Thanks in advance

3 2

About lifecycle of RGW
by VÔ VI 27 Nov '23

27 Nov '23

Hi community, I'm running a large s3 running with ingress and backend rgw, total traffic of my cluster ~30Gbps real time traffic. I' am have a problem with lifecycle in rgw, object can't delete with following log Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 1 lifecycle: ERROR: publishing notification failed, with error: -2 Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 0 lifecycle: ERROR: remove_expired_obj :ec3204cam05[f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.2]):003_8792934df4756401_231127_1701067328.134916_15b007878cf011ee960ba1c41ff4eaf9_3007777.test (2) No such file or directory wp_thrd: 4, 2 Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 0 lifecycle: ERROR: remove_expired_obj :ec3204cam05[f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.2]):003_8792934df4756401_231127_1701067328.134916_15b007878cf011ee960ba1c41ff4eaf9_3007777.test (2) No such file or directory wp_thrd: 4, 2 Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 1 lifecycle: ERROR: publishing notification failed, with error: -2 Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 0 lifecycle: ERROR: remove_expired_obj :ec3204cam05[f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.2]):003_8792934df4756401_231127_1701067327.693377_14a3adc68cf011ee960ba1c41ff4eaf9_3007677.test (2) No such file or directory wp_thrd: 4, 2 Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 0 lifecycle: ERROR: remove_expired_obj :ec3204cam05[f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.2]):003_8792934df4756401_231127_1701067327.693377_14a3adc68cf011ee960ba1c41ff4eaf9_3007677.test (2) No such file or directory wp_thrd: 4, 2 My cluster setting rgw_lc_max_worker = 6 rgw_lc_debug_interval=60 (setting during testing) I am testing bucket lifecycle expire every 5 minute and 1 minute radosgw-lc list of bucket is processing "bucket": ":ec3204cam05:f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.2", "started": "Mon, 27 Nov 2023 07:29:58 GMT", "status": "PROCESSING" s3cmd getlifecycle s3://ec3204cam04 <?xml version="1.0" ?> <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <ID>Expire after 5 day</ID> <Prefix/> <Status>Enabled</Status> <Expiration> <Days>5</Days> </Expiration> </Rule> </LifecycleConfiguration> s3cmd getlifecycle s3://ec3204cam05 <?xml version="1.0" ?> <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <ID>Expire after 1 day</ID> <Prefix/> <Status>Enabled</Status> <Expiration> <Days>1</Days> </Expiration> </Rule> </LifecycleConfiguration> I don't know why the rgw show log can't delete object with no such file, how i can find the problem, I hope someone can help me Thanks

1 0

Re: openstack Vm shutoff by itself
by AJ_ sunny 27 Nov '23

27 Nov '23

++adding @ceph-users-confirm+4555fdc6282a38c849f4d27a40339f1b7e4bde74@ceph.io <ceph-users-confirm+4555fdc6282a38c849f4d27a40339f1b7e4bde74(a)ceph.io> ++Adding dev(a)ceph.io Thanks,&, Regards Arihant Jain On Mon, 27 Nov, 2023, 7:48 am AJ_ sunny, <jains8550(a)gmail.com> wrote: > Hi team, > > After doing above changes I am still getting the issue in which machine > continuously went shutdown > > In nova-compute logs I am getting only this footprint > > Logs:- > 2023-10-16 08:48:10.971 7 WARNING nova.compute.manager > [req-c7b731db-2b61-400e-917f-8645c9984696 f226d81a45dd46488fb2e19515 848 > 316d215042914de190f5f9e1c8466bf0 default default] [instance: > 4b04d3f1-1fbd-4b63-b693-a0ef316ecff3] Received unexpected - vent > network-vif-plugged-f191f6c8-dff5-4c1b-94b3-8d91aa6ff5ac for instance with > vm_state active and task_state None. 2023-10-21 22:42:44.589 7 INFO > nova.compute.manager [-] [instance: 4b04d3f1-1fbd-4b63-b693-a0ef316ecff3] > VM Stopped (Lifecyc Event) > > 2023-10-21 22:42:44.683 7 INFO nova.compute.manager > [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d -] [instance: 4b04d3f1- > fbd-4b63-b693-a0ef316ecff3] During _sync_instance_power_state the DB > power_state (1) does not match the vm_power_state from ti e hypervisor (4). > Updating power_state in the DB to match the hypervisor. > > 2023-10-21 22:42:44.811 7 WARNING nova.compute.manager > [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d ----] [instance: 4b04d3f > 1-1fbd-4b63-b693-a0ef316ecff3] Instance shutdown by itself. Calling the > stop API. Current vm_state: active, current task_state : None, original DB > power_state: 1, current VM power_state: 4 2023-10-21 22:42:44.977 7 INFO > nova.compute.manager [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d -] > [instance: 4b04d3f1-1 > > fbd-4b63-b693-a0ef316ecff3] Instance is already powered off in the > hypervisor when stop is called. > > > And in this architecture we are using ceph is the backend storage for > Nova,glance & cinder > When machine auto goes down and if i try to start the machine it will go > in error i.e. in Vm console is show I/O ERROR during boot so first we need > to rebuild the volume from ceph side then I have to start the machine > Rbd object-map rebuild<volume-id> > Openstack server start <server-id> > > So this issue is showing two faces one from ceph side and another from > nova-compute log > can someone please help me out to fix out this issue asap > > Thanks & Regards > Arihant Jain > > On Tue, 24 Oct, 2023, 4:56 pm , <smooney(a)redhat.com> wrote: > >> On Tue, 2023-10-24 at 10:11 +0530, AJ_ sunny wrote: >> > Hi team, >> > >> > Vm is not shutting off by owner from inside its automatically went to >> > shutdown i.e. libvirt lifecycle stop event triggering >> > In my nova.conf configuration I am using ram_allocation_ratio = 1.5 >> > And previously I tried to set in nova.conf >> > Sync_power_state_interval = -1 but still facing the same problem >> > OOM might be causing this issue >> > Can you please give me some idea to fix this issue if OOM is the cause >> the general answer is swap. >> >> nova should alwasy be deployed with swap even if you do not have over >> commit enabled. >> there are a few reason for this the first being python allocates memory >> diffently if >> any swap is aviable, even 1G is enough to have it not try to commit all >> memory. so >> when swap is aviable the nova/neutron agents will use much less resident >> memeory even with >> out usign any of the swap space. >> >> we have some docs about this downstream >> >> https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/17… >> >> if you are being ultra conservative we recommend allocating (ram * >> allocation ratio) in swap so in your case allcoate >> 1.5 times your ram as swap. we woudl expect the actul useage of swap to >> be a small fraction of that however so we >> also provide a formula for >> >> overcommit_ratio = NovaRAMAllocationRatio - 1 >> Minimum swap size (MB) = (total_RAM * overcommit_ratio) + >> RHEL_min_swap >> Recommended swap size (MB) = total_RAM * (overcommit_ratio + >> percentage_of_RAM_to_use_for_swap) >> >> so say your host had 64G of ram with an allocation ratio of 1.5 and a min >> swap percentaiong of 25% >> the conserviver swap recommentation would be >> >> (64*(0.5+0.25)) + disto_min_swap >> (64*0.75) + 4G = 52G of recommended swap >> >> if your wondering why we add a min swap precentage and disto min swap its >> basically to acocund for the >> Qemu and host OS memory overhead as well as the memory used by the >> nova/neutron agents and libvirt/ovs >> >> >> if your not using memory over commit my general recommdation is if you >> have less then 64G of ram allcoate 16G if you >> have more then 256G of ram allocate 64G and you should be fine. when you >> do use memofy over commit you must >> have at least enouch swap to account for the qemu overhead of all >> instance + the over committed memory. >> >> >> the other common cause of OOM errors is if you are using numa affinity >> and the guest dont request >> hw:mem_page_size=<something> without setting a mem_page_size request we >> dont do numa aware memory placement. the kernel >> OOM system works >> on a per numa node basis, numa affintiy does not supprot memory over >> commit either so that is likly not your issue. >> i jsut said i woudl mention it to cover all basis. >> >> regards >> sean >> >> >> >> > >> > >> > Thanks & Regards >> > Arihant Jain >> > >> > On Mon, 23 Oct, 2023, 11:29 pm , <smooney(a)redhat.com> wrote: >> > >> > > On Mon, 2023-10-23 at 13:19 -0400, Jonathan Proulx wrote: >> > > > >> > > > I've seen similar log traces with overcommitted memory when the >> > > > hypervisor runs out of physical memory and OOM killer gets the VM >> > > > process. >> > > > >> > > > This is an unusuall configuration (I think) but if the VM owner >> claims >> > > > they didn't power down the VM internally you might look at the local >> > > > hypevisor logs to see if the VM process crashed or was killed for >> some >> > > > other reason. >> > > yep OOM events are one common causes fo this. >> > > >> > > nova is bacialy just saying "hay you said this vm should be active >> but its >> > > not, im going to update the db to reflect >> > > reality." you can turn that off with >> > > >> > > >> https://docs.openstack.org/nova/latest/configuration/config.html#workaround… >> > > or >> > > >> > > >> https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.sy… >> > > either disabel the sync via setign the interval to -1 >> > > or disable haneling the virt lifecycle events. >> > > >> > > i would recommend the sync_power_state_interval approach but again if >> vms >> > > are stopping >> > > and you dont know why you likely should discover why rahter then just >> > > turning if the update of the nova db to reflect >> > > the actual sate. >> > > >> > > > >> > > > -Jon >> > > > >> > > > On Mon, Oct 23, 2023 at 02:02:26PM +0100, smooney(a)redhat.com wrote: >> > > > :On Mon, 2023-10-23 at 17:45 +0530, AJ_ sunny wrote: >> > > > :> Hi team, >> > > > :> >> > > > :> I am using openstack kolla ansible on wallaby version and >> currently I >> > > am >> > > > :> facing issue with virtual machine, vm is shutoff by itself and >> and >> > > from log >> > > > :> it seems libvirt lifecycle stop event triggering again and again >> > > > :> >> > > > :> Logs:- >> > > > :> 2023-10-16 08:48:10.971 7 WARNING nova.compute.manager >> > > > :> [req-c7b731db-2b61-400e-917f-8645c9984696 >> f226d81a45dd46488fb2e19515 >> > > 848 >> > > > :> 316d215042914de190f5f9e1c8466bf0 default default] [instance: >> > > > :> 4b04d3f1-1fbd-4b63-b693-a0ef316ecff3] Received unexpected - vent >> > > > :> network-vif-plugged-f191f6c8-dff5-4c1b-94b3-8d91aa6ff5ac for >> instance >> > > with >> > > > :> vm_state active and task_state None. 2023-10-21 22:42:44.589 7 >> INFO >> > > > :> nova.compute.manager [-] [instance: >> > > 4b04d3f1-1fbd-4b63-b693-a0ef316ecff3] >> > > > :> VM Stopped (Lifecyc Event) >> > > > :> >> > > > :> 2023-10-21 22:42:44.683 7 INFO nova.compute.manager >> > > > :> [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d -] [instance: 4b04d3f1- >> > > > :> fbd-4b63-b693-a0ef316ecff3] During _sync_instance_power_state >> the DB >> > > > :> power_state (1) does not match the vm_power_state from ti e >> > > hypervisor (4). >> > > > :> Updating power_state in the DB to match the hypervisor. >> > > > :> >> > > > :> 2023-10-21 22:42:44.811 7 WARNING nova.compute.manager >> > > > :> [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d ----] [instance: >> 4b04d3f >> > > > :> 1-1fbd-4b63-b693-a0ef316ecff3] Instance shutdown by itself. >> Calling >> > > the >> > > > :> stop API. Current vm_state: active, current task_state : None, >> > > original DB >> > > > :> power_state: 1, current VM power_state: 4 2023-10-21 >> 22:42:44.977 7 >> > > INFO >> > > > :> nova.compute.manager [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d -] >> > > > :> [instance: 4b04d3f1-1 >> > > > :> >> > > > :> fbd-4b63-b693-a0ef316ecff3] Instance is already powered off in >> the >> > > > :> hypervisor when stop is called. >> > > > : >> > > > :that sounds like the guest os shutdown the vm. >> > > > :i.e. somethign in the guest ran sudo poweroff >> > > > :then nova detected teh vm was stoped by the user and updated its >> db to >> > > match >> > > > :that. >> > > > : >> > > > :that is the expected beahvior wehn you have the power sync enabled. >> > > > :it is enabled by default. >> > > > :> >> > > > :> >> > > > :> Thanks & Regards >> > > > :> Arihant Jain >> > > > :> +91 8299719369 >> > > > : >> > > > >> > > >> > > >> >>

1 1

Quincy patch upgrade
by Ben 27 Nov '23

27 Nov '23

Dear Cephers, With improvement on dmclock in 17.2.7, we are considering upgrade from 17.2.5 to 17.2.7. seeing this which is worrysome: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/SG7CKALU3AI… any suggestions for an easy upgrade? Regards, Ben

1 0

Full cluster outage when ECONNREFUSED is triggered
by Denis Krienbühl 24 Nov '23

24 Nov '23

Hi We’ve recently had a serious outage at work, after a host had a network problem: - We rebooted a single host in a cluster of fifteen hosts across three racks. - The single host had a bad network configuration after booting, causing it to send some packets to the wrong network. - One network still worked and offered a connection to the mons. - The other network connection was bad. Packets were refused, not dropped. - Due to osd_fast_fail_on_connection_refused=true, the broken host forced the mons to take all other OSDs down (immediate failure). - Only after shutting down the faulty host, was it possible to start the shut down OSDs, to restore the cluster. We have since solved the problem by removing the default route that caused the packets to end up in the wrong network, where they were summarily rejected by a firewall. That is, we made sure that packets would be dropped in the future, not rejected. Still, I figured I’ll send this experience of ours to this mailing list, as this seems to be something others might encounter as well. In the following PR, that introduced osd_fast_fail_on_connection_refused, there’s this description: > This changeset adds additional handler (handle_refused()) to the dispatchers > and code that detects when connection attempt fails with ECONNREFUSED error > (connection refused) which is a clear indication that host is alive, but > daemon isn't, so daemons can instantly mark the other side as undoubtly > downed without the need for grace timer. And this comment: > As for flapping, we discussed it on ceph-devel ml > and came to conclusion that it requires either broken firewall or network > configuration to cause this, and these are more serious issues that should > be resolved first before worrying about OSDs flapping (either way, flapping > OSDs could be good for getting someone's attention). https://github.com/ceph/ceph/pull/8558https://github.com/ceph/ceph/pull/8558 It has left us wondering if these are the right assumptions. An ECONNREFUSED condition can bring down a whole cluster, and I wonder if there should be some kind of safe-guard to ensure that this is avoided. One badly configured host should generally not be able do that, and if the packets are dropped, instead of refused, the cluster notices that the OSD down reports come only from one host, and acts accordingly. What do you think? Does this warrant a change in Ceph? I’m happy to provide details and create a ticket. Cheers, Denis

4 9

[CEPH] Ceph multi nodes failed
by Nguyễn Hữu Khôi 24 Nov '23

24 Nov '23

Hello guys. I see many docs and threads talking about osd failed. I have a question: how many nodes in a cluster can be failed. I am using ec 8 + 2(10 osd nodes) and when I shutdown 2 nodes then my cluster crashes, It cannot write anymore. Thank you. Regards Nguyen Huu Khoi

4 6

Object size
by Miroslav Svoboda 24 Nov '23

24 Nov '23

Hi, please, is it better to reduce the default object size from 4MB to some smaller value for the rbd image where there will be a lot of small mail and webhosting files? Thanks Svoboda Miroslav

2 1

CEPH Daemon container CentOS Stream 8 over CentOS Stream 9 host
by Gaël THEROND 24 Nov '23

24 Nov '23

Hi team, I’m experimenting a bit CentOS Stream 9 on our infrastructure as we’re migrating away from CentOS Stream 8. As our deployment model is an hyperconverged one, I have CEPH and OPENSTACK running on the same hosts (OSDs+NOVA/CINDER). That prohibits me to let CEPH nodes running on CentOS Stream 8. However, I’ve noticed an issue related to LVM, which is a bit annoying, when running CentOS Stream 8 based ceph/daemon container over a CentOS Stream 9 node. Indeed, when the ceph-volume lvm batch is processed and perform the lvm subcommand process call, the pvcreate do not persist anymore on the host and so consequently, when you reboot, all ceph related disks disappear as LVM can’t find their PVID or device references. Even a pvscan doesn’t find them. The only workaround that we found out right now is to pvcreate the host disks, pvremove them, dd them and then run the installation process again, from my understanding it kinda warmup the host lvm system and consequently let it keep up with the ceph-volume run later on. For additional information: We deploy the cluster using ceph-ansible stable-6.0 branch (pacific). The container come from quay.io and is 6.0.11 CentOS Stream 8 release. This release is working perfectly on a CentOS Stream 8 based host. One weird thing that we catched is that on a CentOS Stream 8 host using this CentOS Stream 8 based image, lvm/dm are creating new dm block devices named after ceph vg/lv where on a CentOS Stream 9 based host, lvm/dm create the new vg/lv as symlink to dm-xY dm block devices. And last question, is there any planned build of the ceph/daemon:6.0.11 or greater image based on CentOS Stream 9 ? Thanks a lot!

1 0

[MDS] mds stuck in laggy state, CephFS unusable
by kvesligaj＠cs.hr 24 Nov '23

24 Nov '23

Hi, we're having a peculiar issue which we found out during HA/DR testing in our Ceph cluster. Basic info about cluster: Version: Quincy (17.2.6) 5 nodes configured in stretch cluster (2 DCs with one arbiter node which is also admin node for the cluster) On every node beside the admin node we have OSD and MON services. We have 3 MGR instances in cluster. Specific thing that we wanted to test is multiple CephFS with each having multiple MDS (with HA in mind). We deployed MDS on every node, increased max_mds to 2 for every CephFS and other two MDS-es are in standby-replay mode (they are automatically configured during CephFS creation to follow specific CephFS - join_fscid). We did multiple tests and when we have only one CephFS it behaves as expected (two MDS are in up:active state and clients can connect and interact with CephFS as if nothing has happened). When we test with multiple CephFS (two for example) and we shutdown two nodes one of MDS is stuck in up:active laggy state and when this happens the CephFS for which this happens is unusable, client hangs and it is stuck like that until we power on other DC. This happens even when there are no clients connected to this specific CephFS. We can provide additional logs and do any tests necessary. We checked the usual culprits and our nodes don't show any excessive CPU or memory usage. We would appreciate any help.

1 0

CephFS - MDS removed from map - filesystem keeps to be stopped
by Denis Polom 24 Nov '23

24 Nov '23

Hi running Ceph Pacific 16.2.13. we had full CephFS filesystem and after adding new HW we tried to start it but our MDS daemons are pushed to be standby and are removed from MDS map. Filesystem was broken, so we repaired it with: # ceph fs fail cephfs # cephfs-journal-tool --rank=cephfs:0 event recover_dentries summary # cephfs-journal-tool --rank=cephfs:0 journal reset then I started ceph-mds service and marked rank as repaired mds after some time has switched to standby. Log is bellow. I would appreciate any help to resolve this situation. Thank you. from log: 2023-11-22T14:11:49.212+0100 7f5dc155e700 1 mds.0.9604 handle_mds_map i am now mds.0.9604 2023-11-22T14:11:49.212+0100 7f5dc155e700 1 mds.0.9604 handle_mds_map state change up:rejoin --> up:active 2023-11-22T14:11:49.212+0100 7f5dc155e700 1 mds.0.9604 recovery_done -- successful recovery! 2023-11-22T14:11:49.212+0100 7f5dc155e700 1 mds.0.9604 active_start 2023-11-22T14:11:49.216+0100 7f5dc155e700 1 mds.0.9604 cluster recovered. 2023-11-22T14:11:49.216+0100 7f5dc3d63700 0 --1- [v2:10.245.4.103:6800/1548097835,v1:10.245.4.103:6801/1548097835] >> v1:10.245.8.127:0/2123529386 conn(0x55a60627a800 0x55a606e5b000 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0). handle_connect_message_2 accept peer reset, then tried to connect to us, replacing 2023-11-22T14:11:49.216+0100 7f5dc4564700 0 --1- [v2:10.245.4.103:6800/1548097835,v1:10.245.4.103:6801/1548097835] >> v1:10.245.6.88:0/1899426587 conn(0x55a60627ac00 0x55a6070d0000 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).h andle_connect_message_2 accept peer reset, then tried to connect to us, replacing 2023-11-22T14:11:49.216+0100 7f5dc4564700 0 --1- [v2:10.245.4.103:6800/1548097835,v1:10.245.4.103:6801/1548097835] >> v1:10.245.4.216:0/2058542052 conn(0x55a6070c9800 0x55a6070d1800 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0). handle_connect_message_2 accept peer reset, then tried to connect to us, replacing 2023-11-22T14:11:49.216+0100 7f5dc3d63700 0 --1- [v2:10.245.4.103:6800/1548097835,v1:10.245.4.103:6801/1548097835] >> v1:10.245.4.220:0/1549374180 conn(0x55a60708d000 0x55a6070d0800 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0). handle_connect_message_2 accept peer reset, then tried to connect to us, replacing 2023-11-22T14:11:49.216+0100 7f5dc4d65700 0 --1- [v2:10.245.4.103:6800/1548097835,v1:10.245.4.103:6801/1548097835] >> v1:10.245.8.180:0/270666178 conn(0x55a60703a000 0x55a6070cf800 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).h andle_connect_message_2 accept peer reset, then tried to connect to us, replacing 2023-11-22T14:11:49.216+0100 7f5dc4d65700 0 --1- [v2:10.245.4.103:6800/1548097835,v1:10.245.4.103:6801/1548097835] >> v1:10.245.8.178:0/3673271488 conn(0x55a6070c9400 0x55a6070d1000 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0). handle_connect_message_2 accept peer reset, then tried to connect to us, replacing 2023-11-22T14:11:49.216+0100 7f5dc4d65700 0 --1- [v2:10.245.4.103:6800/1548097835,v1:10.245.4.103:6801/1548097835] >> v1:10.245.4.167:0/2667964940 conn(0x55a6070c9c00 0x55a607112000 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0). handle_connect_message_2 accept peer reset, then tried to connect to us, replacing 2023-11-22T14:11:49.216+0100 7f5dc3d63700 0 --1- [v2:10.245.4.103:6800/1548097835,v1:10.245.4.103:6801/1548097835] >> v1:10.245.6.70:0/3181830075 conn(0x55a607116000 0x55a607112800 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).h andle_connect_message_2 accept peer reset, then tried to connect to us, replacing 2023-11-22T14:11:49.216+0100 7f5dc4564700 0 --1- [v2:10.245.4.103:6800/1548097835,v1:10.245.4.103:6801/1548097835] >> v1:10.245.6.72:0/3744737352 conn(0x55a60627a800 0x55a606e5b000 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).h andle_connect_message_2 accept peer reset, then tried to connect to us, replacing 2023-11-22T14:11:49.216+0100 7f5dc3d63700 0 --1- [v2:10.245.4.103:6800/1548097835,v1:10.245.4.103:6801/1548097835] >> v1:10.244.18.140:0/1607447464 conn(0x55a60627ac00 0x55a6070d0000 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0) .handle_connect_message_2 accept peer reset, then tried to connect to us, replacing 2023-11-22T14:11:49.220+0100 7f5dc155e700 1 mds.mds1 Updating MDS map to version 9608 from mon.1 2023-11-22T14:11:49.220+0100 7f5dc155e700 1 mds.0.9604 handle_mds_map i am now mds.0.9604 2023-11-22T14:11:49.220+0100 7f5dc155e700 1 mds.0.9604 handle_mds_map state change up:active --> up:stopping 2023-11-22T14:11:52.412+0100 7f5dc3562700 1 mds.mds1 asok_command: client ls {prefix=client ls} (starting...) 2023-11-22T14:11:57.412+0100 7f5dc3562700 1 mds.mds1 asok_command: client ls {prefix=client ls} (starting...) 2023-11-22T14:12:02.416+0100 7f5dc3562700 1 mds.mds1 asok_command: client ls {prefix=client ls} (starting...) 2023-11-22T14:12:07.420+0100 7f5dc3562700 1 mds.mds1 asok_command: client ls {prefix=client ls} (starting...) 2023-11-22T14:12:12.420+0100 7f5dc3562700 1 mds.mds1 asok_command: client ls {prefix=client ls} (starting...) 2023-11-22T14:12:13.552+0100 7f5dc155e700 1 mds.mds1 Updating MDS map to version 9609 from mon.1 2023-11-22T14:12:13.552+0100 7f5dc155e700 1 mds.mds1 Map removed me [mds.mds1{0:5320528} state up:stopping seq 67 addr [v2:10.245.4.103:6800/1548097835,v1:10.245.4.103:6801/1548097835] compat {c=[1],r=[1],i=[7ff]}] from cluster; respawnin g! See cluster/monitor logs for details. 2023-11-22T14:12:13.552+0100 7f5dc155e700 1 mds.mds1 respawn!

2 1

CLT Meeting minutes 2023-11-23
by Nizamudeen A 23 Nov '23

23 Nov '23

Hello, etherpad history lost need a way to recover from DB or find another way to back things up discuss the quincy/dashboard-v3 backports? was tabled from 11/1 https://github.com/ceph/ceph/pull/54252 agreement is to not backport breaking features to stable branches. 18.2.1 LRC upgrade affected by https://tracker.ceph.com/issues/62570#note-4 https://ceph-storage.slack.com/archives/C1HFJ4VTN/p1700575544548809 Figure out the reproducer and add tests https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?… is the trigger on the kernel client side? Clients with that patch should work with the server-side code that broke older ones Suggestion to introduce a matrix of pre-built "older" kernels into the fs suite gibba vs LRC upgrades LRC shouldn't be updated often, it should act more like a production cluster RCs for reef, quincy and pacific for next week when there is more time to discuss Regards, -- Nizamudeen A Software Engineer Red Hat <https://www.redhat.com/> <https://www.redhat.com/>

3 2

Erasure vs replica
by Albert Shih 23 Nov '23

23 Nov '23

Hi everyone, I just like to know what's your opinion about the reliability of erasure coding. Of course I can understand if we want the «best of the best of the best» ;-) I can choose the replica method. I heard in many location “replica” are more reliable, “replica” are more efficient etc... Yes...well since 25 years I'm using raid (5, 6, lvm, raidz1, raidz2, etc.) I never loose data only once when a firmware bug in some xxxxx card crash the raid volume. Now 25 years later lot of people recommend to use replica so if I buy XTo I'm only going to have X/3 To (vs raidz2 where I loose 2 disks over 9-12 disks). So my question are : Anyone use in large scale erasure coding for critical (same level as raidz1/raid5 ou raidz2/raid6) ? Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: jeu. 23 nov. 2023 14:51:28 CET

4 3

cephadm vs ceph.conf
by Albert Shih 23 Nov '23

23 Nov '23

Hi everyone. Still me with my newbie question....sorry. I'm using cephadm to deploy my ceph cluster, but when I search in the documentation «docs.ceph.com» I see in some place like https://docs.ceph.com/en/latest/rados/configuration/pool-pg-config-ref/ to change something in the /etc/ceph/ceph.conf. How that is taking account by cephadm ? I see in the docker container they have a overlay for /etc/ceph/ceph.conf. Should I modify the ceph.conf (vi/emacs) directly ? Should I modify the ceph.conf (vi/emacs) directly and restart something ? Should I use some cephadm shell and don't manually touche ceph.conf ? And what about the future ? Regards. -- Albert SHIH 🦫 🐸 France Heure locale/Local time: jeu. 23 nov. 2023 15:21:47 CET

4 6

How to use hardware
by Albert Shih 23 Nov '23

23 Nov '23

Hi everyone, In the purpose to deploy a medium size of ceph cluster (300 OSD) we have 6 bare-metal server for the OSD, and 5 bare-metal server for the service (MDS, Mon, etc.) Those 5 bare-metal server have each 48 cores and 256 Gb. What would be the smartest way to use those 5 server, I see two way : first : Server 1 : MDS,MON, grafana, prometheus, webui Server 2: MON Server 3: MON Server 4 : MDS Server 5 : MDS so 3 MDS, 3 MON. and we can loose 2 servers. Second KVM on each server Server 1 : 3 VM : One for grafana & CIe, and 1 MDS, 2 MON other server : 1 MDS, 1 MON in total : 5 MDS, 5 MON and we can loose 4 servers. So on paper it's seem the second are smarter, but it's also more complex, so my question are «is it worth the complexity to have 5 MDS/MON for 300 OSD». Important : The main goal of this ceph cluster are not to get the maximum I/O speed, I would not say the speed is not a factor, but it's not the main point. Regards. -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: ven. 17 nov. 2023 10:49:27 CET

5 8

ceph-exporter binds to IPv4 only
by Stefan Kooman 23 Nov '23

23 Nov '23

Hi, In a IPv6 only deployment the ceph-exporter daemons are not listening on IPv6 address(es). This can be fixed by editing the unit.run file of the ceph-exporter by changing "--addrs=0.0.0.0" to "--addrs=::". Is this configurable? So that cephadm deploys ceph-exporter with proper unit.run arguments? Gr. Stefan ... who really thinks the Ceph test lab should have an IPv6 only test environment to catch these things

1 1

Service Discovery issue in Reef 18.2.0 release ( upgrading )
by Brent Kennedy 22 Nov '23

22 Nov '23

Greetings group! We recently reloaded a cluster from scratch using cephadm and reef. The cluster came up, no issues. We then decided to upgrade two existing cephadm clusters that were on quincy. Those two clusters came up just fine but there is an issue with the Grafana graphs on both clusters ( which were working before the upgrade ). They are now blank. There is an error in the Prometheus alerts (PrometheusJobMissing) that is alerting and it states the following: The prometheus job that scrapes from Ceph is no longer defined, this will effectively mean you'll have no metrics or alerts for the cluster. Please review the job definitions in the prometheus.yml file of the prometheus instance. summary: The scrape job for Ceph is missing from Prometheus When I look at the Prometheus.yml file on the performance monitoring node, this is what is there( I replaced ip with x.x.x.x ): global: scrape_interval: 10s evaluation_interval: 10s rule_files: - /etc/prometheus/alerting/* alerting: alertmanagers: - scheme: http http_sd_configs: - url: http://x.x.x.x:8765/sd/prometheus/sd-config?service=alertmanager scrape_configs: - job_name: 'ceph' honor_labels: true http_sd_configs: - url: http://x.x.x.x:8765/sd/prometheus/sd-config?service=mgr-prometheus - job_name: 'node' http_sd_configs: - url: http://x.x.x.x:8765/sd/prometheus/sd-config?service=node-exporter - job_name: 'ceph-exporter' honor_labels: true http_sd_configs: - url: http://x.x.x.x:8765/sd/prometheus/sd-config?service=ceph-exporter When I open a run "netstat -ntlp" on the active mgr node, I see the 8765 port being used by docker. However, when I try to use the chrome browser to access the URLs listed in the Prometheus.yml file, the page times out. However, if I do this with the active manager on the cluster that was installed from scratch ( and not upgraded ), the URL for that cluster returns output( different for each URL ). So it appears to me that the service discovery function is not working for upgrades from quincy. Also, the ceph-exporter service was not installed on the cluster during the upgrade process. I manually added the service when I noticed that it was not there ( when comparing the from scratch cluster to the upgraded cluster ). Not sure if this will help or is even related, but I saw it in the cephadm log: 2023-11-15T04:22:30.789998+0000 mgr. CEPH-MON-01.mlmups (mgr.144601) 753 : cephadm 4 host CEPH-MON-02 `cephadm gather-facts` failed: Cannot decode JSON: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1425, in _run_cephadm_json return json.loads(''.join(out)) File "/lib64/python3.6/json/__init__.py", line 354, in loads return _default_decoder.decode(s) File "/lib64/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/lib64/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Is there any way to fix the service discovery? Thanks! -Brent Existing Clusters: Test: Reef 18.2.0 ( all virtual on nvme ) US Production(HDD): Reef 18.2.0 with 11 osd servers, 3 mons, 4 gateways, 2 iscsi gateways UK Production(HDD): Nautilus 14.2.22 with 18 osd servers, 3 mons, 4 gateways, 2 iscsi gateways US Production(SSD): Reef 18.2.0 Cephadm with 6 osd servers, 5 mons, 4 gateways UK Production(SSD): Reef 18.2.0 cephadm with 7 osd servers, 5 mons, 4 gateways

2 2

CloudStack and Ceph Day 2024
by 42on - Michiel Manten 22 Nov '23

22 Nov '23

Hello Ceph users, Together with ShapeBlue and Adyen, 42on is organizing a CloudStack and Ceph Day; this time in Amsterdam, The Netherlands. We are planning this for February 8 | 2024. We want to create a technical event that shares updates on both technologies, as well as 'use cases', stories, challenges and perhaps some crazy ideas or configurations. Let’s share information and make Ceph even bigger and better! I am still looking for some speakers who would love to share something about their Ceph infrastructure or configuration. As small as it might seem to you, any ideas are welcome and if you are in doubt about the subject, please message or call me to discuss. I would love to hear about your ideas. To RSVP and more information can be found here: https://www.eventbrite.nl/e/cloudstack-and-ceph-day-netherlands-2024-ticket… We hope to see you in Amsterdam once more. Sincerely, Michiel Manten

1 0

No SSL Dashboard working after installing mgr crt|key with RSA/4096 secp384r1
by Ackermann, Christoph 22 Nov '23

22 Nov '23

Hello all, today i got a new certificate for our internal domain based on RSA/4096 secp384r1. After inserting CRT and Key i got both "...updated" messages. After checking the dashboard i got an empty page and this error: health: HEALTH_ERR Module 'dashboard' has failed: key type unsupported So we tried to go back to the original state by removing CRT anf KEY but without success. The new key seems to stuck into the config [root@cephxxxx ~]# ceph config-key get mgr/dashboard/crt -----BEGIN CERTIFICATE----- MIIFqTCCBJGgAwIBAgIMB5tjLSz264Ic8zeHMA0GCSqGSIb3DQEBCwUAMEwxCzAJ [...] ItzkEzq4SZ6V1Jhuf4bFlOMBVAKgAwZ90gXlguoiFFQu5+NIqNljZ8Jz7d0jhH43 e3zhm5sn21+eIqRbiQ== -----END CERTIFICATE----- [root@cephxxxx ~]# ceph config-key get mgr/dashboard/key *Error ENOENT: * We tried to generate a self signed Cert but no luck. It looks like manger stays in an intermediate state. The only way to get back the dashboard is to disable SSL and use plain http. Can somebody explain this behaviour? Maybe secp384r1 elliptic curves aren't supported? How can we clean up SSL configuration? Thanks, Christoph Ackermann Ps we checked some Information like https://tracker.ceph.com/issues/57924#change-227744 and others but no luck...

2 3

[RGW][STS] How to use Roles to limit access to only buckets of one user?
by Rudenko Aleksandr 21 Nov '23

21 Nov '23

Hi, I have setup with one default tenant and next user/bucket structure: user1 bucket1 bucket11 user2 bucket2 user3 bucket3 IAM and STS APIs are enabled, user1 has roles=* capabilities. When user1 permit user2 to assume role with next permission policy: {"Version":"2012-10-17", "Statement":[ {"Effect":"Allow", "Action":["s3:*"], "Resource":"arn:aws:s3:::*"}] } user2 can use temporary credentials (after AssumeRole action) which give him access to ALL buckets of ALL users in this tenant (bucket3 for example). But I expect that access should be limited by user1 own buckets. I understand that roles=* caps it’s some kind of global admin permissions, but this caps are so powerful and give access to all buckets through roles. How can I use Role to limit access to only all own buckets? Now I can specify one or few exact buckets or ALL (*) in tenant. In AWS, we can give access to all buckets (*) but it will be all buckets in our account, not all buckets in S3.

1 0

RadosGW public HA traffic - best practices?
by Boris Behrens 21 Nov '23

21 Nov '23

Hi, I am looking for some experience on how people make their RGW public. Currently we use the follow: 3 IP addresses that get distributed via keepalived between three HAproxy instances, which then balance to three RGWs. The caveat is, that keepalived is PITA to get working in distributing a set of IP addresses, and it doesn't scale very well (up and down). The upside is, that it is really stable and customer nearly never have an availability problem. And we have 3 IPs that make some sort of LB. It serves up to 24Gbit in peak times, when all those backup jobs are running at night. But today I thought, what will happen if I just ditch the keepalived and configure thos addresses static to the haproxy hosts? How bad will the impact to a customer if I reboot one haproxy? Is there an easier, more scalable way if I want to spread the load even further without having an ingress HW LB (what I don't have)? I have a lot of hosts that would be able to host some POD with a haproxy and a RGW as container together, or even host the RGW alone in a container. It would just need to bridge two networks. But I currently do not have a way to use BGP to have one IP address split between a set of RGW instances. So long story short: What are your easy setups to serve public RGW traffic with some sort of HA and LB (without using a big HW LB that is capable of 100GBit traffic)? And have you experienced problems when you do not shift around IP addresses. Cheers Boris

4 4

really need help how to save old client out of hang?
by zxcs 21 Nov '23

21 Nov '23

Hi, Experts, we have an cephfs cluster 16.2.* run with multi active mds, and we have some old machine run with ubuntu 16.04 , so we mount these client using ceph-fuse. After a full mds process restart, all of these old ubuntu 16.04 clients cannot connect to ceph , `ls -lrth` or `df -hT` hang on the client node. and we can see mds log said `evicting unresponsive client *** after waiting ** seconds during restart` also checking the client node, no ceph-fuse process, (ps -ef | grep ceph or ps -ef | grep fuse). but there is a remount process (ps -ef | grep mount) in the client node. root 176516 1 0 01:38 ? 00:00:00 mount -i -o remount /data cannot kill this process using `sudo kill -9 176516`. we really need experts help us how to save this client out of hang. we cannot restart this client node due to there are some other critical service. Thanks a ton. zx

2 1

Previously synced bucket resharded after sync removed
by Szabo, Istvan (Agoda) 21 Nov '23

21 Nov '23

Hi, I had a multisite bucket which I've removed from sync completely and resharded on the master zone the bucket which was successful. On the 2nd site (which was expected) can't list anything inside that bucket anymore which is okay, the issue is how I can delete the data somehow? It was 50TB data there which I'd like to cleanup but at the moment everything shows 0, if I check the user space usage or the bucket space usage, all 0, however the data I'm sure still there because the used space is still the same as before in my cluster. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Re: CFP closing soon: Everything Open 2024 (Gladstone, Queensland, Australia, April 16-18)
by Tim Serong 21 Nov '23

21 Nov '23

Update: the CFP has been extended 'til November 30 (see http://lists.linux.org.au/pipermail/eo-announce/2023-November/000011.html) On 11/17/23 14:09, Tim Serong wrote: > Everything Open (auspiced by Linux Australia) is happening again in > 2024. The CFP closes at the end of this weekend (November 19): > > https://2024.everythingopen.au/programme/proposals/ > > More details below. > > -------- Forwarded Message -------- > Date: Sun, 15 Oct 2023 09:16:31 +1000 > From: Everything Open <contact(a)everythingopen.au> > To: eo-announce(a)lists.linux.org.au, announce(a)lists.linux.org.au > Subject: [Announce] Everything Open 2024: Call for Sessions Now Open > User-Agent: Roundcube Webmail/1.1.1 > > Submit your session proposals today - the Everything Open 2024 Call for > Sessions is now open. > > > ## Call for Sessions > > We invite you to submit a session proposal on a topic you are familiar > with via our proposals portal at > https://2024.everythingopen.au/programme/proposals/. > The Call for Sessions will remain open until 11:59pm on Sunday 19 > November 2023 anywhere on earth (AoE). > > There will be multiple streams catering for a wide range of interest > areas across the many facets of open technology, including Linux, open > source software, open hardware, standards, formats and documentation, > and our communities. > In keeping with the conference’s aim to be inclusive to all community > members, presentations can be aimed at any level, ranging from technical > deep-dives through to beginner and intermediate level presentations for > those who are newer to the subject. > > There will be two types of sessions at Everything Open: talks and > tutorials. Talks will nominally be 45 minutes long on a single topic > presented in lecture format. We will also have a few short talk slots of > 25 minutes available, which are perfect for people new to presenting at > a conference. Tutorials are interactive and hands-on in nature, > presented in classroom format. > Each accepted session will receive one Professional level ticket to > attend the conference. > > The Session Selection Committee is looking forward to reading your > submissions. We would also like to thank them for coming together and > volunteering their time to help put this conference together. > > > ## Sponsor Early > > As usual, we have a range of sponsorship opportunities available, for > the conference overall as well as the ability to contribute towards > specific parts of the event. > We encourage you to sponsor the conference early, to get the maximum > promotion during the lead up to the event. > If you or your organisation is interested in sponsoring Everything Open, > please get in touch via > https://2024.everythingopen.au/sponsors/prospectus/. > > ---- > Read this online at > https://2024.everythingopen.au/news/call-for-sessions-open/ > > > _______________________________________________ > announce mailing list > announce(a)lists.linux.org.au > http://lists.linux.org.au/mailman/listinfo/announce > > ----- End forwarded message ----- > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io

1 0

Why is min_size of erasure pools set to k+1
by Vladimir Brik 20 Nov '23

20 Nov '23

Could someone help me understand why it's a bad idea to set min_size of erasure-coded pools to k? From what I've read, the argument for k+1 is that if min_size is k and you lose an OSD during recovery after a failure of m OSDs, data will become unavailable. But how does setting min_size to k+1 help? If m=2, if you experience a double failure followed by another failure during recovery you still lost 3 OSDs and therefore your data because the pool wasn't set up to handle 3 concurrent failures, and the value of min_size is irrelevant. https://github.com/ceph/ceph/pull/8008 mentions inability to peer if min_size = k, but I don't understand why. Does that mean that if min_size=k and I lose m OSDs, and then an OSD is restarted during recovery, PGs will not peer even after the restarted OSD comes back online? Vlad

2 1

Re: Bug fixes in 17.2.7
by Konstantin Shalygin 20 Nov '23

20 Nov '23

Hi, > On Nov 20, 2023, at 19:24, Tobias Kulschewski <T.Kulschewski(a)uni-bonn.de> wrote: > > do you have a rough estimate of when this will happen? > > Not at this year I think. For now precedence for a 18.2.1 and last release of Pacific But you can request shaman build, and clone repo for your local usage k

1 0

Bug fixes in 17.2.7
by Tobias Kulschewski 20 Nov '23

20 Nov '23

Hi guys, thank you for releasing 17.2.7! Just wanted to ask, if the bug with the multipart upload [1] has been fixed in 17.2.7? When are you planning on fixing this bug? Best, Tobias [1] https://tracker.ceph.com/issues/58879_

2 1

blustore osd nearfull but no pgs on it
by Debian 20 Nov '23

20 Nov '23

Hi, after a massive rebalance(tunables) my small SSD-OSDs are getting full, i changed my crush rules so there are actual no pgs/pools on it, but the disks stay full: ceph version 14.2.21 (5ef401921d7a88aea18ec7558f7f9374ebd8f5a6) nautilus (stable) ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME 158 ssd 0.21999 1.00000 224 GiB 194 GiB 193 GiB 22 MiB 1002 MiB 30 GiB 86.68 1.49 0 up osd.158 inferring bluefs devices from bluestore path 1 : device size 0x37e4400000 : own 0x[1ad3f00000~23c600000] = 0x23c600000 : using 0x39630000(918 MiB) : bluestore has 0x46e2d0000(18 GiB) available when i recreate the osd the osd gets full again any suggestion? thx & best regards

5 11

304 response is not RFC9110 compliant
by Ondřej Kukla 20 Nov '23

20 Nov '23

Hello, I’ve noticed that 304 response from s3 and s3website api is not RFC9110 compliant. This is an issue especially for caching the content when you have a cache-control header set on the object. There was an old Issue and PR from 2020 fixing this issue but it was completely ignored. I’ve created a new issue so it will be able to get back to it. https://tracker.ceph.com/issues/63507 Regards, Ondrej

1 0

Rgw object deletion
by mahnoosh shahidi 20 Nov '23

20 Nov '23

Hi everyone, In a pacific cluster, deleting S3 objects finish successfully but the deleted objects still exist in the bucket list and we can delete them over again without any error. Apparently object deletion does not update bucket index data. The behaviour happens randomly in different buckets. This can be fixed by* radosgw-admin bucket check --fix --check-objects *command. Any idea why this is happening? and how can I fix it? Best Regards, Mahnoosh

2 1

Re: reef 18.2.1 QE Validation status
by Redouane Kachach 17 Nov '23

17 Nov '23

Hi Yuri, I've just backported to reef several fixes that I introduced in the last months for the rook orchestrator. Most of them are fixes for dashboard issues/crashes that only happen on Rook environments. The PR [1] has all the changes and it was merged into reef this morning. We really need these changes to be part of the next reef release as the upcoming Rook stable version will be based on it. Please, can you include those changes in the upcoming reef 18.2.1 release? [1] https://github.com/ceph/ceph/pull/54224 Thanks a lot, Redouane. On Mon, Nov 13, 2023 at 6:03 PM Yuri Weinstein <yweinste(a)redhat.com> wrote: > ---------- Forwarded message --------- > From: Venky Shankar <vshankar(a)redhat.com> > Date: Thu, Nov 9, 2023 at 11:52 PM > Subject: Re: [ceph-users] Re: reef 18.2.1 QE Validation status > To: Yuri Weinstein <yweinste(a)redhat.com> > Cc: dev <dev(a)ceph.io>, ceph-users <ceph-users(a)ceph.io> > > > Hi Yuri, > > On Fri, Nov 10, 2023 at 4:55 AM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > > > > I've updated all approvals and merged PRs in the tracker and it looks > > like we are ready for gibba, LRC upgrades pending approval/update from > > Venky. > > The smoke test failure is caused by missing (kclient) patches in > Ubuntu 20.04 that certain parts of the fs suite (via smoke tests) rely > on. More details here > > https://tracker.ceph.com/issues/63488#note-8 > > The kclient tests in smoke pass with other distro's and the fs suite > tests have been reviewed and look good. Run details are here > > https://tracker.ceph.com/projects/cephfs/wiki/Reef#07-Nov-2023 > > The smoke failure is noted as a known issue for now. Consider this run > as "fs approved". > > > > > On Thu, Nov 9, 2023 at 1:31 PM Radoslaw Zarzynski <rzarzyns(a)redhat.com> > wrote: > > > > > > rados approved! > > > > > > Details are here: > https://tracker.ceph.com/projects/rados/wiki/REEF#1821-Review. > > > > > > On Mon, Nov 6, 2023 at 10:33 PM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > > > > > > > > Details of this release are summarized here: > > > > > > > > https://tracker.ceph.com/issues/63443#note-1 > > > > > > > > Seeking approvals/reviews for: > > > > > > > > smoke - Laura, Radek, Prashant, Venky (POOL_APP_NOT_ENABLE failures) > > > > rados - Neha, Radek, Travis, Ernesto, Adam King > > > > rgw - Casey > > > > fs - Venky > > > > orch - Adam King > > > > rbd - Ilya > > > > krbd - Ilya > > > > upgrade/quincy-x (reef) - Laura PTL > > > > powercycle - Brad > > > > perf-basic - Laura, Prashant (POOL_APP_NOT_ENABLE failures) > > > > > > > > Please reply to this email with approval and/or trackers of known > > > > issues/PRs to address them. > > > > > > > > TIA > > > > YuriW > > > > _______________________________________________ > > > > Dev mailing list -- dev(a)ceph.io > > > > To unsubscribe send an email to dev-leave(a)ceph.io > > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > -- > Cheers, > Venky > >

7 22

cephadm user on cephadm rpm package
by Luis Domingues 17 Nov '23

17 Nov '23

Hi, I noticed when installing the cephadm rpm package, to bootstrap a cluster for example, that a user cephadm was created. But I do not see it used anywhere. What is the purpose of creating a user on the machine we install the local binary of cephadm? Luis Domingues Proton AG

2 3

blustore osd nearfull but no pgs on it
by Debian 17 Nov '23

17 Nov '23

Hi, after a massive rebalance(tunables) my small SSD-OSDs are getting full, i changed my crush rules so there are actual no pgs/pools on it, but the disks stay full: ceph version 14.2.21 (5ef401921d7a88aea18ec7558f7f9374ebd8f5a6) nautilus (stable) ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME 158 ssd 0.21999 1.00000 224 GiB 194 GiB 193 GiB 22 MiB 1002 MiB 30 GiB 86.68 1.49 0 up osd.158 inferring bluefs devices from bluestore path 1 : device size 0x37e4400000 : own 0x[1ad3f00000~23c600000] = 0x23c600000 : using 0x39630000(918 MiB) : bluestore has 0x46e2d0000(18 GiB) available when i recreate the osd the osd gets full again any suggestion? thx & best regards

1 0

Issue with using the block device inside a pod.
by Kushagr Gupta 17 Nov '23

17 Nov '23

Hi Team, Components: Kubernetes, Ceph Problem statement: We are trying to integrate Ceph with kubernetes. We are unable to utilize the block volume mode in the pod. Description: OS: Almalinux 8.8 Ceph version: 18.2 Kubernetes version: 1.28.2 We have deployed a single node kubernetes cluster and a 3 node ceph cluster. We are trying to integrate kubernetes with ceph using csi-rbd plugin. We have used the following link to do the same: https://docs.ceph.com/en/reef/rbd/rbd-kubernetes/ We are strugling with the usage of the pvc we created in the block mode. Kindly refer the "raw-block-pvc-4.yaml". We created a pod using the file "raw-block-pod-4.yaml". The pod was created successfully. The device path we used was /dev/xvda. We are unable to use this path inside the pod. Its not gettting shown in the lsblk of the pod and its not even shown in the df -kh of the pod. When we try to mount /dev/xvda on /tmp but we get permission denied error. We get a similar error when we try to make partiotion of the disk. The partition gets created but partiotion table is not updated. At the same time we are able to format the disk using the command mkfs.ext4 /dev/xvda. Kindly refer the output: " [root@kube-cluster-1 ceph_provisioner]# k exec -it pod-with-raw-block-volume-2 sh kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. sh-4.4# hostname pod-with-raw-block-volume-2 sh-4.4# df -kh Filesystem Size Used Avail Use% Mounted on overlay 70G 14G 57G 19% / tmpfs 64M 0 64M 0% /dev tmpfs 32G 0 32G 0% /sys/fs/cgroup /dev/mapper/almalinux-root 70G 14G 57G 19% /etc/hosts shm 64M 0 64M 0% /dev/shm tmpfs 63G 12K 63G 1% /run/secrets/ kubernetes.io/serviceaccount tmpfs 32G 0 32G 0% /proc/acpi tmpfs 32G 0 32G 0% /proc/scsi tmpfs 32G 0 32G 0% /sys/firmware sh-4.4# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 1G 0 loop loop1 7:1 0 500G 0 loop sda 8:0 0 446.6G 0 disk |-sda1 8:1 200M 0 part |-sda2 8:2 1M 0 part |-sda3 8:3 1G 0 part `-sda4 8:4 0 445.4G 0 part sdb 8:16 0 893.3G 0 disk rbd0 252:0 0 1G 0 disk rbd1 252:16 0 500G 0 disk rbd2 252:32 0 100G 0 disk sh-4.4# ls /dev core fd full mqueue null ptmx pts random shm stderr stdin stdout termination-log tty urandom xvda zero sh-4.4# " Altough we are able to see a volume corresponding to the PV. But we are not able to use the block device. Could anyone help me out. Thanks and Regards, Kushagra Gupta

2 1

Re: Large size differences between pgs
by Miroslav Svoboda 17 Nov '23

17 Nov '23

Namely, the problem what I am trying to solve is that with such a large cluster I will lose a lot of capacity like unused. I have deviation set to value 1 at the balancer, that is, if I'm not mistaken +-1pg per OSD and then due to the size dispersion between the largest and smallest PGs on the one OSDs host.Svoboda Miroslav -------- Původní zpráva --------Od: Anthony D'Atri <aad(a)dreamsnake.net> Datum: 15.11.23 21:54 (GMT+01:00) Komu: Miroslav Svoboda <miroslav.svoboda(a)metode.cz> Předmět: Re: [ceph-users] Large size differences between pgs How are you determining PG size?> On Nov 15, 2023, at 15:46, Miroslav Svoboda <miroslav.svoboda(a)metode.cz> wrote:> > Hi,is it possible decrease large size differences between pgs? I have 5PB cluster and differences between smalest and bigest pgs are somewhere about 25GB.thanks,Svoboda Miroslav> _______________________________________________> ceph-users mailing list -- ceph-users(a)ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Different behaviors for ceph kernel client in limiting IOPS when data pool enters `nearfull`?
by Matt Larson 17 Nov '23

17 Nov '23

On CentOS 7 systems with the CephFS kernel client, if the data pool has a `nearfull` status there is a slight reduction in write speeds (possibly 20-50% fewer IOPS). On a similar Rocky 8 system with the CephFS kernel client, if the data pool has `nearfull` status, a similar test shows write speeds at different block sizes shows the IOPS < 150 bottlenecked vs the typical write performance that might be with 20000-30000 IOPS at a particular block size. Is there any way to avoid the extremely bottlenecked IOPS seen on the Rocky 8 system CephFS kernel clients during the `nearfull` condition or to have behavior more similar to the CentOS 7 CephFS clients? Do different OS or Linux kernels have greatly different ways they respond or limit on the IOPS? Are there any options to adjust how they limit on IOPS? Thanks, Matt -- Matt Larson, PhD Madison, WI 53705 U.S.A.

3 6

CFP closing soon: Everything Open 2024 (Gladstone, Queensland, Australia, April 16-18)
by Tim Serong 17 Nov '23

17 Nov '23

Everything Open (auspiced by Linux Australia) is happening again in 2024. The CFP closes at the end of this weekend (November 19): https://2024.everythingopen.au/programme/proposals/ More details below. -------- Forwarded Message -------- Date: Sun, 15 Oct 2023 09:16:31 +1000 From: Everything Open <contact(a)everythingopen.au> To: eo-announce(a)lists.linux.org.au, announce(a)lists.linux.org.au Subject: [Announce] Everything Open 2024: Call for Sessions Now Open User-Agent: Roundcube Webmail/1.1.1 Submit your session proposals today - the Everything Open 2024 Call for Sessions is now open. ## Call for Sessions We invite you to submit a session proposal on a topic you are familiar with via our proposals portal at https://2024.everythingopen.au/programme/proposals/. The Call for Sessions will remain open until 11:59pm on Sunday 19 November 2023 anywhere on earth (AoE). There will be multiple streams catering for a wide range of interest areas across the many facets of open technology, including Linux, open source software, open hardware, standards, formats and documentation, and our communities. In keeping with the conference’s aim to be inclusive to all community members, presentations can be aimed at any level, ranging from technical deep-dives through to beginner and intermediate level presentations for those who are newer to the subject. There will be two types of sessions at Everything Open: talks and tutorials. Talks will nominally be 45 minutes long on a single topic presented in lecture format. We will also have a few short talk slots of 25 minutes available, which are perfect for people new to presenting at a conference. Tutorials are interactive and hands-on in nature, presented in classroom format. Each accepted session will receive one Professional level ticket to attend the conference. The Session Selection Committee is looking forward to reading your submissions. We would also like to thank them for coming together and volunteering their time to help put this conference together. ## Sponsor Early As usual, we have a range of sponsorship opportunities available, for the conference overall as well as the ability to contribute towards specific parts of the event. We encourage you to sponsor the conference early, to get the maximum promotion during the lead up to the event. If you or your organisation is interested in sponsoring Everything Open, please get in touch via https://2024.everythingopen.au/sponsors/prospectus/. ---- Read this online at https://2024.everythingopen.au/news/call-for-sessions-open/ _______________________________________________ announce mailing list announce(a)lists.linux.org.au http://lists.linux.org.au/mailman/listinfo/announce ----- End forwarded message -----

1 0

Upgrading From RHCS v4 to OSS Ceph
by jarulsam＠uwyo.edu 16 Nov '23

16 Nov '23

Hi everyone, I have a storage cluster running RHCS v4 (old, I know) and am looking to upgrade it soon. I would also like to migrate from RHCS to the open source version of Ceph at some point, as our support contract with RedHat for Ceph is likely going to not be renewed going forward. I was wondering if anyone has any advice as to how to upgrade our cluster with minimal production impact. I have the following server configuration: + 3x monitors + 3x metadata servers + 2x RadosGWs with 2x servers running HAproxy and keepalived for HA RadosGWs. + 19x OSDs - 110TB HDD and 1TB NVMe each. (Total ~2.1PB raw) Currently, I have RHCS v4 installed baremetal on RHEL 7. I see that newer versions of Ceph require containerized deployments so I am thinking it is best to first migrate to a containerized installation then try and upgrade everything else. My first inclination is to do the upgrade like this: 1. Move existing installation to containerized, maintain all the same versions and OS installations. 2. Pull one monitor, fresh reinstall RHEL 9, reinstall RHCS v4, readd to cluster. Repeat for all the monitors. 3. Pull one MDS, do the same as step 2 but for MDS. 4. Pull one RadosGW, do the same as step 2 but for RadosGW. 5. Pull one OSD, rebalance, fresh reinstall RHEL 9, reinstall RHCS v4, readd to cluster, rebalance. Repeat for all OSDs. 6. Upgrade RHCS to OSS Ceph Octopus -> Pacific -> Quincy -> Reef. Does such a plan seem reasonable? Are there any major pitfalls of an approach like this? Ideally, I would just rebuild an entire new cluster on Ceph Reef, however there are obvious budgetary issues with such a plan. My biggest concerns are with moving to a containerized installation, then migrating from RHCS to OSS Ceph. Any advice or feedback is much appreciated. Best, Josh

2 1

Debian 12 support
by nessero karuzo 16 Nov '23

16 Nov '23

Hi to all ceph community. I have a question about Debian 12 support for ceph 17. I didn’t find repo for that release at https://download.ceph.com/, does it supports and how to install ceph Quincy on Debian 12 ? Thank you to those who will answer.

9 12

ceph-volume lvm new-db throws errors
by Giuliano Maggi 16 Nov '23

16 Nov '23

Hi, ceph-volume lvm new-db does not work as expected due to missing /var/lib/ceph/osd/ceph-OSDID. However, the database device seems to be added to the OSD. Until now, it is unclear to me if the DB device was actually successfully added to the OSD or this is a bug in ceph version 17.2.7 The detailed explanation is in a github ticket: https://github.com/ceph/ceph/pull/52941#issuecomment-1814251890 does anyone else have this issue ? Any feedback is welcome of course. Thanks, Giuliano.

1 0