October 2023 - ceph-users

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

multifs and snapshots
by Dmitry Melekhov 12 Nov '24

12 Nov '24

Hello! There is note here https://docs.ceph.com/en/reef/dev/cephfs-snapshots/ about multiply filesystems and snapshots: If each FS gets its own pool things probably work, but this isn’t tested and may not be true. Is it still untested ? Thank you!

3 4

Help needed to configure erasure coding LRC plugin
by Michel Jouvin 16 Oct '24

16 Oct '24

Hi, As discussed in another thread (Crushmap rule for multi-datacenter erasure coding), I'm trying to create an EC pool spanning 3 datacenters (datacenters are present in the crushmap), with the objective to be resilient to 1 DC down, at least keeping the readonly access to the pool and if possible the read-write access, and have a storage efficiency better than 3 replica (let say a storage overhead <= 2). In the discussion, somebody mentioned LRC plugin as a possible jerasure alternative to implement this without tweaking the crushmap rule to implement the 2-step OSD allocation. I looked at the documentation (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/) but I have some questions if someone has experience/expertise with this LRC plugin. I tried to create a rule for using 5 OSDs per datacenter (15 in total), with 3 (9 in total) being data chunks and others being coding chunks. For this, based of my understanding of examples, I used k=9, m=3, l=4. Is it right? Is this configuration equivalent, in terms of redundancy, to a jerasure configuration with k=9, m=6? The resulting rule, which looks correct to me, is: -------- { "rule_id": 6, "rule_name": "test_lrc_2", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 15, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -4, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 5, "type": "host" }, { "op": "emit" } ] } ------------ Unfortunately, it doesn't work as expected: a pool created with this rule ends up with its pages active+undersize, which is unexpected for me. Looking at 'ceph health detail` output, I see for each page something like: pg 52.14 is stuck undersized for 27m, current state active+undersized, last acting [90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647] For each PG, there is 3 '2147483647' entries and I guess it is the reason of the problem. What are these entries about? Clearly it is not OSD entries... Looks like a negative number, -1, which in terms of crushmap ID is the crushmap root (named "default" in our configuration). Any trivial mistake I would have made? Thanks in advance for any help or for sharing any successful configuration? Best regards, Michel

4 24

Not all Bucket Shards being used
by Christian Kugler 01 Oct '24

01 Oct '24

Hi, I have trouble with large OMAP files in a cluster in the RGW index pool. Some background information about the cluster: There is CephFS and RBD usage on the main cluster but for this issue I think only S3 is interesting. There is one realm, one zonegroup with two zones which have a bidirectional sync set up. Since this does not allow for autoresharding we have to do it by hand in this cluster – looking forward to Reef! From the logs: cluster 2023-07-17T22:59:03.018722+0000 osd.75 (osd.75) 623978 : cluster [WRN] Large omap object found. Object: 34:bcec3016:::.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5:head PG: 34.680c373d (34.5) Key count: 962091 Size (bytes): 277963182 The offending bucket looks like this: # radosgw-admin bucket stats \ | jq '.[] | select(.marker =="3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9") |"\(.num_shards) \(.usage["rgw.main"].num_objects)"' -r 131 9463833 Last week the number of objects was about 12 million. Which is why I reshareded the offending bucket twice, I think. Once to 129 and the second time to 131 because I wanted some leeway (or lieway? scnr, Sage). Unfortunately, even after a week the objects were still to big (the log line above is quite recent), so I looked into it again. # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.0 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.1 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.2 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.3 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.4 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.6 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.7 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.8 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.9 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.10 # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V \ |xargs -IOMAP sh -c \ 'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l' 1013854 1011007 1012287 1011232 1013565 998262 1012777 1012713 1012230 1010690 997111 Apparently, only 11 shards are in use. This would explain why the "Key usage" (from the log line) is about ten times higher than I would expect. How can I deal with this issue? One thing I could try to fix this would be to reshard to a lower number, but I am not sure if there are any risks associated with "downsharding". After that I could reshard to something like 97. Or I could directly "downshard" to 97. Also, the second zone has a similar problem, but as the error messsage lets me know, this would be a bad idea. Will it just take more time until the sharding is transferred to the seconds zone? Best, Christian Kugler

2 5

Fixing BlueFS spillover (pacific 16.2.14)
by Chris Dunlop 09 Jul '24

09 Jul '24

Hi, tl;dr why are my osds still spilling? I've recently upgraded to 16.2.14 from 16.2.9 and started receiving bluefs spillover warnings (due to the "fix spillover alert" per the 16.2.14 release notes). E.g. from 'ceph health detail', the warning on one of these (there are a few): osd.76 spilled over 128 KiB metadata from 'db' device (56 GiB used of 60 GiB) to slow device This is a 15T HDD with only a 60G SSD for the db so it's not surprising it spilled as it's way below the recommendation for rbd usage at db size 1-2% of the storage size. There was some spare space on the db ssd so I increased the size of the db LV up over 400G and did an bluefs-bdev-expand. However, days later, I'm still getting the spillover warning for that osd, including after running a manual compact: # ceph tell osd.76 compact See attached perf-dump-76 for the perf dump output: # cephadm enter --name 'osd.76' ceph daemon 'osd.76' perf dump" | jq -r '.bluefs' In particular, if my understanding is correct, that's telling me the db available size is 487G (i.e. the LV expand worked), of which it's using 59G, and there's 128K spilled to the slow device: "db_total_bytes": 512309059584, # 487G "db_used_bytes": 63470305280, # 59G "slow_used_bytes": 131072, # 128K A "bluefs stats" also says the db is using 128K of slow storage (although perhaps it's getting the info from the same place as the perf dump?): # ceph tell osd.76 bluefs stats 1 : device size 0x7747ffe000 : using 0xea6200000(59 GiB) 2 : device size 0xe8d7fc00000 : using 0x6554d689000(6.3 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 8.8 MiB 1 WAL 0 B 2.5 GiB 0 B 0 B 0 B 751 MiB 8 DB 0 B 56 GiB 128 KiB 0 B 0 B 50 GiB 842 SLOW 0 B 0 B 0 B 0 B 0 B 0 B 0 TOTAL 0 B 58 GiB 128 KiB 0 B 0 B 0 B 850 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 3.9 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 71 GiB 282 MiB 0 B 0 B 62 GiB SLOW 0 B 0 B 0 B 0 B 0 B 0 B TOTAL 0 B 74 GiB 282 MiB 0 B 0 B 0 B >> SIZE << 0 B 453 GiB 14 TiB I had a look at the "DUMPING STATS" output in the logs bug I don't know how to interpret it. I did try calculating the total of the sizes on the "Sum" lines but that comes to 100G so I don't know what that all means. See attached log-stats-76. I also tried "ceph-kvstore-tool bluestore-kv ... stats": $ { cephadm unit --fsid $clusterid --name osd.76 stop cephadm shell --fsid $clusterid --name osd.76 -- ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-76 stats cephadm unit --fsid $clusterid --name osd.76 start } Output attached as bluestore-kv-stats-76. I can't see anything interesting in there, although again I don't really know how to interpret it. So... why is this osd db still spilling onto slow storage, and how do I fix things so it's no longer using the slow storage? And a bonus issue... on another osd that hasn't yet been resized (i.e. again with a grossly undersized 60G db on SSD with a 15T HDD) I'm also getting a spillover warning. The "bluefs stats" seems to be saying the db is NOT currently spilling (i.e. "0 B" the DB/SLOW position in the matrix), but there's "something" currently using 59G on the slow device: $ ceph tell osd.85 bluefs stats 1 : device size 0xeffffe000 : using 0x3a3900000(15 GiB) 2 : device size 0xe8d7fc00000 : using 0x7aea7434000(7.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 7.4 MiB 1 WAL 0 B 564 MiB 0 B 0 B 0 B 132 MiB 2 DB 0 B 11 GiB 0 B 0 B 0 B 8.1 GiB 177 SLOW 0 B 3.0 GiB 59 GiB 0 B 0 B 56 GiB 898 TOTAL 0 B 13 GiB 59 GiB 0 B 0 B 0 B 1072 MAXIMUMS: LOG 0 B 24 MiB 0 B 0 B 0 B 20 MiB WAL 0 B 2.8 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 22 GiB 448 KiB 0 B 0 B 18 GiB SLOW 0 B 3.3 GiB 62 GiB 0 B 0 B 62 GiB TOTAL 0 B 27 GiB 62 GiB 0 B 0 B 0 B >> SIZE << 0 B 57 GiB 14 TiB Is there anywhere that describes how to interpret this output, and specifically, what stuff is going into the SLOW row? Seemingly there's 898 "files" there, but not LOG, WAL or DB files - so what are they? Cheers, Chris

4 7

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

Re: MDS crashes to damaged metadata
by Patrick Donnelly 10 Jun '24

10 Jun '24

On Thu, Dec 15, 2022 at 9:32 AM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): > > 2022-12-08T13:05:48.919+0100 7f440afec700 -1 /build/ceph-16.2.10/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f440afec700 time 2022-12-08T13:05:48.921223+0100 > /build/ceph-16.2.10/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE) > > 12 Minutes lates the unlink_local error crashes appeared again. This time with a new file. During debugging we noticed a MTU mismatch between MDS (1500) and client (9000) with cephfs kernel mount. The client is also creating the snapshots via mkdir in the .snap directory. > > We disabled snapshot creation for now, but really need this feature. I uploaded the mds logs of the first crash along with the information above to https://tracker.ceph.com/issues/38452 > > I would greatly appreciate it, if you could answer me the following question: > > Is the Bug related to our MTU Mismatch? We fixed the MTU Issue going back to 1500 on all nodes in the ceph public network on the weekend also. I doubt it. > If you need a debug level 20 log of the ScatterLock for further analysis, i could schedule snapshots at the end of our workdays and increase the debug level 5 Minutes arround snap shot creation. This would be very helpful! -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 4

16.2.14: [progress WARNING root] complete: ev {UUID} does not exist
by Zakhar Kirpichenko 03 May '24

03 May '24

Hi, Mgr of my cluster logs this every few seconds: [progress WARNING root] complete: ev 7de5bb74-790b-4fda-8838-e4af4af18c62 does not exist [progress WARNING root] complete: ev fff93fce-b630-4141-81ee-19e7a3e61483 does not exist [progress WARNING root] complete: ev a02f6966-5b9f-49e8-89c4-b4fb8e6f4423 does not exist [progress WARNING root] complete: ev 8d318560-ff1a-477f-9386-43f6b51080bf does not exist [progress WARNING root] complete: ev ff3740a9-6434-470a-808f-a2762fb542a0 does not exist [progress WARNING root] complete: ev 7d0589f1-545e-4970-867b-8482ce48d7f0 does not exist [progress WARNING root] complete: ev 78d57e43-5be5-43f0-8b1a-cdc60e410892 does not exist I would appreciate an advice on what these warnings mean and how they can be resolved. Best regards, Zakhar

4 7

Re: Recoveries without any misplaced objects?
by Hector Martin 24 Apr '24

24 Apr '24

On 29/05/2023 20.55, Anthony D'Atri wrote: > Check the uptime for the OSDs in question I restarted all my OSDs within the past 10 days or so. Maybe OSD restarts are somehow breaking these stats? > >> On May 29, 2023, at 6:44 AM, Hector Martin <marcan(a)marcan.st> wrote: >> >> Hi, >> >> I'm watching a cluster finish a bunch of backfilling, and I noticed that >> quite often PGs end up with zero misplaced objects, even though they are >> still backfilling. >> >> Right now the cluster is down to 6 backfilling PGs: >> >> data: >> volumes: 1/1 healthy >> pools: 6 pools, 268 pgs >> objects: 18.79M objects, 29 TiB >> usage: 49 TiB used, 25 TiB / 75 TiB avail >> pgs: 262 active+clean >> 6 active+remapped+backfilling >> >> But there are no misplaced objects, and the misplaced column in `ceph pg >> dump` is zero for all PGs. >> >> If I do a `ceph pg dump_json`, I can see `num_objects_recovered` >> increasing for these PGs... but the misplaced count is still 0. >> >> Is there something else that would cause recoveries/backfills other than >> misplaced objects? Or perhaps there is a bug somewhere causing the >> misplaced object count to be misreported as 0 sometimes? >> >> # ceph -v >> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy >> (stable) >> >> - Hector >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > - Hector

3 3

Status of IPv4 / IPv6 dual stack?
by Robert Sander 23 Apr '24

23 Apr '24

Hi, as the documentation sends mixed signals in https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#ipv… "Note Binding to IPv4 is enabled by default, so if you just add the option to bind to IPv6 you’ll actually put yourself into dual stack mode." and https://docs.ceph.com/en/latest/rados/configuration/msgr2/#address-formats "Note The ability to bind to multiple ports has paved the way for dual-stack IPv4 and IPv6 support. That said, dual-stack operation is not yet supported as of Quincy v17.2.0." just the quick questions: Is a dual stacked networking with IPv4 and IPv6 now supported or not? From which version on is it considered stable? Are OSDs now able to register themselves with two IP addresses in the cluster map? MONs too? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

6 6

Ceph 16.2.x mon compactions, disk writes
by Zakhar Kirpichenko 17 Apr '24

17 Apr '24

Hi, Monitors in our 16.2.14 cluster appear to quite often run "manual compaction" tasks: debug 2023-10-09T09:30:53.888+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696843853892760, "job": 64225, "event": "flush_started", "num_memtables": 1, "num_entries": 715, "num_deletes": 251, "total_data_size": 3870352, "memory_usage": 3886744, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:30:53.904+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:30:53.910204) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:30:53.911004) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:32:08.956+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696843928961390, "job": 64228, "event": "flush_started", "num_memtables": 1, "num_entries": 1580, "num_deletes": 502, "total_data_size": 8404605, "memory_usage": 8465840, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:32:08.972+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:32:08.977739) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'logm .. 'logm; will stop at (end) debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:32:08.978512) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'logm .. 'logm; will stop at (end) debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.028+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696844009033151, "job": 64231, "event": "flush_started", "num_memtables": 1, "num_entries": 1430, "num_deletes": 251, "total_data_size": 8975535, "memory_usage": 9035920, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:33:29.044+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:33:29.049585) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:33:29.050355) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'paxos .. 'paxos; will stop at (end) I have removed a lot of interim log messages to save space. During each compaction the monitor process writes approximately 500-600 MB of data to disk over a short period of time. These writes add up to tens of gigabytes per hour and hundreds of gigabytes per day. Monitor rocksdb and compaction options are default: "mon_compact_on_bootstrap": "false", "mon_compact_on_start": "false", "mon_compact_on_trim": "true", "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", Is this expected behavior? Is this something I can adjust in order to extend the system storage life? Best regards, Zakhar

5 32

A couple OSDs not starting after host reboot
by Alison Peisker 11 Apr '24

11 Apr '24

Hi all, We rebooted all the nodes in our 17.2.5 cluster after performing kernel updates, but 2 of the OSDs on different nodes are not coming back up. This is a production cluster using cephadm. The error message from the OSD log is ceph-osd[87340]: ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-665: (2) No such file or directory The error message from ceph-volume is 2023-08-23T16:12:43.452-0500 7f0cad968600 2 bluestore(/dev/mapper/ceph--febad5a5--ba44--41aa--a39e--b9897f757752-osd--block--87e548f4--b9b5--4ed8--aca8--de703a341a50) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input We tried restarting the daemons and rebooting the node again, but still see the same error. Has anyone experienced this issue before? How do we fix this? Thanks, Alison

9 13

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

Why you might want packages not containers for Ceph deployments
by Matthew Vernon 25 Mar '24

25 Mar '24

Hi, In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0]. I'd like to suggest, briefly, that: * containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes. For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice. Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages): * you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice) We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part. Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems! I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder". I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :) Regards, and apologies for the wall of text, Matthew [0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

26 65

Upgrading from Pacific to Quincy fails with "Unexpected error"
by Reza Bakhshayeshi 13 Mar '24

13 Mar '24

Hi all, I have a problem regarding upgrading Ceph cluster from Pacific to Quincy version with cephadm. I have successfully upgraded the cluster to the latest Pacific (16.2.11). But when I run the following command to upgrade the cluster to 17.2.5, After upgrading 3/4 mgrs, the upgrade process stops with "Unexpected error". (everything is on a private network) ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v17.2.5 I also tried the 17.2.4 version. cephadm fails to check the hosts' status and marks them as offline: cephadm 2023-04-06T10:19:59.998510+0000 mgr.host9.arhpnd (mgr.4516356) 5782 : cephadm [DBG] host host4 (x.x.x.x) failed check cephadm 2023-04-06T10:19:59.998553+0000 mgr.host9.arhpnd (mgr.4516356) 5783 : cephadm [DBG] Host "host4" marked as offline. Skipping daemon refresh cephadm 2023-04-06T10:19:59.998581+0000 mgr.host9.arhpnd (mgr.4516356) 5784 : cephadm [DBG] Host "host4" marked as offline. Skipping gather facts refresh cephadm 2023-04-06T10:19:59.998609+0000 mgr.host9.arhpnd (mgr.4516356) 5785 : cephadm [DBG] Host "host4" marked as offline. Skipping network refresh cephadm 2023-04-06T10:19:59.998633+0000 mgr.host9.arhpnd (mgr.4516356) 5786 : cephadm [DBG] Host "host4" marked as offline. Skipping device refresh cephadm 2023-04-06T10:19:59.998659+0000 mgr.host9.arhpnd (mgr.4516356) 5787 : cephadm [DBG] Host "host4" marked as offline. Skipping osdspec preview refresh cephadm 2023-04-06T10:19:59.998682+0000 mgr.host9.arhpnd (mgr.4516356) 5788 : cephadm [DBG] Host "host4" marked as offline. Skipping autotune cluster 2023-04-06T10:20:00.000151+0000 mon.host8 (mon.0) 158587 : cluster [ERR] Health detail: HEALTH_ERR 9 hosts fail cephadm check; Upgrade: failed due to an unexpected exception cluster 2023-04-06T10:20:00.000191+0000 mon.host8 (mon.0) 158588 : cluster [ERR] [WRN] CEPHADM_HOST_CHECK_FAILED: 9 hosts fail cephadm check cluster 2023-04-06T10:20:00.000202+0000 mon.host8 (mon.0) 158589 : cluster [ERR] host host7 (x.x.x.x) failed check: Unable to reach remote host host7. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000213+0000 mon.host8 (mon.0) 158590 : cluster [ERR] host host2 (x.x.x.x) failed check: Unable to reach remote host host2. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000220+0000 mon.host8 (mon.0) 158591 : cluster [ERR] host host8 (x.x.x.x) failed check: Unable to reach remote host host8. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000228+0000 mon.host8 (mon.0) 158592 : cluster [ERR] host host4 (x.x.x.x) failed check: Unable to reach remote host host4. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000240+0000 mon.host8 (mon.0) 158593 : cluster [ERR] host host3 (x.x.x.x) failed check: Unable to reach remote host host3. Process exited with non-zero exit status 3 and here are some outputs of the commands: [root@host8 ~]# ceph -s cluster: id: xxx health: HEALTH_ERR 9 hosts fail cephadm check Upgrade: failed due to an unexpected exception services: mon: 5 daemons, quorum host8,host1,host7,host2,host9 (age 2w) mgr: host9.arhpnd(active, since 105m), standbys: host8.jowfih, host1.warjsr, host2.qyavjj mds: 1/1 daemons up, 3 standby osd: 37 osds: 37 up (since 8h), 37 in (since 3w) data: io: client: progress: Upgrade to 17.2.5 (0s) [............................] [root@host8 ~]# ceph orch upgrade status { "target_image": "my-private-repo/quay-io/ceph/ceph@sha256 :34c763383e3323c6bb35f3f2229af9f466518d9db926111277f5e27ed543c427", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "3/59 daemons upgraded", "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception", "is_paused": true } [root@host8 ~]# ceph cephadm check-host host7 check-host failed: Host 'host7' not found. Use 'ceph orch host ls' to see all managed hosts. [root@host8 ~]# ceph versions { "mon": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 1, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 }, "osd": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 37 }, "mds": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 4 }, "overall": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 47, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 } } The strange thing is I can rollback the cluster status by failing to not-upgraded mgr like this: ceph mgr fail ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v16.2.11 Would you happen to have any idea about this? Best regards, Reza

4 9

Upgrading nautilus / centos7 to octopus / ubuntu 20.04. - Suggestions and hints?
by Götz Reinicke 17 Jan '24

17 Jan '24

Hi, As I’v read and thought a lot about the migration as this is a bigger project, I was wondering if anyone has done that already and might share some notes or playbooks, because in all readings there where some parts missing or miss understandable to me. I do have some different approaches in mind, so may be you have some suggestions or hints. a) upgrade nautilus on centos 7 with the few missing features like dashboard and prometheus. After that migrate one node after an other to ubuntu 20.04 with octopus and than upgrade ceph to the recent stable version. b) migrate one node after an other to ubuntu 18.04 with nautilus and then upgrade to octupus and after that to ubuntu 20.04. or c) upgrade one node after an other to ubuntu 20.04 with octopus and join it to the cluster until all nodes are upgraded. For test I tried c) with a mon node, but adding that to the cluster fails with some failed state, still probing for the other mons. (I dont have the right log at hand right now.) So my questions are: a) What would be the best (most stable) migration path and b) is it in general possible to add a new octopus mon (not upgraded one) to a nautilus cluster, where the other mons are still on nautilus? I hope my thoughts and questions are understandable :) Thanks for any hint and suggestion. Best . Götz

7 11

Re: increasing number of (deep) scrubs
by Frank Schilder 13 Dec '23

13 Dec '23

Hi Dan, thanks for your answer. I don't have a problem with increasing osd_max_scrubs (=1 at the moment) as such. I would simply prefer a somewhat finer grained way of controlling scrubbing than just doubling or tripling it right away. Some more info. These 2 pools are data pools for a large FS. Unfortunately, we have a large percentage of small files, which is a pain for recovery and seemingly also for deep scrubbing. Our OSDs are about 25% used and I had to increase the warning interval already to 2 weeks. With all the warning grace parameters this means that we manage to deep scrub everything about every month. I need to plan for 75% utilisation and a 3 months period is a bit far on the risky side. Our data is to a large percentage cold data. Client reads will not do the check for us, we need to combat bit-rot pro-actively. The reasons I'm interested in parameters initiating more scrubs while also converting more scrubs into deep scrubs are, that 1) scrubs seem to complete very fast. I almost never catch a PG in state "scrubbing", I usually only see "deep scrubbing". 2) I suspect the low deep-scrub count is due to a low number of deep-scrubs scheduled and not due to conflicting per-OSD deep scrub reservations. With the OSD count we have and the distribution over 12 servers I would expect at least a peak of 50% OSDs being active in scrubbing instead of the 25% peak I'm seeing now. It ought to be possible to schedule more PGs for deep scrub than actually are. 3) Every OSD having only 1 deep scrub active seems to have no measurable impact on user IO. If I could just get more PGs scheduled with 1 deep scrub per OSD it would already help a lot. Once this is working, I can eventually increase osd_max_scrubs when the OSDs fill up. For now I would just like that (deep) scrub scheduling looks a bit harder and schedules more eligible PGs per time unit. If we can get deep scrubbing up to an average of 42PGs completing per hour with keeping osd_max_scrubs=1 to maintain current IO impact, we should be able to complete a deep scrub with 75% full OSDs in about 30 days. This is the current tail-time with 25% utilisation. I believe currently a deep scrub of a PG in these pools takes 2-3 hours. Its just a gut feeling from some repair and deep-scrub commands, I would need to check logs for more precise info. Increasing osd_max_scrubs would then be a further and not the only option to push for more deep scrubbing. My expectation would be that values of 2-3 are fine due to the increasingly higher percentage of cold data for which no interference with client IO will happen. Hope that makes sense and there is a way beyond bumping osd_max_scrubs to increase the number of scheduled and executed deep scrubs. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dvanders(a)gmail.com> Sent: 05 January 2023 15:36 To: Frank Schilder Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] increasing number of (deep) scrubs Hi Frank, What is your current osd_max_scrubs, and why don't you want to increase it? With 8+2, 8+3 pools each scrub is occupying the scrub slot on 10 or 11 OSDs, so at a minimum it could take 3-4x the amount of time to scrub the data than if those were replicated pools. If you want the scrub to complete in time, you need to increase the amount of scrub slots accordingly. On the other hand, IMHO the 1-week deadline for deep scrubs is often much too ambitious for large clusters -- increasing the scrub intervals is one solution, or I find it simpler to increase mon_warn_pg_not_scrubbed_ratio and mon_warn_pg_not_deep_scrubbed_ratio until you find a ratio that works for your cluster. Of course, all of this can impact detection of bit-rot, which anyway can be covered by client reads if most data is accessed periodically. But if the cluster is mostly idle or objects are generally not read, then it would be preferable to increase slots osd_max_scrubs. Cheers, Dan On Tue, Jan 3, 2023 at 2:30 AM Frank Schilder <frans(a)dtu.dk> wrote: > > Hi all, > > we are using 16T and 18T spinning drives as OSDs and I'm observing that they are not scrubbed as often as I would like. It looks like too few scrubs are scheduled for these large OSDs. My estimate is as follows: we have 852 spinning OSDs backing a 8+2 pool with 2024 and an 8+3 pool with 8192 PGs. On average I see something like 10PGs of pool 1 and 12 PGs of pool 2 (deep) scrubbing. This amounts to only 232 out of 852 OSDs scrubbing and seems to be due to a conservative rate of (deep) scrubs being scheduled. The PGs (dep) scrub fairly quickly. > > I would like to increase gently the number of scrubs scheduled for these drives and *not* the number of scrubs per OSD. I'm looking at parameters like: > > osd_scrub_backoff_ratio > osd_deep_scrub_randomize_ratio > > I'm wondering if lowering osd_scrub_backoff_ratio to 0.5 and, maybe, increasing osd_deep_scrub_randomize_ratio to 0.2 would have the desired effect? Are there other parameters to look at that allow gradual changes in the number of scrubs going on? > > Thanks a lot for your help! > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 3

CephFS pool not releasing space after data deletion
by Kuhring, Mathias 05 Dec '23

05 Dec '23

Dear Ceph users, Our CephFS is not releasing/freeing up space after deleting hundreds of terabytes of data. By now, this drives us in a "nearfull" osd/pool situation and thus throttles IO. We are on ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable). Recently, we moved a bunch of data to a new pool with better EC. This was done by adding a new EC pool to the FS. Then assigning the FS root to the new EC pool via the directory layout xattr (so all new data is written to the new pool). And finally copying old data to new folders. I swapped the data as follows to remain the old directory structures. I also made snapshots for validation purposes. So basically: cp -r mymount/mydata/ mymount/new/ # this creates copy on new pool mkdir mymount/mydata/.snap/tovalidate mkdir mymount/new/mydata/.snap/tovalidate mv mymount/mydata/ mymount/old/ mv mymount/new/mydata mymount/ I could see the increase of data in the new pool as expected (ceph df). I compared the snapshots with hashdeep to make sure the new data is alright. Then I went ahead deleting the old data, basically: rmdir mymount/old/mydata/.snap/* # this also included a bunch of other older snapshots rm -r mymount/old/mydata At first we had a bunch of PGs with snaptrim/snaptrim_wait. But they are done for quite some time now. And now, already two weeks later the size of the old pool still hasn't really decreased. I'm still waiting for around 500 TB to be released (and much more is planned). I honestly have no clue, where to go from here. From my point of view (i.e. the CephFS mount), the data is gone. I also never hard/soft-linked it anywhere. This doesn't seem to be a regular issue. At least I couldn't find anything related or resolved in the docs or user list, yet. If anybody has an idea how to resolve this, I would highly appreciate it. Best Wishes, Mathias

3 4

v17.2.7 Quincy released
by Yuri Weinstein 12 Nov '23

12 Nov '23

We're happy to announce the 7th backport release in the Quincy series. https://ceph.io/en/news/blog/2023/v17-2-7-quincy-released/ Notable Changes --------------- * `ceph mgr dump` command now displays the name of the Manager module that registered a RADOS client in the `name` field added to elements of the `active_clients` array. Previously, only the address of a module's RADOS client was shown in the `active_clients` array. * mClock Scheduler: The mClock scheduler (default scheduler in Quincy) has undergone significant usability and design improvements to address the slow backfill issue. Some important changes are: * The 'balanced' profile is set as the default mClock profile because it represents a compromise between prioritizing client IO or recovery IO. Users can then choose either the 'high_client_ops' profile to prioritize client IO or the 'high_recovery_ops' profile to prioritize recovery IO. * QoS parameters including reservation and limit are now specified in terms of a fraction (range: 0.0 to 1.0) of the OSD's IOPS capacity. * The cost parameters (osd_mclock_cost_per_io_usec_* and osd_mclock_cost_per_byte_usec_*) have been removed. The cost of an operation is now determined using the random IOPS and maximum sequential bandwidth capability of the OSD's underlying device. * Degraded object recovery is given higher priority when compared to misplaced object recovery because degraded objects present a data safety issue not present with objects that are merely misplaced. Therefore, backfilling operations with the 'balanced' and 'high_client_ops' mClock profiles may progress slower than what was seen with the 'WeightedPriorityQueue' (WPQ) scheduler. * The QoS allocations in all mClock profiles are optimized based on the above fixes and enhancements. * For more detailed information see: https://docs.ceph.com/en/quincy/rados/configuration/mclock-config-ref/ * RGW: S3 multipart uploads using Server-Side Encryption now replicate correctly in multi-site. Previously, the replicas of such objects were corrupted on decryption. A new tool, ``radosgw-admin bucket resync encrypted multipart``, can be used to identify these original multipart uploads. The ``LastModified`` timestamp of any identified object is incremented by 1 nanosecond to cause peer zones to replicate it again. For multi-site deployments that make any use of Server-Side Encryption, we recommended running this command against every bucket in every zone after all zones have upgraded. * CephFS: MDS evicts clients which are not advancing their request tids which causes a large buildup of session metadata resulting in the MDS going read-only due to the RADOS operation exceeding the size threshold. `mds_session_metadata_threshold` config controls the maximum size that a (encoded) session metadata can grow. * CephFS: After recovering a Ceph File System post following the disaster recovery procedure, the recovered files under `lost+found` directory can now be deleted. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-17.2.7.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: b12291d110049b2f35e32e0de30d70e9a4c060d2

2 1

MDS stuck in rejoin
by Frank Schilder 10 Nov '23

10 Nov '23

Hi all, we had a client with the warning "[WRN] MDS_CLIENT_OLDEST_TID: 1 clients failing to advance oldest client/flush tid". I looked at the client and there was nothing going on, so I rebooted it. After the client was back, the message was still there. To clean this up I failed the MDS. Unfortunately, the MDS that took over is remained stuck in rejoin without doing anything. All that happened in the log was: [root@ceph-10 ceph]# tail -f ceph-mds.ceph-10.log 2023-07-20T15:54:29.147+0200 7fedb9c9f700 1 mds.2.896604 rejoin_start 2023-07-20T15:54:29.161+0200 7fedb9c9f700 1 mds.2.896604 rejoin_joint_start 2023-07-20T15:55:28.005+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896614 from mon.4 2023-07-20T15:56:00.278+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896615 from mon.4 [...] 2023-07-20T16:02:54.935+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896653 from mon.4 2023-07-20T16:03:07.276+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896654 from mon.4 After some time I decided to give another fail a try and, this time, the replacement daemon went to active state really fast. If I have a message like the above, what is the clean way of getting the client clean again (version: 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable))? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 16

Ceph OSD reported Slow operations
by prabhav＠cdac.in 08 Nov '23

08 Nov '23

In a production setup of 36 OSDs( SAS disks) totalling 180 TB allocated to a single Ceph Cluster with 3 monitors and 3 managers. There were 830 volumes and VMs created in Openstack with Ceph as a backend. On Sep 21, users reported slowness in accessing the VMs. Analysing the logs lead us to problem with SAS , Network congestion and Ceph configuration( as all default values were used). We updated the Network from 1Gbps to 10Gbps for public and cluster networking. There was no change. The ceph benchmark performance showed that 28 OSDs out of 36 OSDs reported very low IOPS of 30 to 50 while the remaining showed 300+ IOPS. We gradually started reducing the load on the ceph cluster and now the volumes count is 650. Now the slow operations has gradually reduced but I am aware that this is not the solution. Ceph configuration is updated with increasing the osd_journal_size to 10 GB, osd_max_backfills = 1 osd_recovery_max_active = 1 osd_recovery_op_priority = 1 bluestore_cache_trim_max_skip_pinned=10000 After one month, now we faced another issue with Mgr daemon stopped in all 3 quorums and 16 OSDs went down. From the ceph-mon,ceph-mgr.log could not get the reason. Please guide me as its a production setup

3 3

owner locked out of bucket via bucket policy
by Wesley Dillingham 08 Nov '23

08 Nov '23

I have a bucket which got injected with bucket policy which locks the bucket even to the bucket owner. The bucket now cannot be accessed (even get its info or delete bucket policy does not work) I have looked in the radosgw-admin command for a way to delete a bucket policy but do not see anything. I presume I will need to somehow remove the bucket policy from however it is stored in the bucket metadata / omap etc. If anyone can point me in the right direction on that I would appreciate it. Thanks Respectfully, *Wes Dillingham* wes(a)wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>

4 15

6.5 CephFS client - ceph_cap_reclaim_work [ceph] / ceph_con_workfn [libceph] hogged CPU
by Stefan Kooman 08 Nov '23

08 Nov '23

Hi, Since the 6.5 kernel addressed the issue with regards to regression in the readahead handling code... we went ahead and installed this kernel for a couple of mail / web clusters (Ubuntu 6.5.1-060501-generic #202309020842 SMP PREEMPT_DYNAMIC Sat Sep 2 08:48:34 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux). Since then we occasionally see the following being logged by the kernel: [Sun Sep 10 07:19:00 2023] workqueue: delayed_work [ceph] hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND [Sun Sep 10 08:41:24 2023] workqueue: ceph_con_workfn [libceph] hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND [Sun Sep 10 11:05:55 2023] workqueue: delayed_work [ceph] hogged CPU for >10000us 8 times, consider switching to WQ_UNBOUND [Sun Sep 10 12:54:38 2023] workqueue: ceph_con_workfn [libceph] hogged CPU for >10000us 8 times, consider switching to WQ_UNBOUND [Sun Sep 10 19:06:37 2023] workqueue: ceph_con_workfn [libceph] hogged CPU for >10000us 16 times, consider switching to WQ_UNBOUND [Mon Sep 11 10:53:33 2023] workqueue: ceph_con_workfn [libceph] hogged CPU for >10000us 32 times, consider switching to WQ_UNBOUND [Tue Sep 12 10:14:03 2023] workqueue: ceph_con_workfn [libceph] hogged CPU for >10000us 64 times, consider switching to WQ_UNBOUND [Tue Sep 12 11:14:33 2023] workqueue: ceph_cap_reclaim_work [ceph] hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND We wonder if this is a new phenomenon, or that it's rather logged in the new kernel and it was not before. However, we have hit a few OOM situations since we switched to the new kernel because of ceph_cap_reclaim_work events (OOM is because Apache threads keep piling up as it cannot access CephFS). We then also see MDS slow ops reported. This might be related to a backup job that is running on a backup server. We did not observe this behavior on 5.12.19 kernel. Ceph cluster is on 16.2.11 currently. Anyone has some insight on this? Thanks, Stefan

3 12

17.2.7 quincy
by Matthew Darwin 03 Nov '23

03 Nov '23

Hi all, I see17.2.7 quincy is published as debian-bullseye packages. So I tried it on a test cluster. I must say I was not expecting the big dashboard change in a patch release. Also all the "cluster utilization" numbers are all blank now (any way to fix it?), so the dashboard is much less usable now. Thoughts?

2 7

Re: RGW access logs with bucket name
by Boris Behrens 02 Nov '23

02 Nov '23

Bringing up that topic again: is it possible to log the bucket name in the rgw client logs? currently I am only to know the bucket name when someone access the bucket via https://TLD/bucket/object instead of https://bucket.TLD/object. Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens <bb(a)kervyn.de>: > Hi, > I am looking forward to move our logs from > /var/log/ceph/ceph-client...log to our logaggregator. > > Is there a way to have the bucket name in the log file? > > Or can I write the rgw_enable_ops_log into a file? Maybe I could work with > this. > > Cheers and happy new year > Boris > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

4 6

"cephadm version" in reef returns "AttributeError: 'CephadmContext' object has no attribute 'fsid'"
by Martin Conway 02 Nov '23

02 Nov '23

Hi I wrote before about issues I was having with cephadm in 18.2.0 Sorry, I didn't see the helpful replies because my mail service binned the responses. I still can't get the reef version of cephadm to work properly. I had updated the system rpm to reef (ceph repo) and also upgraded the containerised ceph daemons to reef before my first email. Both the system package cephadm and the one found at /var/lib/ceph/${fsid}/cephadm.* return the same error when running "cephadm version" Traceback (most recent call last): File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 9468, in <module> main() File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 9456, in main r = ctx.func(ctx) File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2108, in _infer_image ctx.image = infer_local_ceph_image(ctx, ctx.container_engine.path) File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2191, in infer_local_ceph_image container_info = get_container_info(ctx, daemon, daemon_name is not None) File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2154, in get_container_info matching_daemons = [d for d in daemons if daemon_name_or_type(d) == daemon_filter and d['fsid'] == ctx.fsid] File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2154, in <listcomp> matching_daemons = [d for d in daemons if daemon_name_or_type(d) == daemon_filter and d['fsid'] == ctx.fsid] File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 217, in __getattr__ return super().__getattribute__(name) AttributeError: 'CephadmContext' object has no attribute 'fsid' I am running into other issues as well, but I think they may point back to this issue of "'CephadmContext' object has no attribute 'fsid'" Any help would be appreciated. Regards, Martin Conway IT and Digital Media Manager Research School of Physics Australian National University Canberra ACT 2601 +61 2 6125 1599 https://physics.anu.edu.au<https://physics.anu.edu.au/>

4 8

Moving devices to a different device class?
by Matt Larson 01 Nov '23

01 Nov '23

I am looking to create a new pool that would be backed by a particular set of drives that are larger nVME SSDs (Intel SSDPF2NV153TZ, 15TB drives). Particularly, I am wondering about what is the best way to move devices from one pool and to direct them to be used in a new pool to be created. In this case, the documentation suggests I could want to assign them to a new device-class and have a placement rule that targets that device-class in the new pool. Currently the Ceph cluster has two device classes 'hdd' and 'ssd', and the larger 15TB drives were automatically assigned to the 'ssd' device class that is in use by a different pool. The `ssd` device classes are used in a placement rule targeting that class. The documentation describes that I could set a device class for an OSD with a command like: `ceph osd crush set-device-class CLASS OSD_ID [OSD_ID ..]` Class names can be arbitrary strings like 'big_nvme". Before setting a new device class to an OSD that already has an assigned device class, should use `ceph osd crush rm-device-class ssd osd.XX`. Can I proceed to directly remove these OSDs from the current device class and assign to a new device class? Should they be moved one by one? What is the way to safely protect data from the existing pool that they are mapped to? Thanks, Matt -- Matt Larson, PhD Madison, WI 53705 U.S.A.

4 5

find PG with large omap object
by Frank Schilder 31 Oct '23

31 Oct '23

Hi all, we had a bunch of large omap object warnings after a user deleted a lot of files on a ceph fs with snapshots. After the snapshots were rotated out, all but one of these warnings disappeared over time. However, one warning is stuck and I wonder if its something else. Is there a reasonable way (say, one-liner with no more than 120 characters) to get ceph to tell me which PG this is coming from? I just want to issue a deep scrub to check if it disappears and going through the logs and querying every single object for its key count seems a bit of a hassle for something that ought to be part of "ceph health detail". Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 6

Add nats_adapter
by Vahideh Alinouri 30 Oct '23

30 Oct '23

Dear Ceph Users, I am requesting the backporting changes related to the nats_adapter.lua. This feature is in a version newer than pacific, but we don't have it in pacific version. I would greatly appreciate it if someone from the Ceph development team backport this change to the pacific version. Best regards, Vahideh Alinouri

2 1

Re: Status of Quincy 17.2.5 ?
by Christian Rohmann 30 Oct '23

30 Oct '23

Hey everyone, On 20/10/2022 10:12, Christian Rohmann wrote: > 1) May I bring up again my remarks about the timing: > > On 19/10/2022 11:46, Christian Rohmann wrote: > >> I believe the upload of a new release to the repo prior to the >> announcement happens quite regularly - it might just be due to the >> technical process of releasing. >> But I agree it would be nice to have a more "bit flip" approach to >> new releases in the repo and not have the packages appear as updates >> prior to the announcement and final release and update notes. > By my observations sometimes there are packages available on the > download servers via the "last stable" folders such as > https://download.ceph.com/debian-quincy/ quite some time before the > announcement of a release is out. > I know it's hard to time this right with mirrors requiring some time > to sync files, but would be nice to not see the packages or have > people install them before there are the release notes and potential > pointers to changes out. Todays 16.2.11 release shows the exact issue I described above .... 1) 16.2.11 packages are already available via e.g. https://download.ceph.com/debian-pacific 2) release notes not yet merged: (https://github.com/ceph/ceph/pull/49839) thus https://ceph.io/en/news/blog/2022/v16-2-11-pacific-released/ show a 404 :-) 3) No announcement like https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/QOCU563UD3… to the ML yet. Regards Christian

2 2

dashboard ERROR exception
by farhad kh 30 Oct '23

30 Oct '23

i use ceph 17.2.6 and when i deploy two number of separate rgw realm with zonegroup and zone , dashboard enabled access for bouth object gateway and i can create user and bucket and etc .but when i trying create bucket in on of object gatways .i get this error in below: ------------ debug 2023-10-29T12:19:50.697+0000 7fd203a26700 0 [dashboard ERROR rest_client] RGW REST API failed PUT req status: 400 debug 2023-10-29T12:19:50.697+0000 7fd203a26700 0 [dashboard ERROR exception] Dashboard Exception Traceback (most recent call last): File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 304, in create lock_enabled) File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 534, in func_wrapper **kwargs) File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 563, in create_bucket return request(data=data, headers=headers) File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 323, in __call__ data, raw_content, headers) File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 452, in do_request resp.content) dashboard.rest_client.RequestException: RGW REST API failed request with status code 400 (b'{"Code":"InvalidLocationConstraint","Message":"The specified location-constr' b'aint is not valid","BucketName":"farhad2","RequestId":"tx000003fa9d80c50a79d' b'b6-00653e4de6-285b3-test","HostId":"285b3-test-test"}') During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler return handler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 258, in inner ret = func(*args, **kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py", line 191, in wrapper return func(*vpath, **params) File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 315, in create raise DashboardException(e, http_status_code=500, component='rgw') dashboard.exceptions.DashboardException: RGW REST API failed request with status code 400 (b'{"Code":"InvalidLocationConstraint","Message":"The specified location-constr' b'aint is not valid","BucketName":"farhad2","RequestId":"tx000003fa9d80c50a79d' b'b6-00653e4de6-285b3-test","HostId":"285b3-test-test"}') debug 2023-10-29T12:19:50.701+0000 7fd203a26700 0 [dashboard INFO request] [192.168.0.1:55833] [POST] [500] [0.031s] [admin] [252.0B] /api/rgw/bucket debug 2023-10-29T12:19:50.713+0000 7fd204a28700 0 [dashboard ERROR rest_client] RGW REST API failed GET req status: 404 debug 2023-10-29T12:19:50.715+0000 7fd204a28700 0 [dashboard ERROR exception] Dashboard Exception Traceback (most recent call last): File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 145, in proxy result = instance.proxy(method, path, params, None) File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 513, in proxy params, data) File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 534, in func_wrapper **kwargs) File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 507, in _proxy_request raw_content=True) File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 323, in __call__ data, raw_content, headers) File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 452, in do_request resp.content) dashboard.rest_client.RequestException: RGW REST API failed request with status code 404 (b'{"Code":"NoSuchBucket","RequestId":"tx0000086cdcd9547b301e2-00653e4de6-285b3' b'-test","HostId":"285b3-test-test"}') During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler return handler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 258, in inner ret = func(*args, **kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py", line 191, in wrapper return func(*vpath, **params) File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 275, in get result = self.proxy(daemon_name, 'GET', 'bucket', {'bucket': bucket}) File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 151, in proxy raise DashboardException(e, http_status_code=http_status_code, component='rgw') dashboard.exceptions.DashboardException: RGW REST API failed request with status code 404 (b'{"Code":"NoSuchBucket","RequestId":"tx0000086cdcd9547b301e2-00653e4de6-285b3' b'-test","HostId":"285b3-test-test"}') -------------------------------------------------- my cluster have tow realm : 1) rgw-realm= test , rgw-zonegroup=test(master) ,zone= test( master ) 2) rgw-realm= velero(default), rgw-zonegroup=velero(master) ,zone= velero( master And another question, is it possible that I can create users and buckets with the same name on each rgw-realm?For example, user Yan on both.

1 0

Stickyness of writing vs full network storage writing
by Hans Kaiser 28 Oct '23

28 Oct '23

3 3

Re: Ceph - Error ERANGE: (34) Numerical result out of range
by Eugen Block 28 Oct '23

28 Oct '23

So this is a new host (you didn't provide the osd tree)? In that case I would compare the ceph.conf files between a working and this failing host, and paste it here (mask sensitive data). It looks like the connection to the MONs is successful though, and "ceph-volume create" worked as well. You could try to avoid a crush update on start: [osd] osd crush update on start = false Or you could also try to manually assign the location: [osd.301] osd crush location = "root=ssd" Try one option at a time to see which one works (if at all). Zitat von Pardhiv Karri <meher4india(a)gmail.com>: > Hi Eugen, > > Thank you for the reply. For some reason I'm not getting individual reply > but only the digest. Below is the ceph -s output (renamed hostnames) and > the command I am using to create a bluestore OSD. It should create a OSD > with its hostname and then the OSD should be up but it is not creating the > host and just a rogue OSD which is down. > > [root@hbmon1 ~]# ceph -s > cluster: > id: f1579737-d2c9-49ab-a6fa-8ca952488120 > health: HEALTH_WARN > 116896/167701779 objects misplaced (0.070%) > > services: > mon: 3 daemons, quorum hbmon1,hbmon2,hbmon3 > mgr: hbmon2(active), standbys: hbmon1, hbmon3 > osd: 721 osds: 717 up, 716 in; 60 remapped pgs > rgw: 1 daemon active > > data: > pools: 13 pools, 32384 pgs > objects: 55.90M objects, 324TiB > usage: 973TiB used, 331TiB / 1.27PiB avail > pgs: 116896/167701779 objects misplaced (0.070%) > 32294 active+clean > 59 active+remapped+backfill_wait > 27 active+clean+scrubbing+deep > 3 active+clean+scrubbing > 1 active+remapped+backfilling > > io: > client: 237MiB/s rd, 635MiB/s wr, 10.66kop/s rd, 6.98kop/s wr > recovery: 12.9MiB/s, 1objects/s > > [root@hbmon1 ~]# > > > Command used to create OSD, "ceph-volume lvm create --data /dev/sda" > > > > Debug log output of OSD creation command. > > [root@dra1361 ~]# ceph-volume lvm create --data /dev/sda > Running command: /usr/bin/ceph-authtool --gen-print-key > Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd > --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new > 21e9a327-ada5-4734-ab5d-7be333d4f3cf > Running command: vgcreate --force --yes > ceph-81236ab2-f6e0-4cc3-9815-95c8dd16c6ef /dev/sda > stdout: Physical volume "/dev/sda" successfully created. > stdout: Volume group "ceph-81236ab2-f6e0-4cc3-9815-95c8dd16c6ef" > successfully created > Running command: lvcreate --yes -l 100%FREE -n > osd-block-21e9a327-ada5-4734-ab5d-7be333d4f3cf > ceph-81236ab2-f6e0-4cc3-9815-95c8dd16c6ef > stdout: Logical volume "osd-block-21e9a327-ada5-4734-ab5d-7be333d4f3cf" > created. > Running command: /usr/bin/ceph-authtool --gen-print-key > Running command: mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-301 > --> Absolute path not found for executable: restorecon > --> Ensure $PATH environment variable contains common executable locations > Running command: chown -h ceph:ceph > /dev/ceph-81236ab2-f6e0-4cc3-9815-95c8dd16c6ef/osd-block-21e9a327-ada5-4734-ab5d-7be333d4f3cf > Running command: chown -R ceph:ceph /dev/dm-0 > Running command: ln -s > /dev/ceph-81236ab2-f6e0-4cc3-9815-95c8dd16c6ef/osd-block-21e9a327-ada5-4734-ab5d-7be333d4f3cf > /var/lib/ceph/osd/ceph-301/block > Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring > /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o > /var/lib/ceph/osd/ceph-301/activate.monmap > stderr: 2023-10-27 19:48:57.789631 7ff36a340700 2 Event(0x7ff3640e2950 > nevent=5000 time_id=1).set_owner idx=0 owner=140683435575040 > 2023-10-27 19:48:57.789713 7ff369b3f700 2 Event(0x7ff36410f670 nevent=5000 > time_id=1).set_owner idx=1 owner=140683427182336 > 2023-10-27 19:48:57.789771 7ff36933e700 2 Event(0x7ff36413c4e0 nevent=5000 > time_id=1).set_owner idx=2 owner=140683418789632 > stderr: 2023-10-27 19:48:57.790044 7ff36c135700 1 Processor -- start > 2023-10-27 19:48:57.790100 7ff36c135700 1 -- - start start > 2023-10-27 19:48:57.790352 7ff36c135700 1 -- - --> 10.51.228.32:6789/0 -- > auth(proto 0 38 bytes epoch 0) v1 -- 0x7ff364175e70 con 0 > 2023-10-27 19:48:57.790368 7ff36c135700 1 -- - --> 10.51.228.33:6789/0 -- > auth(proto 0 38 bytes epoch 0) v1 -- 0x7ff3641762b0 con 0 > stderr: 2023-10-27 19:48:57.791313 7ff369b3f700 1 -- > 10.51.228.213:0/2678799534 learned_addr learned my addr > 10.51.228.213:0/2678799534 > stderr: 2023-10-27 19:48:57.791740 7ff36933e700 2 -- > 10.51.228.213:0/2678799534 >> 10.51.228.32:6789/0 conn(0x7ff36417f4e0 :-1 > s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got > newly_acked_seq 0 vs out_seq 0 > 2023-10-27 19:48:57.791763 7ff369b3f700 2 -- 10.51.228.213:0/2678799534 >> > 10.51.228.33:6789/0 conn(0x7ff36417be80 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ > pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0 > stderr: 2023-10-27 19:48:57.792414 7ff353fff700 1 -- > 10.51.228.213:0/2678799534 <== mon.1 10.51.228.33:6789/0 1 ==== mon_map > magic: 0 v1 ==== 442+0+0 (171445244 0 0) 0x7ff360001690 con 0x7ff36417be80 > 2023-10-27 19:48:57.792544 7ff353fff700 1 -- 10.51.228.213:0/2678799534 > <== mon.1 10.51.228.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 > ==== 33+0+0 (2209822748 0 0) 0x7ff3641762b0 con 0x7ff36417be80 > 2023-10-27 19:48:57.792686 7ff353fff700 1 -- 10.51.228.213:0/2678799534 > --> 10.51.228.33:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- > 0x7ff34c001880 con 0 > 2023-10-27 19:48:57.792722 7ff353fff700 1 -- 10.51.228.213:0/2678799534 > <== mon.0 10.51.228.32:6789/0 1 ==== mon_map magic: 0 v1 ==== 442+0+0 > (171445244 0 0) 0x7ff354001710 con 0x7ff36417f4e0 > stderr: 2023-10-27 19:48:57.792776 7ff353fff700 1 -- > 10.51.228.213:0/2678799534 <== mon.0 10.51.228.32:6789/0 2 ==== > auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (782387931 0 0) > 0x7ff354001c10 con 0x7ff36417f4e0 > 2023-10-27 19:48:57.792832 7ff353fff700 1 -- 10.51.228.213:0/2678799534 > --> 10.51.228.32:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- > 0x7ff34c0035b0 con 0 > stderr: 2023-10-27 19:48:57.793541 7ff353fff700 1 -- > 10.51.228.213:0/2678799534 <== mon.1 10.51.228.33:6789/0 3 ==== > auth_reply(proto 2 0 (0) Success) v1 ==== 222+0+0 (713751175 0 0) > 0x7ff3600022d0 con 0x7ff36417be80 > 2023-10-27 19:48:57.793740 7ff353fff700 1 -- 10.51.228.213:0/2678799534 > --> 10.51.228.33:6789/0 -- auth(proto 2 181 bytes epoch 0) v1 -- > 0x7ff34c0024d0 con 0 > 2023-10-27 19:48:57.793774 7ff353fff700 1 -- 10.51.228.213:0/2678799534 > <== mon.0 10.51.228.32:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 > ==== 222+0+0 (1047601458 0 0) 0x7ff3540022d0 con 0x7ff36417f4e0 > stderr: 2023-10-27 19:48:57.793868 7ff353fff700 1 -- > 10.51.228.213:0/2678799534 --> 10.51.228.32:6789/0 -- auth(proto 2 181 > bytes epoch 0) v1 -- 0x7ff34c005f10 con 0 > stderr: 2023-10-27 19:48:57.794682 7ff353fff700 1 -- > 10.51.228.213:0/2678799534 <== mon.1 10.51.228.33:6789/0 4 ==== > auth_reply(proto 2 0 (0) Success) v1 ==== 612+0+0 (1210392221 0 0) > 0x7ff360002cd0 con 0x7ff36417be80 > stderr: 2023-10-27 19:48:57.794875 7ff353fff700 1 -- > 10.51.228.213:0/2678799534 >> 10.51.228.32:6789/0 conn(0x7ff36417f4e0 :-1 > s=STATE_OPEN pgs=316443717 cs=1 l=1).mark_down > 2023-10-27 19:48:57.794897 7ff353fff700 2 -- 10.51.228.213:0/2678799534 >> > 10.51.228.32:6789/0 conn(0x7ff36417f4e0 :-1 s=STATE_OPEN pgs=316443717 cs=1 > l=1)._stop > 2023-10-27 19:48:57.794955 7ff353fff700 1 -- 10.51.228.213:0/2678799534 > --> 10.51.228.33:6789/0 -- mon_subscribe({monmap=0+}) v2 -- 0x7ff364176570 > con 0 > 2023-10-27 19:48:57.795071 7ff36c135700 1 -- 10.51.228.213:0/2678799534 > --> 10.51.228.33:6789/0 -- mon_subscribe({mgrmap=0+}) v2 -- 0x7ff364176b70 > con 0 > stderr: 2023-10-27 19:48:57.795186 7ff36c135700 1 -- > 10.51.228.213:0/2678799534 --> 10.51.228.33:6789/0 -- > mon_subscribe({osdmap=0}) v2 -- 0x7ff3641843c0 con 0 > stderr: 2023-10-27 19:48:57.795919 7ff353fff700 1 -- > 10.51.228.213:0/2678799534 <== mon.1 10.51.228.33:6789/0 5 ==== mon_map > magic: 0 v1 ==== 442+0+0 (171445244 0 0) 0x7ff3600032f0 con 0x7ff36417be80 > 2023-10-27 19:48:57.796020 7ff353fff700 1 -- 10.51.228.213:0/2678799534 > <== mon.1 10.51.228.33:6789/0 6 ==== mgrmap(e 255) v1 ==== 580+0+0 > (3748818868 0 0) 0x7ff3600037e0 con 0x7ff36417be80 > stderr: 2023-10-27 19:48:57.797732 7ff353fff700 1 -- > 10.51.228.213:0/2678799534 <== mon.1 10.51.228.33:6789/0 7 ==== > osd_map(1497788..1497788 src has 1496148..1497788) v3 ==== 383089+0+0 > (4062048124 0 0) 0x7ff34c0024d0 con 0x7ff36417be80 > stderr: 2023-10-27 19:48:57.797968 7ff36933e700 2 -- > 10.51.228.213:0/2678799534 >> 10.51.228.33:6800/5258 conn(0x7ff34c00ebb0 > :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got > newly_acked_seq 0 vs out_seq 0 > stderr: 2023-10-27 19:48:57.804679 7ff36c135700 1 -- > 10.51.228.213:0/2678799534 --> 10.51.228.33:6789/0 -- > mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- > 0x7ff364099090 con 0 > stderr: 2023-10-27 19:48:57.807820 7ff353fff700 1 -- > 10.51.228.213:0/2678799534 <== mon.1 10.51.228.33:6789/0 8 ==== > mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== > 72+0+66166 (1092875540 0 105479317) 0x7ff360072010 con 0x7ff36417be80 > stderr: 2023-10-27 19:48:57.894128 7ff36c135700 1 -- > 10.51.228.213:0/2678799534 --> 10.51.228.33:6789/0 -- > mon_command({"prefix": "mon getmap"} v 0) v1 -- 0x7ff3640d9070 con 0 > stderr: 2023-10-27 19:48:57.894988 7ff353fff700 1 -- > 10.51.228.213:0/2678799534 <== mon.1 10.51.228.33:6789/0 9 ==== > mon_command_ack([{"prefix": "mon getmap"}]=0 got monmap epoch 4 v4) v1 ==== > 76+0+438 (3852220838 0 1414311087) 0x7ff360062000 con 0x7ff36417be80 > stderr: got monmap epoch 4 > stderr: 2023-10-27 19:48:57.899563 7ff36c135700 1 -- > 10.51.228.213:0/2678799534 >> 10.51.228.33:6800/5258 conn(0x7ff34c00ebb0 > :-1 s=STATE_OPEN pgs=915966 cs=1 l=1).mark_down > 2023-10-27 19:48:57.899603 7ff36c135700 2 -- 10.51.228.213:0/2678799534 >> > 10.51.228.33:6800/5258 conn(0x7ff34c00ebb0 :-1 s=STATE_OPEN pgs=915966 cs=1 > l=1)._stop > 2023-10-27 19:48:57.899637 7ff36c135700 1 -- 10.51.228.213:0/2678799534 >> > 10.51.228.33:6789/0 conn(0x7ff36417be80 :-1 s=STATE_OPEN pgs=494490687 cs=1 > l=1).mark_down > 2023-10-27 19:48:57.899644 7ff36c135700 2 -- 10.51.228.213:0/2678799534 >> > 10.51.228.33:6789/0 conn(0x7ff36417be80 :-1 s=STATE_OPEN pgs=494490687 cs=1 > l=1)._stop > stderr: 2023-10-27 19:48:57.900080 7ff36c135700 1 -- > 10.51.228.213:0/2678799534 shutdown_connections > stderr: 2023-10-27 19:48:57.900797 7ff36c135700 1 -- > 10.51.228.213:0/2678799534 shutdown_connections > stderr: 2023-10-27 19:48:57.901041 7ff36c135700 1 -- > 10.51.228.213:0/2678799534 wait complete. > 2023-10-27 19:48:57.901079 7ff36c135700 1 -- 10.51.228.213:0/2678799534 >> > 10.51.228.213:0/2678799534 conn(0x7ff3641698a0 :-1 s=STATE_NONE pgs=0 cs=0 > l=0).mark_down > 2023-10-27 19:48:57.901090 7ff36c135700 2 -- 10.51.228.213:0/2678799534 >> > 10.51.228.213:0/2678799534 conn(0x7ff3641698a0 :-1 s=STATE_NONE pgs=0 cs=0 > l=0)._stop > Running command: ceph-authtool /var/lib/ceph/osd/ceph-301/keyring > --create-keyring --name osd.301 --add-key > AQAnFDxlSUVdERAAvv6P4q/MGml/tPx9Kka77w== > stdout: creating /var/lib/ceph/osd/ceph-301/keyring > added entity osd.301 auth auth(auid = 18446744073709551615 > key=AQAnFDxlSUVdERAAvv6P4q/MGml/tPx9Kka77w== with 0 caps) > Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-301/keyring > Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-301/ > Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore > bluestore --mkfs -i 301 --monmap /var/lib/ceph/osd/ceph-301/activate.monmap > --keyfile - --osd-data /var/lib/ceph/osd/ceph-301/ --osd-uuid > 21e9a327-ada5-4734-ab5d-7be333d4f3cf --setuser ceph --setgroup ceph > --> ceph-volume lvm prepare successful for: /dev/sda > Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-301 > Running command: ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev > /dev/ceph-81236ab2-f6e0-4cc3-9815-95c8dd16c6ef/osd-block-21e9a327-ada5-4734-ab5d-7be333d4f3cf > --path /var/lib/ceph/osd/ceph-301 > Running command: ln -snf > /dev/ceph-81236ab2-f6e0-4cc3-9815-95c8dd16c6ef/osd-block-21e9a327-ada5-4734-ab5d-7be333d4f3cf > /var/lib/ceph/osd/ceph-301/block > Running command: chown -h ceph:ceph /var/lib/ceph/osd/ceph-301/block > Running command: chown -R ceph:ceph /dev/dm-0 > Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-301 > Running command: systemctl enable > ceph-volume@lvm-301-21e9a327-ada5-4734-ab5d-7be333d4f3cf > stderr: Created symlink > /etc/systemd/system/multi-user.target.wants/ceph-volume(a)lvm-301-21e9a327-ada5-4734-ab5d-7be333d4f3cf.service > → /lib/systemd/system/ceph-volume@.service. > Running command: systemctl enable --runtime ceph-osd@301 > Running command: systemctl start ceph-osd@301 > --> ceph-volume lvm activate successful for osd ID: 301 > --> ceph-volume lvm create successful for: /dev/sda > [root@dra1361 ~]# > > > Log file of OSD 301 (modified the IP address for security reasons): > > > 2023-10-27 20:02:25.597254 7fbe3eee5e40 0 osd.301 0 crush map has features > 288232575208783872, adjusting msgr requires for osds > 2023-10-27 20:02:25.597293 7fbe3eee5e40 0 osd.301 0 load_pgs > 2023-10-27 20:02:25.597301 7fbe3eee5e40 0 osd.301 0 load_pgs opened 0 pgs > 2023-10-27 20:02:25.597303 7fbe3eee5e40 2 osd.301 0 superblock: I am > osd.301 > 2023-10-27 20:02:25.597304 7fbe3eee5e40 0 osd.301 0 using weightedpriority > op queue with priority op cut off at 64. > 2023-10-27 20:02:25.597390 7fbe3eee5e40 1 Processor -- start > 2023-10-27 20:02:25.597594 7fbe3eee5e40 1 Processor -- start > 2023-10-27 20:02:25.597933 7fbe3eee5e40 1 Processor -- start > 2023-10-27 20:02:25.598011 7fbe3eee5e40 1 Processor -- start > 2023-10-27 20:02:25.598086 7fbe3eee5e40 1 Processor -- start > 2023-10-27 20:02:25.598224 7fbe3eee5e40 1 Processor -- start > 2023-10-27 20:02:25.598467 7fbe3eee5e40 1 Processor -- start > 2023-10-27 20:02:25.598833 7fbe3eee5e40 -1 osd.301 0 log_to_monitors > {default=true} > 2023-10-27 20:02:25.599904 7fbe3eee5e40 1 -- 10.10.21.213:6800/20559 --> > 10.10.21.32:6789/0 -- auth(proto 0 28 bytes epoch 0) v1 -- 0x55cf49fcd180 > con 0 > 2023-10-27 20:02:25.599922 7fbe3eee5e40 1 -- 10.10.21.213:6800/20559 --> > 10.10.21.33:6789/0 -- auth(proto 0 28 bytes epoch 0) v1 -- 0x55cf49fcd400 > con 0 > 2023-10-27 20:02:25.601479 7fbe3de99700 2 -- 10.10.21.213:6800/20559 >> > 10.10.21.33:6789/0 conn(0x55cf4a291800 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ > pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0 > 2023-10-27 20:02:25.601712 7fbe3d698700 2 -- 10.10.21.213:6800/20559 >> > 10.10.21.32:6789/0 conn(0x55cf4a293000 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ > pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0 > 2023-10-27 20:02:25.602181 7fbe3508b700 1 -- 10.10.21.213:6800/20559 <== > mon.1 10.10.21.33:6789/0 1 ==== mon_map magic: 0 v1 ==== 442+0+0 (171445244 > 0 0) 0x55cf4a29efc0 con 0x55cf4a291800 > 2023-10-27 20:02:25.602305 7fbe3508b700 1 -- 10.10.21.213:6800/20559 <== > mon.1 10.10.21.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== > 33+0+0 (1293857214 0 0) 0x55cf49fcd400 con 0x55cf4a291800 > 2023-10-27 20:02:25.602475 7fbe3508b700 1 -- 10.10.21.213:6800/20559 --> > 10.10.21.33:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x55cf49fcd900 > con 0 > 2023-10-27 20:02:25.602507 7fbe3508b700 1 -- 10.10.21.213:6800/20559 <== > mon.0 10.10.21.32:6789/0 1 ==== mon_map magic: 0 v1 ==== 442+0+0 (171445244 > 0 0) 0x55cf4a29efc0 con 0x55cf4a293000 > 2023-10-27 20:02:25.602557 7fbe3508b700 1 -- 10.10.21.213:6800/20559 <== > mon.0 10.10.21.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== > 33+0+0 (2698054748 0 0) 0x55cf49fcd180 con 0x55cf4a293000 > 2023-10-27 20:02:25.602627 7fbe3508b700 1 -- 10.10.21.213:6800/20559 --> > 10.10.21.32:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x55cf49fcd400 > con 0 > 2023-10-27 20:02:25.603288 7fbe3508b700 1 -- 10.10.21.213:6800/20559 <== > mon.1 10.10.21.33:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== > 206+0+0 (3831096342 0 0) 0x55cf49fcd900 con 0x55cf4a291800 > 2023-10-27 20:02:25.603488 7fbe3508b700 1 -- 10.10.21.213:6800/20559 --> > 10.10.21.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x55cf49fcd180 > con 0 > 2023-10-27 20:02:25.603707 7fbe3508b700 1 -- 10.10.21.213:6800/20559 <== > mon.0 10.10.21.32:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== > 206+0+0 (1800961675 0 0) 0x55cf49fcd400 con 0x55cf4a293000 > 2023-10-27 20:02:25.603889 7fbe3508b700 1 -- 10.10.21.213:6800/20559 --> > 10.10.21.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x55cf49fcd900 > con 0 > 2023-10-27 20:02:25.604366 7fbe3508b700 1 -- 10.10.21.213:6800/20559 <== > mon.1 10.10.21.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== > 596+0+0 (1171153923 0 0) 0x55cf49fcd180 con 0x55cf4a291800 > 2023-10-27 20:02:25.604560 7fbe3508b700 1 -- 10.10.21.213:6800/20559 >> > 10.10.21.32:6789/0 conn(0x55cf4a293000 :-1 s=STATE_OPEN pgs=316446787 cs=1 > l=1).mark_down > 2023-10-27 20:02:25.604572 7fbe3508b700 2 -- 10.10.21.213:6800/20559 >> > 10.10.21.32:6789/0 conn(0x55cf4a293000 :-1 s=STATE_OPEN pgs=316446787 cs=1 > l=1)._stop > 2023-10-27 20:02:25.604622 7fbe3508b700 1 monclient: found mon.or1dra1301 > 2023-10-27 20:02:25.604663 7fbe3508b700 1 -- 10.10.21.213:6800/20559 --> > 10.10.21.33:6789/0 -- mon_subscribe({monmap=0+}) v2 -- 0x55cf49dc9680 con 0 > 2023-10-27 20:02:25.604712 7fbe3508b700 1 -- 10.10.21.213:6800/20559 --> > 10.10.21.33:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x55cf49fcd400 > con 0 > 2023-10-27 20:02:25.604792 7fbe3eee5e40 5 monclient: authenticate success, > global_id 1528606696 > 2023-10-27 20:02:25.605306 7fbe3508b700 1 -- 10.10.21.213:6800/20559 <== > mon.1 10.10.21.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 442+0+0 (171445244 > 0 0) 0x55cf4a29f200 con 0x55cf4a291800 > 2023-10-27 20:02:25.605398 7fbe3508b700 1 -- 10.10.21.213:6800/20559 <== > mon.1 10.10.21.33:6789/0 6 ==== auth_reply(proto 2 0 (0) Success) v1 ==== > 194+0+0 (2618733096 0 0) 0x55cf49fcd400 con 0x55cf4a291800 > 2023-10-27 20:02:25.605807 7fbe3eee5e40 1 -- 10.10.21.213:6800/20559 --> > 10.10.21.33:6789/0 -- mon_command({"prefix": "osd crush set-device-class", > "class": "ssd", "ids": ["301"]} v 0) v1 -- 0x55cf49dc98c0 con 0 > 2023-10-27 20:02:25.608362 7fbe3508b700 1 -- 10.10.21.213:6800/20559 <== > mon.1 10.10.21.33:6789/0 7 ==== mon_command_ack([{"prefix": "osd crush > set-device-class", "class": "ssd", "ids": ["301"]}]=0 osd.301 already set > to class ssdset-device-class item id 301 name 'osd.301' device_class 'ssd': > no change v1497811) v1 ==== 211+0+0 (3030640430 0 0) 0x55cf49dc98c0 con > 0x55cf4a291800 > 2023-10-27 20:02:25.608668 7fbe3eee5e40 1 -- 10.10.21.213:6800/20559 --> > 10.10.21.33:6789/0 -- mon_command({"prefix": "osd crush create-or-move", > "id": 301, "weight":3.4931, "args": ["host=or1dra1361", "root=default"]} v > 0) v1 -- 0x55cf49dc9b00 con 0 > 2023-10-27 20:02:25.611784 7fbe3508b700 1 -- 10.10.21.213:6800/20559 <== > mon.1 10.10.21.33:6789/0 8 ==== mon_command_ack([{"prefix": "osd crush > create-or-move", "id": 301, "weight":3.4931, "args": ["host=or1dra1361", > "root=default"]}]=-34 (34) Numerical result out of range v1497811) v1 ==== > 179+0+0 (1380436622 0 0) 0x55cf49dc9b00 con 0x55cf4a291800 > 2023-10-27 20:02:25.612011 7fbe3eee5e40 -1 osd.301 0 > mon_cmd_maybe_osd_create fail: '(34) Numerical result out of range': (34) > Numerical result out of range > 2023-10-27 20:02:25.612070 7fbe3eee5e40 -1 osd.301 0 init unable to > update_crush_location: (34) Numerical result out of range > [root@dra1361 /var/log/ceph]# > > > > Thanks, > Pardhiv Karri > > > > > > > On Fri, Oct 27, 2023 at 7:00 AM <ceph-users-request(a)ceph.io> wrote: > >> Send ceph-users mailing list submissions to >> ceph-users(a)ceph.io >> >> To subscribe or unsubscribe via email, send a message with subject or >> body 'help' to >> ceph-users-request(a)ceph.io >> >> You can reach the person managing the list at >> ceph-users-owner(a)ceph.io >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of ceph-users digest..." >> >> Today's Topics: >> >> 1. Re: Ceph - Error ERANGE: (34) Numerical result out of range >> (Eugen Block) >> 2. Re: [EXTERNAL] [Pacific] ceph orch device ls do not returns any HDD >> (Patrick Begou) >> 3. Re: [ext] CephFS pool not releasing space after data deletion >> (Kuhring, Mathias) >> 4. Re: "cephadm version" in reef returns "AttributeError: >> 'CephadmContext' object has no attribute 'fsid'" >> (John Mulligan) >> >> >> ---------------------------------------------------------------------- >> >> Date: Fri, 27 Oct 2023 11:56:38 +0000 >> From: Eugen Block <eblock(a)nde.ag> >> Subject: [ceph-users] Re: Ceph - Error ERANGE: (34) Numerical result >> out of range >> To: ceph-users(a)ceph.io >> Message-ID: >> <20231027115638.Horde.48HDZ8Azsv-ho_0pQNe0p-s(a)webmail.nde.ag> >> Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes >> >> Hi, >> >> please provide more information about your cluster, like 'ceph -s', >> 'ceph osd tree' and the exact procedure you used to create the OSDs. >> From your last post it seems like the OSD creation failed and this >> might be just a consequence of that? Do you have the logs from the OSD >> creation as well? Not just the logs from the failing OSD start. >> >> Thanks, >> Eugen >> >> Zitat von Pardhiv Karri <meher4india(a)gmail.com>: >> >> > Hi, >> > Trying to move a node/host under a new SSD root and getting below error. >> > Has anyone seen it and know the fix? the pg_num and pgp_num are same for >> > all pools so that is not the issue. >> > >> > [root@hbmon1 ~]# ceph osd crush move hbssdhost1 root=ssd >> > Error ERANGE: (34) Numerical result out of range >> > [root@hbmon1 ~]# >> > >> > Thanks, >> > Pardhiv >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> >> ------------------------------ >> >> Date: Fri, 27 Oct 2023 15:35:37 +0200 >> From: Patrick Begou <Patrick.Begou(a)univ-grenoble-alpes.fr> >> Subject: [ceph-users] Re: [EXTERNAL] [Pacific] ceph orch device ls do >> not returns any HDD >> To: ceph-users(a)ceph.io >> Message-ID: >> <1fe5ff89-5b96-20e6-988d-ab2dd514ca2f(a)univ-grenoble-alpes.fr> >> Content-Type: text/plain; charset=UTF-8; format=flowed >> >> Hi all, >> >> First of all I apologize if I've not done things correctly but these are >> some tests results. >> >> 1) I've compiled the main branch in a fresh podman container (Alma Linux >> 8) and installed. Successfull! >> 2) I have done a copy of the /etc/ceph directory of the host (member of >> the ceph cluster in Pacific 16.2.14) in this container (good or bad idea ?) >> 3) "ceph-volume inventory" works but with some error messages: >> >> [root@74285dcfa91f etc]# ceph-volume inventory >> stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or >> /sys expected. >> stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or >> /sys expected. >> stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or >> /sys expected. >> stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or >> /sys expected. >> stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or >> /sys expected. >> >> Device Path Size Device nodes rotates available >> Model name >> /dev/sdc 232.83 GB sdc True True >> SAMSUNG HE253GJ >> /dev/sda 232.83 GB sda True False >> SAMSUNG HE253GJ >> /dev/sdb 465.76 GB sdb True False >> WDC WD5003ABYX-1 >> 4) ceph version show: >> [root@74285dcfa91f etc]# ceph -v >> ceph version 18.0.0-6846-g2706ecac4a9 >> (2706ecac4a90447420904e42d6e0445134dff2be) reef (dev) >> >> >> 5) lsblk works (container launched with "--privileged" flag) >> [root@74285dcfa91f etc]# lsblk >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT >> sda 8:0 1 232.9G 0 disk >> |-sda1 8:1 3.9G 0 part >> |-sda2 8:2 1 3.9G 0 part [SWAP] >> `-sda3 8:3 1 225G 0 part >> sdb 8:16 1 465.8G 0 disk >> sdc 8:32 1 232.9G 0 disk >> >> But some commands do not works (my setup or ceph ?) >> >> [root@74285dcfa91f etc]# ceph orch device zap >> mostha1.legi.grenoble-inp.fr /dev/sdc --force >> Error EINVAL: Device path '/dev/sdc' not found on host >> 'mostha1.legi.grenoble-inp.fr' >> [root@74285dcfa91f etc]# >> >> [root@74285dcfa91f etc]# ceph orch device ls >> [root@74285dcfa91f etc]# >> >> Patrick >> >> >> Le 24/10/2023 à 22:43, Zack Cerza a écrit : >> > That's correct - it's the removable flag that's causing the disks to >> > be excluded. >> > >> > I actually just merged this PR last week: >> > https://github.com/ceph/ceph/pull/49954 >> > >> > One of the changes it made was to enable removable (but not USB) >> > devices, as there are vendors that report hot-swappable drives as >> > removable. Patrick, it looks like this may resolve your issue as well. >> > >> > >> > On Tue, Oct 24, 2023 at 5:57 AM Eugen Block <eblock(a)nde.ag> wrote: >> >> Hi, >> >> >> >>> May be because they are hot-swappable hard drives. >> >> yes, that's my assumption as well. >> >> >> >> >> >> Zitat von Patrick Begou <Patrick.Begou(a)univ-grenoble-alpes.fr>: >> >> >> >>> Hi Eugen, >> >>> >> >>> Yes Eugen, all the devices /dev/sd[abc] have the removable flag set >> >>> to 1. May be because they are hot-swappable hard drives. >> >>> >> >>> I have contacted the commit author Zack Cerza and he asked me for >> >>> some additional tests too this morning. I add him in copy to this >> >>> mail. >> >>> >> >>> Patrick >> >>> >> >>> Le 24/10/2023 à 12:57, Eugen Block a écrit : >> >>>> Hi, >> >>>> >> >>>> just to confirm, could you check that the disk which is *not* >> >>>> discovered by 16.2.11 has a "removable" flag? >> >>>> >> >>>> cat /sys/block/sdX/removable >> >>>> >> >>>> I could reproduce it as well on a test machine with a USB thumb >> >>>> drive (live distro) which is excluded in 16.2.11 but is shown in >> >>>> 16.2.10. Although I'm not a developer I tried to understand what >> >>>> changes were made in >> >>>> >> https://github.com/ceph/ceph/pull/46375/files#diff-330f9319b0fe352dff0486f6… >> and there's this >> >>>> line: >> >>>> >> >>>>> if get_file_contents(os.path.join(_sys_block_path, dev, >> >>>>> 'removable')) == "1": >> >>>>> continue >> >>>> The thumb drive is removable, of course, apparently that is filtered >> here. >> >>>> >> >>>> Regards, >> >>>> Eugen >> >>>> >> >>>> Zitat von Patrick Begou <Patrick.Begou(a)univ-grenoble-alpes.fr>: >> >>>> >> >>>>> Le 23/10/2023 à 03:04, 544463199(a)qq.com a écrit : >> >>>>>> I think you can try to roll back this part of the python code and >> >>>>>> wait for your good news :) >> >>>>> >> >>>>> Not so easy 😕 >> >>>>> >> >>>>> >> >>>>> [root@e9865d9a7f41 ceph]# git revert >> >>>>> 4fc6bc394dffaf3ad375ff29cbb0a3eb9e4dbefc >> >>>>> Auto-merging src/ceph-volume/ceph_volume/tests/util/test_device.py >> >>>>> CONFLICT (content): Merge conflict in >> >>>>> src/ceph-volume/ceph_volume/tests/util/test_device.py >> >>>>> Auto-merging src/ceph-volume/ceph_volume/util/device.py >> >>>>> CONFLICT (content): Merge conflict in >> >>>>> src/ceph-volume/ceph_volume/util/device.py >> >>>>> Auto-merging src/ceph-volume/ceph_volume/util/disk.py >> >>>>> CONFLICT (content): Merge conflict in >> >>>>> src/ceph-volume/ceph_volume/util/disk.py >> >>>>> error: could not revert 4fc6bc394df... ceph-volume: Optionally >> >>>>> consume loop devices >> >>>>> >> >>>>> Patrick >> >>>>> _______________________________________________ >> >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >> >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >>>> >> >>>> _______________________________________________ >> >>>> ceph-users mailing list -- ceph-users(a)ceph.io >> >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >>> _______________________________________________ >> >>> ceph-users mailing list -- ceph-users(a)ceph.io >> >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> >> >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> ------------------------------ >> >> Date: Fri, 27 Oct 2023 13:52:03 +0000 >> From: "Kuhring, Mathias" <mathias.kuhring(a)bih-charite.de> >> Subject: [ceph-users] Re: [ext] CephFS pool not releasing space after >> data deletion >> To: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io>, "frans(a)dtu.dk" >> <frans(a)dtu.dk> >> Message-ID: <a5bb6a0a-aab5-402d-8ee3-68eccabb7b6b(a)bih-charite.de> >> Content-Type: text/plain; charset="utf-8" >> >> Dear ceph users, >> >> We are wondering, if this might be the same issue as with this bug: >> https://tracker.ceph.com/issues/52581 >> >> Except that we seem to have been snapshots dangling on the old pool. >> And the bug report snapshots dangling on the new pool. >> But maybe it's both? >> >> I mean, once the global root layout was created to a new pool, >> the new pool became in charge for snapshooting at least of new data, right? >> What about data which is overwritten? Is there a conflict of >> responsibility? >> >> We do have similar listings of snaps with "ceph osd pool ls detail", I >> think: >> >> 0|0[root@osd-1 ~]# ceph osd pool ls detail | grep -B 1 removed_snaps_queue >> pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 1 >> object_hash rjenkins pg_num 115 pgp_num 107 pg_num_target 32 >> pgp_num_target 32 autoscale_mode on last_change 803558 lfor >> 0/803250/803248 flags hashpspool,selfmanaged_snaps stripe_width 0 >> expected_num_objects 1 application cephfs >> removed_snaps_queue >> >> [3541~1,36e4~1,379f~2,3862~1,3876~1,387d~1,388b~1,389a~1,38a6~1,38bc~1,3993~1,3999~1,39a0~1,39a7~1,39ae~1,39b5~3,39be~1,39c5~1,39cc~1] >> -- >> pool 3 'hdd_ec' erasure profile hdd_ec size 3 min_size 2 crush_rule 3 >> object_hash rjenkins pg_num 2048 pgp_num 2048 autoscale_mode off >> last_change 803558 lfor 0/87229/87229 flags >> hashpspool,ec_overwrites,selfmanaged_snaps stripe_width 8192 application >> cephfs >> removed_snaps_queue >> >> [3541~1,36e4~1,379f~2,3862~1,3876~1,387d~1,388b~1,389a~1,38a6~1,38bc~1,3993~1,3999~1,39a0~1,39a7~1,39ae~1,39b5~3,39be~1,39c5~1,39cc~1] >> -- >> pool 20 'hdd_ec_8_2_pool' erasure profile hdd_ec_8_2_profile size 10 >> min_size 9 crush_rule 5 object_hash rjenkins pg_num 8192 pgp_num 8192 >> autoscale_mode off last_change 803558 lfor 0/0/681917 flags >> hashpspool,ec_overwrites,selfmanaged_snaps stripe_width 32768 >> application cephfs >> removed_snaps_queue >> >> [3541~1,36e4~1,379f~2,3862~1,3876~1,387d~1,388b~1,389a~1,38a6~1,38bc~1,3993~1,3999~1,39a0~1,39a7~1,39ae~1,39b5~3,39be~1,39c5~1,39cc~1] >> >> >> Here, pool hdd_ec_8_2_pool is the one we recently assigned to the root >> layout. >> Pool hdd_ec is the one which was assigned before and which won't release >> space (at least where I know of). >> >> Is this removed_snaps_queue the same as removed_snaps in the bug issue >> (i.e. the label was renamed)? >> And is it normal that all queues list the same info or should this be >> different per pool? >> Might this be related to pools having now share responsibility over some >> snaps due to layout changes? >> >> And for the big question: >> How can I actually trigger/speedup the removal of those snaps? >> I find the removed_snaps/removed_snaps_queue mentioned a few times in >> the user list. >> But never with some conclusive answer how to deal with them. >> And the only mentions in the docs are just change logs. >> >> I also looked into and started cephfs stray scrubbing: >> >> https://docs.ceph.com/en/latest/cephfs/scrub/#evaluate-strays-using-recursi… >> But according to the status output, no scrubbing is actually active. >> >> I would appreciate any further ideas. Thanks a lot. >> >> Best Wishes, >> Mathias >> >> On 10/23/2023 12:42 PM, Kuhring, Mathias wrote: >> > Dear Ceph users, >> > >> > Our CephFS is not releasing/freeing up space after deleting hundreds of >> > terabytes of data. >> > By now, this drives us in a "nearfull" osd/pool situation and thus >> > throttles IO. >> > >> > We are on ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) >> > quincy (stable). >> > >> > Recently, we moved a bunch of data to a new pool with better EC. >> > This was done by adding a new EC pool to the FS. >> > Then assigning the FS root to the new EC pool via the directory layout >> xattr >> > (so all new data is written to the new pool). >> > And finally copying old data to new folders. >> > >> > I swapped the data as follows to remain the old directory structures. >> > I also made snapshots for validation purposes. >> > >> > So basically: >> > cp -r mymount/mydata/ mymount/new/ # this creates copy on new pool >> > mkdir mymount/mydata/.snap/tovalidate >> > mkdir mymount/new/mydata/.snap/tovalidate >> > mv mymount/mydata/ mymount/old/ >> > mv mymount/new/mydata mymount/ >> > >> > I could see the increase of data in the new pool as expected (ceph df). >> > I compared the snapshots with hashdeep to make sure the new data is >> alright. >> > >> > Then I went ahead deleting the old data, basically: >> > rmdir mymount/old/mydata/.snap/* # this also included a bunch of other >> > older snapshots >> > rm -r mymount/old/mydata >> > >> > At first we had a bunch of PGs with snaptrim/snaptrim_wait. >> > But they are done for quite some time now. >> > And now, already two weeks later the size of the old pool still hasn't >> > really decreased. >> > I'm still waiting for around 500 TB to be released (and much more is >> > planned). >> > >> > I honestly have no clue, where to go from here. >> > From my point of view (i.e. the CephFS mount), the data is gone. >> > I also never hard/soft-linked it anywhere. >> > >> > This doesn't seem to be a regular issue. >> > At least I couldn't find anything related or resolved in the docs or >> > user list, yet. >> > If anybody has an idea how to resolve this, I would highly appreciate it. >> > >> > Best Wishes, >> > Mathias >> > >> > >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> -- >> Mathias Kuhring >> >> Dr. rer. nat. >> Bioinformatician >> HPC & Core Unit Bioinformatics >> Berlin Institute of Health at Charité (BIH) >> >> E-Mail: mathias.kuhring(a)bih-charite.de >> Mobile: +49 172 3475576 >> >> >> ------------------------------ >> >> Date: Fri, 27 Oct 2023 09:57:31 -0400 >> From: John Mulligan <phlogistonjohn(a)asynchrono.us> >> Subject: [ceph-users] Re: "cephadm version" in reef returns >> "AttributeError: 'CephadmContext' object has no attribute 'fsid'" >> To: ceph-users(a)ceph.io >> Message-ID: >> < >> 7411686.LvFx2qVVIh(a)li-241d88cc-27c5-11b2-a85c-c640472b3c85.ibm.com> >> Content-Type: text/plain; charset="us-ascii" >> >> On Friday, October 27, 2023 2:40:17 AM EDT Eugen Block wrote: >> > Are the issues you refer to the same as before? I don't think this >> > version issue is the root cause, I do see it as well in my test >> > cluster(s) but the rest works properly except for the tag issue I >> > already reported which you can easily fix by setting the config value >> > for the default image >> > ( >> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/LASBJCSPFGD >> > YAWPVE2YLV2ZLF3HC5SLS/#LASBJCSPFGDYAWPVE2YLV2ZLF3HC5SLS). Or are there >> new >> > issues you encountered? >> >> >> I concur. That `cephadm version` failure is/was a known issue but should >> not >> be the cause of any other issues. On the main branch `cephadm version` no >> longer fails this way - rather, it reports the version of a cephadm build >> and >> no longer inspects a container image. We can look into backporting this >> before the next reef release. >> >> The issue related to the container image tag that Eugen filed has also >> been >> fixed on reef. Thanks for filing that. >> >> Martin you may want to retry things after the next reef release. >> Unfortunately, I don't know when that is planned but I think it's soonish. >> >> > >> > Zitat von Martin Conway <martin.conway(a)anu.edu.au>: >> > > I just had another look through the issues tracker and found this >> > > bug already listed. >> > > https://tracker.ceph.com/issues/59428 >> > > >> > > I need to go back to the other issues I am having and figure out if >> > > they are related or something different. >> > > >> > > >> > > Hi >> > > >> > > I wrote before about issues I was having with cephadm in 18.2.0 >> > > Sorry, I didn't see the helpful replies because my mail service >> > > binned the responses. >> > > >> > > I still can't get the reef version of cephadm to work properly. >> > > >> > > I had updated the system rpm to reef (ceph repo) and also upgraded >> > > the containerised ceph daemons to reef before my first email. >> > > >> > > Both the system package cephadm and the one found at >> > > /var/lib/ceph/${fsid}/cephadm.* return the same error when running >> > > "cephadm version" >> > > >> > > Traceback (most recent call last): >> > > File >> > > >> > > >> "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4 >> > > e", line 9468, in <module> >> > > >> > > main() >> > > >> > > File >> > > >> > > >> "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4 >> > > e", line 9456, in main >> > > >> > > r = ctx.func(ctx) >> > > >> > > File >> > > >> > > >> "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4 >> > > e", line 2108, in _infer_image >> > > >> > > ctx.image = infer_local_ceph_image(ctx, ctx.container_engine.path) >> > > >> > > File >> > > >> > > >> "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4 >> > > e", line 2191, in infer_local_ceph_image >> > > >> > > container_info = get_container_info(ctx, daemon, daemon_name is not >> > > None) >> > > >> > > File >> > > >> > > >> "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4 >> > > e", line 2154, in get_container_info >> > > >> > > matching_daemons = [d for d in daemons if daemon_name_or_type(d) >> > > >> > > == daemon_filter and d['fsid'] == ctx.fsid] >> > > >> > > File >> > > >> > > >> "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4 >> > > e", line 2154, in <listcomp> >> > > >> > > matching_daemons = [d for d in daemons if daemon_name_or_type(d) >> > > >> > > == daemon_filter and d['fsid'] == ctx.fsid] >> > > >> > > File >> > > >> > > >> "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4 >> > > e", line 217, in __getattr__ >> > > >> > > return super().__getattribute__(name) >> > > >> > > AttributeError: 'CephadmContext' object has no attribute 'fsid' >> > > >> > > _______________________________________________ >> > > ceph-users mailing list -- ceph-users(a)ceph.io >> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> >> >> ------------------------------ >> >> Subject: Digest Footer >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s >> >> >> ------------------------------ >> >> End of ceph-users Digest, Vol 112, Issue 119 >> ******************************************** >> > > > -- > *Pardhiv Karri* > "Rise and Rise again until LAMBS become LIONS"

1 0

Problem with upgrade
by Jorge Garcia 28 Oct '23

28 Oct '23

I'm trying to upgrade our 3-monitor cluster from Centos 7 and Nautilus to Rocky 9 and Quincy. This has been a very slow process of upgrading one thing, running the cluster for a while, then upgrading the next thing. I first upgraded to the last Centos 7 and upgraded to Octopus. That worked fine. Then I was going to upgrade the OS to Rocky 9 while staying on Octopus, but then found out that Octopus is not available for Rocky 9. So I broke my own rule and upgraded one of the monitor (and manager) nodes to Rocky 9 and Pacific, then rejoined it to the cluster. That seemed to work just fine. Feeling bold, I upgraded the second monitor and manager node to Rocky 9 and Pacific. That also seemed to work fine, with the cluster showing all the monitors and managers running. But now, if I shut down the last "Octopus" monitor, the cluster becomes unresponsive. This only happens when I shut down the Octopus monitor. If I shut down one of the Pacific monitors, the cluster keeps responding with the expected: "HEALTH_WARN 1/3 mons down" and then goes back to normal when the monitor process is started again. Is this expected? What am I missing? Thanks for any pointers!

3 9

Join us for the User + Dev Meeting, happening tomorrow!
by Laura Flores 27 Oct '23

27 Oct '23

Hi Ceph users and developers, You are invited to join us at the User + Dev meeting tomorrow at 10:00 AM EST! See below for more meeting details. We have two guest speakers joining us tomorrow: 1. "CRUSH Changes at Scale" by Joshua Baergen, Digital Ocean In this talk, Joshua Baergen will discuss the problems that operators encounter with CRUSH changes at scale and how DigitalOcean built pg-remapper to control and speed up CRUSH-induced backfill. 2. "CephFS Management with Ceph Dashboard" by Pedro Gonzalez Gomez, IBM This talk will demonstrate new Dashboard behavior regarding CephFS management. The last part of the meeting will be dedicated to open discussion. Feel free to add questions for the speakers or additional topics under the "Open Discussion" section on the agenda: https://pad.ceph.com/p/ceph-user-dev-monthly-minutes If you have an idea for a focus topic you'd like to present at a future meeting, you are welcome to submit it to this Google Form: https://docs.google.com/forms/d/e/1FAIpQLSdboBhxVoBZoaHm8xSmeBoemuXoV_rmh4v… Any Ceph user or developer is eligible to submit! Thanks, Laura Flores Meeting link: https://meet.jit.si/ceph-user-dev-monthly Time conversions: UTC: Thursday, October 19, 14:00 UTC Mountain View, CA, US: Thursday, October 19, 7:00 PDT Phoenix, AZ, US: Thursday, October 19, 7:00 MST Denver, CO, US: Thursday, October 19, 8:00 MDT Huntsville, AL, US: Thursday, October 19, 9:00 CDT Raleigh, NC, US: Thursday, October 19, 10:00 EDT London, England: Thursday, October 19, 15:00 BST Paris, France: Thursday, October 19, 16:00 CEST Helsinki, Finland: Thursday, October 19, 17:00 EEST Tel Aviv, Israel: Thursday, October 19, 17:00 IDT Pune, India: Thursday, October 19, 19:30 IST Brisbane, Australia: Friday, October 20, 0:00 AEST Singapore, Asia: Thursday, October 19, 22:00 +08 Auckland, New Zealand: Friday, October 20, 3:00 NZDT -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 1

Solution for heartbeat and slow ops warning
by huongnv 27 Oct '23

27 Oct '23

Hi guys, Do you can find this solution for issue with long hearbeat and warning slow ops? Thank you so much.

1 0

Stickyness of writing vs full network storage writing
by Hans Kaiser 27 Oct '23

27 Oct '23

1 0

[Pacific] ceph orch device ls do not returns any HDD
by Patrick Begou 27 Oct '23

27 Oct '23

Hi everyone I'm new to CEPH, just a french 4 days training session with Octopus on VMs that convince me to build my first cluster. At this time I have 4 old identical nodes for testing with 3 HDDs each, 2 network interfaces and running Alma Linux8 (el8). I try to replay the training session but it fails, breaking the web interface because of some problems with podman 4.2 not compatible with Octopus. So I try to deploy Pacific with cephadm tool on my first node (mostha1) (to enable testing also an upgrade later). dnf -y install https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noar… monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }') cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \ --initial-dashboard-user admceph \ --allow-fqdn-hostname --cluster-network 10.1.0.0/16 This was sucessfull. But running "*c**eph orch device ls*" do not show any HDD even if I have /dev/sda (used by the OS), /dev/sdb and /dev/sdc The web interface shows a row capacity which is an aggregate of the sizes of the 3 HDDs for the node. I've also tried to reset /dev/sdb but cephadm do not see it: [ceph: root@mostha1 /]# ceph orch device zap mostha1.legi.grenoble-inp.fr /dev/sdb --force Error EINVAL: Device path '/dev/sdb' not found on host 'mostha1.legi.grenoble-inp.fr' On my first attempt with octopus, I was able to list the available HDD with this command line. Before moving to Pacific, the OS on this node has been reinstalled from scratch. Any advices for a CEPH beginner ? Thanks Patrick

9 51

Ceph - Error ERANGE: (34) Numerical result out of range
by Pardhiv Karri 27 Oct '23

27 Oct '23

Hi, Trying to move a node/host under a new SSD root and getting below error. Has anyone seen it and know the fix? the pg_num and pgp_num are same for all pools so that is not the issue. [root@hbmon1 ~]# ceph osd crush move hbssdhost1 root=ssd Error ERANGE: (34) Numerical result out of range [root@hbmon1 ~]# Thanks, Pardhiv

2 1

How to trigger scrubbing in Ceph on-demand ?
by Jayjeet Chakraborty 26 Oct '23

26 Oct '23

Hi all, I am trying to trigger deep scrubbing in Ceph reef (18.2.0) on demand on a set of files that I randomly write to CephFS. I have tried both invoking deep-scrub on CephFS using ceph tell and just deep scrubbing a particular PG. Unfortunately, none of that seems to be working for me. I am monitoring the ceph status output, it never shows any scrubbing information. Can anyone please help me out on this ? In a nutshell, I need Ceph to scrub for me anytime I want. I am using Ceph with default configs for scrubbing. Thanks all. Best Regards, *Jayjeet Chakraborty* Ph.D. Student Department of Computer Science and Engineering University of California, Santa Cruz *Email: jayjeetc(a)ucsc.edu <jayjeetc(a)ucsc.edu>*

3 4

quincy v17.2.7 QE Validation status
by Yuri Weinstein 26 Oct '23

26 Oct '23

Details of this release are summarized here: https://tracker.ceph.com/issues/63219#note-2 Release Notes - TBD Issue https://tracker.ceph.com/issues/63192 appears to be failing several runs. Should it be fixed for this release? Seeking approvals/reviews for: smoke - Laura rados - Laura, Radek, Travis, Ernesto, Adam King rgw - Casey fs - Venky orch - Adam King rbd - Ilya krbd - Ilya upgrade/quincy-p2p - Known issue IIRC, Casey pls confirm/approve client-upgrade-quincy-reef - Laura powercycle - Brad pls confirm ceph-volume - Guillaume pls take a look Please reply to this email with approval and/or trackers of known issues/PRs to address them. Josh, Neha - gibba and LRC upgrades -- N/A for quincy now after reef release. Thx YuriW

10 25

[quincy - 17.2.6] Lua scripting in the rados gateway - HTTP_REMOTE-ADDR missing
by stephan＠gridscale.io 26 Oct '23

26 Oct '23

Hi Ceph users, currently I'm using the lua script feature in radosgw to send "put_obj" and "get_obj" requests stats to a mongo db. So far it's working quite well but I miss a field which is very important for us for traffic stats. Im looking for the HTTP_REMOTE-ADDR field which is available in the ops_log but couldn't find it in here https://docs.ceph.com/en/quincy/radosgw/lua-scripting/#request-fields Does someone know how to get this field via lua script? Cheers Stephan

2 2

Enterprise SSD require for Ceph Reef Cluster
by Nafiz Imtiaz 26 Oct '23

26 Oct '23

Hi, We have a ceph cluster running reef version. We want to buy some enterprise ssd for our ceph cluster, and our prepared storage size is 1.92TB. For that, we have selected the Intel model. Please give a review about this model, and if you have any other model preference, please share with us. Thank you Brand: Intel SSD: 1.92TB 2.5'' Enterprise SATA, 6Gb/s Model: D3-S4510 Regards, Nafiz Imtiaz Assistant Manager, Product Development IT Division Bangladesh Export Import Company Ltd.

1 0

init unable to update_crush_location: (34) Numerical result out of range
by Pardhiv Karri 25 Oct '23

25 Oct '23

Hi, Getting an error while adding a new node/OSD with bluestore OSDs to the cluster. The OSD is added without any host and is down, tried to bring it up didn't work. The same method to add in other clusters doesn't have any issue. Any idea what the problem is? Ceph Version: ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable) Ceph Health: OK 2023-10-25 20:40:40.867878 7f1f478cde40 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1698266440867866, "job": 1, "event": "recovery_started", "log_files": [270]} 2023-10-25 20:40:40.867883 7f1f478cde40 4 rocksdb: [/build/ceph-U0cfoi/ceph-12.2.11/src/rocksdb/db/db_impl_open.cc:482] Recovering log #270 mode 0 2023-10-25 20:40:40.867904 7f1f478cde40 4 rocksdb: [/build/ceph-U0cfoi/ceph-12.2.11/src/rocksdb/db/version_set.cc:2395] Creating manifest 272 2023-10-25 20:40:40.869553 7f1f478cde40 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1698266440869548, "job": 1, "event": "recovery_finished"} 2023-10-25 20:40:40.870924 7f1f478cde40 4 rocksdb: [/build/ceph-U0cfoi/ceph-12.2.11/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x55c9061ba000 2023-10-25 20:40:40.870964 7f1f478cde40 1 bluestore(/var/lib/ceph/osd/ceph-721) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 2023-10-25 20:40:40.871234 7f1f478cde40 1 freelist init 2023-10-25 20:40:40.871293 7f1f478cde40 1 bluestore(/var/lib/ceph/osd/ceph-721) _open_alloc opening allocation metadata 2023-10-25 20:40:40.871314 7f1f478cde40 1 bluestore(/var/lib/ceph/osd/ceph-721) _open_alloc loaded 3.49TiB in 1 extents 2023-10-25 20:40:40.874700 7f1f478cde40 0 <cls> /build/ceph-U0cfoi/ceph-12.2.11/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs 2023-10-25 20:40:40.874721 7f1f478cde40 0 _get_class not permitted to load sdk 2023-10-25 20:40:40.874955 7f1f478cde40 0 _get_class not permitted to load kvs 2023-10-25 20:40:40.875638 7f1f478cde40 0 _get_class not permitted to load lua 2023-10-25 20:40:40.875724 7f1f478cde40 0 <cls> /build/ceph-U0cfoi/ceph-12.2.11/src/cls/hello/cls_hello.cc:296: loading cls_hello 2023-10-25 20:40:40.875776 7f1f478cde40 0 osd.721 0 crush map has features 288232575208783872, adjusting msgr requires for clients 2023-10-25 20:40:40.875780 7f1f478cde40 0 osd.721 0 crush map has features 288232575208783872 was 8705, adjusting msgr requires for mons 2023-10-25 20:40:40.875784 7f1f478cde40 0 osd.721 0 crush map has features 288232575208783872, adjusting msgr requires for osds 2023-10-25 20:40:40.875837 7f1f478cde40 0 osd.721 0 load_pgs 2023-10-25 20:40:40.875840 7f1f478cde40 0 osd.721 0 load_pgs opened 0 pgs 2023-10-25 20:40:40.875844 7f1f478cde40 0 osd.721 0 using weightedpriority op queue with priority op cut off at 64. 2023-10-25 20:40:40.877401 7f1f478cde40 -1 osd.721 0 log_to_monitors {default=true} 2023-10-25 20:40:40.888408 7f1f478cde40 -1 osd.721 0 mon_cmd_maybe_osd_create fail: '(34) Numerical result out of range': (34) Numerical result out of range 2023-10-25 20:40:40.891367 7f1f478cde40 -1 osd.721 0 mon_cmd_maybe_osd_create fail: '(34) Numerical result out of range': (34) Numerical result out of range 2023-10-25 20:40:40.891409 7f1f478cde40 -1 osd.721 0 init unable to update_crush_location: (34) Numerical result out of range Thanks, Pardhiv

1 0

cephadm failing to add hosts despite a working SSH connection
by Michel Jouvin 25 Oct '23

25 Oct '23

Hi, I'm struggling with a problem to add cephadm some hosts in our Quincy cluster. "ceph orch host add host addr" fails with the famous "missing 2 required positional arguments: 'hostname' and 'addr'" because of bug https://tracker.ceph.com/issues/59081 but looking at cephadm messages with "ceph -W cephadm", I can see: -------- Log: Opening SSH connection to 10.81.22.183, port 22 [conn=736] Connected to SSH server at 10.81.22.183, port 22 [conn=736] Local address: 10.81.22.151, port 53640 [conn=736] Peer address: 10.81.22.183, port 22 [conn=736] Login timeout expired [conn=736] Aborting connection Traceback (most recent call last): (removed) cephadm.ssh.HostConnectionError: Failed to connect to jc-rgw3 (10.81.22.183). Login timeout expired Log: Opening SSH connection to 10.81.22.183, port 22 [conn=736] Connected to SSH server at 10.81.22.183, port 22 [conn=736] Local address: 10.81.22.151, port 53640 [conn=736] Peer address: 10.81.22.183, port 22 [conn=736] Login timeout expired [conn=736] Aborting connection -------- It is very strange for me because " ssh -i /tmp/cephadm_identity_xxx 10.81.22.183" is working fine |when executed in the active mgr container. | |The host I'm trying to add is a RGW that has 3 active network connections: Ceph public network, our intranet network (used for managing the server) and the network of the application that will use the RGW. It seems to be somewhat related to this network configuration as main cluster servers (MONs, OSDs) which have only the the 2 Ceph networks and the intranet one don't suffer the same problem. In particular, what is strange is that I can successfully add the host if I use its intranet adress rather than the Ceph public network one (|||10.81.22.183) in the cephadm command. I have 3 hosts sharing the same network configuration and having the same problem. Any hint or suggestion to troubleshoot further this problem would be highly appreciated! Best regards, Michel

1 1

Re: radosgw - octopus - 500 Bad file descriptor on upload
by BEAUDICHON Hubert (Acoss) 25 Oct '23

25 Oct '23

Hi, We encountered the same kind of error for one of our users. CEPH Version : 16.2.10 2023-10-24T17:57:22.438+0200 7fc27ab44700 0 WARNING: set_req_state_err err_no=125 resorting to 500 2023-10-24T17:57:22.439+0200 7fc584957700 0 req 12200560481916573577 143.735748291s ERROR: RESTFUL_IO(s)->complete_header() returned err=Bad file descriptor 2023-10-24T17:57:22.439+0200 7fbecfaed700 1 ====== req done req=0x7fbdb86ab600 op status=-125 http_status=500 latency=143.735748291s ====== 2023-10-24T17:57:22.439+0200 7fbecfaed700 1 beast: 0x7fbdb86ab600: 10.227.131.117 - dev-centralog-save [24/Oct/2023:17:54:58.703 +0200] "PUT <obfuscate>&partNumber=1 HTTP/1.1" 500 58720313 - "<agent>" - latency=143.735748291s I haven't got any clue on the cause...

2 1

Ceph Leadership Team notes 10/25
by Dan van der Ster 25 Oct '23

25 Oct '23

Hi all, Here are this week's notes from the CLT: * Collective review of the Reef/Squid "State of Cephalopod" slides. * Smoke test suite was unscheduled but it's back on now. * Releases: * 17.2.7: about to start building last week, delayed by a few issues (https://tracker.ceph.com/issues/63257, https://tracker.ceph.com/issues/63305, https://github.com/ceph/ceph/pull/54169) ceph_exporter test coverage will be prioritized. * 18.2.1: all PRs in testing or merged. * Ceph Board approved a new Foundation member tiers model, Silver, Gold, Platinum, Diamond. Working on implementation with LF. -- dan

1 0

Ceph 16.2.14: pgmap updated every few seconds for no apparent reason
by Zakhar Kirpichenko 25 Oct '23

25 Oct '23

Hi, I am investigating excessive mon writes in our cluster and wondering whether excessive pgmap updates could be the culprit. Basically pgmap is updated every few seconds, sometimes over ten times per minute, in a healthy cluster with no OSD and/or PG changes: Oct 13 11:03:03 ceph03 bash[4019]: cluster 2023-10-13T11:03:01.515438+0000 mgr.ceph01.vankui (mgr.336635131) 838252 : cluster [DBG] pgmap v606575: 2400 pgs: 5 active+clean+scrubbing+deep, 2395 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 60 MiB/s rd, 109 MiB/s wr, 5.65k op/s Oct 13 11:03:04 ceph03 bash[4019]: cluster 2023-10-13T11:03:03.520953+0000 mgr.ceph01.vankui (mgr.336635131) 838253 : cluster [DBG] pgmap v606576: 2400 pgs: 5 active+clean+scrubbing+deep, 2395 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 64 MiB/s rd, 128 MiB/s wr, 5.76k op/s Oct 13 11:03:06 ceph03 bash[4019]: cluster 2023-10-13T11:03:05.524474+0000 mgr.ceph01.vankui (mgr.336635131) 838255 : cluster [DBG] pgmap v606577: 2400 pgs: 5 active+clean+scrubbing+deep, 2395 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 64 MiB/s rd, 122 MiB/s wr, 5.57k op/s Oct 13 11:03:08 ceph03 bash[4019]: cluster 2023-10-13T11:03:07.530484+0000 mgr.ceph01.vankui (mgr.336635131) 838256 : cluster [DBG] pgmap v606578: 2400 pgs: 5 active+clean+scrubbing+deep, 2395 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 79 MiB/s rd, 127 MiB/s wr, 6.62k op/s Oct 13 11:03:10 ceph03 bash[4019]: cluster 2023-10-13T11:03:09.533337+0000 mgr.ceph01.vankui (mgr.336635131) 838258 : cluster [DBG] pgmap v606579: 2400 pgs: 5 active+clean+scrubbing+deep, 2395 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 66 MiB/s rd, 104 MiB/s wr, 5.38k op/s Oct 13 11:03:12 ceph03 bash[4019]: cluster 2023-10-13T11:03:11.537908+0000 mgr.ceph01.vankui (mgr.336635131) 838259 : cluster [DBG] pgmap v606580: 2400 pgs: 5 active+clean+scrubbing+deep, 2395 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 85 MiB/s rd, 121 MiB/s wr, 6.43k op/s Oct 13 11:03:13 ceph03 bash[4019]: cluster 2023-10-13T11:03:13.543490+0000 mgr.ceph01.vankui (mgr.336635131) 838260 : cluster [DBG] pgmap v606581: 2400 pgs: 5 active+clean+scrubbing+deep, 2395 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 78 MiB/s rd, 127 MiB/s wr, 6.54k op/s Oct 13 11:03:16 ceph03 bash[4019]: cluster 2023-10-13T11:03:15.547122+0000 mgr.ceph01.vankui (mgr.336635131) 838262 : cluster [DBG] pgmap v606582: 2400 pgs: 5 active+clean+scrubbing+deep, 2395 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 71 MiB/s rd, 122 MiB/s wr, 6.08k op/s Oct 13 11:03:18 ceph03 bash[4019]: cluster 2023-10-13T11:03:17.553180+0000 mgr.ceph01.vankui (mgr.336635131) 838263 : cluster [DBG] pgmap v606583: 2400 pgs: 1 active+clean+scrubbing, 5 active+clean+scrubbing+deep, 2394 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 75 MiB/s rd, 176 MiB/s wr, 6.83k op/s Oct 13 11:03:20 ceph03 bash[4019]: cluster 2023-10-13T11:03:19.555960+0000 mgr.ceph01.vankui (mgr.336635131) 838264 : cluster [DBG] pgmap v606584: 2400 pgs: 1 active+clean+scrubbing, 5 active+clean+scrubbing+deep, 2394 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 58 MiB/s rd, 161 MiB/s wr, 5.55k op/s Oct 13 11:03:22 ceph03 bash[4019]: cluster 2023-10-13T11:03:21.560597+0000 mgr.ceph01.vankui (mgr.336635131) 838266 : cluster [DBG] pgmap v606585: 2400 pgs: 1 active+clean+scrubbing, 5 active+clean+scrubbing+deep, 2394 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 62 MiB/s rd, 221 MiB/s wr, 6.19k op/s Oct 13 11:03:24 ceph03 bash[4019]: cluster 2023-10-13T11:03:23.565974+0000 mgr.ceph01.vankui (mgr.336635131) 838267 : cluster [DBG] pgmap v606586: 2400 pgs: 1 active+clean+scrubbing, 5 active+clean+scrubbing+deep, 2394 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 50 MiB/s rd, 246 MiB/s wr, 5.93k op/s Oct 13 11:03:26 ceph03 bash[4019]: cluster 2023-10-13T11:03:25.569471+0000 mgr.ceph01.vankui (mgr.336635131) 838269 : cluster [DBG] pgmap v606587: 2400 pgs: 1 active+clean+scrubbing, 5 active+clean+scrubbing+deep, 2394 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 41 MiB/s rd, 240 MiB/s wr, 4.99k op/s Oct 13 11:03:28 ceph03 bash[4019]: cluster 2023-10-13T11:03:27.575618+0000 mgr.ceph01.vankui (mgr.336635131) 838270 : cluster [DBG] pgmap v606588: 2400 pgs: 4 active+clean+scrubbing+deep, 2396 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 44 MiB/s rd, 259 MiB/s wr, 5.38k op/s Oct 13 11:03:30 ceph03 bash[4019]: cluster 2023-10-13T11:03:29.578262+0000 mgr.ceph01.vankui (mgr.336635131) 838271 : cluster [DBG] pgmap v606589: 2400 pgs: 4 active+clean+scrubbing+deep, 2396 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 31 MiB/s rd, 195 MiB/s wr, 4.06k op/s Oct 13 11:03:32 ceph03 bash[4019]: cluster 2023-10-13T11:03:31.582849+0000 mgr.ceph01.vankui (mgr.336635131) 838272 : cluster [DBG] pgmap v606590: 2400 pgs: 4 active+clean+scrubbing+deep, 2396 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 35 MiB/s rd, 197 MiB/s wr, 4.43k op/s Oct 13 11:03:34 ceph03 bash[4019]: cluster 2023-10-13T11:03:33.588249+0000 mgr.ceph01.vankui (mgr.336635131) 838274 : cluster [DBG] pgmap v606591: 2400 pgs: 4 active+clean+scrubbing+deep, 2396 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 39 MiB/s rd, 138 MiB/s wr, 4.50k op/s Oct 13 11:03:36 ceph03 bash[4019]: cluster 2023-10-13T11:03:35.591837+0000 mgr.ceph01.vankui (mgr.336635131) 838276 : cluster [DBG] pgmap v606592: 2400 pgs: 4 active+clean+scrubbing+deep, 2396 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 35 MiB/s rd, 92 MiB/s wr, 3.90k op/s Oct 13 11:03:37 ceph03 bash[4019]: cluster 2023-10-13T11:03:37.597899+0000 mgr.ceph01.vankui (mgr.336635131) 838277 : cluster [DBG] pgmap v606593: 2400 pgs: 4 active+clean+scrubbing+deep, 2396 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 42 MiB/s rd, 77 MiB/s wr, 4.47k op/s Oct 13 11:03:39 ceph03 bash[4019]: cluster 2023-10-13T11:03:39.600591+0000 mgr.ceph01.vankui (mgr.336635131) 838278 : cluster [DBG] pgmap v606594: 2400 pgs: 4 active+clean+scrubbing+deep, 2396 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 34 MiB/s rd, 39 MiB/s wr, 3.61k op/s Oct 13 11:03:41 ceph03 bash[4019]: cluster 2023-10-13T11:03:41.605546+0000 mgr.ceph01.vankui (mgr.336635131) 838279 : cluster [DBG] pgmap v606595: 2400 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 2395 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 41 MiB/s rd, 42 MiB/s wr, 4.21k op/s Oct 13 11:03:43 ceph03 bash[4019]: cluster 2023-10-13T11:03:43.610694+0000 mgr.ceph01.vankui (mgr.336635131) 838281 : cluster [DBG] pgmap v606596: 2400 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 2395 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 49 MiB/s rd, 49 MiB/s wr, 4.59k op/s Oct 13 11:03:45 ceph03 bash[4019]: cluster 2023-10-13T11:03:45.614721+0000 mgr.ceph01.vankui (mgr.336635131) 838283 : cluster [DBG] pgmap v606597: 2400 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 2395 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 44 MiB/s rd, 49 MiB/s wr, 4.15k op/s Within 14 hours of today pgmap was updated 19960 times for no apparent reason. The cluster is healthy and nothing is going on apart from normal I/O. Are frequent pgmap updates expected behavior? What may be causing these updates? I would very much appreciate your comments and suggestions. Best regards, Zakhar

3 5

Combining masks in ceph config
by Frank Schilder 25 Oct '23

25 Oct '23

Hi all, I have a case where I want to set options for a set of HDDs under a common sub-tree with root A. I have also HDDs in another disjoint sub-tree with root B. Therefore, I would like to do something like ceph config set osd/class:hdd,datacenter:A option value The above does not give a syntax error, but I'm also not sure it does the right thing. Does the above mean "class:hdd and datacenter:A" or does it mean "for OSDs with device class 'hdd,datacenter:A'"? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

Quincy: failure to enable mgr rgw module if not --force
by Michel Jouvin 25 Oct '23

25 Oct '23

Hi, I'm trying to use the rgw mgr module to configure RGWs. Unfortunately it is not present in 'ceph mgr module ls' list and any attempt to enable it suggests that one mgr doesn't support it and that --force should be added. Adding --force effectively enabled it. It is strange as it is a brand new cluster, created in Quincy, using cephadm. Why this need for --force? And it seems that even if the module is listed as enabled, the 'ceph rgw' command is not recognized and the help is not available for the rgw subcommand? What are we doing wrong? Cheers, Michel

3 3

Dashboard crash with rook/reef and external prometheus
by r-ceph＠rich0.org 24 Oct '23

24 Oct '23

I'm fairly new to the community so I figured I'd ask about this here before creating an issue - I'm not sure how supported this config is. I am running rook v1.12.6 and ceph 18.2.0. I've enabled the dashboard in the CRD and it has been working for a while. However, the charts are empty. I do have Prometheus+Grafana running on my cluster, and I can access many of the ceph metrics from there. With the upgrade to reef I noticed that many of the qunicy dashboard elements have been replaced by charts, so I wanted to get those working. I discovered that if I run ceph dashboard set-prometheus-api-host <url> the charts immediately are populated (including historical data). However, when I do this I rapidly start getting ceph health alerts due to a crashing mgr module. If I set the prometheus api host url to '' the crashes stop accumulating, though this disables the charts. I am running the prometheus-community/prometheus-25.2.0 chart. Various ceph grafana dashboards that I've found published work fine. The following are relevant dumps. Please let me know if you have any ideas, or if I should go ahead and create an issue for this... mgr console output during crash: debug 2023-10-24T15:11:23.498+0000 7fc81fa5d700 -1 mgr.server reply reply (2) No such file or directory This Orchestrator does not support `orch prometheus access info` debug 2023-10-24T15:11:23.502+0000 7fc7ea3f3700 0 [dashboard INFO request] [::ffff:10.1.0.106:49760] [GET] [200] [0.012s] [admin] [101.0B] /api/health/get_cluster_capacity debug 2023-10-24T15:11:23.502+0000 7fc813985700 0 [stats WARNING root] cmdtag not found in client metadata debug 2023-10-24T15:11:23.502+0000 7fc813985700 0 [stats WARNING root] cmdtag not found in client metadata debug 2023-10-24T15:11:23.502+0000 7fc7e83ef700 0 [dashboard INFO request] [::ffff:10.1.0.106:5580] [GET] [200] [0.011s] [admin] [73.0B] /api/osd/settings debug 2023-10-24T15:11:23.506+0000 7fc85411a700 0 log_channel(audit) log [DBG] : from='mon.2 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": "json"}]: dispatch debug 2023-10-24T15:11:23.506+0000 7fc813985700 0 [stats WARNING root] cmdtag not found in client metadata debug 2023-10-24T15:11:23.506+0000 7fc7e9bf2700 0 [dashboard INFO request] [::ffff:10.1.0.106:20241] [GET] [200] [0.014s] [admin] [34.0B] /api/prometheus/rules debug 2023-10-24T15:11:23.630+0000 7fc7ebbf6700 0 [dashboard INFO orchestrator] is orchestrator available: True, debug 2023-10-24T15:11:23.734+0000 7fc7ebbf6700 0 [dashboard INFO orchestrator] is orchestrator available: True, debug 2023-10-24T15:11:23.802+0000 7fc86511c700 0 log_channel(cluster) log [DBG] : pgmap v126: 617 pgs: 53 active+remapped+backfill_wait, 2 active+remapped+backfilling, 562 active+clean; 34 TiB data, 68 TiB used, 64 TiB / 132 TiB avail; 2.4 MiB/s rd, 93 KiB/s wr, 21 op/s; 1213586/22505781 objects misplaced (5.392%) debug 2023-10-24T15:11:23.862+0000 7fc7ebbf6700 0 [dashboard INFO orchestrator] is orchestrator available: True, debug 2023-10-24T15:11:23.962+0000 7fc7ebbf6700 0 [dashboard INFO orchestrator] is orchestrator available: True, debug 2023-10-24T15:11:24.058+0000 7fc7ebbf6700 0 [dashboard INFO orchestrator] is orchestrator available: True, debug 2023-10-24T15:11:24.158+0000 7fc7ebbf6700 0 [dashboard INFO orchestrator] is orchestrator available: True, debug 2023-10-24T15:11:24.270+0000 7fc7ebbf6700 0 [dashboard INFO orchestrator] is orchestrator available: True, debug 2023-10-24T15:11:24.546+0000 7fc7ebbf6700 0 [dashboard INFO orchestrator] is orchestrator available: True, debug 2023-10-24T15:11:24.654+0000 7fc7ebbf6700 0 [dashboard INFO orchestrator] is orchestrator available: True, debug 2023-10-24T15:11:24.654+0000 7fc7ebbf6700 0 [dashboard INFO request] [::ffff:10.1.0.106:13711] [GET] [200] [1.170s] [admin] [3.2K] /api/health/minimal debug 2023-10-24T15:11:25.802+0000 7fc86511c700 0 log_channel(cluster) log [DBG] : pgmap v127: 617 pgs: 53 active+remapped+backfill_wait, 2 active+remapped+backfilling, 562 active+clean; 34 TiB data, 68 TiB used, 64 TiB / 132 TiB avail; 1.1 MiB/s rd, 53 KiB/s wr, 17 op/s; 1213586/22505781 objects misplaced (5.392%) debug 2023-10-24T15:11:27.802+0000 7fc86511c700 0 log_channel(cluster) log [DBG] : pgmap v128: 617 pgs: 53 active+remapped+backfill_wait, 2 active+remapped+backfilling, 562 active+clean; 34 TiB data, 68 TiB used, 64 TiB / 132 TiB avail; 1.8 MiB/s rd, 58 KiB/s wr, 18 op/s; 1213586/22505781 objects misplaced (5.392%) debug 2023-10-24T15:11:28.494+0000 7fc813985700 0 [stats WARNING root] cmdtag not found in client metadata debug 2023-10-24T15:11:28.498+0000 7fc7eb3f5700 0 [dashboard INFO request] [::ffff:10.1.0.106:20241] [GET] [200] [0.011s] [admin] [73.0B] /api/osd/settings debug 2023-10-24T15:11:28.498+0000 7fc85411a700 0 log_channel(audit) log [DBG] : from='mon.2 -' entity='mon.' cmd=[{"prefix": "orch prometheus access info"}]: dispatch debug 2023-10-24T15:11:28.502+0000 7fc7ec3f7700 0 [dashboard INFO request] [::ffff:10.1.0.106:5580] [GET] [200] [0.006s] [admin] [102.0B] /api/health/get_cluster_capacity debug 2023-10-24T15:11:28.502+0000 7fc7e93f1700 0 [dashboard INFO request] [::ffff:10.1.0.106:44009] [GET] [200] [0.005s] [admin] [22.0B] /api/prometheus/notifications debug 2023-10-24T15:11:28.502+0000 7fc81fa5d700 -1 Remote method threw exception: Traceback (most recent call last): File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 675, in get_prometheus_access_info raise NotImplementedError() NotImplementedError ceph crash info dump: { "backtrace": [ " File \"/usr/share/ceph/mgr/orchestrator/_interface.py\", line 675, in get_prometheus_access_info\n raise NotImplementedError()", "NotImplementedError" ], "ceph_version": "18.2.0", "crash_id": "2023-10-24T14:59:52.921427Z_08e06575-0431-47fe-afc5-be8e4a7d1144", "entity_name": "mgr.a", "mgr_module": "rook", "mgr_module_caller": "ActivePyModule::dispatch_remote get_prometheus_access_info", "mgr_python_exception": "NotImplementedError", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "8", "os_version_id": "8", "process_name": "ceph-mgr", "stack_sig": "bbf52dcdbbe54d67edf59ebdb5d201fffd921db5a9dd4431964c2aaac2250c7e", "timestamp": "2023-10-24T14:59:52.921427Z", "utsname_hostname": "k8s4", "utsname_machine": "x86_64", "utsname_release": "5.15.0-87-generic", "utsname_sysname": "Linux", "utsname_version": "#97-Ubuntu SMP Mon Oct 2 21:09:21 UTC 2023" } -- Rich

1 0

traffic by IP address / bucket / user
by Boris Behrens 24 Oct '23

24 Oct '23

Hi, did someone have a solution ready to monitor traffic by IP address? Cheers Boris

2 1

Modify user op status=-125
by mahnoosh shahidi 24 Oct '23

24 Oct '23

Hi all, I couldn't understand what does the status -125 mean from the docs. I'm getting 500 response status code when I call rgw admin APIs and the only log in the rgw log files is as follows. s3:get_obj recalculating target initializing for trans_id = tx00000aa90f570fb8281cf-006537bf9e-84395fa-default s3:get_obj reading permissions getting op 1 s3:put_obj verifying requester s3:put_obj normalizing buckets and tenants s3:put_obj init permissions s3:put_obj recalculating target s3:put_obj reading permissions s3:put_obj init op s3:put_obj verifying op mask s3:put_obj verifying op permissions s3:put_obj verifying op params s3:put_obj pre-executing s3:put_obj executing :modify_user completing WARNING: set_req_state_err err_no=125 resorting to 500 :modify_user op status=-125 :modify_user http status=500 ====== req done req=0x7f3f85a78620 op status=-125 http_status=500 latency=0.076000459s ====== Can anyone explain what this error means and why it's happening? Best Regards, Mahnoosh

2 4

RadosGW load balancing with Kubernetes + ceph orch
by Janek Bevendorff 23 Oct '23

23 Oct '23

Hey all, My Ceph cluster is managed mostly by cephadm / ceph orch to avoid circular dependencies between in our infrastructure deployment. Our RadosGW endpoints, however, are managed by Kubernetes, since it provides proper load balancing and service health checks. This leaves me in the unsatisfactory situation that Ceph complains about 'stray' RGW daemons in the cluster. The only two solutions to this that I found were a) to turn of the warning, which applies to all daemons and not just the RGWs (not pretty!), or b) to move the deployment out of Kubernetes. For the latter, I could define external Endpoints in Kubernetes, so that I still have load balancing, but then I don't have proper health checks any more. Meaning, if one of the RGW endpoints goes down, requests to our S3 endpoint will intermittently time out in round-robin fashion (not pretty at all!). Can you think of a better option to solve this? I would already be satisfied with turning off the warning for RGW daemons only, but there doesn't seem to be a config option for that. Thanks Janek

1 0

Ceph orch OSD redeployment after boot on stateless RAM root
by Janek Bevendorff 23 Oct '23

23 Oct '23

Hi, I recently moved from a manual Ceph deployment using Saltstack to a hybrid of Saltstack and cephadm / ceph orch. We are provisioning our Ceph hosts using a stateless PXE RAM root, so I definitely need Saltstack to bootstrap at least the Ceph APT repository and the MON/MGR deployment. After that, ceph orch can take over and deploy the remaining daemons. The MONs/MGRs are deployed after each reboot with cephadm deploy --name mon.{{ ceph.node_id }} --fsid {{ ceph.conf.global.fsid }} --config /etc/ceph/ceph.conf cephadm deploy --name mgr.{{ ceph.node_id }} --fsid {{ ceph.conf.global.fsid }} --config /etc/ceph/ceph.conf (the MON store is provided in /var/lib/ceph/{{ ceph.conf.global.fsid }}/mon.{{ ceph.node_id }}). Since cephadm ceph-volume lvm activate --all is broken (see https://tracker.ceph.com/issues/55395) I am activating each OSD individually like this: cephadm deploy --name osd.{{ osd_id }} --fsid {{ ceph.conf.global.fsid }} --osd-fsid {{ osd_fsid }} --config /etc/ceph/ceph.conf Now my question: Is there a better way to do this and can ceph orch take care of this in the same way it deploys my MDS? All OSDs are listed as <unmanaged> in ceph orch ls (I think this is by design?) and I cannot find a way to activate them automatically via ceph orch when the host boots up. I tried ceph cephadm osd activate HOSTNAME, but all I get is "Created no osd(s) on host HOSTNAME; already created?" The docs only talk about how I can create new OSDs, but not how I can automatically redeploy existing OSDs after a fresh boot. It seems like it is generally assumed that OSD deployments are persistent and next time the host boots, systemd simply activates the existing units. I'd be glad about any hints! Janek

1 0

ATTN: DOCS rgw bucket pubsub notification.
by Artem Torubarov 22 Oct '23

22 Oct '23

Hi, The latest documentation <https://docs.ceph.com/en/latest/radosgw/s3-notification-compatibility/#supp…> says that: "pulling and acking of events stored in Ceph (as an internal destination)." is a valid source for s3 notifications. If i got it right, pull-based subscription was possible only with radosgw PubSub sync module, which was removed in reef release <https://docs.ceph.com/en/latest/releases/reef/#:~:text=The%20pubsub%20funct…> . Suggested changes: Currently, we support: HTTP/S, Kafka and AMQP. And also support pulling and acking of events stored in Ceph (as an internal destination). Best regards, Artem

3 2

Ceph 16.2.14: OSDs randomly crash in bstore_kv_sync
by Zakhar Kirpichenko 20 Oct '23

20 Oct '23

Hi, After upgrading to Ceph 16.2.14 we had several OSD crashes in bstore_kv_sync thread: 1. "assert_thread_name": "bstore_kv_sync", 2. "backtrace": [ 3. "/lib64/libpthread.so.0(+0x12cf0) [0x7ff2f6750cf0]", 4. "gsignal()", 5. "abort()", 6. "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x564dc5f87d0b]", 7. "/usr/bin/ceph-osd(+0x584ed4) [0x564dc5f87ed4]", 8. "(RocksDBBlueFSVolumeSelector::sub_usage(void*, bluefs_fnode_t const&)+0x15e) [0x564dc6604a9e]", 9. "(BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x77d) [0x564dc66951cd]", 10. "(BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0x90) [0x564dc6695670]", 11. "(BlueFS::fsync(BlueFS::FileWriter*)+0x18b) [0x564dc66b1a6b]", 12. "(BlueRocksWritableFile::Sync()+0x18) [0x564dc66c1768]", 13. "(rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x564dc6b6496f]", 14. "(rocksdb::WritableFileWriter::SyncInternal(bool)+0x402) [0x564dc6c761c2]", 15. "(rocksdb::WritableFileWriter::Sync(bool)+0x88) [0x564dc6c77808]", 16. "(rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0x309) [0x564dc6b780c9]", 17. "(rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x2629) [0x564dc6b80c69]", 18. "(rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x21) [0x564dc6b80e61]", 19. "(RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x84) [0x564dc6b1f644]", 20. "(RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x9a) [0x564dc6b2004a]", 21. "(BlueStore::_kv_sync_thread()+0x30d8) [0x564dc6602ec8]", 22. "(BlueStore::KVSyncThread::entry()+0x11) [0x564dc662ab61]", 23. "/lib64/libpthread.so.0(+0x81ca) [0x7ff2f67461ca]", 24. "clone()" 25. ], I am attaching two instances of crash info for further reference: https://pastebin.com/E6myaHNU OSD configuration is rather simple and close to default: osd.6 dev bluestore_cache_size_hdd 4294967296 osd.6 dev bluestore_cache_size_ssd 4294967296 osd advanced debug_rocksdb 1/5 osd advanced osd_max_backfills 2 osd basic osd_memory_target 17179869184 osd advanced osd_recovery_max_active 2 osd advanced osd_scrub_sleep 0.100000 osd advanced rbd_balance_parent_reads false debug_rocksdb is a recent change, otherwise this configuration has been running without issues for months. The crashes happened on two different hosts with identical hardware, the hosts and storage (NVME DB/WAL, HDD block) don't exhibit any issues. We have not experienced such crashes with Ceph < 16.2.14. Is this a known issue, or should I open a bug report? Best regards, Zakhar

3 13

fixing future rctime
by MARTEL Arnaud 20 Oct '23

20 Oct '23

Hi all, I have some troubles with my backup script because there are few files, in a deep sub-directory, with a creation/modification date in the future (for example: 2040-02-06 18:00:00). As my script uses the ceph.dir.rctime extended attribute to identify the files and directories to backup, it now browses and sync a lot of unchanged sub-directories… I tried a lot of things, including remove and recreate the files so they have now the current datetime, but rctime is never updated. Even when I remove the last directory (ie: the one where the files are located), rctime is not updated for the parent directories. Has someone a trick to reset rctime to current datetime (or any other solution to remove this inconsistent value of rctime) with quincy (17.2.6) ? Regards, Arnaud

2 2

Specify priority for active MGR and MDS
by Nicolas FONTAINE 19 Oct '23

19 Oct '23

Hi everyone, Is there a way to specify which MGR and which MDS should be the active one? Thanks, Nicolas.

3 2

Turn off Dashboard CephNodeDiskspaceWarning for specific drives?
by Daniel Brown 19 Oct '23

19 Oct '23

Greetings - Forgive me if this is an elementary question - am fairly new to running CEPH. Have searched but didn’t see anything specific that came up. Is there any way to disable the disk space warnings (CephNodeDiskspaceWarning) for specific drives or filesystems on my CEPH servers? Running 18.2.0, installed with cephadm on Ubuntu 22.04 on Arm. Keep seeing these warnings in the Dashboard for /boot/firmware, which, in my opinion shouldn’t really be something that ceph needs to worry about - or at least, should be something I can configure it to ignore. Thanks in advance. Dan.

2 1

How do you handle large Ceph object storage cluster?
by pawel.przestrzelski＠gmail.com 19 Oct '23

19 Oct '23

Hi Everyone, My company is dealing with quite large Ceph cluster (>10k OSDs, >60 PB of data). It is entirely dedicated to object storage with S3 interface. Maintenance and its extension are getting more and more problematic and time consuming. We consider to split it to two or more completely separate clusters (without replication of data among them) and create S3 layer of abstraction with some additional metadata that will allow us to use these 2+ physically independent instances as a one logical cluster. Additionally, newest data is the most demanded data, so we have to spread it equally among clusters to avoid skews in cluster load. Do you have any similar experience? How did you handle it? Maybe you have some advice? I'm not a Ceph expert. I'm just a Ceph's user and software developer who does not like to duplicate someone's job. Best, Paweł

3 2

Remove empty orphaned PGs not mapped to a pool
by Malte Stroem 19 Oct '23

19 Oct '23

Hello, we removed an SSD cache tier and its pool. The PGs for the pool do still exist. The cluster is healthy. The PGs are empty and they reside on the cache tier pool's SSDs. We like to take out the disks but it is not possible. The cluster sees the PGs and answers with a HEALTH_WARN. Because of the replication of three there are still 128 PGs on three of the 24 OSDs. We were able to remove the other OSDs. Summary: - pool removed - 3 x 128 empty PGs still exist - 3 of 24 OSDs still exist How is it possible to remove these empty and healthy PGs? The only way I found was something like: ceph pg {pg-id} mark_unfound_lost delete Is that the right way? Some output of: ceph pg ls-by-osd 23 PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP 3.0 0 0 0 0 0 0 0 0 active+clean 27h 0'0 2627265:196316 [15,6,23]p15 [15,6,23]p15 2023-09-28T12:41:52.982955+0200 2023-09-27T06:48:23.265838+0200 3.1 0 0 0 0 0 0 0 0 active+clean 9h 0'0 2627266:19330 [6,23,15]p6 [6,23,15]p6 2023-09-29T06:30:57.630016+0200 2023-09-27T22:58:21.992451+0200 3.2 0 0 0 0 0 0 0 0 active+clean 2h 0'0 2627265:1135185 [23,15,6]p23 [23,15,6]p23 2023-09-29T13:42:07.346658+0200 2023-09-24T14:31:52.844427+0200 3.3 0 0 0 0 0 0 0 0 active+clean 13h 0'0 2627266:193170 [6,15,23]p6 [6,15,23]p6 2023-09-29T01:56:54.517337+0200 2023-09-27T17:47:24.961279+0200 3.4 0 0 0 0 0 0 0 0 active+clean 14h 0'0 2627265:2343551 [23,6,15]p23 [23,6,15]p23 2023-09-29T00:47:47.548860+0200 2023-09-25T09:39:51.259304+0200 3.5 0 0 0 0 0 0 0 0 active+clean 2h 0'0 2627265:194111 [15,6,23]p15 [15,6,23]p15 2023-09-29T13:28:48.879959+0200 2023-09-26T15:35:44.217302+0200 3.6 0 0 0 0 0 0 0 0 active+clean 6h 0'0 2627265:2345717 [23,15,6]p23 [23,15,6]p23 2023-09-29T09:26:02.534825+0200 2023-09-27T21:56:57.500126+0200 Best regards, Malte

4 12

stuck MDS warning: Client HOST failing to respond to cache pressure
by Frank Schilder 19 Oct '23

19 Oct '23

Hi all, I'm affected by a stuck MDS warning for 2 clients: "failing to respond to cache pressure". This is a false alarm as no MDS is under any cache pressure. The warning is stuck already for a couple of days. I found some old threads about cases where the MDS does not update flags/triggers for this warning in certain situations. Dating back to luminous and I'm probably hitting one of these. In these threads I could find a lot except for instructions for how to clear this out in a nice way. Is there something I can do on the clients to clear this warning? I don't want to evict/reboot just because of that. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 7

How to deal with increasing HDD sizes ? 1 OSD for 2 LVM-packed HDDs ?
by Renaud Jean Christophe Miel 19 Oct '23

19 Oct '23

Hi, Use case: * Ceph cluster with old nodes having 6TB HDDs * Add new node with new 12TB HDDs Is it supported/recommended to pack 2 6TB HDDs handled by 2 old OSDs into 1 12TB LVM disk handled by 1 new OSD ? Regards, Renaud Miel

5 5

Nautilus - Octopus upgrade - more questions
by Dave Hall 18 Oct '23

18 Oct '23

Hello, I have a Nautilus cluster built using Ceph packages from Debian 10 Backports, deployed with Ceph-Ansible. I see that Debian does not offer Ceph 15/Octopus packages. However, download.ceph.com does offer such packages. Question: Is it a safe upgrade to install the download.ceph.com packages over top of the buster-backports packages? If so, the next question is how to deploy this? Should I pull down an appropriate version of Ceph-Ansible and use the rolling-upgrade playbook? Or just apg-get -f dist-upgrade the new Ceph packages into place? BTW, in the long run I'll probably want to get to container-based Reef, but I need to keep a stable cluster throughout. Any advice or reassurance much appreciated. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

4 3

Time Estimation for cephfs-data-scan scan_links
by Odair M. 18 Oct '23

18 Oct '23

Hello, I've encountered an issue where the metadata pool has corrupted a cache inode, leading to an MDS rank abort in the 'reconnect' state. To address this, I'm following the "USING AN ALTERNATE METADATA POOL FOR RECOVERY" section from the documentation [1]. However, I've observed that the cephfs-data-scan scan_links step has been running for over 24 hours on 35 TB of data, which is replicated across 3 OSDs, resulting in more than 100 TB of raw data. Does anyone have an estimation on the duration for this step? Additional detail: The corrupted mds log: -9> 2023-10-11T10:13:22.254-0300 7ff901f75700 10 monclient: get_auth_request con 0x559bf41e4400 auth_method 0 -8> 2023-10-11T10:13:22.254-0300 7ff8ff770700 5 mds.barril12 handle_mds_map old map epoch 472481 <= 472481, discarding -7> 2023-10-11T10:13:22.254-0300 7ff8ff770700 0 mds.0.cache missing dir for * (which maps to *) on [inode 0x10021afaf90 [...392,head] /dbteamvenv/ auth v98534854 snaprealm=0x559bf427ce00 f(v60 m2023-10-06T15:35:03.278089-0300 9=0+9) n(v141971 rc2023-10-09T18:41:19.742089-0300 b1424948533453 139810=131460+8350) (iversion lock) 0x559bf4298580] -6> 2023-10-11T10:13:22.254-0300 7ff8ff770700 0 mds.0.cache missing dir ino 0x20005dd786b -5> 2023-10-11T10:13:22.254-0300 7ff902776700 10 monclient: get_auth_request con 0x559bf4142c00 auth_method 0 -4> 2023-10-11T10:13:22.258-0300 7ff902f77700 5 mds.beacon.barril12 received beacon reply up:rejoin seq 4 rtt 1.09601 -3> 2023-10-11T10:13:22.258-0300 7ff8ff770700 -1 ./src/mds/MDCache.cc: In function 'void MDCache::handle_cache_rejoin_weak(ceph::cref_t<MMDSCacheRejoin>&)' thread 7ff8ff770700 time 2023-10-11T10:13:22.259535-0300 ./src/mds/MDCache.cc: 4462: FAILED ceph_assert(diri) ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x124) [0x7ff904a5b282] 2: /usr/lib/ceph/libceph-common.so.2(+0x25b420) [0x7ff904a5b420] 3: (MDCache::handle_cache_rejoin_weak(boost::intrusive_ptr<MMDSCacheRejoin const> const&)+0x20de) [0x559bf0a9da6e] 4: (MDCache::dispatch(boost::intrusive_ptr<Message const> const&)+0x424) [0x559bf0aa2a64] 5: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x5c0) [0x559bf0930580] 6: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x58) [0x559bf0930b78] 7: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1bf) [0x559bf090b5df] 8: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x468) [0x7ff904ca71d8] 9: (DispatchQueue::entry()+0x5ef) [0x7ff904ca48df] 10: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff904d681cd] 11: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7ff905680ea7] 12: clone() -2> 2023-10-11T10:13:22.258-0300 7ff902f77700 10 monclient: get_auth_request con 0x559bf41e4c00 auth_method 0 -1> 2023-10-11T10:13:22.258-0300 7ff902f77700 10 monclient: get_auth_request con 0x559bf41e5400 auth_method 0 0> 2023-10-11T10:13:22.262-0300 7ff8ff770700 -1 *** Caught signal (Aborted) ** in thread 7ff8ff770700 thread_name:ms_dispatch ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140) [0x7ff90568c140] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17e) [0x7ff904a5b2dc] 5: /usr/lib/ceph/libceph-common.so.2(+0x25b420) [0x7ff904a5b420] 6: (MDCache::handle_cache_rejoin_weak(boost::intrusive_ptr<MMDSCacheRejoin const> const&)+0x20de) [0x559bf0a9da6e] 7: (MDCache::dispatch(boost::intrusive_ptr<Message const> const&)+0x424) [0x559bf0aa2a64] 8: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x5c0) [0x559bf0930580] 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x58) [0x559bf0930b78] 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1bf) [0x559bf090b5df] 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x468) [0x7ff904ca71d8] 12: (DispatchQueue::entry()+0x5ef) [0x7ff904ca48df] 13: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff904d681cd] 14: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7ff905680ea7] 15: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Ceph Cluster status: barril1:~# ceph status cluster: id: c30ecc8d-440e-4608-b3fe-5020337ae11d health: HEALTH_ERR 2 filesystems are degraded 2 filesystems are offline services: mon: 5 daemons, quorum barril4,barril3,barril2,barril1,urquell (age 32h) mgr: barril2(active, since 32h), standbys: barril3, barril4, urquell, barril1 mds: 0/10 daemons up (10 failed), 9 standby osd: 48 osds: 48 up (since 32h), 48 in (since 2M); 22 remapped pgs rgw: 4 daemons active (4 hosts, 1 zones) data: volumes: 0/2 healthy, 2 failed pools: 12 pools, 1475 pgs objects: 50.89M objects, 72 TiB usage: 207 TiB used, 148 TiB / 355 TiB avail pgs: 579358/152674596 objects misplaced (0.379%) 1449 active+clean 22 active+remapped+backfilling 4 active+clean+scrubbing+deep io: client: 7.2 MiB/s rd, 1.2 MiB/s wr, 342 op/s rd, 367 op/s wr recovery: 26 MiB/s, 13 keys/s, 26 objects/s progress: Global Recovery Event (19h) [===========================.] (remaining: 17m) Ceph fs status: barril1:~# ceph fs status cephfs - 0 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed 1 failed 2 failed 3 failed 4 failed 5 failed 6 failed 7 failed 8 failed POOL TYPE USED AVAIL cephfs_metadata metadata 1045G 35.6T cephfs.c3sl.data data 114T 35.6T c3sl - 0 clients ==== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed POOL TYPE USED AVAIL cephfs.c3sl.meta metadata 28.2G 35.6T cephfs.c3sl.data data 114T 35.6T STANDBY MDS barril2 barril4 barril42 barril33 barril13 barril23 barril43 barril1 barril12 MDS version: ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) ceph health detail: barril1:~# ceph health detail HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline [WRN] FS_DEGRADED: 2 filesystems are degraded fs cephfs is degraded fs c3sl is degraded [ERR] MDS_ALL_DOWN: 2 filesystems are offline fs cephfs is offline because no MDS is active for it. fs c3sl is offline because no MDS is active for it. [1]: https://docs.ceph.com/en/reef/cephfs/disaster-recovery-experts/#using-an-al… Best regards, Odair M. Ditkun Jr

4 4

How to confirm cache hit rate in ceph osd.
by mitsu 18 Oct '23

18 Oct '23

Hi, I'd like to know cache hit rate in ceph osd. I installed prometheus and grafana. But there aren't cache hit rate on grafana dashbords... Does Ceph have a cache hit rate counter? I'd like to know the impact of READ performance on Ceph cluster. Regards, -- Mitsumasa KONDO

1 0

Dashboard and Object Gateway
by Tim Holloway 17 Oct '23

17 Oct '23

First, an abject apology for the horrors I'm about to unveil. I made a cold migration from GlusterFS to Ceph a few months back, so it was a learn-/screwup/-as-you-go affair. For reasons of presumed compatibility with some of my older servers, I started with Ceph Octopus. Unfortunately, Octopus seems to have been a nexus of transitions from older Ceph organization and management to a newer (cephadm) system combined with a relocation of many ceph resources and compounded by stale bits of documentation (notably some references to SysV procedures and an obsolete installer that doesn't even come with Octopus). A far bigger problem was a known issue where actions would be scheduled but never executed if the system was even slightly dirty. And of course, since my system was hopelessly dirty, that was a major issue. Finally I took a risk and bumped up to Pacific, where that issue no longer exists. I won't say that I'm 100% clean even now, but at least the remaining crud is in areas where it cannot do any harm. Presumably. Given that, the only bar now remaining to total joy has been my inability to connect via the Ceph Dashboard to the Object Gateway. This seems to be an oft-reported problem, but generally referenced relative to higher-level administrative interfaces like Kubernetes and rook. I'm interfacing more directly, however. Regardless, the error reported is notably familiar: [quote] The Object Gateway Service is not configured Error connecting to Object Gateway: RGW REST API failed request with status code 404 (b'{"Code":"NoSuchBucket","Message":"","BucketName":"default","RequestI d":"tx00' b'000dd0c65b8bda685b4-00652d8e0f-5e3a9b- default","HostId":"5e3a9b-default-defa' b'ult"}') Please consult the documentation on how to configure and enable the Object Gateway management functionality. [/quote] In point of fact, what this REALLY means in my case is that the bucket that is supposed to contain the necessary information for the dashboard and rgw to communicate has not been created. Presumably that SHOULDhave been done by the "ceph dashboard set-rgw-credentials" command, but apparently isn't, because the default zone has no buckets at all, much less one named "default". By way of reference, the dashboard is definitely trying to interact with the rgw container, because trying object gateway options on the dashboard result in the container logging the following. beast: 0x7efd29621620: 10.0.1.16 - dashboard [16/Oct/2023:19:25:03.678 +0000] "GET /default/metadata/user?myself HTTP/1.1" 404 To make everything happy, I'd be glad to accept instructions on how to manually brute-force construct this bucket. Of course, as a cleaner long-term solution, it would be nice if the failure to create could be detected and logged. And of course, the ultimate solution: something that would assist in making whatever processes are unhappy be happy. Thanks, Tim

3 6

Unable to delete rbd images
by Mohammad Alam 17 Oct '23

17 Oct '23

Hello All, Greetings. We've a Ceph Cluster with the version *ceph version 14.2.16-402-g7d47dbaf4d (7d47dbaf4d0960a2e910628360ae36def84ed913) nautilus (stable) ========================================= Issues: Can't able to delete rbd images We have deleted target from the dashboard and now trying to delete rbd images from cli but not able to delete. when we ran "rbd rm -f tegile-500tb -p iscsi-images" its returning 2023-10-16 15:22:16.719 7f90bb332700 -1 librbd::image::PreRemoveRequest: 0x7f90a80041a0 check_image_watchers: image has watchers - not removing Removing image: 0% complete...failed. rbd: error: image still has watchers This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. ============================ It is also not being deleted from dashboard. ============================ Even we tried to list the watcher but it is not returning anything like no such file or directory , ============================ "rbd info iscsi-images/tegile-500tb" rbd: error opening image tegile-500tb: (2) No such file or directory ============================ It is not showing on "rbd showmapped" output as well for that particular images, hence we can not unmap it. We can not restart iscsi gateway because that is being running and we can not interrupt it. =========================== Suggest how to fix this issue,

2 1

Ceph 16.2.14: how to set mon_rocksdb_options to enable RocksDB compression?
by Zakhar Kirpichenko 17 Oct '23

17 Oct '23

Hi, I'm still trying to fight large Ceph monitor writes. One option I considered is enabling RocksDB compression, as our nodes have more than sufficient RAM and CPU. Unfortunately, monitors seem to completely ignore the compression setting: I tried: - setting ceph config set mon.ceph05 mon_rocksdb_options "write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true", restarting the test monitor. The monitor started with no RocksDB compression: debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Compression algorithms supported: debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kZSTDNotFinalCompression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kXpressCompression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kLZ4HCCompression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kLZ4Compression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kBZip2Compression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kZlibCompression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kSnappyCompression supported: 1 ... debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Options.compression: NoCompression debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Options.bottommost_compression: Disabled - setting ceph config set mon mon_rocksdb_options "write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true", restarting the test monitor. The monitor started with no RocksDB compression, the same way as above. In each case config options were correctly set and readable with config get. I also found a suggestion in ceph-users ( https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/KJM232IHN7…) to set compression in a similar manner. Unfortunately, these options appear to be ignored. How can I enable RocksDB compression in Ceph monitors? I would very much appreciate your advices and comments. Best regards, Zakhar

3 8

RGW: How to trigger to recalculate the bucket stats?
by Huy Nguyen 17 Oct '23

17 Oct '23

Hi, For some reason, I need to recalculate the bucket stats. Is this possible? Thanks

1 0

NFS - HA and Ingress completion note?
by andreas＠lsit.ucsb.edu 17 Oct '23

17 Oct '23

NFS - HA and Ingress: [ https://docs.ceph.com/en/latest/mgr/nfs/#ingress ] Referring to Note#2, is NFS high-availability functionality considered complete (and stable)?

1 0

Time to Upgrade from Nautilus
by Dave Hall 15 Oct '23

15 Oct '23

Hello. It's been a while. For the past couple years I've had a cluster running Nautilus on Debian 10 using the Debian Ceph packages, and deployed with Ceph-Ansible. It's not a huge cluster - 10 OSD nodes with 80 x 12TB HDD OSDs, plus 3 management nodes, and about 40% full at the moment - but it is a critical resource for one of our researchers. Back then I had some misgivings about non-Debian packages and also about containerized Ceph. I don't know if my feelings about these things have changed that much, but it's time to upgrade, and, with the advent of cephadm it looks like it's just better to stay mainstream. So I'm looking for advice on how to get from where I'm at to at least Pacific or Quincy. I've read a little in the last couple days. I've seen various opinions on (not) skipping releases and on when to switch to cephadm. I'm also concerned about cleaning up those old Debian packages - will there be a point where I can 'apt-get purge' them without harming the cluster. One particular thing: The upgrade instructions in various places on docs.ceph.com say something like Upgrade monitors by installing the new packages and restarting the monitor daemons. To me this is kind of vague. Perhaps there is a different concept fo 'packages' within the cephadm environment. I could really use some clarification on this. I'd also consider decommissioning a few nodes, setting up a new cluster on fresh Debian installs. and migrating the data and remaining nodes. This would be a long and painful process - decommission a node, move it, move some data, decommission another node - and I don't know what effect it would have on external references to our object store. Please advise. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

3 3

Is nfs-ganesha + kerberos actually a thing?
by Torkil Svensgaard 13 Oct '23

13 Oct '23

Hi We have kerberos working with bare metal kernel NFS exporting RBDs. I can see in the ceph documentation[1] that nfs-ganesha should work with kerberos but I'm having little luck getting it to work. This bit from the container log seems to suggest that some plumbing is missing? " 13/10/2023 08:09:12 : epoch 6528fb25 : ceph-flash1 : ganesha.nfsd-2[main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/var/run/ganesha) already exists 13/10/2023 08:09:12 : epoch 6528fb25 : ceph-flash1 : ganesha.nfsd-2[main] find_keytab_entry :NFS CB :WARN :Configuration file does not specify default realm while getting default realm name 13/10/2023 08:09:12 : epoch 6528fb25 : ceph-flash1 : ganesha.nfsd-2[main] gssd_refresh_krb5_machine_credential :NFS CB :CRIT :ERROR: gssd_refresh_krb5_machine_credential: no usable keytab entry found in keytab /etc/krb5.keytab for connection with host localhost 13/10/2023 08:09:12 : epoch 6528fb25 : ceph-flash1 : ganesha.nfsd-2[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328160:0) " Thoughts? Mvh. Torkil [1] https://docs.ceph.com/en/quincy/mgr/nfs/#create-cephfs-export -- Torkil Svensgaard Systems Administrator Danish Research Centre for Magnetic Resonance DRCMR, Section 714 Copenhagen University Hospital Amager and Hvidovre Kettegaard Allé 30, 2650 Hvidovre, Denmark

2 5

Please help collecting stats of Ceph monitor disk writes
by Zakhar Kirpichenko 13 Oct '23

13 Oct '23

Hi! Further to my thread "Ceph 16.2.x mon compactions, disk writes" ( https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/XGCI2LFW5RH…) where we have established that Ceph monitors indeed write considerable amounts of data to disks, I would like to request fellow Ceph users to provide feedback and help gather some statistics regarding whether this happens on all clusters or on some specific subset of clusters. The procedure is rather simple and won't take much of your time. If you are willing to help, please follow this procedure: --------- 1. Install iotop and run the following command on any of your monitor nodes: iotop -ao -bn 2 -d 300 2>&1 | grep -E "TID|ceph-mon" This will collect a 5-minute disk I/O statistics and produce an output containing the stats for Ceph monitor threads running on the node: TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 4854 be/4 167 8.62 M 2.27 G 0.00 % 0.72 % ceph-mon -n mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true [rocksdb:low0] 4919 be/4 167 0.00 B 39.43 M 0.00 % 0.02 % ceph-mon -n mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true [ms_dispatch] 4855 be/4 167 8.00 K 19.55 M 0.00 % 0.00 % ceph-mon -n mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true [rocksdb:high0] We're particularly interested in the amount of written data. --------- 2. Optional: collect the number of "manual compaction" events from the monitor. This step will depend on how your monitor runs. My cluster is managed by cephadm and monitors run in docker containers, thus I can do something like this, where MYMONCONTAINERID is the container ID of Ceph monitor: # date; d=$(date +'%Y-%m-%d'); docker logs MYMONCONTAINERID 2>&1 | grep $d | grep -ci "manual compaction from" Fri 13 Oct 2023 06:29:39 AM UTC 580 Alternatively, I could run the command against the log file MYMONLOGFILE, whose location I obtained with docker inspect: # date; d=$(date +'%Y-%m-%d'); grep $d MYMONLOGFILE | grep -ci "manual compaction from" Fri 13 Oct 2023 06:35:27 AM UTC 588 If you run monitors with podman or without containerization, please get this information the way that is most convenient in your setup. --------- 3. Optional: collect the monitor store.db size. Usually the monitor store.db is available at /var/lib/ceph/FSID/mon.NAME/store.db/, for example: # du -hs /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/ 642M /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/ --------- 4. Optional: collect Ceph cluster version and status. For example: root@ceph01:/# ceph version; ceph -s ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) cluster: id: 3f50555a-ae2a-11eb-a2fc-ffde44714d86 health: HEALTH_OK services: mon: 5 daemons, quorum ceph01,ceph03,ceph04,ceph05,ceph02 (age 2w) mgr: ceph01.vankui(active, since 13d), standbys: ceph02.shsinf osd: 96 osds: 96 up (since 2w), 95 in (since 3w) data: pools: 10 pools, 2400 pgs objects: 6.30M objects, 16 TiB usage: 61 TiB used, 716 TiB / 777 TiB avail pgs: 2396 active+clean 3 active+clean+scrubbing+deep 1 active+clean+scrubbing io: client: 71 MiB/s rd, 60 MiB/s wr, 2.94k op/s rd, 2.56k op/s wr --------- 5. Reply to this thread and submit the collected information. For example: 1) iotop results: ... Paste data obtained in step 1) 2) manual compactions: ... Paste data obtained in step 2), or put "N/A" 3) monitor store.db size: ... Paste data obtained in step 3), or put "N/A" 4) cluster version and status: ... Paste data obtained in step 4), or put "N/A" ------------- I would very much appreciate your effort and help with gathering these stats. Please don't hesitate to contact me with any questions or concerns. Best regards, Zakhar

5 6

CephFS: convert directory into subvolume
by Eugen Block 12 Oct '23

12 Oct '23

Hi, while writing a response to [1] I tried to convert an existing directory within a single cephfs into a subvolume. According to [2] that should be possible, I'm just wondering how to confirm that it actually worked. Because setting the xattr works fine, the directory just doesn't show up in the subvolume ls command. This is what I tried (in Reef and Pacific): # one "regular" subvolume already exists $ ceph fs subvolume ls cephfs [ { "name": "subvol1" } ] # mounted / and created new subdir $ mkdir /mnt/volumes/subvol2 $ setfattr -n ceph.dir.subvolume -v 1 /mnt/volumes/subvol2 # still only one subvolume $ ceph fs subvolume ls cephfs [ { "name": "subvol1" } ] I also tried it directly underneath /mnt: $ mkdir /mnt/subvol2 $ setfattr -n ceph.dir.subvolume -v 1 /mnt/subvol2 But still no subvolume2 available. What am I missing here? Thanks Eugen [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/G4ZWGGUPPFQ… [2] https://www.spinics.net/lists/ceph-users/msg72341.html

3 7

Clients failing to respond to capability release
by Tim Bishop 12 Oct '23

12 Oct '23

Hi, I've seen this issue mentioned in the past, but with older releases. So I'm wondering if anybody has any pointers. The Ceph cluster is running Pacific 16.2.13 on Ubuntu 20.04. Almost all clients are working fine, with the exception of our backup server. This is using the kernel CephFS client on Ubuntu 22.04 with kernel 6.2.0 [1] (so I suspect a newer Ceph version?). The backup server has multiple (12) CephFS mount points. One of them, the busiest, regularly causes this error on the cluster: HEALTH_WARN 1 clients failing to respond to capability release [WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability release mds.mds-server(mds.0): Client backupserver:cephfs-backupserver failing to respond to capability release client_id: 521306112 And occasionally, which may be unrelated, but occurs at the same time: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests mds.mds-server(mds.0): 1 slow requests are blocked > 30 secs The second one clears itself, but the first sticks until I can unmount the filesystem on the client after the backup completes. It appears that whilst it's in this stuck state there may be one or more directory trees that are inaccessible to all clients. The backup server is walking the whole tree but never gets stuck itself, so either the inaccessible directory entry is caused after it has gone past, or it's not affected. Maybe the backup server is holding a directory when it shouldn't? It may be that an upgrade to Quincy resolves this, since it's more likely to be inline with the kernel client version wise, but I don't want to knee-jerk upgrade just to try and fix this problem. Thanks for any advice. Tim. [1] The reason for the newer kernel is that the backup performance from CephFS was terrible with older kernels. This newer kernel does at least resolve that issue.

4 5

slow recovery with Quincy
by Ben 12 Oct '23

12 Oct '23

Dear cephers, with one osd down(200GB/9.1TB data), rebalance takes 3 hours still in progress. Client bandwidth can go as high as 200MB/s. With little client request throughput, recovery goes at couple MB/s. I wonder if there is configuration to polish for improvement. It runs with quincy 17.2.5, deployed by cephadm. The slowness can do harm in peak hours of usage. Best wishes, Ben ------------------------------------------------- volumes: 1/1 healthy pools: 8 pools, 209 pgs objects: 93.04M objects, 4.8 TiB usage: 15 TiB used, 467 TiB / 482 TiB avail pgs: 1206837/279121971 objects degraded (0.432%) 208 active+clean 1 active+undersized+degraded+remapped+backfilling io: client: 80 KiB/s rd, 420 KiB/s wr, 12 op/s rd, 29 op/s wr recovery: 6.2 MiB/s, 113 objects/s

2 2

Unable to fix 1 Inconsistent PG
by samdto987＠gmail.com 11 Oct '23

11 Oct '23

Hello All, Greetings. We've a Ceph Cluster with the version *ceph version 14.2.16-402-g7d47dbaf4d (7d47dbaf4d0960a2e910628360ae36def84ed913) nautilus (stable) =================================== Issues: 1 pg in inconsistent state and does not recover. # ceph -s cluster: id: 30d6f7ee-fa02-4ab3-8a09-9321c8002794 health: HEALTH_ERR 2 large omap objects 1 pools have many more objects per pg than average 159224 scrub errors Possible data damage: 1 pg inconsistent 2 pgs not deep-scrubbed in time 2 pgs not scrubbed in time # ceph health detail HEALTH_ERR 2 large omap objects; 1 pools have many more objects per pg than average; 159224 scrub errors; Possible data damage: 1 pg inconsistent; 2 pgs not deep-scrubbed in time; 2 pgs not scrubbed in time LARGE_OMAP_OBJECTS 2 large omap objects 2 large objects found in pool 'default.rgw.log' Search the cluster log for 'Large omap object found' for more details. MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average pool iscsi-images objects per pg (541376) is more than 14.9829 times cluster average (36133) OSD_SCRUB_ERRORS 159224 scrub errors PG_DAMAGED Possible data damage: 1 pg inconsistent pg 15.f4f is active+clean+inconsistent, acting [238,106,402,266,374,498,590,627,684,73,66] PG_NOT_DEEP_SCRUBBED 2 pgs not deep-scrubbed in time pg 1.5c not deep-scrubbed since 2021-04-05 23:20:13.714446 pg 1.55 not deep-scrubbed since 2021-04-11 07:12:37.185074 PG_NOT_SCRUBBED 2 pgs not scrubbed in time pg 1.5c not scrubbed since 2023-07-10 21:15:50.352848 pg 1.55 not scrubbed since 2023-06-24 10:02:10.038311 ====================================== We have implemented below command to resolve it 1. We have ran pg repair command "ceph pg repair 15.f4f 2. We have restarted associated OSDs that is mapped to pg 15.f4f 3. We tuned osd_max_scrubs value and set it to 9. 4. We have done scrub and deep scrub by ceph pg scrub 15.4f4 & ceph pg deep-scrub 15.f4f 5. We also tried to ceph-objectstore-tool command to fix it ============================================== We have checked the logs of the primary OSD of the respective inconsistent PG and found the below errors. [ERR] : 15.f4fs0 shard 402(2) 15:f2f3fff4:::94a51ddb-a94f-47bc-9068-509e8c09af9a.7862003.20_c%2f4%2fd61%2f885%2f49627697%2f192_1.ts:head : missing /var/log/ceph/ceph-osd.238.log:339:2023-10-06 00:37:06.410 7f65024cb700 -1 log_channel(cluster) log [ERR] : 15.f4fs0 shard 266(3) 15:f2f00002:::94a51ddb-a94f-47bc-9068-509e8c09af9a.11432468.3_TN8QHE_04.20.2020_08.41%2fCV_MAGNETIC%2fV_274396%2fCHUNK_2440801%2fSFILE_CONTAINER_031.FOLDER%2f3:head : missing /var/log/ceph/ceph-osd.238.log:340:2023-10-06 00:37:06.410 7f65024cb700 -1 log_channel(cluster) log [ERR] : 15.f4fs0 shard 402(2) 15:f2f00002:::94a51ddb-a94f-47bc-9068-509e8c09af9a.11432468.3_TN8QHE_04.20.2020_08.41%2fCV_MAGNETIC%2fV_274396%2fCHUNK_2440801%2fSFILE_CONTAINER_031.FOLDER%2f3:head : missing /var/log/ceph/ceph-osd.238.log:341:2023-10-06 00:37:06.410 7f65024cb700 -1 log_channel(cluster) log [ERR] : 15.f4fs0 shard 590(6) 15:f2f00002:::94a51ddb-a94f-47bc-9068-509e8c09af9a.11432468.3_TN8QHE_04.20.2020_08.41%2fCV_MAGNETIC%2fV_274396%2fCHUNK_2440801%2fSFILE_CONTAINER_031.FOLDER%2f3:head : missing =============================== and also we noticed that the no. of scrub errors in ceph health status are matching with the ERR log entries in the primary OSD logs of the inconsistent PG as below grep -Hn 'ERR' /var/log/ceph/ceph-osd.238.log|wc -l 159226 ================================ Ceph is cleaning the scrub errors but rate of scrub repair is very slow (avg of 200 scrub errors per day) ,we want to increase the rate of scrub error repair to finish the cleanup of pending 159224 scrub errors. #ceph pg 15.f4f query { "state": "active+clean+inconsistent", "snap_trimq": "[]", "snap_trimq_len": 0, "epoch": 409009, "up": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "acting": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "acting_recovery_backfill": [ "66(10)", "73(9)", "106(1)", "238(0)", "266(3)", "374(4)", "402(2)", "498(5)", "590(6)", "627(7)", "684(8)" ], "info": { "pgid": "15.f4fs0", "last_update": "409009'7998", "last_complete": "409009'7998", "log_tail": "382701'4900", "last_user_version": 592883, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [], "history": { "epoch_created": 19813, "epoch_pool_created": 16141, "last_epoch_started": 407097, "last_interval_started": 407096, "last_epoch_clean": 407097, "last_interval_clean": 407096, "last_epoch_split": 19849, "last_epoch_marked_full": 0, "same_up_since": 407096, "same_interval_since": 407096, "same_primary_since": 407038, "last_scrub": "408987'7946", "last_scrub_stamp": "2023-10-06 00:53:53.705241", "last_deep_scrub": "408987'7946", "last_deep_scrub_stamp": "2023-10-06 00:53:53.705241", "last_clean_scrub_stamp": "2023-09-28 14:48:47.098935" }, "stats": { "version": "409009'7998", "reported_seq": "379048", "reported_epoch": "409009", "state": "active+clean+inconsistent", "last_fresh": "2023-10-06 15:49:09.174662", "last_change": "2023-10-06 00:53:53.705308", "last_active": "2023-10-06 15:49:09.174662", "last_peered": "2023-10-06 15:49:09.174662", "last_clean": "2023-10-06 15:49:09.174662", "last_became_active": "2023-10-03 19:55:56.742034", "last_became_peered": "2023-10-03 19:55:56.742034", "last_unstale": "2023-10-06 15:49:09.174662", "last_undegraded": "2023-10-06 15:49:09.174662", "last_fullsized": "2023-10-06 15:49:09.174662", "mapping_epoch": 407096, "log_start": "382701'4900", "ondisk_log_start": "382701'4900", "created": 19813, "last_epoch_clean": 407097, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "408987'7946", "last_scrub_stamp": "2023-10-06 00:53:53.705241", "last_deep_scrub": "408987'7946", "last_deep_scrub_stamp": "2023-10-06 00:53:53.705241", "last_clean_scrub_stamp": "2023-09-28 14:48:47.098935", "log_size": 3098, "ondisk_log_size": 3098, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 45852508474, "num_objects": 40234, "num_object_clones": 0, "num_object_copies": 442574, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 40234, "num_whiteouts": 0, "num_read": 23940, "num_read_kb": 2900611, "num_write": 34950, "num_write_kb": 2440513, "num_scrub_errors": 159224, "num_shallow_scrub_errors": 159224, "num_deep_scrub_errors": 0, "num_objects_recovered": 86741, "num_bytes_recovered": 97127524421, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 42593 }, "up": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "acting": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 238, "acting_primary": 238, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 407097, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, "peer_info": [ { "peer": "66(10)", "pgid": "15.f4fs10", "last_update": "409009'7998", "last_complete": "407068'7791", "log_tail": "378965'4700", "last_user_version": 592676, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19813, "epoch_pool_created": 16141, "last_epoch_started": 407097, "last_interval_started": 407096, "last_epoch_clean": 407097, "last_interval_clean": 407096, "last_epoch_split": 19849, "last_epoch_marked_full": 0, "same_up_since": 407096, "same_interval_since": 407096, "same_primary_since": 407038, "last_scrub": "408987'7946", "last_scrub_stamp": "2023-10-06 00:53:53.705241", "last_deep_scrub": "408987'7946", "last_deep_scrub_stamp": "2023-10-06 00:53:53.705241", "last_clean_scrub_stamp": "2023-09-28 14:48:47.098935" }, "stats": { "version": "407068'7791", "reported_seq": "376441", "reported_epoch": "407068", "state": "active+clean+inconsistent", "last_fresh": "2023-10-03 19:07:04.483241", "last_change": "2023-10-03 18:02:27.182058", "last_active": "2023-10-03 19:07:04.483241", "last_peered": "2023-10-03 19:07:04.483241", "last_clean": "2023-10-03 19:07:04.483241", "last_became_active": "2023-10-03 18:02:27.181798", "last_became_peered": "2023-10-03 18:02:27.181798", "last_unstale": "2023-10-03 19:07:04.483241", "last_undegraded": "2023-10-03 19:07:04.483241", "last_fullsized": "2023-10-03 19:07:04.483241", "mapping_epoch": 407096, "log_start": "378965'4700", "ondisk_log_start": "378965'4700", "created": 19813, "last_epoch_clean": 407068, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "407037'7783", "last_scrub_stamp": "2023-10-03 17:14:13.232978", "last_deep_scrub": "406839'7692", "last_deep_scrub_stamp": "2023-09-28 14:48:47.098935", "last_clean_scrub_stamp": "2023-09-28 14:48:47.098935", "log_size": 3091, "ondisk_log_size": 3091, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 45930070381, "num_objects": 40379, "num_object_clones": 0, "num_object_copies": 444169, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 40379, "num_whiteouts": 0, "num_read": 23277, "num_read_kb": 2877425, "num_write": 34413, "num_write_kb": 2409735, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 86740, "num_bytes_recovered": 97127524421, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 42593 }, "up": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "acting": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 238, "acting_primary": 238, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 407097, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "73(9)", "pgid": "15.f4fs9", "last_update": "409009'7998", "last_complete": "409009'7998", "log_tail": "378965'4700", "last_user_version": 592677, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [], "history": { "epoch_created": 19813, "epoch_pool_created": 16141, "last_epoch_started": 407097, "last_interval_started": 407096, "last_epoch_clean": 407097, "last_interval_clean": 407096, "last_epoch_split": 19849, "last_epoch_marked_full": 0, "same_up_since": 407096, "same_interval_since": 407096, "same_primary_since": 407038, "last_scrub": "408987'7946", "last_scrub_stamp": "2023-10-06 00:53:53.705241", "last_deep_scrub": "408987'7946", "last_deep_scrub_stamp": "2023-10-06 00:53:53.705241", "last_clean_scrub_stamp": "2023-09-28 14:48:47.098935" }, "stats": { "version": "407095'7792", "reported_seq": "376677", "reported_epoch": "407095", "state": "active+undersized+degraded+inconsistent", "last_fresh": "2023-10-03 19:54:53.799024", "last_change": "2023-10-03 19:53:22.457623", "last_active": "2023-10-03 19:54:53.799024", "last_peered": "2023-10-03 19:54:53.799024", "last_clean": "2023-10-03 19:49:47.048960", "last_became_active": "2023-10-03 19:53:22.457623", "last_became_peered": "2023-10-03 19:53:22.457623", "last_unstale": "2023-10-03 19:54:53.799024", "last_undegraded": "2023-10-03 19:53:22.379335", "last_fullsized": "2023-10-03 19:53:22.379134", "mapping_epoch": 407096, "log_start": "378965'4700", "ondisk_log_start": "378965'4700", "created": 19813, "last_epoch_clean": 407093, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "407037'7783", "last_scrub_stamp": "2023-10-03 17:14:13.232978", "last_deep_scrub": "406839'7692", "last_deep_scrub_stamp": "2023-09-28 14:48:47.098935", "last_clean_scrub_stamp": "2023-09-28 14:48:47.098935", "log_size": 3092, "ondisk_log_size": 3092, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 45929876177, "num_objects": 40378, "num_object_clones": 0, "num_object_copies": 444158, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 40378, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 40378, "num_whiteouts": 0, "num_read": 23280, "num_read_kb": 2877429, "num_write": 34414, "num_write_kb": 2409735, "num_scrub_errors": 159632, "num_shallow_scrub_errors": 159632, "num_deep_scrub_errors": 0, "num_objects_recovered": 86740, "num_bytes_recovered": 97127524421, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 42593 }, "up": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "acting": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "avail_no_missing": [ "238(0)", "73(9)", "106(1)", "266(3)", "374(4)", "402(2)", "498(5)", "590(6)", "627(7)", "684(8)" ], "object_location_counts": [ { "shards": "73(9),106(1),238(0),266(3),374(4),402(2),498(5),590(6),627(7),684(8)", "objects": 40378 } ], "blocked_by": [], "up_primary": 238, "acting_primary": 238, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 407097, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "106(1)", "pgid": "15.f4fs1", "last_update": "409009'7998", "last_complete": "409009'7998", "log_tail": "378965'4700", "last_user_version": 592677, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19813, "epoch_pool_created": 16141, "last_epoch_started": 407097, "last_interval_started": 407096, "last_epoch_clean": 407097, "last_interval_clean": 407096, "last_epoch_split": 19849, "last_epoch_marked_full": 0, "same_up_since": 407096, "same_interval_since": 407096, "same_primary_since": 407038, "last_scrub": "408987'7946", "last_scrub_stamp": "2023-10-06 00:53:53.705241", "last_deep_scrub": "408987'7946", "last_deep_scrub_stamp": "2023-10-06 00:53:53.705241", "last_clean_scrub_stamp": "2023-09-28 14:48:47.098935" }, "stats": { "version": "407095'7792", "reported_seq": "376677", "reported_epoch": "407095", "state": "active+undersized+degraded+inconsistent", "last_fresh": "2023-10-03 19:54:53.799024", "last_change": "2023-10-03 19:53:22.457623", "last_active": "2023-10-03 19:54:53.799024", "last_peered": "2023-10-03 19:54:53.799024", "last_clean": "2023-10-03 19:49:47.048960", "last_became_active": "2023-10-03 19:53:22.457623", "last_became_peered": "2023-10-03 19:53:22.457623", "last_unstale": "2023-10-03 19:54:53.799024", "last_undegraded": "2023-10-03 19:53:22.379335", "last_fullsized": "2023-10-03 19:53:22.379134", "mapping_epoch": 407096, "log_start": "378965'4700", "ondisk_log_start": "378965'4700", "created": 19813, "last_epoch_clean": 407093, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "407037'7783", "last_scrub_stamp": "2023-10-03 17:14:13.232978", "last_deep_scrub": "406839'7692", "last_deep_scrub_stamp": "2023-09-28 14:48:47.098935", "last_clean_scrub_stamp": "2023-09-28 14:48:47.098935", "log_size": 3092, "ondisk_log_size": 3092, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 45929876177, "num_objects": 40378, "num_object_clones": 0, "num_object_copies": 444158, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 40378, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 40378, "num_whiteouts": 0, "num_read": 23280, "num_read_kb": 2877429, "num_write": 34414, "num_write_kb": 2409735, "num_scrub_errors": 159632, "num_shallow_scrub_errors": 159632, "num_deep_scrub_errors": 0, "num_objects_recovered": 86740, "num_bytes_recovered": 97127524421, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 42593 }, "up": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "acting": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "avail_no_missing": [ "238(0)", "73(9)", "106(1)", "266(3)", "374(4)", "402(2)", "498(5)", "590(6)", "627(7)", "684(8)" ], "object_location_counts": [ { "shards": "73(9),106(1),238(0),266(3),374(4),402(2),498(5),590(6),627(7),684(8)", "objects": 40378 } ], "blocked_by": [], "up_primary": 238, "acting_primary": 238, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 407097, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "266(3)", "pgid": "15.f4fs3", "last_update": "409009'7998", "last_complete": "409009'7998", "log_tail": "378965'4700", "last_user_version": 592677, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19813, "epoch_pool_created": 16141, "last_epoch_started": 407097, "last_interval_started": 407096, "last_epoch_clean": 407097, "last_interval_clean": 407096, "last_epoch_split": 19849, "last_epoch_marked_full": 0, "same_up_since": 407096, "same_interval_since": 407096, "same_primary_since": 407038, "last_scrub": "408987'7946", "last_scrub_stamp": "2023-10-06 00:53:53.705241", "last_deep_scrub": "408987'7946", "last_deep_scrub_stamp": "2023-10-06 00:53:53.705241", "last_clean_scrub_stamp": "2023-09-28 14:48:47.098935" }, "stats": { "version": "407095'7792", "reported_seq": "376677", "reported_epoch": "407095", "state": "active+undersized+degraded+inconsistent", "last_fresh": "2023-10-03 19:54:53.799024", "last_change": "2023-10-03 19:53:22.457623", "last_active": "2023-10-03 19:54:53.799024", "last_peered": "2023-10-03 19:54:53.799024", "last_clean": "2023-10-03 19:49:47.048960", "last_became_active": "2023-10-03 19:53:22.457623", "last_became_peered": "2023-10-03 19:53:22.457623", "last_unstale": "2023-10-03 19:54:53.799024", "last_undegraded": "2023-10-03 19:53:22.379335", "last_fullsized": "2023-10-03 19:53:22.379134", "mapping_epoch": 407096, "log_start": "378965'4700", "ondisk_log_start": "378965'4700", "created": 19813, "last_epoch_clean": 407093, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "407037'7783", "last_scrub_stamp": "2023-10-03 17:14:13.232978", "last_deep_scrub": "406839'7692", "last_deep_scrub_stamp": "2023-09-28 14:48:47.098935", "last_clean_scrub_stamp": "2023-09-28 14:48:47.098935", "log_size": 3092, "ondisk_log_size": 3092, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 45929876177, "num_objects": 40378, "num_object_clones": 0, "num_object_copies": 444158, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 40378, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 40378, "num_whiteouts": 0, "num_read": 23280, "num_read_kb": 2877429, "num_write": 34414, "num_write_kb": 2409735, "num_scrub_errors": 159632, "num_shallow_scrub_errors": 159632, "num_deep_scrub_errors": 0, "num_objects_recovered": 86740, "num_bytes_recovered": 97127524421, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 42593 }, "up": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "acting": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "avail_no_missing": [ "238(0)", "73(9)", "106(1)", "266(3)", "374(4)", "402(2)", "498(5)", "590(6)", "627(7)", "684(8)" ], "object_location_counts": [ { "shards": "73(9),106(1),238(0),266(3),374(4),402(2),498(5),590(6),627(7),684(8)", "objects": 40378 } ], "blocked_by": [], "up_primary": 238, "acting_primary": 238, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 407097, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "374(4)", "pgid": "15.f4fs4", "last_update": "409009'7998", "last_complete": "409009'7998", "log_tail": "378965'4700", "last_user_version": 592677, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19813, "epoch_pool_created": 16141, "last_epoch_started": 407097, "last_interval_started": 407096, "last_epoch_clean": 407097, "last_interval_clean": 407096, "last_epoch_split": 19849, "last_epoch_marked_full": 0, "same_up_since": 407096, "same_interval_since": 407096, "same_primary_since": 407038, "last_scrub": "408987'7946", "last_scrub_stamp": "2023-10-06 00:53:53.705241", "last_deep_scrub": "408987'7946", "last_deep_scrub_stamp": "2023-10-06 00:53:53.705241", "last_clean_scrub_stamp": "2023-09-28 14:48:47.098935" }, "stats": { "version": "407095'7792", "reported_seq": "376677", "reported_epoch": "407095", "state": "active+undersized+degraded+inconsistent", "last_fresh": "2023-10-03 19:54:53.799024", "last_change": "2023-10-03 19:53:22.457623", "last_active": "2023-10-03 19:54:53.799024", "last_peered": "2023-10-03 19:54:53.799024", "last_clean": "2023-10-03 19:49:47.048960", "last_became_active": "2023-10-03 19:53:22.457623", "last_became_peered": "2023-10-03 19:53:22.457623", "last_unstale": "2023-10-03 19:54:53.799024", "last_undegraded": "2023-10-03 19:53:22.379335", "last_fullsized": "2023-10-03 19:53:22.379134", "mapping_epoch": 407096, "log_start": "378965'4700", "ondisk_log_start": "378965'4700", "created": 19813, "last_epoch_clean": 407093, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "407037'7783", "last_scrub_stamp": "2023-10-03 17:14:13.232978", "last_deep_scrub": "406839'7692", "last_deep_scrub_stamp": "2023-09-28 14:48:47.098935", "last_clean_scrub_stamp": "2023-09-28 14:48:47.098935", "log_size": 3092, "ondisk_log_size": 3092, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 45929876177, "num_objects": 40378, "num_object_clones": 0, "num_object_copies": 444158, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 40378, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 40378, "num_whiteouts": 0, "num_read": 23280, "num_read_kb": 2877429, "num_write": 34414, "num_write_kb": 2409735, "num_scrub_errors": 159632, "num_shallow_scrub_errors": 159632, "num_deep_scrub_errors": 0, "num_objects_recovered": 86740, "num_bytes_recovered": 97127524421, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 42593 }, "up": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "acting": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "avail_no_missing": [ "238(0)", "73(9)", "106(1)", "266(3)", "374(4)", "402(2)", "498(5)", "590(6)", "627(7)", "684(8)" ], "object_location_counts": [ { "shards": "73(9),106(1),238(0),266(3),374(4),402(2),498(5),590(6),627(7),684(8)", "objects": 40378 } ], "blocked_by": [], "up_primary": 238, "acting_primary": 238, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 407097, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "402(2)", "pgid": "15.f4fs2", "last_update": "409009'7998", "last_complete": "409009'7998", "log_tail": "378965'4700", "last_user_version": 592677, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [], "history": { "epoch_created": 19813, "epoch_pool_created": 16141, "last_epoch_started": 407097, "last_interval_started": 407096, "last_epoch_clean": 407097, "last_interval_clean": 407096, "last_epoch_split": 19849, "last_epoch_marked_full": 0, "same_up_since": 407096, "same_interval_since": 407096, "same_primary_since": 407038, "last_scrub": "408987'7946", "last_scrub_stamp": "2023-10-06 00:53:53.705241", "last_deep_scrub": "408987'7946", "last_deep_scrub_stamp": "2023-10-06 00:53:53.705241", "last_clean_scrub_stamp": "2023-09-28 14:48:47.098935" }, "stats": { "version": "407095'7792", "reported_seq": "376677", "reported_epoch": "407095", "state": "active+undersized+degraded+inconsistent", "last_fresh": "2023-10-03 19:54:53.799024", "last_change": "2023-10-03 19:53:22.457623", "last_active": "2023-10-03 19:54:53.799024", "last_peered": "2023-10-03 19:54:53.799024", "last_clean": "2023-10-03 19:49:47.048960", "last_became_active": "2023-10-03 19:53:22.457623", "last_became_peered": "2023-10-03 19:53:22.457623", "last_unstale": "2023-10-03 19:54:53.799024", "last_undegraded": "2023-10-03 19:53:22.379335", "last_fullsized": "2023-10-03 19:53:22.379134", "mapping_epoch": 407096, "log_start": "378965'4700", "ondisk_log_start": "378965'4700", "created": 19813, "last_epoch_clean": 407093, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "407037'7783", "last_scrub_stamp": "2023-10-03 17:14:13.232978", "last_deep_scrub": "406839'7692", "last_deep_scrub_stamp": "2023-09-28 14:48:47.098935", "last_clean_scrub_stamp": "2023-09-28 14:48:47.098935", "log_size": 3092, "ondisk_log_size": 3092, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 45929876177, "num_objects": 40378, "num_object_clones": 0, "num_object_copies": 444158, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 40378, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 40378, "num_whiteouts": 0, "num_read": 23280, "num_read_kb": 2877429, "num_write": 34414, "num_write_kb": 2409735, "num_scrub_errors": 159632, "num_shallow_scrub_errors": 159632, "num_deep_scrub_errors": 0, "num_objects_recovered": 86740, "num_bytes_recovered": 97127524421, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 42593 }, "up": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "acting": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "avail_no_missing": [ "238(0)", "73(9)", "106(1)", "266(3)", "374(4)", "402(2)", "498(5)", "590(6)", "627(7)", "684(8)" ], "object_location_counts": [ { "shards": "73(9),106(1),238(0),266(3),374(4),402(2),498(5),590(6),627(7),684(8)", "objects": 40378 } ], "blocked_by": [], "up_primary": 238, "acting_primary": 238, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 407097, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "498(5)", "pgid": "15.f4fs5", "last_update": "409009'7998", "last_complete": "409009'7998", "log_tail": "378965'4700", "last_user_version": 592677, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19813, "epoch_pool_created": 16141, "last_epoch_started": 407097, "last_interval_started": 407096, "last_epoch_clean": 407097, "last_interval_clean": 407096, "last_epoch_split": 19849, "last_epoch_marked_full": 0, "same_up_since": 407096, "same_interval_since": 407096, "same_primary_since": 407038, "last_scrub": "408987'7946", "last_scrub_stamp": "2023-10-06 00:53:53.705241", "last_deep_scrub": "408987'7946", "last_deep_scrub_stamp": "2023-10-06 00:53:53.705241", "last_clean_scrub_stamp": "2023-09-28 14:48:47.098935" }, "stats": { "version": "407095'7792", "reported_seq": "376677", "reported_epoch": "407095", "state": "active+undersized+degraded+inconsistent", "last_fresh": "2023-10-03 19:54:53.799024", "last_change": "2023-10-03 19:53:22.457623", "last_active": "2023-10-03 19:54:53.799024", "last_peered": "2023-10-03 19:54:53.799024", "last_clean": "2023-10-03 19:49:47.048960", "last_became_active": "2023-10-03 19:53:22.457623", "last_became_peered": "2023-10-03 19:53:22.457623", "last_unstale": "2023-10-03 19:54:53.799024", "last_undegraded": "2023-10-03 19:53:22.379335", "last_fullsized": "2023-10-03 19:53:22.379134", "mapping_epoch": 407096, "log_start": "378965'4700", "ondisk_log_start": "378965'4700", "created": 19813, "last_epoch_clean": 407093, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "407037'7783", "last_scrub_stamp": "2023-10-03 17:14:13.232978", "last_deep_scrub": "406839'7692", "last_deep_scrub_stamp": "2023-09-28 14:48:47.098935", "last_clean_scrub_stamp": "2023-09-28 14:48:47.098935", "log_size": 3092, "ondisk_log_size": 3092, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 45929876177, "num_objects": 40378, "num_object_clones": 0, "num_object_copies": 444158, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 40378, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 40378, "num_whiteouts": 0, "num_read": 23280, "num_read_kb": 2877429, "num_write": 34414, "num_write_kb": 2409735, "num_scrub_errors": 159632, "num_shallow_scrub_errors": 159632, "num_deep_scrub_errors": 0, "num_objects_recovered": 86740, "num_bytes_recovered": 97127524421, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 42593 }, "up": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "acting": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "avail_no_missing": [ "238(0)", "73(9)", "106(1)", "266(3)", "374(4)", "402(2)", "498(5)", "590(6)", "627(7)", "684(8)" ], "object_location_counts": [ { "shards": "73(9),106(1),238(0),266(3),374(4),402(2),498(5),590(6),627(7),684(8)", "objects": 40378 } ], "blocked_by": [], "up_primary": 238, "acting_primary": 238, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 407097, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "590(6)", "pgid": "15.f4fs6", "last_update": "409009'7998", "last_complete": "409009'7998", "log_tail": "378965'4700", "last_user_version": 592677, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [], "history": { "epoch_created": 19813, "epoch_pool_created": 16141, "last_epoch_started": 407097, "last_interval_started": 407096, "last_epoch_clean": 407097, "last_interval_clean": 407096, "last_epoch_split": 19849, "last_epoch_marked_full": 0, "same_up_since": 407096, "same_interval_since": 407096, "same_primary_since": 407038, "last_scrub": "408987'7946", "last_scrub_stamp": "2023-10-06 00:53:53.705241", "last_deep_scrub": "408987'7946", "last_deep_scrub_stamp": "2023-10-06 00:53:53.705241", "last_clean_scrub_stamp": "2023-09-28 14:48:47.098935" }, "stats": { "version": "407095'7792", "reported_seq": "376677", "reported_epoch": "407095", "state": "active+undersized+degraded+inconsistent", "last_fresh": "2023-10-03 19:54:53.799024", "last_change": "2023-10-03 19:53:22.457623", "last_active": "2023-10-03 19:54:53.799024", "last_peered": "2023-10-03 19:54:53.799024", "last_clean": "2023-10-03 19:49:47.048960", "last_became_active": "2023-10-03 19:53:22.457623", "last_became_peered": "2023-10-03 19:53:22.457623", "last_unstale": "2023-10-03 19:54:53.799024", "last_undegraded": "2023-10-03 19:53:22.379335", "last_fullsized": "2023-10-03 19:53:22.379134", "mapping_epoch": 407096, "log_start": "378965'4700", "ondisk_log_start": "378965'4700", "created": 19813, "last_epoch_clean": 407093, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "407037'7783", "last_scrub_stamp": "2023-10-03 17:14:13.232978", "last_deep_scrub": "406839'7692", "last_deep_scrub_stamp": "2023-09-28 14:48:47.098935", "last_clean_scrub_stamp": "2023-09-28 14:48:47.098935", "log_size": 3092, "ondisk_log_size": 3092, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 45929876177, "num_objects": 40378, "num_object_clones": 0, "num_object_copies": 444158, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 40378, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 40378, "num_whiteouts": 0, "num_read": 23280, "num_read_kb": 2877429, "num_write": 34414, "num_write_kb": 2409735, "num_scrub_errors": 159632, "num_shallow_scrub_errors": 159632, "num_deep_scrub_errors": 0, "num_objects_recovered": 86740, "num_bytes_recovered": 97127524421, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 42593 }, "up": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "acting": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "avail_no_missing": [ "238(0)", "73(9)", "106(1)", "266(3)", "374(4)", "402(2)", "498(5)", "590(6)", "627(7)", "684(8)" ], "object_location_counts": [ { "shards": "73(9),106(1),238(0),266(3),374(4),402(2),498(5),590(6),627(7),684(8)", "objects": 40378 } ], "blocked_by": [], "up_primary": 238, "acting_primary": 238, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 407097, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "627(7)", "pgid": "15.f4fs7", "last_update": "409009'7998", "last_complete": "409009'7998", "log_tail": "378965'4700", "last_user_version": 592677, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19813, "epoch_pool_created": 16141, "last_epoch_started": 407097, "last_interval_started": 407096, "last_epoch_clean": 407097, "last_interval_clean": 407096, "last_epoch_split": 19849, "last_epoch_marked_full": 0, "same_up_since": 407096, "same_interval_since": 407096, "same_primary_since": 407038, "last_scrub": "408987'7946", "last_scrub_stamp": "2023-10-06 00:53:53.705241", "last_deep_scrub": "408987'7946", "last_deep_scrub_stamp": "2023-10-06 00:53:53.705241", "last_clean_scrub_stamp": "2023-09-28 14:48:47.098935" }, "stats": { "version": "407095'7792", "reported_seq": "376677", "reported_epoch": "407095", "state": "active+undersized+degraded+inconsistent", "last_fresh": "2023-10-03 19:54:53.799024", "last_change": "2023-10-03 19:53:22.457623", "last_active": "2023-10-03 19:54:53.799024", "last_peered": "2023-10-03 19:54:53.799024", "last_clean": "2023-10-03 19:49:47.048960", "last_became_active": "2023-10-03 19:53:22.457623", "last_became_peered": "2023-10-03 19:53:22.457623", "last_unstale": "2023-10-03 19:54:53.799024", "last_undegraded": "2023-10-03 19:53:22.379335", "last_fullsized": "2023-10-03 19:53:22.379134", "mapping_epoch": 407096, "log_start": "378965'4700", "ondisk_log_start": "378965'4700", "created": 19813, "last_epoch_clean": 407093, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "407037'7783", "last_scrub_stamp": "2023-10-03 17:14:13.232978", "last_deep_scrub": "406839'7692", "last_deep_scrub_stamp": "2023-09-28 14:48:47.098935", "last_clean_scrub_stamp": "2023-09-28 14:48:47.098935", "log_size": 3092, "ondisk_log_size": 3092, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 45929876177, "num_objects": 40378, "num_object_clones": 0, "num_object_copies": 444158, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 40378, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 40378, "num_whiteouts": 0, "num_read": 23280, "num_read_kb": 2877429, "num_write": 34414, "num_write_kb": 2409735, "num_scrub_errors": 159632, "num_shallow_scrub_errors": 159632, "num_deep_scrub_errors": 0, "num_objects_recovered": 86740, "num_bytes_recovered": 97127524421, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 42593 }, "up": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "acting": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "avail_no_missing": [ "238(0)", "73(9)", "106(1)", "266(3)", "374(4)", "402(2)", "498(5)", "590(6)", "627(7)", "684(8)" ], "object_location_counts": [ { "shards": "73(9),106(1),238(0),266(3),374(4),402(2),498(5),590(6),627(7),684(8)", "objects": 40378 } ], "blocked_by": [], "up_primary": 238, "acting_primary": 238, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 407097, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "684(8)", "pgid": "15.f4fs8", "last_update": "409009'7998", "last_complete": "409009'7998", "log_tail": "378965'4700", "last_user_version": 592677, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [], "history": { "epoch_created": 19813, "epoch_pool_created": 16141, "last_epoch_started": 407097, "last_interval_started": 407096, "last_epoch_clean": 407097, "last_interval_clean": 407096, "last_epoch_split": 19849, "last_epoch_marked_full": 0, "same_up_since": 407096, "same_interval_since": 407096, "same_primary_since": 407038, "last_scrub": "408987'7946", "last_scrub_stamp": "2023-10-06 00:53:53.705241", "last_deep_scrub": "408987'7946", "last_deep_scrub_stamp": "2023-10-06 00:53:53.705241", "last_clean_scrub_stamp": "2023-09-28 14:48:47.098935" }, "stats": { "version": "407095'7792", "reported_seq": "376677", "reported_epoch": "407095", "state": "active+undersized+degraded+inconsistent", "last_fresh": "2023-10-03 19:54:53.799024", "last_change": "2023-10-03 19:53:22.457623", "last_active": "2023-10-03 19:54:53.799024", "last_peered": "2023-10-03 19:54:53.799024", "last_clean": "2023-10-03 19:49:47.048960", "last_became_active": "2023-10-03 19:53:22.457623", "last_became_peered": "2023-10-03 19:53:22.457623", "last_unstale": "2023-10-03 19:54:53.799024", "last_undegraded": "2023-10-03 19:53:22.379335", "last_fullsized": "2023-10-03 19:53:22.379134", "mapping_epoch": 407096, "log_start": "378965'4700", "ondisk_log_start": "378965'4700", "created": 19813, "last_epoch_clean": 407093, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "407037'7783", "last_scrub_stamp": "2023-10-03 17:14:13.232978", "last_deep_scrub": "406839'7692", "last_deep_scrub_stamp": "2023-09-28 14:48:47.098935", "last_clean_scrub_stamp": "2023-09-28 14:48:47.098935", "log_size": 3092, "ondisk_log_size": 3092, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 45929876177, "num_objects": 40378, "num_object_clones": 0, "num_object_copies": 444158, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 40378, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 40378, "num_whiteouts": 0, "num_read": 23280, "num_read_kb": 2877429, "num_write": 34414, "num_write_kb": 2409735, "num_scrub_errors": 159632, "num_shallow_scrub_errors": 159632, "num_deep_scrub_errors": 0, "num_objects_recovered": 86740, "num_bytes_recovered": 97127524421, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 42593 }, "up": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "acting": [ 238, 106, 402, 266, 374, 498, 590, 627, 684, 73, 66 ], "avail_no_missing": [ "238(0)", "73(9)", "106(1)", "266(3)", "374(4)", "402(2)", "498(5)", "590(6)", "627(7)", "684(8)" ], "object_location_counts": [ { "shards": "73(9),106(1),238(0),266(3),374(4),402(2),498(5),590(6),627(7),684(8)", "objects": 40378 } ], "blocked_by": [], "up_primary": 238, "acting_primary": 238, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 407097, "hit_set_history": { "current_last_update": "0'0", "history": [] } } ], "recovery_state": [ { "name": "Started/Primary/Active", "enter_time": "2023-10-03 19:55:56.164849", "might_have_unfound": [ { "osd": "66(10)", "status": "already probed" }, { "osd": "73(9)", "status": "already probed" }, { "osd": "106(1)", "status": "already probed" }, { "osd": "266(3)", "status": "already probed" }, { "osd": "374(4)", "status": "already probed" }, { "osd": "402(2)", "status": "already probed" }, { "osd": "498(5)", "status": "already probed" }, { "osd": "590(6)", "status": "already probed" }, { "osd": "627(7)", "status": "already probed" }, { "osd": "684(8)", "status": "already probed" } ], "recovery_progress": { "backfill_targets": [], "waiting_on_backfill": [], "last_backfill_started": "MIN", "backfill_info": { "begin": "MIN", "end": "MIN", "objects": [] }, "peer_backfill_info": [], "backfills_in_flight": [], "recovering": [], "pg_backend": { "recovery_ops": [], "read_ops": [] } }, "scrub": { "scrubber.epoch_start": "407096", "scrubber.active": false, "scrubber.state": "INACTIVE", "scrubber.start": "MIN", "scrubber.end": "MIN", "scrubber.max_end": "MIN", "scrubber.subset_last_update": "0'0", "scrubber.deep": false, "scrubber.waiting_on_whom": [] } }, { "name": "Started", "enter_time": "2023-10-03 19:55:55.247621" } ], "agent_state": {} } node1:~ #

3 5

CLT weekly notes October 11th 2023
by Adam King 11 Oct '23

11 Oct '23

Here are the notes from this week's CLT call. The call focused heavily on release process, specifically around figuring out which patches are required for a release. - 17.2.7 status - A few more FS PRs and one core PR then we can start release process - Trying to finalize list of PRs needed for 18.2.1 - General discussion about the process for getting the list of required PRs for a given release - Using per-release github milestones. E.g. a milestone specifically for 18.2.1 rather than just reef - Would require fixing some scripts that refer to the milestone - https://github.com/ceph/ceph/blob/main/src/script/backport-resolve-issue - For now, continue using etherpad until something more automated exists - Create pads a lot earlier - Could use existing clt call to try to finalize required PRs for releases - should be on agenda for every clt call - couple of build related PRs that were stalled. - for a while, it's not possible to build w/FIO - PR https://github.com/ceph/ceph/pull/53346 - for a while, it's not possible to "make (or ninja) install" with dashboard disabled - PR https://github.com/ceph/ceph/pull/52313 - Some more general discussion of how to get more attention for build PRs - Laura will start grouping some build PRs with RADOS PRs for build/testing in the ci - Can make CI builds with CMAKE_BUILD_TYPE=Debug - https://github.com/ceph/ceph-build/pull/2167 - https://github.com/ceph/ceph/pull/53855#issuecomment-1751367302 - https://shaman.ceph.com/builds/ceph/wip-batrick-testing-20231006.014828-deb… - relies on us removing centos 8 from all testing suites and dropping that as a build target - Last Pacific? - Yes, 17.2.7, then 18.2.1, then 16.2.15 (final) - PTLs will need to go through and find what backports still need to get into pacific - A lot of open pacific backports right now Thanks, - Adam King

1 0

Ceph 18: Unable to delete image after imcomplete migration "image being migrated"
by Rhys Goodwin 11 Oct '23

11 Oct '23

Hi Folks, I'm running Ceph 18 with OpenStack for my lab (and home services) in a 3 node cluster on Ubuntu 22.04. I'm quite new to these platforms. Just learning. This is my build, for what it's worth: https://blog.rhysgoodwin.com/it/openstack-ceph-hyperconverged/ I got myself into some trouble as follows. This is the sequence of events: I don't recall when but at some stage I must have tried an image migration from one pool to another. The source pool/image is infra-pool/sophosbuild I don't know what the target would have been. In any case on my travels, I found the infra-pool/sophosbuild image in the trash: rhys@hcn03:/imagework# rbd trash ls --all infra-pool 65a87bb2472fe sophosbuild I tried to delete it but got the following: rhys@hcn03:/imagework# rbd trash rm infra-pool/65a87bb2472fe 2023-10-06T04:23:13.775+0000 7f28bbfff640 -1 librbd::image::RefreshRequest: image being migrated 2023-10-06T04:23:13.775+0000 7f28bbfff640 -1 librbd::image::OpenRequest: failed to refresh image: (30) Read-only file system 2023-10-06T04:23:13.775+0000 7f28bbfff640 -1 librbd::ImageState: 0x7f28a804b600 failed to open image: (30) Read-only file system 2023-10-06T04:23:13.775+0000 7f28a2ffd640 -1 librbd::image::RemoveRequest: 0x7f28a8000b90 handle_open_image: error opening image: (30) Read-only file system rbd: remove error: (30) Read-only file systemRemoving image: 0% complete...failed. Next, I tried to restore the image, and this also failed: rhys@hcn03:/imagework:# rbd trash restore infra-pool/65a87bb2472fe librbd::api::Trash: restore: Current trash source 'migration' does not match expected: user,mirroring,unknown (4) Probably stupidly, I followed the steps in this post: https://www.spinics.net/lists/ceph-users/msg72786.html to change offset 07 from 02 (TRASH_IMAGE_SOURCE_MIGRATION) in omap value to 00(TRASH_IMAGE_SOURCE_USER) After this I was able to restore the image successfully. However, I still could not delete it: rhys@hcn03:/imagework:# rbd rm infra-pool/sophosbuild 2023-10-06T05:52:30.708+0000 7ff5937fe640 -1 librbd::image::RefreshRequest: image being migrated 2023-10-06T05:52:30.708+0000 7ff5937fe640 -1 librbd::image::OpenRequest: failed to refresh image: (30) Read-only file system 2023-10-06T05:52:30.708+0000 7ff5937fe640 -1 librbd::ImageState: 0x564d3f83d680 failed to open image: (30) Read-only file system Removing image: 0% complete...failed.rbd: delete error: (30) Read-only file system I tried to abort the migration with: root@hcn03:/imagework# rbd migration abort infra-pool/sophosbuild This took a few mins but failed at 99% (sorry, terminal scroll back lost) So now I'm stuck, I don't know how to get rid of this image and while everything is otherwise healthy in the cluster, the dashboard is throwing errors when it tries to enumerate the images in that pool. I'm considering migrating the good images off this pool and deleing the pool. But I don't even know if I'll be allowed to delete the pool while this issue is present. Any advice would be much appreciated. Kind regards, Rhys

3 8

cephadm configuration in git
by Kamil Madac 11 Oct '23

11 Oct '23

Hello ceph community, Currently we have deployed ceph clusters with ceph-ansible and whole configuration (number od daemons, osd configurations, rgw configurations, crush configuration, ...) of each cluster is stored in git and ansible variables and we can recreate clusters with ceph-ansible in case we need it. To change the configuration of a cluster we change appropriate Ansible variable, we test it on testing cluster and if new configuration works correctly we apply it on prod cluster. Is it possible to do it with cephadm? Is it possible to have some config files in git and then apply same cluster configuration on multiple clusters? Or is this approach not aligned with cephadm and we should do it different way? Kamil Madac

3 2

What's the best practices of accessing ceph over flaky network connection?
by nanericwang＠gmail.com 11 Oct '23

11 Oct '23

(transferred from https://github.com/ceph/ceph-csi/discussions/4181) > What's the best practices of accessing ceph over flaky network connection? For example, can I setup a local dm-cache binding ceph with a local SSD to buffer the I/O? Thanks. A flaky network will usually be quite problematic. There is no guarantee that data has not been modified on an other system once the network comes back after a temporary interruption. In case the data was changed remotely, and something wrote to the local dm-cache device, a split-brain can happen. Someone needs to decide which side of the data is the right one. Maybe it is possible to tune RBD for working with dm-cache and network interruptions. Or, for CephFS, you may want to look into FS-Cache (I don't know if CephFS supports that though). The best venue to ask about guidance and experience from others is at https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/ . The discussions in this GitHub project are for Ceph-CSI, which is _only_ a driver to provision/mount Ceph based storage.

1 0

ceph-dashboard python warning with new pyo3 0.17 lib (debian12)
by DERUMIER, Alexandre 11 Oct '23

11 Oct '23

Hi, on debian12, ceph-dashboard is throwing a warning "Module 'dashboard' has failed dependency: PyO3 modules may only be initialized once per interpreter process" Seem to be related to pyo3 0.17 change https://github.com/PyO3/pyo3/blob/7bdc504252a2f972ba3490c44249b202a4ce6180/… " Each #[pymodule] can now only be initialized once per process To make PyO3 modules sound in the presence of Python sub-interpreters, for now it has been necessary to explicitly disable the ability to initialize a #[pymodule] more than once in the same process. Attempting to do this will now raise an ImportError. "

3 3

Copying big objects (>5GB) doesn't work after upgrade to Quincy on S3
by Arvydas Opulskis 11 Oct '23

11 Oct '23

Hi all, after upgrading our cluster from Nautilus -> Pacific -> Quincy we noticed we can't copy bigger objects anymore via S3. An error we get: "Aws::S3::Errors::EntityTooLarge (Aws::S3::Errors::EntityTooLarge)" After some tests we have following findings: * Problems starts for objects bigger than 5 GB (multipart limit) * Issue starts after upgrading to Quincy (17.2.6). In latest Pacific (16.2.13) it works fine. * For Quincy it works ok with AWS S3 CLI "cp" command, but doesn't work using AWS Ruby3 SDK client with copy_object command. * For Pacific setup both clients work ok * From RGW logs seems like AWS S3 CLI client handles multipart copying "under the hood", so it is succesful. It is stated in AWS documentation, that for uploads (and copying) bigger than 5GB files we should use multi part API for AWS S3. For some reason it worked for years in Ceph and stopped working after Quincy release, even I couldn't find something in release notes addressing this change. So, is this change permanent and should be considered as bug fix? Both Pacific and Quincy clusters were running on Rocky 8.6 OS, using Beast frontend. Arvydas

2 2

Re: Problem: Upgrading CEPH Pacific to Quincy resulted in CEPH Storage pool to stop functioning.
by Konstantin Shalygin 11 Oct '23

11 Oct '23

Hi, You need revert your packages from Quincy to Pacific. `dnf downgrade ceph-mon` command should help with this k Sent from my iPhone > On Oct 11, 2023, at 03:22, Waywatcher <sconnary32(a)gmail.com> wrote: > > I am unable to get any of the current monitors to run. They all fail to start

1 0

Problem: Upgrading CEPH Pacific to Quincy resulted in CEPH Storage pool to stop functioning.
by Waywatcher 11 Oct '23

11 Oct '23

I upgraded my CEPH cluster without properly following the mon upgrade so they were no longer on leveldb. Proxmox and CEPH were updated to the latest for the current release. https://pve.proxmox.com/wiki/Ceph_Pacific_to_Quincy 1. The upgrade to Quincy states a recommendation that Mons are using RocksDB. 2. Leveldb support has been removed from quincy. The monitors were still running as leveldb. 1. Does this mean the mons cannot work at all since they are levelDB? I upgraded all nodes to the quincy release 17.2.6 and restarted the mons. At this point the cluster stopped responding. `ceph` commands do not work since the service fails to start. Are there steps for recovery? 1) Roll back to Pacific without being able to use CEPH commands (ceph orch upgrade start --ceph-version <version>). 2) Rebuild the monitors using data from the OSDs while maintaining Quincy release. 3) Is this actually related to the bug about 17.2.6 (which is what Proxmox/CEPH upgrades to) https://tracker.ceph.com/issues/58156 ? I ran the upgrade on another cluster prior to this without issue. The Mons were set with RocksDB and running on Quincy 17.2.6. I appreciate any suggestions.

3 2

Hardware recommendations for a Ceph cluster
by Gustavo Fahnle 10 Oct '23

10 Oct '23

Hi, Currently, I have an OpenStack installation with a Ceph cluster consisting of 4 servers for OSD, each with 16TB SATA HDDs. My intention is to add a second, independent Ceph cluster to provide faster disks for OpenStack VMs. The idea for this second cluster is to exclusively provide RBD services to OpenStack. I plan to start with a cluster composed of 3 mon/mgr nodes similar to what we currently have (3 virtualized servers with VMware) with 4 cores, 8GB of memory, 80GB disk and 10GB network each server. In the current cluster, these nodes have low resource consumption, less than 10% CPU usage, 40% memory usage, and less than 100Mb/s of network usage. For the OSDs, I'm thinking of starting with 3 or 4 servers, specifically Supermicro AS-1114S-WN10RT, each with: 1 AMD EPYC 7713P Gen 3 processor (64 Core, 128 Threads, 2.0GHz) 256GB of RAM 2 x NVME 1TB for the operating system 10 x NVME Kingston DC1500M U.2 7.68TB for the OSDs Two Intel NIC E810-XXVDA2 25GbE Dual Port (2 x SFP28) PCIe 4.0 x8 cards Connected to 2 MikroTik CRS518-16XS-2XQ-RM switches at 100GbE per server Connection to OpenStack would be via 4 x 10GB to our core switch. I would like to hear opinions about this configuration, recommendations, criticisms, etc. If any of you have references or experience with any of the components in this initial configuration, they would be very welcome. Thank you very much in advance. Gustavo Fahnle

3 5

Announcing go-ceph v0.24.0
by John Mulligan 10 Oct '23

10 Oct '23

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.24.0 Changes include fixes to the rgw admin and rbd packages. More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. -- John Mulligan phlogistonjohn(a)asynchrono.us jmulligan(a)redhat.com

1 0

Re: Nothing provides libthrift-0.14.0.so()(64bit)
by Graham Derryberry 10 Oct '23

10 Oct '23

I have just started adding a ceph client on a rocky 9 system to our ceph cluster (we're on quincy 17.2.6) and just discovered that epel 9 now provides thrift-0.15.0-2.el9 not thrift-0.14.0-7.el9 as of June 21 2023. So the Nothing provides libthrift-0.14.0.so()(64bit) error has returned! Recommendations?

2 1

outdated mds slow requests
by Ben 10 Oct '23

10 Oct '23

Hi, It is running 17.2.5. there are slow requests warnings in cluster log. ceph tell mds.5 dump_ops_in_flight, get the following. These look like outdated and clients were k8s pods. There are warning of the kind in other mds as well. How could they be cleaned from warnings safely? Many thanks. { "ops": [ { "description": "peer_request(mds.3:5311742.0 authpin)", "initiated_at": "2023-09-14T12:25:43.092201+0000", "age": 926013.05098558997, "duration": 926013.051015759, "type_data": { "flag_point": "dispatched", "reqid": "mds.3:5311742", "op_type": "peer_request", "leader_info": { "leader": "3" }, "request_info": { "attempt": 0, "op_type": "authpin", "lock_type": 0, "object_info": "0x60001205d6d.head", "srcdnpath": "", "destdnpath": "", "witnesses": "", "has_inode_export": false, "inode_export_v": 0, "op_stamp": "0.000000" }, "events": [ { "time": "2023-09-14T12:25:43.092201+0000", "event": "initiated" }, { "time": "2023-09-14T12:25:43.092202+0000", "event": "throttled" }, { "time": "2023-09-14T12:25:43.092201+0000", "event": "header_read" }, { "time": "2023-09-14T12:25:43.092207+0000", "event": "all_read" }, { "time": "2023-09-14T12:25:43.092218+0000", "event": "dispatched" } ] } }, { "description": "peer_request(mds.3:5311743.0 authpin)", "initiated_at": "2023-09-14T12:25:43.092371+0000", "age": 926013.05081614305, "duration": 926013.05089185503, "type_data": { "flag_point": "dispatched", "reqid": "mds.3:5311743", "op_type": "peer_request", "leader_info": { "leader": "3" }, "request_info": { "attempt": 0, "op_type": "authpin", "lock_type": 0, "object_info": "0x60001205d6d.head", "srcdnpath": "", "destdnpath": "", "witnesses": "", "has_inode_export": false, "inode_export_v": 0, "op_stamp": "0.000000" }, "events": [ { "time": "2023-09-14T12:25:43.092371+0000", "event": "initiated" }, { "time": "2023-09-14T12:25:43.092371+0000", "event": "throttled" }, { "time": "2023-09-14T12:25:43.092371+0000", "event": "header_read" }, { "time": "2023-09-14T12:25:43.092374+0000", "event": "all_read" }, { "time": "2023-09-14T12:25:43.092381+0000", "event": "dispatched" } ] } }, { "description": "peer_request(mds.4:4503615.0 authpin)", "initiated_at": "2023-09-14T13:40:25.150040+0000", "age": 921530.99314722, "duration": 921530.99326053297, "type_data": { "flag_point": "dispatched", "reqid": "mds.4:4503615", "op_type": "peer_request", "leader_info": { "leader": "4" }, "request_info": { "attempt": 0, "op_type": "authpin", "lock_type": 0, "object_info": "0x60001205c4f.head", "srcdnpath": "", "destdnpath": "", "witnesses": "", "has_inode_export": false, "inode_export_v": 0, "op_stamp": "0.000000" }, "events": [ { "time": "2023-09-14T13:40:25.150040+0000", "event": "initiated" }, { "time": "2023-09-14T13:40:25.150040+0000", "event": "throttled" }, { "time": "2023-09-14T13:40:25.150040+0000", "event": "header_read" }, { "time": "2023-09-14T13:40:25.150045+0000", "event": "all_read" }, { "time": "2023-09-14T13:40:25.150053+0000", "event": "dispatched" } ] } }, { "description": "client_request(client.460983:5731820 getattr pAsLsXsFs #0x60001205c4f 2023-09-14T13:40:25.144336+0000 caller_uid=0, caller_gid=0{})", "initiated_at": "2023-09-14T13:40:25.150176+0000", "age": 921530.99301089498, "duration": 921530.99316312897, "type_data": { "flag_point": "failed to authpin, inode is being exported", "reqid": "client.460983:5731820", "op_type": "client_request", "client_info": { "client": "client.460983", "tid": 5731820 }, "events": [ { "time": "2023-09-14T13:40:25.150176+0000", "event": "initiated" }, { "time": "2023-09-14T13:40:25.150177+0000", "event": "throttled" }, { "time": "2023-09-14T13:40:25.150176+0000", "event": "header_read" }, { "time": "2023-09-14T13:40:25.150180+0000", "event": "all_read" }, { "time": "2023-09-14T13:40:25.150186+0000", "event": "dispatched" }, { "time": "2023-09-14T13:40:25.150195+0000", "event": "failed to authpin, inode is being exported" } ] } } ], "num_ops": 4 }

2 4

cephadm, cannot use ECDSA key with quincy
by Paul JURCO 10 Oct '23

10 Oct '23

Resent due to moderation when using web interface. Hi ceph users, We have a few clusters with quincy 17.2.6 and we are preparing to migrate from ceph-deploy to cephadm for better management. We are using Ubuntu20 with latest updates (latest openssh). While testing the migration to cephadm on a test cluster with octopus (v16 latest) we had no issues replacing ceph generated cert/key with our own CA signed certs (ECDSA). After upgrading to quincy the test cluster and test again the migration we cannot add hosts due to the errors below, ssh access errors specified a while ago in a tracker. We use the following type of certs: Type: ecdsa-sha2-nistp384-cert-v01(a)openssh.com user certificate The certificate works everytime when using ssh client from shell to connect to all hosts in the cluster. We do a ceph mgr fail every time we replace cert/key so they are restarted. ----- cephadm logs from mgr ------ Oct 06 09:23:27 ceph-m2 bash[1363]: Log: Opening SSH connection to 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connected to SSH server at 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Local address: 10.10.12.160, port 51870 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Peer address: 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Beginning auth for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Auth failed for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connection failure: Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Aborting connection Oct 06 09:23:27 ceph-m2 bash[1363]: Traceback (most recent call last): Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 111, in redirect_log Oct 06 09:23:27 ceph-m2 bash[1363]: yield Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 90, in _remote_connection Oct 06 09:23:27 ceph-m2 bash[1363]: preferred_auth=['publickey'], options=ssh_options) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect Oct 06 09:23:27 ceph-m2 bash[1363]: 'Opening SSH connection to') Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect Oct 06 09:23:27 ceph-m2 bash[1363]: await conn.wait_established() Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established Oct 06 09:23:27 ceph-m2 bash[1363]: await self._waiter Oct 06 09:23:27 ceph-m2 bash[1363]: asyncssh.misc.PermissionDenied: Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: During handling of the above exception, another exception occurred: Oct 06 09:23:27 ceph-m2 bash[1363]: Traceback (most recent call last): Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper Oct 06 09:23:27 ceph-m2 bash[1363]: return OrchResult(f(*args, **kwargs)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2810, in apply Oct 06 09:23:27 ceph-m2 bash[1363]: results.append(self._apply(spec)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2558, in _apply Oct 06 09:23:27 ceph-m2 bash[1363]: return self._add_host(cast(HostSpec, spec)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1434, in _add_host Oct 06 09:23:27 ceph-m2 bash[1363]: ip_addr = self._check_valid_addr(spec.hostname, spec.addr) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1415, in _check_valid_addr Oct 06 09:23:27 ceph-m2 bash[1363]: error_ok=True, no_fsid=True)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 615, in wait_async Oct 06 09:23:27 ceph-m2 bash[1363]: return self.event_loop.get_result(coro) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 56, in get_result Oct 06 09:23:27 ceph-m2 bash[1363]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result Oct 06 09:23:27 ceph-m2 bash[1363]: return self.__get_result() Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result Oct 06 09:23:27 ceph-m2 bash[1363]: raise self._exception Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1361, in _run_cephadm Oct 06 09:23:27 ceph-m2 bash[1363]: await self.mgr.ssh._remote_connection(host, addr) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in _remote_connection Oct 06 09:23:27 ceph-m2 bash[1363]: raise Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ Oct 06 09:23:27 ceph-m2 bash[1363]: self.gen.throw(type, value, traceback) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 123, in redirect_log Oct 06 09:23:27 ceph-m2 bash[1363]: raise HostConnectionError(msg, host, addr) Oct 06 09:23:27 ceph-m2 bash[1363]: cephadm.ssh.HostConnectionError: Failed to connect to ceph-m1 (10.10.10.232). Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: Log: Opening SSH connection to 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connected to SSH server at 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Local address: 10.10.12.160, port 51870 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Peer address: 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Beginning auth for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Auth failed for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connection failure: Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Aborting connection Oct 06 09:23:27 ceph-m2 bash[1363]: debug 2023-10-06T09:23:27.081+0000 7f78d86d8700 -1 log_channel(cephadm) log [ERR] : Failed to connect to ceph-m1 (10.10.10.232). Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: Log: Opening SSH connection to 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connected to SSH server at 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Local address: 10.10.12.160, port 51870 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Peer address: 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Beginning auth for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Auth failed for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connection failure: Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Aborting connection Oct 06 09:23:27 ceph-m2 bash[1363]: Traceback (most recent call last): Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 111, in redirect_log Oct 06 09:23:27 ceph-m2 bash[1363]: yield Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 90, in _remote_connection Oct 06 09:23:27 ceph-m2 bash[1363]: preferred_auth=['publickey'], options=ssh_options) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect Oct 06 09:23:27 ceph-m2 bash[1363]: 'Opening SSH connection to') Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect Oct 06 09:23:27 ceph-m2 bash[1363]: await conn.wait_established() Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established Oct 06 09:23:27 ceph-m2 bash[1363]: await self._waiter Oct 06 09:23:27 ceph-m2 bash[1363]: asyncssh.misc.PermissionDenied: Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: During handling of the above exception, another exception occurred: Oct 06 09:23:27 ceph-m2 bash[1363]: Traceback (most recent call last): Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper Oct 06 09:23:27 ceph-m2 bash[1363]: return OrchResult(f(*args, **kwargs)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2810, in apply Oct 06 09:23:27 ceph-m2 bash[1363]: results.append(self._apply(spec)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2558, in _apply Oct 06 09:23:27 ceph-m2 bash[1363]: return self._add_host(cast(HostSpec, spec)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1434, in _add_host Oct 06 09:23:27 ceph-m2 bash[1363]: ip_addr = self._check_valid_addr(spec.hostname, spec.addr) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1415, in _check_valid_addr Oct 06 09:23:27 ceph-m2 bash[1363]: error_ok=True, no_fsid=True)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 615, in wait_async Oct 06 09:23:27 ceph-m2 bash[1363]: return self.event_loop.get_result(coro) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 56, in get_result Oct 06 09:23:27 ceph-m2 bash[1363]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result Oct 06 09:23:27 ceph-m2 bash[1363]: return self.__get_result() Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result Oct 06 09:23:27 ceph-m2 bash[1363]: raise self._exception Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1361, in _run_cephadm Oct 06 09:23:27 ceph-m2 bash[1363]: await self.mgr.ssh._remote_connection(host, addr) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in _remote_connection Oct 06 09:23:27 ceph-m2 bash[1363]: raise Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ Oct 06 09:23:27 ceph-m2 bash[1363]: self.gen.throw(type, value, traceback) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 123, in redirect_log Oct 06 09:23:27 ceph-m2 bash[1363]: raise HostConnectionError(msg, host, addr) Oct 06 09:23:27 ceph-m2 bash[1363]: cephadm.ssh.HostConnectionError: Failed to connect to ceph-m1 (10.10.10.232). Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: Log: Opening SSH connection to 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connected to SSH server at 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Local address: 10.10.12.160, port 51870 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Peer address: 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Beginning auth for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Auth failed for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connection failure: Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Aborting connection Oct 06 09:23:27 ceph-m2 bash[1363]: debug 2023-10-06T09:23:27.081+0000 7f78d86d8700 -1 mgr handle_command module 'orchestrator' command handler threw exception: __init__() missing 2 required positional arguments: > Oct 06 09:23:27 ceph-m2 bash[1363]: debug 2023-10-06T09:23:27.093+0000 7f78d86d8700 -1 mgr.server reply reply (22) Invalid argument Traceback (most recent call last): Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/mgr_module.py", line 1756, in _handle_command Oct 06 09:23:27 ceph-m2 bash[1363]: return self.handle_command(inbuf, cmd) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 171, in handle_command Oct 06 09:23:27 ceph-m2 bash[1363]: return dispatch[cmd['prefix']].call(self, cmd, inbuf) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call Oct 06 09:23:27 ceph-m2 bash[1363]: return self.func(mgr, **kwargs) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 107, in <lambda> Oct 06 09:23:27 ceph-m2 bash[1363]: wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) # noqa: E731 Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 96, in wrapper Oct 06 09:23:27 ceph-m2 bash[1363]: return func(*args, **kwargs) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/module.py", line 356, in _add_host Oct 06 09:23:27 ceph-m2 bash[1363]: return self._apply_misc([s], False, Format.plain) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/module.py", line 1092, in _apply_misc Oct 06 09:23:27 ceph-m2 bash[1363]: raise_if_exception(completion) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 225, in raise_if_exception Oct 06 09:23:27 ceph-m2 bash[1363]: e = pickle.loads(c.serialized_exception) Oct 06 09:23:27 ceph-m2 bash[1363]: TypeError: __init__() missing 2 required positional arguments: 'hostname' and 'addr' ----- cephadm logs from mgr ------ ----- sshd logs DEBUG3 level ------ Oct 6 09:33:09 ceph-m1 sshd[57168]: debug2: input_userauth_request: try method publickey [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug2: userauth_pubkey: valid user root querying public key ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzO------------ [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: userauth_pubkey: test pkalg ecdsa-sha2-nistp384 pkblob ECDSA SHA256:m6Q0ZQVjjDLWxbmCn0hcGQ2---------- [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_key_allowed entering [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_send entering: type 22 [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_key_allowed: waiting for MONITOR_ANS_KEYALLOWED [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_receive_expect entering: type 23 [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_receive entering [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_receive entering Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: monitor_read: checking request 22 Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_answer_keyallowed entering Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_answer_keyallowed: key_from_blob: 0x5568f0aa7880 Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: temporarily_use_uid: 0/0 (e=0/0) Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: trying public key file /etc/ssh/fake_authorized_keys Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: fd 5 clearing O_NONBLOCK Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: restore_uid: 0/0 Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_answer_keyallowed: publickey authentication test: ECDSA key is not allowed Oct 6 09:33:09 ceph-m1 sshd[57168]: Failed publickey for root from 10.10.12.160 port 40854 ssh2: ECDSA SHA256:m6Q0ZQVjjDLWxbmCn0hcGQ24gbpk------------- Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_send entering: type 23 Oct 6 09:33:09 ceph-m1 sshd[57168]: debug2: userauth_pubkey: authenticated 0 pkalg ecdsa-sha2-nistp384 [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: user_specific_delay: user specific delay 0.000ms [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: ensure_minimum_time_since: elapsed 8.263ms, delaying 8.080ms (requested 8.171ms) [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: userauth_finish: failure partial=0 next methods="publickey" [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: send packet: type 51 [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: Connection closed by authenticating user root 10.10.12.160 port 40854 [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: do_cleanup [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: PAM: sshpam_thread_cleanup entering [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: monitor_read_log: child log fd closed Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_receive entering Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: do_cleanup Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: PAM: cleanup Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: PAM: sshpam_thread_cleanup entering Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: Killing privsep child 57169 Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: audit_event: unhandled event 12 Oct 6 09:33:09 ceph-m1 sshd[757]: debug1: main_sigchld_handler: Child exited --------------- I get "ECDSA key is not allowed" above. From sshd logs, it looks like the client is not sending what is required or in the expected format. Now, what was changed in quincy/mgr on ssh client? Is anyone else using ECDSA keys and it works with quincy? I could not find in PRs something specific to this that could block the access, but it might be. Any suggestion? Thank you! Paul

2 2

[RGW] Is there a way for a user to change is secret key or create other keys ?
by Gilles Mocellin 09 Oct '23

09 Oct '23

Hello Cephers, I was using Ceph with OpenStack, and users could add, remove credentials with `openstack ec2 credentials` commands. But, we are moving our Object Storage service to a new cluster, and didn't want to tie it with OpenStack. Is there a way to have a bit of self service for Rados Gateway, at leas for creating, deleting, changing S3 keys ? It does not seem to be part of S3 APIs. It's certainly doable with Ceph RGW admin API, but with which tool that a standard user can use ? The Ceph Dashboard does not seem a good idea. Roles are global, nothing that can be scoped to a tenant. Some S3 browsers exist (https://github.com/nimbis/s3commander) but never with some management like changing S3 keys. Certainly because it's not in the "standard" S3 API. Perhaps Ceph can provide a client-side dashboard, which can be exposed externally, aside the actual admin dashboard, which will stay inside ? Regards, -- Gilles

2 2

Next quincy point release 17.2.7
by Yuri Weinstein 09 Oct '23

09 Oct '23

Hello We are getting very close to the next Quincy point release 17.2.7 Here is the list of must-have PRs https://pad.ceph.com/p/quincy_17.2.7_prs We will start the release testing/review/approval process as soon as all PRs from this list are merged. If you see something missing please speak up and the dev leads will make a decision on including it in this release. TIA

5 5

Random issues with Reef
by Martin Conway 09 Oct '23

09 Oct '23

Hi I have been using Ceph for many years now, and recently upgraded to Reef. Seems I made the jump too quickly, as I have been hitting a few issues. I can't find any mention of them in the bug reports. I thought I would share them here in case it is something to do with my setup. On V18.2.0 cephadm version Fails with the following output: Traceback (most recent call last): File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/usr/sbin/cephadm/__main__.py", line 10096, in <module> File "/usr/sbin/cephadm/__main__.py", line 10084, in main File "/usr/sbin/cephadm/__main__.py", line 2240, in _infer_image File "/usr/sbin/cephadm/__main__.py", line 2338, in infer_local_ceph_image File "/usr/sbin/cephadm/__main__.py", line 2301, in get_container_info File "/usr/sbin/cephadm/__main__.py", line 2301, in <listcomp> File "/usr/sbin/cephadm/__main__.py", line 222, in __getattr__ AttributeError: 'CephadmContext' object has no attribute 'fsid' I don't know if it is related, but cephadm adopt --style legacy --name osd.X Tries to use a V15 image which then fails to start after being imported. The OSD in question has an SSD device from block.db if that is relevant. Using the latest head version of cephadm from github let me work around this issue, but the adopted OSDs were running 18.0.0-6603-g6c4ed58a and needed to be upgraded to 18.2.0. Also the command: ceph orch upgrade start -ceph_version v18.2.0 Does not work, it fails to find the right image. From memory I think it tried to pull quay.io/ceph/ceph:v18:v18.2.0 Ceph orch upgrade start quay.io/ceph/ceph:v18.2.0 Does work as expected. Let me know if there is any other information that would be helpful, but I have since worked around these issues and have my ceph back in a happy state. Regards, Martin Conway IT and Digital Media Manager Research School of Physics Australian National University Canberra ACT 2601 +61 2 6125 1599 https://physics.anu.edu.au<https://physics.anu.edu.au/>

2 2

Received signal: Hangup from killall
by Rok Jaklič 09 Oct '23

09 Oct '23

Hi, yesterday we changed RGW from civetweb to beast and at 04:02 RGW stopped working; we had to restart it in the morning. In one rgw log for previous day we can see: 2023-10-06T04:02:01.105+0200 7fb71d45d700 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 3202663) UID: 0 and in the next day log we can see: 2023-10-06T04:02:01.133+0200 7fb71d45d700 -1 received signal: Hangup from (PID: 3202664) UID: 0 and after that no requests came. We had to restart rgw. In ceph.conf we have something like [client.radosgw.ctplmon2] host = ctplmon2 log_file = /var/log/ceph/client.radosgw.ctplmon2.log rgw_dns_name = ctplmon2 rgw_frontends = "beast ssl_endpoint=0.0.0.0:4443 ssl_certificate=..." rgw_max_put_param_size = 15728640 We assume it has something to do with logrotate. /etc/logrotate.d/ceph: /var/log/ceph/*.log { rotate 90 daily compress sharedscripts postrotate killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror || pkill -1 -x "ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph-fuse|radosgw|rbd-mirror|cephfs-mirror" || true endscript missingok notifempty su root ceph } ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) And ideas why this happend? Kind regards, Rok

1 1

Ceph 16.2.x excessive logging, how to reduce?
by Zakhar Kirpichenko 09 Oct '23

09 Oct '23

Hi, Our Ceph 16.2.x cluster managed by cephadm is logging a lot of very detailed messages, Ceph logs alone on hosts with monitors and several OSDs has already eaten through 50% of the endurance of the flash system drives over a couple of years. Cluster logging settings are default, and it seems that all daemons are writing lots and lots of debug information to the logs, such as for example: https://pastebin.com/ebZq8KZk (it's just a snippet, but there's lots and lots of various information). Is there a way to reduce the amount of logging and, for example, limit the logging to warnings or important messages so that it doesn't include every successful authentication attempt, compaction etc, etc, when the cluster is healthy and operating normally? I would very much appreciate your advice on this. Best regards, Zakhar

4 12

Manual resharding with multisite
by Yixin Jin 09 Oct '23

09 Oct '23

Hi folks, I am aware that dynamic resharding isn't supported before Reef with multisite. However, does manual resharding work? It doesn't seem to be so, either. First of all, running "bucket reshard" has to be in the master zone. But if the objects of that bucket isn't in the master zone, resharding in the master zone seems to render those objects inaccessible in the zone that actually has them. So, what is recommended practice of resharding with multiste? No resharding at all? Thanks, Yixin

3 2