September 2023 - ceph-users

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

multifs and snapshots
by Dmitry Melekhov 12 Nov '24

12 Nov '24

Hello! There is note here https://docs.ceph.com/en/reef/dev/cephfs-snapshots/ about multiply filesystems and snapshots: If each FS gets its own pool things probably work, but this isn’t tested and may not be true. Is it still untested ? Thank you!

3 4

Help needed to configure erasure coding LRC plugin
by Michel Jouvin 16 Oct '24

16 Oct '24

Hi, As discussed in another thread (Crushmap rule for multi-datacenter erasure coding), I'm trying to create an EC pool spanning 3 datacenters (datacenters are present in the crushmap), with the objective to be resilient to 1 DC down, at least keeping the readonly access to the pool and if possible the read-write access, and have a storage efficiency better than 3 replica (let say a storage overhead <= 2). In the discussion, somebody mentioned LRC plugin as a possible jerasure alternative to implement this without tweaking the crushmap rule to implement the 2-step OSD allocation. I looked at the documentation (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/) but I have some questions if someone has experience/expertise with this LRC plugin. I tried to create a rule for using 5 OSDs per datacenter (15 in total), with 3 (9 in total) being data chunks and others being coding chunks. For this, based of my understanding of examples, I used k=9, m=3, l=4. Is it right? Is this configuration equivalent, in terms of redundancy, to a jerasure configuration with k=9, m=6? The resulting rule, which looks correct to me, is: -------- { "rule_id": 6, "rule_name": "test_lrc_2", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 15, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -4, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 5, "type": "host" }, { "op": "emit" } ] } ------------ Unfortunately, it doesn't work as expected: a pool created with this rule ends up with its pages active+undersize, which is unexpected for me. Looking at 'ceph health detail` output, I see for each page something like: pg 52.14 is stuck undersized for 27m, current state active+undersized, last acting [90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647] For each PG, there is 3 '2147483647' entries and I guess it is the reason of the problem. What are these entries about? Clearly it is not OSD entries... Looks like a negative number, -1, which in terms of crushmap ID is the crushmap root (named "default" in our configuration). Any trivial mistake I would have made? Thanks in advance for any help or for sharing any successful configuration? Best regards, Michel

4 24

Not all Bucket Shards being used
by Christian Kugler 01 Oct '24

01 Oct '24

Hi, I have trouble with large OMAP files in a cluster in the RGW index pool. Some background information about the cluster: There is CephFS and RBD usage on the main cluster but for this issue I think only S3 is interesting. There is one realm, one zonegroup with two zones which have a bidirectional sync set up. Since this does not allow for autoresharding we have to do it by hand in this cluster – looking forward to Reef! From the logs: cluster 2023-07-17T22:59:03.018722+0000 osd.75 (osd.75) 623978 : cluster [WRN] Large omap object found. Object: 34:bcec3016:::.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5:head PG: 34.680c373d (34.5) Key count: 962091 Size (bytes): 277963182 The offending bucket looks like this: # radosgw-admin bucket stats \ | jq '.[] | select(.marker =="3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9") |"\(.num_shards) \(.usage["rgw.main"].num_objects)"' -r 131 9463833 Last week the number of objects was about 12 million. Which is why I reshareded the offending bucket twice, I think. Once to 129 and the second time to 131 because I wanted some leeway (or lieway? scnr, Sage). Unfortunately, even after a week the objects were still to big (the log line above is quite recent), so I looked into it again. # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.0 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.1 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.2 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.3 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.4 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.6 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.7 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.8 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.9 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.10 # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V \ |xargs -IOMAP sh -c \ 'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l' 1013854 1011007 1012287 1011232 1013565 998262 1012777 1012713 1012230 1010690 997111 Apparently, only 11 shards are in use. This would explain why the "Key usage" (from the log line) is about ten times higher than I would expect. How can I deal with this issue? One thing I could try to fix this would be to reshard to a lower number, but I am not sure if there are any risks associated with "downsharding". After that I could reshard to something like 97. Or I could directly "downshard" to 97. Also, the second zone has a similar problem, but as the error messsage lets me know, this would be a bad idea. Will it just take more time until the sharding is transferred to the seconds zone? Best, Christian Kugler

2 5

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

Re: MDS crashes to damaged metadata
by Patrick Donnelly 10 Jun '24

10 Jun '24

On Thu, Dec 15, 2022 at 9:32 AM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): > > 2022-12-08T13:05:48.919+0100 7f440afec700 -1 /build/ceph-16.2.10/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f440afec700 time 2022-12-08T13:05:48.921223+0100 > /build/ceph-16.2.10/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE) > > 12 Minutes lates the unlink_local error crashes appeared again. This time with a new file. During debugging we noticed a MTU mismatch between MDS (1500) and client (9000) with cephfs kernel mount. The client is also creating the snapshots via mkdir in the .snap directory. > > We disabled snapshot creation for now, but really need this feature. I uploaded the mds logs of the first crash along with the information above to https://tracker.ceph.com/issues/38452 > > I would greatly appreciate it, if you could answer me the following question: > > Is the Bug related to our MTU Mismatch? We fixed the MTU Issue going back to 1500 on all nodes in the ceph public network on the weekend also. I doubt it. > If you need a debug level 20 log of the ScatterLock for further analysis, i could schedule snapshots at the end of our workdays and increase the debug level 5 Minutes arround snap shot creation. This would be very helpful! -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 4

16.2.14: [progress WARNING root] complete: ev {UUID} does not exist
by Zakhar Kirpichenko 03 May '24

03 May '24

Hi, Mgr of my cluster logs this every few seconds: [progress WARNING root] complete: ev 7de5bb74-790b-4fda-8838-e4af4af18c62 does not exist [progress WARNING root] complete: ev fff93fce-b630-4141-81ee-19e7a3e61483 does not exist [progress WARNING root] complete: ev a02f6966-5b9f-49e8-89c4-b4fb8e6f4423 does not exist [progress WARNING root] complete: ev 8d318560-ff1a-477f-9386-43f6b51080bf does not exist [progress WARNING root] complete: ev ff3740a9-6434-470a-808f-a2762fb542a0 does not exist [progress WARNING root] complete: ev 7d0589f1-545e-4970-867b-8482ce48d7f0 does not exist [progress WARNING root] complete: ev 78d57e43-5be5-43f0-8b1a-cdc60e410892 does not exist I would appreciate an advice on what these warnings mean and how they can be resolved. Best regards, Zakhar

4 7

Re: Recoveries without any misplaced objects?
by Hector Martin 24 Apr '24

24 Apr '24

On 29/05/2023 20.55, Anthony D'Atri wrote: > Check the uptime for the OSDs in question I restarted all my OSDs within the past 10 days or so. Maybe OSD restarts are somehow breaking these stats? > >> On May 29, 2023, at 6:44 AM, Hector Martin <marcan(a)marcan.st> wrote: >> >> Hi, >> >> I'm watching a cluster finish a bunch of backfilling, and I noticed that >> quite often PGs end up with zero misplaced objects, even though they are >> still backfilling. >> >> Right now the cluster is down to 6 backfilling PGs: >> >> data: >> volumes: 1/1 healthy >> pools: 6 pools, 268 pgs >> objects: 18.79M objects, 29 TiB >> usage: 49 TiB used, 25 TiB / 75 TiB avail >> pgs: 262 active+clean >> 6 active+remapped+backfilling >> >> But there are no misplaced objects, and the misplaced column in `ceph pg >> dump` is zero for all PGs. >> >> If I do a `ceph pg dump_json`, I can see `num_objects_recovered` >> increasing for these PGs... but the misplaced count is still 0. >> >> Is there something else that would cause recoveries/backfills other than >> misplaced objects? Or perhaps there is a bug somewhere causing the >> misplaced object count to be misreported as 0 sometimes? >> >> # ceph -v >> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy >> (stable) >> >> - Hector >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > - Hector

3 3

Status of IPv4 / IPv6 dual stack?
by Robert Sander 23 Apr '24

23 Apr '24

Hi, as the documentation sends mixed signals in https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#ipv… "Note Binding to IPv4 is enabled by default, so if you just add the option to bind to IPv6 you’ll actually put yourself into dual stack mode." and https://docs.ceph.com/en/latest/rados/configuration/msgr2/#address-formats "Note The ability to bind to multiple ports has paved the way for dual-stack IPv4 and IPv6 support. That said, dual-stack operation is not yet supported as of Quincy v17.2.0." just the quick questions: Is a dual stacked networking with IPv4 and IPv6 now supported or not? From which version on is it considered stable? Are OSDs now able to register themselves with two IP addresses in the cluster map? MONs too? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

6 6

A couple OSDs not starting after host reboot
by Alison Peisker 11 Apr '24

11 Apr '24

Hi all, We rebooted all the nodes in our 17.2.5 cluster after performing kernel updates, but 2 of the OSDs on different nodes are not coming back up. This is a production cluster using cephadm. The error message from the OSD log is ceph-osd[87340]: ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-665: (2) No such file or directory The error message from ceph-volume is 2023-08-23T16:12:43.452-0500 7f0cad968600 2 bluestore(/dev/mapper/ceph--febad5a5--ba44--41aa--a39e--b9897f757752-osd--block--87e548f4--b9b5--4ed8--aca8--de703a341a50) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input We tried restarting the daemons and rebooting the node again, but still see the same error. Has anyone experienced this issue before? How do we fix this? Thanks, Alison

9 13

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

Why you might want packages not containers for Ceph deployments
by Matthew Vernon 25 Mar '24

25 Mar '24

Hi, In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0]. I'd like to suggest, briefly, that: * containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes. For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice. Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages): * you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice) We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part. Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems! I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder". I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :) Regards, and apologies for the wall of text, Matthew [0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

26 65

Upgrading from Pacific to Quincy fails with "Unexpected error"
by Reza Bakhshayeshi 13 Mar '24

13 Mar '24

Hi all, I have a problem regarding upgrading Ceph cluster from Pacific to Quincy version with cephadm. I have successfully upgraded the cluster to the latest Pacific (16.2.11). But when I run the following command to upgrade the cluster to 17.2.5, After upgrading 3/4 mgrs, the upgrade process stops with "Unexpected error". (everything is on a private network) ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v17.2.5 I also tried the 17.2.4 version. cephadm fails to check the hosts' status and marks them as offline: cephadm 2023-04-06T10:19:59.998510+0000 mgr.host9.arhpnd (mgr.4516356) 5782 : cephadm [DBG] host host4 (x.x.x.x) failed check cephadm 2023-04-06T10:19:59.998553+0000 mgr.host9.arhpnd (mgr.4516356) 5783 : cephadm [DBG] Host "host4" marked as offline. Skipping daemon refresh cephadm 2023-04-06T10:19:59.998581+0000 mgr.host9.arhpnd (mgr.4516356) 5784 : cephadm [DBG] Host "host4" marked as offline. Skipping gather facts refresh cephadm 2023-04-06T10:19:59.998609+0000 mgr.host9.arhpnd (mgr.4516356) 5785 : cephadm [DBG] Host "host4" marked as offline. Skipping network refresh cephadm 2023-04-06T10:19:59.998633+0000 mgr.host9.arhpnd (mgr.4516356) 5786 : cephadm [DBG] Host "host4" marked as offline. Skipping device refresh cephadm 2023-04-06T10:19:59.998659+0000 mgr.host9.arhpnd (mgr.4516356) 5787 : cephadm [DBG] Host "host4" marked as offline. Skipping osdspec preview refresh cephadm 2023-04-06T10:19:59.998682+0000 mgr.host9.arhpnd (mgr.4516356) 5788 : cephadm [DBG] Host "host4" marked as offline. Skipping autotune cluster 2023-04-06T10:20:00.000151+0000 mon.host8 (mon.0) 158587 : cluster [ERR] Health detail: HEALTH_ERR 9 hosts fail cephadm check; Upgrade: failed due to an unexpected exception cluster 2023-04-06T10:20:00.000191+0000 mon.host8 (mon.0) 158588 : cluster [ERR] [WRN] CEPHADM_HOST_CHECK_FAILED: 9 hosts fail cephadm check cluster 2023-04-06T10:20:00.000202+0000 mon.host8 (mon.0) 158589 : cluster [ERR] host host7 (x.x.x.x) failed check: Unable to reach remote host host7. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000213+0000 mon.host8 (mon.0) 158590 : cluster [ERR] host host2 (x.x.x.x) failed check: Unable to reach remote host host2. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000220+0000 mon.host8 (mon.0) 158591 : cluster [ERR] host host8 (x.x.x.x) failed check: Unable to reach remote host host8. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000228+0000 mon.host8 (mon.0) 158592 : cluster [ERR] host host4 (x.x.x.x) failed check: Unable to reach remote host host4. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000240+0000 mon.host8 (mon.0) 158593 : cluster [ERR] host host3 (x.x.x.x) failed check: Unable to reach remote host host3. Process exited with non-zero exit status 3 and here are some outputs of the commands: [root@host8 ~]# ceph -s cluster: id: xxx health: HEALTH_ERR 9 hosts fail cephadm check Upgrade: failed due to an unexpected exception services: mon: 5 daemons, quorum host8,host1,host7,host2,host9 (age 2w) mgr: host9.arhpnd(active, since 105m), standbys: host8.jowfih, host1.warjsr, host2.qyavjj mds: 1/1 daemons up, 3 standby osd: 37 osds: 37 up (since 8h), 37 in (since 3w) data: io: client: progress: Upgrade to 17.2.5 (0s) [............................] [root@host8 ~]# ceph orch upgrade status { "target_image": "my-private-repo/quay-io/ceph/ceph@sha256 :34c763383e3323c6bb35f3f2229af9f466518d9db926111277f5e27ed543c427", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "3/59 daemons upgraded", "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception", "is_paused": true } [root@host8 ~]# ceph cephadm check-host host7 check-host failed: Host 'host7' not found. Use 'ceph orch host ls' to see all managed hosts. [root@host8 ~]# ceph versions { "mon": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 1, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 }, "osd": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 37 }, "mds": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 4 }, "overall": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 47, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 } } The strange thing is I can rollback the cluster status by failing to not-upgraded mgr like this: ceph mgr fail ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v16.2.11 Would you happen to have any idea about this? Best regards, Reza

4 9

Upgrading nautilus / centos7 to octopus / ubuntu 20.04. - Suggestions and hints?
by Götz Reinicke 17 Jan '24

17 Jan '24

Hi, As I’v read and thought a lot about the migration as this is a bigger project, I was wondering if anyone has done that already and might share some notes or playbooks, because in all readings there where some parts missing or miss understandable to me. I do have some different approaches in mind, so may be you have some suggestions or hints. a) upgrade nautilus on centos 7 with the few missing features like dashboard and prometheus. After that migrate one node after an other to ubuntu 20.04 with octopus and than upgrade ceph to the recent stable version. b) migrate one node after an other to ubuntu 18.04 with nautilus and then upgrade to octupus and after that to ubuntu 20.04. or c) upgrade one node after an other to ubuntu 20.04 with octopus and join it to the cluster until all nodes are upgraded. For test I tried c) with a mon node, but adding that to the cluster fails with some failed state, still probing for the other mons. (I dont have the right log at hand right now.) So my questions are: a) What would be the best (most stable) migration path and b) is it in general possible to add a new octopus mon (not upgraded one) to a nautilus cluster, where the other mons are still on nautilus? I hope my thoughts and questions are understandable :) Thanks for any hint and suggestion. Best . Götz

7 11

Re: increasing number of (deep) scrubs
by Frank Schilder 13 Dec '23

13 Dec '23

Hi Dan, thanks for your answer. I don't have a problem with increasing osd_max_scrubs (=1 at the moment) as such. I would simply prefer a somewhat finer grained way of controlling scrubbing than just doubling or tripling it right away. Some more info. These 2 pools are data pools for a large FS. Unfortunately, we have a large percentage of small files, which is a pain for recovery and seemingly also for deep scrubbing. Our OSDs are about 25% used and I had to increase the warning interval already to 2 weeks. With all the warning grace parameters this means that we manage to deep scrub everything about every month. I need to plan for 75% utilisation and a 3 months period is a bit far on the risky side. Our data is to a large percentage cold data. Client reads will not do the check for us, we need to combat bit-rot pro-actively. The reasons I'm interested in parameters initiating more scrubs while also converting more scrubs into deep scrubs are, that 1) scrubs seem to complete very fast. I almost never catch a PG in state "scrubbing", I usually only see "deep scrubbing". 2) I suspect the low deep-scrub count is due to a low number of deep-scrubs scheduled and not due to conflicting per-OSD deep scrub reservations. With the OSD count we have and the distribution over 12 servers I would expect at least a peak of 50% OSDs being active in scrubbing instead of the 25% peak I'm seeing now. It ought to be possible to schedule more PGs for deep scrub than actually are. 3) Every OSD having only 1 deep scrub active seems to have no measurable impact on user IO. If I could just get more PGs scheduled with 1 deep scrub per OSD it would already help a lot. Once this is working, I can eventually increase osd_max_scrubs when the OSDs fill up. For now I would just like that (deep) scrub scheduling looks a bit harder and schedules more eligible PGs per time unit. If we can get deep scrubbing up to an average of 42PGs completing per hour with keeping osd_max_scrubs=1 to maintain current IO impact, we should be able to complete a deep scrub with 75% full OSDs in about 30 days. This is the current tail-time with 25% utilisation. I believe currently a deep scrub of a PG in these pools takes 2-3 hours. Its just a gut feeling from some repair and deep-scrub commands, I would need to check logs for more precise info. Increasing osd_max_scrubs would then be a further and not the only option to push for more deep scrubbing. My expectation would be that values of 2-3 are fine due to the increasingly higher percentage of cold data for which no interference with client IO will happen. Hope that makes sense and there is a way beyond bumping osd_max_scrubs to increase the number of scheduled and executed deep scrubs. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dvanders(a)gmail.com> Sent: 05 January 2023 15:36 To: Frank Schilder Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] increasing number of (deep) scrubs Hi Frank, What is your current osd_max_scrubs, and why don't you want to increase it? With 8+2, 8+3 pools each scrub is occupying the scrub slot on 10 or 11 OSDs, so at a minimum it could take 3-4x the amount of time to scrub the data than if those were replicated pools. If you want the scrub to complete in time, you need to increase the amount of scrub slots accordingly. On the other hand, IMHO the 1-week deadline for deep scrubs is often much too ambitious for large clusters -- increasing the scrub intervals is one solution, or I find it simpler to increase mon_warn_pg_not_scrubbed_ratio and mon_warn_pg_not_deep_scrubbed_ratio until you find a ratio that works for your cluster. Of course, all of this can impact detection of bit-rot, which anyway can be covered by client reads if most data is accessed periodically. But if the cluster is mostly idle or objects are generally not read, then it would be preferable to increase slots osd_max_scrubs. Cheers, Dan On Tue, Jan 3, 2023 at 2:30 AM Frank Schilder <frans(a)dtu.dk> wrote: > > Hi all, > > we are using 16T and 18T spinning drives as OSDs and I'm observing that they are not scrubbed as often as I would like. It looks like too few scrubs are scheduled for these large OSDs. My estimate is as follows: we have 852 spinning OSDs backing a 8+2 pool with 2024 and an 8+3 pool with 8192 PGs. On average I see something like 10PGs of pool 1 and 12 PGs of pool 2 (deep) scrubbing. This amounts to only 232 out of 852 OSDs scrubbing and seems to be due to a conservative rate of (deep) scrubs being scheduled. The PGs (dep) scrub fairly quickly. > > I would like to increase gently the number of scrubs scheduled for these drives and *not* the number of scrubs per OSD. I'm looking at parameters like: > > osd_scrub_backoff_ratio > osd_deep_scrub_randomize_ratio > > I'm wondering if lowering osd_scrub_backoff_ratio to 0.5 and, maybe, increasing osd_deep_scrub_randomize_ratio to 0.2 would have the desired effect? Are there other parameters to look at that allow gradual changes in the number of scrubs going on? > > Thanks a lot for your help! > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 3

MDS stuck in rejoin
by Frank Schilder 10 Nov '23

10 Nov '23

Hi all, we had a client with the warning "[WRN] MDS_CLIENT_OLDEST_TID: 1 clients failing to advance oldest client/flush tid". I looked at the client and there was nothing going on, so I rebooted it. After the client was back, the message was still there. To clean this up I failed the MDS. Unfortunately, the MDS that took over is remained stuck in rejoin without doing anything. All that happened in the log was: [root@ceph-10 ceph]# tail -f ceph-mds.ceph-10.log 2023-07-20T15:54:29.147+0200 7fedb9c9f700 1 mds.2.896604 rejoin_start 2023-07-20T15:54:29.161+0200 7fedb9c9f700 1 mds.2.896604 rejoin_joint_start 2023-07-20T15:55:28.005+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896614 from mon.4 2023-07-20T15:56:00.278+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896615 from mon.4 [...] 2023-07-20T16:02:54.935+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896653 from mon.4 2023-07-20T16:03:07.276+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896654 from mon.4 After some time I decided to give another fail a try and, this time, the replacement daemon went to active state really fast. If I have a message like the above, what is the clean way of getting the client clean again (version: 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable))? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 16

6.5 CephFS client - ceph_cap_reclaim_work [ceph] / ceph_con_workfn [libceph] hogged CPU
by Stefan Kooman 08 Nov '23

08 Nov '23

Hi, Since the 6.5 kernel addressed the issue with regards to regression in the readahead handling code... we went ahead and installed this kernel for a couple of mail / web clusters (Ubuntu 6.5.1-060501-generic #202309020842 SMP PREEMPT_DYNAMIC Sat Sep 2 08:48:34 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux). Since then we occasionally see the following being logged by the kernel: [Sun Sep 10 07:19:00 2023] workqueue: delayed_work [ceph] hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND [Sun Sep 10 08:41:24 2023] workqueue: ceph_con_workfn [libceph] hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND [Sun Sep 10 11:05:55 2023] workqueue: delayed_work [ceph] hogged CPU for >10000us 8 times, consider switching to WQ_UNBOUND [Sun Sep 10 12:54:38 2023] workqueue: ceph_con_workfn [libceph] hogged CPU for >10000us 8 times, consider switching to WQ_UNBOUND [Sun Sep 10 19:06:37 2023] workqueue: ceph_con_workfn [libceph] hogged CPU for >10000us 16 times, consider switching to WQ_UNBOUND [Mon Sep 11 10:53:33 2023] workqueue: ceph_con_workfn [libceph] hogged CPU for >10000us 32 times, consider switching to WQ_UNBOUND [Tue Sep 12 10:14:03 2023] workqueue: ceph_con_workfn [libceph] hogged CPU for >10000us 64 times, consider switching to WQ_UNBOUND [Tue Sep 12 11:14:33 2023] workqueue: ceph_cap_reclaim_work [ceph] hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND We wonder if this is a new phenomenon, or that it's rather logged in the new kernel and it was not before. However, we have hit a few OOM situations since we switched to the new kernel because of ceph_cap_reclaim_work events (OOM is because Apache threads keep piling up as it cannot access CephFS). We then also see MDS slow ops reported. This might be related to a backup job that is running on a backup server. We did not observe this behavior on 5.12.19 kernel. Ceph cluster is on 16.2.11 currently. Anyone has some insight on this? Thanks, Stefan

3 12

Re: RGW access logs with bucket name
by Boris Behrens 02 Nov '23

02 Nov '23

Bringing up that topic again: is it possible to log the bucket name in the rgw client logs? currently I am only to know the bucket name when someone access the bucket via https://TLD/bucket/object instead of https://bucket.TLD/object. Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens <bb(a)kervyn.de>: > Hi, > I am looking forward to move our logs from > /var/log/ceph/ceph-client...log to our logaggregator. > > Is there a way to have the bucket name in the log file? > > Or can I write the rgw_enable_ops_log into a file? Maybe I could work with > this. > > Cheers and happy new year > Boris > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

4 6

Re: Status of Quincy 17.2.5 ?
by Christian Rohmann 30 Oct '23

30 Oct '23

Hey everyone, On 20/10/2022 10:12, Christian Rohmann wrote: > 1) May I bring up again my remarks about the timing: > > On 19/10/2022 11:46, Christian Rohmann wrote: > >> I believe the upload of a new release to the repo prior to the >> announcement happens quite regularly - it might just be due to the >> technical process of releasing. >> But I agree it would be nice to have a more "bit flip" approach to >> new releases in the repo and not have the packages appear as updates >> prior to the announcement and final release and update notes. > By my observations sometimes there are packages available on the > download servers via the "last stable" folders such as > https://download.ceph.com/debian-quincy/ quite some time before the > announcement of a release is out. > I know it's hard to time this right with mirrors requiring some time > to sync files, but would be nice to not see the packages or have > people install them before there are the release notes and potential > pointers to changes out. Todays 16.2.11 release shows the exact issue I described above .... 1) 16.2.11 packages are already available via e.g. https://download.ceph.com/debian-pacific 2) release notes not yet merged: (https://github.com/ceph/ceph/pull/49839) thus https://ceph.io/en/news/blog/2022/v16-2-11-pacific-released/ show a 404 :-) 3) No announcement like https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/QOCU563UD3… to the ML yet. Regards Christian

2 2

[Pacific] ceph orch device ls do not returns any HDD
by Patrick Begou 27 Oct '23

27 Oct '23

Hi everyone I'm new to CEPH, just a french 4 days training session with Octopus on VMs that convince me to build my first cluster. At this time I have 4 old identical nodes for testing with 3 HDDs each, 2 network interfaces and running Alma Linux8 (el8). I try to replay the training session but it fails, breaking the web interface because of some problems with podman 4.2 not compatible with Octopus. So I try to deploy Pacific with cephadm tool on my first node (mostha1) (to enable testing also an upgrade later). dnf -y install https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noar… monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }') cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \ --initial-dashboard-user admceph \ --allow-fqdn-hostname --cluster-network 10.1.0.0/16 This was sucessfull. But running "*c**eph orch device ls*" do not show any HDD even if I have /dev/sda (used by the OS), /dev/sdb and /dev/sdc The web interface shows a row capacity which is an aggregate of the sizes of the 3 HDDs for the node. I've also tried to reset /dev/sdb but cephadm do not see it: [ceph: root@mostha1 /]# ceph orch device zap mostha1.legi.grenoble-inp.fr /dev/sdb --force Error EINVAL: Device path '/dev/sdb' not found on host 'mostha1.legi.grenoble-inp.fr' On my first attempt with octopus, I was able to list the available HDD with this command line. Before moving to Pacific, the OS on this node has been reinstalled from scratch. Any advices for a CEPH beginner ? Thanks Patrick

9 51

Specify priority for active MGR and MDS
by Nicolas FONTAINE 19 Oct '23

19 Oct '23

Hi everyone, Is there a way to specify which MGR and which MDS should be the active one? Thanks, Nicolas.

3 2

Remove empty orphaned PGs not mapped to a pool
by Malte Stroem 19 Oct '23

19 Oct '23

Hello, we removed an SSD cache tier and its pool. The PGs for the pool do still exist. The cluster is healthy. The PGs are empty and they reside on the cache tier pool's SSDs. We like to take out the disks but it is not possible. The cluster sees the PGs and answers with a HEALTH_WARN. Because of the replication of three there are still 128 PGs on three of the 24 OSDs. We were able to remove the other OSDs. Summary: - pool removed - 3 x 128 empty PGs still exist - 3 of 24 OSDs still exist How is it possible to remove these empty and healthy PGs? The only way I found was something like: ceph pg {pg-id} mark_unfound_lost delete Is that the right way? Some output of: ceph pg ls-by-osd 23 PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP 3.0 0 0 0 0 0 0 0 0 active+clean 27h 0'0 2627265:196316 [15,6,23]p15 [15,6,23]p15 2023-09-28T12:41:52.982955+0200 2023-09-27T06:48:23.265838+0200 3.1 0 0 0 0 0 0 0 0 active+clean 9h 0'0 2627266:19330 [6,23,15]p6 [6,23,15]p6 2023-09-29T06:30:57.630016+0200 2023-09-27T22:58:21.992451+0200 3.2 0 0 0 0 0 0 0 0 active+clean 2h 0'0 2627265:1135185 [23,15,6]p23 [23,15,6]p23 2023-09-29T13:42:07.346658+0200 2023-09-24T14:31:52.844427+0200 3.3 0 0 0 0 0 0 0 0 active+clean 13h 0'0 2627266:193170 [6,15,23]p6 [6,15,23]p6 2023-09-29T01:56:54.517337+0200 2023-09-27T17:47:24.961279+0200 3.4 0 0 0 0 0 0 0 0 active+clean 14h 0'0 2627265:2343551 [23,6,15]p23 [23,6,15]p23 2023-09-29T00:47:47.548860+0200 2023-09-25T09:39:51.259304+0200 3.5 0 0 0 0 0 0 0 0 active+clean 2h 0'0 2627265:194111 [15,6,23]p15 [15,6,23]p15 2023-09-29T13:28:48.879959+0200 2023-09-26T15:35:44.217302+0200 3.6 0 0 0 0 0 0 0 0 active+clean 6h 0'0 2627265:2345717 [23,15,6]p23 [23,15,6]p23 2023-09-29T09:26:02.534825+0200 2023-09-27T21:56:57.500126+0200 Best regards, Malte

4 12

CephFS: convert directory into subvolume
by Eugen Block 12 Oct '23

12 Oct '23

Hi, while writing a response to [1] I tried to convert an existing directory within a single cephfs into a subvolume. According to [2] that should be possible, I'm just wondering how to confirm that it actually worked. Because setting the xattr works fine, the directory just doesn't show up in the subvolume ls command. This is what I tried (in Reef and Pacific): # one "regular" subvolume already exists $ ceph fs subvolume ls cephfs [ { "name": "subvol1" } ] # mounted / and created new subdir $ mkdir /mnt/volumes/subvol2 $ setfattr -n ceph.dir.subvolume -v 1 /mnt/volumes/subvol2 # still only one subvolume $ ceph fs subvolume ls cephfs [ { "name": "subvol1" } ] I also tried it directly underneath /mnt: $ mkdir /mnt/subvol2 $ setfattr -n ceph.dir.subvolume -v 1 /mnt/subvol2 But still no subvolume2 available. What am I missing here? Thanks Eugen [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/G4ZWGGUPPFQ… [2] https://www.spinics.net/lists/ceph-users/msg72341.html

3 7

Clients failing to respond to capability release
by Tim Bishop 12 Oct '23

12 Oct '23

Hi, I've seen this issue mentioned in the past, but with older releases. So I'm wondering if anybody has any pointers. The Ceph cluster is running Pacific 16.2.13 on Ubuntu 20.04. Almost all clients are working fine, with the exception of our backup server. This is using the kernel CephFS client on Ubuntu 22.04 with kernel 6.2.0 [1] (so I suspect a newer Ceph version?). The backup server has multiple (12) CephFS mount points. One of them, the busiest, regularly causes this error on the cluster: HEALTH_WARN 1 clients failing to respond to capability release [WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability release mds.mds-server(mds.0): Client backupserver:cephfs-backupserver failing to respond to capability release client_id: 521306112 And occasionally, which may be unrelated, but occurs at the same time: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests mds.mds-server(mds.0): 1 slow requests are blocked > 30 secs The second one clears itself, but the first sticks until I can unmount the filesystem on the client after the backup completes. It appears that whilst it's in this stuck state there may be one or more directory trees that are inaccessible to all clients. The backup server is walking the whole tree but never gets stuck itself, so either the inaccessible directory entry is caused after it has gone past, or it's not affected. Maybe the backup server is holding a directory when it shouldn't? It may be that an upgrade to Quincy resolves this, since it's more likely to be inline with the kernel client version wise, but I don't want to knee-jerk upgrade just to try and fix this problem. Thanks for any advice. Tim. [1] The reason for the newer kernel is that the backup performance from CephFS was terrible with older kernels. This newer kernel does at least resolve that issue.

4 5

ceph-dashboard python warning with new pyo3 0.17 lib (debian12)
by DERUMIER, Alexandre 11 Oct '23

11 Oct '23

Hi, on debian12, ceph-dashboard is throwing a warning "Module 'dashboard' has failed dependency: PyO3 modules may only be initialized once per interpreter process" Seem to be related to pyo3 0.17 change https://github.com/PyO3/pyo3/blob/7bdc504252a2f972ba3490c44249b202a4ce6180/… " Each #[pymodule] can now only be initialized once per process To make PyO3 modules sound in the presence of Python sub-interpreters, for now it has been necessary to explicitly disable the ability to initialize a #[pymodule] more than once in the same process. Attempting to do this will now raise an ImportError. "

3 3

outdated mds slow requests
by Ben 10 Oct '23

10 Oct '23

Hi, It is running 17.2.5. there are slow requests warnings in cluster log. ceph tell mds.5 dump_ops_in_flight, get the following. These look like outdated and clients were k8s pods. There are warning of the kind in other mds as well. How could they be cleaned from warnings safely? Many thanks. { "ops": [ { "description": "peer_request(mds.3:5311742.0 authpin)", "initiated_at": "2023-09-14T12:25:43.092201+0000", "age": 926013.05098558997, "duration": 926013.051015759, "type_data": { "flag_point": "dispatched", "reqid": "mds.3:5311742", "op_type": "peer_request", "leader_info": { "leader": "3" }, "request_info": { "attempt": 0, "op_type": "authpin", "lock_type": 0, "object_info": "0x60001205d6d.head", "srcdnpath": "", "destdnpath": "", "witnesses": "", "has_inode_export": false, "inode_export_v": 0, "op_stamp": "0.000000" }, "events": [ { "time": "2023-09-14T12:25:43.092201+0000", "event": "initiated" }, { "time": "2023-09-14T12:25:43.092202+0000", "event": "throttled" }, { "time": "2023-09-14T12:25:43.092201+0000", "event": "header_read" }, { "time": "2023-09-14T12:25:43.092207+0000", "event": "all_read" }, { "time": "2023-09-14T12:25:43.092218+0000", "event": "dispatched" } ] } }, { "description": "peer_request(mds.3:5311743.0 authpin)", "initiated_at": "2023-09-14T12:25:43.092371+0000", "age": 926013.05081614305, "duration": 926013.05089185503, "type_data": { "flag_point": "dispatched", "reqid": "mds.3:5311743", "op_type": "peer_request", "leader_info": { "leader": "3" }, "request_info": { "attempt": 0, "op_type": "authpin", "lock_type": 0, "object_info": "0x60001205d6d.head", "srcdnpath": "", "destdnpath": "", "witnesses": "", "has_inode_export": false, "inode_export_v": 0, "op_stamp": "0.000000" }, "events": [ { "time": "2023-09-14T12:25:43.092371+0000", "event": "initiated" }, { "time": "2023-09-14T12:25:43.092371+0000", "event": "throttled" }, { "time": "2023-09-14T12:25:43.092371+0000", "event": "header_read" }, { "time": "2023-09-14T12:25:43.092374+0000", "event": "all_read" }, { "time": "2023-09-14T12:25:43.092381+0000", "event": "dispatched" } ] } }, { "description": "peer_request(mds.4:4503615.0 authpin)", "initiated_at": "2023-09-14T13:40:25.150040+0000", "age": 921530.99314722, "duration": 921530.99326053297, "type_data": { "flag_point": "dispatched", "reqid": "mds.4:4503615", "op_type": "peer_request", "leader_info": { "leader": "4" }, "request_info": { "attempt": 0, "op_type": "authpin", "lock_type": 0, "object_info": "0x60001205c4f.head", "srcdnpath": "", "destdnpath": "", "witnesses": "", "has_inode_export": false, "inode_export_v": 0, "op_stamp": "0.000000" }, "events": [ { "time": "2023-09-14T13:40:25.150040+0000", "event": "initiated" }, { "time": "2023-09-14T13:40:25.150040+0000", "event": "throttled" }, { "time": "2023-09-14T13:40:25.150040+0000", "event": "header_read" }, { "time": "2023-09-14T13:40:25.150045+0000", "event": "all_read" }, { "time": "2023-09-14T13:40:25.150053+0000", "event": "dispatched" } ] } }, { "description": "client_request(client.460983:5731820 getattr pAsLsXsFs #0x60001205c4f 2023-09-14T13:40:25.144336+0000 caller_uid=0, caller_gid=0{})", "initiated_at": "2023-09-14T13:40:25.150176+0000", "age": 921530.99301089498, "duration": 921530.99316312897, "type_data": { "flag_point": "failed to authpin, inode is being exported", "reqid": "client.460983:5731820", "op_type": "client_request", "client_info": { "client": "client.460983", "tid": 5731820 }, "events": [ { "time": "2023-09-14T13:40:25.150176+0000", "event": "initiated" }, { "time": "2023-09-14T13:40:25.150177+0000", "event": "throttled" }, { "time": "2023-09-14T13:40:25.150176+0000", "event": "header_read" }, { "time": "2023-09-14T13:40:25.150180+0000", "event": "all_read" }, { "time": "2023-09-14T13:40:25.150186+0000", "event": "dispatched" }, { "time": "2023-09-14T13:40:25.150195+0000", "event": "failed to authpin, inode is being exported" } ] } } ], "num_ops": 4 }

2 4

Ceph 16.2.x excessive logging, how to reduce?
by Zakhar Kirpichenko 09 Oct '23

09 Oct '23

Hi, Our Ceph 16.2.x cluster managed by cephadm is logging a lot of very detailed messages, Ceph logs alone on hosts with monitors and several OSDs has already eaten through 50% of the endurance of the flash system drives over a couple of years. Cluster logging settings are default, and it seems that all daemons are writing lots and lots of debug information to the logs, such as for example: https://pastebin.com/ebZq8KZk (it's just a snippet, but there's lots and lots of various information). Is there a way to reduce the amount of logging and, for example, limit the logging to warnings or important messages so that it doesn't include every successful authentication attempt, compaction etc, etc, when the cluster is healthy and operating normally? I would very much appreciate your advice on this. Best regards, Zakhar

4 12

CEPH complete cluster failure: unknown PGS
by v1tnam＠gmail.com 05 Oct '23

05 Oct '23

I have an 8-node cluster with old hardware. a week ago 4 nodes went down and the CEPH cluster went nuts. All pgs became unknown and montors took too long to be in sync. So i reduced the number of mons to one and mgrs to one as well Now the recovery starts with 100% unknown pgs and then pgs start to move ot inactive . It generally fails to recover in the middle and starts from scratch. It's hold hardware and OSDs have lots of slow ops and probably number of bad sectors as well Any suggestions on how to tackle this. It's a nautilus cluster and pretty old (8-year old hardware) Thanks

3 2

RGW multisite - requesting help for fixing error_code: 125
by Jayanth Reddy 04 Oct '23

04 Oct '23

Hello Users, We're running 2 Ceph clusters with v17.2.6 and noticing the error message in # radosgw-admin sync error list *"message": "failed to sync bucket instance: (125) Operation canceled"* We've the output as below, [ { "shard_id": 0, "entries": [ { "id": "1_1690711173.869335_133603.1", "section": "data", "name": "b1:d09d3d16-8601-448b-bf3d-609b8a29647d.38987.1:2897", "timestamp": "2023-07-30T09:59:33.869335Z", "info": { "source_zone": "d09d3d16-8601-448b-bf3d-609b8a29647d", "error_code": 125, "message": "failed to sync bucket instance: (125) Operation canceled" } }, { "id": "1_1690711175.505687_133683.1", "section": "data", "name": "b1:d09d3d16-8601-448b-bf3d-609b8a29647d.38987.1:1719", "timestamp": "2023-07-30T09:59:35.505687Z", "info": { "source_zone": "d09d3d16-8601-448b-bf3d-609b8a29647d", "error_code": 125, "message": "failed to sync bucket instance: (125) Operation canceled" } }, and with around 26236 errors # radosgw-admin sync error list | grep -i "(125) Operation canceled" | wc -l 26236 I'm trying to fix these by rewriting objects but I'm having trouble finding the exact object names and the procedure. Any help is really appreciated! Thanks, Jayanth

2 1

Balancer blocked as autoscaler not acting on scaling change
by bc10＠sanger.ac.uk 04 Oct '23

04 Oct '23

Hi Folks, We are currently running with one nearfull OSD and 15 nearfull pools. The most full OSD is about 86% full but the average is 58% full. However, the balancer is skipping a pool on which the autoscaler is trying to complete a pg_num reduction from 131,072 to 32,768 (default.rgw.buckets.data pool). However, the autoscaler has been working on this for the last 20 days, it works through a list of objects that are misplaced but when it gets close to the end, more objects get added to the list. This morning I observed the list get down to c. 7,000 objects misplaced with 2 PGs active+remapped+backfilling, one PG completed the backfilling then the list shot up to c. 70,000 objects misplaced with 3 PGs active+remapped+backfilling. Has anyone come across this behaviour before? If so, what was your remediation? Thanks in advance for sharing. Bruno Cluster details: 3,068 OSDs when all running, c. 60 per storage node OS: Ubuntu 20.04 Ceph: Pacific 16.2.13 from Ubuntu Cloud Archive Use case: S3 storage and OpenStack backend, all pools three-way replicated

5 4

Re: VM hangs when overwriting a file on erasure coded RBD
by peter.linder＠fiberdirekt.se 04 Oct '23

04 Oct '23

Yes, this is all set up. It was working fine until after the problem with the osd host that lost the cluster/sync network occured. There are a few other VMs that keep running along fine without this issue. I've restarted the problematic VM without success (that is, creating a file works, but overwriting it still hangs right away). fsck runs fine so reading the whole image works. I'm kind of stumped as to what can cause this. Because of the lengthy recovery, and then pg autoscaler currenty doing things there are currently lots of PGs that haven't been scrubbed, but I doubt that is an issue here. Den 2023-09-29 kl. 18:52, skrev Anthony D'Atri: > EC for RBD wasn't possible until Luminous IIRC, so I had to ask. You have a replicated metadata pool defined? Does proxmox know that this is an EC pool? When connecting it needs to know both the metata and data pools. > >> On Sep 29, 2023, at 12:49, peter.linder(a)fiberdirekt.se wrote: >> >> (sorry for duplicate emails) >> >> This turns out to be a good question actually. >> >> The cluster is running Quincy, 17.2.6. >> >> The compute node that is running the VM is proxmox, version 7.4-3. Supposedly this is fairly new, but the version of librbd1 claims to be 14.2.21 when I check with "apt list". We are not using proxmox's own ceph cluster release. However we haven't had any issues with this setup before, but we haven't been using neither erasure coded pools nor had the node-half-dead problem for such a long time. >> >> The VM is configured using proxmox which is not libvirt but similar, and krbd is not enabled. I don't know for sure if proxmox has its own librbd linked in qemu/kvm. >> >> "ceph features" looks like this: >> >> { >> "mon": [ >> { >> "features": "0x3f01cfbf7ffdffff", >> "release": "luminous", >> "num": 5 >> } >> ], >> "osd": [ >> { >> "features": "0x3f01cfbf7ffdffff", >> "release": "luminous", >> "num": 24 >> } >> ], >> "client": [ >> { >> "features": "0x3f01cfb87fecffff", >> "release": "luminous", >> "num": 4 >> }, >> { >> "features": "0x3f01cfbf7ffdffff", >> "release": "luminous", >> "num": 12 >> } >> ], >> "mgr": [ >> { >> "features": "0x3f01cfbf7ffdffff", >> "release": "luminous", >> "num": 2 >> } >> ] >> } >> >> Regards, >> >> Peter >> >> >> Den 2023-09-29 kl. 17:55, skrev Anthony D'Atri: >>> Which Ceph releases are installed on the VM and the back end? Is the VM using librbd through libvirt, or krbd? >>> >>>> On Sep 29, 2023, at 09:09, Peter Linder <peter.linder(a)fiberdirekt.se> wrote: >>>> >>>> Dear all, >>>> >>>> I have a problem that after an OSD host lost connection to the sync/cluster rear network for many hours (the public network was online), a test VM using RBD cant overwrite its files. I can create a new file inside it just fine, but not overwrite it, the process just hangs. >>>> >>>> The VM's disk is on an erasure coded data pool and a replicated pool in front of it. EC overwrites is on for the pool. >>>> >>>> The custer consists of 5 hosts and 4 OSDs on each, and separate hosts for compute. There is a public and separate cluster network, separated. In this case, the AOC cable to the cluster network went link down on a host and it had to be replaced and the host was rebooted. Recovery took about a week to complete. The host was half-down for about 12 hours like this. >>>> >>>> I have some other VMs as well with images in the same pool (totally 4), and they seem to work fine, it is just this one that cant overwrite. >>>> >>>> I'm thinking there is somehow something wrong with just this image? >>>> >>>> Regards, >>>> >>>> Peter >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

cephfs health warn
by Ben 04 Oct '23

04 Oct '23

Hi, see below for details of warnings. the cluster is running 17.2.5. the warnings have been around for a while. one concern of mine is num_segments growing over time. clients with warn of MDS_CLIENT_OLDEST_TID increase from 18 to 25 as well. The nodes are with kernel 4.19.0-91.82.42.uelc20.x86_64. It looks like bugs with client library. And rebooting nodes with problem will fix it for short period of time? Any suggestions from community for fixing? Thanks, Ben [root@8cd2c0657c77 /]# ceph health detail HEALTH_WARN 6 hosts fail cephadm check; 2 clients failing to respond to capability release; 25 clients failing to advance oldest client/flush tid; 3 MDSs report slow requests; 3 MDSs behind on trimming [WRN] CEPHADM_HOST_CHECK_FAILED: 6 hosts fail cephadm check host host15w (192.168.31.33) failed check: Unable to reach remote host host15w. Process exited with non-zero exit status 1 host host20w (192.168.31.38) failed check: Unable to reach remote host host20w. Process exited with non-zero exit status 1 host host19w (192.168.31.37) failed check: Unable to reach remote host host19w. Process exited with non-zero exit status 1 host host17w (192.168.31.35) failed check: Unable to reach remote host host17w. Process exited with non-zero exit status 1 host host18w (192.168.31.36) failed check: Unable to reach remote host host18w. Process exited with non-zero exit status 1 host host16w (192.168.31.34) failed check: Unable to reach remote host host16w. Process exited with non-zero exit status 1 [WRN] MDS_CLIENT_LATE_RELEASE: 2 clients failing to respond to capability release mds.code-store.host18w.fdsqff(mds.1): Client k8s-node36 failing to respond to capability release client_id: 460983 mds.code-store.host16w.vucirx(mds.3): Client failing to respond to capability release client_id: 460983 [WRN] MDS_CLIENT_OLDEST_TID: 25 clients failing to advance oldest client/flush tid mds.code-store.host18w.fdsqff(mds.1): Client k8s-node36 failing to advance its oldest client/flush tid. client_id: 460983 mds.code-store.host18w.fdsqff(mds.1): Client failing to advance its oldest client/flush tid. client_id: 460226 mds.code-store.host18w.fdsqff(mds.1): Client k8s-node32 failing to advance its oldest client/flush tid. client_id: 239797 mds.code-store.host15w.reolpx(mds.5): Client k8s-node34 failing to advance its oldest client/flush tid. client_id: 460226 mds.code-store.host15w.reolpx(mds.5): Client k8s-node32 failing to advance its oldest client/flush tid. client_id: 239797 mds.code-store.host15w.reolpx(mds.5): Client failing to advance its oldest client/flush tid. client_id: 460983 mds.code-store.host18w.rtyvdy(mds.7): Client k8s-node34 failing to advance its oldest client/flush tid. client_id: 460226 mds.code-store.host18w.rtyvdy(mds.7): Client failing to advance its oldest client/flush tid. client_id: 239797 mds.code-store.host18w.rtyvdy(mds.7): Client k8s-node36 failing to advance its oldest client/flush tid. client_id: 460983 mds.code-store.host17w.kcdopb(mds.2): Client failing to advance its oldest client/flush tid. client_id: 239797 mds.code-store.host17w.kcdopb(mds.2): Client failing to advance its oldest client/flush tid. client_id: 460983 mds.code-store.host17w.kcdopb(mds.2): Client k8s-node34 failing to advance its oldest client/flush tid. client_id: 460226 mds.code-store.host17w.kcdopb(mds.2): Client k8s-node24 failing to advance its oldest client/flush tid. client_id: 12072730 mds.code-store.host20w.bfoftp(mds.4): Client k8s-node32 failing to advance its oldest client/flush tid. client_id: 239797 mds.code-store.host20w.bfoftp(mds.4): Client k8s-node36 failing to advance its oldest client/flush tid. client_id: 460983 mds.code-store.host19w.ywrmiz(mds.6): Client k8s-node24 failing to advance its oldest client/flush tid. client_id: 12072730 mds.code-store.host19w.ywrmiz(mds.6): Client k8s-node34 failing to advance its oldest client/flush tid. client_id: 460226 mds.code-store.host19w.ywrmiz(mds.6): Client failing to advance its oldest client/flush tid. client_id: 239797 mds.code-store.host19w.ywrmiz(mds.6): Client failing to advance its oldest client/flush tid. client_id: 460983 mds.code-store.host16w.vucirx(mds.3): Client failing to advance its oldest client/flush tid. client_id: 460983 mds.code-store.host16w.vucirx(mds.3): Client failing to advance its oldest client/flush tid. client_id: 460226 mds.code-store.host16w.vucirx(mds.3): Client failing to advance its oldest client/flush tid. client_id: 239797 mds.code-store.host17w.pdziet(mds.0): Client k8s-node32 failing to advance its oldest client/flush tid. client_id: 239797 mds.code-store.host17w.pdziet(mds.0): Client k8s-node34 failing to advance its oldest client/flush tid. client_id: 460226 mds.code-store.host17w.pdziet(mds.0): Client k8s-node36 failing to advance its oldest client/flush tid. client_id: 460983 [WRN] MDS_SLOW_REQUEST: 3 MDSs report slow requests mds.code-store.host15w.reolpx(mds.5): 4 slow requests are blocked > 5 secs mds.code-store.host20w.bfoftp(mds.4): 6 slow requests are blocked > 5 secs mds.code-store.host16w.vucirx(mds.3): 97 slow requests are blocked > 5 secs [WRN] MDS_TRIM: 3 MDSs behind on trimming mds.code-store.host15w.reolpx(mds.5): Behind on trimming (25831/128) max_segments: 128, num_segments: 25831 mds.code-store.host20w.bfoftp(mds.4): Behind on trimming (27605/128) max_segments: 128, num_segments: 27605 mds.code-store.host16w.vucirx(mds.3): Behind on trimming (28676/128) max_segments: 128, num_segments: 28676

2 8

radosgw-admin sync error trim seems to do nothing
by Richard Bade 03 Oct '23

03 Oct '23

Hi Matthew, At least for nautilus (14.2.22) i have discovered through trial and error that you need to specify a beginning or end date. Something like this: radosgw-admin sync error trim --end-date="2023-08-20 23:00:00" --rgw-zone={your_zone_name} I specify the zone as there's a error list for each zone. Hopefully that helps. Rich ------------------------------ Date: Sat, 19 Aug 2023 12:48:55 -0400 From: Matthew Darwin <bugs(a)mdarwin.ca> Subject: [ceph-users] radosgw-admin sync error trim seems to do nothing To: Ceph Users <ceph-users(a)ceph.io> Message-ID: <95e7edfd-ca29-fc0e-a30a-987f1c43e2d4(a)mdarwin.ca> Content-Type: text/plain; charset=UTF-8; format=flowed Hello all, "radosgw-admin sync error list" returns errors from 2022. I want to clear those out. I tried "radosgw-admin sync error trim" but it seems to do nothing. The man page seems to offer no suggestions https://protect-au.mimecast.com/s/26o0CzvkGRhLoOXfXjZR3?domain=docs.ceph.com Any ideas what I need to do to remove old errors? (or at least I want to see more recent errors) ceph version 17.2.6 (quincy) Thanks.

2 2

Impacts on doubling the size of pgs in a rbd pool?
by Hervé Ballans 03 Oct '23

03 Oct '23

Hi all, I have a Ceph cluster on Quincy (17.2.6), with 3 pools (1 rbd and 1 CephFS volume), each configured with 3 replicas. $ sudo ceph osd pool ls detail pool 7 'cephfs_data_home' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode on last_change 6287147 lfor 0/5364613/5364611 flags hashpspool stripe_width 0 application cephfs pool 8 'cephfs_metadata_home' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 6333341 lfor 0/6333341/6333339 flags hashpspool stripe_width 0 application cephfs pool 9 'rbd_backup_vms' replicated size 3 min_size 2 crush_rule 2 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 6365131 lfor 0/211948/249421 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd pool 10 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode warn last_change 6365131 flags hashpspool stripe_width 0 pg_num_min 1 application mgr,mgr_devicehealth $ sudo ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 306 TiB 186 TiB 119 TiB 119 TiB 39.00 nvme 4.4 TiB 4.3 TiB 118 GiB 118 GiB 2.63 TOTAL 310 TiB 191 TiB 119 TiB 119 TiB 38.49 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL cephfs_data_home 7 512 12 TiB 28.86M 12 TiB 12.85 27 TiB cephfs_metadata_home 8 32 33 GiB 3.63M 33 GiB 0.79 1.3 TiB rbd_backup_vms 9 1024 24 TiB 6.42M 24 TiB 58.65 5.6 TiB .mgr 10 1 35 MiB 9 35 MiB 0 12 TiB I am going to extend the rbd pool (rbd_backup_vms), currently used at 60%. This pool contains 60 disks, i.e. 20 disks by rack in the crushmap. This pool is used for storing VM disk images (available to a separate ProxmoxVE cluster) For this purpose, I am going to add 42 disks of the same size as those currently in the pool, i.e. 14 additional disks on each rack. Currently, this pool is configured with 1024 pgs. Before this operation, I would like to extend the number of pgs, let's say 2048 (i.e. double). I wonder about the overall impact of this change on the cluster. I guess that the heavy moves in the pgs will have a strong impact regarding the iops? I have two questions: 1) Is it useful to make this modification before adding the new OSDs? (I'm afraid of warnings about full or nearfull pgs if not) 2) are there any configuration recommendations in order to minimize these anticipated impacts? Thank you! Cheers, Hervé

3 3

SSD SATA performance
by Murilo Morais 30 Sep '23

30 Sep '23

Good morning everybody! Guys, I have 9x Kingston DC600M/1920 SSDs (SATA) in 3x DL380e, using the P420 (I still don't have an HBA to perform the exchange) in RAID 0. The device's specifications indicate that it achieves 94k/78k RAND-RW IOPS at 4K. I'm using it exclusively for VMs with RBD (I'm using OpenStack). (pool size 3) Performing sequential testing directly on the device I can easily beat these rates, but with random tests I reach fixed rates of 15k, and I think the device just doesn't deliver that. On VMs I get rates of 20k randwrite. What I've already tried: Disable Controller Cache Enable HP Smart Path (it ended up worsening SSD performance) Change the scheduler to "none" I haven't tried putting the controller in HBA mode yet, as the boot disks are on the P420, I still have to align this issue to perform the test. I would like to know if I can improve these rates or if it is just that. Thanks in advance!

1 0

Re: VM hangs when overwriting a file on erasure coded RBD
by peter.linder＠fiberdirekt.se 29 Sep '23

29 Sep '23

(sorry for duplicate emails) This turns out to be a good question actually. The cluster is running Quincy, 17.2.6. The compute node that is running the VM is proxmox, version 7.4-3. Supposedly this is fairly new, but the version of librbd1 claims to be 14.2.21 when I check with "apt list". We are not using proxmox's own ceph cluster release. However we haven't had any issues with this setup before, but we haven't been using neither erasure coded pools nor had the node-half-dead problem for such a long time. The VM is configured using proxmox which is not libvirt but similar, and krbd is not enabled. I don't know for sure if proxmox has its own librbd linked in qemu/kvm. "ceph features" looks like this: { "mon": [ { "features": "0x3f01cfbf7ffdffff", "release": "luminous", "num": 5 } ], "osd": [ { "features": "0x3f01cfbf7ffdffff", "release": "luminous", "num": 24 } ], "client": [ { "features": "0x3f01cfb87fecffff", "release": "luminous", "num": 4 }, { "features": "0x3f01cfbf7ffdffff", "release": "luminous", "num": 12 } ], "mgr": [ { "features": "0x3f01cfbf7ffdffff", "release": "luminous", "num": 2 } ] } Regards, Peter Den 2023-09-29 kl. 17:55, skrev Anthony D'Atri: > Which Ceph releases are installed on the VM and the back end? Is the VM using librbd through libvirt, or krbd? > >> On Sep 29, 2023, at 09:09, Peter Linder <peter.linder(a)fiberdirekt.se> wrote: >> >> Dear all, >> >> I have a problem that after an OSD host lost connection to the sync/cluster rear network for many hours (the public network was online), a test VM using RBD cant overwrite its files. I can create a new file inside it just fine, but not overwrite it, the process just hangs. >> >> The VM's disk is on an erasure coded data pool and a replicated pool in front of it. EC overwrites is on for the pool. >> >> The custer consists of 5 hosts and 4 OSDs on each, and separate hosts for compute. There is a public and separate cluster network, separated. In this case, the AOC cable to the cluster network went link down on a host and it had to be replaced and the host was rebooted. Recovery took about a week to complete. The host was half-down for about 12 hours like this. >> >> I have some other VMs as well with images in the same pool (totally 4), and they seem to work fine, it is just this one that cant overwrite. >> >> I'm thinking there is somehow something wrong with just this image? >> >> Regards, >> >> Peter >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Join us for the User + Dev Relaunch, happening this Thursday!
by Laura Flores 29 Sep '23

29 Sep '23

Hi Ceph users and developers, We invite you to join us at the User + Dev Relaunch, happening this Thursday at 10:00 AM EST! See below for more meeting details. Also see this blog post to read more about the relaunch: https://ceph.io/en/news/blog/2023/user-dev-meeting-relaunch/ We have two guest speakers who will present their focus topics during the first 40 minutes of the session: 1. "What to do when Ceph isn't Ceph-ing" by Cory Snyder Topics include troubleshooting tips, effective ways to gather help from the community, ways to improve cluster health and insights, and more! 2. "Ceph Usability Improvements" by Jonas Sterr A continuation of a talk from Cephalocon 2023, updated after trying out the Reef Dashboard. The last 20 minutes of the meeting will be dedicated to open discussion. Feel free to add questions for the speakers or additional topics under the "Open Discussion" section on the agenda: https://pad.ceph.com/p/ceph-user-dev-monthly-minutes If you have an idea for a focus topic you'd like to present at a future meeting, you are welcome to submit it to this Google Form: https://docs.google.com/forms/d/e/1FAIpQLSdboBhxVoBZoaHm8xSmeBoemuXoV_rmh4v… Any Ceph user or developer is eligible to submit! Thanks, Laura Flores Meeting link: https://meet.jit.si/ceph-user-dev-monthly Time conversions: UTC: Thursday, September 21, 14:00 UTC Mountain View, CA, US: Thursday, September 21, 7:00 PDT Phoenix, AZ, US: Thursday, September 21, 7:00 MST Denver, CO, US: Thursday, September 21, 8:00 MDT Huntsville, AL, US: Thursday, September 21, 9:00 CDT Raleigh, NC, US: Thursday, September 21, 10:00 EDT London, England: Thursday, September 21, 15:00 BST Paris, France: Thursday, September 21, 16:00 CEST Helsinki, Finland: Thursday, September 21, 17:00 EEST Tel Aviv, Israel: Thursday, September 21, 17:00 IDT Pune, India: Thursday, September 21, 19:30 IST Brisbane, Australia: Friday, September 22, 0:00 AEST Singapore, Asia: Thursday, September 21, 22:00 +08 Auckland, New Zealand: Friday, September 22, 2:00 NZST -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

3 4

VM hangs when overwriting a file on erasure coded RBD
by Peter Linder 29 Sep '23

29 Sep '23

Dear all, I have a problem that after an OSD host lost connection to the sync/cluster rear network for many hours (the public network was online), a test VM using RBD cant overwrite its files. I can create a new file inside it just fine, but not overwrite it, the process just hangs. The VM's disk is on an erasure coded data pool and a replicated pool in front of it. EC overwrites is on for the pool. The custer consists of 5 hosts and 4 OSDs on each, and separate hosts for compute. There is a public and separate cluster network, separated. In this case, the AOC cable to the cluster network went link down on a host and it had to be replaced and the host was rebooted. Recovery took about a week to complete. The host was half-down for about 12 hours like this. I have some other VMs as well with images in the same pool (totally 4), and they seem to work fine, it is just this one that cant overwrite. I'm thinking there is somehow something wrong with just this image? Regards, Peter

2 1

After upgrading from 17.2.6 to 18.2.0, OSDs are very frequently restarting due to livenessprobe failures
by sbengeri＠gmail.com 28 Sep '23

28 Sep '23

Since upgrading to 18.2.0 , OSDs are very frequently restarting due to livenessprobe failures making the cluster unusable. Has anyone else seen this behavior? Upgrade path: ceph 17.2.6 to 18.2.0 (and rook from 1.11.9 to 1.12.1) on ubuntu 20.04 kernel 5.15.0-79-generic Thanks.

6 9

Snap_schedule does not always work.
by Kushagr Gupta 28 Sep '23

28 Sep '23

Hi Teams, *Ceph-version*: Quincy, Reef *OS*: Almalinux 8 *Issue*: snap_schedule doesn't create the scheduled snapshots consistently. *Description:* Hi team, We are currently working in a 3-node ceph cluster. We are currently exploring the scheduled snapshot capability of the ceph-mgr module. To enable/configure scheduled snapshots, we followed the following link: https://docs.ceph.com/en/quincy/cephfs/snap-schedule/ Using this we were able to schedule a snapshot for the subvolume which we created. Initially, it was not working but after two days it started working. Later on a fresh cluster we again tried to schedule the snapshot for a subvolume and it worked fine. But next time when we again tried the snapshot creation on a fresh cluster it didn't work this time. We have observed this inconsistency in scheduling the snapshots for a long time now. Everytime we follow the exact same steps to add a snap_schedule but sometimes it works and sometimes it doesn't. We have a chronyd service running and the timezone set to UTC timezone. Could you please help us out with this? Kindly let me know if you require any kind of logs for this Thanks and Regards, Kushagra Gupta

2 3

Re: Not able to find a standardized restoration procedure for subvolume snapshots.
by Kushagr Gupta 27 Sep '23

27 Sep '23

Hi Team, Any update on this? Thanks and Regards, Kushagra Gupta On Tue, Sep 5, 2023 at 10:51 AM Kushagr Gupta <kushagrguptasps.mun(a)gmail.com> wrote: > *Ceph-version*: Quincy > *OS*: Centos 8 stream > > *Issue*: Not able to find a standardized restoration procedure for > subvolume snapshots. > > *Description:* > Hi team, > > We are currently working in a 3-node ceph cluster. > We are currently exploring the scheduled snapshot capability of the > ceph-mgr module. > To enable/configure scheduled snapshots, we followed the following link: > > https://docs.ceph.com/en/quincy/cephfs/snap-schedule/ > > The scheduled snapshots are working as expected. But we are unable to find > any standardized restoration procedure for the same. > > We have found the following link( not official documentation): > https://www.suse.com/support/kb/doc/?id=000019627 > > We have also found a link of cloning a new subvolume from snapshots: > https://docs.ceph.com/en/reef/cephfs/fs-volumes/ > (Section: Cloning Snapshots) > > Is there a standard procedure to restore from a snapshot. > By this I mean, is there some kind of command link maybe > ceph fs subvolume snapshot restore <snapshot-name> > > Or any other procedure please let us know. > > Thanks and Regards, > Kushagra Gupta >

2 2

Quincy NFS ingress failover
by Thorne Lawler 27 Sep '23

27 Sep '23

Fellow cephalopods, I'm trying to get quick, seamless NFS failover happening on my four-node Ceph cluster. I followed the instructions here: https://docs.ceph.com/en/latest/cephadm/services/nfs/#high-availability-nfs but testing shows that failover doesn't happen. When I placed node 2 ("san2") in maintenance mode, the NFS service shut down: Aug 24 14:19:03 san2 ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66-nfs-xcpnfs-1-0-san2-datsvq[1962479]: 24/08/2023 04:19:03 : epoch 64b8af5a : san2 : ganesha.nfsd-8[Admin] do_shutdown :MAIN :EVENT :Removing all exports. Aug 24 14:19:13 san2 bash[3235994]: time="2023-08-24T14:19:13+10:00" level=warning msg="StopSignal SIGTERM failed to stop container ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66-nfs-xcpnfs-1-0-san2-datsvq in 10 seconds, resorting to SIGKILL" Aug 24 14:19:13 san2 bash[3235994]: ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66-nfs-xcpnfs-1-0-san2-datsvq Aug 24 14:19:13 san2 systemd[1]:ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66@nfs.xcpnfs.1.0.san2.datsvq.servic <mailto:ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66@nfs.xcpnfs.1.0.san2.datsvq.servic>e: Main process exited, code=exited, status=137/n/a Aug 24 14:19:14 san2 systemd[1]:ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66@nfs.xcpnfs.1.0.san2.datsvq.servic <mailto:ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66@nfs.xcpnfs.1.0.san2.datsvq.servic>e: Failed with result 'exit-code'. Aug 24 14:19:14 san2 systemd[1]: Stopped Ceph nfs.xcpnfs.1.0.san2.datsvq for e2f1b934-ed43-11ec-80fa-04421a1a1d66. And that's it. The ingress IP didn't move. More odd, the cluster seems to have placed the ingress IP on node 1 (san1) but seems to be using the NFS service on node 2. Do I need to more tightly connect the NFS service to the keepalive and haproxy services, or do I need to expand the ingress services to refer to multiple NFS services? Thank you. -- Regards, Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170 _DDNS /_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

3 9

CVE-2023-43040 - Improperly verified POST keys in Ceph RGW?
by Christian Rohmann 27 Sep '23

27 Sep '23

Hey Ceph-users, i just noticed there is a post to oss-security (https://www.openwall.com/lists/oss-security/2023/09/26/10) about a security issue with Ceph RGW. Signed by IBM / Redhat and including a patch by DO. I also raised an issue on the tracker (https://tracker.ceph.com/issues/63004) about this, as I could not find one yet. It seems a weird way of disclosing such a thing and am wondering if anybody knew any more about this? Regards Christian

2 1

Ceph leadership team notes 9/27
by Gregory Farnum 27 Sep '23

27 Sep '23

Hi everybody, The CLT met today as usual. We only had a few topics under discussion: * the User + Dev relaunch went off well! We’d like reliable recordings and have found Jitsi to be somewhat glitchy; Laura will communicate about workarounds for that while we work on a longer-term solution (self-hosting Jitsi has a better reputation and is a possibility). We also discussed a GitHub repo for hosting presentation files, and organizing them on the website. * CVE handling. As noted elsewhere on the mailing list, CVE-2023-43040 (a privilege escalation impacting RGW) was disclosed elsewhere, and we do not have coordinated releases for it. This was not deemed important enough on the security list for that effort, but we do want to be more prepared for it than we were — our CVE handling process has broken down a bit since some of the CVE work is now being handled by IBM instead of Red Hat. Tech leads and IBM employees will be working on refining that so we have better disclosures. Also, if you were previously on the security mailing list and a did not see these emails, please reach out to the team — some subscribers were lost and not recovered in the lab disaster end of last year. (For obvious reasons this is a closed list — if you do not work for a Linux distribution or at a large deployer with established relationships in Ceph and security communities, it’s hard for us to put you there.) -Greg

1 0

Dashboard daemon logging not working
by Thomas Bennett 27 Sep '23

27 Sep '23

Hey, Has anyone else had issues with exploring Loki after deploying ceph monitoring services <https://docs.ceph.com/en/latest/cephadm/services/monitoring/>? I'm running 17.2.6. When clicking on the Ceph dashboard daemon logs (i.e Cluster -> Logs -> Daemon Logs), it took me through to an embedded Grafana dashboard for "Dashboard1" so it's not working for me. I found a workaround by enabling viewer role edit permissions. So I added viewers_can_edit = true to my grafana.ini. After I fixed this, the 'Explore' button appeared in my Grafana dashboard and I could explore the log files. If you've hit the same problem and have a better solution, please let me know. For anyone who has the same problem and want more details of how I fixed it, here is what I did: From my cephadm shell: ceph config-key get mgr/cephadm/services/grafana/grafana.ini > /tmp/grafana.ini Edit /tmp/grafana.ini and add the line below in red. # {{ cephadm_managed }} # Source /usr/share/ceph/mgr/cephadm/templates/services/grafana/grafana.ini.j2 [users] default_theme = light *viewers_can_edit = true* ... Then updated the config: ceph config-key set mgr/cephadm/services/grafana/grafana.ini -i /tmp/grafana.ini Then a reconfig and restart grafana: ceph orch reconfig grafana ceph orch restart grafana Cheers, Tom

1 0

Cephadm specs application order
by Luis Domingues 27 Sep '23

27 Sep '23

Hi, We are playing a little bit with OSD specs on a test cluster, and we ended up having nodes that match more than 1 OSD spec. (currently 4 or 5). And there is something we did not get yet. Is there any order cephadm will apply the sepcs? Are the specs sorted in any way inside cephadm? We understand that for a specific spec, cephadm will try to match nodes by host, label and then host_pattern. Our question is more at spec level, and the order cephadm will "loop" the specs. I hope I was clear enough. Thanks, Luis Domingues Proton AG

1 0

set proxy for ceph installation
by Majid Varzideh 27 Sep '23

27 Sep '23

hi friends i have deployed my first node in cluster. we dont have direct internet on my server so i have to set proxy for that.i set it /etc/environment /etc/profile but i get bellow error 2023-09-26 17:09:38,254 7f04058b4b80 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.io/ceph/ceph:v17', 'pull'] 2023-09-26 17:09:38,302 7f04058b4b80 INFO Pulling container image quay.io/ceph/ceph:v17... 2023-09-26 17:09:42,083 7f04058b4b80 INFO Non-zero exit code 125 from /usr/bin/podman pull quay.io/ceph/ceph:v17 2023-09-26 17:09:42,083 7f04058b4b80 INFO /usr/bin/podman: stderr Trying to pull quay.io/ceph/ceph:v17... 2023-09-26 17:09:42,084 7f04058b4b80 INFO /usr/bin/podman: stderr time="2023-09-26T17:09:38+03:30" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://quay.io/ceph/ceph:v17: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 34.228.154.221:443: connect: connection refused" 2023-09-26 17:09:42,084 7f04058b4b80 INFO /usr/bin/podman: stderr time="2023-09-26T17:09:39+03:30" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://quay.io/ceph/ceph:v17: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 3.220.246.53:443: connect: connection refused" 2023-09-26 17:09:42,084 7f04058b4b80 INFO /usr/bin/podman: stderr time="2023-09-26T17:09:40+03:30" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://quay.io/ceph/ceph:v17: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 18.213.60.205:443: connect: connection refused" 2023-09-26 17:09:42,084 7f04058b4b80 INFO /usr/bin/podman: stderr Error: initializing source docker://quay.io/ceph/ceph:v17: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp 34.231.182.47:443: connect: connection refused 2023-09-26 17:09:42,084 7f04058b4b80 ERROR ERROR: Failed command: /usr/bin/podman pull quay.io/ceph/ceph:v17 would you please help. thanks,

3 2

CEPH zero iops after upgrade to Reef and manual read balancer
by Mosharaf Hossain 26 Sep '23

26 Sep '23

Hello Folks We've recently performed an upgrade on our Cephadm cluster, transitioning from Ceph Quiency to Reef. However, following the manual implementation of a read balancer in the Reef cluster, we've experienced a significant slowdown in client I/O operations within the Ceph cluster, affecting both client bandwidth and overall cluster performance. This slowdown has resulted in unresponsiveness across all virtual machines within the cluster, despite the fact that the cluster exclusively utilizes SSD storage." Kindly guide us to move forward. Regards Mosharaf Hossain

3 8

replacing storage server host (not drives)
by Wyll Ingersoll 26 Sep '23

26 Sep '23

We have a storage node that is failing, but the disks themselves are not. What is the recommended procedure for replacing the host itself without destroying the OSDs or losing data? This cluster is running ceph 16.2.11 using ceph orchestrator with docker containers on Ubuntu 20.04 (focal). thank you, Wyllys Ingersoll

2 1

pgs incossistent every day same osd
by Jorge JP 26 Sep '23

26 Sep '23

Hello, First, sorry for my english... Since a few weeks, I receive every day notifies with HEALTH ERR in my ceph. The notifies are related to inconssistent pgs and ever are on same osd. I ran smartctl test to the disk osd assigned and the result is "passed". Should replace the disk by other new? Regards!

2 2

rbd rados cephfs libs compilation
by Arnaud Morin 26 Sep '23

26 Sep '23

Hey ceph users! I'd like to compile the different lib-xyz from ceph (rados, rbd, cephfs) with ENABLE_SHARED=OFF (I want static libraries). Since few days I am struggling building the whole ceph repo on debian 12. Is there any way to build only the libraries? I dont need ceph, but only the client side libraries. Regards, Arnaud.

1 0

Re: Libcephfs : ceph_readdirplus_r() with ceph_ll_lookup_vino() : ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)
by Joseph Fernandes 26 Sep '23

26 Sep '23

CC ceph-user (Apologies, Forgot last time I replied) On Tue, Sep 26, 2023 at 1:11 PM Joseph Fernandes <josephaug26(a)gmail.com> wrote: > Hello Venky, > > Did you get a chance to look into the updated program? Am I > missing something? > I suppose it's something trivial I am missing, As I see these API used in > NFS Ganesha also and would have been tested there. > > https://github.com/nfs-ganesha/nfs-ganesha/blob/2a57b6d53295426247b200cd100… > > Thanks, > ~Joe > > On Fri, Sep 22, 2023 at 10:52 PM Joseph Fernandes <josephaug26(a)gmail.com> > wrote: > >> Hello Venky, >> >> Nice to hear from you :) Hope you are doing well. >> >> I tried as you suggested, >> >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# mkdir >> dir1 dir2 >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# echo >> "Hello Worldls!" > file2 >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# echo >> "Hello Worldls!" > file1 >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# ls >> dir1 dir2 file1 file2 >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# >> >> Create a new snapshot called "sofs-4-6" >> >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# ceph fs >> subvolume snapshot ls cephfs 4 >> [ >> { >> "name": "sofs-4-1" >> }, >> { >> "name": "sofs-4-2" >> }, >> { >> "name": "sofs-4-3" >> }, >> { >> "name": "sofs-4-4" >> }, >> { >> "name": "sofs-4-5" >> }, >> { >> "name": "sofs-4-6" >> } >> ] >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# >> >> And Delete the file and directory from the active filessystem >> >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# rm -rf * >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# ls >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# cd .snap >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root/.snap# ls >> _sofs-4-6_1099511627778 >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root/.snap# cd >> _sofs-4-6_1099511627778/ >> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root/.snap/_sofs-4-6_1099511627778# >> ls >> dir1 dir2 file1 file2 >> root@ss-joe-01 >> (bash):/mnt/cephfs/volumes/_nogroup/4/user_root/.snap/_sofs-4-6_1099511627778# >> >> Now I modified the program and executed it, I am getting the same result. >> >> root@ss-joe-01(bash):/home/hydrauser# ./snapshot_inode_lookup >> >> =================================/volumes/_nogroup/4/user_root/===================================== >> >> Path/Name :"/volumes/_nogroup/4/user_root/" >> Inode Address : 0x7f7c10008f00 >> Inode Number : 1099511628293 >> Snapshot Number : 18446744073709551614 >> Inode Number : 1099511628293 >> Snapshot Number : 18446744073709551614 >> . Ino: 1099511628293 SnapId: 18446744073709551614 Address: 0x7f7c10008f00 >> .. Ino: 1099511627778 SnapId: 18446744073709551614 Address: 0x7f7c100086f0 >> >> >> =================================1099511628293:7===================================== >> >> Path/Name :"1099511628293:7" >> Inode Address : 0x7f7c10009710 >> Inode Number : 1099511628293 >> Snapshot Number : 7 >> Inode Number : 1099511628293 >> Snapshot Number : 7 >> . Ino: 1099511628293 SnapId: 7 Address: 0x7f7c10009710 >> .. Ino: 1099511628293 SnapId: 7 Address: 0x7f7c10009710 >> >> >> =================================/volumes/_nogroup/4/user_root/.snap/_sofs-4-6_1099511627778===================================== >> >> Path/Name >> :"/volumes/_nogroup/4/user_root/.snap/_sofs-4-6_1099511627778" >> Inode Address : 0x7f7c10009710 >> Inode Number : 1099511628293 >> Snapshot Number : 7 >> Inode Number : 1099511628293 >> Snapshot Number : 7 >> . Ino: 1099511628293 SnapId: 7 Address: 0x7f7c10009710 >> .. Ino: 1099511628293 SnapId: 18446744073709551615 Address: 0x55efc15b4640 >> file1 Ino: 1099511628297 SnapId: 7 Address: 0x7f7c1000a030 >> dir1 Ino: 1099511628294 SnapId: 7 Address: 0x7f7c1000a720 >> dir2 Ino: 1099511628295 SnapId: 7 Address: 0x7f7c1000ada0 >> file2 Ino: 1099511628296 SnapId: 7 Address: 0x7f7c1000b420 >> >> >> =================================1099511628293:7===================================== >> >> Path/Name :"1099511628293:7" >> Inode Address : 0x7f7c10009710 >> Inode Number : 1099511628293 >> Snapshot Number : 7 >> Inode Number : 1099511628293 >> Snapshot Number : 7 >> . Ino: 1099511628293 SnapId: 7 Address: 0x7f7c10009710 >> .. Ino: 1099511628293 SnapId: 18446744073709551615 Address: 0x55efc15b4640 >> file1 Ino: 1099511628297 SnapId: 7 Address: 0x7f7c1000a030 >> dir1 Ino: 1099511628294 SnapId: 7 Address: 0x7f7c1000a720 >> dir2 Ino: 1099511628295 SnapId: 7 Address: 0x7f7c1000ada0 >> file2 Ino: 1099511628296 SnapId: 7 Address: 0x7f7c1000b420 >> >> root@ss-joe-01(bash):/home/hydrauser# >> >> I have attached the modified program and ceph client logs from this run. >> >> Cheers, >> Joe >> >> On Fri, Sep 22, 2023 at 8:54 PM Venky Shankar <vshankar(a)redhat.com> >> wrote: >> >>> Hi Joseph, >>> >>> On Fri, Sep 22, 2023 at 5:27 PM Joseph Fernandes <josephaug26(a)gmail.com> >>> wrote: >>> > >>> > Hello All, >>> > >>> > I found a weird issue with ceph_readdirplus_r() when used along >>> > with ceph_ll_lookup_vino(). >>> > On ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) >>> quincy >>> > (stable) >>> > >>> > Any help is really appreciated. >>> > >>> > Thanks in advance, >>> > -Joe >>> > >>> > Test Scenario : >>> > >>> > A. Create a Ceph Fs Subvolume "4" and created a directory in root of >>> > subvolume "user_root" >>> > >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23# >>> > ceph fs subvolume ls cephfs >>> > [ >>> > { >>> > "name": "4" >>> > } >>> > ] >>> > root@ss-joe-01 >>> > >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23# >>> > >>> > >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23# >>> > ls -l >>> > total 0 >>> > drwxrwxrwx 2 root root 0 Sep 22 09:16 user_root >>> > root@ss-joe-01 >>> > >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23# >>> > >>> > >>> > B. In the "user_root" directory create some files and directories >>> > >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# >>> > mkdir dir1 dir2 >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# >>> > ls >>> > dir1 dir2 >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# >>> > echo >>> > "Hello Worldls!" > file1 >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# >>> > echo "Hello Worldls!" > file2 >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# >>> > ls >>> > dir1 dir2 file1 file2 >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# >>> > cat file* >>> > Hello Worldls! >>> > Hello Worldls! >>> > >>> > >>> > C. Create a subvolume snapshot "sofs-4-5". Please ignore the older >>> > snapshots. >>> > >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23# >>> > ceph fs subvolume snapshot ls cephfs 4 >>> > [ >>> > { >>> > "name": "sofs-4-1" >>> > }, >>> > { >>> > "name": "sofs-4-2" >>> > }, >>> > { >>> > "name": "sofs-4-3" >>> > }, >>> > { >>> > "name": "sofs-4-4" >>> > }, >>> > { >>> > "name": "sofs-4-5" >>> > } >>> > ] >>> > root@ss-joe-01 >>> > >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23# >>> > >>> > Here "sofs-4-5" has snapshot id 6. >>> > Got this from libcephfs and have verified at Line >>> > snapshot_inode_lookup.cpp#L212. (Attached to the email) >>> > >>> > >>> > #Content within the snapshot >>> > >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23# >>> > cd .snap/ >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/.snap# >>> > ls >>> > _sofs-4-1_1099511627778 _sofs-4-2_1099511627778 >>> _sofs-4-3_1099511627778 >>> > _sofs-4-4_1099511627778 _sofs-4-5_1099511627778 >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/.snap# >>> > cd _sofs-4-5_1099511627778/ >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/.snap/_sofs-4-5_1099511627778# >>> > ls >>> > user_root >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/.snap/_sofs-4-5_1099511627778# >>> > cd user_root/ >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/.snap/_sofs-4-5_1099511627778/user_root# >>> > ls >>> > dir1 dir2 file1 file2 >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/.snap/_sofs-4-5_1099511627778/user_root# >>> > cat file* >>> > Hello Worldls! >>> > Hello Worldls! >>> > root@ss-joe-01 >>> > >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/.snap/_sofs-4-5_1099511627778/user_root# >>> > >>> > >>> > D. Delete all the files and directories in "user_root" >>> > >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# >>> > rm -rf * >>> > root@ss-joe-01 >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# >>> > ls >>> > root@ss-joe-01 >>> > >>> (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# >>> > >>> > >>> > E. Using Libcephfs in a C++ program do the following,(Attached to this >>> > email) >>> > >>> > 1. Get the Inode of "user_root" using ceph_ll_walk(). >>> > 2. Open the directory using Inode received from ceph_ll_walk() >>> and do >>> > ceph_readdirplus_r() >>> > We don't see any dentries(except "." and "..") as we have deleted >>> all >>> > files and directories in the active filesystem. This is expected and >>> > correct! >>> > >>> > >>> > >>> > >>> =================================/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root/===================================== >>> > >>> > Path/Name >>> > >>> :"/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root/" >>> > Inode Address : 0x7f5ce0009900 >>> > Inode Number : 1099511629282 >>> > Snapshot Number : 18446744073709551614 >>> > Inode Number : 1099511629282 >>> > Snapshot Number : 18446744073709551614 >>> > . Ino: 1099511629282 SnapId: 18446744073709551614 Address: >>> 0x7f5ce0009900 >>> > .. Ino: 1099511627779 SnapId: 18446744073709551614 Address: >>> > 0x7f5ce00090f0 >>> > >>> > >>> > 3. Using ceph_ll_lookup_vino() get the Inode * of "user_root" for >>> > snapshot 6, Here "sofs-4-5" has snapshot id 6. >>> >>> Try this for path "/volumes/_nogroup/4/" (snapshots live a level-up in >>> the subvolume path) for the ceph version you use. >>> >>> > Got this from libcephfs and have verified at Line >>> > snapshot_inode_lookup.cpp#L212. (Attached to the email >>> > 4. Open the directory using Inode * received from >>> ceph_ll_lookup_vino() >>> > and do ceph_readdirplus_r() >>> > We don't see any dentries (except "." and "..") This is NOT >>> expected and >>> > NOT correct, as there are files and directories in the snaphot 6. >>> > >>> > >>> > >>> =================================1099511629282:6===================================== >>> > >>> > Path/Name :"1099511629282:6" >>> > Inode Address : 0x7f5ce000a110 >>> > Inode Number : 1099511629282 >>> > Snapshot Number : 6 >>> > Inode Number : 1099511629282 >>> > Snapshot Number : 6 >>> > . Ino: 1099511629282 SnapId: 6 Address: 0x7f5ce000a110 >>> > .. Ino: 1099511629282 SnapId: 6 Address: 0x7f5ce000a110 >>> > >>> > >>> > 5. Get the Inode of "user_root/ .snap/_sofs-4-5_1099511627778 / " >>> using >>> > ceph_ll_walk(). >>> > 6. Open the directory using Inode received from ceph_ll_walk() >>> and do >>> > ceph_readdirplus_r() >>> > We see ALL dentries of all files and directories in the snapshot. >>> This >>> > is expected and correct! >>> > >>> > >>> > >>> =================================/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root/.snap/_sofs-4-5_1099511627778/===================================== >>> > >>> > Path/Name >>> > >>> :"/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root/.snap/_sofs-4-5_1099511627778/" >>> > Inode Address : 0x7f5ce000a110 >>> > Inode Number : 1099511629282 >>> > Snapshot Number : 6 >>> > Inode Number : 1099511629282 >>> > Snapshot Number : 6 >>> > . Ino: 1099511629282 SnapId: 6 Address: 0x7f5ce000a110 >>> > .. Ino: 1099511629282 SnapId: 18446744073709551615 Address: >>> > 0x5630ab946340 >>> > file1 Ino: 1099511628291 SnapId: 6 Address: 0x7f5ce000aa90 >>> > dir1 Ino: 1099511628289 SnapId: 6 Address: 0x7f5ce000b180 >>> > dir2 Ino: 1099511628290 SnapId: 6 Address: 0x7f5ce000b800 >>> > file2 Ino: 1099511628292 SnapId: 6 Address: 0x7f5ce000be80 >>> > >>> > 7. Now Again using ceph_ll_lookup_vino() get the Inode * of >>> "user_root" >>> > for snapshot 6, Here "sofs-4-5" has snapshot id 6. >>> > 8. Open the directory using Inode * received from >>> > ceph_ll_lookup_vino() and do ceph_readdirplus_r() >>> > Now! we see all the files and Directories in the snapshot! >>> > >>> > >>> > >>> =================================1099511629282:6===================================== >>> > >>> > Path/Name :"1099511629282:6" >>> > Inode Address : 0x7f5ce000a110 >>> > Inode Number : 1099511629282 >>> > Snapshot Number : 6 >>> > Inode Number : 1099511629282 >>> > Snapshot Number : 6 >>> > . Ino: 1099511629282 SnapId: 6 Address: 0x7f5ce000a110 >>> > .. Ino: 1099511629282 SnapId: 18446744073709551615 Address: >>> > 0x5630ab946340 >>> > file1 Ino: 1099511628291 SnapId: 6 Address: 0x7f5ce000aa90 >>> > dir1 Ino: 1099511628289 SnapId: 6 Address: 0x7f5ce000b180 >>> > dir2 Ino: 1099511628290 SnapId: 6 Address: 0x7f5ce000b800 >>> > file2 Ino: 1099511628292 SnapId: 6 Address: 0x7f5ce000be80 >>> > >>> > >>> > Am I missing something using these APIs? >>> > >>> > File attached to this email >>> > Full out of the program attached to the email. >>> > - snapshot_inode_lookup.cpp_output.txt <Attached> >>> > C++ Program - snapshot_inode_lookup.cpp <Attached> >>> > /etc/ceph/ceph.conf - <attached> >>> > Ceph Client Log during the run of this C++ program - >>> client.log<attached> >>> > >>> > Compile Command: >>> > g++ -o snapshot_inode_lookup ./snapshot_inode_lookup.cpp -g -ldl -ldw >>> > -lcephfs -lboost_filesystem --std=c++17 >>> > >>> > Linux Details, >>> > root@ss-joe-01(bash):/home/hydrauser# uname -a >>> > Linux ss-joe-01 5.10.0-23-amd64 #1 SMP Debian 5.10.179-1 (2023-05-12) >>> > x86_64 GNU/Linux >>> > root@ss-joe-01(bash):/home/hydrauser# >>> > >>> > Ceph Details, >>> > >>> > root@ss-joe-01(bash):/home/hydrauser# ceph -v >>> > ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy >>> > (stable) >>> > root@ss-joe-01(bash):/home/hydrauser# >>> > root@ss-joe-01(bash):/home/hydrauser# ceph -s >>> > cluster: >>> > id: fb43d857-d165-4189-87fc-cf1debce9170 >>> > health: HEALTH_OK >>> > >>> > services: >>> > mon: 3 daemons, quorum ss-joe-01,ss-joe-02,ss-joe-03 (age 4d) >>> > mgr: ss-joe-01(active, since 4d), standbys: ss-joe-03, ss-joe-02 >>> > mds: 1/1 daemons up >>> > osd: 3 osds: 3 up (since 4d), 3 in (since 4d) >>> > >>> > data: >>> > volumes: 1/1 healthy >>> > pools: 3 pools, 49 pgs >>> > objects: 39 objects, 1.0 MiB >>> > usage: 96 MiB used, 30 GiB / 30 GiB avail >>> > pgs: 49 active+clean >>> > >>> > root@ss-joe-01(bash):/home/hydrauser# >>> > root@ss-joe-01(bash):/home/hydrauser# dpkg -l | grep ceph >>> > ii ceph 17.2.5-1~bpo11+1 >>> > amd64 distributed storage and file system >>> > ii ceph-base 17.2.5-1~bpo11+1 >>> > amd64 common ceph daemon libraries and management >>> tools >>> > ii ceph-base-dbg 17.2.5-1~bpo11+1 >>> > amd64 debugging symbols for ceph-base >>> > ii ceph-common 17.2.5-1~bpo11+1 >>> > amd64 common utilities to mount and interact with a >>> ceph >>> > storage cluster >>> > ii ceph-common-dbg 17.2.5-1~bpo11+1 >>> > amd64 debugging symbols for ceph-common >>> > ii ceph-fuse 17.2.5-1~bpo11+1 >>> > amd64 FUSE-based client for the Ceph distributed file >>> > system >>> > ii ceph-fuse-dbg 17.2.5-1~bpo11+1 >>> > amd64 debugging symbols for ceph-fuse >>> > ii ceph-mds 17.2.5-1~bpo11+1 >>> > amd64 metadata server for the ceph distributed file >>> system >>> > ii ceph-mds-dbg 17.2.5-1~bpo11+1 >>> > amd64 debugging symbols for ceph-mds >>> > ii ceph-mgr 17.2.5-1~bpo11+1 >>> > amd64 manager for the ceph distributed storage system >>> > ii ceph-mgr-cephadm 17.2.5-1~bpo11+1 >>> > all cephadm orchestrator module for ceph-mgr >>> > ii ceph-mgr-dashboard 17.2.5-1~bpo11+1 >>> > all dashboard module for ceph-mgr >>> > ii ceph-mgr-dbg 17.2.5-1~bpo11+1 >>> > amd64 debugging symbols for ceph-mgr >>> > ii ceph-mgr-diskprediction-local 17.2.5-1~bpo11+1 >>> > all diskprediction-local module for ceph-mgr >>> > ii ceph-mgr-k8sevents 17.2.5-1~bpo11+1 >>> > all kubernetes events module for ceph-mgr >>> > ii ceph-mgr-modules-core 17.2.5-1~bpo11+1 >>> > all ceph manager modules which are always enabled >>> > ii ceph-mon 17.2.5-1~bpo11+1 >>> > amd64 monitor server for the ceph storage system >>> > ii ceph-mon-dbg 17.2.5-1~bpo11+1 >>> > amd64 debugging symbols for ceph-mon >>> > ii ceph-osd 17.2.5-1~bpo11+1 >>> > amd64 OSD server for the ceph storage system >>> > ii ceph-osd-dbg 17.2.5-1~bpo11+1 >>> > amd64 debugging symbols for ceph-osd >>> > ii ceph-volume 17.2.5-1~bpo11+1 >>> > all tool to facilidate OSD deployment >>> > ii cephadm 17.2.5-1~bpo11+1 >>> > amd64 cephadm utility to bootstrap ceph daemons with >>> > systemd and containers >>> > ii libcephfs2 17.2.5-1~bpo11+1 >>> > amd64 Ceph distributed file system client library >>> > ii libcephfs2-dbg 17.2.5-1~bpo11+1 >>> > amd64 debugging symbols for libcephfs2 >>> > ii libsqlite3-mod-ceph 17.2.5-1~bpo11+1 >>> > amd64 SQLite3 VFS for Ceph >>> > ii libsqlite3-mod-ceph-dbg 17.2.5-1~bpo11+1 >>> > amd64 debugging symbols for libsqlite3-mod-ceph >>> > ii python3-ceph-argparse 17.2.5-1~bpo11+1 >>> > all Python 3 utility libraries for Ceph CLI >>> > ii python3-ceph-common 17.2.5-1~bpo11+1 >>> > all Python 3 utility libraries for Ceph >>> > ii python3-cephfs 17.2.5-1~bpo11+1 >>> > amd64 Python 3 libraries for the Ceph libcephfs >>> library >>> > ii python3-cephfs-dbg 17.2.5-1~bpo11+1 >>> > amd64 Python 3 libraries for the Ceph libcephfs >>> library >>> > root@ss-joe-01(bash):/home/hydrauser# >>> > _______________________________________________ >>> > ceph-users mailing list -- ceph-users(a)ceph.io >>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> >>> >>> -- >>> Cheers, >>> Venky >>> >>>

1 0

CephFS warning: clients laggy due to laggy OSDs
by Janek Bevendorff 26 Sep '23

26 Sep '23

Hey all, Since the upgrade to Ceph 16.2.14, I keep seeing the following warning: 10 client(s) laggy due to laggy OSDs ceph health detail shows it as: [WRN] MDS_CLIENTS_LAGGY: 10 client(s) laggy due to laggy OSDs mds.***(mds.3): Client *** is laggy; not evicted because some OSD(s) is/are laggy more of this... When I restart the client(s) or the affected MDS daemons, the message goes away and then comes back after a while. ceph osd perf does not list any laggy OSDs (a few with 10-60ms ping, but overwhelmingly < 1ms), so I'm on a total loss what this even means. I have never seen this message before nor was I able to find anything about it. Do you have any idea what this message actually means and how I can get rid of it? Thanks Janek

4 9

September Ceph Science Virtual User Group
by Kevin Hrpcek 25 Sep '23

25 Sep '23

Hey all, We will be having a Ceph science/research/big cluster call on Wednesday September 27th. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research environments where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. NOTE: The change to using Jitsi for the meeting. We are no longer using the bluejeans meeting links. The ceph calendar event does not yet reflect this and has the wrong day as well. Pad URL: https://pad.ceph.com/p/Ceph_Science_User_Group_20230927 Virtual event details: September 27th, 2023 14:00 UTC 4pm Central European 9am Central US Description: Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://meet.jit.si/ceph-science-wg Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS/TROPICS Space Science & Engineering Center University of Wisconsin-Madison

1 0

rgw: strong consistency for (bucket) policy settings?
by Matthias Ferdinand 25 Sep '23

25 Sep '23

Hi, while I don't currently use rgw, I still am curious about consistency guarantees. Usually, S3 has strong read-after-write consistency guarantees (for requests that do not overlap). According to https://docs.ceph.com/en/latest/dev/radosgw/bucket_index/ in Ceph this is also true for per-object ACLs. Is there also a strong consistency guarantee for (bucket) policies? The documentation at https://docs.ceph.com/en/latest/radosgw/bucketpolicy/ apparently does not say anything about this. How would multiple rgw instances synchronize a policy change? Is this effective immediate with strong consistency or is there some propagation delay (hopefully on with some upper bound)? Best regards Matthias

3 6

In which cases can the "mon_osd_full_ratio" and the "mon_osd_backfillfull_ratio" be exceeded ?
by Raphael Laguerre 25 Sep '23

25 Sep '23

Hello, In which cases can the "mon_osd_full_ratio" and the "mon_osd_backfillfull_ratio" be exceeded ? More specifically, in case a subset of OSDs fail, if there isn't any more space left in the remaining OSDs to migrate the PGs of the failed OSDs without exceeding either the "mon_osd_full_ratio" or the "mon_osd_backfillfull_ratio of at least one OSD, how will the cluster behave ? Will it exceed the "mon_osd_full_ratio" or the "mon_osd_backfillfull_ratio" of one or more OSDs to ensure an "active+clean" state ? Best regards, Raphaël

1 0

How to properly remove of cluster_network
by Jan Marek 25 Sep '23

25 Sep '23

Hello, I would like to remove cluster_network, because I'm using for in 10Gbps adapters, but for public_network I have two 25Gbps adapters in LAG group... I have cluster with orchestrator. # ceph config dump ... global advanced cluster_network 172.30.0.0/16 global advanced public_network a.b.7.0/24 mon advanced public_network a.b.7.0/24 ... How to do it safely? Will be correct to only set: ceph config set global cluster_network a.b.7.0/24 ? Have I then restart mon processes and osd processes? Many thanks for advices. Sincerely Jan -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html

1 0

How to use STS Lite correctly?
by Huy Nguyen 23 Sep '23

23 Sep '23

Hi, I have a Ceph cluster v16.2.10 To use STS lite, my configures are like the following: ceph.conf ... [client.rgw.ss-rgw-01] host = ss-rgw-01 rgw_frontends = beast port=8080 rgw_zone=backup-hapu admin_socket = /var/run/ceph/ceph-client.rgw.ss-rgw-01 rgw_sts_key = qekd3Rd5zXr0adQx rgw_s3_auth_use_sts = true $ radosgw-admin role list { "RoleId": "778865a0-bc7b-49d4-aed5-a952ac9d5593", "RoleName": "backup-sts", "Path": "/", "Arn": "arn:aws:iam:::role/backup-sts", "CreateDate": "2022-01-04T10:17:32.373Z", "MaxSessionDuration": 3600, "AssumeRolePolicyDocument": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":[\"arn:aws:iam:::user/backup-service\"]},\"Action\":[\"sts:AssumeRole\"]}]}" }, $ radosgw-admin role policy get --role-name backup-sts --policy-name AllowAccessAllBucket { "Permission policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Action\":[\"s3:*\"],\"Resource\":\"arn:aws:s3:::*/*\"}]}" } Then I use the credential of backup-service user to assume role: sts_client = boto3.client('sts', aws_access_key_id=access_key, aws_secret_access_key=secret_key, endpoint_url=endpoint_url, region_name='backup') response = sts_client.assume_role( RoleArn='arn:aws:iam:::role/backup-sts', RoleSessionName='Alice2', DurationSeconds=3600) s3client = boto3.client('s3', aws_access_key_id = response['Credentials']['AccessKeyId'], aws_secret_access_key = response['Credentials']['SecretAccessKey'], aws_session_token = response['Credentials']['SessionToken'], endpoint_url=endpoint_url, region_name='backup') response = s3client.list_buckets() And the result is AccessDenied, but I can't figure out what I was missing Traceback (most recent call last): File "fff.py", line 52, in <module> response = s3client.list_buckets() File "/home/huynnp/.local/lib/python3.8/site-packages/botocore/client.py", line 535, in _api_call return self._make_api_call(operation_name, kwargs) File "/home/huynnp/.local/lib/python3.8/site-packages/botocore/client.py", line 980, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListBuckets operation: Unknown Does my configuration or code is wrong? Thanks in advance

1 0

Re: S3website range requests - possible issue
by Ondřej Kukla 22 Sep '23

22 Sep '23

I was checking the tracker again and I found already fixed issue that seems to be connected with this issue. https://tracker.ceph.com/issues/44508 Here is the PR that fixes it https://github.com/ceph/ceph/pull/33807 What I’m still not understanding is why this is only happening when using s3website api. Is there someone who could shed some light on this? Regards, Ondrej

3 4

millions of hex 80 0_0000 omap keys in single index shard for single bucket
by Christopher Durham 22 Sep '23

22 Sep '23

I am using ceph 17.2.6 on Rocky 8. I have a system that started giving me large omap object warnings. I tracked this down to a specific index shard for a single s3 bucket. rados -p <indexpool> listomapkeys .dir.<zoneid>.bucketid.nn.shardid shows over 3 million keys for that shard. There are only about 2 million objects in the entire bucket according to a listing of the bucket and radosgw-admin bucket stats --bucket bucketname. No other shard has anywhere near this many index objects. Perhaps it should be noted that this shard is the highest numbered shard for this bucket. For a bucket with 16 shards, this is shard 15. If I look at the list of omapkeys generated, there are *many* beginning with "<80>0_0000", almost the entire set of the three + million keys in the shard. These are index objects in the so-called 'ugly' namespace. The rest ofthey omapkeys appear to be normal. The 0_0000 after the <80> indicates some sort of 'bucket log index' according to src/cls/rgw/cls_rgw.cc. However, using some sed magic previously discussed here, I ran: rados -p <indexpool> getomapval .dir.<zoneid>.bucketid.nn.shardid --omap-key-file /tmp/key.txt Where /tmp/key.txt contains only the funny <80>0_0000 key name without a newline The output of this shows, in a hex dump, the object name to which the index refers, which was at one time a valid object. However, that object no longer exists in the bucket, and based on expiration policy, was previously deleted. Let's say, in the hex dump, that the object was: foo/bar/baz/object1.bin The prefix foo/bar/baz/ used to have 32 objects, say foo/bar/baz/{object1.bin, object2.bin, ... } An s3api listing shows that those objects no longer exist (and that is OK, as they were previously deleted). BUT, now, there is a weirdo object left in the bucket: foo/bar/baz/ <- with the slash at the end, and it is an object not a PRE (fix). All objects under foo/ have a 3 day lifecycle expiration. If I wait(at most) 3 days, the weirdo object with '/' at the end will be deleted, or I can delete it manually using aws s3api. But either way, the log index objects, <80>0_0000.... remain. The bucket in question is heavily used. But with over 3 million of these <80>0_0000 objects (and growing) in a single shard, I am currently at a loss as to what to do or how to stop this from occuring. I've poked around at a few other buckets, and I found a few others that have this problem, but not enoughto cause a large omap warning. (A few hundred <80>0_000.... index objects in a shard), no where near enoughto cause the large omap warning that led me to this post. Any ideas?

2 5

Libcephfs : ceph_readdirplus_r() with ceph_ll_lookup_vino() : ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)
by Joseph Fernandes 22 Sep '23

22 Sep '23

Hello All, I found a weird issue with ceph_readdirplus_r() when used along with ceph_ll_lookup_vino(). On ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) Any help is really appreciated. Thanks in advance, -Joe Test Scenario : A. Create a Ceph Fs Subvolume "4" and created a directory in root of subvolume "user_root" root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23# ceph fs subvolume ls cephfs [ { "name": "4" } ] root@ss-joe-01 (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23# root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23# ls -l total 0 drwxrwxrwx 2 root root 0 Sep 22 09:16 user_root root@ss-joe-01 (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23# B. In the "user_root" directory create some files and directories root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# mkdir dir1 dir2 root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# ls dir1 dir2 root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# echo "Hello Worldls!" > file1 root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# echo "Hello Worldls!" > file2 root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# ls dir1 dir2 file1 file2 root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# cat file* Hello Worldls! Hello Worldls! C. Create a subvolume snapshot "sofs-4-5". Please ignore the older snapshots. root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23# ceph fs subvolume snapshot ls cephfs 4 [ { "name": "sofs-4-1" }, { "name": "sofs-4-2" }, { "name": "sofs-4-3" }, { "name": "sofs-4-4" }, { "name": "sofs-4-5" } ] root@ss-joe-01 (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23# Here "sofs-4-5" has snapshot id 6. Got this from libcephfs and have verified at Line snapshot_inode_lookup.cpp#L212. (Attached to the email) #Content within the snapshot root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23# cd .snap/ root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/.snap# ls _sofs-4-1_1099511627778 _sofs-4-2_1099511627778 _sofs-4-3_1099511627778 _sofs-4-4_1099511627778 _sofs-4-5_1099511627778 root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/.snap# cd _sofs-4-5_1099511627778/ root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/.snap/_sofs-4-5_1099511627778# ls user_root root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/.snap/_sofs-4-5_1099511627778# cd user_root/ root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/.snap/_sofs-4-5_1099511627778/user_root# ls dir1 dir2 file1 file2 root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/.snap/_sofs-4-5_1099511627778/user_root# cat file* Hello Worldls! Hello Worldls! root@ss-joe-01 (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/.snap/_sofs-4-5_1099511627778/user_root# D. Delete all the files and directories in "user_root" root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# rm -rf * root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# ls root@ss-joe-01 (bash):/mnt/cephfs/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root# E. Using Libcephfs in a C++ program do the following,(Attached to this email) 1. Get the Inode of "user_root" using ceph_ll_walk(). 2. Open the directory using Inode received from ceph_ll_walk() and do ceph_readdirplus_r() We don't see any dentries(except "." and "..") as we have deleted all files and directories in the active filesystem. This is expected and correct! =================================/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root/===================================== Path/Name :"/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root/" Inode Address : 0x7f5ce0009900 Inode Number : 1099511629282 Snapshot Number : 18446744073709551614 Inode Number : 1099511629282 Snapshot Number : 18446744073709551614 . Ino: 1099511629282 SnapId: 18446744073709551614 Address: 0x7f5ce0009900 .. Ino: 1099511627779 SnapId: 18446744073709551614 Address: 0x7f5ce00090f0 3. Using ceph_ll_lookup_vino() get the Inode * of "user_root" for snapshot 6, Here "sofs-4-5" has snapshot id 6. Got this from libcephfs and have verified at Line snapshot_inode_lookup.cpp#L212. (Attached to the email 4. Open the directory using Inode * received from ceph_ll_lookup_vino() and do ceph_readdirplus_r() We don't see any dentries (except "." and "..") This is NOT expected and NOT correct, as there are files and directories in the snaphot 6. =================================1099511629282:6===================================== Path/Name :"1099511629282:6" Inode Address : 0x7f5ce000a110 Inode Number : 1099511629282 Snapshot Number : 6 Inode Number : 1099511629282 Snapshot Number : 6 . Ino: 1099511629282 SnapId: 6 Address: 0x7f5ce000a110 .. Ino: 1099511629282 SnapId: 6 Address: 0x7f5ce000a110 5. Get the Inode of "user_root/ .snap/_sofs-4-5_1099511627778 / " using ceph_ll_walk(). 6. Open the directory using Inode received from ceph_ll_walk() and do ceph_readdirplus_r() We see ALL dentries of all files and directories in the snapshot. This is expected and correct! =================================/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root/.snap/_sofs-4-5_1099511627778/===================================== Path/Name :"/volumes/_nogroup/4/f0fae76f-196d-4ebd-b8d0-528985505b23/user_root/.snap/_sofs-4-5_1099511627778/" Inode Address : 0x7f5ce000a110 Inode Number : 1099511629282 Snapshot Number : 6 Inode Number : 1099511629282 Snapshot Number : 6 . Ino: 1099511629282 SnapId: 6 Address: 0x7f5ce000a110 .. Ino: 1099511629282 SnapId: 18446744073709551615 Address: 0x5630ab946340 file1 Ino: 1099511628291 SnapId: 6 Address: 0x7f5ce000aa90 dir1 Ino: 1099511628289 SnapId: 6 Address: 0x7f5ce000b180 dir2 Ino: 1099511628290 SnapId: 6 Address: 0x7f5ce000b800 file2 Ino: 1099511628292 SnapId: 6 Address: 0x7f5ce000be80 7. Now Again using ceph_ll_lookup_vino() get the Inode * of "user_root" for snapshot 6, Here "sofs-4-5" has snapshot id 6. 8. Open the directory using Inode * received from ceph_ll_lookup_vino() and do ceph_readdirplus_r() Now! we see all the files and Directories in the snapshot! =================================1099511629282:6===================================== Path/Name :"1099511629282:6" Inode Address : 0x7f5ce000a110 Inode Number : 1099511629282 Snapshot Number : 6 Inode Number : 1099511629282 Snapshot Number : 6 . Ino: 1099511629282 SnapId: 6 Address: 0x7f5ce000a110 .. Ino: 1099511629282 SnapId: 18446744073709551615 Address: 0x5630ab946340 file1 Ino: 1099511628291 SnapId: 6 Address: 0x7f5ce000aa90 dir1 Ino: 1099511628289 SnapId: 6 Address: 0x7f5ce000b180 dir2 Ino: 1099511628290 SnapId: 6 Address: 0x7f5ce000b800 file2 Ino: 1099511628292 SnapId: 6 Address: 0x7f5ce000be80 Am I missing something using these APIs? File attached to this email Full out of the program attached to the email. - snapshot_inode_lookup.cpp_output.txt <Attached> C++ Program - snapshot_inode_lookup.cpp <Attached> /etc/ceph/ceph.conf - <attached> Ceph Client Log during the run of this C++ program - client.log<attached> Compile Command: g++ -o snapshot_inode_lookup ./snapshot_inode_lookup.cpp -g -ldl -ldw -lcephfs -lboost_filesystem --std=c++17 Linux Details, root@ss-joe-01(bash):/home/hydrauser# uname -a Linux ss-joe-01 5.10.0-23-amd64 #1 SMP Debian 5.10.179-1 (2023-05-12) x86_64 GNU/Linux root@ss-joe-01(bash):/home/hydrauser# Ceph Details, root@ss-joe-01(bash):/home/hydrauser# ceph -v ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) root@ss-joe-01(bash):/home/hydrauser# root@ss-joe-01(bash):/home/hydrauser# ceph -s cluster: id: fb43d857-d165-4189-87fc-cf1debce9170 health: HEALTH_OK services: mon: 3 daemons, quorum ss-joe-01,ss-joe-02,ss-joe-03 (age 4d) mgr: ss-joe-01(active, since 4d), standbys: ss-joe-03, ss-joe-02 mds: 1/1 daemons up osd: 3 osds: 3 up (since 4d), 3 in (since 4d) data: volumes: 1/1 healthy pools: 3 pools, 49 pgs objects: 39 objects, 1.0 MiB usage: 96 MiB used, 30 GiB / 30 GiB avail pgs: 49 active+clean root@ss-joe-01(bash):/home/hydrauser# root@ss-joe-01(bash):/home/hydrauser# dpkg -l | grep ceph ii ceph 17.2.5-1~bpo11+1 amd64 distributed storage and file system ii ceph-base 17.2.5-1~bpo11+1 amd64 common ceph daemon libraries and management tools ii ceph-base-dbg 17.2.5-1~bpo11+1 amd64 debugging symbols for ceph-base ii ceph-common 17.2.5-1~bpo11+1 amd64 common utilities to mount and interact with a ceph storage cluster ii ceph-common-dbg 17.2.5-1~bpo11+1 amd64 debugging symbols for ceph-common ii ceph-fuse 17.2.5-1~bpo11+1 amd64 FUSE-based client for the Ceph distributed file system ii ceph-fuse-dbg 17.2.5-1~bpo11+1 amd64 debugging symbols for ceph-fuse ii ceph-mds 17.2.5-1~bpo11+1 amd64 metadata server for the ceph distributed file system ii ceph-mds-dbg 17.2.5-1~bpo11+1 amd64 debugging symbols for ceph-mds ii ceph-mgr 17.2.5-1~bpo11+1 amd64 manager for the ceph distributed storage system ii ceph-mgr-cephadm 17.2.5-1~bpo11+1 all cephadm orchestrator module for ceph-mgr ii ceph-mgr-dashboard 17.2.5-1~bpo11+1 all dashboard module for ceph-mgr ii ceph-mgr-dbg 17.2.5-1~bpo11+1 amd64 debugging symbols for ceph-mgr ii ceph-mgr-diskprediction-local 17.2.5-1~bpo11+1 all diskprediction-local module for ceph-mgr ii ceph-mgr-k8sevents 17.2.5-1~bpo11+1 all kubernetes events module for ceph-mgr ii ceph-mgr-modules-core 17.2.5-1~bpo11+1 all ceph manager modules which are always enabled ii ceph-mon 17.2.5-1~bpo11+1 amd64 monitor server for the ceph storage system ii ceph-mon-dbg 17.2.5-1~bpo11+1 amd64 debugging symbols for ceph-mon ii ceph-osd 17.2.5-1~bpo11+1 amd64 OSD server for the ceph storage system ii ceph-osd-dbg 17.2.5-1~bpo11+1 amd64 debugging symbols for ceph-osd ii ceph-volume 17.2.5-1~bpo11+1 all tool to facilidate OSD deployment ii cephadm 17.2.5-1~bpo11+1 amd64 cephadm utility to bootstrap ceph daemons with systemd and containers ii libcephfs2 17.2.5-1~bpo11+1 amd64 Ceph distributed file system client library ii libcephfs2-dbg 17.2.5-1~bpo11+1 amd64 debugging symbols for libcephfs2 ii libsqlite3-mod-ceph 17.2.5-1~bpo11+1 amd64 SQLite3 VFS for Ceph ii libsqlite3-mod-ceph-dbg 17.2.5-1~bpo11+1 amd64 debugging symbols for libsqlite3-mod-ceph ii python3-ceph-argparse 17.2.5-1~bpo11+1 all Python 3 utility libraries for Ceph CLI ii python3-ceph-common 17.2.5-1~bpo11+1 all Python 3 utility libraries for Ceph ii python3-cephfs 17.2.5-1~bpo11+1 amd64 Python 3 libraries for the Ceph libcephfs library ii python3-cephfs-dbg 17.2.5-1~bpo11+1 amd64 Python 3 libraries for the Ceph libcephfs library root@ss-joe-01(bash):/home/hydrauser#

2 4

multiple rgw instances with same cephx key
by Boris Behrens 22 Sep '23

22 Sep '23

Hi, is it possible to use one cephx key for multiple parallel running RGW? Maybe I could just use the same 'name' and the same key for all of the RGW instances? I plan to start RGWs all over the place in container and let BGP handle the traffic. But I don't know how to create on demand keys, that get removed when the RGW shuts down. I don't want to use the orchestrator for this, because I would need to add all the compute nodes to it and there might be other processes in place that add FW rules in our provisioning. Cheers Boris

1 0

Querying the most recent snapshot
by Dominique Ramaekers 22 Sep '23

22 Sep '23

Hi, A question to avoid using a to elaborate method in finding de most recent snapshot of a RBD-image. So, what would be the preferred way to find the latest snapshot of this image? root@hvs001:/# rbd snap ls libvirt-pool/CmsrvDOM2-MULTIMEDIA SNAPID NAME SIZE PROTECTED TIMESTAMP 223 snap_5 435 GiB yes Fri Sep 15 15:33:39 2023 262 snap_1 435 GiB yes Mon Sep 18 15:39:36 2023 280 snap_3 435 GiB yes Wed Sep 20 15:39:42 2023 I would tend to select the highest snapid. But at some point, the next snapid will restart at 1? So maybe not the best idea. I could select by date/time but I don't have an easy way to convert the text string to an timestamp... I've looked in the rbd help snap ls, there seem to be now way to sort, format the output of the timestamp,... Any advice will be appreciated. Greetings, Dominique.

2 1

RGW External IAM Authorization
by Seena Fallah 21 Sep '23

21 Sep '23

Hi Community, I recently proposed a new authorization mechanism for RGW that can let the RGW daemon ask an external service to authorize a request based on AWS S3 IAM tags (that means the external service would receive the same env as an IAM policy doc would have to evaluate the policy). You can find the documentation of the implementation here: https://github.com/clwluvw/ceph/blob/rgw-external-iam/doc/radosgw/external-… And the PR here: https://github.com/ceph/ceph/pull/53345 We would love to hear feedback if anyone else feels this would be a need for them and what you would think about the APIs. Best,

1 0

Recently started OSD crashes (or messages thereof)
by Luke Hall 21 Sep '23

21 Sep '23

Hi, Since the recent update to 16.2.14-1~bpo11+1 on Debian Bullseye I've started seeing OSD crashes being registered almost daily across all six physical machines (6xOSD disks per machine). There's a --block-db for each osd on a LV from an NVMe. If anyone has any idea what might be causing these I'd appreciate some insight. Happy to provide any other info which might be useful. Thanks, Luke { "assert_condition": "cur2 >= p.length", "assert_file": "./src/os/bluestore/BlueStore.h", "assert_func": "virtual void RocksDBBlueFSVolumeSelector::sub_usage(void*, const bluefs_fnode_t&)", "assert_line": 3875, "assert_msg": "./src/os/bluestore/BlueStore.h: In function 'virtual void RocksDBBlueFSVolumeSelector::sub_usage(void*, const bluefs_fnode_t&)' thread 7f7f54f25700 time 2023-09-20T14:24:00.455721+0100\n./src/os/bluestore/BlueStore.h: 3875: FAILED ceph_assert(cur2 >= p.length)\n", "assert_thread_name": "bstore_kv_sync", "backtrace": [ "/lib/x86_64-linux-gnu/libpthread.so.0(+0x13140) [0x7f7f68632140]", "gsignal()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55b22a49b5fa]", "/usr/bin/ceph-osd(+0xac673b) [0x55b22a49b73b]", "(RocksDBBlueFSVolumeSelector::sub_usage(void*, bluefs_fnode_t const&)+0x11e) [0x55b22ab0077e]", "(BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x5bd) [0x55b22ab9b8ed]", "(BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0x9a) [0x55b22ab9bd7a]", "(BlueFS::fsync(BlueFS::FileWriter*)+0x79) [0x55b22aba97a9]", "(BlueRocksWritableFile::Sync()+0x15) [0x55b22abbf405]", "(rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x3f) [0x55b22b0914d1]", "(rocksdb::WritableFileWriter::SyncInternal(bool)+0x1f4) [0x55b22b26b7c6]", "(rocksdb::WritableFileWriter::Sync(bool)+0x18c) [0x55b22b26b1f8]", "(rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0x366) [0x55b22b0e4a98]", "(rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x12cc) [0x55b22b0e0c5a]", "(rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x4a) [0x55b22b0df92a]", "(RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x82) [0x55b22b036c42]", "(RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x96) [0x55b22b037cc6]", "(BlueStore::_kv_sync_thread()+0x1201) [0x55b22aafc891]", "(BlueStore::KVSyncThread::entry()+0xd) [0x55b22ab2792d]", "/lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f7f68626ea7]", "clone()" ], "ceph_version": "16.2.14", "crash_id": "2023-09-20T13:24:00.562318Z_beb5c664-9ffb-4a4e-8c61-166865fd4e0b", "entity_name": "osd.8", "os_id": "11", "os_name": "Debian GNU/Linux 11 (bullseye)", "os_version": "11 (bullseye)", "os_version_id": "11", "process_name": "ceph-osd", "stack_sig": "90d1fb6954f0f5b1e98659a93a1b9ce5a5a42cd5e0b2990a65dc336567adcb26", "timestamp": "2023-09-20T13:24:00.562318Z", "utsname_hostname": "cphosd02", "utsname_machine": "x86_64", "utsname_release": "5.10.0-23-amd64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP Debian 5.10.179-1 (2023-05-12)" } -- All postal correspondence to: The Positive Internet Company, 24 Ganton Street, London. W1F 7QY *Follow us on Twitter* @posipeople The Positive Internet Company Limited is registered in England and Wales. Registered company number: 3673639. VAT no: 726 7072 28. Registered office: Northside House, Mount Pleasant, Barnet, Herts, EN4 9EE.

2 1

backfill_wait preventing deep scrubs
by Frank Schilder 21 Sep '23

21 Sep '23

Hi all, I replaced a disk in our octopus cluster and it is rebuilding. I noticed that since the replacement there is no scrubbing going on. Apparently, an OSD having a PG in backfill_wait state seems to block deep scrubbing all other PGs on that OSD as well - at least this is how it looks. Some numbers: the pool in question has 8192 PGs with EC 8+3 and ca 850 OSDs. A total of 144 PGs needed backfilling (were remapped after replacing the disk). After about 2 days we are down to 115 backfill_wait + 3 backfilling. It will take a bit more than a week to complete. There is plenty of time and IOP/s available to deep-scrub PGs on the side, but since the backfill started there is zero scrubbing/deep scrubbing going on and "PGs not deep scrubbed in time" messages are piling up. Is there a way to allow (deep) scrub in this situation? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 2

After power outage, osd do not restart
by Patrick Begou 21 Sep '23

21 Sep '23

Hi, After a power outage on my test ceph cluster, 2 osd fail to restart. The log file show: 8e5f-00266cf8869c(a)osd.2.service: Failed with result 'timeout'. Sep 21 11:55:02 mostha1 systemd[1]: Failed to start Ceph osd.2 for 250f9864-0142-11ee-8e5f-00266cf8869c. Sep 21 11:55:12 mostha1 systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c(a)osd.2.service: Service RestartSec=10s expired, scheduling restart. Sep 21 11:55:12 mostha1 systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c(a)osd.2.service: Scheduled restart job, restart counter is at 2. Sep 21 11:55:12 mostha1 systemd[1]: Stopped Ceph osd.2 for 250f9864-0142-11ee-8e5f-00266cf8869c. Sep 21 11:55:12 mostha1 systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c(a)osd.2.service: Found left-over process 1858 (bash) in control group while starting unit. Ignoring. Sep 21 11:55:12 mostha1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Sep 21 11:55:12 mostha1 systemd[1]: ceph-250f9864-0142-11ee-8e5f-00266cf8869c(a)osd.2.service: Found left-over process 2815 (podman) in control group while starting unit. Ignoring. This is not critical as it is a test cluster and it is actually rebalancing on other osd but I would like to know how to return to HEALTH_OK status. Smartctl show the HDD are OK. So is there a way to recover the osd from this state ? Version is 15.2.17 (juste moved from 15.2.13 to 15.2.17 yesterday, will try to move to latest versions as soon as this problem is solved) Thanks Patrick

3 6

ceph orch osd data_allocate_fraction does not work
by Boris Behrens 21 Sep '23

21 Sep '23

I have a use case where I want to only use a small portion of the disk for the OSD and the documentation states that I can use data_allocation_fraction [1] But cephadm can not use this and throws this error: /usr/bin/podman: stderr ceph-volume lvm batch: error: unrecognized arguments: --data-allocate-fraction 0.1 So, what I actually want to achieve: Split up a single SSD into: 3-5x block.db for spinning disks (5x 320GB or 3x 500GB regarding if I have 8TB HDDs or 16TB HDDs) 1x SSD OSD (100G) for RGW index / meta pools 1x SSD OSD (100G) for RGW gc pool because of this bug [2] My service definition looks like this: service_type: osd service_id: hdd-8tb placement: host_pattern: '*' crush_device_class: hdd spec: data_devices: rotational: 1 size: ':9T' db_devices: rotational: 0 limit: 5 size: '1T:2T' encrypted: true block_db_size: 320000000000 --- service_type: osd service_id: hdd-16tb placement: host_pattern: '*' crush_device_class: hdd spec: data_devices: rotational: 1 size: '14T:' db_devices: rotational: 0 limit: 1 size: '1T:2T' encrypted: true block_db_size: 500000000000 --- service_type: osd service_id: gc placement: host_pattern: '*' crush_device_class: gc spec: data_devices: rotational: 0 size: '1T:2T' encrypted: true data_allocate_fraction: 0.05 --- service_type: osd service_id: ssd placement: host_pattern: '*' crush_device_class: ssd spec: data_devices: rotational: 0 size: '1T:2T' encrypted: true data_allocate_fraction: 0.05 [1] https://docs.ceph.com/en/pacific/cephadm/services/osd/#ceph.deployment.driv… [2] https://tracker.ceph.com/issues/53585 -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 1

OSD not starting after being mounted with ceph-objectstore-tool --op fuse
by Budai Laszlo 21 Sep '23

21 Sep '23

Hello, I have a problem with an OSD not starting after being mounted offline using the ceph-objectstore-tool --op fuse command. The cephadm orch ps now shows me the osd in error state: osd.0 storage1 error 2m ago 5h - 4096M <unknown> <unknown> <unknown> If I'm checkung the logs on the node I can see the following messages in the system journal: Sep 21 10:26:13 storage1 systemd[1]: Started Ceph osd.0 for 82eb0cee-583a-11ee-b10b-abe63a69ab28. Sep 21 10:26:14 storage1 bash[50983]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 Sep 21 10:26:14 storage1 bash[50983]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/ceph--aac54f64--d2a7--42e6> Sep 21 10:26:14 storage1 bash[50983]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/ceph--aac54f64--d2a7--42e6--a09d--1373e3524414-osd--block--57cfd62d--ae4d--4cae--8c64--be255837> Sep 21 10:26:14 storage1 bash[50983]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 Sep 21 10:26:14 storage1 bash[50983]: Running command: /usr/bin/ln -s /dev/mapper/ceph--aac54f64--d2a7--42e6--a09d--1373e3524414-osd--block--57cfd62d--ae4d--4cae--8c64--be25583728fa /var/lib> Sep 21 10:26:14 storage1 bash[50983]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 Sep 21 10:26:14 storage1 bash[50983]: --> ceph-volume raw activate successful for osd ID: 0 Sep 21 10:26:14 storage1 bash[51214]: debug 2023-09-21T10:26:14.607+0000 7f91c87cd540 0 set uid:gid to 167:167 (ceph:ceph) Sep 21 10:26:14 storage1 bash[51214]: debug 2023-09-21T10:26:14.607+0000 7f91c87cd540 0 ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable), process ceph-osd, pid> Sep 21 10:26:14 storage1 bash[51214]: debug 2023-09-21T10:26:14.607+0000 7f91c87cd540 0 pidfile_write: ignore empty --pid-file Sep 21 10:26:14 storage1 bash[51214]: debug 2023-09-21T10:26:14.611+0000 7f91c87cd540 1 bdev(0x55d79b319400 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block Sep 21 10:26:14 storage1 bash[51214]: debug 2023-09-21T10:26:14.611+0000 7f91c87cd540 1 bdev(0x55d79b319400 /var/lib/ceph/osd/ceph-0/block) open size 10733223936 (0x27fc00000, 10 GiB) block> Sep 21 10:26:14 storage1 bash[51214]: debug 2023-09-21T10:26:14.611+0000 7f91c87cd540 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 Sep 21 10:26:14 storage1 bash[51214]: debug 2023-09-21T10:26:14.611+0000 7f91c87cd540 1 bdev(0x55d79b318c00 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block Sep 21 10:26:14 storage1 bash[51214]: debug 2023-09-21T10:26:14.611+0000 7f91c87cd540 1 bdev(0x55d79b318c00 /var/lib/ceph/osd/ceph-0/block) open size 10733223936 (0x27fc00000, 10 GiB) block> Sep 21 10:26:14 storage1 bash[51214]: debug 2023-09-21T10:26:14.611+0000 7f91c87cd540 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 10 GiB Sep 21 10:26:14 storage1 bash[51214]: debug 2023-09-21T10:26:14.611+0000 7f91c87cd540 1 bdev(0x55d79b318c00 /var/lib/ceph/osd/ceph-0/block) close Sep 21 10:26:14 storage1 bash[51214]: debug 2023-09-21T10:26:14.899+0000 7f91c87cd540 1 bdev(0x55d79b319400 /var/lib/ceph/osd/ceph-0/block) close Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.143+0000 7f91c87cd540 0 starting osd.0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.143+0000 7f91c87cd540 -1 Falling back to public interface Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.175+0000 7f91c87cd540 0 load: jerasure load: lrc Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.175+0000 7f91c87cd540 1 bdev(0x55d79c120000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.179+0000 7f91c87cd540 -1 bdev(0x55d79c120000 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.179+0000 7f91c87cd540 1 bdev(0x55d79c120000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.179+0000 7f91c87cd540 -1 bdev(0x55d79c120000 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.179+0000 7f91c87cd540 1 mClockScheduler: set_max_osd_capacity #op shards: 5 max osd capacity(iops) per shard: 63.00 Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.179+0000 7f91c87cd540 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0114000 Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.179+0000 7f91c87cd540 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000026 Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.183+0000 7f91c87cd540 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.183+0000 7f91c87cd540 1 mClockScheduler: set_max_osd_capacity #op shards: 5 max osd capacity(iops) per shard: 63.00 Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.183+0000 7f91c87cd540 0 osd.0:0.OSDShard using op scheduler mClockScheduler Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.183+0000 7f91c87cd540 1 bdev(0x55d79c120000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.183+0000 7f91c87cd540 -1 bdev(0x55d79c120000 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.183+0000 7f91c87cd540 1 mClockScheduler: set_max_osd_capacity #op shards: 5 max osd capacity(iops) per shard: 63.00 Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.183+0000 7f91c87cd540 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0114000 Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.183+0000 7f91c87cd540 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000026 Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.183+0000 7f91c87cd540 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.183+0000 7f91c87cd540 0 osd.0:1.OSDShard using op scheduler mClockScheduler Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.183+0000 7f91c87cd540 1 bdev(0x55d79c120000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.183+0000 7f91c87cd540 -1 bdev(0x55d79c120000 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.183+0000 7f91c87cd540 1 mClockScheduler: set_max_osd_capacity #op shards: 5 max osd capacity(iops) per shard: 63.00 Sep 21 10:26:15 storage1 bash[51214]: debug 2023-09-21T10:26:15.183+0000 7f91c87cd540 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0114000 Sep 21 10:26:15 storage1 systemd[1]: ceph-82eb0cee-583a-11ee-b10b-abe63a69ab28(a)osd.0.service: Main process exited, code=exited, status=1/FAILURE Sep 21 10:26:16 storage1 systemd[1]: ceph-82eb0cee-583a-11ee-b10b-abe63a69ab28(a)osd.0.service: Failed with result 'exit-code'. Any idea how to bring the osd back into the cluster? Thank you, Laszlo

1 0

Error adding OSD
by Budai Laszlo 21 Sep '23

21 Sep '23

Hi all, I am trying to add an OSD using cephadm but it fails with the message found below. Do you have any ide what may be wrong? The given device used to be in the cluster but it has been removed, and now the device appears as available in the `ceph orch device ls`. Thank you, Laszlo root@monitor1:~# ceph orch device ls| grep storage3 storage3 /dev/sdb hdd ATA_QEMU_HARDDISK_QM00002 10.7G No 15m ago Insufficient space (<10 extents) on vgs, LVM detected, locked storage3 /dev/sdc hdd ATA_QEMU_HARDDISK_QM00003 10.7G Yes 15m ago storage3 /dev/sdd hdd ATA_QEMU_HARDDISK_QM00004 8589M No 15m ago locked root@monitor1:~# Here it is the error: Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1756, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 171, in handle_command return dispatch[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 107, in <lambda> wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) # noqa: E731 File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 96, in wrapper return func(*args, **kwargs) File "/usr/share/ceph/mgr/orchestrator/module.py", line 843, in _daemon_add_osd raise_if_exception(completion) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 228, in raise_if_exception raise e RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/314d068c-56ee-11ee-87e2-cd6d389cbfb8/config/ceph.conf Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:6b0a24e3146d4723700ce6579d40e6016b2c63d9bf90422653f2d4caa49be232 -e NODE_NAME=storage3 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=None -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/314d068c-56ee-11ee-87e2-cd6d389cbfb8:/var/run/ceph:z -v /var/log/ceph/314d068c-56ee-11ee-87e2-cd6d389cbfb8:/var/log/ceph:z -v /var/lib/ceph/314d068c-56ee-11ee-87e2-cd6d389cbfb8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp4r8kteec:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmppw___l6k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:6b0a24e3146d4723700ce6579d40e6016b2c63d9bf90422653f2d4caa49be232 lvm batch --no-auto /dev/sdc --yes --no-systemd /usr/bin/docker: stderr --> passed data devices: 1 physical, 0 LVM /usr/bin/docker: stderr --> relative data size: 1.0 /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new d90dcdee-035c-4f3c-80f6-5d3eed25d598 /usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-cf156193-5f39-4bfd-91c0-4e1d50fe0e4e /dev/sdc /usr/bin/docker: stderr stdout: Physical volume "/dev/sdc" successfully created. /usr/bin/docker: stderr stdout: Volume group "ceph-cf156193-5f39-4bfd-91c0-4e1d50fe0e4e" successfully created /usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 2559 -n osd-block-d90dcdee-035c-4f3c-80f6-5d3eed25d598 ceph-cf156193-5f39-4bfd-91c0-4e1d50fe0e4e /usr/bin/docker: stderr stdout: Logical volume "osd-block-d90dcdee-035c-4f3c-80f6-5d3eed25d598" created. /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /usr/bin/docker: stderr Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-6 /usr/bin/docker: stderr Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-cf156193-5f39-4bfd-91c0-4e1d50fe0e4e/osd-block-d90dcdee-035c-4f3c-80f6-5d3eed25d598 /usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 /usr/bin/docker: stderr Running command: /usr/bin/ln -s /dev/ceph-cf156193-5f39-4bfd-91c0-4e1d50fe0e4e/osd-block-d90dcdee-035c-4f3c-80f6-5d3eed25d598 /var/lib/ceph/osd/ceph-6/block /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-6/activate.monmap /usr/bin/docker: stderr stderr: got monmap epoch 3 /usr/bin/docker: stderr --> Creating keyring file for osd.6 /usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6/keyring /usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6/ /usr/bin/docker: stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 6 --monmap /var/lib/ceph/osd/ceph-6/activate.monmap --keyfile - --osdspec-affinity None --osd-data /var/lib/ceph/osd/ceph-6/ --osd-uuid d90dcdee-035c-4f3c-80f6-5d3eed25d598 --setuser ceph --setgroup ceph /usr/bin/docker: stderr stderr: 2023-09-20T19:34:34.916+0000 7f77400f1540 -1 bluestore(/var/lib/ceph/osd/ceph-6/) _read_fsid unparsable uuid /usr/bin/docker: stderr stderr: 2023-09-20T19:34:35.152+0000 7f77400f1540 -1 bluestore(/var/lib/ceph/osd/ceph-6//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-09-20T19:34:35.152+0000 7f77400f1540 -1 bluestore(/var/lib/ceph/osd/ceph-6//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-09-20T19:34:35.152+0000 7f77400f1540 -1 bluestore(/var/lib/ceph/osd/ceph-6//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-09-20T19:34:35.156+0000 7f77400f1540 -1 bluestore(/var/lib/ceph/osd/ceph-6//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-09-20T19:34:35.156+0000 7f77400f1540 -1 bluestore(/var/lib/ceph/osd/ceph-6//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-09-20T19:34:35.156+0000 7f77400f1540 -1 bluestore(/var/lib/ceph/osd/ceph-6//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-09-20T19:34:35.712+0000 7f77400f1540 -1 bluestore(/var/lib/ceph/osd/ceph-6//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-09-20T19:34:35.712+0000 7f77400f1540 -1 bdev(0x563bb89db400 /var/lib/ceph/osd/ceph-6//block) open open got: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-09-20T19:34:35.712+0000 7f77400f1540 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied /usr/bin/docker: stderr stderr: 2023-09-20T19:34:35.712+0000 7f77400f1540 -1 ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-6/: (13) Permission denied /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.6 --yes-i-really-mean-it /usr/bin/docker: stderr stderr: purged osd.6 /usr/bin/docker: stderr --> Zapping: /dev/ceph-cf156193-5f39-4bfd-91c0-4e1d50fe0e4e/osd-block-d90dcdee-035c-4f3c-80f6-5d3eed25d598 /usr/bin/docker: stderr --> Unmounting /var/lib/ceph/osd/ceph-6 /usr/bin/docker: stderr Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-6 /usr/bin/docker: stderr stderr: umount: /var/lib/ceph/osd/ceph-6 unmounted /usr/bin/docker: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-cf156193-5f39-4bfd-91c0-4e1d50fe0e4e/osd-block-d90dcdee-035c-4f3c-80f6-5d3eed25d598 bs=1M count=10 conv=fsync /usr/bin/docker: stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-cf156193-5f39-4bfd-91c0-4e1d50fe0e4e /usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-cf156193-5f39-4bfd-91c0-4e1d50fe0e4e /usr/bin/docker: stderr stderr: Removing ceph--cf156193--5f39--4bfd--91c0--4e1d50fe0e4e-osd--block--d90dcdee--035c--4f3c--80f6--5d3eed25d598 (253:1) /usr/bin/docker: stderr stderr: Archiving volume group "ceph-cf156193-5f39-4bfd-91c0-4e1d50fe0e4e" metadata (seqno 5). /usr/bin/docker: stderr Releasing logical volume "osd-block-d90dcdee-035c-4f3c-80f6-5d3eed25d598" /usr/bin/docker: stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-cf156193-5f39-4bfd-91c0-4e1d50fe0e4e" (seqno 6). /usr/bin/docker: stderr stdout: Logical volume "osd-block-d90dcdee-035c-4f3c-80f6-5d3eed25d598" successfully removed /usr/bin/docker: stderr stderr: Removing physical volume "/dev/sdc" from volume group "ceph-cf156193-5f39-4bfd-91c0-4e1d50fe0e4e" /usr/bin/docker: stderr stdout: Volume group "ceph-cf156193-5f39-4bfd-91c0-4e1d50fe0e4e" successfully removed /usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/sdc /usr/bin/docker: stderr stdout: Labels on physical volume "/dev/sdc" successfully wiped. /usr/bin/docker: stderr --> Zapping successful for OSD: 6 /usr/bin/docker: stderr Traceback (most recent call last): /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in <module> /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ /usr/bin/docker: stderr self.main(self.argv) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc /usr/bin/docker: stderr return f(*a, **kw) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/docker: stderr instance.main() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/docker: stderr instance.main() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/docker: stderr return func(*a, **kw) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 441, in main /usr/bin/docker: stderr self._execute(plan) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 460, in _execute /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/docker: stderr return func(*a, **kw) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create /usr/bin/docker: stderr prepare_step.safe_prepare(args) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare /usr/bin/docker: stderr self.prepare() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/docker: stderr return func(*a, **kw) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 394, in prepare /usr/bin/docker: stderr osd_fsid, /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 119, in prepare_bluestore /usr/bin/docker: stderr db=db /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 484, in osd_mkfs_bluestore /usr/bin/docker: stderr raise RuntimeError('Command failed with exit code %s: %s' % (returncode, ' '.join(command))) /usr/bin/docker: stderr RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 6 --monmap /var/lib/ceph/osd/ceph-6/activate.monmap --keyfile - --osdspec-affinity None --osd-data /var/lib/ceph/osd/ceph-6/ --osd-uuid d90dcdee-035c-4f3c-80f6-5d3eed25d598 --setuser ceph --setgroup ceph Traceback (most recent call last): File "/var/lib/ceph/314d068c-56ee-11ee-87e2-cd6d389cbfb8/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 9653, in <module> main() File "/var/lib/ceph/314d068c-56ee-11ee-87e2-cd6d389cbfb8/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 9641, in main r = ctx.func(ctx) File "/var/lib/ceph/314d068c-56ee-11ee-87e2-cd6d389cbfb8/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 2153, in _infer_config return func(ctx) File "/var/lib/ceph/314d068c-56ee-11ee-87e2-cd6d389cbfb8/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 2069, in _infer_fsid return func(ctx) File "/var/lib/ceph/314d068c-56ee-11ee-87e2-cd6d389cbfb8/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 2181, in _infer_image return func(ctx) File "/var/lib/ceph/314d068c-56ee-11ee-87e2-cd6d389cbfb8/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 2056, in _validate_fsid return func(ctx) File "/var/lib/ceph/314d068c-56ee-11ee-87e2-cd6d389cbfb8/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 6254, in command_ceph_volume out, err, code = call_throws(ctx, c.run_cmd(), verbosity=CallVerbosity.QUIET_UNLESS_ERROR) File "/var/lib/ceph/314d068c-56ee-11ee-87e2-cd6d389cbfb8/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 1853, in call_throws raise RuntimeError('Failed command: %s' % ' '.join(command)) RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:6b0a24e3146d4723700ce6579d40e6016b2c63d9bf90422653f2d4caa49be232 -e NODE_NAME=storage3 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=None -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/314d068c-56ee-11ee-87e2-cd6d389cbfb8:/var/run/ceph:z -v /var/log/ceph/314d068c-56ee-11ee-87e2-cd6d389cbfb8:/var/log/ceph:z -v /var/lib/ceph/314d068c-56ee-11ee-87e2-cd6d389cbfb8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp4r8kteec:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmppw___l6k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:6b0a24e3146d4723700ce6579d40e6016b2c63d9bf90422653f2d4caa49be232 lvm batch --no-auto /dev/sdc --yes --no-systemd

1 0

S3website range requests - possible issue
by Ondřej Kukla 20 Sep '23

20 Sep '23

Hello, In our deployment we are using the mix of s3 and s3website RGW. I’ve noticed strange behaviour when sending range requests to the s3website RGWs that I’m not able to replicate on the s3 ones. I’ve created a simple wrk LUA script to test sending range requests on tiny ranges so the issue is easily seen. When sending these requests against s3 RGW I can see that the amount of data read from Ceph is ± equivalent to what the RGW sends to the client. This change very dramatically when I’m doing the same test against s3website RGW. The read from Ceph is huge (3Gb/s compared to ~22Mb/s on s3 RGW) I seems to me like the RGW is reading the whole files and then sending just the range which is different then what s3 does. I do not understand why would s3website need to read that much from Ceph and I believe this is a bug - I was looking through the tracker and wasn’t able to find anything related to s3website and range requests. Did anyone else noticed this issue? You can replicate it by running this wrk command wrk -t56 -c500 -d5m http://${rgwipaddress}:8080/${bucket}/videos/ -s wrk-range-small.lua wrk script -- Initialize the pseudo random number generator math.randomseed( os.time()) math.random(); math.random(); math.random() i = 1 function request() if i == 8 then i = 1 end local nrangefrom = math.random() local nrangeto = math.random(100) local path = wrk.path url = path..i..".mp4" wrk.headers["Range"] = nrangefrom.."-"..nrangeto i = i+1 return wrk.format(nil, url) end Kind regards, Ondrej

1 1

cephfs mount 'stalls'
by Marc 20 Sep '23

20 Sep '23

I am still on nautilus and some clients are still on centos7 which mount the cephfs. These mounts stall at some point. Currently I am mounting with something like this in the fstab. id=cephfsclientid,client_mountpoint=/cephfs/test /mnt/test fuse.ceph noauto,_netdev,noatime,x-systemd.device-timeout=30,x-systemd.mount-timeout=30,x-systemd.automount,x-systemd.idle-timeout=30 0 0 When the mount stalls I am fixing it with a umount -l, but it would be nicer of course when it would not behave like this. Can this be fixed on el7 and Nautilus, like with different mount options or so?

3 4

Re: S3website range requests - possible issue
by Ondřej Kukla 20 Sep '23

20 Sep '23

I was checking the tracker again and I found already fixed issue that seems to be connected with this issue. https://tracker.ceph.com/issues/44508 Here is the PR that fixes it https://github.com/ceph/ceph/pull/33807 What I’m still not understanding is why this is only happening when using s3website api. Is there someone who could shed some light on this? Regards, Ondrej

1 0

libceph: mds1 IP+PORT wrong peer at address
by Frank Schilder 20 Sep '23

20 Sep '23

Hi all, we seem to have hit a bug in the ceph fs kernel client and I just want to confirm what action to take. We get the error "wrong peer at address" in dmesg and some jobs on that server seem to get stuck in fs access; log extract below. I found these 2 tracker items https://tracker.ceph.com/issues/23883 and https://tracker.ceph.com/issues/41519, which don't seem to have fixes. My questions: - Is this harmless or does it indicate invalid/corrupted client cache entries? - How to resolve, ignore, umount+mount or reboot? Here an extract from the dmesg log, the error has survived a couple of MDS restarts already: [Mon Mar 6 12:56:46 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:05:18 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-1572619386 [Mon Mar 6 13:05:18 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:13:50 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-1572619386 [Mon Mar 6 13:13:50 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state OPEN) [Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state OPEN) [Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start [Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start [Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success [Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success [Mon Mar 6 13:18:13 2023] ceph: update_snap_trace error -22 [Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state OPEN) [Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state OPEN) [Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed [Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect success [Mon Mar 6 13:18:29 2023] ceph: mds7 reconnect success [Mon Mar 6 13:18:35 2023] ceph: update_snap_trace error -22 [Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed [Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed [Mon Mar 6 13:22:22 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Mon Mar 6 13:22:22 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:30:54 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [...] [Thu Mar 9 09:37:24 2023] slurm.epilog.cl (31457): drop_caches: 3 [Thu Mar 9 09:38:26 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:38:26 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 09:46:58 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:46:58 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 09:55:30 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:55:30 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 10:04:02 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 10:04:02 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 6

No snap_schedule module in Octopus
by Patrick Begou 20 Sep '23

20 Sep '23

Hi, I'm working on a small POC for a ceph setup on 4 old C6100 power-edge. I had to install Octopus since latest versions were unable to detect the HDD (too old hardware ??). No matter, this is only for training and understanding Ceph environment. My installation is based on https://download.ceph.com/rpm-15.2.12/el8/noarch/cephadm-15.2.12-0.el8.noar… bootstrapped. I'm reaching the point to automate the snapshots (I can create snapshot by hand without any problem). The documentation https://download.ceph.com/rpm-15.2.12/el8/noarch/cephadm-15.2.12-0.el8.noar… says to use the snap_schedule module but this module does not exist. # ceph mgr module ls | jq -r '.enabled_modules []' cephadm dashboard iostat prometheus restful Have I missed something ? Is there some additional install steps to do for this module ? Thanks for your help. Patrick

3 8

Ceph MDS OOM in combination with 6.5.1 kernel client
by Stefan Kooman 19 Sep '23

19 Sep '23

Hi List, For those of you that are brave enough to run 6.5 CephFS kernel client, we are seeing some interesting things happening. Some of this might be related to this thread [1]. On a couple of shared webhosting platforms we are running CephFS with 6.5.1 kernel. We have disabled "workqueue.cpu_intensive_thresh_us=0" (to prevent CephFS events from seen as cpu intensive). We have seen two MDS OOM situations after that. The MDS allocates ~ 60 GiB of RAM above baseline in ~ 50 seconds. In both OOM situations, a little before the OOM happens, there is a spike of network traffic going out of the MDS to a kernel client (6.5.1). That node gets ~ 700 MiB/s of MDS traffic for also ~ 50 seconds before the MDS process gets killed. Nothing is logged about this. Ceph is HEALTH_OK, no logging by kernel client or MDS whatsoever. The MDS rejoins and is up and active after a couple of minutes. There is no increased load on the MDS or the client that explain this (for as far as we can see). At this point I don't expect anyone to tell me based on these symptoms what the issue is. But if you encounter similar issues, please update this thread. I'm pretty certain we are hitting a bug (or bugs), as the MDS should not blow itself up like that in any case (but evict the client (that misbehaves?). Ceph MDS 16.2.11, MDS_MEMORY_TARGET=160GiB. Gr. Stefan [1]: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YR5UNKBOKDH…

2 1

MDS_CACHE_OVERSIZED, what is this a symptom of?
by Pedro Lopes 19 Sep '23

19 Sep '23

So I'm getting this warning (although there are no noticeable problems in the cluster): $ ceph health detail HEALTH_WARN 1 MDSs report oversized cache [WRN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache mds.storefs-b(mds.0): MDS cache is too large (7GB/4GB); 0 inodes in use by clients, 0 stray files Ceph FS status: $ ceph fs status storefs - 20 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active storefs-a Reqs: 0 /s 1385k 1385k 113k 193k 0-s standby-replay storefs-b Evts: 0 /s 3123k 3123k 33.5k 0 POOL TYPE USED AVAIL storefs-metadata metadata 19.4G 12.6T storefs-pool4x data 4201M 9708G storefs-pool2x data 2338G 18.9T MDS version: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) What is telling me? Is it just that case of the cache size needing to be bigger? Is it a problem with the clients holding onto some kind of reference (documentation says this can be a cause, but now how to check for it). Thanks in advance, Pedro Lopes

2 2

Upgrading OS [and ceph release] nondestructively for oldish Ceph cluster
by Sam Skipsey 19 Sep '23

19 Sep '23

Hello all, We've had a Nautilus [latest releases] cluster for some years now, and are planning the upgrade process - both moving off Centos7 [ideally to a RHEL9 compatible spin like Alma 9 or Rocky 9] and also moving to a newer Ceph release [ideally Pacific or higher to avoid too many later upgrades needed]. As far as ceph release upgrades go, I understand the process in general. What I'm less certain about (and more nervous about from a potential data loss perspective) is the OS upgrade. For Ceph bluestore OSDs, I assume all the relevant metadata is on the OSD disk [or on the separate disk configured for RocksDB etc if you have nvme], and none is on the OS itself? For Mons and Mgrs, what stuff do I need to retain across the OS upgrade to have things "just work" [since they're relatively stateless, I assume mostly the /etc/ceph/ stuff and ceph cluster keys?] For the MDS, I assume it's similar to MGRS? The MDS, IIRC, mainly works as a caching layer so I assume there's not much state that can be lost permanently? Has anyone gone through this process who would be happy to share their experience? (There's not a lot on this on the wider internet - lots on upgrading ceph, much less on the OS) Sam

4 3

Quincy 17.2.6 - Rados gateway crash -
by xadhoom76＠gmail.com 19 Sep '23

19 Sep '23

Hi, we have service that is still crashing when S3 client (veeam backup) start to write data main log from rgw service req 13170422438428971730 0.008000086s s3:get_obj WARNING: couldn't find acl header for object, generating default 2023-07-20T14:36:45.331+0000 7fa5adb4c700 -1 *** Caught signal (Aborted) ** And " 2023-07-19T22:04:15.968+0000 7ff07305b700 1 beast: 0x7fefc7178710: 172.16.199.11 - veeam90 [19/Jul/2023:22:04:15.948 +0000] "PUT /veeam90/Veeam/Backu p/veeam90/Clients/%7Bd14cd688-57b4-4809-a1d9-14cafd191b11%7D/34387bbd-bec9-4a40-a04d-6a890d5d6407/CloudStg/Data/%7Bf687ee0f-fb50-4ded-b3a8-3f67ca7f244 b%7D/%7B6f31c277-734c-46fd-98d5-c560aa6dc776%7D/144113_f3fd31c9ee2a45aeeadda0de3cbc9064_00000000000000000000000000000000 HTTP/1.1" 200 63422 - "APN/1. 0 Veeam/1.0 Backup/12.0" - latency=0.020000216s 2023-07-19T22:04:15.972+0000 7ff08307b700 1 ====== starting new request req=0x7fefc7682710 ===== 2023-07-19T22:04:15.972+0000 7ff087083700 1 ====== starting new request req=0x7fefc737c710 ===== 2023-07-19T22:04:15.972+0000 7ff071057700 1 ====== starting new request req=0x7fefc72fb710 ===== 2023-07-19T22:04:15.972+0000 7ff0998a8700 1 ====== starting new request req=0x7fefc71f9710 ===== 2023-07-19T22:04:15.972+0000 7fefe473e700 -1 *** Caught signal (Aborted) ** in thread 7fefe473e700 thread_name:radosgw ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7ff102d62cf0] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9009b) [0x7ff101d5209b] 5: /lib64/libstdc++.so.6(+0x9653c) [0x7ff101d5853c] 6: /lib64/libstdc++.so.6(+0x95559) [0x7ff101d57559] 7: __gxx_personality_v0() 8: /lib64/libgcc_s.so.1(+0x10b03) [0x7ff101736b03] 9: _Unwind_Resume() 10: /lib64/libradosgw.so.2(+0x538c5b) [0x7ff105246c5b] ---------------------- -- -10> 2023-07-19T22:04:15.972+0000 7ff071057700 2 req 8167590275148061076 0.000000000s s3:put_obj pre-executing -9> 2023-07-19T22:04:15.972+0000 7ff071057700 2 req 8167590275148061076 0.000000000s s3:put_obj check rate limiting -8> 2023-07-19T22:04:15.972+0000 7ff071057700 2 req 8167590275148061076 0.000000000s s3:put_obj executing -7> 2023-07-19T22:04:15.972+0000 7ff0998a8700 1 ====== starting new request req=0x7fefc71f9710 ===== -6> 2023-07-19T22:04:15.972+0000 7ff0998a8700 2 req 15658207768827051601 0.000000000s initializing for trans_id = tx00000d94d21014832be51-0064b85 ddf-3dfe-backup -5> 2023-07-19T22:04:15.972+0000 7ff0998a8700 2 req 15658207768827051601 0.000000000s getting op 1 -4> 2023-07-19T22:04:15.972+0000 7ff0998a8700 2 req 15658207768827051601 0.000000000s s3:put_obj verifying requester -3> 2023-07-19T22:04:15.972+0000 7ff0998a8700 2 req 15658207768827051601 0.000000000s s3:put_obj normalizing buckets and tenants -2> 2023-07-19T22:04:15.972+0000 7ff0998a8700 2 req 15658207768827051601 0.000000000s s3:put_obj init permissions -1> 2023-07-19T22:04:15.972+0000 7ff011798700 2 req 15261257039771290446 0.024000257s s3:put_obj completing 0> 2023-07-19T22:04:15.972+0000 7fefe473e700 -1 *** Caught signal (Aborted) ** in thread 7fefe473e700 thread_name:radosgw " Anyone have this issue ? Thanks

5 5

Error EPERM: error setting 'osd_op_queue' to 'wpq': (1) Operation not permitted
by Nikolaos Dandoulakis 18 Sep '23

18 Sep '23

Hi, After upgrading our cluster to 17.2.6 all OSDs appear to have "osd_op_queue": "mclock_scheduler" (used to be wpq). As we see several OSDs reporting unjustifiable heavy load, we would like to revert this back to "wpq" but any attempt yells the following error: root@store14:~# ceph tell osd.71 config set osd_op_queue wpq Error EPERM: error setting 'osd_op_queue' to 'wpq': (1) Operation not permitted I cannot find anywhere why this is happening, I am guessing another setting needs to be changed as well. Has anybody resolved this? Best, Nick The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th' ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

3 2

openstack rgw swift -- reef vs quincy
by Shashi Dahal 18 Sep '23

18 Sep '23

Hi All, We have 3 openstack clusters, each with their own ceph. The openstack versions are identical( using openstack-ansible) and all rgw-keystone related configs are also the same. The only difference is the ceph version .. one is pacific, quincy while the other (new) one is reef. The issue with reef is: Horizon >> Object Storage >> Containers >> Create New Container In storage-policy , there is nothing in reef vs default-placement in quincy and pacific. without any policy selected ( due to the form being blank), the "submit" button to create the container is disabled. via openstack cli, we are able to create the container, and once created, we can use horizon to upload/download images etc. When doing container show ( in the cli/horizon) it shows that the policy is default-placement. Can someone guide us on how to troubleshoot and correct this? -- Cheers, Shashi

2 1

rbd-mirror and DR test
by Kamil Madac 18 Sep '23

18 Sep '23

One of our customers is currently facing a challenge in testing our disaster recovery (DR) procedures on a pair of Ceph clusters (Quincy version 17.2.5). Our issue revolves around the need to resynchronize data after conducting a DR procedure test. In small-scale scenarios, this may not be a significant problem. However, when dealing with terabytes of data, it becomes a considerable challenge. In a typical DR procedure, there are two sites, Site A and Site B. The process involves demoting Site A and promoting Site B, followed by the reverse operation to ensure data resynchronization. However, our specific challenge lies in the fact that, in our case: - Site A is running and serving production traffic, Site B is just for DR purposes. - Network connectivity between Site A and Site B is deliberately disrupted. - A "promote" operation is enforced (--force) on Site B, creating a split-brain situation. - Data access and modifications are performed on Site B during this state. - To revert to the original configuration, we must demote Site B, but the only way to re-establish RBD mirroring is by forcing a full resynchronization, essentially recopying the entire dataset. Given these circumstances, we are interested in how to address this challenge efficiently, especially when dealing with large datasets (TBs of data). Are there alternative approaches, best practices, or recommendations such that we won't need to fully resync site A to site B in order to reestablish rbd-mirror? Thank you very much for any advice. Kamil Madac

1 0

python error when adding subvolume permission in cli
by metrax＠metrax.eu 18 Sep '23

18 Sep '23

Hi, at one of my clusters, i'm having a problem with authorizing new clients to an existing share. The problem exists on all of the 3 nodes with the same error. Due to a similar command is working on another cluster, something seems to be wrong here. I also tried to delete and recreate the fs and the mds, but it didn't help. The servers are running on Debian 12.1 bookworm with Proxmox. Ceph version is 17.2.6 from the proxmox repositories. Please help me to find the cause of this problem. Thank you very much. Robert ~# ceph fs subvolume authorize shared cweb01lab glcweb01a.lab.xyz.host --access_level=rwp --verbose parsed_args: Namespace(completion=False, help=False, cephconf=None, input_file=None, output_file=None, setuser=None, setgroup=None, client_id=None, client_name=None, cluster=None, admin_socket=None, status=False, watch=False, watch_debug=False, watch_info=False, watch_sec=False, watch_warn=False, watch_error=False, watch_channel=None, version=False, verbose=True, output_format=None, cluster_timeout=None, block=False, period=1), childargs: ['fs', 'subvolume', 'authorize', 'shared', 'cweb01lab', 'glcweb01a.lab.xyz.host', '--access_level=rwp'] cmd000: pg stat cmd001: pg getmap cmd002: pg dump [<dumpcontents:all|summary|sum|delta|pools|osds|pgs|pgs_brief>...] cmd003: pg dump_json [<dumpcontents:all|summary|sum|pools|osds|pgs>...] cmd004: pg dump_pools_json cmd005: pg ls-by-pool <poolstr> [<states>...] cmd006: pg ls-by-primary <id|osd.id> [<pool:int>] [<states>...] cmd007: pg ls-by-osd <id|osd.id> [<pool:int>] [<states>...] cmd008: pg ls [<pool:int>] [<states>...] cmd009: pg dump_stuck [<stuckops:inactive|unclean|stale|undersized|degraded>...] [<threshold:int>] cmd010: pg debug <debugop:unfound_objects_exist|degraded_pgs_exist> cmd011: pg scrub <pgid> cmd012: pg deep-scrub <pgid> cmd013: pg repair <pgid> cmd014: pg force-recovery <pgid>... cmd015: pg force-backfill <pgid>... cmd016: pg cancel-force-recovery <pgid>... cmd017: pg cancel-force-backfill <pgid>... cmd018: osd perf cmd019: osd df [<output_method:plain|tree>] [<filter_by:class|name>] [<filter>] cmd020: osd blocked-by cmd021: osd pool stats [<pool_name>] cmd022: osd pool scrub <who>... cmd023: osd pool deep-scrub <who>... cmd024: osd pool repair <who>... cmd025: osd pool force-recovery <who>... cmd026: osd pool force-backfill <who>... cmd027: osd pool cancel-force-recovery <who>... cmd028: osd pool cancel-force-backfill <who>... cmd029: osd reweight-by-utilization [<oload:int>] [<max_change:float>] [<max_osds:int>] [--no-increasing] cmd030: osd test-reweight-by-utilization [<oload:int>] [<max_change:float>] [<max_osds:int>] [--no-increasing] cmd031: osd reweight-by-pg [<oload:int>] [<max_change:float>] [<max_osds:int>] [<pools>...] cmd032: osd test-reweight-by-pg [<oload:int>] [<max_change:float>] [<max_osds:int>] [<pools>...] cmd033: osd destroy <id|osd.id> [--force] [--yes-i-really-mean-it] cmd034: osd purge <id|osd.id> [--force] [--yes-i-really-mean-it] cmd035: osd safe-to-destroy <ids>... cmd036: osd ok-to-stop <ids>... [<max:int>] cmd037: osd scrub <who> cmd038: osd deep-scrub <who> cmd039: osd repair <who> cmd040: service dump cmd041: service status cmd042: config show <who> [<key>] cmd043: config show-with-defaults <who> cmd044: device ls cmd045: device info <devid> cmd046: device ls-by-daemon <who> cmd047: device ls-by-host <host> cmd048: device set-life-expectancy <devid> <from> [<to>] cmd049: device rm-life-expectancy <devid> cmd050: alerts send cmd051: balancer status cmd052: balancer mode <mode:none|crush-compat|upmap> cmd053: balancer on cmd054: balancer off cmd055: balancer pool ls cmd056: balancer pool add <pools>... cmd057: balancer pool rm <pools>... cmd058: balancer eval-verbose [<option>] cmd059: balancer eval [<option>] cmd060: balancer optimize <plan> [<pools>...] cmd061: balancer show <plan> cmd062: balancer rm <plan> cmd063: balancer reset cmd064: balancer dump <plan> cmd065: balancer ls cmd066: balancer execute <plan> cmd067: crash info <id> cmd068: crash post cmd069: crash ls [--format <value>] cmd070: crash ls-new [--format <value>] cmd071: crash rm <id> cmd072: crash prune <keep:int> cmd073: crash archive <id> cmd074: crash archive-all cmd075: crash stat cmd076: crash json_report <hours:int> cmd077: device query-daemon-health-metrics <who> cmd078: device scrape-daemon-health-metrics <who> cmd079: device scrape-health-metrics [<devid>] cmd080: device get-health-metrics <devid> [<sample>] cmd081: device check-health cmd082: device monitoring on cmd083: device monitoring off cmd084: device predict-life-expectancy <devid> cmd085: influx config-set <key> <value> cmd086: influx config-show cmd087: influx send cmd088: influx config-show cmd089: influx config-set <key> <value> cmd090: influx send cmd091: insights cmd092: insights prune-health [<hours:int>] cmd093: iostat [<width:int>] [--print-header] cmd094: fs snapshot mirror enable [<fs_name>] cmd095: fs snapshot mirror disable [<fs_name>] cmd096: fs snapshot mirror peer_add <fs_name> [<remote_cluster_spec>] [<remote_fs_name>] [<remote_mon_host>] [<cephx_key>] cmd097: fs snapshot mirror peer_list [<fs_name>] cmd098: fs snapshot mirror peer_remove <fs_name> [<peer_uuid>] cmd099: fs snapshot mirror peer_bootstrap create <fs_name> <client_name> [<site_name>] cmd100: fs snapshot mirror peer_bootstrap import <fs_name> [<token>] cmd101: fs snapshot mirror add <fs_name> [<path>] cmd102: fs snapshot mirror remove <fs_name> [<path>] cmd103: fs snapshot mirror dirmap <fs_name> [<path>] cmd104: fs snapshot mirror show distribution [<fs_name>] cmd105: fs snapshot mirror daemon status cmd106: nfs export create cephfs <cluster_id> <pseudo_path> <fsname> [<path>] [--readonly] [--client_addr <value>...] [--squash <value>] [--sectype <value>...] cmd107: nfs export create rgw <cluster_id> <pseudo_path> [<bucket>] [<user_id>] [--readonly] [--client_addr <value>...] [--squash <value>] [--sectype <value>...] cmd108: nfs export rm <cluster_id> <pseudo_path> cmd109: nfs export delete <cluster_id> <pseudo_path> cmd110: nfs export ls <cluster_id> [--detailed] cmd111: nfs export info <cluster_id> <pseudo_path> cmd112: nfs export get <cluster_id> <pseudo_path> cmd113: nfs export apply <cluster_id> cmd114: nfs cluster create <cluster_id> [<placement>] [--ingress] [--virtual_ip <value>] [--port <int>] cmd115: nfs cluster rm <cluster_id> cmd116: nfs cluster delete <cluster_id> cmd117: nfs cluster ls cmd118: nfs cluster info [<cluster_id>] cmd119: nfs cluster config get <cluster_id> cmd120: nfs cluster config set <cluster_id> cmd121: nfs cluster config reset <cluster_id> cmd122: device ls-lights cmd123: device light <enable:on|off> <devid> [<light_type:ident|fault>] [--force] cmd124: orch host add <hostname> [<addr>] [<labels>...] [--maintenance] cmd125: orch host rm <hostname> [--force] [--offline] cmd126: orch host drain <hostname> [--force] cmd127: orch host set-addr <hostname> <addr> cmd128: orch host ls [--format {plain|json|json-pretty|yaml|xml-pretty|xml}] [--host_pattern <value>] [--label <value>] [--host_status <value>] cmd129: orch host label add <hostname> <label> cmd130: orch host label rm <hostname> <label> [--force] cmd131: orch host ok-to-stop <hostname> cmd132: orch host maintenance enter <hostname> [--force] cmd133: orch host maintenance exit <hostname> cmd134: orch host rescan <hostname> [--with-summary] cmd135: orch device ls [<hostname>...] [--format {plain|json|json-pretty|yaml|xml-pretty|xml}] [--refresh] [--wide] cmd136: orch device zap <hostname> <path> [--force] cmd137: orch ls [<service_type>] [<service_name>] [--export] [--format {plain|json|json-pretty|yaml|xml-pretty|xml}] [--refresh] cmd138: orch ps [<hostname>] [--service_name <value>] [--daemon_type <value>] [--daemon_id <value>] [--format {plain|json|json-pretty|yaml|xml-pretty|xml}] [--refresh] cmd139: orch apply osd [--all-available-devices] [--format {plain|json|json-pretty|yaml|xml-pretty|xml}] [--unmanaged] [--dry-run] [--no-overwrite] cmd140: orch daemon add osd [<svc_arg>] [<method:raw|lvm>] cmd141: orch osd rm <osd_id>... [--replace] [--force] [--zap] cmd142: orch osd rm stop <osd_id>... cmd143: orch osd rm status [--format {plain|json|json-pretty|yaml|xml-pretty|xml}] cmd144: orch daemon add [<daemon_type:mon|mgr|rbd-mirror|cephfs-mirror|crash|alertmanager|grafana|node-exporter|ceph-exporter|prometheus|loki|promtail|mds|rgw|nfs|iscsi|snmp-gateway>] [<placement>] cmd145: orch daemon add mds <fs_name> [<placement>] cmd146: orch daemon add rgw <svc_id> [<placement>] [--port <int>] [--ssl] cmd147: orch daemon add nfs <svc_id> [<placement>] cmd148: orch daemon add iscsi <pool> <api_user> <api_password> [<trusted_ip_list>] [<placement>] cmd149: orch <action:start|stop|restart|redeploy|reconfig|rotate-key> <service_name> cmd150: orch daemon <action:start|stop|restart|reconfig|rotate-key> <name> cmd151: orch daemon redeploy <name> [<image>] cmd152: orch daemon rm <names>... [--force] cmd153: orch rm <service_name> [--force] cmd154: orch apply [<service_type:mon|mgr|rbd-mirror|cephfs-mirror|crash|alertmanager|grafana|node-exporter|ceph-exporter|prometheus|loki|promtail|mds|rgw|nfs|iscsi|snmp-gateway>] [<placement>] [--dry-run] [--format {plain|json|json-pretty|yaml|xml-pretty|xml}] [--unmanaged] [--no-overwrite] cmd155: orch apply mds <fs_name> [<placement>] [--dry-run] [--unmanaged] [--format {plain|json|json-pretty|yaml|xml-pretty|xml}] [--no-overwrite] cmd156: orch apply rgw <svc_id> [<placement>] [--realm <value>] [--zone <value>] [--port <int>] [--ssl] [--dry-run] [--format {plain|json|json-pretty|yaml|xml-pretty|xml}] [--unmanaged] [--no-overwrite] cmd157: orch apply nfs <svc_id> [<placement>] [--format {plain|json|json-pretty|yaml|xml-pretty|xml}] [--port <int>] [--dry-run] [--unmanaged] [--no-overwrite] cmd158: orch apply iscsi <pool> <api_user> <api_password> [<trusted_ip_list>] [<placement>] [--unmanaged] [--dry-run] [--format {plain|json|json-pretty|yaml|xml-pretty|xml}] [--no-overwrite] cmd159: orch apply snmp-gateway <snmp_version:V2c|V3> <destination> [<port:int>] [<engine_id>] [<auth_protocol:MD5|SHA>] [<privacy_protocol:DES|AES>] [<placement>] [--unmanaged] [--dry-run] [--format {plain|json|json-pretty|yaml|xml-pretty|xml}] [--no-overwrite] cmd160: orch set backend [<module_name>] cmd161: orch pause cmd162: orch resume cmd163: orch cancel cmd164: orch status [--detail] [--format {plain|json|json-pretty|yaml|xml-pretty|xml}] cmd165: orch tuned-profile apply [<profile_name>] [<placement>] [<settings>] [--no-overwrite] cmd166: orch tuned-profile rm <profile_name> cmd167: orch tuned-profile ls [--format {plain|json|json-pretty|yaml|xml-pretty|xml}] cmd168: orch tuned-profile add-setting <profile_name> <setting> <value> cmd169: orch tuned-profile rm-setting <profile_name> <setting> cmd170: orch upgrade check [<image>] [<ceph_version>] cmd171: orch upgrade ls [<image>] [--tags] [--show-all-versions] cmd172: orch upgrade status cmd173: orch upgrade start [<image>] [--daemon_types <value>] [--hosts <value>] [--services <value>] [--limit <int>] [--ceph_version <value>] cmd174: orch upgrade pause cmd175: orch upgrade resume cmd176: orch upgrade stop cmd177: osd perf query add <query:client_id|rbd_image_id|all_subkeys> cmd178: osd perf query remove <query_id:int> cmd179: osd perf counters get <query_id:int> cmd180: osd pool autoscale-status [--format <value>] cmd181: osd pool set threshold <num:float> cmd182: osd pool get noautoscale cmd183: osd pool unset noautoscale cmd184: osd pool set noautoscale cmd185: progress cmd186: progress json cmd187: progress clear cmd188: progress on cmd189: progress off cmd190: prometheus file_sd_config cmd191: healthcheck history ls [--format {plain|json|json-pretty|yaml}] cmd192: healthcheck history clear cmd193: rbd mirror snapshot schedule add <level_spec> <interval> [<start_time>] cmd194: rbd mirror snapshot schedule remove <level_spec> [<interval>] [<start_time>] cmd195: rbd mirror snapshot schedule list [<level_spec>] cmd196: rbd mirror snapshot schedule status [<level_spec>] cmd197: rbd perf image stats [<pool_spec>] [<sort_by:write_ops|write_bytes|write_latency|read_ops|read_bytes|read_latency>] cmd198: rbd perf image counters [<pool_spec>] [<sort_by:write_ops|write_bytes|write_latency|read_ops|read_bytes|read_latency>] cmd199: rbd task add flatten <image_spec> cmd200: rbd task add remove <image_spec> cmd201: rbd task add trash remove <image_id_spec> cmd202: rbd task add migration execute <image_spec> cmd203: rbd task add migration commit <image_spec> cmd204: rbd task add migration abort <image_spec> cmd205: rbd task cancel <task_id> cmd206: rbd task list [<task_id>] cmd207: rbd trash purge schedule add <level_spec> <interval> [<start_time>] cmd208: rbd trash purge schedule remove <level_spec> [<interval>] [<start_time>] cmd209: rbd trash purge schedule list [<level_spec>] cmd210: rbd trash purge schedule status [<level_spec>] cmd211: restful create-key <key_name> cmd212: restful delete-key <key_name> cmd213: restful list-keys cmd214: restful create-self-signed-cert cmd215: restful restart cmd216: mgr self-test python-version cmd217: mgr self-test run cmd218: mgr self-test background start <workload:command_spam|throw_exception|shutdown> cmd219: mgr self-test background stop cmd220: mgr self-test config get <key> cmd221: mgr self-test config get_localized <key> cmd222: mgr self-test remote cmd223: mgr self-test module <module> cmd224: mgr self-test cluster-log <channel> <priority> <message> cmd225: mgr self-test health set <checks> cmd226: mgr self-test health clear [<checks>...] cmd227: mgr self-test insights_set_now_offset <hours:int> cmd228: mgr self-test eval [<s>] cmd229: fs snap-schedule status [<path>] [<fs>] [--format <value>] cmd230: fs snap-schedule list <path> [--recursive] [--fs <value>] [--format <value>] cmd231: fs snap-schedule add <path> <snap_schedule> [<start>] [<fs>] cmd232: fs snap-schedule remove <path> [<repeat>] [<start>] [<fs>] cmd233: fs snap-schedule retention add <path> <retention_spec_or_period> [<retention_count>] [<fs>] cmd234: fs snap-schedule retention remove <path> <retention_spec_or_period> [<retention_count>] [<fs>] cmd235: fs snap-schedule activate <path> [<repeat>] [<start>] [<fs>] cmd236: fs snap-schedule deactivate <path> [<repeat>] [<start>] [<fs>] cmd237: fs perf stats [<mds_rank>] [<client_id>] [<client_ip>] cmd238: fs status [<fs>] [--format <value>] cmd239: osd status [<bucket>] cmd240: telegraf config-show cmd241: telegraf config-set <key> <value> cmd242: telegraf send cmd243: telemetry status cmd244: telemetry diff cmd245: telemetry on [<license>] cmd246: telemetry off cmd247: telemetry enable channel all [<channels>...] cmd248: telemetry enable channel [<channels>...] cmd249: telemetry disable channel all [<channels>...] cmd250: telemetry disable channel [<channels>...] cmd251: telemetry channel ls cmd252: telemetry collection ls cmd253: telemetry send [<endpoint:ceph|device>...] [<license>] cmd254: telemetry show [<channels>...] cmd255: telemetry preview [<channels>...] cmd256: telemetry show-device cmd257: telemetry preview-device cmd258: telemetry show-all cmd259: telemetry preview-all cmd260: test_orchestrator load_data cmd261: fs volume ls cmd262: fs volume create <name> [<placement>] cmd263: fs volume rm <vol_name> [<yes-i-really-mean-it>] cmd264: fs volume rename <vol_name> <new_vol_name> [--yes-i-really-mean-it] cmd265: fs volume info <vol_name> [--human-readable] cmd266: fs subvolumegroup ls <vol_name> cmd267: fs subvolumegroup create <vol_name> <group_name> [<size:int>] [<pool_layout>] [<uid:int>] [<gid:int>] [<mode>] cmd268: fs subvolumegroup rm <vol_name> <group_name> [--force] cmd269: fs subvolumegroup info <vol_name> <group_name> cmd270: fs subvolumegroup resize <vol_name> <group_name> <new_size> [--no-shrink] cmd271: fs subvolumegroup exist <vol_name> cmd272: fs subvolume ls <vol_name> [<group_name>] cmd273: fs subvolume create <vol_name> <sub_name> [<size:int>] [<group_name>] [<pool_layout>] [<uid:int>] [<gid:int>] [<mode>] [--namespace-isolated] cmd274: fs subvolume rm <vol_name> <sub_name> [<group_name>] [--force] [--retain-snapshots] cmd275: fs subvolume authorize <vol_name> <sub_name> <auth_id> [<group_name>] [<access_level>] [<tenant_id>] [--allow-existing-id] cmd276: fs subvolume deauthorize <vol_name> <sub_name> <auth_id> [<group_name>] cmd277: fs subvolume authorized_list <vol_name> <sub_name> [<group_name>] cmd278: fs subvolume evict <vol_name> <sub_name> <auth_id> [<group_name>] cmd279: fs subvolumegroup getpath <vol_name> <group_name> cmd280: fs subvolume getpath <vol_name> <sub_name> [<group_name>] cmd281: fs subvolume info <vol_name> <sub_name> [<group_name>] cmd282: fs subvolume exist <vol_name> [<group_name>] cmd283: fs subvolume metadata set <vol_name> <sub_name> <key_name> <value> [<group_name>] cmd284: fs subvolume metadata get <vol_name> <sub_name> <key_name> [<group_name>] cmd285: fs subvolume metadata ls <vol_name> <sub_name> [<group_name>] cmd286: fs subvolume metadata rm <vol_name> <sub_name> <key_name> [<group_name>] [--force] cmd287: fs subvolumegroup pin <vol_name> <group_name> <pin_type:export|distributed|random> <pin_setting> cmd288: fs subvolumegroup snapshot ls <vol_name> <group_name> cmd289: fs subvolumegroup snapshot create <vol_name> <group_name> <snap_name> cmd290: fs subvolumegroup snapshot rm <vol_name> <group_name> <snap_name> [--force] cmd291: fs subvolume snapshot ls <vol_name> <sub_name> [<group_name>] cmd292: fs subvolume snapshot create <vol_name> <sub_name> <snap_name> [<group_name>] cmd293: fs subvolume snapshot info <vol_name> <sub_name> <snap_name> [<group_name>] cmd294: fs subvolume snapshot metadata set <vol_name> <sub_name> <snap_name> <key_name> <value> [<group_name>] cmd295: fs subvolume snapshot metadata get <vol_name> <sub_name> <snap_name> <key_name> [<group_name>] cmd296: fs subvolume snapshot metadata ls <vol_name> <sub_name> <snap_name> [<group_name>] cmd297: fs subvolume snapshot metadata rm <vol_name> <sub_name> <snap_name> <key_name> [<group_name>] [--force] cmd298: fs subvolume snapshot rm <vol_name> <sub_name> <snap_name> [<group_name>] [--force] cmd299: fs subvolume resize <vol_name> <sub_name> <new_size> [<group_name>] [--no-shrink] cmd300: fs subvolume pin <vol_name> <sub_name> <pin_type:export|distributed|random> <pin_setting> [<group_name>] cmd301: fs subvolume snapshot protect <vol_name> <sub_name> <snap_name> [<group_name>] cmd302: fs subvolume snapshot unprotect <vol_name> <sub_name> <snap_name> [<group_name>] cmd303: fs subvolume snapshot clone <vol_name> <sub_name> <snap_name> <target_sub_name> [<pool_layout>] [<group_name>] [<target_group_name>] cmd304: fs clone status <vol_name> <clone_name> [<group_name>] cmd305: fs clone cancel <vol_name> <clone_name> [<group_name>] cmd306: zabbix config-show cmd307: zabbix config-set <key> <value> cmd308: zabbix send cmd309: zabbix discovery cmd310: pg map <pgid> cmd311: pg repeer <pgid> cmd312: osd last-stat-seq <id|osd.id> cmd313: auth export [<entity>] cmd314: auth get <entity> cmd315: auth get-key <entity> cmd316: auth print-key <entity> cmd317: auth print_key <entity> cmd318: auth list cmd319: auth ls cmd320: auth import cmd321: auth add <entity> [<caps>...] cmd322: auth get-or-create-key <entity> [<caps>...] cmd323: auth get-or-create <entity> [<caps>...] cmd324: auth get-or-create-pending <entity> cmd325: auth clear-pending <entity> cmd326: auth commit-pending <entity> cmd327: fs authorize <filesystem> <entity> <caps>... cmd328: auth caps <entity> <caps>... cmd329: auth del <entity> cmd330: auth rm <entity> cmd331: compact cmd332: fsid cmd333: log <logtext>... cmd334: log last [<num:int>] [<level:debug|info|sec|warn|error>] [<channel:*|cluster|audit|cephadm>] cmd335: status cmd336: health [<detail:detail>] cmd337: health mute <code> [<ttl>] [--sticky] cmd338: health unmute [<code>] cmd339: time-sync-status cmd340: df [<detail:detail>] cmd341: report [<tags>...] cmd342: features cmd343: quorum_status cmd344: mon ok-to-stop <ids>... cmd345: mon ok-to-add-offline cmd346: mon ok-to-rm <id> cmd347: tell <type.id> <args>... cmd348: version cmd349: node ls [<type:all|osd|mon|mds|mgr>] cmd350: mon scrub cmd351: mon metadata [<id>] cmd352: mon count-metadata <property> cmd353: mon versions cmd354: versions cmd355: mds stat cmd356: fs dump [<epoch:int>] cmd357: mds metadata [<who>] cmd358: mds count-metadata <property> cmd359: mds versions cmd360: mds ok-to-stop <ids>... cmd361: mds freeze <role_or_gid> <val> cmd362: mds set_state <gid:int> <state:int> cmd363: mds fail <role_or_gid> cmd364: mds repaired <role> cmd365: mds rm <gid:int> cmd366: mds rmfailed <role> [--yes-i-really-mean-it] cmd367: mds compat show cmd368: fs compat show <fs_name> cmd369: mds compat rm_compat <feature:int> cmd370: mds compat rm_incompat <feature:int> cmd371: fs new <fs_name> <metadata> <data> [--force] [--allow-dangerous-metadata-overlay] [<fscid:int>] [--recover] cmd372: fs fail <fs_name> cmd373: fs rm <fs_name> [--yes-i-really-mean-it] cmd374: fs reset <fs_name> [--yes-i-really-mean-it] cmd375: fs ls cmd376: fs get <fs_name> cmd377: fs set <fs_name> <var:max_mds|max_file_size|allow_new_snaps|inline_data|cluster_down|allow_dirfrags|balancer|standby_count_wanted|session_timeout|session_autoclose|allow_standby_replay|down|joinable|min_compat_client> <val> [--yes-i-really-mean-it] [--yes-i-really-really-mean-it] cmd378: fs flag set <flag_name:enable_multiple> <val> [--yes-i-really-mean-it] cmd379: fs feature ls cmd380: fs lsflags <fs_name> cmd381: fs compat <fs_name> <subop:rm_compat|rm_incompat|add_compat|add_incompat> <feature:int> [<feature_str>] cmd382: fs required_client_features <fs_name> <subop:add|rm> <val> cmd383: fs add_data_pool <fs_name> <pool> cmd384: fs rm_data_pool <fs_name> <pool> cmd385: fs set_default <fs_name> cmd386: fs set-default <fs_name> cmd387: fs mirror enable <fs_name> cmd388: fs mirror disable <fs_name> cmd389: fs mirror peer_add <fs_name> <uuid> <remote_cluster_spec> <remote_fs_name> cmd390: fs mirror peer_remove <fs_name> <uuid> cmd391: fs rename <fs_name> <new_fs_name> [--yes-i-really-mean-it] cmd392: mon dump [<epoch:int>] cmd393: mon stat cmd394: mon getmap [<epoch:int>] cmd395: mon add <name> <addr> [<location>...] cmd396: mon rm <name> cmd397: mon remove <name> cmd398: mon feature ls [--with-value] cmd399: mon feature set <feature_name> [--yes-i-really-mean-it] cmd400: mon set-rank <name> <rank:int> cmd401: mon set-addrs <name> <addrs> cmd402: mon set-weight <name> <weight:int> cmd403: mon enable-msgr2 cmd404: mon set election_strategy <strategy> cmd405: mon add disallowed_leader <name> cmd406: mon rm disallowed_leader <name> cmd407: mon set_location <name> <args>... cmd408: mon enable_stretch_mode <tiebreaker_mon> <new_crush_rule> <dividing_bucket> cmd409: mon set_new_tiebreaker <name> [--yes-i-really-mean-it] cmd410: osd stat cmd411: osd dump [<epoch:int>] cmd412: osd info [<id|osd.id>] cmd413: osd tree [<epoch:int>] [<states:up|down|in|out|destroyed>...] cmd414: osd tree-from [<epoch:int>] <bucket> [<states:up|down|in|out|destroyed>...] cmd415: osd ls [<epoch:int>] cmd416: osd getmap [<epoch:int>] cmd417: osd getcrushmap [<epoch:int>] cmd418: osd getmaxosd cmd419: osd ls-tree [<epoch:int>] <name> cmd420: osd find <id|osd.id> cmd421: osd metadata [<id|osd.id>] cmd422: osd count-metadata <property> cmd423: osd versions cmd424: osd numa-status cmd425: osd map <pool> <object> [<nspace>] cmd426: osd lspools cmd427: osd crush rule list cmd428: osd crush rule ls cmd429: osd crush rule ls-by-class <class> cmd430: osd crush rule dump [<name>] cmd431: osd crush dump cmd432: osd setcrushmap [<prior_version:int>] cmd433: osd crush set [<prior_version:int>] cmd434: osd crush add-bucket <name> <type> [<args>...] cmd435: osd crush rename-bucket <srcname> <dstname> cmd436: osd crush set <id|osd.id> <weight:float> <args>... cmd437: osd crush add <id|osd.id> <weight:float> <args>... cmd438: osd crush set-all-straw-buckets-to-straw2 cmd439: osd crush class create <class> cmd440: osd crush class rm <class> cmd441: osd crush set-device-class <class> <ids>... cmd442: osd crush rm-device-class <ids>... cmd443: osd crush class rename <srcname> <dstname> cmd444: osd crush create-or-move <id|osd.id> <weight:float> <args>... cmd445: osd crush move <name> <args>... cmd446: osd crush swap-bucket <source> <dest> [--yes-i-really-mean-it] cmd447: osd crush link <name> <args>... cmd448: osd crush rm <name> [<ancestor>] cmd449: osd crush remove <name> [<ancestor>] cmd450: osd crush unlink <name> [<ancestor>] cmd451: osd crush reweight-all cmd452: osd crush reweight <name> <weight:float> cmd453: osd crush reweight-subtree <name> <weight:float> cmd454: osd crush tunables <profile:legacy|argonaut|bobtail|firefly|hammer|jewel|optimal|default> cmd455: osd crush set-tunable <tunable:straw_calc_version> <value:int> cmd456: osd crush get-tunable <tunable:straw_calc_version> cmd457: osd crush show-tunables cmd458: osd crush rule create-simple <name> <root> <type> [<mode:firstn|indep>] cmd459: osd crush rule create-replicated <name> <root> <type> [<class>] cmd460: osd crush rule create-erasure <name> [<profile>] cmd461: osd crush rule rm <name> cmd462: osd crush rule rename <srcname> <dstname> cmd463: osd crush tree [--show-shadow] cmd464: osd crush ls <node> cmd465: osd crush class ls cmd466: osd crush class ls-osd <class> cmd467: osd crush get-device-class <ids>... cmd468: osd crush weight-set ls cmd469: osd crush weight-set dump cmd470: osd crush weight-set create-compat cmd471: osd crush weight-set create <pool> <mode:flat|positional> cmd472: osd crush weight-set rm <pool> cmd473: osd crush weight-set rm-compat cmd474: osd crush weight-set reweight <pool> <item> <weight:float>... cmd475: osd crush weight-set reweight-compat <item> <weight:float>... cmd476: osd setmaxosd <newmax:int> cmd477: osd set-full-ratio <ratio:float> cmd478: osd set-backfillfull-ratio <ratio:float> cmd479: osd set-nearfull-ratio <ratio:float> cmd480: osd get-require-min-compat-client cmd481: osd set-require-min-compat-client <version> [--yes-i-really-mean-it] cmd482: osd pause cmd483: osd unpause cmd484: osd erasure-code-profile set <name> [<profile>...] [--force] cmd485: osd erasure-code-profile get <name> cmd486: osd erasure-code-profile rm <name> cmd487: osd erasure-code-profile ls cmd488: osd set <key:full|pause|noup|nodown|noout|noin|nobackfill|norebalance|norecover|noscrub|nodeep-scrub|notieragent|nosnaptrim|pglog_hardlimit> [--yes-i-really-mean-it] cmd489: osd unset <key:full|pause|noup|nodown|noout|noin|nobackfill|norebalance|norecover|noscrub|nodeep-scrub|notieragent|nosnaptrim> cmd490: osd require-osd-release <release:octopus|pacific|quincy> [--yes-i-really-mean-it] cmd491: osd down <ids>... [--definitely-dead] cmd492: osd stop <ids>... cmd493: osd out <ids>... cmd494: osd in <ids>... cmd495: osd rm <ids>... cmd496: osd add-noup <ids>... cmd497: osd add-nodown <ids>... cmd498: osd add-noin <ids>... cmd499: osd add-noout <ids>... cmd500: osd rm-noup <ids>... cmd501: osd rm-nodown <ids>... cmd502: osd rm-noin <ids>... cmd503: osd rm-noout <ids>... cmd504: osd set-group <flags> <who>... cmd505: osd unset-group <flags> <who>... cmd506: osd reweight <id|osd.id> <weight:float> cmd507: osd reweightn <weights> cmd508: osd force-create-pg <pgid> [--yes-i-really-mean-it] cmd509: osd pg-temp <pgid> [<id|osd.id>...] cmd510: osd pg-upmap <pgid> <id|osd.id>... cmd511: osd rm-pg-upmap <pgid> cmd512: osd pg-upmap-items <pgid> <id|osd.id>... cmd513: osd rm-pg-upmap-items <pgid> cmd514: osd primary-temp <pgid> <id|osd.id> cmd515: osd primary-affinity <id|osd.id> <weight:float> cmd516: osd destroy-actual <id|osd.id> [--yes-i-really-mean-it] cmd517: osd purge-new <id|osd.id> [--yes-i-really-mean-it] cmd518: osd purge-actual <id|osd.id> [--yes-i-really-mean-it] cmd519: osd lost <id|osd.id> [--yes-i-really-mean-it] cmd520: osd create [<uuid>] [<id|osd.id>] cmd521: osd new <uuid> [<id|osd.id>] cmd522: osd blocklist [<range>] <blocklistop:add|rm> <addr> [<expire:float>] cmd523: osd blocklist ls cmd524: osd blocklist clear cmd525: osd blacklist <blacklistop:add|rm> <addr> [<expire:float>] cmd526: osd blacklist ls cmd527: osd blacklist clear cmd528: osd pool mksnap <pool> <snap> cmd529: osd pool rmsnap <pool> <snap> cmd530: osd pool ls [<detail:detail>] cmd531: osd pool create <pool> [<pg_num:int>] [<pgp_num:int>] [<pool_type:replicated|erasure>] [<erasure_code_profile>] [<rule>] [<expected_num_objects:int>] [<size:int>] [<pg_num_min:int>] [<pg_num_max:int>] [<autoscale_mode:on|off|warn>] [--bulk] [<target_size_bytes:int>] [<target_size_ratio:float>] cmd532: osd pool delete <pool> [<pool2>] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] cmd533: osd pool rm <pool> [<pool2>] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] cmd534: osd pool rename <srcpool> <destpool> cmd535: osd pool get <pool> <var:size|min_size|pg_num|pgp_num|crush_rule|hashpspool|nodelete|nopgchange|nosizechange|write_fadvise_dontneed|noscrub|nodeep-scrub|hit_set_type|hit_set_period|hit_set_count|hit_set_fpp|use_gmt_hitset|target_max_objects|target_max_bytes|cache_target_dirty_ratio|cache_target_dirty_high_ratio|cache_target_full_ratio|cache_min_flush_age|cache_min_evict_age|erasure_code_profile|min_read_recency_for_promote|all|min_write_recency_for_promote|fast_read|hit_set_grade_decay_rate|hit_set_search_last_n|scrub_min_interval|scrub_max_interval|deep_scrub_interval|recovery_priority|recovery_op_priority|scrub_priority|compression_mode|compression_algorithm|compression_required_ratio|compression_max_blob_size|compression_min_blob_size|csum_type|csum_min_block|csum_max_block|allow_ec_overwrites|fingerprint_algorithm|pg_autoscale_mode|pg_autoscale_bias|pg_num_min|pg_num_max|target_size_bytes|target_size_ratio|dedup_tier|dedup_chunk_algorithm|dedup_cdc_chunk_size|eio|bulk> cmd536: osd pool set <pool> <var:size|min_size|pg_num|pgp_num|pgp_num_actual|crush_rule|hashpspool|nodelete|nopgchange|nosizechange|write_fadvise_dontneed|noscrub|nodeep-scrub|hit_set_type|hit_set_period|hit_set_count|hit_set_fpp|use_gmt_hitset|target_max_bytes|target_max_objects|cache_target_dirty_ratio|cache_target_dirty_high_ratio|cache_target_full_ratio|cache_min_flush_age|cache_min_evict_age|min_read_recency_for_promote|min_write_recency_for_promote|fast_read|hit_set_grade_decay_rate|hit_set_search_last_n|scrub_min_interval|scrub_max_interval|deep_scrub_interval|recovery_priority|recovery_op_priority|scrub_priority|compression_mode|compression_algorithm|compression_required_ratio|compression_max_blob_size|compression_min_blob_size|csum_type|csum_min_block|csum_max_block|allow_ec_overwrites|fingerprint_algorithm|pg_autoscale_mode|pg_autoscale_bias|pg_num_min|pg_num_max|target_size_bytes|target_size_ratio|dedup_tier|dedup_chunk_algorithm|dedup_cdc_chunk_size|eio|bulk> <val> [--yes-i-really-mean-it] cmd537: osd pool set-quota <pool> <field:max_objects|max_bytes> <val> cmd538: osd pool get-quota <pool> cmd539: osd pool application enable <pool> <app> [--yes-i-really-mean-it] cmd540: osd pool application disable <pool> <app> [--yes-i-really-mean-it] cmd541: osd pool application set <pool> <app> <key> <value> cmd542: osd pool application rm <pool> <app> <key> cmd543: osd pool application get [<pool>] [<app>] [<key>] cmd544: osd utilization cmd545: osd force_healthy_stretch_mode [--yes-i-really-mean-it] cmd546: osd force_recovery_stretch_mode [--yes-i-really-mean-it] cmd547: osd tier add <pool> <tierpool> [--force-nonempty] cmd548: osd tier rm <pool> <tierpool> cmd549: osd tier remove <pool> <tierpool> cmd550: osd tier cache-mode <pool> <mode:writeback|readproxy|readonly|none> [--yes-i-really-mean-it] cmd551: osd tier set-overlay <pool> <overlaypool> cmd552: osd tier rm-overlay <pool> cmd553: osd tier remove-overlay <pool> cmd554: osd tier add-cache <pool> <tierpool> <size:int> cmd555: config-key get <key> cmd556: config-key set <key> [<val>] cmd557: config-key put <key> [<val>] cmd558: config-key del <key> cmd559: config-key rm <key> cmd560: config-key exists <key> cmd561: config-key list cmd562: config-key ls cmd563: config-key dump [<key>] cmd564: mgr stat cmd565: mgr dump [<epoch:int>] cmd566: mgr fail [<who>] cmd567: mgr module ls cmd568: mgr services cmd569: mgr module enable <module> [--force] cmd570: mgr module disable <module> cmd571: mgr metadata [<who>] cmd572: mgr count-metadata <property> cmd573: mgr versions cmd574: config set <who> <name> <value> [--force] cmd575: config rm <who> <name> cmd576: config get <who> [<key>] cmd577: config dump cmd578: config help <key> cmd579: config ls cmd580: config assimilate-conf cmd581: config log [<num:int>] cmd582: config reset <num:int> cmd583: config generate-minimal-conf cmd584: injectargs <injected_args>... cmd585: smart [<devid>] cmd586: mon_status cmd587: heap <heapcmd:dump|start_profiler|stop_profiler|release|stats> [<value>] cmd588: connection scores dump cmd589: connection scores reset cmd590: sync_force [--yes-i-really-mean-it] cmd591: add_bootstrap_peer_hint <addr> cmd592: add_bootstrap_peer_hintv <addrv> cmd593: quorum enter cmd594: quorum exit cmd595: ops cmd596: sessions cmd597: dump_historic_ops cmd598: dump_historic_slow_ops validate_command: fs subvolume authorize shared cweb01lab glcweb01a.lab.xyz.host --access_level=rwp better match: 0.5 > 0.0: pg stat better match: 0.5 > 0.5: pg getmap better match: 0.5 > 0.5: pg dump [<dumpcontents:all|summary|sum|delta|pools|osds|pgs|pgs_brief>...] better match: 0.5 > 0.5: pg dump_json [<dumpcontents:all|summary|sum|pools|osds|pgs>...] better match: 0.5 > 0.5: pg dump_pools_json better match: 0.5 > 0.5: pg ls-by-pool <poolstr> [<states>...] better match: 0.5 > 0.5: pg ls-by-primary <id|osd.id> [<pool:int>] [<states>...] better match: 0.5 > 0.5: pg ls-by-osd <id|osd.id> [<pool:int>] [<states>...] better match: 0.5 > 0.5: pg ls [<pool:int>] [<states>...] better match: 0.5 > 0.5: pg dump_stuck [<stuckops:inactive|unclean|stale|undersized|degraded>...] [<threshold:int>] better match: 0.5 > 0.5: pg debug <debugop:unfound_objects_exist|degraded_pgs_exist> better match: 0.5 > 0.5: pg scrub <pgid> better match: 0.5 > 0.5: pg deep-scrub <pgid> better match: 0.5 > 0.5: pg repair <pgid> better match: 0.5 > 0.5: pg force-recovery <pgid>... better match: 0.5 > 0.5: pg force-backfill <pgid>... better match: 0.5 > 0.5: pg cancel-force-recovery <pgid>... better match: 0.5 > 0.5: pg cancel-force-backfill <pgid>... better match: 0.5 > 0.5: osd perf better match: 0.5 > 0.5: osd df [<output_method:plain|tree>] [<filter_by:class|name>] [<filter>] better match: 0.5 > 0.5: osd blocked-by better match: 0.5 > 0.5: osd pool stats [<pool_name>] better match: 0.5 > 0.5: osd pool scrub <who>... better match: 0.5 > 0.5: osd pool deep-scrub <who>... better match: 0.5 > 0.5: osd pool repair <who>... better match: 0.5 > 0.5: osd pool force-recovery <who>... better match: 0.5 > 0.5: osd pool force-backfill <who>... better match: 0.5 > 0.5: osd pool cancel-force-recovery <who>... better match: 0.5 > 0.5: osd pool cancel-force-backfill <who>... better match: 0.5 > 0.5: osd reweight-by-utilization [<oload:int>] [<max_change:float>] [<max_osds:int>] [--no-increasing] better match: 0.5 > 0.5: osd test-reweight-by-utilization [<oload:int>] [<max_change:float>] [<max_osds:int>] [--no-increasing] better match: 0.5 > 0.5: osd reweight-by-pg [<oload:int>] [<max_change:float>] [<max_osds:int>] [<pools>...] better match: 0.5 > 0.5: osd test-reweight-by-pg [<oload:int>] [<max_change:float>] [<max_osds:int>] [<pools>...] better match: 0.5 > 0.5: osd destroy <id|osd.id> [--force] [--yes-i-really-mean-it] better match: 0.5 > 0.5: osd purge <id|osd.id> [--force] [--yes-i-really-mean-it] better match: 0.5 > 0.5: osd safe-to-destroy <ids>... better match: 0.5 > 0.5: osd ok-to-stop <ids>... [<max:int>] better match: 0.5 > 0.5: osd scrub <who> better match: 0.5 > 0.5: osd deep-scrub <who> better match: 0.5 > 0.5: osd repair <who> better match: 0.5 > 0.5: service dump better match: 0.5 > 0.5: service status better match: 0.5 > 0.5: config show <who> [<key>] better match: 0.5 > 0.5: config show-with-defaults <who> better match: 0.5 > 0.5: device ls better match: 0.5 > 0.5: device info <devid> better match: 0.5 > 0.5: device ls-by-daemon <who> better match: 0.5 > 0.5: device ls-by-host <host> better match: 0.5 > 0.5: device set-life-expectancy <devid> <from> [<to>] better match: 0.5 > 0.5: device rm-life-expectancy <devid> better match: 0.5 > 0.5: alerts send better match: 0.5 > 0.5: balancer status better match: 0.5 > 0.5: balancer mode <mode:none|crush-compat|upmap> better match: 0.5 > 0.5: balancer on better match: 0.5 > 0.5: balancer off better match: 0.5 > 0.5: balancer pool ls better match: 0.5 > 0.5: balancer pool add <pools>... better match: 0.5 > 0.5: balancer pool rm <pools>... better match: 0.5 > 0.5: balancer eval-verbose [<option>] better match: 0.5 > 0.5: balancer eval [<option>] better match: 0.5 > 0.5: balancer optimize <plan> [<pools>...] better match: 0.5 > 0.5: balancer show <plan> better match: 0.5 > 0.5: balancer rm <plan> better match: 0.5 > 0.5: balancer reset better match: 0.5 > 0.5: balancer dump <plan> better match: 0.5 > 0.5: balancer ls better match: 0.5 > 0.5: balancer execute <plan> better match: 0.5 > 0.5: crash info <id> better match: 0.5 > 0.5: crash post better match: 0.5 > 0.5: crash ls [--format <value>] better match: 0.5 > 0.5: crash ls-new [--format <value>] better match: 0.5 > 0.5: crash rm <id> better match: 0.5 > 0.5: crash prune <keep:int> better match: 0.5 > 0.5: crash archive <id> better match: 0.5 > 0.5: crash archive-all better match: 0.5 > 0.5: crash stat better match: 0.5 > 0.5: crash json_report <hours:int> better match: 0.5 > 0.5: device query-daemon-health-metrics <who> better match: 0.5 > 0.5: device scrape-daemon-health-metrics <who> better match: 0.5 > 0.5: device scrape-health-metrics [<devid>] better match: 0.5 > 0.5: device get-health-metrics <devid> [<sample>] better match: 0.5 > 0.5: device check-health better match: 0.5 > 0.5: device monitoring on better match: 0.5 > 0.5: device monitoring off better match: 0.5 > 0.5: device predict-life-expectancy <devid> better match: 0.5 > 0.5: influx config-set <key> <value> better match: 0.5 > 0.5: influx config-show better match: 0.5 > 0.5: influx send better match: 0.5 > 0.5: influx config-show better match: 0.5 > 0.5: influx config-set <key> <value> better match: 0.5 > 0.5: influx send better match: 0.5 > 0.5: insights better match: 0.5 > 0.5: insights prune-health [<hours:int>] better match: 0.5 > 0.5: iostat [<width:int>] [--print-header] better match: 1.5 > 0.5: fs snapshot mirror enable [<fs_name>] better match: 1.5 > 1.5: fs snapshot mirror disable [<fs_name>] better match: 1.5 > 1.5: fs snapshot mirror peer_add <fs_name> [<remote_cluster_spec>] [<remote_fs_name>] [<remote_mon_host>] [<cephx_key>] better match: 1.5 > 1.5: fs snapshot mirror peer_list [<fs_name>] better match: 1.5 > 1.5: fs snapshot mirror peer_remove <fs_name> [<peer_uuid>] better match: 1.5 > 1.5: fs snapshot mirror peer_bootstrap create <fs_name> <client_name> [<site_name>] better match: 1.5 > 1.5: fs snapshot mirror peer_bootstrap import <fs_name> [<token>] better match: 1.5 > 1.5: fs snapshot mirror add <fs_name> [<path>] better match: 1.5 > 1.5: fs snapshot mirror remove <fs_name> [<path>] better match: 1.5 > 1.5: fs snapshot mirror dirmap <fs_name> [<path>] better match: 1.5 > 1.5: fs snapshot mirror show distribution [<fs_name>] better match: 1.5 > 1.5: fs snapshot mirror daemon status better match: 1.5 > 1.5: fs snap-schedule status [<path>] [<fs>] [--format <value>] better match: 1.5 > 1.5: fs snap-schedule list <path> [--recursive] [--fs <value>] [--format <value>] better match: 1.5 > 1.5: fs snap-schedule add <path> <snap_schedule> [<start>] [<fs>] better match: 1.5 > 1.5: fs snap-schedule remove <path> [<repeat>] [<start>] [<fs>] better match: 1.5 > 1.5: fs snap-schedule retention add <path> <retention_spec_or_period> [<retention_count>] [<fs>] better match: 1.5 > 1.5: fs snap-schedule retention remove <path> <retention_spec_or_period> [<retention_count>] [<fs>] better match: 1.5 > 1.5: fs snap-schedule activate <path> [<repeat>] [<start>] [<fs>] better match: 1.5 > 1.5: fs snap-schedule deactivate <path> [<repeat>] [<start>] [<fs>] better match: 1.5 > 1.5: fs perf stats [<mds_rank>] [<client_id>] [<client_ip>] better match: 1.5 > 1.5: fs status [<fs>] [--format <value>] better match: 1.5 > 1.5: fs volume ls better match: 1.5 > 1.5: fs volume create <name> [<placement>] better match: 1.5 > 1.5: fs volume rm <vol_name> [<yes-i-really-mean-it>] better match: 1.5 > 1.5: fs volume rename <vol_name> <new_vol_name> [--yes-i-really-mean-it] better match: 1.5 > 1.5: fs volume info <vol_name> [--human-readable] better match: 1.5 > 1.5: fs subvolumegroup ls <vol_name> better match: 1.5 > 1.5: fs subvolumegroup create <vol_name> <group_name> [<size:int>] [<pool_layout>] [<uid:int>] [<gid:int>] [<mode>] better match: 1.5 > 1.5: fs subvolumegroup rm <vol_name> <group_name> [--force] better match: 1.5 > 1.5: fs subvolumegroup info <vol_name> <group_name> better match: 1.5 > 1.5: fs subvolumegroup resize <vol_name> <group_name> <new_size> [--no-shrink] better match: 1.5 > 1.5: fs subvolumegroup exist <vol_name> better match: 2.5 > 1.5: fs subvolume ls <vol_name> [<group_name>] better match: 2.5 > 2.5: fs subvolume create <vol_name> <sub_name> [<size:int>] [<group_name>] [<pool_layout>] [<uid:int>] [<gid:int>] [<mode>] [--namespace-isolated] better match: 2.5 > 2.5: fs subvolume rm <vol_name> <sub_name> [<group_name>] [--force] [--retain-snapshots] better match: 6.5 > 2.5: fs subvolume authorize <vol_name> <sub_name> <auth_id> [<group_name>] [<access_level>] [<tenant_id>] [--allow-existing-id] bestcmds_sorted: [{'flags': 8, 'help': 'Allow a cephx auth ID access to a subvolume', 'module': 'mgr', 'perm': 'rw', 'sig': [argdesc(<class 'ceph_argparse.CephPrefix'>, req=True, positional=True, name=prefix, n=1, numseen=0, prefix=fs), argdesc(<class 'ceph_argparse.CephPrefix'>, req=True, positional=True, name=prefix, n=1, numseen=0, prefix=subvolume), argdesc(<class 'ceph_argparse.CephPrefix'>, req=True, positional=True, name=prefix, n=1, numseen=0, prefix=authorize), argdesc(<class 'ceph_argparse.CephString'>, req=True, positional=True, name=vol_name, n=1, numseen=0), argdesc(<class 'ceph_argparse.CephString'>, req=True, positional=True, name=sub_name, n=1, numseen=0), argdesc(<class 'ceph_argparse.CephString'>, req=True, positional=True, name=auth_id, n=1, numseen=0), argdesc(<class 'ceph_argparse.CephString'>, req=False, positional=True, name=group_name, n=1, numseen=0), argdesc(<class 'ceph_argparse.CephString'>, req=False, positional=True, name=access_level, n=1, numseen=0), argdesc(<class 'ceph_argparse.CephString'>, req=False, positional=True, name=tenant_id, n=1, numseen=0), argdesc(<class 'ceph_argparse.CephBool'>, req=False, positional=True, name=allow_existing_id, n=1, numseen=0)]}] Submitting command: {'prefix': 'fs subvolume authorize', 'vol_name': 'shared', 'sub_name': 'cweb01lab', 'auth_id': 'glcweb01a.lab.xyz.host', 'access_level': 'rwp', 'target': ('mon-mgr', '')} submit {"prefix": "fs subvolume authorize", "vol_name": "shared", "sub_name": "cweb01lab", "auth_id": "glcweb01a.lab.xyz.host", "access_level": "rwp", "target": ["mon-mgr", ""]} to mon-mgr Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1756, in _handle_command return self.handle_command(inbuf, cmd) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/volumes/module.py", line 533, in handle_command return handler(inbuf, cmd) ^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/volumes/module.py", line 38, in wrap return f(self, inbuf, cmd) ^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/volumes/module.py", line 631, in _cmd_fs_subvolume_authorize return self.vc.authorize_subvolume(vol_name=cmd['vol_name'], ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 304, in authorize_subvolume key = subvolume.authorize(authid, accesslevel, tenant_id, allow_existing_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_v1.py", line 415, in authorize key = self._authorize_subvolume(auth_id, access_level, existing_caps) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_v1.py", line 446, in _authorize_subvolume key = self._authorize(auth_id, access_level, existing_caps) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_v1.py", line 483, in _authorize return allow_access(self.mgr, client_entity, want_mds_cap, want_osd_cap, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/volumes/fs/operations/access.py", line 93, in allow_access caps = json.loads(out) ^^^^^^^^^^^^^^^ File "/lib/python3.11/json/__init__.py", line 346, in loads return _default_decoder.decode(s) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/lib/python3.11/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/lib/python3.11/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

1 0

Make ceph orch daemons reboot safe
by Boris Behrens 18 Sep '23

18 Sep '23

Hi, is there a way to have the pods start again after reboot? Currently I need to start them by hand via ceph orch start mon/mgr/osd/... I imagine this will lead to a lot of headache when the ceph cluster gets a powercycle and the mon pods will not start automatically. I've spun up a test cluster and there the pods start very fast. On the legacy test cluster, which got adopted to cephadm, it does not. Cheers Boris

3 5

radosgw bucket usage metrics gone after created in a loop 64K buckets
by Szabo, Istvan (Agoda) 18 Sep '23

18 Sep '23

Hi, Last week we've created for a user 64K buckets to be able to properly shard their huge amount of objects and I can see that the "radosgw_usage_bucket" metrics disappeared from 10pm that day when the mass creation happened in our octopus 15.2.17 cluster. In the logs I don't really see anything useful. Is there any limitation that I might have hit? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 1

Rebuilding data resiliency after adding new OSD's stuck for so long at 5%
by sharathvuthpala＠gmail.com 18 Sep '23

18 Sep '23

We have a user-provisioned instance( Bare Metal Installation) of OpenShift cluster running on version 4.12 and we are using OpenShift Data Foundation as the Storage System. Earlier we had 3 disks attached to the storage system and 3 OSDs were available in the cluster. Today, while adding additional disks to the storage cluster, we increased the number of disks from 3 to 9, that is 3 per node. The addition of storage capacity was successful, resulting in 6 new OSDs in the cluster. But, after this operation, we noticed that Rebuilding Data Resiliency is stuck at 5% and not moving forward. At the same time, ceph status shows 65% of objects are misplaced and PGs are not in active+clean state. Here is more information about the ceph cluster: sh-4.4$ ceph status cluster: id: 18bf836d-4937-4925-b964-7a026c1d548d health: HEALTH_OK services: mon: 3 daemons, quorum b,u,v (age 2w) mgr: a(active, since 7w) mds: 1/1 daemons up, 1 hot standby osd: 9 osds: 9 up (since 5h), 9 in (since 5h); 191 remapped pgs rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 12 pools, 305 pgs objects: 2.69M objects, 2.9 TiB usage: 8.8 TiB used, 27 TiB / 36 TiB avail pgs: 4723077/8079717 objects misplaced (58.456%) 188 active+remapped+backfill_wait 114 active+clean 3 active+remapped+backfilling io: client: 679 KiB/s rd, 11 MiB/s wr, 13 op/s rd, 622 op/s wr recovery: 20 MiB/s, 89 keys/s, 22 objects/s sh-4.4$ ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.000276", "last_optimize_started": "Tue Sep 12 17:36:03 2023", "mode": "upmap", "optimize_result": "Too many objects (0.581933 > 0.050000) are misplaced; try again later", "plans": [] } One more thing we observed is that the number of misplaced objects is decreasing and also there is a drop in the percentage. What might be the reason behind Rebuilding Data Resiliency is not moving forward? Any inputs would be appreciated. Thanks

3 5

Awful new dashboard in Reef
by Nicola Mori 15 Sep '23

15 Sep '23

Dear Ceph users, I just upgraded my cluster to Reef, and with the new version came also a revamped dashboard. Unfortunately the new dashboard is really awful to me: 1) it's no longer possible to see the status of the PGs: in the old dashboard it was very easy to see e.g. how many PGs were recovering, how many scrubbing etc. by clicking on the PG Status widget. Now the interface shows just how many are Ok and how many are working, without details, and I have to go to the command line to understand what's happening (not really comfortable on mobile) 2) The new timeline graphs do not work properly: changing the time frame sometimes produce empty graphs, 3) The instant values in Cluster utilization are refreshed so slowly that I cannot properly monitor the cluster behavior in real time Is it just me or maybe my impressions are shared by someone else? Is there anything that can be done to improve the situation? Thanks, Nicola

4 10

ceph orchestator pulls strange images from docker.io
by Boris Behrens 15 Sep '23

15 Sep '23

Hi, I currently try to adopt our stage cluster, some hosts just pull strange images. root@0cc47a6df330:/var/lib/containers/storage/overlay-images# podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a532c37ebe42 docker.io/ceph/daemon-base:latest-master-devel -n mgr.0cc47a6df3... 2 minutes ago Up 2 minutes ago ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-mgr-0cc47a6df330-fxrfyl root@0cc47a6df330:~# ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgr.0cc47a6df14e.vqizdz 0cc47a6df14e.f00f.gridscale.dev *:9283 running (3m) 3m ago 3m 10.8M - 16.2.11 de4b0b384ad4 00b02cd82a1c mgr.0cc47a6df330.iijety 0cc47a6df330.f00f.gridscale.dev *:9283 running (5s) 2s ago 4s 10.5M - 17.0.0-7183-g54142666 75e3d7089cea 662c6baa097e mgr.0cc47aad8ce8 0cc47aad8ce8.f00f.gridscale.dev running (65m) 8m ago 60m 553M - 17.2.6 22cd8daf4d70 8145c63fdc44 Any idea what I need to do to change that? -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

4 8

Questions about PG auto-scaling and node addition
by Christophe BAILLON 15 Sep '23

15 Sep '23

Hello, We have a cluster with 21 nodes, each having 12 x 18TB, and 2 NVMe for db/wal. We need to add more nodes. The last time we did this, the PGs remained at 1024, so the number of PGs per OSD decreased. Currently, we are at 43 PGs per OSD. Does auto-scaling work correctly in Ceph version 17.2.5? Should we increase the number of PGs before adding nodes? Should we keep PG auto-scaling active? If we disable auto-scaling, should we increase the number of PGs to reach 100 PGs per OSD? Considering that we use this cluster with a large EC pool (8+3). Thank you for your assistance.

2 2

osd cannot get osdmap
by Nathan Gleason 15 Sep '23

15 Sep '23

Hello, We had a network hiccup with a Ceph cluster and it made several of our osds go out/down. After the network was fixed the osds remain down. We have restarted them in numerous ways and they won’t come up. The logs for the down osds just repeat this line over and over "tick checking mon for new map”. There are osds on the same host that are up so there is connectivity between the osds and mons. Any advice on where to look for a resolution is appreciated. Thanks, Nathan Cluster was built with cephadm Ceph Quincy - 17.2.6 Docker version 23.0.2, build 569dd73 Ubuntu 20.04.6 LTS cluster: id: aa39fa2a-1510-11ee-953a-bd804ec1ea33 health: HEALTH_ERR Failed to apply 1 service(s): nfs.secstorage 1 filesystem is degraded 1 MDSs report slow metadata IOs Module 'cephadm' has failed: Command '['rados', '-n', 'mgr.cphprodc1-11.uuuhug', '-k', '/var/lib/ceph/mgr/ceph-cphprodc1-11.uuuhug/keyring', '-p', '.nfs', '--namespace', 'secstorage', 'rm', 'grace']' timed out after 10 seconds 28 osds down Reduced data availability: 36 pgs stale 2 daemons have recently crashed 1 mgr modules have recently crashed 945514 slow ops, oldest one blocked for 66804 sec, daemons [mon.cphprodc1-10,mon.cphprodc1-11,mon.cphprodc1-13] have slow ops. services: mon: 4 daemons, quorum cphprodc1-10,cphprodc1-11,cphprodc1-12,cphprodc1-13 (age 2h) mgr: cphprodc1-11.uuuhug(active, since 23h), standbys: cphprodc1-10.upwvbg mds: 1/1 daemons up, 1 standby osd: 64 osds: 19 up (since 2d), 47 in (since 23h) data: volumes: 0/1 healthy, 1 recovering pools: 5 pools, 113 pgs objects: 151.91k objects, 592 GiB usage: 840 GiB used, 81 TiB / 82 TiB avail pgs: 65 active+clean 36 stale+active+clean 7 active+clean+scrubbing 5 active+clean+scrubbing+deep osd tree: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 111.78223 root default -5 27.94556 host cphprodc1-10 1 ssd 1.74660 osd.1 down 1.00000 1.00000 5 ssd 1.74660 osd.5 down 1.00000 1.00000 8 ssd 1.74660 osd.8 down 1.00000 1.00000 12 ssd 1.74660 osd.12 down 1.00000 1.00000 14 ssd 1.74660 osd.14 down 1.00000 1.00000 18 ssd 1.74660 osd.18 down 0 1.00000 22 ssd 1.74660 osd.22 down 0 1.00000 26 ssd 1.74660 osd.26 down 0 1.00000 30 ssd 1.74660 osd.30 down 0 1.00000 34 ssd 1.74660 osd.34 down 1.00000 1.00000 37 ssd 1.74660 osd.37 down 1.00000 1.00000 41 ssd 1.74660 osd.41 down 1.00000 1.00000 45 ssd 1.74660 osd.45 up 1.00000 1.00000 48 ssd 1.74660 osd.48 up 1.00000 1.00000 52 ssd 1.74660 osd.52 up 1.00000 1.00000 56 ssd 1.74660 osd.56 up 1.00000 1.00000 -7 27.94556 host cphprodc1-11 2 ssd 1.74660 osd.2 down 0 1.00000 6 ssd 1.74660 osd.6 down 1.00000 1.00000 10 ssd 1.74660 osd.10 down 1.00000 1.00000 16 ssd 1.74660 osd.16 down 1.00000 1.00000 20 ssd 1.74660 osd.20 down 0 1.00000 24 ssd 1.74660 osd.24 down 0 1.00000 28 ssd 1.74660 osd.28 down 0 1.00000 32 ssd 1.74660 osd.32 down 0 1.00000 36 ssd 1.74660 osd.36 down 1.00000 1.00000 40 ssd 1.74660 osd.40 down 1.00000 1.00000 44 ssd 1.74660 osd.44 down 1.00000 1.00000 50 ssd 1.74660 osd.50 up 1.00000 1.00000 54 ssd 1.74660 osd.54 up 1.00000 1.00000 58 ssd 1.74660 osd.58 up 1.00000 1.00000 60 ssd 1.74660 osd.60 up 1.00000 1.00000 62 ssd 1.74660 osd.62 up 1.00000 1.00000 -3 27.94556 host cphprodc1-12 0 ssd 1.74660 osd.0 down 1.00000 1.00000 4 ssd 1.74660 osd.4 down 1.00000 1.00000 7 ssd 1.74660 osd.7 down 1.00000 1.00000 11 ssd 1.74660 osd.11 down 1.00000 1.00000 15 ssd 1.74660 osd.15 down 1.00000 1.00000 19 ssd 1.74660 osd.19 down 0 1.00000 23 ssd 1.74660 osd.23 down 0 1.00000 27 ssd 1.74660 osd.27 down 0 1.00000 31 ssd 1.74660 osd.31 down 0 1.00000 35 ssd 1.74660 osd.35 down 1.00000 1.00000 38 ssd 1.74660 osd.38 down 1.00000 1.00000 42 ssd 1.74660 osd.42 down 1.00000 1.00000 46 ssd 1.74660 osd.46 up 1.00000 1.00000 49 ssd 1.74660 osd.49 up 1.00000 1.00000 53 ssd 1.74660 osd.53 up 1.00000 1.00000 57 ssd 1.74660 osd.57 up 1.00000 1.00000 -9 27.94556 host cphprodc1-13 3 ssd 1.74660 osd.3 down 1.00000 1.00000 9 ssd 1.74660 osd.9 down 1.00000 1.00000 13 ssd 1.74660 osd.13 down 1.00000 1.00000 17 ssd 1.74660 osd.17 down 1.00000 1.00000 21 ssd 1.74660 osd.21 down 0 1.00000 25 ssd 1.74660 osd.25 down 0 1.00000 29 ssd 1.74660 osd.29 down 0 1.00000 33 ssd 1.74660 osd.33 down 0 1.00000 39 ssd 1.74660 osd.39 down 1.00000 1.00000 43 ssd 1.74660 osd.43 down 1.00000 1.00000 47 ssd 1.74660 osd.47 up 1.00000 1.00000 51 ssd 1.74660 osd.51 up 1.00000 1.00000 55 ssd 1.74660 osd.55 up 1.00000 1.00000 59 ssd 1.74660 osd.59 up 1.00000 1.00000 61 ssd 1.74660 osd.61 up 1.00000 1.00000 63 ssd 1.74660 osd.63 up 1.00000 1.00000

2 1

RGW multisite logs (data, md, bilog) not being trimmed automatically?
by Christian Rohmann 14 Sep '23

14 Sep '23

Hey ceph-users, I am running two (now) Quincy clusters doing RGW multi-site replication with only one actually being written to by clients. The other site is intended simply as a remote copy. On the primary cluster I am observing an ever growing (objects and bytes) "sitea.rgw.log" pool, not so on the remote "siteb.rgw.log" which is only 300MiB and around 15k objects with no growth. Metrics show that the growth of pool on primary is linear for at least 6 months, so not sudden spikes or anything. Also sync status appears to be totally happy. There are also no warnings in regards to large OMAPs or anything similar. I was under the impression that RGW will trim its three logs (md, bi, data) automatically and only keep data that has not yet been replicated by the other zonegroup members? The config option "ceph config get mgr rgw_sync_log_trim_interval" is set to 1200, so 20 Minutes. So I am wondering if there might be some inconsistency or how I can best analyze what the cause for the accumulation of log data is? There are older questions on the ML, such as [1], but there was not really a solution or root cause identified. I know there is manual trimming, but I'd rather want to analyze the current situation and figure out what the cause for the lack of auto-trimming is. * Do I need to go through all buckets and count logs and look at their timestamps? Which queries do make sense here? * Is there usually any logging of the log trimming activity that I should expect? Or that might indicate why trimming does not happen? Regards Christian [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/WZCFOAMLWV…

1 2

Re: Not able to find a standardized restoration procedure for subvolume snapshots.
by Lokendra Rathour 14 Sep '23

14 Sep '23

Hi Team, Facing a similar situation, Any help would be appreciated. Thanks once again for the support. -Lokendra On Tue, Sep 5, 2023 at 10:51 AM Kushagr Gupta <kushagrguptasps.mun(a)gmail.com> wrote: > *Ceph-version*: Quincy > *OS*: Centos 8 stream > > *Issue*: Not able to find a standardized restoration procedure for > subvolume snapshots. > > *Description:* > Hi team, > > We are currently working in a 3-node ceph cluster. > We are currently exploring the scheduled snapshot capability of the > ceph-mgr module. > To enable/configure scheduled snapshots, we followed the following link: > > https://docs.ceph.com/en/quincy/cephfs/snap-schedule/ > > The scheduled snapshots are working as expected. But we are unable to find > any standardized restoration procedure for the same. > > We have found the following link( not official documentation): > https://www.suse.com/support/kb/doc/?id=000019627 > > We have also found a link of cloning a new subvolume from snapshots: > https://docs.ceph.com/en/reef/cephfs/fs-volumes/ > (Section: Cloning Snapshots) > > Is there a standard procedure to restore from a snapshot. > By this I mean, is there some kind of command link maybe > ceph fs subvolume snapshot restore <snapshot-name> > > Or any other procedure please let us know. > > Thanks and Regards, > Kushagra Gupta > -- ~ Lokendra skype: lokendrarathour

1 0

Ceph services failing to start after OS upgrade
by hansen.ross＠live.com.au 14 Sep '23

14 Sep '23

Hi There, I have a ceph cluster running on my proxmox system and it all seemed to upgrade successfully however after the reboot my ceph-mon and my ceph-osd services are failing to start or are crashing by the looks of it. ``` ceph version 17.2.6 (810db68029296377607028a6c6da1ec06f5a2b27) quincy (stable) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x3bfd0) [0x7f10aba5afd0] 2: gf_init_hard() 3: gf_init_easy() 4: galois_init_default_field() 5: jerasure_init() 6: __erasure_code_init() 7: (ceph::ErasureCodePluginRegistry::load(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::ErasureCodePlugin**, std::ostream*)+0x2b5) [0x55b04c32c605] 8: (ceph::ErasureCodePluginRegistry::preload(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::ostream*)+0x9f) [0x55b04c32cbaf] 9: (global_init_preload_erasure_code(ceph::common::CephContext const*)+0x7c2) [0x55b04bdd9f92] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x271ca) [0x7f10aba461ca] 12: __libc_start_main() 13: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. ``` I am still quite new to Ceph and would like advice on how to troubleshoot this and get the services working again. Regards Ross

3 2

cannot create new OSDs - ceph version 17.2.6 (810db68029296377607028a6c6da1ec06f5a2b27) quincy (stable)
by Konold, Martin 13 Sep '23

13 Sep '23

Hi, I want to create a new OSD on a 4TB Samsung MZ1L23T8HBLA-00A07 enterprise nvme device in a hyper-converged proxmox 8 environment. Creating the OSD works but it cannot be initialized and therefore not started. In the log I see an entry about a failed assert. ./src/os/bluestore/fastbmap_allocator_impl.cc: 405: FAILED ceph_assert((aligned_extent.length % l0_granularity) == 0) Is this the culprit? In addition at the end of the logfile there is a failed mount and a failed osd init mentioned. 2023-09-11T16:30:04.708+0200 7f99aa28f3c0 -1 bluefs _check_allocations OP_FILE_UPDATE_INC invalid extent 1: 0x140000~10000: duplicate reference, ino 30 2023-09-11T16:30:04.708+0200 7f99aa28f3c0 -1 bluefs mount failed to replay log: (14) Bad address 2023-09-11T16:30:04.708+0200 7f99aa28f3c0 20 bluefs _stop_alloc 2023-09-11T16:30:04.708+0200 7f99aa28f3c0 -1 bluestore(/var/lib/ceph/osd/ceph-43) _open_bluefs failed bluefs mount: (14) Bad address 2023-09-11T16:30:04.708+0200 7f99aa28f3c0 10 bluefs maybe_verify_layout no memorized_layout in bluefs superblock 2023-09-11T16:30:04.708+0200 7f99aa28f3c0 -1 bluestore(/var/lib/ceph/osd/ceph-43) _open_db failed to prepare db environment: 2023-09-11T16:30:04.708+0200 7f99aa28f3c0 1 bdev(0x5565c261fc00 /var/lib/ceph/osd/ceph-43/block) close 2023-09-11T16:30:04.940+0200 7f99aa28f3c0 -1 osd.43 0 OSD:init: unable to mount object store 2023-09-11T16:30:04.940+0200 7f99aa28f3c0 -1 ** ERROR: osd init failed: (5) Input/output error I verified that the hardware of the new nvme is working fine. -- Regards, ppa. Martin Konold -- Viele Grüße ppa. Martin Konold -- Martin Konold - Prokurist, CTO KONSEC GmbH -⁠ make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany

2 3

MDS crash after Disaster Recovery
by Sasha BALLET 13 Sep '23

13 Sep '23

Hi, I'm struggling with a Ceph cluster that doesn't want to end recovery. I suspect there is multiple issues at the same time. So let's start with the most obvious, the MDS daemons are crashing and blocking me from mounting my CephFS. The cluster has been deployed on Debian Buster and later upgraded to Bullseye. There is 4 nodes of 64 OSDs of 12TB. I can't understand which Ceph version it is on because multiple services have different versions (another clue about what could be wrong). There is some 16.2.4, 16.2.5, 16.2.13 and even some 17.2.6 and I think I saw some 15.x.x on the OSDs. I have this log from the service: > journalctl -xe -u ceph-938952d4-7775-11eb-9f42-bc97e19a216a(a)mds.storage.cbi-storage-02.oaelhv.service [...] /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUIL> ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f850b1ed59c] 2: /usr/lib64/ceph/libceph-common.so.2(+0x2767b6) [0x7f850b1ed7b6] 3: (MDCache::journal_cow_dentry(MutationImpl*, EMetaBlob*, CDentry*, snapid_t, CInode**, CDentry::linkage_t*)+0x1215) [0x55768a6ad6b5] 4: (MDCache::journal_dirty_inode(MutationImpl*, EMetaBlob*, CInode*, snapid_t)+0x105) [0x55768a6ad965] 5: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0xc39) [0x55768a78fb09] 6: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x92d) [0x55768a76377d] 7: (MDSContext::complete(int)+0x56) [0x55768a8c0d46] 8: (MDSIOContextBase::complete(int)+0xa3) [0x55768a8c1073] 9: (Filer::C_Probe::finish(int)+0xb5) [0x55768a974af5] 10: (Context::complete(int)+0xd) [0x55768a5b4b6d] 11: (Finisher::finisher_thread_entry()+0x1a5) [0x7f850b28c9d5] 12: /lib64/libpthread.so.0(+0x814a) [0x7f8509f8f14a] 13: clone() debug 0> 2023-09-11T12:55:25.765+0000 7f84fc580700 -1 *** Caught signal (Aborted) ** in thread 7f84fc580700 thread_name:MR_Finisher ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 1: /lib64/libpthread.so.0(+0x12b20) [0x7f8509f99b20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f850b1ed5ed] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2767b6) [0x7f850b1ed7b6] 6: (MDCache::journal_cow_dentry(MutationImpl*, EMetaBlob*, CDentry*, snapid_t, CInode**, CDentry::linkage_t*)+0x1215) [0x55768a6ad6b5] 7: (MDCache::journal_dirty_inode(MutationImpl*, EMetaBlob*, CInode*, snapid_t)+0x105) [0x55768a6ad965] 8: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0xc39) [0x55768a78fb09] 9: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x92d) [0x55768a76377d] 10: (MDSContext::complete(int)+0x56) [0x55768a8c0d46] 11: (MDSIOContextBase::complete(int)+0xa3) [0x55768a8c1073] 12: (Filer::C_Probe::finish(int)+0xb5) [0x55768a974af5] 13: (Context::complete(int)+0xd) [0x55768a5b4b6d] 14: (Finisher::finisher_thread_entry()+0x1a5) [0x7f850b28c9d5] 15: /lib64/libpthread.so.0(+0x814a) [0x7f8509f8f14a] 16: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_pwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test 0/ 5 cephfs_mirror 0/ 5 cephsqlite -2/-2 (syslog threshold) 99/99 (stderr threshold) --- pthread ID / name mapping for recent threads --- 140209120859904 / 140209129252608 / md_submit 140209137645312 / 140209146038016 / MR_Finisher 140209162823424 / PQ_Finisher 140209213179648 / ceph-mds 140209229965056 / safe_timer 140209246750464 / ms_dispatch 140209263535872 / io_context_pool 140209280321280 / admin_socket 140209288713984 / msgr-worker-2 140209297106688 / msgr-worker-1 140209305499392 / msgr-worker-0 140209549580160 / ceph-mds max_recent 10000 max_new 10000 log_file /var/lib/ceph/crash/2023-09-11T12:55:25.769044Z_5229ea26-9bea-4c49-960a-4501b227e545/log --- end dump of recent events --- ceph-938952d4-7775-11eb-9f42-bc97e19a216a(a)mds.storage.cbi-storage-02.oaelhv.service: Main process exited, code=exited, status=134/n/a I tried the Advanced: Metadata Repair Tools https://docs.ceph.com/en/nautilus/cephfs/disaster-recovery-experts/ <https://docs.ceph.com/en/nautilus/cephfs/disaster-recovery-experts/> It goes, "recovery successfull" in the logs, then goes "has been put in standby", then crashes a few seconds or minutes later. I don't know what to do now without breaking everything Thank you for your help Let me know if you need any other informations Have a nice day, Sasha

2 1

Separating Mons and OSDs in Ceph Cluster
by Ramin Najjarbashi 12 Sep '23

12 Sep '23

Hi I am writing to seek guidance and best practices for a maintenance operation in my Ceph cluster. I have an older cluster in which the Monitors (Mons) and Object Storage Devices (OSDs) are currently deployed on the same host. I am interested in separating them while ensuring zero downtime and minimizing risks to the cluster's stability. The primary goal is to deploy new Monitors on different servers without causing service interruptions or disruptions to data availability. The challenge arises because updating the configuration to add new Monitors typically requires a restart of all OSDs, which is less than ideal in terms of maintaining cluster availability. One approach I considered is to reweight all OSDs on the host to zero, allowing data to gradually transfer to other OSDs. Once all data has been safely migrated, I would proceed to remove the old OSDs. Afterward, I would deploy the new Monitors on a different server with the previous IP addresses and deploy the OSDs on the old Monitors' host with new IP addresses. While this approach seems to minimize risks, it can be time-consuming and may not be the most efficient way to achieve the desired separation. I would greatly appreciate the community's insights and suggestions on the best approach to achieve this separation of Mons and OSDs with zero downtime and minimal risk. If there are alternative methods or best practices that can be recommended, please share your expertise.

6 7

[ceph v16.2.10] radosgw crash
by Louis Koo 12 Sep '23

12 Sep '23

2023-08-15T18:15:55.356+0000 7f7916ef3700 -1 *** Caught signal (Aborted) ** in thread 7f7916ef3700 thread_name:radosgw ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f79da065ce0] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f79d905809b] 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f79d905e53c] 6: /lib64/libstdc++.so.6(+0x96597) [0x7f79d905e597] 7: /lib64/libstdc++.so.6(+0x9652e) [0x7f79d905e52e] 8: (spawn::detail::continuation_context::resume()+0x87) [0x7f79e4b70a17] 9: (boost::asio::detail::executor_op<ceph::async::ForwardingHandler<ceph::async::CompletionHandler<spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, std::tuple<boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > > >, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x25a) [0x7f79e4b77b6a] 10: (boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>::operator()()+0x8d) [0x7f79e4b7f93d] 11: (boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>, boost::asio::detail::recycling_allocator<void, boost::asio::detail::thread_info_base::default_tag>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x96) [0x7f79e4b7fca6] 12: (boost::asio::detail::scheduler::run(boost::system::error_code&)+0x4f2) [0x7f79e4b73ad2] 13: /lib64/libradosgw.so.2(+0x430376) [0x7f79e4b56376] 14: /lib64/libstdc++.so.6(+0xc2ba3) [0x7f79d908aba3] 15: /lib64/libpthread.so.0(+0x81ca) [0x7f79da05b1ca] 16: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- -9999> 2023-08-15T18:14:25.987+0000 7f79186f6700 2 req 11141123620750988595 0.001000006s s3:get_obj init permissions -9998> 2023-08-15T18:14:25.987+0000 7f79186f6700 2 req 11141123620750988595 0.001000006s s3:get_obj recalculating target -9997> 2023-08-15T18:14:25.987+0000 7f79186f6700 2 req 11141123620750988595 0.001000006s s3:get_obj reading permissions -9996> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj init op -9995> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj verifying op mask -9994> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj verifying op permissions -9993> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=suliang, acct_name=suliang, subuser=, perm_mask=15, is_admin=0) mask=49 -9992> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Searching permissions for uid=suliang -9991> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Found permission: 15 -9990> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Searching permissions for group=1 mask=49 -9989> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Permissions for group not found -9988> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Searching permissions for group=2 mask=49 -9987> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Permissions for group not found -9986> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=suliang, acct_name=suliang, subuser=, perm_mask=15, is_admin=0), owner=suliang, perm=1 -9985> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj verifying op params -9984> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj pre-executing -9983> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj executing

3 3

MDS daemons don't report any more
by Frank Schilder 12 Sep '23

12 Sep '23

Hi all, I make a weird observation. 8 out of 12 MDS daemons seem not to report to the cluster any more: # ceph fs status con-fs2 - 1625 clients ======= RANK STATE MDS ACTIVITY DNS INOS 0 active ceph-16 Reqs: 0 /s 0 0 1 active ceph-09 Reqs: 128 /s 4251k 4250k 2 active ceph-17 Reqs: 0 /s 0 0 3 active ceph-15 Reqs: 0 /s 0 0 4 active ceph-24 Reqs: 269 /s 3567k 3567k 5 active ceph-11 Reqs: 0 /s 0 0 6 active ceph-14 Reqs: 0 /s 0 0 7 active ceph-23 Reqs: 0 /s 0 0 POOL TYPE USED AVAIL con-fs2-meta1 metadata 2169G 7081G con-fs2-meta2 data 0 7081G con-fs2-data data 1248T 4441T con-fs2-data-ec-ssd data 705G 22.1T con-fs2-data2 data 3172T 4037T STANDBY MDS ceph-08 ceph-10 ceph-12 ceph-13 VERSION DAEMONS None ceph-16, ceph-17, ceph-15, ceph-11, ceph-14, ceph-23, ceph-10, ceph-12 ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) ceph-09, ceph-24, ceph-08, ceph-13 Version is "none" for these and there are no stats. Ceph versions reports only 4 MDSes out of the 12. 8 are not shown at all: [root@gnosis ~]# ceph versions { "mon": { "ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)": 5 }, "mgr": { "ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)": 5 }, "osd": { "ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)": 1282 }, "mds": { "ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)": 4 }, "overall": { "ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)": 1296 } } Ceph status reports everything as up and OK: [root@gnosis ~]# ceph status cluster: id: e4ece518-f2cb-4708-b00f-b6bf511e91d9 health: HEALTH_OK services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 2w) mgr: ceph-03(active, since 61s), standbys: ceph-25, ceph-01, ceph-02, ceph-26 mds: con-fs2:8 4 up:standby 8 up:active osd: 1284 osds: 1282 up (since 31h), 1282 in (since 33h); 567 remapped pgs data: pools: 14 pools, 25065 pgs objects: 2.14G objects, 3.7 PiB usage: 4.7 PiB used, 8.4 PiB / 13 PiB avail pgs: 79908208/18438361040 objects misplaced (0.433%) 23063 active+clean 1225 active+clean+snaptrim_wait 317 active+remapped+backfill_wait 250 active+remapped+backfilling 208 active+clean+snaptrim 2 active+clean+scrubbing+deep io: client: 596 MiB/s rd, 717 MiB/s wr, 4.16k op/s rd, 3.04k op/s wr recovery: 8.7 GiB/s, 3.41k objects/s My first thought is that the status module failed. However, I don't manage to restart it (always on). An MGR fail-over did not help. Any ideas what is going on here? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 3