August 2023 - ceph-users

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

multifs and snapshots
by Dmitry Melekhov 12 Nov '24

12 Nov '24

Hello! There is note here https://docs.ceph.com/en/reef/dev/cephfs-snapshots/ about multiply filesystems and snapshots: If each FS gets its own pool things probably work, but this isn’t tested and may not be true. Is it still untested ? Thank you!

3 4

Help needed to configure erasure coding LRC plugin
by Michel Jouvin 16 Oct '24

16 Oct '24

Hi, As discussed in another thread (Crushmap rule for multi-datacenter erasure coding), I'm trying to create an EC pool spanning 3 datacenters (datacenters are present in the crushmap), with the objective to be resilient to 1 DC down, at least keeping the readonly access to the pool and if possible the read-write access, and have a storage efficiency better than 3 replica (let say a storage overhead <= 2). In the discussion, somebody mentioned LRC plugin as a possible jerasure alternative to implement this without tweaking the crushmap rule to implement the 2-step OSD allocation. I looked at the documentation (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/) but I have some questions if someone has experience/expertise with this LRC plugin. I tried to create a rule for using 5 OSDs per datacenter (15 in total), with 3 (9 in total) being data chunks and others being coding chunks. For this, based of my understanding of examples, I used k=9, m=3, l=4. Is it right? Is this configuration equivalent, in terms of redundancy, to a jerasure configuration with k=9, m=6? The resulting rule, which looks correct to me, is: -------- { "rule_id": 6, "rule_name": "test_lrc_2", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 15, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -4, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 5, "type": "host" }, { "op": "emit" } ] } ------------ Unfortunately, it doesn't work as expected: a pool created with this rule ends up with its pages active+undersize, which is unexpected for me. Looking at 'ceph health detail` output, I see for each page something like: pg 52.14 is stuck undersized for 27m, current state active+undersized, last acting [90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647] For each PG, there is 3 '2147483647' entries and I guess it is the reason of the problem. What are these entries about? Clearly it is not OSD entries... Looks like a negative number, -1, which in terms of crushmap ID is the crushmap root (named "default" in our configuration). Any trivial mistake I would have made? Thanks in advance for any help or for sharing any successful configuration? Best regards, Michel

4 24

Not all Bucket Shards being used
by Christian Kugler 01 Oct '24

01 Oct '24

Hi, I have trouble with large OMAP files in a cluster in the RGW index pool. Some background information about the cluster: There is CephFS and RBD usage on the main cluster but for this issue I think only S3 is interesting. There is one realm, one zonegroup with two zones which have a bidirectional sync set up. Since this does not allow for autoresharding we have to do it by hand in this cluster – looking forward to Reef! From the logs: cluster 2023-07-17T22:59:03.018722+0000 osd.75 (osd.75) 623978 : cluster [WRN] Large omap object found. Object: 34:bcec3016:::.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5:head PG: 34.680c373d (34.5) Key count: 962091 Size (bytes): 277963182 The offending bucket looks like this: # radosgw-admin bucket stats \ | jq '.[] | select(.marker =="3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9") |"\(.num_shards) \(.usage["rgw.main"].num_objects)"' -r 131 9463833 Last week the number of objects was about 12 million. Which is why I reshareded the offending bucket twice, I think. Once to 129 and the second time to 131 because I wanted some leeway (or lieway? scnr, Sage). Unfortunately, even after a week the objects were still to big (the log line above is quite recent), so I looked into it again. # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.0 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.1 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.2 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.3 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.4 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.6 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.7 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.8 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.9 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.10 # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V \ |xargs -IOMAP sh -c \ 'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l' 1013854 1011007 1012287 1011232 1013565 998262 1012777 1012713 1012230 1010690 997111 Apparently, only 11 shards are in use. This would explain why the "Key usage" (from the log line) is about ten times higher than I would expect. How can I deal with this issue? One thing I could try to fix this would be to reshard to a lower number, but I am not sure if there are any risks associated with "downsharding". After that I could reshard to something like 97. Or I could directly "downshard" to 97. Also, the second zone has a similar problem, but as the error messsage lets me know, this would be a bad idea. Will it just take more time until the sharding is transferred to the seconds zone? Best, Christian Kugler

2 5

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

Re: MDS crashes to damaged metadata
by Patrick Donnelly 10 Jun '24

10 Jun '24

On Thu, Dec 15, 2022 at 9:32 AM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): > > 2022-12-08T13:05:48.919+0100 7f440afec700 -1 /build/ceph-16.2.10/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f440afec700 time 2022-12-08T13:05:48.921223+0100 > /build/ceph-16.2.10/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE) > > 12 Minutes lates the unlink_local error crashes appeared again. This time with a new file. During debugging we noticed a MTU mismatch between MDS (1500) and client (9000) with cephfs kernel mount. The client is also creating the snapshots via mkdir in the .snap directory. > > We disabled snapshot creation for now, but really need this feature. I uploaded the mds logs of the first crash along with the information above to https://tracker.ceph.com/issues/38452 > > I would greatly appreciate it, if you could answer me the following question: > > Is the Bug related to our MTU Mismatch? We fixed the MTU Issue going back to 1500 on all nodes in the ceph public network on the weekend also. I doubt it. > If you need a debug level 20 log of the ScatterLock for further analysis, i could schedule snapshots at the end of our workdays and increase the debug level 5 Minutes arround snap shot creation. This would be very helpful! -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 4

Re: Recoveries without any misplaced objects?
by Hector Martin 24 Apr '24

24 Apr '24

On 29/05/2023 20.55, Anthony D'Atri wrote: > Check the uptime for the OSDs in question I restarted all my OSDs within the past 10 days or so. Maybe OSD restarts are somehow breaking these stats? > >> On May 29, 2023, at 6:44 AM, Hector Martin <marcan(a)marcan.st> wrote: >> >> Hi, >> >> I'm watching a cluster finish a bunch of backfilling, and I noticed that >> quite often PGs end up with zero misplaced objects, even though they are >> still backfilling. >> >> Right now the cluster is down to 6 backfilling PGs: >> >> data: >> volumes: 1/1 healthy >> pools: 6 pools, 268 pgs >> objects: 18.79M objects, 29 TiB >> usage: 49 TiB used, 25 TiB / 75 TiB avail >> pgs: 262 active+clean >> 6 active+remapped+backfilling >> >> But there are no misplaced objects, and the misplaced column in `ceph pg >> dump` is zero for all PGs. >> >> If I do a `ceph pg dump_json`, I can see `num_objects_recovered` >> increasing for these PGs... but the misplaced count is still 0. >> >> Is there something else that would cause recoveries/backfills other than >> misplaced objects? Or perhaps there is a bug somewhere causing the >> misplaced object count to be misreported as 0 sometimes? >> >> # ceph -v >> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy >> (stable) >> >> - Hector >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > - Hector

3 3

A couple OSDs not starting after host reboot
by Alison Peisker 11 Apr '24

11 Apr '24

Hi all, We rebooted all the nodes in our 17.2.5 cluster after performing kernel updates, but 2 of the OSDs on different nodes are not coming back up. This is a production cluster using cephadm. The error message from the OSD log is ceph-osd[87340]: ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-665: (2) No such file or directory The error message from ceph-volume is 2023-08-23T16:12:43.452-0500 7f0cad968600 2 bluestore(/dev/mapper/ceph--febad5a5--ba44--41aa--a39e--b9897f757752-osd--block--87e548f4--b9b5--4ed8--aca8--de703a341a50) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input We tried restarting the daemons and rebooting the node again, but still see the same error. Has anyone experienced this issue before? How do we fix this? Thanks, Alison

9 13

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

Why you might want packages not containers for Ceph deployments
by Matthew Vernon 25 Mar '24

25 Mar '24

Hi, In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0]. I'd like to suggest, briefly, that: * containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes. For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice. Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages): * you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice) We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part. Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems! I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder". I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :) Regards, and apologies for the wall of text, Matthew [0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

26 65

Upgrading from Pacific to Quincy fails with "Unexpected error"
by Reza Bakhshayeshi 13 Mar '24

13 Mar '24

Hi all, I have a problem regarding upgrading Ceph cluster from Pacific to Quincy version with cephadm. I have successfully upgraded the cluster to the latest Pacific (16.2.11). But when I run the following command to upgrade the cluster to 17.2.5, After upgrading 3/4 mgrs, the upgrade process stops with "Unexpected error". (everything is on a private network) ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v17.2.5 I also tried the 17.2.4 version. cephadm fails to check the hosts' status and marks them as offline: cephadm 2023-04-06T10:19:59.998510+0000 mgr.host9.arhpnd (mgr.4516356) 5782 : cephadm [DBG] host host4 (x.x.x.x) failed check cephadm 2023-04-06T10:19:59.998553+0000 mgr.host9.arhpnd (mgr.4516356) 5783 : cephadm [DBG] Host "host4" marked as offline. Skipping daemon refresh cephadm 2023-04-06T10:19:59.998581+0000 mgr.host9.arhpnd (mgr.4516356) 5784 : cephadm [DBG] Host "host4" marked as offline. Skipping gather facts refresh cephadm 2023-04-06T10:19:59.998609+0000 mgr.host9.arhpnd (mgr.4516356) 5785 : cephadm [DBG] Host "host4" marked as offline. Skipping network refresh cephadm 2023-04-06T10:19:59.998633+0000 mgr.host9.arhpnd (mgr.4516356) 5786 : cephadm [DBG] Host "host4" marked as offline. Skipping device refresh cephadm 2023-04-06T10:19:59.998659+0000 mgr.host9.arhpnd (mgr.4516356) 5787 : cephadm [DBG] Host "host4" marked as offline. Skipping osdspec preview refresh cephadm 2023-04-06T10:19:59.998682+0000 mgr.host9.arhpnd (mgr.4516356) 5788 : cephadm [DBG] Host "host4" marked as offline. Skipping autotune cluster 2023-04-06T10:20:00.000151+0000 mon.host8 (mon.0) 158587 : cluster [ERR] Health detail: HEALTH_ERR 9 hosts fail cephadm check; Upgrade: failed due to an unexpected exception cluster 2023-04-06T10:20:00.000191+0000 mon.host8 (mon.0) 158588 : cluster [ERR] [WRN] CEPHADM_HOST_CHECK_FAILED: 9 hosts fail cephadm check cluster 2023-04-06T10:20:00.000202+0000 mon.host8 (mon.0) 158589 : cluster [ERR] host host7 (x.x.x.x) failed check: Unable to reach remote host host7. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000213+0000 mon.host8 (mon.0) 158590 : cluster [ERR] host host2 (x.x.x.x) failed check: Unable to reach remote host host2. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000220+0000 mon.host8 (mon.0) 158591 : cluster [ERR] host host8 (x.x.x.x) failed check: Unable to reach remote host host8. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000228+0000 mon.host8 (mon.0) 158592 : cluster [ERR] host host4 (x.x.x.x) failed check: Unable to reach remote host host4. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000240+0000 mon.host8 (mon.0) 158593 : cluster [ERR] host host3 (x.x.x.x) failed check: Unable to reach remote host host3. Process exited with non-zero exit status 3 and here are some outputs of the commands: [root@host8 ~]# ceph -s cluster: id: xxx health: HEALTH_ERR 9 hosts fail cephadm check Upgrade: failed due to an unexpected exception services: mon: 5 daemons, quorum host8,host1,host7,host2,host9 (age 2w) mgr: host9.arhpnd(active, since 105m), standbys: host8.jowfih, host1.warjsr, host2.qyavjj mds: 1/1 daemons up, 3 standby osd: 37 osds: 37 up (since 8h), 37 in (since 3w) data: io: client: progress: Upgrade to 17.2.5 (0s) [............................] [root@host8 ~]# ceph orch upgrade status { "target_image": "my-private-repo/quay-io/ceph/ceph@sha256 :34c763383e3323c6bb35f3f2229af9f466518d9db926111277f5e27ed543c427", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "3/59 daemons upgraded", "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception", "is_paused": true } [root@host8 ~]# ceph cephadm check-host host7 check-host failed: Host 'host7' not found. Use 'ceph orch host ls' to see all managed hosts. [root@host8 ~]# ceph versions { "mon": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 1, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 }, "osd": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 37 }, "mds": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 4 }, "overall": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 47, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 } } The strange thing is I can rollback the cluster status by failing to not-upgraded mgr like this: ceph mgr fail ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v16.2.11 Would you happen to have any idea about this? Best regards, Reza

4 9

Upgrading nautilus / centos7 to octopus / ubuntu 20.04. - Suggestions and hints?
by Götz Reinicke 17 Jan '24

17 Jan '24

Hi, As I’v read and thought a lot about the migration as this is a bigger project, I was wondering if anyone has done that already and might share some notes or playbooks, because in all readings there where some parts missing or miss understandable to me. I do have some different approaches in mind, so may be you have some suggestions or hints. a) upgrade nautilus on centos 7 with the few missing features like dashboard and prometheus. After that migrate one node after an other to ubuntu 20.04 with octopus and than upgrade ceph to the recent stable version. b) migrate one node after an other to ubuntu 18.04 with nautilus and then upgrade to octupus and after that to ubuntu 20.04. or c) upgrade one node after an other to ubuntu 20.04 with octopus and join it to the cluster until all nodes are upgraded. For test I tried c) with a mon node, but adding that to the cluster fails with some failed state, still probing for the other mons. (I dont have the right log at hand right now.) So my questions are: a) What would be the best (most stable) migration path and b) is it in general possible to add a new octopus mon (not upgraded one) to a nautilus cluster, where the other mons are still on nautilus? I hope my thoughts and questions are understandable :) Thanks for any hint and suggestion. Best . Götz

7 11

Re: increasing number of (deep) scrubs
by Frank Schilder 13 Dec '23

13 Dec '23

Hi Dan, thanks for your answer. I don't have a problem with increasing osd_max_scrubs (=1 at the moment) as such. I would simply prefer a somewhat finer grained way of controlling scrubbing than just doubling or tripling it right away. Some more info. These 2 pools are data pools for a large FS. Unfortunately, we have a large percentage of small files, which is a pain for recovery and seemingly also for deep scrubbing. Our OSDs are about 25% used and I had to increase the warning interval already to 2 weeks. With all the warning grace parameters this means that we manage to deep scrub everything about every month. I need to plan for 75% utilisation and a 3 months period is a bit far on the risky side. Our data is to a large percentage cold data. Client reads will not do the check for us, we need to combat bit-rot pro-actively. The reasons I'm interested in parameters initiating more scrubs while also converting more scrubs into deep scrubs are, that 1) scrubs seem to complete very fast. I almost never catch a PG in state "scrubbing", I usually only see "deep scrubbing". 2) I suspect the low deep-scrub count is due to a low number of deep-scrubs scheduled and not due to conflicting per-OSD deep scrub reservations. With the OSD count we have and the distribution over 12 servers I would expect at least a peak of 50% OSDs being active in scrubbing instead of the 25% peak I'm seeing now. It ought to be possible to schedule more PGs for deep scrub than actually are. 3) Every OSD having only 1 deep scrub active seems to have no measurable impact on user IO. If I could just get more PGs scheduled with 1 deep scrub per OSD it would already help a lot. Once this is working, I can eventually increase osd_max_scrubs when the OSDs fill up. For now I would just like that (deep) scrub scheduling looks a bit harder and schedules more eligible PGs per time unit. If we can get deep scrubbing up to an average of 42PGs completing per hour with keeping osd_max_scrubs=1 to maintain current IO impact, we should be able to complete a deep scrub with 75% full OSDs in about 30 days. This is the current tail-time with 25% utilisation. I believe currently a deep scrub of a PG in these pools takes 2-3 hours. Its just a gut feeling from some repair and deep-scrub commands, I would need to check logs for more precise info. Increasing osd_max_scrubs would then be a further and not the only option to push for more deep scrubbing. My expectation would be that values of 2-3 are fine due to the increasingly higher percentage of cold data for which no interference with client IO will happen. Hope that makes sense and there is a way beyond bumping osd_max_scrubs to increase the number of scheduled and executed deep scrubs. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dvanders(a)gmail.com> Sent: 05 January 2023 15:36 To: Frank Schilder Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] increasing number of (deep) scrubs Hi Frank, What is your current osd_max_scrubs, and why don't you want to increase it? With 8+2, 8+3 pools each scrub is occupying the scrub slot on 10 or 11 OSDs, so at a minimum it could take 3-4x the amount of time to scrub the data than if those were replicated pools. If you want the scrub to complete in time, you need to increase the amount of scrub slots accordingly. On the other hand, IMHO the 1-week deadline for deep scrubs is often much too ambitious for large clusters -- increasing the scrub intervals is one solution, or I find it simpler to increase mon_warn_pg_not_scrubbed_ratio and mon_warn_pg_not_deep_scrubbed_ratio until you find a ratio that works for your cluster. Of course, all of this can impact detection of bit-rot, which anyway can be covered by client reads if most data is accessed periodically. But if the cluster is mostly idle or objects are generally not read, then it would be preferable to increase slots osd_max_scrubs. Cheers, Dan On Tue, Jan 3, 2023 at 2:30 AM Frank Schilder <frans(a)dtu.dk> wrote: > > Hi all, > > we are using 16T and 18T spinning drives as OSDs and I'm observing that they are not scrubbed as often as I would like. It looks like too few scrubs are scheduled for these large OSDs. My estimate is as follows: we have 852 spinning OSDs backing a 8+2 pool with 2024 and an 8+3 pool with 8192 PGs. On average I see something like 10PGs of pool 1 and 12 PGs of pool 2 (deep) scrubbing. This amounts to only 232 out of 852 OSDs scrubbing and seems to be due to a conservative rate of (deep) scrubs being scheduled. The PGs (dep) scrub fairly quickly. > > I would like to increase gently the number of scrubs scheduled for these drives and *not* the number of scrubs per OSD. I'm looking at parameters like: > > osd_scrub_backoff_ratio > osd_deep_scrub_randomize_ratio > > I'm wondering if lowering osd_scrub_backoff_ratio to 0.5 and, maybe, increasing osd_deep_scrub_randomize_ratio to 0.2 would have the desired effect? Are there other parameters to look at that allow gradual changes in the number of scrubs going on? > > Thanks a lot for your help! > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 3

MDS stuck in rejoin
by Frank Schilder 10 Nov '23

10 Nov '23

Hi all, we had a client with the warning "[WRN] MDS_CLIENT_OLDEST_TID: 1 clients failing to advance oldest client/flush tid". I looked at the client and there was nothing going on, so I rebooted it. After the client was back, the message was still there. To clean this up I failed the MDS. Unfortunately, the MDS that took over is remained stuck in rejoin without doing anything. All that happened in the log was: [root@ceph-10 ceph]# tail -f ceph-mds.ceph-10.log 2023-07-20T15:54:29.147+0200 7fedb9c9f700 1 mds.2.896604 rejoin_start 2023-07-20T15:54:29.161+0200 7fedb9c9f700 1 mds.2.896604 rejoin_joint_start 2023-07-20T15:55:28.005+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896614 from mon.4 2023-07-20T15:56:00.278+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896615 from mon.4 [...] 2023-07-20T16:02:54.935+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896653 from mon.4 2023-07-20T16:03:07.276+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896654 from mon.4 After some time I decided to give another fail a try and, this time, the replacement daemon went to active state really fast. If I have a message like the above, what is the clean way of getting the client clean again (version: 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable))? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 16

Re: RGW access logs with bucket name
by Boris Behrens 02 Nov '23

02 Nov '23

Bringing up that topic again: is it possible to log the bucket name in the rgw client logs? currently I am only to know the bucket name when someone access the bucket via https://TLD/bucket/object instead of https://bucket.TLD/object. Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens <bb(a)kervyn.de>: > Hi, > I am looking forward to move our logs from > /var/log/ceph/ceph-client...log to our logaggregator. > > Is there a way to have the bucket name in the log file? > > Or can I write the rgw_enable_ops_log into a file? Maybe I could work with > this. > > Cheers and happy new year > Boris > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

4 6

Re: Status of Quincy 17.2.5 ?
by Christian Rohmann 30 Oct '23

30 Oct '23

Hey everyone, On 20/10/2022 10:12, Christian Rohmann wrote: > 1) May I bring up again my remarks about the timing: > > On 19/10/2022 11:46, Christian Rohmann wrote: > >> I believe the upload of a new release to the repo prior to the >> announcement happens quite regularly - it might just be due to the >> technical process of releasing. >> But I agree it would be nice to have a more "bit flip" approach to >> new releases in the repo and not have the packages appear as updates >> prior to the announcement and final release and update notes. > By my observations sometimes there are packages available on the > download servers via the "last stable" folders such as > https://download.ceph.com/debian-quincy/ quite some time before the > announcement of a release is out. > I know it's hard to time this right with mirrors requiring some time > to sync files, but would be nice to not see the packages or have > people install them before there are the release notes and potential > pointers to changes out. Todays 16.2.11 release shows the exact issue I described above .... 1) 16.2.11 packages are already available via e.g. https://download.ceph.com/debian-pacific 2) release notes not yet merged: (https://github.com/ceph/ceph/pull/49839) thus https://ceph.io/en/news/blog/2022/v16-2-11-pacific-released/ show a 404 :-) 3) No announcement like https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/QOCU563UD3… to the ML yet. Regards Christian

2 2

[Pacific] ceph orch device ls do not returns any HDD
by Patrick Begou 27 Oct '23

27 Oct '23

Hi everyone I'm new to CEPH, just a french 4 days training session with Octopus on VMs that convince me to build my first cluster. At this time I have 4 old identical nodes for testing with 3 HDDs each, 2 network interfaces and running Alma Linux8 (el8). I try to replay the training session but it fails, breaking the web interface because of some problems with podman 4.2 not compatible with Octopus. So I try to deploy Pacific with cephadm tool on my first node (mostha1) (to enable testing also an upgrade later). dnf -y install https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noar… monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }') cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \ --initial-dashboard-user admceph \ --allow-fqdn-hostname --cluster-network 10.1.0.0/16 This was sucessfull. But running "*c**eph orch device ls*" do not show any HDD even if I have /dev/sda (used by the OS), /dev/sdb and /dev/sdc The web interface shows a row capacity which is an aggregate of the sizes of the 3 HDDs for the node. I've also tried to reset /dev/sdb but cephadm do not see it: [ceph: root@mostha1 /]# ceph orch device zap mostha1.legi.grenoble-inp.fr /dev/sdb --force Error EINVAL: Device path '/dev/sdb' not found on host 'mostha1.legi.grenoble-inp.fr' On my first attempt with octopus, I was able to list the available HDD with this command line. Before moving to Pacific, the OS on this node has been reinstalled from scratch. Any advices for a CEPH beginner ? Thanks Patrick

9 51

CephFS: convert directory into subvolume
by Eugen Block 12 Oct '23

12 Oct '23

Hi, while writing a response to [1] I tried to convert an existing directory within a single cephfs into a subvolume. According to [2] that should be possible, I'm just wondering how to confirm that it actually worked. Because setting the xattr works fine, the directory just doesn't show up in the subvolume ls command. This is what I tried (in Reef and Pacific): # one "regular" subvolume already exists $ ceph fs subvolume ls cephfs [ { "name": "subvol1" } ] # mounted / and created new subdir $ mkdir /mnt/volumes/subvol2 $ setfattr -n ceph.dir.subvolume -v 1 /mnt/volumes/subvol2 # still only one subvolume $ ceph fs subvolume ls cephfs [ { "name": "subvol1" } ] I also tried it directly underneath /mnt: $ mkdir /mnt/subvol2 $ setfattr -n ceph.dir.subvolume -v 1 /mnt/subvol2 But still no subvolume2 available. What am I missing here? Thanks Eugen [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/G4ZWGGUPPFQ… [2] https://www.spinics.net/lists/ceph-users/msg72341.html

3 7

ceph-dashboard python warning with new pyo3 0.17 lib (debian12)
by DERUMIER, Alexandre 11 Oct '23

11 Oct '23

Hi, on debian12, ceph-dashboard is throwing a warning "Module 'dashboard' has failed dependency: PyO3 modules may only be initialized once per interpreter process" Seem to be related to pyo3 0.17 change https://github.com/PyO3/pyo3/blob/7bdc504252a2f972ba3490c44249b202a4ce6180/… " Each #[pymodule] can now only be initialized once per process To make PyO3 modules sound in the presence of Python sub-interpreters, for now it has been necessary to explicitly disable the ability to initialize a #[pymodule] more than once in the same process. Attempting to do this will now raise an ImportError. "

3 3

radosgw-admin sync error trim seems to do nothing
by Richard Bade 03 Oct '23

03 Oct '23

Hi Matthew, At least for nautilus (14.2.22) i have discovered through trial and error that you need to specify a beginning or end date. Something like this: radosgw-admin sync error trim --end-date="2023-08-20 23:00:00" --rgw-zone={your_zone_name} I specify the zone as there's a error list for each zone. Hopefully that helps. Rich ------------------------------ Date: Sat, 19 Aug 2023 12:48:55 -0400 From: Matthew Darwin <bugs(a)mdarwin.ca> Subject: [ceph-users] radosgw-admin sync error trim seems to do nothing To: Ceph Users <ceph-users(a)ceph.io> Message-ID: <95e7edfd-ca29-fc0e-a30a-987f1c43e2d4(a)mdarwin.ca> Content-Type: text/plain; charset=UTF-8; format=flowed Hello all, "radosgw-admin sync error list" returns errors from 2022. I want to clear those out. I tried "radosgw-admin sync error trim" but it seems to do nothing. The man page seems to offer no suggestions https://protect-au.mimecast.com/s/26o0CzvkGRhLoOXfXjZR3?domain=docs.ceph.com Any ideas what I need to do to remove old errors? (or at least I want to see more recent errors) ceph version 17.2.6 (quincy) Thanks.

2 2

Quincy NFS ingress failover
by Thorne Lawler 27 Sep '23

27 Sep '23

Fellow cephalopods, I'm trying to get quick, seamless NFS failover happening on my four-node Ceph cluster. I followed the instructions here: https://docs.ceph.com/en/latest/cephadm/services/nfs/#high-availability-nfs but testing shows that failover doesn't happen. When I placed node 2 ("san2") in maintenance mode, the NFS service shut down: Aug 24 14:19:03 san2 ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66-nfs-xcpnfs-1-0-san2-datsvq[1962479]: 24/08/2023 04:19:03 : epoch 64b8af5a : san2 : ganesha.nfsd-8[Admin] do_shutdown :MAIN :EVENT :Removing all exports. Aug 24 14:19:13 san2 bash[3235994]: time="2023-08-24T14:19:13+10:00" level=warning msg="StopSignal SIGTERM failed to stop container ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66-nfs-xcpnfs-1-0-san2-datsvq in 10 seconds, resorting to SIGKILL" Aug 24 14:19:13 san2 bash[3235994]: ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66-nfs-xcpnfs-1-0-san2-datsvq Aug 24 14:19:13 san2 systemd[1]:ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66@nfs.xcpnfs.1.0.san2.datsvq.servic <mailto:ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66@nfs.xcpnfs.1.0.san2.datsvq.servic>e: Main process exited, code=exited, status=137/n/a Aug 24 14:19:14 san2 systemd[1]:ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66@nfs.xcpnfs.1.0.san2.datsvq.servic <mailto:ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66@nfs.xcpnfs.1.0.san2.datsvq.servic>e: Failed with result 'exit-code'. Aug 24 14:19:14 san2 systemd[1]: Stopped Ceph nfs.xcpnfs.1.0.san2.datsvq for e2f1b934-ed43-11ec-80fa-04421a1a1d66. And that's it. The ingress IP didn't move. More odd, the cluster seems to have placed the ingress IP on node 1 (san1) but seems to be using the NFS service on node 2. Do I need to more tightly connect the NFS service to the keepalive and haproxy services, or do I need to expand the ingress services to refer to multiple NFS services? Thank you. -- Regards, Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170 _DDNS /_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

3 9

libceph: mds1 IP+PORT wrong peer at address
by Frank Schilder 20 Sep '23

20 Sep '23

Hi all, we seem to have hit a bug in the ceph fs kernel client and I just want to confirm what action to take. We get the error "wrong peer at address" in dmesg and some jobs on that server seem to get stuck in fs access; log extract below. I found these 2 tracker items https://tracker.ceph.com/issues/23883 and https://tracker.ceph.com/issues/41519, which don't seem to have fixes. My questions: - Is this harmless or does it indicate invalid/corrupted client cache entries? - How to resolve, ignore, umount+mount or reboot? Here an extract from the dmesg log, the error has survived a couple of MDS restarts already: [Mon Mar 6 12:56:46 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:05:18 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-1572619386 [Mon Mar 6 13:05:18 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:13:50 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-1572619386 [Mon Mar 6 13:13:50 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state OPEN) [Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state OPEN) [Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start [Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start [Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success [Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success [Mon Mar 6 13:18:13 2023] ceph: update_snap_trace error -22 [Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state OPEN) [Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state OPEN) [Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed [Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect success [Mon Mar 6 13:18:29 2023] ceph: mds7 reconnect success [Mon Mar 6 13:18:35 2023] ceph: update_snap_trace error -22 [Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed [Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed [Mon Mar 6 13:22:22 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Mon Mar 6 13:22:22 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:30:54 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [...] [Thu Mar 9 09:37:24 2023] slurm.epilog.cl (31457): drop_caches: 3 [Thu Mar 9 09:38:26 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:38:26 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 09:46:58 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:46:58 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 09:55:30 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:55:30 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 10:04:02 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 10:04:02 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 6

Quincy 17.2.6 - Rados gateway crash -
by xadhoom76＠gmail.com 19 Sep '23

19 Sep '23

Hi, we have service that is still crashing when S3 client (veeam backup) start to write data main log from rgw service req 13170422438428971730 0.008000086s s3:get_obj WARNING: couldn't find acl header for object, generating default 2023-07-20T14:36:45.331+0000 7fa5adb4c700 -1 *** Caught signal (Aborted) ** And " 2023-07-19T22:04:15.968+0000 7ff07305b700 1 beast: 0x7fefc7178710: 172.16.199.11 - veeam90 [19/Jul/2023:22:04:15.948 +0000] "PUT /veeam90/Veeam/Backu p/veeam90/Clients/%7Bd14cd688-57b4-4809-a1d9-14cafd191b11%7D/34387bbd-bec9-4a40-a04d-6a890d5d6407/CloudStg/Data/%7Bf687ee0f-fb50-4ded-b3a8-3f67ca7f244 b%7D/%7B6f31c277-734c-46fd-98d5-c560aa6dc776%7D/144113_f3fd31c9ee2a45aeeadda0de3cbc9064_00000000000000000000000000000000 HTTP/1.1" 200 63422 - "APN/1. 0 Veeam/1.0 Backup/12.0" - latency=0.020000216s 2023-07-19T22:04:15.972+0000 7ff08307b700 1 ====== starting new request req=0x7fefc7682710 ===== 2023-07-19T22:04:15.972+0000 7ff087083700 1 ====== starting new request req=0x7fefc737c710 ===== 2023-07-19T22:04:15.972+0000 7ff071057700 1 ====== starting new request req=0x7fefc72fb710 ===== 2023-07-19T22:04:15.972+0000 7ff0998a8700 1 ====== starting new request req=0x7fefc71f9710 ===== 2023-07-19T22:04:15.972+0000 7fefe473e700 -1 *** Caught signal (Aborted) ** in thread 7fefe473e700 thread_name:radosgw ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7ff102d62cf0] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9009b) [0x7ff101d5209b] 5: /lib64/libstdc++.so.6(+0x9653c) [0x7ff101d5853c] 6: /lib64/libstdc++.so.6(+0x95559) [0x7ff101d57559] 7: __gxx_personality_v0() 8: /lib64/libgcc_s.so.1(+0x10b03) [0x7ff101736b03] 9: _Unwind_Resume() 10: /lib64/libradosgw.so.2(+0x538c5b) [0x7ff105246c5b] ---------------------- -- -10> 2023-07-19T22:04:15.972+0000 7ff071057700 2 req 8167590275148061076 0.000000000s s3:put_obj pre-executing -9> 2023-07-19T22:04:15.972+0000 7ff071057700 2 req 8167590275148061076 0.000000000s s3:put_obj check rate limiting -8> 2023-07-19T22:04:15.972+0000 7ff071057700 2 req 8167590275148061076 0.000000000s s3:put_obj executing -7> 2023-07-19T22:04:15.972+0000 7ff0998a8700 1 ====== starting new request req=0x7fefc71f9710 ===== -6> 2023-07-19T22:04:15.972+0000 7ff0998a8700 2 req 15658207768827051601 0.000000000s initializing for trans_id = tx00000d94d21014832be51-0064b85 ddf-3dfe-backup -5> 2023-07-19T22:04:15.972+0000 7ff0998a8700 2 req 15658207768827051601 0.000000000s getting op 1 -4> 2023-07-19T22:04:15.972+0000 7ff0998a8700 2 req 15658207768827051601 0.000000000s s3:put_obj verifying requester -3> 2023-07-19T22:04:15.972+0000 7ff0998a8700 2 req 15658207768827051601 0.000000000s s3:put_obj normalizing buckets and tenants -2> 2023-07-19T22:04:15.972+0000 7ff0998a8700 2 req 15658207768827051601 0.000000000s s3:put_obj init permissions -1> 2023-07-19T22:04:15.972+0000 7ff011798700 2 req 15261257039771290446 0.024000257s s3:put_obj completing 0> 2023-07-19T22:04:15.972+0000 7fefe473e700 -1 *** Caught signal (Aborted) ** in thread 7fefe473e700 thread_name:radosgw " Anyone have this issue ? Thanks

5 5

RGW multisite logs (data, md, bilog) not being trimmed automatically?
by Christian Rohmann 14 Sep '23

14 Sep '23

Hey ceph-users, I am running two (now) Quincy clusters doing RGW multi-site replication with only one actually being written to by clients. The other site is intended simply as a remote copy. On the primary cluster I am observing an ever growing (objects and bytes) "sitea.rgw.log" pool, not so on the remote "siteb.rgw.log" which is only 300MiB and around 15k objects with no growth. Metrics show that the growth of pool on primary is linear for at least 6 months, so not sudden spikes or anything. Also sync status appears to be totally happy. There are also no warnings in regards to large OMAPs or anything similar. I was under the impression that RGW will trim its three logs (md, bi, data) automatically and only keep data that has not yet been replicated by the other zonegroup members? The config option "ceph config get mgr rgw_sync_log_trim_interval" is set to 1200, so 20 Minutes. So I am wondering if there might be some inconsistency or how I can best analyze what the cause for the accumulation of log data is? There are older questions on the ML, such as [1], but there was not really a solution or root cause identified. I know there is manual trimming, but I'd rather want to analyze the current situation and figure out what the cause for the lack of auto-trimming is. * Do I need to go through all buckets and count logs and look at their timestamps? Which queries do make sense here? * Is there usually any logging of the log trimming activity that I should expect? Or that might indicate why trimming does not happen? Regards Christian [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/WZCFOAMLWV…

1 2

[ceph v16.2.10] radosgw crash
by Louis Koo 12 Sep '23

12 Sep '23

2023-08-15T18:15:55.356+0000 7f7916ef3700 -1 *** Caught signal (Aborted) ** in thread 7f7916ef3700 thread_name:radosgw ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f79da065ce0] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f79d905809b] 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f79d905e53c] 6: /lib64/libstdc++.so.6(+0x96597) [0x7f79d905e597] 7: /lib64/libstdc++.so.6(+0x9652e) [0x7f79d905e52e] 8: (spawn::detail::continuation_context::resume()+0x87) [0x7f79e4b70a17] 9: (boost::asio::detail::executor_op<ceph::async::ForwardingHandler<ceph::async::CompletionHandler<spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, std::tuple<boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > > >, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x25a) [0x7f79e4b77b6a] 10: (boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>::operator()()+0x8d) [0x7f79e4b7f93d] 11: (boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>, boost::asio::detail::recycling_allocator<void, boost::asio::detail::thread_info_base::default_tag>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x96) [0x7f79e4b7fca6] 12: (boost::asio::detail::scheduler::run(boost::system::error_code&)+0x4f2) [0x7f79e4b73ad2] 13: /lib64/libradosgw.so.2(+0x430376) [0x7f79e4b56376] 14: /lib64/libstdc++.so.6(+0xc2ba3) [0x7f79d908aba3] 15: /lib64/libpthread.so.0(+0x81ca) [0x7f79da05b1ca] 16: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- -9999> 2023-08-15T18:14:25.987+0000 7f79186f6700 2 req 11141123620750988595 0.001000006s s3:get_obj init permissions -9998> 2023-08-15T18:14:25.987+0000 7f79186f6700 2 req 11141123620750988595 0.001000006s s3:get_obj recalculating target -9997> 2023-08-15T18:14:25.987+0000 7f79186f6700 2 req 11141123620750988595 0.001000006s s3:get_obj reading permissions -9996> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj init op -9995> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj verifying op mask -9994> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj verifying op permissions -9993> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=suliang, acct_name=suliang, subuser=, perm_mask=15, is_admin=0) mask=49 -9992> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Searching permissions for uid=suliang -9991> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Found permission: 15 -9990> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Searching permissions for group=1 mask=49 -9989> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Permissions for group not found -9988> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Searching permissions for group=2 mask=49 -9987> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj Permissions for group not found -9986> 2023-08-15T18:14:25.988+0000 7f79186f6700 5 req 11141123620750988595 0.002000013s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=suliang, acct_name=suliang, subuser=, perm_mask=15, is_admin=0), owner=suliang, perm=1 -9985> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj verifying op params -9984> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj pre-executing -9983> 2023-08-15T18:14:25.988+0000 7f79186f6700 2 req 11141123620750988595 0.002000013s s3:get_obj executing

3 3

Re: Debian/bullseye build for reef
by Josh Durgin 07 Sep '23

07 Sep '23

We weren't targeting bullseye once we discovered the compiler version problem, the focus shifted to bookworm. If anyone would like to help maintaining debian builds, or looking into these issues, it would be welcome: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1030129 https://tracker.ceph.com/issues/61845 On Mon, Aug 21, 2023 at 7:50 AM Matthew Darwin <bugs(a)mdarwin.ca> wrote: > Thanks for the link to the issue. Any reason it wasn't added to the > release notes (for bullseye). > > I am also waiting for this to be available to start testing. > On 2023-08-21 10:25, Josh Durgin wrote: > > There was difficulty building on bullseye due to the older version of GCC > available: https://tracker.ceph.com/issues/61845 > > On Mon, Aug 21, 2023 at 3:01 AM Chris Palmer <chris.palmer(a)idnet.com> <chris.palmer(a)idnet.com> wrote: > > > I'd like to try reef, but we are on debian 11 (bullseye). > In the ceph repos, there is debian-quincy/bullseye and > debian-quincy/focal, but under reef there is only focal & jammy. > > Is there a reason why there is no reef/bullseye build? I had thought > that the blocker only affected debian-bookworm builds. > > Thanks, Chris > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

2 2

lack of RGW_API_HOST in ceph dashboard, 17.2.6, causes ceph mgr dashboard problems
by Christopher Durham 05 Sep '23

05 Sep '23

Hi, I am using 17.2.6 on Rocky Linux 8 The ceph mgr dashboard, in my situation, (bare metal install, upgraded from 15->16-> 17.2.6), can no longer hit the ObjectStore->(Daemons,Users,Buckets) pages. When I try to hit those pages, it gives an error: RGW REST API failed request with status code 403 {"Code": "AccessDenied", RequestId: "xxxxxxx", HostId: "yyyy-<my zone>"} The log of the rgw server it hit has: "GET /admin/metadata/user?myself HTTP/1.1" 403 125 It appears that the mgr dashboard setting RGW_API_HOST is no longer an option that can be set, nor does that name exist anywhere under /usr/share/ceph/mgr/dashboard, and: # ceph dashboard set-rgw-api-host <host> is no longer in existence in 17.2.6 However, since my situation is an upgrade, the config value still exists in my config, and I can retrieve it with: # ceph dashboard get-rgw-api-host To get the to work in my situation, I have modified /usr/share/ceph/mgr/dashboard/settings.py and re-added RGW_API_HOST to the Options class using RGW_API_HOST = Settings('', [dict,str]) I then modified /usr/share/ceph/mgr/dashboard/services/rgw_request.py such that each rgw daemon retrieved has its 'host' member set to Settings.RGW_API_HOST. Then after restarting the mgr, I was able to access the Objectstore->(Daemons,Users,Buckets) pages in the dashboard. HOWEVER, I know this is NOT the right way to fix this, it is a hack. It seems like the dashboard is trying to contact an rgw server individually. For us, the RGW_API_HOST is a name in DNS: s3.my.dom, that has multiple A records, one for each of our rgw servers, each of which have the *same* SSL cert with CN and SubjectAltNames that allow the cert to present itself as both s3.my.dom as well as the individual host name (SubjectAltName has ALL the rgw servers in it). This works well for us and has done so since 15.x.y, The endpoint for the zone is set to s3.my.dom. Thus my users only have a single endpoint to care about, unless there is a failure situation onan rgw server. (We have other ways of handling that). Any thoughts on the CORRECT way to handle this so I can have the ceph dashboard work with the ObjectStore->(Daemons,Users,Buckets) pages? Thanks. -Chris

2 2

rgw replication sync issue
by ankit raikwar 04 Sep '23

04 Sep '23

Hello Users, We have the environment as below. Both environments are the zones of one RGW multisite zonegroup, whereas the DC zone is the primary and the DR zone is the secondary at this point. DC Ceph Version: 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Number of rgw daemons : 25 DR Ceph Version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Number of rgw daemons : 25 Environment description: Both the mentioned zones are in production and the RGW multisite bandwidth is over MPLS of around 3 Gbps. Issue description : We have enabled the multisite between DC-DR almost around a month ago. The total data at the DC zone is around 159 TiB and the sync has been going as expected . But when the sync had gone around 120 TiB we saw the speed drastically fell low, the ideal was around 2 Gbps, and it fell way below 10 Mbps though the link is not saturated. After checking "# radosgw-admin sync status " the output says "metadata is caught up with master" and "data is caught up with source" but with almost 25 TB data behind as compared to DC. It also looks like the sync status of the bucket " radosgw-admin bucket sync status --bucket=<bucket-name>" still bucket is behind shards. Attaching the log and the output below. The possibility of issuing a resync of the data from the beginning is quite low and not feasible in our case. The "# radosgw-admin sync error list" output is also attached with some information redacted and we see errors. radosgw-sync status radosgw-admin sync status realm 6a7fab77-64e3-453e-b54b-066bc8af2f00 (realm0) zonegroup be660604-d853-4f8e-a576-579cae2e07c2 (zg0) zone d06a8dd3-5bcb-486c-945b-2a98969ccd5f (fbd) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: d09d3d16-8601-448b-bf3d-609b8a29647d (ahd) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source radosgw-admin bucket sync status --bucket=<bucket-name> realm 6a7fab77-64e3-453e-b54b-066bc8af2f00 (realm0) zonegroup be660604-d853-4f8e-a576-579cae2e07c2 (zg0) zone d06a8dd3-5bcb-486c-945b-2a98969ccd5f (fbd) bucket :tc******rc-b1[d09d3d16-8601-448b-bf3d-609b8a29647d.38987.1]) source zone d09d3d16-8601-448b-bf3d-609b8a29647d (ahd) source bucket :tc*******arc-b1[d09d3d16-8601-448b-bf3d-609b8a29647d.38987.1]) full sync: 14/9221 shards full sync: 49448693 objects completed incremental sync: 9207/9221 shards bucket is behind on 25 shards behind shards: [9,111,590,826,1774,2968,3132,3382,3386,3409,3685,3820,4174,4544,4708,4811,5733,6285,6558,7288,7417,7443,7876,8151,8878] Error: radosgw-admin sync error list "id": "1_1690799008.725414_3926410.1", "section": "data", "name": "bucket0:d09d3d16-8601-448b-bf3d-609b8a29647d.89871.1:1949", "timestamp": "2023-07-31T10:23:28.725414Z", "info": { "source_zone": "d09d3d16-8601-448b-bf3d-609b8a29647d", "error_code": 125, "message": "failed to sync bucket instance: (125) Operation canceled" "id": "1_1690804503.144829_3759212.1", "section": "data", "name": "bucket1:d09d3d16-8601-448b-bf3d-609b8a29647d.38987.1:1232/S01/1/120/2b7ea802-efad-41d3-9d90-9**************523.txt", "timestamp": "2023-07-31T11:54:53.233451Z", "info": { "source_zone": "d09d3d16-8601-448b-bf3d-609b8a29647d", "error_code": 5, "message": "failed to sync object(5) Input/output error" Thanks Ankit

2 3

Is it safe to add different OS but same ceph version to the existing cluster?
by Szabo, Istvan (Agoda) 04 Sep '23

04 Sep '23

Hi, I have an octopus cluster on the latest octopus version with mgr/mon/rgw/osds on centos 8. Is it safe to add an ubuntu osd host with the same octopus version? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 3

Client failing to respond to capability release
by Frank Schilder 01 Sep '23

01 Sep '23

Hi all, I have this warning the whole day already (octopus latest cluster): HEALTH_WARN 4 clients failing to respond to capability release; 1 pgs not deep-scrubbed in time [WRN] MDS_CLIENT_LATE_RELEASE: 4 clients failing to respond to capability release mds.ceph-24(mds.1): Client sn352.hpc.ait.dtu.dk:con-fs2-hpc failing to respond to capability release client_id: 145698301 mds.ceph-24(mds.1): Client sn463.hpc.ait.dtu.dk:con-fs2-hpc failing to respond to capability release client_id: 189511877 mds.ceph-24(mds.1): Client sn350.hpc.ait.dtu.dk:con-fs2-hpc failing to respond to capability release client_id: 189511887 mds.ceph-24(mds.1): Client sn403.hpc.ait.dtu.dk:con-fs2-hpc failing to respond to capability release client_id: 231250695 If I look at the session info from mds.1 for these clients I see this: # ceph tell mds.1 session ls | jq -c '[.[] | {id: .id, h: .client_metadata.hostname, addr: .inst, fs: .client_metadata.root, caps: .num_caps, req: .request_load_avg}]|sort_by(.caps)|.[]' | grep -e 145698301 -e 189511877 -e 189511887 -e 231250695 {"id":189511887,"h":"sn350.hpc.ait.dtu.dk","addr":"client.189511887 v1:192.168.57.221:0/4262844211","fs":"/hpc/groups","caps":2,"req":0} {"id":231250695,"h":"sn403.hpc.ait.dtu.dk","addr":"client.231250695 v1:192.168.58.18:0/1334540218","fs":"/hpc/groups","caps":3,"req":0} {"id":189511877,"h":"sn463.hpc.ait.dtu.dk","addr":"client.189511877 v1:192.168.58.78:0/3535879569","fs":"/hpc/groups","caps":4,"req":0} {"id":145698301,"h":"sn352.hpc.ait.dtu.dk","addr":"client.145698301 v1:192.168.57.223:0/2146607320","fs":"/hpc/groups","caps":7,"req":0} We have mds_min_caps_per_client=4096, so it looks like the limit is well satisfied. Also, the file system is pretty idle at the moment. Why and what exactly is the MDS complaining about here? Thanks and best regards. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 6

When to use the auth profiles simple-rados-client and profile simple-rados-client-with-blocklist?
by Christian Rohmann 01 Sep '23

01 Sep '23

Hey ceph-users, 1) When configuring Gnocchi to use Ceph storage (see https://gnocchi.osci.io/install.html#ceph-requirements) I was wondering if one could use any of the auth profiles like * simple-rados-client * simple-rados-client-with-blocklist ? Or are those for different use cases? 2) I was also wondering why the documentation mentions "(Monitor only)" but then it says "Gives a user read-only permissions for monitor, OSD, and PG data."? 3) And are those profiles really for "read-only" users? Why don't they have "read-only" in their name like the rbd and the corresponding "rbd-read-only" profile? Regards Christian

2 1

Critical Information: DELL/Toshiba SSDs dying after 70,000 hours of operation
by Frédéric Nass 01 Sep '23

01 Sep '23

Hello, This message does not concern Ceph itself but a hardware vulnerability which can lead to permanent loss of data on a Ceph cluster equipped with the same hardware in separate fault domains. The DELL / Toshiba PX02SMF020, PX02SMF040, PX02SMF080 and PX02SMB160 SSD drives of the 13G generation of DELL servers are subject to a vulnerability which renders them unusable after 70,000 hours of operation, i.e. approximately 7 years and 11 months of activity. This topic has been discussed here: https://www.dell.com/community/PowerVault/TOSHIBA-PX02SMF080-has-lost-commu… The risk is all the greater since these disks may die at the same time in the same server leading to the loss of all data in the server. To date, DELL has not provided any firmware fixing this vulnerability, the latest firmware version being "A3B3" released on Sept. 12, 2016: https://www.dell.com/support/home/en-us/ drivers/driversdetails?driverid=hhd9k If your have servers running these drives, check their uptime. If they are close to the 70,000 hour limit, replace them immediately. The smartctl tool does not report the uptime for these SSDs, but if you have HDDs in the server, you can query their SMART status and get their uptime, which should be about the same as the SSDs. The smartctl command is: smartctl -a -d megaraid,XX /dev/sdc (where XX is the iSCSI bus number). We have informed DELL about this but have no information yet on the arrival of a fix. We have lost 6 disks, in 3 different servers, in the last few weeks. Our observation shows that the drives don't survive full shutdown and restart of the machine (power off then power on in iDrac), but they may also die during a single reboot (init 6) or even while the machine is running. Fujitsu released a corrective firmware in June 2021 but this firmware is most certainly not applicable to DELL drives: https://www.fujitsu.com/us/imagesgig5/PY-CIB070-00.pdf Regards, Frederic Sous-direction Infrastructure and Services Direction du Numérique Université de Lorraine

2 2

OSDs spam log with scrub starts
by Adrien Georget 01 Sep '23

01 Sep '23

Hello, On our 16.2.14 CephFS cluster, all OSDs are spamming logs with messages like "log_channel(cluster) log [DBG] : xxx scrub starts". All OSDs are concerned, for different PGs. Cluster is healthy without any recovery ops. For a single PG, we can have hundreds of scrub starts msg in less than an hour. With 720 OSDs (8k PG, EC8+2), it can lead to millions of messages by hour... For example with PG 3.1d57 or||3.1988 : |Aug 31 16:02:09 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-58[1310188]: debug 2023-08-31T14:02:09.453+0000 7fdab1ec4700 0 log_channel(cluster) log [DBG] : 3.1d57 scrub starts|| ||Aug 31 16:02:11 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-58[1310188]: debug 2023-08-31T14:02:11.446+0000 7fdab1ec4700 0 log_channel(cluster) log [DBG] : 3.1d57 scrub starts|| ||Aug 31 16:02:12 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-58[1310188]: debug 2023-08-31T14:02:12.428+0000 7fdab1ec4700 0 log_channel(cluster) log [DBG] : 3.1d57 scrub starts|| ||Aug 31 16:02:13 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-58[1310188]: debug 2023-08-31T14:02:13.456+0000 7fdab1ec4700 0 log_channel(cluster) log [DBG] : 3.1d57 scrub starts|| ||Aug 31 16:02:14 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-58[1310188]: debug 2023-08-31T14:02:14.431+0000 7fdab1ec4700 0 log_channel(cluster) log [DBG] : 3.1d57 scrub starts|| ||Aug 31 16:02:15 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-58[1310188]: debug 2023-08-31T14:02:15.475+0000 7fdab1ec4700 0 log_channel(cluster) log [DBG] : 3.1d57 scrub starts|| ||Aug 31 16:02:21 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-58[1310188]: debug 2023-08-31T14:02:21.516+0000 7fdab1ec4700 0 log_channel(cluster) log [DBG] : 3.1d57 scrub starts|| ||Aug 31 16:02:23 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-58[1310188]: debug 2023-08-31T14:02:23.555+0000 7fdab1ec4700 0 log_channel(cluster) log [DBG] : 3.1d57 scrub starts|| ||Aug 31 16:02:24 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-58[1310188]: debug 2023-08-31T14:02:24.510+0000 7fdab1ec4700 0 log_channel(cluster) log [DBG] : 3.1d57 deep-scrub starts|| Aug 31 16:02:10 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-276[1325507]: debug 2023-08-31T14:02:10.384+0000 7f0606ce3700 0 log_channel(cluster) log [DBG] : 3.1988 deep-scrub starts Aug 31 16:02:11 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-276[1325507]: debug 2023-08-31T14:02:11.377+0000 7f0606ce3700 0 log_channel(cluster) log [DBG] : 3.1988 scrub starts Aug 31 16:02:13 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-276[1325507]: debug 2023-08-31T14:02:13.383+0000 7f0606ce3700 0 log_channel(cluster) log [DBG] : 3.1988 scrub starts Aug 31 16:02:15 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-276[1325507]: debug 2023-08-31T14:02:15.383+0000 7f0606ce3700 0 log_channel(cluster) log [DBG] : 3.1988 deep-scrub starts Aug 31 16:02:17 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-276[1325507]: debug 2023-08-31T14:02:17.336+0000 7f0606ce3700 0 log_channel(cluster) log [DBG] : 3.1988 scrub starts Aug 31 16:02:19 ceph-86cd8a68-7649-11ed-b2be-5cba2c7fdb30-osd-276[1325507]: debug 2023-08-31T14:02:19.328+0000 7f0606ce3700 0 log_channel(cluster) log [DBG] : 3.1988 scrub starts || ||PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN|| ||3.1d57 52757 0 0 0 0 167596026648 0 0 1799 1799 active+clean 2023-08-31T14:27:24.025755+0000 236010'4532653 236011:8745383 [58,421,335,9,59,199,390,481,425,480] 58 [58,421,335,9,59,199,390,481,425,480] 58 231791'4531915 *2023-08-29T22:41:12.266874+0000* 229377'4526369 *2023-08-26T04:30:42.894505+0000* 0| |3.1988 52867 0 0 0 0 168603872808 0 0 1811 1811 active+clean 2023-08-31T14:32:13.361420+0000 236018'4241611 236018:9815753 [276,342,345,299,210,349,85,481,446,46] 276 [276,342,345,299,210,349,85,481,446,46] 276 236012'4241602 *2023-08-31T14:32:13.361343+0000* 228157'4229095 *2023-08-24T05:59:16.573471+0000*| However scrub is working fine, scrub stamp looks OK (2023-08-29 or 2023-08-31) as we have default value for scrub interval (min 24h / max 7days). I tried to play with scrub parameters like osd_scrub_load_threshold (->20), osd_max_scrubs (->3), osd_scrub_*_interval but nothing better. Any idea what's going on and how to fix this? Cheers, Adrien ||

3 3

osdspec_affinity error in the Cephadm module
by Adam Huffman 31 Aug '23

31 Aug '23

I've been having fun today trying to invite a new disk that replaced a failing one into a cluster. One of my attempts to apply an OSD spec was clearly wrong, because I now have this error: Module 'cephadm' has failed: 'osdspec_affinity' and this was the traceback in the mgr logs: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work return f(*arg) File "/usr/share/ceph/mgr/cephadm/serve.py", line 224, in refresh r = self._refresh_host_devices(host) File "/usr/share/ceph/mgr/cephadm/serve.py", line 396, in _refresh_host_devices self.update_osdspec_previews(host) File "/usr/share/ceph/mgr/cephadm/serve.py", line 412, in update_osdspec_previews previews.extend(self.mgr.osd_service.get_previews(search_host)) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 258, in get_previews return self.generate_previews(osdspecs, host) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 291, in generate_previews for host, ds in self.prepare_drivegroup(osdspec): File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 225, in prepare_drivegroup existing_daemons=len(dd_for_spec_and_host)) File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 35, in __init__ self._data = self.assign_devices('data_devices', self.spec.data_devices) File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 19, in wrapper return f(self, ds) File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 134, in assign_devices if lv['osdspec_affinity'] != self.spec.service_id: KeyError: 'osdspec_affinity' This cluster is running 16.2.13. The exported service spec is: service_type: osd service_id: osd_spec-0.3 service_name: osd.osd_spec-0.3 placement: host_pattern: cepho-* spec: data_devices: rotational: true db_devices: model: SSDPE2KE032T8L encrypted: true filter_logic: AND objectstore: bluestore Best Wishes, Adam

3 3

Re: Reef - what happened to OSD spec?
by Nigel Williams 31 Aug '23

31 Aug '23

Thanks Eugen for following up. Sorry my second response was incomplete. I can confirm that it works as expected too. My confusion was that the section from the online documentation seemed to be missing/moved, and when it initially failed I wrongly thought that the OSD-add process had changed in the Reef release. There might still need to be a way that "destroy" does additional clean-up to clear remnants of LVM fingerprints on the devices as this tripped me up when the OSDspec apply failed due to "filesystem on device" checks. Documentation has been improved and OSD spec is now under this heading for Reef: https://docs.ceph.com/en/reef/cephadm/services/osd/#advanced-osd-service-sp…

2 2

Pacific 16.2.14 debian Incomplete
by Reed Dier 31 Aug '23

31 Aug '23

6 10

Re: cephfs snapshot mirror peer_bootstrap import hung
by Adiga, Anantha 31 Aug '23

31 Aug '23

Hi Venky, Could this be the reason that the peer-bootstrap import is hanging? how do I upgrade cephfs-mirror to Quincy? root@fl31ca104ja0201:/# cephfs-mirror --version ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable) root@fl31ca104ja0201:/# ceph version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) root@fl31ca104ja0201:/# Thank you, Anantha From: Adiga, Anantha Sent: Monday, August 7, 2023 11:21 AM To: 'Venky Shankar' <vshankar(a)redhat.com>; 'ceph-users(a)ceph.io' <ceph-users(a)ceph.io> Subject: RE: [ceph-users] Re: cephfs snapshot mirror peer_bootstrap import hung Hi Venky, I tried on another secondary Quincy cluster and it is the same problem. The peer_bootstrap mport command hangs. root@fl31ca104ja0201:/# ceph fs snapshot mirror peer_bootstrap import cephfs eyJmc2lkIjogIjJlYWMwZWEwLTYwNDgtNDQ0Zi04NGIyLThjZWVmZWQyN2E1YiIsICJmaWxlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiwgInNpdGVfbmFtZSI6ICJzaGdSLXNpdGUiLCAia2V5IjogIkFRQ0lGdEZrSStTTE5oQUFXbWV6MkRKcEg5ZUdyYnhBOWVmZG9BPT0iLCAibW9uX2hvc3QiOiAiW3YyOjEwLjIzOS4xNTUuMTg6MzMwMC8wLHYxOjEwLjIzOS4xNTUuMTg6Njc4OS8wXSBbdjI6MTAuMjM5LjE1NS4xOTozMzAwLzAsdjE6MTAuMjM5LjE1NS4xOTo2Nzg5LzBdIFt2MjoxMC4yMzkuMTU1LjIwOjMzMDAvMCx2MToxMC4yMzkuMTU1LjIwOjY3ODkvMF0ifQ== …… ……. ..command does not complete..waits here ^C to exit. Thereafter some commands do not complete… root@fl31ca104ja0201:/# ceph -s cluster: id: d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e health: HEALTH_OK services: mon: 3 daemons, quorum fl31ca104ja0202,fl31ca104ja0203,fl31ca104ja0201 (age 2d) mgr: fl31ca104ja0201.kkoono(active, since 3d), standbys: fl31ca104ja0202, fl31ca104ja0203 mds: 1/1 daemons up, 2 standby osd: 44 osds: 44 up (since 2d), 44 in (since 5w) cephfs-mirror: 1 daemon active (1 hosts) rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 25 pools, 769 pgs objects: 614.40k objects, 1.9 TiB usage: 2.9 TiB used, 292 TiB / 295 TiB avail pgs: 769 active+clean io: client: 32 KiB/s rd, 0 B/s wr, 33 op/s rd, 1 op/s wr root@fl31ca104ja0201:/# root@fl31ca104ja0201:/# ceph fs status cephfs This command also waits. …… I have attached the mgr log root@fl31ca104ja0201:/# ceph service status { "cephfs-mirror": { "5306346": { "status_stamp": "2023-08-07T17:35:56.884907+0000", "last_beacon": "2023-08-07T17:45:01.903540+0000", "status": { "status_json": "{\"1\":{\"name\":\"cephfs\",\"directory_count\":0,\"peers\":{}}}" } } Quincy secondary cluster root@a001s008-zz14l47008:/# ceph mgr module enable mirroring root@a001s008-zz14l47008:/# ceph fs authorize cephfs client.mirror_remote / rwps [client.mirror_remote] key = AQCIFtFkI+SLNhAAWmez2DJpH9eGrbxA9efdoA== root@a001s008-zz14l47008:/# ceph auth get client.mirror_remote [client.mirror_remote] key = AQCIFtFkI+SLNhAAWmez2DJpH9eGrbxA9efdoA== caps mds = "allow rwps fsname=cephfs" caps mon = "allow r fsname=cephfs" caps osd = "allow rw tag cephfs data=cephfs" root@a001s008-zz14l47008:/# root@a001s008-zz14l47008:/# ceph fs snapshot mirror peer_bootstrap create cephfs client.mirror_remote shgR-site {"token": "eyJmc2lkIjogIjJlYWMwZWEwLTYwNDgtNDQ0Zi04NGIyLThjZWVmZWQyN2E1YiIsICJmaWxlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiwgInNpdGVfbmFtZSI6ICJzaGdSLXNpdGUiLCAia2V5IjogIkFRQ0lGdEZrSStTTE5oQUFXbWV6MkRKcEg5ZUdyYnhBOWVmZG9BPT0iLCAibW9uX2hvc3QiOiAiW3YyOjEwLjIzOS4xNTUuMTg6MzMwMC8wLHYxOjEwLjIzOS4xNTUuMTg6Njc4OS8wXSBbdjI6MTAuMjM5LjE1NS4xOTozMzAwLzAsdjE6MTAuMjM5LjE1NS4xOTo2Nzg5LzBdIFt2MjoxMC4yMzkuMTU1LjIwOjMzMDAvMCx2MToxMC4yMzkuMTU1LjIwOjY3ODkvMF0ifQ=="} root@a001s008-zz14l47008:/# Thank you, Anantha From: Adiga, Anantha Sent: Friday, August 4, 2023 11:55 AM To: Venky Shankar <vshankar(a)redhat.com<mailto:vshankar@redhat.com>>; ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> Subject: RE: [ceph-users] Re: cephfs snapshot mirror peer_bootstrap import hung Hi Venky, Thank you so much for the guidance. Attached is the mgr log. Note: the 4th node in the primary cluster has smaller capacity drives, the other 3 nodes have the larger capacity drives. 32 ssd 6.98630 1.00000 7.0 TiB 44 GiB 44 GiB 183 KiB 148 MiB 6.9 TiB 0.62 0.64 40 up osd.32 -7 76.84927 - 77 TiB 652 GiB 648 GiB 20 MiB 3.0 GiB 76 TiB 0.83 0.86 - host fl31ca104ja0203 1 ssd 6.98630 1.00000 7.0 TiB 73 GiB 73 GiB 8.0 MiB 333 MiB 6.9 TiB 1.02 1.06 54 up osd.1 4 ssd 6.98630 1.00000 7.0 TiB 77 GiB 77 GiB 1.1 MiB 174 MiB 6.9 TiB 1.07 1.11 55 up osd.4 7 ssd 6.98630 1.00000 7.0 TiB 47 GiB 47 GiB 140 KiB 288 MiB 6.9 TiB 0.66 0.68 51 up osd.7 10 ssd 6.98630 1.00000 7.0 TiB 75 GiB 75 GiB 299 KiB 278 MiB 6.9 TiB 1.05 1.09 44 up osd.10 13 ssd 6.98630 1.00000 7.0 TiB 94 GiB 94 GiB 1018 KiB 291 MiB 6.9 TiB 1.31 1.36 72 up osd.13 16 ssd 6.98630 1.00000 7.0 TiB 31 GiB 31 GiB 163 KiB 267 MiB 7.0 TiB 0.43 0.45 49 up osd.16 19 ssd 6.98630 1.00000 7.0 TiB 14 GiB 14 GiB 756 KiB 333 MiB 7.0 TiB 0.20 0.21 50 up osd.19 22 ssd 6.98630 1.00000 7.0 TiB 105 GiB 104 GiB 1.3 MiB 313 MiB 6.9 TiB 1.46 1.51 48 up osd.22 25 ssd 6.98630 1.00000 7.0 TiB 17 GiB 16 GiB 257 KiB 272 MiB 7.0 TiB 0.23 0.24 45 up osd.25 28 ssd 6.98630 1.00000 7.0 TiB 72 GiB 72 GiB 6.1 MiB 180 MiB 6.9 TiB 1.01 1.05 43 up osd.28 31 ssd 6.98630 1.00000 7.0 TiB 47 GiB 46 GiB 592 KiB 358 MiB 6.9 TiB 0.65 0.68 56 up osd.31 -9 64.04089 - 64 TiB 728 GiB 726 GiB 17 MiB 1.8 GiB 63 TiB 1.11 1.15 - host fl31ca104ja0302 33 ssd 5.82190 1.00000 5.8 TiB 65 GiB 65 GiB 245 KiB 144 MiB 5.8 TiB 1.09 1.13 47 up osd.33 34 ssd 5.82190 1.00000 5.8 TiB 14 GiB 14 GiB 815 KiB 83 MiB 5.8 TiB 0.24 0.25 55 up osd.34 35 ssd 5.82190 1.00000 5.8 TiB 77 GiB 77 GiB 224 KiB 213 MiB 5.7 TiB 1.30 1.34 44 up osd.35 36 ssd 5.82190 1.00000 5.8 TiB 117 GiB 117 GiB 8.5 MiB 284 MiB 5.7 TiB 1.96 2.03 52 up osd.36 37 ssd 5.82190 1.00000 5.8 TiB 58 GiB 58 GiB 501 KiB 132 MiB 5.8 TiB 0.98 1.01 40 up osd.37 38 ssd 5.82190 1.00000 5.8 TiB 123 GiB 123 GiB 691 KiB 266 MiB 5.7 TiB 2.07 2.14 73 up osd.38 39 ssd 5.82190 1.00000 5.8 TiB 77 GiB 77 GiB 609 KiB 193 MiB 5.7 TiB 1.30 1.34 62 up osd.39 40 ssd 5.82190 1.00000 5.8 TiB 77 GiB 77 GiB 262 KiB 148 MiB 5.7 TiB 1.29 1.34 55 up osd.40 41 ssd 5.82190 1.00000 5.8 TiB 44 GiB 44 GiB 4.4 MiB 140 MiB 5.8 TiB 0.75 0.77 44 up osd.41 42 ssd 5.82190 1.00000 5.8 TiB 45 GiB 45 GiB 886 KiB 135 MiB 5.8 TiB 0.75 0.78 47 up osd.42 43 ssd 5.82190 1.00000 5.8 TiB 28 GiB 28 GiB 187 KiB 104 MiB 5.8 TiB 0.48 0.49 58 up osd.43 [Also: Yesterday I had two cfs-mirror running one on fl31ca104ja0201 and fl31ca104ja0302. The cfs-mirror on fl31ca104ja0201 was stopped. When the import token was run on fl31ca104ja0302, the cfs-mirror log was active. Just in case it is useful I have attached that log (cfsmirror-container.log) as well. ] How can I list the token on the target cluster after running the create peer_bootstrap command? Here is today’s status with your suggestion: There is only one cfs-mirror daemon running now. It is on fl31ca104ja0201 node. root@fl31ca104ja0201:/# ceph -s cluster: id: d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e health: HEALTH_OK services: mon: 3 daemons, quorum fl31ca104ja0202,fl31ca104ja0203,fl31ca104ja0201 (age 7m) mgr: fl31ca104ja0201.kkoono(active, since 13m), standbys: fl31ca104ja0202, fl31ca104ja0203 mds: 1/1 daemons up, 2 standby osd: 44 osds: 44 up (since 7m), 44 in (since 4w) cephfs-mirror: 1 daemon active (1 hosts) rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 25 pools, 769 pgs objects: 614.40k objects, 1.9 TiB usage: 2.8 TiB used, 292 TiB / 295 TiB avail pgs: 769 active+clean io: client: 32 MiB/s rd, 0 B/s wr, 57 op/s rd, 1 op/s wr root@fl31ca104ja0201:/# root@fl31ca104ja0201:/# root@fl31ca104ja0201:/# ceph tell mgr.fl31ca104ja0201.kkoono config set debug_mgr 20 { "success": "" } root@fl31ca104ja0201:/# ceph fs snapshot mirror peer_bootstrap import cephfs eyJmc2lkIjogImE2ZjUyNTk4LWU1Y2QtNGEwOC04NDIyLTdiNmZkYjFkNWRiZSIsICJmaWxlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiwgInNpdGVfbmFtZSI6ICJmbGV4Mi1zaXRlIiwgImtleSI6ICJBUUNmd01sa005MHBMQkFBd1h0dnBwOGowNEl2Qzh0cXBBRzliQT09IiwgIm1vbl9ob3N0IjogIlt2MjoxNzIuMTguNTUuNzE6MzMwMC8wLHYxOjE3Mi4xOC41NS43MTo2Nzg5LzBdIFt2MjoxNzIuMTguNTUuNzM6MzMwMC8wLHYxOjE3Mi4xOC41NS43Mzo2Nzg5LzBdIn0= ^CInterrupted Ctrl-C after 15 min. Once the command is run, the health status goes to WARN . root@fl31ca104ja0201:/# ceph -s cluster: id: d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e health: HEALTH_WARN 6 slow ops, oldest one blocked for 1095 sec, mon.fl31ca104ja0203 has slow ops services: mon: 3 daemons, quorum fl31ca104ja0202,fl31ca104ja0203,fl31ca104ja0201 (age 30m) mgr: fl31ca104ja0201.kkoono(active, since 35m), standbys: fl31ca104ja0202, fl31ca104ja0203 mds: 1/1 daemons up, 2 standby osd: 44 osds: 44 up (since 29m), 44 in (since 4w) cephfs-mirror: 1 daemon active (1 hosts) rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 25 pools, 769 pgs objects: 614.40k objects, 1.9 TiB usage: 2.8 TiB used, 292 TiB / 295 TiB avail pgs: 769 active+clean io: client: 67 KiB/s rd, 0 B/s wr, 68 op/s rd, 21 op/s wr -----Original Message----- From: Venky Shankar <vshankar(a)redhat.com<mailto:vshankar@redhat.com>> Sent: Thursday, August 3, 2023 11:03 PM To: Adiga, Anantha <anantha.adiga(a)intel.com<mailto:anantha.adiga@intel.com>> Cc: ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> Subject: [ceph-users] Re: cephfs snapshot mirror peer_bootstrap import hung Hi Anantha, On Fri, Aug 4, 2023 at 2:27 AM Adiga, Anantha <anantha.adiga(a)intel.com<mailto:anantha.adiga@intel.com>> wrote: > > Hi > > Could you please provide guidance on how to diagnose this issue: > > In this case, there are two Ceph clusters: cluster A, 4 nodes and cluster B, 3 node, in different locations. Both are already running RGW multi-site, A is master. > > Cephfs snapshot mirroring is being configured on the clusters. Cluster A is the primary, cluster B is the peer. Cephfs snapshot mirroring is being configured. The bootstrap import step on the primary node hangs. > > On the target cluster : > --------------------------- > "version": "16.2.5", > "release": "pacific", > "release_type": "stable" > > root@cr21meg16ba0101:/# ceph fs snapshot mirror peer_bootstrap create > cephfs client.mirror_remote flex2-site > {"token": > "eyJmc2lkIjogImE2ZjUyNTk4LWU1Y2QtNGEwOC04NDIyLTdiNmZkYjFkNWRiZSIsICJma > Wxlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiw > gInNpdGVfbmFtZSI6ICJmbGV4Mi1zaXRlIiwgImtleSI6ICJBUUNmd01sa005MHBMQkFBd > 1h0dnBwOGowNEl2Qzh0cXBBRzliQT09IiwgIm1vbl9ob3N0IjogIlt2MjoxNzIuMTguNTU > uNzE6MzMwMC8wLHYxOjE3Mi4xOC41NS43MTo2Nzg5LzBdIFt2MjoxNzIuMTguNTUuNzM6M > zMwMC8wLHYxOjE3Mi4xOC41NS43Mzo2Nzg5LzBdIn0="} Seems fine uptil here. > root@cr21meg16ba0101:/var/run/ceph# > > On the source cluster: > ---------------------------- > "version": "17.2.6", > "release": "quincy", > "release_type": "stable" > > root@fl31ca104ja0201:/# ceph -s > cluster: > id: d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e > health: HEALTH_OK > > services: > mon: 3 daemons, quorum fl31ca104ja0202,fl31ca104ja0203,fl31ca104ja0201 (age 111m) > mgr: fl31ca104ja0201.nwpqlh(active, since 11h), standbys: fl31ca104ja0203, fl31ca104ja0202 > mds: 1/1 daemons up, 2 standby > osd: 44 osds: 44 up (since 111m), 44 in (since 4w) > cephfs-mirror: 1 daemon active (1 hosts) > rgw: 3 daemons active (3 hosts, 1 zones) > > data: > volumes: 1/1 healthy > pools: 25 pools, 769 pgs > objects: 614.40k objects, 1.9 TiB > usage: 2.8 TiB used, 292 TiB / 295 TiB avail > pgs: 769 active+clean > > root@fl31ca104ja0302:/# ceph mgr module enable mirroring module > 'mirroring' is already enabled root@fl31ca104ja0302:/# ceph fs > snapshot mirror peer_bootstrap import cephfs > eyJmc2lkIjogImE2ZjUyNTk4LWU1Y2QtNGEwOC04NDIyLTdiNmZkYjFkNWRiZSIsICJmaW > xlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiwg > InNpdGVfbmFtZSI6ICJmbGV4Mi1zaXRlIiwgImtleSI6ICJBUUNmd01sa005MHBMQkFBd1 > h0dnBwOGowNEl2Qzh0cXBBRzliQT09IiwgIm1vbl9ob3N0IjogIlt2MjoxNzIuMTguNTUu > NzE6MzMwMC8wLHYxOjE3Mi4xOC41NS43MTo2Nzg5LzBdIFt2MjoxNzIuMTguNTUuNzM6Mz > MwMC8wLHYxOjE3Mi4xOC41NS43Mzo2Nzg5LzBdIn0= Going by your description, I'm guessing this is the command that hangs? If that's the case, set `debug_mgr=20`, repeat the token import step and share the ceph-mgr log. Also note that you can check the mirror daemon status as detailed in https://docs.ceph.com/en/latest/dev/cephfs-mirroring/#mirror-daemon-status > > > root@fl31ca104ja0302:/var/run/ceph# ceph --admin-daemon > /var/run/ceph/ceph-client.cephfs-mirror.fl31ca104ja0302.sypagt.7.94083135960976.asok status { > "metadata": { > "ceph_sha1": "d7ff0d10654d2280e08f1ab989c7cdf3064446a5", > "ceph_version": "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)", > "entity_id": "cephfs-mirror.fl31ca104ja0302.sypagt", > "hostname": "fl31ca104ja0302", > "pid": "7", > "root": "/" > }, > "dentry_count": 0, > "dentry_pinned_count": 0, > "id": 5194553, > "inst": { > "name": { > "type": "client", > "num": 5194553 > }, > "addr": { > "type": "v1", > "addr": "10.45.129.5:0", > "nonce": 2497002034 > } > }, > "addr": { > "type": "v1", > "addr": "10.45.129.5:0", > "nonce": 2497002034 > }, > "inst_str": "client.5194553 10.45.129.5:0/2497002034", > "addr_str": "10.45.129.5:0/2497002034", > "inode_count": 1, > "mds_epoch": 118, > "osd_epoch": 6266, > "osd_epoch_barrier": 0, > "blocklisted": false, > "fs_name": "cephfs" > } > > root@fl31ca104ja0302:/home/general# docker logs > ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-cephfs-mirror-fl31ca104ja030 > 2-sypagt --tail 10 debug 2023-08-03T05:24:27.413+0000 7f8eb6fc0280 0 > ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy > (stable), process cephfs-mirror, pid 7 debug > 2023-08-03T05:24:27.413+0000 7f8eb6fc0280 0 pidfile_write: ignore > empty --pid-file debug 2023-08-03T05:24:27.445+0000 7f8eb6fc0280 1 > mgrc service_daemon_register cephfs-mirror.5184622 metadata > {arch=x86_64,ceph_release=quincy,ceph_version=ceph version 17.2.6 > (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy > (stable),ceph_version_short=17.2.6,container_hostname=fl31ca104ja0302, > container_image=quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c<mailto:container_image=quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c> > 64ca62a0b38ab362e614ad671efa4a0547e,cpu=Intel(R) Xeon(R) Gold 6252 CPU > @ 2.10GHz,distro=centos,distro_description=CentOS Stream > 8,distro_version=8,hostname=fl31ca104ja0302,id=fl31ca104ja0302.sypagt, > instance_id=5184622,kernel_description=#82-Ubuntu SMP Tue Jun 6 > 23:10:23 UTC > 2023,kernel_version=5.15.0-75-generic,mem_swap_kb=8388604,mem_total_kb > =527946928,os=Linux} debug 2023-08-03T05:27:10.419+0000 7f8ea1b2c700 > 0 client.5194553 ms_handle_reset on v2:10.45.128.141:3300/0 debug > 2023-08-03T05:50:10.917+0000 7f8ea1b2c700 0 client.5194553 > ms_handle_reset on v2:10.45.128.139:3300/0 > > Thank you, > Anantha > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an > email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io> > -- Cheers, Venky _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

2 4

v16.2.14 Pacific released
by Yuri Weinstein 30 Aug '23

30 Aug '23

We're happy to announce the 14th backport release in the Pacific series. https://ceph.io/en/news/blog/2023/v16-2-14-pacific-released/ Notable Changes --------------- * CEPHFS: After recovering a Ceph File System post following the disaster recovery procedure, the recovered files under lost+found directory can now be deleted. * ceph mgr dump command now displays the name of the mgr module that registered a RADOS client in the name field added to elements of the active_clients array. Previously, only the address of a module's RADOS client was shown in the active_clients array. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-16.2.14.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 238ba602515df21ea7ffc75c88db29f9e5ef12c9

1 0

Multisite RGW setup not working when following the docs step by step
by Petr Bena 30 Aug '23

30 Aug '23

Hello, My goal is to setup multisite RGW with 2 separate CEPH clusters in separate datacenters, where RGW data are being replicated. I created a lab for this purpose in both locations (with latest reef ceph installed using cephadm) and tried to follow this guide: https://docs.ceph.com/en/reef/radosgw/multisite/ Unfortunatelly, even after multiple attempts it always failed when creating a secondary zone. I could succesfully pull the realm from master, but that was pretty much the last trully succesful step. I can notice that immediately after pulling the realm to secondary, radosgw-admin user list returns an empty list (which IMHO should contain replicated user list from master). Continuing by setting default real and zonegroup and creating the secondary zone in secondary cluster I end up having 2 zones in each cluster, both seemingly in same zonegroup, but with replication failing - this is what I see in sync status: (master) [ceph: root@ceph-lab-brn-01 /]# radosgw-admin sync status realm d2c4ebf9-e156-4c4e-9d56-3fff6a652e75 (ceph) zonegroup abc3c0ae-a84d-48d4-8e78-da251eb78781 (cz) zone 97fb5842-713a-4995-8966-5afe1384f17f (cz-brn) current time 2023-08-30T12:58:12Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync no sync (zone is master) 2023-08-30T12:58:13.991+0000 7f583a52c780 0 ERROR: failed to fetch datalog info data sync source: 13a8c663-b241-4d8a-a424-8785fc539ec5 (cz-hol) failed to retrieve sync info: (13) Permission denied (secondary) [ceph: root@ceph-lab-hol-01 /]# radosgw-admin sync status realm d2c4ebf9-e156-4c4e-9d56-3fff6a652e75 (ceph) zonegroup abc3c0ae-a84d-48d4-8e78-da251eb78781 (cz) zone 13a8c663-b241-4d8a-a424-8785fc539ec5 (cz-hol) current time 2023-08-30T12:58:54Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync failed to read sync status: (2) No such file or directory 2023-08-30T12:58:55.617+0000 7ff37c9db780 0 ERROR: failed to fetch datalog info data sync source: 97fb5842-713a-4995-8966-5afe1384f17f (cz-brn) failed to retrieve sync info: (13) Permission denied In master there is one user created during the process (synchronization-user), on slave there are no users and when I try to re-create this synchronization user it complains I shouldn't even try and instead execute the command on master. I can see same realm and zonegroup IDs on both sides, zone list is different though: (master) [ceph: root@ceph-lab-brn-01 /]# radosgw-admin zone list { "default_info": "97fb5842-713a-4995-8966-5afe1384f17f", "zones": [ "cz-brn", "default" ] } (secondary) [ceph: root@ceph-lab-hol-01 /]# radosgw-admin zone list { "default_info": "13a8c663-b241-4d8a-a424-8785fc539ec5", "zones": [ "cz-hol", "default" ] } The permission denied error is puzzling me - could it be because real pull didn't sync the users? I tried this multiple times with clean ceph install on both sides - and always ended up the same. I even tried force creating the same user with same secrets on the other side, but it didn't help. How can I debug what kind of secret is secondary trying to use when communicating with master? Could it be that this multisite RGW setup is not yet truly supported in reef? I noticed that the documentation itself seems written for older ceph versions, as there are no mentions about orchestrator (for example in steps where configuration files of RGW need to be edited, which is done differently when using cephadm). I think that documentation is simply wrong at this time. Either it's missing some crucial steps, or it's outdated or otherwise unclear - simply by following all the steps as outlined there, you are likely to end up the same. Thanks for help!

2 1

radosgw mulsite multi zone configuration: current period realm name not same as in zonegroup
by Adiga, Anantha 30 Aug '23

30 Aug '23

Hi, I have a multi zone configuration with 4 zones. While adding a secondary zone, getting this error: root@cs17ca101ja0702:/# radosgw-admin realm pull --rgw-realm=global --url=http://10.45.128.139:8080 --default --access-key=sync_user --secret=sync_secret request failed: (13) Permission denied If the realm has been changed on the master zone, the master zone's gateway may need to be restarted to recognize this user. root@cs17ca101ja0702:/# The realm name is "global". Is the cause of the error due to the primary cluster having a current period listing the realm name as "default" instead of "global" ? However, the realm id is of realm "global" AND the zonegroup does not list realm name but has the correct realm id. See below. How to fix this issue. root@fl31ca104ja0201:/# radosgw-admin realm get { "id": "3da7b5ea-c44b-4d44-aced-fae2aabce97b", "name": "global", "current_period": "b8bc1187-2a2d-4d9e-b7be-c4f4667e3fa6", "epoch": 2 } root@fl31ca104ja0201:/# radosgw-admin realm get --rgw-realm=global { "id": "3da7b5ea-c44b-4d44-aced-fae2aabce97b", "name": "global", "current_period": "b8bc1187-2a2d-4d9e-b7be-c4f4667e3fa6", "epoch": 2 } root@fl31ca104ja0201:/# radosgw-admin zonegroup list { "default_info": "ec8b68db-1900-464f-a21a-2f6e8c107e94", "zonegroups": [ "alldczg" ] } root@fl31ca104ja0201:/# radosgw-admin zonegroup get --rgw-zonegroup=alldczg { "id": "ec8b68db-1900-464f-a21a-2f6e8c107e94", "name": "alldczg", "api_name": "alldczg", "is_master": "true", "endpoints": [ http://10.45.128.139:8080, http://172.18.55.71:8080, http://10.239.155.23:8080 ], "hostnames": [], "hostnames_s3website": [], "master_zone": "ae267592-7cd8-4d67-8792-adc57d104cd6", "zones": [ { "id": "0962f0b4-beb6-4d07-a64d-07046b81529e", "name": "CRsite", "endpoints": [ http://172.18.55.71:8080 ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" }, { "id": "9129d118-55ac-4859-b339-b8afe0793a80", "name": "BArga", "endpoints": [ http://10.208.11.26:8080 ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" }, { "id": "ae267592-7cd8-4d67-8792-adc57d104cd6", "name": "ORflex2", "endpoints": [ http://10.45.128.139:8080 ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" }, { "id": "f5edeb4b-2a37-413b-8587-0ff40d7647ea", "name": "SHGrasp", "endpoints": [ http://10.239.155.23:8080 ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "3da7b5ea-c44b-4d44-aced-fae2aabce97b", "sync_policy": { "groups": [] } } root@fl31ca104ja0201:/# radosgw-admin period get-current { "current_period": "b8bc1187-2a2d-4d9e-b7be-c4f4667e3fa6" } root@fl31ca104ja0201:/# radosgw-admin period get { "id": "b8bc1187-2a2d-4d9e-b7be-c4f4667e3fa6", "epoch": 42, "predecessor_uuid": "2df86f9a-d267-4b52-a13b-def8e5e612a2", "sync_status": [], "period_map": { "id": "b8bc1187-2a2d-4d9e-b7be-c4f4667e3fa6", "zonegroups": [ { "id": "ec8b68db-1900-464f-a21a-2f6e8c107e94", "name": "alldczg", "api_name": "alldczg", "is_master": "true", "endpoints": [ http://10.45.128.139:8080, http://172.18.55.71:8080, http://10.239.155.23:8080 ], "hostnames": [], "hostnames_s3website": [], "master_zone": "ae267592-7cd8-4d67-8792-adc57d104cd6", "zones": [ { "id": "0962f0b4-beb6-4d07-a64d-07046b81529e", "name": "CRsite", "endpoints": [ http://172.18.55.71:8080 ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" }, { "id": "9129d118-55ac-4859-b339-b8afe0793a80", "name": "BArga", "endpoints": [ http://10.208.11.26:8080 ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" }, { "id": "ae267592-7cd8-4d67-8792-adc57d104cd6", "name": "ORflex2", "endpoints": [ http://10.45.128.139:8080 ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" }, { "id": "f5edeb4b-2a37-413b-8587-0ff40d7647ea", "name": "SHGrasp", "endpoints": [ http://10.239.155.23:8080 ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "3da7b5ea-c44b-4d44-aced-fae2aabce97b", "sync_policy": { "groups": [] } } ], "short_zone_ids": [ { "key": "0962f0b4-beb6-4d07-a64d-07046b81529e", "val": 890360385 }, { "key": "9129d118-55ac-4859-b339-b8afe0793a80", "val": 3855275770 }, { "key": "ae267592-7cd8-4d67-8792-adc57d104cd6", "val": 1499747631 }, { "key": "f5edeb4b-2a37-413b-8587-0ff40d7647ea", "val": 815709635 } ] }, "master_zonegroup": "ec8b68db-1900-464f-a21a-2f6e8c107e94", "master_zone": "ae267592-7cd8-4d67-8792-adc57d104cd6", "period_config": { "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } }, "realm_id": "3da7b5ea-c44b-4d44-aced-fae2aabce97b", "realm_name": "default", "realm_epoch": 2 } root@fl31ca104ja0201:/#

1 1

CLT Meeting minutes 2023-08-30
by Nizamudeen A 30 Aug '23

30 Aug '23

Hello, Finish v18.2.0 upgrade on LRC? It seems to be running v18.1.3 not much of a difference in code commits news on teuthology jobs hanging? cephfs issues because of network troubles Its resolved by Patrick User council discussion follow-up Detailed info on this pad: https://pad.ceph.com/p/user_dev_relaunch First topic will come from David's team 16.2.14 release Pushing to release by this week. Regards, Nizam -- Nizamudeen A Software Engineer Red Hat <https://www.redhat.com/> <https://www.redhat.com/>

1 0

Is there any way to fine tune peering/pg relocation/rebalance?
by Szabo, Istvan (Agoda) 30 Aug '23

30 Aug '23

Hello, Is there a way to somehow fine tune the rebalance even further than basic tuning steps when adding new osds? Today I've added some osd to the index pool and it generated many slow ops due to OSD op latency increase + read operation latency increase = high put get latency. https://ibb.co/album/9mN6GQ osd max backfill, max recovery, recovery ops priority are 1. 1 nvme drive has 4 osd, each osd has around 80pg. The steps how I add the osds: 1. Set norebalance 2. add the osds 3. wait for peering 4. unset rebalance It takes like 15-20 mins to became normal without interrupting the rebalance the user traffic. Thank you, Istvan ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 5

hardware setup recommendations wanted
by Kai Zimmer 29 Aug '23

29 Aug '23

Dear listers, my employer already has a production Ceph cluster running but we need a second one. I just wanted to ask your opininion on the following setup. It is planned for 500 TB net capacity, expandable to 2 PB. I expect the number of OSD servers to double in the next 4 years. Erasure Code 3:2 will be used for OSDs. Usage will be file storage, Rados block devices and S3: 5x OSD servers (12x18 TB Toshiba MG09SCA18TE SAS spinning disks for data, 2x512 GB Samsung PM9A1 M.2 NVME SSD 0,55 DWPD for system, 1xAMD 7313P CPU with 16 cores @3GHz, 256 GB RAM, LSI SAS 9500 HBA, Broadcom P425G network adapter 4x25 Gbit/s) 3x MON servers (1x2 TB Samsung PM9A1 M.2 NVME SSD 0,55 DWPD for system, 2x1.6TB Kioxia CD6-V SSD 3.0 DWPD for data, 2x Broadcom P210/N210 network 4x10 GBit/s, 1xAMD 7232P CPU with 8 cores @3.1 GHz, 64 GB RAM) 3x MDS servers (1x2 TB Samsung PM9A1 M.2 NVME SSD 0,55 DWPD for system, 2x1.6 TB Kioxia CD6-V SSD 3.0 DWPD for data, 2x Broadcom P210/N210 network 4x10 GBit/s, 1xAMD 7313P CPU with 16 cores @3 GHz, 128 GB RAM) OSD servers will be connected via 2x25 GBit fibre interfaces "backend" to 2x Mikrotik CRS518-16XS-2XQ (which are connected for high-availability via 100 GBit) For the "frontend" connection to servers/clients via 2x10 GBit we're looking into 3x Mikrotik CRS326-24S+2Q+RM (which are connected for high-availability via 40 GBit) Especially for the "frontend" switches i'm looking for alternatives. Currently we use Huawei C6810-32T16A4Q-LI models with 2x33 LACP connections connected via 10 GBit/s RJ45. But those had ports blocking after a number of errors which resulted in some trouble. We'd like to avoid IOS and clones in general and would prefer a decent web interface. Any comments/recommendations? Best regards, Kai

2 1

two ways of adding OSDs? LVN vs ceph orch daemon add
by Giuliano Maggi 29 Aug '23

29 Aug '23

Hi, I am learning about Ceph, and I found this two ways of adding OSDs: https://docs.ceph.com/en/quincy/install/manual-deployment/#short-form <https://docs.ceph.com/en/quincy/install/manual-deployment/#short-form> (via LVM) AND https://docs.ceph.com/en/quincy/cephadm/services/osd/#creating-new-osds <https://docs.ceph.com/en/quincy/cephadm/services/osd/#creating-new-osds> (ceph orch daemon add osd *<host>*:*<device-path>*) Are these two ways equivalents? Thanks, Giuliano,

2 1

Reef - what happened to OSD spec?
by Nigel Williams 29 Aug '23

29 Aug '23

We upgraded to Reef from Quincy, all went smoothly (thanks Ceph developers!) When adding OSDs, the process seems to have changed, the docs no longer mention OSD spec, and giving it a try it fails when it bumps into the root drive (which has an active LVM). I expect I can add a filter to avoid it. But is using the OSD spec ( https://docs.ceph.com/en/octopus/cephadm/drivegroups/) approach now deprecated? Is the web-interface now the preferred way? thanks.

2 2

Questions since updating to 18.0.2
by Curt 28 Aug '23

28 Aug '23

Hello, We recently upgraded our cluster to version 18 and I've noticed some things that I'd like feedback on before I go down a rabbit hole for non-issues. cephadm was used for the upgrade and there were no issues. Cluster is 56 OSD's spinners for right now only used for RBD images. I've noticed active scrubs/deep scrubs. I don't remember seeing a large amount before, usually around 20-30 scrubs and 15 deep I think, now I will have 70 scrubs and 70 deep scrubs happening. Which I thought were limited to 1 per OSD or am I misunderstanding osd_max_scrubs? Everything on the cluster is currently at default values. The other thing I've noticed is since the upgrade it seems like any time backfill happens the client io drops, but neither is high to begin with, 30MiB/s read/write client IO drops to 10-15 with 200MiB/s backfill. Before upgrading backfill would be hitting 5-600 with 30 clientio. I realize lots of things could affect this and it could be separate from the cluster, I'm still investigating, but wanted to mention it incase someone could recommend a check or some change to Reef that could cause this. mclock profile is client_io. Thanks, Curt

1 0

16.2.14 pacific QE validation status
by Yuri Weinstein 28 Aug '23

28 Aug '23

Details of this release are summarized here: https://tracker.ceph.com/issues/62527#note-1 Release Notes - TBD Seeking approvals for: smoke - Venky rados - Radek, Laura rook - Sébastien Han cephadm - Adam K dashboard - Ernesto rgw - Casey rbd - Ilya krbd - Ilya fs - Venky, Patrick upgrade/pacific-p2p - Laura powercycle - Brad (SELinux denials) Thx YuriW

10 17

Status of diskprediction MGR module?
by Robert Sander 28 Aug '23

28 Aug '23

Hi, Several years ago the diskprediction module was added to the MGR collecting SMART data from the OSDs. There were local and cloud modes available claiming different accuracies. Now only the local mode remains. What is the current status of that MGR module (diskprediction_local)? We have a cluster where SMART data is available from the disks (tested with smartctl and visible in the Ceph dashboard), but even with an enabled diskprediction_local module no health and lifetime info is shown. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

3 3

What does 'removed_snaps_queue' [d5~3] means?
by Work Ceph 28 Aug '23

28 Aug '23

Hello guys, We are facing/seeing an unexpected mark in one of our pools. Do you guys know what does "removed_snaps_queue" it mean? We see some notation such as "d5~3" after this tag. What does it mean? We tried to look into the docs, but could not find anything meaningful. We are running Ceph Octopus on top of Ubuntu 18.04.

2 3

Windows 2016 RBD Driver install failure
by Robert Ford 28 Aug '23

28 Aug '23

Hello, We have been running into an issue installing the pacific windows rbd driver on windows 2016. It has no issues with either 2019 or 2022. It looks like it fails at checkpoint creation. We are installing it as admin. Has anyone seen this before or know of a solution? The closest thing I can find to why it wont install: ******* Product: D:\software\ceph_pacific_beta.msi ******* Action: INSTALL ******* CommandLine: ********** MSI (s) (CC:24) [12:31:30:315]: Machine policy value 'DisableUserInstalls' is 0 MSI (s) (CC:24) [12:31:30:315]: Note: 1: 2203 2: C:\windows\Installer\inprogressinstallinfo.ipi 3: -2147287038 MSI (s) (CC:24) [12:31:30:315]: Machine policy value 'LimitSystemRestoreCheckpointing' is 0 MSI (s) (CC:24) [12:31:30:315]: Note: 1: 1715 2: Ceph for Windows MSI (s) (CC:24) [12:31:30:315]: Calling SRSetRestorePoint API. dwRestorePtType: 0, dwEventType: 102, llSequenceNumber: 0, szDescription: "Installed Ceph for Windows". MSI (s) (CC:24) [12:31:30:315]: The call to SRSetRestorePoint API failed. Returned status: 0. GetLastError() returned: 127 -- -- Robert Ford GoDaddy | SRE III 9519020587 Phoenix, AZ rford(a)godaddy.com

2 1

rbd export-diff/import-diff hangs
by Tony Liu 28 Aug '23

28 Aug '23

Hi, I'm using rbd import and export to copy image from one cluster to another. Also using import-diff and export-diff to update image in remote cluster. For example, "rbd --cluster local export-diff ... | rbd --cluster remote import-diff ...". Sometimes, the whole command is stuck. I can't tell it's stuck on which end of the pipe. I did some search, [1] seems the same issue and [2] is also related. Wonder if there is any way to identify where it's stuck and get more debugging info. Given [2], I'd suspect the import-diff is stuck, cause rbd client is importing to the remote cluster. Networking latency could be involved here? Ping latency is 7~8 ms. Any comments is appreciated! [1] https://bugs.launchpad.net/cinder/+bug/2031897 [2] https://stackoverflow.com/questions/69858763/ceph-rbd-import-hangs Thanks! Tony

3 5

rbd export with export-format 2 exports all snapshots?
by Tony Liu 28 Aug '23

28 Aug '23

Hi, Say, source image has snapshot s1, s2 and s3. I expect "export" behaves the same as "deep cp", when specify a snapshot, with "--export-format 2", only the specified snapshot and all snapshots earlier than that will be exported. What I see is that, no matter which snapshot I specify, "export" with "--export-format 2" always exports the whole image with all snapshots. Is this expected? Could anyone help to clarify? Thanks! Tony

2 2

Can ceph-volume manage the LVs optionally used for DB / WAL at all?
by Christian Rohmann 26 Aug '23

26 Aug '23

Hey ceph-users, I was wondering if ceph-volume did anything in regards to the management (creation, setting metadata, ....) of LVs which are used for DB / WAL of an OSD? Reading the documentation at https://docs.ceph.com/en/latest/man/8/ceph-volume/#new-db is seems to indicate that the LV to be used as e.g. DB needs to be created manually (without ceph-volume) and exist prior to using ceph-volume to move the DB to that LV? I suppose the same is true for "ceph-volume lvm create" or "ceph-volume lvm prepare" and "--block.db" It's not that creating a few LVs is hard... it's just that ceph volume does apply some structure to the naming of LVM VGs and LVs on the OSD device and also adds metadata. That would then be up to the user, right? Regards Christian

2 5

cephadm to setup wal/db on nvme
by Satish Patel 25 Aug '23

25 Aug '23

Folks, I have 3 nodes with each having 1x NvME (1TB) and 3x 2.9TB SSD. Trying to build ceph storage using cephadm on Ubuntu 22.04 distro. If I want to use NvME for Journaling (WAL/DB) for my SSD based OSDs then how does cephadm handle it? Trying to find a document where I can tell cephadm to deploy wal/db on nvme so it can speed up write optimization. Do I need to create or cephadm will create each partition for the number of OSD? Help me to understand how it works and is it worth doing?

3 4

lun allocation failure
by Opánszki Gábor 25 Aug '23

25 Aug '23

Hi folks, we deployed new reef cluster to our lab. all of the nodes are up and running, but we can't allocate lun to target. on the gui we got "disk create/update failed on ceph-iscsigw0. LUN allocation failure" message. We created images on gui do you have any idea? Thanks root@ceph-mgr0:~# ceph -s *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** cluster: id: ad0aede2-4100-11ee-bc14-1c40244f5c21 health: HEALTH_OK services: mon: 5 daemons, quorum ceph-mgr0,ceph-mgr1,ceph-osd5,ceph-osd7,ceph-osd6 (age 28h) mgr: ceph-mgr0.sapbav(active, since 45h), standbys: ceph-mgr1.zwzyuc osd: 44 osds: 44 up (since 4h), 44 in (since 4h) tcmu-runner: 1 portal active (1 hosts) data: pools: 5 pools, 3074 pgs objects: 27 objects, 453 KiB usage: 30 GiB used, 101 TiB / 101 TiB avail pgs: 3074 active+clean io: client: 2.7 KiB/s rd, 2 op/s rd, 0 op/s wr root@ceph-mgr0:~# root@ceph-mgr0:~# rados lspools .mgr ace1 1T-r3-01 ace0 x root@ceph-mgr0:~# rbd ls 1T-r3-01 111 aaaa bb pool2 teszt root@ceph-mgr0:~# rbd ls x x-a root@ceph-mgr0:~# root@ceph-mgr0:~# rbd info 1T-r3-01/111 rbd image '111': size 1 GiB in 256 objects order 22 (4 MiB objects) snapshot_count: 0 id: 5f927ce161de block_name_prefix: rbd_data.5f927ce161de format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Aug 24 17:33:37 2023 access_timestamp: Thu Aug 24 17:33:37 2023 modify_timestamp: Thu Aug 24 17:33:37 2023 root@ceph-mgr0:~# rbd info 1T-r3-01/aaaa rbd image 'aaaa': size 1 GiB in 256 objects order 22 (4 MiB objects) snapshot_count: 0 id: 5f926a0e299f block_name_prefix: rbd_data.5f926a0e299f format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Aug 24 17:18:06 2023 access_timestamp: Thu Aug 24 17:18:06 2023 modify_timestamp: Thu Aug 24 17:18:06 2023 root@ceph-mgr0:~# rbd info x/x-a rbd image 'x-a': size 1 GiB in 256 objects order 22 (4 MiB objects) snapshot_count: 0 id: 5f922dbdf6c6 block_name_prefix: rbd_data.5f922dbdf6c6 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Aug 24 17:48:28 2023 access_timestamp: Thu Aug 24 17:48:28 2023 modify_timestamp: Thu Aug 24 17:48:28 2023 root@ceph-mgr0:~# root@ceph-mgr0:~# ceph orch ls --service_type iscsi *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** NAME PORTS RUNNING REFRESHED AGE PLACEMENT iscsi.gw-1 ?:5000 2/2 4m ago 6m ceph-iscsigw0;ceph-iscsigw1 root@ceph-mgr0:~# GW: root@ceph-iscsigw0:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d677a8abd2d8 quay.io/ceph/ceph "/usr/bin/rbd-target…" 6 seconds ago Up 5 seconds ceph-ad0aede2-4100-11ee-bc14-1c40244f5c21-iscsi-gw-1-ceph-iscsigw0-fmuyhi ead503586cdd quay.io/ceph/ceph "/usr/bin/tcmu-runner" 6 seconds ago Up 5 seconds ceph-ad0aede2-4100-11ee-bc14-1c40244f5c21-iscsi-gw-1-ceph-iscsigw0-fmuyhi-tcmu 3ae0014bcc41 quay.io/ceph/ceph "/usr/bin/ceph-crash…" About an hour ago Up About an hour ceph-ad0aede2-4100-11ee-bc14-1c40244f5c21-crash-ceph-iscsigw0 1a7bc044ed8a quay.io/ceph/ceph "/usr/bin/ceph-expor…" About an hour ago Up About an hour ceph-ad0aede2-4100-11ee-bc14-1c40244f5c21-ceph-exporter-ceph-iscsigw0 c746a4da2bbb quay.io/prometheus/node-exporter:v1.5.0 "/bin/node_exporter …" About an hour ago Up About an hour ceph-ad0aede2-4100-11ee-bc14-1c40244f5c21-node-exporter-ceph-iscsigw0 root@ceph-iscsigw0:~# docker exec -it d677a8abd2d8 /bin/bash [root@ceph-iscsigw0 /]# gwcli ls o- / ......................................................................................................................... [...] o- cluster ......................................................................................................... [Clusters: 1] | o- ceph ............................................................................................................ [HEALTH_OK] | o- pools .......................................................................................................... [Pools: 5] | | o- .mgr .................................................................. [(x3), Commit: 0.00Y/33602764M (0%), Used: 3184K] | | o- 1T-r3-01 ................................................................ [(x3), Commit: 0.00Y/5793684M (0%), Used: 108K] | | o- ace0 ................................................................... [(2+1), Commit: 0.00Y/11587368M (0%), Used: 24K] | | o- ace1 ................................................................... [(2+1), Commit: 0.00Y/55665220M (0%), Used: 12K] | | o- x ....................................................................... [(x3), Commit: 0.00Y/33602764M (0%), Used: 36K] | o- topology ............................................................................................... [OSDs: 44,MONs: 5] o- disks ....................................................................................................... [0.00Y, Disks: 0] o- iscsi-targets ............................................................................... [DiscoveryAuth: None, Targets: 1] o- iqn.2001-07.com.ceph:1692892702115 ................................................................ [Auth: None, Gateways: 2] o- disks .......................................................................................................... [Disks: 0] o- gateways ............................................................................................ [Up: 2/2, Portals: 2] | o- ceph-iscsigw0 ............................................................................ [10.202.5.21,10.202.4.21 (UP)] | o- ceph-iscsigw1 ............................................................................ [10.202.3.21,10.202.2.21 (UP)] o- host-groups .................................................................................................. [Groups : 0] o- hosts ....................................................................................... [Auth: ACL_ENABLED, Hosts: 0] [root@ceph-iscsigw0 /]#

2 2

Problem when configuring S3 website domain go through Cloudflare DNS proxy
by Huy Nguyen 25 Aug '23

25 Aug '23

Hi, Currently, I trying to create a CNAME record point to a s3 website, for example: s3.example.com => s3.example.com.s3-website.myceph.com. So in this way, my subdomain s3. will have https. But then only http works. If I go to https://s3.example.com, it shows the metadata of index.html: This XML file does not appear to have any style information associated with it. The document tree is shown below. <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Name>s3.example.com</Name> <Prefix/> <MaxKeys>1000</MaxKeys> <IsTruncated>false</IsTruncated> <Contents> <Key>index.html</Key> <LastModified>2023-08-24T10:03:14.046Z</LastModified> <ETag>"8e26caf000875221bf89d95f7f244927"</ETag> <Size>295</Size> <StorageClass>STANDARD</StorageClass> <Owner> <ID>d92ac19d934a4e9b90e7707372c64996</ID> <DisplayName>foo(a)example.com</DisplayName> </Owner> <Type>Normal</Type> </Contents> <Marker/> </ListBucketResult> Here is my rgw configuration: rgw_resolve_cname = true rgw_enable_static_website = true rgw_dns_s3website_name = ss-website.example.com rgw_trust_forwarded_https = true So how to make the https show the content of index.html (not its metadata)? Thanks in advance.

1 1

User + Dev Monthly Meeting Minutes 2023-08-24
by Laura Flores 24 Aug '23

24 Aug '23

Is there going to be another Pacific point release (16.2.14) in the pipeline? - Yes, 16.2.14 is going through QA right now. See https://www.spinics.net/lists/ceph-users/msg78528.html for updates. Need pacific backport for https://tracker.ceph.com/issues/59478 - Laura will check on this, although a Pacific backport is unlikely due to incompatibilities from the scrub backend refactoring. There are inconsistencies with the `ceph config dump` normal vs. json output. A fix has been proposed in https://tracker.ceph.com/issues/62379. Question for users: Will this change break any existing automation? See the tracker for more details, and reach out to @Sridhar Seshasayee <sseshasa(a)redhat.com> with any questions. -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

2 2

Rados object transformation
by Yixin Jin 24 Aug '23

24 Aug '23

Hi folks, Is it possible through object classes to transform object content? For example, I'd like this transformer to change the content of the object when it is read and when it is written. In this way, I can potentially encrypt the object content in storage without the need to make ceph/osd to do encryption/decryption. It could be taken care of by the object class itself. Thanks,Yixin

2 2

snaptrim number of objects
by Angelo Höngens 23 Aug '23

23 Aug '23

Hey guys, I'm trying to figure out what's happening to my backup cluster that often grinds to a halt when cephfs automatically removes snapshots. Almost all OSD's go to 100% CPU, ceph complains about slow ops, and CephFS stops doing client i/o. I'm graphing the cumulative value of the snaptrimq_len value, and that slowly decreases over time. One night it takes an hour, but other days, like today, my cluster has been down for almost 20 hours, and I think we're half way. Funny thing is that in both cases, the snaptrimq_len value initially goes to the same value, around 3000, and then slowly decreases, but my guess is that the number of objects that need to be trimmed varies hugely every day. Is there a way to show the size of cephfs snapshots, or get the number of objects or bytes that need snaptrimming? Perhaps I can graph that and see where the differences are. That won't explain why my cluster bogs down, but at least it gives some visibility. Running 17.2.6 everywhere by the way. Angelo.

7 9

User + Dev Monthly Meeting happening next week
by Laura Flores 23 Aug '23

23 Aug '23

Hi everyone, The User + Dev Monthly Meeting is happening next week on* Thursday, August 24th* *@ 2:00 PM UTC *at this link: https://meet.jit.si/ceph-user-dev-monthly (Note that the date has been rescheduled from the original date, August 17th.) Please add any topics you'd like to discuss to the agenda: https://pad.ceph.com/p/ceph-user-dev-monthly-minutes Thanks, Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 1

Patch change for CephFS subvolume
by Michal Strnad 23 Aug '23

23 Aug '23

Hi! I'm trying to figure out how to specify the path for a CephFS subvolume, as it's intended to represent a user's home directory. By default, it's located at /volumes/_nogroup/$NAME/$UUID. Is it possible to change this path somehow, or is using symbolic links the only option? Thank you Michal Strnad

3 5

[quincy] Migrating ceph cluster to new network, bind OSDs to multple public_nework
by Boris Behrens 23 Aug '23

23 Aug '23

Hi, I need to migrate a storage cluster to a new network. I added the new network to the ceph config via: ceph config set global public_network "old_network/64, new_network/64" I've added a set of new mon daemons with IP addresses in the new network and they are added to the quorum and seem to work as expected. But when I restart the OSD daemons, the do not bind to the new addresses. I would have expected that the OSDs try to bind to all networks but they are only bound to the old_network. The idea was to add the new set of network config to the current storage hosts, bind everything to ip addresses in both networks, shift over workload, and then remove the old network.

4 11

Re: Listing S3 buckets of a tenant using admin API
by Michal Strnad 23 Aug '23

23 Aug '23

Hi. Did you manage to access the specific bucket? Is it possible and if so, how to obtain information about a bucket storage_class and placement through the API, in addition to the regular information? Thanks Michal On 2/1/22 18:01, Daniel Iwan wrote: > It seems that REST API and CLI support listing either all buckets in the > system or buckets owned by a specific user by providing uid. > https://docs.ceph.com/en/latest/radosgw/adminops/#get-bucket-info > > I would like to list bucket of a specific tenant, possibly with stats, > therefore I would prefer to avoid fetching information for all buckets and > filtering in the application layer. > > Looks like --tenant in CLI would be ideal, but at the moment it expects > --uid also be provided. > > Is there any way to achieve that at the moment? > I'm on Ceph 16.2.7 > > Regards > Daniel Iwan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Create OSDs MANUALLY
by Alfredo Rezinovsky 23 Aug '23

23 Aug '23

SSD driver works awful when full. Even if I set DB to ssd for 4 OSDs and theres 2 the dashboard daemon allocates all the ssd. I want to partition only 70% of the SSD for DB/WAL and leave the rest for SSD maneouvering. There's a way to create an OSD telling manually disk or partitions to user for data and DB (like the way I used to do it with ceph-deploy)? -- Alfrenovsky

2 1

ceph osd error log
by Peter 22 Aug '23

22 Aug '23

Hi Ceph community, My cluster has lots of logs regarding an error that ceph-osd. I am encountering the following error message in the logs: Aug 22 00:01:28 host008 ceph-osd[3877022]: 2023-08-22T00:01:28.347-0700 7fef85251700 -1 Fail to open '/proc/3850681/cmdline' error = (2) No such file or directory My cluster is working healthy and I am looking to gain a better understanding of this error and its implications for the system's functioning to avoid protencial issue in the future. root@ceph001:~# ceph -v ceph version 16.2.13 (b81a1d7f978c8d41cf452da7af14e190542d2ee2) pacific (stable)

1 0

EC pool degrades when adding device-class to crush rule
by Lars Fenneberg 22 Aug '23

22 Aug '23

Hi all! The cluster was installed before device classes were a thing so as a prepartion to install some SSDs into a Ceph cluster with OSDs on 7 maschines I migrated all replicated pools to a CRUSH rule with device-class set. Lots of misplaced objects (probably because of changed ids in the CRUSH tree) but the cluster was HEALTH_OK the whole time. I'm now testing the migration of a 4+2 erasure coded pool to also include the device-class. Here are the old and new CRUSH rules: # Old rule default.rgw.buckets.data { id 3 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step chooseleaf indep 0 type host step emit } # New rule default.rgw.buckets.data_hdd { id 4 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step chooseleaf indep 0 type host step emit } After changing the rule with ceph osd pool set default.rgw.buckets.data crush_rule default.rgw.buckets.data_hdd I'm getting quite a few degraded placement groups and I'm not sure if I still have a redundant setup or not during the migration: [root@master1 tmp]# ceph version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) [root@master1 tmp]# ceph -s cluster: id: 3b46f93c-788a-11e9-bc8c-bcaec503b525 health: HEALTH_WARN Degraded data redundancy: 4903/15474 objects degraded (31.685%), 12 pgs degraded services: mon: 5 daemons, quorum master1.dev,master2.dev,master3.dev,master4.dev,master5.dev (age 10m) mgr: master3.dev(active, since 8m), standbys: master5.dev mds: 1/1 daemons up, 1 standby osd: 14 osds: 14 up (since 8m), 14 in (since 9m) rgw: 4 daemons active (4 hosts, 1 zones) data: volumes: 1/1 healthy pools: 21 pools, 586 pgs objects: 2.70k objects, 365 MiB usage: 281 GiB used, 279 GiB / 560 GiB avail pgs: 4903/15474 objects degraded (31.685%) 7054/15474 objects misplaced (45.586%) 554 active+clean 20 active+recovering 12 active+recovery_wait+degraded io: recovery: 545 KiB/s, 18 objects/s [root@master1 tmp]# ceph health detail HEALTH_WARN Degraded data redundancy: 4903/15474 objects degraded (31.685%), 12 pgs degraded [WRN] PG_DEGRADED: Degraded data redundancy: 4903/15474 objects degraded (31.685%), 12 pgs degraded pg 19.0 is active+recovery_wait+degraded, acting [9,2,3,13,11,5] pg 19.2 is active+recovery_wait+degraded, acting [2,6,0,11,7,12] pg 19.5 is active+recovery_wait+degraded, acting [5,1,0,11,4,12] pg 19.6 is active+recovery_wait+degraded, acting [1,6,3,8,7,11] pg 19.7 is active+recovery_wait+degraded, acting [13,6,10,3,7,8] pg 19.b is active+recovery_wait+degraded, acting [4,1,13,6,11,9] pg 19.c is active+recovery_wait+degraded, acting [8,6,11,7,13,2] pg 19.e is active+recovery_wait+degraded, acting [6,8,0,11,12,4] pg 19.f is active+recovery_wait+degraded, acting [5,10,13,1,0,4] pg 19.10 is active+recovery_wait+degraded, acting [3,11,5,9,7,1] pg 19.1a is active+recovery_wait+degraded, acting [7,0,8,1,11,5] pg 19.1f is active+recovery_wait+degraded, acting [7,5,8,3,1,11] [root@master1 tmp]# ceph osd pool list detail | grep default.rgw.buckets.data pool 19 'default.rgw.buckets.data' erasure profile default.rgw.buckets.data size 6 min_size 5 crush_rule 4 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 174 flags hashpspool stripe_width 16384 pg_num_min 32 target_size_ratio 69.8 application rgw The output of ceph health detail looks to me as if all the six chunks of the erasure coded data are still there as expected. But why are these placement groups shown as degraded? I'd expect only misplaced object chunks in this scenario the same as with replicated pools. Is there actually any reduction in redundancy or not? The cluster recovers after some time. I've also tested with more (256) placement groups but the outcome is the same. Thanks, LF. -- Lars Fenneberg, lf(a)elemental.net

2 2

Global recovery event but HEALTH_OK
by Alfredo Daniel Rezinovsky 22 Aug '23

22 Aug '23

I had many movement in my cluster. Broken node, replacement, rebalancing. Noy I'm stuck in upgrade to 18.2.0 (mgr and mon upgraded) and the cluster is in "Global Recovery Event" The health is OK I don't know how to search for the problem

2 1

Upcoming change to fix "ceph config dump" output inconsistency.
by Sridhar Seshasayee 22 Aug '23

22 Aug '23

Hello Everyone, Recently, an issue related to inconsistency in the output of "ceph config dump" command was reported. The inconsistency is between the normal (non-pretty-print) and pretty-print outputs. The non-pretty print output displays the localized option name whereas the pretty-print output displays the normalized option name. For e.g., Normalized: mgr/dashboard/ssl_server_port Localized: mgr/dashboard/x/ssl_server_port The fix ensures that the localized option name is shown in all cases. The issue is tracked in https://tracker.ceph.com/issues/62379 and the fix is not yet merged. This is to give a heads up in case you have any kind of automation that relies on the pretty-printed output (json, xml). This fix would soon be made available in upstream and downstream branches. If you have any concerns around this change, please let us know. Thanks, -Sridhar

1 0

Debian/bullseye build for reef
by Chris Palmer 21 Aug '23

21 Aug '23

I'd like to try reef, but we are on debian 11 (bullseye). In the ceph repos, there is debian-quincy/bullseye and debian-quincy/focal, but under reef there is only focal & jammy. Is there a reason why there is no reef/bullseye build? I had thought that the blocker only affected debian-bookworm builds. Thanks, Chris

3 4

osd: why not use aio in read?
by Xinying Song 21 Aug '23

21 Aug '23

Hi, guys: I'm using ceph 14 on HDD, and observed obvious high latency for pg.lock(). Further inspection shows the root cause seems to be the function pgbackend->objects_read_sync() called in PrimaryLogPG::do_read() which will hold the pg lock until the disk read finish. My question is why not use aio for read like what we are doing for write in bluestore? Is there any known problem for aio read in osd? Thanks in advance, Xinying Song

1 0

OSD delete vs destroy vs purge
by Nicola Mori 21 Aug '23

21 Aug '23

Dear Ceph users, I see that the OSD page of the Ceph dashboard offers three possibilities for "removing" an OSD: delete, destroy and purge. The delete operation has the possibility to flag the "Preserve OSD ID(s) for replacement." option. I searched for explanations of the differences between the three commands but I didn't find anything definitive, so I'd need some help with this. Thanks in advance, Nicola

3 4

Decrepit ceph cluster performance
by J David 21 Aug '23

21 Aug '23

I have a cluster that is performing *very* poorly. It has 12 physical servers and 72 400GB SATA Intel DC mixed-use SSD OSDs (S3700's & S3710's). All servers have bonded 10GB NICs for 20Gbps aggregate networking to a dedicated switch. At least one CPU core and 8GB RAM per OSD. No SMART errors. All SSDs have media wearout indicators >90% remaining. The cluster is Quincy 17.2.6 under cephadm. (It was recently upgraded from Octopus.) Its primary use is serving RBD VM block devices for Proxmox, but it is a standalone cluster. (The RBD bench above was run from one of the mon servers, so Proxmox is not at all involved here.) The rados bench for this is... problematic: gridfury@f13:~$ rados bench -p docker 60 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 60 seconds or 0 objects Object prefix: benchmark_data_f13_857310 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 220 204 815.924 816 0.0339855 0.0617994 2 16 252 236 471.942 128 0.0360448 0.0584063 3 16 431 415 553.261 716 0.0544013 0.11399 4 16 533 517 516.93 408 0.0536192 0.103342 5 16 553 537 429.541 80 0.0465437 0.120551 6 16 675 659 439.271 488 0.0390309 0.13919 7 16 704 688 393.087 116 0.0344105 0.134758 8 16 935 919 459.433 924 0.0443338 0.138301 9 16 1167 1151 511.48 928 0.0407041 0.124401 10 16 1219 1203 481.129 208 0.0408885 0.121026 11 16 1219 1203 437.389 0 - 0.121026 12 16 1333 1317 438.934 228 0.0436965 0.141108 13 16 1333 1317 405.169 0 - 0.141108 14 16 1333 1317 376.228 0 - 0.141108 15 16 1333 1317 351.146 0 - 0.141108 16 16 1333 1317 329.199 0 - 0.141108 17 16 1383 1367 321.597 40 0.0738412 0.198283 18 16 1621 1605 356.612 952 0.0396665 0.179042 19 16 1880 1864 392.361 1036 0.0349097 0.162829 2023-08-14T01:27:04.758111+0000 min lat: 0.0238844 max lat: 5.29731 avg lat: 0.152446 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 20 16 2111 2095 418.936 924 0.0581478 0.152446 21 16 2352 2336 444.881 964 0.0639009 0.143467 22 16 2592 2576 468.288 960 0.0921614 0.136478 23 16 2828 2812 488.965 944 0.0587983 0.130606 24 16 3050 3034 505.586 888 0.106586 0.126353 25 16 3167 3151 504.08 468 0.0606541 0.126778 26 16 3408 3392 521.763 964 0.0356258 0.12252 27 16 3644 3628 537.395 944 0.045652 0.118972 28 16 3844 3828 546.769 800 0.0328637 0.116728 29 16 3870 3854 531.501 104 0.0418716 0.11621 30 16 3870 3854 513.784 0 - 0.11621 31 16 3870 3854 497.21 0 - 0.11621 32 16 3870 3854 481.673 0 - 0.11621 33 16 3870 3854 467.076 0 - 0.11621 34 16 3870 3854 453.339 0 - 0.11621 35 16 3870 3854 440.386 0 - 0.11621 36 16 3870 3854 428.152 0 - 0.11621 37 16 3870 3854 416.581 0 - 0.11621 38 16 3870 3854 405.618 0 - 0.11621 39 16 3870 3854 395.218 0 - 0.11621 2023-08-14T01:27:24.761559+0000 min lat: 0.0238844 max lat: 5.29731 avg lat: 0.11621 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 40 16 3870 3854 385.337 0 - 0.11621 41 16 3870 3854 375.938 0 - 0.11621 42 16 3870 3854 366.987 0 - 0.11621 43 16 3870 3854 358.452 0 - 0.11621 44 16 3870 3854 350.306 0 - 0.11621 45 16 3870 3854 342.521 0 - 0.11621 46 16 3870 3854 335.075 0 - 0.11621 47 16 3870 3854 327.946 0 - 0.11621 48 16 3870 3854 321.114 0 - 0.11621 49 16 3870 3854 314.56 0 - 0.11621 50 16 3870 3854 308.269 0 - 0.11621 51 16 3870 3854 302.224 0 - 0.11621 52 16 3870 3854 296.412 0 - 0.11621 53 16 3870 3854 290.82 0 - 0.11621 54 16 3870 3854 285.434 0 - 0.11621 55 16 3870 3854 280.244 0 - 0.11621 56 16 3870 3854 275.24 0 - 0.11621 57 16 3870 3854 270.411 0 - 0.11621 58 16 3870 3854 265.749 0 - 0.11621 59 16 3870 3854 261.245 0 - 0.11621 2023-08-14T01:27:44.765065+0000 min lat: 0.0238844 max lat: 5.29731 avg lat: 0.11621 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 60 16 3870 3854 256.89 0 - 0.11621 61 16 3870 3854 252.679 0 - 0.11621 62 16 3870 3854 248.603 0 - 0.11621 63 16 3870 3854 244.657 0 - 0.11621 64 16 3870 3854 240.835 0 - 0.11621 65 16 3870 3854 237.129 0 - 0.11621 Total time run: 65.6256 Total writes made: 3870 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 235.884 Stddev Bandwidth: 375.227 Max bandwidth (MB/sec): 1036 Min bandwidth (MB/sec): 0 Average IOPS: 58 Stddev IOPS: 93.8068 Max IOPS: 259 Min IOPS: 0 Average Latency(s): 0.271253 Stddev Latency(s): 2.43809 Max latency(s): 37.7948 Min latency(s): 0.0238844 58 IOPs average is... less than I might hope for from this hardware. But those long strings of 0's... I feel like 0 IOPS is inarguably too low. At the time I ran this, I had ceph status running and saw +laggy pgs and actually caught a message: 36 slow ops, oldest one blocked for 37 sec, daemons [osd.10,osd.12,osd.13,osd.14,osd.15,osd.17,osd.2,osd.25,osd.28,osd.3]... So I checked a few osd logs; there is nothing during the test, but several have stuff immediately after like this: Aug 14 01:27:49 f13 ceph-osd[651856]: osd.2 pg_epoch: 110829 pg[2.591( v 110828'143791701 (110828'143791003,110828'143791701] local-lis/les=107797/107798 n=826 ec=1/1 lis/c=107797/107797 les/c/f=107798/107798/43927 sis=110829) [2] r=0 lpr=110829 pi=[10779 7,110829)/1 luod=0'0 lua=107760'143777660 crt=110828'143791701 lcod 110828'143791700 mlcod 0'0 active mbc={}] start_peering_interval up [48,2] -> [2], acting [48,2] -> [2], acting_primary 48 -> 2, up_primary 48 -> 2, role 1 -> 0, features acting 454013832 0759226367 upacting 4540138320759226367 Aug 14 01:27:49 f13 ceph-osd[651856]: osd.2 pg_epoch: 110829 pg[2.591( v 110828'143791701 (110828'143791003,110828'143791701] local-lis/les=107797/107798 n=826 ec=1/1 lis/c=107797/107797 les/c/f=107798/107798/43927 sis=110829) [2] r=0 lpr=110829 pi=[10779 7,110829)/1 crt=110828'143791701 lcod 110828'143791700 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary Aug 14 01:27:49 f13 ceph-osd[651856]: osd.2 pg_epoch: 110829 pg[5.b( v 98878'16 (0'0,98878'16] local-lis/les=107797/107798 n=4 ec=98376/98376 lis/c=107797/107797 les/c/f=107798/107798/0 sis=110829) [2,14] r=0 lpr=110829 pi=[107797,110829)/1 crt=98878'16 l cod 0'0 mlcod 0'0 active mbc={}] start_peering_interval up [2,48,14] -> [2,14], acting [2,48,14] -> [2,14], acting_primary 2 -> 2, up_primary 2 -> 2, role 0 -> 0, features acting 4540138320759226367 upacting 4540138320759226367 Aug 14 01:27:49 f13 ceph-osd[651856]: osd.2 pg_epoch: 110829 pg[5.b( v 98878'16 (0'0,98878'16] local-lis/les=107797/107798 n=4 ec=98376/98376 lis/c=107797/107797 les/c/f=107798/107798/0 sis=110829) [2,14] r=0 lpr=110829 pi=[107797,110829)/1 crt=98878'16 l cod 0'0 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary Aug 14 01:27:49 f13 ceph-osd[651856]: osd.2 pg_epoch: 110829 pg[2.30c( v 110828'58472450 (107939'58468576,110828'58472450] local-lis/les=107797/107798 n=840 ec=1/1 lis/c=107797/107797 les/c/f=107798/107800/43927 sis=110829) [2] r=0 lpr=110829 pi=[107797,1 10829)/1 luod=110828'58472444 crt=110828'58472450 lcod 110828'58472443 mlcod 110828'58472443 active mbc={255={}}] start_peering_interval up [2,48] -> [2], acting [2,48] -> [2], acting_primary 2 -> 2, up_primary 2 -> 2, role 0 -> 0, features acting 4540138 320759226367 upacting 4540138320759226367 Aug 14 01:27:49 f13 ceph-osd[651856]: osd.2 pg_epoch: 110829 pg[2.326( v 110828'84969926 (110432'84966024,110828'84969926] local-lis/les=107797/107798 n=831 ec=1/1 lis/c=107797/107797 les/c/f=107798/107798/43927 sis=110829) [2] r=0 lpr=110829 pi=[107797,1 10829)/1 luod=110828'84969917 crt=110828'84969926 lcod 110828'84969916 mlcod 110828'84969916 active mbc={255={}}] start_peering_interval up [2,48] -> [2], acting [2,48] -> [2], acting_primary 2 -> 2, up_primary 2 -> 2, role 0 -> 0, features acting 4540138 320759226367 upacting 4540138320759226367 Aug 14 01:27:49 f13 ceph-osd[651856]: osd.2 pg_epoch: 110829 pg[2.30c( v 110828'58472450 (107939'58468576,110828'58472450] local-lis/les=107797/107798 n=840 ec=1/1 lis/c=107797/107797 les/c/f=107798/107800/43927 sis=110829) [2] r=0 lpr=110829 pi=[107797,1 10829)/1 crt=110828'58472450 lcod 110828'58472443 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary Aug 14 01:27:49 f13 ceph-osd[651856]: osd.2 pg_epoch: 110829 pg[2.326( v 110828'84969926 (110432'84966024,110828'84969926] local-lis/les=107797/107798 n=831 ec=1/1 lis/c=107797/107797 les/c/f=107798/107798/43927 sis=110829) [2] r=0 lpr=110829 pi=[107797,1 10829)/1 crt=110828'84969926 lcod 110828'84969916 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary Aug 14 01:27:57 f13 ceph-osd[651856]: --2- [v2:192.168.2.213:6804/3119259815,v1:192.168.2.213:6805/3119259815] >> [v2:192.168.2.225:6808/1596776081,v1:192.168.2.225:6809/1596776081] conn(0x55c3dc924400 0x55c3b8555b80 crc :-1 s=SESSION_ACCEPTING pgs=167 cs =0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_reconnect no existing connection exists, reseting client Aug 14 01:27:58 f13 ceph-osd[651856]: osd.2 pg_epoch: 110833 pg[2.591( v 110828'143791701 (110828'143791003,110828'143791701] local-lis/les=110829/110830 n=826 ec=1/1 lis/c=110829/107797 les/c/f=110830/107798/43927 sis=110833 pruub=15.513665199s) [48,2] r =1 lpr=110833 pi=[107797,110833)/1 crt=110828'143791701 lcod 110828'143791700 mlcod 0'0 active pruub 22348.017578125s@ mbc={}] start_peering_interval up [2] -> [48,2], acting [2] -> [48,2], acting_primary 2 -> 48, up_primary 2 -> 48, role 0 -> 1, features acting 4540138320759226367 upacting 4540138320759226367 Aug 14 01:27:58 f13 ceph-osd[651856]: osd.2 pg_epoch: 110833 pg[2.591( v 110828'143791701 (110828'143791003,110828'143791701] local-lis/les=110829/110830 n=826 ec=1/1 lis/c=110829/107797 les/c/f=110830/107798/43927 sis=110833 pruub=15.513574600s) [48,2] r =1 lpr=110833 pi=[107797,110833)/1 crt=110828'143791701 lcod 110828'143791700 mlcod 0'0 unknown NOTIFY pruub 22348.017578125s@ mbc={}] state<Start>: transitioning to Stray Aug 14 01:27:58 f13 ceph-osd[651856]: osd.2 pg_epoch: 110833 pg[5.b( v 98878'16 (0'0,98878'16] local-lis/les=110829/110830 n=4 ec=98376/98376 lis/c=110829/107797 les/c/f=110830/107798/0 sis=110833 pruub=15.519681931s) [2,48,14] r=0 lpr=110833 pi=[107797,1 10833)/1 crt=98878'16 lcod 0'0 mlcod 0'0 active pruub 22348.023437500s@ mbc={}] start_peering_interval up [2,14] -> [2,48,14], acting [2,14] -> [2,48,14], acting_primary 2 -> 2, up_primary 2 -> 2, role 0 -> 0, features acting 4540138320759226367 upacting 4540138320759226367 Aug 14 01:27:58 f13 ceph-osd[651856]: osd.2 pg_epoch: 110833 pg[5.b( v 98878'16 (0'0,98878'16] local-lis/les=110829/110830 n=4 ec=98376/98376 lis/c=110829/107797 les/c/f=110830/107798/0 sis=110833 pruub=15.519681931s) [2,48,14] r=0 lpr=110833 pi=[107797,1 10833)/1 crt=98878'16 lcod 0'0 mlcod 0'0 unknown pruub 22348.023437500s@ mbc={}] state<Start>: transitioning to Primary Aug 14 01:27:58 f13 ceph-osd[651856]: osd.2 pg_epoch: 110833 pg[2.326( v 110828'84969926 (110432'84966024,110828'84969926] local-lis/les=110829/110830 n=831 ec=1/1 lis/c=110829/107797 les/c/f=110830/107798/43927 sis=110833 pruub=15.512885094s) [2,48] r=0 lpr=110833 pi=[107797,110833)/1 crt=110828'84969926 lcod 110828'84969916 mlcod 0'0 active pruub 22348.017578125s@ mbc={}] start_peering_interval up [2] -> [2,48], acting [2] -> [2,48], acting_primary 2 -> 2, up_primary 2 -> 2, role 0 -> 0, features acting 4540138320759226367 upacting 4540138320759226367 Aug 14 01:27:58 f13 ceph-osd[651856]: osd.2 pg_epoch: 110833 pg[2.326( v 110828'84969926 (110432'84966024,110828'84969926] local-lis/les=110829/110830 n=831 ec=1/1 lis/c=110829/107797 les/c/f=110830/107798/43927 sis=110833 pruub=15.512885094s) [2,48] r=0 lpr=110833 pi=[107797,110833)/1 crt=110828'84969926 lcod 110828'84969916 mlcod 0'0 unknown pruub 22348.017578125s@ mbc={}] state<Start>: transitioning to Primary Aug 14 01:27:58 f13 ceph-osd[651856]: osd.2 pg_epoch: 110833 pg[2.30c( v 110828'58472450 (107939'58468576,110828'58472450] local-lis/les=110829/110830 n=840 ec=1/1 lis/c=110829/107797 les/c/f=110830/107800/43927 sis=110833 pruub=15.512117386s) [2,48] r=0 lpr=110833 pi=[107797,110833)/1 crt=110828'58472450 lcod 110828'58472443 mlcod 0'0 active pruub 22348.017578125s@ mbc={}] start_peering_interval up [2] -> [2,48], acting [2] -> [2,48], acting_primary 2 -> 2, up_primary 2 -> 2, role 0 -> 0, features acting 4540138320759226367 upacting 4540138320759226367 Aug 14 01:27:58 f13 ceph-osd[651856]: osd.2 pg_epoch: 110833 pg[2.30c( v 110828'58472450 (107939'58468576,110828'58472450] local-lis/les=110829/110830 n=840 ec=1/1 lis/c=110829/107797 les/c/f=110830/107800/43927 sis=110833 pruub=15.512117386s) [2,48] r=0 lpr=110833 pi=[107797,110833)/1 crt=110828'58472450 lcod 110828'58472443 mlcod 0'0 unknown pruub 22348.017578125s@ mbc={}] state<Start>: transitioning to Primary I also captured iostat -x 5 from several of the OSDs during the test, including one of the ones identified as having blocked IO. It shows 3-15% utilized with 30-200 IOPS (mostly write). r_await average 0.22, w_await average 0.46. Nothing in dmesg or syslog. Is there anything to be done to further investigate or remediate this? This cluster is pretty old and I'm starting to think we should just scrap it as hopeless. But that's a lot of hardware to write off if there's any possible way to fix this. Thanks for any advice!

5 10

v18.2.0 Reef released
by Yuri Weinstein 19 Aug '23

19 Aug '23

We're very happy to announce the first stable release of the Reef series. We express our gratitude to all members of the Ceph community who contributed by proposing pull requests, testing this release, providing feedback, and offering valuable suggestions. Major Changes from Quincy: - RADOS: RocksDB has been upgraded to version 7.9.2. - RADOS: There have been significant improvements to RocksDB iteration overhead and performance. - RADOS: The perf dump and perf schema commands have been deprecated in favor of the new counter dump and counter schema commands. - RADOS: Cache tiering is now deprecated. - RADOS: A new feature, the "read balancer", is now available, which allows users to balance primary PGs per pool on their clusters. - RGW: Bucket resharding is now supported for multi-site configurations. - RGW: There have been significant improvements to the stability and consistency of multi-site replication. - RGW: Compression is now supported for objects uploaded with Server-Side Encryption. - Dashboard: There is a new Dashboard page with improved layout. Active alerts and some important charts are now displayed inside cards. - RBD: Support for layered client-side encryption has been added. - Telemetry: Users can now opt in to participate in a leaderboard in the telemetry public dashboards. We encourage you to read the full release notes at https://ceph.io/en/news/blog/2023/v18-2-0-reef-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-18.2.0.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 5dd24139a1eada541a3bc16b6941c5dde975e26d Did you know? Every Ceph release is built and tested on resources funded directly by the non-profit Ceph Foundation. If you would like to support this and our other efforts, please consider joining now https://ceph.io/en/foundation/.

4 3

radosgw-admin sync error trim seems to do nothing
by Matthew Darwin 19 Aug '23

19 Aug '23

Hello all, "radosgw-admin sync error list" returns errors from 2022. I want to clear those out. I tried "radosgw-admin sync error trim" but it seems to do nothing. The man page seems to offer no suggestions https://docs.ceph.com/en/quincy/man/8/radosgw-admin/ Any ideas what I need to do to remove old errors? (or at least I want to see more recent errors) ceph version 17.2.6 (quincy) Thanks.

1 0

Degraded FS on 18.2.0 - two monitors per host????
by Robert W. Eckert 18 Aug '23

18 Aug '23

Hi - I have a 4 node cluster, and started to have some odd access issues to my file system "Home" When I started investigating, saw the message "1 MDSs behind on trimming", but I also noticed that I seem to have 2 MDSs running on each server - 3 Daemons up, with 5 standby. Is this expected behavior after the upgrade to 18.2? or did something go wrong? [root@cube ~]# ceph status cluster: id: fe3a7cb0-69ca-11eb-8d45-c86000d08867 health: HEALTH_WARN 1 filesystem is degraded 1 MDSs behind on trimming services: mon: 3 daemons, quorum rhel1,cube,hiho (age 23m) mgr: hiho.bphqff(active, since 23m), standbys: rhel1.owrvaz, cube.sdhftu mds: 3/3 daemons up, 5 standby osd: 16 osds: 16 up (since 23m), 16 in (since 26h) rgw: 4 daemons active (4 hosts, 1 zones) data: volumes: 0/1 healthy, 1 recovering pools: 12 pools, 769 pgs objects: 3.64M objects, 3.1 TiB usage: 17 TiB used, 49 TiB / 65 TiB avail pgs: 765 active+clean 4 active+clean+scrubbing+deep io: client: 154 MiB/s rd, 38 op/s rd, 0 op/s wr [root@cube ~]# ceph health detail HEALTH_WARN 1 filesystem is degraded; 1 MDSs behind on trimming [WRN] FS_DEGRADED: 1 filesystem is degraded fs home is degraded [WRN] MDS_TRIM: 1 MDSs behind on trimming mds.home.story.sodtjs(mds.0): Behind on trimming (5546/128) max_segments: 128, num_segments: 5546 [root@cube ~]# ceph fs status home home - 10 clients ==== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay home.story.sodtjs<http://home.story.sodtjs> 802k 766k 36.7k 0 1 resolve home.cube.xljmfz<http://home.cube.xljmfz> 735k 680k 39.0k 0 2 resolve home.rhel1.nwpmbg<http://home.rhel1.nwpmbg> 322k 316k 17.5k 0 POOL TYPE USED AVAIL home.meta metadata 361G 14.9T home.data data 9206G 14.9T STANDBY MDS home.rhel1.ffrufi<http://home.rhel1.ffrufi> home.hiho.mssdyh<http://home.hiho.mssdyh> home.cube.kmpbku<http://home.cube.kmpbku> home.hiho.cfuswn<http://home.hiho.cfuswn> home.story.gmieio<http://home.story.gmieio> MDS version: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)

2 2

Re: [ceph v16.2.10] radosgw crash
by 1187873955 18 Aug '23

18 Aug '23

Here is  the backtrace info in  core file (gdb) bt #0  0x00007f79da065b7f in raise () from /lib64/libpthread.so.0 #1  0x00007f79e5303563 in reraise_fatal (signum=6) at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/src/global/signal_handler.cc:332 #2  handle_fatal_signal (signum=6) at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/src/global/signal_handler.cc:332 #3  <signal handler called> #4  0x00007f79d86b6a9f in raise () from /lib64/libc.so.6 #5  0x00007f79d8689e05 in abort () from /lib64/libc.so.6 #6  0x00007f79d905809b in __gnu_cxx::__verbose_terminate_handler() [clone .cold.1] () from /lib64/libstdc++.so.6 #7  0x00007f79d905e53c in __cxxabiv1::__terminate(void (*)()) () from /lib64/libstdc++.so.6 #8  0x00007f79d905e597 in std::terminate() () from /lib64/libstdc++.so.6 #9  0x00007f79d905e52e in std::rethrow_exception(std::__exception_ptr::exception_ptr) () from /lib64/libstdc++.so.6 #10 0x00007f79e4b70a17 in spawn::detail::continuation_context::resume (this=<optimized out>) at /usr/include/c++/8/bits/exception_ptr.h:107 #11 0x00007f79e4b77b6a in spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >::operator() (value=..., ec=..., this=0x7f7916eeedd0) at /usr/include/c++/8/bits/shared_ptr_base.h:1018 #12 std::__invoke_impl<void, spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > (     __f=...) at /usr/include/c++/8/bits/invoke.h:60 #13 std::__invoke<spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > (__fn=...)     at /usr/include/c++/8/bits/invoke.h:95 #14 std::__apply_impl<spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, std::tuple<boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, 0ul, 1ul> (__t=..., __f=...) at /usr/include/c++/8/tuple:1678 #15 std::apply<spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, std::tuple<boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > > (     __t=..., __f=...) at /usr/include/c++/8/tuple:1687 #16 ceph::async::CompletionHandler<spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, std::tuple<boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > >::operator()() && (this=0x7f7916eeedd0) at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/src/common/async/bind_handler.h:52 #17 ceph::async::ForwardingHandler<ceph::async::CompletionHandler<spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, std::tuple<boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > > >::operator()<>() (this=0x7f7916eeedd0) at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/src/common/async/forward_handler.h:47 #18 boost::asio::asio_handler_invoke<ceph::async::ForwardingHandler<ceph::async::CompletionHandler<spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, std::tuple<boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > > > > (function=...)     at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/handler_invoke_hook.hpp:69 #19 boost_asio_handler_invoke_helpers::invoke<ceph::async::ForwardingHandler<ceph::async::CompletionHandler<spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, std::tuple<boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > > >, ceph::async::ForwardingHandler<ceph::async::CompletionHandler<spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, std::tuple<boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > > > > (context=..., function=...)     at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/detail/handler_invoke_helpers.hpp:37 #20 boost::asio::detail::executor_op<ceph::async::ForwardingHandler<ceph::async::CompletionHandler<spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, std::tuple<boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > > >, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete (owner=0x55fd64378910,     base=<optimized out>) at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/detail/executor_op.hpp:70 #21 0x00007f79e4b7f93d in boost::asio::detail::scheduler_operation::complete (bytes_transferred=0, ec=..., owner=<optimized out>, this=<optimized out>)     at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/detail/scheduler_operation.hpp:40 #22 boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>::operator() (this=this@entry=0x7f7916eeeed0)     at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/detail/impl/strand_executor_service.hpp:90 #23 0x00007f79e4b7fca6 in boost::asio::asio_handler_invoke<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const> > (function=...) --Type <RET> for more, q to quit, c to continue without paging--     at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/handler_invoke_hook.hpp:67 #24 boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>, boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const> > (context=..., function=...)     at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/detail/handler_invoke_helpers.hpp:37 #25 boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>, boost::asio::detail::recycling_allocator<void, boost::asio::detail::thread_info_base::default_tag>, boost::asio::detail::scheduler_operation>::do_complete (owner=0x55fd63db8300, base=<optimized out>)     at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/detail/executor_op.hpp:70 #26 0x00007f79e4b73ad2 in boost::asio::detail::scheduler_operation::complete (bytes_transferred=0, ec=..., owner=0x55fd63db8300, this=<optimized out>)     at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/detail/scheduler_operation.hpp:40 #27 boost::asio::detail::scheduler::do_run_one (ec=..., this_thread=..., lock=..., this=0x55fd63db8300)     at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/detail/impl/scheduler.ipp:447 #28 boost::asio::detail::scheduler::run (this=0x55fd63db8300, ec=...) at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/detail/impl/scheduler.ipp:200 #29 0x00007f79e4b56376 in boost::asio::io_context::run (this=<optimized out>, ec=...)     at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/impl/io_context.ipp:71 #30 (anonymous namespace)::AsioFrontend::<lambda()>::operator() (__closure=0x55fd640cf148) at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/src/rgw/rgw_asio_frontend.cc:1025 #31 std::__invoke_impl<void, (anonymous namespace)::AsioFrontend::run()::<lambda()> > (__f=...) at /usr/include/c++/8/bits/invoke.h:60 #32 std::__invoke<(anonymous namespace)::AsioFrontend::run()::<lambda()> > (__fn=...) at /usr/include/c++/8/bits/invoke.h:95 #33 std::thread::_Invoker<std::tuple<(anonymous namespace)::AsioFrontend::run()::<lambda()> > >::_M_invoke<0> (this=0x55fd640cf148) at /usr/include/c++/8/thread:244 #34 std::thread::_Invoker<std::tuple<(anonymous namespace)::AsioFrontend::run()::<lambda()> > >::operator() (this=0x55fd640cf148) at /usr/include/c++/8/thread:253 #35 std::thread::_State_impl<std::thread::_Invoker<std::tuple<(anonymous namespace)::AsioFrontend::run()::<lambda()> > > >::_M_run(void) (this=0x55fd640cf140)     at /usr/include/c++/8/thread:196 #36 0x00007f79d908aba3 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #37 0x00007f79da05b1ca in start_thread () from /lib64/libpthread.so.0 #38 0x00007f79d86a1dd3 in clone () from /lib64/libc.so.6

1 0

Re: Check allocated RGW bucket/object size after enabling Bluestore compression
by Konstantin Shalygin 17 Aug '23

17 Aug '23

Hi, What you mean, Bluestore compression? The rgw compression is a rados compression, not the compress by rgw itself. You can setup different storage classes and upload to same pool uncompressed, or compressed objects The compression ratio you can determine with exporter [1] [1] https://github.com/blemmenes/radosgw_usage_exporter k Sent from my iPhone > On 17 Aug 2023, at 17:58, yosr.kchaou96(a)gmail.com wrote: > I see that if we enable RGW compression, we can see the actual (before compression) and utilized (after compression) size of the bucket. > Is it possible to find the same information but with Bluestore compression ?

3 4

Check allocated RGW bucket/object size after enabling Bluestore compression
by yosr.kchaou96＠gmail.com 17 Aug '23

17 Aug '23

Hello, I have enabled Bluestore compression and I see that my data is being compressed by checking the following metrics: bash-4.4$ ceph tell osd.0 perf dump | grep compre "compress_success_count": 24064, "compress_rejected_count": 1, "bluestore_compressed": 1617592, "bluestore_compressed_allocated": 98844672, "bluestore_compressed_original": 197689344, "bluestore_extent_compress": 107478, So far I was able to check the size the compressed data only on osd and pool level, but not on S3 bucket or object level. I see that if we enable RGW compression, we can see the actual (before compression) and utilized (after compression) size of the bucket. Is it possible to find the same information but with Bluestore compression ? Thanks Yosr

1 0

Lost buckets when moving OSD location
by Huy Nguyen 17 Aug '23

17 Aug '23

Hi, I'm using a v16.2.13 Ceph cluster. Yesterday, I just add some SSD node for replace HDD node. During the process, 1 SSD node have different MTU that cause the some PGs become unactive for a while. But after change the MTU, all the PGs is active+clean now. But after that, I can't access some buckets despite their metadata is still there: $ radosgw-admin bucket stats --bucket new-bucket failure: (2002) Unknown error 2002: $ radosgw-admin bucket list --uid owner_id [ ..., "new-bucket", "tesst" ] $ radosgw-admin metadata get bucket.instance:new-bucket:699abe5c-9603-4718-9073-938888ed8bbb.4961564.2 { "key": "bucket.instance:new-bucket:699abe5c-9603-4718-9073-938888ed8bbb.4961564.2", "ver": { "tag": "_ppS69qh0QYSDPuH1I-kp51a", "ver": 32 }, "mtime": "2023-04-03T01:54:24.445009Z", "data": { "bucket_info": { "bucket": { "name": "new-bucket", "marker": "699abe5c-9603-4718-9073-938888ed8bbb.4961564.2", "bucket_id": "699abe5c-9603-4718-9073-938888ed8bbb.4961564.2", "tenant": "", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" } }, "creation_time": "2022-03-15T08:14:13.199771Z", "owner": "0e0481b66c814e0c9d34b5bee8779205", "flags": 6, "zonegroup": "a453e527-0ded-4e8f-a51e-7eff2293b056", "placement_rule": "default-placement", "has_instance_obj": "true", "quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "num_shards": 0, "bi_shard_hash_type": 0, "requester_pays": "false", "has_website": "false", "swift_versioning": "false", "swift_ver_location": "", "index_type": 0, "mdsearch_config": [], "reshard_status": 0, "new_bucket_instance_id": "" }, "attrs": [ { "key": "user.rgw.acl", "val": "AgJNAQAAAwI+AAAAIAAAADBlMDQ4MWI2NmM4MTRlMGM5ZDM0YjViZWU4Nzc5MjA1FgAAAGFuaHZ0MTIxMjEyQHZjY2xvdWQudm4EAwMBAAABAQAAACAAAAAwZTA0ODFiNjZjODE0ZTBjOWQzNGI1YmVlODc3OTIwNQ8AAAACAAAAAAAAAAUDLAAAAAICBAAAAAIAAAAAAAAAAAAAAAAAAAACAgQAAAABAAAAAAAAAAEAAAAAAAAAIAAAADBlMDQ4MWI2NmM4MTRlMGM5ZDM0YjViZWU4Nzc5MjA1BQNiAAAAAgIEAAAAAAAAACAAAAAwZTA0ODFiNjZjODE0ZTBjOWQzNGI1YmVlODc3OTIwNQAAAAAAAAAAAgIEAAAADwAAABYAAABhbmh2dDEyMTIxMkB2Y2Nsb3VkLnZuAAAAAAAAAAABAAAAAQAAAAEAAAAAAAAA" }, { "key": "user.rgw.iam-policy", "val": "eyJWZXJzaW9uIjogIjIwMTItMTAtMTciLCAiU3RhdGVtZW50IjogW3siRWZmZWN0IjogIkFsbG93IiwgIlByaW5jaXBhbCI6IHsiQVdTIjogWyJhcm46YXdzOmlhbTo6OnVzZXIvdW5kZWZpbmVkOmh1eW5ucC1jcmVhdGUtc3VidXNlci1mcm9tLXJnd2FkbWluIl19LCAiQWN0aW9uIjogWyJzMzpDcmVhdGVCdWNrZXQiLCAiczM6RGVsZXRlQnVja2V0Il0sICJSZXNvdXJjZSI6IFsiYXJuOmF3czpzMzo6Om5ldy1idWNrZXQiLCAiYXJuOmF3czpzMzo6Om5ldy1idWNrZXQvKiJdfV19" } ] } } Do any of you have any ideas on how to get these buckets back? I really appreciate it. Thanks

1 0

Ceph Tech Talk for August 2023: Making Tehthology Friendly
by Mike Perez 17 Aug '23

17 Aug '23

Hi everyone, Join us tomorrow at 15:00 UTC to hear from our Google Summer of Code and Outreachy interns, Devansh Singh and Medhavi Singh, in this next Ceph Tech Talk on Making Teuthology Friendly. https://ceph.io/en/community/tech-talks/ If you want to give atechnicalpresentation for CephTechTalks, please contact me directly with a title and description. Thank you! -- Mike Perez Community Manager Ceph Foundation

1 1

ceph orch upgrade stuck between 16.2.7 and 16.2.13
by Robert Sander 16 Aug '23

16 Aug '23

Hi, A healthy 16.2.7 cluster should get an upgrade to 16.2.13. ceph orch upgrade start --ceph-version 16.2.13 did upgrade MONs, MGRs and 25% of the OSDs and is now stuck. We tried several "ceph orch upgrade stop" and starts again. We "failed" the active MGR but no progress. We set the debug logging with "ceph config set mgr mgr/cephadm/log_to_cluster_level debug" but it only tells that it starts: 2023-08-15T09:05:58.548896+0200 mgr.cephmon01 [INF] Upgrade: Started with target quay.io/ceph/ceph:v16.2.13 How can we check what is happening (or not happening) here? How do we get cephadm to complete the task? Current status is: # ceph orch upgrade status { "target_image": "quay.io/ceph/ceph:v16.2.13", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "", "message": "", "is_paused": false } # ceph -s cluster: id: 3098199a-c7f5-4baf-901c-f178131be6f4 health: HEALTH_WARN There are daemons running an older version of ceph services: mon: 5 daemons, quorum cephmon02,cephmon01,cephmon03,cephmon04,cephmon05 (age 4d) mgr: cephmon03(active, since 8d), standbys: cephmon01, cephmon02 mds: 2/2 daemons up, 1 standby, 2 hot standby osd: 202 osds: 202 up (since 11d), 202 in (since 13d) rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 2/2 healthy pools: 11 pools, 4961 pgs objects: 98.84M objects, 347 TiB usage: 988 TiB used, 1.3 PiB / 2.3 PiB avail pgs: 4942 active+clean 19 active+clean+scrubbing+deep io: client: 89 MiB/s rd, 598 MiB/s wr, 25 op/s rd, 157 op/s wr progress: Upgrade to quay.io/ceph/ceph:v16.2.13 (0s) [............................] # ceph versions { "mon": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 3 }, "osd": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 48, "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 154 }, "mds": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 5 }, "rgw": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 2 }, "overall": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 56, "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 161 } } Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

3 8

CephFS metadata outgrow DISASTER during recovery
by Jakub Petrzilka 16 Aug '23

16 Aug '23

Hello everyone! Recently we had a very nasty incident with one of our CEPH storages. During basic backfill recovery operation due to faulty disk CephFS metadata started growing exponentially until they used all available space and whole cluster DIED. Usage graph screenshot in attachment. Everything was very fast and even when the OSDs were marked full they tripped failsafe and ate all the free blocks, still trying to allocate space and completely died without possibility to even start them again. Only solution was to copy whole bluestore to bigger SSD and resize underlying BS device. Just about 1/3 was able to start after moving but it was enough since we have very redundant settings for cephfs metadata. Basically metadata were moved from 12x 240g SSD to 12x 500GB SSD to have enough space to start again. Brief info about the cluster: - CephFS data are stored on ~500x 8TB SAS HDD using 10+2 ECC in 18 hosts. - CephFS metadata are stored on ~12x 500GB SAS/SATA SSD using 5x replication on 6 hosts. - Version was one of the latest 16.x.x Pacific at the time of the incident. - 3x Mon+mgr and 2 active and 2 hot standby MDS are on separate virtual servers. - typical file size to be stored is from hundreds of MBs to tens of GBs. - this cluster is not the biggest, not having the most HDDs, no special config, I simply see nothing special about this cluster. During investigation I found out the following: - Metadata are outgrowing any time recovery is running on any of maintained clusters (~15 clusters of different usages and sizes) but not this much, this was an extreme situation. - after recovery finish size went fine again. - i think there is slight correlation with recovery width (objects to be touched by recovery in order to recovery everything) and recovery (time) length. But i have no proof. - nothing much else I would like to find out why this happened because i think this can happen again sometime and someone might lose some data if they have less luck. Any ideas are appreciated, or even info if anyone have seen any similar behavior or if i am the only one struggling with issue like this :) Kind regards, Jakub Petrzilka

3 5

Re: Can't join new mon - lossy channel, failing
by Josef Johansson 16 Aug '23

16 Aug '23

Hi, Let's do some serious necromancy here. I just had this exact problem. Turns out that after rebooting all nodes (one at the time of course), the monitor could join perfectly. Why? You tell me. We did not see any traces of the ip address in any dumps that we could get a hold of. I restarted all ceph-mgr beforehand as well. Med vänliga hälsningar Josef Johansson On 10/5/21 15:37, Konstantin Shalygin wrote: > As last resort we've change ipaddr of this host, and mon successfully joined to quorum. When revert ipaddr back - mon can't join, we think there something on switch side or on old mon's side. From old mon's I was checked new mon process connectivity via telnet - all works > It's good to make a some reproducer of this network problem to know what exactly message of ceph protocol is broken > > > > k > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 3

Cephadm adoption - service reconfiguration changes container image
by Iain Stott 16 Aug '23

16 Aug '23

Hi Everyone, We are looking at migrating all our production clusters from ceph-ansible to cephadm. We are currently experiencing an issue where when reconfiguring a service through ceph orch, it will change the running container image for that service which has led to the mgr services running an earlier version than the rest of the cluster, this has caused cephadm/ceph orch to not be able to manage services in the cluster. Does anyone have any help with this as I cannot find anything in the docs that would correct this. Cheers Iain [root@de1-ceph-mon-ceph-site-a-1 ~]# cephadm --image quay.io/ceph/ceph:v16.2.9 adopt --style legacy --name mon.$(hostname -s) [root@de1-ceph-mon-ceph-site-a-1 ~]# cat ./mon.yaml service_type: mon service_name: mon placement: host_pattern: '*mon*' extra_container_args: - -v - /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.client.admin.keyring [root@de1-ceph-mon-ceph-site-a-1 ~]# cephadm shell -- ceph config get mgr mgr/cephadm/container_image_base quay.io/ceph/ceph [root@de1-ceph-mon-ceph-site-a-1 ~]# podman ps -a | grep mon 55fe8bc10476 quay.io/ceph/ceph:v16.2.9 -n mgr.de1-ceph-m... 5 minutes ago Up 5 minutes ceph-da9e9837-a3cf-4482-9a13-790a721598cd-mgr-de1-ceph-mon-ceph-site-a-1 [root@de1-ceph-mon-ceph-site-a-1 ~]# cat ./mon.yaml | cephadm --image quay.io/ceph/ceph:v16.2.9 shell -- ceph orch apply -i - Inferring fsid da9e9837-a3cf-4482-9a13-790a721598cd Scheduled mon update... [root@de1-ceph-mon-ceph-site-a-1 ~]# podman ps -a | grep mon ecec1d62c719 docker.io/ceph/daemon-base:latest-pacific-devel -n mon.de1-ceph-m... 25 seconds ago Up 26 seconds ceph-da9e9837-a3cf-4482-9a13-790a721598cd-mon-de1-ceph-mon-ceph-site-a-1 [root@de1-ceph-mon-ceph-site-a-2 ~]# ceph versions { "mon": { "ceph version 16.2.5-387-g7282d81d (7282d81d2c500b5b0e929c07971b72444c6ac424) pacific (stable)": 3 }, "mgr": { "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 3 }, "osd": { "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 8 }, "mds": {}, "rgw": { "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 3 }, "overall": { "ceph version 16.2.5-387-g7282d81d (7282d81d2c500b5b0e929c07971b72444c6ac424) pacific (stable)": 3, "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 14 } } Iain Stott OpenStack Engineer Iain.Stott(a)thehutgroup.com [THG Ingenuity Logo]<https://www.thg.com> [https://i.imgur.com/wbpVRW6.png]<https://www.linkedin.com/company/thgplc/?originalSubdomain=uk> [https://i.imgur.com/c3040tr.png] <https://twitter.com/thgplc?lang=en>

4 4

Multisite s3 website slow period update
by Ondřej Kukla 15 Aug '23

15 Aug '23

Hello, I’ve had a quite unpleasant experience today that I would like to share. In our setup we use two set’s of RGW one that has only s3 and admin API and a second set with s3website and admin API. I was changing the global quota setting which means that I’ve then needed to commit the updated period. The first set of s3 RGWs updated without issue, but the s3website RGWs not. They were somehow stuck at the period update took minutes in some cases meaning service disruption for MINUTES! When I’ve looked at the RGW logs I’ve found this line that seems to show the issue. rgw realm reloader: Pausing frontends for realm update... req 3935991300378050219 61.212696075s s3:get_obj iterate_obj() failed with -104 We are running on ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Did someone had a similar issue? Is there a something we can do about it? Is there an option to update the period machine by machine or some other way that would let us to update the period and not disrupt the service? Your help is much appreciated. Kind regards, Ondrej

1 0

Announcing go-ceph v0.23.0
by Sven Anderson 15 Aug '23

15 Aug '23

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.23.0 The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Sven

1 0

OSD containers lose connectivity after change from Rocky 8.7->9.2
by Dan O'Brien 15 Aug '23

15 Aug '23

I recently updated one of the hosts (an older Dell PowerEdge R515) in my Ceph Quincy (17.2.6) cluster. I needed to change the IP address, so I removed the host from the cluster (gracefully removed OSDs and daemons, then removed the host). I also took the opportunity to upgrade the host from Rocky 8.7 to 9.2 before re-joining it to the cluster with cephadm. I zapped the storage, so for all intents and purposes, it should have been a completely clean instance and went smoothly. I have 2 other hosts (new Dell PowerEdge R450s) using Rocky 9.2 with no problems. Before the upgrade, the R515 host was well-behaved and unremarkable. Our cluster is connected to our internal network, and has a 10G private network used for interconnect between the nodes. Since the upgrade, the OSDs on the R515 host regularly, after a period of minutes to hours (usually a few hours), drop out of the cluster. I can restart the OSDs and they immediately reconnect and join the cluster, which returns to a HEALTHY state after a short period. The OSD logs show Aug 15 06:53:57 ceph99.cecnet.gmu.edu ceph-osd[193725]: log_channel(cluster) log [WRN] : Monitor daemon marked osd.9 down, but it is still running Aug 15 06:53:57 ceph99.cecnet.gmu.edu ceph-osd[193725]: log_channel(cluster) log [DBG] : map e17993 wrongly marked me down at e17988 Aug 15 06:53:57 ceph99.cecnet.gmu.edu ceph-osd[193725]: osd.9 17993 start_waiting_for_healthy Aug 15 06:53:57 ceph99.cecnet.gmu.edu ceph-osd[193725]: osd.9 pg_epoch: 17988 pg[16.f( v 16315'2765702 (15197'2757704,16315'2765702] local-lis/les=17902/17903 n=188 ec=211/211 lis/c=17902/17902 les/c/f=17903/17903/0 sis=17988 pruub=8.000660896s) [23,18] r=-1 lpr=1798> Aug 15 06:53:57 ceph99.cecnet.gmu.edu ceph-osd[193725]: osd.9 17993 is_healthy false -- only 0/10 up peers (less than 33%) Aug 15 06:53:57 ceph99.cecnet.gmu.edu ceph-osd[193725]: osd.9 17993 not healthy; waiting to boot The MON logs show Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.9 reported immediately failed by osd.3 Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.9 failed (root=default,pod=openstack,host=ceph99) (connection refused reported by osd.3) Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.11 reported immediately failed by osd.3 Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.11 failed (root=default,pod=openstack,host=ceph99) (connection refused reported by osd.3) Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.12 reported immediately failed by osd.3 Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.12 failed (root=default,pod=openstack,host=ceph99) (connection refused reported by osd.3) Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.10 reported immediately failed by osd.3 Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.10 failed (root=default,pod=openstack,host=ceph99) (connection refused reported by osd.3) Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.9 reported immediately failed by osd.3 Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.10 reported immediately failed by osd.3 Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.11 reported immediately failed by osd.3 Aug 15 06:53:53 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.12 reported immediately failed by osd.3 Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: mon.os-storage-1(a)1(peon).osd e17989 e17989: 26 total, 23 up, 26 in Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: mon.os-storage-1(a)1(peon).osd e17989 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: 15.13 scrub starts Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: Health check failed: 4 osds down (OSD_DOWN) Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: Health check failed: 2 hosts (4 osds) down (OSD_HOST_DOWN) Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osdmap e17988: 26 total, 22 up, 26 in Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.11 marked itself dead as of e17988 Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: from='mgr.16700599 10.192.126.85:0/2567473893' entity='mgr.os-storage.cecnet.gmu.edu.mouglb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: pgmap v121406: 374 pgs: 3 peering, 20 stale+active+clean, 3 active+remapped+backfilling, 348 active+clean; 2.1 TiB data, 4.1 TiB used, 76 TiB / 80 TiB avail; 102 B/s rd, 338 KiB/s wr, 6 op/s; 36715/1268774 objects > Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: Health check cleared: OSD_HOST_DOWN (was: 2 hosts (4 osds) down) Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: from='mgr.16700599 10.192.126.85:0/2567473893' entity='mgr.os-storage.cecnet.gmu.edu.mouglb' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.11 [v2:10.192.126.76:6808/3768579449,v1:10.192.126.76:6809/3768579449] boot Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osdmap e17989: 26 total, 23 up, 26 in Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.12 marked itself dead as of e17989 Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.11 reported immediately failed by osd.24 Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.11 failed (root=default,pod=openstack,host=ceph99) (connection refused reported by osd.24) Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.11 reported immediately failed by osd.5 Aug 15 06:53:54 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: osd.11 reported immediately failed by osd.3 Aug 15 06:53:55 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: mon.os-storage-1(a)1(peon).osd e17990 e17990: 26 total, 23 up, 26 in Aug 15 06:53:55 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: Monitor daemon marked osd.11 down, but it is still running Aug 15 06:53:55 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: map e17988 wrongly marked me down at e17988 Aug 15 06:53:55 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: 25.16 continuing backfill to osd.22 from (17451'16681348,17987'16685608] 25:6977376b:::rbd_data.4040a64151e028.000000000000a3c3:head to 17987'16685608 Aug 15 06:53:55 os-storage-1.cecnet.gmu.edu ceph-mon[4684]: Monitor daemon marked osd.12 down, but it is still running The system logs show no problems around the time of the drop. My best guess at the moment is that it's a networking issue with podman, but I've found no evidence of a problem. Output from ethtool doesn't show any problems [root@ceph99 ~]# ethtool -S enp3s0f0 | egrep 'error|drop|timeout' rx_errors: 0 tx_errors: 0 rx_dropped: 0 tx_dropped: 0 rx_over_errors: 0 rx_crc_errors: 0 rx_frame_errors: 0 rx_fifo_errors: 0 rx_missed_errors: 0 tx_aborted_errors: 0 tx_carrier_errors: 0 tx_fifo_errors: 0 tx_heartbeat_errors: 0 tx_timeout_count: 0 rx_length_errors: 0 rx_long_length_errors: 0 rx_short_length_errors: 0 rx_csum_offload_errors: 0 tx_hwtstamp_timeouts: 0 So far I have: - Updated the Intel X550-T NIC's firmware and driver (ixgbe) to the latest from Intel - Reverted the kernel, podman and NetworkManager packages to match the other Rocky 9.2 hosts that are working - Reverted the Intel driver to the ixgbe included in the kernel - sworn, cried and pleaded with the gods to spare me further anguish, to no avail. Other, possibly relevant, information: - There's no MON or MGR daemons on the host; just node exporter, crash, alerter and promtail. Is there anything else I should be looking at before I remove the host from the cluster and re-install Rocky 8.7 (and hope it works again)? This host was used while we were standing up the cluster and is due to be retired as we repurpose some of our other storage (standalone NFS servers with RAID6) and move them into the cluster.

1 0

Re: ceph orch upgrade stuck between 16.2.7 and 16.2.13
by Robert Sander 15 Aug '23

15 Aug '23

On 8/15/23 11:16, Curt wrote: > Probably not the issue, but do all your osd servers have internet > access? I've had a similar experience when one of our osd servers > default gateway got changed, so it was just waiting to download and took > a bit to timeout. Yes, all nodes can manually pull the image from quay.io. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: help, ceph fs status stuck with no response
by Patrick Donnelly 15 Aug '23

15 Aug '23

On Tue, Aug 8, 2023 at 1:18 AM Zhang Bao <lonsdale8734(a)gmail.com> wrote: > > Hi, thanks for your help. > > I am using ceph Pacific 16.2.7. > > Before my Ceph stuck at `ceph fs status fsname`, one of my cephfs became readonly. Probably the ceph-mgr is stuck (the "volumes" plugin) somehow talking to the read-only CephFS. That's not a scenario we've tested well. > The metadata pool of the readonly cephfs grew up from 10GB to 3TB. Then I shut down the readonly mds. Your metadata pool grew from 10GB to 3TB in read-only mode? That's unbelievable! We would need a lot more information to help figure the cause. Such as: `ceph tell mds.X perf dump`, `ceph tell mds.X status`, `ceph fs dump`, and `ceph tell mgr.X perf dump` while this is occurring. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

radosgw-admin command hangs out ,many hours
by nguyenvandiep＠baoviet.com.vn 14 Aug '23

14 Aug '23

We r using ceph 16.2.4 While init rados gw service, we run : " ceph orch apply rgw S3GW --placement="1 cephgw03uat" radosgw-admin user create --uid=rgwadmin --display-name=rgwadmin --system " but the command stuck, return nothing, hust HANGS. we add "--verbose" option and see log overlays: uncrognized mount option "volatile" or missing value how can we pass it

2 1

Lots of space allocated in completely empty OSDs
by Andras Pataki 14 Aug '23

14 Aug '23

Here is a strange problem that I don't seem to be able to figure out. Some of our OSDs that have zero weight, and no PGs have lots of allocated space: [root@cephosd0032 ~]# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 2925 ssd 0 1.00000 1.1 TiB 126 GiB 124 GiB 3.2 MiB 1021 MiB 984 GiB 11.32 0.15 0 up 2926 ssd 0 1.00000 1.1 TiB 126 GiB 124 GiB 3.2 MiB 1021 MiB 984 GiB 11.33 0.15 0 up 2927 ssd 0 1.00000 1.1 TiB 125 GiB 124 GiB 3.2 MiB 1021 MiB 984 GiB 11.31 0.15 0 up 2928 ssd 0 1.00000 1.1 TiB 126 GiB 124 GiB 3.2 MiB 1021 MiB 984 GiB 11.32 0.15 0 up so 120GB+ allocated, but no PGs. The cluster is clean, all PGs are active+clean, no rebalancing/recovery/etc. happening. There has been no rebalancing/recovery for at least a couple of days (after having added some nodes to the cluster). Doing a perf dump on the OSD confirms the space usage - that it has 130GB or so allocated: [root@cephosd0032 ~]# ceph daemon /var/run/ceph/ceph-osd.2925.asok perf dump .... "bluestore_allocated": 133277466624, "bluestore_stored": 133014427663, ... Bringing the OSD down - ceph-objectstore-tool claims that there are no objects stored in the OSD: [root@cephosd0032 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2925 --op list [root@cephosd0032 ~]# No fragmentation on the OSD either: [root@cephosd0032 ~]# ceph daemon /var/run/ceph/ceph-osd.2925.asok bluestore allocator fragmentation block { "fragmentation_rating": 1.2165147867554808e-08 } BlueFS also claims not to use significant space: [root@cephosd0032 ~]# ceph daemon /var/run/ceph/ceph-osd.2925.asok bluefs stats 1 : device size 0x11548000000 : own 0x[8518510000~b175d0000] = 0xb175d0000 : using 0xd1e0000(210 MiB) : bluestore has 0xeb3b630000(941 GiB) available wal_total:0, db_total:1131368205516, slow_total:0 So the puzzle is, what is stored in that 130GB allocated space? The cluster is Octopus 15.2.17. I came to looking at this by a mismatch in free space on our OSDs that are in service. After adding some nodes and waiting for the rebalance to finish, the free space didn't increase by the amount of space that was added, although the total space in the cluster did increase correctly (and data in pools didn't increase by a significant amount to explain this). So there is a bit of a mystery regarding what a bunch of space is allocated for. Any ideas/pointers would be appreciated. Andras

3 2

librbd 4k read/write?
by Murilo Morais 13 Aug '23

13 Aug '23

Good afternoon everybody! I have the following scenario: Pool RBD replication x3 5 hosts with 12 SAS spinning disks each I'm using exactly the following line with FIO to test: fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -size=10G -iodepth=16 -rw=write -filename=./test.img If I increase the blocksize I can easily reach 1.5 GBps or more. But when I use blocksize in 4K I get a measly 12 Megabytes per second, which is quite annoying. I achieve the same rate if rw=read. If I use librbd's cache I get a considerable improvement in writing, but reading remains the same. I already tested with rbd_read_from_replica_policy=balance but I didn't notice any difference. I tried to leave readahead enabled by setting rbd_readahead_disable_after_bytes=0 but I didn't see any difference in sequential reading either. Note: I tested it on another smaller cluster, with 36 SAS disks and got the same result. I don't know exactly what to look for or configure to have any improvement.

9 15

Puzzle re 'ceph: mds0 session blocklisted"
by Harry G Coin 12 Aug '23

12 Aug '23

Can anyone help me understand seemingly contradictory cephfs error messages? I have a RHEL ceph client that mounts a cephfs file system via autofs. Very typical. After boot, when a user first uses the mount, for example 'ls /mountpoint' , all appears normal to the user. But on the system console I get, every time after first boot: ... [ 412.762310] Key type dns_resolver registered [ 412.912107] Key type ceph registered [ 412.925268] libceph: loaded (mon/osd proto 15/24) [ 413.110488] ceph: loaded (mds proto 32) [ 413.124870] libceph: mon3 (2)[fc00:1002:c7::44]:3300 session established [ 413.128298] libceph: client56471655 fsid 4067126d-01cb-40af-824a-881c130140f8 [ 413.355716] ceph: mds0 session blocklisted ... The autofs line is /mountpoint -fstype=ceph,fs=cephlibraryfs,fsid=really-big-number,name=cephuser,secretfile=/etc/ceph/secret1.key,ms_mode=crc,relatime,recover_session=clean,mount_timeout=15,fscontext="system_u:object_r:cephfs_t:s0" [fc00:1002:c7::41]:3300,[fc00:1002:c7::42]:3300,[fc00:1002:c7::43]:3300,[fc00:1002:c7::44]:3300:/ 'blocklisting' is, well, 'bad'... but there's no obvious user effect. Is there an 'unobvious' problem? What am I missing? Ceph pacific everywhere latest. Thanks Harry Coin

2 1

CEPHADM_STRAY_DAEMON
by Tyler Jurgens 11 Aug '23

11 Aug '23

Hey, sorry, I'm a ceph noob here, but I'm trying to figure out how to clear this up. I've got 8 servers (ceph-admin, ceph-mon(1->3), ceph-osd(1->4) as part of my ceph deployment (reef, upgraded from quincy), and when I originally deployed it Ceph used random servers for the mon daemon. Wanting to target the intended servers for ceph, I created a label (mon) and assigned that label to my three mon hosts (ceph orch host label add ceph-mon1 mon), etc. Then I assigned the mon daemon to my hosts with the mon label: ceph orch apply mon label:mon. However, now I have this health message: stray daemon mon.ceph-osd3 on host ceph-mon3 not managed by cephadm stray daemon mon.ceph-osd2 on host ceph-osd2 not managed by cephadm How do I clean that up? The docs ( https://docs.ceph.com/en/reef/cephadm/operations/#cephadm-stray-daemon) say to add the hosts with ceph orch host add <hostname> but those hosts with stray daemons are already added. I tried adding them again, but it didn't make a difference. All hosts are/were always managed by cephadm (edited) I also rebooted one of the hosts, that didn't make a difference. Any ideas on what I can try? Cheers, Tyler

3 2

Re: ceph-volume lvm new-db fails
by Christian Rohmann 11 Aug '23

11 Aug '23

On 11/05/2022 23:21, Joost Nieuwenhuijse wrote: > After a reboot the OSD turned out to be corrupt. Not sure if > ceph-volume lvm new-db caused the problem, or failed because of > another problem. I just ran into the same issue trying to add a db to an existing OSD. Apparently this is a known bug: https://tracker.ceph.com/issues/55260 It's already fixed master, but the backports are all still pending ... Regards Christian

1 1

how to set load balance on multi active mds?
by zxcs 11 Aug '23

11 Aug '23

Hi, experts, we have a product env build with ceph version 16.2.11 pacific, and using CephFS. Also enable multi active mds(more than 10), but we usually see load unbalance on our client request with these mds. see below picture. the top 1 mds has 32.2k client request. and the last one only 331. this always lead our cluster into very bad situation. say many MDS report slow requests… ... 7 MDSs report slow requests 1 MDSs behind on trimming … So our question is how to set those mdss load balance? Could any one please help to shed some light here? Thanks a ton! Thanks, xz

3 4

Ceph bucket notification events stop working
by daniel.yordanov1＠swisscom.com 10 Aug '23

10 Aug '23

Hello, We started to use the Ceph bucket notification events with subscription to an HTTP endpoint. We encountered an issue when the receiver endpoint was changed. Which means the events from Ceph weren't consumed. We deleted the bucket notifications and the topic, and created a new topic with the new endpoint and new bucket notifications. (We are using the REST api to create bucket notifications and topics. We also used the CLI commands, but there we found out that deleting a topic doesn't delete the notifications that are subscribed to it. Ceph version is Pacific.) From that moment we didn't receive any more notification events to our new endpoint. We tried many times to create new topics and new bucket notifications, but we don't receive anymore events to our endpoint. We suspect that the notification queues don't get fully cleaned and they stay in some broken state. We have been able to reproduce this locally and the only solution was to wipe all the containers and recreate them. The problem is that this issue is on a staging environment where we cannot destroy everything. We are looking for a solution or a command to clean the notification queues, to be able to start anew. We also are looking for a way to know programatically if the notifications broke and have a way to automatically recover as such a flaw is critical for our application. Thanks for your time! Daniel Yordanov

2 2

libcephfs init hangs, is there a 'timeout' argument?
by Harry G Coin 09 Aug '23

09 Aug '23

Libcephfs's 'init' call hangs when passed arguments that once worked normally, but later refer to a cluster that's either broken, is on its way out of service, has too few mons, etc. At least the python libcephfs wrapper hangs on init. Of course mount and session timeouts work, but is there a way to error out a failed init call and not just hang the client? Thanks!

1 0