July 2023 - ceph-users - lists.ceph.io

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

Help needed to configure erasure coding LRC plugin
by Michel Jouvin 16 Oct '24

16 Oct '24

Hi, As discussed in another thread (Crushmap rule for multi-datacenter erasure coding), I'm trying to create an EC pool spanning 3 datacenters (datacenters are present in the crushmap), with the objective to be resilient to 1 DC down, at least keeping the readonly access to the pool and if possible the read-write access, and have a storage efficiency better than 3 replica (let say a storage overhead <= 2). In the discussion, somebody mentioned LRC plugin as a possible jerasure alternative to implement this without tweaking the crushmap rule to implement the 2-step OSD allocation. I looked at the documentation (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/) but I have some questions if someone has experience/expertise with this LRC plugin. I tried to create a rule for using 5 OSDs per datacenter (15 in total), with 3 (9 in total) being data chunks and others being coding chunks. For this, based of my understanding of examples, I used k=9, m=3, l=4. Is it right? Is this configuration equivalent, in terms of redundancy, to a jerasure configuration with k=9, m=6? The resulting rule, which looks correct to me, is: -------- { "rule_id": 6, "rule_name": "test_lrc_2", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 15, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -4, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 5, "type": "host" }, { "op": "emit" } ] } ------------ Unfortunately, it doesn't work as expected: a pool created with this rule ends up with its pages active+undersize, which is unexpected for me. Looking at 'ceph health detail` output, I see for each page something like: pg 52.14 is stuck undersized for 27m, current state active+undersized, last acting [90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647] For each PG, there is 3 '2147483647' entries and I guess it is the reason of the problem. What are these entries about? Clearly it is not OSD entries... Looks like a negative number, -1, which in terms of crushmap ID is the crushmap root (named "default" in our configuration). Any trivial mistake I would have made? Thanks in advance for any help or for sharing any successful configuration? Best regards, Michel

4 24

Not all Bucket Shards being used
by Christian Kugler 01 Oct '24

01 Oct '24

Hi, I have trouble with large OMAP files in a cluster in the RGW index pool. Some background information about the cluster: There is CephFS and RBD usage on the main cluster but for this issue I think only S3 is interesting. There is one realm, one zonegroup with two zones which have a bidirectional sync set up. Since this does not allow for autoresharding we have to do it by hand in this cluster – looking forward to Reef! From the logs: cluster 2023-07-17T22:59:03.018722+0000 osd.75 (osd.75) 623978 : cluster [WRN] Large omap object found. Object: 34:bcec3016:::.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5:head PG: 34.680c373d (34.5) Key count: 962091 Size (bytes): 277963182 The offending bucket looks like this: # radosgw-admin bucket stats \ | jq '.[] | select(.marker =="3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9") |"\(.num_shards) \(.usage["rgw.main"].num_objects)"' -r 131 9463833 Last week the number of objects was about 12 million. Which is why I reshareded the offending bucket twice, I think. Once to 129 and the second time to 131 because I wanted some leeway (or lieway? scnr, Sage). Unfortunately, even after a week the objects were still to big (the log line above is quite recent), so I looked into it again. # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.0 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.1 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.2 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.3 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.4 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.6 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.7 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.8 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.9 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.10 # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V \ |xargs -IOMAP sh -c \ 'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l' 1013854 1011007 1012287 1011232 1013565 998262 1012777 1012713 1012230 1010690 997111 Apparently, only 11 shards are in use. This would explain why the "Key usage" (from the log line) is about ten times higher than I would expect. How can I deal with this issue? One thing I could try to fix this would be to reshard to a lower number, but I am not sure if there are any risks associated with "downsharding". After that I could reshard to something like 97. Or I could directly "downshard" to 97. Also, the second zone has a similar problem, but as the error messsage lets me know, this would be a bad idea. Will it just take more time until the sharding is transferred to the seconds zone? Best, Christian Kugler

2 5

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

Re: MDS crashes to damaged metadata
by Patrick Donnelly 10 Jun '24

10 Jun '24

On Thu, Dec 15, 2022 at 9:32 AM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): > > 2022-12-08T13:05:48.919+0100 7f440afec700 -1 /build/ceph-16.2.10/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f440afec700 time 2022-12-08T13:05:48.921223+0100 > /build/ceph-16.2.10/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE) > > 12 Minutes lates the unlink_local error crashes appeared again. This time with a new file. During debugging we noticed a MTU mismatch between MDS (1500) and client (9000) with cephfs kernel mount. The client is also creating the snapshots via mkdir in the .snap directory. > > We disabled snapshot creation for now, but really need this feature. I uploaded the mds logs of the first crash along with the information above to https://tracker.ceph.com/issues/38452 > > I would greatly appreciate it, if you could answer me the following question: > > Is the Bug related to our MTU Mismatch? We fixed the MTU Issue going back to 1500 on all nodes in the ceph public network on the weekend also. I doubt it. > If you need a debug level 20 log of the ScatterLock for further analysis, i could schedule snapshots at the end of our workdays and increase the debug level 5 Minutes arround snap shot creation. This would be very helpful! -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 4

Re: Recoveries without any misplaced objects?
by Hector Martin 24 Apr '24

24 Apr '24

On 29/05/2023 20.55, Anthony D'Atri wrote: > Check the uptime for the OSDs in question I restarted all my OSDs within the past 10 days or so. Maybe OSD restarts are somehow breaking these stats? > >> On May 29, 2023, at 6:44 AM, Hector Martin <marcan(a)marcan.st> wrote: >> >> Hi, >> >> I'm watching a cluster finish a bunch of backfilling, and I noticed that >> quite often PGs end up with zero misplaced objects, even though they are >> still backfilling. >> >> Right now the cluster is down to 6 backfilling PGs: >> >> data: >> volumes: 1/1 healthy >> pools: 6 pools, 268 pgs >> objects: 18.79M objects, 29 TiB >> usage: 49 TiB used, 25 TiB / 75 TiB avail >> pgs: 262 active+clean >> 6 active+remapped+backfilling >> >> But there are no misplaced objects, and the misplaced column in `ceph pg >> dump` is zero for all PGs. >> >> If I do a `ceph pg dump_json`, I can see `num_objects_recovered` >> increasing for these PGs... but the misplaced count is still 0. >> >> Is there something else that would cause recoveries/backfills other than >> misplaced objects? Or perhaps there is a bug somewhere causing the >> misplaced object count to be misreported as 0 sometimes? >> >> # ceph -v >> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy >> (stable) >> >> - Hector >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > - Hector

3 3

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

Why you might want packages not containers for Ceph deployments
by Matthew Vernon 25 Mar '24

25 Mar '24

Hi, In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0]. I'd like to suggest, briefly, that: * containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes. For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice. Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages): * you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice) We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part. Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems! I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder". I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :) Regards, and apologies for the wall of text, Matthew [0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

26 65

Upgrading from Pacific to Quincy fails with "Unexpected error"
by Reza Bakhshayeshi 13 Mar '24

13 Mar '24

Hi all, I have a problem regarding upgrading Ceph cluster from Pacific to Quincy version with cephadm. I have successfully upgraded the cluster to the latest Pacific (16.2.11). But when I run the following command to upgrade the cluster to 17.2.5, After upgrading 3/4 mgrs, the upgrade process stops with "Unexpected error". (everything is on a private network) ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v17.2.5 I also tried the 17.2.4 version. cephadm fails to check the hosts' status and marks them as offline: cephadm 2023-04-06T10:19:59.998510+0000 mgr.host9.arhpnd (mgr.4516356) 5782 : cephadm [DBG] host host4 (x.x.x.x) failed check cephadm 2023-04-06T10:19:59.998553+0000 mgr.host9.arhpnd (mgr.4516356) 5783 : cephadm [DBG] Host "host4" marked as offline. Skipping daemon refresh cephadm 2023-04-06T10:19:59.998581+0000 mgr.host9.arhpnd (mgr.4516356) 5784 : cephadm [DBG] Host "host4" marked as offline. Skipping gather facts refresh cephadm 2023-04-06T10:19:59.998609+0000 mgr.host9.arhpnd (mgr.4516356) 5785 : cephadm [DBG] Host "host4" marked as offline. Skipping network refresh cephadm 2023-04-06T10:19:59.998633+0000 mgr.host9.arhpnd (mgr.4516356) 5786 : cephadm [DBG] Host "host4" marked as offline. Skipping device refresh cephadm 2023-04-06T10:19:59.998659+0000 mgr.host9.arhpnd (mgr.4516356) 5787 : cephadm [DBG] Host "host4" marked as offline. Skipping osdspec preview refresh cephadm 2023-04-06T10:19:59.998682+0000 mgr.host9.arhpnd (mgr.4516356) 5788 : cephadm [DBG] Host "host4" marked as offline. Skipping autotune cluster 2023-04-06T10:20:00.000151+0000 mon.host8 (mon.0) 158587 : cluster [ERR] Health detail: HEALTH_ERR 9 hosts fail cephadm check; Upgrade: failed due to an unexpected exception cluster 2023-04-06T10:20:00.000191+0000 mon.host8 (mon.0) 158588 : cluster [ERR] [WRN] CEPHADM_HOST_CHECK_FAILED: 9 hosts fail cephadm check cluster 2023-04-06T10:20:00.000202+0000 mon.host8 (mon.0) 158589 : cluster [ERR] host host7 (x.x.x.x) failed check: Unable to reach remote host host7. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000213+0000 mon.host8 (mon.0) 158590 : cluster [ERR] host host2 (x.x.x.x) failed check: Unable to reach remote host host2. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000220+0000 mon.host8 (mon.0) 158591 : cluster [ERR] host host8 (x.x.x.x) failed check: Unable to reach remote host host8. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000228+0000 mon.host8 (mon.0) 158592 : cluster [ERR] host host4 (x.x.x.x) failed check: Unable to reach remote host host4. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000240+0000 mon.host8 (mon.0) 158593 : cluster [ERR] host host3 (x.x.x.x) failed check: Unable to reach remote host host3. Process exited with non-zero exit status 3 and here are some outputs of the commands: [root@host8 ~]# ceph -s cluster: id: xxx health: HEALTH_ERR 9 hosts fail cephadm check Upgrade: failed due to an unexpected exception services: mon: 5 daemons, quorum host8,host1,host7,host2,host9 (age 2w) mgr: host9.arhpnd(active, since 105m), standbys: host8.jowfih, host1.warjsr, host2.qyavjj mds: 1/1 daemons up, 3 standby osd: 37 osds: 37 up (since 8h), 37 in (since 3w) data: io: client: progress: Upgrade to 17.2.5 (0s) [............................] [root@host8 ~]# ceph orch upgrade status { "target_image": "my-private-repo/quay-io/ceph/ceph@sha256 :34c763383e3323c6bb35f3f2229af9f466518d9db926111277f5e27ed543c427", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "3/59 daemons upgraded", "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception", "is_paused": true } [root@host8 ~]# ceph cephadm check-host host7 check-host failed: Host 'host7' not found. Use 'ceph orch host ls' to see all managed hosts. [root@host8 ~]# ceph versions { "mon": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 1, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 }, "osd": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 37 }, "mds": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 4 }, "overall": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 47, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 } } The strange thing is I can rollback the cluster status by failing to not-upgraded mgr like this: ceph mgr fail ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v16.2.11 Would you happen to have any idea about this? Best regards, Reza

4 9

Re: increasing number of (deep) scrubs
by Frank Schilder 13 Dec '23

13 Dec '23

Hi Dan, thanks for your answer. I don't have a problem with increasing osd_max_scrubs (=1 at the moment) as such. I would simply prefer a somewhat finer grained way of controlling scrubbing than just doubling or tripling it right away. Some more info. These 2 pools are data pools for a large FS. Unfortunately, we have a large percentage of small files, which is a pain for recovery and seemingly also for deep scrubbing. Our OSDs are about 25% used and I had to increase the warning interval already to 2 weeks. With all the warning grace parameters this means that we manage to deep scrub everything about every month. I need to plan for 75% utilisation and a 3 months period is a bit far on the risky side. Our data is to a large percentage cold data. Client reads will not do the check for us, we need to combat bit-rot pro-actively. The reasons I'm interested in parameters initiating more scrubs while also converting more scrubs into deep scrubs are, that 1) scrubs seem to complete very fast. I almost never catch a PG in state "scrubbing", I usually only see "deep scrubbing". 2) I suspect the low deep-scrub count is due to a low number of deep-scrubs scheduled and not due to conflicting per-OSD deep scrub reservations. With the OSD count we have and the distribution over 12 servers I would expect at least a peak of 50% OSDs being active in scrubbing instead of the 25% peak I'm seeing now. It ought to be possible to schedule more PGs for deep scrub than actually are. 3) Every OSD having only 1 deep scrub active seems to have no measurable impact on user IO. If I could just get more PGs scheduled with 1 deep scrub per OSD it would already help a lot. Once this is working, I can eventually increase osd_max_scrubs when the OSDs fill up. For now I would just like that (deep) scrub scheduling looks a bit harder and schedules more eligible PGs per time unit. If we can get deep scrubbing up to an average of 42PGs completing per hour with keeping osd_max_scrubs=1 to maintain current IO impact, we should be able to complete a deep scrub with 75% full OSDs in about 30 days. This is the current tail-time with 25% utilisation. I believe currently a deep scrub of a PG in these pools takes 2-3 hours. Its just a gut feeling from some repair and deep-scrub commands, I would need to check logs for more precise info. Increasing osd_max_scrubs would then be a further and not the only option to push for more deep scrubbing. My expectation would be that values of 2-3 are fine due to the increasingly higher percentage of cold data for which no interference with client IO will happen. Hope that makes sense and there is a way beyond bumping osd_max_scrubs to increase the number of scheduled and executed deep scrubs. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dvanders(a)gmail.com> Sent: 05 January 2023 15:36 To: Frank Schilder Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] increasing number of (deep) scrubs Hi Frank, What is your current osd_max_scrubs, and why don't you want to increase it? With 8+2, 8+3 pools each scrub is occupying the scrub slot on 10 or 11 OSDs, so at a minimum it could take 3-4x the amount of time to scrub the data than if those were replicated pools. If you want the scrub to complete in time, you need to increase the amount of scrub slots accordingly. On the other hand, IMHO the 1-week deadline for deep scrubs is often much too ambitious for large clusters -- increasing the scrub intervals is one solution, or I find it simpler to increase mon_warn_pg_not_scrubbed_ratio and mon_warn_pg_not_deep_scrubbed_ratio until you find a ratio that works for your cluster. Of course, all of this can impact detection of bit-rot, which anyway can be covered by client reads if most data is accessed periodically. But if the cluster is mostly idle or objects are generally not read, then it would be preferable to increase slots osd_max_scrubs. Cheers, Dan On Tue, Jan 3, 2023 at 2:30 AM Frank Schilder <frans(a)dtu.dk> wrote: > > Hi all, > > we are using 16T and 18T spinning drives as OSDs and I'm observing that they are not scrubbed as often as I would like. It looks like too few scrubs are scheduled for these large OSDs. My estimate is as follows: we have 852 spinning OSDs backing a 8+2 pool with 2024 and an 8+3 pool with 8192 PGs. On average I see something like 10PGs of pool 1 and 12 PGs of pool 2 (deep) scrubbing. This amounts to only 232 out of 852 OSDs scrubbing and seems to be due to a conservative rate of (deep) scrubs being scheduled. The PGs (dep) scrub fairly quickly. > > I would like to increase gently the number of scrubs scheduled for these drives and *not* the number of scrubs per OSD. I'm looking at parameters like: > > osd_scrub_backoff_ratio > osd_deep_scrub_randomize_ratio > > I'm wondering if lowering osd_scrub_backoff_ratio to 0.5 and, maybe, increasing osd_deep_scrub_randomize_ratio to 0.2 would have the desired effect? Are there other parameters to look at that allow gradual changes in the number of scrubs going on? > > Thanks a lot for your help! > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 3

MDS stuck in rejoin
by Frank Schilder 10 Nov '23

10 Nov '23

Hi all, we had a client with the warning "[WRN] MDS_CLIENT_OLDEST_TID: 1 clients failing to advance oldest client/flush tid". I looked at the client and there was nothing going on, so I rebooted it. After the client was back, the message was still there. To clean this up I failed the MDS. Unfortunately, the MDS that took over is remained stuck in rejoin without doing anything. All that happened in the log was: [root@ceph-10 ceph]# tail -f ceph-mds.ceph-10.log 2023-07-20T15:54:29.147+0200 7fedb9c9f700 1 mds.2.896604 rejoin_start 2023-07-20T15:54:29.161+0200 7fedb9c9f700 1 mds.2.896604 rejoin_joint_start 2023-07-20T15:55:28.005+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896614 from mon.4 2023-07-20T15:56:00.278+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896615 from mon.4 [...] 2023-07-20T16:02:54.935+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896653 from mon.4 2023-07-20T16:03:07.276+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896654 from mon.4 After some time I decided to give another fail a try and, this time, the replacement daemon went to active state really fast. If I have a message like the above, what is the clean way of getting the client clean again (version: 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable))? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 16

Re: RGW access logs with bucket name
by Boris Behrens 02 Nov '23

02 Nov '23

Bringing up that topic again: is it possible to log the bucket name in the rgw client logs? currently I am only to know the bucket name when someone access the bucket via https://TLD/bucket/object instead of https://bucket.TLD/object. Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens <bb(a)kervyn.de>: > Hi, > I am looking forward to move our logs from > /var/log/ceph/ceph-client...log to our logaggregator. > > Is there a way to have the bucket name in the log file? > > Or can I write the rgw_enable_ops_log into a file? Maybe I could work with > this. > > Cheers and happy new year > Boris > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

4 6

Re: Status of Quincy 17.2.5 ?
by Christian Rohmann 30 Oct '23

30 Oct '23

Hey everyone, On 20/10/2022 10:12, Christian Rohmann wrote: > 1) May I bring up again my remarks about the timing: > > On 19/10/2022 11:46, Christian Rohmann wrote: > >> I believe the upload of a new release to the repo prior to the >> announcement happens quite regularly - it might just be due to the >> technical process of releasing. >> But I agree it would be nice to have a more "bit flip" approach to >> new releases in the repo and not have the packages appear as updates >> prior to the announcement and final release and update notes. > By my observations sometimes there are packages available on the > download servers via the "last stable" folders such as > https://download.ceph.com/debian-quincy/ quite some time before the > announcement of a release is out. > I know it's hard to time this right with mirrors requiring some time > to sync files, but would be nice to not see the packages or have > people install them before there are the release notes and potential > pointers to changes out. Todays 16.2.11 release shows the exact issue I described above .... 1) 16.2.11 packages are already available via e.g. https://download.ceph.com/debian-pacific 2) release notes not yet merged: (https://github.com/ceph/ceph/pull/49839) thus https://ceph.io/en/news/blog/2022/v16-2-11-pacific-released/ show a 404 :-) 3) No announcement like https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/QOCU563UD3… to the ML yet. Regards Christian

2 2

[Pacific] ceph orch device ls do not returns any HDD
by Patrick Begou 27 Oct '23

27 Oct '23

Hi everyone I'm new to CEPH, just a french 4 days training session with Octopus on VMs that convince me to build my first cluster. At this time I have 4 old identical nodes for testing with 3 HDDs each, 2 network interfaces and running Alma Linux8 (el8). I try to replay the training session but it fails, breaking the web interface because of some problems with podman 4.2 not compatible with Octopus. So I try to deploy Pacific with cephadm tool on my first node (mostha1) (to enable testing also an upgrade later). dnf -y install https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noar… monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }') cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \ --initial-dashboard-user admceph \ --allow-fqdn-hostname --cluster-network 10.1.0.0/16 This was sucessfull. But running "*c**eph orch device ls*" do not show any HDD even if I have /dev/sda (used by the OS), /dev/sdb and /dev/sdc The web interface shows a row capacity which is an aggregate of the sizes of the 3 HDDs for the node. I've also tried to reset /dev/sdb but cephadm do not see it: [ceph: root@mostha1 /]# ceph orch device zap mostha1.legi.grenoble-inp.fr /dev/sdb --force Error EINVAL: Device path '/dev/sdb' not found on host 'mostha1.legi.grenoble-inp.fr' On my first attempt with octopus, I was able to list the available HDD with this command line. Before moving to Pacific, the OS on this node has been reinstalled from scratch. Any advices for a CEPH beginner ? Thanks Patrick

9 51

ceph-dashboard python warning with new pyo3 0.17 lib (debian12)
by DERUMIER, Alexandre 11 Oct '23

11 Oct '23

Hi, on debian12, ceph-dashboard is throwing a warning "Module 'dashboard' has failed dependency: PyO3 modules may only be initialized once per interpreter process" Seem to be related to pyo3 0.17 change https://github.com/PyO3/pyo3/blob/7bdc504252a2f972ba3490c44249b202a4ce6180/… " Each #[pymodule] can now only be initialized once per process To make PyO3 modules sound in the presence of Python sub-interpreters, for now it has been necessary to explicitly disable the ability to initialize a #[pymodule] more than once in the same process. Attempting to do this will now raise an ImportError. "

3 3

libceph: mds1 IP+PORT wrong peer at address
by Frank Schilder 20 Sep '23

20 Sep '23

Hi all, we seem to have hit a bug in the ceph fs kernel client and I just want to confirm what action to take. We get the error "wrong peer at address" in dmesg and some jobs on that server seem to get stuck in fs access; log extract below. I found these 2 tracker items https://tracker.ceph.com/issues/23883 and https://tracker.ceph.com/issues/41519, which don't seem to have fixes. My questions: - Is this harmless or does it indicate invalid/corrupted client cache entries? - How to resolve, ignore, umount+mount or reboot? Here an extract from the dmesg log, the error has survived a couple of MDS restarts already: [Mon Mar 6 12:56:46 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:05:18 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-1572619386 [Mon Mar 6 13:05:18 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:13:50 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-1572619386 [Mon Mar 6 13:13:50 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state OPEN) [Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state OPEN) [Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start [Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start [Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success [Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success [Mon Mar 6 13:18:13 2023] ceph: update_snap_trace error -22 [Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state OPEN) [Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state OPEN) [Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed [Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect success [Mon Mar 6 13:18:29 2023] ceph: mds7 reconnect success [Mon Mar 6 13:18:35 2023] ceph: update_snap_trace error -22 [Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed [Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed [Mon Mar 6 13:22:22 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Mon Mar 6 13:22:22 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:30:54 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [...] [Thu Mar 9 09:37:24 2023] slurm.epilog.cl (31457): drop_caches: 3 [Thu Mar 9 09:38:26 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:38:26 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 09:46:58 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:46:58 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 09:55:30 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:55:30 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 10:04:02 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 10:04:02 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 6

Quincy 17.2.6 - Rados gateway crash -
by xadhoom76＠gmail.com 19 Sep '23

19 Sep '23

Hi, we have service that is still crashing when S3 client (veeam backup) start to write data main log from rgw service req 13170422438428971730 0.008000086s s3:get_obj WARNING: couldn't find acl header for object, generating default 2023-07-20T14:36:45.331+0000 7fa5adb4c700 -1 *** Caught signal (Aborted) ** And " 2023-07-19T22:04:15.968+0000 7ff07305b700 1 beast: 0x7fefc7178710: 172.16.199.11 - veeam90 [19/Jul/2023:22:04:15.948 +0000] "PUT /veeam90/Veeam/Backu p/veeam90/Clients/%7Bd14cd688-57b4-4809-a1d9-14cafd191b11%7D/34387bbd-bec9-4a40-a04d-6a890d5d6407/CloudStg/Data/%7Bf687ee0f-fb50-4ded-b3a8-3f67ca7f244 b%7D/%7B6f31c277-734c-46fd-98d5-c560aa6dc776%7D/144113_f3fd31c9ee2a45aeeadda0de3cbc9064_00000000000000000000000000000000 HTTP/1.1" 200 63422 - "APN/1. 0 Veeam/1.0 Backup/12.0" - latency=0.020000216s 2023-07-19T22:04:15.972+0000 7ff08307b700 1 ====== starting new request req=0x7fefc7682710 ===== 2023-07-19T22:04:15.972+0000 7ff087083700 1 ====== starting new request req=0x7fefc737c710 ===== 2023-07-19T22:04:15.972+0000 7ff071057700 1 ====== starting new request req=0x7fefc72fb710 ===== 2023-07-19T22:04:15.972+0000 7ff0998a8700 1 ====== starting new request req=0x7fefc71f9710 ===== 2023-07-19T22:04:15.972+0000 7fefe473e700 -1 *** Caught signal (Aborted) ** in thread 7fefe473e700 thread_name:radosgw ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7ff102d62cf0] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9009b) [0x7ff101d5209b] 5: /lib64/libstdc++.so.6(+0x9653c) [0x7ff101d5853c] 6: /lib64/libstdc++.so.6(+0x95559) [0x7ff101d57559] 7: __gxx_personality_v0() 8: /lib64/libgcc_s.so.1(+0x10b03) [0x7ff101736b03] 9: _Unwind_Resume() 10: /lib64/libradosgw.so.2(+0x538c5b) [0x7ff105246c5b] ---------------------- -- -10> 2023-07-19T22:04:15.972+0000 7ff071057700 2 req 8167590275148061076 0.000000000s s3:put_obj pre-executing -9> 2023-07-19T22:04:15.972+0000 7ff071057700 2 req 8167590275148061076 0.000000000s s3:put_obj check rate limiting -8> 2023-07-19T22:04:15.972+0000 7ff071057700 2 req 8167590275148061076 0.000000000s s3:put_obj executing -7> 2023-07-19T22:04:15.972+0000 7ff0998a8700 1 ====== starting new request req=0x7fefc71f9710 ===== -6> 2023-07-19T22:04:15.972+0000 7ff0998a8700 2 req 15658207768827051601 0.000000000s initializing for trans_id = tx00000d94d21014832be51-0064b85 ddf-3dfe-backup -5> 2023-07-19T22:04:15.972+0000 7ff0998a8700 2 req 15658207768827051601 0.000000000s getting op 1 -4> 2023-07-19T22:04:15.972+0000 7ff0998a8700 2 req 15658207768827051601 0.000000000s s3:put_obj verifying requester -3> 2023-07-19T22:04:15.972+0000 7ff0998a8700 2 req 15658207768827051601 0.000000000s s3:put_obj normalizing buckets and tenants -2> 2023-07-19T22:04:15.972+0000 7ff0998a8700 2 req 15658207768827051601 0.000000000s s3:put_obj init permissions -1> 2023-07-19T22:04:15.972+0000 7ff011798700 2 req 15261257039771290446 0.024000257s s3:put_obj completing 0> 2023-07-19T22:04:15.972+0000 7fefe473e700 -1 *** Caught signal (Aborted) ** in thread 7fefe473e700 thread_name:radosgw " Anyone have this issue ? Thanks

5 5

RGW multisite logs (data, md, bilog) not being trimmed automatically?
by Christian Rohmann 14 Sep '23

14 Sep '23

Hey ceph-users, I am running two (now) Quincy clusters doing RGW multi-site replication with only one actually being written to by clients. The other site is intended simply as a remote copy. On the primary cluster I am observing an ever growing (objects and bytes) "sitea.rgw.log" pool, not so on the remote "siteb.rgw.log" which is only 300MiB and around 15k objects with no growth. Metrics show that the growth of pool on primary is linear for at least 6 months, so not sudden spikes or anything. Also sync status appears to be totally happy. There are also no warnings in regards to large OMAPs or anything similar. I was under the impression that RGW will trim its three logs (md, bi, data) automatically and only keep data that has not yet been replicated by the other zonegroup members? The config option "ceph config get mgr rgw_sync_log_trim_interval" is set to 1200, so 20 Minutes. So I am wondering if there might be some inconsistency or how I can best analyze what the cause for the accumulation of log data is? There are older questions on the ML, such as [1], but there was not really a solution or root cause identified. I know there is manual trimming, but I'd rather want to analyze the current situation and figure out what the cause for the lack of auto-trimming is. * Do I need to go through all buckets and count logs and look at their timestamps? Which queries do make sense here? * Is there usually any logging of the log trimming activity that I should expect? Or that might indicate why trimming does not happen? Regards Christian [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/WZCFOAMLWV…

1 2

Critical Information: DELL/Toshiba SSDs dying after 70,000 hours of operation
by Frédéric Nass 01 Sep '23

01 Sep '23

Hello, This message does not concern Ceph itself but a hardware vulnerability which can lead to permanent loss of data on a Ceph cluster equipped with the same hardware in separate fault domains. The DELL / Toshiba PX02SMF020, PX02SMF040, PX02SMF080 and PX02SMB160 SSD drives of the 13G generation of DELL servers are subject to a vulnerability which renders them unusable after 70,000 hours of operation, i.e. approximately 7 years and 11 months of activity. This topic has been discussed here: https://www.dell.com/community/PowerVault/TOSHIBA-PX02SMF080-has-lost-commu… The risk is all the greater since these disks may die at the same time in the same server leading to the loss of all data in the server. To date, DELL has not provided any firmware fixing this vulnerability, the latest firmware version being "A3B3" released on Sept. 12, 2016: https://www.dell.com/support/home/en-us/ drivers/driversdetails?driverid=hhd9k If your have servers running these drives, check their uptime. If they are close to the 70,000 hour limit, replace them immediately. The smartctl tool does not report the uptime for these SSDs, but if you have HDDs in the server, you can query their SMART status and get their uptime, which should be about the same as the SSDs. The smartctl command is: smartctl -a -d megaraid,XX /dev/sdc (where XX is the iSCSI bus number). We have informed DELL about this but have no information yet on the arrival of a fix. We have lost 6 disks, in 3 different servers, in the last few weeks. Our observation shows that the drives don't survive full shutdown and restart of the machine (power off then power on in iDrac), but they may also die during a single reboot (init 6) or even while the machine is running. Fujitsu released a corrective firmware in June 2021 but this firmware is most certainly not applicable to DELL drives: https://www.fujitsu.com/us/imagesgig5/PY-CIB070-00.pdf Regards, Frederic Sous-direction Infrastructure and Services Direction du Numérique Université de Lorraine

2 2

CephFS metadata outgrow DISASTER during recovery
by Jakub Petrzilka 16 Aug '23

16 Aug '23

Hello everyone! Recently we had a very nasty incident with one of our CEPH storages. During basic backfill recovery operation due to faulty disk CephFS metadata started growing exponentially until they used all available space and whole cluster DIED. Usage graph screenshot in attachment. Everything was very fast and even when the OSDs were marked full they tripped failsafe and ate all the free blocks, still trying to allocate space and completely died without possibility to even start them again. Only solution was to copy whole bluestore to bigger SSD and resize underlying BS device. Just about 1/3 was able to start after moving but it was enough since we have very redundant settings for cephfs metadata. Basically metadata were moved from 12x 240g SSD to 12x 500GB SSD to have enough space to start again. Brief info about the cluster: - CephFS data are stored on ~500x 8TB SAS HDD using 10+2 ECC in 18 hosts. - CephFS metadata are stored on ~12x 500GB SAS/SATA SSD using 5x replication on 6 hosts. - Version was one of the latest 16.x.x Pacific at the time of the incident. - 3x Mon+mgr and 2 active and 2 hot standby MDS are on separate virtual servers. - typical file size to be stored is from hundreds of MBs to tens of GBs. - this cluster is not the biggest, not having the most HDDs, no special config, I simply see nothing special about this cluster. During investigation I found out the following: - Metadata are outgrowing any time recovery is running on any of maintained clusters (~15 clusters of different usages and sizes) but not this much, this was an extreme situation. - after recovery finish size went fine again. - i think there is slight correlation with recovery width (objects to be touched by recovery in order to recovery everything) and recovery (time) length. But i have no proof. - nothing much else I would like to find out why this happened because i think this can happen again sometime and someone might lose some data if they have less luck. Any ideas are appreciated, or even info if anyone have seen any similar behavior or if i am the only one struggling with issue like this :) Kind regards, Jakub Petrzilka

3 5

MDS nodes blocklisted
by Nathan Harper 04 Aug '23

04 Aug '23

Hi, We're having sporadic problems with a CephFS filesystem where MDSs end up on the OSD blocklist. We're still digging around looking for a cause (Ceph related or other infrastructure cause). The cluster isn't massive (68 OSDs spread over 34 hosts), each host is a VM, with MGR/MON/MDS on non-OSD hosts. Running Ceph 16.2.10 Any suggestions for debugging this further?

3 2

Luminous Bluestore issues and RGW Multi-site Recovery
by Gregory O'Neill 03 Aug '23

03 Aug '23

Hello, I have two main questions here. 1. What can I do when `ceph-bluestore-tool` outputs a stack trace for `fsck`? 2. How does one recover from lost PGs / data corruption in an RGW Multi-site setup? --- I have a Luminous 12.2.12 cluster built on ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64 for all daemons, no ceph packages are installed on the systems. The OSD nodes have 128GB RAM, 6 SATA SSDs (Micron 5200, 2TB) and 1 NVMe SSD split into 4 OSDs. osd_memory_target is set to 10GB and the OSD nodes have 128GB of RAM. That should put me at 100/128GB used. There are 3 PGs down, 3 of the OSDs that had those PGs won't stay online, and they crash fairly quickly after starting. These are running on SATA SSDs which are being replaced with NVMe SSDs. Crush re-weighting the SATA drives down causes some SATA OSDs to crash and some NVMe drives have slow or blocked ops (related to the down PGs). I installed the ceph-osd package on one OSD host. When I ran `ceph-bluestore-tool`, I got a bunch of tcmalloc and unexpected aio errors. Exact output below. I also tried `ceph-objectstore-tool` but received similar results. I cloned the other OSD that has the affected PGs to have a copy I can work on, but I got the exact same results as before. --- From what I can see, this is likely due to bad drives and automation trying to restart down OSDs several times. With 3 down PGs, I am assuming my next step would be to mark those PGs lost. From there, I am unsure what the recovery procedure is to sync "clean" data from other zones into the cluster that was impacted. Is RGW able to handle this? Do I need to use `rclone`? --- $ ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-11 fsck tcmalloc: large alloc 1283989504 bytes == 0x557fdbe46000 @ 0x7fc87e4126d0 0x7fc873354ae9 0x7fc873356073 0x557f89d3d680 0x557f89d2ebcd 0x557f89d30524 0x557f89d318ef 0x557f89d33147 0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a 0x557f89a2c5e1 0x7fc87299d2e1 0x557f89ab03fa (nil) tcmalloc: large alloc 2567970816 bytes == 0x5580286c8000 @ 0x7fc87e4126d0 0x7fc873354ae9 0x7fc873356073 0x557f89d3d680 0x557f89d2ebcd 0x557f89d30524 0x557f89d318ef 0x557f89d33147 0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a 0x557f89a2c5e1 0x7fc87299d2e1 0x557f89ab03fa (nil) tcmalloc: large alloc 5135933440 bytes == 0x5580c17ca000 @ 0x7fc87e4126d0 0x7fc873354ae9 0x7fc873356073 0x557f89d3d680 0x557f89d2ebcd 0x557f89d30524 0x557f89d318ef 0x557f89d33147 0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a 0x557f89a2c5e1 0x7fc87299d2e1 0x557f89ab03fa (nil) tcmalloc: large alloc 3025510400 bytes == 0x557f8f6e6000 @ 0x7fc87e4126d0 0x7fc873354ae9 0x7fc87335582b 0x557f89d75d19 0x557f89d2edda 0x557f89d30524 0x557f89d318ef 0x557f89d33147 0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a 0x557f89a2c5e1 0x7fc87299d2e1 0x557f89ab03fa (nil) tcmalloc: large alloc 2269913088 bytes == 0x55832469e000 @ 0x7fc87e3f2e50 0x7fc87e4121b9 0x7fc8756ca4f7 0x7fc8756cd304 0x557f89cc4661 0x557f89ad0858 0x557f89ad2224 0x557f89cb7b1d 0x557f89de584c 0x557f89de6a7e 0x557f89e05e7b 0x557f89d2cf48 0x557f89d2efd2 0x557f89d30524 0x557f89d318ef 0x557f89d33147 0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a 0x557f89a2c5e1 0x7fc87299d2e1 0x557f89ab03fa (nil) 2023-07-30 08:27:27.531919 7fc86f689700 -1 bdev(0x557f8add4240 /var/lib/ceph/osd/ceph-11/block) aio to 929504952320~2269908992 but returned: 2147479552/build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc86f689700 time 2023-07-30 08:27:27.532004 /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0 == "unexpected aio error") ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7fc8757242c2] 2: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7] 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d] 4: (()+0x74a4) [0x7fc8740104a4] 5: (clone()+0x3f) [0x7fc872a65d0f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 2023-07-30 08:27:27.544215 7fc86f689700 -1 /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc86f689700 time 2023-07-30 08:27:27.532004 /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0 == "unexpected aio error") ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7fc8757242c2] 2: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7] 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d] 4: (()+0x74a4) [0x7fc8740104a4] 5: (clone()+0x3f) [0x7fc872a65d0f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -1> 2023-07-30 08:27:27.531919 7fc86f689700 -1 bdev(0x557f8add4240 /var/lib/ceph/osd/ceph-11/block) aio to 929504952320~2269908992 but returned: 2147479552 0> 2023-07-30 08:27:27.544215 7fc86f689700 -1 /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc86f689700 time 2023-07-30 08:27:27.532004 /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0 == "unexpected aio error") ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7fc8757242c2] 2: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7] 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d] 4: (()+0x74a4) [0x7fc8740104a4] 5: (clone()+0x3f) [0x7fc872a65d0f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. *** Caught signal (Aborted) ** in thread 7fc86f689700 thread_name:bstore_aio ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (()+0x424fc4) [0x557f89d25fc4] 2: (()+0x110e0) [0x7fc87401a0e0] 3: (gsignal()+0xcf) [0x7fc8729affff] 4: (abort()+0x16a) [0x7fc8729b142a] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x7fc87572444e] 6: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7] 7: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d] 8: (()+0x74a4) [0x7fc8740104a4] 9: (clone()+0x3f) [0x7fc872a65d0f] 2023-07-30 08:27:27.549175 7fc86f689700 -1 *** Caught signal (Aborted) ** in thread 7fc86f689700 thread_name:bstore_aio ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (()+0x424fc4) [0x557f89d25fc4] 2: (()+0x110e0) [0x7fc87401a0e0] 3: (gsignal()+0xcf) [0x7fc8729affff] 4: (abort()+0x16a) [0x7fc8729b142a] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x7fc87572444e] 6: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7] 7: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d] 8: (()+0x74a4) [0x7fc8740104a4] 9: (clone()+0x3f) [0x7fc872a65d0f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 0> 2023-07-30 08:27:27.549175 7fc86f689700 -1 *** Caught signal (Aborted) ** in thread 7fc86f689700 thread_name:bstore_aio ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (()+0x424fc4) [0x557f89d25fc4] 2: (()+0x110e0) [0x7fc87401a0e0] 3: (gsignal()+0xcf) [0x7fc8729affff] 4: (abort()+0x16a) [0x7fc8729b142a] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x7fc87572444e] 6: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7] 7: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d] 8: (()+0x74a4) [0x7fc8740104a4] 9: (clone()+0x3f) [0x7fc872a65d0f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Aborted $ ceph-objectstore-tool --data-path=/var/lib/ceph/osd/ceph-11 --op list-pgs tcmalloc: large alloc 1283989504 bytes == 0x5649b1bdc000 @ 0x7f3af5e756d0 0x7f3aeafbbae9 0x7f3aeafbd073 0x56495defb9e0 0x56495deed01d 0x56495deee974 0x56495deefd3f 0x56495def1597 0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4 0x56495d7de4db 0x7f3aea6042e1 0x56495d86853a (nil) tcmalloc: large alloc 2567970816 bytes == 0x5649fe45e000 @ 0x7f3af5e756d0 0x7f3aeafbbae9 0x7f3aeafbd073 0x56495defb9e0 0x56495deed01d 0x56495deee974 0x56495deefd3f 0x56495def1597 0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4 0x56495d7de4db 0x7f3aea6042e1 0x56495d86853a (nil) tcmalloc: large alloc 5135933440 bytes == 0x564a97560000 @ 0x7f3af5e756d0 0x7f3aeafbbae9 0x7f3aeafbd073 0x56495defb9e0 0x56495deed01d 0x56495deee974 0x56495deefd3f 0x56495def1597 0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4 0x56495d7de4db 0x7f3aea6042e1 0x56495d86853a (nil) tcmalloc: large alloc 3025510400 bytes == 0x56496547c000 @ 0x7f3af5e756d0 0x7f3aeafbbae9 0x7f3aeafbc82b 0x56495df34079 0x56495deed22a 0x56495deee974 0x56495deefd3f 0x56495def1597 0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4 0x56495d7de4db 0x7f3aea6042e1 0x56495d86853a (nil) tcmalloc: large alloc 2269913088 bytes == 0x564cfa402000 @ 0x7f3af5e55e50 0x7f3af5e751b9 0x7f3aed12d4f7 0x7f3aed130304 0x56495de9fbc1 0x56495de7a5f8 0x56495de7bfc4 0x56495de9307d 0x56495dfa32dc 0x56495dfa450e 0x56495dfc34db 0x56495deeb398 0x56495deed422 0x56495deee974 0x56495deefd3f 0x56495def1597 0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4 0x56495d7de4db 0x7f3aea6042e1 0x56495d86853a (nil) /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f3ae72f0700 time 2023-07-30 08:37:16.531432 /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0 == "unexpected aio error") ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f3aed1872c2] 2: (KernelDevice::_aio_thread()+0x1377) [0x56495de9ca27] 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x56495dea27bd] 4: (()+0x74a4) [0x7f3aeba734a4] 5: (clone()+0x3f) [0x7f3aea6ccd0f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. *** Caught signal (Aborted) ** in thread 7f3ae72f0700 thread_name:bstore_aio ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (()+0x94a0f4) [0x56495debe0f4] 2: (()+0x110e0) [0x7f3aeba7d0e0] 3: (gsignal()+0xcf) [0x7f3aea616fff] 4: (abort()+0x16a) [0x7f3aea61842a] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x7f3aed18744e] 6: (KernelDevice::_aio_thread()+0x1377) [0x56495de9ca27] 7: (KernelDevice::AioCompletionThread::entry()+0xd) [0x56495dea27bd] 8: (()+0x74a4) [0x7f3aeba734a4] 9: (clone()+0x3f) [0x7f3aea6ccd0f] Aborted -- Gregory O’Neill

3 3

ref v18.2.0 QE Validation status
by Yuri Weinstein 03 Aug '23

03 Aug '23

Details of this release are summarized here: https://tracker.ceph.com/issues/62231#note-1 Seeking approvals/reviews for: smoke - Laura, Radek rados - Neha, Radek, Travis, Ernesto, Adam King rgw - Casey fs - Venky orch - Adam King rbd - Ilya krbd - Ilya upgrade-clients:client-upgrade* - in progress powercycle - Brad Please reply to this email with approval and/or trackers of known issues/PRs to address them. bookworm distro support is an outstanding issue. TIA YuriW

11 19

1 Large omap object found
by Mark Johnson 02 Aug '23

02 Aug '23

I've been going round and round in circles trying to work this one out but I'm getting nowhere. We're running a 4 node quincy cluster (17.2.6) which recently reported the following: ceph.log-20230729.gz:2023-07-28T08:31:42.390003+0000 osd.26 (osd.26) 13834 : cluster [WRN] Large omap object found. Object: 5:6c65dd84:users.uid::callrecordings$callrecordings_rw.buckets:head PG: 5.21bba636 (5.16) Key count: 378454 Size (bytes): 75565579 This happened a week or so ago (only the key count was only just over the 200000 threshold on that occasion) and after much searching around, I found an article that suggested a deep scrub on the pg would likely resolve the issue, so I forced a deep scrub and shortly after, the warning cleared. Came into the office today to discover the above. It's on the same PG as before which is in the default.rgw.meta pool. This time, after forcing a deep-scrub on that PG, nothing changed. I did it a second time just to be sure but got the same result. I keep finding a suse article that simply suggests increasing the threshold to the previous default of 2,000,000, but other articles I read say it was lowered for a reason and that by the time it hits that figure, it's too late so I don't want to just mask it. Problem is that I don't really understand it. I found a thread here from a bit over two years ago but their issue was in the default.rgw.buckets.index pool. A step in the solution was to list out the problematic object id and check the objects per shard however, if I issue the command "rados -p default.rgw.meta ls" it returns nothing. I get a big list from "rados -p default.rgw.buckets.index ls" just nothing from the first pool. I think it may be because the meta pool isn't indexed based on something I read, but I really don't know what I'm talking about tbh. I don't know if this is helpful, but if I list out all the PGs for that pool, there are 32 PGs and 5.16 shows 80186950 bytes and 401505 keys. PG 5.c has 75298 and 384 keys. The remaining 30 PGs show zero bytes and zero keys. I'm really not sure how to troubleshoot and resolve from here. For the record, dynamic resharding is enabled in that no options have been set in the config and that is the default setting. Based on the suse article I mentioned which also references the default.rgw.meta pool, I'm gathering our issue is because we have so many buckets that are all owned by the one user and the solution is either: * delete unused buckets * create multiple users and spread buckets evenly across all users (not something we can do) * increase the threshold to stop the warning Problem is that I'm having trouble verifying this is the issue. I've tried dumping out bucket stats to a file (radosgw-admin bucket stats > bucket_stats.txt) but after three hours this is still running with no output. Thanks for your time, Mark

3 12

RGW multi-site recovery
by Gregory O'Neill 31 Jul '23

31 Jul '23

Hello, I have 3 Luminous (12.2.12) clusters setup with rgw multi-site. One cluster recently suffered data loss from PGs lost to bad SSDs. We're looking at how to recover from this scenario. The other two clusters are healthy. Will RGW sync the missing data? Do buckets and objects need to be restored using rclone? From `radosgw-admin sync error list`, these are the error types: $ cat sync-error-list | jq -r '.[].entries[].info.message' | sort -h | uniq -c | sort -h 2 failed to sync bucket instance: (13) Permission denied 3 failed to sync object(13) Permission denied 100 failed to sync bucket instance: (125) Operation canceled 239 failed to sync bucket instance: (5) Input/output error 404 failed to read remote metadata entry: (11) Resource temporarily unavailable 2148 failed to sync object(5) Input/output error -- Gregory O’Neill

1 0

LARGE_OMAP_OBJECTS warning and bucket has lot of unknown objects and 1999 shards.
by Uday Bhaskar Jalagam 31 Jul '23

31 Jul '23

Hello Everyone , I am getting [WRN] LARGE_OMAP_OBJECTS: 18 large omap objects warning in one of my clusters . I see one of the buckets has a huge number of shards 1999 and "num_objects": 221185360 when I check bucket stats using radosgw-admin bucket stats . However I see only 8 files when I actually do list bucket using python boto3. I am not sure what those objects are missing somewhere . # radosgw-admin bucket stats --bucket=epbucket { "bucket": "epbucket", "num_shards": 1999, "tenant": "", "zonegroup": "ac361c18-7420-4739-8e92-73109fc8ec80", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "f3e47d62-b9ff-4f38-b863-d7bfc155d698.35744173.268063", "marker": "f3e47d62-b9ff-4f38-b863-d7bfc155d698.35744173.1225", "index_type": "Normal", "owner": "user", "ver": "0#3,1#1,2#1,3#9,4#1,5#3,6#1,7#3,8#1,9#8,10#7,11#1,12#3,13#1,14#1,15#1,16#1,17#1,18#5,19#1,20#1,21#3,22#3,23#1,24#3,25#1,26#1,27#1,28#3,29#3,30#2,31#3,32#2,33#1,34#3,35#1,36#3,37#1,38#3,39#1,40#3,41#1,42#1,43#1,44#1,45#1,46#3,47#1,48#3,49#1,50#3,51#1,52#1,53#1,54#1,55#1,56#3,57#1,58#1,59#5,60#1,61#1,62#1,63#3,64#1,65#1,66#3,67#1,68#5,69#1,70#1,71#3,72#1,73#3,74#2,75#3,76#1,77#3,78#1,79#1,80#3,81#3,82#3,83#3,84#1,85#1,86#3,87#3,88#8,89#1,90#1,91#1,92#5,93#7,94#1,95#1,96#3,97#1,98#5,99#1,100#1,101#3,102#5,103#3,104#1,105#1,106#1,107#1,108#1,109#239050,110#1,111#1,112#1,113#5,114#1,115#1,116#1,117#3,118#3,119#9,120#1,121#1,122#3,123#5,124#3,125#3,126#1,127#1,128#1,129#3,130#3,131#3,132#3,133#5,134#1,135#1,136#1,137#1,138#5,139#3,140#1,141#1,142#3,143#3,144#1,145#1,146#3,147#3,148#3,149#1,150#3,151#1,152#1,153#3,154#7,155#1,156#5,157#3,158#3,159#3,160#5,161#3,162#3,163#3,164#3,165#1,166#1,167#5,168#3,169#1,170#1,171#1,172#4,173#1,174#12,175#1,176#3,177#3,178#3,179#1,180#3,181#33,182#1,183#1,184#1,185#1,186#3,187#3,188#11,189#3,190#5,191#3,192#1,193#5,194#3,195#2,196#3,197#1,198#3,199#1,200#7,201#3,202#7,203#1,204#3,205#3,206#1,207#1,208#3,209#1,210#7,211#3,212#2,213#9,214#1,215#3,216#1,217#1,218#7,219#1,220#1,221#5,222#1,223#1,224#3,225#1,226#1,227#3,228#3,229#1,230#5,231#5,232#1,233#3,234#1,235#1,236#5,237#1,238#1,239#3,240#7,241#3,242#1,243#3,244#1,245#1,246#3,247#1,248#3,249#3,250#1,251#1,252#3,253#1,254#3,255#1,256#3,257#1,258#1,259#7,260#9,261#3,262#1,263#1,264#3,265#1,266#1,267#5,268#1,269#1,270#1,271#1,272#1,273#3,274#1,275#1,276#1,277#3,278#1,279#7,280#1,281#238803,282#3,283#1,284#1,285#1,286#1,287#3,288#1,289#1,290#1,291#16,292#3,293#3,294#1,295#1,296#5,297#3,298#3,299#7,300#5,301#5,302#1,303#1,304#1,305#3,306#3,307#5,308#1,309#1,310#3,311#3,312#1,313#1,314#5,315#3,316#7,317#1,318#1,319#3,320#3,321#3,322#1,323#9,324#3,325#1,326#5,327#1,328#3,329#9,330#3,331#1,332#3,333#3,334#3,335#3,336#3,337#9,338#1,339#1,340#1,341#3,342#3,343#3,344#2,345#3,346#3,347#1,348#3,349#3,350#5,351#1,352#5,353#8,354#5,355#1,356#3,357#1,358#3,359#1,360#1,361#1,362#1,363#5,364#1,365#1,366#1,367#1,368#1,369#1,370#3,371#3,372#3,373#3,374#1,375#3,376#5,377#8,378#1,379#1,380#1,381#1,382#3,383#3,384#1,385#3,386#1,387#1,388#1,389#2,390#1,391#1,392#1,393#1,394#5,395#5,396#1,397#5,398#3,399#1,400#1,401#7,402#1,403#5,404#1,405#1,406#3,407#3,408#3,409#1,410#1,411#1,412#1,413#1,414#1,415#1,416#1,417#1,418#3,419#3,420#1,421#5,422#5,423#5,424#1,425#1,426#1,427#1,428#9,429#1,430#3,431#1,432#3,433#3,434#1,435#1,436#3,437#5,438#3,439#3,440#1,441#1,442#1,443#1,444#1,445#5,446#3,447#9,448#1,449#1,450#1,451#1,452#1,453#3,454#1,455#1,456#3,457#5,458#5,459#3,460#9,461#1,462#1,463#1,464#1,465#1,466#1,467#3,468#1,469#3,470#8,471#1,472#1,473#1,474#7,475#3,476#1,477#11,478#10,479#1,480#1,481#1,482#3,483#1,484#7,485#1,486#1,487#3,488#8,489#1,490#1,491#8,492#3,493#1,494#3,495#5,496#1,497#1,498#1,499#3,500#1,501#5,502#1,503#1,504#3,505#1,506#1,507#1,508#3,509#3,510#3,511#1,512#7,513#3,514#1,515#3,516#5,517#1,518#3,519#1,520#3,521#3,522#3,523#5,524#3,525#3,526#1,527#1,528#1,529#1,530#1,531#3,532#1,533#3,534#5,535#5,536#1,537#3,538#3,539#5,540#1,541#1,542#1,543#1,544#7,545#3,546#1,547#1,548#1,549#3,550#1,551#1,552#3,553#1,554#3,555#3,556#1,557#3,558#5,559#3,560#3,561#8,562#1,563#3,564#7,565#3,566#5,567#1,568#3,569#3,570#3,571#3,572#5,573#1,574#1,575#5,576#5,577#3,578#7,579#1,580#1,581#5,582#1,583#3,584#1,585#1,586#9,587#9,588#3,589#1,590#1,591#3,592#3,593#1,594#10,595#3,596#1,597#3,598#1,599#1,600#1,601#3,602#3,603#1,604#3,605#5,606#1,607#1,608#1,609#10,610#3,611#264711,612#3,613#5,614#3,615#3,616#1,617#1,618#7,619#3,620#3,621#1,622#3,623#1,624#3,625#1,626#3,627#1,628#1,629#3,630#1,631#2,632#233833,633#1,634#1,635#5,636#1,637#1,638#1,639#1,640#3,641#1,642#1,643#3,644#1,645#1,646#10,647#1,648#1,649#1,650#3,651#1,652#1,653#5,654#1,655#7,656#7,657#1,658#1,659#5,660#3,661#3,662#1,663#3,664#1,665#1,666#3,667#1,668#7,669#1,670#3,671#3,672#1,673#1,674#1,675#1,676#1,677#3,678#1,679#11,680#1,681#7,682#5,683#1,684#1,685#1,686#1,687#3,688#1,689#1,690#3,691#3,692#1,693#1,694#1,695#1,696#1,697#1,698#1,699#5,700#1,701#1,702#1,703#1,704#3,705#1,706#8,707#3,708#3,709#8,710#1,711#2,712#3,713#1,714#1,715#10,716#3,717#1,718#1,719#3,720#1,721#7,722#3,723#1,724#5,725#1,726#5,727#1,728#3,729#1,730#3,731#1,732#7,733#3,734#3,735#1,736#5,737#1,738#5,739#1,740#1,741#1,742#1,743#1,744#1,745#3,746#1,747#1,748#1,749#5,750#12,751#1,752#9,753#7,754#5,755#1,756#1,757#1,758#1,759#5,760#1,761#3,762#3,763#3,764#1,765#1,766#11,767#5,768#3,769#3,770#5,771#3,772#1,773#1,774#1,775#1,776#7,777#5,778#3,779#3,780#2,781#5,782#1,783#1,784#5,785#1,786#3,787#1,788#3,789#5,790#10,791#3,792#7,793#1,794#1,795#3,796#1,797#3,798#1,799#1,800#1,801#1,802#3,803#1,804#3,805#1,806#1,807#1,808#1,809#3,810#1,811#3,812#2,813#1,814#1,815#1,816#3,817#1,818#3,819#1,820#3,821#1,822#1,823#1,824#5,825#5,826#3,827#1,828#3,829#3,830#1,831#1,832#3,833#5,834#3,835#3,836#3,837#7,838#7,839#3,840#1,841#3,842#1,843#1,844#1,845#12,846#1,847#3,848#1,849#1,850#1,851#1,852#1,853#5,854#3,855#1,856#7,857#8,858#3,859#1,860#3,861#1,862#1,863#1,864#1,865#1,866#1,867#5,868#3,869#1,870#3,871#3,872#7,873#9,874#7,875#1,876#1,877#3,878#1,879#3,880#1,881#1,882#1,883#1,884#3,885#7,886#1,887#1,888#1,889#3,890#1,891#1,892#1,893#1,894#3,895#1,896#1,897#3,898#1,899#3,900#3,901#1,902#1,903#1,904#1,905#1,906#1,907#3,908#3,909#3,910#3,911#1,912#1,913#5,914#3,915#3,916#1,917#1,918#1,919#1,920#3,921#1,922#7,923#2,924#3,925#1,926#3,927#1,928#7,929#1,930#5,931#1,932#3,933#3,934#1,935#1,936#3,937#1,938#3,939#1,940#5,941#1,942#5,943#1,944#1,945#12,946#1,947#5,948#1,949#1,950#3,951#7,952#1,953#1,954#1,955#3,956#1,957#5,958#3,959#3,960#3,961#3,962#3,963#3,964#3,965#3,966#1,967#3,968#3,969#3,970#1,971#18,972#3,973#3,974#1,975#1,976#1,977#3,978#1,979#1,980#1,981#1,982#3,983#8,984#5,985#1,986#8,987#3,988#1,989#1,990#1,991#1,992#1,993#5,994#1,995#373523,996#3,997#3,998#3,999#1,1000#1,1001#1,1002#1,1003#1,1004#3,1005#1,1006#1,1007#1,1008#1,1009#7,1010#1,1011#3,1012#1,1013#1,1014#1,1015#1,1016#3,1017#3,1018#1,1019#5,1020#1,1021#1,1022#5,1023#3,1024#1,1025#1,1026#7,1027#3,1028#3,1029#3,1030#1,1031#3,1032#1,1033#1,1034#3,1035#9,1036#1,1037#1,1038#3,1039#5,1040#5,1041#5,1042#3,1043#1,1044#3,1045#7,1046#1,1047#3,1048#3,1049#1,1050#1,1051#1,1052#3,1053#1,1054#3,1055#3,1056#5,1057#8,1058#3,1059#1,1060#3,1061#3,1062#7,1063#1,1064#1,1065#7,1066#1,1067#1,1068#1,1069#3,1070#3,1071#3,1072#3,1073#7,1074#1,1075#3,1076#1,1077#3,1078#1,1079#1,1080#5,1081#1,1082#1,1083#1,1084#3,1085#1,1086#3,1087#1,1088#1,1089#1,1090#1,1091#1,1092#5,1093#1,1094#5,1095#3,1096#1,1097#1,1098#3,1099#1,1100#1,1101#5,1102#1,1103#5,1104#3,1105#1,1106#1,1107#3,1108#3,1109#1,1110#1,1111#1,1112#1,1113#3,1114#1,1115#3,1116#3,1117#1,1118#1,1119#1,1120#1,1121#1,1122#3,1123#1,1124#5,1125#1,1126#1,1127#5,1128#9,1129#5,1130#3,1131#1,1132#7,1133#1,1134#3,1135#2,1136#1,1137#1,1138#1,1139#1,1140#1,1141#1,1142#5,1143#3,1144#1,1145#5,1146#10,1147#235181,1148#1,1149#3,1150#1,1151#3,1152#1,1153#3,1154#7,1155#1,1156#1,1157#3,1158#1,1159#1,1160#3,1161#5,1162#1,1163#1,1164#1,1165#1,1166#5,1167#1,1168#3,1169#1,1170#1,1171#3,1172#3,1173#2,1174#1,1175#1,1176#1,1177#1,1178#1,1179#1,1180#5,1181#5,1182#1,1183#1,1184#1,1185#3,1186#1,1187#1,1188#1,1189#1,1190#5,1191#9,1192#3,1193#8,1194#1,1195#1,1196#1,1197#1,1198#1,1199#1,1200#3,1201#1,1202#1,1203#1,1204#3,1205#1,1206#1,1207#1,1208#1,1209#1,1210#1,1211#1,1212#1,1213#2,1214#1,1215#1,1216#3,1217#3,1218#3,1219#3,1220#1,1221#3,1222#1,1223#1,1224#3,1225#17,1226#3,1227#3,1228#239050,1229#1,1230#3,1231#3,1232#1,1233#5,1234#3,1235#1,1236#1,1237#1,1238#1,1239#1,1240#1,1241#5,1242#1,1243#1,1244#3,1245#5,1246#1,1247#3,1248#1,1249#3,1250#1,1251#3,1252#3,1253#1,1254#1,1255#1,1256#1,1257#1,1258#1,1259#3,1260#5,1261#3,1262#3,1263#1,1264#1,1265#1,1266#1,1267#1,1268#5,1269#3,1270#3,1271#5,1272#1,1273#1,1274#1,1275#5,1276#3,1277#1,1278#8,1279#5,1280#5,1281#1,1282#5,1283#1,1284#1,1285#5,1286#1,1287#3,1288#1,1289#1,1290#3,1291#1,1292#1,1293#3,1294#1,1295#1,1296#1,1297#9,1298#3,1299#1,1300#1,1301#1,1302#3,1303#1,1304#3,1305#3,1306#1,1307#3,1308#1,1309#1,1310#3,1311#3,1312#1,1313#3,1314#1,1315#1,1316#1,1317#5,1318#3,1319#1,1320#1,1321#1,1322#1,1323#3,1324#1,1325#3,1326#8,1327#1,1328#7,1329#6,1330#1,1331#1,1332#1,1333#3,1334#1,1335#1,1336#1,1337#1,1338#1,1339#1,1340#3,1341#3,1342#1,1343#3,1344#5,1345#1,1346#1,1347#1,1348#3,1349#3,1350#1,1351#1,1352#1,1353#1,1354#8,1355#1,1356#3,1357#3,1358#3,1359#3,1360#1,1361#1,1362#1,1363#5,1364#3,1365#1,1366#1,1367#3,1368#5,1369#5,1370#1,1371#1,1372#1,1373#3,1374#1,1375#1,1376#1,1377#7,1378#1,1379#3,1380#1,1381#1,1382#1,1383#1,1384#1,1385#6,1386#1,1387#1,1388#3,1389#1,1390#1,1391#1,1392#1,1393#3,1394#1,1395#3,1396#1,1397#1,1398#1,1399#1,1400#5,1401#3,1402#5,1403#3,1404#5,1405#5,1406#3,1407#1,1408#3,1409#1,1410#3,1411#1,1412#1,1413#3,1414#1,1415#1,1416#5,1417#3,1418#12,1419#3,1420#1,1421#1,1422#3,1423#1,1424#5,1425#7,1426#3,1427#1,1428#1,1429#1,1430#3,1431#1,1432#3,1433#1,1434#1,1435#1,1436#1,1437#1,1438#3,1439#1,1440#1,1441#3,1442#3,1443#1,1444#1,1445#1,1446#1,1447#1,1448#3,1449#3,1450#1,1451#1,1452#3,1453#1,1454#1,1455#1,1456#3,1457#1,1458#5,1459#3,1460#1,1461#3,1462#1,1463#1,1464#1,1465#1,1466#1,1467#1,1468#6,1469#1,1470#1,1471#1,1472#1,1473#1,1474#1,1475#1,1476#5,1477#1,1478#3,1479#8,1480#1,1481#3,1482#1,1483#1,1484#7,1485#1,1486#1,1487#3,1488#3,1489#1,1490#1,1491#1,1492#1,1493#1,1494#1,1495#5,1496#5,1497#1,1498#8,1499#1,1500#3,1501#3,1502#1,1503#1,1504#5,1505#1,1506#1,1507#3,1508#3,1509#3,1510#1,1511#3,1512#3,1513#1,1514#1,1515#1,1516#1,1517#3,1518#1,1519#3,1520#1,1521#1,1522#3,1523#1,1524#7,1525#1,1526#1,1527#1,1528#1,1529#1,1530#3,1531#1,1532#1,1533#1,1534#3,1535#5,1536#3,1537#3,1538#7,1539#3,1540#1,1541#1,1542#1,1543#3,1544#1,1545#1,1546#1,1547#1,1548#1,1549#7,1550#1,1551#1,1552#3,1553#1,1554#5,1555#3,1556#1,1557#1,1558#1,1559#5,1560#1,1561#1,1562#1,1563#1,1564#5,1565#3,1566#3,1567#3,1568#1,1569#3,1570#1,1571#3,1572#1,1573#13,1574#1,1575#1,1576#1,1577#1,1578#1,1579#1,1580#1,1581#1,1582#3,1583#1,1584#1,1585#1,1586#3,1587#1,1588#3,1589#5,1590#10,1591#3,1592#3,1593#1,1594#7,1595#1,1596#3,1597#5,1598#1,1599#1,1600#5,1601#3,1602#1,1603#1,1604#1,1605#3,1606#1,1607#1,1608#5,1609#15,1610#3,1611#3,1612#3,1613#1,1614#3,1615#1,1616#1,1617#1,1618#3,1619#3,1620#1,1621#3,1622#7,1623#1,1624#5,1625#1,1626#3,1627#3,1628#1,1629#1,1630#1,1631#3,1632#1,1633#1,1634#1,1635#1,1636#1,1637#3,1638#1,1639#3,1640#1,1641#1,1642#1,1643#1,1644#3,1645#1,1646#1,1647#3,1648#5,1649#1,1650#1,1651#1,1652#7,1653#3,1654#1,1655#1,1656#13,1657#1,1658#3,1659#3,1660#9,1661#1,1662#7,1663#3,1664#3,1665#1,1666#1,1667#1,1668#1,1669#1,1670#1,1671#16,1672#3,1673#1,1674#3,1675#1,1676#1,1677#1,1678#3,1679#1,1680#1,1681#1,1682#1,1683#1,1684#3,1685#1,1686#1,1687#1,1688#5,1689#5,1690#1,1691#3,1692#1,1693#1,1694#3,1695#3,1696#1,1697#5,1698#1,1699#3,1700#1,1701#1,1702#1,1703#3,1704#1,1705#5,1706#1,1707#9,1708#1,1709#1,1710#3,1711#3,1712#1,1713#5,1714#1,1715#1,1716#1,1717#1,1718#1,1719#1,1720#3,1721#3,1722#1,1723#1,1724#3,1725#1,1726#1,1727#1,1728#1,1729#3,1730#1,1731#3,1732#3,1733#10,1734#3,1735#3,1736#3,1737#3,1738#1,1739#1,1740#5,1741#1,1742#1,1743#1,1744#3,1745#1,1746#5,1747#3,1748#5,1749#1,1750#3,1751#9,1752#3,1753#1,1754#1,1755#3,1756#1,1757#3,1758#1,1759#3,1760#3,1761#3,1762#3,1763#1,1764#1,1765#5,1766#1,1767#1,1768#5,1769#5,1770#1,1771#1,1772#7,1773#7,1774#5,1775#3,1776#3,1777#1,1778#3,1779#1,1780#1,1781#3,1782#1,1783#1,1784#3,1785#1,1786#1,1787#1,1788#1,1789#1,1790#3,1791#3,1792#3,1793#1,1794#1,1795#1,1796#3,1797#1,1798#8,1799#3,1800#5,1801#9,1802#4,1803#3,1804#3,1805#1,1806#1,1807#3,1808#3,1809#3,1810#1,1811#5,1812#3,1813#5,1814#10,1815#1,1816#3,1817#1,1818#1,1819#3,1820#1,1821#1,1822#1,1823#3,1824#1,1825#1,1826#5,1827#3,1828#1,1829#3,1830#8,1831#3,1832#1,1833#3,1834#5,1835#1,1836#3,1837#3,1838#1,1839#3,1840#1,1841#3,1842#1,1843#1,1844#1,1845#8,1846#1,1847#1,1848#7,1849#1,1850#1,1851#3,1852#1,1853#1,1854#1,1855#1,1856#3,1857#1,1858#3,1859#1,1860#1,1861#589361,1862#1,1863#1,1864#2,1865#3,1866#4,1867#3,1868#1,1869#3,1870#3,1871#7,1872#1,1873#1,1874#1,1875#1,1876#1,1877#9,1878#7,1879#1,1880#1,1881#1,1882#3,1883#3,1884#1,1885#10,1886#8,1887#1,1888#1,1889#1,1890#1,1891#1,1892#1,1893#12,1894#1,1895#1,1896#1,1897#3,1898#1,1899#1,1900#5,1901#12,1902#1,1903#1,1904#1,1905#1,1906#1,1907#1,1908#1,1909#1,1910#3,1911#1,1912#1,1913#1,1914#7,1915#1,1916#5,1917#1,1918#1,1919#233919,1920#1,1921#3,1922#1,1923#1,1924#1,1925#3,1926#1,1927#3,1928#1,1929#1,1930#1,1931#1,1932#1,1933#1,1934#3,1935#1,1936#1,1937#3,1938#3,1939#1,1940#1,1941#3,1942#1,1943#1,1944#3,1945#1,1946#1,1947#7,1948#1,1949#3,1950#3,1951#1,1952#1,1953#1,1954#1,1955#9,1956#1,1957#1,1958#1,1959#1,1960#1,1961#1,1962#3,1963#1,1964#1,1965#2,1966#1,1967#3,1968#4,1969#1,1970#3,1971#1,1972#5,1973#1,1974#3,1975#1,1976#1,1977#5,1978#1,1979#8,1980#1,1981#1,1982#5,1983#1,1984#1,1985#1,1986#1,1987#1,1988#1,1989#1,1990#1,1991#1,1992#1,1993#1,1994#1,1995#3,1996#1,1997#1,1998#3", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0,128#0,129#0,130#0,131#0,132#0,133#0,134#0,135#0,136#0,137#0,138#0,139#0,140#0,141#0,142#0,143#0,144#0,145#0,146#0,147#0,148#0,149#0,150#0,151#0,152#0,153#0,154#0,155#0,156#0,157#0,158#0,159#0,160#0,161#0,162#0,163#0,164#0,165#0,166#0,167#0,168#0,169#0,170#0,171#0,172#0,173#0,174#0,175#0,176#0,177#0,178#0,179#0,180#0,181#0,182#0,183#0,184#0,185#0,186#0,187#0,188#0,189#0,190#0,191#0,192#0,193#0,194#0,195#0,196#0,197#0,198#0,199#0,200#0,201#0,202#0,203#0,204#0,205#0,206#0,207#0,208#0,209#0,210#0,211#0,212#0,213#0,214#0,215#0,216#0,217#0,218#0,219#0,220#0,221#0,222#0,223#0,224#0,225#0,226#0,227#0,228#0,229#0,230#0,231#0,232#0,233#0,234#0,235#0,236#0,237#0,238#0,239#0,240#0,241#0,242#0,243#0,244#0,245#0,246#0,247#0,248#0,249#0,250#0,251#0,252#0,253#0,254#0,255#0,256#0,257#0,258#0,259#0,260#0,261#0,262#0,263#0,264#0,265#0,266#0,267#0,268#0,269#0,270#0,271#0,272#0,273#0,274#0,275#0,276#0,277#0,278#0,279#0,280#0,281#0,282#0,283#0,284#0,285#0,286#0,287#0,288#0,289#0,290#0,291#0,292#0,293#0,294#0,295#0,296#0,297#0,298#0,299#0,300#0,301#0,302#0,303#0,304#0,305#0,306#0,307#0,308#0,309#0,310#0,311#0,312#0,313#0,314#0,315#0,316#0,317#0,318#0,319#0,320#0,321#0,322#0,323#0,324#0,325#0,326#0,327#0,328#0,329#0,330#0,331#0,332#0,333#0,334#0,335#0,336#0,337#0,338#0,339#0,340#0,341#0,342#0,343#0,344#0,345#0,346#0,347#0,348#0,349#0,350#0,351#0,352#0,353#0,354#0,355#0,356#0,357#0,358#0,359#0,360#0,361#0,362#0,363#0,364#0,365#0,366#0,367#0,368#0,369#0,370#0,371#0,372#0,373#0,374#0,375#0,376#0,377#0,378#0,379#0,380#0,381#0,382#0,383#0,384#0,385#0,386#0,387#0,388#0,389#0,390#0,391#0,392#0,393#0,394#0,395#0,396#0,397#0,398#0,399#0,400#0,401#0,402#0,403#0,404#0,405#0,406#0,407#0,408#0,409#0,410#0,411#0,412#0,413#0,414#0,415#0,416#0,417#0,418#0,419#0,420#0,421#0,422#0,423#0,424#0,425#0,426#0,427#0,428#0,429#0,430#0,431#0,432#0,433#0,434#0,435#0,436#0,437#0,438#0,439#0,440#0,441#0,442#0,443#0,444#0,445#0,446#0,447#0,448#0,449#0,450#0,451#0,452#0,453#0,454#0,455#0,456#0,457#0,458#0,459#0,460#0,461#0,462#0,463#0,464#0,465#0,466#0,467#0,468#0,469#0,470#0,471#0,472#0,473#0,474#0,475#0,476#0,477#0,478#0,479#0,480#0,481#0,482#0,483#0,484#0,485#0,486#0,487#0,488#0,489#0,490#0,491#0,492#0,493#0,494#0,495#0,496#0,497#0,498#0,499#0,500#0,501#0,502#0,503#0,504#0,505#0,506#0,507#0,508#0,509#0,510#0,511#0,512#0,513#0,514#0,515#0,516#0,517#0,518#0,519#0,520#0,521#0,522#0,523#0,524#0,525#0,526#0,527#0,528#0,529#0,530#0,531#0,532#0,533#0,534#0,535#0,536#0,537#0,538#0,539#0,540#0,541#0,542#0,543#0,544#0,545#0,546#0,547#0,548#0,549#0,550#0,551#0,552#0,553#0,554#0,555#0,556#0,557#0,558#0,559#0,560#0,561#0,562#0,563#0,564#0,565#0,566#0,567#0,568#0,569#0,570#0,571#0,572#0,573#0,574#0,575#0,576#0,577#0,578#0,579#0,580#0,581#0,582#0,583#0,584#0,585#0,586#0,587#0,588#0,589#0,590#0,591#0,592#0,593#0,594#0,595#0,596#0,597#0,598#0,599#0,600#0,601#0,602#0,603#0,604#0,605#0,606#0,607#0,608#0,609#0,610#0,611#0,612#0,613#0,614#0,615#0,616#0,617#0,618#0,619#0,620#0,621#0,622#0,623#0,624#0,625#0,626#0,627#0,628#0,629#0,630#0,631#0,632#0,633#0,634#0,635#0,636#0,637#0,638#0,639#0,640#0,641#0,642#0,643#0,644#0,645#0,646#0,647#0,648#0,649#0,650#0,651#0,652#0,653#0,654#0,655#0,656#0,657#0,658#0,659#0,660#0,661#0,662#0,663#0,664#0,665#0,666#0,667#0,668#0,669#0,670#0,671#0,672#0,673#0,674#0,675#0,676#0,677#0,678#0,679#0,680#0,681#0,682#0,683#0,684#0,685#0,686#0,687#0,688#0,689#0,690#0,691#0,692#0,693#0,694#0,695#0,696#0,697#0,698#0,699#0,700#0,701#0,702#0,703#0,704#0,705#0,706#0,707#0,708#0,709#0,710#0,711#0,712#0,713#0,714#0,715#0,716#0,717#0,718#0,719#0,720#0,721#0,722#0,723#0,724#0,725#0,726#0,727#0,728#0,729#0,730#0,731#0,732#0,733#0,734#0,735#0,736#0,737#0,738#0,739#0,740#0,741#0,742#0,743#0,744#0,745#0,746#0,747#0,748#0,749#0,750#0,751#0,752#0,753#0,754#0,755#0,756#0,757#0,758#0,759#0,760#0,761#0,762#0,763#0,764#0,765#0,766#0,767#0,768#0,769#0,770#0,771#0,772#0,773#0,774#0,775#0,776#0,777#0,778#0,779#0,780#0,781#0,782#0,783#0,784#0,785#0,786#0,787#0,788#0,789#0,790#0,791#0,792#0,793#0,794#0,795#0,796#0,797#0,798#0,799#0,800#0,801#0,802#0,803#0,804#0,805#0,806#0,807#0,808#0,809#0,810#0,811#0,812#0,813#0,814#0,815#0,816#0,817#0,818#0,819#0,820#0,821#0,822#0,823#0,824#0,825#0,826#0,827#0,828#0,829#0,830#0,831#0,832#0,833#0,834#0,835#0,836#0,837#0,838#0,839#0,840#0,841#0,842#0,843#0,844#0,845#0,846#0,847#0,848#0,849#0,850#0,851#0,852#0,853#0,854#0,855#0,856#0,857#0,858#0,859#0,860#0,861#0,862#0,863#0,864#0,865#0,866#0,867#0,868#0,869#0,870#0,871#0,872#0,873#0,874#0,875#0,876#0,877#0,878#0,879#0,880#0,881#0,882#0,883#0,884#0,885#0,886#0,887#0,888#0,889#0,890#0,891#0,892#0,893#0,894#0,895#0,896#0,897#0,898#0,899#0,900#0,901#0,902#0,903#0,904#0,905#0,906#0,907#0,908#0,909#0,910#0,911#0,912#0,913#0,914#0,915#0,916#0,917#0,918#0,919#0,920#0,921#0,922#0,923#0,924#0,925#0,926#0,927#0,928#0,929#0,930#0,931#0,932#0,933#0,934#0,935#0,936#0,937#0,938#0,939#0,940#0,941#0,942#0,943#0,944#0,945#0,946#0,947#0,948#0,949#0,950#0,951#0,952#0,953#0,954#0,955#0,956#0,957#0,958#0,959#0,960#0,961#0,962#0,963#0,964#0,965#0,966#0,967#0,968#0,969#0,970#0,971#0,972#0,973#0,974#0,975#0,976#0,977#0,978#0,979#0,980#0,981#0,982#0,983#0,984#0,985#0,986#0,987#0,988#0,989#0,990#0,991#0,992#0,993#0,994#0,995#0,996#0,997#0,998#0,999#0,1000#0,1001#0,1002#0,1003#0,1004#0,1005#0,1006#0,1007#0,1008#0,1009#0,1010#0,1011#0,1012#0,1013#0,1014#0,1015#0,1016#0,1017#0,1018#0,1019#0,1020#0,1021#0,1022#0,1023#0,1024#0,1025#0,1026#0,1027#0,1028#0,1029#0,1030#0,1031#0,1032#0,1033#0,1034#0,1035#0,1036#0,1037#0,1038#0,1039#0,1040#0,1041#0,1042#0,1043#0,1044#0,1045#0,1046#0,1047#0,1048#0,1049#0,1050#0,1051#0,1052#0,1053#0,1054#0,1055#0,1056#0,1057#0,1058#0,1059#0,1060#0,1061#0,1062#0,1063#0,1064#0,1065#0,1066#0,1067#0,1068#0,1069#0,1070#0,1071#0,1072#0,1073#0,1074#0,1075#0,1076#0,1077#0,1078#0,1079#0,1080#0,1081#0,1082#0,1083#0,1084#0,1085#0,1086#0,1087#0,1088#0,1089#0,1090#0,1091#0,1092#0,1093#0,1094#0,1095#0,1096#0,1097#0,1098#0,1099#0,1100#0,1101#0,1102#0,1103#0,1104#0,1105#0,1106#0,1107#0,1108#0,1109#0,1110#0,1111#0,1112#0,1113#0,1114#0,1115#0,1116#0,1117#0,1118#0,1119#0,1120#0,1121#0,1122#0,1123#0,1124#0,1125#0,1126#0,1127#0,1128#0,1129#0,1130#0,1131#0,1132#0,1133#0,1134#0,1135#0,1136#0,1137#0,1138#0,1139#0,1140#0,1141#0,1142#0,1143#0,1144#0,1145#0,1146#0,1147#0,1148#0,1149#0,1150#0,1151#0,1152#0,1153#0,1154#0,1155#0,1156#0,1157#0,1158#0,1159#0,1160#0,1161#0,1162#0,1163#0,1164#0,1165#0,1166#0,1167#0,1168#0,1169#0,1170#0,1171#0,1172#0,1173#0,1174#0,1175#0,1176#0,1177#0,1178#0,1179#0,1180#0,1181#0,1182#0,1183#0,1184#0,1185#0,1186#0,1187#0,1188#0,1189#0,1190#0,1191#0,1192#0,1193#0,1194#0,1195#0,1196#0,1197#0,1198#0,1199#0,1200#0,1201#0,1202#0,1203#0,1204#0,1205#0,1206#0,1207#0,1208#0,1209#0,1210#0,1211#0,1212#0,1213#0,1214#0,1215#0,1216#0,1217#0,1218#0,1219#0,1220#0,1221#0,1222#0,1223#0,1224#0,1225#0,1226#0,1227#0,1228#0,1229#0,1230#0,1231#0,1232#0,1233#0,1234#0,1235#0,1236#0,1237#0,1238#0,1239#0,1240#0,1241#0,1242#0,1243#0,1244#0,1245#0,1246#0,1247#0,1248#0,1249#0,1250#0,1251#0,1252#0,1253#0,1254#0,1255#0,1256#0,1257#0,1258#0,1259#0,1260#0,1261#0,1262#0,1263#0,1264#0,1265#0,1266#0,1267#0,1268#0,1269#0,1270#0,1271#0,1272#0,1273#0,1274#0,1275#0,1276#0,1277#0,1278#0,1279#0,1280#0,1281#0,1282#0,1283#0,1284#0,1285#0,1286#0,1287#0,1288#0,1289#0,1290#0,1291#0,1292#0,1293#0,1294#0,1295#0,1296#0,1297#0,1298#0,1299#0,1300#0,1301#0,1302#0,1303#0,1304#0,1305#0,1306#0,1307#0,1308#0,1309#0,1310#0,1311#0,1312#0,1313#0,1314#0,1315#0,1316#0,1317#0,1318#0,1319#0,1320#0,1321#0,1322#0,1323#0,1324#0,1325#0,1326#0,1327#0,1328#0,1329#0,1330#0,1331#0,1332#0,1333#0,1334#0,1335#0,1336#0,1337#0,1338#0,1339#0,1340#0,1341#0,1342#0,1343#0,1344#0,1345#0,1346#0,1347#0,1348#0,1349#0,1350#0,1351#0,1352#0,1353#0,1354#0,1355#0,1356#0,1357#0,1358#0,1359#0,1360#0,1361#0,1362#0,1363#0,1364#0,1365#0,1366#0,1367#0,1368#0,1369#0,1370#0,1371#0,1372#0,1373#0,1374#0,1375#0,1376#0,1377#0,1378#0,1379#0,1380#0,1381#0,1382#0,1383#0,1384#0,1385#0,1386#0,1387#0,1388#0,1389#0,1390#0,1391#0,1392#0,1393#0,1394#0,1395#0,1396#0,1397#0,1398#0,1399#0,1400#0,1401#0,1402#0,1403#0,1404#0,1405#0,1406#0,1407#0,1408#0,1409#0,1410#0,1411#0,1412#0,1413#0,1414#0,1415#0,1416#0,1417#0,1418#0,1419#0,1420#0,1421#0,1422#0,1423#0,1424#0,1425#0,1426#0,1427#0,1428#0,1429#0,1430#0,1431#0,1432#0,1433#0,1434#0,1435#0,1436#0,1437#0,1438#0,1439#0,1440#0,1441#0,1442#0,1443#0,1444#0,1445#0,1446#0,1447#0,1448#0,1449#0,1450#0,1451#0,1452#0,1453#0,1454#0,1455#0,1456#0,1457#0,1458#0,1459#0,1460#0,1461#0,1462#0,1463#0,1464#0,1465#0,1466#0,1467#0,1468#0,1469#0,1470#0,1471#0,1472#0,1473#0,1474#0,1475#0,1476#0,1477#0,1478#0,1479#0,1480#0,1481#0,1482#0,1483#0,1484#0,1485#0,1486#0,1487#0,1488#0,1489#0,1490#0,1491#0,1492#0,1493#0,1494#0,1495#0,1496#0,1497#0,1498#0,1499#0,1500#0,1501#0,1502#0,1503#0,1504#0,1505#0,1506#0,1507#0,1508#0,1509#0,1510#0,1511#0,1512#0,1513#0,1514#0,1515#0,1516#0,1517#0,1518#0,1519#0,1520#0,1521#0,1522#0,1523#0,1524#0,1525#0,1526#0,1527#0,1528#0,1529#0,1530#0,1531#0,1532#0,1533#0,1534#0,1535#0,1536#0,1537#0,1538#0,1539#0,1540#0,1541#0,1542#0,1543#0,1544#0,1545#0,1546#0,1547#0,1548#0,1549#0,1550#0,1551#0,1552#0,1553#0,1554#0,1555#0,1556#0,1557#0,1558#0,1559#0,1560#0,1561#0,1562#0,1563#0,1564#0,1565#0,1566#0,1567#0,1568#0,1569#0,1570#0,1571#0,1572#0,1573#0,1574#0,1575#0,1576#0,1577#0,1578#0,1579#0,1580#0,1581#0,1582#0,1583#0,1584#0,1585#0,1586#0,1587#0,1588#0,1589#0,1590#0,1591#0,1592#0,1593#0,1594#0,1595#0,1596#0,1597#0,1598#0,1599#0,1600#0,1601#0,1602#0,1603#0,1604#0,1605#0,1606#0,1607#0,1608#0,1609#0,1610#0,1611#0,1612#0,1613#0,1614#0,1615#0,1616#0,1617#0,1618#0,1619#0,1620#0,1621#0,1622#0,1623#0,1624#0,1625#0,1626#0,1627#0,1628#0,1629#0,1630#0,1631#0,1632#0,1633#0,1634#0,1635#0,1636#0,1637#0,1638#0,1639#0,1640#0,1641#0,1642#0,1643#0,1644#0,1645#0,1646#0,1647#0,1648#0,1649#0,1650#0,1651#0,1652#0,1653#0,1654#0,1655#0,1656#0,1657#0,1658#0,1659#0,1660#0,1661#0,1662#0,1663#0,1664#0,1665#0,1666#0,1667#0,1668#0,1669#0,1670#0,1671#0,1672#0,1673#0,1674#0,1675#0,1676#0,1677#0,1678#0,1679#0,1680#0,1681#0,1682#0,1683#0,1684#0,1685#0,1686#0,1687#0,1688#0,1689#0,1690#0,1691#0,1692#0,1693#0,1694#0,1695#0,1696#0,1697#0,1698#0,1699#0,1700#0,1701#0,1702#0,1703#0,1704#0,1705#0,1706#0,1707#0,1708#0,1709#0,1710#0,1711#0,1712#0,1713#0,1714#0,1715#0,1716#0,1717#0,1718#0,1719#0,1720#0,1721#0,1722#0,1723#0,1724#0,1725#0,1726#0,1727#0,1728#0,1729#0,1730#0,1731#0,1732#0,1733#0,1734#0,1735#0,1736#0,1737#0,1738#0,1739#0,1740#0,1741#0,1742#0,1743#0,1744#0,1745#0,1746#0,1747#0,1748#0,1749#0,1750#0,1751#0,1752#0,1753#0,1754#0,1755#0,1756#0,1757#0,1758#0,1759#0,1760#0,1761#0,1762#0,1763#0,1764#0,1765#0,1766#0,1767#0,1768#0,1769#0,1770#0,1771#0,1772#0,1773#0,1774#0,1775#0,1776#0,1777#0,1778#0,1779#0,1780#0,1781#0,1782#0,1783#0,1784#0,1785#0,1786#0,1787#0,1788#0,1789#0,1790#0,1791#0,1792#0,1793#0,1794#0,1795#0,1796#0,1797#0,1798#0,1799#0,1800#0,1801#0,1802#0,1803#0,1804#0,1805#0,1806#0,1807#0,1808#0,1809#0,1810#0,1811#0,1812#0,1813#0,1814#0,1815#0,1816#0,1817#0,1818#0,1819#0,1820#0,1821#0,1822#0,1823#0,1824#0,1825#0,1826#0,1827#0,1828#0,1829#0,1830#0,1831#0,1832#0,1833#0,1834#0,1835#0,1836#0,1837#0,1838#0,1839#0,1840#0,1841#0,1842#0,1843#0,1844#0,1845#0,1846#0,1847#0,1848#0,1849#0,1850#0,1851#0,1852#0,1853#0,1854#0,1855#0,1856#0,1857#0,1858#0,1859#0,1860#0,1861#0,1862#0,1863#0,1864#0,1865#0,1866#0,1867#0,1868#0,1869#0,1870#0,1871#0,1872#0,1873#0,1874#0,1875#0,1876#0,1877#0,1878#0,1879#0,1880#0,1881#0,1882#0,1883#0,1884#0,1885#0,1886#0,1887#0,1888#0,1889#0,1890#0,1891#0,1892#0,1893#0,1894#0,1895#0,1896#0,1897#0,1898#0,1899#0,1900#0,1901#0,1902#0,1903#0,1904#0,1905#0,1906#0,1907#0,1908#0,1909#0,1910#0,1911#0,1912#0,1913#0,1914#0,1915#0,1916#0,1917#0,1918#0,1919#0,1920#0,1921#0,1922#0,1923#0,1924#0,1925#0,1926#0,1927#0,1928#0,1929#0,1930#0,1931#0,1932#0,1933#0,1934#0,1935#0,1936#0,1937#0,1938#0,1939#0,1940#0,1941#0,1942#0,1943#0,1944#0,1945#0,1946#0,1947#0,1948#0,1949#0,1950#0,1951#0,1952#0,1953#0,1954#0,1955#0,1956#0,1957#0,1958#0,1959#0,1960#0,1961#0,1962#0,1963#0,1964#0,1965#0,1966#0,1967#0,1968#0,1969#0,1970#0,1971#0,1972#0,1973#0,1974#0,1975#0,1976#0,1977#0,1978#0,1979#0,1980#0,1981#0,1982#0,1983#0,1984#0,1985#0,1986#0,1987#0,1988#0,1989#0,1990#0,1991#0,1992#0,1993#0,1994#0,1995#0,1996#0,1997#0,1998#0", "mtime": "2023-07-11T17:16:21.434940Z", "creation_time": "2022-06-14T23:31:53.669280Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#,128#,129#,130#,131#,132#,133#,134#,135#,136#,137#,138#,139#,140#,141#,142#,143#,144#,145#,146#,147#,148#,149#,150#,151#,152#,153#,154#,155#,156#,157#,158#,159#,160#,161#,162#,163#,164#,165#,166#,167#,168#,169#,170#,171#,172#,173#,174#,175#,176#,177#,178#,179#,180#,181#,182#,183#,184#,185#,186#,187#,188#,189#,190#,191#,192#,193#,194#,195#,196#,197#,198#,199#,200#,201#,202#,203#,204#,205#,206#,207#,208#,209#,210#,211#,212#,213#,214#,215#,216#,217#,218#,219#,220#,221#,222#,223#,224#,225#,226#,227#,228#,229#,230#,231#,232#,233#,234#,235#,236#,237#,238#,239#,240#,241#,242#,243#,244#,245#,246#,247#,248#,249#,250#,251#,252#,253#,254#,255#,256#,257#,258#,259#,260#,261#,262#,263#,264#,265#,266#,267#,268#,269#,270#,271#,272#,273#,274#,275#,276#,277#,278#,279#,280#,281#,282#,283#,284#,285#,286#,287#,288#,289#,290#,291#,292#,293#,294#,295#,296#,297#,298#,299#,300#,301#,302#,303#,304#,305#,306#,307#,308#,309#,310#,311#,312#,313#,314#,315#,316#,317#,318#,319#,320#,321#,322#,323#,324#,325#,326#,327#,328#,329#,330#,331#,332#,333#,334#,335#,336#,337#,338#,339#,340#,341#,342#,343#,344#,345#,346#,347#,348#,349#,350#,351#,352#,353#,354#,355#,356#,357#,358#,359#,360#,361#,362#,363#,364#,365#,366#,367#,368#,369#,370#,371#,372#,373#,374#,375#,376#,377#,378#,379#,380#,381#,382#,383#,384#,385#,386#,387#,388#,389#,390#,391#,392#,393#,394#,395#,396#,397#,398#,399#,400#,401#,402#,403#,404#,405#,406#,407#,408#,409#,410#,411#,412#,413#,414#,415#,416#,417#,418#,419#,420#,421#,422#,423#,424#,425#,426#,427#,428#,429#,430#,431#,432#,433#,434#,435#,436#,437#,438#,439#,440#,441#,442#,443#,444#,445#,446#,447#,448#,449#,450#,451#,452#,453#,454#,455#,456#,457#,458#,459#,460#,461#,462#,463#,464#,465#,466#,467#,468#,469#,470#,471#,472#,473#,474#,475#,476#,477#,478#,479#,480#,481#,482#,483#,484#,485#,486#,487#,488#,489#,490#,491#,492#,493#,494#,495#,496#,497#,498#,499#,500#,501#,502#,503#,504#,505#,506#,507#,508#,509#,510#,511#,512#,513#,514#,515#,516#,517#,518#,519#,520#,521#,522#,523#,524#,525#,526#,527#,528#,529#,530#,531#,532#,533#,534#,535#,536#,537#,538#,539#,540#,541#,542#,543#,544#,545#,546#,547#,548#,549#,550#,551#,552#,553#,554#,555#,556#,557#,558#,559#,560#,561#,562#,563#,564#,565#,566#,567#,568#,569#,570#,571#,572#,573#,574#,575#,576#,577#,578#,579#,580#,581#,582#,583#,584#,585#,586#,587#,588#,589#,590#,591#,592#,593#,594#,595#,596#,597#,598#,599#,600#,601#,602#,603#,604#,605#,606#,607#,608#,609#,610#,611#,612#,613#,614#,615#,616#,617#,618#,619#,620#,621#,622#,623#,624#,625#,626#,627#,628#,629#,630#,631#,632#,633#,634#,635#,636#,637#,638#,639#,640#,641#,642#,643#,644#,645#,646#,647#,648#,649#,650#,651#,652#,653#,654#,655#,656#,657#,658#,659#,660#,661#,662#,663#,664#,665#,666#,667#,668#,669#,670#,671#,672#,673#,674#,675#,676#,677#,678#,679#,680#,681#,682#,683#,684#,685#,686#,687#,688#,689#,690#,691#,692#,693#,694#,695#,696#,697#,698#,699#,700#,701#,702#,703#,704#,705#,706#,707#,708#,709#,710#,711#,712#,713#,714#,715#,716#,717#,718#,719#,720#,721#,722#,723#,724#,725#,726#,727#,728#,729#,730#,731#,732#,733#,734#,735#,736#,737#,738#,739#,740#,741#,742#,743#,744#,745#,746#,747#,748#,749#,750#,751#,752#,753#,754#,755#,756#,757#,758#,759#,760#,761#,762#,763#,764#,765#,766#,767#,768#,769#,770#,771#,772#,773#,774#,775#,776#,777#,778#,779#,780#,781#,782#,783#,784#,785#,786#,787#,788#,789#,790#,791#,792#,793#,794#,795#,796#,797#,798#,799#,800#,801#,802#,803#,804#,805#,806#,807#,808#,809#,810#,811#,812#,813#,814#,815#,816#,817#,818#,819#,820#,821#,822#,823#,824#,825#,826#,827#,828#,829#,830#,831#,832#,833#,834#,835#,836#,837#,838#,839#,840#,841#,842#,843#,844#,845#,846#,847#,848#,849#,850#,851#,852#,853#,854#,855#,856#,857#,858#,859#,860#,861#,862#,863#,864#,865#,866#,867#,868#,869#,870#,871#,872#,873#,874#,875#,876#,877#,878#,879#,880#,881#,882#,883#,884#,885#,886#,887#,888#,889#,890#,891#,892#,893#,894#,895#,896#,897#,898#,899#,900#,901#,902#,903#,904#,905#,906#,907#,908#,909#,910#,911#,912#,913#,914#,915#,916#,917#,918#,919#,920#,921#,922#,923#,924#,925#,926#,927#,928#,929#,930#,931#,932#,933#,934#,935#,936#,937#,938#,939#,940#,941#,942#,943#,944#,945#,946#,947#,948#,949#,950#,951#,952#,953#,954#,955#,956#,957#,958#,959#,960#,961#,962#,963#,964#,965#,966#,967#,968#,969#,970#,971#,972#,973#,974#,975#,976#,977#,978#,979#,980#,981#,982#,983#,984#,985#,986#,987#,988#,989#,990#,991#,992#,993#,994#,995#,996#,997#,998#,999#,1000#,1001#,1002#,1003#,1004#,1005#,1006#,1007#,1008#,1009#,1010#,1011#,1012#,1013#,1014#,1015#,1016#,1017#,1018#,1019#,1020#,1021#,1022#,1023#,1024#,1025#,1026#,1027#,1028#,1029#,1030#,1031#,1032#,1033#,1034#,1035#,1036#,1037#,1038#,1039#,1040#,1041#,1042#,1043#,1044#,1045#,1046#,1047#,1048#,1049#,1050#,1051#,1052#,1053#,1054#,1055#,1056#,1057#,1058#,1059#,1060#,1061#,1062#,1063#,1064#,1065#,1066#,1067#,1068#,1069#,1070#,1071#,1072#,1073#,1074#,1075#,1076#,1077#,1078#,1079#,1080#,1081#,1082#,1083#,1084#,1085#,1086#,1087#,1088#,1089#,1090#,1091#,1092#,1093#,1094#,1095#,1096#,1097#,1098#,1099#,1100#,1101#,1102#,1103#,1104#,1105#,1106#,1107#,1108#,1109#,1110#,1111#,1112#,1113#,1114#,1115#,1116#,1117#,1118#,1119#,1120#,1121#,1122#,1123#,1124#,1125#,1126#,1127#,1128#,1129#,1130#,1131#,1132#,1133#,1134#,1135#,1136#,1137#,1138#,1139#,1140#,1141#,1142#,1143#,1144#,1145#,1146#,1147#,1148#,1149#,1150#,1151#,1152#,1153#,1154#,1155#,1156#,1157#,1158#,1159#,1160#,1161#,1162#,1163#,1164#,1165#,1166#,1167#,1168#,1169#,1170#,1171#,1172#,1173#,1174#,1175#,1176#,1177#,1178#,1179#,1180#,1181#,1182#,1183#,1184#,1185#,1186#,1187#,1188#,1189#,1190#,1191#,1192#,1193#,1194#,1195#,1196#,1197#,1198#,1199#,1200#,1201#,1202#,1203#,1204#,1205#,1206#,1207#,1208#,1209#,1210#,1211#,1212#,1213#,1214#,1215#,1216#,1217#,1218#,1219#,1220#,1221#,1222#,1223#,1224#,1225#,1226#,1227#,1228#,1229#,1230#,1231#,1232#,1233#,1234#,1235#,1236#,1237#,1238#,1239#,1240#,1241#,1242#,1243#,1244#,1245#,1246#,1247#,1248#,1249#,1250#,1251#,1252#,1253#,1254#,1255#,1256#,1257#,1258#,1259#,1260#,1261#,1262#,1263#,1264#,1265#,1266#,1267#,1268#,1269#,1270#,1271#,1272#,1273#,1274#,1275#,1276#,1277#,1278#,1279#,1280#,1281#,1282#,1283#,1284#,1285#,1286#,1287#,1288#,1289#,1290#,1291#,1292#,1293#,1294#,1295#,1296#,1297#,1298#,1299#,1300#,1301#,1302#,1303#,1304#,1305#,1306#,1307#,1308#,1309#,1310#,1311#,1312#,1313#,1314#,1315#,1316#,1317#,1318#,1319#,1320#,1321#,1322#,1323#,1324#,1325#,1326#,1327#,1328#,1329#,1330#,1331#,1332#,1333#,1334#,1335#,1336#,1337#,1338#,1339#,1340#,1341#,1342#,1343#,1344#,1345#,1346#,1347#,1348#,1349#,1350#,1351#,1352#,1353#,1354#,1355#,1356#,1357#,1358#,1359#,1360#,1361#,1362#,1363#,1364#,1365#,1366#,1367#,1368#,1369#,1370#,1371#,1372#,1373#,1374#,1375#,1376#,1377#,1378#,1379#,1380#,1381#,1382#,1383#,1384#,1385#,1386#,1387#,1388#,1389#,1390#,1391#,1392#,1393#,1394#,1395#,1396#,1397#,1398#,1399#,1400#,1401#,1402#,1403#,1404#,1405#,1406#,1407#,1408#,1409#,1410#,1411#,1412#,1413#,1414#,1415#,1416#,1417#,1418#,1419#,1420#,1421#,1422#,1423#,1424#,1425#,1426#,1427#,1428#,1429#,1430#,1431#,1432#,1433#,1434#,1435#,1436#,1437#,1438#,1439#,1440#,1441#,1442#,1443#,1444#,1445#,1446#,1447#,1448#,1449#,1450#,1451#,1452#,1453#,1454#,1455#,1456#,1457#,1458#,1459#,1460#,1461#,1462#,1463#,1464#,1465#,1466#,1467#,1468#,1469#,1470#,1471#,1472#,1473#,1474#,1475#,1476#,1477#,1478#,1479#,1480#,1481#,1482#,1483#,1484#,1485#,1486#,1487#,1488#,1489#,1490#,1491#,1492#,1493#,1494#,1495#,1496#,1497#,1498#,1499#,1500#,1501#,1502#,1503#,1504#,1505#,1506#,1507#,1508#,1509#,1510#,1511#,1512#,1513#,1514#,1515#,1516#,1517#,1518#,1519#,1520#,1521#,1522#,1523#,1524#,1525#,1526#,1527#,1528#,1529#,1530#,1531#,1532#,1533#,1534#,1535#,1536#,1537#,1538#,1539#,1540#,1541#,1542#,1543#,1544#,1545#,1546#,1547#,1548#,1549#,1550#,1551#,1552#,1553#,1554#,1555#,1556#,1557#,1558#,1559#,1560#,1561#,1562#,1563#,1564#,1565#,1566#,1567#,1568#,1569#,1570#,1571#,1572#,1573#,1574#,1575#,1576#,1577#,1578#,1579#,1580#,1581#,1582#,1583#,1584#,1585#,1586#,1587#,1588#,1589#,1590#,1591#,1592#,1593#,1594#,1595#,1596#,1597#,1598#,1599#,1600#,1601#,1602#,1603#,1604#,1605#,1606#,1607#,1608#,1609#,1610#,1611#,1612#,1613#,1614#,1615#,1616#,1617#,1618#,1619#,1620#,1621#,1622#,1623#,1624#,1625#,1626#,1627#,1628#,1629#,1630#,1631#,1632#,1633#,1634#,1635#,1636#,1637#,1638#,1639#,1640#,1641#,1642#,1643#,1644#,1645#,1646#,1647#,1648#,1649#,1650#,1651#,1652#,1653#,1654#,1655#,1656#,1657#,1658#,1659#,1660#,1661#,1662#,1663#,1664#,1665#,1666#,1667#,1668#,1669#,1670#,1671#,1672#,1673#,1674#,1675#,1676#,1677#,1678#,1679#,1680#,1681#,1682#,1683#,1684#,1685#,1686#,1687#,1688#,1689#,1690#,1691#,1692#,1693#,1694#,1695#,1696#,1697#,1698#,1699#,1700#,1701#,1702#,1703#,1704#,1705#,1706#,1707#,1708#,1709#,1710#,1711#,1712#,1713#,1714#,1715#,1716#,1717#,1718#,1719#,1720#,1721#,1722#,1723#,1724#,1725#,1726#,1727#,1728#,1729#,1730#,1731#,1732#,1733#,1734#,1735#,1736#,1737#,1738#,1739#,1740#,1741#,1742#,1743#,1744#,1745#,1746#,1747#,1748#,1749#,1750#,1751#,1752#,1753#,1754#,1755#,1756#,1757#,1758#,1759#,1760#,1761#,1762#,1763#,1764#,1765#,1766#,1767#,1768#,1769#,1770#,1771#,1772#,1773#,1774#,1775#,1776#,1777#,1778#,1779#,1780#,1781#,1782#,1783#,1784#,1785#,1786#,1787#,1788#,1789#,1790#,1791#,1792#,1793#,1794#,1795#,1796#,1797#,1798#,1799#,1800#,1801#,1802#,1803#,1804#,1805#,1806#,1807#,1808#,1809#,1810#,1811#,1812#,1813#,1814#,1815#,1816#,1817#,1818#,1819#,1820#,1821#,1822#,1823#,1824#,1825#,1826#,1827#,1828#,1829#,1830#,1831#,1832#,1833#,1834#,1835#,1836#,1837#,1838#,1839#,1840#,1841#,1842#,1843#,1844#,1845#,1846#,1847#,1848#,1849#,1850#,1851#,1852#,1853#,1854#,1855#,1856#,1857#,1858#,1859#,1860#,1861#,1862#,1863#,1864#,1865#,1866#,1867#,1868#,1869#,1870#,1871#,1872#,1873#,1874#,1875#,1876#,1877#,1878#,1879#,1880#,1881#,1882#,1883#,1884#,1885#,1886#,1887#,1888#,1889#,1890#,1891#,1892#,1893#,1894#,1895#,1896#,1897#,1898#,1899#,1900#,1901#,1902#,1903#,1904#,1905#,1906#,1907#,1908#,1909#,1910#,1911#,1912#,1913#,1914#,1915#,1916#,1917#,1918#,1919#,1920#,1921#,1922#,1923#,1924#,1925#,1926#,1927#,1928#,1929#,1930#,1931#,1932#,1933#,1934#,1935#,1936#,1937#,1938#,1939#,1940#,1941#,1942#,1943#,1944#,1945#,1946#,1947#,1948#,1949#,1950#,1951#,1952#,1953#,1954#,1955#,1956#,1957#,1958#,1959#,1960#,1961#,1962#,1963#,1964#,1965#,1966#,1967#,1968#,1969#,1970#,1971#,1972#,1973#,1974#,1975#,1976#,1977#,1978#,1979#,1980#,1981#,1982#,1983#,1984#,1985#,1986#,1987#,1988#,1989#,1990#,1991#,1992#,1993#,1994#,1995#,1996#,1997#,1998#", "usage": { "rgw.main": { "size": 209866840, "size_actual": 209903616, "size_utilized": 209866840, "size_kb": 204949, "size_kb_actual": 204984, "size_kb_utilized": 204949, "num_objects": 169433938 }, "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 837, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 1, "num_objects": 31 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } I am not sure what those objects are and why they have so many shards, since actual readable objects are less than 10 . Any help with that is really appreciated.

2 2

Blank dashboard
by Curt 31 Jul '23

31 Jul '23

Hello, This is a strange one for me. My ceph dashboard just stopped loading, nothing but a white page. I don't see anything in the logs and on browser side the only error I see is Failed to load resource: net::ERR_CONTENT_LENGTH_MISMATCH in chrome and Uncaught SyntaxError: expected expression, got end of script for Firefox on the file main.ddd4de0999172734.js. Nothing in the logs files, put mgr up to 20 on the log level. Any suggestions? Thanks, Curt

2 2

[rbd-mirror] can't enable journal-based image mirroring
by Tony Liu 31 Jul '23

31 Jul '23

Hi, The Ceph cluster is with Pacific v16.2.10. "rbd mirror image enable <image> journal" seems not working. Any clues what I'm missing? There is no error messages from the CLI. Any way to troubleshooting? ``` # rbd mirror pool info volume-ssd Mode: image Site Name: 35d050c0-77c0-11eb-9242-2cea7ff9d07c Peer Sites: UUID: 86eacc0f-6657-4742-8daf-2942ea23affd Name: qa Mirror UUID: Direction: rx-tx Client: client.infra # rbd feature enable volume-ssd/volume-aceee005-265e-44ea-a591-b6dda639a76b journaling # rbd mirror image enable volume-ssd/volume-aceee005-265e-44ea-a591-b6dda639a76b journal # rbd info volume-ssd/volume-aceee005-265e-44ea-a591-b6dda639a76b rbd image 'volume-aceee005-265e-44ea-a591-b6dda639a76b': size 40 GiB in 10240 objects order 22 (4 MiB objects) snapshot_count: 0 id: e6fce674350e55 block_name_prefix: rbd_data.e6fce674350e55 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling op_features: flags: create_timestamp: Sun Jul 30 22:21:09 2023 access_timestamp: Mon Jul 31 00:36:20 2023 modify_timestamp: Mon Jul 31 08:01:26 2023 parent: image/801d1850-de2f-443b-a30c-71966e90c118@snap overlap: 10 GiB journal: e6fce674350e55 mirroring state: disabled # rbd mirror image status volume-ssd/volume-aceee005-265e-44ea-a591-b6dda639a76b rbd: mirroring not enabled on the image ``` Thanks! Tony

1 1

Some Ceph OSD metrics are zero
by GOSSET, Alexandre 31 Jul '23

31 Jul '23

Hello, We have a cluster in version 17.2.5 and on 2 nodes (out of 5), some metrics are at 0 on all OSD. The problem is really global to the node. I performed an upgrade to 17.2.6 on the cluster like this : * upgrade all monitors * upgrade all mgr * then reboot each node one by one (there was a pending upgrade of linux kernel image) to upgrade all OSDs This fixed for one node of the cluster but not the other so I stil don't understand. Here is an example of such OSD : { "AsyncMessenger::Worker-0": { "msgr_recv_messages": 6843132, "msgr_send_messages": 6880681, "msgr_recv_bytes": 63926805259, "msgr_send_bytes": 3298578019, "msgr_created_connections": 354, "msgr_active_connections": 20, "msgr_running_total_time": 361.495252541, "msgr_running_send_time": 88.204996929, "msgr_running_recv_time": 157.610653554, "msgr_running_fast_dispatch_time": 94.944377551, "msgr_send_messages_queue_lat": { "avgcount": 6880679, "sum": 398.607948692, "avgtime": 0.000057931 }, "msgr_handle_ack_lat": { "avgcount": 8509969, "sum": 1.558485881, "avgtime": 0.000000183 } }, "AsyncMessenger::Worker-1": { "msgr_recv_messages": 2332821, "msgr_send_messages": 2357957, "msgr_recv_bytes": 17554824362, "msgr_send_bytes": 3407896999, "msgr_created_connections": 177, "msgr_active_connections": 18, "msgr_running_total_time": 112.862256919, "msgr_running_send_time": 34.874196263, "msgr_running_recv_time": 46.995625507, "msgr_running_fast_dispatch_time": 25.082966322, "msgr_send_messages_queue_lat": { "avgcount": 2357955, "sum": 85.725096545, "avgtime": 0.000036355 }, "msgr_handle_ack_lat": { "avgcount": 1140003, "sum": 0.241074745, "avgtime": 0.000000211 } }, "AsyncMessenger::Worker-2": { "msgr_recv_messages": 4326174, "msgr_send_messages": 4330697, "msgr_recv_bytes": 31752213275, "msgr_send_bytes": 3567666813, "msgr_created_connections": 100, "msgr_active_connections": 20, "msgr_running_total_time": 218.260025879, "msgr_running_send_time": 58.635931148, "msgr_running_recv_time": 92.152198211, "msgr_running_fast_dispatch_time": 54.438590557, "msgr_send_messages_queue_lat": { "avgcount": 4330696, "sum": 250.493265066, "avgtime": 0.000057841 }, "msgr_handle_ack_lat": { "avgcount": 4229647, "sum": 0.806930547, "avgtime": 0.000000190 } }, "bluefs": { "db_total_bytes": 1920362086400, "db_used_bytes": 1089011712, "wal_total_bytes": 0, "wal_used_bytes": 0, "slow_total_bytes": 0, "slow_used_bytes": 0, "num_files": 125, "log_bytes": 2646016, "log_compactions": 812, "logged_bytes": 13500940288, "files_written_wal": 1575, "files_written_sst": 182, "bytes_written_wal": 25542598656, "bytes_written_sst": 2466263040, "bytes_written_slow": 0, "max_bytes_wal": 0, "max_bytes_db": 2035351552, "max_bytes_slow": 0, "alloc_unit_main": 0, "alloc_unit_db": 65536, "alloc_unit_wal": 0, "read_random_count": 201339, "read_random_bytes": 3080889493, "read_random_disk_count": 101652, "read_random_disk_bytes": 2645938201, "read_random_disk_bytes_wal": 0, "read_random_disk_bytes_db": 2645938201, "read_random_disk_bytes_slow": 0, "read_random_buffer_count": 110722, "read_random_buffer_bytes": 434951292, "read_count": 80250, "read_bytes": 1159472505, "read_disk_count": 68083, "read_disk_bytes": 1053351936, "read_disk_bytes_wal": 0, "read_disk_bytes_db": 1053356032, "read_disk_bytes_slow": 0, "read_prefetch_count": 78665, "read_prefetch_bytes": 1108068939, "compact_lat": { "avgcount": 812, "sum": 1.246095373, "avgtime": 0.001534600 }, "compact_lock_lat": { "avgcount": 812, "sum": 0.466894228, "avgtime": 0.000574992 }, "alloc_slow_fallback": 0, "alloc_slow_size_fallback": 0, "read_zeros_candidate": 0, "read_zeros_errors": 0 }, "bluestore": { "allocated": 180780802048, "stored": 188948817556, "fragmentation_micros": 2, "alloc_unit": 4096, "state_prepare_lat": { "avgcount": 3979669, "sum": 426.352439741, "avgtime": 0.000107132 }, "state_aio_wait_lat": { "avgcount": 3979669, "sum": 662.940570063, "avgtime": 0.000166581 }, "state_io_done_lat": { "avgcount": 3979669, "sum": 7.323425924, "avgtime": 0.000001840 }, "state_kv_queued_lat": { "avgcount": 3979669, "sum": 1900.856846444, "avgtime": 0.000477641 }, "state_kv_commiting_lat": { "avgcount": 3979669, "sum": 3192.442512062, "avgtime": 0.000802187 }, "state_kv_done_lat": { "avgcount": 3979669, "sum": 1.143240407, "avgtime": 0.000000287 }, "state_finishing_lat": { "avgcount": 3979665, "sum": 0.363208449, "avgtime": 0.000000091 }, "state_done_lat": { "avgcount": 3979658, "sum": 123255.522846135, "avgtime": 0.030971385 }, "state_deferred_queued_lat": { "avgcount": 356134, "sum": 455582.851504787, "avgtime": 1.279245597 }, "state_deferred_aio_wait_lat": { "avgcount": 356134, "sum": 64.258012775, "avgtime": 0.000180432 }, "state_deferred_cleanup_lat": { "avgcount": 356134, "sum": 58342.990966896, "avgtime": 0.163823142 }, "txc_commit_lat": { "avgcount": 3979669, "sum": 6190.519490871, "avgtime": 0.001555536 }, "txc_throttle_lat": { "avgcount": 3979669, "sum": 61.112847717, "avgtime": 0.000015356 }, "txc_submit_lat": { "avgcount": 3979669, "sum": 495.416650342, "avgtime": 0.000124486 }, "txc_count": 3979669, "read_onode_meta_lat": { "avgcount": 1197781, "sum": 19.722806609, "avgtime": 0.000016466 }, "read_wait_aio_lat": { "avgcount": 929167, "sum": 537.061469380, "avgtime": 0.000578003 }, "csum_lat": { "avgcount": 2133026, "sum": 13.996152575, "avgtime": 0.000006561 }, "read_eio": 0, "reads_with_retries": 0, "read_lat": { "avgcount": 268614, "sum": 520.246577964, "avgtime": 0.001936781 }, "kv_flush_lat": { "avgcount": 3291697, "sum": 403.833685777, "avgtime": 0.000122682 }, "kv_commit_lat": { "avgcount": 3291697, "sum": 2311.502280053, "avgtime": 0.000702222 }, "kv_sync_lat": { "avgcount": 3291697, "sum": 2715.335965830, "avgtime": 0.000824904 }, "kv_final_lat": { "avgcount": 3291667, "sum": 65.461458609, "avgtime": 0.000019887 }, "write_big": 3712142, "write_big_bytes": 94437191680, "write_big_blobs": 4685989, "write_big_deferred": 0, "write_small": 584115, "write_small_bytes": 960510935, "write_small_unused": 13820, "write_small_pre_read": 567335, "write_pad_bytes": 217785305, "write_penalty_read_ops": 688111, "write_new": 4702769, "issued_deferred_writes": 567335, "issued_deferred_write_bytes": 2323804160, "submitted_deferred_writes": 251455, "submitted_deferred_write_bytes": 1235841024, "write_big_skipped_blobs": 0, "write_big_skipped_bytes": 0, "write_small_skipped": 0, "write_small_skipped_bytes": 0, "compressed": 0, "compressed_allocated": 0, "compressed_original": 0, "compress_lat": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "decompress_lat": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "compress_success_count": 0, "compress_rejected_count": 0, "onodes": 42417, "onodes_pinned": 0, "onode_hits": 8904311, "onode_misses": 87317, "onode_shard_hits": 6533806, "onode_shard_misses": 270931, "onode_extents": 1288567, "onode_blobs": 1116911, "buffers": 956, "buffer_bytes": 4562944, "buffer_hit_bytes": 652455498, "buffer_miss_bytes": 115020768720, "onode_reshard": 54721, "blob_split": 1236, "extent_compress": 4199105, "gc_merged": 0, "omap_seek_to_first_lat": { "avgcount": 31054, "sum": 1.622677452, "avgtime": 0.000052253 }, "omap_upper_bound_lat": { "avgcount": 4, "sum": 0.016788155, "avgtime": 0.004197038 }, "omap_lower_bound_lat": { "avgcount": 82, "sum": 0.001473844, "avgtime": 0.000017973 }, "omap_next_lat": { "avgcount": 791854, "sum": 2.400923607, "avgtime": 0.000003032 }, "omap_get_keys_lat": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "omap_get_values_lat": { "avgcount": 26905, "sum": 4.060382074, "avgtime": 0.000150915 }, "omap_clear_lat": { "avgcount": 3, "sum": 0.001249818, "avgtime": 0.000416606 }, "clist_lat": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "remove_lat": { "avgcount": 35724, "sum": 28.130138084, "avgtime": 0.000787429 }, "truncate_lat": { "avgcount": 6300, "sum": 0.361881579, "avgtime": 0.000057441 } }, "bluestore-pricache": { "target_bytes": 4294967296, "mapped_bytes": 2529673216, "unmapped_bytes": 230645760, "heap_bytes": 2760318976, "cache_bytes": 2845415832 }, "bluestore-pricache:data": { "pri0_bytes": 0, "pri1_bytes": 0, "pri2_bytes": 0, "pri3_bytes": 0, "pri4_bytes": 24576, "pri5_bytes": 57344, "pri6_bytes": 81920, "pri7_bytes": 0, "pri8_bytes": 0, "pri9_bytes": 0, "pri10_bytes": 0, "pri11_bytes": 77126679, "reserved_bytes": 73704425, "committed_bytes": 150994944 }, "bluestore-pricache:kv": { "pri0_bytes": 67108864, "pri1_bytes": 0, "pri2_bytes": 0, "pri3_bytes": 4375736, "pri4_bytes": 0, "pri5_bytes": 112032, "pri6_bytes": 0, "pri7_bytes": 0, "pri8_bytes": 0, "pri9_bytes": 0, "pri10_bytes": 0, "pri11_bytes": 578450095, "reserved_bytes": 71373561, "committed_bytes": 721420288 }, "bluestore-pricache:kv_onode": { "pri0_bytes": 67108864, "pri1_bytes": 0, "pri2_bytes": 0, "pri3_bytes": 0, "pri4_bytes": 0, "pri5_bytes": 0, "pri6_bytes": 0, "pri7_bytes": 0, "pri8_bytes": 0, "pri9_bytes": 0, "pri10_bytes": 5099312, "pri11_bytes": 51417786, "reserved_bytes": 77700630, "committed_bytes": 201326592 }, "bluestore-pricache:meta": { "pri0_bytes": 0, "pri1_bytes": 90067, "pri2_bytes": 90067, "pri3_bytes": 1200904, "pri4_bytes": 1486119, "pri5_bytes": 3047296, "pri6_bytes": 3632737, "pri7_bytes": 0, "pri8_bytes": 0, "pri9_bytes": 0, "pri10_bytes": 0, "pri11_bytes": 578450095, "reserved_bytes": 83091355, "committed_bytes": 671088640 }, "cct": { "total_workers": 16, "unhealthy_workers": 0 }, "finisher-commit_finisher": { "queue_len": 0, "complete_latency": { "avgcount": 286, "sum": 0.262805597, "avgtime": 0.000918900 } }, "finisher-objecter-finisher-0": { "queue_len": 0, "complete_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluestore_alloc_bytes": 125943744, "bluestore_alloc_items": 8193228, "bluestore_cache_data_bytes": 1884109, "bluestore_cache_data_items": 829, "bluestore_cache_onode_bytes": 28842880, "bluestore_cache_onode_items": 42416, "bluestore_cache_meta_bytes": 50594253, "bluestore_cache_meta_items": 5315174, "bluestore_cache_other_bytes": 229414948, "bluestore_cache_other_items": 23631685, "bluestore_Buffer_bytes": 108416, "bluestore_Buffer_items": 968, "bluestore_Extent_bytes": 61847760, "bluestore_Extent_items": 1288495, "bluestore_Blob_bytes": 140450128, "bluestore_Blob_items": 1254019, "bluestore_SharedBlob_bytes": 125093808, "bluestore_SharedBlob_items": 1116909, "bluestore_inline_bl_bytes": 369130, "bluestore_inline_bl_items": 725, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_txc_bytes": 5888, "bluestore_txc_items": 8, "bluestore_writing_deferred_bytes": 3317593, "bluestore_writing_deferred_items": 794, "bluestore_writing_bytes": 380783, "bluestore_writing_items": 47, "bluefs_bytes": 114472, "bluefs_items": 6282, "bluefs_file_reader_bytes": 705280, "bluefs_file_reader_items": 72, "bluefs_file_writer_bytes": 768, "bluefs_file_writer_items": 4, "buffer_anon_bytes": 13417560, "buffer_anon_items": 53741, "buffer_meta_bytes": 3034064, "buffer_meta_items": 34478, "osd_bytes": 1998496, "osd_items": 173, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 205086276, "osd_pglog_items": 814012, "osdmap_bytes": 515864, "osdmap_items": 11775, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "mds_co_bytes": 0, "mds_co_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "op_active": 0, "op_laggy": 0, "op_send": 0, "op_send_bytes": 0, "op_resend": 0, "op_reply": 0, "oplen_avg": { "avgcount": 0, "sum": 0 }, "op": 0, "op_r": 0, "op_w": 0, "op_rmw": 0, "op_pg": 0, "osdop_stat": 0, "osdop_create": 0, "osdop_read": 0, "osdop_write": 0, "osdop_writefull": 0, "osdop_writesame": 0, "osdop_append": 0, "osdop_zero": 0, "osdop_truncate": 0, "osdop_delete": 0, "osdop_mapext": 0, "osdop_sparse_read": 0, "osdop_clonerange": 0, "osdop_getxattr": 0, "osdop_setxattr": 0, "osdop_cmpxattr": 0, "osdop_rmxattr": 0, "osdop_resetxattrs": 0, "osdop_call": 0, "osdop_watch": 0, "osdop_notify": 0, "osdop_src_cmpxattr": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_other": 0, "linger_active": 0, "linger_send": 0, "linger_resend": 0, "linger_ping": 0, "poolop_active": 0, "poolop_send": 0, "poolop_resend": 0, "poolstat_active": 0, "poolstat_send": 0, "poolstat_resend": 0, "statfs_active": 0, "statfs_send": 0, "statfs_resend": 0, "command_active": 0, "command_send": 0, "command_resend": 0, "map_epoch": 12426, "map_full": 1, "map_inc": 287, "osd_sessions": 0, "osd_session_open": 0, "osd_session_close": 0, "osd_laggy": 0, "omap_wr": 0, "omap_rd": 0, "omap_del": 0 }, "osd": { "op_wip": 0, "op": 0, "op_in_bytes": 0, "op_out_bytes": 0, "op_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_process_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_prepare_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_r": 0, "op_r_out_bytes": 0, "op_r_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_r_process_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_r_prepare_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_w": 0, "op_w_in_bytes": 0, "op_w_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_w_process_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_w_prepare_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_rw": 0, "op_rw_in_bytes": 0, "op_rw_out_bytes": 0, "op_rw_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_rw_process_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_rw_prepare_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_before_queue_op_lat": { "avgcount": 3965535, "sum": 333.780003675, "avgtime": 0.000084170 }, "op_before_dequeue_op_lat": { "avgcount": 3965535, "sum": 434.995411447, "avgtime": 0.000109694 }, "subop": 3944307, "subop_in_bytes": 98738982548, "subop_latency": { "avgcount": 3944307, "sum": 6756.175478507, "avgtime": 0.001712892 }, "subop_w": 3944307, "subop_w_in_bytes": 98738982548, "subop_w_latency": { "avgcount": 3944307, "sum": 6756.175478507, "avgtime": 0.001712892 }, "subop_pull": 0, "subop_pull_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "subop_push": 0, "subop_push_in_bytes": 0, "subop_push_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "pull": 0, "push": 0, "push_out_bytes": 0, "recovery_ops": 0, "recovery_bytes": 0, "loadavg": 40, "cached_crc": 0, "cached_crc_adjusted": 0, "missed_crc": 0, "numpg": 173, "numpg_primary": 0, "numpg_replica": 173, "numpg_stray": 0, "numpg_removing": 0, "heartbeat_to_peers": 10, "map_messages": 2085, "map_message_epochs": 2223, "map_message_epoch_dups": 1935, "messages_delayed_for_map": 0, "osd_map_cache_hit": 50571, "osd_map_cache_miss": 1, "osd_map_cache_miss_low": 0, "osd_map_cache_miss_low_avg": { "avgcount": 0, "sum": 0 }, "osd_map_bl_cache_hit": 39, "osd_map_bl_cache_miss": 317, "stat_bytes": 1920362086400, "stat_bytes_used": 181869830144, "stat_bytes_avail": 1738492256256, "copyfrom": 0, "tier_promote": 0, "tier_flush": 0, "tier_flush_fail": 0, "tier_try_flush": 0, "tier_try_flush_fail": 0, "tier_evict": 0, "tier_whiteout": 0, "tier_dirty": 0, "tier_clean": 0, "tier_delay": 0, "tier_proxy_read": 0, "tier_proxy_write": 0, "agent_wake": 0, "agent_skip": 0, "agent_flush": 0, "agent_evict": 0, "object_ctx_cache_hit": 0, "object_ctx_cache_total": 0, "op_cache_hit": 0, "osd_tier_flush_lat": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "osd_tier_promote_lat": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "osd_tier_r_lat": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "osd_pg_info": 3979232, "osd_pg_fastinfo": 3825183, "osd_pg_biginfo": 33887 }, "recoverystate_perf": { "initial_latency": { "avgcount": 173, "sum": 6.060385414, "avgtime": 0.035031129 }, "started_latency": { "avgcount": 173, "sum": 35.126913452, "avgtime": 0.203045742 }, "reset_latency": { "avgcount": 346, "sum": 696.523139606, "avgtime": 2.013072657 }, "start_latency": { "avgcount": 346, "sum": 0.000675707, "avgtime": 0.000001952 }, "primary_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "peering_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "backfilling_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "waitremotebackfillreserved_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "waitlocalbackfillreserved_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "notbackfilling_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "repnotrecovering_latency": { "avgcount": 88, "sum": 14.954158341, "avgtime": 0.169933617 }, "repwaitrecoveryreserved_latency": { "avgcount": 88, "sum": 43.849329328, "avgtime": 0.498287833 }, "repwaitbackfillreserved_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "reprecovering_latency": { "avgcount": 88, "sum": 16.136443540, "avgtime": 0.183368676 }, "activating_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "waitlocalrecoveryreserved_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "waitremoterecoveryreserved_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "recovering_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "recovered_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "clean_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "active_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "replicaactive_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "stray_latency": { "avgcount": 346, "sum": 212.848549950, "avgtime": 0.615169219 }, "getinfo_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "getlog_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "waitactingchange_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "incomplete_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "down_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "getmissing_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "waitupthru_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "notrecovering_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "rocksdb": { "get_latency": { "avgcount": 790064, "sum": 51.480388766, "avgtime": 0.000065159 }, "submit_latency": { "avgcount": 3979669, "sum": 377.602827079, "avgtime": 0.000094882 }, "submit_sync_latency": { "avgcount": 3291698, "sum": 1909.117762596, "avgtime": 0.000579979 }, "compact": 0, "compact_range": 0, "compact_queue_merge": 0, "compact_queue_len": 0, "rocksdb_write_wal_time": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "rocksdb_write_memtable_time": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "rocksdb_write_delay_time": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "rocksdb_write_pre_and_post_time": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } } } We monitor osd.op_w_latency to check for problems on our clusters. Does someone know how to fix this ? Best regards, [cid:e90311cc-51d8-4266-b5ee-347da8fe1b72] Alexandre GOSSET Ingénieur Système et Réseau Fixe : +33 (0)3 59 82 20 20 Cyllene Lille - 393 rue du général de Gaulle - 59700 Marcq-en-baroeul www.groupe-cyllene.com<http://www.groupe-cyllene.com/>

1 0

cephadm logs
by Luis Domingues 31 Jul '23

31 Jul '23

Hi, Quick question about cephadm and its logs. On my cluster I have every logs that goes to journald. But on each machine, I still have /var/log/ceph/cephadm.log that is alive. Is there a way to make cephadm log to journald instead of a file? If yes did I miss it on the documentation? Of if not is there any reason to log into a file while everything else logs to journald? Thanks Luis Domingues Proton AG

3 3

OSD stuck on booting state after upgrade (v15.2.17 -> v17.2.6)
by s.smagul94＠gmail.com 31 Jul '23

31 Jul '23

Updating the Ceph cluster from the Octopus version (v15.2.17) to Quincy (v17.2.6). We used ceph-deploy to update all ceph packages on all hosts, and then we restarted the services one by one (mon -> mgr -> osd -> rgw). During the restart on the first node, all osd encountered an issue where they didn't change to the "up" state and got stuck in "booting" state. # ceph daemon osd.3 status { "cluster_fsid": "f95b201c-4cd6-4c36-a54e-7f2b68608b8f", "osd_fsid": "b0141718-a2ac-4a26-808b-17b6741b789e", "whoami": 3, "state": "booting", "oldest_map": 4437792, "newest_map": 4441114, "num_pgs": 29 } While changing "ceph osd require-osd-release quincy," the monitor service crashed with errors. # ceph report | jq '.osdmap.require_osd_release' "nautilus" -2> 2023-07-25T12:10:20.977+0600 7f245a84f700 5 mon.ceph-ph-mon1-dc3(a)0(leader).paxos(paxos updating c 81819224..81819937) is_readable = 1 - now=2023-07-25T12:10:20.981801+0600 lease_expire=2023-07-25T12:10:25.959818+0600 has v0 lc 81819937 -1> 2023-07-25T12:10:20.997+0600 7f245a84f700 -1 /build/ceph-17.2.6/src/mon/OSDMonitor.cc: In function 'bool OSDMonitor::prepare_command_impl(MonOpRequestRef, const cmdmap_t&)' thread 7f245a84f700 time 2023-07-25T12:10:20.981991+0600 /build/ceph-17.2.6/src/mon/OSDMonitor.cc: 11631: FAILED ceph_assert(osdmap.require_osd_release >= ceph_release_t::octopus) ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7f24629d3878] 2: /usr/lib/ceph/libceph-common.so.2(+0x27da8a) [0x7f24629d3a8a] 3: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr<MonOpRequest>, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::variant<std::__cxx11::basic_stri ng<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basi c_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > >, std::less<void>, std::allocator<std::pair<std: :__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std: :vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<lo ng, std::allocator<long> >, std::vector<double, std::allocator<double> > > > > > const&)+0xcb13) [0x5569f209a823] 4: (OSDMonitor::prepare_command(boost::intrusive_ptr<MonOpRequest>)+0x45f) [0x5569f20ab89f] 5: (OSDMonitor::prepare_update(boost::intrusive_ptr<MonOpRequest>)+0x162) [0x5569f20baa42] 6: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x716) [0x5569f201fd86] 7: (PaxosService::C_RetryMessage::_finish(int)+0x6c) [0x5569f1f4f93c] 8: (C_MonOp::finish(int)+0x4b) [0x5569f1ebbb3b] 9: (Context::complete(int)+0xd) [0x5569f1ebaa0d] 10: (void finish_contexts<std::__cxx11::list<Context*, std::allocator<Context*> > >(ceph::common::CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0xb0) [0x5569f1ef11e0] 11: (Paxos::finish_round()+0xb1) [0x5569f2015a61] 12: (Paxos::handle_last(boost::intrusive_ptr<MonOpRequest>)+0x11e3) [0x5569f20172a3] 13: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x49f) [0x5569f2019f7f] 14: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x14f4) [0x5569f1eb7f34] 15: (Monitor::_ms_dispatch(Message*)+0xa68) [0x5569f1eb8bd8] 16: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5d) [0x5569f1ef2c4d] 17: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x460) [0x7f2462c71da0] 18: (DispatchQueue::entry()+0x58f) [0x7f2462c6f63f] 19: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2462d40b61] 20: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f24624f4609] 21: clone() 0> 2023-07-25T12:10:21.009+0600 7f245a84f700 -1 *** Caught signal (Aborted) ** in thread 7f245a84f700 thread_name:ms_dispatch ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f24625003c0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b7) [0x7f24629d38e0] 5: /usr/lib/ceph/libceph-common.so.2(+0x27da8a) [0x7f24629d3a8a] 6: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr<MonOpRequest>, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::variant<std::__cxx11::basic_stri ng<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basi c_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > >, std::less<void>, std::allocator<std::pair<std: :__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std: :vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<lo ng, std::allocator<long> >, std::vector<double, std::allocator<double> > > > > > const&)+0xcb13) [0x5569f209a823] 7: (OSDMonitor::prepare_command(boost::intrusive_ptr<MonOpRequest>)+0x45f) [0x5569f20ab89f] 8: (OSDMonitor::prepare_update(boost::intrusive_ptr<MonOpRequest>)+0x162) [0x5569f20baa42] 9: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x716) [0x5569f201fd86] 10: (PaxosService::C_RetryMessage::_finish(int)+0x6c) [0x5569f1f4f93c] 11: (C_MonOp::finish(int)+0x4b) [0x5569f1ebbb3b] 12: (Context::complete(int)+0xd) [0x5569f1ebaa0d] 13: (void finish_contexts<std::__cxx11::list<Context*, std::allocator<Context*> > >(ceph::common::CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0xb0) [0x5569f1ef11e0] 14: (Paxos::finish_round()+0xb1) [0x5569f2015a61] 15: (Paxos::handle_last(boost::intrusive_ptr<MonOpRequest>)+0x11e3) [0x5569f20172a3] 16: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x49f) [0x5569f2019f7f] 17: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x14f4) [0x5569f1eb7f34] 18: (Monitor::_ms_dispatch(Message*)+0xa68) [0x5569f1eb8bd8] 19: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5d) [0x5569f1ef2c4d] 20: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x460) [0x7f2462c71da0] 21: (DispatchQueue::entry()+0x58f) [0x7f2462c6f63f] 22: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2462d40b61] 23: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f24624f4609] 24: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Now the cluster looks as follows: # ceph -s cluster: id: f95b201c-4cd6-4c36-a54e-7f2b68608b8f health: HEALTH_WARN noout flag(s) set 12 osds down 1 host (12 osds) down all OSDs are running octopus or later but require_osd_release < octopus Degraded data redundancy: 5731463/11463184 objects degraded (49.999%), 240 pgs degraded, 315 pgs undersized services: mon: 3 daemons, quorum ceph-ph-mon1-dc3,hw-ceph-ph3-dc3,hw-ceph-ph4-dc3 (age 68m) mgr: ceph-ph-mon1-dc3.alahd.kz.test.bash.kz(active, since 45m) osd: 25 osds: 13 up (since 26h), 25 in (since 21h); 6 remapped pgs flags noout rgw: 1 daemon active (1 hosts, 1 zones) data: pools: 8 pools, 321 pgs objects: 5.73M objects, 4.4 TiB usage: 8.8 TiB used, 32 TiB / 40 TiB avail pgs: 5731463/11463184 objects degraded (49.999%) 129/11463184 objects misplaced (0.001%) 240 active+undersized+degraded 75 active+undersized 6 active+clean+remapped # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 42.23981 - 22 TiB 4.7 TiB 4.6 TiB 1.2 GiB 22 GiB 17 TiB 0 0 - root default -2 20.19991 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host hw-ceph-ph3 0 hdd 1.81999 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down osd.0 1 hdd 1.81999 1.00000 1.8 TiB 401 GiB 396 GiB 76 MiB 1.2 GiB 1.4 TiB 21.47 0.98 0 down osd.1 2 hdd 1.81999 1.00000 1.8 TiB 365 GiB 362 GiB 45 KiB 1.1 GiB 1.5 TiB 19.59 0.90 0 down osd.2 3 hdd 1.81999 1.00000 1.8 TiB 584 GiB 580 GiB 825 KiB 1.7 GiB 1.3 TiB 31.29 1.43 0 down osd.3 4 hdd 1.81999 1.00000 1.8 TiB 365 GiB 362 GiB 621 KiB 1.1 GiB 1.5 TiB 19.57 0.90 0 down osd.4 5 hdd 1.81999 1.00000 1.8 TiB 583 GiB 579 GiB 31 KiB 1.7 GiB 1.3 TiB 31.25 1.43 0 down osd.5 6 hdd 1.81999 1.00000 1.8 TiB 365 GiB 362 GiB 43 KiB 1.1 GiB 1.5 TiB 19.55 0.90 0 down osd.6 7 hdd 1.81999 1.00000 1.8 TiB 365 GiB 362 GiB 12 KiB 1.1 GiB 1.5 TiB 19.58 0.90 0 down osd.7 8 hdd 1.81999 1.00000 1.8 TiB 365 GiB 362 GiB 27 KiB 1.1 GiB 1.5 TiB 19.58 0.90 0 down osd.8 9 hdd 1.81999 1.00000 1.8 TiB 547 GiB 543 GiB 47 KiB 1.6 GiB 1.3 TiB 29.32 1.34 0 down osd.9 10 hdd 1.81999 1.00000 1.8 TiB 330 GiB 327 GiB 68 KiB 987 MiB 1.5 TiB 17.67 0.81 0 down osd.10 12 ssd 0.17999 1.00000 186 GiB 1.2 GiB 67 MiB 1.1 GiB 86 MiB 185 GiB 0.65 0.03 0 down osd.12 -3 22.03990 - 22 TiB 4.7 TiB 4.6 TiB 1.2 GiB 22 GiB 17 TiB 21.11 0.97 - host hw-ceph-ph4 11 hdd 1.81999 1.00000 1.8 TiB 550 GiB 546 GiB 1012 KiB 2.3 GiB 1.3 TiB 29.48 1.35 34 up osd.11 13 hdd 1.81999 1.00000 1.8 TiB 332 GiB 329 GiB 2.8 MiB 1.5 GiB 1.5 TiB 17.78 0.82 22 up osd.13 14 hdd 1.81999 1.00000 1.8 TiB 550 GiB 547 GiB 4.4 MiB 2.1 GiB 1.3 TiB 29.50 1.35 25 up osd.14 15 hdd 1.81999 1.00000 1.8 TiB 259 GiB 256 GiB 1.6 MiB 1.2 GiB 1.6 TiB 13.86 0.64 20 up osd.15 16 hdd 1.81999 1.00000 1.8 TiB 477 GiB 474 GiB 2.7 MiB 1.9 GiB 1.4 TiB 25.58 1.17 23 up osd.16 17 hdd 1.81999 1.00000 1.8 TiB 403 GiB 400 GiB 1.7 MiB 1.7 GiB 1.4 TiB 21.63 0.99 32 up osd.17 18 hdd 1.81999 1.00000 1.8 TiB 294 GiB 291 GiB 79 MiB 1.4 GiB 1.5 TiB 15.79 0.72 25 up osd.18 19 hdd 1.81999 1.00000 1.8 TiB 294 GiB 291 GiB 1.6 MiB 1.4 GiB 1.5 TiB 15.76 0.72 27 up osd.19 20 hdd 1.81999 1.00000 1.8 TiB 477 GiB 474 GiB 2.3 MiB 1.9 GiB 1.4 TiB 25.60 1.17 23 up osd.20 21 hdd 1.81999 1.00000 1.8 TiB 473 GiB 470 GiB 23 MiB 2.0 GiB 1.4 TiB 25.36 1.16 24 up osd.21 22 hdd 1.81999 1.00000 1.8 TiB 404 GiB 401 GiB 3.6 MiB 2.3 GiB 1.4 TiB 21.69 0.99 22 up osd.22 23 hdd 1.81999 1.00000 1.8 TiB 258 GiB 255 GiB 2.7 MiB 1.3 GiB 1.6 TiB 13.84 0.63 18 up osd.23 24 ssd 0.20000 1.00000 238 GiB 2.2 GiB 67 MiB 1.1 GiB 1.0 GiB 236 GiB 0.91 0.04 32 up osd.24 TOTAL 40 TiB 8.8 TiB 8.8 TiB 2.3 GiB 34 GiB 32 TiB 21.81 MIN/MAX VAR: 0/1.43 STDDEV: 8.97 We checked the network connectivity between the hosts, and everything is fine there. We have not restarted the OSD services on the second node because we suspect that the entire cluster might crash. Please give advise how to solve the problem

5 6

configure rgw
by Tony Liu 30 Jul '23

30 Jul '23

Hi, I'm using Pacific v16.2.10 container image, deployed by cephadm. I used to manually build config file for rgw, deploy rgw, put config file in place and restart rgw. It works fine. Now, I'd like to put rgw config into config db. I tried with client.rgw, but the config is not taken by rgw. Also "config show" doesn't work. It always says "no config state". ``` # ceph orch ps | grep rgw rgw.qa.ceph-1.hzfrwq ceph-1 10.250.80.100:80 running (10m) 10m ago 53m 51.4M - 16.2.10 32214388de9d 13169a213bc5 # ceph config get client.rgw | grep frontends client.rgw basic rgw_frontends beast port=8086 * # ceph config show rgw.qa.ceph-1.hzfrwq Error ENOENT: no config state for daemon rgw.qa.ceph-1.hzfrwq # ceph config show client.rgw.qa.ceph-1.hzfrwq Error ENOENT: no config state for daemon client.rgw.qa.ceph-1.hzfrwq # radosgw-admin --show-config -n client.rgw.qa.ceph-1.hzfrwq | grep frontends rgw_frontends = beast port=7480 ``` Any clues what I am missing here? Thanks! Tony

1 2

precise/best way to check ssd usage
by Marc 29 Jul '23

29 Jul '23

Currently I am checking usage on ssd drives with ceph osd df| egrep 'CLASS|ssd' I have a use % between 48% and 57%, and assume that with a node failure 1/3 (only using 3x repl.) of this 57% needs to be able to migrate and added to a different node. Is there a better way of checking this (on old Nautilus)?

2 2

Reef release candidate - v18.1.3
by Yuri Weinstein 28 Jul '23

28 Jul '23

This is the third and possibly last release candidate for Reef. The Reef release comes with a new RockDB version (7.9.2) [0], which incorporates several performance improvements and features. Our internal testing doesn't show any side effects from the new version, but we are very eager to hear community feedback on it. This is the first release to have the ability to tune RockDB settings per column family [1], which allows for more granular tunings to be applied to different kinds of data stored in RocksDB. A new set of settings has been used in Reef to optimize performance for most kinds of workloads with a slight penalty in some cases, outweighed by large improvements in use cases such as RGW, in terms of compactions and write amplification. We would highly encourage community members to give these a try against their performance benchmarks and use cases. The detailed list of changes in terms of RockDB and BlueStore can be found in https://pad.ceph.com/p/reef-rc-relnotes. If any of our community members would like to help us with performance investigations or regression testing of the Reef release candidate, please feel free to provide feedback via email or in https://pad.ceph.com/p/reef_scale_testing. For more active discussions, please use the #ceph-at-scale slack channel in ceph-storage.slack.com. This RC has gone thru partial testing due to issues we are experiencing in the sepia lab. Please try it out and report any issues you encounter. Happy testing! Thanks, YuriW Get the release from * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-18.1.3.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: f594a0802c34733bb06e5993bc4bdb085c9a5f3f

1 0

MON sync time depends on outage duration
by Eugen Block 28 Jul '23

28 Jul '23

Hi *, I'm investigating an interesting issue on two customer clusters (used for mirroring) I've not solved yet, but today we finally made some progress. Maybe someone has an idea where to look next, I'd appreciate any hints or comments. These are two (latest) Octopus clusters, main usage currently is RBD mirroring with snapshot mode (around 500 RBD images are synced every 30 minutes). They noticed very long startup times of MON daemons after reboot, times between 10 and 30 minutes (reboot time already subtracted). These delays are present on both sites. Today we got a maintenance window and started to check in more detail by just restarting the MON service (joins quorum within seconds), then stopping the MON service and wait a few minutes (still joins quorum within seconds). And then we stopped the service and waited for more than 5 minutes, simulating a reboot, and then we were able to reproduce it. The sync then takes around 15 minutes, we verified with other MONs as well. The MON store is around 2 GB of size (on HDD), I understand that the sync itself can take some time, but what is the threshold here? I tried to find a hint in the MON config, searching for timeouts with 300 seconds, there were only a few matches (mon_session_timeout is one of them), but I'm not sure if they can explain this behavior. Investigating the MON store (ceph-monstore-tool dump-keys) I noticed that there were more than 42 Million osd_snap keys, which is quite a lot and would explain the size of the MON store. But I'm also not sure if it's related to the long syncing process. Does that sound familiar to anyone? Thanks, Eugen

4 13

Multiple object instances with null version id
by Huy Nguyen 28 Jul '23

28 Jul '23

Hi, I have a Ceph cluster in v16.2.13. I'm not sure why does this happen and how to clean it? [2023-07-12 21:23:13 +07] 299B STANDARD null v18 PUT index.txt [2023-07-12 21:27:54 +07] 299B STANDARD null v17 PUT index.txt [2023-07-12 21:48:01 +07] 299B STANDARD null v16 PUT index.txt [2023-07-12 21:42:24 +07] 299B STANDARD null v15 PUT index.txt [2023-07-12 21:03:42 +07] 299B STANDARD null v14 PUT index.txt [2023-07-12 21:16:25 +07] 299B STANDARD null v13 PUT index.txt [2023-07-12 21:09:27 +07] 299B STANDARD null v12 PUT index.txt [2023-07-12 22:01:28 +07] 299B STANDARD null v11 PUT index.txt [2023-07-25 08:33:03 +07] 0B null v10 DEL index.txt [2023-07-12 21:31:26 +07] 299B STANDARD null v9 PUT index.txt [2023-07-12 21:08:35 +07] 299B STANDARD null v8 PUT index.txt [2023-07-12 21:19:28 +07] 299B STANDARD null v7 PUT index.txt [2023-07-12 21:11:53 +07] 299B STANDARD null v6 PUT index.txt [2023-07-12 23:13:52 +07] 299B STANDARD null v5 PUT index.txt [2023-07-12 22:00:38 +07] 299B STANDARD null v4 PUT index.txt [2023-07-12 23:12:09 +07] 299B STANDARD null v3 PUT index.txt [2023-07-12 23:20:50 +07] 299B STANDARD null v2 PUT index.txt [2023-07-12 21:42:00 +07] 299B STANDARD null v1 PUT index.txt I tried to delete the object, but it only create delete marker. I can't even specify the versionid because all of them is null. I also tried to find that object with rados ls, and it returned only 1 object (which should be 18): 17a4ce99-009e-40f2-a2d2-2afc218ebd9b.876888518.16_airbnbnova/files/category/index.txt rados rm this object doesn't help anything Anyone have any ideal? Thanks

1 1

Ceph 17.2.6 alert-manager receives error 500 from inactive MGR
by Robert Sander 27 Jul '23

27 Jul '23

Hi, we noticed a strange error message in the logfiles: The alert-manager deployed with cephadm receives a HTTP 500 error from the inactive MGR when trying to call the URI /api/prometheus_receiver: Jul 25 09:35:25 alert-manager conmon[2426]: level=error ts=2023-07-25T07:35:25.171Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=45 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: unexpected status code 500: https://mgr001.example.net:8443/api/prometheus_receiver; ceph-dashboard/webhook[2]: notify retry canceled after 8 attempts: unexpected status code 500: https://mgr003.example.net:8443/api/prometheus_receiver" Jul 25 09:35:25 alert-manager conmon[2426]: level=warn ts=2023-07-25T07:35:25.175Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[2] msg="Notify attempt failed, will retry later" attempts=1 err="unexpected status code 500: https://mgr003.example.net:8443/api/prometheus_receiver" Jul 25 09:35:25 alert-manager conmon[2426]: level=warn ts=2023-07-25T07:35:25.177Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="unexpected status code 500: https://mgr001.example.net:8443/api/prometheus_receiver" Jul 25 09:35:35 alert-manager conmon[2426]: level=error ts=2023-07-25T07:35:35.171Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=45 err="ceph-dashboard/webhook[2]: notify retry canceled after 7 attempts: unexpected status code 500: https://mgr003.example.net:8443/api/prometheus_receiver; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: unexpected status code 500: https://mgr001.example.net:8443/api/prometheus_receiver" Jul 25 09:35:35 alert-manager conmon[2426]: level=warn ts=2023-07-25T07:35:35.176Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[2] msg="Notify attempt failed, will retry later" attempts=1 err="unexpected status code 500: https://mgr003.example.net:8443/api/prometheus_receiver" Jul 25 09:35:35 alert-manager conmon[2426]: level=warn ts=2023-07-25T07:35:35.176Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="unexpected status code 500: https://mgr001.example.net:8443/api/prometheus_receiver" This is from the logfile of mgr002, which was passive first and then became active. After being active the errors on the MGR where gone but showed on the newly passive MGR. Jul 25 09:25:25 mgr002 ceph-mgr[1841]: [dashboard INFO request] [::ffff:10.54.226.222:49904] [POST] [500] [0.002s] [513.0B] [581dce66-9c65-4e84-a41a-8d72b450791e] /api/prometheus_receiver Jul 25 09:25:25 mgr002 ceph-mgr[1841]: [dashboard ERROR request] [::ffff:10.54.226.222:49904] [POST] [500] [0.001s] [513.0B] [26e1854a-3b93-49c4-8afc-1a96426a3dab] /api/prometheus_receiver Jul 25 09:25:25 mgr002 ceph-mgr[1841]: [dashboard ERROR request] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request _id": "26e1854a-3b93-49c4-8afc-1a96426a3dab"} '] Jul 25 09:25:25 mgr002 ceph-mgr[1841]: [dashboard INFO request] [::ffff:10.54.226.222:49904] [POST] [500] [0.002s] [513.0B] [26e1854a-3b93-49c4-8afc-1a96426a3dab] /api/prometheus_receiver Jul 25 09:25:26 mgr002 ceph-mgr[1841]: [dashboard ERROR request] [::ffff:10.54.226.222:49904] [POST] [500] [0.001s] [513.0B] [46d7e78c-49d5-4652-9877-973129ad3977] /api/prometheus_receiver Jul 25 09:25:26 mgr002 ceph-mgr[1841]: [dashboard ERROR request] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request _id": "46d7e78c-49d5-4652-9877-973129ad3977"} '] Jul 25 09:25:26 mgr002 ceph-mgr[1841]: [dashboard INFO request] [::ffff:10.54.226.222:49904] [POST] [500] [0.002s] [513.0B] [46d7e78c-49d5-4652-9877-973129ad3977] /api/prometheus_receiver Jul 25 09:25:27 mgr002 ceph-mgr[1841]: [dashboard ERROR request] [::ffff:10.54.226.222:49904] [POST] [500] [0.002s] [513.0B] [a9b25e54-f1e1-42eb-90b2-af5aa22769cf] /api/prometheus_receiver Jul 25 09:25:27 mgr002 ceph-mgr[1841]: [dashboard ERROR request] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request _id": "a9b25e54-f1e1-42eb-90b2-af5aa22769cf"} '] Jul 25 09:25:27 mgr002 ceph-mgr[1841]: [dashboard INFO request] [::ffff:10.54.226.222:49904] [POST] [500] [0.002s] [513.0B] [a9b25e54-f1e1-42eb-90b2-af5aa22769cf] /api/prometheus_receiver Jul 25 09:25:28 mgr002 ceph-mgr[1841]: mgr handle_mgr_map Activating! Jul 25 09:25:28 mgr002 ceph-mgr[1841]: mgr handle_mgr_map I am now activating We have a test cluster running also with version 17.2.6 where this does not happen. In this test cluster the passive MGRs return an HTTP code 204 when the alert-manager tries to request /api/prometheus_receiver. What is happening here? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

2 2

inactive PGs looking for a non existent OSD
by Alfredo Rezinovsky 27 Jul '23

27 Jul '23

I had a problem with a server, hardware completely broken. "ceph orch rm host" hanged, even with force and offline options I reinstalled other server with the same IP address and then I removed the OSD with: ceph osd purge osd.10 ceph osd purge osd.11 Now I have 0.342% pgs not active with ceph pg <pg.id> query I can see the PG is blocked by a non existent OSD.10 or 11 (in the other problematic PG) I already tried setting osd_find_best_info_ignore_history_les = false in the intervening OSDs and restarted them with some luck (I had 3 non active PGs, now I have 2) Also after that another OSD keeps restarting. Fixed that by setting the reweight to 0 and still waiting until the OSD is empty to destroy it. -- Alfrenovsky

2 1

PG backfilled slow
by Peter 26 Jul '23

26 Jul '23

Hi all, I need replace some disk due to bad sector. I have crush out these disks and ceph did backfilling and migrate data as I want. However, I could see these OSD has one or more PG left after a day wait and backfilling really slow. Now it has only one backfilling PG at the same time. host001:~# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 122 hdd 9.37500 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 1 up 123 hdd 9.37500 1.00000 9.4 TiB 2.1 TiB 1.8 TiB 224 KiB 4.7 GiB 7.3 TiB 22.06 0.69 64 up 124 hdd 9.37500 1.00000 9.4 TiB 2.0 TiB 1.7 TiB 211 KiB 4.4 GiB 7.4 TiB 21.14 0.67 61 up 125 hdd 9.37500 1.00000 9.4 TiB 2.2 TiB 1.9 TiB 218 KiB 5.0 GiB 7.2 TiB 22.94 0.72 67 up 126 hdd 9.37500 1.00000 9.4 TiB 2.3 TiB 2.0 TiB 235 KiB 4.7 GiB 7.1 TiB 24.50 0.77 72 up 127 hdd 9.37500 1.00000 9.4 TiB 2.4 TiB 2.1 TiB 248 KiB 5.5 GiB 6.9 TiB 25.91 0.82 77 up 128 hdd 9.37500 1.00000 9.4 TiB 2.2 TiB 1.9 TiB 349 KiB 5.0 GiB 7.2 TiB 23.52 0.74 69 up 129 hdd 9.37500 1.00000 9.4 TiB 2.1 TiB 1.8 TiB 216 KiB 4.6 GiB 7.3 TiB 22.62 0.71 66 up 130 hdd 9.37500 1.00000 9.4 TiB 2.5 TiB 2.2 TiB 244 KiB 5.3 GiB 6.9 TiB 26.51 0.83 79 up 131 hdd 9.37500 1.00000 9.4 TiB 2.1 TiB 1.8 TiB 230 KiB 4.0 GiB 7.3 TiB 22.09 0.70 64 up 132 hdd 9.37500 1.00000 9.4 TiB 2.2 TiB 2.0 TiB 231 KiB 5.1 GiB 7.1 TiB 23.93 0.75 70 up 133 hdd 9.37500 1.00000 9.4 TiB 2.7 TiB 2.4 TiB 479 KiB 6.1 GiB 6.7 TiB 28.92 0.91 87 up 134 hdd 9.37500 1.00000 9.4 TiB 2.3 TiB 2.1 TiB 225 KiB 4.9 GiB 7.0 TiB 25.02 0.79 74 up 135 hdd 9.37500 1.00000 9.4 TiB 2.0 TiB 1.7 TiB 395 KiB 4.5 GiB 7.4 TiB 21.46 0.68 62 up 136 hdd 9.37500 1.00000 9.4 TiB 2.8 TiB 2.5 TiB 294 KiB 5.6 GiB 6.6 TiB 29.52 0.93 89 up 137 hdd 9.37500 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 2 up 138 hdd 9.37500 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 5 up 139 hdd 9.37500 1.00000 9.4 TiB 2.4 TiB 2.2 TiB 259 KiB 5.3 GiB 6.9 TiB 25.94 0.82 77 up 140 hdd 9.37500 1.00000 9.4 TiB 2.5 TiB 2.2 TiB 355 KiB 4.8 GiB 6.9 TiB 26.86 0.85 80 up 141 hdd 9.37500 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 1 up 142 hdd 9.37500 1.00000 9.4 TiB 2.6 TiB 2.3 TiB 1.6 GiB 4.9 GiB 6.8 TiB 27.43 0.86 83 up 143 hdd 9.37500 1.00000 9.4 TiB 2.7 TiB 2.4 TiB 276 KiB 5.7 GiB 6.7 TiB 28.64 0.90 86 up 144 hdd 9.37500 1.00000 9.4 TiB 2.5 TiB 2.2 TiB 256 KiB 5.5 GiB 6.9 TiB 26.77 0.84 80 up 145 hdd 9.37500 1.00000 9.4 TiB 2.3 TiB 2.0 TiB 248 KiB 5.0 GiB 7.1 TiB 24.46 0.77 72 up 146 hdd 9.37500 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 1 up 147 hdd 9.37500 1.00000 9.4 TiB 2.2 TiB 1.9 TiB 237 KiB 5.1 GiB 7.2 TiB 23.53 0.74 69 up 148 hdd 9.37500 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 1 up host001:~# ceph pg dump_stuck PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 5.3fd active+remapped+backfill_wait [145,158,151] 145 [145,151,126] 145 5.3a8 active+remapped+backfilling [136,133,158] 136 [136,133,167] 136 5.2e0 active+remapped+backfill_wait [147,158,135] 147 [147,135,166] 147 5.294 active+remapped+backfill_wait [147,128,164] 147 [147,128,138] 147 5.ef active+remapped+backfill_wait [123,134,158] 123 [123,148,137] 123 5.116 active+remapped+backfill_wait [123,166,145] 123 [123,166,138] 123 5.1e8 active+remapped+backfill_wait [127,158,157] 127 [127,157,161] 127 5.106 active+remapped+backfill_wait [124,158,144] 124 [124,144,167] 124 5.1c active+remapped+backfill_wait [128,158,155] 128 [128,155,140] 128 5.2ef active+remapped+backfill_wait [128,163,153] 128 [128,163,137] 128 5.1e0 active+remapped+backfill_wait [129,158,153] 129 [129,153,162] 129 5.1d2 active+remapped+backfill_wait [128,168,149] 128 [128,168,146] 128 5.167 active+remapped+backfill_wait [129,142,158] 129 [129,142,168] 129 5.f1 active+remapped+backfill_wait [124,147,158] 124 [124,147,168] 124 5.2c active+remapped+backfill_wait [129,159,154] 129 [129,159,141] 129 5.12b active+remapped+backfill_wait [128,169,157] 128 [128,169,138] 128 5.3eb active+remapped+backfill_wait [136,158,149] 136 [136,149,127] 136 5.6e active+remapped+backfill_wait [136,168,152] 136 [136,168,122] 136 5.3d8 active+remapped+backfill_wait [124,147,134] 124 [124,147,138] 124 5.b4 active+remapped+backfill_wait [123,142,166] 123 [123,166,138] 123 5.1f5 active+remapped+backfill_wait [145,153,158] 145 [145,153,169] 145 5.19c active+remapped+backfill_wait [129,158,151] 129 [129,151,164] 129 5.b3 active+remapped+backfill_wait [124,143,158] 124 [124,143,155] 124 5.108 active+remapped+backfill_wait [136,158,133] 136 [136,133,153] 136 Anyone can suggest what to do to fasten this process. Thanks, Peter

2 1

cephbot - a Slack bot for Ceph has been added to the github.com/ceph project
by David Turner 26 Jul '23

26 Jul '23

cephbot [1] is a project that I've been working on and using for years now and it has been added to the github.com/ceph project to increase visibility for other people that would like to implement slack-ops for their Ceph clusters. The instructions show how to set it up so that read-only operations can be performed from Slack for security purposes, but there are settings that could make it possible to lock down who can communicate with cephbot which could make it relatively secure to run administrative tasks as well. Ask here or in the Ceph Slack instance if you have any questions about its uses, implementation, or would like to contribute. I hope you find it as useful as I have. David Turner Sony Interactive Entertainment [1] https://github.com/ceph/cephbot-slack

2 1

Ceph Leadership Team Meeting, 2023-07-26 Minutes
by Casey Bodley 26 Jul '23

26 Jul '23

Welcome to Aviv Caro as new Ceph NVMe-oF lead Reef status: * reef 18.1.3 built, gibba cluster upgraded, plan to publish this week * https://pad.ceph.com/p/reef_final_blockers all resolved except for bookworm builds https://tracker.ceph.com/issues/61845 * only blockers will merge to reef so the release matches final rc Planning for distribution updates earlier in release process: * centos 9 testing wasn't enabled for reef until very late -- partly because of missing python dependencies -- required fixes to test suites of every component so we couldn't merge until everything was fixed * also applies to major dependencies like boost and rocksdb -- boost upgrade on main disrupted testing on other release branches -- build containerization in CI would help a lot here. discussion continues tomorrow in Ceph Infrastructure meeting Improving the documentation/procedure for deploying a vstart cluster: * including installation of dependencies and compilation -- add test coverage on fresh distros to verify that all required dependencies are installed * README.md will be the canonical guide CDS concluded yesterday: * recordings at https://ceph.io/en/community/events/2023/ceph-developer-summit-squid/ * component leads to update ceph backlog on trello

1 0

RGWs offline after upgrade to Nautilus
by Ben.Zieglmeier 26 Jul '23

26 Jul '23

Hello, We have an RGW cluster that was recently upgraded from 12.2.11 to 14.2.22. The upgrade went mostly fine, though now several of our RGWs will not start. One RGW is working fine, the rest will not initialize. They are on a crash loop. This is part of a multisite configuration, and is currently not the master zone. Current master zone is running 14.2.22. These are the only two zones in the zonegroup. After turning debug up to 20, these are the log snippets between each crash: ``` 2023-07-20 14:29:56.371 7fd8dec40900 20 RGWRados::pool_iterate: got periods.1b6e1a93-98ba-4378-bc5c-d36cd5542f11.52 2023-07-20 14:29:56.371 7fd8dec40900 20 RGWRados::pool_iterate: got periods.1b6e1a93-98ba-4378-bc5c-d36cd5542f11.54 2023-07-20 14:29:56.371 7fd8dec40900 20 RGWRados::pool_iterate: got realms_names. <redacted> 2023-07-20 14:29:56.371 7fd8dec40900 20 RGWRados::pool_iterate: got <redacted> 2023-07-20 14:29:56.371 7fd8dec40900 20 rados->read ofs=0 len=0 2023-07-20 14:29:56.371 7fd8dec40900 20 rados_obj.operate() r=-2 bl.length=0 2023-07-20 14:29:56.371 7fd8dec40900 20 rados->read ofs=0 len=0 2023-07-20 14:29:56.373 7fd8dec40900 20 rados_obj.operate() r=-2 bl.length=0 2023-07-20 14:29:56.373 7fd8dec40900 20 rados->read ofs=0 len=0 2023-07-20 14:29:56.373 7fd8dec40900 20 rados_obj.operate() r=-2 bl.length=0 2023-07-20 14:29:56.373 7fd8dec40900 20 rados->read ofs=0 len=0 2023-07-20 14:29:56.373 7fd8dec40900 20 rados_obj.operate() r=0 bl.length=46 2023-07-20 14:29:56.373 7fd8dec40900 20 rados->read ofs=0 len=0 2023-07-20 14:29:56.373 7fd8dec40900 20 rados_obj.operate() r=0 bl.length=114 2023-07-20 14:29:56.373 7fd8dec40900 20 rados->read ofs=0 len=0 2023-07-20 14:29:56.373 7fd8dec40900 20 rados_obj.operate() r=0 bl.length=46 2023-07-20 14:29:56.373 7fd8dec40900 20 rados->read ofs=0 len=0 2023-07-20 14:29:56.374 7fd8dec40900 20 rados_obj.operate() r=0 bl.length=686 2023-07-20 14:29:56.374 7fd8dec40900 20 period zonegroup init ret 0 2023-07-20 14:29:56.374 7fd8dec40900 20 period zonegroup name <redacted> 2023-07-20 14:29:56.374 7fd8dec40900 20 using current period zonegroup <redacted> 2023-07-20 14:29:56.374 7fd8dec40900 20 rados->read ofs=0 len=0 2023-07-20 14:29:56.374 7fd8dec40900 20 rados_obj.operate() r=0 bl.length=46 2023-07-20 14:29:56.374 7fd8dec40900 20 rados->read ofs=0 len=0 2023-07-20 14:29:56.375 7fd8dec40900 20 rados_obj.operate() r=0 bl.length=903 2023-07-20 14:29:56.375 7fd8dec40900 10 Cannot find current period zone using local zone 2023-07-20 14:29:56.375 7fd8dec40900 20 rados->read ofs=0 len=0 2023-07-20 14:29:56.375 7fd8dec40900 20 rados_obj.operate() r=0 bl.length=903 2023-07-20 14:29:56.375 7fd8dec40900 20 zone <redacted> 2023-07-20 14:29:56.375 7fd8dec40900 20 generating connection object for zone <redacted> id f10b465f-bf18-47d0-a51c-ca4f17118ee1 2023-07-20 14:34:56.198 7fd8cafe8700 -1 Initialization timeout, failed to initialize ``` I’ve checked all file permissions, filesystem free space, disabled selinux and firewalld, tried turning up the initialization timeout to 600, and tried removing all non-essential config from ceph.conf. All produce the same results. I would greatly appreciate any other ideas or insight. Thanks, Ben

3 3

Re: 1 PG stucked in "active+undersized+degraded for long time
by Matthew Leonard (BLOOMBERG/ 120 PARK) 26 Jul '23

26 Jul '23

Assuming you're running systemctl OSDs you can run the following command on the host that OSD 343 resides on. systemctl restart ceph-osd@343 From: siddhit.renake(a)nxtgen.com At: 07/20/23 13:44:36 UTC-4:00To: ceph-users(a)ceph.io Subject: [ceph-users] Re: 1 PG stucked in "active+undersized+degraded for long time What should be appropriate way to restart primary OSD in this case (343) ? _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 2

ceph quincy repo update to debian bookworm...?
by Christian Peters 26 Jul '23

26 Jul '23

Hi ceph users/maintainers, I installed ceph quincy on debian bullseye as a ceph client and now want to update to bookworm. I see that there is at the moment only bullseye supported. https://download.ceph.com/debian-quincy/dists/bullseye/ Will there be an update of deb https://download.coeh.com/debian-quincy/ bullseye main to deb https://download.coeh.com/debian-quincy/ boowkworm main in the near future!? Regards, Christian

3 2

cephadm and kernel memory usage
by Luis Domingues 26 Jul '23

26 Jul '23

Hi, So after, looking into OSDs memory usage, which seem to be fine, on a v16.2.13 running with cephadm, on el8, it seems that the kernel is using a lot of memory. # smem -t -w -k Area Used Cache Noncache firmware/hardware 0 0 0 kernel image 0 0 0 kernel dynamic memory 65.0G 18.6G 46.4G userspace memory 50.1G 260.5M 49.9G free memory 9.9G 9.9G 0 ---------------------------------------------------------- 125.0G 28.8G 96.3G Comparing with a similar other cluster, same OS, same ceph version, but running packages instead if containers, and machines have a little bit more memory: # smem -t -w -k Area Used Cache Noncache firmware/hardware 0 0 0 kernel image 0 0 0 kernel dynamic memory 52.8G 50.5G 2.4G userspace memory 123.9G 198.5M 123.7G free memory 10.6G 10.6G 0 ---------------------------------------------------------- 187.3G 61.3G 126.0G Does anyone have an idea why when using containers with podman the kernel needs a lot more memory? Luis Domingues Proton AG

2 6

Signature V4 for Ceph 16.2.4 ( Pacific )
by nguyenvandiep＠baoviet.com.vn 25 Jul '23

25 Jul '23

Can anyone help me ? I need to know do Ceph 16.2.4 support Signature V4 for S3 API ? IF yes, pls guide us Thank u all

2 1

Failing to restart mon and mgr daemons on Pacific
by Renata Callado Borges 25 Jul '23

25 Jul '23

Dear all, How are you? I have a cluster on Pacific with 3 hosts, each one with 1 mon, 1 mgr and 12 OSDs. One of the hosts, darkside1, has been out of quorum according to ceph status. Systemd showed 4 services dead, two mons and two mgrs. I managed to systemctl restart one mon and one mgr, but even after several attempts, the remaining mon and mgr services, when asked to restart, keep returning to a failed state after a few seconds. They try to auto-restart and then go into a failed state where systemd requires me to manually set them to "reset-failed" before trying to start again. But they never stay up. There are no clear messages about the issue in /var/log/ceph/cephadm.log. The host is still out of quorum. I have failed to "turn on debug" as per https://docs.ceph.com/en/pacific/rados/troubleshooting/log-and-debug/. It seems I do not know the proper incantantion for "ceph daemon X config show", no string for X seems to satisfy this command. I have tried adding this: [mon] debug mon = 20 To my ceph.conf, but no additional lines of log are sent to /var/log/cephadm.log so I'm sorry I can´t provide more details. Could someone help me debug this situation? I am sure that if just reboot the machine, it will start up the services properly, as it always has done, but I would prefer to fix this without this action. Cordially, Renata.

2 4

Does ceph permit the definition of new classes?
by wodel youchi 24 Jul '23

24 Jul '23

Hi, Can I define new device classes in ceph, I know that there are hdd, ssd and nvme, but can I define other classes? Regards. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

3 2

Adding datacenter level to CRUSH tree causes rebalancing
by Niklas Hambüchen 24 Jul '23

24 Jul '23

Hi Ceph users, I have a Ceph 16.2.7 cluster that so far has been replicated over the `host` failure domain. All `hosts` have been chosen to be in different `datacenter`s, so that was sufficient. Now I wish to add more hosts, including some in already-used data centers, so I'm planning to use CRUSH's `datacenter` failure domain instead. My problem is that when I add the `datacenter`s into the CRUSH tree, Ceph decides that it should now rebalance the entire cluster. This seems unnecessary, and wrong. Before, `ceph osd tree` (some OSDs omitted for legibility): ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 440.73514 root default -3 146.43625 host node-4 2 hdd 14.61089 osd.2 up 1.00000 1.00000 3 hdd 14.61089 osd.3 up 1.00000 1.00000 -7 146.43625 host node-5 14 hdd 14.61089 osd.14 up 1.00000 1.00000 15 hdd 14.61089 osd.15 up 1.00000 1.00000 -10 146.43625 host node-6 26 hdd 14.61089 osd.26 up 1.00000 1.00000 27 hdd 14.61089 osd.27 up 1.00000 1.00000 After assigning of `datacenter` crush buckets: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 440.73514 root default -18 146.43625 datacenter FSN-DC16 -7 146.43625 host node-5 14 hdd 14.61089 osd.14 up 1.00000 1.00000 15 hdd 14.61089 osd.15 up 1.00000 1.00000 -17 146.43625 datacenter FSN-DC18 -10 146.43625 host node-6 26 hdd 14.61089 osd.26 up 1.00000 1.00000 27 hdd 14.61089 osd.27 up 1.00000 1.00000 -16 146.43625 datacenter FSN-DC4 -3 146.43625 host node-4 2 hdd 14.61089 osd.2 up 1.00000 1.00000 3 hdd 14.61089 osd.3 up 1.00000 1.00000 This shows that the tree is essentially unchanged, it just "gained a level". In `ceph status` I now get: pgs: 1167541260/1595506041 objects misplaced (73.177%) If I remove the `datacenter` level again, then the misplacement disappears. On a minimal testing cluster, this misplacement issue did not appear. Why does Ceph think that these objects are misplaced when I add the datacenter level? Is there a more correct way to do this? Thanks!

4 6

mds terminated
by dxodnd＠naver.com 24 Jul '23

24 Jul '23

hello. I am using ROK CEPH and have 20 MDSs in use. 10 are in rank 0-9 and 10 are in standby. I have one ceph filesystem, and 2 mds are trimming. Under one FILESYSTEM, there are 6 MDSs in RESOLVE, 1 MDS in REPLAY, and 3 in ACTIVE. For some reason, since 36 hours ago, RESOLVE is stuck in TRIMMING, and so are the MDSs in REPLAY. I've also tried FAILing each MDS, but to no avail. I think something should change when the MDS in REPLAY goes to RESOLVE, but I don't know what. Even looking at the logs of the REPLAY MDS, it's hard to see any messages other than it is TERMINATED every 11 minutes. I'm desperate for someone's help.

4 6

cephfs - unable to create new subvolume
by karon karon 24 Jul '23

24 Jul '23

Hello, I recently use cephfs in version 17.2.6 I have a pool named "*data*" and a fs "*kube*" it was working fine until a few days ago, now i can no longer create a new subvolume*, *it gives me the following error: Error EINVAL: invalid value specified for ceph.dir.subvolume > here is the command used: ceph fs subvolume create kube newcsivol --pool_layout data > from what I understand it seems that it creates the subvolume but immediately puts it in the trash !? here is the log : 2023-06-23T08:30:53.307+0000 7f2b929d2700 0 log_channel(audit) log [DBG] : > from='client.86289 -' entity='client.admin' cmd=[{"prefix": "fs subvolume > create", "vol_name": "kube", "sub_name": "newcsivol", "group_name": "csi", > "pool_layout": "data", "target": ["mon-mgr", ""]}]: dispatch > 2023-06-23T08:30:53.307+0000 7f2b8a1d1700 0 [volumes INFO volumes.module] > Starting _cmd_fs_subvolume_create(group_name:csi, pool_layout:data, > prefix:fs subvolume create, sub_name:newcsivol, target:['mon-mgr', ''], > vol_name:kube) < "" > 2023-06-23T08:30:53.327+0000 7f2b8a1d1700 0 [volumes INFO > volumes.fs.operations.versions.subvolume_v2] cleaning up subvolume with > path: newcsivol > 2023-06-23T08:30:53.331+0000 7f2b8a1d1700 0 [volumes INFO > volumes.fs.operations.versions.subvolume_base] subvolume path > 'b'/volumes/csi/newcsivol'' moved to trashcan > 2023-06-23T08:30:53.331+0000 7f2b8a1d1700 0 [volumes INFO > volumes.fs.async_job] queuing job for volume 'kube' > 2023-06-23T08:30:53.335+0000 7f2b8a1d1700 0 [volumes INFO volumes.module] > Finishing _cmd_fs_subvolume_create(group_name:csi, pool_layout:data, > prefix:fs subvolume create, sub_name:newcsivol, target:['mon-mgr', ''], > vol_name:kube) < "" > 2023-06-23T08:30:53.335+0000 7f2b8a1d1700 -1 mgr.server reply reply (22) > Invalid argument invalid value specified for ceph.dir.subvolume > 2023-06-23T08:30:53.339+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.339+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.339+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.339+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.339+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.363+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.363+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.363+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.363+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.363+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.383+0000 7f2b479c2700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.383+0000 7f2b479c2700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.383+0000 7f2b479c2700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.383+0000 7f2b479c2700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.383+0000 7f2b479c2700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.507+0000 7f2b3ff33700 0 [prometheus INFO > cherrypy.access.139824530773776] 192.168.240.231 - - [23/Jun/2023:08:30:53] > "GET /metrics HTTP/1.1" 200 194558 "" "Prometheus/2.33.4" > 2023-06-23T08:30:54.219+0000 7f2b3ddaf700 0 [dashboard INFO request] [ > 172.29.2.142:33040] [GET] [200] [0.003s] [admin] [22.0B] > /api/prometheus/notifications > 2023-06-23T08:30:54.223+0000 7f2b929d2700 0 log_channel(audit) log [DBG] > : from='mon.0 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": > "json"}]: dispatch > 2023-06-23T08:30:54.227+0000 7f2b3a5a8700 0 [dashboard INFO request] [ > 172.29.2.142:49348] [GET] [200] [0.019s] [admin] [22.0B] /api/prometheus > 2023-06-23T08:30:54.227+0000 7f2b929d2700 0 log_channel(audit) log [DBG] > : from='mon.0 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": > "json"}]: dispatch > 2023-06-23T08:30:54.231+0000 7f2b3d5ae700 0 [dashboard INFO request] [ > 172.29.2.142:39414] [GET] [200] [0.022s] [admin] [9.3K] > /api/prometheus/rules > 2023-06-23T08:30:54.275+0000 7f2ba39d4700 0 log_channel(cluster) log > [DBG] : pgmap v2116480: 145 pgs: 145 active+clean; 2.8 GiB data, 12 GiB > used, 1.5 TiB / 1.5 TiB avail; 5.5 KiB/s wr, 0 op/s > my fs info : # ceph fs ls > name: kube, metadata pool: metadata, data pools: [data ] > thank for your help best regards Karim

3 3

MDS cache is too large and crashes
by Sake Ceph 24 Jul '23

24 Jul '23

At 01:27 this morning I received the first email about MDS cache is too large (mailing happens every 15 minutes if something happens). Looking into it, it was again a standby-replay host which stops working. At 01:00 a few rsync processes start in parallel on a client machine. This copies data from a NFS share to Cephfs share to sync the latest changes. (we want to switch to Cephfs in the near future). This crashing of the standby-replay mds happend a couple times now, so I think it would be good to get some help. Where should I look next? Some cephfs information ---------------------------------- # ceph fs status atlassian-opl - 8 clients ============= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active atlassian-opl.mds5.zsxfep Reqs: 0 /s 7830 7803 635 3706 0-s standby-replay atlassian-opl.mds6.svvuii Evts: 0 /s 3139 1924 461 0 POOL TYPE USED AVAIL cephfs.atlassian-opl.meta metadata 2186M 1161G cephfs.atlassian-opl.data data 23.0G 1161G atlassian-prod - 12 clients ============== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active atlassian-prod.mds1.msydxf Reqs: 0 /s 2703k 2703k 905k 1585 1 active atlassian-prod.mds2.oappgu Reqs: 0 /s 961k 961k 317k 622 2 active atlassian-prod.mds3.yvkjsi Reqs: 0 /s 2083k 2083k 670k 443 0-s standby-replay atlassian-prod.mds4.qlvypn Evts: 0 /s 352k 352k 102k 0 1-s standby-replay atlassian-prod.mds5.egsdfl Evts: 0 /s 873k 873k 277k 0 2-s standby-replay atlassian-prod.mds6.ghonso Evts: 0 /s 2317k 2316k 679k 0 POOL TYPE USED AVAIL cephfs.atlassian-prod.meta metadata 58.8G 1161G cephfs.atlassian-prod.data data 5492G 1161G MDS version: ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) When looking at the log on the MDS server, I've got the following: 2023-07-21T01:21:01.942+0000 7f668a5e0700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2023-07-21T01:23:13.856+0000 7f6688ddd700 1 mds.atlassian-prod.pwsoel13143.qlvypn Updating MDS map to version 5671 from mon.1 2023-07-21T01:23:18.369+0000 7f6688ddd700 1 mds.atlassian-prod.pwsoel13143.qlvypn Updating MDS map to version 5672 from mon.1 2023-07-21T01:23:31.719+0000 7f6688ddd700 1 mds.atlassian-prod.pwsoel13143.qlvypn Updating MDS map to version 5673 from mon.1 2023-07-21T01:23:35.769+0000 7f6688ddd700 1 mds.atlassian-prod.pwsoel13143.qlvypn Updating MDS map to version 5674 from mon.1 2023-07-21T01:28:23.764+0000 7f6688ddd700 1 mds.atlassian-prod.pwsoel13143.qlvypn Updating MDS map to version 5675 from mon.1 2023-07-21T01:29:13.657+0000 7f6688ddd700 1 mds.atlassian-prod.pwsoel13143.qlvypn Updating MDS map to version 5676 from mon.1 2023-07-21T01:33:43.886+0000 7f6688ddd700 1 mds.atlassian-prod.pwsoel13143.qlvypn Updating MDS map to version 5677 from mon.1 (and another 20 lines about updating MDS map) Alert mailings: Mail at 01:27 ---------------------------------- HEALTH_WARN --- New --- [WARN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache mds.atlassian-prod.mds4.qlvypn(mds.0): MDS cache is too large (13GB/9GB); 0 inodes in use by clients, 0 stray files === Full health status === [WARN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache mds.atlassian-prod.mds4.qlvypn(mds.0): MDS cache is too large (13GB/9GB); 0 inodes in use by clients, 0 stray files Mail at 03:27 ---------------------------------- HEALTH_OK --- Cleared --- [WARN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache mds.atlassian-prod.mds4.qlvypn(mds.0): MDS cache is too large (14GB/9GB); 0 inodes in use by clients, 0 stray files === Full health status === Mail at 04:12 ---------------------------------- HEALTH_WARN --- New --- [WARN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache mds.atlassian-prod.mds4.qlvypn(mds.0): MDS cache is too large (15GB/9GB); 0 inodes in use by clients, 0 stray files === Full health status === [WARN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache mds.atlassian-prod.mds4.qlvypn(mds.0): MDS cache is too large (15GB/9GB); 0 inodes in use by clients, 0 stray files Best regards, Sake

3 3

quincy 17.2.6 - write performance continuously slowing down until OSD restart needed
by Nikola Ciprich 22 Jul '23

22 Jul '23

Hello dear CEPH users and developers, we're dealing with strange problems.. we're having 12 node alma linux 9 cluster, initially installed CEPH 15.2.16, then upgraded to 17.2.5. It's running bunch of KVM virtual machines accessing volumes using RBD. everything is working well, but there is strange and for us quite serious issue - speed of write operations (both sequential and random) is constantly degrading drastically to almost unusable numbers (in ~1week it drops from ~70k 4k writes/s from 1 VM to ~7k writes/s) When I restart all OSD daemons, numbers immediately return to normal.. volumes are stored on replicated pool of 4 replicas, on top of 7*12 = 84 INTEL SSDPE2KX080T8 NVMEs. I've updated cluster to 17.2.6 some time ago, but the problem persists. This is especially annoying in connection with https://tracker.ceph.com/issues/56896 as restarting OSDs is quite painfull when half of them crash.. I don't see anything suspicious, nodes load is quite low, no logs errors, network latency and throughput is OK too Anyone having simimar issue? I'd like to ask for hints on what should I check further.. we're running lots of 14.2.x and 15.2.x clusters, none showing similar issue, so I'm suspecting this is something related to quincy thanks a lot in advance with best regards nikola ciprich -- ------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis(a)linuxbox.cz -------------------------------------

7 16

upload-part-copy gets access denied after cluster upgrade
by Motahare S 22 Jul '23

22 Jul '23

Hello everyone, We have a ceph cluster which was recently updated from octopus(15.2.12) to pacific(16.2.13). There has been a problem in multi part upload, which is, when doing UPLOAD_PART_COPY from a valid and existing previously uploaded part, it gets 403, ONLY WHEN IT'S CALLED BY SERVICE-USER. The same scenario gets a 200 response by a full-access sub-user, and both sub-user and service-user get 200 on the same scenario in octopus version. The policy for service user access is as below: { "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::user/wid:suserid" }, "Action": "*", "Resource": [ "arn:aws:s3:::bucketname", "arn:aws:s3:::bucketname/*" ] } ] } Note that this very service-user can perform a multi-part upload without any problem on both versions, only the upload_part_copy and only on pacific, it gets 403; which makes it unlikely to be an access problem. Has anyone encountered this issue? I performed multi-part upload using boto3 but there has been the same issue on other clients as well. regards

2 1

OSD tries (and fails) to scrub the same PGs over and over
by Vladimir Brik 22 Jul '23

22 Jul '23

I have a PG that hasn't been scrubbed in over a month and not deep-scrubbed in over two months. I tried forcing with `ceph pg (deep-)scrub` but with no success. Looking at the logs of that PG's primary OSD it looks like every once in a while it attempts (and apparently fails) to scrub that PG, along with two others, over and over. For example: 2023-07-19T16:26:07.082 ... 24.3ea scrub starts 2023-07-19T16:26:10.284 ... 27.aae scrub starts 2023-07-19T16:26:11.169 ... 24.aa scrub starts 2023-07-19T16:26:12.153 ... 24.3ea scrub starts 2023-07-19T16:26:13.346 ... 27.aae scrub starts 2023-07-19T16:26:16.239 ... 24.aa scrub starts ... Lines like that are repeated throughout the log file. Has anyone seen something similar? How can I debug this? I am running 17.2.5 Vlad

2 3

July Ceph Science Virtual User Group
by Kevin Hrpcek 21 Jul '23

21 Jul '23

Hey all, We will be having a Ceph science/research/big cluster call on Wednesday July 26th. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research environments where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. NOTE: The change to using Jitsi for the meeting. We are no longer using the bluejeans meeting links. The ceph calendar event does not yet reflect this and has the wrong day as well. Pad URL: https://pad.ceph.com/p/Ceph_Science_User_Group_20230726 Ceph calendar event details: July 26th, 2023 14:00 UTC 4pm Central European 9am Central US Description: Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://meet.jit.si/ceph-science-wg Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS/TROPICS Space Science & Engineering Center University of Wisconsin-Madison

1 0

what is the point of listing "auth: unable to find a keyring on /etc/ceph/ceph.client nfs-ganesha
by Marc 21 Jul '23

21 Jul '23

I need some help understanding this. I have configured nfs-ganesha for cephfs using something like this in ganesha.conf FSAL { Name = CEPH; User_Id = "testing.nfs"; Secret_Access_Key = "AAAAAAAAAAAAAAA=="; } But I contstantly have these messages in de ganesha logs, 6x per user_id auth: unable to find a keyring on /etc/ceph/ceph.client.testing I thought this was a ganesha authentication order issue, but they[1] say it has to do with ceph. I am still on Nautilus so maybe this has been fixed in newer releases. I still have a hard time understanding why this is an issue of ceph (libraries). [1] https://github.com/nfs-ganesha/nfs-ganesha/issues/974

2 3

Regressed tail (p99.99+) write latency for RBD workloads in Quincy (vs. pre-Pacific)?
by Tyler Stachecki 21 Jul '23

21 Jul '23

Hi all, Has anyone else noticed any p99.99+ tail latency regression for RBD workloads in Quincy vs. pre-Pacific, i.e., before the kv_onode cache existed? Some notes from what I have seen thus far: * Restarting OSDs temporarily resolves the problem... then as activity accrues over time, the problem becomes appreciably worse * In comparing profiles of running OSDs, I've noticed that the bluestore block allocators are comparatively more active than in old releases (even though the fragmentation scores of the Quincy OSDs are far better in this case) * The new kv_onode cache often looks like it is often bursting at the seams, whereas the kv/meta/data caches have breathing room I am becoming increasingly confident that the observations are related, though I have not dived enough into bluestore to reason about how/when onodes are allocated on disk to complete the circle. Anyways, I am posting this to see if perhaps the defaults for the priority cache for the new kv_onode slab needs a slight nudge. You can observe them on OSDs with debug_bluestore 20/20 for a second and grepping for cache_size. Cheers, Tyler

1 0

index object in shard begins with hex 80
by Christopher Durham 20 Jul '23

20 Jul '23

Hi, I am using ceph 17.2.6 on rocky linux 8. I got a large omap object warning today. Ok, So I tracked it down to a shard for a bucket in the index pool of an s3 pool. However, when lisitng the omapkeys with: # rados -p pool.index listomapkeys .dir.zone.bucketid.xx.indexshardnumber it is clear that the problem is caused by many omapkeys with the following name format: <80>0_00004771163.3444695458.6 A hex dump of the output of the listomapkeys command above indicates that the first 'character' is indeed hex 80, but as there is no equivalent ascii for hex 80, I am not sure how to 'get at' those keys to see the values, delete them, etc. The index keys not of the format above appear to be fine, indicating s3 object names as expected. The rest of the index shards for the bucket are reasonable and have less than osd_deep_scrub_large_omap_object_key_threshold index objects , and the overall total of objects in the bucket is way less than osd_deep_scrub_large_omap_object_key_threshold*num_shards. These weird objects seem to be created occasionally.........????? Yes, the bucket is used heavily. Any advice here? -Chris

2 4

ceph-mgr ssh connections left open
by Wyll Ingersoll 20 Jul '23

20 Jul '23

Every night at midnight, our ceph-mgr daemons open up ssh connections to the other nodes and then leaves them open. Eventually they become zombies. I cannot figure out what module is causing this or how to turn it off. If left unchecked over days/weeks, the zombie ssh connections just keep growing, the only way to clear them is to restart ceph-mgr services. Any idea what is causing this or how it can be disabled? Example: ceph 1350387 1350373 7 Jul17 ? 01:19:39 /usr/bin/ceph-mgr -n mgr.mon03 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix ceph 1350548 1350387 0 Jul17 ? 00:00:01 ssh -C -F /tmp/cephadm-conf-d0khggdz -i /tmp/cephadm-identity-onf2msju -o ServerAliveInterval=7 -o ServerAliveCountMax=3 xxx(a)10.4.1.11 sudo python ceph 1350549 1350387 0 Jul17 ? 00:00:02 ssh -C -F /tmp/cephadm-conf-d0khggdz -i /tmp/cephadm-identity-onf2msju -o ServerAliveInterval=7 -o ServerAliveCountMax=3 xxx(a)10.4.1.41 sudo python ceph 1350550 1350387 0 Jul17 ? 00:00:01 ssh -C -F /tmp/cephadm-conf-d0khggdz -i /tmp/cephadm-identity-onf2msju -o ServerAliveInterval=7 -o ServerAliveCountMax=3 xxx(a)10.4.1.42 sudo python ceph 1350551 1350387 0 Jul17 ? 00:00:01 ssh -C -F /tmp/cephadm-conf-d0khggdz -i /tmp/cephadm-identity-onf2msju -o ServerAliveInterval=7 -o ServerAliveCountMax=3 xxx(a)10.4.1.22 sudo python ceph 1350552 1350387 0 Jul17 ? 00:00:01 ssh -C -F /tmp/cephadm-conf-d0khggdz -i /tmp/cephadm-identity-onf2msju -o ServerAliveInterval=7 -o ServerAliveCountMax=3 xxx(a)10.4.1.23 sudo python root 1350553 902 0 Jul17 ? 00:00:00 sshd: xxx [priv] ceph 1350554 1350387 0 Jul17 ? 00:00:01 ssh -C -F /tmp/cephadm-conf-d0khggdz -i /tmp/cephadm-identity-onf2msju -o ServerAliveInterval=7 -o ServerAliveCountMax=3 xxx(a)10.4.1.105 sudo pytho ceph 1350556 1350387 0 Jul17 ? 00:00:01 ssh -C -F /tmp/cephadm-conf-d0khggdz -i /tmp/cephadm-identity-onf2msju -o ServerAliveInterval=7 -o ServerAliveCountMax=3 xxx(a)10.4.1.21 sudo python ceph 1350557 1350387 0 Jul17 ? 00:00:01 ssh -C -F /tmp/cephadm-conf-d0khggdz -i /tmp/cephadm-identity-onf2msju -o ServerAliveInterval=7 -o ServerAliveCountMax=3 xxx(a)10.4.1.101 sudo pytho ceph 1350559 1350387 0 Jul17 ? 00:00:01 ssh -C -F /tmp/cephadm-conf-d0khggdz -i /tmp/cephadm-identity-onf2msju -o ServerAliveInterval=7 -o ServerAliveCountMax=3 xxx(a)10.4.1.102 sudo pytho Our current list of ceph-mgr modules enabled and default is: "always_on_modules": [ "balancer", "crash", "devicehealth", "orchestrator", "pg_autoscaler", "progress", "rbd_support", "status", "telemetry", "volumes" ], "enabled_modules": [ "cephadm", "dashboard", "diskprediction_local", "nfs", "prometheus", "restful" ],

2 3

Re: Workload that delete 100 M object daily via lifecycle
by Hoan Nguyen Van 20 Jul '23

20 Jul '23

You can enable debug lc to test and tuning rgw lc parameter.

3 2

rgw multisite sync not syncing data, error: RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data log shards
by Christian Rohmann 20 Jul '23

20 Jul '23

Hey ceph-users, I setup a multisite sync between two freshly setup Octopus clusters. In the first cluster I created a bucket with some data just to test the replication of actual data later. I then followed the instructions on https://docs.ceph.com/en/octopus/radosgw/multisite/#migrating-a-single-site… to add a second zone. Things went well and both zones are now happily reaching each other and the API endpoints are talking. Also the metadata is in sync already - both sides are happy and I can see bucket listings and users are "in sync": > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > metadata sync no sync (zone is master) > data sync source: c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > and on the other side ... > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > metadata sync syncing > full sync: 0/64 shards > incremental sync: 64/64 shards > metadata is caught up with master > data sync source: 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > also the newly created buckets (read: their metadata) is synced. What is apparently not working in the sync of actual data. Upon startup the radosgw on the second site shows: > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: start > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: realm > epoch=2 period id=f4553d7c-5cc5-4759-9253-9a22b051e736 > 2021-06-25T16:15:11.525+0000 7fe71dff3700 0 > RGW-SYNC:data:sync:init_data_sync_status: ERROR: failed to read remote > data log shards > also when issuing # radosgw-admin data sync init --source-zone obst-rgn it throws > 2021-06-25T16:20:29.167+0000 7f87c2aec080 0 > RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data > log shards Does anybody have any hints on where to look for what could be broken here? Thanks a bunch, Regards Christian

3 4

cephadm does not redeploy OSD
by Luis Domingues 20 Jul '23

20 Jul '23

Hi, We are running a ceph cluster managed with cephadm v16.2.13. Recently we needed to change a disk, and we replaced it with: ceph orch osd rm 37 --replace. It worked fine, the disk was drained and the OSD marked as destroy. However, after changing the disk, no OSD was created. Looking to the db device, the partition for db for OSD 37 was still there. So we destroyed it using: ceph-volume lvm zap --osd-id=37 --destroy. But we still have no OSD redeployed. Here we have our spec: --- service_type: osd service_id: osd-hdd placement: label: osds spec: data_devices: rotational: 1 encrypted: true db_devices: size: '1TB:2TB' db_slots: 12 And the disk looks good: HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS node05 /dev/nvme2n1 ssd SAMSUNG MZPLJ1T6HBJR-00007_S55JNG0R600357 1600G 12m ago LVM detected, locked node05 /dev/sdk hdd SEAGATE_ST10000NM0206_ZA21G2170000C7240KPF 10.0T Yes 12m ago And VG on db_device looks to have enough space: ceph-33b06f1a-f6f6-57cf-9ca8-6e4aa81caae0 1 11 0 wz--n- <1.46t 173.91g If I remove the db_devices and db_slots from the specs, and do a dry run, the orchestrator seems to see the new disk as available: ceph orch apply -i osd_specs.yml --dry-run WARNING! Dry-Runs are snapshots of a certain point in time and are bound to the current inventory setup. If any of these conditions change, the preview will be invalid. Please make sure to have a minimal timeframe between planning and applying the specs. #################### SERVICESPEC PREVIEWS #################### +---------+------+--------+-------------+ |SERVICE |NAME |ADD_TO |REMOVE_FROM | +---------+------+--------+-------------+ +---------+------+--------+-------------+ ################ OSDSPEC PREVIEWS ################ +---------+---------+-------------------------+----------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+---------+-------------------------+----------+----+-----+ |osd |osd-hdd |node05 |/dev/sdk |- |- | +---------+---------+-------------------------+----------+----+-----+ But as soon as I add db_devices back, the orchestrator is happy as it is, like there is nothing to do: ceph orch apply -i osd_specs.yml --dry-run WARNING! Dry-Runs are snapshots of a certain point in time and are bound to the current inventory setup. If any of these conditions change, the preview will be invalid. Please make sure to have a minimal timeframe between planning and applying the specs. #################### SERVICESPEC PREVIEWS #################### +---------+------+--------+-------------+ |SERVICE |NAME |ADD_TO |REMOVE_FROM | +---------+------+--------+-------------+ +---------+------+--------+-------------+ ################ OSDSPEC PREVIEWS ################ +---------+------+------+------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+------+------+------+----+-----+ I do not know why ceph will not use this disk, and I do not know where to look. It seems logs are not saying anything. And the weirdest thing, another disk was replaced on the same machine, and it went without any issues. Luis Domingues Proton AG

2 5

User + Dev Monthly Meeting happening tomorrow
by Laura Flores 19 Jul '23

19 Jul '23

Hi everyone, The User + Dev Monthly Meeting is happening tomorrow, July 20th at 2:00 PM UTC at this link: https://meet.jit.si/ceph-user-dev-monthly Please add any topics you'd like to discuss to the agenda: https://pad.ceph.com/p/ceph-user-dev-monthly-minutes Thanks, Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 1

Ceph Leadership Team Meeting, 2023-07-19 Minutes
by Patrick Donnelly 19 Jul '23

19 Jul '23

Hi folks, Today we discussed: - Reef is almost ready! The remaining issues are tracked in [1]. In particular, an epel9 package is holding back the release. - Vincent Hsu, Storage Group CTO of IBM, presented a proposal outline for a Ceph Foundation Client Council. This council would be composed of 10-25 invited significant operators or users of Ceph. The function of the council is to provide essential feedback on use-cases, pain-points, and successes arising during their use of Ceph. This feedback will be used to steer development and initiatives. More information on this will be forthcoming once the proposal is finalized. The monthly user <-> dev meeting will be reevaluated in light of this, possibly continuing on as usual. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 1

RBD image QoS rbd_qos_write_bps_limit and rbd_qos_bps_limit and mkfs performance
by Engelmann Florian 19 Jul '23

19 Jul '23

Hi, I noticed an incredible high performance drop with mkfs.ext4 (as well as mkfs.xfs) when setting (almost) "any" value for rbd_qos_write_bps_limit (or rbd_qos_bps_limit). Baseline: 4TB rbd volume rbd_qos_write_bps_limit = 0 mkfs.ext4: real 0m6.688s user 0m0.000s sys 0m0.006s 50GB/s: 4TB rbd volume rbd_qos_write_bps_limit = 53687091200 mkfs.ext4: real 1m22.217s user 0m0.009s sys 0m0.000s 5GB/s: 4TB rbd volume rbd_qos_write_bps_limit = 5368709120 mkfs.ext4: real 13m39.770s user 0m0.008s sys 0m0.034s 500MB/s: 4TB rbd volume rbd_qos_write_bps_limit = 524288000 mkfs.ext4: test still runing... I can provide the result if needed. The tests are running on a client vm (Ubuntu 22.04) using Qemu/libvirt. Using the same values with Qemu/libvirt QoS does not affect mkfs performance. https://libvirt.org/formatdomain.html#block-i-o-tuning Ceph Version: 16.2.11 Qemu: 6.2.0 Libvirt: 8.0.0 Kernel (hypervisor host): 5.19.0-35-generic librbd1 (hypervisor host): 17.2.5 Could anyone pls confirm and explain what's going on? All the best, Florian

2 3

librbd hangs during large backfill
by fb2cd0fc-933c-4cfe-b534-93d67045a088＠simplelogin.com 19 Jul '23

19 Jul '23

Starting on Friday, as part of adding a new pod of 12 servers, we initiated a reweight on roughly 384 drives; from 0.1 to 0.25. Something about the resulting large backfill is causing librbd to hang, requiring server restarts. The volumes are showing buffer i/o errors when this happens.We are currently using hybrid OSDs with both SSD and traditional spinning disks. The current status of the cluster is: ceph --version ceph version 14.2.22 Cluster Kernel 5.4.49-200 { "mon": { "ceph version 14.2.22 nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.22 nautilus (stable)": 3 }, "osd": { "ceph version 14.2.21 nautilus (stable)": 368, "ceph version 14.2.22 (stable)": 2055 }, "mds": {}, "rgw": { "ceph version 14.2.22 (stable)": 7 }, "overall": { "ceph version 14.2.21 (stable)": 368, "ceph version 14.2.22 (stable)": 2068 } } HEALTH_WARN, noscrub,nodeep-scrub flag(s) set. pgs: 6815703/11016906121 objects degraded (0.062%) 2814059622/11016906121 objects misplaced (25.543%). The client servers are on 3.10.0-1062.1.2.el7.x86_6 We have found a couple of issues that look relevant: https://tracker.ceph.com/issues/19385 https://tracker.ceph.com/issues/18807 Has anyone experienced anything like this before? Does anyone have any recommendations as to settings that can help alleviate this while the backfill completes? An example of the buffer ii/o errors: Jul 17 06:36:08 host8098 kernel: buffer_io_error: 22 callbacks suppressed Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-4, logical block 0, async page read Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-4, logical block 0, async page read Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-4, logical block 0, async page read Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-4, logical block 0, async page read Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-4, logical block 0, async page read Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-4, logical block 0, async page read Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-4, logical block 3, async page read Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-5, logical block 511984, async page read Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-6, logical block 3487657728, async page read Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-6, logical block 3487657729, async page read

4 4

1 PG stucked in "active+undersized+degraded for long time
by siddhit.renake＠nxtgen.com 19 Jul '23

19 Jul '23

Hello All, Ceph version: 14.2.5-382-g8881d33957 (8881d33957b54b101eae9c7627b351af10e87ee8) nautilus (stable) Issue: 1 PG stucked in "active+undersized+degraded for long time Degraded data redundancy: 44800/8717052637 objects degraded (0.001%), 1 pg degraded, 1 pg undersized #ceph pg dump_stuck PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 15.28f0 active+undersized+degraded [2147483647,343,355,415,426,640,302,392,78,202,607] 343 [2147483647,343,355,415,426,640,302,392,78,202,607] 343 PG Query: #ceph pg 15.28f0 query { "state": "active+undersized+degraded", "snap_trimq": "[]", "snap_trimq_len": 0, "epoch": 303362, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting_recovery_backfill": [ "78(8)", "202(9)", "302(6)", "343(1)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "info": { "pgid": "15.28f0s1", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598853, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "303161'598853", "reported_seq": "3594551", "reported_epoch": "303362", "state": "active+undersized+degraded", "last_fresh": "2023-06-20 19:03:59.135295", "last_change": "2023-06-20 15:11:12.569114", "last_active": "2023-06-20 19:03:59.135295", "last_peered": "2023-06-20 19:03:59.135295", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2023-06-20 15:11:12.569114", "last_became_peered": "2023-06-20 15:11:12.569114", "last_unstale": "2023-06-20 19:03:59.135295", "last_undegraded": "2023-06-20 15:11:10.430426", "last_fullsized": "2023-06-20 15:11:10.430154", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3028, "ondisk_log_size": 3028, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54989065178, "num_objects": 44800, "num_object_clones": 0, "num_object_copies": 492800, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44800, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44800, "num_whiteouts": 0, "num_read": 201078, "num_read_kb": 30408632, "num_write": 219335, "num_write_kb": 59459782, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44800 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, "peer_info": [ { "peer": "78(8)", "pgid": "15.28f0s8", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "202(9)", "pgid": "15.28f0s9", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "302(6)", "pgid": "15.28f0s6", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "355(2)", "pgid": "15.28f0s2", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "392(7)", "pgid": "15.28f0s7", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "415(3)", "pgid": "15.28f0s3", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "426(4)", "pgid": "15.28f0s4", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "607(10)", "pgid": "15.28f0s10", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "640(5)", "pgid": "15.28f0s5", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } } ], "recovery_state": [ { "name": "Started/Primary/Active", "enter_time": "2023-06-20 15:11:10.430274", "might_have_unfound": [], "recovery_progress": { "backfill_targets": [], "waiting_on_backfill": [], "last_backfill_started": "MIN", "backfill_info": { "begin": "MIN", "end": "MIN", "objects": [] }, "peer_backfill_info": [], "backfills_in_flight": [], "recovering": [], "pg_backend": { "recovery_ops": [], "read_ops": [] } }, "scrub": { "scrubber.epoch_start": "0", "scrubber.active": false, "scrubber.state": "INACTIVE", "scrubber.start": "MIN", "scrubber.end": "MIN", "scrubber.max_end": "MIN", "scrubber.subset_last_update": "0'0", "scrubber.deep": false, "scrubber.waiting_on_whom": [] } }, { "name": "Started", "enter_time": "2023-06-20 15:11:09.294876" } ], "agent_state": {} } Require assistance to to make this PG active+clean

3 4

Another Pacific point release?
by Ponnuvel Palaniyappan 19 Jul '23

19 Jul '23

Hi, The typical EOL date (2023-06-01) has already passed for Pacific. Just wondering if there's going to be another Pacific point release (16.2.14) in the pipeline. -- Regards, Ponnuvel P

4 3

replacing all disks in a stretch mode ceph cluster
by Zoran Bošnjak 19 Jul '23

19 Jul '23

Hello ceph users, my ceph configuration is - ceph version 17.2.5 on ubuntu 20.04 - stretch mode - 2 rooms with OSDs and monitors + additional room for the tiebreaker monitor - 4 OSD servers in each room - 6 OSDs per OSD server - ceph installation/administration is manual (without ansible, orch... or any other tool like this) Ceph health is currently OK. Raw usage is around 60%, Pools usage is below 75% I need to replace all OSD disks in the cluster with larger capacity disks (500G to 1000G). So the eventual configuration will contain the same number of OSDs and servers. I understand I can replace OSDs one by one, following the documented procedure (removing old and adding new OSD to the configuration) and waiting for health OK. But in this case, ceph will probably copy data around like crazy after each step. So, my question is: What is the recommended procedure in this case of replacing ALL disks and keeping the ceph operational during the upgrade? In particular: Should I use any of "nobackfill, norebalance, norecover..." flags during the process? If yes, which? Should I do one OSD at the time, server at the time or even room at the time? Thanks for the suggestions. regards, Zoran

3 2

Workload that delete 100 M object daily via lifecycle
by Ha Nguyen Van 19 Jul '23

19 Jul '23

Hi Experts, We plan to setup a Ceph Object to support a S3 workload, that will need to delete 100M file daily via lifecycle. Appreciate your suggestion and setting to handle this kind of scenario Best Regards, Ha

4 5

Re: librbd hangs during large backfill
by Jack Hayhurst 18 Jul '23

18 Jul '23

We did have a peering storm, we're past that portion of the backfill and still experiencing new instances of rbd volumes hanging. It is for sure not just the peering storm. We've got 22.184% objects misplaced yet, with a bunch of pgs left to backfill (like 75k). Our rbd poll is using about 1.7PiB of storage, so we're looking at like 370TiB yet to backfill, rough estimate. This specific pool is using replicated encoding, with size=3. RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 21 PiB 11 PiB 10 PiB 10 PiB 48.73 TOTAL 21 PiB 11 PiB 10 PiB 10 PiB 48.73 POOLS: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL pool1 4 32768 574 TiB 147.16M 1.7 PiB 68.87 260 TiB We did see a lot of rbd volumes that hung, often giving the buffer i/o errors previously sent - whether that was the peering storm or backfills is uncertain. As suggested, we've already been detaching/reattaching the rbd volumes, pushing the primary active osd for pgs to another, and sometimes rebooting the kernel on the vm to clear the io queue. A combination of those brings the rbd volume block device back for a while. We're no longer in a peering storm and we're seeing the rbd volumes going into an unresponsive state again - including osds where they were unresponsive, we did things and got them responsive, and then they went unresponsive again. All pgs are in an active state, some active+remapped+backfilling, some active+undersized+remapped+backfilling, etc. We also run the object gateway off the same cluster with the same backfill, the object gateway is not experiencing issues. Also the osds patricipating in the backfill are not saturated with i/o, or seeing abnormal load for our usual backfill operations. But with the continuing backfill, we're seeing rbd volumes on active pgs going back into a blocked state. We can do about the same with detaching the volume / bouncing the pg to a new primary acting osd, but we'd rather have these stop going unresponsive in the first place. Any suggestions towards that direction are greatly appreciated.

1 0

CEPHADM_FAILED_SET_OPTION
by Arnoud de Jonge 18 Jul '23

18 Jul '23

Hi, After having set up RadosGW with keystone authentication the cluster shows this warning: # ceph health detail HEALTH_WARN Failed to set 1 option(s) [WRN] CEPHADM_FAILED_SET_OPTION: Failed to set 1 option(s) Failed to set rgw.fra option rgw_keystone_implicit_tenants: config set failed: error parsing value: 'True' is not one of the permitted values: false, true, swift, s3, both, 0, 1, none retval: -22 I might have made a typo but that has been corrected. # ceph config dump |grep rgw_keystone_implicit_tenants client.rgw.fra.controller1.lushdc advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller1.pznmuf advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller1.tdrqot advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller2.ndxaet advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller2.rodqxh advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller2.wyhjuk advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller3.boasgr advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller3.lkczbl advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller3.qxctee advanced rgw_keystone_implicit_tenants true * # ceph --version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Any idea on how to fix this issue? Thanks, Arnoud.

2 1

OSD memory usage after cephadm adoption
by Luis Domingues 17 Jul '23

17 Jul '23

Hi everyone, We recently migrate a cluster from ceph-ansible to cephadm. Everything went as expected. But now we have some alerts on high memory usage. Cluster is running ceph 16.2.13. Of course, after adoption OSDs ended up in the <unmanaged> zone: NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd 88 7m ago - <unmanaged> But the weirdest thing I observed, is that the OSDs seem to use more memory that the mem limit: NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID osd.0 <node> running (5d) 2m ago 5d 19.7G 6400M 16.2.13 327f301eff51 ca07fe74a0fa osd.1 <node> running (5d) 2m ago 5d 7068M 6400M 16.2.13 327f301eff51 6223ed8e34e9 osd.10 <node> running (5d) 10m ago 5d 7235M 6400M 16.2.13 327f301eff51 073ddc0d7391 osd.100 <node> running (5d) 2m ago 5d 7118M 6400M 16.2.13 327f301eff51 b7f9238c0c24 Does anybody knows why OSDs would use more memory than the limit? Thanks Luis Domingues Proton AG

4 7

Re: RBD with PWL cache shows poor performance compared to cache device
by Matthew Booth 16 Jul '23

16 Jul '23

On Wed, 28 Jun 2023 at 22:44, Ilya Dryomov <idryomov(a)redhat.com> wrote: >> ** TL;DR >> >> In testing, the write latency performance of a PWL-cache backed RBD >> disk was 2 orders of magnitude worse than the disk holding the PWL >> cache. >> >> ** Summary >> >> I was hoping that PWL cache might be a good solution to the problem of >> write latency requirements of etcd when running a kubernetes control >> plane on ceph. Etcd is extremely write latency sensitive and becomes >> unstable if write latency is too high. The etcd workload can be >> characterised by very small (~4k) writes with a queue depth of 1. >> Throughput, even on a busy system, is normally very low. As etcd is >> distributed and can safely handle the loss of un-flushed data from a >> single node, a local ssd PWL cache for etcd looked like an ideal >> solution. > > > Right, this is exactly the use case that the PWL cache is supposed to address. Good to know! >> My expectation was that adding a PWL cache on a local SSD to an >> RBD-backed would improve write latency to something approaching the >> write latency performance of the local SSD. However, in my testing >> adding a PWL cache to an rbd-backed VM increased write latency by >> approximately 4x over not using a PWL cache. This was over 100x more >> than the write latency performance of the underlying SSD. >> >> My expectation was based on the documentation here: >> https://docs.ceph.com/en/quincy/rbd/rbd-persistent-write-log-cache/ >> >> “The cache provides two different persistence modes. In >> persistent-on-write mode, the writes are completed only when they are >> persisted to the cache device and will be readable after a crash. In >> persistent-on-flush mode, the writes are completed as soon as it no >> longer needs the caller’s data buffer to complete the writes, but does >> not guarantee that writes will be readable after a crash. The data is >> persisted to the cache device when a flush request is received.” >> >> ** Method >> >> 2 systems, 1 running single-node Ceph Quincy (17.2.6), the other >> running libvirt and mounting a VM’s disk with librbd (also 17.2.6) >> from the first node. >> >> All performance testing is from the libvirt system. I tested write >> latency performance: >> >> * Inside the VM without a PWL cache >> * Of the PWL device directly from the host (direct to filesystem, no VM) >> * Inside the VM with a PWL cache >> >> I am testing with fio. Specifically I am running a containerised test, >> executed with: >> podman run --volume .:/var/lib/etcd:Z quay.io/openshift-scale/etcd-perf >> >> This container runs: >> fio --rw=write --ioengine=sync --fdatasync=1 >> --directory=/var/lib/etcd --size=100m --bs=8000 --name=etcd_perf >> --output-format=json --runtime=60 --time_based=1 >> >> And extracts sync.lat_ns.percentile["99.000000"] > > > Matthew, do you have the rest of the fio output captured? It would be interesting to see if it's just the 99th percentile that is bad or the PWL cache is worse in general. Sure. With PWL cache: https://paste.openstack.org/show/820504/ Without PWL cache: https://paste.openstack.org/show/b35e71zAwtYR2hjmSRtR/ With PWL cache, 'rbd_cache'=false: https://paste.openstack.org/show/byp8ZITPzb3r9bb06cPf/ >> ** Results >> >> All results were stable across multiple runs within a small margin of error. >> >> * rbd no cache: 1417216 ns >> * pwl cache device: 44288 ns >> * rbd with pwl cache: 5210112 ns >> >> Note that by adding a PWL cache we increase write latency by >> approximately 4x, which is more than 100x than the underlying device. >> >> ** Hardware >> >> 2 x Dell R640s, each with Xeon Silver 4216 CPU @ 2.10GHz and 192G RAM >> Storage under test: 2 x SAMSUNG MZ7KH480HAHQ0D3 SSDs attached to PERC >> H730P Mini (Embedded) >> >> OS installed on rotational disks >> >> N.B. Linux incorrectly detects these disks as rotational, which I >> assume relates to weird behaviour by the PERC controller. I remembered >> to manually correct this on the ‘client’ machine for the PWL cache, >> but at OSD configuration time ceph would have detected them as >> rotational. They are not rotational. >> >> ** Ceph Configuration >> >> CentOS Stream 9 >> >> # ceph version >> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy >> (stable) >> >> Single node installation with cephadm. 2 OSDs, one on each SSD. >> 1 pool with size 2 >> >> ** Client Configuration >> >> Fedora 38 >> Librbd1-17.2.6-3.fc38.x86_64 >> >> PWL cache is XFS filesystem with 4k block size, matching the >> underlying device. The filesystem uses the whole block device. There >> is no other load on the system. >> >> ** RBD Configuration >> >> # rbd config image list libvirt-pool/pwl-test | grep cache >> rbd_cache true > > > I wonder if rbd_cache should have been set to false here to disable the default volatile cache. Other than that, I don't see anything obviously wrong with the configuration at first sight. I added some full output for this above. > > -- > Ilya > >> config >> rbd_cache_block_writes_upfront false >> config >> rbd_cache_max_dirty 25165824 >> config >> rbd_cache_max_dirty_age 1.000000 >> config >> rbd_cache_max_dirty_object 0 >> config >> rbd_cache_policy writeback >> pool >> rbd_cache_size 33554432 >> config >> rbd_cache_target_dirty 16777216 >> config >> rbd_cache_writethrough_until_flush true >> pool >> rbd_parent_cache_enabled false >> config >> rbd_persistent_cache_mode ssd >> pool >> rbd_persistent_cache_path /var/lib/libvirt/images/pwl >> pool >> rbd_persistent_cache_size 1073741824 >> config >> rbd_plugins pwl_cache >> pool >> >> # rbd status libvirt-pool/pwl-test >> Watchers: >> watcher=10.1.240.27:0/1406459716 client.14475 >> cookie=140282423200720 >> Persistent cache state: >> host: dell-r640-050 >> path: >> /var/lib/libvirt/images/pwl/rbd-pwl.libvirt-pool.37e947fd216b.pool >> size: 1 GiB >> mode: ssd >> stats_timestamp: Mon Jun 26 11:29:21 2023 >> present: true empty: false clean: true >> allocated: 180 MiB >> cached: 135 MiB >> dirty: 0 B >> free: 844 MiB >> hits_full: 1 / 0% >> hits_partial: 3 / 0% >> misses: 21952 >> hit_bytes: 6 KiB / 0% >> miss_bytes: 349 MiB -- Matthew Booth

4 16

OSD crash after server reboot
by pedro.martin＠vodafone.com 15 Jul '23

15 Jul '23

Hi We are facing error with OSD crash after reboot of the server where it is installed We rebooted servers in our ceph cluster for a patching and after rebooting two OSD where crashing One of them finally recovered but the other is still down Cluster is currently rebalancing objects : # ceph status cluster: id: 62d303dc-e46b-4863-93b3-7ee995594dd1 health: HEALTH_ERR clients are using insecure global_id reclaim mons are allowing insecure global_id reclaim 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged mons ac,ae,v are low on available space 4/1100134 objects unfound (0.000%) 1 osds down 1 host (1 osds) down 1 nearfull osd(s) Reduced data availability: 9 pgs inactive, 8 pgs down, 1 pg incomplete Possible data damage: 4 pgs recovery_unfound Degraded data redundancy: 12/2204629 objects degraded (0.001%), 5 pgs degraded, 14 pgs undersized 13 pool(s) nearfull 236 daemons have recently crashed services: mon: 3 daemons, quorum v,ac,ae (age 13h) mgr: a(active, since 24m) mds: myfs:0/1 2 up:standby, 1 damaged osd: 7 osds: 6 up (since 68s), 7 in (since 18m); 131 remapped pgs rgw: 1 daemon active (harbor.object.store.a) task status: data: pools: 13 pools, 337 pgs objects: 1.10M objects, 2.3 TiB usage: 5.6 TiB used, 3.0 TiB / 8.5 TiB avail pgs: 2.671% pgs not active 12/2204629 objects degraded (0.001%) 616501/2204629 objects misplaced (27.964%) 4/1100134 objects unfound (0.000%) 179 active+clean 78 active+clean+remapped 52 active+remapped+backfill_wait 13 active+undersized 8 down 4 active+recovery_unfound+degraded 1 active+undersized+degraded 1 incomplete 1 active+remapped+backfilling io: client: 8.0 KiB/s wr, 0 op/s rd, 0 op/s wr recovery: 1.4 MiB/s, 7 objects/s These are the last lines from the OSD crash log. We are no sure why this is crashing :( 4(1) r=-1 lpr=69148 pi=[69137,69148)/2 crt=7010'401 mlcod 0'0 remapped NOTIFY m=276 mbc={}] exit Started/Stray 1.009572 7 0.001960 -19> 2023-07-15T06:58:40.723+0000 7f444a0b0700 3 osd.0 69149 handle_osd_map epochs [69149,69149], i have 69149, src has [65617,69149] -18> 2023-07-15T06:58:40.723+0000 7f444109e700 5 osd.0 pg_epoch: 69149 pg[10.14( v 9188'79 (0'0,9188'79] local-lis/les=69148/69149 n=4 ec=8825/8825 lis/c=69148/69143 les/c/f=69149/69144/0 sis=69148) [0,3] r=0 lpr=69148 pi=[66972,69148)/5 crt=9188'79 lcod 0'0 mlcod 0'0 active mbc={}] enter Started/Primary/Active/Clean -17> 2023-07-15T06:58:40.723+0000 7f444009c700 5 osd.0 pg_epoch: 69149 pg[8.7s0( v 7010'401 lc 0'0 (0'0,7010'401] local-lis/les=0/0 n=151 ec=77/77 lis/c=69144/69144 les/c/f=69145/69145/0 sis=69148) [0,4,6]/[NONE,4,6]p4(1) r=-1 lpr=69148 pi=[69137,69148)/2 crt=7010'401 mlcod 0'0 remapped NOTIFY m=276 mbc={}] enter Started/ReplicaActive -16> 2023-07-15T06:58:40.723+0000 7f444009c700 5 osd.0 pg_epoch: 69149 pg[8.7s0( v 7010'401 lc 0'0 (0'0,7010'401] local-lis/les=0/0 n=151 ec=77/77 lis/c=69144/69144 les/c/f=69145/69145/0 sis=69148) [0,4,6]/[NONE,4,6]p4(1) r=-1 lpr=69148 pi=[69137,69148)/2 crt=7010'401 mlcod 0'0 remapped NOTIFY m=276 mbc={}] enter Started/ReplicaActive/RepNotRecovering -15> 2023-07-15T06:58:40.723+0000 7f444a0b0700 3 osd.0 69149 handle_osd_map epochs [69149,69149], i have 69149, src has [65617,69149] -14> 2023-07-15T06:58:40.724+0000 7f444089d700 5 osd.0 pg_epoch: 69149 pg[10.1f( v 9092'7 lc 8831'4 (0'0,9092'7] local-lis/les=69148/69149 n=2 ec=8825/8825 lis/c=69148/69144 les/c/f=69149/69145/0 sis=69148) [6,0] r=1 lpr=69148 pi=[69137,69148)/2 luod=0'0 crt=9092'7 lcod 0'0 mlcod 0'0 active m=2 mbc={}] exit Started/ReplicaActive/RepNotRecovering 0.001805 4 0.000066 -13> 2023-07-15T06:58:40.724+0000 7f444089d700 5 osd.0 pg_epoch: 69149 pg[10.1f( v 9092'7 lc 8831'4 (0'0,9092'7] local-lis/les=69148/69149 n=2 ec=8825/8825 lis/c=69148/69144 les/c/f=69149/69145/0 sis=69148) [6,0] r=1 lpr=69148 pi=[69137,69148)/2 luod=0'0 crt=9092'7 lcod 0'0 mlcod 0'0 active m=2 mbc={}] enter Started/ReplicaActive/RepWaitRecoveryReserved -12> 2023-07-15T06:58:40.724+0000 7f444089d700 5 osd.0 pg_epoch: 69149 pg[10.1f( v 9092'7 lc 8831'4 (0'0,9092'7] local-lis/les=69148/69149 n=2 ec=8825/8825 lis/c=69148/69144 les/c/f=69149/69145/0 sis=69148) [6,0] r=1 lpr=69148 pi=[69137,69148)/2 luod=0'0 crt=9092'7 lcod 0'0 mlcod 0'0 active m=2 mbc={}] exit Started/ReplicaActive/RepWaitRecoveryReserved 0.000043 1 0.000053 -11> 2023-07-15T06:58:40.724+0000 7f444089d700 5 osd.0 pg_epoch: 69149 pg[10.1f( v 9092'7 lc 8831'4 (0'0,9092'7] local-lis/les=69148/69149 n=2 ec=8825/8825 lis/c=69148/69144 les/c/f=69149/69145/0 sis=69148) [6,0] r=1 lpr=69148 pi=[69137,69148)/2 luod=0'0 crt=9092'7 lcod 0'0 mlcod 0'0 active m=2 mbc={}] enter Started/ReplicaActive/RepRecovering -10> 2023-07-15T06:58:40.725+0000 7f444009c700 5 osd.0 pg_epoch: 69149 pg[7.2( v 268'5 (79'3,268'5] lb MIN local-lis/les=69140/69141 n=0 ec=71/68 lis/c=69147/69143 les/c/f=69148/69144/0 sis=69147) [0,2]/[2,5] r=-1 lpr=69147 pi=[69137,69147)/1 luod=0'0 crt=268'5 lcod 0'0 mlcod 0'0 active+remapped mbc={}] exit Started/ReplicaActive/RepNotRecovering 1.010921 6 0.000077 -9> 2023-07-15T06:58:40.725+0000 7f444009c700 5 osd.0 pg_epoch: 69149 pg[7.2( v 268'5 (79'3,268'5] lb MIN local-lis/les=69140/69141 n=0 ec=71/68 lis/c=69147/69143 les/c/f=69148/69144/0 sis=69147) [0,2]/[2,5] r=-1 lpr=69147 pi=[69137,69147)/1 luod=0'0 crt=268'5 lcod 0'0 mlcod 0'0 active+remapped mbc={}] enter Started/ReplicaActive/RepWaitBackfillReserved -8> 2023-07-15T06:58:40.726+0000 7f444a0b0700 3 osd.0 69149 handle_osd_map epochs [69149,69149], i have 69149, src has [65617,69149] -7> 2023-07-15T06:58:40.726+0000 7f444009c700 5 osd.0 pg_epoch: 69149 pg[12.c( v 65902'349780 (65642'347213,65902'349780] lb 12:30005f02:::1000060a6a4.00000000:head local-lis/les=67222/67223 n=1903 ec=9097/9097 lis/c=69147/69143 les/c/f=69148/69144/0 sis=69147) [0,5]/[5,1] r=-1 lpr=69147 pi=[67170,69147)/2 luod=0'0 crt=65902'349780 mlcod 0'0 active+remapped mbc={}] exit Started/ReplicaActive/RepRecovering 0.996833 5 0.000101 -6> 2023-07-15T06:58:40.726+0000 7f444009c700 5 osd.0 pg_epoch: 69149 pg[12.c( v 65902'349780 (65642'347213,65902'349780] lb 12:30005f02:::1000060a6a4.00000000:head local-lis/les=67222/67223 n=1903 ec=9097/9097 lis/c=69147/69143 les/c/f=69148/69144/0 sis=69147) [0,5]/[5,1] r=-1 lpr=69147 pi=[67170,69147)/2 luod=0'0 crt=65902'349780 mlcod 0'0 active+remapped mbc={}] enter Started/ReplicaActive/RepNotRecovering -5> 2023-07-15T06:58:40.726+0000 7f444009c700 5 osd.0 pg_epoch: 69149 pg[8.7s0( v 7010'401 lc 0'0 (0'0,7010'401] local-lis/les=0/0 n=151 ec=77/77 lis/c=69148/69144 les/c/f=69149/69145/0 sis=69148) [0,4,6]/[NONE,4,6]p4(1) r=-1 lpr=69148 pi=[69137,69148)/2 luod=0'0 crt=7010'401 mlcod 0'0 active+remapped m=276 mbc={}] exit Started/ReplicaActive/RepNotRecovering 0.002781 3 0.000076 -4> 2023-07-15T06:58:40.726+0000 7f444009c700 5 osd.0 pg_epoch: 69149 pg[8.7s0( v 7010'401 lc 0'0 (0'0,7010'401] local-lis/les=0/0 n=151 ec=77/77 lis/c=69148/69144 les/c/f=69149/69145/0 sis=69148) [0,4,6]/[NONE,4,6]p4(1) r=-1 lpr=69148 pi=[69137,69148)/2 luod=0'0 crt=7010'401 mlcod 0'0 active+remapped m=276 mbc={}] enter Started/ReplicaActive/RepWaitRecoveryReserved -3> 2023-07-15T06:58:40.727+0000 7f444089d700 5 osd.0 pg_epoch: 69149 pg[12.1( v 65904'388229 (65626'385027,65904'388229] lb MIN local-lis/les=67170/67171 n=1876 ec=9097/9097 lis/c=69147/69143 les/c/f=69148/69144/0 sis=69147) [5,0]/[5,3] r=-1 lpr=69147 pi=[67170,69147)/3 luod=0'0 crt=65904'388229 lcod 0'0 mlcod 0'0 active+remapped mbc={}] exit Started/ReplicaActive/RepNotRecovering 1.009973 6 0.000367 -2> 2023-07-15T06:58:40.727+0000 7f444089d700 5 osd.0 pg_epoch: 69149 pg[12.1( v 65904'388229 (65626'385027,65904'388229] lb MIN local-lis/les=67170/67171 n=1876 ec=9097/9097 lis/c=69147/69143 les/c/f=69148/69144/0 sis=69147) [5,0]/[5,3] r=-1 lpr=69147 pi=[67170,69147)/3 luod=0'0 crt=65904'388229 lcod 0'0 mlcod 0'0 active+remapped mbc={}] enter Started/ReplicaActive/RepWaitBackfillReserved -1> 2023-07-15T06:58:40.729+0000 7f444109e700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.13/rpm/el8/BUILD/ceph-15.2.13/src/osd/PGLog.cc: In function 'void PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)' thread 7f444109e700 time 2023-07-15T06:58:40.727106+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.13/rpm/el8/BUILD/ceph-15.2.13/src/osd/PGLog.cc: 369: FAILED ceph_assert(log.head >= olog.tail && olog.head >= log.tail) ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x563620b4dbd8] 2: (()+0x507df2) [0x563620b4ddf2] 3: (PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0x1ca1) [0x563620d121f1] 4: (PeeringState::merge_log(ceph::os::Transaction&, pg_info_t&, pg_log_t&, pg_shard_t)+0x75) [0x563620e982c5] 5: (PeeringState::Stray::react(MLogRec const&)+0xcc) [0x563620ed308c] 6: (boost::statechart::simple_state<PeeringState::Stray, PeeringState::Started, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xa5) [0x563620efefb5] 7: (boost::statechart::state_machine<PeeringState::PeeringMachine, PeeringState::Initial, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x5b) [0x563620cf22ab] 8: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PeeringCtx&)+0x2d1) [0x563620ce48a1] 9: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x29c) [0x563620c5bc7c] 10: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x56) [0x563620e8d906] 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x12ef) [0x563620c4e92f] 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x56362128ef84] 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x563621291be4] 14: (()+0x814a) [0x7f446112c14a] 15: (clone()+0x43) [0x7f445fe63f23] 0> 2023-07-15T06:58:40.733+0000 7f444109e700 -1 *** Caught signal (Aborted) ** in thread 7f444109e700 thread_name:tp_osd_tp ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable) 1: (()+0x12b20) [0x7f4461136b20] 2: (gsignal()+0x10f) [0x7f445fd9e7ff] 3: (abort()+0x127) [0x7f445fd88c35] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x563620b4dc29] 5: (()+0x507df2) [0x563620b4ddf2] 6: (PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0x1ca1) [0x563620d121f1] 7: (PeeringState::merge_log(ceph::os::Transaction&, pg_info_t&, pg_log_t&, pg_shard_t)+0x75) [0x563620e982c5] 8: (PeeringState::Stray::react(MLogRec const&)+0xcc) [0x563620ed308c] 9: (boost::statechart::simple_state<PeeringState::Stray, PeeringState::Started, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xa5) [0x563620efefb5] 10: (boost::statechart::state_machine<PeeringState::PeeringMachine, PeeringState::Initial, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x5b) [0x563620cf22ab] 11: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PeeringCtx&)+0x2d1) [0x563620ce48a1] 12: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x29c) [0x563620c5bc7c] 13: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x56) [0x563620e8d906] 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x12ef) [0x563620c4e92f] 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x56362128ef84] 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x563621291be4] 17: (()+0x814a) [0x7f446112c14a] 18: (clone()+0x43) [0x7f445fe63f23] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_rwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test -2/-2 (syslog threshold) 99/99 (stderr threshold) --- pthread ID / name mapping for recent threads --- 7f443e899700 / osd_srv_heartbt 7f443f09a700 / tp_osd_tp 7f443f89b700 / tp_osd_tp 7f444009c700 / tp_osd_tp 7f444089d700 / tp_osd_tp 7f444109e700 / tp_osd_tp 7f444a0b0700 / ms_dispatch 7f444b0b2700 / rocksdb:dump_st 7f444beae700 / fn_anonymous 7f444ceb0700 / cfin 7f444e28c700 / safe_timer 7f444f28e700 / ms_dispatch 7f4451eba700 / bstore_mempool 7f44570ca700 / fn_anonymous 7f44588cd700 / safe_timer 7f445a141700 / safe_timer 7f445a942700 / signal_handler 7f445b944700 / admin_socket 7f445c145700 / service 7f445c946700 / msgr-worker-2 7f445d147700 / msgr-worker-1 7f445d948700 / msgr-worker-0 7f44633cef00 / ceph-osd max_recent 10000 max_new 1000 log_file /var/lib/ceph/crash/2023-07-15T06:58:40.734237Z_21e01469-d6d6-4be4-b913-f9cc55a7ab22/log --- end dump of recent events --- I appreciate any help to point us to a possible troubleshooting path :) thanks a lot and kind regards,

1 0

Re: Ceph Mgr/Dashboard Python depedencies: a new approach
by Casey Bodley 14 Jul '23

14 Jul '23

hi Ernesto and lists, > [1] https://github.com/ceph/ceph/pull/47501 are we planning to backport this to quincy so we can support centos 9 there? enabling that upgrade path on centos 9 was one of the conditions for dropping centos 8 support in reef, which i'm still keen to do if not, can we find another resolution to https://tracker.ceph.com/issues/58832? as i understand it, all of those python packages exist in centos 8. do we know why they were dropped for centos 9? have we looked into making those available in epel? (cc Ken and Kaleb) On Fri, Sep 2, 2022 at 12:01 PM Ernesto Puerta <epuertat(a)redhat.com> wrote: > > Hi Kevin, > >> >> Isn't this one of the reasons containers were pushed, so that the packaging isn't as big a deal? > > > Yes, but the Ceph community has a strong commitment to provide distro packages for those users who are not interested in moving to containers. > >> Is it the continued push to support lots of distros without using containers that is the problem? > > > If not a problem, it definitely makes it more challenging. Compiled components often sort this out by statically linking deps whose packages are not widely available in distros. The approach we're proposing here would be the closest equivalent to static linking for interpreted code (bundling). > > Thanks for sharing your questions! > > Kind regards, > Ernesto > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io

4 10

cephadm upgrade 16.2.10 to 16.2.11: osds crash and get stuck restarting
by Zakhar Kirpichenko 14 Jul '23

14 Jul '23

Hi, Attempted to upgrade 16.2.10 to 16.2.11, 2 OSDs out of many started crashing in a loop on the very 1st host: Jan 25 23:07:53 ceph01 bash[2553123]: Uptime(secs): 0.0 total, 0.0 interval Jan 25 23:07:53 ceph01 bash[2553123]: Flush(GB): cumulative 0.000, interval 0.000 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(GB): cumulative 0.000, interval 0.000 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(Total Files): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(L0 Files): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(Keys): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Jan 25 23:07:53 ceph01 bash[2553123]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Jan 25 23:07:53 ceph01 bash[2553123]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count Jan 25 23:07:53 ceph01 bash[2553123]: ** File Read Latency Histogram By Level [P] ** Jan 25 23:07:53 ceph01 bash[2553123]: debug -10> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986439) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -9> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986493) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -8> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986500) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -7> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986505) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -6> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: [compaction/compaction_job.cc:1676] [O-2] [JOB 9] Compacting 4@0 + 2@1 files to L1, score 1.0 0 Jan 25 23:07:53 ceph01 bash[2553123]: debug -5> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: [compaction/compaction_job.cc:1680] [O-2] Compaction start summary: Base version 33 Base level 0, inputs: [649058(959KB) 649046(1510KB) 649024(1323KB) 649002(1396KB)], [648981(66MB) 648982(52MB)] Jan 25 23:07:53 ceph01 bash[2553123]: debug -4> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1674688072986547, "job": 9, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [649058, 649046, 649024, 649002], "files_L1": [648981, 648982], "score": 1, "input_data_size": 129161327} Jan 25 23:07:53 ceph01 bash[2553123]: debug -3> 2023-01-25T23:07:52.990+0000 7f7c8b66e700 -1 bdev(0x5619bf0ce400 /var/lib/ceph/osd/ceph-3/block) _aio_thread got r=-1 ((1) Operation not permitted) Jan 25 23:07:53 ceph01 bash[2553123]: debug -2> 2023-01-25T23:07:52.990+0000 7f7c8b66e700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.11/rpm/el8/BUILD/ceph-16.2.11/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f7c8b66e700 time 2023-01-25T23:07:52.993976+0000 Jan 25 23:07:53 ceph01 bash[2553123]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.11/rpm/el8/BUILD/ceph-16.2.11/src/blk/kernel/KernelDevice.cc: 604: ceph_abort_msg("Unexpected IO error. This may suggest HW issue. Please check your dmesg!") Jan 25 23:07:53 ceph01 bash[2553123]: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) Jan 25 23:07:53 ceph01 bash[2553123]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x5619b2fc1adc] Jan 25 23:07:53 ceph01 bash[2553123]: 2: (KernelDevice::_aio_thread()+0x1285) [0x5619b3b2a4e5] Jan 25 23:07:53 ceph01 bash[2553123]: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5619b3b357b1] Jan 25 23:07:53 ceph01 bash[2553123]: 4: /lib64/libpthread.so.0(+0x81ca) [0x7f7c978851ca] Jan 25 23:07:53 ceph01 bash[2553123]: 5: clone() Jan 25 23:07:53 ceph01 bash[2553123]: debug -1> 2023-01-25T23:07:52.994+0000 7f7c8b66e700 -1 *** Caught signal (Aborted) ** Jan 25 23:07:53 ceph01 bash[2553123]: in thread 7f7c8b66e700 thread_name:bstore_aio Jan 25 23:07:53 ceph01 bash[2553123]: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) Jan 25 23:07:53 ceph01 bash[2553123]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7c9788fcf0] Jan 25 23:07:53 ceph01 bash[2553123]: 2: gsignal() Jan 25 23:07:53 ceph01 bash[2553123]: 3: abort() Jan 25 23:07:53 ceph01 bash[2553123]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x5619b2fc1bad] Jan 25 23:07:53 ceph01 bash[2553123]: 5: (KernelDevice::_aio_thread()+0x1285) [0x5619b3b2a4e5] Jan 25 23:07:53 ceph01 bash[2553123]: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5619b3b357b1] Jan 25 23:07:53 ceph01 bash[2553123]: 7: /lib64/libpthread.so.0(+0x81ca) [0x7f7c978851ca] OSD kept crashing until the host reboot, an OSD restart wouldn't help. This hasn't happened during any previous upgrades, so was a rather unexpected development. Unclear what caused this, but a host reboot seems to have fixed it. It happened to 1 other OSD on another host, exactly the same symptoms, also solved by a reboot. Best regards, Zakhar

4 4

Multisite sync - zone permission denied
by Szabo, Istvan (Agoda) 14 Jul '23

14 Jul '23

Hi, Have you had the issue with zones are permission denied? failed to retrieve sync info: (13) Permission denied It's a newly added zone, uses the same sync user and credentials but it shows permission denied and I don't see any reason behind. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

resume RBD mirror on another host
by Tony Liu 13 Jul '23

13 Jul '23

Hi, How RBD mirror tracks mirroring process, on local storage? Say, RBD mirror is running on host-1, when host-1 goes down, start RBD mirror on host-2. In that case, is RBD mirror on host-2 going to continue the mirroring? Thanks! Tony

2 3

Ceph Developer Summit - Squid
by Neha Ojha 13 Jul '23

13 Jul '23

Hi everyone, Join us for the Ceph Developer Summit happening virtually from Jul 17 – 24, 2023. We'll be discussing features planned for our next release, Squid, across the course of this summit. The schedule has been published on our website[0] and meetings have been added to the community calendar[1]. You can find agenda items in this etherpad[2]. We look forward to seeing you there! Thanks, Neha [0] https://ceph.io/en/community/events/2023/ceph-developer-summit-squid/ [1] https://calendar.google.com/calendar/u/0/embed?src=9ts9c7lt7u1vic2ijvvqqlfp… [2] https://pad.ceph.com/p/cds-squid

1 0

Cluster down after network outage
by Frank Schilder 13 Jul '23

13 Jul '23

Hi all, we had a network outage tonight (power loss) and restored network in the morning. All OSDs were running during this period. After restoring network peering hell broke loose and the cluster has a hard time coming back up again. OSDs get marked down all the time and come back later. Peering never stops. Below is the current status, I had all OSDs shown as up for a while, but many were not responsive. Are there some flags that help bringing things up in a sequence that causes less overload on the system? [root@gnosis ~]# ceph status cluster: id: XXX health: HEALTH_WARN 2 clients failing to respond to capability release 6 MDSs report slow metadata IOs 3 MDSs report slow requests nodown,noout,nobackfill,norecover flag(s) set 176 osds down Slow OSD heartbeats on back (longest 551718.679ms) Slow OSD heartbeats on front (longest 549598.330ms) Reduced data availability: 8069 pgs inactive, 3786 pgs down, 3161 pgs peering, 1341 pgs stale Degraded data redundancy: 1187354920/16402772667 objects degraded (7.239%), 6222 pgs degraded, 6231 pgs undersized 1 pools nearfull 17386 slow ops, oldest one blocked for 1811 sec, daemons [osd.1128,osd.1152,osd.1154,osd.12,osd.1227,osd.1244,osd.328,osd.354,osd.381,osd.4]... have slow ops. services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 28m) mgr: ceph-25(active, since 30m), standbys: ceph-26, ceph-01, ceph-02, ceph-03 mds: con-fs2:8 4 up:standby 8 up:active osd: 1260 osds: 1082 up (since 6m), 1258 in (since 18m); 266 remapped pgs flags nodown,noout,nobackfill,norecover data: pools: 14 pools, 25065 pgs objects: 1.91G objects, 3.4 PiB usage: 3.1 PiB used, 6.0 PiB / 9.0 PiB avail pgs: 0.626% pgs unknown 31.566% pgs not active 1187354920/16402772667 objects degraded (7.239%) 51/16402772667 objects misplaced (0.000%) 11706 active+clean 4752 active+undersized+degraded 3286 down 2702 peering 799 undersized+degraded+peered 464 stale+down 418 stale+active+undersized+degraded 214 remapped+peering 157 unknown 128 stale+peering 117 stale+remapped+peering 101 stale+undersized+degraded+peered 57 stale+active+undersized+degraded+remapped+backfilling 35 down+remapped 26 stale+undersized+degraded+remapped+backfilling+peered 23 undersized+degraded+remapped+backfilling+peered 14 active+clean+scrubbing+deep 9 stale+active+undersized+degraded+remapped+backfill_wait 7 active+recovering+undersized+degraded 7 stale+active+recovering+undersized+degraded 6 active+undersized+degraded+remapped+backfilling 6 active+undersized 5 active+undersized+degraded+remapped+backfill_wait 5 stale+remapped 4 stale+activating+undersized+degraded 3 active+undersized+remapped 3 stale+undersized+degraded+remapped+backfill_wait+peered 1 activating+undersized+degraded 1 activating+undersized+degraded+remapped 1 undersized+degraded+remapped+backfill_wait+peered 1 stale+active+clean 1 active+recovering 1 stale+down+remapped 1 undersized+peered 1 active+undersized+degraded+remapped 1 active+clean+scrubbing 1 active+clean+remapped 1 active+recovering+degraded io: client: 1.8 MiB/s rd, 18 MiB/s wr, 409 op/s rd, 796 op/s wr Thanks for any hints! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 5

CEPHADM_FAILED_SET_OPTION
by arnoud＠fuga.cloud 13 Jul '23

13 Jul '23

Hi, After having set up RadosGW with keystone authentication the cluster shows this warning: # ceph health detail HEALTH_WARN Failed to set 1 option(s) [WRN] CEPHADM_FAILED_SET_OPTION: Failed to set 1 option(s) Failed to set rgw.fra option rgw_keystone_implicit_tenants: config set failed: error parsing value: 'True' is not one of the permitted values: false, true, swift, s3, both, 0, 1, none retval: -22 I might have made a typo but that has been corrected. # ceph config dump |grep rgw_keystone_implicit_tenants client.rgw.fra.controller1.lushdc advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller1.pznmuf advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller1.tdrqot advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller2.ndxaet advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller2.rodqxh advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller2.wyhjuk advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller3.boasgr advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller3.lkczbl advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller3.qxctee advanced rgw_keystone_implicit_tenants true * # ceph --version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Any idea on how to fix this issue? Thanks, Arnoud. -- Arnoud de Jonge DevOps Engineer FUGA | CLOUD https://fuga.cloud <https://fuga.cloud/> arnoud(a)fuga.cloud <mailto:arnoud@fuga.cloud> Phone +31727513408 <tel://+31727513408> Registration ID 64988767 VAT ID NL855935984B01 Twitter @fugacloud

2 4

Per minor-version view on docs.ceph.com
by Satoru Takeuchi 13 Jul '23

13 Jul '23

Hi, I have a request about docs.ceph.com. Could you provide per minor-version views on docs.ceph.com? Currently, we can select the Ceph version by using `https://docs.ceph.com/en/<version>". In this case, we can use the major version's code names (e.g., "quincy") or "latest". However, we can't use minor version numbers like "v17.2.6". It's convenient for me (and I guess for many other users, too) to be able to select the document for the version which we actually use. In my recent case, I've read the mclock's document of quincy because I use v17.2.6. However, the document has changed a lot from v17.2.6 to the quincy's latest one because of the recent mclock's rework. Thanks, Satoru

3 3

bluestore/bluefs: A large number of unfounded read bandwidth
by yite gu 13 Jul '23

13 Jul '23

Hi, all and Igor Have a case: https://tracker.ceph.com/issues/61973, I'm not sure if it's related to this PR(https://github.com/ceph/ceph/pull/38902), but it looks very similar.

1 0

Reef release candidate - v18.1.2
by Yuri Weinstein 12 Jul '23

12 Jul '23

Hi everyone, This is the second release candidate for Reef. The Reef release comes with a new RockDB version (7.9.2) [0], which incorporates several performance improvements and features. Our internal testing doesn't show any side effects from the new version, but we are very eager to hear community feedback on it. This is the first release to have the ability to tune RockDB settings per column family [1], which allows for more granular tunings to be applied to different kinds of data stored in RocksDB. A new set of settings has been used in Reef to optimize performance for most kinds of workloads with a slight penalty in some cases, outweighed by large improvements in use cases such as RGW, in terms of compactions and write amplification. We would highly encourage community members to give these a try against their performance benchmarks and use cases. The detailed list of changes in terms of RockDB and BlueStore can be found in https://pad.ceph.com/p/reef-rc-relnotes. If any of our community members would like to help us with performance investigations or regression testing of the Reef release candidate, please feel free to provide feedback via email or in https://pad.ceph.com/p/reef_scale_testing. For more active discussions, please use the #ceph-at-scale slack channel in ceph-storage.slack.com. This RC has gone thru partial testing due to issues we are experiencing in the sepia lab. Please try it out and report any issues you encounter. Happy testing! Thanks, YuriW

4 6

radosgw + keystone breaks when projects have - in their names
by Andrew Bogott 12 Jul '23

12 Jul '23

I'm in the process of adding the radosgw service to our OpenStack cloud and hoping to re-use keystone for discovery and auth. Things seem to work fine with many keystone tenants, but as soon as we try to do something in a project with a '-' in its name everything fails. Here's an example, using the openstack swift cli: root@cloudcontrol2001-dev:~# OS_PROJECT_ID="testlabs" openstack container create 'makethiscontainer' +---------------+-------------------+----------------------------------------------------+ | account | container | x-trans-id | +---------------+-------------------+----------------------------------------------------+ | AUTH_testlabs | makethiscontainer | tx0000008c311dbda86c695-0064ac5fad-6927acd-default | +---------------+-------------------+----------------------------------------------------+ root@cloudcontrol2001-dev:~# OS_PROJECT_ID="service" openstack container create 'makethiscontainer' +--------------+-------------------+----------------------------------------------------+ | account | container | x-trans-id | +--------------+-------------------+----------------------------------------------------+ | AUTH_service | makethiscontainer | tx00000b341a22866f65e44-0064ac5fb7-6927acd-default | +--------------+-------------------+----------------------------------------------------+ root@cloudcontrol2001-dev:~# OS_PROJECT_ID="admin-monitoring" openstack container create 'makethiscontainer' Bad Request (HTTP 400) (Request-ID: tx00000f7326bb541b4d2a9-0064ac5fc2-6927acd-default) Before I dive into the source code, is this a known issue and/or something I can configure? Dash-named-projects work fine in keystone and seem to also work fine with standalone rados; I assume the issue is somewhere in the communication between the two. I suspected the implicit user creation code, but that seems to be working properly: # radosgw-admin user list [ "cloudvirt-canary$cloudvirt-canary", "testlabs$testlabs", "paws-dev$paws-dev", "andrewtestproject$andrewtestproject", "admin-monitoring$admin-monitoring", "taavi-test-project$taavi-test-project", "admin$admin", "taavitestproject$taavitestproject", "bastioninfra-codfw1dev$bastioninfra-codfw1dev", ] Here is the radosgw section of my ceph.conf: [client.radosgw] host = 10.192.20.9 keyring = /etc/ceph/ceph.client.radosgw.keyring rgw frontends = "civetweb port=18080" rgw_keystone_verify_ssl = false rgw_keystone_api_version = 3 rgw_keystone_url = https://openstack.codfw1dev.wikimediacloud.org:25000 rgw_keystone_accepted_roles = 'reader, admin, member' rgw_keystone_implicit_tenants = true rgw_keystone_admin_domain = default rgw_keystone_admin_project = service rgw_keystone_admin_user = swift rgw_keystone_admin_password = (redacted) rgw_s3_auth_use_keystone = true rgw_swift_account_in_url = true rgw_user_default_quota_max_objects = 4096 rgw_user_default_quota_max_size = 8589934592 And here's a debug log of a failed transaction: https://phabricator.wikimedia.org/P49539 Thanks in advance!

1 1

Production random data not accessible(NoSuchKey)
by Jonas Nemeiksis 12 Jul '23

12 Jul '23

Hi all, I'm facing a strange problem, where from time to time there are no accessible S3 objects. I've found similar issues [1] , [2] but our clusters have already upgraded to the latest Pacific version. I have noted in the bug report https://tracker.ceph.com/issues/61716 RGW logs [3] Maybe someone has an idea what's wrong? Thanks. [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/WQ2F2GWI2WR… [2] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RS2272EWAGC… [3] https://pastebin.com/ZvBdNi5j -- Jonas

1 0

Cephadm fails to deploy loki with promtail correctly
by Sake Ceph 11 Jul '23

11 Jul '23

I'm not sure if it's a bug with Cephadm, but it looks like it. I've got Loki deployed on one machine and Promtail deployed to all machines. After creating a login, I can view only the logs on the hosts on which Loki is running. When inspecting the Promtail configuration, the configured URL for Loki is set to http://host.containers.internal:3100. Shouldn't this be configured by Cephadm and pointing to the Loki host? This looks a lot like the issues with incorrectly setting the Grafana or Prometheus URL's, bug 57018 is created for this. Should I create another bug report? And does someone know a workaround to set the correct URL for the time being? Best regards, Sake

1 0

cephadm problem with MON deployment
by Adam Huffman 11 Jul '23

11 Jul '23

Hello I'm trying to add MONs in advance of a planned downtime. This has actually ended up removing an existing MON, which isn't helpful. The error I'm seeing is: Invalid argument: /var/lib/ceph/mon/ceph-<hostname>/store.db: does not exist (create_if_missing is false) error opening mon data directory at '/var/lib/ceph/mon/ceph-<hostname>': (22) Invalid argument It appears that the fsid is being stripped, because the directory was there. It's now in /var/lib/ceph/<fsid>/removed This appears to be similar to: https://tracker.ceph.com/issues/45167 which was closed for lack of a reproducer. The command I ran was: sudo ceph orch apply mon --placement="comma-separated hostname list" after running that with "--dry-run". Would be grateful for some advice here - I wasn't expecting to reduce the MON count. Best Wishes, Adam

1 2

RGW dynamic resharding blocks write ops
by Eugen Block 11 Jul '23

11 Jul '23

Hi *, last week I successfully upgraded a customer cluster from Nautilus to Pacific, no real issues, their main use is RGW. A couple of hours after most of the OSDs were upgraded (the RGWs were not yet) their application software reported an error, it couldn't write to a bucket. This error occured again two days ago, in the RGW logs I found the relevant messages that resharding was happening at that time. I'm aware that this is nothing unusual, but I can't find anything helpful how to prevent this except for deactivating dynamic resharding and then manually do it during maintenance windows. We don't know yet if there's really data missing after the bucket access has recovered or not, that still needs to be investigated. Since Nautilus already had dynamic resharding enabled, I wonder if they were just lucky until now, for example resharding happened while no data was being written to the buckets. Or if resharding just didn't happen until then, I have no access to the cluster so I don't have any bucket stats available right now. I found this thread [1] about an approach how to prevent blocked IO but it's from 2019 and I don't know how far that got. There are many users/operators on this list who use RGW more than me, how do you deal with this? Are your clients better prepared for these events? Any comments are appreciated! Thanks, Eugen [1] https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/NG56XXAM5A4JONT4BG…

3 7

Planning cluster
by Jan Marek 11 Jul '23

11 Jul '23

Hello, I have a cluster, which have this configuration: osd pool default size = 3 osd pool default min size = 1 I have 5 monitor nodes and 7 OSD nodes. I have changed a crush map to divide ceph cluster to two datacenters - in the first one will be a part of cluster with 2 copies of data and in the second one will be part of cluster with one copy - only emergency. I still have this cluster in one This cluster have a 1 PiB of raw data capacity, thus it is very expensive add a further 300TB capacity to have 2+2 data redundancy. Will it works? If I turn off the 1/3 location, will it be operational? I believe, it is a better choose, it will. And what if "die" 2/3 location? On this cluster is pool with cephfs - this is a main part of CEPH. Many thanks for your notices. Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html

3 2

mon log file grows huge
by Ben 11 Jul '23

11 Jul '23

Hi, In our cluster monitors' log grows to couple GBs in days. There are quite many debug message from rocksdb, osd, mgr and mds. These should not be necessary with a well-run cluster. How could I close these logging? Thanks, Ben

2 2

ceph quota qustion
by sejun21.kim＠samsung.com 10 Jul '23

10 Jul '23

Hi, I contact you for some question about quota. Situation is following below. 1. I set the user quota 10M 2. Using s3 browser, upload one 12M file 3. The upload failed as i wish, but some object remains in the pool(almost 10M) and s3brower doesn't show failed file. I expected nothing to be left in Ceph. My question is "can user or admin remove the remaining objects?"

3 3

CEPH orch made osd without WAL
by Jan Marek 10 Jul '23

10 Jul '23

Hello, I've tried to add to CEPH cluster OSD node with a 12 rotational disks and 1 NVMe. My YAML was this: service_type: osd service_id: osd_spec_default service_name: osd.osd_spec_default placement: host_pattern: osd8 spec: block_db_size: 64G data_devices: rotational: 1 db_devices: paths: - /dev/nvme0n1 filter_logic: AND objectstore: bluestore Now I have 12 OSD with DB on NVMe device, but without WAL. How I can add WAL to this OSD? NVMe device still have 128GB free place. Thanks a lot. Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html

3 7

Are replicas 4 or 6 safe during network partition? Will there be split-brain?
by jcichra＠cloudflare.com 10 Jul '23

10 Jul '23

Hi, I'm evaluating if an even number of replicas is safe. Are 4 and 6 replicas still safe compared to replicas 5 and 7? With replicas 4 the min_size is 2. Does this mean in a split-brain situation there's a possibility that OSDs on both sides of the split could accept writes? Here's the scenario I'm thinking of. Please correct the assumptions I am making. So far I am unable to create this problem in my testing: There are two sites, A and B. There are 5 mons, 2 in A, 3 in B. Looking at just one PG and 4 replicas, we have 2 replicas in site A and 2 replicas in site B. Site A holds the primary OSD for this PG. When a network split happens, I/O would still be working in site A since there are still 2 OSDs, even without mon quorum. The primary OSD can still reach one of the other OSDs but not a quorum of mons. After a period of time, site B has quorum and changes the topology of the cluster to say it lost site A. Writes are also accepted in site A because min_size 2 is also satisfied. My question is - is this an actual scenario that can happen with an even number of replicas and an even network split? Do I need to go to 5 replicas with a min_size of 3 to prevent an even split from accepting writes on both sides? In that case one side would have 3 copies and the other would have 2. Thanks.

2 1

librbd Python asyncio
by Tony Liu 09 Jul '23

09 Jul '23

Hi, Wondering if there is librbd supporting Python asyncio, or any plan to do that? Thanks! Tony

1 0

immutable bit
by Angelo Höngens 08 Jul '23

08 Jul '23

Hey guys and girls, I noticed CephFS on my kinda default 17.2.6 CephFS volume, it does not support setting the immutable bit. (Want to start using it with the Veeam hardened repo that uses the immutable bit). I do see a lot of very, very old posts with technical details on how to implement it, but is there a way for me to use that yet? Angelo. see https://tracker.ceph.com/issues/10679

2 1