June 2023 - ceph-users - lists.ceph.io

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

Help needed to configure erasure coding LRC plugin
by Michel Jouvin 16 Oct '24

16 Oct '24

Hi, As discussed in another thread (Crushmap rule for multi-datacenter erasure coding), I'm trying to create an EC pool spanning 3 datacenters (datacenters are present in the crushmap), with the objective to be resilient to 1 DC down, at least keeping the readonly access to the pool and if possible the read-write access, and have a storage efficiency better than 3 replica (let say a storage overhead <= 2). In the discussion, somebody mentioned LRC plugin as a possible jerasure alternative to implement this without tweaking the crushmap rule to implement the 2-step OSD allocation. I looked at the documentation (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/) but I have some questions if someone has experience/expertise with this LRC plugin. I tried to create a rule for using 5 OSDs per datacenter (15 in total), with 3 (9 in total) being data chunks and others being coding chunks. For this, based of my understanding of examples, I used k=9, m=3, l=4. Is it right? Is this configuration equivalent, in terms of redundancy, to a jerasure configuration with k=9, m=6? The resulting rule, which looks correct to me, is: -------- { "rule_id": 6, "rule_name": "test_lrc_2", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 15, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -4, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 5, "type": "host" }, { "op": "emit" } ] } ------------ Unfortunately, it doesn't work as expected: a pool created with this rule ends up with its pages active+undersize, which is unexpected for me. Looking at 'ceph health detail` output, I see for each page something like: pg 52.14 is stuck undersized for 27m, current state active+undersized, last acting [90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647] For each PG, there is 3 '2147483647' entries and I guess it is the reason of the problem. What are these entries about? Clearly it is not OSD entries... Looks like a negative number, -1, which in terms of crushmap ID is the crushmap root (named "default" in our configuration). Any trivial mistake I would have made? Thanks in advance for any help or for sharing any successful configuration? Best regards, Michel

4 24

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

Re: MDS crashes to damaged metadata
by Patrick Donnelly 10 Jun '24

10 Jun '24

On Thu, Dec 15, 2022 at 9:32 AM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): > > 2022-12-08T13:05:48.919+0100 7f440afec700 -1 /build/ceph-16.2.10/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f440afec700 time 2022-12-08T13:05:48.921223+0100 > /build/ceph-16.2.10/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE) > > 12 Minutes lates the unlink_local error crashes appeared again. This time with a new file. During debugging we noticed a MTU mismatch between MDS (1500) and client (9000) with cephfs kernel mount. The client is also creating the snapshots via mkdir in the .snap directory. > > We disabled snapshot creation for now, but really need this feature. I uploaded the mds logs of the first crash along with the information above to https://tracker.ceph.com/issues/38452 > > I would greatly appreciate it, if you could answer me the following question: > > Is the Bug related to our MTU Mismatch? We fixed the MTU Issue going back to 1500 on all nodes in the ceph public network on the weekend also. I doubt it. > If you need a debug level 20 log of the ScatterLock for further analysis, i could schedule snapshots at the end of our workdays and increase the debug level 5 Minutes arround snap shot creation. This would be very helpful! -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 4

Re: Recoveries without any misplaced objects?
by Hector Martin 24 Apr '24

24 Apr '24

On 29/05/2023 20.55, Anthony D'Atri wrote: > Check the uptime for the OSDs in question I restarted all my OSDs within the past 10 days or so. Maybe OSD restarts are somehow breaking these stats? > >> On May 29, 2023, at 6:44 AM, Hector Martin <marcan(a)marcan.st> wrote: >> >> Hi, >> >> I'm watching a cluster finish a bunch of backfilling, and I noticed that >> quite often PGs end up with zero misplaced objects, even though they are >> still backfilling. >> >> Right now the cluster is down to 6 backfilling PGs: >> >> data: >> volumes: 1/1 healthy >> pools: 6 pools, 268 pgs >> objects: 18.79M objects, 29 TiB >> usage: 49 TiB used, 25 TiB / 75 TiB avail >> pgs: 262 active+clean >> 6 active+remapped+backfilling >> >> But there are no misplaced objects, and the misplaced column in `ceph pg >> dump` is zero for all PGs. >> >> If I do a `ceph pg dump_json`, I can see `num_objects_recovered` >> increasing for these PGs... but the misplaced count is still 0. >> >> Is there something else that would cause recoveries/backfills other than >> misplaced objects? Or perhaps there is a bug somewhere causing the >> misplaced object count to be misreported as 0 sometimes? >> >> # ceph -v >> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy >> (stable) >> >> - Hector >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > - Hector

3 3

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

Why you might want packages not containers for Ceph deployments
by Matthew Vernon 25 Mar '24

25 Mar '24

Hi, In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0]. I'd like to suggest, briefly, that: * containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes. For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice. Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages): * you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice) We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part. Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems! I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder". I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :) Regards, and apologies for the wall of text, Matthew [0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

26 65

Upgrading from Pacific to Quincy fails with "Unexpected error"
by Reza Bakhshayeshi 13 Mar '24

13 Mar '24

Hi all, I have a problem regarding upgrading Ceph cluster from Pacific to Quincy version with cephadm. I have successfully upgraded the cluster to the latest Pacific (16.2.11). But when I run the following command to upgrade the cluster to 17.2.5, After upgrading 3/4 mgrs, the upgrade process stops with "Unexpected error". (everything is on a private network) ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v17.2.5 I also tried the 17.2.4 version. cephadm fails to check the hosts' status and marks them as offline: cephadm 2023-04-06T10:19:59.998510+0000 mgr.host9.arhpnd (mgr.4516356) 5782 : cephadm [DBG] host host4 (x.x.x.x) failed check cephadm 2023-04-06T10:19:59.998553+0000 mgr.host9.arhpnd (mgr.4516356) 5783 : cephadm [DBG] Host "host4" marked as offline. Skipping daemon refresh cephadm 2023-04-06T10:19:59.998581+0000 mgr.host9.arhpnd (mgr.4516356) 5784 : cephadm [DBG] Host "host4" marked as offline. Skipping gather facts refresh cephadm 2023-04-06T10:19:59.998609+0000 mgr.host9.arhpnd (mgr.4516356) 5785 : cephadm [DBG] Host "host4" marked as offline. Skipping network refresh cephadm 2023-04-06T10:19:59.998633+0000 mgr.host9.arhpnd (mgr.4516356) 5786 : cephadm [DBG] Host "host4" marked as offline. Skipping device refresh cephadm 2023-04-06T10:19:59.998659+0000 mgr.host9.arhpnd (mgr.4516356) 5787 : cephadm [DBG] Host "host4" marked as offline. Skipping osdspec preview refresh cephadm 2023-04-06T10:19:59.998682+0000 mgr.host9.arhpnd (mgr.4516356) 5788 : cephadm [DBG] Host "host4" marked as offline. Skipping autotune cluster 2023-04-06T10:20:00.000151+0000 mon.host8 (mon.0) 158587 : cluster [ERR] Health detail: HEALTH_ERR 9 hosts fail cephadm check; Upgrade: failed due to an unexpected exception cluster 2023-04-06T10:20:00.000191+0000 mon.host8 (mon.0) 158588 : cluster [ERR] [WRN] CEPHADM_HOST_CHECK_FAILED: 9 hosts fail cephadm check cluster 2023-04-06T10:20:00.000202+0000 mon.host8 (mon.0) 158589 : cluster [ERR] host host7 (x.x.x.x) failed check: Unable to reach remote host host7. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000213+0000 mon.host8 (mon.0) 158590 : cluster [ERR] host host2 (x.x.x.x) failed check: Unable to reach remote host host2. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000220+0000 mon.host8 (mon.0) 158591 : cluster [ERR] host host8 (x.x.x.x) failed check: Unable to reach remote host host8. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000228+0000 mon.host8 (mon.0) 158592 : cluster [ERR] host host4 (x.x.x.x) failed check: Unable to reach remote host host4. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000240+0000 mon.host8 (mon.0) 158593 : cluster [ERR] host host3 (x.x.x.x) failed check: Unable to reach remote host host3. Process exited with non-zero exit status 3 and here are some outputs of the commands: [root@host8 ~]# ceph -s cluster: id: xxx health: HEALTH_ERR 9 hosts fail cephadm check Upgrade: failed due to an unexpected exception services: mon: 5 daemons, quorum host8,host1,host7,host2,host9 (age 2w) mgr: host9.arhpnd(active, since 105m), standbys: host8.jowfih, host1.warjsr, host2.qyavjj mds: 1/1 daemons up, 3 standby osd: 37 osds: 37 up (since 8h), 37 in (since 3w) data: io: client: progress: Upgrade to 17.2.5 (0s) [............................] [root@host8 ~]# ceph orch upgrade status { "target_image": "my-private-repo/quay-io/ceph/ceph@sha256 :34c763383e3323c6bb35f3f2229af9f466518d9db926111277f5e27ed543c427", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "3/59 daemons upgraded", "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception", "is_paused": true } [root@host8 ~]# ceph cephadm check-host host7 check-host failed: Host 'host7' not found. Use 'ceph orch host ls' to see all managed hosts. [root@host8 ~]# ceph versions { "mon": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 1, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 }, "osd": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 37 }, "mds": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 4 }, "overall": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 47, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 } } The strange thing is I can rollback the cluster status by failing to not-upgraded mgr like this: ceph mgr fail ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v16.2.11 Would you happen to have any idea about this? Best regards, Reza

4 9

Re: increasing number of (deep) scrubs
by Frank Schilder 13 Dec '23

13 Dec '23

Hi Dan, thanks for your answer. I don't have a problem with increasing osd_max_scrubs (=1 at the moment) as such. I would simply prefer a somewhat finer grained way of controlling scrubbing than just doubling or tripling it right away. Some more info. These 2 pools are data pools for a large FS. Unfortunately, we have a large percentage of small files, which is a pain for recovery and seemingly also for deep scrubbing. Our OSDs are about 25% used and I had to increase the warning interval already to 2 weeks. With all the warning grace parameters this means that we manage to deep scrub everything about every month. I need to plan for 75% utilisation and a 3 months period is a bit far on the risky side. Our data is to a large percentage cold data. Client reads will not do the check for us, we need to combat bit-rot pro-actively. The reasons I'm interested in parameters initiating more scrubs while also converting more scrubs into deep scrubs are, that 1) scrubs seem to complete very fast. I almost never catch a PG in state "scrubbing", I usually only see "deep scrubbing". 2) I suspect the low deep-scrub count is due to a low number of deep-scrubs scheduled and not due to conflicting per-OSD deep scrub reservations. With the OSD count we have and the distribution over 12 servers I would expect at least a peak of 50% OSDs being active in scrubbing instead of the 25% peak I'm seeing now. It ought to be possible to schedule more PGs for deep scrub than actually are. 3) Every OSD having only 1 deep scrub active seems to have no measurable impact on user IO. If I could just get more PGs scheduled with 1 deep scrub per OSD it would already help a lot. Once this is working, I can eventually increase osd_max_scrubs when the OSDs fill up. For now I would just like that (deep) scrub scheduling looks a bit harder and schedules more eligible PGs per time unit. If we can get deep scrubbing up to an average of 42PGs completing per hour with keeping osd_max_scrubs=1 to maintain current IO impact, we should be able to complete a deep scrub with 75% full OSDs in about 30 days. This is the current tail-time with 25% utilisation. I believe currently a deep scrub of a PG in these pools takes 2-3 hours. Its just a gut feeling from some repair and deep-scrub commands, I would need to check logs for more precise info. Increasing osd_max_scrubs would then be a further and not the only option to push for more deep scrubbing. My expectation would be that values of 2-3 are fine due to the increasingly higher percentage of cold data for which no interference with client IO will happen. Hope that makes sense and there is a way beyond bumping osd_max_scrubs to increase the number of scheduled and executed deep scrubs. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dvanders(a)gmail.com> Sent: 05 January 2023 15:36 To: Frank Schilder Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] increasing number of (deep) scrubs Hi Frank, What is your current osd_max_scrubs, and why don't you want to increase it? With 8+2, 8+3 pools each scrub is occupying the scrub slot on 10 or 11 OSDs, so at a minimum it could take 3-4x the amount of time to scrub the data than if those were replicated pools. If you want the scrub to complete in time, you need to increase the amount of scrub slots accordingly. On the other hand, IMHO the 1-week deadline for deep scrubs is often much too ambitious for large clusters -- increasing the scrub intervals is one solution, or I find it simpler to increase mon_warn_pg_not_scrubbed_ratio and mon_warn_pg_not_deep_scrubbed_ratio until you find a ratio that works for your cluster. Of course, all of this can impact detection of bit-rot, which anyway can be covered by client reads if most data is accessed periodically. But if the cluster is mostly idle or objects are generally not read, then it would be preferable to increase slots osd_max_scrubs. Cheers, Dan On Tue, Jan 3, 2023 at 2:30 AM Frank Schilder <frans(a)dtu.dk> wrote: > > Hi all, > > we are using 16T and 18T spinning drives as OSDs and I'm observing that they are not scrubbed as often as I would like. It looks like too few scrubs are scheduled for these large OSDs. My estimate is as follows: we have 852 spinning OSDs backing a 8+2 pool with 2024 and an 8+3 pool with 8192 PGs. On average I see something like 10PGs of pool 1 and 12 PGs of pool 2 (deep) scrubbing. This amounts to only 232 out of 852 OSDs scrubbing and seems to be due to a conservative rate of (deep) scrubs being scheduled. The PGs (dep) scrub fairly quickly. > > I would like to increase gently the number of scrubs scheduled for these drives and *not* the number of scrubs per OSD. I'm looking at parameters like: > > osd_scrub_backoff_ratio > osd_deep_scrub_randomize_ratio > > I'm wondering if lowering osd_scrub_backoff_ratio to 0.5 and, maybe, increasing osd_deep_scrub_randomize_ratio to 0.2 would have the desired effect? Are there other parameters to look at that allow gradual changes in the number of scrubs going on? > > Thanks a lot for your help! > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 3

Re: RGW access logs with bucket name
by Boris Behrens 02 Nov '23

02 Nov '23

Bringing up that topic again: is it possible to log the bucket name in the rgw client logs? currently I am only to know the bucket name when someone access the bucket via https://TLD/bucket/object instead of https://bucket.TLD/object. Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens <bb(a)kervyn.de>: > Hi, > I am looking forward to move our logs from > /var/log/ceph/ceph-client...log to our logaggregator. > > Is there a way to have the bucket name in the log file? > > Or can I write the rgw_enable_ops_log into a file? Maybe I could work with > this. > > Cheers and happy new year > Boris > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

4 6

Re: Status of Quincy 17.2.5 ?
by Christian Rohmann 30 Oct '23

30 Oct '23

Hey everyone, On 20/10/2022 10:12, Christian Rohmann wrote: > 1) May I bring up again my remarks about the timing: > > On 19/10/2022 11:46, Christian Rohmann wrote: > >> I believe the upload of a new release to the repo prior to the >> announcement happens quite regularly - it might just be due to the >> technical process of releasing. >> But I agree it would be nice to have a more "bit flip" approach to >> new releases in the repo and not have the packages appear as updates >> prior to the announcement and final release and update notes. > By my observations sometimes there are packages available on the > download servers via the "last stable" folders such as > https://download.ceph.com/debian-quincy/ quite some time before the > announcement of a release is out. > I know it's hard to time this right with mirrors requiring some time > to sync files, but would be nice to not see the packages or have > people install them before there are the release notes and potential > pointers to changes out. Todays 16.2.11 release shows the exact issue I described above .... 1) 16.2.11 packages are already available via e.g. https://download.ceph.com/debian-pacific 2) release notes not yet merged: (https://github.com/ceph/ceph/pull/49839) thus https://ceph.io/en/news/blog/2022/v16-2-11-pacific-released/ show a 404 :-) 3) No announcement like https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/QOCU563UD3… to the ML yet. Regards Christian

2 2

[Pacific] ceph orch device ls do not returns any HDD
by Patrick Begou 27 Oct '23

27 Oct '23

Hi everyone I'm new to CEPH, just a french 4 days training session with Octopus on VMs that convince me to build my first cluster. At this time I have 4 old identical nodes for testing with 3 HDDs each, 2 network interfaces and running Alma Linux8 (el8). I try to replay the training session but it fails, breaking the web interface because of some problems with podman 4.2 not compatible with Octopus. So I try to deploy Pacific with cephadm tool on my first node (mostha1) (to enable testing also an upgrade later). dnf -y install https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noar… monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }') cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \ --initial-dashboard-user admceph \ --allow-fqdn-hostname --cluster-network 10.1.0.0/16 This was sucessfull. But running "*c**eph orch device ls*" do not show any HDD even if I have /dev/sda (used by the OS), /dev/sdb and /dev/sdc The web interface shows a row capacity which is an aggregate of the sizes of the 3 HDDs for the node. I've also tried to reset /dev/sdb but cephadm do not see it: [ceph: root@mostha1 /]# ceph orch device zap mostha1.legi.grenoble-inp.fr /dev/sdb --force Error EINVAL: Device path '/dev/sdb' not found on host 'mostha1.legi.grenoble-inp.fr' On my first attempt with octopus, I was able to list the available HDD with this command line. Before moving to Pacific, the OS on this node has been reinstalled from scratch. Any advices for a CEPH beginner ? Thanks Patrick

9 51

ceph-dashboard python warning with new pyo3 0.17 lib (debian12)
by DERUMIER, Alexandre 11 Oct '23

11 Oct '23

Hi, on debian12, ceph-dashboard is throwing a warning "Module 'dashboard' has failed dependency: PyO3 modules may only be initialized once per interpreter process" Seem to be related to pyo3 0.17 change https://github.com/PyO3/pyo3/blob/7bdc504252a2f972ba3490c44249b202a4ce6180/… " Each #[pymodule] can now only be initialized once per process To make PyO3 modules sound in the presence of Python sub-interpreters, for now it has been necessary to explicitly disable the ability to initialize a #[pymodule] more than once in the same process. Attempting to do this will now raise an ImportError. "

3 3

libceph: mds1 IP+PORT wrong peer at address
by Frank Schilder 20 Sep '23

20 Sep '23

Hi all, we seem to have hit a bug in the ceph fs kernel client and I just want to confirm what action to take. We get the error "wrong peer at address" in dmesg and some jobs on that server seem to get stuck in fs access; log extract below. I found these 2 tracker items https://tracker.ceph.com/issues/23883 and https://tracker.ceph.com/issues/41519, which don't seem to have fixes. My questions: - Is this harmless or does it indicate invalid/corrupted client cache entries? - How to resolve, ignore, umount+mount or reboot? Here an extract from the dmesg log, the error has survived a couple of MDS restarts already: [Mon Mar 6 12:56:46 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:05:18 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-1572619386 [Mon Mar 6 13:05:18 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:13:50 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-1572619386 [Mon Mar 6 13:13:50 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state OPEN) [Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state OPEN) [Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start [Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start [Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success [Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success [Mon Mar 6 13:18:13 2023] ceph: update_snap_trace error -22 [Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state OPEN) [Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state OPEN) [Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed [Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect success [Mon Mar 6 13:18:29 2023] ceph: mds7 reconnect success [Mon Mar 6 13:18:35 2023] ceph: update_snap_trace error -22 [Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed [Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed [Mon Mar 6 13:22:22 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Mon Mar 6 13:22:22 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:30:54 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [...] [Thu Mar 9 09:37:24 2023] slurm.epilog.cl (31457): drop_caches: 3 [Thu Mar 9 09:38:26 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:38:26 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 09:46:58 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:46:58 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 09:55:30 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:55:30 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 10:04:02 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 10:04:02 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 6

RGW multisite logs (data, md, bilog) not being trimmed automatically?
by Christian Rohmann 14 Sep '23

14 Sep '23

Hey ceph-users, I am running two (now) Quincy clusters doing RGW multi-site replication with only one actually being written to by clients. The other site is intended simply as a remote copy. On the primary cluster I am observing an ever growing (objects and bytes) "sitea.rgw.log" pool, not so on the remote "siteb.rgw.log" which is only 300MiB and around 15k objects with no growth. Metrics show that the growth of pool on primary is linear for at least 6 months, so not sudden spikes or anything. Also sync status appears to be totally happy. There are also no warnings in regards to large OMAPs or anything similar. I was under the impression that RGW will trim its three logs (md, bi, data) automatically and only keep data that has not yet been replicated by the other zonegroup members? The config option "ceph config get mgr rgw_sync_log_trim_interval" is set to 1200, so 20 Minutes. So I am wondering if there might be some inconsistency or how I can best analyze what the cause for the accumulation of log data is? There are older questions on the ML, such as [1], but there was not really a solution or root cause identified. I know there is manual trimming, but I'd rather want to analyze the current situation and figure out what the cause for the lack of auto-trimming is. * Do I need to go through all buckets and count logs and look at their timestamps? Which queries do make sense here? * Is there usually any logging of the log trimming activity that I should expect? Or that might indicate why trimming does not happen? Regards Christian [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/WZCFOAMLWV…

1 2

Critical Information: DELL/Toshiba SSDs dying after 70,000 hours of operation
by Frédéric Nass 01 Sep '23

01 Sep '23

Hello, This message does not concern Ceph itself but a hardware vulnerability which can lead to permanent loss of data on a Ceph cluster equipped with the same hardware in separate fault domains. The DELL / Toshiba PX02SMF020, PX02SMF040, PX02SMF080 and PX02SMB160 SSD drives of the 13G generation of DELL servers are subject to a vulnerability which renders them unusable after 70,000 hours of operation, i.e. approximately 7 years and 11 months of activity. This topic has been discussed here: https://www.dell.com/community/PowerVault/TOSHIBA-PX02SMF080-has-lost-commu… The risk is all the greater since these disks may die at the same time in the same server leading to the loss of all data in the server. To date, DELL has not provided any firmware fixing this vulnerability, the latest firmware version being "A3B3" released on Sept. 12, 2016: https://www.dell.com/support/home/en-us/ drivers/driversdetails?driverid=hhd9k If your have servers running these drives, check their uptime. If they are close to the 70,000 hour limit, replace them immediately. The smartctl tool does not report the uptime for these SSDs, but if you have HDDs in the server, you can query their SMART status and get their uptime, which should be about the same as the SSDs. The smartctl command is: smartctl -a -d megaraid,XX /dev/sdc (where XX is the iSCSI bus number). We have informed DELL about this but have no information yet on the arrival of a fix. We have lost 6 disks, in 3 different servers, in the last few weeks. Our observation shows that the drives don't survive full shutdown and restart of the machine (power off then power on in iDrac), but they may also die during a single reboot (init 6) or even while the machine is running. Fujitsu released a corrective firmware in June 2021 but this firmware is most certainly not applicable to DELL drives: https://www.fujitsu.com/us/imagesgig5/PY-CIB070-00.pdf Regards, Frederic Sous-direction Infrastructure and Services Direction du Numérique Université de Lorraine

2 2

ceph quincy repo update to debian bookworm...?
by Christian Peters 26 Jul '23

26 Jul '23

Hi ceph users/maintainers, I installed ceph quincy on debian bullseye as a ceph client and now want to update to bookworm. I see that there is at the moment only bullseye supported. https://download.ceph.com/debian-quincy/dists/bullseye/ Will there be an update of deb https://download.coeh.com/debian-quincy/ bullseye main to deb https://download.coeh.com/debian-quincy/ boowkworm main in the near future!? Regards, Christian

3 2

cephfs - unable to create new subvolume
by karon karon 24 Jul '23

24 Jul '23

Hello, I recently use cephfs in version 17.2.6 I have a pool named "*data*" and a fs "*kube*" it was working fine until a few days ago, now i can no longer create a new subvolume*, *it gives me the following error: Error EINVAL: invalid value specified for ceph.dir.subvolume > here is the command used: ceph fs subvolume create kube newcsivol --pool_layout data > from what I understand it seems that it creates the subvolume but immediately puts it in the trash !? here is the log : 2023-06-23T08:30:53.307+0000 7f2b929d2700 0 log_channel(audit) log [DBG] : > from='client.86289 -' entity='client.admin' cmd=[{"prefix": "fs subvolume > create", "vol_name": "kube", "sub_name": "newcsivol", "group_name": "csi", > "pool_layout": "data", "target": ["mon-mgr", ""]}]: dispatch > 2023-06-23T08:30:53.307+0000 7f2b8a1d1700 0 [volumes INFO volumes.module] > Starting _cmd_fs_subvolume_create(group_name:csi, pool_layout:data, > prefix:fs subvolume create, sub_name:newcsivol, target:['mon-mgr', ''], > vol_name:kube) < "" > 2023-06-23T08:30:53.327+0000 7f2b8a1d1700 0 [volumes INFO > volumes.fs.operations.versions.subvolume_v2] cleaning up subvolume with > path: newcsivol > 2023-06-23T08:30:53.331+0000 7f2b8a1d1700 0 [volumes INFO > volumes.fs.operations.versions.subvolume_base] subvolume path > 'b'/volumes/csi/newcsivol'' moved to trashcan > 2023-06-23T08:30:53.331+0000 7f2b8a1d1700 0 [volumes INFO > volumes.fs.async_job] queuing job for volume 'kube' > 2023-06-23T08:30:53.335+0000 7f2b8a1d1700 0 [volumes INFO volumes.module] > Finishing _cmd_fs_subvolume_create(group_name:csi, pool_layout:data, > prefix:fs subvolume create, sub_name:newcsivol, target:['mon-mgr', ''], > vol_name:kube) < "" > 2023-06-23T08:30:53.335+0000 7f2b8a1d1700 -1 mgr.server reply reply (22) > Invalid argument invalid value specified for ceph.dir.subvolume > 2023-06-23T08:30:53.339+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.339+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.339+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.339+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.339+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.363+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.363+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.363+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.363+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.363+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.383+0000 7f2b479c2700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.383+0000 7f2b479c2700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.383+0000 7f2b479c2700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.383+0000 7f2b479c2700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.383+0000 7f2b479c2700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.507+0000 7f2b3ff33700 0 [prometheus INFO > cherrypy.access.139824530773776] 192.168.240.231 - - [23/Jun/2023:08:30:53] > "GET /metrics HTTP/1.1" 200 194558 "" "Prometheus/2.33.4" > 2023-06-23T08:30:54.219+0000 7f2b3ddaf700 0 [dashboard INFO request] [ > 172.29.2.142:33040] [GET] [200] [0.003s] [admin] [22.0B] > /api/prometheus/notifications > 2023-06-23T08:30:54.223+0000 7f2b929d2700 0 log_channel(audit) log [DBG] > : from='mon.0 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": > "json"}]: dispatch > 2023-06-23T08:30:54.227+0000 7f2b3a5a8700 0 [dashboard INFO request] [ > 172.29.2.142:49348] [GET] [200] [0.019s] [admin] [22.0B] /api/prometheus > 2023-06-23T08:30:54.227+0000 7f2b929d2700 0 log_channel(audit) log [DBG] > : from='mon.0 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": > "json"}]: dispatch > 2023-06-23T08:30:54.231+0000 7f2b3d5ae700 0 [dashboard INFO request] [ > 172.29.2.142:39414] [GET] [200] [0.022s] [admin] [9.3K] > /api/prometheus/rules > 2023-06-23T08:30:54.275+0000 7f2ba39d4700 0 log_channel(cluster) log > [DBG] : pgmap v2116480: 145 pgs: 145 active+clean; 2.8 GiB data, 12 GiB > used, 1.5 TiB / 1.5 TiB avail; 5.5 KiB/s wr, 0 op/s > my fs info : # ceph fs ls > name: kube, metadata pool: metadata, data pools: [data ] > thank for your help best regards Karim

3 3

quincy 17.2.6 - write performance continuously slowing down until OSD restart needed
by Nikola Ciprich 22 Jul '23

22 Jul '23

Hello dear CEPH users and developers, we're dealing with strange problems.. we're having 12 node alma linux 9 cluster, initially installed CEPH 15.2.16, then upgraded to 17.2.5. It's running bunch of KVM virtual machines accessing volumes using RBD. everything is working well, but there is strange and for us quite serious issue - speed of write operations (both sequential and random) is constantly degrading drastically to almost unusable numbers (in ~1week it drops from ~70k 4k writes/s from 1 VM to ~7k writes/s) When I restart all OSD daemons, numbers immediately return to normal.. volumes are stored on replicated pool of 4 replicas, on top of 7*12 = 84 INTEL SSDPE2KX080T8 NVMEs. I've updated cluster to 17.2.6 some time ago, but the problem persists. This is especially annoying in connection with https://tracker.ceph.com/issues/56896 as restarting OSDs is quite painfull when half of them crash.. I don't see anything suspicious, nodes load is quite low, no logs errors, network latency and throughput is OK too Anyone having simimar issue? I'd like to ask for hints on what should I check further.. we're running lots of 14.2.x and 15.2.x clusters, none showing similar issue, so I'm suspecting this is something related to quincy thanks a lot in advance with best regards nikola ciprich -- ------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis(a)linuxbox.cz -------------------------------------

7 16

rgw multisite sync not syncing data, error: RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data log shards
by Christian Rohmann 20 Jul '23

20 Jul '23

Hey ceph-users, I setup a multisite sync between two freshly setup Octopus clusters. In the first cluster I created a bucket with some data just to test the replication of actual data later. I then followed the instructions on https://docs.ceph.com/en/octopus/radosgw/multisite/#migrating-a-single-site… to add a second zone. Things went well and both zones are now happily reaching each other and the API endpoints are talking. Also the metadata is in sync already - both sides are happy and I can see bucket listings and users are "in sync": > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > metadata sync no sync (zone is master) > data sync source: c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > and on the other side ... > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > metadata sync syncing > full sync: 0/64 shards > incremental sync: 64/64 shards > metadata is caught up with master > data sync source: 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > also the newly created buckets (read: their metadata) is synced. What is apparently not working in the sync of actual data. Upon startup the radosgw on the second site shows: > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: start > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: realm > epoch=2 period id=f4553d7c-5cc5-4759-9253-9a22b051e736 > 2021-06-25T16:15:11.525+0000 7fe71dff3700 0 > RGW-SYNC:data:sync:init_data_sync_status: ERROR: failed to read remote > data log shards > also when issuing # radosgw-admin data sync init --source-zone obst-rgn it throws > 2021-06-25T16:20:29.167+0000 7f87c2aec080 0 > RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data > log shards Does anybody have any hints on where to look for what could be broken here? Thanks a bunch, Regards Christian

3 4

1 PG stucked in "active+undersized+degraded for long time
by siddhit.renake＠nxtgen.com 19 Jul '23

19 Jul '23

Hello All, Ceph version: 14.2.5-382-g8881d33957 (8881d33957b54b101eae9c7627b351af10e87ee8) nautilus (stable) Issue: 1 PG stucked in "active+undersized+degraded for long time Degraded data redundancy: 44800/8717052637 objects degraded (0.001%), 1 pg degraded, 1 pg undersized #ceph pg dump_stuck PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 15.28f0 active+undersized+degraded [2147483647,343,355,415,426,640,302,392,78,202,607] 343 [2147483647,343,355,415,426,640,302,392,78,202,607] 343 PG Query: #ceph pg 15.28f0 query { "state": "active+undersized+degraded", "snap_trimq": "[]", "snap_trimq_len": 0, "epoch": 303362, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting_recovery_backfill": [ "78(8)", "202(9)", "302(6)", "343(1)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "info": { "pgid": "15.28f0s1", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598853, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "303161'598853", "reported_seq": "3594551", "reported_epoch": "303362", "state": "active+undersized+degraded", "last_fresh": "2023-06-20 19:03:59.135295", "last_change": "2023-06-20 15:11:12.569114", "last_active": "2023-06-20 19:03:59.135295", "last_peered": "2023-06-20 19:03:59.135295", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2023-06-20 15:11:12.569114", "last_became_peered": "2023-06-20 15:11:12.569114", "last_unstale": "2023-06-20 19:03:59.135295", "last_undegraded": "2023-06-20 15:11:10.430426", "last_fullsized": "2023-06-20 15:11:10.430154", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3028, "ondisk_log_size": 3028, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54989065178, "num_objects": 44800, "num_object_clones": 0, "num_object_copies": 492800, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44800, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44800, "num_whiteouts": 0, "num_read": 201078, "num_read_kb": 30408632, "num_write": 219335, "num_write_kb": 59459782, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44800 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, "peer_info": [ { "peer": "78(8)", "pgid": "15.28f0s8", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "202(9)", "pgid": "15.28f0s9", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "302(6)", "pgid": "15.28f0s6", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "355(2)", "pgid": "15.28f0s2", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "392(7)", "pgid": "15.28f0s7", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "415(3)", "pgid": "15.28f0s3", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "426(4)", "pgid": "15.28f0s4", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "607(10)", "pgid": "15.28f0s10", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "640(5)", "pgid": "15.28f0s5", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } } ], "recovery_state": [ { "name": "Started/Primary/Active", "enter_time": "2023-06-20 15:11:10.430274", "might_have_unfound": [], "recovery_progress": { "backfill_targets": [], "waiting_on_backfill": [], "last_backfill_started": "MIN", "backfill_info": { "begin": "MIN", "end": "MIN", "objects": [] }, "peer_backfill_info": [], "backfills_in_flight": [], "recovering": [], "pg_backend": { "recovery_ops": [], "read_ops": [] } }, "scrub": { "scrubber.epoch_start": "0", "scrubber.active": false, "scrubber.state": "INACTIVE", "scrubber.start": "MIN", "scrubber.end": "MIN", "scrubber.max_end": "MIN", "scrubber.subset_last_update": "0'0", "scrubber.deep": false, "scrubber.waiting_on_whom": [] } }, { "name": "Started", "enter_time": "2023-06-20 15:11:09.294876" } ], "agent_state": {} } Require assistance to to make this PG active+clean

3 4

Re: RBD with PWL cache shows poor performance compared to cache device
by Matthew Booth 16 Jul '23

16 Jul '23

On Wed, 28 Jun 2023 at 22:44, Ilya Dryomov <idryomov(a)redhat.com> wrote: >> ** TL;DR >> >> In testing, the write latency performance of a PWL-cache backed RBD >> disk was 2 orders of magnitude worse than the disk holding the PWL >> cache. >> >> ** Summary >> >> I was hoping that PWL cache might be a good solution to the problem of >> write latency requirements of etcd when running a kubernetes control >> plane on ceph. Etcd is extremely write latency sensitive and becomes >> unstable if write latency is too high. The etcd workload can be >> characterised by very small (~4k) writes with a queue depth of 1. >> Throughput, even on a busy system, is normally very low. As etcd is >> distributed and can safely handle the loss of un-flushed data from a >> single node, a local ssd PWL cache for etcd looked like an ideal >> solution. > > > Right, this is exactly the use case that the PWL cache is supposed to address. Good to know! >> My expectation was that adding a PWL cache on a local SSD to an >> RBD-backed would improve write latency to something approaching the >> write latency performance of the local SSD. However, in my testing >> adding a PWL cache to an rbd-backed VM increased write latency by >> approximately 4x over not using a PWL cache. This was over 100x more >> than the write latency performance of the underlying SSD. >> >> My expectation was based on the documentation here: >> https://docs.ceph.com/en/quincy/rbd/rbd-persistent-write-log-cache/ >> >> “The cache provides two different persistence modes. In >> persistent-on-write mode, the writes are completed only when they are >> persisted to the cache device and will be readable after a crash. In >> persistent-on-flush mode, the writes are completed as soon as it no >> longer needs the caller’s data buffer to complete the writes, but does >> not guarantee that writes will be readable after a crash. The data is >> persisted to the cache device when a flush request is received.” >> >> ** Method >> >> 2 systems, 1 running single-node Ceph Quincy (17.2.6), the other >> running libvirt and mounting a VM’s disk with librbd (also 17.2.6) >> from the first node. >> >> All performance testing is from the libvirt system. I tested write >> latency performance: >> >> * Inside the VM without a PWL cache >> * Of the PWL device directly from the host (direct to filesystem, no VM) >> * Inside the VM with a PWL cache >> >> I am testing with fio. Specifically I am running a containerised test, >> executed with: >> podman run --volume .:/var/lib/etcd:Z quay.io/openshift-scale/etcd-perf >> >> This container runs: >> fio --rw=write --ioengine=sync --fdatasync=1 >> --directory=/var/lib/etcd --size=100m --bs=8000 --name=etcd_perf >> --output-format=json --runtime=60 --time_based=1 >> >> And extracts sync.lat_ns.percentile["99.000000"] > > > Matthew, do you have the rest of the fio output captured? It would be interesting to see if it's just the 99th percentile that is bad or the PWL cache is worse in general. Sure. With PWL cache: https://paste.openstack.org/show/820504/ Without PWL cache: https://paste.openstack.org/show/b35e71zAwtYR2hjmSRtR/ With PWL cache, 'rbd_cache'=false: https://paste.openstack.org/show/byp8ZITPzb3r9bb06cPf/ >> ** Results >> >> All results were stable across multiple runs within a small margin of error. >> >> * rbd no cache: 1417216 ns >> * pwl cache device: 44288 ns >> * rbd with pwl cache: 5210112 ns >> >> Note that by adding a PWL cache we increase write latency by >> approximately 4x, which is more than 100x than the underlying device. >> >> ** Hardware >> >> 2 x Dell R640s, each with Xeon Silver 4216 CPU @ 2.10GHz and 192G RAM >> Storage under test: 2 x SAMSUNG MZ7KH480HAHQ0D3 SSDs attached to PERC >> H730P Mini (Embedded) >> >> OS installed on rotational disks >> >> N.B. Linux incorrectly detects these disks as rotational, which I >> assume relates to weird behaviour by the PERC controller. I remembered >> to manually correct this on the ‘client’ machine for the PWL cache, >> but at OSD configuration time ceph would have detected them as >> rotational. They are not rotational. >> >> ** Ceph Configuration >> >> CentOS Stream 9 >> >> # ceph version >> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy >> (stable) >> >> Single node installation with cephadm. 2 OSDs, one on each SSD. >> 1 pool with size 2 >> >> ** Client Configuration >> >> Fedora 38 >> Librbd1-17.2.6-3.fc38.x86_64 >> >> PWL cache is XFS filesystem with 4k block size, matching the >> underlying device. The filesystem uses the whole block device. There >> is no other load on the system. >> >> ** RBD Configuration >> >> # rbd config image list libvirt-pool/pwl-test | grep cache >> rbd_cache true > > > I wonder if rbd_cache should have been set to false here to disable the default volatile cache. Other than that, I don't see anything obviously wrong with the configuration at first sight. I added some full output for this above. > > -- > Ilya > >> config >> rbd_cache_block_writes_upfront false >> config >> rbd_cache_max_dirty 25165824 >> config >> rbd_cache_max_dirty_age 1.000000 >> config >> rbd_cache_max_dirty_object 0 >> config >> rbd_cache_policy writeback >> pool >> rbd_cache_size 33554432 >> config >> rbd_cache_target_dirty 16777216 >> config >> rbd_cache_writethrough_until_flush true >> pool >> rbd_parent_cache_enabled false >> config >> rbd_persistent_cache_mode ssd >> pool >> rbd_persistent_cache_path /var/lib/libvirt/images/pwl >> pool >> rbd_persistent_cache_size 1073741824 >> config >> rbd_plugins pwl_cache >> pool >> >> # rbd status libvirt-pool/pwl-test >> Watchers: >> watcher=10.1.240.27:0/1406459716 client.14475 >> cookie=140282423200720 >> Persistent cache state: >> host: dell-r640-050 >> path: >> /var/lib/libvirt/images/pwl/rbd-pwl.libvirt-pool.37e947fd216b.pool >> size: 1 GiB >> mode: ssd >> stats_timestamp: Mon Jun 26 11:29:21 2023 >> present: true empty: false clean: true >> allocated: 180 MiB >> cached: 135 MiB >> dirty: 0 B >> free: 844 MiB >> hits_full: 1 / 0% >> hits_partial: 3 / 0% >> misses: 21952 >> hit_bytes: 6 KiB / 0% >> miss_bytes: 349 MiB -- Matthew Booth

4 16

Re: Ceph Mgr/Dashboard Python depedencies: a new approach
by Casey Bodley 14 Jul '23

14 Jul '23

hi Ernesto and lists, > [1] https://github.com/ceph/ceph/pull/47501 are we planning to backport this to quincy so we can support centos 9 there? enabling that upgrade path on centos 9 was one of the conditions for dropping centos 8 support in reef, which i'm still keen to do if not, can we find another resolution to https://tracker.ceph.com/issues/58832? as i understand it, all of those python packages exist in centos 8. do we know why they were dropped for centos 9? have we looked into making those available in epel? (cc Ken and Kaleb) On Fri, Sep 2, 2022 at 12:01 PM Ernesto Puerta <epuertat(a)redhat.com> wrote: > > Hi Kevin, > >> >> Isn't this one of the reasons containers were pushed, so that the packaging isn't as big a deal? > > > Yes, but the Ceph community has a strong commitment to provide distro packages for those users who are not interested in moving to containers. > >> Is it the continued push to support lots of distros without using containers that is the problem? > > > If not a problem, it definitely makes it more challenging. Compiled components often sort this out by statically linking deps whose packages are not widely available in distros. The approach we're proposing here would be the closest equivalent to static linking for interpreted code (bundling). > > Thanks for sharing your questions! > > Kind regards, > Ernesto > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io

4 10

cephadm upgrade 16.2.10 to 16.2.11: osds crash and get stuck restarting
by Zakhar Kirpichenko 14 Jul '23

14 Jul '23

Hi, Attempted to upgrade 16.2.10 to 16.2.11, 2 OSDs out of many started crashing in a loop on the very 1st host: Jan 25 23:07:53 ceph01 bash[2553123]: Uptime(secs): 0.0 total, 0.0 interval Jan 25 23:07:53 ceph01 bash[2553123]: Flush(GB): cumulative 0.000, interval 0.000 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(GB): cumulative 0.000, interval 0.000 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(Total Files): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(L0 Files): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(Keys): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Jan 25 23:07:53 ceph01 bash[2553123]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Jan 25 23:07:53 ceph01 bash[2553123]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count Jan 25 23:07:53 ceph01 bash[2553123]: ** File Read Latency Histogram By Level [P] ** Jan 25 23:07:53 ceph01 bash[2553123]: debug -10> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986439) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -9> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986493) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -8> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986500) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -7> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986505) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -6> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: [compaction/compaction_job.cc:1676] [O-2] [JOB 9] Compacting 4@0 + 2@1 files to L1, score 1.0 0 Jan 25 23:07:53 ceph01 bash[2553123]: debug -5> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: [compaction/compaction_job.cc:1680] [O-2] Compaction start summary: Base version 33 Base level 0, inputs: [649058(959KB) 649046(1510KB) 649024(1323KB) 649002(1396KB)], [648981(66MB) 648982(52MB)] Jan 25 23:07:53 ceph01 bash[2553123]: debug -4> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1674688072986547, "job": 9, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [649058, 649046, 649024, 649002], "files_L1": [648981, 648982], "score": 1, "input_data_size": 129161327} Jan 25 23:07:53 ceph01 bash[2553123]: debug -3> 2023-01-25T23:07:52.990+0000 7f7c8b66e700 -1 bdev(0x5619bf0ce400 /var/lib/ceph/osd/ceph-3/block) _aio_thread got r=-1 ((1) Operation not permitted) Jan 25 23:07:53 ceph01 bash[2553123]: debug -2> 2023-01-25T23:07:52.990+0000 7f7c8b66e700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.11/rpm/el8/BUILD/ceph-16.2.11/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f7c8b66e700 time 2023-01-25T23:07:52.993976+0000 Jan 25 23:07:53 ceph01 bash[2553123]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.11/rpm/el8/BUILD/ceph-16.2.11/src/blk/kernel/KernelDevice.cc: 604: ceph_abort_msg("Unexpected IO error. This may suggest HW issue. Please check your dmesg!") Jan 25 23:07:53 ceph01 bash[2553123]: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) Jan 25 23:07:53 ceph01 bash[2553123]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x5619b2fc1adc] Jan 25 23:07:53 ceph01 bash[2553123]: 2: (KernelDevice::_aio_thread()+0x1285) [0x5619b3b2a4e5] Jan 25 23:07:53 ceph01 bash[2553123]: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5619b3b357b1] Jan 25 23:07:53 ceph01 bash[2553123]: 4: /lib64/libpthread.so.0(+0x81ca) [0x7f7c978851ca] Jan 25 23:07:53 ceph01 bash[2553123]: 5: clone() Jan 25 23:07:53 ceph01 bash[2553123]: debug -1> 2023-01-25T23:07:52.994+0000 7f7c8b66e700 -1 *** Caught signal (Aborted) ** Jan 25 23:07:53 ceph01 bash[2553123]: in thread 7f7c8b66e700 thread_name:bstore_aio Jan 25 23:07:53 ceph01 bash[2553123]: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) Jan 25 23:07:53 ceph01 bash[2553123]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7c9788fcf0] Jan 25 23:07:53 ceph01 bash[2553123]: 2: gsignal() Jan 25 23:07:53 ceph01 bash[2553123]: 3: abort() Jan 25 23:07:53 ceph01 bash[2553123]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x5619b2fc1bad] Jan 25 23:07:53 ceph01 bash[2553123]: 5: (KernelDevice::_aio_thread()+0x1285) [0x5619b3b2a4e5] Jan 25 23:07:53 ceph01 bash[2553123]: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5619b3b357b1] Jan 25 23:07:53 ceph01 bash[2553123]: 7: /lib64/libpthread.so.0(+0x81ca) [0x7f7c978851ca] OSD kept crashing until the host reboot, an OSD restart wouldn't help. This hasn't happened during any previous upgrades, so was a rather unexpected development. Unclear what caused this, but a host reboot seems to have fixed it. It happened to 1 other OSD on another host, exactly the same symptoms, also solved by a reboot. Best regards, Zakhar

4 4

Reef release candidate - v18.1.2
by Yuri Weinstein 12 Jul '23

12 Jul '23

Hi everyone, This is the second release candidate for Reef. The Reef release comes with a new RockDB version (7.9.2) [0], which incorporates several performance improvements and features. Our internal testing doesn't show any side effects from the new version, but we are very eager to hear community feedback on it. This is the first release to have the ability to tune RockDB settings per column family [1], which allows for more granular tunings to be applied to different kinds of data stored in RocksDB. A new set of settings has been used in Reef to optimize performance for most kinds of workloads with a slight penalty in some cases, outweighed by large improvements in use cases such as RGW, in terms of compactions and write amplification. We would highly encourage community members to give these a try against their performance benchmarks and use cases. The detailed list of changes in terms of RockDB and BlueStore can be found in https://pad.ceph.com/p/reef-rc-relnotes. If any of our community members would like to help us with performance investigations or regression testing of the Reef release candidate, please feel free to provide feedback via email or in https://pad.ceph.com/p/reef_scale_testing. For more active discussions, please use the #ceph-at-scale slack channel in ceph-storage.slack.com. This RC has gone thru partial testing due to issues we are experiencing in the sepia lab. Please try it out and report any issues you encounter. Happy testing! Thanks, YuriW

4 6

CephFS snapshots: impact of moving data
by Kuhring, Mathias 06 Jul '23

06 Jul '23

Dear Ceph community, We want to restructure (i.e. move around) a lot of data (hundreds of terabyte) in our CephFS. And now I was wondering what happens within snapshots when I move data around within a snapshotted folder. I.e. do I need to account for a lot increased storage usage due to older snapshots differing from the new restructured state? In the end it is just metadata changes. Are the snapshots aware of this? Consider the following examples. Copying data: Let's say I have a folder /test, with a file XYZ in sub-folder /test/sub1 and an empty sub-folder /test/sub2. I create snapshot snapA in /test/.snap, copy XYZ to sub-folder /test/sub2, delete it from /test/sub1 and create another snapshot snapB. I would have two snapshots each with distinct copies of XYZ, hence using double the space in the FS: /test/.snap/snapA/sub1/XYZ <-- copy 1 /test/.snap/snapA/sub2/ /test/.snap/snapB/sub1/ /test/.snap/snapB/sub2/XYZ <-- copy 2 Moving data: Let's assume the same structure. But now after creating snapshot snapA, I move XYZ to sub-folder /test/sub2 and then create the other snapshot snapB. The directory tree will look the same. But how is this treated internally? Once I move the data, will there be an actually copy created in snapA to represent the old state? Or will this remain the same data (like a link to the inode or so)? And hence not double the storage used for that file. I couldn't find (or understand) anything related to this in the docs. The closest seems to be the hard-link section here: https://docs.ceph.com/en/quincy/dev/cephfs-snapshots/#hard-links Which unfortunately goes a bit over my head. So I'm not sure if this answers my question. Thank you all for your help. Appreciate it. Best Wishes, Mathias Kuhring

3 2

RGW accessing real source IP address of a client (e.g. in S3 bucket policies)
by Christian Rohmann 06 Jul '23

06 Jul '23

Hello Ceph-Users, context or motivation of my question is S3 bucket policies and other cases using the source IP address as condition. I was wondering if and how RadosGW is able to access the source IP address of clients if receiving their connections via a loadbalancer / reverse proxy like HAProxy. So naturally that is where the connection originates from in that case, rendering a policy based on IP addresses useless. Depending on whether the connection balanced as HTTP or TCP there are two ways to carry information about the actual source: * In case of HTTP via headers like "X-Forwarded-For". This is apparently supported only for logging the source in the "rgw ops log" ([1])? Or is this info used also when evaluating the source IP condition within a bucket policy? * In case of TCP loadbalancing, there is the proxy protocol v2. This unfortunately seems not even supposed by the BEAST library which RGW uses. I opened feature requests ... ** https://tracker.ceph.com/issues/59422 ** https://github.com/chriskohlhoff/asio/issues/1091 ** https://github.com/boostorg/beast/issues/2484 but there is no outcome yet. Regards Christian [1] https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_remote_addr…

3 5

[multisite] The purpose of zonegroup
by Yixin Jin 05 Jul '23

05 Jul '23

Hi folks, In the multisite environment, we can get one realm that contains multiple zonegroups, each in turn can have multiple zones. However, the purpose of zonegroup isn't clear to me. It seems that when a user is created, its metadata is synced to all zones within the same realm, regardless whether they are in different zonegroups or not. The same happens to buckets. Therefore, what is the purpose of having zonegroups? Wouldn't it be easier to just have realm and zones? Thanks,Yixin

4 6

Ceph iSCSI GW not working with VMware VMFS and Windows Clustered Storage Volumes (CSV)
by Work Ceph 04 Jul '23

04 Jul '23

Hello guys, We have a Ceph cluster that runs just fine with Ceph Octopus; we use RBD for some workloads, RadosGW (via S3) for others, and iSCSI for some Windows clients. Recently, we had the need to add some VMWare clusters as clients for the iSCSI GW and also Windows systems with the use of Clustered Storage Volumes (CSV), and we are facing a weird situation. In windows for instance, the iSCSI block can be mounted, formatted and consumed by all nodes, but when we add in the CSV it fails with some generic exception. The same happens in VMWare, when we try to use it with VMFS it fails. We do not seem to find the root cause for these errors. However, the errors seem to be linked to the situation of multiple nodes consuming the same block by shared file systems. Have you guys seen this before? Are we missing some basic configuration in the iSCSI GW?

4 7

Re: Ceph iSCSI GW is too slow when compared with Raw RBD performance
by Work Ceph 04 Jul '23

04 Jul '23

Awesome, thanks for the info! By any chance, do you happen to know what configurations you needed to adjust to make Veeam perform a bit better? On Fri, Jun 23, 2023 at 10:42 AM Anthony D'Atri <aad(a)dreamsnake.net> wrote: > Yes, with someone I did some consulting for. Veeam seems to be one of the > prevalent uses for ceph-iscsi, though I'd try to use the native RBD client > instead if possible. > > Veeam appears by default to store really tiny blocks, so there's a lot of > protocol overhead. I understand that Veeam can be configured to use "large > blocks" that can make a distinct difference. > > > > On Jun 23, 2023, at 09:33, Work Ceph <work.ceph.user.mailing(a)gmail.com> > wrote: > > Great question! > > Yes, one of the slowness was detected in a Veeam setup. Have you > experienced that before? > > On Fri, Jun 23, 2023 at 10:32 AM Anthony D'Atri <aad(a)dreamsnake.net> > wrote: > >> Are you using Veeam by chance? >> >> > On Jun 22, 2023, at 21:18, Work Ceph <work.ceph.user.mailing(a)gmail.com> >> wrote: >> > >> > Hello guys, >> > >> > We have a Ceph cluster that runs just fine with Ceph Octopus; we use RBD >> > for some workloads, RadosGW (via S3) for others, and iSCSI for some >> Windows >> > clients. >> > >> > We started noticing some unexpected performance issues with iSCSI. I >> mean, >> > an SSD pool is reaching 100MB of write speed for an image, when it can >> > reach up to 600MB+ of write speed for the same image when mounted and >> > consumed directly via RBD. >> > >> > Is that performance degradation expected? We would expect some >> degradation, >> > but not as much as this one. >> > >> > Also, we have a question regarding the use of Intel Turbo boost. Should >> we >> > disable it? Is it possible that the root cause of the slowness in the >> iSCSI >> > GW is caused by the use of Intel Turbo boost feature, which reduces the >> > clock of some cores? >> > >> > Any feedback is much appreciated. >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >

2 2

radosgw new zonegroup hammers master with metadata sync
by Boris Behrens 04 Jul '23

04 Jul '23

Hi, yesterday I added a new zonegroup and it looks like it seems to cycle over the same requests over and over again. In the log of the main zone I see these requests: 2023-06-20T09:48:37.979+0000 7f8941fb3700 1 beast: 0x7f8a602f3700: fd00:2380:0:24::136 - - [2023-06-20T09:48:37.979941+0000] "GET /admin/log?type=metadata&id=62&period=e8fc96f1-ae86-4dc1-b432-470b0772fded&max-entries=100&&rgwx-zonegroup=b39392eb-75f8-47f0-b4f3-7d3882930b26 HTTP/1.1" 200 44 - - - Only thing that changes is the &id. We have two other zonegroups that are configured identical (ceph.conf and period) and these don;t seem to spam the main rgw. root@host:~# radosgw-admin sync status realm 5d6f2ea4-b84a-459b-bce2-bccac338b3ef (main) zonegroup b39392eb-75f8-47f0-b4f3-7d3882930b26 (dc3) zone 96f5eca9-425b-4194-a152-86e310e91ddb (dc3) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master root@host:~# radosgw-admin period get { "id": "e8fc96f1-ae86-4dc1-b432-470b0772fded", "epoch": 92, "predecessor_uuid": "5349ac85-3d6d-4088-993f-7a1d4be3835a", "sync_status": [ "", ... "" ], "period_map": { "id": "e8fc96f1-ae86-4dc1-b432-470b0772fded", "zonegroups": [ { "id": "b39392eb-75f8-47f0-b4f3-7d3882930b26", "name": "dc3", "api_name": "dc3", "is_master": "false", "endpoints": [ ], "hostnames": [ ], "hostnames_s3website": [ ], "master_zone": "96f5eca9-425b-4194-a152-86e310e91ddb", "zones": [ { "id": "96f5eca9-425b-4194-a152-86e310e91ddb", "name": "dc3", "endpoints": [ ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "5d6f2ea4-b84a-459b-bce2-bccac338b3ef", "sync_policy": { "groups": [] } }, ... -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

3 5

upgrading from 15.2.17 to 16.2.11 - Health ERROR
by xadhoom76＠gmail.com 04 Jul '23

04 Jul '23

hi , starting upgrade from 15.2.17 i got this error Module 'cephadm' has failed: Expecting value: line 1 column 1 (char 0) Cluster was in health ok before starting.

5 9

db/wal pvmoved ok, but gui show old metadatas
by Christophe BAILLON 03 Jul '23

03 Jul '23

Hello, we have a Ceph 17.2.5 cluster with a total of 26 nodes, where 15 nodes that have faulty NVMe drives, where the db/wal resides (one NVMe for the first 6 OSDs and another for the remaining 6). We replaced them with new drives and pvmoved it to avoid losing the OSDs. So far, there are no issues, and the OSDs are functioning properly. ceph see the correct news disks root@node02:/# ceph daemon osd.26 list_devices [ { "device": "/dev/nvme0n1", "device_id": "INTEL_SSDPEDME016T4S_CVMD516500851P6KGN" }, { "device": "/dev/sdc", "device_id": "SEAGATE_ST18000NM004J_ZR52TT830000C148JFSJ" } ] However, the Cephadm GUI still shows the old NVMe drives and hasn't recognized the device change. How can we make the GUI and Cephadm recognize the new devices? I tried restarting the managers, thinking that it would rescan the OSDs during startup, but it didn't work. If you have any ideas, I would appreciate it. Should I perform something like that: ceph orch daemon reconfig osd.* Thank you for your help.

1 1

RadosGW strange behavior when using a presigned url generated by SDK PHP
by Huy Nguyen 03 Jul '23

03 Jul '23

Hi, I tried to generate a presigned url using SDK PHP, but it doesn't work. (I also tried to use boto3 with the same configures and the url works normally) Here is my php code: <?php require 'aws-autoloader.php'; use Aws\S3\S3Client; use Aws\Exception\AwsException; $s3Client = new Aws\S3\S3Client([ 'version' => '2006-03-01', 'region' => 'us-east-1', 'signature_version' => 'v4', 'use_path_style_endpoint' => true, 'endpoint' => 'http://hn.ss.bfcplatform.vn', 'credentials' => [ 'key' => 'DNMZAFE6G2PP8H9P05UU', 'secret' => 'XXX', ] ]); $cmd = $s3Client->getCommand('PutObject', [ 'Bucket' => 'huynnp-testbucket1', 'Key' => 'testfile.txt', ]); $request = $s3Client->createPresignedRequest($cmd, '+60 minutes'); // Set the expiration time as desired $presignedUrl = (string)$request->getUri(); echo "$presignedUrl"; ?> and then: curl -X PUT -T testfile.txt `php s3.php` <?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><RequestId>tx00000b7bb3b2deb6a6ef2-00649d5ebd-d1d50041-hn-1</RequestId><HostId>d1d50041-hn-1-hn</HostId></Error> I enable the debug_rgw and what I can see is really strange. the domain has been added :8084, so it make "canonical request hash" and "signature" between client and server unmatched. I can't explain why does this happens 2023-06-29T17:10:46.880+0700 7f26014b0700 10 v4 credential format = DNMZAFE6G2PP8H9P05UU/20230629/us-east-1/s3/aws4_request 2023-06-29T17:10:46.880+0700 7f26014b0700 10 access key id = DNMZAFE6G2PP8H9P05UU 2023-06-29T17:10:46.880+0700 7f26014b0700 10 credential scope = 20230629/us-east-1/s3/aws4_request 2023-06-29T17:10:46.880+0700 7f26014b0700 10 req 15647562574720867919 1000005ns canonical headers format = host:hn.ss.bfcplatform.vn:8084 2023-06-29T17:10:46.880+0700 7f26014b0700 10 payload request hash = UNSIGNED-PAYLOAD 2023-06-29T17:10:46.880+0700 7f26014b0700 10 canonical request = PUT /huynnp-testbucket1/testfile.txt X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=DNMZAFE6G2PP8H9P05UU%2F20230629%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230629T101046Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host host:hn.ss.bfcplatform.vn:8084 host UNSIGNED-PAYLOAD 2023-06-29T17:10:46.880+0700 7f26014b0700 10 canonical request hash = d28e6c3104aff99e9928f902892627d2b284a29d489fbb034ed5c90aa21c566a 2023-06-29T17:10:46.880+0700 7f26014b0700 10 string to sign = AWS4-HMAC-SHA256 20230629T101046Z 20230629/us-east-1/s3/aws4_request d28e6c3104aff99e9928f902892627d2b284a29d489fbb034ed5c90aa21c566a

2 4

Quincy osd bench in order to define osd_mclock_max_capacity_iops_[hdd|ssd]
by Rafael Diaz Maurin 30 Jun '23

30 Jun '23

Hello, I've just upgraded a Pacific cluster into Quincy, and all my osd have the low value osd_mclock_max_capacity_iops_hdd : 315.000000. The manuel does not explain how to benchmark the OSD with fio or ceph bench with good options. Can someone have the good ceph bench options or fio options in order to configure osd_mclock_max_capacity_iops_hdd for each osd ? I ran this bench various times on the same OSD (class hdd) and I obtain different results. ceph tell ${osd} cache drop ceph tell ${osd} bench 12288000 4096 4194304 100 example : osd.21 (hdd): osd_mclock_max_capacity_iops_hdd = 315.000000 bench 1 : 3006.2271379745534 bench 2 : 819.503206458996 bench 3 : 946.5406320134085 How can I succeed in getting the good values for the osd_mclock_max_capacity_iops_[hdd|ssd] options ? Thank you for your help, Rafael

2 2

ceph-fuse crash
by hakesudu＠gmail.com 30 Jun '23

30 Jun '23

Hi, I've deployed a ceph-quincy for HPC. Recently, I always encounter the problem of ceph-fuse crash kernel version is 4.18.0-348.el8.0.2.x86_64 here is part of ceph-fuse log: -59> 2023-06-28T09:51:00.452+0800 155546ff7700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -58> 2023-06-28T09:51:00.452+0800 15554cc49700 3 client.159239 ll_opendir 0x10003e1408d.head -57> 2023-06-28T09:51:00.452+0800 15554cc49700 3 client.159239 may_open 0x1554e79123d0 = 0 -56> 2023-06-28T09:51:00.452+0800 15554cc49700 3 client.159239 ll_opendir 0x10003e1408d.head = 0 (0x155328079380) -55> 2023-06-28T09:51:00.453+0800 1555473f9700 3 client.159239 seekdir(0x155328079380, 0) -54> 2023-06-28T09:51:00.452+0800 155546bf5700 5 client.159239 put_cap_ref dropped last FILE_CACHE ref on 0x20004a6e548.head(faked_ino=0 nref=14 ll_ref=2 cap_refs={1024=0,2048=1} open={1=1} mode=100644 size=5626/0 nlink=1 btime=2023-06-05T14:38:36.471178+0800 mtime=2023-06-05T14:38:36.471178+0800 ctime=2023-06-05T14:38:36.471178+0800 change_attr=1 caps=pAsLsXsFscr(0=pAsLsXsFscr) objectset[0x20004a6e548 ts 0/0 objects 1 dirty_or_tx 0] 0x1554e7902300) -53> 2023-06-28T09:51:00.453+0800 155546bf5700 3 client.159239 ll_read 0x15531806b970 0~8192 = 5626 -52> 2023-06-28T09:51:00.453+0800 155546ff7700 3 client.159239 may_lookup 0x155420004840 = 0 -51> 2023-06-28T09:51:00.453+0800 1554a5dee700 3 client.159239 ll_lookup 0x10003e1406f.head html -50> 2023-06-28T09:51:00.453+0800 155546ff7700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -> 0 (100019b7a43) -49> 2023-06-28T09:51:00.453+0800 1554a5dee700 3 client.159239 may_lookup 0x1554c1a89e40 = 0 -48> 2023-06-28T09:51:00.453+0800 1554a7dfe700 3 client.159239 ll_flush 0x15531806b970 0x20004a6e548 -47> 2023-06-28T09:51:00.453+0800 1555469f4700 3 client.159239 ll_lookup 0x100019b7a43.head envs -46> 2023-06-28T09:51:00.453+0800 1555469f4700 3 client.159239 may_lookup 0x155420343eb0 = 0 -45> 2023-06-28T09:51:00.453+0800 1555469f4700 3 client.159239 ll_lookup 0x100019b7a43.head envs -> 0 (200035de5d8) -44> 2023-06-28T09:51:00.453+0800 155545fef700 3 client.159239 ll_release (fh)0x15531806b970 0x20004a6e548 -43> 2023-06-28T09:51:00.453+0800 155546bf5700 3 client.159239 seekdir(0x15544c054010, 1152360438801891331) -42> 2023-06-28T09:51:00.453+0800 155546ff7700 3 client.159239 seekdir(0x15544c029ee0, 1152690945930559491) -41> 2023-06-28T09:51:00.453+0800 15554cc49700 3 client.159239 ll_releasedir 0x15544c029ee0 -40> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 ll_lookup 0x20004a66459.head tests -39> 2023-06-28T09:51:00.453+0800 15554c244700 3 client.159239 ll_lookup 0x100040bd04e.head att5410-w -> 0 (200047cbc02) -38> 2023-06-28T09:51:00.453+0800 1554a7dfe700 3 client.159239 ll_lookup 0x200035de5d8.head steven-colossal -37> 2023-06-28T09:51:00.453+0800 1554a7dfe700 3 client.159239 may_lookup 0x1554c20229c0 = 0 -36> 2023-06-28T09:51:00.453+0800 1554a7dfe700 3 client.159239 ll_lookup 0x200035de5d8.head steven-colossal -> 0 (100040bcf3a) -35> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 may_lookup 0x155533c970f0 = 0 -34> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 ll_lookup 0x20004a66459.head tests -> 0 (20004a6e3e0) -33> 2023-06-28T09:51:00.453+0800 155545bed700 3 client.159239 ll_releasedir 0x15544c054010 -32> 2023-06-28T09:51:00.453+0800 155546bf5700 3 client.159239 ll_getattr 0x200047cbc02.head = 0 -31> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -30> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 may_lookup 0x155420004840 = 0 -29> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -> 0 (100019b7a43) -28> 2023-06-28T09:51:00.453+0800 155545fef700 3 client.159239 ll_lookup 0x100040bcf3a.head lib -27> 2023-06-28T09:51:00.453+0800 155545fef700 3 client.159239 may_lookup 0x1554d7d1d240 = 0 -26> 2023-06-28T09:51:00.453+0800 155545fef700 3 client.159239 ll_lookup 0x100040bcf3a.head lib -> 0 (100040bd018) -25> 2023-06-28T09:51:00.453+0800 1555469f4700 3 client.159239 ll_lookup 0x20004a6e3e0.head test_trainer -24> 2023-06-28T09:51:00.453+0800 1555469f4700 3 client.159239 may_lookup 0x155531e5a6c0 = 0 -23> 2023-06-28T09:51:00.453+0800 15554cc49700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -22> 2023-06-28T09:51:00.453+0800 15554cc49700 3 client.159239 may_lookup 0x155420004840 = 0 -21> 2023-06-28T09:51:00.453+0800 15554cc49700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -> 0 (100019b7a43) -20> 2023-06-28T09:51:00.453+0800 15554c244700 3 client.159239 ll_lookup 0x100019b7a43.head envs -19> 2023-06-28T09:51:00.453+0800 15554c244700 3 client.159239 may_lookup 0x155420343eb0 = 0 -18> 2023-06-28T09:51:00.453+0800 15554c244700 3 client.159239 ll_lookup 0x100019b7a43.head envs -> 0 (200035de5d8) -17> 2023-06-28T09:51:00.453+0800 155546ff7700 3 client.159239 ll_lookup 0x100019b7a43.head envs -16> 2023-06-28T09:51:00.453+0800 155546ff7700 3 client.159239 may_lookup 0x155420343eb0 = 0 -15> 2023-06-28T09:51:00.453+0800 155546ff7700 3 client.159239 ll_lookup 0x100019b7a43.head envs -> 0 (200035de5d8) -14> 2023-06-28T09:51:00.453+0800 155545bed700 3 client.159239 ll_lookup 0x100040bd018.head terminfo -13> 2023-06-28T09:51:00.453+0800 1555475fa700 3 client.159239 ll_lookup 0x1000383ca0a.head train.py -> 0 (1000383caf7) -12> 2023-06-28T09:51:00.453+0800 1555465f2700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -11> 2023-06-28T09:51:00.453+0800 1555465f2700 3 client.159239 may_lookup 0x155420004840 = 0 -10> 2023-06-28T09:51:00.453+0800 1555465f2700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -> 0 (100019b7a43) -9> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 ll_lookup 0x200035de5d8.head steven-colossal -8> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 may_lookup 0x1554c20229c0 = 0 -7> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 ll_lookup 0x200035de5d8.head steven-colossal -> 0 (100040bcf3a) -6> 2023-06-28T09:51:00.453+0800 15554cc49700 3 client.159239 ll_lookup 0x100019b7a43.head envs -5> 2023-06-28T09:51:00.453+0800 1555465f2700 3 client.159239 ll_getattr 0x1000383caf7.head = 0 -4> 2023-06-28T09:51:00.453+0800 155546bf5700 3 client.159239 ll_lookup 0x200035de5d8.head llmzoo -3> 2023-06-28T09:51:00.453+0800 155546bf5700 3 client.159239 may_lookup 0x1554c20229c0 = 0 -2> 2023-06-28T09:51:00.453+0800 155546bf5700 3 client.159239 ll_lookup 0x200035de5d8.head llmzoo -> 0 (10003e11a9e) -1> 2023-06-28T09:51:00.453+0800 15554c646700 3 client.159239 ll_lookup 0x10003e11a9e.head lib 0> 2023-06-28T09:51:00.458+0800 1554a77fb700 -1 *** Caught signal (Segmentation fault) ** in thread 1554a77fb700 thread_name:ceph-fuse ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) 1: /lib64/libpthread.so.0(+0x12ce0) [0x1555535eece0] 2: (Client::_readdir_cache_cb(dir_result_t*, int (*)(void*, dirent*, ceph_statx*, long, Inode*), void*, int, bool)+0x2f4) [0x555555647d64] 3: (Client::readdir_r_cb(dir_result_t*, int (*)(void*, dirent*, ceph_statx*, long, Inode*), void*, unsigned int, unsigned int, bool)+0xae7) [0x55555564cd37] 4: ceph-fuse(+0xadbf8) [0x555555601bf8] 5: /lib64/libfuse.so.2(+0x16706) [0x1555550fd706] 6: /lib64/libfuse.so.2(+0x17868) [0x1555550fe868] 7: /lib64/libfuse.so.2(+0x14440) [0x1555550fb440] 8: /lib64/libpthread.so.0(+0x81cf) [0x1555535e41cf] 9: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_pwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 1/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/ 5 rgw_datacache 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 fuse 2/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test 0/ 5 cephfs_mirror 0/ 5 cephsqlite 0/ 5 seastore 0/ 5 seastore_onode 0/ 5 seastore_odata 0/ 5 seastore_omap 0/ 5 seastore_tm 0/ 5 seastore_cleaner 0/ 5 seastore_lba 0/ 5 seastore_cache 0/ 5 seastore_journal 0/ 5 seastore_device 0/ 5 alienstore 1/ 5 mclock 1/ 5 ceph_exporter -2/-2 (syslog threshold) -1/-1 (stderr threshold) --- pthread ID / name mapping for recent threads --- 1554947e3700 / ceph-fuse 155494fe7700 / ceph-fuse 1554975fa700 / ceph-fuse 1554a57eb700 / ceph-fuse 1554a59ec700 / ceph-fuse 1554a5dee700 / ceph-fuse 1554a5fef700 / ceph-fuse 1554a61f0700 / 1554a63f1700 / ceph-fuse 1554a65f2700 / ceph-fuse 1554a67f3700 / ceph-fuse 1554a6bf5700 / ceph-fuse 1554a6df6700 / ceph-fuse 1554a71f8700 / ceph-fuse 1554a73f9700 / ceph-fuse 1554a75fa700 / 1554a77fb700 / ceph-fuse 1554a7dfe700 / ceph-fuse 1554a7fff700 / ceph-fuse 1555452da700 / ceph-fuse 1555455ea700 / ceph-fuse 1555459ec700 / ceph-fuse 155545bed700 / ceph-fuse 155545fef700 / ceph-fuse 1555465f2700 / ceph-fuse 1555469f4700 / ceph-fuse 155546bf5700 / ceph-fuse 155546df6700 / ceph-fuse 155546ff7700 / ceph-fuse 1555471f8700 / ceph-fuse 1555473f9700 / ceph-fuse 1555475fa700 / ceph-fuse 1555477fb700 / 155547dfe700 / ceph-fuse 155547fff700 / ceph-fuse 15554c244700 / ceph-fuse 15554c646700 / ceph-fuse 15554c847700 / ceph-fuse 15554cc49700 / ceph-fuse 15554ce4a700 / ceph-fuse 15554da50700 / ms_dispatch max_recent 10000 max_new 10000 log_file /var/lib/ceph/crash/2023-06-28T01:51:00.459615Z_3ddbaa44-d8cd-437b-908b-c3772520c7a6/log --- end dump of recent events --- Has anyone encountered this kind of problem?

2 1

Quincy osd bench in order to define osd_mclock_max_capacity_iops_[hdd|ssd]
by Rafael Diaz Maurin 30 Jun '23

30 Jun '23

Hello, I've just upgraded a Pacific cluster into Quincy, and all my osd have the low value osd_mclock_max_capacity_iops_hdd : 315.000000. The manuel does not explain how to benchmark the OSD with fio or ceph bench with good options. Can someone have the good ceph bench options or fio options in order to configure osd_mclock_max_capacity_iops_hdd for each osd ? I ran this bench various times on the same OSD (class hdd) and I obtain different results. ceph tell ${osd} cache drop ceph tell ${osd} bench 12288000 4096 4194304 100 example : osd.21 (hdd): osd_mclock_max_capacity_iops_hdd = 315.000000 bench 1 : 3006.2271379745534 bench 2 : 819.503206458996 bench 3 : 946.5406320134085 How can I succeed in getting the good values for the osd_mclock_max_capacity_iops_[hdd|ssd] options ? Thank you for your help, Rafael

1 0

warning: CEPHADM_APPLY_SPEC_FAIL
by Adiga, Anantha 30 Jun '23

30 Jun '23

Hi, I am not finding any reference to clear this warning AND stop the service. See below After creating OSD with iops_optimized option, this WARN mesg appear. Ceph 17.2.6 [cid:image001.png@01D9AAA5.8639A1F0] 6/29/23 4:10:45 PM [WRN] Health check failed: Failed to apply 1 service(s): osd.iops_optimized (CEPHADM_APPLY_SPEC_FAIL) 6/29/23 4:10:45 PM [ERR] Failed to apply osd.iops_optimized spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: iops_optimized service_name: osd.iops_optimized placement: host_pattern: '*' spec: data_devices: rotational: 0 filter_logic: AND objectstore: bluestore ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/mon.fl31ca104ja0203/config Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e -e NODE_NAME=fl31ca104ja0203 -e #:/var/log/ceph# grep optimi cephadm.log cephadm ['--env', 'CEPH_VOLUME_OSDSPEC_AFFINITY=iops_optimized', '--image', 'quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e', 'ceph-volume', '--fsid', 'd0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e', '--config-json', '-', '--', 'lvm', 'batch', '--no-auto', '/dev/nvme10n1', '/dev/nvme11n1', '/dev/nvme12n1', '/dev/nvme13n1', '/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1', '/dev/nvme5n1', '/dev/nvme6n1', '/dev/nvme7n1', '/dev/nvme8n1', '/dev/nvme9n1', '--yes', '--no-systemd'] 2023-06-29 23:06:28,340 7fc2668a7740 INFO Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e -e NODE_NAME=fl31ca104ja0202 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=iops_optimized -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e:/var/run/ceph:z -v /var/log/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e:/var/log/ceph:z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp1v09i0jx:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphy3pnh46:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e lvm batch --no-auto /dev/nvme10n1 /dev/nvme11n1 /dev/nvme12n1 /dev/nvme13n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 /dev/nvme5n1 /dev/nvme6n1 /dev/nvme7n1 /dev/nvme8n1 /dev/nvme9n1 --yes --no-systemd #:/var/log/ceph# grep optimi cephadm.log cephadm ['--env', 'CEPH_VOLUME_OSDSPEC_AFFINITY=iops_optimized', '--image', 'quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e', 'ceph-volume', '--fsid', 'd0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e', '--config-json', '-', '--', 'lvm', 'batch', '--no-auto', '/dev/nvme10n1', '/dev/nvme11n1', '/dev/nvme12n1', '/dev/nvme13n1', '/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1', '/dev/nvme5n1', '/dev/nvme6n1', '/dev/nvme7n1', '/dev/nvme8n1', '/dev/nvme9n1', '--yes', '--no-systemd'] 2023-06-29 23:06:28,340 7fc2668a7740 INFO Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e<mailto:CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e> -e NODE_NAME=fl31ca104ja0202 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=iops_optimized -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e:/var/run/ceph:z -v /var/log/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e:/var/log/ceph:z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp1v09i0jx:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphy3pnh46:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e<mailto:quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e> lvm batch --no-auto /dev/nvme10n1 /dev/nvme11n1 /dev/nvme12n1 /dev/nvme13n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 /dev/nvme5n1 /dev/nvme6n1 /dev/nvme7n1 /dev/nvme8n1 /dev/nvme9n1 --yes --no-systemd The warning message clears on its on for a few seconds..ceph health status foes green, but comes back. I am not finding any reference to clear this msg AND not able to stop the service. /#ceph orch ls node-exporter ?:9100 4/4 8m ago 10w * osd 33 8m ago - <unmanaged> osd.iops_optimized 0 - 2d * /# ceph orch stop --service_name osd.iops_optimized Error EINVAL: No daemons exist under service name "osd.iops_optimized". View currently running services using "ceph orch ls" /# Thank you, Anantha

1 1

1 pg inconsistent and does not recover
by Niklas Hambüchen 29 Jun '23

29 Jun '23

Hi, I have a 3x-replicated pool with Ceph 12.2.7. One HDD broke, its OSD "2" was automatically marked as "out", the disk was physically replaced by a new one, and that added back in. Now `ceph health detail` continues to permanently show: [ERR] OSD_SCRUB_ERRORS: 1 scrub errors [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent pg 2.87 is active+clean+inconsistent, acting [33,2,20] What exactly is wrong here? Why can Ceph not fix the issue? With BlueStore I have checksums, on two unbroken disks, so what remaining inconsistency can there be? The suggested command in https://docs.ceph.com/en/pacific/rados/operations/pg-repair/#commands-for-d… does not work: # rados list-inconsistent-obj 2.87 No scrub information available for pg 2.87 error 2: (2) No such file or directory Further, I find the documentation in https://docs.ceph.com/en/pacific/rados/operations/pg-repair/#more-informati… extremely unclear. It says > In the case of replicated pools, recovery is beyond the scope of pg repair. while many people on the Internet suggest that `ceph pg repair` might fix the issue. Yet again others claim that Ceph will fix the issue itself. I am hesitant to run "ceph pg repair" without understanding what the problem is and what exactly this will do. I have already reported the "error 2" and the documentation in issue https://tracker.ceph.com/issues/61739 but not received a reply yet, and my cluster stays "inconsistent". How can this be fixed? I would appreciate any help!

5 15

device class for nvme disk is ssd
by Boris Behrens 29 Jun '23

29 Jun '23

Hi, is it a problem that the device class for all my disks is SSD even all of these disks are NVME disks? If it is just a classification for ceph, so I can have pools on SSDs and NVMEs separated I don't care. But maybe ceph handles NVME disks differently internally? I've added them via ceph-volume lvm create --bluestore --data /dev/nvme2n1 and they only show up as ssd root@a0423f621aaa:~# ceph osd metadata osd.0 { "id": 0, "arch": "x86_64", ... "bluefs": "1", "bluefs_dedicated_db": "0", "bluefs_dedicated_wal": "0", "bluefs_single_shared_device": "1", "bluestore_bdev_access_mode": "blk", "bluestore_bdev_block_size": "4096", "bluestore_bdev_dev_node": "/dev/dm-2", "bluestore_bdev_devices": "nvme0n1", "bluestore_bdev_driver": "KernelDevice", "bluestore_bdev_partition_path": "/dev/dm-2", "bluestore_bdev_rotational": "0", "bluestore_bdev_size": "1920378863616", "bluestore_bdev_support_discard": "1", "bluestore_bdev_type": "ssd", "ceph_release": "pacific", "ceph_version": "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)", "ceph_version_short": "16.2.13", "ceph_version_when_created": "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)", "cpu": "Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz", "created_at": "2023-06-20T14:03:35.167741Z", "default_device_class": "ssd", "device_ids": "nvme0n1=SAMSUNG_MZQLB1T9HAJR-00007_S439NF0M506164", "device_paths": "nvme0n1=/dev/disk/by-path/pci-0000:5e:00.0-nvme-1", "devices": "nvme0n1", "distro": "ubuntu", "distro_description": "Ubuntu 20.04.6 LTS", "distro_version": "20.04", ... "journal_rotational": "0", "kernel_description": "#169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023", "kernel_version": "5.4.0-152-generic", "mem_swap_kb": "0", "mem_total_kb": "196668116", "network_numa_unknown_ifaces": "back_iface,front_iface", "objectstore_numa_node": "0", "objectstore_numa_nodes": "0", "os": "Linux", "osd_data": "/var/lib/ceph/osd/ceph-0", "osd_objectstore": "bluestore", "osdspec_affinity": "", "rotational": "0" } Cheers Boris

5 6

CLT Meeting Notes June 28th, 2023
by Adam King 28 Jun '23

28 Jun '23

Reef RC linking failure on Alpine Linux. Do we worry about that? 1. https://tracker.ceph.com/issues/61718 2. Nice to fix, but not a requirement 3. If there are patches available, we should accept them, but probably don't put too much work into it currently debian bullseye build failure on reef rc: 1. https://tracker.ceph.com/issues/61845 2. want to fix before final release clean-up in AuthMonitor – CephFS and core are fine. Any other component interested in? 1. https://github.com/ceph/ceph/pull/52008#issuecomment-1606581139 2. Already tested for rados and cephfs 3. No other components requesting testing Reef rc v18.1.2 in progress 1. Next rc build in progress 2. build issue to be looked at 1. Failure on a jammy arm build, not a platform we test on meaningfully 2. Think this is an infrastructure issue 3. Generally, this shouldn't be a release blocker 4. Priority of arm builds might rise in the future though 5. investigate today, if we can't figure it out quickly, publish rc with a known issue 6. NOTE: Rook expects arm builds to be present 3. Would like to release next rc later this week if things work out 4. Would also like to upgrade lrc CDS agenda https://pad.ceph.com/p/cds-squid 1. leads should add topics 2. plan is for this to happen week of July 17th mempool monitoring in teuthology tests https://github.com/ceph/ceph/pull/51853 1. Just an FYI 2. ceph task will now have ability to dump memtools 3. might add a bit of delay to how long tests take 4. expected to be merged soon 5. will follow up in performance meeting iSCSI packages old/not signed -- want to fix before final release 1. https://tracker.ceph.com/issues/57388 2. tcmu-runner, since containerization of ceph, is being pulled from our build system 3. This tcmu-runner package is not signed 4. ceph-iscsi package is signed, but outdated (seems to be because this is the newest one that is signed and pushed to download.ceph.com) 5. someone with access to tools to sign the packages would have to help fix this 6. been like this for a long time and nobody noticed 7. only ceph-iscsi package, not tcmu-runner, is distributed through download.ceph.com 8. getting updated ceph-iscsi package on download.ceph.com should be done before reef release 9. tcmu-runner inside the container being unsigned is not as big of a deal (was this way in quincy/pacific as well)

1 0

Re: [multisite] period update and zonegroup
by Yixin Jin 28 Jun '23

28 Jun '23

Mystery solved. I loaded the zonegroup up with 100K hostnanes. That caused the period to fail to decode when other zones try to pick up the new period. Apparently, there is a size limit on how big zonegroup info can be but implicitly. Regards, Yixin Sent from Yahoo Mail on Android

1 0

[multisite] period update and zonegroup
by Yixin Jin 28 Jun '23

28 Jun '23

Hi folks, Has anyone experienced the situation where "radosgw-admin period get" shows the latest zonegroup information but "radosgw-admin zonegroup get" shows the old one? I am now seeing this with v17.2.6. I used to play with v17.2.5 but never saw this situation. I tried to do various "period update --commit" and "period pull" to see if I could get the update happen. No avail. Any ideas? Thanks,Yixin

1 0

cephadm, new OSD
by Shashi Dahal 28 Jun '23

28 Jun '23

Hi, I added new OSD on ceph servers. ( orch is cephadm) Its recognized as osd.12 and osd.13 ceph pg dump shows no pg are there in osd 12 and 13 .. they are all empty .. ceph osd tree shows them that they are up. ceph osd df shows them to be all 0 in reweight and size etc. ceph orch device ls shows the status as "Insufficient space (<10 extents) on vgs, LVM detected, locked" However, this is the same status shown in all the existing osds ( working ones also ) I tried ceph osd crush reweight osd.12 1 reweighted item id 12 name 'osd.12' to 1 in crush map but the reweight is still 0 . and these disks are not in the cluster yet. How to add these disks so that they are part of the cluster ? Thanks

2 1

Applying crush rule to existing live pool
by Rok Jaklič 28 Jun '23

28 Jun '23

Hi, I want to place an existing pool with data to ssd-s. I've created crush rule: ceph osd crush rule create-replicated replicated_ssd default host ssd If I apply this rule to the existing pool default.rgw.buckets.index with 180G of data with command: ceph osd pool set default.rgw.buckets.index crush_rule replicated_ssd Will the rgw/cluster be available in the meantime while the cluster moves data? Kind regards, Rok

2 1

Re: ceph-users Digest, Vol 108, Issue 88
by hui chen 28 Jun '23

28 Jun '23

I want to unsubscribe the general mailing list for Ceph users. Sincerely Yours, ------------------------------------------ Ivan E_: chenhui0228(a)gmail.com A_: Wuhan, Hubei, China ------------------------------------------ <ceph-users-request(a)ceph.io> 于2023年6月28日周三 01:54写道： > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. Re: ceph orch host label rm : does not update label removal > (Adiga, Anantha) > 2. Re: RBD with PWL cache shows poor performance compared to cache > device > (Josh Baergen) > 3. Re: RBD with PWL cache shows poor performance compared to cache > device > (Matthew Booth) > > > ---------------------------------------------------------------------- > > Date: Tue, 27 Jun 2023 16:04:05 +0000 > From: "Adiga, Anantha" <anantha.adiga(a)intel.com> > Subject: [ceph-users] Re: ceph orch host label rm : does not update > label removal > To: "Adiga, Anantha" <anantha.adiga(a)intel.com>, "ceph-users(a)ceph.io" > <ceph-users(a)ceph.io> > Message-ID: <CY5PR11MB62118438F851BE59AC91F85BF627A(a)CY5PR11MB6211.namp > rd11.prod.outlook.com> > Content-Type: text/plain; charset="us-ascii" > > Hello, > > This issue is resolved. > > The syntax of providing the labels was not correct. > > -----Original Message----- > From: Adiga, Anantha <anantha.adiga(a)intel.com> > Sent: Thursday, June 22, 2023 1:08 PM > To: ceph-users(a)ceph.io > Subject: [ceph-users] ceph orch host label rm : does not update label > removal > > Hi , > > Not sure if the lables are really removed or the update is not working? > This was taken as a single label: mgrs,ceph osd,rgws.ceph > > > root@fl31ca104ja0201:/# ceph orch host ls > HOST ADDR LABELS > STATUS > fl31ca104ja0201 XX.XX.XXX.139 ceph clients mdss mgrs monitoring mons > osds rgws > fl31ca104ja0202 XX.XX.XXX.140 ceph clients mdss mgrs mons osds rgws > fl31ca104ja0203 XX.XX.XXX.141 ceph clients mdss mgrs mons osds rgws > fl31ca104ja0302 XX.XX.XXX.5 _admin mgrs,ceph osd,rgws.ceph > 4 hosts in cluster > > root@fl31ca104ja0201:/# ceph orch host label rm fl31ca104ja0302 mgrs,ceph > osd,rgws.ceph > Removed label mgrs,ceph osd,rgws.ceph from host fl31ca104ja0302 > > root@fl31ca104ja0201:/# ceph orch host ls > HOST ADDR LABELS > STATUS > fl31ca104ja0201 XX.XX.XXX.139 ceph clients mdss mgrs monitoring mons > osds rgws > fl31ca104ja0202 XX.XX.XXX.140 ceph clients mdss mgrs mons osds rgws > fl31ca104ja0203 XX.XX.XXX.141 ceph clients mdss mgrs mons osds rgws > fl31ca104ja0302 XX.XX.XXX.5 _admin > 4 hosts in cluster > > Thank you, > Anantha > > > > > root@fl31ca104ja0201:/# ceph orch host ls > HOST ADDR LABELS > STATUS > fl31ca104ja0201 XX.XX.XXX.139 ceph clients mdss mgrs monitoring mons > osds rgws > fl31ca104ja0202 XX.XX.XXX.140 ceph clients mdss mgrs mons osds rgws > fl31ca104ja0203 XX.XX.XXX.141 ceph clients mdss mgrs mons osds rgws > fl31ca104ja0302 XX.XX.XXX.5 _admin mgrs,ceph osd,rgws.ceph > 4 hosts in cluster > root@fl31ca104ja0201:/# > root@fl31ca104ja0201:/# > root@fl31ca104ja0201:/# ceph orch host label rm fl31ca104ja0302 rgws.ceph > Removed label rgws.ceph from host fl31ca104ja0302 root@fl31ca104ja0201:/# > ceph orch host ls > HOST ADDR LABELS > STATUS > fl31ca104ja0201 XX.XX.XXX.139 ceph clients mdss mgrs monitoring mons > osds rgws > fl31ca104ja0202 XX.XX.XXX.140 ceph clients mdss mgrs mons osds rgws > fl31ca104ja0203 XX.XX.XXX.141 ceph clients mdss mgrs mons osds rgws > fl31ca104ja0302 XX.XX.XXX.5 _admin mgrs,ceph osd,rgws.ceph > 4 hosts in cluster > root@fl31ca104ja0201:/# ceph orch host label rm fl31ca104ja0302 rgws.ceph > --force Removed label rgws.ceph from host fl31ca104ja0302 > root@fl31ca104ja0201:/# ceph orch host ls > HOST ADDR LABELS > STATUS > fl31ca104ja0201 XX.XX.XXX.139 ceph clients mdss mgrs monitoring mons > osds rgws > fl31ca104ja0202 XX.XX.XXX.140 ceph clients mdss mgrs mons osds rgws > fl31ca104ja0203 XX.XX.XXX.141 ceph clients mdss mgrs mons osds rgws > fl31ca104ja0302 XX.XX.XXX.5 _admin mgrs,ceph osd,rgws.ceph > 4 hosts in cluster > > Regards, > Anantha > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > ------------------------------ > > Date: Tue, 27 Jun 2023 11:20:01 -0600 > From: Josh Baergen <jbaergen(a)digitalocean.com> > Subject: [ceph-users] Re: RBD with PWL cache shows poor performance > compared to cache device > To: Matthew Booth <mbooth(a)redhat.com> > Cc: ceph-users(a)ceph.io > Message-ID: > < > CA+5zLQ+R4Yq+4F4GwTZKqz4VYj5sN7gniSf3X5wnsk06F_UFtQ(a)mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > Hi Matthew, > > We've done a limited amount of work on characterizing the pwl and I think > it suffers the classic problem of some writeback caches in that, once the > cache is saturated, it's actually worse than just being in writethrough. > IIRC the pwl does try to preserve write ordering (unlike the other > writeback/writearound modes) which limits it in the concurrency it can > issue to the backend, which means that even an iodepth=1 test can saturate > the pwl, assuming the backend latency is higher than the pwl latency. > > I _think_ that if you were able to devise a burst test with bursts smaller > than the pwl capacity and gaps in between large enough for the cache to > flush, or if you were to ratelimit I/Os to the pwl, that you should see > closer to the lower latencies that you would expect. > > Josh > > On Tue, Jun 27, 2023 at 9:04 AM Matthew Booth <mbooth(a)redhat.com> wrote: > > > ** TL;DR > > > > In testing, the write latency performance of a PWL-cache backed RBD > > disk was 2 orders of magnitude worse than the disk holding the PWL > > cache. > > > > ** Summary > > > > I was hoping that PWL cache might be a good solution to the problem of > > write latency requirements of etcd when running a kubernetes control > > plane on ceph. Etcd is extremely write latency sensitive and becomes > > unstable if write latency is too high. The etcd workload can be > > characterised by very small (~4k) writes with a queue depth of 1. > > Throughput, even on a busy system, is normally very low. As etcd is > > distributed and can safely handle the loss of un-flushed data from a > > single node, a local ssd PWL cache for etcd looked like an ideal > > solution. > > > > My expectation was that adding a PWL cache on a local SSD to an > > RBD-backed would improve write latency to something approaching the > > write latency performance of the local SSD. However, in my testing > > adding a PWL cache to an rbd-backed VM increased write latency by > > approximately 4x over not using a PWL cache. This was over 100x more > > than the write latency performance of the underlying SSD. > > > > My expectation was based on the documentation here: > > https://docs.ceph.com/en/quincy/rbd/rbd-persistent-write-log-cache/ > > > > “The cache provides two different persistence modes. In > > persistent-on-write mode, the writes are completed only when they are > > persisted to the cache device and will be readable after a crash. In > > persistent-on-flush mode, the writes are completed as soon as it no > > longer needs the caller’s data buffer to complete the writes, but does > > not guarantee that writes will be readable after a crash. The data is > > persisted to the cache device when a flush request is received.” > > > > ** Method > > > > 2 systems, 1 running single-node Ceph Quincy (17.2.6), the other > > running libvirt and mounting a VM’s disk with librbd (also 17.2.6) > > from the first node. > > > > All performance testing is from the libvirt system. I tested write > > latency performance: > > > > * Inside the VM without a PWL cache > > * Of the PWL device directly from the host (direct to filesystem, no VM) > > * Inside the VM with a PWL cache > > > > I am testing with fio. Specifically I am running a containerised test, > > executed with: > > podman run --volume .:/var/lib/etcd:Z > quay.io/openshift-scale/etcd-perf > > > > This container runs: > > fio --rw=write --ioengine=sync --fdatasync=1 > > --directory=/var/lib/etcd --size=100m --bs=8000 --name=etcd_perf > > --output-format=json --runtime=60 --time_based=1 > > > > And extracts sync.lat_ns.percentile["99.000000"] > > > > ** Results > > > > All results were stable across multiple runs within a small margin of > > error. > > > > * rbd no cache: 1417216 ns > > * pwl cache device: 44288 ns > > * rbd with pwl cache: 5210112 ns > > > > Note that by adding a PWL cache we increase write latency by > > approximately 4x, which is more than 100x than the underlying device. > > > > ** Hardware > > > > 2 x Dell R640s, each with Xeon Silver 4216 CPU @ 2.10GHz and 192G RAM > > Storage under test: 2 x SAMSUNG MZ7KH480HAHQ0D3 SSDs attached to PERC > > H730P Mini (Embedded) > > > > OS installed on rotational disks > > > > N.B. Linux incorrectly detects these disks as rotational, which I > > assume relates to weird behaviour by the PERC controller. I remembered > > to manually correct this on the ‘client’ machine for the PWL cache, > > but at OSD configuration time ceph would have detected them as > > rotational. They are not rotational. > > > > ** Ceph Configuration > > > > CentOS Stream 9 > > > > # ceph version > > ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy > > (stable) > > > > Single node installation with cephadm. 2 OSDs, one on each SSD. > > 1 pool with size 2 > > > > ** Client Configuration > > > > Fedora 38 > > Librbd1-17.2.6-3.fc38.x86_64 > > > > PWL cache is XFS filesystem with 4k block size, matching the > > underlying device. The filesystem uses the whole block device. There > > is no other load on the system. > > > > ** RBD Configuration > > > > # rbd config image list libvirt-pool/pwl-test | grep cache > > rbd_cache true > > config > > rbd_cache_block_writes_upfront false > > config > > rbd_cache_max_dirty 25165824 > > config > > rbd_cache_max_dirty_age 1.000000 > > config > > rbd_cache_max_dirty_object 0 > > config > > rbd_cache_policy writeback > > pool > > rbd_cache_size 33554432 > > config > > rbd_cache_target_dirty 16777216 > > config > > rbd_cache_writethrough_until_flush true > > pool > > rbd_parent_cache_enabled false > > config > > rbd_persistent_cache_mode ssd > > pool > > rbd_persistent_cache_path /var/lib/libvirt/images/pwl > > pool > > rbd_persistent_cache_size 1073741824 > > config > > rbd_plugins pwl_cache > > pool > > > > # rbd status libvirt-pool/pwl-test > > Watchers: > > watcher=10.1.240.27:0/1406459716 client.14475 > > cookie=140282423200720 > > Persistent cache state: > > host: dell-r640-050 > > path: > > /var/lib/libvirt/images/pwl/rbd-pwl.libvirt-pool.37e947fd216b.pool > > size: 1 GiB > > mode: ssd > > stats_timestamp: Mon Jun 26 11:29:21 2023 > > present: true empty: false clean: true > > allocated: 180 MiB > > cached: 135 MiB > > dirty: 0 B > > free: 844 MiB > > hits_full: 1 / 0% > > hits_partial: 3 / 0% > > misses: 21952 > > hit_bytes: 6 KiB / 0% > > miss_bytes: 349 MiB > > -- > > Matthew Booth > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > ------------------------------ > > Date: Tue, 27 Jun 2023 18:50:07 +0100 > From: Matthew Booth <mbooth(a)redhat.com> > Subject: [ceph-users] Re: RBD with PWL cache shows poor performance > compared to cache device > To: Josh Baergen <jbaergen(a)digitalocean.com> > Cc: ceph-users(a)ceph.io > Message-ID: > < > CAEkQehcGq-88heq3UN8tsWsTyOfTcm1122Ffxy7PEhXF7Hj1mA(a)mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > On Tue, 27 Jun 2023 at 18:20, Josh Baergen <jbaergen(a)digitalocean.com> > wrote: > > > > Hi Matthew, > > > > We've done a limited amount of work on characterizing the pwl and I > think it suffers the classic problem of some writeback caches in that, once > the cache is saturated, it's actually worse than just being in > writethrough. IIRC the pwl does try to preserve write ordering (unlike the > other writeback/writearound modes) which limits it in the concurrency it > can issue to the backend, which means that even an iodepth=1 test can > saturate the pwl, assuming the backend latency is higher than the pwl > latency. > > What do you mean by saturated here? FWIW I was using the default cache > size of 1G and each test run only wrote ~100MB of data, so I don't > think I ever filled the cache, even with multiple runs. > > > I _think_ that if you were able to devise a burst test with bursts > smaller than the pwl capacity and gaps in between large enough for the > cache to flush, or if you were to ratelimit I/Os to the pwl, that you > should see closer to the lower latencies that you would expect. > > My goal is to characterise the requirements of etcd. Unfortunately I > don't think changing the test would do that. Incidentally, note that > the total bandwidth of an extremely busy etcd is usually very low. > >From memory, the etcd write rate for a system we were debugging whose > etcd was occasionally falling over due to load was only about 5MiB/s. > It's all about write latency of really small writes, not bandwidth. > > Matt > > > > > Josh > > > > On Tue, Jun 27, 2023 at 9:04 AM Matthew Booth <mbooth(a)redhat.com> wrote: > >> > >> ** TL;DR > >> > >> In testing, the write latency performance of a PWL-cache backed RBD > >> disk was 2 orders of magnitude worse than the disk holding the PWL > >> cache. > >> > >> ** Summary > >> > >> I was hoping that PWL cache might be a good solution to the problem of > >> write latency requirements of etcd when running a kubernetes control > >> plane on ceph. Etcd is extremely write latency sensitive and becomes > >> unstable if write latency is too high. The etcd workload can be > >> characterised by very small (~4k) writes with a queue depth of 1. > >> Throughput, even on a busy system, is normally very low. As etcd is > >> distributed and can safely handle the loss of un-flushed data from a > >> single node, a local ssd PWL cache for etcd looked like an ideal > >> solution. > >> > >> My expectation was that adding a PWL cache on a local SSD to an > >> RBD-backed would improve write latency to something approaching the > >> write latency performance of the local SSD. However, in my testing > >> adding a PWL cache to an rbd-backed VM increased write latency by > >> approximately 4x over not using a PWL cache. This was over 100x more > >> than the write latency performance of the underlying SSD. > >> > >> My expectation was based on the documentation here: > >> https://docs.ceph.com/en/quincy/rbd/rbd-persistent-write-log-cache/ > >> > >> “The cache provides two different persistence modes. In > >> persistent-on-write mode, the writes are completed only when they are > >> persisted to the cache device and will be readable after a crash. In > >> persistent-on-flush mode, the writes are completed as soon as it no > >> longer needs the caller’s data buffer to complete the writes, but does > >> not guarantee that writes will be readable after a crash. The data is > >> persisted to the cache device when a flush request is received.” > >> > >> ** Method > >> > >> 2 systems, 1 running single-node Ceph Quincy (17.2.6), the other > >> running libvirt and mounting a VM’s disk with librbd (also 17.2.6) > >> from the first node. > >> > >> All performance testing is from the libvirt system. I tested write > >> latency performance: > >> > >> * Inside the VM without a PWL cache > >> * Of the PWL device directly from the host (direct to filesystem, no VM) > >> * Inside the VM with a PWL cache > >> > >> I am testing with fio. Specifically I am running a containerised test, > >> executed with: > >> podman run --volume .:/var/lib/etcd:Z > quay.io/openshift-scale/etcd-perf > >> > >> This container runs: > >> fio --rw=write --ioengine=sync --fdatasync=1 > >> --directory=/var/lib/etcd --size=100m --bs=8000 --name=etcd_perf > >> --output-format=json --runtime=60 --time_based=1 > >> > >> And extracts sync.lat_ns.percentile["99.000000"] > >> > >> ** Results > >> > >> All results were stable across multiple runs within a small margin of > error. > >> > >> * rbd no cache: 1417216 ns > >> * pwl cache device: 44288 ns > >> * rbd with pwl cache: 5210112 ns > >> > >> Note that by adding a PWL cache we increase write latency by > >> approximately 4x, which is more than 100x than the underlying device. > >> > >> ** Hardware > >> > >> 2 x Dell R640s, each with Xeon Silver 4216 CPU @ 2.10GHz and 192G RAM > >> Storage under test: 2 x SAMSUNG MZ7KH480HAHQ0D3 SSDs attached to PERC > >> H730P Mini (Embedded) > >> > >> OS installed on rotational disks > >> > >> N.B. Linux incorrectly detects these disks as rotational, which I > >> assume relates to weird behaviour by the PERC controller. I remembered > >> to manually correct this on the ‘client’ machine for the PWL cache, > >> but at OSD configuration time ceph would have detected them as > >> rotational. They are not rotational. > >> > >> ** Ceph Configuration > >> > >> CentOS Stream 9 > >> > >> # ceph version > >> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy > (stable) > >> > >> Single node installation with cephadm. 2 OSDs, one on each SSD. > >> 1 pool with size 2 > >> > >> ** Client Configuration > >> > >> Fedora 38 > >> Librbd1-17.2.6-3.fc38.x86_64 > >> > >> PWL cache is XFS filesystem with 4k block size, matching the > >> underlying device. The filesystem uses the whole block device. There > >> is no other load on the system. > >> > >> ** RBD Configuration > >> > >> # rbd config image list libvirt-pool/pwl-test | grep cache > >> rbd_cache true > config > >> rbd_cache_block_writes_upfront false > config > >> rbd_cache_max_dirty 25165824 > config > >> rbd_cache_max_dirty_age 1.000000 > config > >> rbd_cache_max_dirty_object 0 > config > >> rbd_cache_policy writeback > pool > >> rbd_cache_size 33554432 > config > >> rbd_cache_target_dirty 16777216 > config > >> rbd_cache_writethrough_until_flush true > pool > >> rbd_parent_cache_enabled false > config > >> rbd_persistent_cache_mode ssd > pool > >> rbd_persistent_cache_path > /var/lib/libvirt/images/pwl pool > >> rbd_persistent_cache_size 1073741824 > config > >> rbd_plugins pwl_cache > pool > >> > >> # rbd status libvirt-pool/pwl-test > >> Watchers: > >> watcher=10.1.240.27:0/1406459716 client.14475 > cookie=140282423200720 > >> Persistent cache state: > >> host: dell-r640-050 > >> path: > /var/lib/libvirt/images/pwl/rbd-pwl.libvirt-pool.37e947fd216b.pool > >> size: 1 GiB > >> mode: ssd > >> stats_timestamp: Mon Jun 26 11:29:21 2023 > >> present: true empty: false clean: true > >> allocated: 180 MiB > >> cached: 135 MiB > >> dirty: 0 B > >> free: 844 MiB > >> hits_full: 1 / 0% > >> hits_partial: 3 / 0% > >> misses: 21952 > >> hit_bytes: 6 KiB / 0% > >> miss_bytes: 349 MiB > >> -- > >> Matthew Booth > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > -- > Matthew Booth > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > > > ------------------------------ > > End of ceph-users Digest, Vol 108, Issue 88 > ******************************************* >

1 0

RBD with PWL cache shows poor performance compared to cache device
by Matthew Booth 27 Jun '23

27 Jun '23

** TL;DR In testing, the write latency performance of a PWL-cache backed RBD disk was 2 orders of magnitude worse than the disk holding the PWL cache. ** Summary I was hoping that PWL cache might be a good solution to the problem of write latency requirements of etcd when running a kubernetes control plane on ceph. Etcd is extremely write latency sensitive and becomes unstable if write latency is too high. The etcd workload can be characterised by very small (~4k) writes with a queue depth of 1. Throughput, even on a busy system, is normally very low. As etcd is distributed and can safely handle the loss of un-flushed data from a single node, a local ssd PWL cache for etcd looked like an ideal solution. My expectation was that adding a PWL cache on a local SSD to an RBD-backed would improve write latency to something approaching the write latency performance of the local SSD. However, in my testing adding a PWL cache to an rbd-backed VM increased write latency by approximately 4x over not using a PWL cache. This was over 100x more than the write latency performance of the underlying SSD. My expectation was based on the documentation here: https://docs.ceph.com/en/quincy/rbd/rbd-persistent-write-log-cache/ “The cache provides two different persistence modes. In persistent-on-write mode, the writes are completed only when they are persisted to the cache device and will be readable after a crash. In persistent-on-flush mode, the writes are completed as soon as it no longer needs the caller’s data buffer to complete the writes, but does not guarantee that writes will be readable after a crash. The data is persisted to the cache device when a flush request is received.” ** Method 2 systems, 1 running single-node Ceph Quincy (17.2.6), the other running libvirt and mounting a VM’s disk with librbd (also 17.2.6) from the first node. All performance testing is from the libvirt system. I tested write latency performance: * Inside the VM without a PWL cache * Of the PWL device directly from the host (direct to filesystem, no VM) * Inside the VM with a PWL cache I am testing with fio. Specifically I am running a containerised test, executed with: podman run --volume .:/var/lib/etcd:Z quay.io/openshift-scale/etcd-perf This container runs: fio --rw=write --ioengine=sync --fdatasync=1 --directory=/var/lib/etcd --size=100m --bs=8000 --name=etcd_perf --output-format=json --runtime=60 --time_based=1 And extracts sync.lat_ns.percentile["99.000000"] ** Results All results were stable across multiple runs within a small margin of error. * rbd no cache: 1417216 ns * pwl cache device: 44288 ns * rbd with pwl cache: 5210112 ns Note that by adding a PWL cache we increase write latency by approximately 4x, which is more than 100x than the underlying device. ** Hardware 2 x Dell R640s, each with Xeon Silver 4216 CPU @ 2.10GHz and 192G RAM Storage under test: 2 x SAMSUNG MZ7KH480HAHQ0D3 SSDs attached to PERC H730P Mini (Embedded) OS installed on rotational disks N.B. Linux incorrectly detects these disks as rotational, which I assume relates to weird behaviour by the PERC controller. I remembered to manually correct this on the ‘client’ machine for the PWL cache, but at OSD configuration time ceph would have detected them as rotational. They are not rotational. ** Ceph Configuration CentOS Stream 9 # ceph version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Single node installation with cephadm. 2 OSDs, one on each SSD. 1 pool with size 2 ** Client Configuration Fedora 38 Librbd1-17.2.6-3.fc38.x86_64 PWL cache is XFS filesystem with 4k block size, matching the underlying device. The filesystem uses the whole block device. There is no other load on the system. ** RBD Configuration # rbd config image list libvirt-pool/pwl-test | grep cache rbd_cache true config rbd_cache_block_writes_upfront false config rbd_cache_max_dirty 25165824 config rbd_cache_max_dirty_age 1.000000 config rbd_cache_max_dirty_object 0 config rbd_cache_policy writeback pool rbd_cache_size 33554432 config rbd_cache_target_dirty 16777216 config rbd_cache_writethrough_until_flush true pool rbd_parent_cache_enabled false config rbd_persistent_cache_mode ssd pool rbd_persistent_cache_path /var/lib/libvirt/images/pwl pool rbd_persistent_cache_size 1073741824 config rbd_plugins pwl_cache pool # rbd status libvirt-pool/pwl-test Watchers: watcher=10.1.240.27:0/1406459716 client.14475 cookie=140282423200720 Persistent cache state: host: dell-r640-050 path: /var/lib/libvirt/images/pwl/rbd-pwl.libvirt-pool.37e947fd216b.pool size: 1 GiB mode: ssd stats_timestamp: Mon Jun 26 11:29:21 2023 present: true empty: false clean: true allocated: 180 MiB cached: 135 MiB dirty: 0 B free: 844 MiB hits_full: 1 / 0% hits_partial: 3 / 0% misses: 21952 hit_bytes: 6 KiB / 0% miss_bytes: 349 MiB -- Matthew Booth

2 3

Fix for incorrect available space with stretched cluster
by ceph＠paulusma.eu 27 Jun '23

27 Jun '23

Hi all! I was wondering if some could checkout https://tracker.ceph.com/issues/56650 about the incorrect calculation of the cluster size? It's a waist of storage if I can't use more than 50% and it looks like someone got it fixed with the mentioned Pull Request. Best regards, Sake

1 0

ceph orch host label rm : does not update label removal
by Adiga, Anantha 27 Jun '23

27 Jun '23

Hi , Not sure if the lables are really removed or the update is not working? root@fl31ca104ja0201:/# ceph orch host ls HOST ADDR LABELS STATUS fl31ca104ja0201 XX.XX.XXX.139 ceph clients mdss mgrs monitoring mons osds rgws fl31ca104ja0202 XX.XX.XXX.140 ceph clients mdss mgrs mons osds rgws fl31ca104ja0203 XX.XX.XXX.141 ceph clients mdss mgrs mons osds rgws fl31ca104ja0302 XX.XX.XXX.5 _admin mgrs,ceph osd,rgws.ceph 4 hosts in cluster root@fl31ca104ja0201:/# root@fl31ca104ja0201:/# root@fl31ca104ja0201:/# ceph orch host label rm fl31ca104ja0302 rgws.ceph Removed label rgws.ceph from host fl31ca104ja0302 root@fl31ca104ja0201:/# ceph orch host ls HOST ADDR LABELS STATUS fl31ca104ja0201 XX.XX.XXX.139 ceph clients mdss mgrs monitoring mons osds rgws fl31ca104ja0202 XX.XX.XXX.140 ceph clients mdss mgrs mons osds rgws fl31ca104ja0203 XX.XX.XXX.141 ceph clients mdss mgrs mons osds rgws fl31ca104ja0302 XX.XX.XXX.5 _admin mgrs,ceph osd,rgws.ceph 4 hosts in cluster root@fl31ca104ja0201:/# ceph orch host label rm fl31ca104ja0302 rgws.ceph --force Removed label rgws.ceph from host fl31ca104ja0302 root@fl31ca104ja0201:/# ceph orch host ls HOST ADDR LABELS STATUS fl31ca104ja0201 XX.XX.XXX.139 ceph clients mdss mgrs monitoring mons osds rgws fl31ca104ja0202 XX.XX.XXX.140 ceph clients mdss mgrs mons osds rgws fl31ca104ja0203 XX.XX.XXX.141 ceph clients mdss mgrs mons osds rgws fl31ca104ja0302 XX.XX.XXX.5 _admin mgrs,ceph osd,rgws.ceph 4 hosts in cluster Regards, Anantha

1 1

Bluestore compression - Which algo to choose? Zstd really still that bad?
by Christian Rohmann 27 Jun '23

27 Jun '23

Hey ceph-users, we've been using the default "snappy" to have Ceph compress data on certain pools - namely backups / copies of volumes of a VM environment. So it's write once, and no random access. I am now wondering if switching to another algo (there is snappy, zlib, lz4, or zstd) would improve the compression ratio (significantly)? * Does anybody have any real world data on snappy vs. $anyother? Using zstd is tempting as it's used in various other applications (btrfs, MongoDB, ...) for inline-compression with great success. For Ceph though there is a warning ([1]), about it being not recommended in the docs still. But I am wondering if this still stands with e.g. [2] merged. And there was [3] trying to improve the performance, this this reads as it only lead to a dead-end and no code changes? In any case does anybody have any numbers to help with the decision on the compression algo? Regards Christian [1] https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#c… [2] https://github.com/ceph/ceph/pull/33790 [3] https://github.com/facebook/zstd/issues/910

3 3

Radogw ignoring HTTP_X_FORWARDED_FOR header
by Yosr Kchaou 27 Jun '23

27 Jun '23

Hello, We are working on setting up an nginx sidecar container running along a RadosGW container inside the same kubernetes pod. We are facing an issue with getting the right value for the header HTTP_X_FORWARDED_FOR when getting client requests. We need this value to do the source ip check validation. Currently, RGW sees that all requests come from 127.0.0.1. So it is still considering the nginx ip address and not the client who made the request. Even though, we configured our RGW the following rgw_remote_addr_param = HTTP_X_FORWARDED_FOR And from the nginx side, we have the following configuration to set the headers. location / { proxy_pass http://localhost:7480; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-Port $server_port; Any idea what is the issue here ? Thank you in advance, Yoke

3 3

Possible data damage: 1 pg recovery_unfound, 1 pg inconsistent
by Jorge JP 26 Jun '23

26 Jun '23

Hello, After deep-scrub my cluster shown this error: HEALTH_ERR 1/38578006 objects unfound (0.000%); 1 scrub errors; Possible data damage: 1 pg recovery_unfound, 1 pg inconsistent; Degraded data redundancy: 2/77158878 objects degraded (0.000%), 1 pg degraded [WRN] OBJECT_UNFOUND: 1/38578006 objects unfound (0.000%) pg 32.15c has 1 unfound objects [ERR] OSD_SCRUB_ERRORS: 1 scrub errors [ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound, 1 pg inconsistent pg 32.15c is active+recovery_unfound+degraded+inconsistent, acting [49,47], 1 unfound [WRN] PG_DEGRADED: Degraded data redundancy: 2/77158878 objects degraded (0.000%), 1 pg degraded pg 32.15c is active+recovery_unfound+degraded+inconsistent, acting [49,47], 1 unfound I searching in internet how it solves, but I'm confusing.. Anyone can help me? Thank you! (Sorry for my english)

3 6

ceph.conf and two different ceph clusters
by garcetto 26 Jun '23

26 Jun '23

good afternoon, how can i config ceph.conf file on a generic rbd client to say to use two different ceph clusters to access different volumes on them? ceph-cluster-left --> rbd-vol-green ceph-cluster-right --> rbd-vol-blue thank you.

2 1

Re: cephadm and remoto package
by Florian Haas 26 Jun '23

26 Jun '23

Hi Shashi, I just ran into this myself, and I thought I'd share the solution/workaround that I applied. On 15/05/2023 22:08, Shashi Dahal wrote: > Hi, > I followed this documentation: > > https://docs.ceph.com/en/pacific/cephadm/adoption/ > > This is the error I get when trying to enable cephadm. > > ceph mgr module enable cephadm > > Error ENOENT: module 'cephadm' reports that it cannot run on the active > manager daemon: loading remoto library:No module named 'remoto' (pass > --force to force enablement) > > When I import remoto, it imports just fine. > > > OS is ubuntu 20.04 focal As far as I can see, this issue applies to non-containerized Ceph Pacific deployments — such as ones orchestrated with ceph-ansible — running on Debian or Ubuntu. There is no python3-remoto package on those platforms, so you can't install remoto by "regular" installation means (that is, apt/apt-get). It looks to me like this issue was introduced in Pacific, and then went away in Quincy because that release dropped remoto and replaced it with asyncssh (for which a Debian/Ubuntu package does exist). If you start out on Octopus with ceph-ansible and do the Cephadm migration *then*, you're apparently fine too, and you can subsequently use Cephadm to upgrade to Pacific and Quincy. I think it's just this particular combination — (a) run on Debian/Ubuntu, (b) deploy non-containerized, *and* (c) start your deployment on Pacific, where Cephadm adoption breaks. The problem has apparently been known for a while (see https://tracker.ceph.com/issues/43415) but the recommendation appears to have been "just run mgr on a different OS then", which is frequently not a viable option. I tried (like you did, I assume) to just pip-install remoto, and if I opened a Python console and typed "import remoto" it imported just fine, but apparently the cephadm mgr module didn't like that. I've now traced this down to the following line that shows up in the ceph-mgr log if you bump "debug mgr" to 10/10: 2023-06-26T10:01:34.799+0000 7fb0979ba500 10 mgr[py] Computed sys.path '/usr/share/ceph/mgr:/local/lib/python3.8/dist-packages:/lib/python3/dist-packages:/lib/python3.8/dist-packages:lib/python38.zip:/lib/python3.8:/lib/python3.8/lib-dynload' Note the /local/lib/python3.8/dist-packages path, which does not exist on Ubuntu Focal. It's properly /usr/local/lib/python3.8/dist-packages, and this is where "pip install", when run as root outside a virtualenv, installs packages to. I think the incorrect sys.path may actually be a build or packaging bug in the community packages built for Debian/Ubuntu, but I'm not 100% certain. At any rate, the combined workaround for this issue, for me, is: (1) pip install remoto (this installs remoto into /usr/local/lib/python3.8/dist-packages) (2) ln -s /usr/local/lib/python3.8/dist-packages /local/lib/python3.8/dist-packages (this makes pip-installed packages available to ceph-mgr) (3) restart all ceph-mgr instances (4) ceph mgr module enable cephadm Cheers, Florian

1 0

radosgw hang under pressure
by grin 26 Jun '23

26 Jun '23

Hello, ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) There is a single (test) radosgw serving plenty of test traffic. When under heavy req/s ("heavy" in a low sense, about 1k rq/s) it pretty reliably hangs: low traffic threads seem to work (like handling occasional PUTs) but GETs are completely nonresponsive, all attention seems to be spent on futexes. The effect is extremely similar to https://ceph-users.ceph.narkive.com/I4uFVzH9/radosgw-civetweb-hangs-once-ar… (subject: Radosgw (civetweb) hangs once around) except this is quincy so it's beast instead of civetweb. The effect is the same as described there, except the cluster is way smaller (about 20-40 OSDs). I observed that when I start radosgw -f with debug 20/20 it almost never hangs, so my guess is some ugly race condition. However I am a bit clueless how to actually debug it since debugging makes it go away. Debug 1 (default) with -d seems to hang after a while but it's not that simple to induce, I'm still testing under 4/4. Also I do not see much to configure about beast. As to answer the question in the original (2016) thread: - Debian stable - no visible limits issue - no obvious memory leak observed - no other visible resource shortage - strace says everyone's waiting on futexes, about 600-800 threads, apart from the one serving occasional PUTs - tcp port doesn't respond. IRC didn't react. ;-) Thanks, Peter

4 4

copy file in nfs over cephfs error "error: error in file IO (code 11)"
by farhad kh 26 Jun '23

26 Jun '23

hi everybody we have problem with nfs gansha load balancer whene use rsync -avre to copy file from another share to ceph nfs share path we get this error `rsync -rav /mnt/elasticsearch/newLogCluster/acr-202* /archive/Elastic-v7-archive` rsync : close failed on "/archive/Elastic-v7-archive/...." : Input/output error (5) rsync error: error in file IO (code 11) at receiver.c(586) [Receiver=3.1.3]" we used ingress for load balancing nfs service and No other problems are observed in the cluster. Below is information about the pool, volume path and quota ------------ amount10.20.32.161:/volumes/arch-1/arch 30T 5.0T 26T 17% /archive# ceph osd pool get-quota arch-bigdata-data quotas for pool 'arch-bigdata-data': max objects: N/A max bytes : 30 TiB (current num bytes: 5488192308978 bytes) --------------- # ceph fs subvolume info arch-bigdata arch arch-1 { "atime": "2023-06-11 13:32:22", "bytes_pcent": "16.64", "bytes_quota": 32985348833280, "bytes_used": 5488566602388, "created_at": "2023-06-11 13:32:22", "ctime": "2023-06-25 10:45:35", "data_pool": "arch-bigdata-data", "features": [ "snapshot-clone", "snapshot-autoprotect", "snapshot-retention" ], "gid": 0, "mode": 16877, "mon_addrs": [ "10.20.32.153:6789", "10.20.32.155:6789", "10.20.32.154:6789" ], "mtime": "2023-06-25 10:38:48", "path": "/volumes/arch-1/arch/f246a31b-7103-41b9-8005-63d00efe88e4", "pool_namespace": "", "state": "complete", "type": "subvolume", "uid": 0 } .Has anyone ever experienced this error? What way do you suggest to solve it?

1 0

alerts in dashboard
by Ben 26 Jun '23

26 Jun '23

Hi, I got many critical alerts in ceph dashboard. Meanwhile the cluster shows health ok status. See attached screenshot for detail. My questions are, are they real alerts? How to get rid of them? Thanks Ben

3 3

Changing bucket owner in a multi-zonegroup Ceph cluster
by Ramin Najjarbashi 25 Jun '23

25 Jun '23

Hi all I have a Ceph cluster consisting of two zonegroups with metadata syncing enabled. I need to change the owner of a bucket that is located in the secondary zonegroup. I followed the steps below: Unlinked the bucket from the old user on the secondary zonegroup: bash Copy code $ radosgw-admin bucket unlink --uid OLD_UID -b test-change-owner Linked the bucket to the new user on the secondary zonegroup: bash Copy code $ radosgw-admin bucket link --uid NEW_UID -b test-change-owner Changed the owner of the bucket on the primary (master) zonegroup: bash Copy code $ radosgw-admin bucket chown --uid NEW_UID -b test-change-owner After executing the last command on the primary zonegroup, the bucket owner was successfully changed. However, the ownership of the objects within the bucket still remains with the old user. When I executed the same radosgw-admin bucket chown command on the secondary zonegroup, I received a warning about inconsistent metadata between zones, but the bucket owner was changed successfully on the secondary zonegroup. My questions are: What is the best way to change the owner of a bucket in a multi-zonegroup cluster? What are the potential impacts of running the chown command on the secondary zonegroup? Is it possible to have inconsistent metadata between zones in this case?

1 0

OSDs cannot join cluster anymore
by Malte Stroem 24 Jun '23

24 Jun '23

Hello, we removed some nodes from our cluster. This worked without problems. Now, lots of OSDs do not want to join the cluster anymore if we reboot one of the still available nodes. It always runs into timeouts: --> ceph-volume lvm activate successful for osd ID: XX monclient(hunting): authenticate timed out after 300 MONs and MGRs are running fine. Network is working, netcat to the MONs' ports are open. Setting a higher debug level has no effect even if we add it to the ceph.conf file. The PGs are pretty unhappy, e. g.: 7.143 87771 0 0 0 0 314744902235 0 0 10081 10081 down 2023-06-20T09:16:03.546158+0000 961275'1395646 961300:9605547 [209,NONE,NONE] 209 [209,NONE,NONE] 209 961231'1395512 2023-06-19T23:46:40.101791+0000 961231'1395512 2023-06-19T23:46:40.101791+0000 PG query wants us to set an OSD lost however I do not want to do this. OSDs are blocked by OSDs from the removed nodes: ceph osd blocked-by osd num_blocked 152 38 244 41 144 54 ... We added the removed hosts again and tried to start the OSDs on this node and they also failed into the timeout mentioned above. This is a containerized cluster running version 16.2.10. Replication is 3, some pools use an erasure coded profile. Best regards, Malte

4 9

users caps change unexpected
by Alessandro Italiano 23 Jun '23

23 Jun '23

Hi we have a brand new ceph instance deployed by ceph puppet module. We are experiencing a funny issues. users caps change unexpected. logs do not report any message about the user caps even with auth/debug_auth: 5/5 who/what can change the caps ? thanks in advance Ale root@cephmon1:~# ceph version ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) root@cephmon1:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 22.04.1 LTS Release: 22.04 Codename: jammy root@cephmon1:~# root@cephmon1:~# ceph auth del client.cinder-backup updated root@cephmon1:~# root@cephmon1:~# root@cephmon1:~# root@cephmon1:~# date Fri Jun 23 07:52:40 AM CEST 2023 root@cephmon1:~# root@cephmon1:~# ceph auth add client.cinder-backup added key for client.cinder-backup root@cephmon1:~# ceph auth caps client.cinder-backup mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=backup' mgr 'profile rbd pool=backup' updated caps for client.cinder-backup root@cephmon1:~# root@cephmon1:~# root@cephmon1:~# root@cephmon1:~# date Fri Jun 23 07:53:18 AM CEST 2023 root@cephmon1:~# ceph auth list client.cinder-backup key: AQBEM5VkhIfJHBAA6WP9P3HHCTSySdTqZv4Ypg== caps: [mgr] profile rbd pool=backup caps: [mon] allow r caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=backup root@cephmon1:~# ceph auth list client.cinder-backup key: AQAM0nJi8OYAFBAA+p+T2QWtKaq92Z/hFMgF4w== caps: [mgr] profile rbd pool=backups caps: [mon] profile rbd caps: [osd] profile rbd pool=backups root@cephmon1:~# date Fri Jun 23 07:56:42 AM CEST 2023

2 1

Re: Grafana service fails to start due to bad directory name after Quincy upgrade
by Adiga, Anantha 23 Jun '23

23 Jun '23

Hi Eugene, Thank you for your response, here is the update. The upgrade to Quincy was done following the cephadm orch upgrade procedure ceph orch upgrade start --image quay.io/ceph/ceph:v17.2.6 Upgrade completed with out errors. After the upgrade, upon creating the Grafana service from Ceph dashboard, it deployed Grafana 6.7.4. The version is hardcoded in the code, should it not be 8.3.5 as listed below in Quincy documentation? See below [Grafana service started from Cephdashboard] Quincy documentation states: https://docs.ceph.com/en/latest/releases/quincy/ ……documentation snippet Monitoring and alerting: 43 new alerts have been added (totalling 68) improving observability of events affecting: cluster health, monitors, storage devices, PGs and CephFS. Alerts can now be sent externally as SNMP traps via the new SNMP gateway service (the MIB is provided). Improved integrated full/nearfull event notifications. Grafana Dashboards now use grafonnet format (though they’re still available in JSON format). Stack update: images for monitoring containers have been updated. Grafana 8.3.5, Prometheus 2.33.4, Alertmanager 0.23.0 and Node Exporter 1.3.1. This reduced exposure to several Grafana vulnerabilities (CVE-2021-43798, CVE-2021-39226, CVE-2021-43798, CVE-2020-29510, CVE-2020-29511). …………………. I notice that the versions of the remaining stack, that Ceph dashboard deploys, are also older than what is documented. Prometheus 2.7.2, Alertmanager 0.16.2 and Node Exporter 0.17.0. AND 6.7.4 Grafana service reports a few warnings: highlighted below root@fl31ca104ja0201:/home/general# systemctl status ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana.fl31ca104ja0201.service ● ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana.fl31ca104ja0201.service - Ceph grafana.fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e Loaded: loaded (/etc/systemd/system/ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2023-06-13 03:37:58 UTC; 11h ago Main PID: 391896 (bash) Tasks: 53 (limit: 618607) Memory: 17.9M CGroup: /system.slice/system-ceph\x2dd0a3b6e0\x2dd2c3\x2d11ed\x2dbe05\x2da7a3a1d7a87e.slice/ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana.fl31ca104j> ├─391896 /bin/bash /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/unit.run └─391969 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --init --name ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl> -- Logs begin at Sun 2023-06-11 20:41:51 UTC, end at Tue 2023-06-13 15:35:12 UTC. -- Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Created default organization" logger=sqlstore Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Initializing HTTPServer" logger=server Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Initializing BackendPluginManager" logger=server Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Initializing PluginManager" logger=server Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Starting plugin search" logger=plugins Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Initializing HooksService" logger=server Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Initializing OSSLicensingService" logger=server Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Initializing InternalMetricsService" logger=server Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Initializing RemoteCache" logger=server Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Initializing RenderingService" logger=server Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Initializing AlertEngine" logger=server Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Initializing QuotaService" logger=server Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Initializing ServerLockService" logger=server Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Initializing UserAuthTokenService" logger=server Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Initializing DatasourceCacheService" logger=server Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Initializing LoginService" logger=server Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Initializing SearchService" logger=server Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Initializing TracingService" logger=server Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Initializing UsageStatsService" logger=server Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Initializing CleanUpService" logger=server Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Initializing NotificationService" logger=server Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Initializing provisioningServiceImpl" logger=server Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml This warning comes due to the missing “ apiVersion: 1” first line entry in /etc/grafana/provisioning/datasources/ceph-dashboard.yml created by cephadm. If the file is modified to include the apiversion line and restart Grafana service, Is this a known ISSUE ? Here is the content of the ceph-dashboard.yml produced by cephadm deleteDatasources: - name: 'Dashboard1' orgId: 1 - name: 'Loki' orgId: 2 datasources: - name: 'Dashboard1' type: 'prometheus' access: 'proxy' orgId: 1 url: 'http://fl31ca104ja0201.xxx.xxx.com:9095' basicAuth: false isDefault: true editable: false - name: 'Loki' type: 'loki' access: 'proxy' orgId: 2 url: '' basicAuth: false isDefault: true editable: false -------------------------------------------------------------- Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Backend rendering via phantomJS" logger=rendering renderer=phantomJS Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=warn msg="phantomJS is deprecated and will be removed in a future release. You should consider migrating from phantomJS to grafana-image-renderer plugin. Read more at https://grafana.com/docs/grafana/latest/administration/image_rendering/" logger=rendering renderer=phantomJS Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="Initializing Stream Manager" Jun 13 03:37:59 fl31ca104ja0201 bash[391969]: t=2023-06-13T03:37:59+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= I also had to change a few other things to keep all the services running. The last issue that I have not been able to resolve yet is the Cephbash board gives this error even though grafana is running on the same server. However, the grafana dashboard cannot be accessed without tunnelling. [cid:image002.png@01D9A10B.F8B9D220]

3 4

Ceph iSCSI GW is too slow when compared with Raw RBD performance
by Work Ceph 23 Jun '23

23 Jun '23

Hello guys, We have a Ceph cluster that runs just fine with Ceph Octopus; we use RBD for some workloads, RadosGW (via S3) for others, and iSCSI for some Windows clients. We started noticing some unexpected performance issues with iSCSI. I mean, an SSD pool is reaching 100MB of write speed for an image, when it can reach up to 600MB+ of write speed for the same image when mounted and consumed directly via RBD. Is that performance degradation expected? We would expect some degradation, but not as much as this one. Also, we have a question regarding the use of Intel Turbo boost. Should we disable it? Is it possible that the root cause of the slowness in the iSCSI GW is caused by the use of Intel Turbo boost feature, which reduces the clock of some cores? Any feedback is much appreciated.

2 1

changing crush map on the fly?
by Angelo Höngens 23 Jun '23

23 Jun '23

Hey, Just to confirm my understanding: If I set up a 3-osd cluster really fast with an EC42 pool, and I set the crush map to osd failover domain, the data will be distributed among the osd's, and of course there won't be protection against host failure. And yes, I know that's a bad idea, but I need the extra storage really fast, and it's a backup of other data. So availability is important, but now critical. If I then add 5 more hosts a week later, I can just edit the crush map and change the failover domain from osd to host, put the crush map back in, and ceph should automatically distribute all the pg's over the osd's again to be fully host-fault tolerant, right? Am I understanding this correctly? Angelo.

2 1

Removing the encryption: (essentially decrypt) encrypted RGW objects
by Jayanth Reddy 22 Jun '23

22 Jun '23

Hello Users, We've a big cluster (Quincy) with almost 1.7 billion RGW objects, and we've enabled SSE on as per https://docs.ceph.com/en/quincy/radosgw/encryption/#automatic-encryption-fo… (yes, we've chosen this insecure method to store the key) We're now in the process of implementing RGW multisite, but stuck due to https://tracker.ceph.com/issues/46062 and list at https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/PQW66JJ5DCR… Was wondering if there is a way to decrypt the objects in-place with the applied symmetric key. I tried to remove the rgw_crypt_default_encryption_key from the mon configuration database (on a test cluster), but as expected, RGW daemons throw 500 server errors as it can not work on encrypted objects. There is a PR being worked on about introducing the command option at https://github.com/ceph/ceph/pull/51842 but it appears it takes some time to be merged. Cheers, Jayanth Reddy

2 1

How does a "ceph orch restart SERVICE" affect availability?
by Mikael Öhman 22 Jun '23

22 Jun '23

The documentation very briefly explains a few core commands for restarting things; https://docs.ceph.com/en/quincy/cephadm/operations/#starting-and-stopping-d… but I feel I'm lacking quite some details of what is safe to do. I have a system in production, clusters connected via CephFS and some shared block devices. We would like to restart some things due to some new network configurations. Going daemon by daemon would take forever, so I'm curious as to what happens if one tries the command; ceph orch restart osd Will that try to be smart and just restart a few at a time to keep things up and available. Or will it just trigger a restart everywhere simultaneously. I guess in my current scenario, restarting one host at the time makes most sense, with a systemctl restart ceph-{fsid}.target and then checking that "ceph -s" says OK before proceeding to the next host, but I'm still curious as to what the "ceph orch restart xxx" command would do (but not enough to try it out in production) Best regards, Mikael Chalmers University of Technology

2 2

Ceph Pacific bluefs enospc bug with newly created OSDs
by Carsten Grommel 22 Jun '23

22 Jun '23

Hi all, we are experiencing the “bluefs enospc bug” again after redeploying all OSDs of our Pacific Cluster. I know that our cluster is a bit too utilized at the moment with 87.26 % raw usage but still this should not happen afaik. We never hat this problem with previous ceph versions and right now I am kind of out of ideas at how to tackle these crashes. Compacting the database did not help in the past either. Redeploy seems to no help in the long run as well. For documentation I used these commands to redeploy the osds: systemctl stop ceph-osd@${OSDNUM} ceph osd destroy --yes-i-really-mean-it ${OSDNUM} blkdiscard ${DEVICE} sgdisk -Z ${DEVICE} dmsetup remove ${DMDEVICE} ceph-volume lvm create --osd-id ${OSDNUM} --data ${DEVICE} Any ideas or possible solutions on this? I am not yet ready to upgrade our clusters to quincy, also I do presume that this bug is still present in quincy as well? Follow our cluster information: Crash Info: ceph crash info 2023-06-19T21:23:51.285180Z_ac4105d7-cb09-45c8-a6e3-8a6bb6727b25 { "assert_condition": "abort", "assert_file": "/build/ceph/src/os/bluestore/BlueFS.cc", "assert_func": "int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)", "assert_line": 2810, "assert_msg": "/build/ceph/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7fd561810100 time 2023-06-19T23:23:51.261617+0200\n/build/ceph/src/os/bluestore/BlueFS.cc: 2810: ceph_abort_msg(\"bluefs enospc\")\n", "assert_thread_name": "ceph-osd", "backtrace": [ "/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7fd56225f730]", "gsignal()", "abort()", "(ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1a7) [0x557bb3c65762]", "(BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1175) [0x557bb42e7945]", "(BlueFS::_flush(BlueFS::FileWriter*, bool, bool*)+0xa1) [0x557bb42e7ad1]", "(BlueFS::_flush(BlueFS::FileWriter*, bool, std::unique_lock<std::mutex>&)+0x2e) [0x557bb42f803e]", "(BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x11b) [0x557bb431134b]", "(rocksdb::LegacyWritableFileWrapper::Append(rocksdb::Slice const&, rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x44) [0x557bb478e602]", "(rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long)+0x333) [0x557bb4956feb]", "(rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x5d1) [0x557bb4955569]", "(rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0x11d) [0x557bb4b142e1]", "(rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x7d6) [0x557bb4b140ca]", "(rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x48) [0x557bb4b138e0]", "(rocksdb::BlockBasedTableBuilder::Flush()+0x9a) [0x557bb4b13890]", "(rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x192) [0x557bb4b133c8]", "(rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0x773) [0x557bb4a9aa7d]", "(rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x5de) [0x557bb4824676]", "(rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*)+0x1aa0) [0x557bb48232d0]", "(rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)+0x158a) [0x557bb4820846]", "(rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x679) [0x557bb4825b25]", "(rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x52) [0x557bb4824efa]", "(RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xdaf) [0x557bb473b85f]", "(BlueStore::_open_db(bool, bool, bool)+0x44b) [0x557bb41ec20b]", "(BlueStore::_open_db_and_around(bool, bool)+0x2ef) [0x557bb425288f]", "(BlueStore::_mount()+0x9c) [0x557bb42551ec]", "(OSD::init()+0x38a) [0x557bb3d568da]", "main()", "__libc_start_main()", "_start()" ], "ceph_version": "16.2.11", "crash_id": "2023-06-19T21:23:51.285180Z_ac4105d7-cb09-45c8-a6e3-8a6bb6727b25", "entity_name": "osd.39", "os_id": "10", "os_name": "Debian GNU/Linux 10 (buster)", "os_version": "10 (buster)", "os_version_id": "10", "process_name": "ceph-osd", "stack_sig": "23f90145bebe39074210d4a79260e8977aec6b1c4d963740d1a04c3ddd4756a4", "timestamp": "2023-06-19T21:23:51.285180Z", "utsname_hostname": "cloud5-1567", "utsname_machine": "x86_64", "utsname_release": "5.10.144+1-ph", "utsname_sysname": "Linux", "utsname_version": "#1 SMP Mon Sep 26 07:02:56 UTC 2022" } Utilization: ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 168 TiB 21 TiB 146 TiB 146 TiB 87.26 TOTAL 168 TiB 21 TiB 146 TiB 146 TiB 87.26 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 4.7 MiB 48 14 MiB 0 2.1 TiB cephstor5 2 2048 52 TiB 14.27M 146 TiB 95.89 2.1 TiB cephfs_cephstor5_data 3 32 95 MiB 118.52k 1.4 GiB 0.02 2.1 TiB cephfs_cephstor5_metadata 4 16 352 MiB 166 1.0 GiB 0.02 2.1 TiB Versions: ceph versions { "mon": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 3 }, "mgr": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 3 }, "osd": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 48 }, "mds": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 3 }, "overall": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 57 } } Kind regards Carsten Grommel ------------------------------- Profihost GmbH Expo Plaza 1 30539 Hannover Deutschland Tel.: +49 (511) 5151 8181 | Fax.: +49 (511) 5151 8282 URL: http://www.profihost.com | E-Mail: info(a)profihost.com<mailto:info@profihost.com> Sitz der Gesellschaft: Hannover, USt-IdNr. DE249338561 Registergericht: Amtsgericht Hannover, Register-Nr.: HRB 222926 Geschäftsführer: Marc Zocher, Dr. Claus Boyens, Daniel Hagemeier

3 5

How to repair pg in failed_repair state?
by 이 강우 22 Jun '23

22 Jun '23

A lot of pg in inconsistent state occurred. Most of them were repaired with ceph pg repair all, but in the case of 3 pg as shown below, it does not proceed further with failed_repair status. [root@cephvm1 ~]# ceph health detail HEALTH_ERR 30 scrub errors; Too many repaired reads on 7 OSDs; Possible data damage: 3 pgs inconsistent OSD_SCRUB_ERRORS 30 scrub errors OSD_TOO_MANY_REPAIRS Too many repaired reads on 7 OSDs osd.29 had 315 reads repaired osd.23 had 530 reads repaired osd.18 had 69 reads repaired osd.2 had 267 reads repaired osd.0 had 179 reads repaired osd.12 had 513 reads repaired osd.13 had 404 reads repaired PG_DAMAGED Possible data damage: 3 pgs inconsistent pg 2.2f is active+clean+inconsistent+failed_repair, acting [29,13,18] pg 2.46 is active+clean+inconsistent+failed_repair, acting [12,0,29] pg 2.5c is active+clean+inconsistent+failed_repair, acting [12,23,0] The query result of pg 2.2f is as follows, and the problem seems to be that the three peer versions are different. [root@cephvm1 ~]# ceph pg 2.2f query { "state": "active+clean+inconsistent+failed_repair", "snap_trimq": "[]", "snap_trimq_len": 0, "epoch": 426, "up": [ 29, 13, 18 ], "acting": [ 29, 13, 18 ], "acting_recovery_backfill": [ "13", "18", "29" ], "info": { "pgid": "2.2f", "last_update": "426'128436680", "last_complete": "426'128436680", "log_tail": "390'128433627", "last_user_version": 128436529, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [], "history": { "epoch_created": 111, "epoch_pool_created": 67, "last_epoch_started": 426, "last_interval_started": 425, "last_epoch_clean": 426, "last_interval_clean": 425, "last_epoch_split": 111, "last_epoch_marked_full": 0, "same_up_since": 425, "same_interval_since": 425, "same_primary_since": 425, "last_scrub": "426'128436680", "last_scrub_stamp": "2023-06-21 15:57:53.645395", "last_deep_scrub": "426'128436680", "last_deep_scrub_stamp": "2023-06-21 15:57:53.645395", "last_clean_scrub_stamp": "2023-03-28 09:11:29.298557" }, "stats": { "version": "426'128436680", "reported_seq": "128628939", "reported_epoch": "426", "state": "active+clean+inconsistent+failed_repair", "last_fresh": "2023-06-21 15:57:53.645450", "last_change": "2023-06-21 15:57:53.645450", "last_active": "2023-06-21 15:57:53.645450", "last_peered": "2023-06-21 15:57:53.645450", "last_clean": "2023-06-21 15:57:53.645450", "last_became_active": "2023-06-21 14:03:02.233710", "last_became_peered": "2023-06-21 14:03:02.233710", "last_unstale": "2023-06-21 15:57:53.645450", "last_undegraded": "2023-06-21 15:57:53.645450", "last_fullsized": "2023-06-21 15:57:53.645450", "mapping_epoch": 425, "log_start": "390'128433627", "ondisk_log_start": "390'128433627", "created": 111, "last_epoch_clean": 426, "parent": "0.0", "parent_split_bits": 7, "last_scrub": "426'128436680", "last_scrub_stamp": "2023-06-21 15:57:53.645395", "last_deep_scrub": "426'128436680", "last_deep_scrub_stamp": "2023-06-21 15:57:53.645395", "last_clean_scrub_stamp": "2023-03-28 09:11:29.298557", "log_size": 3053, "ondisk_log_size": 3053, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 10888387166, "num_objects": 2610, "num_object_clones": 0, "num_object_copies": 7830, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 2610, "num_whiteouts": 0, "num_read": 191976, "num_read_kb": 10314827, "num_write": 128429383, "num_write_kb": 741542291, "num_scrub_errors": 3, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 3, "num_objects_recovered": 28, "num_bytes_recovered": 113242624, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 28 }, "up": [ 29, 13, 18 ], "acting": [ 29, 13, 18 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 29, "acting_primary": 29, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 426, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, "peer_info": [ { "peer": "13", "pgid": "2.2f", "last_update": "426'128436680", "last_complete": "426'128436680", "log_tail": "390'128433627", "last_user_version": 128436529, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [], "history": { "epoch_created": 111, "epoch_pool_created": 67, "last_epoch_started": 426, "last_interval_started": 425, "last_epoch_clean": 426, "last_interval_clean": 425, "last_epoch_split": 111, "last_epoch_marked_full": 0, "same_up_since": 425, "same_interval_since": 425, "same_primary_since": 425, "last_scrub": "426'128436680", "last_scrub_stamp": "2023-06-21 15:57:53.645395", "last_deep_scrub": "426'128436680", "last_deep_scrub_stamp": "2023-06-21 15:57:53.645395", "last_clean_scrub_stamp": "2023-03-28 09:11:29.298557" }, "stats": { "version": "406'128436652", "reported_seq": "128628750", "reported_epoch": "424", "state": "peering", "last_fresh": "2023-06-21 14:03:00.219516", "last_change": "2023-06-21 14:03:00.219516", "last_active": "2023-06-12 09:43:51.161310", "last_peered": "2023-04-17 12:38:42.363058", "last_clean": "2023-04-17 12:38:42.363058", "last_became_active": "2023-04-17 10:54:25.756138", "last_became_peered": "2023-04-17 10:54:25.756138", "last_unstale": "2023-06-21 14:03:00.219516", "last_undegraded": "2023-06-21 14:03:00.219516", "last_fullsized": "2023-06-21 14:03:00.219516", "mapping_epoch": 425, "log_start": "390'128433627", "ondisk_log_start": "390'128433627", "created": 111, "last_epoch_clean": 418, "parent": "0.0", "parent_split_bits": 7, "last_scrub": "406'128436652", "last_scrub_stamp": "2023-06-21 08:35:59.025077", "last_deep_scrub": "406'128436652", "last_deep_scrub_stamp": "2023-06-21 08:35:59.025077", "last_clean_scrub_stamp": "2023-03-28 09:11:29.298557", "log_size": 3025, "ondisk_log_size": 3025, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 10888387166, "num_objects": 2610, "num_object_clones": 0, "num_object_copies": 7830, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 2610, "num_whiteouts": 0, "num_read": 191976, "num_read_kb": 10314827, "num_write": 128429383, "num_write_kb": 741542291, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 29, 13, 18 ], "acting": [ 29, 13, 18 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 29, "acting_primary": 29, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 426, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "18", "pgid": "2.2f", "last_update": "426'128436680", "last_complete": "426'128436680", "log_tail": "390'128433627", "last_user_version": 128436529, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [], "history": { "epoch_created": 111, "epoch_pool_created": 67, "last_epoch_started": 426, "last_interval_started": 425, "last_epoch_clean": 426, "last_interval_clean": 425, "last_epoch_split": 111, "last_epoch_marked_full": 0, "same_up_since": 425, "same_interval_since": 425, "same_primary_since": 425, "last_scrub": "426'128436680", "last_scrub_stamp": "2023-06-21 15:57:53.645395", "last_deep_scrub": "426'128436680", "last_deep_scrub_stamp": "2023-06-21 15:57:53.645395", "last_clean_scrub_stamp": "2023-03-28 09:11:29.298557" }, "stats": { "version": "406'128436651", "reported_seq": "128628747", "reported_epoch": "406", "state": "active+clean+scrubbing+deep", "last_fresh": "2023-04-17 12:38:42.363058", "last_change": "2023-04-17 12:35:34.783904", "last_active": "2023-04-17 12:38:42.363058", "last_peered": "2023-04-17 12:38:42.363058", "last_clean": "2023-04-17 12:38:42.363058", "last_became_active": "2023-04-17 10:54:25.756138", "last_became_peered": "2023-04-17 10:54:25.756138", "last_unstale": "2023-04-17 12:38:42.363058", "last_undegraded": "2023-04-17 12:38:42.363058", "last_fullsized": "2023-04-17 12:38:42.363058", "mapping_epoch": 425, "log_start": "390'128433627", "ondisk_log_start": "390'128433627", "created": 111, "last_epoch_clean": 406, "parent": "0.0", "parent_split_bits": 7, "last_scrub": "390'128260640", "last_scrub_stamp": "2023-03-28 09:11:29.298557", "last_deep_scrub": "390'125680773", "last_deep_scrub_stamp": "2023-03-24 13:37:20.633751", "last_clean_scrub_stamp": "2023-03-28 09:11:29.298557", "log_size": 3024, "ondisk_log_size": 3024, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 10888387166, "num_objects": 2610, "num_object_clones": 0, "num_object_copies": 7830, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 2610, "num_whiteouts": 0, "num_read": 191976, "num_read_kb": 10314827, "num_write": 128429383, "num_write_kb": 741542291, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 29, 13, 18 ], "acting": [ 29, 13, 18 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 29, "acting_primary": 29, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 426, "hit_set_history": { "current_last_update": "0'0", "history": [] } } ], "recovery_state": [ { "name": "Started/Primary/Active", "enter_time": "2023-06-21 14:03:02.221320", "might_have_unfound": [], "recovery_progress": { "backfill_targets": [], "waiting_on_backfill": [], "last_backfill_started": "MIN", "backfill_info": { "begin": "MIN", "end": "MIN", "objects": [] }, "peer_backfill_info": [], "backfills_in_flight": [], "recovering": [], "pg_backend": { "pull_from_peer": [], "pushing": [] } }, "scrub": { "scrubber.epoch_start": "425", "scrubber.active": false, "scrubber.state": "INACTIVE", "scrubber.start": "MIN", "scrubber.end": "MIN", "scrubber.max_end": "MIN", "scrubber.subset_last_update": "0'0", "scrubber.deep": false, "scrubber.waiting_on_whom": [] } }, { "name": "Started", "enter_time": "2023-06-21 14:03:01.225451" } ], "agent_state": {} } ------------- "last_user_version": 128436529, "version": "426'128436680", "last_user_version": 128436529, "version": "406'128436652", "last_user_version": 128436529, "version": "406'128436651", Can I repari to a specific version?

1 0

RGW: Migrating a long-lived cluster to multi-site, fixing an EC pool mistake
by Christian Theune 21 Jun '23

21 Jun '23

Hi, we are running a cluster that has been alive for a long time and we tread carefully regarding updates. We are still a bit lagging and our cluster (that started around Firefly) is currently at Nautilus. We’re updating and we know we’re still behind, but we do keep running into challenges along the way that typically are still unfixed on main and - as I started with - have to tread carefully. Nevertheless, mistakes happen, and we found ourselves in this situation: we converted our RGW data pool from replicated (n=3) to erasure coded (k=10, m=3, with 17 hosts) but when doing the EC profile selection we missed that our hosts are not evenly balanced (this is a growing cluster and some machines have around 20TiB capacity for the RGW data pool, wheres newer machines have around 160TiB and we rather should have gone with k=4, m=3. In any case, having 13 chunks causes too many hosts to participate in each object. Going for k+m=7 will allow distribution to be more effective as we have 7 hosts that have the 160TiB sizing. Our original migration used the “cache tiering” approach, but that only works once when moving from replicated to EC and can not be used for further migrations. The amount of data is at 215TiB somewhat significant, so using an approach that scales when copying data[1] to avoid ending up with months of migration. I’ve run out of ideas doing this on a low-level (i.e. trying to fix it on a rados/pool level) and I guess we can only fix this on an application level using multi-zone replication. I have the setup nailed in general, but I’m running into issues with buckets in our staging and production environment that have `explicit_placement` pools attached, AFAICT is this an outdated mechanisms but there are no migration tools around. I’ve seen some people talk about patched versions of the `radosgw-admin metadata put` variant that (still) prohibits removing explicit placements. AFAICT those explicit placements will be synced to the secondary zone and the effect that I’m seeing underpins that theory: the sync runs for a while and only a few hundred objects show up in the new zone, as the buckets/objects are already found in the old pool that the new zone uses due to the explicit placement rule. I’m currently running out of ideas, but open for any other options. Looking at https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/ULKK5RU2VXL… I’m wondering whether the relevant patch is available somewhere, or whether I’ll have to try building that patch again on my own. Going through the docs and the code I’m actually wondering whether `explicit_placement` is actually a really crufty residual piece that won’t get used in newer clusters but older clusters don’t really have an option to get away from? In my specific case, the placement rules are identical to the explicit placements that are stored on (apparently older) buckets and the only thing I need to do is to remove them. I can accept a bit of downtime to avoid any race conditions if needed, so maybe having a small tool to just remove those entries while all RGWs are down would be fine. A call to `radosgw-admin bucket stat` takes about 18s for all buckets in production and I guess that would be a good comparison for what timing to expect when running an update on the metadata. I’ll also be in touch with colleagues from Heinlein and 42on but I’m open to other suggestions. Hugs, Christian [1] We currently have 215TiB data in 230M objects. Using the “official” “cache-flush-evict-all” approach was unfeasible here as it only yielded around 50MiB/s. Using cache limits and targetting the cache sizes to 0 caused proper parallelization and was able to flush/evict at almost constant 1GiB/s in the cluster. -- Christian Theune · ct(a)flyingcircus.io · +49 345 219401 0 Flying Circus Internet Operations GmbH · https://flyingcircus.io Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick

2 6

Recover OSDs from folder /var/lib/ceph/uuid/removed
by Malte Stroem 21 Jun '23

21 Jun '23

Hello, is it possible to recover an OSD if it was removed? The systemd service was removed but the block device is still listed under lsblk and the config files are still available under /var/lib/ceph/uuid/removed It is a containerized cluster. So I think we need to add the cephx entries, use ceph-volume, crush, and so on. Best regards, Malte

1 2

[question] Put with "tagging" is slowly?
by Louis Koo 21 Jun '23

21 Jun '23

2023-06-21T02:48:50.754+0000 7f1cd5b84700 1 beast: 0x7f1c4b26e630: 10.x.x.83 - xx [21/Jun/2023:02:48:47.653 +0000] "PUT /zhucan/deb/content/vol-26/chap-41/3a917ec7-02b3-4b45-8c0c-be32f4914708.bytes?tagging HTTP/1.1" 200 0 - "aws-sdk-java/1.12.299 Linux/3.10.0-1127.el7.x86_64 OpenJDK_64-Bit_Server_VM/25.372-b07 java/1.8.0_372 vendor/Red_Hat,_Inc. cfg/retry-mode/legacy" - latency=3.101041317s 2023-06-21T02:48:53.789+0000 7f1cf53c3700 1 beast: 0x7f1c4b26e630: 10.x.x.83 - xx [21/Jun/2023:02:48:50.758 +0000] "PUT /zhucan/deb/content/vol-26/chap-41/3a917ec7-02b3-4b45-8c0c-be32f4914708.properties?tagging HTTP/1.1" 200 0 - "aws-sdk-java/1.12.299 Linux/3.10.0-1127.el7.x86_64 OpenJDK_64-Bit_Server_VM/25.372-b07 java/1.8.0_372 vendor/Red_Hat,_Inc. cfg/retry-mode/legacy" - latency=3.031040430s cost more 3s, why?

1 0

Error while adding host : Error EINVAL: Traceback (most recent call last): File /usr/share/ceph/mgr/mgr_module.py, line 1756, in _handle_command
by Adiga, Anantha 20 Jun '23

20 Jun '23

Hi, I am seeing this error after an offline was deleted and while adding the host again. Thereafter, I have removed the /var/lib/cep folder and removed the ceph quincy image in the offline host. What is the cause of this issue and the solution. root@fl31ca104ja0201:/home/general# cephadm shell Inferring fsid d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e Using recent ceph image quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e<mailto:quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e> root@fl31ca104ja0201:/# root@fl31ca104ja0201:/# ceph orch host rm fl31ca104ja0302 --offline --force Removed offline host 'fl31ca104ja0302' root@fl31ca104ja0201:/# ceph -s cluster: id: d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e health: HEALTH_OK services: mon: 3 daemons, quorum fl31ca104ja0201,fl31ca104ja0202,fl31ca104ja0203 (age 28h) mgr: fl31ca104ja0203(active, since 6d), standbys: fl31ca104ja0202, fl31ca104ja0201 mds: 1/1 daemons up, 2 standby osd: 33 osds: 33 up (since 28h), 33 in (since 28h) rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 24 pools, 737 pgs objects: 613.56k objects, 1.9 TiB usage: 2.9 TiB used, 228 TiB / 231 TiB avail pgs: 737 active+clean io: client: 161 MiB/s rd, 75 op/s rd, 0 op/s wr root@fl31ca104ja0201:/# ceph orch host add fl31ca104ja0302 10.45.219.5 Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1756, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 171, in handle_command return dispatch[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 107, in <lambda> wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) # noqa: E731 File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 96, in wrapper return func(*args, **kwargs) File "/usr/share/ceph/mgr/orchestrator/module.py", line 356, in _add_host return self._apply_misc([s], False, Format.plain) File "/usr/share/ceph/mgr/orchestrator/module.py", line 1092, in _apply_misc raise_if_exception(completion) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 225, in raise_if_exception e = pickle.loads(c.serialized_exception) TypeError: __init__() missing 2 required positional arguments: 'hostname' and 'addr' Thank you, Anantha

2 2

[rgw multisite] Perpetual behind
by Yixin Jin 20 Jun '23

20 Jun '23

Hi ceph gurus, I am experimenting with rgw multisite sync feature using Quincy release (17.2.5). I am using the zone-level sync, not bucket-level sync policy. During my experiment, somehow my setup got into a situation that it doesn't seem to get out of. One zone is perpetually behind the other, although there is no ongoing client request. Here is the output of my "sync status": root@mon1-z1:~# radosgw-admin sync status realm f90e4356-3aa7-46eb-a6b7-117dfa4607c4 (test-realm) zonegroup a5f23c9c-0640-41f2-956f-a8523eccecb3 (zg) zone bbe3e2a1-bdba-4977-affb-80596a6fe2b9 (z1) metadata sync no sync (zone is master) data sync source: 9645a68b-012e-4889-bf24-096e7478f786 (z2) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 14 shards behind shards: [56,61,63,107,108,109,110,111,112,113,114,115,116,117] It stays behind forever while rgw is almost completely idle (1% of CPU). Any suggestion on how to drill deeper to see what happened? Thanks, Yixin

4 4

RGW STS Token Forbidden error since upgrading to Quincy 17.2.6
by Austin Axworthy 20 Jun '23

20 Jun '23

Hi, We are using STS tokens to grant temporary access for users. I was running ceph version 15.2.16 and this method worked as expected until the cluster was upgraded to 17.2.6. Using boto3 I am able to assume the correct role, but when trying to use the temporary credentials a forbidden error is reported. This was working before the upgrade. I have also tested on a 15.2.17 cluster and experience the same issue. I have pasted the created role, and boto3 code I am using as well as the error I am running into. Any insight on this issue would be greatly appreciated. Error: botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden Role: { "RoleId": "499eba48-8431-48f7-9aca-76000b9c01cc", "RoleName": "DefaultNoS3Access", "Path": "/", "Arn": "arn:aws:iam:::role/DefaultNoS3Access", "CreateDate": "2023-05-11T12:04:44.367Z", "MaxSessionDuration": 3600, "AssumeRolePolicyDocument": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principa l\":{\"AWS\":[\"arn:aws:iam:::user/user\"]},\"Action\":[\"sts:AssumeRole\"]} ]}" }, Boto3 Code: response = sts_client.assume_role( RoleArn=role_arn, RoleSessionName="test", Policy=json.dumps( { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObjectAcl", "s3:PutObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::bucket1/*", "arn:aws:s3:::bucket2/*" ] } ] }, separators=(',', ':') ) ) credentials = response['Credentials'] s3_client = boto3.client( 's3', aws_access_key_id=credentials['AccessKeyId'], aws_secret_access_key=credentials['SecretAccessKey'], aws_session_token=credentials['SessionToken'], endpoint_url="http://IP:8080", region_name="" ) bucket_name = 'bucket1' file_key = 'test.txt' local_file_path = '/' s3_client.download_file(bucket_name, file_key, local_file_path)

2 4

osd memory target not work
by farhad kh 20 Jun '23

20 Jun '23

when set osd_memory_target for limitation usage memory for osd disk ,This value is expected to be set for the OSD container .But with the docker stats command, this value is not seen Is my perception of this process wrong? ----------- [root@opcsdfpsbpp0201 ~]# ceph orch ps | grep osd.12 osd.12 opcsdfpsbpp0201 running (9d) 5m ago 9d 1205M 1953M 17.2.6 c9a1062f7289 bf27cfe16046 [root@opcsdfpsbpp0201 ~]# docker stats | grep osd 1253766d6a78 ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-48 0.05% 2.237GiB / 7.732GiB 28.93% 0B / 0B 86.8GB / 562GB 63 2bc012e5c604 ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-67 0.20% 727MiB / 7.732GiB 9.18% 0B / 0B 37.5GB / 1.29TB 63 dc0bf068050b ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-62 0.11% 360.5MiB / 7.732GiB 4.55% 0B / 0B 125MB / 1.85GB 63 c5f119a37652 ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-55 0.12% 312.5MiB / 7.732GiB 3.95% 0B / 0B 86.6MB / 1.66GB 63 7f0b7b61807d ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-5 0.11% 299.4MiB / 7.732GiB 3.78% 0B / 0B 119MB / 1.6GB 63 dadffc77f7b6 ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-40 0.11% 274MiB / 7.732GiB 3.46% 0B / 0B 110MB / 1.5GB 63 e439e58d907e ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-34 0.12% 355.9MiB / 7.732GiB 4.49% 0B / 0B 125MB / 1.78GB 63 5e500e2197d6 ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-25 0.11% 273.3MiB / 7.732GiB 3.45% 0B / 0B 128MB / 1.55GB 63 a63709567669 ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-19 0.11% 714.6MiB / 7.732GiB 9.03% 0B / 0B 89.8MB / 167GB 63 bf27cfe16046 ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-12 0.16% 1.177GiB / 7.732GiB 15.23% 0B / 0B 40.8GB / 644GB 63 ----------- # ceph orch ps | grep osd | grep opcsdfpsbpp0201 osd.5 opcsdfpsbpp0201 running (9d) 6m ago 9d 298M 1953M 17.2.6 c9a1062f7289 7f0b7b61807d osd.12 opcsdfpsbpp0201 running (9d) 6m ago 9d 1205M 1953M 17.2.6 c9a1062f7289 bf27cfe16046 osd.19 opcsdfpsbpp0201 running (9d) 6m ago 9d 704M 1953M 17.2.6 c9a1062f7289 a63709567669 osd.25 opcsdfpsbpp0201 running (9d) 6m ago 9d 273M 1953M 17.2.6 c9a1062f7289 5e500e2197d6 osd.34 opcsdfpsbpp0201 running (9d) 6m ago 9d 355M 1953M 17.2.6 c9a1062f7289 e439e58d907e osd.40 opcsdfpsbpp0201 running (9d) 6m ago 9d 273M 1953M 17.2.6 c9a1062f7289 dadffc77f7b6 osd.48 opcsdfpsbpp0201 running (4h) 6m ago 9d 2290M 1953M 17.2.6 c9a1062f7289 1253766d6a78 osd.55 opcsdfpsbpp0201 running (9d) 6m ago 9d 312M 1953M 17.2.6 c9a1062f7289 c5f119a37652 osd.62 opcsdfpsbpp0201 running (9d) 6m ago 9d 359M 1953M 17.2.6 c9a1062f7289 dc0bf068050b osd.67 opcsdfpsbpp0201 running (9d) 6m ago 9d 727M 1953M 17.2.6 c9a1062f7289 2bc012e5c604 ---------------------------------- #ceph config get mgr osd_memory_target 2048000000

2 1

X large objects found in pool 'XXX.rgw.buckets.index'
by Gilles Mocellin 20 Jun '23

20 Jun '23

Hello, I still have large OMAP objects since a year. These objects are probably from an ancient bucket that has been removed. So I cannot use bilog trim. Depp-scrub dos nothing. Also, even if I don't have a huge cluster (my Object Storage pools is only arounde 10TB), the rgw-orphan-list is too long to run. So, As I have only 6 Large OMAP objects (just above 200000 keys), I would like to find and remove the orphan rados objects. If someone can tell me if I'm right in my assumptions ? Let's take that log : 2023-06-09T00:51:10.222449+0200 osd.66 (osd.66) 12 : cluster [WRN] Large omap object found. Object: 3:56f3a469:::.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.9:head PG: 3.9625cf6a (3.2a) Key count: 200304 Size (bytes): 58606170 I deduce that the bucket-id is : aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2 I cannot find a bucket name with that ID in the metadata list : # radosgw-admin metadata list --metadata-key bucket.instance | grep -i aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2 returns nothing. If I list all objects begining with that bucket ID, I find : gmo_admin@fidcl-mrs4-sto-sds-01:~$ sudo rados -p MRS4.rgw.buckets.index ls | grep aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2 | cat .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.25 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.0 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.15 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.17 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.9 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.26 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.7 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.23 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.2 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.3 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.16 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.13 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.22 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.21 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.4 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.20 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.19 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.11 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.27 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.24 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.28 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.10 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.14 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.18 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.1 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.6 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.8 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.12 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.5 Do you think I'm safe to delete them ? If they are all about a non-existent bucket... How can I be sure that an RGW index object and the omap keys it has, are not used ?

1 0

OpenStack (cinder) volumes retyping on Ceph back-end
by andrea.martra＠oscct.it 20 Jun '23

20 Jun '23

Hello, I configured different back-end storage on OpenStack (Yoga release) and using Ceph (ceph version 17.2.4) with different pools (volumes, cloud-basic, shared-hosting-os, shared-hosting-homes,...) for RBD application. I created different volume types towards each of the backends and everything works perfectly. If I perform a "change volume type" from OpenStack on volumes attached to the VMs the system successfully migrates the volume from the source pool to the destination pool and at the end of the process the volume is visible in the new pool and is removed from the old pool. The problem encountered is that when reconfiguring the VM, to specify the new pool associated with the volumes (performed through a resize of the VM, I haven't found any other method to change the information on the nova/cinder db automatically. I also did some tests shut-off of the VM and modification of the xml through virsh edit and startup the VM) the volume presented to the VM is exactly the version and content on the retype date of the volume itself. All data written and modified after the retype is lost. The VM after the retype continues to work perfectly in RW but the "new" volume created in the new pool is not used to write data and consequently when the VM is shut down all the changes are lost. Do you have any idea how to carry out a check and possibly how to proceed in order not to lose the data of the vm of which I have retyped the volume? The data is written somewhere because the VMs work perfectly. Thank you

3 3

Transmit rate metric based per bucket
by Szabo, Istvan (Agoda) 20 Jun '23

20 Jun '23

Hello, I'd like to know is there a way to query some metrics/logs in octopus (or if has newer version I'm interested for the future too) about the bandwidth used in the bucket for put/get operations? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

CephFS metadata pool grows by two orders of magnitude while trimming (?) snapshots
by Janek Bevendorff 19 Jun '23

19 Jun '23

Hi, Perhaps this is a known issue and I was simply too dumb to find it, but we are having problems with our CephFS metadata pool filling up over night. Our cluster has a small SSD pool of around 15TB which hosts our CephFS metadata pool. Usually, that's more than enough. The normal size of the pool ranges between 200 and 800GiB (which is quite a lot of fluctuation already). Yesterday, we had suddenly had the pool fill up entirely and they only way to fix it was to add more capacity. I increased the pool size to 18TB by adding more SSDs and could resolve the problem. After a couple of hours of reshuffling, the pool size finally went back to 230GiB. But then we had another fill-up tonight to 7.6TiB. Luckily, I had adjusted the weights so that not all disks could fill up entirely like last time, so it ended there. I wasn't really able to identify the problem yesterday, but under the more controllable scenario today, I could check the MDS logs at debug_mds=10 and to me it seems like the problem is caused by snapshot trimming. The logs contain a lot of snapshot-related messages for paths that haven't been touched in a long time. The messages all look something like this: May 31 09:16:48 XXX ceph-mds[2947525]: 2023-05-31T09:16:48.292+0200 7f7ce1bd9700 10 mds.1.cache.ino(0x1000b3c3670) add_client_cap first cap, joining realm snaprealm(0x10000000000 seq 1b1c lc 1b1b cr 1 b1b cps 2 snaps={185f=snap(185f 0x10000000000 'monthly_20221201' 2022-12-01T00:00:01.530830+0100),18de=snap(18de 0x10000000000 'monthly_20230101' 2023-01-01T00:00:04.657252+0100),1941=snap(1941 0x10000000000 ... May 31 09:25:03 XXX ceph-mds[3268481]: 2023-05-31T09:25:03.396+0200 7f0e6a6ca700 10 mds.0.cache | |______ 3 rep [dir 0x100000218fe.101111101* /storage/REDACTED/| ptrwaiter=0 request=0 child=0 frozen=0 subtree=1 replicated=0 dirty=0 waiter=0 authpin=0 tempexporting=0 0x5607759d9600] May 31 09:25:03 XXX ceph-mds[3268481]: 2023-05-31T09:25:03.452+0200 7f0e6a6ca700 10 mds.0.cache | | |____ 4 rep [dir 0x100000ff904.100111101010* /storage/REDACTED/| ptrwaiter=0 request=0 child=0 frozen=0 subtree=1 importing=0 replicated=0 waiter=0 authpin=0 tempexporting=0 0x56034ed25200] May 31 09:25:03 XXX ceph-mds[3268481]: 2023-05-31T09:25:03.716+0200 7f0e6becd700 10 mds.0.server set_trace_dist snaprealm snaprealm(0x10000000000 seq 1b1c lc 1b1b cr 1b1b cps 2 snaps={185f=snap(185f 0x10000000000 'monthly_20221201' 2022-12-01T00:00:01.530830+0100),18de=snap(18de 0x10000000000 'monthly_20230101' 2023-01-01T00:00:04.657252+0100),1941=snap(1941 0x10000000000 'monthly_20230201' 2023-02-01T00:00:01.854059+0100),19a6=snap(19a6 0x10000000000 'monthly_20230301' 2023-03-01T00:00:01.215197+0100),1a24=snap(1a24 0x10000000000 'monthly_20230401' ...) len=384 May 31 09:25:36 deltaweb055 ceph-mds[3268481]: 2023-05-31T09:25:36.076+0200 7f0e6becd700 10 mds.0.cache.ino(0x10004d74911) remove_client_cap last cap, leaving realm snaprealm(0x10000000000 seq 1b1c lc 1b1b cr 1b1b cps 2 snaps={185f=snap(185f 0x10000000000 'monthly_20221201' 2022-12-01T00:00:01.530830+0100),18de=snap(18de 0x10000000000 'monthly_20230101' ...) The daily_*, montly_* etc. names are the names of our regular snapshots. I posted a larger log file snippet using ceph-post-file with the ID: da0eb93d-f340-4457-8a3f-434e8ef37d36 Is it possible that the MDS are trimming old snapshots without taking care not to fill up the entire metadata pool? Cheers Janek

3 16

Starting v17.2.5 RGW SSE with default key (likely others) no longer works
by Jayanth Reddy 19 Jun '23

19 Jun '23

Hello Folks, I've been experimenting with RGW encryption and found this out. Focusing on Quincy and Reef dev, for the SSE (any methods) to work, transit has to be end to end encrypted, however if there is a proxy, then [1] can be made use to tell RGW that SSL is being terminated. As per docs, RGW can still continue to accept SSE if rgw_crypt_require_ssl is set to false as an overriding item for the requirement of encryption in transit. Below are my observations. Until v17.2.3 ( quay.io/ceph/ceph@sha256:43f6e905f3e34abe4adbc9042b9d6f6b625dee8fa8d93c2bae53fa9b61c3df1a), setting the same key as in [2], would show the object unreadable when copied using # rados -p default.rgw.buckets.data get 03c2ef32-b7c8-4e18-8e0c-ebac10a42f10.17254.1_file.plain file.enc The object would be unreadable. The original object is in plain text. Ofcourse, with rgw_crypt_require_ssl to false or [1] However, starting with v17.2.4 onwards and even until my recent testing with reef-dev (18.0.0-4353-g1e3835ab 1e3835abb2d19ce6ac4149c260ef804f1041d751) When I try getting the same object onto the disk using rados command, the object (contains plain text) would still be readable. Has something changed since v17.2.4? I'll also test with Pacific and let you know. Not sure if it affects other SSE mechanisms as well. [1] https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_trust_forwa… [2] https://docs.ceph.com/en/quincy/radosgw/encryption/#automatic-encryption-fo… Thanks, Jayanth Reddy

2 2

header_limit in AsioFrontend class
by Vahideh Alinouri 19 Jun '23

19 Jun '23

Dear Ceph Users, I am writing to request the backporting changes related to the AsioFrontend class and specifically regarding the header_limit value. In the Pacific release of Ceph, the header_limit value in the AsioFrontend class was set to 4096. From Quincy release, there has been a configurable option introduced to set the header_limit value and the default value is 16384. I would greatly appreciate it if someone from the Ceph development team backport this change to the older version. Best regards, Vahideh Alinouri

2 1

Re: same OSD in multiple CRUSH hierarchies
by Budai Laszlo 19 Jun '23

19 Jun '23

Hi, Actually I've learned that it's not needed for a rule to start with a root bucket, so I can heve rules that will only consider a subtree of my total resources, and achieve what I was trying to do with the different disjunct hierarchies. BTW: it is possible to have different trees with different roots, with some OSDs being part of multiple such trees, and create different rules that will start with one root or the other. But I was told that this could mess up the calculation for pg autoscaler and other housekeeping functions. So it seems a better option to have each OSD in one single tree, and use rules that will only consider subtrees ... Regards, Laszlo > Date: Mon, 19 Jun 2023 07:41:35 +0000 > From: Eugen Block<eblock(a)nde.ag> > Subject: [ceph-users] Re: same OSD in multiple CRUSH hierarchies > To:ceph-users@ceph.io > Message-ID: > <20230619074135.Horde.gS8nAKQgZhlbV0HpymJ-lqf(a)webmail.nde.ag> > Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes > > Hi, > I don't think this is going to work. Each OSD belongs to a specific > host and you can't have multiple buckets (e.g. bucket type "host") > with the same name in the crush tree. But if I understand your > requirement correctly, there should be no need to do it this way. If > you structure your crush tree according to your separation > requirements and the critical pools use designated rules, you can > still have a rule that doesn't care about the data separation but > distributes the replicas across the available hosts (given your > failure domain would be "host"), which is already the default for the > replicated_rule. Did I misunderstand something? > > Regards, > Eugen > > > Zitat von Budai Laszlo<laszlo.budai(a)gmail.com>: > >> Hi there, >> >> I'm curious if there is anything against configuring an ODS to be >> part in multiple CRUSH hierarchies. I'm thinking of the following >> scenario: >> >> I want to create pools that are using distinct sets of OSDs. I want >> to make sure that a piece data which replicated at application level >> will not end up on the same OSD. So I would creat multiple CRUSH >> hierarchies (root - host - osd) but using different OSDs in each, >> and different rules that are using those hierarchies. Then I would >> create pools with the different rules, and use those different pools >> for storing the data for the different application instances. But I >> would also like to use the OSDs in the "default hierarchy" set up by >> ceph where all the hosts are in the same root bucket, and have the >> default replicated rule, so my generic data volumes would be able to >> spread across all the OSDs available. >> >> Is there something against this setup? >> >> Thank you for any advice! >> Laszlo >> _______________________________________________ >> ceph-users mailing list --ceph-users(a)ceph.io >> To unsubscribe send an email toceph-users-leave(a)ceph.io

1 0

EC 8+3 Pool PGs stuck in remapped+incomplete
by Jayanth Reddy 19 Jun '23

19 Jun '23

Hello Users, Greetings. We've a Ceph Cluster with the version *ceph version 14.2.5-382-g8881d33957 (8881d33957b54b101eae9c7627b351af10e87ee8) nautilus (stable)* 5 PGs belonging to our RGW 8+3 EC Pool are stuck in incomplete and incomplete+remapped states. Below are the PGs, # ceph pg dump_stuck inactive ok PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 15.251e incomplete [151,464,146,503,166,41,555,542,9,565,268] 151 [151,464,146,503,166,41,555,542,9,565,268] 151 15.3f3 incomplete [584,281,672,699,199,224,239,430,355,504,196] 584 [584,281,672,699,199,224,239,430,355,504,196] 584 15.985 remapped+incomplete [396,690,493,214,319,209,546,91,599,237,352] 396 [2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352] 214 15.39d3 remapped+incomplete [404,221,223,585,38,102,533,471,568,451,195] 404 [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] 223 15.d46 remapped+incomplete [297,646,212,254,110,169,500,372,623,470,678] 297 [2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678] 548 Some of the OSDs had gone down on the cluster. Below is the # ceph status # ceph -s cluster: id: 30d6f7ee-fa02-4ab3-8a09-9321c8002794 health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 1 pools have many more objects per pg than average Reduced data availability: 5 pgs inactive, 5 pgs incomplete Degraded data redundancy: 44798/8718528059 objects degraded (0.001%), 1 pg degraded, 1 pg undersized 22726 pgs not deep-scrubbed in time 23552 pgs not scrubbed in time 77 slow ops, oldest one blocked for 56400 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops. too many PGs per OSD (330 > max 250) services: mon: 3 daemons, quorum brc1mon2,brc1mon3,brc1mon1 (age 2y) mgr: brc1mon2(active, since 8d), standbys: brc1mon1, brc1mon3 mds: cephfs:1 {0=brc1mds2=up:active} 1 up:standby osd: 1012 osds: 698 up (since 14h), 698 in (since 2d); 3 remapped pgs flags noscrub,nodeep-scrub rgw: 2 daemons active (brc1rgw1, brc1rgw2) data: pools: 17 pools, 23552 pgs objects: 863.74M objects, 1.2 PiB usage: 2.4 PiB used, 6.2 PiB / 8.6 PiB avail pgs: 0.021% pgs not active 44798/8718528059 objects degraded (0.001%) 23546 active+clean 3 remapped+incomplete 2 incomplete 1 active+undersized+degraded io: client: 24 MiB/s rd, 3.2 KiB/s wr, 56 op/s rd, 4 op/s wr And the health detail shows as # ceph health detail HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 pools have many more objects per pg than average; Reduced data availability: 5 pgs inactive, 5 pgs incomplete; Degraded data redundancy: 44798/8718528081 objects degraded (0.001%), 1 pg degraded, 1 pg undersized; 22726 pgs not deep-scrubbed in time; 23552 pgs not scrubbed in time; 77 slow ops, oldest one blocked for 56440 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops.; too many PGs per OSD (330 > max 250) OSDMAP_FLAGS noscrub,nodeep-scrub flag(s) set MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average pool iscsi-images objects per pg (540004) is more than 14.7248 times cluster average (36673) PG_AVAILABILITY Reduced data availability: 5 pgs inactive, 5 pgs incomplete pg 15.3f3 is incomplete, acting [584,281,672,699,199,224,239,430,355,504,196] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.985 is remapped+incomplete, acting [2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.d46 is remapped+incomplete, acting [2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.251e is incomplete, acting [151,464,146,503,166,41,555,542,9,565,268] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.39d3 is remapped+incomplete, acting [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') PG_DEGRADED Degraded data redundancy: 44798/8718528081 objects degraded (0.001%), 1 pg degraded, 1 pg undersized pg 15.28f0 is stuck undersized for 67359238.592403, current state active+undersized+degraded, last acting [2147483647,343,355,415,426,640,302,392,78,202,607] PG_NOT_DEEP_SCRUBBED 22726 pgs not deep-scrubbed in time We've the pools as below # ceph osd lspools 1 iscsi-images 2 cephfs_data 3 cephfs_metadata 4 .rgw.root 5 default.rgw.control 6 default.rgw.meta 7 default.rgw.log 8 default.rgw.buckets.index 13 rbd 15 default.rgw.buckets.data 16 default.rgw.buckets.non-ec 19 cephfs_data-ec 22 rbd-ec 23 iscsi-images-ec 24 hpecpool 25 hpec.rgw.buckets.index 26 hpec.rgw.buckets.non-ec We've been struggling for a long time to fix this but out of luck! Our RGW daemons hosted on dedicated machines are continuously failing to respond, being behind a load balancer, LB throws 504 Gateway Timeout as the daemons are failing to respond in the expected time. We perform active health checks from the LB on '/' by HTTP HEAD but these are failing as well, very frequently. Currently we're surviving by writing a script that restarts RGW daemons whenever the LB responds with HTTP status code 504. Any help is highly appreciated! Regards, Jayanth Reddy

3 7

same OSD in multiple CRUSH hierarchies
by Budai Laszlo 19 Jun '23

19 Jun '23

Hi there, I'm curious if there is anything against configuring an ODS to be part in multiple CRUSH hierarchies. I'm thinking of the following scenario: I want to create pools that are using distinct sets of OSDs. I want to make sure that a piece data which replicated at application level will not end up on the same OSD. So I would creat multiple CRUSH hierarchies (root - host - osd) but using different OSDs in each, and different rules that are using those hierarchies. Then I would create pools with the different rules, and use those different pools for storing the data for the different application instances. But I would also like to use the OSDs in the "default hierarchy" set up by ceph where all the hosts are in the same root bucket, and have the default replicated rule, so my generic data volumes would be able to spread across all the OSDs available. Is there something against this setup? Thank you for any advice! Laszlo

2 1

autocaling not work and active+remapped+backfilling
by farhad kh 19 Jun '23

19 Jun '23

hi i have a problem with ceph 17.2.6 , cephfs with mds daemons but see an unusual behavior. create a data pool with default crush rule but data just store in 3 specific osd and other osd is clean PG auto-scaling is also active but its size does not change when the pool is biger I did this manually but the problem was not solved and I got the error pg are not balanced across osds How do I solve this problem? Is this a bug? I did not have this problem in previous versions I solved this problem. There are several identical crash rules in the folder step chooseleaf firstn 0 type host I think this confuses the balancer and autoscale and output for ceph osd pool autoscale-status is empty after remove other crush rules autoscale runing but move data from osd full to clear osd is slow trying with Reducing the weight of filled OSDs, I tried to prioritize the use of other OSDs ceph osd reweight-by-utilization I hope this works Is there a solution that makes the process of autoscaling and cleaning placement groups faster? ------------------------------- [root@opcsdfpsbpp0201 ~]# ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "r3-host", "type": 1, "steps": [ { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "r3", "type": 1, "steps": [ { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ] ------------------------------------ # ceph osd status | grep back 23 opcsdfpsbpp0211 1900G 147G 0 0 0 0 backfillfull,exists,up 48 opcsdfpsbpp0201 1900G 147G 0 0 0 0 backfillfull,exists,up 61 opcsdfpsbpp0205 1900G 147G 0 0 0 0 backfillfull,exists,up ---------------------------------------------- Every 2.0s: ceph -s opcsdfpsbpp0201: Sun Jun 18 11:44:29 2023 cluster: id: 79a2627c-0821-11ee-a494-00505695c58c health: HEALTH_WARN 3 backfillfull osd(s) 6 pool(s) backfillfull services: mon: 3 daemons, quorum opcsdfpsbpp0201,opcsdfpsbpp0205,opcsdfpsbpp0203 (age 6d) mgr: opcsdfpsbpp0201.vttwxa(active, since 5d), standbys: opcsdfpsbpp0205.tpodbs, opcsdfpsbpp0203.jwjkcl mds: 1/1 daemons up, 2 standby osd: 74 osds: 74 up (since 7d), 74 in (since 7d); 107 remapped pgs data: volumes: 1/1 healthy pools: 6 pools, 359 pgs objects: 599.64k objects, 2.2 TiB usage: 8.1 TiB used, 140 TiB / 148 TiB avail pgs: 923085/1798926 objects misplaced (51.313%) 252 active+clean 87 active+remapped+backfill_wait 20 active+remapped+backfilling io: client: 255 B/s rd, 0 op/s rd, 0 op/s wr recovery: 33 MiB/s, 8 objects/s progress: Global Recovery Event (5h) [===================.........] (remaining: 2h)

1 0

cephfs mount with kernel driver
by farhad kh 19 Jun '23

19 Jun '23

I noticed that in my scenario, when I mount cephfs via the kernel module, it directly copies to one or three of the OSDs. And the writing speed of the client is higher than the speed of replication and auto scaling This causes the writing operation to stop as soon as those OSDs are filled, and the error of free space is not available What should be done to solve this problem? Is there a way to increase the speed of scaling or moving objects in OSD? Or a way to mount cephfs that does not have these problems?

1 0

Re: EC 8+3 Pool PGs stuck in remapped+incomplete
by Jayanth Reddy 17 Jun '23

17 Jun '23

Hello Anthony / Users, After some initial analysis, I had increased max_pg_per_osd to 480, but we're out of luck. Also tried force-backfill and force-repair as well. On querying PG using *# ceph pg **<pg.ID> query* the output says blocked_by 3 to 4 OSDs which are out of the cluster already. Guessing if these have to do something with the recovery. Thanks, Jayanth Reddy On Sat, Jun 17, 2023 at 4:17 PM Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > Your cluster’s configuration is preventing CRUSH from calculating full > placements > > set max_pg_per_osd = 1000, either in central config (or ceph.conf if you > have it set there now). > > If you have it set in ceph.conf, you may need to serially restart the mons. > > ceph osd down 214 > sleep 60 > ceph osd down 223 > sleep 60 > ceph osd down 548 > sleep 60 > ceph osd down 584 > > > > > > > > On Jun 17, 2023, at 2:22 AM, Jayanth Reddy <jayanthreddy5666(a)gmail.com> > wrote: > > > > Hello Users, > > Greetings. We've a Ceph Cluster with the version > > *ceph version 14.2.5-382-g8881d33957 > > (8881d33957b54b101eae9c7627b351af10e87ee8) nautilus (stable)* > > > > 5 PGs belonging to our RGW 8+3 EC Pool are stuck in incomplete and > > incomplete+remapped states. Below are the PGs, > > > > # ceph pg dump_stuck inactive > > ok > > PG_STAT STATE UP > > UP_PRIMARY ACTING > > ACTING_PRIMARY > > 15.251e incomplete [151,464,146,503,166,41,555,542,9,565,268] > > 151 > > [151,464,146,503,166,41,555,542,9,565,268] 151 > > 15.3f3 incomplete [584,281,672,699,199,224,239,430,355,504,196] > > 584 > > [584,281,672,699,199,224,239,430,355,504,196] 584 > > 15.985 remapped+incomplete [396,690,493,214,319,209,546,91,599,237,352] > > 396 > > > [2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352] > > 214 > > 15.39d3 remapped+incomplete [404,221,223,585,38,102,533,471,568,451,195] > > 404 > > [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] > > 223 > > 15.d46 remapped+incomplete [297,646,212,254,110,169,500,372,623,470,678] > > 297 > > [2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678] > > 548 > > > > Some of the OSDs had gone down on the cluster. Below is the # ceph status > > > > # ceph -s > > cluster: > > id: 30d6f7ee-fa02-4ab3-8a09-9321c8002794 > > health: HEALTH_WARN > > noscrub,nodeep-scrub flag(s) set > > 1 pools have many more objects per pg than average > > Reduced data availability: 5 pgs inactive, 5 pgs incomplete > > Degraded data redundancy: 44798/8718528059 objects degraded > > (0.001%), 1 pg degraded, 1 pg undersized > > 22726 pgs not deep-scrubbed in time > > 23552 pgs not scrubbed in time > > 77 slow ops, oldest one blocked for 56400 sec, daemons > > [osd.214,osd.223,osd.548,osd.584] have slow ops. > > too many PGs per OSD (330 > max 250) > > > > services: > > mon: 3 daemons, quorum brc1mon2,brc1mon3,brc1mon1 (age 2y) > > mgr: brc1mon2(active, since 8d), standbys: brc1mon1, brc1mon3 > > mds: cephfs:1 {0=brc1mds2=up:active} 1 up:standby > > osd: 1012 osds: 698 up (since 14h), 698 in (since 2d); 3 remapped pgs > > flags noscrub,nodeep-scrub > > rgw: 2 daemons active (brc1rgw1, brc1rgw2) > > > > data: > > pools: 17 pools, 23552 pgs > > objects: 863.74M objects, 1.2 PiB > > usage: 2.4 PiB used, 6.2 PiB / 8.6 PiB avail > > pgs: 0.021% pgs not active > > 44798/8718528059 objects degraded (0.001%) > > 23546 active+clean > > 3 remapped+incomplete > > 2 incomplete > > 1 active+undersized+degraded > > > > io: > > client: 24 MiB/s rd, 3.2 KiB/s wr, 56 op/s rd, 4 op/s wr > > > > And the health detail shows as > > > > # ceph health detail > > HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 pools have many more > > objects per pg than average; Reduced data availability: 5 pgs inactive, 5 > > pgs incomplete; Degraded data redundancy: 44798/8718528081 objects > degraded > > (0.001%), 1 pg degraded, 1 pg undersized; 22726 pgs not deep-scrubbed in > > time; 23552 pgs not scrubbed in time; 77 slow ops, oldest one blocked for > > 56440 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops.; too > > many PGs per OSD (330 > max 250) > > OSDMAP_FLAGS noscrub,nodeep-scrub flag(s) set > > MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average > > pool iscsi-images objects per pg (540004) is more than 14.7248 times > > cluster average (36673) > > PG_AVAILABILITY Reduced data availability: 5 pgs inactive, 5 pgs > incomplete > > pg 15.3f3 is incomplete, acting > > [584,281,672,699,199,224,239,430,355,504,196] (reducing pool > > default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs > for > > 'incomplete') > > pg 15.985 is remapped+incomplete, acting > > > [2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352] > > (reducing pool default.rgw.buckets.data min_size from 9 may help; search > > ceph.com/docs for 'incomplete') > > pg 15.d46 is remapped+incomplete, acting > > [2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678] > > (reducing pool default.rgw.buckets.data min_size from 9 may help; search > > ceph.com/docs for 'incomplete') > > pg 15.251e is incomplete, acting > > [151,464,146,503,166,41,555,542,9,565,268] (reducing pool > > default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs > for > > 'incomplete') > > pg 15.39d3 is remapped+incomplete, acting > > [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] > > (reducing pool default.rgw.buckets.data min_size from 9 may help; search > > ceph.com/docs for 'incomplete') > > PG_DEGRADED Degraded data redundancy: 44798/8718528081 objects degraded > > (0.001%), 1 pg degraded, 1 pg undersized > > pg 15.28f0 is stuck undersized for 67359238.592403, current state > > active+undersized+degraded, last acting > > [2147483647,343,355,415,426,640,302,392,78,202,607] > > PG_NOT_DEEP_SCRUBBED 22726 pgs not deep-scrubbed in time > > > > We've the pools as below > > > > # ceph osd lspools > > 1 iscsi-images > > 2 cephfs_data > > 3 cephfs_metadata > > 4 .rgw.root > > 5 default.rgw.control > > 6 default.rgw.meta > > 7 default.rgw.log > > 8 default.rgw.buckets.index > > 13 rbd > > 15 default.rgw.buckets.data > > 16 default.rgw.buckets.non-ec > > 19 cephfs_data-ec > > 22 rbd-ec > > 23 iscsi-images-ec > > 24 hpecpool > > 25 hpec.rgw.buckets.index > > 26 hpec.rgw.buckets.non-ec > > > > > > We've been struggling for a long time to fix this but out of luck! Our > RGW > > daemons hosted on dedicated machines are continuously failing to respond, > > being behind a load balancer, LB throws 504 Gateway Timeout as the > daemons > > are failing to respond in the expected time. We perform active health > > checks from the LB on '/' by HTTP HEAD but these are failing as well, > very > > frequently. Currently we're surviving by writing a script that restarts > RGW > > daemons whenever the LB responds with HTTP status code 504. Any help is > > highly appreciated! > > > > Regards, > > Jayanth Reddy > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

OSD stuck down
by Nicola Mori 16 Jun '23

16 Jun '23

Dear Ceph users, after a host reboot one of the OSDs is now stuck down (and out). I tried several times to restart it and even to reboot the host, but it still remains down. # ceph -s cluster: id: b1029256-7bb3-11ec-a8ce-ac1f6b627b45 health: HEALTH_WARN 4 OSD(s) have spurious read errors (muted: OSD_SLOW_PING_TIME_BACK OSD_SLOW_PING_TIME_FRONT) services: mon: 5 daemons, quorum bofur,balin,aka,romolo,dwalin (age 16h) mgr: bofur.tklnrn(active, since 16h), standbys: aka.wzystq, balin.hvunfe mds: 2/2 daemons up, 1 standby osd: 104 osds: 103 up (since 16h), 103 in (since 13h); 4 remapped pgs data: volumes: 1/1 healthy pools: 3 pools, 529 pgs objects: 18.85M objects, 41 TiB usage: 56 TiB used, 139 TiB / 195 TiB avail pgs: 68130/150150628 objects misplaced (0.045%) 522 active+clean 4 active+remapped+backfilling 3 active+clean+scrubbing+deep io: recovery: 46 MiB/s, 21 objects/s The host is reachable (its other OSDs are in) and from the systemd logs of the OSD I don't see anything wrong: $ sudo systemctl status ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)osd.34 ● ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)osd.34.service - Ceph osd.34 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 Loaded: loaded (/etc/systemd/system/ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45@.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2023-06-12 17:00:25 CEST; 15h ago Main PID: 36286 (bash) Tasks: 11 (limit: 152154) Memory: 20.0M CGroup: /system.slice/system-ceph\x2db1029256\x2d7bb3\x2d11ec\x2da8ce\x2dac1f6b627b45.slice/ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)osd.34.service ├─36286 /bin/bash /var/lib/ceph/b1029256-7bb3-11ec-a8ce-ac1f6b627b45/osd.34/unit.run └─36657 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45-osd-34 --pids-limit=0 -e CONTAINER_IMAGE=snack14/ceph-wizard@sha> Jun 12 17:00:25 balin systemd[1]: Started Ceph osd.34 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45. Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-34 Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-34 --no-mon-config --dev /dev/mapper/ceph--9a4c3927--d3da--4b49--80fe--6cdc00c7897c-osd--block--36d2f793--e5c7--4247--a314--bcc40389d50d Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/ceph--9a4c3927--d3da--4b49--80fe--6cdc00c7897c-osd--block--36d2f793--e5c7--4247--a314--bcc40389d50d Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-6 Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/ln -s /dev/mapper/ceph--9a4c3927--d3da--4b49--80fe--6cdc00c7897c-osd--block--36d2f793--e5c7--4247--a314--bcc40389d50d /var/lib/ceph/osd/ceph-34/block Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-34 Jun 12 17:00:27 balin bash[36306]: --> ceph-volume raw activate successful for osd ID: 34 Jun 12 17:00:29 balin bash[36657]: debug 2023-06-12T15:00:29.066+0000 7f818e356540 -1 Falling back to public interface I'd need some help to understand how to fix this. Thank you, Nicola

5 9

Improving write performance on ceph 17.6.2 HDDs + DB/WAL storage on nvme
by alexey.blinkov＠gmail.com 16 Jun '23

16 Jun '23

Greetings. My cluster consists of 3 nodes. Each node has 4 OSD HDDs with a capacity of 6 TB each and 1 nvme for db/wal storage. 2 * 10Gbps network assembled in bond and some parameters changed in rc.local to improve performance. They are below: #Set network interface buffer size ethtool -G eno1 rx 4096 tx 4096; ethtool -G eno2 rx 4096 tx 4096 #Set txqueuelen eno1, eno2, bond0, vmbr0 ip link set eno1 txqueuelen 10000 ip link set eno2 txqueuelen 10000 ip link set bond0 txqueuelen 20000 ip link set vmbr0 txqueuelen 20000 ip link set vmbr0.4040 txqueuelen 10000 ip link set vmbr0.4043 txqueuelen 10000 ip link set vmbr0.4045 txqueuelen 10000 ip link set vmbr0.4053 txqueuelen 10000 #Disable Offload ethtool -K eno1 gso off gro off lro off tso off ethtool -K eno2 gso off gro off lro off tso off ethtool -K eno1 rx on tx on sg on ethtool -K eno2 rx on tx on sg on ethtool -L eno1 combined 8 ethtool -L eno2 combined 8 ethtool -K eno1 rxhash on ethtool -K eno2 rxhash on rss-ladder eno1 0 rss-ladder eno2 1 With the default settings of ceph.conf, when testing a 1 GB file with the SEQ1MQ8T1 profile inside the kvm virtual machine, I get the following write throughput: 152 MB/s The bandwidth of each of the disks, if you do not use CEPH, is at least 200 MB/s, and I have 4 of them per node. In total, this is 800 MB/s. If we discard the overhead and abstraction levels, a loss of 25% of performance is possible, this should be +- 600 MB/s I may be wrong, but I'm purely theoretically guessing. When testing and monitoring disk load with the nmon utility, I see write rates of 30 to 50 MB/s per node disk. The network cannot be a bottleneck, since the nodes are empty and there is only 1 virtual machine on them that generates the load. iperf3 from one node to another passes at least 8 Gbit/s, which in the test will completely allow to drive 4.8 Gbit/s equal to 600 MB/s Write caching in the virtual machine is disabled. I started changing the [osd] parameters in the ceph.conf file with these changes, I managed to increase the read speed by 2 times from 195 MB/s to 550 MB/s, but the write speed did not budge. The conversation now is not about IOPS, but about the write speed. What needs to be changed? Below is my ceph.conf [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster_network = 10.50.251.0/24 fsid = 7d00b675-2f1e-47ff-a71c-b95d1745bc39 mon_allow_pool_delete = true mon_host = 10.50.250.1 10.50.250.2 10.50.250.3 ms_bind_ipv4 = true ms_bind_ipv6 = false osd_pool_default_min_size = 2 osd_pool_default_size = 3 public_network = 10.50.250.0/24 [client] keyring = /etc/pve/priv/$cluster.$name.keyring #[osd] # bluestore_compression_mode = none # bluestore_cache_autotune = true # bluestore_cache_size = 3221225472 # bluestore_cache_kv_ratio = 0.2 # bluestore_cache_kv_max = 1610612736 # bluestore_min_alloc_size_hdd = 65536 # bluestore_max_alloc_size_hdd = 67108864 # bluestore_min_alloc_size_ssd = 16384 # bluestore_max_alloc_size_ssd = 33554432 # bluestore_throttle_bytes = 268435456 # bluestore_throttle_deferred_bytes = 1073741824 # bluestore_rocksdb_options = write_buffer_size=268435456;max_write_buffer_number=4 # osd_op_num_threads_per_shard = 4 # osd_client_message_cap = 1024 # osd_client_message_size_cap = 536870912 [mds] keyring = /var/lib/ceph/mds/ceph-$id/keyring [mds.nd01] host = nd01 mds_standby_for_name = pve [mds.nd02] host = nd02 mds_standby_for_name = pve [mds.nd03] host = nd03 mds_standby_for_name = pve [mon.nd01] public_addr = 10.50.250.1 [mon.nd02] public_addr = 10.50.250.2 [mon.nd03] public_addr = 10.50.250.3 Perhaps I do not quite understand the philosophy, logic and functionality of CEPH? I am aware that there may be competition for resources and so on. But I see a clear 50-50 write throughput split when another VM runs a similar test. And at a speed of 150 MB / s, it is very scary that at some point there will be competition between virtual machines and the write speed may drop to very low

1 0

ceph blocklist
by Budai Laszlo 16 Jun '23

16 Jun '23

Hello everyone, can someone explain, or direct me to some documentation that explains the role of the blocklists (former blacklist). What do they useful for? how do they work? Thank you, Laszlo

1 0

How to change RGW certificate in Cephadm?
by Kai Stian Olstad 16 Jun '23

16 Jun '23

When I enabled RGW in cephadm I used this spec file rgw.yml service_type: rgw service_id: pech placement: label: cog spec: ssl: true rgw_frontend_ssl_certificate: | -----BEGIN CERTIFICATE----- <snip /> -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- <snip /> -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- <snip /> -----END CERTIFICATE----- -----BEGIN RSA PRIVATE KEY----- <snip /> -----END RSA PRIVATE KEY----- And enabled it with ceph orch apply -i /etc/ceph/rgw.yml The certificate is about to expire so I would like to update it. I updated rgw.yml spec with the new certificate and run ceph orch apply -i /etc/ceph/rgw.yml But nothing happened, so I tried to redeploy one of them with ceph orch daemon redeploy rgw.pech.pech-mon-3.upnvrd It redeployed the RGW, but still uses the old certificate. ceph config-key list | grep rgw gives me two keys of interest mgr/cephadm/spec.rgw.pech and rgw/cert/rgw.pech The content of mgr/cephadm/spec.rgw.pech is the new spec file with the updated certificates, but the rgw/cert/rgw.pech only contains certificate and private key, but the certificate is the old ones about to expire. I have looked in the documentation and can't find how to update the certificate for RGW. Can anyone shed some light on how to replace the certificate? -- Kai Stian Olstad

2 5

RGW versioned bucket index issues
by Cory Snyder 16 Jun '23

16 Jun '23

Hi all, I wanted to call attention to some RGW issues that we've observed on a Pacific cluster over the past several weeks. The problems relate to versioned buckets and index entries that can be left behind after transactions complete abnormally. The scenario is multi-faceted and we're still investigating some of the details, but I wanted to provide a big-picture summary of what we've found so far. It looks like most of these issues should be reproducible on versions before and after Pacific as well. I'll enumerate the individual issues below: 1. PUT requests during reshard of versioned bucket fail with 404 and leave behind dark data Tracker: https://tracker.ceph.com/issues/61359 2. When bucket index ops are cancelled it can leave behind zombie index entries This one was merged a few months ago and did make the v16.2.13 release, but in our case we had billions of extra index entries by the time that we had upgraded to the patched version. Tracker: https://tracker.ceph.com/issues/58673 3. Issuing a delete for a key that already has a delete marker as the current version leaves behind index entries and OLH objects Note that the tracker's original description describes the problem a bit differently, but I've clarified the nature of the issue in a comment. Tracker: https://tracker.ceph.com/issues/59663 The extra index entries and OLH objects that are left behind due to these sorts of issues are obviously annoying in regards to the fact that they unnecessarily consume space, but we've found that they can also cause severe performance degradation for bucket listings, lifecycle processing, and other ops indirectly due to higher osd latencies. The reason for the performance impact is that bucket listing calls must repeatedly perform additional OSD ops until they find the requisite number of entries to return. The OSD cls method for bucket listing also does its own internal iteration for the same purpose. Since these entries are invalid, they are skipped. In the case that we observed, where some of our bucket indexes were filled with a sea of contiguous leftover entries, the process of continually iterating over and skipping invalid entries caused enormous read amplification. I believe that the following tracker is describing symptoms that are related to the same issue: https://tracker.ceph.com/issues/59164. Note that this can also cause LC processing to repeatedly fail in cases where there are enough contiguous invalid entries, since the OSD cls code eventually gives up and returns an error that isn't handled. The severity of these issues likely varies greatly based upon client behavior. If anyone has experienced similar problems, we'd love to hear about the nature of how they've manifested for you so that we can be more confident that we've plugged all of the holes. Thanks, Cory Snyder 11:11 Systems

3 5

degraded objects increasing
by Angelo Höngens 16 Jun '23

16 Jun '23

Hey guys, I'm trying to understand what is happening in my cluster, I see the number of degraded objects increasing, while all OSD's are still up and running. Can someone explain what's happening? I would expect the number of misplaced objects to increase when ceph's balancing algorithm decides blocks should be on different OSDs, but I would only expect the degraded objects to increase when OSD's die?? 6/15/23 11:31:25 AM[WRN]Health check update: Degraded data redundancy: 809035/219783315 objects degraded (0.368%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 6/15/23 11:31:20 AM[WRN]Health check update: Degraded data redundancy: 809044/219788544 objects degraded (0.368%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 6/15/23 11:31:15 AM[WRN]Health check update: Degraded data redundancy: 809044/219788616 objects degraded (0.368%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 6/15/23 11:31:10 AM[WRN]Health check update: Degraded data redundancy: 808944/219777540 objects degraded (0.368%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 6/15/23 11:31:05 AM[WRN]Health check update: Degraded data redundancy: 808944/219776271 objects degraded (0.368%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 6/15/23 11:31:00 AM[WRN]Health check update: Degraded data redundancy: 808821/219683475 objects degraded (0.368%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 6/15/23 11:30:55 AM[WRN]Health check update: Degraded data redundancy: 808740/219672240 objects degraded (0.368%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 6/15/23 11:30:50 AM[WRN]Health check update: Degraded data redundancy: 808667/219645417 objects degraded (0.368%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) health: health: HEALTH_WARN Degraded data redundancy: 810779/220602543 objects degraded (0.368%), 6 pgs degraded, 6 pgs undersized services: mon: 3 daemons, quorum ceph-mon01,ceph-mon02,ceph-mon03 (age 12h) mgr: ceph-mon01.vzbglj(active, since 12h), standbys: ceph-mon02.qtuntk mds: 1/1 daemons up, 2 standby osd: 118 osds: 118 up (since 12h), 118 in (since 23h); 510 remapped pgs data: volumes: 1/1 healthy pools: 5 pools, 1249 pgs objects: 44.55M objects, 60 TiB usage: 118 TiB used, 2.0 PiB / 2.1 PiB avail pgs: 810779/220602543 objects degraded (0.368%) 39586388/220602543 objects misplaced (17.945%) 739 active+clean 406 active+remapped+backfill_wait 98 active+remapped+backfilling 6 active+undersized+degraded+remapped+backfilling io: client: 585 KiB/s rd, 22 MiB/s wr, 357 op/s rd, 1.41k op/s wr recovery: 150 MiB/s, 82 objects/s Kind regards, Angelo Hongens

3 2

Restful API and Cephfs quota usage
by Sake 16 Jun '23

16 Jun '23

1 1

Unable to start MDS and access CephFS after upgrade to 17.2.6
by Ben Stöver 15 Jun '23

15 Jun '23

Hi everyone, as discussed on this list before, we had an issue upgrading the metadata servers while performing an upgrade from 17.2.5 to 17.2.6. (See also https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/U3VEPCXYDYO… .) We had to pause the upgrade leaving us in a state where the MDS were running on the old version, while most other cluster components were already running the newer version. In the meantime we were able solve this issue and also reduced the number of active MDS from 2 to 1. Today we resumed the upgrade and all components are on version 17.2.6 now. Right after the upgrade a file system inconsistency (backtrace) was found, which we solved by scrubbing. Shortly after users coming back on the cluster using the CephFS again, all MDS failed and cannot be started again. The first MDS gets stuck in state "replay (laggy)", while both standby MDS crash right away ("Caught signal (Segmentation fault)"). All MDS report multiple of the following problems in the log: - "replayed ESubtreeMap at xxx subtree root 0x1 not in cache" - "replayed ESubtreeMap at xxx subtree root 0x1 is not mine in cache (it's -2,-2)" As a result, we are currently not able to access our CephFS anymore. Does anyone have an idea how to solve this or what could be the cause? (I have found this ticket, which seems to be very similar but there does not seem to be a solution mentioned there. https://bugzilla.redhat.com/show_bug.cgi?id=2056935 ) Below are some parts of the MDS logs that seem relevant to us for this issue. We are thankful for any ideas. :-) Best Ben Log excerpt of Active MDS (replay): -140> 2023-06-14T07:51:59.585+0000 7feb588c0700 5 asok(0x55e56e23a000) register_command objecter_requests hook 0x55e56e1ca380 -139> 2023-06-14T07:51:59.585+0000 7feb588c0700 10 monclient: _renew_subs -138> 2023-06-14T07:51:59.585+0000 7feb588c0700 10 monclient: _send_mon_message to mon.ceph-service-01 at v2:131.220.126.65:3300/0 -137> 2023-06-14T07:51:59.585+0000 7feb588c0700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) -136> 2023-06-14T07:51:59.585+0000 7feb588c0700 4 mds.0.purge_queue operator(): data pool 3 not found in OSDMap -135> 2023-06-14T07:51:59.585+0000 7feb588c0700 4 mds.0.0 apply_blocklist: killed 0, blocklisted sessions (0 blocklist entries, 0) -134> 2023-06-14T07:51:59.585+0000 7feb588c0700 1 mds.0.172004 handle_mds_map i am now mds.0.172004 -133> 2023-06-14T07:51:59.585+0000 7feb588c0700 1 mds.0.172004 handle_mds_map state change up:standby --> up:replay -132> 2023-06-14T07:51:59.585+0000 7feb588c0700 5 mds.beacon.fdi-cephfs.ceph-service-01.gfwudy set_want_state: up:standby -> up:replay -131> 2023-06-14T07:51:59.585+0000 7feb588c0700 1 mds.0.172004 replay_start -130> 2023-06-14T07:51:59.585+0000 7feb588c0700 1 mds.0.172004 waiting for osdmap 341143 (which blocklists prior instance) -129> 2023-06-14T07:51:59.585+0000 7feb588c0700 10 monclient: _send_mon_message to mon.ceph-service-01 at v2:131.220.126.65:3300/0 -128> 2023-06-14T07:51:59.585+0000 7feb548b8700 2 mds.0.cache Memory usage: total 300004, rss 34196, heap 182556, baseline 182556, 0 / 0 inodes have caps, 0 caps, 0 caps per inode -127> 2023-06-14T07:51:59.603+0000 7feb588c0700 10 monclient: _renew_subs -126> 2023-06-14T07:51:59.603+0000 7feb588c0700 10 monclient: _send_mon_message to mon.ceph-service-01 at v2:131.220.126.65:3300/0 -125> 2023-06-14T07:51:59.603+0000 7feb588c0700 10 monclient: handle_get_version_reply finishing 1 version 341143 -124> 2023-06-14T07:51:59.603+0000 7feb528b4700 2 mds.0.172004 Booting: 0: opening inotable -123> 2023-06-14T07:51:59.603+0000 7feb528b4700 2 mds.0.172004 Booting: 0: opening sessionmap -122> 2023-06-14T07:51:59.603+0000 7feb528b4700 2 mds.0.172004 Booting: 0: opening mds log -121> 2023-06-14T07:51:59.603+0000 7feb528b4700 5 mds.0.log open discovering log bounds -120> 2023-06-14T07:51:59.603+0000 7feb528b4700 2 mds.0.172004 Booting: 0: opening purge queue (async) -119> 2023-06-14T07:51:59.603+0000 7feb528b4700 4 mds.0.purge_queue open: opening -118> 2023-06-14T07:51:59.603+0000 7feb528b4700 1 mds.0.journaler.pq(ro) recover start -117> 2023-06-14T07:51:59.603+0000 7feb528b4700 1 mds.0.journaler.pq(ro) read_head -116> 2023-06-14T07:51:59.603+0000 7feb520b3700 4 mds.0.journalpointer Reading journal pointer '400.00000000' -115> 2023-06-14T07:51:59.603+0000 7feb528b4700 2 mds.0.172004 Booting: 0: loading open file table (async) -114> 2023-06-14T07:51:59.604+0000 7feb528b4700 2 mds.0.172004 Booting: 0: opening snap table -113> 2023-06-14T07:51:59.604+0000 7feb5c0c7700 10 monclient: get_auth_request con 0x55e56ef7e000 auth_method 0 -112> 2023-06-14T07:51:59.604+0000 7feb5b8c6700 10 monclient: get_auth_request con 0x55e56ef7e800 auth_method 0 -111> 2023-06-14T07:51:59.604+0000 7feb5b0c5700 10 monclient: get_auth_request con 0x55e56ef7f000 auth_method 0 -110> 2023-06-14T07:51:59.604+0000 7feb5c0c7700 10 monclient: get_auth_request con 0x55e56ef92000 auth_method 0 -109> 2023-06-14T07:51:59.605+0000 7feb538b6700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 18366857216, expire 18367220777, write 18367220777, stream_format 1). probing for end of log (from 18367220777)... -108> 2023-06-14T07:51:59.605+0000 7feb538b6700 1 mds.0.journaler.pq(ro) probing for end of the log -107> 2023-06-14T07:51:59.605+0000 7feb520b3700 1 mds.0.journaler.mdlog(ro) recover start -106> 2023-06-14T07:51:59.605+0000 7feb520b3700 1 mds.0.journaler.mdlog(ro) read_head -105> 2023-06-14T07:51:59.605+0000 7feb520b3700 4 mds.0.log Waiting for journal 0x200 to recover... -104> 2023-06-14T07:51:59.605+0000 7feb5b8c6700 10 monclient: get_auth_request con 0x55e56ef7f400 auth_method 0 -103> 2023-06-14T07:51:59.605+0000 7feb5c0c7700 10 monclient: get_auth_request con 0x55e56ef92800 auth_method 0 -102> 2023-06-14T07:51:59.605+0000 7feb5b0c5700 10 monclient: get_auth_request con 0x55e56ef7f800 auth_method 0 -101> 2023-06-14T07:51:59.606+0000 7feb528b4700 1 mds.0.journaler.mdlog(ro) _finish_read_head loghead(trim 320109193199616, expire 320109196838948, write 320109612836052, stream_format 1). probing for end of log (from 320109612836052)... -100> 2023-06-14T07:51:59.606+0000 7feb528b4700 1 mds.0.journaler.mdlog(ro) probing for end of the log -99> 2023-06-14T07:51:59.606+0000 7feb5b0c5700 10 monclient: get_auth_request con 0x55e56f2ea800 auth_method 0 -98> 2023-06-14T07:51:59.606+0000 7feb5b8c6700 10 monclient: get_auth_request con 0x55e56f2ea000 auth_method 0 -97> 2023-06-14T07:51:59.606+0000 7feb538b6700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 18367220777 (header had 18367220777). recovered. -96> 2023-06-14T07:51:59.606+0000 7feb538b6700 4 mds.0.purge_queue operator(): open complete -95> 2023-06-14T07:51:59.606+0000 7feb538b6700 1 mds.0.journaler.pq(ro) set_writeable -94> 2023-06-14T07:51:59.607+0000 7feb528b4700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 320109613139120 (header had 320109612836052). recovered. -93> 2023-06-14T07:51:59.607+0000 7feb520b3700 4 mds.0.log Journal 0x200 recovered. -92> 2023-06-14T07:51:59.607+0000 7feb520b3700 4 mds.0.log Recovered journal 0x200 in format 1 -91> 2023-06-14T07:51:59.607+0000 7feb520b3700 2 mds.0.172004 Booting: 1: loading/discovering base inodes -90> 2023-06-14T07:51:59.607+0000 7feb520b3700 0 mds.0.cache creating system inode with ino:0x100 -89> 2023-06-14T07:51:59.607+0000 7feb520b3700 0 mds.0.cache creating system inode with ino:0x1 -88> 2023-06-14T07:51:59.608+0000 7feb5c0c7700 10 monclient: get_auth_request con 0x55e56f2eb400 auth_method 0 -87> 2023-06-14T07:51:59.615+0000 7feb528b4700 2 mds.0.172004 Booting: 2: replaying mds log -86> 2023-06-14T07:51:59.615+0000 7feb528b4700 2 mds.0.172004 Booting: 2: waiting for purge queue recovered -85> 2023-06-14T07:51:59.615+0000 7feb5b8c6700 10 monclient: get_auth_request con 0x55e56f1cec00 auth_method 0 -84> 2023-06-14T07:51:59.615+0000 7feb5b0c5700 10 monclient: get_auth_request con 0x55e56f2eb800 auth_method 0 -83> 2023-06-14T07:51:59.615+0000 7feb5c0c7700 10 monclient: get_auth_request con 0x55e56f1ce400 auth_method 0 -82> 2023-06-14T07:51:59.615+0000 7feb5b8c6700 10 monclient: get_auth_request con 0x55e56ef7e400 auth_method 0 -81> 2023-06-14T07:51:59.615+0000 7feb5b0c5700 10 monclient: get_auth_request con 0x55e56f1cf400 auth_method 0 -80> 2023-06-14T07:51:59.634+0000 7feb510b1700 -1 log_channel(cluster) log [ERR] : replayed ESubtreeMap at 320109197452549 subtree root 0x1 not in cache -79> 2023-06-14T07:51:59.634+0000 7feb510b1700 0 mds.0.journal journal subtrees: {0x1=[],0x100=[]} -78> 2023-06-14T07:51:59.634+0000 7feb510b1700 0 mds.0.journal journal ambig_subtrees: -77> 2023-06-14T07:51:59.637+0000 7feb510b1700 -1 log_channel(cluster) log [ERR] : replayed ESubtreeMap at 320109198067908 subtree root 0x1 not in cache -76> 2023-06-14T07:51:59.637+0000 7feb510b1700 0 mds.0.journal journal subtrees: {0x1=[],0x100=[]} -75> 2023-06-14T07:51:59.637+0000 7feb510b1700 0 mds.0.journal journal ambig_subtrees: -74> 2023-06-14T07:51:59.640+0000 7feb510b1700 -1 log_channel(cluster) log [ERR] : replayed ESubtreeMap at 320109198682422 subtree root 0x1 not in cache -73> 2023-06-14T07:51:59.640+0000 7feb510b1700 0 mds.0.journal journal subtrees: {0x1=[],0x100=[]} Log excerpt of standby MDS: -23> 2023-06-14T07:48:56.590+0000 7f14fcb97700 -1 log_channel(cluster) log [ERR] : replayed ESubtreeMap at 320109205645870 subtree root 0x1 is not mine in cache (it's -2,-2) -22> 2023-06-14T07:48:56.590+0000 7f14fcb97700 0 mds.0.journal journal subtrees: {0x1=[],0x100=[]} -21> 2023-06-14T07:48:56.590+0000 7f14fcb97700 0 mds.0.journal journal ambig_subtrees: -20> 2023-06-14T07:48:56.592+0000 7f14fcb97700 -1 log_channel(cluster) log [ERR] : replayed ESubtreeMap at 320109206260384 subtree root 0x1 is not mine in cache (it's -2,-2) -19> 2023-06-14T07:48:56.592+0000 7f14fcb97700 0 mds.0.journal journal subtrees: {0x1=[],0x100=[]} -18> 2023-06-14T07:48:56.592+0000 7f14fcb97700 0 mds.0.journal journal ambig_subtrees: -17> 2023-06-14T07:48:56.595+0000 7f14fcb97700 -1 log_channel(cluster) log [ERR] : replayed ESubtreeMap at 320109206875743 subtree root 0x1 is not mine in cache (it's -2,-2) -16> 2023-06-14T07:48:56.595+0000 7f14fcb97700 0 mds.0.journal journal subtrees: {0x1=[],0x100=[]} -15> 2023-06-14T07:48:56.595+0000 7f14fcb97700 0 mds.0.journal journal ambig_subtrees: -14> 2023-06-14T07:48:56.598+0000 7f14fcb97700 -1 log_channel(cluster) log [ERR] : replayed ESubtreeMap at 320109207491102 subtree root 0x1 is not mine in cache (it's -2,-2) -13> 2023-06-14T07:48:56.598+0000 7f14fcb97700 0 mds.0.journal journal subtrees: {0x1=[],0x100=[]} -12> 2023-06-14T07:48:56.598+0000 7f14fcb97700 0 mds.0.journal journal ambig_subtrees: -11> 2023-06-14T07:48:56.601+0000 7f14fcb97700 -1 log_channel(cluster) log [ERR] : replayed ESubtreeMap at 320109208105616 subtree root 0x1 is not mine in cache (it's -2,-2) -10> 2023-06-14T07:48:56.601+0000 7f14fcb97700 0 mds.0.journal journal subtrees: {0x1=[],0x100=[]} -9> 2023-06-14T07:48:56.601+0000 7f14fcb97700 0 mds.0.journal journal ambig_subtrees: -8> 2023-06-14T07:48:56.638+0000 7f1507bad700 10 monclient: get_auth_request con 0x556935cb3400 auth_method 0 -7> 2023-06-14T07:48:56.677+0000 7f15073ac700 10 monclient: get_auth_request con 0x556935c92c00 auth_method 0 -6> 2023-06-14T07:48:56.687+0000 7f1506bab700 10 monclient: get_auth_request con 0x5569359f6400 auth_method 0 -5> 2023-06-14T07:48:56.967+0000 7f1507bad700 10 monclient: get_auth_request con 0x556935cb2800 auth_method 0 -4> 2023-06-14T07:48:57.008+0000 7f15073ac700 10 monclient: get_auth_request con 0x556935cb8400 auth_method 0 -3> 2023-06-14T07:48:57.159+0000 7f1506bab700 10 monclient: get_auth_request con 0x556935cb3800 auth_method 0 -2> 2023-06-14T07:48:57.169+0000 7f1507bad700 10 monclient: get_auth_request con 0x556935cb9000 auth_method 0 -1> 2023-06-14T07:48:57.197+0000 7f15073ac700 10 monclient: get_auth_request con 0x5569359f7400 auth_method 0 0> 2023-06-14T07:48:57.317+0000 7f14fcb97700 -1 *** Caught signal (Segmentation fault) ** in thread 7f14fcb97700 thread_name:md_log_replay

3 2

User + Dev Monthly Meeting Minutes 2023-06-15
by Laura Flores 15 Jun '23

15 Jun '23

Topics: Reef rc is out! https://www.spinics.net/lists/ceph-users/msg77659.html Minutes: [Andras Pataki] Wants to test out Reef as a fresh deploy and as an upgrade; current hardware issues but will update when ready. [Andras Pataki] Is concerned about some performance issues in pacific - Would recommend talking to Mark Nelson about your use case and how it would be affected by any pacific performance issues [Andras Pataki] What are the major changes in Reef? - Major changes in Reef include a RocksDB version upgrade and new RocksDB default tunings [Bryan Stillwell] Was there an improvement to the balancer? - Yes, the read balancer was added as an offline tool available in Reef. [Bryan Stillwell] Were there any improvements to MDS? - Would recommend asking Patrick Donnelly or Greg Farnum -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

RGW bucket not getting create for recreated user via ceph Dashboard
by Maaz Azmi 15 Jun '23

15 Jun '23

Dear Team, I am facing one issue, which could be a possible bug, but not able to find any solution. We are unable to create a bucket on ceph from ceph-dashboard. Bucket is getting created with a fresh/different name but once I am trying with a deleted bucket name then it is not getting created. I have tested on (Octopus and Pacific version). Steps to reproduce. a. Create user RGW user from ceph Dashboard. (user1) b. Create a bucket with the bucket owner as user1. (bucket1) c. Delete the bucket d Delete user. e. Create a user again with the same name (user1) f. Create a new bucket with the bucket owner as user1. (Bucket2) I get below error message: RGW REST API failed request with status code 403 (b'{"Code":"InvalidAccessKeyId","RequestId":"tx00000457ff5169168a9e3-00648afcd0' b'-fdd1c2-dev","HostId":"fdd1c2-dev-india"}') So it seems the ceph dashboard has stored the user (keys) (created in step a.) somewhere and when the same user is created it is not getting updated at ceph dashboard cache. For a temporary fix I have restarted the ceph mgr and then I was able to create a bucket with the same user name. Regards, Maaz Regards, Maaz

1 0

Bottleneck between loadbalancer and rgws
by Szabo, Istvan (Agoda) 15 Jun '23

15 Jun '23

Hi, I have a dedicated loadbalancer pairs separated on 2x baremetal servers and behind the haproxy balancers I have 3 mon/mgr/rgw nodes. Each rgw node has 2rgw on it so in the cluster altogether 6, (now I just added one more so currently 9). Today I see pretty high GET latency in the cluster (3-4s) and seems like the limitations are the gateways: https://i.ibb.co/ypXFL34/1.png In this netstat seems like maxed out the established connections around 2-3k. When I've added one more gateway it increased. Seems like the gateway node or the gateway instance has some limitation. What is the value which is around 1000,I haven't really found it and affect GET and limit the connections on linux? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 3

[rgw] Multi-zone per storage cluster
by Yixin Jin 14 Jun '23

14 Jun '23

Hi folks, For multi-site environment, it seems that the common practice is to have one rgw zone backed by one storage cluster. However, after gaining some experience with such a setup, it appears to me that it is possible to have multiple zones with the same storage cluster. The first evidence is that when configuring the multi-site setup, the default zone "default" is still there and it needs a separate step to remove. It seems to me that there are 2 zones co-existing. The second evidence is that I can do "radosgw-admin zone create" multiple times at the same cluster with different zone names and make them all join the same zonegroup, although only one of them is the default. So, is it true that by design, we can have multiple zones per cluster? If so, is it a by-product of the design or a feature? Are there any such use cases in production? Thanks a lot Yixin

1 0