May 2023 - ceph-users - lists.ceph.io

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

Help needed to configure erasure coding LRC plugin
by Michel Jouvin 16 Oct '24

16 Oct '24

Hi, As discussed in another thread (Crushmap rule for multi-datacenter erasure coding), I'm trying to create an EC pool spanning 3 datacenters (datacenters are present in the crushmap), with the objective to be resilient to 1 DC down, at least keeping the readonly access to the pool and if possible the read-write access, and have a storage efficiency better than 3 replica (let say a storage overhead <= 2). In the discussion, somebody mentioned LRC plugin as a possible jerasure alternative to implement this without tweaking the crushmap rule to implement the 2-step OSD allocation. I looked at the documentation (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/) but I have some questions if someone has experience/expertise with this LRC plugin. I tried to create a rule for using 5 OSDs per datacenter (15 in total), with 3 (9 in total) being data chunks and others being coding chunks. For this, based of my understanding of examples, I used k=9, m=3, l=4. Is it right? Is this configuration equivalent, in terms of redundancy, to a jerasure configuration with k=9, m=6? The resulting rule, which looks correct to me, is: -------- { "rule_id": 6, "rule_name": "test_lrc_2", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 15, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -4, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 5, "type": "host" }, { "op": "emit" } ] } ------------ Unfortunately, it doesn't work as expected: a pool created with this rule ends up with its pages active+undersize, which is unexpected for me. Looking at 'ceph health detail` output, I see for each page something like: pg 52.14 is stuck undersized for 27m, current state active+undersized, last acting [90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647] For each PG, there is 3 '2147483647' entries and I guess it is the reason of the problem. What are these entries about? Clearly it is not OSD entries... Looks like a negative number, -1, which in terms of crushmap ID is the crushmap root (named "default" in our configuration). Any trivial mistake I would have made? Thanks in advance for any help or for sharing any successful configuration? Best regards, Michel

4 24

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

Re: MDS crashes to damaged metadata
by Patrick Donnelly 10 Jun '24

10 Jun '24

On Thu, Dec 15, 2022 at 9:32 AM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): > > 2022-12-08T13:05:48.919+0100 7f440afec700 -1 /build/ceph-16.2.10/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f440afec700 time 2022-12-08T13:05:48.921223+0100 > /build/ceph-16.2.10/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE) > > 12 Minutes lates the unlink_local error crashes appeared again. This time with a new file. During debugging we noticed a MTU mismatch between MDS (1500) and client (9000) with cephfs kernel mount. The client is also creating the snapshots via mkdir in the .snap directory. > > We disabled snapshot creation for now, but really need this feature. I uploaded the mds logs of the first crash along with the information above to https://tracker.ceph.com/issues/38452 > > I would greatly appreciate it, if you could answer me the following question: > > Is the Bug related to our MTU Mismatch? We fixed the MTU Issue going back to 1500 on all nodes in the ceph public network on the weekend also. I doubt it. > If you need a debug level 20 log of the ScatterLock for further analysis, i could schedule snapshots at the end of our workdays and increase the debug level 5 Minutes arround snap shot creation. This would be very helpful! -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 4

Re: Recoveries without any misplaced objects?
by Hector Martin 24 Apr '24

24 Apr '24

On 29/05/2023 20.55, Anthony D'Atri wrote: > Check the uptime for the OSDs in question I restarted all my OSDs within the past 10 days or so. Maybe OSD restarts are somehow breaking these stats? > >> On May 29, 2023, at 6:44 AM, Hector Martin <marcan(a)marcan.st> wrote: >> >> Hi, >> >> I'm watching a cluster finish a bunch of backfilling, and I noticed that >> quite often PGs end up with zero misplaced objects, even though they are >> still backfilling. >> >> Right now the cluster is down to 6 backfilling PGs: >> >> data: >> volumes: 1/1 healthy >> pools: 6 pools, 268 pgs >> objects: 18.79M objects, 29 TiB >> usage: 49 TiB used, 25 TiB / 75 TiB avail >> pgs: 262 active+clean >> 6 active+remapped+backfilling >> >> But there are no misplaced objects, and the misplaced column in `ceph pg >> dump` is zero for all PGs. >> >> If I do a `ceph pg dump_json`, I can see `num_objects_recovered` >> increasing for these PGs... but the misplaced count is still 0. >> >> Is there something else that would cause recoveries/backfills other than >> misplaced objects? Or perhaps there is a bug somewhere causing the >> misplaced object count to be misreported as 0 sometimes? >> >> # ceph -v >> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy >> (stable) >> >> - Hector >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > - Hector

3 3

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

Why you might want packages not containers for Ceph deployments
by Matthew Vernon 25 Mar '24

25 Mar '24

Hi, In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0]. I'd like to suggest, briefly, that: * containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes. For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice. Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages): * you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice) We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part. Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems! I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder". I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :) Regards, and apologies for the wall of text, Matthew [0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

26 65

Upgrading from Pacific to Quincy fails with "Unexpected error"
by Reza Bakhshayeshi 13 Mar '24

13 Mar '24

Hi all, I have a problem regarding upgrading Ceph cluster from Pacific to Quincy version with cephadm. I have successfully upgraded the cluster to the latest Pacific (16.2.11). But when I run the following command to upgrade the cluster to 17.2.5, After upgrading 3/4 mgrs, the upgrade process stops with "Unexpected error". (everything is on a private network) ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v17.2.5 I also tried the 17.2.4 version. cephadm fails to check the hosts' status and marks them as offline: cephadm 2023-04-06T10:19:59.998510+0000 mgr.host9.arhpnd (mgr.4516356) 5782 : cephadm [DBG] host host4 (x.x.x.x) failed check cephadm 2023-04-06T10:19:59.998553+0000 mgr.host9.arhpnd (mgr.4516356) 5783 : cephadm [DBG] Host "host4" marked as offline. Skipping daemon refresh cephadm 2023-04-06T10:19:59.998581+0000 mgr.host9.arhpnd (mgr.4516356) 5784 : cephadm [DBG] Host "host4" marked as offline. Skipping gather facts refresh cephadm 2023-04-06T10:19:59.998609+0000 mgr.host9.arhpnd (mgr.4516356) 5785 : cephadm [DBG] Host "host4" marked as offline. Skipping network refresh cephadm 2023-04-06T10:19:59.998633+0000 mgr.host9.arhpnd (mgr.4516356) 5786 : cephadm [DBG] Host "host4" marked as offline. Skipping device refresh cephadm 2023-04-06T10:19:59.998659+0000 mgr.host9.arhpnd (mgr.4516356) 5787 : cephadm [DBG] Host "host4" marked as offline. Skipping osdspec preview refresh cephadm 2023-04-06T10:19:59.998682+0000 mgr.host9.arhpnd (mgr.4516356) 5788 : cephadm [DBG] Host "host4" marked as offline. Skipping autotune cluster 2023-04-06T10:20:00.000151+0000 mon.host8 (mon.0) 158587 : cluster [ERR] Health detail: HEALTH_ERR 9 hosts fail cephadm check; Upgrade: failed due to an unexpected exception cluster 2023-04-06T10:20:00.000191+0000 mon.host8 (mon.0) 158588 : cluster [ERR] [WRN] CEPHADM_HOST_CHECK_FAILED: 9 hosts fail cephadm check cluster 2023-04-06T10:20:00.000202+0000 mon.host8 (mon.0) 158589 : cluster [ERR] host host7 (x.x.x.x) failed check: Unable to reach remote host host7. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000213+0000 mon.host8 (mon.0) 158590 : cluster [ERR] host host2 (x.x.x.x) failed check: Unable to reach remote host host2. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000220+0000 mon.host8 (mon.0) 158591 : cluster [ERR] host host8 (x.x.x.x) failed check: Unable to reach remote host host8. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000228+0000 mon.host8 (mon.0) 158592 : cluster [ERR] host host4 (x.x.x.x) failed check: Unable to reach remote host host4. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000240+0000 mon.host8 (mon.0) 158593 : cluster [ERR] host host3 (x.x.x.x) failed check: Unable to reach remote host host3. Process exited with non-zero exit status 3 and here are some outputs of the commands: [root@host8 ~]# ceph -s cluster: id: xxx health: HEALTH_ERR 9 hosts fail cephadm check Upgrade: failed due to an unexpected exception services: mon: 5 daemons, quorum host8,host1,host7,host2,host9 (age 2w) mgr: host9.arhpnd(active, since 105m), standbys: host8.jowfih, host1.warjsr, host2.qyavjj mds: 1/1 daemons up, 3 standby osd: 37 osds: 37 up (since 8h), 37 in (since 3w) data: io: client: progress: Upgrade to 17.2.5 (0s) [............................] [root@host8 ~]# ceph orch upgrade status { "target_image": "my-private-repo/quay-io/ceph/ceph@sha256 :34c763383e3323c6bb35f3f2229af9f466518d9db926111277f5e27ed543c427", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "3/59 daemons upgraded", "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception", "is_paused": true } [root@host8 ~]# ceph cephadm check-host host7 check-host failed: Host 'host7' not found. Use 'ceph orch host ls' to see all managed hosts. [root@host8 ~]# ceph versions { "mon": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 1, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 }, "osd": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 37 }, "mds": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 4 }, "overall": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 47, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 } } The strange thing is I can rollback the cluster status by failing to not-upgraded mgr like this: ceph mgr fail ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v16.2.11 Would you happen to have any idea about this? Best regards, Reza

4 9

Re: increasing number of (deep) scrubs
by Frank Schilder 13 Dec '23

13 Dec '23

Hi Dan, thanks for your answer. I don't have a problem with increasing osd_max_scrubs (=1 at the moment) as such. I would simply prefer a somewhat finer grained way of controlling scrubbing than just doubling or tripling it right away. Some more info. These 2 pools are data pools for a large FS. Unfortunately, we have a large percentage of small files, which is a pain for recovery and seemingly also for deep scrubbing. Our OSDs are about 25% used and I had to increase the warning interval already to 2 weeks. With all the warning grace parameters this means that we manage to deep scrub everything about every month. I need to plan for 75% utilisation and a 3 months period is a bit far on the risky side. Our data is to a large percentage cold data. Client reads will not do the check for us, we need to combat bit-rot pro-actively. The reasons I'm interested in parameters initiating more scrubs while also converting more scrubs into deep scrubs are, that 1) scrubs seem to complete very fast. I almost never catch a PG in state "scrubbing", I usually only see "deep scrubbing". 2) I suspect the low deep-scrub count is due to a low number of deep-scrubs scheduled and not due to conflicting per-OSD deep scrub reservations. With the OSD count we have and the distribution over 12 servers I would expect at least a peak of 50% OSDs being active in scrubbing instead of the 25% peak I'm seeing now. It ought to be possible to schedule more PGs for deep scrub than actually are. 3) Every OSD having only 1 deep scrub active seems to have no measurable impact on user IO. If I could just get more PGs scheduled with 1 deep scrub per OSD it would already help a lot. Once this is working, I can eventually increase osd_max_scrubs when the OSDs fill up. For now I would just like that (deep) scrub scheduling looks a bit harder and schedules more eligible PGs per time unit. If we can get deep scrubbing up to an average of 42PGs completing per hour with keeping osd_max_scrubs=1 to maintain current IO impact, we should be able to complete a deep scrub with 75% full OSDs in about 30 days. This is the current tail-time with 25% utilisation. I believe currently a deep scrub of a PG in these pools takes 2-3 hours. Its just a gut feeling from some repair and deep-scrub commands, I would need to check logs for more precise info. Increasing osd_max_scrubs would then be a further and not the only option to push for more deep scrubbing. My expectation would be that values of 2-3 are fine due to the increasingly higher percentage of cold data for which no interference with client IO will happen. Hope that makes sense and there is a way beyond bumping osd_max_scrubs to increase the number of scheduled and executed deep scrubs. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dvanders(a)gmail.com> Sent: 05 January 2023 15:36 To: Frank Schilder Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] increasing number of (deep) scrubs Hi Frank, What is your current osd_max_scrubs, and why don't you want to increase it? With 8+2, 8+3 pools each scrub is occupying the scrub slot on 10 or 11 OSDs, so at a minimum it could take 3-4x the amount of time to scrub the data than if those were replicated pools. If you want the scrub to complete in time, you need to increase the amount of scrub slots accordingly. On the other hand, IMHO the 1-week deadline for deep scrubs is often much too ambitious for large clusters -- increasing the scrub intervals is one solution, or I find it simpler to increase mon_warn_pg_not_scrubbed_ratio and mon_warn_pg_not_deep_scrubbed_ratio until you find a ratio that works for your cluster. Of course, all of this can impact detection of bit-rot, which anyway can be covered by client reads if most data is accessed periodically. But if the cluster is mostly idle or objects are generally not read, then it would be preferable to increase slots osd_max_scrubs. Cheers, Dan On Tue, Jan 3, 2023 at 2:30 AM Frank Schilder <frans(a)dtu.dk> wrote: > > Hi all, > > we are using 16T and 18T spinning drives as OSDs and I'm observing that they are not scrubbed as often as I would like. It looks like too few scrubs are scheduled for these large OSDs. My estimate is as follows: we have 852 spinning OSDs backing a 8+2 pool with 2024 and an 8+3 pool with 8192 PGs. On average I see something like 10PGs of pool 1 and 12 PGs of pool 2 (deep) scrubbing. This amounts to only 232 out of 852 OSDs scrubbing and seems to be due to a conservative rate of (deep) scrubs being scheduled. The PGs (dep) scrub fairly quickly. > > I would like to increase gently the number of scrubs scheduled for these drives and *not* the number of scrubs per OSD. I'm looking at parameters like: > > osd_scrub_backoff_ratio > osd_deep_scrub_randomize_ratio > > I'm wondering if lowering osd_scrub_backoff_ratio to 0.5 and, maybe, increasing osd_deep_scrub_randomize_ratio to 0.2 would have the desired effect? Are there other parameters to look at that allow gradual changes in the number of scrubs going on? > > Thanks a lot for your help! > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 3

Re: RGW access logs with bucket name
by Boris Behrens 02 Nov '23

02 Nov '23

Bringing up that topic again: is it possible to log the bucket name in the rgw client logs? currently I am only to know the bucket name when someone access the bucket via https://TLD/bucket/object instead of https://bucket.TLD/object. Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens <bb(a)kervyn.de>: > Hi, > I am looking forward to move our logs from > /var/log/ceph/ceph-client...log to our logaggregator. > > Is there a way to have the bucket name in the log file? > > Or can I write the rgw_enable_ops_log into a file? Maybe I could work with > this. > > Cheers and happy new year > Boris > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

4 6

Re: Status of Quincy 17.2.5 ?
by Christian Rohmann 30 Oct '23

30 Oct '23

Hey everyone, On 20/10/2022 10:12, Christian Rohmann wrote: > 1) May I bring up again my remarks about the timing: > > On 19/10/2022 11:46, Christian Rohmann wrote: > >> I believe the upload of a new release to the repo prior to the >> announcement happens quite regularly - it might just be due to the >> technical process of releasing. >> But I agree it would be nice to have a more "bit flip" approach to >> new releases in the repo and not have the packages appear as updates >> prior to the announcement and final release and update notes. > By my observations sometimes there are packages available on the > download servers via the "last stable" folders such as > https://download.ceph.com/debian-quincy/ quite some time before the > announcement of a release is out. > I know it's hard to time this right with mirrors requiring some time > to sync files, but would be nice to not see the packages or have > people install them before there are the release notes and potential > pointers to changes out. Todays 16.2.11 release shows the exact issue I described above .... 1) 16.2.11 packages are already available via e.g. https://download.ceph.com/debian-pacific 2) release notes not yet merged: (https://github.com/ceph/ceph/pull/49839) thus https://ceph.io/en/news/blog/2022/v16-2-11-pacific-released/ show a 404 :-) 3) No announcement like https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/QOCU563UD3… to the ML yet. Regards Christian

2 2

[Pacific] ceph orch device ls do not returns any HDD
by Patrick Begou 27 Oct '23

27 Oct '23

Hi everyone I'm new to CEPH, just a french 4 days training session with Octopus on VMs that convince me to build my first cluster. At this time I have 4 old identical nodes for testing with 3 HDDs each, 2 network interfaces and running Alma Linux8 (el8). I try to replay the training session but it fails, breaking the web interface because of some problems with podman 4.2 not compatible with Octopus. So I try to deploy Pacific with cephadm tool on my first node (mostha1) (to enable testing also an upgrade later). dnf -y install https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noar… monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }') cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \ --initial-dashboard-user admceph \ --allow-fqdn-hostname --cluster-network 10.1.0.0/16 This was sucessfull. But running "*c**eph orch device ls*" do not show any HDD even if I have /dev/sda (used by the OS), /dev/sdb and /dev/sdc The web interface shows a row capacity which is an aggregate of the sizes of the 3 HDDs for the node. I've also tried to reset /dev/sdb but cephadm do not see it: [ceph: root@mostha1 /]# ceph orch device zap mostha1.legi.grenoble-inp.fr /dev/sdb --force Error EINVAL: Device path '/dev/sdb' not found on host 'mostha1.legi.grenoble-inp.fr' On my first attempt with octopus, I was able to list the available HDD with this command line. Before moving to Pacific, the OS on this node has been reinstalled from scratch. Any advices for a CEPH beginner ? Thanks Patrick

9 51

libceph: mds1 IP+PORT wrong peer at address
by Frank Schilder 20 Sep '23

20 Sep '23

Hi all, we seem to have hit a bug in the ceph fs kernel client and I just want to confirm what action to take. We get the error "wrong peer at address" in dmesg and some jobs on that server seem to get stuck in fs access; log extract below. I found these 2 tracker items https://tracker.ceph.com/issues/23883 and https://tracker.ceph.com/issues/41519, which don't seem to have fixes. My questions: - Is this harmless or does it indicate invalid/corrupted client cache entries? - How to resolve, ignore, umount+mount or reboot? Here an extract from the dmesg log, the error has survived a couple of MDS restarts already: [Mon Mar 6 12:56:46 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:05:18 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-1572619386 [Mon Mar 6 13:05:18 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:13:50 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-1572619386 [Mon Mar 6 13:13:50 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state OPEN) [Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state OPEN) [Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start [Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start [Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success [Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success [Mon Mar 6 13:18:13 2023] ceph: update_snap_trace error -22 [Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state OPEN) [Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state OPEN) [Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed [Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect success [Mon Mar 6 13:18:29 2023] ceph: mds7 reconnect success [Mon Mar 6 13:18:35 2023] ceph: update_snap_trace error -22 [Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed [Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed [Mon Mar 6 13:22:22 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Mon Mar 6 13:22:22 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:30:54 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [...] [Thu Mar 9 09:37:24 2023] slurm.epilog.cl (31457): drop_caches: 3 [Thu Mar 9 09:38:26 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:38:26 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 09:46:58 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:46:58 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 09:55:30 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:55:30 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 10:04:02 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 10:04:02 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 6

quincy 17.2.6 - write performance continuously slowing down until OSD restart needed
by Nikola Ciprich 22 Jul '23

22 Jul '23

Hello dear CEPH users and developers, we're dealing with strange problems.. we're having 12 node alma linux 9 cluster, initially installed CEPH 15.2.16, then upgraded to 17.2.5. It's running bunch of KVM virtual machines accessing volumes using RBD. everything is working well, but there is strange and for us quite serious issue - speed of write operations (both sequential and random) is constantly degrading drastically to almost unusable numbers (in ~1week it drops from ~70k 4k writes/s from 1 VM to ~7k writes/s) When I restart all OSD daemons, numbers immediately return to normal.. volumes are stored on replicated pool of 4 replicas, on top of 7*12 = 84 INTEL SSDPE2KX080T8 NVMEs. I've updated cluster to 17.2.6 some time ago, but the problem persists. This is especially annoying in connection with https://tracker.ceph.com/issues/56896 as restarting OSDs is quite painfull when half of them crash.. I don't see anything suspicious, nodes load is quite low, no logs errors, network latency and throughput is OK too Anyone having simimar issue? I'd like to ask for hints on what should I check further.. we're running lots of 14.2.x and 15.2.x clusters, none showing similar issue, so I'm suspecting this is something related to quincy thanks a lot in advance with best regards nikola ciprich -- ------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis(a)linuxbox.cz -------------------------------------

7 16

rgw multisite sync not syncing data, error: RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data log shards
by Christian Rohmann 20 Jul '23

20 Jul '23

Hey ceph-users, I setup a multisite sync between two freshly setup Octopus clusters. In the first cluster I created a bucket with some data just to test the replication of actual data later. I then followed the instructions on https://docs.ceph.com/en/octopus/radosgw/multisite/#migrating-a-single-site… to add a second zone. Things went well and both zones are now happily reaching each other and the API endpoints are talking. Also the metadata is in sync already - both sides are happy and I can see bucket listings and users are "in sync": > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > metadata sync no sync (zone is master) > data sync source: c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > and on the other side ... > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > metadata sync syncing > full sync: 0/64 shards > incremental sync: 64/64 shards > metadata is caught up with master > data sync source: 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > also the newly created buckets (read: their metadata) is synced. What is apparently not working in the sync of actual data. Upon startup the radosgw on the second site shows: > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: start > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: realm > epoch=2 period id=f4553d7c-5cc5-4759-9253-9a22b051e736 > 2021-06-25T16:15:11.525+0000 7fe71dff3700 0 > RGW-SYNC:data:sync:init_data_sync_status: ERROR: failed to read remote > data log shards > also when issuing # radosgw-admin data sync init --source-zone obst-rgn it throws > 2021-06-25T16:20:29.167+0000 7f87c2aec080 0 > RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data > log shards Does anybody have any hints on where to look for what could be broken here? Thanks a bunch, Regards Christian

3 4

Re: Ceph Mgr/Dashboard Python depedencies: a new approach
by Casey Bodley 14 Jul '23

14 Jul '23

hi Ernesto and lists, > [1] https://github.com/ceph/ceph/pull/47501 are we planning to backport this to quincy so we can support centos 9 there? enabling that upgrade path on centos 9 was one of the conditions for dropping centos 8 support in reef, which i'm still keen to do if not, can we find another resolution to https://tracker.ceph.com/issues/58832? as i understand it, all of those python packages exist in centos 8. do we know why they were dropped for centos 9? have we looked into making those available in epel? (cc Ken and Kaleb) On Fri, Sep 2, 2022 at 12:01 PM Ernesto Puerta <epuertat(a)redhat.com> wrote: > > Hi Kevin, > >> >> Isn't this one of the reasons containers were pushed, so that the packaging isn't as big a deal? > > > Yes, but the Ceph community has a strong commitment to provide distro packages for those users who are not interested in moving to containers. > >> Is it the continued push to support lots of distros without using containers that is the problem? > > > If not a problem, it definitely makes it more challenging. Compiled components often sort this out by statically linking deps whose packages are not widely available in distros. The approach we're proposing here would be the closest equivalent to static linking for interpreted code (bundling). > > Thanks for sharing your questions! > > Kind regards, > Ernesto > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io

4 10

cephadm upgrade 16.2.10 to 16.2.11: osds crash and get stuck restarting
by Zakhar Kirpichenko 14 Jul '23

14 Jul '23

Hi, Attempted to upgrade 16.2.10 to 16.2.11, 2 OSDs out of many started crashing in a loop on the very 1st host: Jan 25 23:07:53 ceph01 bash[2553123]: Uptime(secs): 0.0 total, 0.0 interval Jan 25 23:07:53 ceph01 bash[2553123]: Flush(GB): cumulative 0.000, interval 0.000 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(GB): cumulative 0.000, interval 0.000 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(Total Files): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(L0 Files): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(Keys): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Jan 25 23:07:53 ceph01 bash[2553123]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Jan 25 23:07:53 ceph01 bash[2553123]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count Jan 25 23:07:53 ceph01 bash[2553123]: ** File Read Latency Histogram By Level [P] ** Jan 25 23:07:53 ceph01 bash[2553123]: debug -10> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986439) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -9> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986493) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -8> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986500) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -7> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986505) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -6> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: [compaction/compaction_job.cc:1676] [O-2] [JOB 9] Compacting 4@0 + 2@1 files to L1, score 1.0 0 Jan 25 23:07:53 ceph01 bash[2553123]: debug -5> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: [compaction/compaction_job.cc:1680] [O-2] Compaction start summary: Base version 33 Base level 0, inputs: [649058(959KB) 649046(1510KB) 649024(1323KB) 649002(1396KB)], [648981(66MB) 648982(52MB)] Jan 25 23:07:53 ceph01 bash[2553123]: debug -4> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1674688072986547, "job": 9, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [649058, 649046, 649024, 649002], "files_L1": [648981, 648982], "score": 1, "input_data_size": 129161327} Jan 25 23:07:53 ceph01 bash[2553123]: debug -3> 2023-01-25T23:07:52.990+0000 7f7c8b66e700 -1 bdev(0x5619bf0ce400 /var/lib/ceph/osd/ceph-3/block) _aio_thread got r=-1 ((1) Operation not permitted) Jan 25 23:07:53 ceph01 bash[2553123]: debug -2> 2023-01-25T23:07:52.990+0000 7f7c8b66e700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.11/rpm/el8/BUILD/ceph-16.2.11/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f7c8b66e700 time 2023-01-25T23:07:52.993976+0000 Jan 25 23:07:53 ceph01 bash[2553123]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.11/rpm/el8/BUILD/ceph-16.2.11/src/blk/kernel/KernelDevice.cc: 604: ceph_abort_msg("Unexpected IO error. This may suggest HW issue. Please check your dmesg!") Jan 25 23:07:53 ceph01 bash[2553123]: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) Jan 25 23:07:53 ceph01 bash[2553123]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x5619b2fc1adc] Jan 25 23:07:53 ceph01 bash[2553123]: 2: (KernelDevice::_aio_thread()+0x1285) [0x5619b3b2a4e5] Jan 25 23:07:53 ceph01 bash[2553123]: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5619b3b357b1] Jan 25 23:07:53 ceph01 bash[2553123]: 4: /lib64/libpthread.so.0(+0x81ca) [0x7f7c978851ca] Jan 25 23:07:53 ceph01 bash[2553123]: 5: clone() Jan 25 23:07:53 ceph01 bash[2553123]: debug -1> 2023-01-25T23:07:52.994+0000 7f7c8b66e700 -1 *** Caught signal (Aborted) ** Jan 25 23:07:53 ceph01 bash[2553123]: in thread 7f7c8b66e700 thread_name:bstore_aio Jan 25 23:07:53 ceph01 bash[2553123]: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) Jan 25 23:07:53 ceph01 bash[2553123]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7c9788fcf0] Jan 25 23:07:53 ceph01 bash[2553123]: 2: gsignal() Jan 25 23:07:53 ceph01 bash[2553123]: 3: abort() Jan 25 23:07:53 ceph01 bash[2553123]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x5619b2fc1bad] Jan 25 23:07:53 ceph01 bash[2553123]: 5: (KernelDevice::_aio_thread()+0x1285) [0x5619b3b2a4e5] Jan 25 23:07:53 ceph01 bash[2553123]: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5619b3b357b1] Jan 25 23:07:53 ceph01 bash[2553123]: 7: /lib64/libpthread.so.0(+0x81ca) [0x7f7c978851ca] OSD kept crashing until the host reboot, an OSD restart wouldn't help. This hasn't happened during any previous upgrades, so was a rather unexpected development. Unclear what caused this, but a host reboot seems to have fixed it. It happened to 1 other OSD on another host, exactly the same symptoms, also solved by a reboot. Best regards, Zakhar

4 4

upgrading from 15.2.17 to 16.2.11 - Health ERROR
by xadhoom76＠gmail.com 04 Jul '23

04 Jul '23

hi , starting upgrade from 15.2.17 i got this error Module 'cephadm' has failed: Expecting value: line 1 column 1 (char 0) Cluster was in health ok before starting.

5 9

CephFS metadata pool grows by two orders of magnitude while trimming (?) snapshots
by Janek Bevendorff 19 Jun '23

19 Jun '23

Hi, Perhaps this is a known issue and I was simply too dumb to find it, but we are having problems with our CephFS metadata pool filling up over night. Our cluster has a small SSD pool of around 15TB which hosts our CephFS metadata pool. Usually, that's more than enough. The normal size of the pool ranges between 200 and 800GiB (which is quite a lot of fluctuation already). Yesterday, we had suddenly had the pool fill up entirely and they only way to fix it was to add more capacity. I increased the pool size to 18TB by adding more SSDs and could resolve the problem. After a couple of hours of reshuffling, the pool size finally went back to 230GiB. But then we had another fill-up tonight to 7.6TiB. Luckily, I had adjusted the weights so that not all disks could fill up entirely like last time, so it ended there. I wasn't really able to identify the problem yesterday, but under the more controllable scenario today, I could check the MDS logs at debug_mds=10 and to me it seems like the problem is caused by snapshot trimming. The logs contain a lot of snapshot-related messages for paths that haven't been touched in a long time. The messages all look something like this: May 31 09:16:48 XXX ceph-mds[2947525]: 2023-05-31T09:16:48.292+0200 7f7ce1bd9700 10 mds.1.cache.ino(0x1000b3c3670) add_client_cap first cap, joining realm snaprealm(0x10000000000 seq 1b1c lc 1b1b cr 1 b1b cps 2 snaps={185f=snap(185f 0x10000000000 'monthly_20221201' 2022-12-01T00:00:01.530830+0100),18de=snap(18de 0x10000000000 'monthly_20230101' 2023-01-01T00:00:04.657252+0100),1941=snap(1941 0x10000000000 ... May 31 09:25:03 XXX ceph-mds[3268481]: 2023-05-31T09:25:03.396+0200 7f0e6a6ca700 10 mds.0.cache | |______ 3 rep [dir 0x100000218fe.101111101* /storage/REDACTED/| ptrwaiter=0 request=0 child=0 frozen=0 subtree=1 replicated=0 dirty=0 waiter=0 authpin=0 tempexporting=0 0x5607759d9600] May 31 09:25:03 XXX ceph-mds[3268481]: 2023-05-31T09:25:03.452+0200 7f0e6a6ca700 10 mds.0.cache | | |____ 4 rep [dir 0x100000ff904.100111101010* /storage/REDACTED/| ptrwaiter=0 request=0 child=0 frozen=0 subtree=1 importing=0 replicated=0 waiter=0 authpin=0 tempexporting=0 0x56034ed25200] May 31 09:25:03 XXX ceph-mds[3268481]: 2023-05-31T09:25:03.716+0200 7f0e6becd700 10 mds.0.server set_trace_dist snaprealm snaprealm(0x10000000000 seq 1b1c lc 1b1b cr 1b1b cps 2 snaps={185f=snap(185f 0x10000000000 'monthly_20221201' 2022-12-01T00:00:01.530830+0100),18de=snap(18de 0x10000000000 'monthly_20230101' 2023-01-01T00:00:04.657252+0100),1941=snap(1941 0x10000000000 'monthly_20230201' 2023-02-01T00:00:01.854059+0100),19a6=snap(19a6 0x10000000000 'monthly_20230301' 2023-03-01T00:00:01.215197+0100),1a24=snap(1a24 0x10000000000 'monthly_20230401' ...) len=384 May 31 09:25:36 deltaweb055 ceph-mds[3268481]: 2023-05-31T09:25:36.076+0200 7f0e6becd700 10 mds.0.cache.ino(0x10004d74911) remove_client_cap last cap, leaving realm snaprealm(0x10000000000 seq 1b1c lc 1b1b cr 1b1b cps 2 snaps={185f=snap(185f 0x10000000000 'monthly_20221201' 2022-12-01T00:00:01.530830+0100),18de=snap(18de 0x10000000000 'monthly_20230101' ...) The daily_*, montly_* etc. names are the names of our regular snapshots. I posted a larger log file snippet using ceph-post-file with the ID: da0eb93d-f340-4457-8a3f-434e8ef37d36 Is it possible that the MDS are trimming old snapshots without taking care not to fill up the entire metadata pool? Cheers Janek

3 16

RGW versioned bucket index issues
by Cory Snyder 16 Jun '23

16 Jun '23

Hi all, I wanted to call attention to some RGW issues that we've observed on a Pacific cluster over the past several weeks. The problems relate to versioned buckets and index entries that can be left behind after transactions complete abnormally. The scenario is multi-faceted and we're still investigating some of the details, but I wanted to provide a big-picture summary of what we've found so far. It looks like most of these issues should be reproducible on versions before and after Pacific as well. I'll enumerate the individual issues below: 1. PUT requests during reshard of versioned bucket fail with 404 and leave behind dark data Tracker: https://tracker.ceph.com/issues/61359 2. When bucket index ops are cancelled it can leave behind zombie index entries This one was merged a few months ago and did make the v16.2.13 release, but in our case we had billions of extra index entries by the time that we had upgraded to the patched version. Tracker: https://tracker.ceph.com/issues/58673 3. Issuing a delete for a key that already has a delete marker as the current version leaves behind index entries and OLH objects Note that the tracker's original description describes the problem a bit differently, but I've clarified the nature of the issue in a comment. Tracker: https://tracker.ceph.com/issues/59663 The extra index entries and OLH objects that are left behind due to these sorts of issues are obviously annoying in regards to the fact that they unnecessarily consume space, but we've found that they can also cause severe performance degradation for bucket listings, lifecycle processing, and other ops indirectly due to higher osd latencies. The reason for the performance impact is that bucket listing calls must repeatedly perform additional OSD ops until they find the requisite number of entries to return. The OSD cls method for bucket listing also does its own internal iteration for the same purpose. Since these entries are invalid, they are skipped. In the case that we observed, where some of our bucket indexes were filled with a sea of contiguous leftover entries, the process of continually iterating over and skipping invalid entries caused enormous read amplification. I believe that the following tracker is describing symptoms that are related to the same issue: https://tracker.ceph.com/issues/59164. Note that this can also cause LC processing to repeatedly fail in cases where there are enough contiguous invalid entries, since the OSD cls code eventually gives up and returns an error that isn't handled. The severity of these issues likely varies greatly based upon client behavior. If anyone has experienced similar problems, we'd love to hear about the nature of how they've manifested for you so that we can be more confident that we've plugged all of the holes. Thanks, Cory Snyder 11:11 Systems

3 5

bucket notification retries
by Yuval Lifshitz 12 Jun '23

12 Jun '23

Dear Community, I would like to collect your feedback on this issue. This is a followup from a discussion that started in the RGW refactoring meeting on 31-May-23 (thanks @Krunal Chheda <kchheda3(a)bloomberg.net> for bringing up this topic!). Currently persistent notifications are retried indefinitely. The only limiting mechanism that exists is that all notifications to a specific topic are stored in one RADOS object (of size 128MB). Assuming notifications are ~1KB at most, this would give us at least 128K notifications that can wait in the queue. When the queue fills up (e.g. kafka broker is down for 20 minutes, we are sending ~100 notifications per second) we start sending "slow down" replies to the client, and in this case the S3 operation will not be performed. This means that, for example, an outage of the kafka system would eventually cause an outage of our service. Note that this may also be a result of a misconfiguration of the kafka broker, or decommissioning of a broker. To avoid that, we propose several options: * use a fifo instead of a queue. This would allow us to hold more than 128K messages - survive longer broker outages, and at a higher message rate. there should still probably be a limit set on the size of the fifo * define maximum number of retries allowed for a notification * define maximum time the notification may stay in the queue before it is removed We should probably start with these definitions done as topic attributes, reflecting our delivery guarantees for this specific destination. Will try to capture the results of the discussion in this tracker: https://tracker.ceph.com/issues/61532 Thanks, Yuval

2 3

ceph Pacific - MDS activity freezes when one the MDSs is restarted
by Emmanuel Jaep 09 Jun '23

09 Jun '23

Hi, we are currently running a ceph fs cluster at the following version: MDS version: ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) The cluster is composed of 7 active MDSs and 1 standby MDS: RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active icadmin012 Reqs: 73 /s 1938k 1880k 85.3k 92.8k 1 active icadmin008 Reqs: 206 /s 2375k 2375k 7081 171k 2 active icadmin007 Reqs: 91 /s 5709k 5256k 149k 299k 3 active icadmin014 Reqs: 93 /s 679k 664k 40.1k 216k 4 active icadmin013 Reqs: 86 /s 3585k 3569k 12.7k 197k 5 active icadmin011 Reqs: 72 /s 225k 221k 8611 164k 6 active icadmin015 Reqs: 87 /s 1682k 1610k 27.9k 274k POOL TYPE USED AVAIL cephfs_metadata metadata 8552G 22.3T cephfs_data data 226T 22.3T STANDBY MDS icadmin006 When I restart one of the active MDSs, the standby MDS becomes active and its state becomes "replay". So far, so good! However, only one of the other "active" MDSs seems to remain active. All activities drop from the other ones: RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active icadmin012 Reqs: 0 /s 1938k 1881k 85.3k 9720 1 active icadmin008 Reqs: 0 /s 2375k 2375k 7080 2505 2 active icadmin007 Reqs: 2 /s 5709k 5256k 149k 26.5k 3 active icadmin014 Reqs: 0 /s 679k 664k 40.1k 3259 4 replay icadmin006 801k 801k 1279 0 5 active icadmin011 Reqs: 0 /s 225k 221k 8611 9241 6 active icadmin015 Reqs: 0 /s 1682k 1610k 27.9k 34.8k POOL TYPE USED AVAIL cephfs_metadata metadata 8539G 22.8T cephfs_data data 225T 22.8T STANDBY MDS icadmin013 In effect, the cluster becomes almost unavailable until the newly promoted MDS finishes rejoining the cluster. Obviously, this defeats the purpose of having 7MDSs. Is this behavior? If not, what configuration items should I check to go back to "normal" operations? Best, Emmanuel

5 12

RadosGW S3 API Multi-Tenancy
by Brad House 08 Jun '23

08 Jun '23

I'm in the process of exploring if it is worthwhile to add RadosGW to our existing ceph cluster. We've had a few internal requests for exposing the S3 API for some of our business units, right now we just use the ceph cluster for VM disk image storage via RBD. Everything looks pretty straight forward until we hit multitenancy. The page on multi-tenancy doesn't dive into permission delegation: https://docs.ceph.com/en/quincy/radosgw/multitenancy/ The end goal I want is to be able to create a single user per tenant (Business Unit) which will act as their 'administrator', where they can then do basically whatever they want under their tenant sandbox (though I don't think we need more advanced cases like creations of roles or policies, just create/delete their own users, buckets, objects). I was hopeful this would just work, and I asked on the ceph IRC channel on OFTC and was told once I grant a user caps="users=*", they would then be allowed to create users *outside* of their own tenant using the Rados Admin API and that I should explore IAM roles. I think it would make sense to add a feature, such as a flag that can be set on a user, to ensure they stay in their "sandbox". I'd assume this is probably a common use-case. Anyhow, if its possible to do today using iam roles/policies, then great, unfortunately this is my first time looking at this stuff and there are some things not immediately obvious. I saw this online about AWS itself and creating a permissions boundary, but that's for allowing creation of roles within a boundary: https://www.qloudx.com/delegate-aws-iam-user-and-role-creation-without-givi… I'm not sure what "Action" is associated with the Rados Admin API create user for applying a boundary that the user can only create users with the same tenant name. https://docs.ceph.com/en/quincy/radosgw/adminops/#create-user Any guidance on this would be extremely helpful. Thanks! -Brad

1 1

slow mds requests with random read test
by Ben 07 Jun '23

07 Jun '23

Hi, We are performing couple performance tests on CephFS using fio. fio is run in k8s pod and 3 pods will be up running mounting the same pvc to CephFS volume. Here is command line for random read: fio -direct=1 -iodepth=128 -rw=randread -ioengine=libaio -bs=4k -size=1G -numjobs=5 -runtime=500 -group_reporting -directory=/tmp/cache -name=Rand_Read_Testing_$BUILD_TIMESTAMP The random read is performed very slow. Here is the cluster log from dashboard: 5/30/23 8:13:16 PM [INF] Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 5/30/23 8:13:16 PM [INF] Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 5/30/23 8:13:16 PM [INF] MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 30 secs, oldest blocked for 33 secs 5/30/23 8:13:16 PM [INF] MDS health message cleared (mds.?): 1 slow requests are blocked > 30 secs 5/30/23 8:13:14 PM [WRN] Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 5/30/23 8:13:13 PM [INF] MDS health message cleared (mds.?): 1 slow requests are blocked > 30 secs 5/30/23 8:13:08 PM [WRN] Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 5/30/23 8:13:08 PM [WRN] Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 5/30/23 8:13:08 PM [WRN] slow request 34.213327 seconds old, received at 2023-05-30T12:12:33.951399+0000: client_request(client.270564:1406144 getattr pAsLsXsFs #0x700000103d0 2023-05-30T12:12:33.947323+0000 caller_uid=0, caller_gid=0{}) currently failed to rdlock, waiting 5/30/23 8:13:08 PM [WRN] 1 slow requests, 1 included below; oldest blocked for > 34.213328 secs 5/30/23 8:13:07 PM [WRN] slow request 33.169703 seconds old, received at 2023-05-30T12:12:33.952078+0000: peer_request:client.270564:1406144 currently dispatched 5/30/23 8:13:07 PM [WRN] 1 slow requests, 1 included below; oldest blocked for > 33.169704 secs 5/30/23 8:13:04 PM [INF] Cluster is now healthy 5/30/23 8:13:04 PM [INF] Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 5/30/23 8:13:04 PM [INF] Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 5/30/23 8:13:04 PM [INF] MDS health message cleared (mds.?): 9 slow metadata IOs are blocked > 30 secs, oldest blocked for 45 secs 5/30/23 8:13:04 PM [INF] MDS health message cleared (mds.?): 2 slow requests are blocked > 30 secs 5/30/23 8:12:57 PM [WRN] 2 slow requests, 0 included below; oldest blocked for > 44.954377 secs 5/30/23 8:12:52 PM [WRN] 2 slow requests, 0 included below; oldest blocked for > 39.954313 secs 5/30/23 8:12:48 PM [WRN] Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 5/30/23 8:12:47 PM [WRN] slow request 34.935921 seconds old, received at 2023-05-30T12:12:12.185614+0000: client_request(client.270564:1406139 create #0x7000001045b/atomic7966567911433736706tmp 2023-05-30T12:12:12.182999+0000 caller_uid=0, caller_gid=0{}) currently submit entry: journal_and_reply 5/30/23 8:12:47 PM [WRN] slow request 34.954254 seconds old, received at 2023-05-30T12:12:12.167281+0000: client_request(client.270564:1406138 rename #0x70000010457/build.xml #0x70000010457/atomic6590865221269854506tmp 2023-05-30T12:12:12.162999+0000 caller_uid=0, caller_gid=0{}) currently submit entry: journal_and_reply 5/30/23 8:12:47 PM [WRN] 2 slow requests, 2 included below; oldest blocked for > 34.954254 secs 5/30/23 8:12:44 PM [WRN] Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 5/30/23 8:12:41 PM [INF] Cluster is now healthy 5/30/23 8:12:41 PM [INF] Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 5/30/23 8:12:41 PM [INF] MDS health message cleared (mds.?): 1 slow requests are blocked > 30 secs 5/30/23 8:12:40 PM [INF] Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 5/30/23 8:12:40 PM [INF] MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 30 secs, oldest blocked for 38 secs However, random write test is performing very good. Any suggestions on the problem? Thanks, Ben

2 3

Orchestration seems not to work
by Thomas Widhalm 07 Jun '23

07 Jun '23

Hi, I'm in the process of upgrading my cluster from 17.2.5 to 17.2.6 but the following problem existed when I was still everywhere on 17.2.5 . I had a major issue in my cluster which could be solved with a lot of your help and even more trial and error. Right now it seems that most is already fixed but I can't rule out that there's still some problem hidden. The very issue I'm asking about started during the repair. When I want to orchestrate the cluster, it logs the command but it doesn't do anything. No matter if I use ceph dashboard or "ceph orch" in "cephadm shell". I don't get any error message when I try to deploy new services, redeploy them etc. The log only says "scheduled" and that's it. Same when I change placement rules. Usually I use tags. But since they don't work anymore, too, I tried host and umanaged. No success. The only way I can actually start and stop containers is via systemctl from the host itself. When I run "ceph orch ls" or "ceph orch ps" I see services I deployed for testing being deleted (for weeks now). Ans especially a lot of old MDS are listed as "error" or "starting". The list doesn't match reality at all because I had to start them by hand. I tried "ceph mgr fail" and even a complete shutdown of the whole cluster with all nodes including all mgs, mds even osd - everything during a maintenance window. Didn't change anything. Could you help me? To be honest I'm still rather new to Ceph and since I didn't find anything in the logs that caught my eye I would be thankful for hints how to debug. Cheers, Thomas -- http://www.widhalm.or.at GnuPG : 6265BAE6 , A84CB603 Threema: H7AV7D33 Telegram, Signal: widhalmt(a)widhalm.or.at

3 19

Encryption per user Howto
by huxiaoyu＠horebdata.cn 07 Jun '23

07 Jun '23

Dear Ceph folks, Recently one of our clients approached us with a request on encrpytion per user, i.e. using individual encrytion key for each user and encryption files and object store. Does anyone know (or have experience) how to do with CephFS and Ceph RGW? Any suggestionns or comments are highly appreciated, best regards, Samuel huxiaoyu(a)horebdata.cn

8 27

reef v18.1.0 QE Validation status
by Yuri Weinstein 07 Jun '23

07 Jun '23

Details of this release are summarized here: https://tracker.ceph.com/issues/61515#note-1 Release Notes - TBD Seeking approvals/reviews for: rados - Neha, Radek, Travis, Ernesto, Adam King (we still have to merge https://github.com/ceph/ceph/pull/51788 for the core) rgw - Casey fs - Venky orch - Adam King rbd - Ilya krbd - Ilya upgrade/octopus-x - deprecated upgrade/pacific-x - known issues, Ilya, Laura? upgrade/reef-p2p - N/A clients upgrades - not run yet powercycle - Brad ceph-volume - in progress Please reply to this email with approval and/or trackers of known issues/PRs to address them. gibba upgrade was done and will need to be done again this week. LRC upgrade TBD TIA

7 9

Unexpected behavior of directory mtime after being set explicitly
by Sandip Divekar 06 Jun '23

06 Jun '23

Hi Team, I'm writing to bring to your attention an issue we have encountered with the "mtime" (modification time) behavior for directories in the Ceph filesystem. Upon observation, we have noticed that when the mtime of a directory (let's say: dir1) is explicitly changed in CephFS, subsequent additions of files or directories within 'dir1' fail to update the directory's mtime as expected. This behavior appears to be specific to CephFS - we have reproduced this issue on both Quincy and Pacific. Similar steps work as expected in the ext4 filesystem amongst others. Reproduction steps: 1. Create a directory - mkdir dir1 2. Modify mtime using the touch command - touch dir1 3. Create a file or directory inside of 'dir1' - mkdir dir1/dir2 Expected result: mtime for dir1 should change to the time the file or directory was created in step 3 Actual result: there was no change to the mtime for 'dir1' Note : For more detail, kindly find the attached logs. Our queries are : 1. Is this expected behavior for CephFS? 2. If so, can you explain why the directory behavior is inconsistent depending on whether the mtime for the directory has previously been manually updated. Best Regards, Sandip Divekar Component QA Lead SDET.

7 14

Re: CEPH Version choice
by Frank Schilder 02 Jun '23

02 Jun '23

Hi Marc, I uploaded all scripts and a rudimentary readme to https://github.com/frans42/cephfs-bench . I hope it is sufficient to get started. I'm afraid its very much tailored to our deployment and I can't make it fully configurable anytime soon. I hope it serves a purpose though - at least I discovered a few bugs with it. We actually kept the benchmark running through an upgrade from mimic to octopus. Was quite interesting to see how certain performance properties change with that. This benchmark makes it possible to compare versions with live timings coming in. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Marc <Marc(a)f1-outsourcing.eu> Sent: Monday, May 15, 2023 11:28 PM To: Frank Schilder Subject: RE: [ceph-users] Re: CEPH Version choice > I planned to put it on-line. The hold-back is that the main test is un- > taring a nasty archive and this archive might contain personal > information, so I can't just upload it as is. I can try to put together > a similar archive from public sources. Please give me a bit of time. I'm > also a bit under stress right now with our users being hit by an FS meta > data corruption. That's also why I'm a bit trigger happy. > Ok thanks, very nice, no hurry!!!

2 2

NFS export of 2 disjoint sub-dir mounts
by Frank Schilder 02 Jun '23

02 Jun '23

Hi all, I have a problem with exporting 2 different sub-folder ceph-fs kernel mounts via nfsd to the same IP address. The top-level structure on the ceph fs is something like /A/S1 and /A/S2. On a file server I mount /A/S1 and /A/S2 as two different file systems under /mnt/S1 and /mnt/S2 using the ceph fs kernel client. Then, these 2 mounts are exported with lines like these in /etc/exports: /mnt/S1 -options NET /mnt/S2 -options IP IP is an element of NET, meaning that the host at IP should be the only host being able to access /mnt/S1 and /mnt/S2. What we observe is that any attempt to mount the export /mnt/S1 on the host at IP results in /mnt/S2 being mounted instead. My first guess was that here we have a clash of fsids and the ceph fs is simply reporting the same fsid to nfsd and, hence, nfsd thinks both mountpoints contain the same. So I modified the second export line to /mnt/S2 -options,fsid=100 IP to no avail. The two folders are completely disjoint, neither symlinks nor hard-links between them. So it should be safe to export these as 2 different file systems. Exporting such constructs to non-overlapping networks/IPs works as expected - even when exporting subdirs of a dir (like exporting /A/B and /A/B/C from the same file server to strictly different IPs). It seems the same-IP config that breaks expectations. Am I missing here a magic -yes-i-really-know-what-i-am-doing hack? The file server is on AlmaLinux release 8.7 (Stone Smilodon) and all ceph packages match the ceph version octopus latest of our cluster. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 1

PGs incomplete - Data loss
by Benno Wulf 01 Jun '23

01 Jun '23

Hi guys, I'm awake since 36h and try to restore a broken ceph Pool (2 PGs incomplete) My vm are all broken. Some Boot, some Dont Boot... Also I have 5 removed disk with Data of that Pool "in my Hands" - Dont ask... So my question is it possible to restore Data of these other disks and "add" them thee others for healing? Best regards Ben

2 1

cephadm does not honor container_image default value
by Daniel Krambrock 01 Jun '23

01 Jun '23

Hello. I think i found a bug in cephadm/ceph orch: Redeploying a container image (tested with alertmanager) after removing a custom `mgr/cephadm/container_image_alertmanager` value, deploys the previous container image and not the default container image. I'm running `cephadm` from ubuntu 22.04 pkg 17.2.5-0ubuntu0.22.04.3 and `ceph` version 17.2.6. Here is an example. Node clrz20-08 is the node altermanager is running on, clrz20-01 the node I'm controlling ceph from: * Get alertmanager version ``` root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name == "alertmanager")| .container_image_name' "quay.io/prometheus/alertmanager:v0.23.0" ``` * Set alertmanager image ``` root@clrz20-01:~# ceph config set mgr mgr/cephadm/container_image_alertmanager quay.io/prometheus/alertmanager root@clrz20-01:~# ceph config get mgr mgr/cephadm/container_image_alertmanager quay.io/prometheus/alertmanager ``` * redeploy altermanager ``` root@clrz20-01:~# ceph orch redeploy alertmanager Scheduled to redeploy alertmanager.clrz20-08 on host 'clrz20-08' ``` * Get alertmanager version ``` root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name == "alertmanager")| .container_image_name' "quay.io/prometheus/alertmanager:latest" ``` * Remove alertmanager image setting, revert to default: ``` root@clrz20-01:~# ceph config rm mgr mgr/cephadm/container_image_alertmanager root@clrz20-01:~# ceph config get mgr mgr/cephadm/container_image_alertmanager quay.io/prometheus/alertmanager:v0.23.0 ``` * redeploy altermanager ``` root@clrz20-01:~# ceph orch redeploy alertmanager Scheduled to redeploy alertmanager.clrz20-08 on host 'clrz20-08' ``` * Get alertmanager version ``` root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name == "alertmanager")| .container_image_name' "quay.io/prometheus/alertmanager:latest" ``` -> `mgr/cephadm/container_image_alertmanager` is set to `quay.io/prometheus/alertmanager:v0.23.0`, but redeploy uses `quay.io/prometheus/alertmanager:latest`. This looks like a bug. * Set alertmanager image explicitly to the default value ``` root@clrz20-01:~# ceph config set mgr mgr/cephadm/container_image_alertmanager quay.io/prometheus/alertmanager:v0.23.0 root@clrz20-01:~# ceph config get mgr mgr/cephadm/container_image_alertmanager quay.io/prometheus/alertmanager:v0.23.0 ``` * redeploy altermanager ``` root@clrz20-01:~# ceph orch redeploy alertmanager Scheduled to redeploy alertmanager.clrz20-08 on host 'clrz20-08' ``` * Get alertmanager version ``` root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name == "alertmanager")| .container_image_name' "quay.io/prometheus/alertmanager:v0.23.0" ``` -> Setting `mgr/cephadm/container_image_alertmanager` to the default setting fixes the issue. Bests, Daniel

3 3

Small RGW objects and RADOS 64KB minimun size
by Loïc Dachary 31 May '23

31 May '23

Bonjour, Reading Karan's blog post about benchmarking the insertion of billions objects to Ceph via S3 / RGW[0] from last year, it reads: > we decided to lower bluestore_min_alloc_size_hdd to 18KB and re-test. As represented in chart-5, the object creation rate found to be notably reduced after lowering the bluestore_min_alloc_size_hdd parameter from 64KB (default) to 18KB. As such, for objects larger than the bluestore_min_alloc_size_hdd , the default values seems to be optimal, smaller objects further require more investigation if you intended to reduce bluestore_min_alloc_size_hdd parameter. There also is a mail thread dated 2018 on this topic as well, with the same conclusion although using RADOS directly and not RGW[3]. I read the RGW data layout page in the documentation[1] and concluded that by default every object inserted with S3 / RGW will indeed use at least 64kb. A pull request from last year[2] seems to confirm it and also suggests modifying bluestore_min_alloc_size_hdd has adverse side effects. That being said, I'm curious to know if people developed strategies to cope with this overhead. Someone mentioned packing objects together client side to make them larger. But maybe there are simpler ways to do the same? Cheers [0] https://www.redhat.com/en/blog/scaling-ceph-billion-objects-and-beyond [1] https://docs.ceph.com/en/latest/radosgw/layout/ [2] https://github.com/ceph/ceph/pull/32809 [3] https://www.spinics.net/lists/ceph-users/msg45755.html -- Loïc Dachary, Artisan Logiciel Libre

4 6

BlueStore fragmentation woes
by Hector Martin 31 May '23

31 May '23

Hi, I've been seeing relatively large fragmentation numbers on all my OSDs: ceph daemon osd.13 bluestore allocator score block { "fragmentation_rating": 0.77251526920454427 } These aren't that old, as I recreated them all around July last year. They mostly hold CephFS data with erasure coding, with a mix of large and small files. The OSDs are at around 80%-85% utilization right now. Most of the data was written sequentially when the OSDs were created (I rsynced everything from a remote backup). Since then more data has been added, but not particularly quickly. At some point I noticed pathologically slow writes, and I couldn't figure out what was wrong. Eventually I did some block tracing and noticed the I/Os were very small, even though CephFS-side I was just writing one large file sequentially, and that's when I stumbled upon the free space fragmentation problem. Indeed, deleting some large files opened up some larger free extents and resolved the problem, but only until those get filled up and I'm back to fragmented tiny extents. So effectively I'm stuck at the current utilization, as trying to fill them up any more just slows down to an absolute crawl. I'm adding a few more OSDs and plan on doing the dance of removing one OSD at a time and replacing it with another one to hopefully improve the situation, but obviously this is going to take forever. Is there any plan for offering a defrag tool of some sort for bluestore? - Hector

7 27

[Pacific] Admin keys no longer works I get access denied URGENT!!!
by wodel youchi 31 May '23

31 May '23

Hi, After a wrong manipulation, the admin key no longer works, it seems it has been modified. My cluster is built using containers. When I execute ceph -s I get [root@controllera ceph]# ceph -s 2023-05-31T11:33:20.940+0100 7ff7b2d13700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] 2023-05-31T11:33:20.940+0100 7ff7b1d11700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] 2023-05-31T11:33:20.940+0100 7ff7b2512700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] [errno 13] RADOS permission denied (error connecting to the cluster) From the log file I am getting : May 31 11:03:02 controllera docker[214909]: debug 2023-05-31T11:03:02.714+0100 7fcfc0c91700 0 cephx server client.admin: unexpected key: req.key=5fea877f2a68548b expected_key=8c2074e03ffa449a How can I recover the correct key? Regards.

2 1

all buckets mtime = "0.000000" after upgrade to 17.2.6
by alyarb＠gmail.com 31 May '23

31 May '23

I was running on 17.2.5 since October, and just upgraded to 17.2.6, and now the "mtime" property on all my buckets is 0.000000. On all previous versions going back to Nautilus this wasn't an issue, and we do like to have that value present. radosgw-admin has no quick way to get the last object in the bucket. Here's my tracker submission: https://tracker.ceph.com/issues/61264#change-239348

2 1

MDS corrupt (also RADOS-level copy?)
by Jake Grimmett 31 May '23

31 May '23

Dear All, we are trying to recover from what we suspect is a corrupt MDS :( and have been following the guide here: <https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/> Symptoms: MDS SSD pool (2TB) filled completely over the weekend, normally uses less than 400GB, resulting in MDS crash. We added 4 x extra SSD to increase pool capacity to 3.5TB, however MDS did not recover # ceph fs status cephfs2 - 0 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed 1 resolve wilma-s3 8065 8063 8047 0 2 resolve wilma-s2 901k 802k 34.4k 0 POOL TYPE USED AVAIL mds_ssd metadata 2296G 3566G primary_fs_data data 0 3566G ec82pool data 2168T 3557T STANDBY MDS wilma-s1 wilma-s4 setting "ceph mds repaired 0" causes rank 0 to restart, and then immediately fail. Following the disaster-recovery-experts guide, the first step we did was to export the MDS journals, e.g: # cephfs-journal-tool --rank=cephfs2:0 journal export /root/backup.bin.0 journal is 9744716714163~658103700 wrote 658103700 bytes at offset 9744716714163 to /root/backup.bin.0 so far so good, however when we try to backup the final MDS the process consumes all available RAM (470GB) and needs to be killed after 14 minutes. # cephfs-journal-tool --rank=cephfs2:2 journal export /root/backup.bin.2 similarly, "recover_dentries summary" consumes all RAM when applied to MDS 2 # cephfs-journal-tool --rank=cephfs2:2 event recover_dentries summary We successfully ran "cephfs-journal-tool --rank=cephfs2:0 event recover_dentries summary" and "cephfs-journal-tool --rank=cephfs2:1 event recover_dentries summary" at this point, we tried to follow the instructions and make a RADOS level copy of the journal data, however the link in the docs doesn't explain how to do this and just points to <http://tracker.ceph.com/issues/9902> At this point we are tempted to reset the journal on MDS 2, but wanted to get a feeling from others about how dangerous this could be? We have a backup, but as there is 1.8PB of data, it's going to take a few weeks to restore.... any ideas gratefully received. Jake -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK.

2 3

how to use ctdb_mutex_ceph_rados_helper
by Angelo Höngens 31 May '23

31 May '23

Hey, I have a test setup with a 3-node samba cluster. This cluster consists of 3 vm's storing its locks on a replicated gluster volume. I want to switch to 2 physical smb-gateways for performance reasons (not enough money for 3), and since the 2-node cluster can't get quorum, I hope to switch to storing the ctdb lock in ceph and hope that will work reliably. (experiences with 2 node SMB clusters?) I am looking into the ctdb rados helper: [cluster] recovery lock = !/usr/lib/x86_64-linux-gnu/ctdb/ctdb_mutex_ceph_rados_helper ceph client.tenant1 cephfs_metadata ctdb_lock Now I do have a bit of experience with cephfs, rbd and rgw, but not rados. How do I give the user client.tenant1 permissions? We have a single cephfs, with 4 different tenants (departments). Each department has their own samba cluster. We're using cephfs permissions to limit the tenants to their own path (I hope). example of ceph auth: client.tenant1 key: ***** caps: [mds] allow rws fsname=cephfs path=/tenant1 caps: [mon] allow r fsname=cephfs caps: [osd] allow rw tag cephfs data=cephfs If I try some stuff manually (without really knowing how to specify objects or what that means), I get this permission denied error: root@tenant1-1:~# /usr/lib/x86_64-linux-gnu/ctdb/ctdb_mutex_ceph_rados_helper ceph client.tenant1 cephfs_metadata tenant1/ctdb_lock 1 /usr/lib/x86_64-linux-gnu/ctdb/ctdb_mutex_ceph_rados_helper: Failed to get lock on RADOS object 'tenant1/ctdb_lock' - (Operation not permitted) Angelo.

1 0

Important: RGW multisite bug may silently corrupt encrypted objects on replication
by Casey Bodley 31 May '23

31 May '23

Our downstream QE team recently observed an md5 mismatch of replicated objects when testing rgw's server-side encryption in multisite. This corruption is specific to s3 multipart uploads, and only affects the replicated copy - the original object remains intact. The bug likely affects Ceph releases all the way back to Luminous where server-side encryption was first introduced. To expand on the cause of this corruption: Encryption of multipart uploads requires special handling around the part boundaries, because each part is uploaded and encrypted separately. In multisite, objects are replicated in their encrypted form, and multipart uploads are replicated as a single part. As a result, the replicated copy loses its knowledge about the original part boundaries required to decrypt the data correctly. We don't have a fix yet, but we're tracking it in https://tracker.ceph.com/issues/46062. The fix will only modify the replication logic, so won't repair any objects that have already replicated incorrectly. We'll need to develop a radosgw-admin command to search for affected objects and reschedule their replication. In the meantime, I can only advise multisite users to avoid using encryption for multipart uploads. If you'd like to scan your cluster for existing encrypted multipart uploads, you can identify them with a s3 HeadObject request. The response would include a x-amz-server-side-encryption header, and the ETag header value (with "s removed) would be longer than 32 characters (multipart ETags are in the special form "<md5sum>-<num parts>"). Take care not to delete the corrupted replicas, because an active-active multisite configuration would go on to delete the original copy.

2 4

Custom CRUSH maps HOWTO?
by Thorne Lawler 31 May '23

31 May '23

Hi folks! I have a Ceph production 17.2.6 cluster with 6 machines in it - four newer, faster machines with 4x3.84TB NVME drives each, and two with 24x1.68TB SAS disks each. I know I should have done something smart with the CRUSH maps for this up front, but until now I have shied away from CRUSH maps as they sound really complex. Right now my cluster's performance, especially write performance, is not what it needs to be, and I am looking for advice: 1. How should I be structuring my crush map, and why? 2. How does one actually edit and manage a CRUSH map? What /commands/ does one use? This isn't clear at all in the documentation. Are there any GUI tools out there for managing CRUSH? 3. Is this going to impact production performance or availability while I'm configuring it? I have tens of thousands of users relying on this thing, so I can't take any risks. Thanks in advance! -- Regards, Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170 _DDNS /_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

3 3

Ceph client version vs server version inter-operability
by Mark Kirkwood 31 May '23

31 May '23

Hi, We are running a ceph cluster that is currently on Luminous. At this point most of our clients are also Luminous, but as we provision new client hosts we are using client versions that are more recent (e.g Octopus, Pacific and more recently Quincy). Is this safe? Is there a known list of what client versions are compatible with what server version? We are only using RBD and are specifying rbd_default_features (the same) on all server and client hosts. regards Mark

1 0

Seeking feedback on Improving cephadm bootstrap process
by Redouane Kachach 30 May '23

30 May '23

Dear ceph community, As you are aware, cephadm has become the default tool for installing Ceph on bare-metal systems. Currently, during the bootstrap process of a new cluster, if the user interrupts the process manually or if there are any issues causing the bootstrap process to fail, cephadm leaves behind the failed cluster files and processes on the current host. While this can be beneficial for debugging and resolving issues related to the cephadm bootstrap process, it can create difficulties for inexperienced users who need to delete the faulty cluster and proceed with the Ceph installation. The problem described in the tracker https://tracker.ceph.com/issues/57016 is a good example of this issue.In the cephadm development team, we are considering ways to enhance the user experience during the bootstrap of a new cluster. We have discussed the following options:1) Retain the cluster files without deleting them, but provide the user with a clear command to remove the broken/faulty cluster. 2) Automatically delete the broken/failed ceph installation and offer an option for the user to disable this behavior if desired.Both options have their advantages and disadvantages, which is why we are seeking your feedback. We would like to know which option you prefer and the reasoning behind your choice. Please provide reasonable arguments to justify your preference.Your feedback will be taken into careful consideration when we work on improving the ceph bootstrap process.Thank you, Redouane, On behalf of cephadm dev team.

6 8

RBD imagem mirroring doubt
by Work Ceph 30 May '23

30 May '23

Hello guys, What would happen if we set up an RBD mirroring configuration, and in the target system (the system where the RBD image is mirrored) we create snapshots of this image? Would that cause some problems? Also, what happens if we delete the source RBD image? Would that trigger a deletion in the target system RBD image as well? Thanks in advance!

1 0

Re: Newer linux kernel cephfs clients is more trouble?
by Dan van der Ster 30 May '23

30 May '23

Hi, Sorry for poking this old thread, but does this issue still persist in the 6.3 kernels? Cheers, Dan ______________________________ Clyso GmbH | https://www.clyso.com On Wed, Dec 7, 2022 at 3:42 AM William Edwards <wedwards(a)cyberfusion.nl> wrote: > > > > Op 7 dec. 2022 om 11:59 heeft Stefan Kooman <stefan(a)bit.nl> het volgende geschreven: > > > > On 5/13/22 09:38, Xiubo Li wrote: > >>> On 5/12/22 12:06 AM, Stefan Kooman wrote: > >>> Hi List, > >>> > >>> We have quite a few linux kernel clients for CephFS. One of our customers has been running mainline kernels (CentOS 7 elrepo) for the past two years. They started out with 3.x kernels (default CentOS 7), but upgraded to mainline when those kernels would frequently generate MDS warnings like "failing to respond to capability release". That worked fine until 5.14 kernel. 5.14 and up would use a lot of CPU and *way* more bandwidth on CephFS than older kernels (order of magnitude). After the MDS was upgraded from Nautilus to Octopus that behavior is gone (comparable CPU / bandwidth usage as older kernels). However, the newer kernels are now the ones that give "failing to respond to capability release", and worse, clients get evicted (unresponsive as far as the MDS is concerned). Even the latest 5.17 kernels have that. No difference is observed between using messenger v1 or v2. MDS version is 15.2.16. > >>> Surprisingly the latest stable kernels from CentOS 7 work flawlessly now. Although that is good news, newer operating systems come with newer kernels. > >>> > >>> Does anyone else observe the same behavior with newish kernel clients? > >> There have some known bugs, which have been fixed or under fixing recently, even in the mainline and, not sure whether are they related. Such as [1][2][3][4]. More detail please see ceph-client repo testing branch [5]. > > > > None of the issues you mentioned were related. We gained some more experience with newer kernel clients, specifically on Ubuntu Focal / Jammy (5.15). Performance issues seem to arise in certain workloads, specifically load-balanced Apache shared web hosting clusters with CephFS. We have tested linux kernel clients from 5.8 up to and including 6.0 with a production workload and the short summary is: > > > > < 5.13, everything works fine > > 5.13 and up is giving issues > > I see this issue on 6.0.0 as well. > > > > > We tested the 5.13.-rc1 as well, and already that kernel is giving issues. So something has changed in 5.13 that results in performance regression in certain workloads. And I wonder if it has something to do with the changes related to fscache that have, and are, happening in the kernel. These web servers might access the same directories / files concurrently. > > > > Note: we have quite a few 5.15 kernel clients not doing any (load-balanced) web based workload (container clusters on CephFS) that don't have any performance issue running these kernels. > > > > Issue: poor CephFS performance > > Symptom / result: excessive CephFS network usage (order of magnitude higher than for older kernels not having this issue), within a minute there are a bunch of slow web service processes, claiming loads of virtual memory, that result in heavy swap usage and basically rendering the node unusable slow. > > > > Other users that replied to this thread experienced similar symptoms. It is reproducible on both CentOS (EPEL mainline kernels) as well as on Ubuntu (hwe as well as default relase kernel). > > > > MDS version used: 15.2.16 (with a backported patch from 15.2.17) (single active / standby-replay) > > > > Does this ring a bell? > > > > Gr. Stefan > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 2

Pacific - mds How to know how many sequences still to be replayed
by Emmanuel Jaep 29 May '23

29 May '23

Hi, I just restarted one of our mds servers. I can find some "progress" in logs as below: mds.beacon.icadmin006 Sending beacon up:replay seq 461 mds.beacon.icadmin006 received beacon reply up:replay seq 461 rtt 0 How I know how long is the sequence (ie. when the node will be finished replaying)? Best, Emmanuel

1 0

Troubleshooting "N slow requests are blocked > 30 secs" on Pacific
by Emmanuel Jaep 29 May '23

29 May '23

Hi, we are running a cephfs cluster with the following version: ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) Several MDSs are reporting slow requests: HEALTH_WARN 4 MDSs report slow requests [WRN] MDS_SLOW_REQUEST: 4 MDSs report slow requests mds.icadmin011(mds.5): 1 slow requests are blocked > 30 secs mds.icadmin015(mds.6): 2 slow requests are blocked > 30 secs mds.icadmin006(mds.4): 8 slow requests are blocked > 30 secs mds.icadmin007(mds.2): 2 slow requests are blocked > 30 secs According to Quincy's documentation ( https://docs.ceph.com/en/quincy/cephfs/troubleshooting/) this can be investigated by issuing: ceph mds.icadmin011 dump cache /tmp/dump.txt Unfortunately, this command fails: no valid command found; 10 closest matches: pg stat pg getmap pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] pg dump_json [all|summary|sum|pools|osds|pgs...] pg dump_pools_json pg ls-by-pool <poolstr> [<states>...] pg ls-by-primary <id|osd.id> [<pool:int>] [<states>...] pg ls-by-osd <id|osd.id> [<pool:int>] [<states>...] pg ls [<pool:int>] [<states>...] pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [<threshold:int>] Error EINVAL: invalid command I imagine that it is related to the fact that we are running the Pacific version and not the Quincy version. When looking at the Pacific's documentation ( https://docs.ceph.com/en/pacific/cephfs/health-messages/) I should: > Use the ops admin socket command to list outstanding metadata operations. Unfortunately, I fail to really understand what I'm supposed to do. Can someone give a pointer? Best, Emmanuel

2 7

Recoveries without any misplaced objects?
by Hector Martin 29 May '23

29 May '23

Hi, I'm watching a cluster finish a bunch of backfilling, and I noticed that quite often PGs end up with zero misplaced objects, even though they are still backfilling. Right now the cluster is down to 6 backfilling PGs: data: volumes: 1/1 healthy pools: 6 pools, 268 pgs objects: 18.79M objects, 29 TiB usage: 49 TiB used, 25 TiB / 75 TiB avail pgs: 262 active+clean 6 active+remapped+backfilling But there are no misplaced objects, and the misplaced column in `ceph pg dump` is zero for all PGs. If I do a `ceph pg dump_json`, I can see `num_objects_recovered` increasing for these PGs... but the misplaced count is still 0. Is there something else that would cause recoveries/backfills other than misplaced objects? Or perhaps there is a bug somewhere causing the misplaced object count to be misreported as 0 sometimes? # ceph -v ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) - Hector

1 0

[Ceph | Quency ]The scheduled snapshots are not getting created till we create a manual backup.
by Lokendra Rathour 29 May '23

29 May '23

Hi Team, *Problem:* Create scheduled snapshots of the ceph subvolume. *Expected Result:* The scheduled snapshots should be created at the given scheduled time. *Actual Result:* The scheduled snapshots are not getting created till we create a manual backup. *Description:* *Ceph version: 17(quincy)* OS: Centos/Almalinux The scheduled snapshot creation is not working and we were only able to see the following logs in the file "ceph-mgr.storagenode3.log": *2023-05-29T04:59:35.101+0000 7f4cd3ad8700 0 [snap_schedule INFO mgr_util] scanning for idle connections..* *2023-05-29T04:59:35.101+0000 7f4cd3ad8700 0 [snap_schedule DEBUG mgr_util] fs_name (cephfs) connections ([])* *2023-05-29T04:59:35.101+0000 7f4cd3ad8700 0 [snap_schedule INFO mgr_util] cleaning up connections: [* The command which we were executing to add the snapshot schedule: *ceph fs snap-schedule add /volumes/<subvolume_group>/<subvolume> <frequency_of_snapshot> <Timeinformat_yyyymmddTHH:MM:ss>* *eg.* *ceph fs snap-schedule add /volumes/xyz/test_restore_53 1h 2023-05-26T11:05:00* We can make sure that the schedule has been created using the following commands: *#ceph fs snap-schedule list / --recursive=true* *#ceph fs snap-schedule status /volumes/xyz/test_restore_53* Even though we created the snapshot schedule, snapshots were not getting created. We then tried creating a manual snapshot for one of the sub-volumes using the following command: *#ceph fs subvolume snapshot create cephfs <subvolume> <snapshot-name> --group_name <subvolume_group>* *eg. ceph fs subvolume snapshot create cephfs test_restore_53 snapshot-1 --group_name xyz* To check the snapshots created we can use the following command: *ceph fs subvolume snapshot ls cephfs <subvolume> <snapshot-name> <subvolume_group>* *eg. ceph fs subvolume snapshot ls cephfs test_restore_53 snapshot-1 xyz* To delete the manually created snapshot: *ceph fs subvolume snapshot rm cephfs <subvolume> <snapshot-name> <subvolume_group>* *eg. ceph fs subvolume snapshot rm cephfs test_restore_53 snapshot-1 xyz* To our surprise, the scheduled snapshots started working. We also applied the retention policy and seems to be working fine. We re-tested this understanding for another subvolume. And the scheduled snapshots only started once we triggered a manual snapshot. Could you please help us out with this? -- ~ Lokendra skype: lokendrarathour

2 1

Creating a bucket with bucket constructor in Ceph v16.2.7
by Ramin Najjarbashi 29 May '23

29 May '23

Hi I'm currently using Ceph version 16.2.7 and facing an issue with bucket creation in a multi-zone configuration. My setup includes two zone groups: ZG1 (Master) and ZG2, with one zone in each zone group (zone-1 in ZG1 and zone-2 in ZG2). The objective is to create buckets in a specific zone group (ZG2) using the bucket constructor. However, despite setting the desired zone group (abrak) in the request, the bucket is still being created in the master zone group (ZG1). I have defined the following endpoint pattern for each zone group: s3.{zg}.mydomain.com I am using the s3cmd client to interact with the Ceph cluster. I have ensured that I provide the necessary endpoint and region information while executing the bucket creation command. Despite my efforts, the bucket consistently gets created in ZG1 instead of ZG2. - Ceph Version: 16.2.7 - Zone Group 1 (ZG1) Endpoint: http://s3.zonegroup1.mydomain.com - Zone Group 2 (ZG2) Endpoint: http://s3.zonegroup2.mydomain.com - Desired Bucket Creation Region: zg2-api-name have reviewed the Ceph documentation and made necessary configuration changes, but I have not been able to achieve the desired result.I kindly request your assistance in understanding why the bucket constructor is not honoring the specified region and always defaults to ZG1. I would greatly appreciate any insights, recommendations, or potential solutions to resolve this issue. Thank you for your time and support. --------------------------------- Here are the details of my setup: --------------------------------- ```sh s3cmd --region zg2-api-name mb s3://test-zg2s3cmd info s3://test-zg2s3://test-zg2/ (bucket): Location: zg2-api-name Payer: BucketOwner Expiration Rule: none Policy: none CORS: none ACL: development: FULL_CONTROL ``` this is my config file: ```ini [default] access_key = <KEY> secret_key = <SECRET> host_base = s3.zonegroup1.mydomain.com host_bucket = s3.%(location)s.mydomain.com #host_bucket = %(bucket)s.s3.zonegroup1.mydomain.com #host_bucket = s3.%(location)s.mydomain.com #host_bucket = s3.%(region)s.mydomain.com bucket_location = zg1-api-name use_https = False ``` Zonegroup configuration for the `zonegroup1` region: ```json { "id": "fb3f818a-ca9b-4b12-b431-7cdcd80006d", "name": "zg1-api-name", "api_name": "zg1-api-name", "is_master": "false", "endpoints": [ "http://s3.zonegroup1.mydomain.com", ], "hostnames": [ "s3.zonegroup1.mydomain.com", ], "hostnames_s3website": [ "s3-website.zonegroup1.mydomain.com", ], "master_zone": "at2-stg-zone", "zones": [ { "id": "at2-stg-zone", "name": "at2-stg-zone", "endpoints": [ "http://s3.zonegroup1.mydomain.com" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "fa2f8194-4a9d-4b98-b411-9cdcd1e5506a", "sync_policy": { "groups": [] } } ``` Zonegroup configuration for the `zonegroup2` region: ```json { "id": "a513d60c-44a2-4289-a23d-b7a511be6ee4", "name": "zg2-api-name", "api_name": "zg2-api-name", "is_master": "false", "endpoints": [ "http://s3.zonegroup2.mydomain.com" ], "hostnames": [ "s3.zonegroup2.mydomain.com" ], "hostnames_s3website": [], "master_zone": "zonegroup2-sh-1", "zones": [ { "id": "zonegroup2-sh-1", "name": "zonegroup2-sh-1", "endpoints": [ "http://s3.zonegroup2.mydomain.com" ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "fa2f8194-4a9d-4b98-b411-9cdcd1e5506a", "sync_policy": { "groups": [] } } ```

3 4

Can lifecycle transition cause a high load if many buckets do transition at the same time?
by huy nguyen 29 May '23

29 May '23

Hi, I have a plan to use the second storage class for cold data in the near future. But I have a question: if many old buckets with many objects are set to transition at the same time, can Ceph handle it by default, or do I have to configure QoS to avoid the high load? Thanks!

1 0

Ceph iscsi gateway semi deprecation warning?
by Mark Kirkwood 29 May '23

29 May '23

I am looking at using an iscsi gateway in front of a ceph setup. However the warning in the docs is concerning: The iSCSI gateway is in maintenance as of November 2022. This means that it is no longer in active development and will not be updated to add new features. Does this mean I should be wary of using it, or is it simply that it does all the stuff it needs to and no further development is needed? regards Mark

3 2

fail delete "daemon(s) not managed by cephadm"
by farhad kh 28 May '23

28 May '23

hi everyone i have a warning ` 1 stray daemon(s) not managed by cephadm` # ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon mon.apcepfpspsp0111 on host apcepfpspsp0111 not managed by cephadm and in detail : # ceph crash info 2023-01-17T18:25:19.601062Z_4affd3b9-f486-4ae5-801b-ede231e2b624 { "archived": "2023-01-18 19:31:58.200096", "assert_condition": "abort", "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.5/rpm/el8/BUILD/ceph-17.2.5/src/mon/MonitorDBStore.h", "assert_func": "int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)", "assert_line": 355, "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.5/rpm/el8/BUILD/ceph-17.2.5/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7fabeb737700 time 2023-01-17T18:25:19.568046+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.5/rpm/el8/BUILD/ceph-17.2.5/src/mon/MonitorDBStore.h: 355: ceph_abort_msg(\"failed to write to db\")\n", "assert_thread_name": "ms_dispatch", "backtrace": [ "/lib64/libpthread.so.0(+0x12cf0) [0x7fabf5949cf0]", "gsignal()", "abort()", "(ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x197) [0x7fabf799eb5f]", "(MonitorDBStore::apply_transaction(std::shared_ptr<MonitorDBStore::Transaction>)+0x88f) [0x55fe9cc3a68f]", "(Elector::persist_epoch(unsigned int)+0x184) [0x55fe9ccc64c4]", "(ElectionLogic::bump_epoch(unsigned int)+0x5d) [0x55fe9ccccd5d]", "(ElectionLogic::propose_classic_prefix(int, unsigned int)+0x3c7) [0x55fe9ccce8e7]", "(ElectionLogic::propose_classic_handler(int, unsigned int)+0x2b) [0x55fe9cccef9b]", "(Elector::handle_propose(boost::intrusive_ptr<MonOpRequest>)+0x6f5) [0x55fe9ccc6175]", "(Elector::dispatch(boost::intrusive_ptr<MonOpRequest>)+0xcdb) [0x55fe9ccc756b]", "(Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x11c2) [0x55fe9cc0aeb2]", "(Monitor::_ms_dispatch(Message*)+0x406) [0x55fe9cc0b5e6]", "(Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5d) [0x55fe9cc3bdad]", "(Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x478) [0x7fabf7c18c88]", "(DispatchQueue::entry()+0x50f) [0x7fabf7c160cf]", "(DispatchQueue::DispatchThread::entry()+0x11) [0x7fabf7cdd8f1]", "/lib64/libpthread.so.0(+0x81ca) [0x7fabf593f1ca]", "clone()" ], "ceph_version": "17.2.5", "crash_id": "2023-01-17T18:25:19.601062Z_4affd3b9-f486-4ae5-801b-ede231e2b624", "entity_name": "mon.apcepfpspsp0101", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "8", "os_version_id": "8", "process_name": "ceph-mon", "stack_sig": "74f29b024a3ec0f1395579505c5728e8ad5fd5a77c4a92437d696fa6fd1a8e20", "timestamp": "2023-01-17T18:25:19.601062Z", "utsname_hostname": "apcepfpspsp0101", "utsname_machine": "x86_64", "utsname_release": "5.4.17-2136.310.7.1.el8uek.x86_64", "utsname_sysname": "Linux", "utsname_version": "#2 SMP Wed Aug 17 15:14:08 PDT 2022" } but i have 3 instance from mon modul NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 3/3 2m ago 9M label:mon crash 6/6 9m ago 9M * grafana ?:3000 3/3 2m ago 4M count-per-host:1;label:mon mds.rook 3/3 2m ago 8M apcepfpspsp0101;apcepfpspsp0103;apcepfpspsp0105;count:3 mgr 2/2 2m ago 9M count:2 mon 3/3 2m ago 2h label:mon node-exporter ?:9100 6/6 9m ago 9M * osd 3 2m ago - <unmanaged> osd.cost_capacity 15 9m ago 4M * prometheus ?:9095 3/3 2m ago 9M label:mon whene i want remove this get # ceph orch daemon rm mon.apcepfpspsp0111 --force Error EINVAL: Unable to find daemon(s) ['mon.apcepfpspsp0111'] I don't have such a daemon mon.apcepfpspsp0101 apcepfpspsp0101 running (88m) mon.apcepfpspsp0103 apcepfpspsp0103 running (87m) mon.apcepfpspsp0105 apcepfpspsp0105 running (87m) What should I do to remove this warning?

2 1

cephfs-data-scan with multiple data pools
by Justin Li 26 May '23

26 May '23

Dear All, I'm trying to recover failed MDS metadata by following the link below but having troubles. Thanks in advance. Question1: how to scan 2 data pools with scan_extents (cmd 1). The cmd didn't work with two pools specified. Should I scan one then the other? Question2: As to scan_inodes (cmd 2), should I only specify the first data pool per the document. I'm concerned if the 2nd pool is not scanned then that'll cause metadata loss. *my fs name: cephfs, data pools: cephfs_hdd, cephfs_ssd* cmd 1: cephfs-data-scan scan_extents --filesystem cephfs cephfs_hdd cephfs_ssd cmd 2: cephfs-data-scan scan_inodes --filesystem cephfs cephfs_hdd cephfs-data-scan scan_extents [<data pool> [<extra data pool> ...]]cephfs-data-scan scan_inodes [<data pool>]cephfs-data-scan scan_links Note, the data pool parameters for ‘scan_extents’, ‘scan_inodes’ and ‘cleanup’ commands are optional, and usually the tool will be able to detect the pools automatically. Still you may override this. The ‘scan_extents’ command needs all data pools to be specified,* while ‘scan_inodes’ and ‘cleanup’ commands need only the main data pool.* *https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/ <https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/>* -- Best Regards, *Justin Li* IT Support/Systems Administrator *Justin.Li2030(a)Gmail.com <Justin.Li2030(a)Gmail.com>* <http://www.linkedin.com/in/justinli7>

1 0

Unable to online CephFS, MDS segfaults during mds log replay
by Alfred Heisner 26 May '23

26 May '23

Hello, I have a Ceph deployment using CephFS. Recently MDS failed and cannot start. Attempting to start MDS for this filesystem results in nearly immediate segfault in MDS. Logs below. cephfs-journal-tool shows Overall journal integrity state OK root@proxmox-2:/var/log/ceph# cephfs-journal-tool --rank=galaxy:all journal inspect Overall journal integrity: OK Stack dump / log from MDS: -14> 2023-05-26T15:01:09.204-0500 7f27c24b2700 1 mds.0.journaler.mdlog(ro) probing for end of the log -13> 2023-05-26T15:01:09.208-0500 7f27c34b4700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 4194304, expire 4194607, write 4194607, stream_format 1). probing for end of log (from 4194607)... -12> 2023-05-26T15:01:09.208-0500 7f27c34b4700 1 mds.0.journaler.pq(ro) probing for end of the log -11> 2023-05-26T15:01:09.412-0500 7f27c24b2700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 2388235687 (header had 2388213543). recovered. -10> 2023-05-26T15:01:09.412-0500 7f27c34b4700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 4194607 (header had 4194607). recovered. -9> 2023-05-26T15:01:09.412-0500 7f27c34b4700 4 mds.0.purge_queue operator(): open complete -8> 2023-05-26T15:01:09.412-0500 7f27c34b4700 1 mds.0.journaler.pq(ro) set_writeable -7> 2023-05-26T15:01:09.412-0500 7f27c1cb1700 4 mds.0.log Journal 0x200 recovered. -6> 2023-05-26T15:01:09.412-0500 7f27c1cb1700 4 mds.0.log Recovered journal 0x200 in format 1 -5> 2023-05-26T15:01:09.412-0500 7f27c1cb1700 2 mds.0.6403 Booting: 1: loading/discovering base inodes -4> 2023-05-26T15:01:09.412-0500 7f27c1cb1700 0 mds.0.cache creating system inode with ino:0x100 -3> 2023-05-26T15:01:09.412-0500 7f27c1cb1700 0 mds.0.cache creating system inode with ino:0x1 -2> 2023-05-26T15:01:09.416-0500 7f27c24b2700 2 mds.0.6403 Booting: 2: replaying mds log -1> 2023-05-26T15:01:09.416-0500 7f27c24b2700 2 mds.0.6403 Booting: 2: waiting for purge queue recovered 0> 2023-05-26T15:01:09.428-0500 7f27c0caf700 -1 *** Caught signal (Segmentation fault) ** in thread 7f27c0caf700 thread_name:md_log_replay ceph version 17.2.6 (995dec2cdae920da21db2d455e55efbc339bde24) quincy (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140) [0x7f27cd70c140] 2: (EMetaBlob::replay(MDSRank*, LogSegment*, MDPeerUpdate*)+0x66c2) [0x563540fc7372] 3: (EUpdate::replay(MDSRank*)+0x3c) [0x563540fc8abc] 4: (MDLog::_replay_thread()+0x7cb) [0x563540f4d0fb] 5: (MDLog::ReplayThread::entry()+0xd) [0x563540c1fbfd] 6: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f27cd700ea7] 7: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. What are the safest steps to recovery at this point? Thanks, Al

1 0

Pacific - MDS behind on trimming
by Emmanuel Jaep 26 May '23

26 May '23

Hi, lately, we have had some issues with our MDSs (Ceph version 16.2.10 Pacific). Part of them are related to MDS being behind on trimming. I checked the documentation and found the following information ( https://docs.ceph.com/en/pacific/cephfs/health-messages/) > CephFS maintains a metadata journal that is divided into *log segments*. The length of journal (in number of segments) is controlled by the setting mds_log_max_segments, and when the number of segments exceeds that setting the MDS starts writing back metadata so that it can remove (trim) the oldest segments. If this writeback is happening too slowly, or a software bug is preventing trimming, then this health message may appear. The threshold for this message to appear is controlled by the config option mds_log_warn_factor, the default is 2.0. Some resources on the web (https://www.suse.com/support/kb/doc/?id=000019740) indicated that a solution would be to change the `mds_log_max_segments`. Which I did: ``` ceph --cluster floki tell mds.* injectargs '--mds_log_max_segments=400000' ``` Of course, the warning disappeared, but I have a feeling that I just hid the problem. Pushing a value to 400'000 when the default value is 512 is a lot. Why is the trimming not taking place? How can I troubleshoot this further? Best, Emmanuel

2 1

mds dump inode crashes file system
by Frank Schilder 26 May '23

26 May '23

Hi all, I have an annoying problem with a specific ceph fs client. We have a file server on which we re-export kernel mounts via samba (all mounts with noshare option). On one of these re-exports we have recurring problems. Today I caught it with 2023-05-10T13:39:50.963685+0200 mds.ceph-23 (mds.1) 1761 : cluster [WRN] client.205899841 isn't responding to mclientcaps(revoke), ino 0x20011d3e5cb pending pAsLsXsFscr issued pAsLsXsFscr, sent 61.705410 seconds ago and I wanted to look up what path the inode 0x20011d3e5cb points to. Unfortunately, the command ceph tell "mds.*" dump inode 0x20011d3e5cb crashes an MDS in a way that it restarts itself, but doesn't seem to come back clean (it does not fail over to a stand-by). If I repeat the command above, it crashes the MDS again. Execution on other MDS daemons succeeds, for example: # ceph tell "mds.ceph-24" dump inode 0x20011d3e5cb 2023-05-10T14:14:37.091+0200 7fa47ffff700 0 client.210149523 ms_handle_reset on v2:192.168.32.88:6800/3216233914 2023-05-10T14:14:37.124+0200 7fa4857fa700 0 client.210374440 ms_handle_reset on v2:192.168.32.88:6800/3216233914 dump inode failed, wrong inode number or the inode is not cached The caps recall gets the client evicted at some point but it doesn't manage to come back clean. On a single ceph fs mount point I see this # ls /shares/samba/rit-oil ls: cannot access '/shares/samba/rit-oil': Stale file handle All other mount points are fine, just this one acts up. A "mount -o remount /shares/samba/rit-oil" crashed the entire server and I had to do a cold reboot. On reboot I see this message: https://imgur.com/a/bOSLxBb , which only occurs on this one file server (we are running a few of those). Does this point to a more serious problem, like a file system corruption? Should I try an fs scrub on the corresponding path? Some info about the system: The file server's kernel version is quite recent, updated two weeks ago: $ uname -r 4.18.0-486.el8.x86_64 # cat /etc/redhat-release CentOS Stream release 8 Our ceph cluster is octopus latest and we use the packages from the octopus el8 repo on this server. We have several such shares and they all work fine. It is only on one share where we have persistent problems with the mount point hanging or the server freezing and crashing. After working hours I will try a proper fail of the "broken" MDS to see if I can execute the dump inode command without it crashing everything. In the mean time, any hints would be appreciated. I see that we have an exceptionally large MDS log for the problematic one. Any hint what to look for would be appreciated, it contains a lot from the recovery operations: # pdsh -w ceph-[08-17,23-24] ls -lh "/var/log/ceph/ceph-mds.ceph-??.log" ceph-23: -rw-r--r--. 1 ceph ceph 15M May 10 14:28 /var/log/ceph/ceph-mds.ceph-23.log *** huge *** ceph-24: -rw-r--r--. 1 ceph ceph 14K May 10 14:28 /var/log/ceph/ceph-mds.ceph-24.log ceph-10: -rw-r--r--. 1 ceph ceph 394 May 10 14:02 /var/log/ceph/ceph-mds.ceph-10.log ceph-13: -rw-r--r--. 1 ceph ceph 394 May 10 14:02 /var/log/ceph/ceph-mds.ceph-13.log ceph-08: -rw-r--r--. 1 ceph ceph 394 May 10 14:02 /var/log/ceph/ceph-mds.ceph-08.log ceph-15: -rw-r--r--. 1 ceph ceph 14K May 10 14:28 /var/log/ceph/ceph-mds.ceph-15.log ceph-17: -rw-r--r--. 1 ceph ceph 14K May 10 14:28 /var/log/ceph/ceph-mds.ceph-17.log ceph-14: -rw-r--r--. 1 ceph ceph 16K May 10 14:28 /var/log/ceph/ceph-mds.ceph-14.log ceph-09: -rw-r--r--. 1 ceph ceph 16K May 10 14:28 /var/log/ceph/ceph-mds.ceph-09.log ceph-16: -rw-r--r--. 1 ceph ceph 15K May 10 14:28 /var/log/ceph/ceph-mds.ceph-16.log ceph-11: -rw-r--r--. 1 ceph ceph 14K May 10 14:28 /var/log/ceph/ceph-mds.ceph-11.log ceph-12: -rw-r--r--. 1 ceph ceph 394 May 10 14:02 /var/log/ceph/ceph-mds.ceph-12.log Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 24

Multi region RGW Config Questions - Quincy
by Deep Dish 26 May '23

26 May '23

Hello, I have a Qunicy (17.2.6) cluster, looking to create a multi-zone / multi-region RGW service and have a few questions with respect to published docs - https://docs.ceph.com/en/quincy/radosgw/multisite/. In general, I understand the process as: 1. Create a new REALM, ZONEGROUP, ZONE: radosgw-admin realm create --rgw-realm=my_new_realm [--default] radosgw-admin zonegroup create --rgw-zonegroup=my_country --endpoints=http://rgw1:80 --rgw-realm=my_new_realm --master –default radosgw-admin zone create --rgw-zonegroup=my_country --rgw-zone=my-region *\* --master --default *\* --endpoints={http://fqdn}[,{http://fqdn}] ## Question: If I have multiple RGWs deployed on my cluster, do I specify all of them as individual endpoints? OR specifying one rgw automatically propagates config throughout all? 2. Create SYSTEM user radosgw-admin user create --uid="synchronization-user" --display-name="Synchronization User" --system radosgw-admin zone modify --rgw-zone={zone-name} --access-key={access-key} --secret={secret} radosgw-admin period update --commit ## Question: The SYSTEM user is used only for replication? Will creating new REALM, ZONGROUP, ZONE reset any administrative access to management of RGWs through ceph-dashboard? 3. Remove DEFAULT REALM, ZONEGROUP, ZONE and supporting pools radosgw-admin zonegroup delete --rgw-zonegroup=default --rgw-zone=default radosgw-admin period update --commit radosgw-admin zone delete --rgw-zone=default radosgw-admin period update --commit radosgw-admin zonegroup delete --rgw-zonegroup=default radosgw-admin period update --commit ceph osd pool rm default.rgw.control default.rgw.control --yes-i-really-really-mean-it ceph osd pool rm default.rgw.data.root default.rgw.data.root --yes-i-really-really-mean-it ceph osd pool rm default.rgw.gc default.rgw.gc --yes-i-really-really-mean-it ceph osd pool rm default.rgw.log default.rgw.log --yes-i-really-really-mean-it ceph osd pool rm default.rgw.users.uid default.rgw.users.uid --yes-i-really-really-mean-it 4. UPDATING CEPH CONFIG FILE / RGW CONFIG VIA CEPH ORCH # QUESTION: Since I’m using ceph orch would I simply set rgw_zone property via CLUSTER -> CONFIGURATION on ceph-dashboard? Thank you.

1 0

ln: failed to create hard link 'file name': Read-only file system
by Frank Schilder 26 May '23

26 May '23

Hi all, on an NFS re-export of a ceph-fs (kernel client) I observe a very strange error. I'm un-taring a larger package (1.2G) and after some time I get these errors: ln: failed to create hard link 'file name': Read-only file system The strange thing is that this seems only temporary. When I used "ln src dst" for manual testing, the command failed as above. However, after that I tried "ln -v src dst" and this command created the hard link with exactly the same path arguments. During the period when the error occurs, I can't see any FS in read-only mode, neither on the NFS client nor the NFS server. Funny thing is that file creation and write still works, its only the hard-link creation that fails. For details, the set-up is: file-server: mount ceph-fs at /shares/path, export /shares/path as nfs4 to other server other server: mount /shares/path as NFS More precisely, on the file-server: fstab: MON-IPs:/shares/folder /shares/nfs/folder ceph defaults,noshare,name=NAME,secretfile=sec.file,mds_namespace=FS-NAME,_netdev 0 0 exports: /shares/nfs/folder -no_root_squash,rw,async,mountpoint,no_subtree_check DEST-IP On the host at DEST-IP: fstab: FILE-SERVER-IP:/shares/nfs/folder /mnt/folder nfs defaults,_netdev 0 0 Both, the file server and the client server are virtual machines. The file server is on Centos 8 stream (4.18.0-338.el8.x86_64) and the client machine is on AlmaLinux 8 (4.18.0-425.13.1.el8_7.x86_64). When I change the NFS export from "async" to "sync" everything works. However, that's a rather bad workaround and not a solution. Although this looks like an NFS issue, I'm afraid it is a problem with hard links and ceph-fs. It looks like a race with scheduling and executing operations on the ceph-fs kernel mount. Has anyone seen something like that? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 13

`ceph features` on Nautilus still reports "luminous"
by Oliver Schmidt 26 May '23

26 May '23

Dear Ceph community, on our way towards getting our cluster to a current Ceph release, we updated all hosts and clients to Nautilus 14.2.22. But despite setting `ceph osd set-require-min-compat-client nautilus`, the release reported by `ceph features` is still "luminous". Is this supposed to be like this? If not, does anyone have an idea what might be missing to make the features being reported as "nautilus" as well? ``` ~ # ceph mon dump epoch 66 fsid b67bad36-3273-11e3-a2ed-0200000311bf last_changed 2022-12-12 12:20:39.244333 created 2013-10-11 14:57:32.291514 min_mon_release 14 (nautilus) 0: [v2:172.20.4.10:3300/0,v1:172.20.4.10:6789/0] mon.host1 1: [v2:172.20.4.100:3300/0,v1:172.20.4.100:6789/0] mon.host2 2: [v2:172.20.4.101:3300/0,v1:172.20.4.101:6789/0] mon.host3 dumped monmap epoch 66 ~ # ceph features { "mon": [ { "features": "0x3ffddff8ffecffff", "release": "luminous", "num": 3 } ], "osd": [ { "features": "0x3ffddff8ffecffff", "release": "luminous", "num": 14 } ], "client": [ { "features": "0x3ffddff8ffecffff", "release": "luminous", "num": 137 } ], "mgr": [ { "features": "0x3ffddff8ffecffff", "release": "luminous", "num": 3 } ] } ``` All the best -- Oliver Schmidt · os(a)flyingcircus.io · Systems Engineer Flying Circus Internet Operations GmbH · http://flyingcircus.io Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick

5 6

Re: [Help appreciated] ceph mds damaged
by Justin Li 26 May '23

26 May '23

Sorry Patrick, last email was restricted as attachment size. I attached a link for you to download the log. Thanks. https://drive.google.com/drive/folders/1bV_X7vyma_-gTfLrPnEV27QzsdmgyK4g?us… Justin Li Senior Technical Officer School of Information Technology Faculty of Science, Engineering and Built Environment For ICT Support please see https://www.deakin.edu.au/sebeicthelp Deakin University Melbourne Burwood Campus, 221 Burwood Highway, Burwood, VIC 3125 +61 3 9246 8932 Justin.li(a)deakin.edu.au http://www.deakin.edu.au/ Deakin University CRICOS Provider Code 00113B Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone. Deakin University does not warrant that this email and any attachments are error or virus free. -----Original Message----- From: Justin Li Sent: Wednesday, May 24, 2023 8:21 AM To: Patrick Donnelly <pdonnell(a)redhat.com> Cc: ceph-users(a)ceph.io Subject: RE: [ceph-users] [Help appreciated] ceph mds damaged Hi Patrick, I attached two logs here. Those two servers are one of the monitors and MDSs. Let me know if you need more logs. Thanks. Justin Li Senior Technical Officer School of Information Technology Faculty of Science, Engineering and Built Environment For ICT Support please see https://www.deakin.edu.au/sebeicthelp Deakin University Melbourne Burwood Campus, 221 Burwood Highway, Burwood, VIC 3125 +61 3 9246 8932 Justin.li(a)deakin.edu.au http://www.deakin.edu.au/ Deakin University CRICOS Provider Code 00113B Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone. Deakin University does not warrant that this email and any attachments are error or virus free. -----Original Message----- From: Patrick Donnelly <pdonnell(a)redhat.com> Sent: Wednesday, May 24, 2023 7:35 AM To: Justin Li <justin.li(a)deakin.edu.au> Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] [Help appreciated] ceph mds damaged Hello Justin, On Tue, May 23, 2023 at 4:55 PM Justin Li <justin.li(a)deakin.edu.au> wrote: > > Dear All, > > After a unsuccessful upgrade to pacific, MDS were offline and could not get back on. Checked the MDS log and found below. See cluster info from below as well. Appreciate it if anyone can point me to the right direction. Thanks. > > > MDS log: > > 2023-05-24T06:21:36.831+1000 7efe56e7d700 1 mds.0.cache.den(0x600 > 1005480d3b2) loaded already corrupt dentry: [dentry > #0x100/stray0/1005480d3b2 [19ce,head] rep(a)0,-2.0<mailto:rep@0,-2.0> > NULL (dversion lock) pv=0 v=2154265030 ino=(nil) state=0 > 0x556433addb80] > > -5> 2023-05-24T06:21:36.831+1000 7efe56e7d700 -1 mds.0.damage > notify_dentry Damage to dentries in fragment * of ino 0x600is fatal > because it is a system directory for this rank > > -4> 2023-05-24T06:21:36.831+1000 7efe56e7d700 5 mds.beacon.posco > set_want_state: up:active -> down:damaged > > -3> 2023-05-24T06:21:36.831+1000 7efe56e7d700 5 mds.beacon.posco > Sending beacon down:damaged seq 5339 > > -2> 2023-05-24T06:21:36.831+1000 7efe56e7d700 10 monclient: > _send_mon_message to mon.ceph-3 at v2:10.120.0.146:3300/0 > > -1> 2023-05-24T06:21:37.659+1000 7efe60690700 5 mds.beacon.posco > received beacon reply down:damaged seq 5339 rtt 0.827966 > > 0> 2023-05-24T06:21:37.659+1000 7efe56e7d700 1 mds.posco respawn! > > > Cluster info: > root@ceph-1:~# ceph -s > cluster: > id: e2b93a76-2f97-4b34-8670-727d6ac72a64 > health: HEALTH_ERR > 1 filesystem is degraded > 1 filesystem is offline > 1 mds daemon damaged > > services: > mon: 3 daemons, quorum ceph-1,ceph-2,ceph-3 (age 26h) > mgr: ceph-3(active, since 15h), standbys: ceph-1, ceph-2 > mds: 0/1 daemons up, 3 standby > osd: 135 osds: 133 up (since 10h), 133 in (since 2w) > > data: > volumes: 0/1 healthy, 1 recovering; 1 damaged > pools: 4 pools, 4161 pgs > objects: 230.30M objects, 276 TiB > usage: 836 TiB used, 460 TiB / 1.3 PiB avail > pgs: 4138 active+clean > 13 active+clean+scrubbing > 10 active+clean+scrubbing+deep > > > > root@ceph-1:~# ceph health detail > HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds > daemon damaged [WRN] FS_DEGRADED: 1 filesystem is degraded > fs cephfs is degraded > [ERR] MDS_ALL_DOWN: 1 filesystem is offline > fs cephfs is offline because no MDS is active for it. > [ERR] MDS_DAMAGE: 1 mds daemon damaged > fs cephfs mds.0 is damaged Do you have a complete log you can share? Try: https://docs.ceph.com/en/quincy/man/8/ceph-post-file/ To get your upgrade to complete, you may set: ceph config set mds mds_go_bad_corrupt_dentry false -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone. Deakin University does not warrant that this email and any attachments are error or virus free.

2 4

Feature/Change Request: Don't send alert emails for --sticky muted WARN conditions
by Edward R Huyer 25 May '23

25 May '23

I recently upgraded to Quincy, and toggled on the BULK flag of a few pools. As a result, my cluster has been spending the last several days shuffling data while growing the pool pg counts. That in turn has resulted in a steadily increasing number of pgs being flagged PG_NOT_DEEP_SCRUBBED. And that has resulted in my getting hundreds of alert emails about pgs not being deep scrubbed, because I get a new email whenever the count changes. I tried using "ceph health mute PG_NOT_DEEP_SCRUBBED --sticky", but all that did (in terms of the email alerts) was make the emails say "HEALTH_OK" instead of "HEALTH WARN", which is less than helpful. I haven't found a way to stop the cluster from sending me these alert emails other than turning off email notifications entirely. If there is one, I'd love to know what it is. If not, I feel like there ought to be one, either as part of muting the health warning, or as a separate toggle. Hundreds of emails over what is expected behavior is rather silly. ----- Edward Huyer Golisano College of Computing and Information Sciences Rochester Institute of Technology Golisano 70-2373 152 Lomb Memorial Drive Rochester, NY 14623 585-475-6651 erhvks(a)rit.edu<mailto:erhvks@rit.edu> Obligatory Legalese: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this information.

1 0

MDS Upgrade from 17.2.5 to 17.2.6 not possible
by Henning Achterrath 25 May '23

25 May '23

Hi all, we did a major update from Pacific to Quincy (17.2.5) a month ago without any problems. Now we have tried a minor update from 17.2.5 to 17.2.6 (ceph orch upgrade). It stucks at mds upgrade phase. At this point the cluster tries to scale down mds (ceph fs set max_mds 1). We waited a few hours. We are running two active mds with 1 standby. No subdir pinning configured. CephFS data pool: 575 TB While Upgrading, Rank 1 MDS remains in state stopping. During this state clients are not able to reconnect. So we paused this upgrade and set max_mds to 2 back again and fail rank 1. After that, standby becomes active. In the mds (rank 1 in stopping state) logs we can see: waiting for strays to migrate In our second try, we have evicted all clients first without success. We make daily snapshots on / and rotate them via snapshot scheduler after one week. Is there a way to get rid of stray entries without scale down mds or do we have to wait longer? We had about the same amount of strays before we did the major upgrade. So, it is a bit curious. Current output from ceph perf dump Rank0: "num_strays": 417304, "num_strays_delayed": 3, "num_strays_enqueuing": 0, "strays_created": 567879, "strays_enqueued": 561803, "strays_reintegrated": 13751, "strays_migrated": 4, Rank1: ceph daemon mds.fdi-cephfs.ceph-service-13.rwdkqs perf dump | grep stray "num_strays": 172528, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "strays_created": 418365, "strays_enqueued": 396142, "strays_reintegrated": 67406, "strays_migrated": 4, Any help would be appreciated. best regards Henning

6 7

sub-read of librados
by johnnyjohnnypd＠gmail.com 25 May '23

25 May '23

Hi! I am trying to read only part of an object by specifying the non-trivial offset and length of the read function: `librados::IoCtxImpl::read(const object_t& oid, bufferlist& bl, size_t len, uint64_t off)` from `IoCtxImpl.cc`. However, after connecting to an erasure code pool (e.g., 12+4), I try to read data from a randomly chosen OSD (that is, 1/12 of the object), but the results of command `vmstat -d` and `iostat` show that the entire object was read, since read operations appeared on all 12 OSDs. So, I wonder if librados doesn't support the real sub-read of an object, and what should I do if I want to implement this function. Thanks!

1 0

mgr memory usage constantly increasing
by Tobias Hachmer 25 May '23

25 May '23

Hello list, we have found that the active mgr process in our 3-node CEPH cluster takes a lot of memory. After start the memory usage is constantly increasing. After 6 days the process takes ~67GB: ~# ps -p 7371 -o rss,%mem,cmd RSS %MEM CMD 71053880 26.9 /usr/bin/ceph-mgr -n mgr.hostname.nvwzhc -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false Cluster Specs: - 3-node 3-way mirror - each node has 256GB memory - each node has 10x 7.68TB NVMe, each NVMe is split into 2 OSDs - monitoring node is a separate VM (separate Hypervisor) - iSCSI and NFS Gateway are located on 2 separate VMs (separate Hypervisor) - main purpose is CephFS with currently ~15.42 million objects Is this normal behaviour or might we have a misconfiguration somewhere? What can we do to dig into this further? ~# ceph status cluster: id: f5129939-964b-11ed-bb6a-f7caa5af2f56 health: HEALTH_OK services: mon: 3 daemons, quorum host1,host2,host3 (age 6d) mgr: host2.nvwzhc(active, since 6d), standbys: host1.wwczzn mds: 1/1 daemons up, 1 standby, 1 hot standby osd: 60 osds: 60 up (since 6d), 60 in (since 8w) tcmu-runner: 2 portals active (2 hosts) data: volumes: 1/1 healthy pools: 6 pools, 2161 pgs objects: 15.42M objects, 45 TiB usage: 135 TiB used, 75 TiB / 210 TiB avail pgs: 2161 active+clean Thanks and kind regards Tobias Hachmer

3 4

Ceph Tech Talk For May 2023: RGW Lua Scripting Code Walkthrough
by Mike Perez 24 May '23

24 May '23

Hello everyone, Join us on May 24th at 17:00 UTC for a long overdue Ceph Tech Talk! This month, Yuval Lifshitz will give an RGW Lua Scripting Code Walkthrough. https://ceph.io/en/community/tech-talks/ You can also see Yuval's previous presentation at Ceph Month 2021, From Open Source to Open Ended in Ceph with Lua. https://www.youtube.com/watch?v=anQJugs27hE If you want to give a technical presentation for Ceph Tech Talks, please contact me directly with a title and description. Thank you! -- Mike Perez Community Manager Ceph Foundation

2 1

Training on ceph fs
by Emmanuel Jaep 24 May '23

24 May '23

Hi, I inherited a ceph fs cluster. Even if I have years of experience in systems management, I fail to grasp the complete logic of it fully. From what I found on the web, the documentation is either too "high level" or too detailed. Do you know any good resources to get fully acquainted with ceph (specifically, ceph fs)? Or any live training I could attend? Thanks in advance for your help, Emmanuel

2 1

Re: MDS crashes to damaged metadata
by Stefan Kooman 24 May '23

24 May '23

On 12/15/22 15:31, Stolte, Felix wrote: > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): Are you willing to share this script? I would like to use it to scan our CephFS before upgrading to 16.2.13. Do you run this script when the filesystem is online / active? Thanks, Gr. Stefan

2 3

cephfs max_file_size
by Dietmar Rieder 24 May '23

24 May '23

Hi, can the cephfs "max_file_size" setting be changed at any point in the lifetime of a cephfs? Or is it critical for existing data if it is changed after some time? Is there anything to consider when changing, let's say, from 1TB (default) to 4TB ? We are running the latest Nautilus release, BTW. Thanks in advance Dietmar

3 5

Slow recovery on Quincy
by Sake Paulusma 24 May '23

24 May '23

We noticed extremely slow performance when remapping is necessary. We didn't do anything special other than assigning the correct device_class (to ssd). When checking ceph status, we notice the number of objects recovering is around 17-25 (with watch -n 1 -c ceph status). How can we increase the recovery process? There isn't any client load, because we're going to migrate to this cluster in the future, so only a rsync once a while is being executed. [ceph: root@pwsoel12998 /]# ceph status cluster: id: da3ca2e4-ee5b-11ed-8096-0050569e8c3b health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set services: mon: 5 daemons, quorum pqsoel12997,pqsoel12996,pwsoel12994,pwsoel12998,prghygpl03 (age 3h) mgr: pwsoel12998.ylvjcb(active, since 3h), standbys: pqsoel12997.gagpbt mds: 4/4 daemons up, 2 standby osd: 32 osds: 32 up (since 73m), 32 in (since 6d); 10 remapped pgs flags noscrub,nodeep-scrub data: volumes: 2/2 healthy pools: 5 pools, 193 pgs objects: 13.97M objects, 853 GiB usage: 3.5 TiB used, 12 TiB / 16 TiB avail pgs: 755092/55882956 objects misplaced (1.351%) 183 active+clean 10 active+remapped+backfilling io: recovery: 2.3 MiB/s, 20 objects/s

8 17

[cephfs-data-scan] Estimate time for scanning extents and inodes
by Justin Li 24 May '23

24 May '23

Dear All, I'm using metadata repair tools to repair a damaged MDS following below document. My storage has about 276TB data. Cephfs-data-scan is using 32 workers. How long will it take to finish scanning extents? What about scanning inodes? It has run 6 hours and metadata pool dropped 1G. Is this normal? Thanks. ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 1.3 PiB 454 TiB 835 TiB 835 TiB 64.79 ssd 7.1 TiB 6.2 TiB 878 GiB 878 GiB 12.15 TOTAL 1.3 PiB 460 TiB 836 TiB 836 TiB 64.50 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL cephfs_data 1 4096 276 TiB 222.96M 828 TiB 71.28 111 TiB cephfs_metadata 2 32 101 GiB 5.83M 302 GiB 0.09 111 TiB device_health_metrics 10 1 239 MiB 251 716 MiB 0 112 TiB cephfs_ssd 11 32 117 GiB 1.49M 361 GiB 5.83 1.9 TiB https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/ Justin Li Senior Technical Officer School of Information Technology Faculty of Science, Engineering and Built Environment Request for assistance can be lodged to the SIT Technical Team using this form<https://deakinesmprod.service-now.com/esc?id=sc_cat_item&sys_id=7afa8fa5db6…> Deakin University Melbourne Burwood Campus, 221 Burwood Highway, Burwood, VIC 3125 +61 3 9246 8932 justin.li(a)deakin.edu.au<mailto:justin.li@deakin.edu.au> http://www.deakin.edu.au<http://www.deakin.edu.au/> Deakin University CRICOS Provider Code 00113B Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone. Deakin University does not warrant that this email and any attachments are error or virus free. Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone. Deakin University does not warrant that this email and any attachments are error or virus free.

1 0

RGW: How to restore the index and metadata of a bucket
by huy nguyen 24 May '23

24 May '23

There is a test bucket, I have removed its index and metadata: radosgw-admin bi purge --bucket abccc --yes-i-really-mean-it radosgw-admin metadata rm bucket.instance:abccc:17a4ce99-009e-40f2-a2d2-2afc218ebd9b.425824299.4 Now the index and metadata is gone, but how to clean its data? Or is there any way to restore the index and metadata? I have tried to fix the index, but it doesn't work: radosgw-admin bucket check --check-objects --fix --bucket abccc 2023-05-24 16:46:41.896 7f32d5890a80 -1 ERROR: get_bucket_instance_from_oid failed: -2 2023-05-24 16:46:41.896 7f32d5890a80 0 could not get bucket info for bucket=abccc Same result with bucket rm: radosgw-admin bucket rm --bucket abccc --purge-objects 2023-05-24 16:47:49.439 7fafb7939a80 -1 ERROR: get_bucket_instance_from_oid failed: -2 2023-05-24 16:47:49.439 7fafb7939a80 0 could not get bucket info for bucket=abccc

1 0

Timeout in Dashboard
by mailing-lists 24 May '23

24 May '23

Hey all, im facing a "minor" problem. I do not always get results when going to the dashboard, under Block->Images in the tab Images or Namespaces. The little refresh button will keep spinning and sometimes after several minutes it will finally show something. That is odd, because from the shell I am not seeing any problems like that. rbd namespace ls as well as: rbd ls --namespace someNS ...will give immediately their results. Does anyone know whats causing this? I did not yet find anything in the logs. Best Ken

1 0

[Help appreciated] ceph mds damaged
by Justin Li 23 May '23

23 May '23

Dear All, After a unsuccessful upgrade to pacific, MDS were offline and could not get back on. Checked the MDS log and found below. See cluster info from below as well. Appreciate it if anyone can point me to the right direction. Thanks. MDS log: 2023-05-24T06:21:36.831+1000 7efe56e7d700 1 mds.0.cache.den(0x600 1005480d3b2) loaded already corrupt dentry: [dentry #0x100/stray0/1005480d3b2 [19ce,head] rep(a)0,-2.0<mailto:rep@0,-2.0> NULL (dversion lock) pv=0 v=2154265030 ino=(nil) state=0 0x556433addb80] -5> 2023-05-24T06:21:36.831+1000 7efe56e7d700 -1 mds.0.damage notify_dentry Damage to dentries in fragment * of ino 0x600is fatal because it is a system directory for this rank -4> 2023-05-24T06:21:36.831+1000 7efe56e7d700 5 mds.beacon.posco set_want_state: up:active -> down:damaged -3> 2023-05-24T06:21:36.831+1000 7efe56e7d700 5 mds.beacon.posco Sending beacon down:damaged seq 5339 -2> 2023-05-24T06:21:36.831+1000 7efe56e7d700 10 monclient: _send_mon_message to mon.ceph-3 at v2:10.120.0.146:3300/0 -1> 2023-05-24T06:21:37.659+1000 7efe60690700 5 mds.beacon.posco received beacon reply down:damaged seq 5339 rtt 0.827966 0> 2023-05-24T06:21:37.659+1000 7efe56e7d700 1 mds.posco respawn! Cluster info: root@ceph-1:~# ceph -s cluster: id: e2b93a76-2f97-4b34-8670-727d6ac72a64 health: HEALTH_ERR 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged services: mon: 3 daemons, quorum ceph-1,ceph-2,ceph-3 (age 26h) mgr: ceph-3(active, since 15h), standbys: ceph-1, ceph-2 mds: 0/1 daemons up, 3 standby osd: 135 osds: 133 up (since 10h), 133 in (since 2w) data: volumes: 0/1 healthy, 1 recovering; 1 damaged pools: 4 pools, 4161 pgs objects: 230.30M objects, 276 TiB usage: 836 TiB used, 460 TiB / 1.3 PiB avail pgs: 4138 active+clean 13 active+clean+scrubbing 10 active+clean+scrubbing+deep root@ceph-1:~# ceph health detail HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged [WRN] FS_DEGRADED: 1 filesystem is degraded fs cephfs is degraded [ERR] MDS_ALL_DOWN: 1 filesystem is offline fs cephfs is offline because no MDS is active for it. [ERR] MDS_DAMAGE: 1 mds daemon damaged fs cephfs mds.0 is damaged Justin Li Senior Technical Officer School of Information Technology Faculty of Science, Engineering and Built Environment Request for assistance can be lodged to the SIT Technical Team using this form<https://deakinesmprod.service-now.com/esc?id=sc_cat_item&sys_id=7afa8fa5db6…> Deakin University Melbourne Burwood Campus, 221 Burwood Highway, Burwood, VIC 3125 +61 3 9246 8932 justin.li(a)deakin.edu.au<mailto:justin.li@deakin.edu.au> http://www.deakin.edu.au<http://www.deakin.edu.au/> Deakin University CRICOS Provider Code 00113B Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone. Deakin University does not warrant that this email and any attachments are error or virus free. Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone. Deakin University does not warrant that this email and any attachments are error or virus free.

3 4

Unable to change port range for OSDs in Pacific
by Renata Callado Borges 23 May '23

23 May '23

Dear all, How are you? I have a Pacific 3 nodes cluster, and the machines do double-duty as Ceph nodes and as Slurm clients. (I am well aware that this is not desirable, but my client wants it like this anyway). Our Slurm install uses the port 6818 for slurmd everywhere. In one of our Ceph/Slurm nodes, Ceph decided that port 6818 is great for an OSD. This prevents slurmd from running properly. Changing the slurmd port causes the Slurm master, slurmctld, to misread the OSD communication as Slurm "Insane length messages". I have tried unsuccessfully to change this port in Slurm and in Ceph. I wonder if someone here can help me limit the ports Ceph uses for its OSDs. I have tried this with no success: [root@darkside1 ~]# cat /etc/ceph/ceph.conf~ # minimal ceph.conf for 1902a026-496d-11ed-b43e-08c0eb320ec2 [global] fsid = 1902a026-496d-11ed-b43e-08c0eb320ec2 mon_host = [v2:172.22.132.188:3300/0,v1:172.22.132.188:6789/0] [osd] ms_bind_port_min = 6830 ms_bind_port_max = 7300 Then I restarted with "systemctl restart ceph.target" but the OSD keeps being re-bound to 6818. I also tried the same config but with the options under the [global] section. No luck there either. Tried reboot the Ceph/Slurm machine, the OSD is re-bound in 6818 also. Could someone help? Thanks in advance! Cordially, Renata.

2 2

Grafana service fails to start due to bad directory name after Quincy upgrade
by Adiga, Anantha 23 May '23

23 May '23

Hi Upgraded from Pacific 16.2.5 to 17.2.6 on May 8th However, Grafana fails to start due to bad folder path :/tmp# journalctl -u ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201 -n 25 -- Logs begin at Sun 2023-05-14 20:05:52 UTC, end at Tue 2023-05-16 19:07:51 UTC. -- May 16 19:05:00 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:00 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:00 fl31ca104ja0201 bash[2575021]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:00 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:00 fl31ca104ja0201 bash[2575030]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:00 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:10 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:10 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:10 fl31ca104ja0201 bash[2575273]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:10 fl31ca104ja0201 bash[2575282]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:20 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:20 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:20 fl31ca104ja0201 bash[2575369]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:20 fl31ca104ja0201 bash[2575370]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:30 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Start request r> May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:30 fl31ca104ja0201 systemd[1]: Failed to start Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ESCOC 19:07:51 UTC. -- 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory -be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Main process exited, code=exited, status=127/n/a ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory -be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Failed with result 'exit-code'. -be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Scheduled restart job, restart counter is at 3. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory -be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Main process exited, code=exited, status=127/n/a ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory -be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Failed with result 'exit-code'. -be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Scheduled restart job, restart counter is at 4. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. Check if path exists: #<mailto:root@fl31ca104ja0201:/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201#> ls /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run ls: cannot access '/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run': No such file or directory Check if grafana.fl31ca104ja0201 directory exists: #<mailto:root@fl31ca104ja0201:/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201#> ls -l rt/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/unit.run total 36 drwxr-xr-x 4 167 167 4096 Apr 20 08:05 data drwxr-xr-x 3 167 167 4096 Apr 20 08:13 etc -rw------- 1 167 167 48 Apr 20 08:13 unit.created -rw------- 1 167 167 390 May 8 16:12 unit.stop -rw------- 1 167 167 390 May 8 16:12 unit.poststop -rw------- 1 167 167 365 May 8 16:12 unit.meta -rw------- 1 167 167 32 May 8 16:12 unit.image -rw------- 1 167 167 38 May 8 16:12 unit.configured -rw------- 1 167 167 1063 May 8 16:12 unit.run cat unit.run set -e # grafana.fl31ca104ja0201 ! /usr/bin/docker rm -f ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana.fl31ca104ja0201 2> /dev/null ! /usr/bin/docker rm -f ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 2> /dev/null /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --init --name ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 --user 472 -e CONTAINER_IMAGE=docker.io/grafana/grafana:6.7.4 -e NODE_NAME=fl31ca104ja0201 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/grafana.ini:/etc/grafana/grafana.ini:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/certs:/etc/grafana/certs:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/data/grafana.db:/var/lib/grafana/grafana.db:Z docker.io/grafana/grafana:6.7.4 Thank you, Anantha

3 5

Deleting a CephFS volume
by Conrad Hoffmann 23 May '23

23 May '23

Hi all, I'm having difficulties removing a CephFS volume that I set up for testing. I've been through this with RBDs, so I do know about `mon_allow_pool_delete`. However, it doesn't help in this case. It is a cluster with 3 monitors. You can find a console log of me verifying that `mon_allow_pool_delete` is indeed true on all monitors but still fail to remove the volume here: https://paste.sr.ht/~bitfehler/4b317e5b8c5823583c1b26b42e8495f54cbd5e2a I am happy to post config or logs as needed, I wasn't sure which ones might be relevant. I scanned monitor and MDS logs, but nothing stuck out. Using the volume worked fine in tests, so I don't think it is broken or anything. Any ideas would be greatly appreciated! Thanks a bunch, Conrad

3 4

Advices : RGW and Citrix ADC
by BEAUDICHON Hubert (Acoss) 22 May '23

22 May '23

Hi everyone, Is there any advices, best practices for a Citrix ADC VIP configuration in front of a dual RadosGW (same zone) ? I find many documentations for HAProxy but nothing for this hardware. For information, the CEPH cluster version is Pacific 16.2.10. Thanks for your help. Regards, Hubert [cid:image001.png@01D98CA0.D075D720] Hubert Beaudichon Secteur Infrastructures / Gamme CET-Infrastructures SDAT UT AUTOMATION ET CLOUD Hubert.beaudichon(a)acoss.fr<mailto:Hubert.beaudichon@acoss.fr> 01 82 30 77 03 * 06 00 00 00 00 36 rue de Valmy * 93108 Montreuil CEDEX www.urssaf.fr<http://www.urssaf.fr> [cid:image002.png@01D98CA1.F1ADE8F0]<https://www.youtube.com/channel/UCSjS8hOXCs5FlAnTi74rPrA>[cid:image003.png@01D98CA1.F1ADE8F0]<https://www.linkedin.com/company/acoss/>[cid:image002.png@01D98CA1.F1ADE8F0]<https://twitter.com/urssaf>[cid:image005.png@01D98CA0.D075D720]

1 0

RGW: are there any ways to limit access from user to a storage class?
by huy nguyen 22 May '23

22 May '23

There are tags for placement, so any user that has the tag is allowed to access the placement. But I can't find how to do it for storage class? Anyone have any ideal? Thanks.

1 0

Colocated RGW using same port, but different IPs
by Ondřej Kukla 22 May '23

22 May '23

Hello, As a part of performance/HW utilization testing I would like to compare how a VM with single RGW instance compares to a VM with 2 or more colocated RGWs. I’m deploying Quincy using Ansible but I’m struggling with setting the config to use the same port across all RGW instances, but to have a different IP as a beast endpoint. I would like to ask if anyone has any suggestion on this topic. Is this even supported scenario or it is better to have the instances using different ports? Any help is much appreciated. Kind regards, Ondrej

1 0

MDS migrations
by Odair M. 21 May '23

21 May '23

Hi, I'm doing a master's degree in distributed file systems, specifically CephFS in HPC, and I have some questions about the metadata migration process of MDSes that balances the MDS cluster. * Sage Weil Ph.D. thesis' and the documentation in the Ceph website say that before migrating to balance the MDS, the first step is to freeze the subtree and when the migration is complete, to unfreeze it. Doesn't it affect the performance since the subtree stays unavailable until migration is over? * Could you explain why the freezes are required at all? * Is It possible to change the migration process to migrate a copy of the inodes and then migrate inodes that have modifications after the migration? Like taking a snapshot of the subtree and migrate that, and then migrate any changes that happened after the snapshot migration is done, reducing the time spent with the subtree frozen. -- Odair M. Ditkun Jr Master's networks and distributed systems — University of State Parana — Brazil

1 0

RGW: request fields in Lua script return wrong data?
by viplanghe6＠gmail.com 21 May '23

21 May '23

Thanks for your answer. I was able to get the Lua debug log. But I think some request fields don't work: I have this lua script, for example: if Request.HTTP.StorageClass == 'COLD' then RGWDebugLog(Request.RGWOp .. " request with StorageClass: " .. Request.HTTP.StorageClass .. " Obj name: " .. Request.Object.Name .. " Obj ID: " .. Request.Object.Id .. " Size: " .. Request.Object.Size .. " MTime: " .. Request.Object.MTime ) end I put an 1kb object using boto3: s3.Object(bucket, 'test-object6').put(Body="0"*1000, ACL='public-read', StorageClass='COLD') But the size and mtime field provided wrong data? Lua INFO: put_obj request with StorageClass: COLD Obj name: test-object6 Obj ID: test-object6 Size: 0 MTime: 1970-01-01 08:00:00

3 3

Dedicated radosgw gateways
by Michal Strnad 19 May '23

19 May '23

Hi all, at Cephalocon 2023, it was mentioned several times that for service tasks such as data deletion via garbage collection or data replication in S3 via zoning, it is good to do them on dedicated radosgw gateways and not mix them with gateways used by users. How can this be achieved? How can we isolate these tasks? Will using dedicated keyrings instead of admin keys be sufficient? How do you operate this in your environment? Thx Michal

6 10

Ceph OSDs suddenly use public network for heardbeat_check
by Lee, H. (Hurng-Chun) 19 May '23

19 May '23

Hi all, We are running the Nautilus cluster. Today due to UPS work, we shut down the whole cluster. After we start the cluster, many OSDs go down and they seem to start doing the heardbeat_check using the public network. For example, we see the following logs: --- 2023-05-16 19:35:29.254 7efcd4ce7700 -1 osd.101 42916 heartbeat_check: no reply from 131.174.45.223:6825 osd.185 ever on either front or back, first ping sent 2023-05-16 19:34:48.593701 (oldest deadline 2023-05-16 19:35:08.593701) --- While I was expect the heardbeat to go through the cluster network, e.g. instead of 131.174.45.223, it should use 172.20.128.223. In fact, when we start up the cluster, we don't have DNS available to resolve the IP addresses, and for a short while, all OSDs are located in a new host called "localhost.localdomain". At that point, I fixed it by setting the static hostname using `hostnamectl set -hostname xxx`. Now we cannot bring the cluster back to healthy state. We are stuck at: --- cluster: id: 86c9bc85-b7f3-49a1-9e1f-8c9f2b31fca8 health: HEALTH_ERR 1 filesystem is degraded 1 filesystem has a failed mds daemon 1 filesystem is offline insufficient standby MDS daemons available pauserd,pausewr,noout,nobackfill,norebalance,norecover flag(s) set 88 osds down Reduced data availability: 2544 pgs inactive, 2369 pgs down, 159 pgs peering, 294 pgs stale Degraded data redundancy: 870424/2714593746 objects degraded (0.032%), 30 pgs degraded, 9 pgs undersized 8631 slow ops, oldest one blocked for 803 sec, mon.ceph- mon01 has slow ops services: mon: 3 daemons, quorum ceph-mon01,ceph-mon02,ceph-mon03 (age 2h) mgr: ceph-mon02(active, since 2h), standbys: ceph-mon01, ceph-mon03 mds: cephfs:0/1, 1 failed osd: 191 osds: 103 up (since 1.46827s), 191 in (since 4w) flags pauserd,pausewr,noout,nobackfill,norebalance,norecover data: pools: 2 pools, 2560 pgs objects: 456.13M objects, 657 TiB usage: 1.1 PiB used, 638 TiB / 1.7 PiB avail pgs: 100.000% pgs not active 870424/2714593746 objects degraded (0.032%) 2087 down 282 stale+down 129 peering 32 stale+peering 30 undersized+degraded+peered --- Any idea we could fix it and get the OSDs to use cluster network to do heartbeat checks. Any help would be highly appreciated. Thank you very much. Cheers, Hong -- Hurng-Chun (Hong) Lee, PhD ICT manager Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive Neuroimaging Radboud University Nijmegen e-mail: h.lee(a)donders.ru.nl tel: +31(0)631132518 web: http://www.ru.nl/donders/ pgp: 3AC505B2B787A8ABE2C551B1362976D838ABF09E * Mon, Tue and Thu at Trigon; Wed and Fri working from home

3 3

ls: cannot access '/cephfs': Stale file handle
by Harry G Coin 18 May '23

18 May '23

I have two autofs entries that mount the same cephfs file system to two different mountpoints. Accessing the first of the two fails with 'stale file handle'. The second works normally. Other than the name of the mount point, the lines in autofs are identical. No amount of 'umount -f' or restarting autofs resolves it. Any ideas?

2 1

User + Dev Monthly Meeting Minutes 2023-05-18
by Laura Flores 18 May '23

18 May '23

Hey all, Here are the minutes from today's meeting: Reef RC0 status? - ETA is the last week of May - Several users have volunteered to help with Reef scale testing; efforts are documented in this etherpad: https://pad.ceph.com/p/reef_scale_testing [Ben Gao] feature request: for non root user with password to install cluster with cephadm - There was work done on this previously, but it was closed in this PR: https://github.com/ceph/ceph/pull/44085 - Before, Sebastian questioned the use case of this feature and overhead - Ben's use case: Customer network security policy; currently ceph admin only supports root install - Adam King: We would be open to reviewing any patches submitted, but ultimately ssh keys are a better option. [Andras Pataki] Crush upmap fix: https://tracker.ceph.com/issues/51729 - There is a fix, but it needs to be well tested, as this is a high-traffic area of code - Andras is interested in testing the fix in a test environment; would need to upgrade the OSDs to include the crushwrapper code fix - Laura recommends against testing the fix on production clusters - Update the Tracker if you need a test build See you all next month! - Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

CLT Meeting minutes 17/05/23
by Nizamudeen A 18 May '23

18 May '23

Hey all, Ceph Quarterly announcement [Josh and Zac] One page digest that may published quarterly Planning for 1st of June, September and December Reef RC https://pad.ceph.com/p/reef_scale_testing https://pad.ceph.com/p/ceph-user-dev-monthly-minutes#L17 ETA last week of May Missing CentOS 9 Python deps Ken Dreyer has volunteered to help get the packages in epel Lab update There have been reimaging issues and kernel timeouts seen in the lab; infra team is working through fixing it Please raise infrastructure trackers for any bugs you see, as we have started to do monthly infrastructure bug scrubs Regards, Nizam

1 0

Deleting millions of objects
by Rok Jaklič 18 May '23

18 May '23

Hi, I would like to delete millions of objects in RGW instance with: mc rm --recursive --force ceph/archive/veeam but it seems it allows only 1000 (or 1002 exactly) removals per command. How can I delete/remove all objects with some prefix? Kind regards, Rok

7 9

Discussion thread for Known Pacific Performance Regressions
by Mark Nelson 17 May '23

17 May '23

Hi Everyone, This email was originally posted to dev(a)ceph.io, but Marc mentioned that he thought this would be useful to post on the user list so I'm re-posting here as well. David Orman mentioned in the CLT meeting this morning that there are a number of people on the mailing list asking about performance regressions in Pacific+ vs older releases. I want to document a couple of the bigger ones that we know about for the community's benefit. I want to be clear that Pacific does have a number of performance improvements over previous releases, and we do have tests showing improvement relative to nautilus (especially RBD on NVMe drives). Some of these regressions are going to have a bigger effect for some users than others. Having said that, let's get into them. ********** Regression #1: RocksDB Log File Recycling ********** Effects: More metadata updates to the underlying FS, higher write-amplification (Observed by Digital Ocean), Slower performance especially when the WAL device is saturated. When bluestore was created back in 2015 Sage implemented an optimization in RocksDB that allowed WAL log files to be recycled. The idea is that instead of deleting logs when they are flushed, rocksdb can simply reuse them. The benefit here is that it allows records to be written and fadatasync can be called without touching the inode for every IO. Sage did a pretty good job of explaining the benefit in the PR available here: https://github.com/facebook/rocksdb/pull/746 After much discussion, that PR was merged and received a couple of bug fixes over the years: Locking bug fix from Somnath back in 2016: https://github.com/facebook/rocksdb/pull/1313 Another bug fix from ajkr in 2020: https://github.com/facebook/rocksdb/pull/5900 In 2020, the RocksDB folks discovered there is a fundamental flaw in the way that the original PR works. It turns out that the feature to recycle log files is incompatible with RocksDB's kPointInTimeRecovery, kAbsoluteConsistency, and kTolerateCorruptedTailRecords recovery modes. One of the later PR's included a very good and concise description of the problem: "The two features are naturally incompatible. WAL recycling expects the recovery to succeed upon encountering a corrupt record at the point where new data ends and recycled data remains at the tail. However, WALRecoveryMode::kTolerateCorruptedTailRecords must fail upon encountering any such corrupt record, as it cannot differentiate between this and a real corruption, which would cause committed updates to be truncated." More background discussion on the RocksDB side available in these PRs and comments: https://github.com/facebook/rocksdb/pull/6351 https://github.com/facebook/rocksdb/pull/6351#issuecomment-672838284 https://github.com/facebook/rocksdb/pull/7252 https://github.com/facebook/rocksdb/pull/7271 On the Ceph side, there was a PR to try to re-enable the old behavior which we rejected as unsafe based on the analysis by the RocksDB folks (which we agree with): https://github.com/ceph/ceph/pull/36579 Sage also commented about a potential way forward: https://github.com/ceph/ceph/pull/36579#issuecomment-870884583 "tbh I think the best approach would be to create a new WAL file format that (1) is 4k block aligned and (2) has a header for each block that indicates the generation # for that log file (so we can see whether what we read is from a previous pass or corruption). That would be a fair bit of effort, though." On a side note, Igor tried to also disable WAL file recycling as a backport to Octopus but was thwarted by a BlueFS bug. That PR was eventually reverted leaving the old (dangerous!) behavior being left in place: https://github.com/ceph/ceph/pull/45040 https://github.com/ceph/ceph/pull/47053 The gist of it is that releases of Ceph older than Pacific are benefiting from the speed improvement of log file recycling but may be vulnerable to the issue as described above. This is likely one of the more impactful regressions that people upgrading to Pacific or later releases are seeing. Josh Baergen from Digital Ocean followed up that there is a slew of additional information on this issue in the following tracker as well: https://tracker.ceph.com/issues/58530 ********** Regression #1 Potential Fixes *********** Josh Baergen also mentioned that the write-amplification effect that was observed due to this issue is mitigated in by https://github.com/ceph/ceph/pull/48915 which was merged into 16.2.11 back in December. That however does not improve write IOPS amplification. Beyond that, we could follow Sage's idea and try to implement a new WAL file format. The risks here are that it could be a lot of work and we don't know if there is really any appetite on the RocksDB side to merge something like this upstream. My personal take is that we're already kind of abusing the RocksDB WAL for short lived PG log updates and I'm not thrilled about trying to add further code into RocksDB to try and support our use cases (though there is benefit here that goes beyond Ceph). We already maintain a custom version of RocksDB's LRU cache in our code to tie into our memory autotuning system but it would be really nice to avoid custom code like that in the future. One alternative: Igor Fedetov implemented a prototype WAL inside bluestore itself and we saw very good initial results from it with the RocksDB WAL disabled. These can be seen on slide 24 of my performance deck from Cephalocon 2023: https://www.linkedin.com/in/markhpc/overlay/experience/2113859303/multiple-… If Igor (or others) want to continue this work, I personally would be in favor of trying to move the WAL into Bluestore itself. I suspect we can make better decisions about PG log life cycles and have better BlueFS integration than what RocksDB provides us. Igor probably has a better idea of the pitfalls here though so I think we should hear out his thoughts on whether this is the right path forward. Igor also mentioned that he is continuing to work on his Bluestore WAL prototype with promising results, but that PG Log will (as expected) likely require a different solution that looks more like a specialized ring buffer. I think moving the WAL out of RocksDB is a good step toward that eventual goal. ********** Regression #2: (re-)Enabling BlueFS Buffered IO ********** Effects: Works around unexpected readahead behavior in RocksDB by utilizing underlying kernel page cache. Hurts write performance on fast devices. We're stuck between a bit of a rock and a hard place here. Over the years we have sea-sawed back and forth regarding when we should or should not use buffered IO: https://github.com/ceph/ceph/pull/11012 https://github.com/ceph/ceph/pull/11059 https://github.com/ceph/ceph/pull/18172 https://github.com/ceph/ceph/pull/20542 https://github.com/ceph/ceph/pull/34224 https://github.com/ceph/ceph/pull/38044 <-- lots of discussion here The gist of it is that there are upsides and downside to having bluefs_buffered_io=true. Direct IO is faster in some scenarios, especially more recent write tests on NVMe drives. The trade off is that RocksDB really seems to benefit from kernel buffer cache and there are other scenarios where bluefs_buffered_io is a big win. 2 years ago Adam and I did a walkthrough of the RocksDB code to try to understand the behavior regarding RocksDB readahead and we couldn't understand why it was re-reading data from the file system so often (or in the case of buffered IO the page cache!). I wrote up our walkthrough of the code here: https://github.com/ceph/ceph/pull/38044#issuecomment-790157415 ********** Regression #2 Potential Fixes ********** In a recent discussion with Mark Callaghan (of MyRocks/RocksDB performance tuning fame), he pointed out that RocksDB has an option to pre-populate the block cache with the data from SSTs created by memtable flush and that might help when O_DIRECT is used: https://github.com/facebook/rocksdb/blob/main/include/rocksdb/table.h#L600 We may want to experiment to see if this helps keep the block cache pre-populated after compaction and avoid (re)reads from the disk during iteration. We also might want to revisit this topic in general with the compact on iteration feature that was recently added and backported to pacific in 16.2.13. I'm still a little concerned however that we were seeing repeated overlapping reads for the same ranges during iteration that I would have expected to be cached by RocksDB on a previous read. Ultimately I think many of us would prefer to move entirely to directIO but there's more work to do to figure this one out. Josh Baergen provided further advice here: They have had good luck enabling buffered IO for rgw bucket index OSDs and disabling it everywhere else. This assumes that bucket indexes are on their own dedicated OSDs though, and personally I am a bit wary of hitting slow cases in RocksDB even on "regular" OSDs, but this might be something to consider as they've had good luck with this configuration for over a year. ********** Regression #3: RadosGW Coroutine and Request Timeout Changes ********** Effects: Higher RadosGW CPU usage, lower performance, especially for small object workloads Back when Pacific was released it was observed that RadosGW was showing much higher CPU usage and lower performance vs Nautilus for small (4KB) objects. It's likely that larger objects may be affected, though to a lesser degree A git bisection was performed and the results are summarized in the introduction section of the folliowing RGW performance analysis blog post here: https://ceph.io/en/news/blog/2023/reef-freeze-rgw-performance/ The bisection uncovered two primary PRs that were causing performance regression: https://github.com/ceph/ceph/pull/31580 https://github.com/ceph/ceph/pull/35355 The good news is that once those PRs were identified, the RGW team started working to improve things, especially for #35355: https://github.com/ceph/ceph/pull/43761 <-- Fixes issues introduced in #35355, backported to Pacific in 2022 *********** Regression #3 Potential Fixes ********** Quincy (and due to the backport likely Pacific) is showing significantly better behavior in recent tests due to PR #43761. The effects of #31580 are still present, but are considered a necessary trade-off. Other improvements since then may be helping, but we'll need to continue to make up the difference in other areas and start really investigating where we are spending cycles/time, especially in Reef. ********** Regression #4: Gradually slowing down OSDs ********** Effects: Significant slowdown after 1-2 weeks of OSD runtime Igor Fedetov pointed this one out in discussion earlier today: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/OXAUOK7CQXW… This one is pretty new and there is not much there yet other than perhaps low memory (and cache?) usage despite regular IO workload. Onode misses can absolutely cause performance degradation, but it's not clear yet whether this is memory related issue or something else. More investigation needed. Hopefully we'll get perf data from the users who encountered it to help diagnose what's going on here. ********** Conclusion ********** There may be other performance issues that I'm not remembering, but these are the big ones I can think of off the top of my head at the moment. Hopefully this helps clarify what's going on if people are seeing a regression, what to look for, and if they are hitting it, the why behind it. Thanks, Mark -- Best Regards, Mark Nelson Head of R&D (USA) Clyso GmbH p: +49 89 21552391 12 a: Loristraße 8 | 80335 München | Germany w: https://clyso.com | e: mark.nelson(a)clyso.com We are hiring: https://www.clyso.com/jobs/

2 2

Re: Enable Centralized Logging in Dashboard.
by Vladimir Brik 17 May '23

17 May '23

How do I create a user name and password that I could use to log in to grafana? Vlad On 11/16/22 08:42, E Taka wrote: > Thank you, Nizam. I wasn't aware that the Dashboard login is not the same > as the grafana login. Now I have accass to the logfiles. > > Am Mi., 16. Nov. 2022 um 15:06 Uhr schrieb Nizamudeen A <nia(a)redhat.com>: > >> Hi, >> >> Did you login to the grafana dashboard? For centralized logging you'll >> need to login to the >> grafana using your grafana username and password. If you do that and >> refresh the dashboard, >> I think the Loki page should be visible from the Daemon Logs page. >> >> Regards, >> Nizam >> >> On Wed, Nov 16, 2022 at 7:31 PM E Taka <0etaka0(a)gmail.com> wrote: >> >>> Ceph: 17.2.5, dockerized with Ubuntu 20.04 >>> >>> Hi all, >>> >>> I try to enable the Centralized Logging in Dashboard as described in >>> >>> >>> https://docs.ceph.com/en/quincy/cephadm/services/monitoring/#cephadm-monito… >>> >>> Logging inti files is enabled: >>> ceph config set global log_to_file true >>> ceph config set global mon_cluster_log_to_file true >>> >>> Loki is deployed at one host, promtail on every host: >>> >>> >>> service_type: loki >>> service_name: loki >>> placement: >>> hosts: >>> - ceph00 >>> --- >>> service_type: promtail >>> service_name: promtail >>> placement: >>> host_pattern: '*' >>> >>> >>> After applying the YAML above the log messages in »ceph -W cephadm« look >>> good (deploying loki+promtail and reconfiguring grafana). But the >>> Dashboard >>> "Cluster → Logs → Daemon Logs" just shows a standard grafana page without >>> any buttons for the Ceph Cluster. Its URL is https://ceph00. >>> >>> [mydomain]:3000/explore?orgId=1&left=["now-1h","now","Loki",{"refId":"A"}]&kiosk. >>> >>> Did I miss something for the centralized Logging in the Dashboard? >>> >>> Thank! >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

new install or change default registry to private registry
by farhad kh 17 May '23

17 May '23

i try deploy cluster from private registry and used this command {cephadm bootstrap ---mon-ip 10.10.128.68 --registry-url my.registry.xo --registry-username myuser1 --registry-password mypassword1 --dashboard-password-noupdate --initial-dashboard-password P@ssw0rd } even i changed section Default container images in cephadm script # Default container images ----------------------------------------------------- DEFAULT_IMAGE = ' my.registry.xo/ceph/ceph:v17' DEFAULT_IMAGE_IS_MASTER = False DEFAULT_IMAGE_RELEASE = 'quincy' DEFAULT_PROMETHEUS_IMAGE = ' my.registry.xo/ceph/prometheus:v2.33.4' DEFAULT_LOKI_IMAGE = ' my.registry.xo/ceph/loki:2.4.0' DEFAULT_PROMTAIL_IMAGE = my.registry.xo/ceph/promtail:2.4.0' DEFAULT_NODE_EXPORTER_IMAGE = ' my.registry.xo/ceph/node-exporter:v1.3.1' DEFAULT_ALERT_MANAGER_IMAGE = ' my.registry.xo/ceph/alertmanager:v0.23.0' DEFAULT_GRAFANA_IMAGE = ' my.registry.xo/ceph/ceph-grafana:8.3.5' DEFAULT_HAPROXY_IMAGE = ' my.registry.xo/ceph/haproxy:2.3' DEFAULT_KEEPALIVED_IMAGE = ' my.registry.xo/ceph//keepalived:2.1.5' DEFAULT_SNMP_GATEWAY_IMAGE = ' my.registry.xo/ceph/ snmp-notifier:v1.2.1' DEFAULT_REGISTRY = ' my.registry.xo' # normalize unqualified digests to this # ------------------------------------------------------------------------------ to my private registy but when i deploy new cluster cephadm try pull prometheus , node-exporter and ceph-grafana from quay.io . Why does it happen and what should I do to change it? Even the default registry is not changed #ceph config get mgr mgr/cephadm/default_registry docker.io Can someone guide me, what should I do?

2 1

how to enable multisite resharding feature?
by Alexander Mamonov 17 May '23

17 May '23

https://docs.ceph.com/en/latest/radosgw/multisite/#feature-resharding When I try this I get: root@ceph-m-02:~# radosgw-admin zone modify --rgw-zone=sel --enable-feature=resharding ERROR: invalid flag --enable-feature=resharding root@ceph-m-02:~# ceph version ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)

2 1

CEPH Version choice
by Tino Todino 17 May '23

17 May '23

Hi all, I've been reading through this email list for a while now, but one thing that I'm curious about is why a lot of installations out there aren't upgraded to the latest version of CEPH (Quincy). What are the main reasons for not upgrading to the latest and greatest? Thanks. Tino This E-mail is intended solely for the person or organisation to which it is addressed. It may contain privileged or confidential information and, if you are not the intended recipient, you must not copy, distribute or take any action in reliance upon it. Any views or opinions presented are solely those of the author and do not necessarily represent those of Marlan Maritime Technologies Ltd. If you have received this E-mail in error, please notify us as soon as possible and delete it from your computer. Marlan Maritime Technologies Ltd Registered in England & Wales 323 Mariners House, Norfolk Street, Liverpool. L1 0BG Company No. 08492427.

10 14

Best practice for expanding Ceph cluster
by huxiaoyu＠horebdata.cn 17 May '23

17 May '23

Dear Ceph folks, I am writing to ask for advice on best practice of expanding ceph cluster. We are running an 8-node Ceph cluster and RGW, and would like to add another 10 node, each of which have 10x 12TB HDD. The current 8-node has ca. 400TB user data. I am wondering whether to add 10 nodes at one shot and let the cluster to rebalance, or divide into 5 steps, each of which add 2 nodes and rebalance step by step? I do not know what would be the advantages or disadvantages with the one shot scheme vs 5 bataches of adding 2 nodes step-by-step. Any suggestions, experience sharing or advice are highly appreciated. thanks a lot in advance, Samuel huxiaoyu(a)horebdata.cn

4 6

CFP NOW OPEN: Ceph Days Vancouver Co-located with OpenInfra Summit
by Mike Perez 17 May '23

17 May '23

Hi everyone, I invite you to Ceph Days Vancouver on June 15, co-located with the OpenInfra Summit. Ceph Days are one-day events dedicated to multiple breakout and BoF sessions with a wide range of topics around Ceph. You can receive a nice discount to attend Ceph Days and the OpenInfra Summit if you register before May 12! https://ceph.io/en/community/events/2023/ceph-days-vancouver/ Our Call for proposals is open until May 17, and we have the following suggested topics: - Ceph operations, management, and development - New and proposed Ceph features, development status - Ceph development roadmap - Best practices - Ceph use-cases, solution architectures, and user experiences - Ceph performance and optimization - Platform Integrations - Kubernetes, OpenShift - OpenStack (Cinder, Manila, etc.) - Spark - Multi-site and multi-cluster data services - Persistent memory, ZNS SSDs, SMR HDDs, DPUs, and other new hardware technologies - Storage management, monitoring, and deployment automation - Experiences deploying and operating Ceph in production and/or at scale - Small-scale or edge deployments - Long-term, archival storage - Data compression, deduplication, and storage optimization - Developer processes, tools, challenges - Ceph testing infrastructure, tools - Ceph community issues, outreach, and project governance - Ceph documentation, training, and learner experience CFP: https://survey.zohopublic.com/zs/TVCCCQ See you all there soon! -- Mike Perez Community Manager Ceph Foundation

1 1

FW: Public Access URL returns "NoSuchBucket" when rgw_swift_account_in_url is True
by Beaman, Joshua 17 May '23

17 May '23

Thought I might forward this to the users list in case anyone else is experiencing or knows how to resolve. Thank you, Josh Beaman From: Beaman, Joshua <Joshua_Beaman(a)comcast.com> Date: Tuesday, May 16, 2023 at 3:23 PM To: dev(a)ceph.io <dev(a)ceph.io> Subject: Public Access URL returns "NoSuchBucket" when rgw_swift_account_in_url is True We have a continuing problem with Openstack Swift Public URLs returning “NoSuchBucket” even though the S3 equivalent URL works fine. This was not a problem in our Nautilus 14.2.22 clusters, but after upgrading to 16.2.9 or 16.2.10, this regression bug has returned. All 13 of our prod clusters were upgraded to 16.2.11 and still experience the problem, and now I’ve upgraded 1 prod cluster to Quincy/17.2.6 and it is still present. I’ve submitted the following bug report: https://tracker.ceph.com/issues/58019 Thanks in advance, Josh Beaman

1 0

Bucket-level redirect_zone
by Yixin Jin 16 May '23

16 May '23

Hi folks, I created a feature request ticket to call for bucket-level redirect_zone (https://tracker.ceph.com/issues/61199) which is basically an extension from zone-level redirect_zone. I found it helpful in realizing CopyObject with (x-copy-source) in multisite environments where bucket content don't exist in all zones. This feature is similar to what Matt Benjamin suggested about the concept of "bucket residence". In my own prototyping affect, redirecting feature at bucket level is fairly straightforward. Making use of it with CopyObject is trickier because I haven't found a good way to get the source object policy when RGWCopyObj::verify_permission() is called. Anyway, I will continue to explore this idea and hope to get more support of it. Thanks, Yixin

2 1

User + Dev Monthly Meeting happening this Thursday
by Laura Flores 16 May '23

16 May '23

Hi Ceph Users, The User + Dev Monthly Meeting is coming up next week on *Thursday, May 17th* *@* *14:00 UTC* (time conversions below). See meeting details at the bottom of this email. Please add any topics you'd like to discuss to the agenda: https://pad.ceph.com/p/ceph-user-dev-monthly-minutes See you there, Laura Flores Meeting link: https://meet.jit.si/ceph-user-dev-monthly Time Conversions: UTC: Wednesday, May 17, 14:00 UTC Mountain View, CA, US: Wednesday, May 17, 7:00 PDT Phoenix, AZ, US: Wednesday, May 17, 7:00 MST Denver, CO, US: Wednesday, May 17, 8:00 MDT Huntsville, AL, US: Wednesday, May 17, 9:00 CDT Raleigh, NC, US: Wednesday, May 17, 10:00 EDT London, England: Wednesday, May 17, 15:00 BST Paris, France: Wednesday, May 17, 16:00 CEST Helsinki, Finland: Wednesday, May 17, 17:00 EEST Tel Aviv, Israel: Wednesday, May 17, 17:00 IDT Pune, India: Wednesday, May 17, 19:30 IST Brisbane, Australia: Thursday, May 18, 0:00 AEST Singapore, Asia: Wednesday, May 17, 22:00 +08 Auckland, New Zealand: Thursday, May 18, 2:00 NZST -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 1

rbd mirror snapshot trash
by Eugen Block 16 May '23

16 May '23

Good morning, I would be grateful if anybody could shed some light on this, I can't reproduce it in my lab clusters so I was hoping for the community. A customer has 2 clusters with rbd mirroring (snapshots) enabled, it seems to work fine, they have regular checks and the images on the remote site are synced correctly. While investigating a different issue (could be related though) we noticed that there are quite a lot of snapshots in the trash namespace but for unkown reasons they are not purged. This is some example output from the remote site (the primary site looks similar): ---snip--- # rbd snap ls --all <pool>/<image> SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 23962608 42c2e4c3-e1ab-480f-9d29-aab5555c751b 20 GiB Tue Sep 20 20:33:18 2022 trash (.mirror.non_primary.61ef3a1e-6e5b-4147-ac62-552e4776dd70.ef345a74-2585-48ac-8e25-09b15c20c877) 96796437 24cc96d4-51c6-447e-870d-1545d8ec9308 20 GiB Tue Apr 18 12:15:28 2023 trash (.mirror.non_primary.61ef3a1e-6e5b-4147-ac62-552e4776dd70.a6ae46e4-cbb5-4264-b2e4-6b565820aa16) 110025019 .mirror.non_primary.61ef3a1e-6e5b-4147-ac62-552e4776dd70.0fc73e88-6671-4ba8-a118-8fcbcd422c22 20 GiB Mon May 15 16:27:39 2023 mirror (non-primary peer_uuids:[] 18ac37b9-a6c6-4f64-ba43-a1f0d02b3f96:115986586 copied) ---snip--- They're on latest Octopus and use all config defaults. There are more than 500 images in that pool with a snapshot schedule of 3 minutes for each images. A couple of questions arised and hopefully at least some of them can be answered: - Why are there snapshots in the trash namespace from September 2022, how can they be cleaned up and why aren't they cleaned up automatically? Not all images have trash entries though. - Could disabling mirroring for those images help getting rid of the trash, then enable mirroring again? - Is the default osd_max_trimming_pgs = 2 a bottleneck here? They have a week old sst files in the store.db, leading to long startup duration for MONs after reboot/restart. Would increasing that value help getting rid of the trash entries and maybe also trim the mon store? - Regarding the general snapshot mirroring procedure, the default of rbd_mirroring_max_mirroring_snapshots is 5, but I assume that the number of active snapshots would only grow in case of a disruption between those sites, correct? If the sync works there's no need to keep 5 snapshots, right? I'm still looking into these things myself but I'd appreciate anyone chiming in here. Thanks! Eugen

2 2

Telemetry and Redmine sync
by Yaarit Hatuka 16 May '23

16 May '23

Hi everyone, Over this weekend we will run a sync between telemetry crashes and Redmine tracker issues. This might affect your inbox, depending on your Redmine email notification setup. You can set up filters for these emails to skip your inbox. Thanks, Yaarit

2 1

cephadm and remoto package
by Shashi Dahal 15 May '23

15 May '23

Hi, I followed this documentation: https://docs.ceph.com/en/pacific/cephadm/adoption/ This is the error I get when trying to enable cephadm. ceph mgr module enable cephadm Error ENOENT: module 'cephadm' reports that it cannot run on the active manager daemon: loading remoto library:No module named 'remoto' (pass --force to force enablement) When I import remoto, it imports just fine. OS is ubuntu 20.04 focal -- Cheers, Shashi

1 0

Quincy Ceph-orchestrator and multipath SAS
by Deep Dish 15 May '23

15 May '23

Hello, I have a few hosts about to add into a cluster that have a multipath storage config for SAS devices. Is this supported on Quincy, and how would ceph-orchestrator and / or ceph-volume handle multipath storage? Here's a snip of lsblk output of a host in question: # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS ... sdc 8:32 0 9.1T 0 disk └─mpathh 253:4 0 9.1T 0 mpath sdd 8:48 0 9.1T 0 disk └─mpathi 253:5 0 9.1T 0 mpath sde 8:64 0 7.3T 0 disk └─mpathj 253:6 0 7.3T 0 mpath sdf 8:80 0 7.3T 0 disk └─mpathl 253:7 0 7.3T 0 mpath sdg 8:96 0 7.2T 0 disk └─mpathk 253:8 0 7.2T 0 mpath sdh 8:112 0 7.3T 0 disk └─mpathe 253:9 0 7.3T 0 mpath sdi 8:128 0 7.3T 0 disk └─mpathg 253:10 0 7.3T 0 mpath sdj 8:144 0 7.3T 0 disk └─mpathf 253:11 0 7.3T 0 mpath sdk 8:160 0 7.3T 0 disk └─mpathc 253:12 0 7.3T 0 mpath sdl 8:176 0 7.3T 0 disk └─mpathd 253:13 0 7.3T 0 mpath ...

2 1

Octopus on Ubuntu 20.04.6 LTS with kernel 5
by Szabo, Istvan (Agoda) 15 May '23

15 May '23

Hi, In octopus documentation we can see kernel 4 as recommended, however we've changed our test cluster yesterday from centos 7 / 8 to Ubuntu 20.04.6 LTS with kernel 5.4.0-148 and seems working, I just want to make sure before I move to prod there isn't any caveats. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

4 5

Upgrade Ceph cluster + radosgw from 14.2.18 to latest 15
by viplanghe6＠gmail.com 15 May '23

15 May '23

Hi, I want to upgrade my old Ceph cluster + Radosgw from v14 to v15. But I'm not using cephadm and I'm not sure how to limit errors as much as possible during the upgrade process? Here is my upgrade steps: Firstly, upgrade from 14.2.18 to 14.2.22 (latest nautilus version) Then, upgrade it from 14.2.22 to 15.2.17 (latest octopus version) service restart order: - restart monitors (sleep ~10) - restart managers (sleep ~5) - restart metadata OSDs (sleep ~30) - restart data OSDs (sleep ~30) - restart radosgw (sleep ~20) Is there anything wrong with these steps? Does 15.2.17 is the most stable version of Octopus? Thanks.

3 7

RGW: Using Lua script to monitor storage class put operations
by viplanghe6＠gmail.com 14 May '23

14 May '23

I have a lua script that read Storageclass header of any put request (as I understand): local function isempty(input) return input == nil or input == '' end if Request.RGWOp == 'put_obj' then RGWDebugLog("Put_Obj with StorageClass: " .. Request.HTTP.StorageClass ) end Then apply the script: radosgw-admin script put --infile=storage_class.lua --context=preRequest Then put an object using boto3: s3.Object(bucket, 'test-object').put(Body="0"*100, ACL='public-read', StorageClass='COLD') But I can't find any log about it in radosgw log? grep -i Lua /var/log/ceph/ceph-client.rgw.ceph1.log Anyone have any ideal? Thanks

2 1

Unable to Retrieve Usage Logs in Ceph RGW with Boto3
by Ramin Najjarbashi 13 May '23

13 May '23

Hi I am currently working on retrieving user usage and usage logs from Ceph RGW using Boto3. While I am able to successfully obtain user usage and all bucket usage with the following model and code: ```python params = { "start-date": "2023-05-13T00:00:00Z", "end-date": "2023-05-13T23:59:59Z", } print(json.dumps(s3_client.get_usage_stats(**params), indent=2)) ``` ```json { "version": 1.0, "merge": { "operations": { "GetUsageStats": { "name": "GetUsageStats", "http": { "method": "GET", "requestUri": "/?usage&start-date={start-date}&end-date={end-date}", "responseCode": 200 }, "input": { "shape": "GetUsageLogInput" }, "output": { "shape": "GetUsageStatsOutput" }, "documentationUrl": " https://docs.ceph.com/docs/master/radosgw/s3/serviceops#get-usage-stats", "documentation": "Get usage stats for the user" } }, "shapes": { "GetUsageLogInput": { "type": "structure", "members": { "start-date": { "shape": "Timestamp", "location": "uri", "documentation": "The start date and time for the log entries to retrieve." }, "end-date": { "shape": "Timestamp", "location": "uri", "documentation": "The end date and time for the log entries to retrieve." } }, "documentation": "Input shape for the GetUsageLog operation." }, "Timestamp": { "type": "string", "documentation": "A timestamp in ISO 8601 format." }, "GetUsageStatsOutput": { "type": "structure", "members": { "Summary": { "shape": "UsageStatsSummary", "documentation": "Summary of usage stats." }, "CapacityUsed": { "shape": "CapacityUsedUsage", "documentation": "Usage stats for capacity used." } } }, "CapacityUsedUsage": { "type": "structure", "members": { "User": { "shape": "UserUsageEntries", "documentation": "Usage stats for the user." } } }, "UserUsageEntries": { "type": "structure", "members": { "Buckets": { "shape": "BucketsUsageEntries", "documentation": "Usage stats for buckets." } } }, "BucketsUsageEntries": { "type": "structure", "members": { "Entry": { "shape": "UsageEntries", "documentation": "List of usage entries for buckets." } } }, "UsageEntries": { "type": "list", "member": { "shape": "BucketEntry" }, "documentation": "List of usage entries." }, "BucketEntry": { "type": "structure", "members": { "Bucket": { "shape": "String", "documentation": "Name of the bucket." }, "Bytes_Rounded": { "shape": "Bytes_Rounded", "documentation": "Number of bytes (rounded)." }, "Bytes": { "shape": "Bytes", "documentation": "Number of bytes." } } }, "UsageStatsSummary": { "type": "structure", "members": { "QuotaMaxBytes": { "shape": "QuotaMaxBytes", "documentation": "Maximum quota in bytes." }, "QuotaMaxBuckets": { "shape": "QuotaMaxBuckets", "documentation": "Maximum quota for buckets." }, "QuotaMaxObjCount": { "shape": "QuotaMaxObjCount", "documentation": "Maximum quota for object count." }, "QuotaMaxBytesPerBucket": { "shape": "QuotaMaxBytesPerBucket", "documentation": "Maximum quota in bytes per bucket." }, "QuotaMaxObjCountPerBucket": { "shape": "QuotaMaxObjCountPerBucket", "documentation": "Maximum quota for object count per bucket." }, "TotalBytes": { "shape": "TotalBytes", "documentation": "Total number of bytes." }, "TotalBytesRounded": { "shape": "TotalBytesRounded", "documentation": "Total number of bytes (rounded)." }, "TotalEntries": { "shape": "TotalEntries", "documentation": "Total number of entries." } } }, "BytesReceived": { "type": "integer", "documentation": "Number of bytes received." }, "QuotaMaxBytes": { "type": "integer", "documentation": "Maximum quota in bytes." }, "QuotaMaxBuckets": { "type": "integer", "documentation": "Maximum quota for buckets." }, "QuotaMaxObjCount": { "type": "integer", "documentation": "Maximum quota for object count." }, "QuotaMaxBytesPerBucket": { "type": "integer", "documentation": "Maximum quota in bytes per bucket." }, "QuotaMaxObjCountPerBucket": { "type": "integer", "documentation": "Maximum quota for object count per bucket." }, "TotalBytesRounded": { "type": "integer", "documentation": "Total number of bytes (rounded)." }, "TotalBytes": { "type": "integer", "documentation": "Total number of bytes." }, "TotalEntries": { "type": "integer", "documentation": "Total number of entries." }, "String": { "type": "string", "documentation": "A string value." }, "Bytes_Rounded": { "type": "integer", "documentation": "Number of bytes (rounded)." }, "Bytes": { "type": "integer", "documentation": "Number of bytes." } }, "documentation": "Model for retrieving usage statistics." } } ``` and this is my output: ``` "Summary": { "QuotaMaxBytes": -1, "QuotaMaxBuckets": 1000, "QuotaMaxObjCount": -1, "QuotaMaxBytesPerBucket": -1, "QuotaMaxObjCountPerBucket": -1, "TotalBytes": 52696, "TotalBytesRounded": 53248, "TotalEntries": 1 }, "CapacityUsed": { "User": { "Buckets": { "Entry": [ { "Bucket": "opa-test", "Bytes_Rounded": 53248, "Bytes": 52696 }, { "Bucket": "ramfrom", "Bytes_Rounded": 0, "Bytes": 0 } } } } } ``` The issue I am facing is that the usage logs are not being displayed. Upon investigating the code in the send_response() function of the RGWGetUsage_ObjStore_S3 class, I found the following conditions: ```cpp if (show_log_sum) { formatter->open_array_section("Summary"); map<string, rgw_usage_log_entry>::iterator siter; for (siter = summary_map.begin(); siter != summary_map.end(); ++siter) { const rgw_usage_log_entry& entry = siter->second; formatter->open_object_section("User"); formatter->dump_string("User", siter->first); dump_usage_categories_info(formatter, entry, &categories); rgw_usage_data total_usage; entry.sum(total_usage, categories); formatter->open_object_section("Total"); encode_json("BytesSent", total_usage.bytes_sent, formatter); encode_json("BytesReceived", total_usage.bytes_received, formatter); encode_json("Ops", total_usage.ops, formatter); encode_json("SuccessfulOps", total_usage.successful_ops, formatter); formatter->close_section(); // total formatter->close_section(); // user } ``` According to the code, the show_log_sum parameter must be true to display the summary log entries, and I have confirmed that it is indeed set to true. Additionally, the show_log_entries parameter must also be true to retrieve the usage log entries, and I have verified that it is set to true as well. I am using Ceph version 16.2.12, and I have set the RGW debug level to 99. Here is a snippet of the log output: ```log 2023-05-13T20:39:17.111+0330 7f346da3d700 1 ====== starting new request req= 0x7f3504a0e620 ===== 2023-05-13T20:39:17.147+0330 7f346da3d700 2 req 11854699226730900750 0.035999894s initializing for trans_id = tx00000a4845a234864ad0e-00645fc43d-fb27-default 2023-05-13T20:39:17.283+0330 7f346da3d700 10 req 11854699226730900750 0.171999499s rgw api priority: s3=8 s3website=7 2023-05-13T20:39:17.287+0330 7f346da3d700 10 req 11854699226730900750 0.175999492s host=s3.abrak.stage.test.com 2023-05-13T20:39:17.291+0330 7f346da3d700 20 req 11854699226730900750 0.179999471s subdomain= domain=s3.abrak.stage.test.com in_hosted_domain=1 in_hosted_domain_s3website=0 2023-05-13T20:39:17.291+0330 7f346da3d700 20 req 11854699226730900750 0.179999471s final domain/bucket subdomain= domain=s3.abrak.stage.test.com in_hosted_domain=1 in_hosted_domain_s3website=0 s->info.domain=s3.abrak. stage.test.com s->info.request_uri=/ 2023-05-13T20:39:17.315+0330 7f346da3d700 10 req 11854699226730900750 0.207999393s meta>> HTTP_X_AMZ_CONTENT_SHA256 2023-05-13T20:39:17.319+0330 7f346da3d700 10 req 11854699226730900750 0.207999393s meta>> HTTP_X_AMZ_DATE 2023-05-13T20:39:17.319+0330 7f346da3d700 10 req 11854699226730900750 0.207999393s x>> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-05-13T20:39:17.319+0330 7f346da3d700 10 req 11854699226730900750 0.207999393s x>> x-amz-date:20230513T170916Z 2023-05-13T20:39:17.347+0330 7f346da3d700 10 req 11854699226730900750 0.235999316s name: usage val: 2023-05-13T20:39:17.347+0330 7f346da3d700 10 req 11854699226730900750 0.235999316s name: start-date val: 2023-05-13T00:00:00Z 2023-05-13T20:39:17.347+0330 7f346da3d700 10 req 11854699226730900750 0.235999316s name: end-date val: 2023-05-13T23:59:59Z 2023-05-13T20:39:17.355+0330 7f346da3d700 20 req 11854699226730900750 0.243999273s get_handler handler=26RGWHandler_REST_Service_S3 2023-05-13T20:39:17.383+0330 7f346da3d700 10 req 11854699226730900750 0.271999210s handler=26RGWHandler_REST_Service_S3 2023-05-13T20:39:17.383+0330 7f346da3d700 2 req 11854699226730900750 0.271999210s getting op 0 2023-05-13T20:39:17.423+0330 7f346da3d700 10 req 11854699226730900750 0.311999112s s3:get_self_usage scheduling with throttler client=3 cost=1 2023-05-13T20:39:17.431+0330 7f346da3d700 10 req 11854699226730900750 0.319999069s s3:get_self_usage op=23RGWGetUsage_ObjStore_S3 2023-05-13T20:39:17.431+0330 7f346da3d700 2 req 11854699226730900750 0.319999069s s3:get_self_usage verifying requester 2023-05-13T20:39:17.435+0330 7f346da3d700 20 req 11854699226730900750 0.323999047s s3:get_self_usage rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy 2023-05-13T20:39:17.447+0330 7f346da3d700 20 req 11854699226730900750 0.335999012s s3:get_self_usage rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3:: S3AnonymousEngine 2023-05-13T20:39:17.451+0330 7f346da3d700 20 req 11854699226730900750 0.339999020s s3:get_self_usage rgw::auth::s3::S3AnonymousEngine denied with reason=-1 2023-05-13T20:39:17.451+0330 7f346da3d700 20 req 11854699226730900750 0.339999020s s3:get_self_usage rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3:: LocalEngine 2023-05-13T20:39:17.455+0330 7f346da3d700 10 req 11854699226730900750 0.343998998s v4 signature format = 2e4a95626b3633439d13f5769d6a0be142c30823192837c3599fbb3f758d7c40 2023-05-13T20:39:17.463+0330 7f346da3d700 10 req 11854699226730900750 0.351998985s v4 credential format = OSDDH3XY9FPGHJ07O7JP/20230513/us-east-1/s3/ aws4_request 2023-05-13T20:39:17.463+0330 7f346da3d700 10 req 11854699226730900750 0.351998985s access key id = OSDDH3XY9FPGHJ07O7JP 2023-05-13T20:39:17.463+0330 7f346da3d700 10 req 11854699226730900750 0.351998985s credential scope = 20230513/us-east-1/s3/aws4_request 2023-05-13T20:39:17.471+0330 7f346da3d700 10 req 11854699226730900750 0.359998941s canonical headers format = host:s3.abrak.stage.test.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20230513T170916Z 2023-05-13T20:39:17.479+0330 7f346da3d700 10 req 11854699226730900750 0.367998898s payload request hash = e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-05-13T20:39:17.483+0330 7f346da3d700 10 req 11854699226730900750 0.371998906s canonical request = GET / end-date=2023-05-13T23%3A59%3A59Z&start-date=2023-05-13T00%3A00%3A00Z&usage= host:s3.abrak.stage.test.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20230513T170916Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-05-13T20:39:17.483+0330 7f346da3d700 10 req 11854699226730900750 0.371998906s canonical request hash = 060e8a249e9f3708393aa7d2cfde262976bdd4d10216be12ca74ae7961e3994f 2023-05-13T20:39:17.483+0330 7f346da3d700 10 req 11854699226730900750 0.371998906s string to sign = AWS4-HMAC-SHA256 20230513T170916Z 20230513/us-east-1/s3/aws4_request 060e8a249e9f3708393aa7d2cfde262976bdd4d10216be12ca74ae7961e3994f 2023-05-13T20:39:17.511+0330 7f346da3d700 10 req 11854699226730900750 0.399998844s date_k = fe25d3b269a34fa9c86caf0b35d6a46791ddd7fbcd0650a212fb7528a2dce06d 2023-05-13T20:39:17.511+0330 7f346da3d700 10 req 11854699226730900750 0.399998844s region_k = 03b05d2d23e0b150ae09d8baf554e8a8bc34887951f47b5925dead931c7faeaa 2023-05-13T20:39:17.511+0330 7f346da3d700 10 req 11854699226730900750 0.399998844s service_k = d10551f64ed49675cde1f1b609ee5b54b513b077e2c00198ade53b689dee17df 2023-05-13T20:39:17.511+0330 7f346da3d700 10 req 11854699226730900750 0.399998844s signing_k = c35cb1ca41d5cc1c5a49429c50afe3c840d0b67e81df0faf8122d979294098e7 2023-05-13T20:39:17.511+0330 7f346da3d700 10 req 11854699226730900750 0.399998844s generated signature = 2e4a95626b3633439d13f5769d6a0be142c30823192837c3599fbb3f758d7c40 2023-05-13T20:39:17.511+0330 7f346da3d700 15 req 11854699226730900750 0.399998844s s3:get_self_usage string_to_sign=AWS4-HMAC-SHA256 20230513T170916Z 20230513/us-east-1/s3/aws4_request 060e8a249e9f3708393aa7d2cfde262976bdd4d10216be12ca74ae7961e3994f 2023-05-13T20:39:17.511+0330 7f346da3d700 15 req 11854699226730900750 0.399998844s s3:get_self_usage server signature= 2e4a95626b3633439d13f5769d6a0be142c30823192837c3599fbb3f758d7c40 2023-05-13T20:39:17.511+0330 7f346da3d700 15 req 11854699226730900750 0.399998844s s3:get_self_usage client signature= 2e4a95626b3633439d13f5769d6a0be142c30823192837c3599fbb3f758d7c40 2023-05-13T20:39:17.511+0330 7f346da3d700 15 req 11854699226730900750 0.399998844s s3:get_self_usage compare=0 2023-05-13T20:39:17.519+0330 7f346da3d700 20 req 11854699226730900750 0.407998830s s3:get_self_usage rgw::auth::s3::LocalEngine granted access 2023-05-13T20:39:17.519+0330 7f346da3d700 20 req 11854699226730900750 0.407998830s s3:get_self_usage rgw::auth::s3::AWSAuthStrategy granted access 2023-05-13T20:39:17.519+0330 7f346da3d700 2 req 11854699226730900750 0.407998830s s3:get_self_usage normalizing buckets and tenants 2023-05-13T20:39:17.519+0330 7f346da3d700 10 req 11854699226730900750 0.407998830s s->object=<NULL> s->bucket= 2023-05-13T20:39:17.519+0330 7f346da3d700 2 req 11854699226730900750 0.407998830s s3:get_self_usage init permissions 2023-05-13T20:39:17.535+0330 7f346da3d700 20 req 11854699226730900750 0.423998743s s3:get_self_usage get_system_obj_state: rctx=0x7f3504a0ca28 obj=default.rgw. meta:users.uid:development state=0x56448667e0a0 s->prefetch_data=0 2023-05-13T20:39:17.535+0330 7f346da3d700 10 req 11854699226730900750 0.423998743s s3:get_self_usage cache get: name=default.rgw.meta+users.uid+development : hit (requested=0x6, cached=0x17) 2023-05-13T20:39:17.539+0330 7f346da3d700 20 req 11854699226730900750 0.427998751s s3:get_self_usage get_system_obj_state: s->obj_tag was set empty 2023-05-13T20:39:17.539+0330 7f346da3d700 20 req 11854699226730900750 0.427998751s s3:get_self_usage Read xattr: user.rgw.idtag 2023-05-13T20:39:17.539+0330 7f346da3d700 10 req 11854699226730900750 0.427998751s s3:get_self_usage cache get: name=default.rgw.meta+users.uid+development : hit (requested=0x3, cached=0x17) 2023-05-13T20:39:17.679+0330 7f346da3d700 2 req 11854699226730900750 0.567998350s s3:get_self_usage recalculating target 2023-05-13T20:39:17.679+0330 7f346da3d700 2 req 11854699226730900750 0.567998350s s3:get_self_usage reading permissions 2023-05-13T20:39:17.687+0330 7f346da3d700 2 req 11854699226730900750 0.575998306s s3:get_self_usage init op 2023-05-13T20:39:17.687+0330 7f346da3d700 2 req 11854699226730900750 0.575998306s s3:get_self_usage verifying op mask 2023-05-13T20:39:17.695+0330 7f346da3d700 20 req 11854699226730900750 0.583998263s s3:get_self_usage required_mask= 1 user.op_mask=7 2023-05-13T20:39:17.695+0330 7f346da3d700 2 req 11854699226730900750 0.583998263s s3:get_self_usage verifying op permissions 2023-05-13T20:39:17.695+0330 7f346da3d700 2 req 11854699226730900750 0.583998263s s3:get_self_usage verifying op params 2023-05-13T20:39:17.695+0330 7f346da3d700 2 req 11854699226730900750 0.583998263s s3:get_self_usage pre-executing 2023-05-13T20:39:17.695+0330 7f346da3d700 2 req 11854699226730900750 0.583998263s s3:get_self_usage executing 2023-05-13T20:39:21.023+0330 7f343c1da700 2 req 11854699226730900750 3.911988497s s3:get_self_usage completing 2023-05-13T20:39:21.067+0330 7f343c1da700 30 AccountingFilter::send_status: e=0, sent=17, total=0 2023-05-13T20:39:21.071+0330 7f343c1da700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2023-05-13T20:39:21.075+0330 7f343c1da700 30 AccountingFilter:: send_chunked_transfer_encoding: e=0, sent=28, total=0 2023-05-13T20:39:21.079+0330 7f343c1da700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2023-05-13T20:39:21.107+0330 7f343b1d8700 30 AccountingFilter:: complete_header: e=0, sent=156, total=0 2023-05-13T20:39:21.107+0330 7f343b1d8700 30 AccountingFilter::set_account: e=1 2023-05-13T20:39:21.199+0330 7f3457a11700 30 AccountingFilter::send_body: e= 1, sent=4051, total=0 2023-05-13T20:39:21.199+0330 7f3457a11700 30 AccountingFilter:: complete_request: e=1, sent=5, total=4051 2023-05-13T20:39:21.199+0330 7f3457a11700 30 req 11854699226730900750 4.087987900s log_usage: bucket_name= tenant=, bytes_sent=4056, bytes_received=0, success= 1 2023-05-13T20:39:21.199+0330 7f3457a11700 2 req 11854699226730900750 4.087987900s s3:get_self_usage op status=0 2023-05-13T20:39:21.199+0330 7f3457a11700 2 req 11854699226730900750 4.087987900s s3:get_self_usage http status=200 2023-05-13T20:39:21.219+0330 7f3457a11700 1 ====== req done req= 0x7f3504a0e620 op status=0 http_status=200 latency=4.103988171s ====== 2023-05-13T20:39:21.227+0330 7f3457a11700 1 beast: 0x7f3504a0e620: 192.168. 200.1 - development [13/May/2023:20:39:17.059 +0330] "GET /?usage&start-date=2023-05-13T00%3A00%3A00Z&end-date=2023-05-13T23%3A59%3A59Z HTTP/1.0" 200 4056 - "Boto3/1.26.131 Python/3.10.11 Darwin/22.4.0 Botocore/1.29.131" - latency=4.103988171s ``` I would greatly appreciate any guidance or insights into why I am unable to retrieve the usage logs. How can I ensure that the usage logs are included in the response? Thank you for your assistance! P.S. I have also verified the response using tools like Wireshark and tcpdump, and I can confirm that the responses received are consistent with the output I provided earlier.

1 0

MDS "newly corrupt dentry" after patch version upgrade
by Janek Bevendorff 12 May '23

12 May '23

Hi, After a patch version upgrade from 16.2.10 to 16.2.12, our rank 0 MDS fails start start. After replaying the journal, it just crashes with [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/storage [2,head] auth (dversion lock) Immediately after the upgrade, I had it running shortly, but then it decided to crash for unknown reasons and I cannot get it back up. We have five ranks in total, the other four seem to be fine. I backed up the journal and tried to run cephfs-journal-tool --rank=cephfs.storage:0 event recover_dentries summary, but it never finishes only eats up a lot of RAM. I stopped it after an hour and 50GB RAM. Resetting the journal makes the MDS crash with a missing inode error on another top-level directory, so I re-imported the backed-up journal. Is there any way to recover from this without rebuilding the whole file system? Thanks Janek Here's the full crash log: May 02 16:16:53 xxx077 ceph-mds[3047358]: -29> 2023-05-02T16:16:52.761+0200 7f51f878b700 1 mds.0.1711712 Finished replaying journal May 02 16:16:53 xxx077 ceph-mds[3047358]: -28> 2023-05-02T16:16:52.761+0200 7f51f878b700 1 mds.0.1711712 making mds journal writeable May 02 16:16:53 xxx077 ceph-mds[3047358]: -27> 2023-05-02T16:16:52.761+0200 7f51f878b700 1 mds.0.journaler.mdlog(ro) set_writeable May 02 16:16:53 xxx077 ceph-mds[3047358]: -26> 2023-05-02T16:16:52.761+0200 7f51f878b700 2 mds.0.1711712 i am not alone, moving to state resolve May 02 16:16:53 xxx077 ceph-mds[3047358]: -25> 2023-05-02T16:16:52.761+0200 7f51f878b700 3 mds.0.1711712 request_state up:resolve May 02 16:16:53 xxx077 ceph-mds[3047358]: -24> 2023-05-02T16:16:52.761+0200 7f51f878b700 5 mds.beacon.xxx077 set_want_state: up:replay -> up:resolve May 02 16:16:53 xxx077 ceph-mds[3047358]: -23> 2023-05-02T16:16:52.761+0200 7f51f878b700 5 mds.beacon.xxx077 Sending beacon up:resolve seq 15 May 02 16:16:53 xxx077 ceph-mds[3047358]: -22> 2023-05-02T16:16:52.761+0200 7f51f878b700 10 monclient: _send_mon_message to mon.xxx056 at v2:141.54.133.56:3300/0 May 02 16:16:53 xxx077 ceph-mds[3047358]: -21> 2023-05-02T16:16:53.113+0200 7f51fef98700 10 monclient: tick May 02 16:16:53 xxx077 ceph-mds[3047358]: -20> 2023-05-02T16:16:53.113+0200 7f51fef98700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-05-02T16:16:23.118186+0200) May 02 16:16:53 xxx077 ceph-mds[3047358]: -19> 2023-05-02T16:16:53.373+0200 7f51fff9a700 1 mds.xxx077 Updating MDS map to version 1711713 from mon.1 May 02 16:16:53 xxx077 ceph-mds[3047358]: -18> 2023-05-02T16:16:53.373+0200 7f51fff9a700 1 mds.0.1711712 handle_mds_map i am now mds.0.1711712 May 02 16:16:53 xxx077 ceph-mds[3047358]: -17> 2023-05-02T16:16:53.373+0200 7f51fff9a700 1 mds.0.1711712 handle_mds_map state change up:replay --> up:resolve May 02 16:16:53 xxx077 ceph-mds[3047358]: -16> 2023-05-02T16:16:53.373+0200 7f51fff9a700 1 mds.0.1711712 resolve_start May 02 16:16:53 xxx077 ceph-mds[3047358]: -15> 2023-05-02T16:16:53.373+0200 7f51fff9a700 1 mds.0.1711712 reopen_log May 02 16:16:53 xxx077 ceph-mds[3047358]: -14> 2023-05-02T16:16:53.373+0200 7f51fff9a700 1 mds.0.1711712 recovery set is 1,2,3,4 May 02 16:16:53 xxx077 ceph-mds[3047358]: -13> 2023-05-02T16:16:53.373+0200 7f51fff9a700 1 mds.0.1711712 recovery set is 1,2,3,4 May 02 16:16:53 xxx077 ceph-mds[3047358]: -12> 2023-05-02T16:16:53.373+0200 7f5202fa0700 10 monclient: get_auth_request con 0x5574fe74c400 auth_method 0 May 02 16:16:53 xxx077 ceph-mds[3047358]: -11> 2023-05-02T16:16:53.373+0200 7f52037a1700 10 monclient: get_auth_request con 0x5574fe40fc00 auth_method 0 May 02 16:16:53 xxx077 ceph-mds[3047358]: -10> 2023-05-02T16:16:53.373+0200 7f520279f700 10 monclient: get_auth_request con 0x5574f932fc00 auth_method 0 May 02 16:16:53 xxx077 ceph-mds[3047358]: -9> 2023-05-02T16:16:53.373+0200 7f520279f700 10 monclient: get_auth_request con 0x5574ffce2000 auth_method 0 May 02 16:16:53 xxx077 ceph-mds[3047358]: -8> 2023-05-02T16:16:53.377+0200 7f5202fa0700 5 mds.beacon.xxx077 received beacon reply up:resolve seq 15 rtt 0.616008 May 02 16:16:53 xxx077 ceph-mds[3047358]: -7> 2023-05-02T16:16:53.393+0200 7f51fff9a700 5 mds.xxx077 handle_mds_map old map epoch 1711713 <= 1711713, discarding May 02 16:16:53 xxx077 ceph-mds[3047358]: -6> 2023-05-02T16:16:53.393+0200 7f51fff9a700 5 mds.xxx077 handle_mds_map old map epoch 1711713 <= 1711713, discarding May 02 16:16:53 xxx077 ceph-mds[3047358]: -5> 2023-05-02T16:16:53.393+0200 7f51fff9a700 5 mds.xxx077 handle_mds_map old map epoch 1711713 <= 1711713, discarding May 02 16:16:53 xxx077 ceph-mds[3047358]: -4> 2023-05-02T16:16:53.393+0200 7f51fff9a700 5 mds.xxx077 handle_mds_map old map epoch 1711713 <= 1711713, discarding May 02 16:16:53 xxx077 ceph-mds[3047358]: -3> 2023-05-02T16:16:53.545+0200 7f51fff9a700 -1 mds.0.cache.den(0x1 storage) newly corrupt dentry to be committed: [dentry #0x1/storage [2,head] auth (dversion lock) v=78956500 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x5574f932db80] May 02 16:16:53 xxx077 ceph-mds[3047358]: -2> 2023-05-02T16:16:53.545+0200 7f51fff9a700 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/storage [2,head] auth (dversion lock) v=78956500 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x5574f932db80] May 02 16:16:53 xxx077 ceph-mds[3047358]: -1> 2023-05-02T16:16:53.549+0200 7f51fff9a700 -1 /build/ceph-16.2.12/src/mds/CDentry.cc: In function 'bool CDentry::check_corruption(bool)' thread 7f51fff9a700 time 2023-05-02T16:16:53.549536+0200 /build/ceph-16.2.12/src/mds/CDentry.cc: 697: ceph_abort_msg("abort() called") ceph version 16.2.12 (5a2d516ce4b134bfafc80c4274532ac0d56fc1e2) pacific (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe0) [0x7f52054c3495] 2: (CDentry::check_corruption(bool)+0x86b) [0x5574f7e3a91b] 3: (EMetaBlob::add_dir_context(CDir*, int)+0x507) [0x5574f7f9f9d7] 4: (MDCache::create_subtree_map()+0x13e1) [0x5574f7d20dc1] 5: (MDLog::_journal_segment_subtree_map(MDSContext*)+0x4d) [0x5574f7f2949d] 6: (MDLog::_submit_entry(LogEvent*, MDSLogContextBase*)+0x206) [0x5574f7f29866] 7: (MDCache::log_leader_commit(metareqid_t)+0x277) [0x5574f7ccc3c7] 8: (MDCache::finish_committed_leaders()+0x87) [0x5574f7ccd0d7] 9: (MDCache::maybe_resolve_finish()+0x78) [0x5574f7d38358] 10: (MDCache::handle_resolve(boost::intrusive_ptr<MMDSResolve const> const&)+0x1e02) [0x5574f7d454c2] 11: (MDCache::dispatch(boost::intrusive_ptr<Message const> const&)+0x144) [0x5574f7d47bd4] 12: (MDSRank::handle_message(boost::intrusive_ptr<Message const> const&)+0x733) [0x5574f7bac793] 13: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x643) [0x5574f7bcacb3] 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x5c) [0x5574f7bcb34c] 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1d6) [0x5574f7b9f226] 16: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x460) [0x7f5205714020] 17: (DispatchQueue::entry()+0x58f) [0x7f52057118bf] 18: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f52057df261] 19: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f5205205609] 20: clone() May 02 16:16:53 xxx077 ceph-mds[3047358]: 0> 2023-05-02T16:16:53.553+0200 7f51fff9a700 -1 *** Caught signal (Aborted) ** in thread 7f51fff9a700 thread_name:ms_dispatch ceph version 16.2.12 (5a2d516ce4b134bfafc80c4274532ac0d56fc1e2) pacific (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f5205211420] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1af) [0x7f52054c3564] 5: (CDentry::check_corruption(bool)+0x86b) [0x5574f7e3a91b] 6: (EMetaBlob::add_dir_context(CDir*, int)+0x507) [0x5574f7f9f9d7] 7: (MDCache::create_subtree_map()+0x13e1) [0x5574f7d20dc1] 8: (MDLog::_journal_segment_subtree_map(MDSContext*)+0x4d) [0x5574f7f2949d] 9: (MDLog::_submit_entry(LogEvent*, MDSLogContextBase*)+0x206) [0x5574f7f29866] 10: (MDCache::log_leader_commit(metareqid_t)+0x277) [0x5574f7ccc3c7] 11: (MDCache::finish_committed_leaders()+0x87) [0x5574f7ccd0d7] 12: (MDCache::maybe_resolve_finish()+0x78) [0x5574f7d38358] 13: (MDCache::handle_resolve(boost::intrusive_ptr<MMDSResolve const> const&)+0x1e02) [0x5574f7d454c2] 14: (MDCache::dispatch(boost::intrusive_ptr<Message const> const&)+0x144) [0x5574f7d47bd4] 15: (MDSRank::handle_message(boost::intrusive_ptr<Message const> const&)+0x733) [0x5574f7bac793] 16: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x643) [0x5574f7bcacb3] 17: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x5c) [0x5574f7bcb34c] 18: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1d6) [0x5574f7b9f226] 19: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x460) [0x7f5205714020] 20: (DispatchQueue::entry()+0x58f) [0x7f52057118bf] 21: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f52057df261] 22: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f5205205609] 23: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. May 02 16:16:53 xxx077 ceph-mds[3047358]: --- logging levels --- May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 none May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 1 lockdep May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 1 context May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 1 crush May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mds May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mds_balancer May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mds_locker May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mds_log May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mds_log_expire May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mds_migrator May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 1 buffer May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 1 timer May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 1 filer May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 1 striper May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 1 objecter May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 rados May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 rbd May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 rbd_mirror May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 rbd_replay May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 rbd_pwl May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 journaler May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 objectcacher May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 immutable_obj_cache May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 client May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 osd May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 optracker May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 objclass May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 3 filestore May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 3 journal May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 0 ms May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mon May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/10 monc May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 paxos May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 tp May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 auth May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 crypto May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 1 finisher May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 1 reserver May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 heartbeatmap May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 perfcounter May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 rgw May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 rgw_sync May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/10 civetweb May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 javaclient May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 asok May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 1 throttle May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 0 refs May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 compressor May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 bluestore May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 bluefs May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 3 bdev May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 kstore May 02 16:16:53 xxx077 ceph-mds[3047358]: 4/ 5 rocksdb May 02 16:16:53 xxx077 ceph-mds[3047358]: 4/ 5 leveldb May 02 16:16:53 xxx077 ceph-mds[3047358]: 4/ 5 memdb May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 fuse May 02 16:16:53 xxx077 ceph-mds[3047358]: 2/ 5 mgr May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mgrc May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 dpdk May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 eventtrace May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 prioritycache May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 test May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 cephfs_mirror May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 cephsqlite May 02 16:16:53 xxx077 ceph-mds[3047358]: 99/99 (syslog threshold) May 02 16:16:53 xxx077 ceph-mds[3047358]: -2/-2 (stderr threshold) May 02 16:16:53 xxx077 ceph-mds[3047358]: --- pthread ID / name mapping for recent threads --- May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990037739264 / May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990054524672 / May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990062917376 / MR_Finisher May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990079702784 / PQ_Finisher May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990096488192 / ms_dispatch May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990130059008 / ceph-mds May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990146844416 / safe_timer May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990163629824 / ms_dispatch May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990180415232 / io_context_pool May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990197200640 / admin_socket May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990205593344 / msgr-worker-2 May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990213986048 / msgr-worker-1 May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990222378752 / msgr-worker-0 May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990239524736 / ceph-mds May 02 16:16:53 xxx077 ceph-mds[3047358]: max_recent 10000 May 02 16:16:53 xxx077 ceph-mds[3047358]: max_new 10000 May 02 16:16:53 xxx077 ceph-mds[3047358]: log_file /var/lib/ceph/crash/2023-05-02T14:16:53.555508Z_0b05f5cb-130c-4979-95ca-1ba7f31cf7e5/log May 02 16:16:53 xxx077 ceph-mds[3047358]: --- end dump of recent events --- -- Bauhaus-Universität Weimar Bauhausstr. 9a, R308 99423 Weimar, Germany Phone: +49 3643 58 3577 www.webis.de

3 9

Radosgw multisite replication issues
by Tarrago, Eli (RIS-BCT) 12 May '23

12 May '23

Good Afternoon, I am experiencing an issue where east-1 is no longer able to replicate from west-1, however, after a realm pull, west-1 is now able to replicate from east-1. In other words: West <- Can Replicate <- East West -> Cannot Replicate -> East After confirming the access and secret keys are identical on both sides, I restarted all radosgw services. Here is the current status of the cluster below. Thank you for your help, Eli Tarrago root@east01:~# radosgw-admin zone get { "id": "ddd66ab8-0417-46ee-a53b-043352a63f93", "name": "rgw-east", "domain_root": "rgw-east.rgw.meta:root", "control_pool": "rgw-east.rgw.control", "gc_pool": "rgw-east.rgw.log:gc", "lc_pool": "rgw-east.rgw.log:lc", "log_pool": "rgw-east.rgw.log", "intent_log_pool": "rgw-east.rgw.log:intent", "usage_log_pool": "rgw-east.rgw.log:usage", "roles_pool": "rgw-east.rgw.meta:roles", "reshard_pool": "rgw-east.rgw.log:reshard", "user_keys_pool": "rgw-east.rgw.meta:users.keys", "user_email_pool": "rgw-east.rgw.meta:users.email", "user_swift_pool": "rgw-east.rgw.meta:users.swift", "user_uid_pool": "rgw-east.rgw.meta:users.uid", "otp_pool": "rgw-east.rgw.otp", "system_key": { "access_key": "PxxxxxxxxxxxxxxxxW", "secret_key": "Hxxxxxxxxxxxxxxxx6" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "rgw-east.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "rgw-east.rgw.buckets.data" } }, "data_extra_pool": "rgw-east.rgw.buckets.non-ec", "index_type": 0 } } ], "realm_id": "98e0e391-16fb-48da-80a5-08437fd81789", "notif_pool": "rgw-east.rgw.log:notif" } root@west01:~# radosgw-admin zone get { "id": "b2a4a31c-1505-4fdc-b2e0-ea07d9463da1", "name": "rgw-west", "domain_root": "rgw-west.rgw.meta:root", "control_pool": "rgw-west.rgw.control", "gc_pool": "rgw-west.rgw.log:gc", "lc_pool": "rgw-west.rgw.log:lc", "log_pool": "rgw-west.rgw.log", "intent_log_pool": "rgw-west.rgw.log:intent", "usage_log_pool": "rgw-west.rgw.log:usage", "roles_pool": "rgw-west.rgw.meta:roles", "reshard_pool": "rgw-west.rgw.log:reshard", "user_keys_pool": "rgw-west.rgw.meta:users.keys", "user_email_pool": "rgw-west.rgw.meta:users.email", "user_swift_pool": "rgw-west.rgw.meta:users.swift", "user_uid_pool": "rgw-west.rgw.meta:users.uid", "otp_pool": "rgw-west.rgw.otp", "system_key": { "access_key": "PxxxxxxxxxxxxxxW", "secret_key": "Hxxxxxxxxxxxxxx6" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "rgw-west.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "rgw-west.rgw.buckets.data" } }, "data_extra_pool": "rgw-west.rgw.buckets.non-ec", "index_type": 0 } } ], "realm_id": "98e0e391-16fb-48da-80a5-08437fd81789", "notif_pool": "rgw-west.rgw.log:notif" east01:~# radosgw-admin metadata sync status { "sync_status": { "info": { "status": "init", "num_shards": 0, "period": "", "realm_epoch": 0 }, "markers": [] }, "full_sync": { "total": 0, "complete": 0 } } west01:~# radosgw-admin metadata sync status { "sync_status": { "info": { "status": "sync", "num_shards": 64, "period": "44b6b308-e2d8-4835-8518-c90447e7b55c", "realm_epoch": 3 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 46, "pos": 0, "timestamp": "0.000000", "realm_epoch": 3 } }, #### goes on for a long time… { "key": 63, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 3 } } ] }, "full_sync": { "total": 46, "complete": 46 } } east01:~# radosgw-admin sync status realm 98e0e391-16fb-48da-80a5-08437fd81789 (rgw-blobs) zonegroup 0e0faf4e-39f5-402e-9dbb-4a1cdc249ddd (EastWestceph) zone ddd66ab8-0417-46ee-a53b-043352a63f93 (rgw-east) metadata sync no sync (zone is master) 2023-04-20T19:03:13.388+0000 7f25fa036c80 0 ERROR: failed to fetch datalog info data sync source: b2a4a31c-1505-4fdc-b2e0-ea07d9463da1 (rgw-west) failed to retrieve sync info: (13) Permission denied west01:~# radosgw-admin sync status realm 98e0e391-16fb-48da-80a5-08437fd81789 (rgw-blobs) zonegroup 0e0faf4e-39f5-402e-9dbb-4a1cdc249ddd (EastWestceph) zone b2a4a31c-1505-4fdc-b2e0-ea07d9463da1 (rgw-west) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: ddd66ab8-0417-46ee-a53b-043352a63f93 (rgw-east) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 16 shards behind shards: [5,56,62,65,66,70,76,86,87,94,104,107,111,113,120,126] oldest incremental change not applied: 2023-04-20T19:02:48.783283+0000 [5] east01:~# radosgw-admin zonegroup get { "id": "0e0faf4e-39f5-402e-9dbb-4a1cdc249ddd", "name": "EastWestceph", "api_name": "EastWestceph", "is_master": "true", "endpoints": [ http://east01.example.net:8080, http://east02.example.net:8080, http://east03.example.net:8080, http://west01.example.net:8080, http://west02.example.net:8080, http://west03.example.net:8080 ], "hostnames": [ "eastvip.example.net", "westvip.example.net" ], "hostnames_s3website": [], "master_zone": "ddd66ab8-0417-46ee-a53b-043352a63f93", "zones": [ { "id": "b2a4a31c-1505-4fdc-b2e0-ea07d9463da1", "name": "rgw-west", "endpoints": [ http://west01.example.net:8080, http://west02.example.net:8080, http://west03.example.net:8080 ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" }, { "id": "ddd66ab8-0417-46ee-a53b-043352a63f93", "name": "rgw-east", "endpoints": [ http://east01.example.net:8080, http://east02.example.net:8080, http://east03.example.net:8080 ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "98e0e391-16fb-48da-80a5-08437fd81789", "sync_policy": { "groups": [] } } ________________________________ The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message.

4 6

client isn't responding to mclientcaps(revoke), pending pAsLsXsFsc issued pAsLsXsFsc
by Frank Schilder 12 May '23

12 May '23

Hi all, I think we might be hitting a known problem (https://tracker.ceph.com/issues/57244) I don't want to fail the mds yet, because we have troubles with older kclients that miss the mds restart and hold on to cache entries referring to the killed instance, leading to hanging jobs on our HPC cluster. I have seen this issue before and there was a process in D-state that dead-locked itself. Usually, killing this process succeeded and resolved the issue. However, this time I can't find such a process. The tracker mentions that one can delete the file/folder. I have the inode number, but really don't want to start a find on a 1.5PB file system. Is there a better way to find what path is causing the issue (ask the MDS directly, look at a cache dump, or similar)? Is there an alternative to deletion or MDS fail? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 8

docker restarting lost all managers accidentally
by Ben 12 May '23

12 May '23

Hi, This cluster is deployed by cephadm 17.2.5,containerized. It ends up in this(no active mgr): [root@8cd2c0657c77 /]# ceph -s cluster: id: ad3a132e-e9ee-11ed-8a19-043f72fb8bf9 health: HEALTH_WARN 6 hosts fail cephadm check no active mgr 1/3 mons down, quorum h18w,h19w Degraded data redundancy: 781908/2345724 objects degraded (33.333%), 101 pgs degraded, 209 pgs undersized services: mon: 3 daemons, quorum h18w,h19w (age 19m), out of quorum: h15w mgr: no daemons active (since 5h) mds: 1/1 daemons up, 1 standby osd: 9 osds: 6 up (since 5h), 6 in (since 5h) rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 1/1 healthy pools: 8 pools, 209 pgs objects: 781.91k objects, 152 GiB usage: 312 GiB used, 54 TiB / 55 TiB avail pgs: 781908/2345724 objects degraded (33.333%) 108 active+undersized 101 active+undersized+degraded I checked the h20w, there is a manager container running with log: debug 2023-05-10T12:43:23.315+0000 7f5e152ec000 0 monclient(hunting): authenticate timed out after 300 debug 2023-05-10T12:48:23.318+0000 7f5e152ec000 0 monclient(hunting): authenticate timed out after 300 debug 2023-05-10T12:53:23.318+0000 7f5e152ec000 0 monclient(hunting): authenticate timed out after 300 debug 2023-05-10T12:58:23.319+0000 7f5e152ec000 0 monclient(hunting): authenticate timed out after 300 debug 2023-05-10T13:03:23.319+0000 7f5e152ec000 0 monclient(hunting): authenticate timed out after 300 debug 2023-05-10T13:08:23.319+0000 7f5e152ec000 0 monclient(hunting): authenticate timed out after 300 debug 2023-05-10T13:13:23.319+0000 7f5e152ec000 0 monclient(hunting): authenticate timed out after 300 any idea to get a mgr up running again through cephadm? Thanks, Ben

2 2

Lua scripting in the rados gateway
by Thomas Bennett 11 May '23

11 May '23

Hi ceph users, I've been trying out the lua scripting for the rados gateway (thanks Yuval). As in my previous email I mentioned that there is an error when trying to load the luasocket module. However, I thought it was a good time to report on my progress. My 'hello world' example below is called *test.lua* below includes the following checks: 1. Can I write to the debug log? 2. Can I use the lua socket package to do something stupid but intersting, like connect to a webservice? Before you continue reading this, you might need to know that I run all ceph processes in a *CentOS Stream release 8 *container deployed using ceph orchestrator running *Ceph v17.2.5*, so please view the information below in that context. For anyone looking for a reference, I suggest going to the ceph lua rados gateway documentation at radosgw/lua-scripting <https://docs.ceph.com/en/quincy/radosgw/lua-scripting/>. There are two new switches you need to know about in the radosgw-admin: - *script* -> loading your lua script - *script-package* -> loading supporting packages for your script - e.i. luasocket in this case. For a basic setup, you'll need to have a few dependencies in your containers: - cephadm container: requires luarocks (I've checked the code - it runs a luarocks search command) - radosgw container: requires luarocks, gcc, make, m4, wget (wget just in case). To achieve the above, I updated the container image for our running system. I needed to do this because I needed to redeploy the rados gateway container to inject the lua script packages into the radosgw runtime process. This will start with a fresh container based on the global config *container_image* setting on your running system. For us this is currently captured in *quay.io/tsolo/ceph:v17.2.5-3 <http://quay.io/tsolo/ceph:v17.2.5-3>* and included the following exta steps (including installing the lua dev from an rpm because there is no centos package in yum): yum install luarocks gcc make wget m4 rpm -i https://rpmfind.net/linux/centos/8-stream/PowerTools/x86_64/os/Packages/lua… You will notice that I've included a compiler and compiler support into the image. This is because luarocks on the radosgw to compile luasocket (the package I want to install). This will happen at start time when the radosgw is restarted from ceph orch. In the cephadm container I still need to update our cephadm shell so I need to install luarocks by hand: yum install luarocks Then set thew updated image to use: ceph config set global container_image quay.io/tsolo/ceph:v17.2.5-3 I now create a file called: *test.lua* in the cephadm container. This contains the following lines to write to the log and then do a get request to google. This is not practical in production, but it serves the purpose of testing the infrastructure: RGWDebugLog("Tsolo start lua script") local LuaSocket = require("socket") client = LuaSocket.connect("google.com", 80) client:send("GET / HTTP/1.0\r\nHost: google.com\r\n\r\n") while true do s, status, partial = client:receive('*a') RGWDebugLog(s or partial) if status == "closed" then break end end client:close() RGWDebugLog("Tsolo stop lua") Next I run: radosgw-admin script-package add --package=luasocket --allow-compilation And then list the added package to make sure it is there: radosgw-admin script-package list Note - at this point the radosgw has not been modified, it must first be restarted. Then I put the *test.lua *script into the pre request context: radosgw-admin script put --infile=test.lua --context=preRequest You also need to raise the debug log level on the running rados gateway: ceph daemon /var/run/ceph/ceph-client.rgw.xxx.xxx-cms1.xxxxx.x.xxxxxxxxxxxxxx.asok config set debug_rgw 20 Inside the radosgw container I apply my fix (as per previous email): cp -ru /tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/lib64/* /tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/lib/ Outside on the host running the radosgw-admin container I follow the journalctl for the radosgw container (to get the logs): journalctl -fu ceph-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx@rgw. xxx.xxx-cms1.xxxxx.x.xxxxxxxxxxxxxx.service Then I run an s3cmd to put data in via the rados gateway and check the journalctl logs and see: Apr 25 20:54:47 brp-ceph-cms1 radosgw[60901]: Lua INFO: Tsolo start lua Apr 25 20:54:47 brp-ceph-cms1 radosgw[60901]: Lua INFO: HTTP/1.0 301 Moved Permanently Apr 25 20:54:47 brp-ceph-cms1 radosgw[60901]: Lua INFO: Apr 25 20:54:47 brp-ceph-cms1 radosgw[60901]: Lua INFO: Tsolo stop lua Apr 25 20:54:47 brp-ceph-cms1 radosgw[60901]: Lua INFO: Tsolo start lua Apr 25 20:54:48 brp-ceph-cms1 radosgw[60901]: Lua INFO: HTTP/1.0 301 Moved Permanently Apr 25 20:54:48 brp-ceph-cms1 radosgw[60901]: Lua INFO: Apr 25 20:54:48 brp-ceph-cms1 radosgw[60901]: Lua INFO: Tsolo stop lua So the script worked :) If you want to see where the luarocks libraries have been installed, look in the /tmp/ directory of the radosgw container after you redeploy it and you will find the content in /tmp/luarocks. Conclusions: There was a bit to figure out to get this working, but now that I've got this simple test working I think there is a lot more to look into and discover and use w.r.t. this powerful tool. Cheers, Tom

2 5

multisite sync and multipart uploads
by Yixin Jin 11 May '23

11 May '23

Hi guys, With Quincy release, does anyone know how multisite sync deals with multipart uploads? I mean those part objects of some incomplete multipart uploads. Are those objects also sync-ed over either with full-sync or incremental sync? I did a quick experiment and notice that these objects are not sync-ed over. Is it intentional or is there a defect of it? Thanks, Yixin

2 1

multisite synchronization and multipart uploads
by Yixin Jin 11 May '23

11 May '23

Hi folks,

1 0

Recovering from OSD with corrupted DB
by Jessica Sol 11 May '23

11 May '23

Hey all, I recently had two OSDs fail. The first one I just removed and expected replication to fix for me. Replication froze and I restarted the OSDs after seeing heartbeat failures. It allowed replication to resume but one of the OSD's RocksDB became corrupted, showing this error when I try to bring it up: rocksdb: verify_sharding unable to list column families: Corruption: CURRENT file does not end with newline bluestore(/var/lib/ceph/osd/ceph-1) _open_db erroring opening db: osd.1 0 OSD:init: unable to mount object store ** ERROR: osd init failed: (5) Input/output error I found a relevant bug report[1] with a workaround, but it requires ceph-bluestore-tool to have bluefs-import, which was only added in 16.2.11, I'm on 16.2.9 right now. Is it safe to do a patch upgrade from 16.2.9 to 16.2.11 without having an OSD running? Or an alternative way of running the tool without upgrading? Is the tool even safe to run on data from 16.2.9? Thanks, Jess [1] https://tracker.ceph.com/issues/47330

1 0

Protected snapshot deleted when RBD image is deleted
by Work Ceph 10 May '23

10 May '23

Hello guys, We have a doubt regarding snapshot management, when a protected snapshot is created, should it be deleted when its RBD image is removed from the system? If not, how can we list orphaned snapshots in a pool?

2 3

Ceph Leadership Team Meeting Minutes - May 10, 2023
by Ernesto Puerta 10 May '23

10 May '23

Hi Cephers, These are the topics that we just covered in today's meeting: - *Issues recording our meetings in Jitsi (Mike Perez)* - David Orman suggested using a self-hosted Jitsi instance: https://jitsi.github.io/handbook/docs/devops-guide/. Tested on a single container, with 4 cores for 4-5 attendees a call. - Help needed from Dan Mick & Adam Kraitman. - Ilya: this might scale for smaller meetings (dailies), but not for the larger ones (CDM, User-Devs, etc.). - *Reef release status review* - Josh: CentOS 9 blocked by Dashboard/Cephadm Python deps missing in CentOS 9. Casey: It's blocking teuthology testing. - Already discussed in devs mailing list a couple of months ago, for Quincy + Centos 9 (the missing packages would be the same). - Ken Dreyer preferred to keep the legacy approach (distro packages) instead of embedding Python deps. - Casey, Matt & Ernesto to resume that discussion with Ken Dreyer. - Other issues: - Radek: - Performance issue with RocksDB config. - Mismatching client-server features during upgrade. msgr encoder-decoder issues, introduced feature bit for Squid ( https://github.com/ceph/ceph/commit/1049d3e5eff0b7fa4fc9e5853494cb21c10b290a) Performance concerns. To be further discussed at Mark's Perf meeting (incl. Yuval) - Paul Cuzner perf testing Reef: higher CPU usage in Reef vs Quincy (more time spent in RocksDB get calls) - Target GA remains this June. Missing CentOS 9 packages becoming a blocker issue for the release. - *Perf regression in Pacific vs Nautilus (David Orman)* - https://tracker.ceph.com/issues/58530 - Mark: Missing change in upstream Rocksdb project. - David Orman: Is this degradation still happening in newer Rocksdb versions (reef)? Mark: No reason to think otherwise. Kind Regards, Ernesto

1 0

How can I use not-replicated pool (replication 1 or raid-0)
by mhnx 10 May '23

10 May '23

Hello. I have a 10 node cluster. I want to create a non-replicated pool (replication 1) and I want to ask some questions about it: Let me tell you my use case: - I don't care about losing data, - All of my data is JUNK and these junk files are usually between 1KB to 32MB. - These files will be deleted in 5 days. - Writable space and I/O speed is more important. - I have high Write/Read/Delete operations, minimum 200GB a day. I'm afraid that, in any failure, I won't be able to access the whole cluster. Losing data is okay but I have to ignore missing files, remove the data from the cluster and continue with existing data and while doing this, I want to be able to write new data to the cluster. My questions are: 1- To reach this goal do you have any recommendations? 2- With this setup, what potential problems do you have in mind? 3- I think Erasure Coding is not a choice because of the performance problems and slow file deletion. With this I/O need EC will miss files and leaks may happen (I've seen before on Nautilus). 4- You read my needs, is there a better way to do this? Thank you for the answers. Best regards.

3 8

Re: 17.2.6 Dashboard/RGW Signature Mismatch
by Ondřej Kukla 10 May '23

10 May '23

Hello, I have found out that the issue seems to be in this change - https://github.com/ceph/ceph/pull/47207 When I’ve commented out the change and replaced it with the previous value the dashboard works as expected. Ondrej

1 0

Re: 17.2.6 Dashboard/RGW Signature Mismatch
by Ondřej Kukla 10 May '23

10 May '23

Hello, I have found out that the issue seems to be in this change - https://github.com/ceph/ceph/pull/47207 When I’ve commented out the change and replaced it with the previous value the dashboard works as expected. Ondrej

1 0

non root deploy ceph 17.2.5 failed
by Ben 10 May '23

10 May '23

Hi, with following command: sudo cephadm --docker bootstrap --mon-ip 10.1.32.33 --skip-monitoring-stack --ssh-user deployer the user deployer has passwordless sudo configuration. I can see the error below: debug 2023-05-04T12:46:43.268+0000 7fc5ddc2e700 0 [cephadm ERROR cephadm.ssh] Unable to write szhyf-xx1d002-hx15w:/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e: scp: /tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e.new: Permission denied Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/ssh.py", line 222, in _write_remote_file await asyncssh.scp(f.name, (conn, tmp_path)) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp await source.run(srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run await self._send_files(path, b'') File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files await self._send_file(srcpath, dstpath, attrs) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file await self._make_cd_request(b'C', attrs, size, srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request self._fs.basename(path)) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request raise exc Any ideas on this? Thanks, Ben

3 10

cephadm docker registry
by Satish Patel 10 May '23

10 May '23

Folks, I am trying to install ceph on 10 node clusters and planning to use cephadm. My question is if next year i will add new nodes to this cluster then what docker image version cephadm will use to add new nodes? Are there any local registry can i create one to copy images locally? How does cephadm control images?

2 1

v16.2.13 Pacific released
by Yuri Weinstein 10 May '23

10 May '23

We're happy to announce the 13th backport release in the Pacific series. https://ceph.io/en/news/blog/2023/v16-2-13-pacific-released/ Notable Changes --------------- * CEPHFS: Rename the `mds_max_retries_on_remount_failure` option to `client_max_retries_on_remount_failure` and move it from mds.yaml.in to mds-client.yaml.in because this option was only used by MDS client from its birth. * `ceph mgr dump` command now outputs `last_failure_osd_epoch` and `active_clients` fields at the top level. Previously, these fields were output under `always_on_modules` field. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-16.2.13.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 5378749ba6be3a0868b51803968ee9cde4833a3e

3 2

ceph status is warning with "266 pgs not deep-scrubbed in time"
by Louis Koo 10 May '23

10 May '23

ceph version is: v16.2.10 But I had close the ration with "ceph config set mon mon_warn_pg_not_deep_scrubbed_ratio 0" show the prints: [root@smd-node01 deeproute]# ceph config get mon mon_warn_pg_not_deep_scrubbed_ratio 0.000000

1 0

Change in DMARC handling for the list
by Dan Mick 09 May '23

09 May '23

Several users have complained for some time that our DMARC/DKIM handling is not correct. I've recently had time to go study DMARC, DKIM, SPF, SRS, and other tasty morsels of initialisms, and have thus made a change to how Mailman handles DKIM signatures for the list: If a domain advertises that it will reject or quarantine messages that fail DKIM (through its DMARC policy in the DNS text record _dmarc.<domain>), the message will be rewritten to be "From" ceph.io, and SPF should be correct. I do not know if it will regenerate a DKIM signature in that case for what is now it's own message. The From: address will say something like "From Original Sender via ceph-users <ceph-users(a)ceph.io> so it's somewhat clear who first sent the message, and Reply-To will be set to Original Sender. Again, this will only happen for senders from domains that advertise a strict DMARC policy. This does not include gmail.com, surprisingly. Let me know if you notice anything that seems to have gotten worse. Next on the list is to investigate if DKIM-signing outbound messages, or at least ones that don't already have an ARC-Seal, is appropriate and/or workable.

2 1

ERROR: Distro uos version 20 not supported
by Ben 09 May '23

09 May '23

Hi, This seems not very relevant since all ceph components are running in containers. Any ideas to get over this issue? Any other ideas or suggestions on this kind of deployment? sudo ./cephadm --image 10.21.22.1:5000/ceph:v17.2.5-20230316 --docker bootstrap --mon-ip 10.21.22.1 --skip-monitoring-stack Creating directory /etc/ceph for ceph.conf Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... No time sync service is running; checked for ['chrony.service', 'chronyd.service', 'systemd-timesyncd.service', 'ntpd.service', 'ntp.service', 'ntpsec.service', 'openntpd.service'] ERROR: Distro uos version 20 not supported uname -a Linux aaaa 4.19.0-91.82.42.uelc20.x86_64 #1 SMP Sat May 15 13:50:04 CST 2021 x86_64 x86_64 x86_64 GNU/Linux Thank you in advance Ben

1 1

pg deep-scrub issue
by Peter 08 May '23

08 May '23

Dear all, I am writing to seek your assistance in resolving an issue with my Ceph cluster. Currently, the cluster is experiencing a problem where the number of Placement Groups (PGs) that need to undergo deepscrub and regular scrub cannot be completed in a timely manner. I have noticed that these PGs appear to be concentrated on a single OSD. I am seeking your guidance on how to address this issue and would appreciate any insights or suggestions you may have. Please find attached the Ceph health detail output for your reference. HEALTH_WARN 13 pgs not deep-scrubbed in time; 7 pgs not scrubbed in time [WRN] PG_NOT_DEEP_SCRUBBED: 13 pgs not deep-scrubbed in time pg 4.426 not deep-scrubbed since 2023-04-22T10:00:21.529716+0800 pg 4.f0 not deep-scrubbed since 2023-04-22T04:55:17.868881+0800 pg 4.b9 not deep-scrubbed since 2023-04-22T16:47:25.219603+0800 pg 4.87 not deep-scrubbed since 2023-04-22T20:01:02.508600+0800 pg 4.31 not deep-scrubbed since 2023-04-23T00:27:39.299893+0800 pg 4.5b9 not deep-scrubbed since 2023-04-19T21:03:47.041934+0800 pg 4.68a not deep-scrubbed since 2023-04-21T19:52:39.251293+0800 pg 4.6a4 not deep-scrubbed since 2023-04-22T16:20:51.078431+0800 pg 4.6ec not deep-scrubbed since 2023-04-21T11:20:33.661595+0800 pg 4.7a4 not deep-scrubbed since 2023-04-20T22:30:44.506420+0800 pg 4.7a2 not deep-scrubbed since 2023-04-16T12:05:56.586205+0800 pg 4.7b4 not deep-scrubbed since 2023-04-17T15:50:10.595292+0800 pg 4.7c8 not deep-scrubbed since 2023-04-19T15:10:12.673655+0800 [WRN] PG_NOT_SCRUBBED: 7 pgs not scrubbed in time pg 4.31e not scrubbed since 2023-04-24T13:34:26.103257+0800 pg 4.5b9 not scrubbed since 2023-04-24T07:20:53.891175+0800 pg 4.68a not scrubbed since 2023-04-24T03:37:58.070854+0800 pg 4.7a4 not scrubbed since 2023-04-24T02:55:25.912789+0800 pg 4.7b4 not scrubbed since 2023-04-24T10:04:46.889422+0800 pg 4.7c8 not scrubbed since 2023-04-24T13:36:07.284271+0800 pg 4.7d2 not scrubbed since 2023-04-24T14:47:19.365551+0800 Peter

3 3

16.2.13 pacific QE validation status
by Yuri Weinstein 08 May '23

08 May '23

Details of this release are summarized here: https://tracker.ceph.com/issues/59542#note-1 Release Notes - TBD Seeking approvals for: smoke - Radek, Laura rados - Radek, Laura rook - Sébastien Han cephadm - Adam K dashboard - Ernesto rgw - Casey rbd - Ilya krbd - Ilya fs - Venky, Patrick upgrade/octopus-x (pacific) - Laura (look the same as in 16.2.8) upgrade/pacific-p2p - Laura powercycle - Brad (SELinux denials) ceph-volume - Guillaume, Adam K Thx YuriW

10 26

rgw service fails to start with zone not found
by Adiga, Anantha 08 May '23

08 May '23

Hi, An existing multisite configuration was removed. But the radosgw services still see the old zone name and fail to start. journalctl -u ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)rgw.default.default.fl31ca104ja0201.ninovs ........... May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740 0 deferred set uid:gid to 167:167 (ceph:ceph) May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740 0 ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989> May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740 0 framework: beast May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740 0 framework conf key: port, val: 8080 May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740 1 radosgw_Main not setting numa affinity May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.901+0000 7f2f47634740 1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0 May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.901+0000 7f2f47634740 1 D3N datacache enabled: 0 May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.917+0000 7f2f47634740 0 rgw main: ERROR: could not find zone (fl2site2) May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.917+0000 7f2f47634740 0 rgw main: ERROR: failed to start notify service> May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.917+0000 7f2f47634740 0 rgw main: ERROR: failed to init services (ret=(> May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.921+0000 7f2f47634740 -1 Couldn't init storage provider (RADOS) May 08 16:10:49 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)rgw.default.default.fl31ca104ja0201.ninovs.service: Main pr> May 08 16:10:49 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)rgw.default.default.fl31ca104ja0201.ninovs.service: Failed Here is the current configuration root@fl31ca104ja0201:/# radosgw-admin period get { "id": "729f7cef-6340-4750-b3ae-9164177c0df3", "epoch": 1, "predecessor_uuid": "1d124ad5-57b0-41de-8def-823bd40f72aa", "sync_status": [], "period_map": { "id": "729f7cef-6340-4750-b3ae-9164177c0df3", "zonegroups": [ { "id": "21b8306c-be43-4567-a0c5-74ab69937535", "name": "default", "api_name": "default", "is_master": "true", "endpoints": [], "hostnames": [], "hostnames_s3website": [], "master_zone": "ff468492-8a14-414b-a8cf-7c20ab699af3", "zones": [ { "id": "ff468492-8a14-414b-a8cf-7c20ab699af3", "name": "default", "endpoints": [], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "8999e04c-17a4-4150-8845-cecd6672d312", "sync_policy": { "groups": [] } } ], "short_zone_ids": [ { "key": "ff468492-8a14-414b-a8cf-7c20ab699af3", "val": 3883136521 } ] }, "master_zonegroup": "21b8306c-be43-4567-a0c5-74ab69937535", "master_zone": "ff468492-8a14-414b-a8cf-7c20ab699af3", "period_config": { "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_ratelimit": { "max_read_ops": 0, "max_write_ops": 0, "max_read_bytes": 0, "max_write_bytes": 0, "enabled": false }, "bucket_ratelimit": { "max_read_ops": 0, "max_write_ops": 0, "max_read_bytes": 0, "max_write_bytes": 0, "enabled": false }, "anonymous_ratelimit": { "max_read_ops": 0, "max_write_ops": 0, "max_read_bytes": 0, "max_write_bytes": 0, "enabled": false } }, "realm_id": "8999e04c-17a4-4150-8845-cecd6672d312", "realm_name": "default", "realm_epoch": 2 } root@fl31ca104ja0201:/# radosgw-admin realm list { "default_info": "8999e04c-17a4-4150-8845-cecd6672d312", "realms": [ "default" ] } root@fl31ca104ja0201:/# radosgw-admin zonegroup list { "default_info": "21b8306c-be43-4567-a0c5-74ab69937535", "zonegroups": [ "default" ] } root@fl31ca104ja0201:/# radosgw-admin zone list { "default_info": "", "zones": [ "default" ] } Where is the radosgw service getting the old zone info from? How to fix this configuration? Thank you, Anantha

3 4

Ceph Host offline after performing dnf upgrade on RHEL 8.7 host
by Mevludin Blazevic 08 May '23

08 May '23

Hi all, after I performed a minor RHEL package upgrade (8.7 -> 8.7) in one of our Ceph hosts, I get a Ceph warning describing that cephadm "Can't communicate with remote host `...`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory, although Python3 is installed. However, I figured out that these packages were updated: Installing: kernel 4.18.0-425.19.2.el8_7 kernel-core 4.18.0-425.19.2.el8_7 kernel-modules 4.18.0-425.19.2.el8_7 Upgrading: NetworkManager 1:1.40.0-6.el8_7 NetworkManager-config-server 1:1.40.0-6.el8_7 NetworkManager-initscripts-updown 1:1.40.0-6.el8_7 NetworkManager-libnm 1:1.40.0-6.el8_7 NetworkManager-team 1:1.40.0-6.el8_7 NetworkManager-tui 1:1.40.0-6.el8_7 authselect 1.2.5-2.el8_7 authselect-compat 1.2.5-2.el8_7 authselect-libs 1.2.5-2.el8_7 bpftool 4.18.0-425.19.2.el8_7 buildah 1:1.27.3-1.module+el8.7.0+17824+66a0202b cockpit-podman 53-1.module+el8.7.0+17824+66a0202b conmon 3:2.1.4-1.module+el8.7.0+17824+66a0202b container-selinux 2:2.189.0-1.module+el8.7.0+17824+66a0202b containernetworking-plugins 1:1.1.1-3.module+el8.7.0+17824+66a0202b containers-common 2:1-46.module+el8.7.0+17824+66a0202b criu 3.15-3.module+el8.7.0+17824+66a0202b curl 7.61.1-25.el8_7.3 dbus 1:1.12.8-23.el8_7.1 dbus-common 1:1.12.8-23.el8_7.1 dbus-daemon 1:1.12.8-23.el8_7.1 dbus-libs 1:1.12.8-23.el8_7.1 dbus-tools 1:1.12.8-23.el8_7.1 device-mapper-multipath 0.8.4-28.el8_7.3 device-mapper-multipath-libs 0.8.4-28.el8_7.3 dhcp-client 12:4.3.6-48.el8_7.1 dhcp-common 12:4.3.6-48.el8_7.1 dhcp-libs 12:4.3.6-48.el8_7.1 dracut 049-218.git20221019.el8_7 dracut-config-rescue 049-218.git20221019.el8_7 dracut-network 049-218.git20221019.el8_7 dracut-squash 049-218.git20221019.el8_7 emacs-filesystem 1:26.1-7.el8_7.1 epel-release 8-19.el8 expat 2.2.5-10.el8_7.1 fuse-overlayfs 1.9-1.module+el8.7.0+17824+66a0202b gnutls 3.6.16-6.el8_7 grub2-common 1:2.02-142.el8_7.3 grub2-pc 1:2.02-142.el8_7.3 grub2-pc-modules 1:2.02-142.el8_7.3 grub2-tools 1:2.02-142.el8_7.3 grub2-tools-efi 1:2.02-142.el8_7.3 grub2-tools-extra 1:2.02-142.el8_7.3 grub2-tools-minimal 1:2.02-142.el8_7.3 insights-client 3.1.7-9.el8_7 iptables 1.8.4-23.el8_7.1 iptables-ebtables 1.8.4-23.el8_7.1 iptables-libs 1.8.4-23.el8_7.1 iwl100-firmware 39.31.5.1-111.el8_7.1 iwl1000-firmware 1:39.31.5.1-111.el8_7.1 iwl105-firmware 18.168.6.1-111.el8_7.1 iwl135-firmware 18.168.6.1-111.el8_7.1 iwl2000-firmware 18.168.6.1-111.el8_7.1 iwl2030-firmware 18.168.6.1-111.el8_7.1 iwl3160-firmware 1:25.30.13.0-111.el8_7.1 iwl5000-firmware 8.83.5.1_1-111.el8_7.1 iwl5150-firmware 8.24.2.2-111.el8_7.1 iwl6000-firmware 9.221.4.1-111.el8_7.1 iwl6000g2a-firmware 18.168.6.1-111.el8_7.1 iwl6000g2b-firmware 18.168.6.1-111.el8_7.1 iwl6050-firmware 41.28.5.1-111.el8_7.1 iwl7260-firmware 1:25.30.13.0-111.el8_7.1 kernel-tools 4.18.0-425.19.2.el8_7 kernel-tools-libs 4.18.0-425.19.2.el8_7 kmod-kvdo 6.2.7.17-88.el8_7 kpartx 0.8.4-28.el8_7.3 libblkid 2.32.1-39.el8_7 libcurl 7.61.1-25.el8_7.3 libertas-usb8388-firmware 2:20220726-111.git150864a4.el8_7 libfdisk 2.32.1-39.el8_7 libgcc 8.5.0-16.el8_7 libgomp 8.5.0-16.el8_7 libipa_hbac 2.7.3-4.el8_7.3 libksba 1.3.5-9.el8_7 libmount 2.32.1-39.el8_7 libnfsidmap 1:2.3.3-57.el8_7.1 libslirp 4.4.0-1.module+el8.7.0+17824+66a0202b libsmartcols 2.32.1-39.el8_7 libsmbclient 4.16.4-6.el8_7 libsolv 0.7.20-4.el8_7 libsss_autofs 2.7.3-4.el8_7.3 libsss_certmap 2.7.3-4.el8_7.3 libsss_idmap 2.7.3-4.el8_7.3 libsss_nss_idmap 2.7.3-4.el8_7.3 libsss_sudo 2.7.3-4.el8_7.3 libstdc++ 8.5.0-16.el8_7 libtasn1 4.13-4.el8_7 libuuid 2.32.1-39.el8_7 libwbclient 4.16.4-6.el8_7 libxml2 2.9.7-15.el8_7.1 linux-firmware 20220726-111.git150864a4.el8_7 nss 3.79.0-11.el8_7 nss-softokn 3.79.0-11.el8_7 nss-softokn-freebl 3.79.0-11.el8_7 nss-sysinit 3.79.0-11.el8_7 nss-util 3.79.0-11.el8_7 openssh 8.0p1-17.el8_7 openssh-clients 8.0p1-17.el8_7 openssh-server 8.0p1-17.el8_7 openssl 1:1.1.1k-9.el8_7 openssl-libs 1:1.1.1k-9.el8_7 platform-python 3.6.8-48.el8_7.1 platform-python-setuptools 39.2.0-6.el8_7.1 podman 3:4.2.0-8.module+el8.7.0+17824+66a0202b podman-catatonit 3:4.2.0-8.module+el8.7.0+17824+66a0202b python3-libs 3.6.8-48.el8_7.1 python3-libxml2 2.9.7-15.el8_7.1 python3-perf 4.18.0-425.19.2.el8_7 python3-setuptools 39.2.0-6.el8_7.1 python3-setuptools-wheel 39.2.0-6.el8_7.1 python3-sssdconfig 2.7.3-4.el8_7.3 rhc 1:0.2.1-12.el8_7 rsync 3.1.3-19.el8_7.1 runc 1:1.1.4-1.module+el8.7.0+17824+66a0202b samba-client-libs 4.16.4-6.el8_7 samba-common 4.16.4-6.el8_7 samba-common-libs 4.16.4-6.el8_7 selinux-policy 3.14.3-108.el8_7.2 selinux-policy-targeted 3.14.3-108.el8_7.2 slirp4netns 1.2.0-2.module+el8.7.0+17824+66a0202b sos 4.5.1-3.el8 sqlite 3.26.0-17.el8_7 sqlite-libs 3.26.0-17.el8_7 sssd 2.7.3-4.el8_7.3 sssd-ad 2.7.3-4.el8_7.3 sssd-client 2.7.3-4.el8_7.3 sssd-common 2.7.3-4.el8_7.3 sssd-common-pac 2.7.3-4.el8_7.3 sssd-ipa 2.7.3-4.el8_7.3 sssd-kcm 2.7.3-4.el8_7.3 sssd-krb5 2.7.3-4.el8_7.3 sssd-krb5-common 2.7.3-4.el8_7.3 sssd-ldap 2.7.3-4.el8_7.3 sssd-nfs-idmap 2.7.3-4.el8_7.3 sssd-proxy 2.7.3-4.el8_7.3 sudo 1.8.29-8.el8_7.1 systemd 239-68.el8_7.4 systemd-libs 239-68.el8_7.4 systemd-pam 239-68.el8_7.4 systemd-udev 239-68.el8_7.4 tar 2:1.30-6.el8_7.1 tzdata 2023c-1.el8 util-linux 2.32.1-39.el8_7 util-linux-user 2.32.1-39.el8_7 zlib 1.2.11-21.el8_7 Is something odd with the updated Python3 packages? Best, Mevludin

1 1

Os changed to Ubuntu, device class not shown
by Szabo, Istvan (Agoda) 08 May '23

08 May '23

Hi, We have an octopus cluster where we want to move from centos to Ubuntu, after activate all the osd, class is not shown in ceph osd tree. However ceph-volume list shows the crush device class :/ Should I just add it or? ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Testing to simulate real-world usage scenarios
by Michal Strnad 07 May '23

07 May '23

Hi everyone, how do you test features that are used by your users before updating/upgrading to a newer version of Ceph? For example, if you have users or customers using RBD images with special sets of permissions/caps, or users/customers using S3 with versioning... None of these should be broken by the new code. Do you use any framework, set of scripts, or anything else? So far, I have only found Teuthology for integration tests and then for S3 [https://github.com/ceph/s3-tests] Thank you Michal

1 0

splitting client and replication traffic
by Justin Ryan 06 May '23

06 May '23

hiya folks! i am excited to be learning about ceph in my homelab, where i have a couple trios of machines service, currently, a cephfs, but i’m looking fw to using ceph for block devices for vms, etc.. my phyiscal machines have multiple network interfaces, one trio of them are ARM machines with up to 4x SFP+ 10GBe interfaces. I have a couple of 10GBe switches - one which has exactly three ports I can connect these nodes to for general distribution to the rest of the network, and another… I would like to configure my Ceph cluster to replicate over a different network interface than it serves clients over.. Is this a common configuration, and/or can anyone provide me some guidance?! Thanks in advance! Best! J -- Justin Alan Ryan

2 1

Unable to restart mds - mds crashes almost immediately after finishing recovery
by Emmanuel Jaep 05 May '23

05 May '23

Hi, I just inherited a ceph storage. Therefore, my level of confidence with the tool is certainly less than ideal. We currently have an mds server that refuses to come back online. While reviewing the logs, I can see that, upon mds start, the recovery goes well: ``` -10> 2023-05-03T08:12:43.632+0200 7f345d00b700 1 mds.4.2638711 cluster recovered. 12: (MDCache::_open_ino_traverse_dir(inodeno_t, MDCache::open_ino_info_t&, int)+0xbf) [0x558323d602df] ``` However, rights after this message, ceph handles a couple of clients request: ``` -9> 2023-05-03T08:12:43.632+0200 7f345d00b700 4 mds.4.2638711 set_osd_epoch_barrier: epoch=249241 -8> 2023-05-03T08:12:43.632+0200 7f3459003700 2 mds.4.cache Memory usage: total 2739784, rss 2321188, heap 348412, baseline 315644, 0 / 765023 inodes have caps, 0 caps, 0 caps per inode -7> 2023-05-03T08:12:43.688+0200 7f3458802700 4 mds.4.server handle_client_request client_request(client.108396030:57271 lookup #0x70001516236/012385530.npy 2023-05-02T20:37:19.675666+0200 RETRY=6 caller_uid=135551, caller_gid=11157{0,4,27,11157,}) v5 -6> 2023-05-03T08:12:43.688+0200 7f3458802700 4 mds.4.server handle_client_request client_request(client.104073212:5109945 readdir #0x70001516236 2023-05-02T20:36:29.517066+0200 RETRY=6 caller_uid=180090, caller_gid=11157{0,4,27,11157,}) v5 -5> 2023-05-03T08:12:43.688+0200 7f3458802700 4 mds.4.server handle_client_request client_request(client.104288735:3008344 readdir #0x70001516236 2023-05-02T20:36:29.520801+0200 RETRY=6 caller_uid=135551, caller_gid=11157{0,4,27,11157,}) v5 -4> 2023-05-03T08:12:43.688+0200 7f3458802700 4 mds.4.server handle_client_request client_request(client.8558540:46306346 readdir #0x700019ba15e 2023-05-01T21:26:34.303697+0200 RETRY=49 caller_uid=0, caller_gid=0{}) v2 -3> 2023-05-03T08:12:43.688+0200 7f3458802700 4 mds.4.server handle_client_request client_request(client.96913903:2156912 create #0x1000b37db9a/street-photo-3.png 2023-05-01T17:27:37.454042+0200 RETRY=59 caller_uid=271932, caller_gid=30034{}) v2 -2> 2023-05-03T08:12:43.688+0200 7f345d00b700 5 mds.icadmin006 handle_mds_map old map epoch 2638715 <= 2638715, discarding ``` and crashes: ``` -1> 2023-05-03T08:12:43.692+0200 7f345d00b700 -1 /build/ceph-16.2.10/src/mds/Server.cc: In function 'void Server::handle_client_open(MDRequestRef&)' thread 7f345d00b700 time 2023-05-03T08:12:43.694660+0200 /build/ceph-16.2.10/src/mds/Server.cc: 4240: FAILED ceph_assert(cur->is_auth()) ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f3462533d65] 2: /usr/lib/ceph/libceph-common.so.2(+0x265f6d) [0x7f3462533f6d] 3: (Server::handle_client_open(boost::intrusive_ptr<MDRequestImpl>&)+0x1834) [0x558323c89f04] 4: (Server::handle_client_openc(boost::intrusive_ptr<MDRequestImpl>&)+0x28f) [0x558323c925ef] 5: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0xa45) [0x558323cc3575] 6: (MDCache::dispatch_request(boost::intrusive_ptr<MDRequestImpl>&)+0x3d) [0x558323d7460d] 7: (MDSContext::complete(int)+0x61) [0x558323f68681] 8: (MDCache::_open_remote_dentry_finish(CDentry*, inodeno_t, MDSContext*, bool, int)+0x3e) [0x558323d3edce] 9: (C_MDC_OpenRemoteDentry::finish(int)+0x3e) [0x558323de6cce] 10: (MDSContext::complete(int)+0x61) [0x558323f68681] 11: (MDCache::open_ino_finish(inodeno_t, MDCache::open_ino_info_t&, int)+0xcf) [0x558323d5ff2f] 12: (MDCache::_open_ino_traverse_dir(inodeno_t, MDCache::open_ino_info_t&, int)+0xbf) [0x558323d602df] 13: (MDSContext::complete(int)+0x61) [0x558323f68681] 14: (MDSRank::_advance_queues()+0x88) [0x558323c23c38] 15: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x1fa) [0x558323c24a1a] 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x5e) [0x558323c254fe] 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1d6) [0x558323bfd906] 18: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x460) [0x7f34627854e0] 19: (DispatchQueue::entry()+0x58f) [0x7f3462782d7f] 20: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f346284eee1] 21: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f3462278609] 22: clone() 0> 2023-05-03T08:12:43.700+0200 7f345d00b700 -1 *** Caught signal (Aborted) ** in thread 7f345d00b700 thread_name:ms_dispatch ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x143c0) [0x7f34622843c0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1ad) [0x7f3462533dc0] 5: /usr/lib/ceph/libceph-common.so.2(+0x265f6d) [0x7f3462533f6d] 6: (Server::handle_client_open(boost::intrusive_ptr<MDRequestImpl>&)+0x1834) [0x558323c89f04] 7: (Server::handle_client_openc(boost::intrusive_ptr<MDRequestImpl>&)+0x28f) [0x558323c925ef] 8: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0xa45) [0x558323cc3575] 9: (MDCache::dispatch_request(boost::intrusive_ptr<MDRequestImpl>&)+0x3d) [0x558323d7460d] 10: (MDSContext::complete(int)+0x61) [0x558323f68681] 11: (MDCache::_open_remote_dentry_finish(CDentry*, inodeno_t, MDSContext*, bool, int)+0x3e) [0x558323d3edce] 12: (C_MDC_OpenRemoteDentry::finish(int)+0x3e) [0x558323de6cce] 13: (MDSContext::complete(int)+0x61) [0x558323f68681] 14: (MDCache::open_ino_finish(inodeno_t, MDCache::open_ino_info_t&, int)+0xcf) [0x558323d5ff2f] 15: (MDCache::_open_ino_traverse_dir(inodeno_t, MDCache::open_ino_info_t&, int)+0xbf) [0x558323d602df] 16: (MDSContext::complete(int)+0x61) [0x558323f68681] 17: (MDSRank::_advance_queues()+0x88) [0x558323c23c38] 18: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x1fa) [0x558323c24a1a] 19: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x5e) [0x558323c254fe] 20: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1d6) [0x558323bfd906] 21: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x460) [0x7f34627854e0] 22: (DispatchQueue::entry()+0x58f) [0x7f3462782d7f] 23: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f346284eee1] 24: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f3462278609] 25: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_pwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 fuse 2/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test 0/ 5 cephfs_mirror 0/ 5 cephsqlite -2/-2 (syslog threshold) -1/-1 (stderr threshold) --- pthread ID / name mapping for recent threads --- 139862749464320 / 139862757857024 / md_submit 139862766249728 / 139862774642432 / MR_Finisher 139862791427840 / PQ_Finisher 139862799820544 / mds_rank_progr 139862808213248 / ms_dispatch 139862841784064 / ceph-mds 139862858569472 / safe_timer 139862875354880 / ms_dispatch 139862892140288 / io_context_pool 139862908925696 / admin_socket 139862917318400 / msgr-worker-2 139862925711104 / msgr-worker-1 139862934103808 / msgr-worker-0 139862951257984 / ceph-mds max_recent 10000 max_new 10000 log_file /var/log/ceph/floki-mds.icadmin006.log --- end dump of recent events --- ``` How could I troubleshoot that further? Thanks in advance for your help, Emmanuel

3 3

osd pause
by Thomas Bennett 05 May '23

05 May '23

Hi, FYI - This might be pedantic, but there does not seem to be any difference between using these two sets of commands: - ceph osd pause / ceph osd unpause - ceph osd set pause / ceph osd unset pause I can see that they both set/unset the pauserd,pausewr flags, but since they don't report anything else, I assume they do exactly the same thing. I also assumed it only stopped reads/writes to the OSDs, but I found this openattic post <https://openattic.org/posts/how-to-do-a-ceph-cluster-maintenanceshutdown/> which had this comment: Pausing the cluster means that you can't see when OSDs come back up again and no map update will happen. I didn't know that but it seems pretty useful knowing. Pausing is mentioned when posts talking about shutting down Ceph cluster for maintenance but often it's added as optional. Does anyone know what is the original intended purpose of pausing and when/why would you use it? Also - can I assume that pause will complete any current write/read ops on OSDs before pausing? Thanks, Tom

1 0

rbd map: corrupt full osdmap (-22) when
by Kamil Madac 05 May '23

05 May '23

Hi, We deployed pacific cluster 16.2.12 with cephadm. We experience following error during rbd map: [Wed May 3 08:59:11 2023] libceph: mon2 (1)[2a00:da8:ffef:1433::]:6789 session established [Wed May 3 08:59:11 2023] libceph: another match of type 1 in addrvec [Wed May 3 08:59:11 2023] libceph: corrupt full osdmap (-22) epoch 200 off 1042 (000000009876284d of 000000000cb24b58-0000000080b70596) [Wed May 3 08:59:11 2023] osdmap: 00000000: 08 07 7d 10 00 00 09 01 5d 09 00 00 a2 22 3b 86 ..}.....]....";. [Wed May 3 08:59:11 2023] osdmap: 00000010: e4 f5 11 ed 99 ee 47 75 ca 3c ad 23 c8 00 00 00 ......Gu.<.#.... [Wed May 3 08:59:11 2023] osdmap: 00000020: 21 68 4a 64 98 d2 5d 2e 84 fd 50 64 d9 3a 48 26 !hJd..]...Pd.:H& [Wed May 3 08:59:11 2023] osdmap: 00000030: 02 00 00 00 01 00 00 00 00 00 00 00 1d 05 71 01 ..............q. .... Linux Kernel is 6.1.13 and the important thing is that we are using ipv6 addresses for connection to ceph nodes. We were able to map rbd from client with kernel 5.10, but in prod environment we are not allowed to use that kernel. What could be the reason for such behavior on newer kernels and how to troubleshoot it? Here is output of ceph osd dump: # ceph osd dump epoch 200 fsid a2223b86-e4f5-11ed-99ee-4775ca3cad23 created 2023-04-27T12:18:41.777900+0000 modified 2023-05-02T12:09:40.642267+0000 flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit crush_version 34 full_ratio 0.95 backfillfull_ratio 0.9 nearfull_ratio 0.85 require_min_compat_client luminous min_compat_client jewel require_osd_release pacific stretch_mode_enabled false pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 183 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr_devicehealth pool 2 'idp' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 48 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd max_osd 3 osd.0 up in weight 1 up_from 176 up_thru 182 down_at 172 last_clean_interval [170,171) [v2:[2a00:da8:ffef:1431::]:6800/805023868,v1:[2a00:da8:ffef:1431::]:6801/805023868,v2: 0.0.0.0:6802/805023868,v1:0.0.0.0:6803/805023868] [v2:[2a00:da8:ffef:1431::]:6804/805023868,v1:[2a00:da8:ffef:1431::]:6805/805023868,v2: 0.0.0.0:6806/805023868,v1:0.0.0.0:6807/805023868] exists,up e8fd0ee2-ea63-4d02-8f36-219d36869078 osd.1 up in weight 1 up_from 136 up_thru 182 down_at 0 last_clean_interval [0,0) [v2:[2a00:da8:ffef:1432::]:6800/2172723816,v1:[2a00:da8:ffef:1432::]:6801/2172723816,v2: 0.0.0.0:6802/2172723816,v1:0.0.0.0:6803/2172723816] [v2:[2a00:da8:ffef:1432::]:6804/2172723816,v1:[2a00:da8:ffef:1432::]:6805/2172723816,v2: 0.0.0.0:6806/2172723816,v1:0.0.0.0:6807/2172723816] exists,up 0b7b5628-9273-4757-85fb-9c16e8441895 osd.2 up in weight 1 up_from 182 up_thru 182 down_at 178 last_clean_interval [123,177) [v2:[2a00:da8:ffef:1433::]:6800/887631330,v1:[2a00:da8:ffef:1433::]:6801/887631330,v2: 0.0.0.0:6802/887631330,v1:0.0.0.0:6803/887631330] [v2:[2a00:da8:ffef:1433::]:6804/887631330,v1:[2a00:da8:ffef:1433::]:6805/887631330,v2: 0.0.0.0:6806/887631330,v1:0.0.0.0:6807/887631330] exists,up 21f8d0d5-6a3f-4f78-96c8-8ec4e4f78a01 Thank you. -- Kamil Madac

3 5

Re: 17.2.6 fs 'ls' ok, but 'cat' 'operation not permitted' puzzle
by Harry G Coin 05 May '23

05 May '23

This problem of inaccessible file systems post upgrade by other than client.admin date back from v14 carries on through v17. It also applies to any case of specifying other than the default pool names for new file systems. Solved because Curt remembered link on this list. (Thanks Curt!) Here's what the official ceph docs ought have provided, for others who hit this. YMMV: IF you have ceph file systems which have data and meta data pools that were specified in the 'ceph fs new' command (meaning not left to the defaults which create them for you), OR you have an existing ceph file system and are upgrading to a new major version of ceph THEN for the documented 'ceph fs authorize...' commands to do as documented (and avoid strange 'operation not permitted' errors when doing file I/O or similar security related problems for all but such as the client.admin user), you must first run: ceph osd pool application set <your metadata pool name> cephfs metadata <your ceph fs filesystem name> and ceph osd pool application set <your data pool name> cephfs data <your ceph fs filesystem name> Otherwise when the OSD's get a request to read or write data (not the directory info, but file data) they won't know which ceph file system name to look up, nevermind the names you may have chosen for the pools, as the 'defaults' themselves changed in the major releases, from data pool=fsname metadata pool=fsname_metadata to data pool=fsname.data and metadata pool=fsname.meta as the ceph revisions came and went. Any setup that just used 'client.admin' for all mounts didn't see the problem as the admin key gave blanket permission. A temporary 'fix' is to change mount requests to the 'client.admin' and associated key. A less drastic but still half-fix is to change the osd cap for your user to just 'caps osd = "allow rw" and delete "tag cephfs data=...." The only documentation I could find for this upgrade security-related ceph-ending catastrophe was in the NFS, not cephfs docs: https://docs.ceph.com/en/latest/cephfs/nfs/ and the Genius level much appreciated pointer from Curt here: On 5/2/23 14:21, Curt wrote: > This thread might be of use, it's an older version of ceph 14, but > might still apply, > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/23FDDSYBCDV… > ? > > On Tue, May 2, 2023 at 11:06 PM Harry G Coin <hgcoin(a)gmail.com> wrote: > > In 17.2.6 is there a security requirement that pool names > supporting a > ceph fs filesystem match the filesystem name.data for the data and > name.meta for the associated metadata pool? (multiple file systems > are > enabled) > > I have filesystems from older versions with the data pool name > matching > the filesystem and appending _metadata for that, > > and even older filesystems with the pool name as in 'library' and > 'library_metadata' supporting a filesystem called 'libraryfs' > > The pools all have the cephfs tag. > > But using the documented: > > ceph fs authorize libraryfs client.basicuser / rw > > command allows the root user to mount and browse the library > directory > tree, but fails with 'operation not permitted' when even reading > any file. > > However, changing the client.basicuser osd auth to 'allow rw' > instead of > 'allow rw tag...' allows normal operations. > > So: > > [client.basicuser] > key = <key stuff>== > caps mds = "allow rw fsname=libraryfs" > caps mon = "allow r fsname=libraryfs" > caps osd = "allow rw" > > works, but the same with > > caps osd = "allow rw tag cephfs data=libraryfs" > > leads to the 'operation not permitted' on read, or write or any > actual > access. > > It remains a puzzle. Help appreciated! > > Were there upgrade instructions about that, any help pointing me > to them? > > Thanks > > Harry Coin > Rock Stable Systems > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

3 2

CephFS Scrub Questions
by Chris Palmer 04 May '23

04 May '23

Hi Grateful if someone could clarify some things about CephFS Scrubs: 1) Am I right that a command such as "ceph tell mds.cephfs:0 scrub start / recursive" only triggers a forward scrub (not a backward scrub)? 2) I couldn't find any reference to forward scrubs being done automatically and was wondering whether I should do them using cron? But then I saw an undated (but I think a little elderly) presentation by Greg Farnum that states that "forward scrub...runs continuously in the background". Is that still correct (for Quincy), and if so what controls the frequency? 3) Are backward scrubs always manual, using the 3 cephfs-data-scan phases? 4) Are regular backward scrubs recommended, or only if there is indication of a problem? (With due regard to the amount of time they may take...) Thanks for any advice. Regards, Chris

2 1

Radosgw: ssl_private_key could not find the file even if it existed
by viplanghe6＠gmail.com 04 May '23

04 May '23

The radosgw has been configured like this: [client.rgw.ceph1] host = ceph1 rgw_frontends = beast port=8080 ssl_port=443 ssl_certificate=/root/ssl/ca.crt ssl_private_key=/root/ssl/ca.key #rgw_frontends = beast port=8080 ssl_port=443 ssl_certificate=/root/ssl/ca.crt ssl_private_key=config://rgw/cert/default/ca.key admin_socket = /var/run/ceph/ceph-client.rgw.ceph1 but I'm getting this error: failed to add ssl_private_key=/root/ssl/ca.key: No such file or directory I also tried to import the key into ceph db and provided the path with config://, but it doesn't work too. Anyone have any ideal? Thanks

2 1

Frequent calling monitor election
by Frank Schilder 04 May '23

04 May '23

Hi all, our monitors have enjoyed democracy since the beginning. However, I don't share a sudden excitement about voting: 2/9/23 4:42:30 PM[INF]overall HEALTH_OK 2/9/23 4:42:30 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 4:42:26 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:42:26 PM[INF]mon.ceph-26 calling monitor election 2/9/23 4:42:26 PM[INF]mon.ceph-25 calling monitor election 2/9/23 4:42:26 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:40:00 PM[INF]overall HEALTH_OK 2/9/23 4:30:00 PM[INF]overall HEALTH_OK 2/9/23 4:24:34 PM[INF]overall HEALTH_OK 2/9/23 4:24:34 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 4:24:29 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-03 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-26 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-25 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:24:04 PM[INF]overall HEALTH_OK 2/9/23 4:24:03 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 4:23:59 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:23:59 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:20:00 PM[INF]overall HEALTH_OK 2/9/23 4:10:00 PM[INF]overall HEALTH_OK 2/9/23 4:00:00 PM[INF]overall HEALTH_OK 2/9/23 3:50:00 PM[INF]overall HEALTH_OK 2/9/23 3:43:13 PM[INF]overall HEALTH_OK 2/9/23 3:43:13 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 3:43:08 PM[INF]mon.ceph-01 calling monitor election 2/9/23 3:43:08 PM[INF]mon.ceph-26 calling monitor election 2/9/23 3:43:08 PM[INF]mon.ceph-25 calling monitor election We moved a switch from one rack to another and after the switch came beck up, the monitors frequently bitch about who is the alpha. How do I get them to focus more on their daily duties again? Thanks for any help! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 9

pg upmap primary
by Nguetchouang Ngongang Kevin 04 May '23

04 May '23

Hello, I have a question, when happened when i delete a pg on which i set a particular osd as primary using the pg-upmap-primary command ? -- Nguetchouang Ngongang Kevin ENS de Lyon https://perso.ens-lyon.fr/kevin.nguetchouang/

2 1

回复：ceph-users Digest, Vol 107, Issue 20
by 余晟佐 04 May '23

04 May '23

help ------------------ 原始邮件 ------------------ 发件人: "ceph-users" <ceph-users-request(a)ceph.io>; 发送时间: 2023年5月4日(星期四) 下午4:40 收件人: "ceph-users"<ceph-users(a)ceph.io>; 主题: ceph-users Digest, Vol 107, Issue 20 Send ceph-users mailing list submissions to ceph-users(a)ceph.io To subscribe or unsubscribe via email, send a message with subject or body 'help' to ceph-users-request(a)ceph.io You can reach the person managing the list at ceph-users-owner(a)ceph.io When replying, please edit your Subject line so it is more specific than "Re: Contents of ceph-users digest..." Today's Topics:    1. Initialization timeout, failed to initialize (Vitaly Goot)    2. Re: MDS crash on FAILED ceph_assert(cur->is_auth())       (Peter van Heusden)    3. Re: MDS "newly corrupt dentry" after patch version upgrade       (Janek Bevendorff)    4. Best practice for expanding Ceph cluster (huxiaoyu(a)horebdata.cn)    5. Re: 16.2.13 pacific QE validation status (Guillaume Abrioux) ---------------------------------------------------------------------- Date: Thu, 04 May 2023 01:50:12 -0000 From: "Vitaly Goot" <vitaly.goot(a)gmail.com> Subject: [ceph-users] Initialization timeout, failed to initialize To: ceph-users(a)ceph.io Message-ID: <168316501216.1713.9594013921879975501@mailman-web> Content-Type: text/plain; charset="utf-8" playing with MULTI-SITE zones for CEPH Object Gateway ceph version: 17.2.5 my setup: 3 zone multi-site; 3-way full sync mode; each zone has 3 machines -> RGW+MON+OSD running load test:  3000 concurrent uploads of 1M object after about 3-4 minutes of load RGW machine get stuck, on 2 zone out of 3 RGW is not responding (e.g. curl $RGW:80) attempt to restart RGW ends up with `Initialization timeout, failed to initialize` here is a backtrace from gdb with a backtrace where it hangs after restart: (gdb) inf thr   Id   Target Id                                           Frame * 1    Thread 0x7fa7d3abbcc0 (LWP 30791) "radosgw"         futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffc7f7a2438) at ../sysdeps/nptl/futex-internal.h:183 ... (gdb) bt #0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffc7f7a2438) at ../sysdeps/nptl/futex-internal.h:183 #1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffc7f7a2488, cond=0x7ffc7f7a2410) at pthread_cond_wait.c:508 #2  __pthread_cond_wait (cond=cond@entry=0x7ffc7f7a2410, mutex=0x7ffc7f7a2488) at pthread_cond_wait.c:647 #3  0x00007fa7d7097e42 in ceph::condition_variable_debug::wait (this=this@entry=0x7ffc7f7a2410, lock=...) at ../src/common/mutex_debug.h:148 #4  0x00007fa7d7953cba in ceph::condition_variable_debug::wait<librados::IoCtxImpl::operate(const object_t&, ObjectOperation*, ceph::real_time*, int)::<lambda()> > (pred=..., lock=..., this=0x7ffc7f7a2410) at ../src/librados/IoCtxImpl.cc:672 #5  librados::IoCtxImpl::operate (this=this@entry=0x558347c21010, oid=..., o=0x558347e12310, pmtime=<optimized out>, flags=<optimized out>) at ../src/librados/IoCtxImpl.cc:672 #6  0x00007fa7d792bd55 in librados::v14_2_0::IoCtx::operate (this=this@entry=0x558347e44760, oid="notify.0", o=o@entry=0x7ffc7f7a2690, flags=flags@entry=0) at ../src/librados/librados_cxx.cc:1536 #7  0x00007fa7d9490ad1 in rgw_rados_operate (dpp=<optimized out>, ioctx=..., oid="notify.0", op=op@entry=0x7ffc7f7a2690, y=..., flags=0) at ../src/rgw/rgw_tools.cc:277 #8  0x00007fa7d9627e0f in RGWSI_RADOS::Obj::operate (this=this@entry=0x558347e44710, dpp=<optimized out>, op=op@entry=0x7ffc7f7a2690, y=..., flags=flags@entry=0) at ../src/rgw/services/svc_rados.h:112 #9  0x00007fa7d96209a5 in RGWSI_Notify::init_watch (this=this@entry=0x558347c49530, dpp=<optimized out>, y=...) at ../src/rgw/services/svc_notify.cc:214 #10 0x00007fa7d962161b in RGWSI_Notify::do_start (this=0x558347c49530, y=..., dpp=<optimized out>) at ../src/rgw/services/svc_notify.cc:277 #11 0x00007fa7d8f17bcf in RGWServiceInstance::start (this=0x558347c49530, y=..., dpp=<optimized out>) at ../src/rgw/rgw_service.cc:331 #12 0x00007fa7d8f1a260 in RGWServices_Def::init (this=this@entry=0x558347de90a0, cct=<optimized out>, have_cache=<optimized out>, raw=raw@entry=false, run_sync=<optimized out>, y=..., dpp=<optimized out>) at /usr/include/c++/9/bits/unique_ptr.h:360 #13 0x00007fa7d8f1cc40 in RGWServices::do_init (this=this@entry=0x558347de90a0, _cct=<optimized out>, have_cache=<optimized out>, raw=raw@entry=false, run_sync=<optimized out>, y=..., dpp=<optimized out>) at ../src/rgw/rgw_service.cc:284 #14 0x00007fa7d92a7b1f in RGWServices::init (dpp=<optimized out>, y=..., run_sync=<optimized out>, have_cache=<optimized out>, cct=<optimized out>, this=0x558347de90a0) at ../src/rgw/rgw_service.h:153 #15 RGWRados::init_svc (this=this@entry=0x558347de8dc0, raw=raw@entry=false, dpp=<optimized out>) at ../src/rgw/rgw_rados.cc:1380 #16 0x00007fa7d930f241 in RGWRados::initialize (this=0x558347de8dc0, dpp=<optimized out>) at ../src/rgw/rgw_rados.cc:1400 #17 0x00007fa7d944f85f in RGWRados::initialize (dpp=<optimized out>, _cct=0x558347c6a320, this=<optimized out>) at ../src/rgw/rgw_rados.h:586 #18 StoreManager::init_storage_provider (dpp=<optimized out>, dpp@entry=0x7ffc7f7a2e90, cct=cct@entry=0x558347c6a320, svc="rados", use_gc_thread=use_gc_thread@entry=true, use_lc_thread=use_lc_thread@entry=true, quota_threads=quota_threads@entry=true, run_sync_thread=true, run_reshard_thread=true, use_cache=true,     use_gc=true) at ../src/rgw/rgw_sal.cc:55 #19 0x00007fa7d8e7367a in StoreManager::get_storage (use_gc=true, use_cache=true, run_reshard_thread=true, run_sync_thread=true, quota_threads=true, use_lc_thread=true, use_gc_thread=true, svc="rados", cct=0x558347c6a320, dpp=0x7ffc7f7a2e90) at /usr/include/c++/9/bits/basic_string.h:267 #20 radosgw_Main (argc=<optimized out>, argv=<optimized out>) at ../src/rgw/rgw_main.cc:372 #21 0x0000558347883f56 in main (argc=<optimized out>, argv=<optimized out>) at ../src/rgw/radosgw.cc:12 (gdb) #0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffc7f7a2438) at ../sysdeps/nptl/futex-internal.h:183 #1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffc7f7a2488, cond=0x7ffc7f7a2410) at pthread_cond_wait.c:508 #2  __pthread_cond_wait (cond=cond@entry=0x7ffc7f7a2410, mutex=0x7ffc7f7a2488) at pthread_cond_wait.c:647 #3  0x00007fa7d7097e42 in ceph::condition_variable_debug::wait (this=this@entry=0x7ffc7f7a2410, lock=...) at ../src/common/mutex_debug.h:148 #4  0x00007fa7d7953cba in ceph::condition_variable_debug::wait<librados::IoCtxImpl::operate(const object_t&, ObjectOperation*, ceph::real_time*, int)::<lambda()> > (pred=..., lock=..., this=0x7ffc7f7a2410) at ../src/librados/IoCtxImpl.cc:672 #5  librados::IoCtxImpl::operate (this=this@entry=0x558347c21010, oid=..., o=0x558347e12310, pmtime=<optimized out>, flags=<optimized out>) at ../src/librados/IoCtxImpl.cc:672 #6  0x00007fa7d792bd55 in librados::v14_2_0::IoCtx::operate (this=this@entry=0x558347e44760, oid="notify.0", o=o@entry=0x7ffc7f7a2690, flags=flags@entry=0) at ../src/librados/librados_cxx.cc:1536 #7  0x00007fa7d9490ad1 in rgw_rados_operate (dpp=<optimized out>, ioctx=..., oid="notify.0", op=op@entry=0x7ffc7f7a2690, y=..., flags=0) at ../src/rgw/rgw_tools.cc:277 #8  0x00007fa7d9627e0f in RGWSI_RADOS::Obj::operate (this=this@entry=0x558347e44710, dpp=<optimized out>, op=op@entry=0x7ffc7f7a2690, y=..., flags=flags@entry=0) at ../src/rgw/services/svc_rados.h:112 #9  0x00007fa7d96209a5 in RGWSI_Notify::init_watch (this=this@entry=0x558347c49530, dpp=<optimized out>, y=...) at ../src/rgw/services/svc_notify.cc:214 #10 0x00007fa7d962161b in RGWSI_Notify::do_start (this=0x558347c49530, y=..., dpp=<optimized out>) at ../src/rgw/services/svc_notify.cc:277 #11 0x00007fa7d8f17bcf in RGWServiceInstance::start (this=0x558347c49530, y=..., dpp=<optimized out>) at ../src/rgw/rgw_service.cc:331 #12 0x00007fa7d8f1a260 in RGWServices_Def::init (this=this@entry=0x558347de90a0, cct=<optimized out>, have_cache=<optimized out>, raw=raw@entry=false, run_sync=<optimized out>, y=..., dpp=<optimized out>) at /usr/include/c++/9/bits/unique_ptr.h:360 #13 0x00007fa7d8f1cc40 in RGWServices::do_init (this=this@entry=0x558347de90a0, _cct=<optimized out>, have_cache=<optimized out>, raw=raw@entry=false, run_sync=<optimized out>, y=..., dpp=<optimized out>) at ../src/rgw/rgw_service.cc:284 #14 0x00007fa7d92a7b1f in RGWServices::init (dpp=<optimized out>, y=..., run_sync=<optimized out>, have_cache=<optimized out>, cct=<optimized out>, this=0x558347de90a0) at ../src/rgw/rgw_service.h:153 #15 RGWRados::init_svc (this=this@entry=0x558347de8dc0, raw=raw@entry=false, dpp=<optimized out>) at ../src/rgw/rgw_rados.cc:1380 #16 0x00007fa7d930f241 in RGWRados::initialize (this=0x558347de8dc0, dpp=<optimized out>) at ../src/rgw/rgw_rados.cc:1400 #17 0x00007fa7d944f85f in RGWRados::initialize (dpp=<optimized out>, _cct=0x558347c6a320, this=<optimized out>) at ../src/rgw/rgw_rados.h:586 #18 StoreManager::init_storage_provider (dpp=<optimized out>, dpp@entry=0x7ffc7f7a2e90, cct=cct@entry=0x558347c6a320, svc="rados", use_gc_thread=use_gc_thread@entry=true, use_lc_thread=use_lc_thread@entry=true, quota_threads=quota_threads@entry=true, run_sync_thread=true, run_reshard_thread=true, use_cache=true,     use_gc=true) at ../src/rgw/rgw_sal.cc:55 #19 0x00007fa7d8e7367a in StoreManager::get_storage (use_gc=true, use_cache=true, run_reshard_thread=true, run_sync_thread=true, quota_threads=true, use_lc_thread=true, use_gc_thread=true, svc="rados", cct=0x558347c6a320, dpp=0x7ffc7f7a2e90) at /usr/include/c++/9/bits/basic_string.h:267 #20 radosgw_Main (argc=<optimized out>, argv=<optimized out>) at ../src/rgw/rgw_main.cc:372 #21 0x0000558347883f56 in main (argc=<optimized out>, argv=<optimized out>) at ../src/rgw/radosgw.cc:12 Any suggestion on what can be a problem and how to reset RGW so it will be able to start normally? ------------------------------ Date: Thu, 4 May 2023 09:13:56 +0200 From: Peter van Heusden <pvh(a)sanbi.ac.za> Subject: [ceph-users] Re: MDS crash on FAILED ceph_assert(cur->is_auth()) Cc: ceph-users(a)ceph.io Message-ID: <CAK1reXhEDjfKuLmuyus0RT09mwecRmP=LGLcoSKWeZ+pu+YXJQ(a)mail.gmail.com> Content-Type: text/plain; charset="UTF-8" Hi Emmaneul It was a while ago, but as I recall I evicted all clients and that allowed me to restart the MDS servers. There was something clearly "broken" in how at least one of the clients was interacting with the system. Peter On Thu, 4 May 2023 at 07:18, Emmanuel Jaep <emmanuel.jaep(a)gmail.com> wrote: > Hi, > > did you finally figure out what happened? > I do have the same behavior and we can't get the mds to start again... > > Thanks, > > Emmanuel > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > ------------------------------ Date: Thu, 4 May 2023 09:15:38 +0200 From: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de> Subject: [ceph-users] Re: MDS "newly corrupt dentry" after patch version upgrade To: Patrick Donnelly <pdonnell(a)redhat.com> Cc: ceph-users <ceph-users(a)ceph.io> Message-ID: <591f410c-aabc-72af-36d0-478ce8d09028(a)uni-weimar.de> Content-Type: text/plain; charset=UTF-8; format=flowed After running the tool for 11 hours straight, it exited with the following exception: Traceback (most recent call last):    File "/home/webis/first-damage.py", line 156, in <module>      traverse(f, ioctx)    File "/home/webis/first-damage.py", line 84, in traverse      for (dnk, val) in it:    File "rados.pyx", line 1389, in rados.OmapIterator.__next__    File "rados.pyx", line 318, in rados.decode_cstr UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 8: invalid start byte Does that mean that the last inode listed in the output file is corrupt? Any way I can fix it? The output file has 14 million lines. We have about 24.5 million objects in the metadata pool. Janek On 03/05/2023 14:20, Patrick Donnelly wrote: > On Wed, May 3, 2023 at 4:33 AM Janek Bevendorff > <janek.bevendorff(a)uni-weimar.de> wrote: >> Hi Patrick, >> >>> I'll try that tomorrow and let you know, thanks! >> I was unable to reproduce the crash today. Even with >> mds_abort_on_newly_corrupt_dentry set to true, all MDS booted up >> correctly (though they took forever to rejoin with logs set to 20). >> >> To me it looks like the issue has resolved itself overnight. I had run a >> recursive scrub on the file system and another snapshot was taken, in >> case any of those might have had an effect on this. It could also be the >> case that the (supposedly) corrupt journal entry has simply been >> committed now and hence doesn't trigger the assertion any more. Is there >> any way I can verify this? > You can run: > > https://github.com/ceph/ceph/blob/main/src/tools/cephfs/first-damage.py > > Just do: > > python3 first-damage.py --memo run.1 <meta pool> > > No need to do any of the other steps if you just want a read-only check. > -- Bauhaus-Universität Weimar Bauhausstr. 9a, R308 99423 Weimar, Germany Phone: +49 3643 58 3577 www.webis.de ------------------------------ Date: Thu, 4 May 2023 10:38:30 +0200 From: "huxiaoyu(a)horebdata.cn" <huxiaoyu(a)horebdata.cn> Subject: [ceph-users] Best practice for expanding Ceph cluster To: ceph-users <ceph-users(a)ceph.io> Message-ID: <985B22F000D9DE88+2023050410382589829410(a)horebdata.cn> Content-Type: text/plain;       charset="us-ascii" Dear Ceph folks, I am writing to ask for advice on best practice of expanding ceph cluster. We are running an 8-node Ceph cluster and RGW, and would like to add another 10 node, each of which have 10x 12TB HDD. The current 8-node has ca. 400TB user data. I am wondering whether to add 10 nodes at one shot and let the cluster to rebalance, or divide into 5 steps, each of which add 2 nodes and rebalance step by step?  I do not know what would be the advantages or disadvantages with the one shot scheme vs 5 bataches of adding 2 nodes step-by-step. Any suggestions, experience sharing or advice are highly appreciated. thanks a lot in advance, Samuel huxiaoyu(a)horebdata.cn ------------------------------ Date: Thu, 4 May 2023 10:40:02 +0200 From: Guillaume Abrioux <gabrioux(a)redhat.com> Subject: [ceph-users] Re: 16.2.13 pacific QE validation status To: Laura Flores <lflores(a)redhat.com> Cc: Yuri Weinstein <yweinste(a)redhat.com>, Radoslaw Zarzynski <rzarzyns(a)redhat.com>, dev <dev(a)ceph.io>, ceph-users <ceph-users(a)ceph.io> Message-ID: <CANqTTH5ba9qf3xStCcCZr24n5GPyq0Eeimw3Seha1MZ6wna5nA(a)mail.gmail.com> Content-Type: text/plain; charset="UTF-8" ceph-volume approved https://jenkins.ceph.com/job/ceph-volume-test/553/ On Wed, 3 May 2023 at 22:43, Guillaume Abrioux <gabrioux(a)redhat.com> wrote: > The failure seen in ceph-volume tests isn't related. > That being said, it needs to be fixed to have a better view of the current > status. > > On Wed, 3 May 2023 at 21:00, Laura Flores <lflores(a)redhat.com> wrote: > >> upgrade/octopus-x (pacific) is approved. Went over failures with Adam >> King and it was decided they are not release blockers. >> >> On Wed, May 3, 2023 at 1:53 PM Yuri Weinstein <yweinste(a)redhat.com> >> wrote: >> >>> upgrade/octopus-x (pacific) - Laura >>> ceph-volume - Guillaume >>> >>> + 2 PRs are the remaining issues >>> >>> Josh FYI >>> >>> On Wed, May 3, 2023 at 11:50 AM Radoslaw Zarzynski <rzarzyns(a)redhat.com> >>> wrote: >>> > >>> > rados approved. >>> > >>> > Big thanks to Laura for helping with this! >>> > >>> > On Thu, Apr 27, 2023 at 11:21 PM Yuri Weinstein <yweinste(a)redhat.com> >>> wrote: >>> > > >>> > > Details of this release are summarized here: >>> > > >>> > > https://tracker.ceph.com/issues/59542#note-1 >>> > > Release Notes - TBD >>> > > >>> > > Seeking approvals for: >>> > > >>> > > smoke - Radek, Laura >>> > > rados - Radek, Laura >>> > >   rook - Sébastien Han >>> > >   cephadm - Adam K >>> > >   dashboard - Ernesto >>> > > >>> > > rgw - Casey >>> > > rbd - Ilya >>> > > krbd - Ilya >>> > > fs - Venky, Patrick >>> > > upgrade/octopus-x (pacific) - Laura (look the same as in 16.2.8) >>> > > upgrade/pacific-p2p - Laura >>> > > powercycle - Brad (SELinux denials) >>> > > ceph-volume - Guillaume, Adam K >>> > > >>> > > Thx >>> > > YuriW >>> > > _______________________________________________ >>> > > Dev mailing list -- dev(a)ceph.io >>> > > To unsubscribe send an email to dev-leave(a)ceph.io >>> > >>> _______________________________________________ >>> Dev mailing list -- dev(a)ceph.io >>> To unsubscribe send an email to dev-leave(a)ceph.io >>> >> >> >> -- >> >> Laura Flores >> >> She/Her/Hers >> >> Software Engineer, Ceph Storage <https://ceph.io> >> >> Chicago, IL >> >> lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> >> M: +17087388804 >> >> >> _______________________________________________ >> Dev mailing list -- dev(a)ceph.io >> To unsubscribe send an email to dev-leave(a)ceph.io >> > > > -- > > *Guillaume Abrioux*Senior Software Engineer > -- *Guillaume Abrioux*Senior Software Engineer ------------------------------ Subject: Digest Footer _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s ------------------------------ End of ceph-users Digest, Vol 107, Issue 20 *******************************************

1 0

MDS crash on FAILED ceph_assert(cur->is_auth())
by Peter van Heusden 04 May '23

04 May '23

I am running Ceph 15.2.13 on CentOS 7.9.2009 and recently my MDS servers have started failing with the error message In function 'void Server::handle_client_open(MDRequestRef&)' thread 7f0ca9908700 time 2021-06-28T09:21:11.484768+0200 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/15.2.13/rpm/el7/BUILD/ceph-15.2.13/src/mds/Server.cc: 4149: FAILED ceph_assert(cur->is_auth()) Complete log is: https://gist.github.com/pvanheus/4da555a6de6b5fa5e46cbf74f5500fbd ceph status output is: # ceph status cluster: id: ed7b2c16-b053-45e2-a1fe-bf3474f90508 health: HEALTH_WARN 30 OSD(s) experiencing BlueFS spillover insufficient standby MDS daemons available 1 MDSs report slow requests 2 mgr modules have failed dependencies 4347046/326505282 objects misplaced (1.331%) 6 nearfull osd(s) 23 pgs not deep-scrubbed in time 23 pgs not scrubbed in time 8 pool(s) nearfull services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 22m) mgr: ceph-mon1(active, since 11w), standbys: ceph-mon2, ceph-mon3 mds: SANBI_FS:2 {0=ceph-mon1=up:active(laggy or crashed),1=ceph-mon2=up:stopping} osd: 54 osds: 54 up (since 2w), 54 in (since 11w); 50 remapped pgs data: pools: 8 pools, 833 pgs objects: 42.37M objects, 89 TiB usage: 159 TiB used, 105 TiB / 264 TiB avail pgs: 4347046/326505282 objects misplaced (1.331%) 782 active+clean 49 active+clean+remapped 1 active+clean+scrubbing+deep 1 active+clean+remapped+scrubbing io: client: 29 KiB/s rd, 427 KiB/s wr, 37 op/s rd, 48 op/s wr When restarting a MDS it goes through states replace, reconnect, resolve and finally sets itself to active before this crash happens. Any advice on what to do? Thanks, Peter P.S. apologies if you received this email more than once - I have had some trouble figuring out the correct mailing list to use.

3 5

Initialization timeout, failed to initialize
by Vitaly Goot 04 May '23

04 May '23

playing with MULTI-SITE zones for CEPH Object Gateway ceph version: 17.2.5 my setup: 3 zone multi-site; 3-way full sync mode; each zone has 3 machines -> RGW+MON+OSD running load test: 3000 concurrent uploads of 1M object after about 3-4 minutes of load RGW machine get stuck, on 2 zone out of 3 RGW is not responding (e.g. curl $RGW:80) attempt to restart RGW ends up with `Initialization timeout, failed to initialize` here is a backtrace from gdb with a backtrace where it hangs after restart: (gdb) inf thr Id Target Id Frame * 1 Thread 0x7fa7d3abbcc0 (LWP 30791) "radosgw" futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffc7f7a2438) at ../sysdeps/nptl/futex-internal.h:183 ... (gdb) bt #0 futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffc7f7a2438) at ../sysdeps/nptl/futex-internal.h:183 #1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffc7f7a2488, cond=0x7ffc7f7a2410) at pthread_cond_wait.c:508 #2 __pthread_cond_wait (cond=cond@entry=0x7ffc7f7a2410, mutex=0x7ffc7f7a2488) at pthread_cond_wait.c:647 #3 0x00007fa7d7097e42 in ceph::condition_variable_debug::wait (this=this@entry=0x7ffc7f7a2410, lock=...) at ../src/common/mutex_debug.h:148 #4 0x00007fa7d7953cba in ceph::condition_variable_debug::wait<librados::IoCtxImpl::operate(const object_t&, ObjectOperation*, ceph::real_time*, int)::<lambda()> > (pred=..., lock=..., this=0x7ffc7f7a2410) at ../src/librados/IoCtxImpl.cc:672 #5 librados::IoCtxImpl::operate (this=this@entry=0x558347c21010, oid=..., o=0x558347e12310, pmtime=<optimized out>, flags=<optimized out>) at ../src/librados/IoCtxImpl.cc:672 #6 0x00007fa7d792bd55 in librados::v14_2_0::IoCtx::operate (this=this@entry=0x558347e44760, oid="notify.0", o=o@entry=0x7ffc7f7a2690, flags=flags@entry=0) at ../src/librados/librados_cxx.cc:1536 #7 0x00007fa7d9490ad1 in rgw_rados_operate (dpp=<optimized out>, ioctx=..., oid="notify.0", op=op@entry=0x7ffc7f7a2690, y=..., flags=0) at ../src/rgw/rgw_tools.cc:277 #8 0x00007fa7d9627e0f in RGWSI_RADOS::Obj::operate (this=this@entry=0x558347e44710, dpp=<optimized out>, op=op@entry=0x7ffc7f7a2690, y=..., flags=flags@entry=0) at ../src/rgw/services/svc_rados.h:112 #9 0x00007fa7d96209a5 in RGWSI_Notify::init_watch (this=this@entry=0x558347c49530, dpp=<optimized out>, y=...) at ../src/rgw/services/svc_notify.cc:214 #10 0x00007fa7d962161b in RGWSI_Notify::do_start (this=0x558347c49530, y=..., dpp=<optimized out>) at ../src/rgw/services/svc_notify.cc:277 #11 0x00007fa7d8f17bcf in RGWServiceInstance::start (this=0x558347c49530, y=..., dpp=<optimized out>) at ../src/rgw/rgw_service.cc:331 #12 0x00007fa7d8f1a260 in RGWServices_Def::init (this=this@entry=0x558347de90a0, cct=<optimized out>, have_cache=<optimized out>, raw=raw@entry=false, run_sync=<optimized out>, y=..., dpp=<optimized out>) at /usr/include/c++/9/bits/unique_ptr.h:360 #13 0x00007fa7d8f1cc40 in RGWServices::do_init (this=this@entry=0x558347de90a0, _cct=<optimized out>, have_cache=<optimized out>, raw=raw@entry=false, run_sync=<optimized out>, y=..., dpp=<optimized out>) at ../src/rgw/rgw_service.cc:284 #14 0x00007fa7d92a7b1f in RGWServices::init (dpp=<optimized out>, y=..., run_sync=<optimized out>, have_cache=<optimized out>, cct=<optimized out>, this=0x558347de90a0) at ../src/rgw/rgw_service.h:153 #15 RGWRados::init_svc (this=this@entry=0x558347de8dc0, raw=raw@entry=false, dpp=<optimized out>) at ../src/rgw/rgw_rados.cc:1380 #16 0x00007fa7d930f241 in RGWRados::initialize (this=0x558347de8dc0, dpp=<optimized out>) at ../src/rgw/rgw_rados.cc:1400 #17 0x00007fa7d944f85f in RGWRados::initialize (dpp=<optimized out>, _cct=0x558347c6a320, this=<optimized out>) at ../src/rgw/rgw_rados.h:586 #18 StoreManager::init_storage_provider (dpp=<optimized out>, dpp@entry=0x7ffc7f7a2e90, cct=cct@entry=0x558347c6a320, svc="rados", use_gc_thread=use_gc_thread@entry=true, use_lc_thread=use_lc_thread@entry=true, quota_threads=quota_threads@entry=true, run_sync_thread=true, run_reshard_thread=true, use_cache=true, use_gc=true) at ../src/rgw/rgw_sal.cc:55 #19 0x00007fa7d8e7367a in StoreManager::get_storage (use_gc=true, use_cache=true, run_reshard_thread=true, run_sync_thread=true, quota_threads=true, use_lc_thread=true, use_gc_thread=true, svc="rados", cct=0x558347c6a320, dpp=0x7ffc7f7a2e90) at /usr/include/c++/9/bits/basic_string.h:267 #20 radosgw_Main (argc=<optimized out>, argv=<optimized out>) at ../src/rgw/rgw_main.cc:372 #21 0x0000558347883f56 in main (argc=<optimized out>, argv=<optimized out>) at ../src/rgw/radosgw.cc:12 (gdb) #0 futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffc7f7a2438) at ../sysdeps/nptl/futex-internal.h:183 #1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffc7f7a2488, cond=0x7ffc7f7a2410) at pthread_cond_wait.c:508 #2 __pthread_cond_wait (cond=cond@entry=0x7ffc7f7a2410, mutex=0x7ffc7f7a2488) at pthread_cond_wait.c:647 #3 0x00007fa7d7097e42 in ceph::condition_variable_debug::wait (this=this@entry=0x7ffc7f7a2410, lock=...) at ../src/common/mutex_debug.h:148 #4 0x00007fa7d7953cba in ceph::condition_variable_debug::wait<librados::IoCtxImpl::operate(const object_t&, ObjectOperation*, ceph::real_time*, int)::<lambda()> > (pred=..., lock=..., this=0x7ffc7f7a2410) at ../src/librados/IoCtxImpl.cc:672 #5 librados::IoCtxImpl::operate (this=this@entry=0x558347c21010, oid=..., o=0x558347e12310, pmtime=<optimized out>, flags=<optimized out>) at ../src/librados/IoCtxImpl.cc:672 #6 0x00007fa7d792bd55 in librados::v14_2_0::IoCtx::operate (this=this@entry=0x558347e44760, oid="notify.0", o=o@entry=0x7ffc7f7a2690, flags=flags@entry=0) at ../src/librados/librados_cxx.cc:1536 #7 0x00007fa7d9490ad1 in rgw_rados_operate (dpp=<optimized out>, ioctx=..., oid="notify.0", op=op@entry=0x7ffc7f7a2690, y=..., flags=0) at ../src/rgw/rgw_tools.cc:277 #8 0x00007fa7d9627e0f in RGWSI_RADOS::Obj::operate (this=this@entry=0x558347e44710, dpp=<optimized out>, op=op@entry=0x7ffc7f7a2690, y=..., flags=flags@entry=0) at ../src/rgw/services/svc_rados.h:112 #9 0x00007fa7d96209a5 in RGWSI_Notify::init_watch (this=this@entry=0x558347c49530, dpp=<optimized out>, y=...) at ../src/rgw/services/svc_notify.cc:214 #10 0x00007fa7d962161b in RGWSI_Notify::do_start (this=0x558347c49530, y=..., dpp=<optimized out>) at ../src/rgw/services/svc_notify.cc:277 #11 0x00007fa7d8f17bcf in RGWServiceInstance::start (this=0x558347c49530, y=..., dpp=<optimized out>) at ../src/rgw/rgw_service.cc:331 #12 0x00007fa7d8f1a260 in RGWServices_Def::init (this=this@entry=0x558347de90a0, cct=<optimized out>, have_cache=<optimized out>, raw=raw@entry=false, run_sync=<optimized out>, y=..., dpp=<optimized out>) at /usr/include/c++/9/bits/unique_ptr.h:360 #13 0x00007fa7d8f1cc40 in RGWServices::do_init (this=this@entry=0x558347de90a0, _cct=<optimized out>, have_cache=<optimized out>, raw=raw@entry=false, run_sync=<optimized out>, y=..., dpp=<optimized out>) at ../src/rgw/rgw_service.cc:284 #14 0x00007fa7d92a7b1f in RGWServices::init (dpp=<optimized out>, y=..., run_sync=<optimized out>, have_cache=<optimized out>, cct=<optimized out>, this=0x558347de90a0) at ../src/rgw/rgw_service.h:153 #15 RGWRados::init_svc (this=this@entry=0x558347de8dc0, raw=raw@entry=false, dpp=<optimized out>) at ../src/rgw/rgw_rados.cc:1380 #16 0x00007fa7d930f241 in RGWRados::initialize (this=0x558347de8dc0, dpp=<optimized out>) at ../src/rgw/rgw_rados.cc:1400 #17 0x00007fa7d944f85f in RGWRados::initialize (dpp=<optimized out>, _cct=0x558347c6a320, this=<optimized out>) at ../src/rgw/rgw_rados.h:586 #18 StoreManager::init_storage_provider (dpp=<optimized out>, dpp@entry=0x7ffc7f7a2e90, cct=cct@entry=0x558347c6a320, svc="rados", use_gc_thread=use_gc_thread@entry=true, use_lc_thread=use_lc_thread@entry=true, quota_threads=quota_threads@entry=true, run_sync_thread=true, run_reshard_thread=true, use_cache=true, use_gc=true) at ../src/rgw/rgw_sal.cc:55 #19 0x00007fa7d8e7367a in StoreManager::get_storage (use_gc=true, use_cache=true, run_reshard_thread=true, run_sync_thread=true, quota_threads=true, use_lc_thread=true, use_gc_thread=true, svc="rados", cct=0x558347c6a320, dpp=0x7ffc7f7a2e90) at /usr/include/c++/9/bits/basic_string.h:267 #20 radosgw_Main (argc=<optimized out>, argv=<optimized out>) at ../src/rgw/rgw_main.cc:372 #21 0x0000558347883f56 in main (argc=<optimized out>, argv=<optimized out>) at ../src/rgw/radosgw.cc:12 Any suggestion on what can be a problem and how to reset RGW so it will be able to start normally?

1 0

test
by Dan Mick 04 May '23

04 May '23

making some DMARC-related changes. Ignore this please.

1 0

To drain unmanaged OSDs by an oversight
by Ackermann, Christoph 03 May '23

03 May '23

Hello List, i made a mistake, draining a host instead of entering it into Maintenance Mode (for OS reboot). :-/ After "Stop Drain" and restore of original "crush reweight" values, so far everything looks fine. cluster: health: HEALTH_OK services: [..] osd: 79 osds: 78 up (since 3h), 78 in (since 6w); 166 remapped pgs [..] And some minor objects being misplaced.. But I can't remove/solve this annoying message "deleting" on these host OSDs. [image: Auswahl_2023-05-03_14-17.png] Does someone have a hint for me? Thanks, Christoph

1 0

MAnual Upgrade from Octopus Ubuntu 18.04 to Quincy 20.04
by Iban Cabrillo 03 May '23

03 May '23

Dear Cephers, We are planing the dist upgrade from Octopus to Quincy in the next weeks. The first step is the linux version upgrade from Ubuntu 18.04 to Ubuntu 20.04 from some big ODS servers runnign this OS version. we just have a look at ( Upgrading non-cephadm clusters [ https://ceph.io/en/news/blog/2022/v17-2-0-quincy-released/#upgrading-non-ce… | ¶ ] ): https://ceph.io/en/news/blog/2022/v17-2-0-quincy-released/ Is any advise or suggestion before to start the procedure ? regards, I -- ================================================================ Ibán Cabrillo Bartolomé Instituto de Física de Cantabria (IFCA-CSIC) Santander, Spain Tel: +34942200969/+34669930421 Responsible for advanced computing service (RSC) ========================================================================================= ========================================================================================= All our suppliers must know and accept IFCA policy available at: https://confluence.ifca.es/display/IC/Information+Security+Policy+for+Exter… ==========================================================================================

1 0

Flushing stops as copy-from message being throttled
by lingu2008 03 May '23

03 May '23

Hi all, On one server with a cache tier on Samsung PM983 SSDs for an EC base tier on HDDs, I find the cache tier stops flushing or evicting when the cache tier is near full. With quite some gdb-debugging, I find the problem may be with the throttling mechanism. When the write traffic is high, the cache tier quickly fills its maximum request count and throttles further requests. Then flush stops because copy-from requests are throttled by the cache tier OSD. Ironically, the 256 requests already accepted by the cache tier cannot proceed, either, because the cache tier is full and cannot flush/evict. While we may advise cache tier should not go full, this deadlock situation is not entirely comprehensible to me because a full cache usually can flush/evict as long as the base tier has space. I wonder whether there has been some specific reasons for this behavior. My test environment is with version 15.2.17 but the code in 17.2.2 appears to handle this part of logic in the same way. Cheers, lin

2 1

RBD mirroring, asking for clarification
by wodel youchi 03 May '23

03 May '23

Hi, When using rbd mirroring, the mirroring concerns the images only, not the whole pool? So, we don't need to have a dedicated pool in the destination site to be mirrored, the only obligation is that the mirrored pools must have the same name. In other words, We create two pools with the same name, one on the source site the other on the destination site, we create the mirror link (one way or two ways replication), then we choose what images to sync. Both pools can be used simultaneously on both sites, it's the mirrored images that cannot be used simultaneously, only promoted ones. Is this correct? Regards.

2 5

architecture help (iscsi, rbd, backups?)
by Angelo Höngens 02 May '23

02 May '23

Hey guys and girls, I'm working on a project to build storage for one of our departments, and I want to ask you guys and girls for input on the high-level overview part. It's a long one, I hope you read along and comment. SUMMARY I made a plan last year to build a 'storage solution' including ceph and some windows VM's to expose the data over SMB to clients. A year later I finally have the hardware, built a ceph cluster, and I'm doing tests. Ceph itself runs great, but when I wanted to start exposing the data using iscsi to our VMware farm, I ran into some issues. I know the iscsi gateways will introduce some new performance bottlenecks, but I'm seeing really slow performance, still working on that. But then I ran into the warning on the iscsi gateway page: "The iSCSI gateway is in maintenance as of November 2022. This means that it is no longer in active development and will not be updated to add new features.". Wait, what? Why!? What does this mean? Does this mean that iSCSI is now 'feature complete' and will still be supported the next 5 years, or will it be deprecated in the future? I tried searching, but couldn't find any info on the decision and the roadmap. My goal is to build a future-proof setup, and using deprecated components should not be part of that of course. If the iscsi gateway will still be supported the next few years and I can iron out the performance issues, I can still go on with my original plan. If not, I have to go back to the drawing board. And maybe you guys would advise me to take another route anyway. GOALS My goals/considerations are: - we want >1PB of storage capacity for cheap (on a tight budget) for research data. Most of it is 'store once, read sometimes'. <1% of the data is 'hot'. - focus is on capacity, but it would be nice to have > 200MB/s of sequential write/read performance and not 'totally suck' on random i/o. Yes, not very well quantified, but ah. Sequential writes are most important. - end users all run Windows computers (mostly VDI's) and a lot of applications require SMB shares. - security is a big thing, we want really tight ACL's, specific monitoring agents, etc. - our data is incredibly important to us, we still want the 3-2-1 backup rule. Primary storage solution, a second storage solution in a different place, and some of the data that is not reproducible is also written to tape. We also want to be protected from ransomware or user errors (so no direct replication to the second storage). - I like open source, reliability, no fork-lift upgrades, no vendor lock-in, blah, well, I'm on the ceph list here, no need to convince you guys ;) - We're hiring a commercial company to do ceph maintenance and support for when I'm on leave or leaving the company, but they won't support clients, backup software, etc, so I want something as simple as possible. We do have multiple Windows/VMware admins, but no other real linux guru's. THE INITIAL PLAN Given these considerations, I ordered two identical clusters, each consisting of 3 monitor nodes and 8 osd nodes, Each osd node has 2 ssd's and 10 capacity disks (EC 4:2 for the data), and each node is connected using a 2x25Gbps bond. Ceph is running like a charm. Now I just have to think about exposing the data to end users, and I've been testing different setups. My original plan was to expose for example 10x100TB rbd images using iSCSI to our VMware farm, formatting the luns with VMFS6, and run for example 2 Windows file servers per datastore on that with a single DFS namespace to end users. Then backup the file servers using our existing Veeam infrastructure to RGW running on the second cluster with an immutable bucket. This way we would have easily defined security boundaries: the clients can only reach the file servers, the file servers only see their local VMDK's, ESX only sees the luns on the iSCSI target, etc. When a file server would be compromised, it would have no access to ceph. We have easy incremental backups, immutability for ransomware protection, etc. And the best part is that the ceph admin can worry about ceph, the vmware admin can focus on ESX, VMFS and all the vmware stuff, and the Windows admins can focus on the Windows boxes, Windows-specific ACLS and tools and Veeam backups and stuff. CURRENT SITUATION I'm building out this plan now, but I'm running into issues with iSCSI. Are any of you doing something similar? What is your iscsi performance compared to direct rbd? In regard to performance: If I take 2 test windows VM's, I put one on an iSCSI datastore and another with direct rbd access using the windows rbd driver, I create a share on those boxes and push data to it, I see different results (of course). Copying some iso images over SMB to the 'windows vm running direct rbd' I see around 800MB/s write, and 200MB/s read, which is pretty okay. When I send data to the 'windows vm running on top of iscsi' it starts writing at around 350MB/s, but after like 10-20 seconds drops to 100MB/s and won't go faster. Reads are anywhere from 40MB/s to 80MB/s, which is not really acceptable. Another really viable and performant scenario would be to have the Windows file servers connect to rbd directly with the windows rbd driver. It seems to work well, it's fast, and you don't have the bottleneck that the iscsi gateway creates. But I see this driver is still in beta. Is anyone using this in production? What are your experiences? We would miss out on the separation of layers and thus have less security, but at the same time, it really increases efficiency and performance. And if I use rbd, then vmware won't see the storage, and I cannot do an image backup using veeam. I could of course do backups of the rbd images, using tools like restic or backy to rgw running on the second cluster with immutable buckets. What are your experiences? Is it easy to do differential backups of lots of 50TB rbd images? Change rate is usually like 0.005% per day or something ;) By the way, we also thought about CephFS, but we have some complex stuff going on with extended ACL's that I don't think will play nice with CephFS, and I think it's a lot more complex to backup CephFS than block images. If you made it here, thank you for your time! I hope you can share thoughts on my questions! Angelo.

6 8

[multisite] "bucket sync status" takes a while
by Yixin Jin 02 May '23

02 May '23

Hi folks, With a multi-site environment, when I create a bucket-level sync policy with a symmetric flow between the master zone and another zone, "bucket sync status" immediately shows that the sync is now enabled in the master zone. But it takes a while for it to show that in the other zone. I tried "period pull" at the other zone and "period push" at the master zone. Neither seem to make a difference. Is there a way to speed up this process? Thanks, Yixin

1 0

17.2.6 fs 'ls' ok, but 'cat' 'operation not permitted' puzzle
by Harry G Coin 02 May '23

02 May '23

In 17.2.6 is there a security requirement that pool names supporting a ceph fs filesystem match the filesystem name.data for the data and name.meta for the associated metadata pool? (multiple file systems are enabled) I have filesystems from older versions with the data pool name matching the filesystem and appending _metadata for that, and even older filesystems with the pool name as in 'library' and 'library_metadata' supporting a filesystem called 'libraryfs' The pools all have the cephfs tag. But using the documented: ceph fs authorize libraryfs client.basicuser / rw command allows the root user to mount and browse the library directory tree, but fails with 'operation not permitted' when even reading any file. However, changing the client.basicuser osd auth to 'allow rw' instead of 'allow rw tag...' allows normal operations. So: [client.basicuser] key = <key stuff>== caps mds = "allow rw fsname=libraryfs" caps mon = "allow r fsname=libraryfs" caps osd = "allow rw" works, but the same with caps osd = "allow rw tag cephfs data=libraryfs" leads to the 'operation not permitted' on read, or write or any actual access. It remains a puzzle. Help appreciated! Were there upgrade instructions about that, any help pointing me to them? Thanks Harry Coin Rock Stable Systems

1 0

Memory leak in MGR after upgrading to pacific.
by Gary Molenkamp 02 May '23

02 May '23

Good morning, After upgrading from Octopus (15.2.17) to Pacific (16.2.12) two days ago, I'm noticing that the MGR daemons keep failing over to standby and then back every 24hrs. Watching the output of 'ceph orch ps' I can see that the memory consumption of the mgr is steadily growing until it becomes unresponsive. When the mgr becomes unresponsive, tasks such as RESTful calls start to fail, and the standby eventually takes over after ~20 minutes. I've included a log of memory consumption (in 10 minute intervals) at the end of this message. While the cluster recovers during this issue, the loss of usage data during the outage, and the fact its occurring is problematic. Any assistance would be appreciated. Note, this is a cluster that has been upgraded from an original jewel based ceph using filestore, through bluestore conversion, container conversion, and now to Pacific. The data below shows memory use with three mgr modules enabled: cephadm, restful, iostat. By disabling iostat, I can reduce the rate of memory consumption increasing to about 200MB/hr. Thanks Gary. Wed Apr 26 12:50:08 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (3h) 7m ago 3h 1099M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 7m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:00:09 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (3h) 6m ago 3h 1161M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 6m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:10:10 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (3h) 6m ago 3h 1230M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 6m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:20:11 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 6m ago 4h 1250M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 6m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:30:12 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 5m ago 4h 1318M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 5m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:40:13 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 5m ago 4h 1379M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 5m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:50:13 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 5m ago 4h 1442M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 5m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:00:14 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 4m ago 4h 1498M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 4m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:10:15 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 4m ago 4h 1554M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 4m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:20:16 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 4m ago 5h 1617M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 3m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:30:17 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 3m ago 5h 1677M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 3m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:40:18 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 3m ago 5h 1735M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 3m ago 18h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:50:19 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 2m ago 5h 1806M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 2m ago 18h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:00:20 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 2m ago 5h 1873M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 2m ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:10:21 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 2m ago 5h 1920M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 2m ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:20:21 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 114s ago 6h 1981M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 113s ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:30:22 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 95s ago 6h 2048M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 95s ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:40:23 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 80s ago 6h 2121M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 80s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:50:24 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 53s ago 6h 2170M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 53s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:00:25 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 35s ago 6h 2238M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 34s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:10:26 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 19s ago 6h 2266M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 19s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:20:27 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 10m ago 7h 2266M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 10m ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:30:27 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 9m ago 7h 2321M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 9m ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:40:28 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 9m ago 7h 2382M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 9m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:50:29 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 9m ago 7h 2449M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 9m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:00:30 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 8m ago 7h 2505M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 8m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:10:31 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 8m ago 7h 2562M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 8m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:20:32 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 8m ago 8h 2628M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 8m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:30:33 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 7m ago 8h 2692M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 7m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:40:34 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 7m ago 8h 2761M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 7m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:50:34 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 7m ago 8h 2811M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 7m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:00:35 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 6m ago 8h 2873M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 6m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:10:36 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 6m ago 8h 2923M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 6m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:20:37 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 6m ago 9h 2995M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 6m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:30:38 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 5m ago 9h 3055M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 5m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:40:39 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 5m ago 9h 3110M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 5m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:50:40 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 5m ago 9h 3166M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 5m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:00:41 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 4m ago 9h 3244M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 4m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:10:42 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 4m ago 9h 3289M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 4m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:20:42 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 4m ago 10h 3368M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 4m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:30:43 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 3m ago 10h 3437M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 3m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:40:44 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 3m ago 10h 3496M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 3m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:50:45 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 3m ago 10h 3546M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 3m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:00:46 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 2m ago 10h 3627M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 2m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:10:47 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 2m ago 10h 3637M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 2m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:20:48 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 92s ago 11h 492M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 91s ago 23h 418M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:31:55 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 51s ago 11h 373M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 51s ago 23h 520M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:41:56 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 32s ago 11h 374M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 32s ago 0h 585M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:51:57 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 17s ago 11h 376M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 17s ago 0h 647M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:01:58 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 10m ago 11h 376M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 10m ago 0h 647M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:11:58 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 9m ago 11h 378M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 9m ago 0h 698M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:21:59 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 9m ago 12h 379M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 9m ago 0h 758M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:32:00 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 9m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 9m ago 0h 825M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:42:01 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 8m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 8m ago 1h 895M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:52:02 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 8m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 8m ago 1h 948M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:02:03 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 8m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 8m ago 1h 1010M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:12:04 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 7m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 7m ago 1h 1071M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:22:05 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 7m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 7m ago 1h 1132M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:32:06 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 7m ago 13h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 7m ago 1h 1183M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:42:06 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 6m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 6m ago 2h 1244M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:52:07 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 6m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 6m ago 2h 1311M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:02:08 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 6m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 6m ago 2h 1380M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:12:09 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 5m ago 13h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 5m ago 2h 1441M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:22:10 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 5m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 5m ago 2h 1508M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:32:11 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 5m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 5m ago 2h 1541M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:42:12 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 4m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 4m ago 3h 1610M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:52:13 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 4m ago 14h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 4m ago 3h 1674M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:02:14 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 4m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 4m ago 3h 1725M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:12:14 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 3m ago 14h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 3m ago 3h 1794M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:22:15 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 3m ago 15h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 3m ago 3h 1844M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:32:16 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 3m ago 15h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 3m ago 3h 1917M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:42:17 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 2m ago 15h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 2m ago 4h 2002M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:52:18 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 2m ago 15h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 2m ago 4h 2056M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:02:19 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 2m ago 15h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 2m ago 4h 2113M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:12:20 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 92s ago 15h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 91s ago 4h 2173M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:22:21 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 73s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 72s ago 4h 2201M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:32:22 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 57s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 57s ago 4h 2264M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:42:22 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 29s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 29s ago 5h 2324M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:52:23 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 10s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 10s ago 5h 2387M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:02:24 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 10m ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 10m ago 5h 2387M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:12:25 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 9m ago 16h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 9m ago 5h 2442M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:22:26 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 9m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 9m ago 5h 2502M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:32:27 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 9m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 9m ago 5h 2552M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:42:28 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 8m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 8m ago 6h 2614M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:52:28 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 8m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 8m ago 6h 2681M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:02:29 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 8m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 8m ago 6h 2748M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:12:30 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 7m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 7m ago 6h 2810M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:22:31 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 7m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 7m ago 6h 2867M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:32:32 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 7m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 7m ago 6h 2916M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:42:33 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 6m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 6m ago 7h 3024M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:52:34 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 6m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 6m ago 7h 3070M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:02:35 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 6m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 6m ago 7h 3127M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:12:36 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 5m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 5m ago 7h 3183M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:22:36 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 5m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 5m ago 7h 3215M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:32:37 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 4m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 4m ago 7h 3290M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:42:38 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 4m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 4m ago 8h 3342M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:52:39 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 4m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 4m ago 8h 3392M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:02:40 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 3m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 3m ago 8h 3451M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:12:41 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 3m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 3m ago 8h 3516M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:22:42 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 3m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 3m ago 8h 3566M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:32:43 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 2m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 2m ago 8h 3632M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:42:44 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 2m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 2m ago 9h 3696M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:52:44 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 2m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 2m ago 9h 3760M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:02:45 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 115s ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 114s ago 9h 3813M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:12:46 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 99s ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 99s ago 9h 3869M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:22:47 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 72s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 72s ago 9h 3923M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:32:48 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 54s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 53s ago 9h 3990M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:42:49 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 38s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 38s ago 10h 4040M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:52:50 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 11s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 10s ago 10h 4097M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 07:02:51 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 10m ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 10m ago 10h 4097M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 07:12:51 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 9m ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 9m ago 10h 4157M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 07:22:52 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (22h) 9m ago 22h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 9m ago 10h 4213M - 16.2.12 921c4e969fff 6fed006895e8 -- Gary Molenkamp Science Technology Services Systems Administrator University of Western Ontario molenkam(a)uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566

1 1

Balancing Reads in Ceph
by Alan Nair 02 May '23

02 May '23

Hi. I am currently using Ceph for replicated storage to store many objects across 5 nodes with 3x replication. When I generate ~1000 read requests to a single object, they all get serviced by the same primary OSD. I would like to balance the reads across the replicas. So I use the following: auto read_op = rados_create_read_op(); rados_read_op_read(read_op, offset, outSize, buffer, &bytes_read, &prval); err = rados_read_op_operate(read_op, pool->ioctx, keyName.c_str(), LIBRADOS_OPERATION_BALANCE_READS); However, this does not seem to balance the reads across replicas. I do not see what I am doing wrong in the above code. Could you please guide me on this? ceph-mon and ceph-osd are run on Ubuntu 22.04 installed via apt-get update ceph ceph-mds ceph-volume If I should ask this question somewhere else, please point me in the right direction. Thanks and regards, Alan. The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th' ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

1 0

PVE CEPH OSD heartbeat show
by Peter 02 May '23

02 May '23

Dear all, We are experiencing with Ceph after deploying it by PVE with the network backed by a 10G Cisco switch with VPC feature on. We are encountering a slow OSD heartbeat and have not been able to identify any network traffic issues. Upon checking, we found that the ping is around 0.1ms, and there is occasional 2% packet loss when using flood ping, but not consistently. We also noticed a large number of UDP port 5405 packets and the 'corosync' process utilizing a significant amount of CPU. When running the 'ceph -s' command, we observed a slow OSD heartbeat on the back and front, with the longest latency being 2250.54ms. We suspect that this may be a network issue, but we are unsure of how Ceph detects such long latency. Additionally, we are wondering if a 2% packet loss can significantly affect Ceph's performance and even cause the OSD process to fail sometimes. We have heard about potential issues with rockdb 6 causing OSD process failures, and we are curious about how to check the rockdb version. Furthermore, we are wondering how severe traffic package loss and latency must be to cause OSD process crashes, and how the monitoring system determines that an OSD is offline. We would greatly appreciate any assistance or insights you could provide on these matters. Thanks,

3 4

Re: cephfs - max snapshot limit?
by MARTEL Arnaud 02 May '23

02 May '23

Hi Venky, > Also, at one point the kclient wasn't able to handle more than 400 snapshots (per file system), but we have come a long way from that and that is not a constraint right now. Does it mean that there is no more limit to the number of snapshots per filesystem? And, if not, do you know what is the max number of snapshots now (per filesystem) ?? Cheers, Arnaud

2 1

[multisite] Resetting an empty bucket
by Yixin Jin 01 May '23

01 May '23

Hi folks, Armed with bucket-specific sync policy feature, I found that we could move objects of a bucket between zones. It is migration via sync followed by object removal at the source. This allows us to better utilize available capacities in different clusters/zones. However, to achieve this, we need a way to reset an empty bucket so that it can serve as a destination for a migration after it serves as a source before. ceph/rgw currently doesn't seem to be able to do that. So I create a feature request for it https://tracker.ceph.com/issues/59593 My own prototype shows that this feature is fairly simply to implement and works well for bucket migration. Cheers, Yixin

2 1

Ceph recovery
by wodel youchi 01 May '23

01 May '23

Hi, When creating a ceph cluster, a failover domain is created, and by default it uses host as a minimal domain, that domain can be modified to chassis, or rack, ...etc. My question is : Suppose I have three osd nodes, my replication is 3 and my failover domain is host, which means that each copy of data is stored on a different node. What happens when one node crashes, does Ceph use the remaining free space on the other two to create the third copy, or the ceph cluster will run in degraded mode, like a RAID5 which lost a disk. Regards.

2 2

Nearly 1 exabyte of Ceph storage
by Yaarit Hatuka 01 May '23

01 May '23

Hi everyone, We are excited to share with you the latest statistics from our Ceph public telemetry dashboards <https://telemetry-public.ceph.com/>. With almost 2,500 clusters reporting nearly 1 exabyte of total storage capacity, we can see a steady growth rate <https://telemetry-public.ceph.com/d/ZFYuv1qWz/telemetry?viewPanel=19&orgId=…> since the telemetry service was launched. One of the things telemetry helps us to understand is version adoption rate. See, for example, the trend of Quincy <https://telemetry-public.ceph.com/d/ZFYuv1qWz/telemetry?viewPanel=28&orgId=…> deployments in the community. We can also learn that the average cluster size grows over time <https://telemetry-public.ceph.com/d/ZFYuv1qWz/telemetry?viewPanel=33&orgId=1>. Ceph telemetry is on an opt-in basis. You can opt-in with: `ceph telemetry on` Learn more here <https://docs.ceph.com/en/latest/mgr/telemetry/#enabling-telemetry>. Help us cross the exabyte mark by opting-in today! Learn more about the latest developments around Telemetry <https://sched.co/1JKZ2> at the upcoming Cephalocon. Thank you for your contribution, Yaarit [image: ceph_telemetry_dashboards.jpg]

2 3

RGW Lua - cancel request
by Ondřej Kukla 01 May '23

01 May '23

Hello, Lately I’ve been playing with Lua scripting on top of RGW. I would like to implement a request blocking based on bucket name -> when there is a dot in a bucket name return error code and a message that this name is invalid. Here is the code I was able to came up with. if string.find(Request.HTTP.URI, '%.') then Request.Response.HTTPStatusCode = 400 Request.Response.HTTPStatus = “InvalidBucketName" Request.Response.Message = “Dots in bucket name are not allowed." end This works fine, but the request for creating a bucket would be processed and the bucket will be created. I thought about a dirty workaround with setting the Request.Bucket.Name to a bucket that already exists but it seems that this field is not writable in Quincy. Is there a way to block the request from processing? Any help is much appreciated. Kind regards, Ondrej

3 3