May 2023 - ceph-users - lists.ceph.io

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

Help needed to configure erasure coding LRC plugin
by Michel Jouvin 16 Oct '24

16 Oct '24

Hi, As discussed in another thread (Crushmap rule for multi-datacenter erasure coding), I'm trying to create an EC pool spanning 3 datacenters (datacenters are present in the crushmap), with the objective to be resilient to 1 DC down, at least keeping the readonly access to the pool and if possible the read-write access, and have a storage efficiency better than 3 replica (let say a storage overhead <= 2). In the discussion, somebody mentioned LRC plugin as a possible jerasure alternative to implement this without tweaking the crushmap rule to implement the 2-step OSD allocation. I looked at the documentation (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/) but I have some questions if someone has experience/expertise with this LRC plugin. I tried to create a rule for using 5 OSDs per datacenter (15 in total), with 3 (9 in total) being data chunks and others being coding chunks. For this, based of my understanding of examples, I used k=9, m=3, l=4. Is it right? Is this configuration equivalent, in terms of redundancy, to a jerasure configuration with k=9, m=6? The resulting rule, which looks correct to me, is: -------- { "rule_id": 6, "rule_name": "test_lrc_2", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 15, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -4, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 5, "type": "host" }, { "op": "emit" } ] } ------------ Unfortunately, it doesn't work as expected: a pool created with this rule ends up with its pages active+undersize, which is unexpected for me. Looking at 'ceph health detail` output, I see for each page something like: pg 52.14 is stuck undersized for 27m, current state active+undersized, last acting [90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647] For each PG, there is 3 '2147483647' entries and I guess it is the reason of the problem. What are these entries about? Clearly it is not OSD entries... Looks like a negative number, -1, which in terms of crushmap ID is the crushmap root (named "default" in our configuration). Any trivial mistake I would have made? Thanks in advance for any help or for sharing any successful configuration? Best regards, Michel

4 24

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

Re: MDS crashes to damaged metadata
by Patrick Donnelly 10 Jun '24

10 Jun '24

On Thu, Dec 15, 2022 at 9:32 AM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): > > 2022-12-08T13:05:48.919+0100 7f440afec700 -1 /build/ceph-16.2.10/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f440afec700 time 2022-12-08T13:05:48.921223+0100 > /build/ceph-16.2.10/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE) > > 12 Minutes lates the unlink_local error crashes appeared again. This time with a new file. During debugging we noticed a MTU mismatch between MDS (1500) and client (9000) with cephfs kernel mount. The client is also creating the snapshots via mkdir in the .snap directory. > > We disabled snapshot creation for now, but really need this feature. I uploaded the mds logs of the first crash along with the information above to https://tracker.ceph.com/issues/38452 > > I would greatly appreciate it, if you could answer me the following question: > > Is the Bug related to our MTU Mismatch? We fixed the MTU Issue going back to 1500 on all nodes in the ceph public network on the weekend also. I doubt it. > If you need a debug level 20 log of the ScatterLock for further analysis, i could schedule snapshots at the end of our workdays and increase the debug level 5 Minutes arround snap shot creation. This would be very helpful! -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 4

Re: Recoveries without any misplaced objects?
by Hector Martin 24 Apr '24

24 Apr '24

On 29/05/2023 20.55, Anthony D'Atri wrote: > Check the uptime for the OSDs in question I restarted all my OSDs within the past 10 days or so. Maybe OSD restarts are somehow breaking these stats? > >> On May 29, 2023, at 6:44 AM, Hector Martin <marcan(a)marcan.st> wrote: >> >> Hi, >> >> I'm watching a cluster finish a bunch of backfilling, and I noticed that >> quite often PGs end up with zero misplaced objects, even though they are >> still backfilling. >> >> Right now the cluster is down to 6 backfilling PGs: >> >> data: >> volumes: 1/1 healthy >> pools: 6 pools, 268 pgs >> objects: 18.79M objects, 29 TiB >> usage: 49 TiB used, 25 TiB / 75 TiB avail >> pgs: 262 active+clean >> 6 active+remapped+backfilling >> >> But there are no misplaced objects, and the misplaced column in `ceph pg >> dump` is zero for all PGs. >> >> If I do a `ceph pg dump_json`, I can see `num_objects_recovered` >> increasing for these PGs... but the misplaced count is still 0. >> >> Is there something else that would cause recoveries/backfills other than >> misplaced objects? Or perhaps there is a bug somewhere causing the >> misplaced object count to be misreported as 0 sometimes? >> >> # ceph -v >> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy >> (stable) >> >> - Hector >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > - Hector

3 3

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

Why you might want packages not containers for Ceph deployments
by Matthew Vernon 25 Mar '24

25 Mar '24

Hi, In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0]. I'd like to suggest, briefly, that: * containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes. For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice. Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages): * you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice) We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part. Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems! I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder". I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :) Regards, and apologies for the wall of text, Matthew [0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

26 65

Upgrading from Pacific to Quincy fails with "Unexpected error"
by Reza Bakhshayeshi 13 Mar '24

13 Mar '24

Hi all, I have a problem regarding upgrading Ceph cluster from Pacific to Quincy version with cephadm. I have successfully upgraded the cluster to the latest Pacific (16.2.11). But when I run the following command to upgrade the cluster to 17.2.5, After upgrading 3/4 mgrs, the upgrade process stops with "Unexpected error". (everything is on a private network) ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v17.2.5 I also tried the 17.2.4 version. cephadm fails to check the hosts' status and marks them as offline: cephadm 2023-04-06T10:19:59.998510+0000 mgr.host9.arhpnd (mgr.4516356) 5782 : cephadm [DBG] host host4 (x.x.x.x) failed check cephadm 2023-04-06T10:19:59.998553+0000 mgr.host9.arhpnd (mgr.4516356) 5783 : cephadm [DBG] Host "host4" marked as offline. Skipping daemon refresh cephadm 2023-04-06T10:19:59.998581+0000 mgr.host9.arhpnd (mgr.4516356) 5784 : cephadm [DBG] Host "host4" marked as offline. Skipping gather facts refresh cephadm 2023-04-06T10:19:59.998609+0000 mgr.host9.arhpnd (mgr.4516356) 5785 : cephadm [DBG] Host "host4" marked as offline. Skipping network refresh cephadm 2023-04-06T10:19:59.998633+0000 mgr.host9.arhpnd (mgr.4516356) 5786 : cephadm [DBG] Host "host4" marked as offline. Skipping device refresh cephadm 2023-04-06T10:19:59.998659+0000 mgr.host9.arhpnd (mgr.4516356) 5787 : cephadm [DBG] Host "host4" marked as offline. Skipping osdspec preview refresh cephadm 2023-04-06T10:19:59.998682+0000 mgr.host9.arhpnd (mgr.4516356) 5788 : cephadm [DBG] Host "host4" marked as offline. Skipping autotune cluster 2023-04-06T10:20:00.000151+0000 mon.host8 (mon.0) 158587 : cluster [ERR] Health detail: HEALTH_ERR 9 hosts fail cephadm check; Upgrade: failed due to an unexpected exception cluster 2023-04-06T10:20:00.000191+0000 mon.host8 (mon.0) 158588 : cluster [ERR] [WRN] CEPHADM_HOST_CHECK_FAILED: 9 hosts fail cephadm check cluster 2023-04-06T10:20:00.000202+0000 mon.host8 (mon.0) 158589 : cluster [ERR] host host7 (x.x.x.x) failed check: Unable to reach remote host host7. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000213+0000 mon.host8 (mon.0) 158590 : cluster [ERR] host host2 (x.x.x.x) failed check: Unable to reach remote host host2. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000220+0000 mon.host8 (mon.0) 158591 : cluster [ERR] host host8 (x.x.x.x) failed check: Unable to reach remote host host8. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000228+0000 mon.host8 (mon.0) 158592 : cluster [ERR] host host4 (x.x.x.x) failed check: Unable to reach remote host host4. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000240+0000 mon.host8 (mon.0) 158593 : cluster [ERR] host host3 (x.x.x.x) failed check: Unable to reach remote host host3. Process exited with non-zero exit status 3 and here are some outputs of the commands: [root@host8 ~]# ceph -s cluster: id: xxx health: HEALTH_ERR 9 hosts fail cephadm check Upgrade: failed due to an unexpected exception services: mon: 5 daemons, quorum host8,host1,host7,host2,host9 (age 2w) mgr: host9.arhpnd(active, since 105m), standbys: host8.jowfih, host1.warjsr, host2.qyavjj mds: 1/1 daemons up, 3 standby osd: 37 osds: 37 up (since 8h), 37 in (since 3w) data: io: client: progress: Upgrade to 17.2.5 (0s) [............................] [root@host8 ~]# ceph orch upgrade status { "target_image": "my-private-repo/quay-io/ceph/ceph@sha256 :34c763383e3323c6bb35f3f2229af9f466518d9db926111277f5e27ed543c427", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "3/59 daemons upgraded", "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception", "is_paused": true } [root@host8 ~]# ceph cephadm check-host host7 check-host failed: Host 'host7' not found. Use 'ceph orch host ls' to see all managed hosts. [root@host8 ~]# ceph versions { "mon": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 1, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 }, "osd": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 37 }, "mds": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 4 }, "overall": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 47, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 } } The strange thing is I can rollback the cluster status by failing to not-upgraded mgr like this: ceph mgr fail ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v16.2.11 Would you happen to have any idea about this? Best regards, Reza

4 9

Re: increasing number of (deep) scrubs
by Frank Schilder 13 Dec '23

13 Dec '23

Hi Dan, thanks for your answer. I don't have a problem with increasing osd_max_scrubs (=1 at the moment) as such. I would simply prefer a somewhat finer grained way of controlling scrubbing than just doubling or tripling it right away. Some more info. These 2 pools are data pools for a large FS. Unfortunately, we have a large percentage of small files, which is a pain for recovery and seemingly also for deep scrubbing. Our OSDs are about 25% used and I had to increase the warning interval already to 2 weeks. With all the warning grace parameters this means that we manage to deep scrub everything about every month. I need to plan for 75% utilisation and a 3 months period is a bit far on the risky side. Our data is to a large percentage cold data. Client reads will not do the check for us, we need to combat bit-rot pro-actively. The reasons I'm interested in parameters initiating more scrubs while also converting more scrubs into deep scrubs are, that 1) scrubs seem to complete very fast. I almost never catch a PG in state "scrubbing", I usually only see "deep scrubbing". 2) I suspect the low deep-scrub count is due to a low number of deep-scrubs scheduled and not due to conflicting per-OSD deep scrub reservations. With the OSD count we have and the distribution over 12 servers I would expect at least a peak of 50% OSDs being active in scrubbing instead of the 25% peak I'm seeing now. It ought to be possible to schedule more PGs for deep scrub than actually are. 3) Every OSD having only 1 deep scrub active seems to have no measurable impact on user IO. If I could just get more PGs scheduled with 1 deep scrub per OSD it would already help a lot. Once this is working, I can eventually increase osd_max_scrubs when the OSDs fill up. For now I would just like that (deep) scrub scheduling looks a bit harder and schedules more eligible PGs per time unit. If we can get deep scrubbing up to an average of 42PGs completing per hour with keeping osd_max_scrubs=1 to maintain current IO impact, we should be able to complete a deep scrub with 75% full OSDs in about 30 days. This is the current tail-time with 25% utilisation. I believe currently a deep scrub of a PG in these pools takes 2-3 hours. Its just a gut feeling from some repair and deep-scrub commands, I would need to check logs for more precise info. Increasing osd_max_scrubs would then be a further and not the only option to push for more deep scrubbing. My expectation would be that values of 2-3 are fine due to the increasingly higher percentage of cold data for which no interference with client IO will happen. Hope that makes sense and there is a way beyond bumping osd_max_scrubs to increase the number of scheduled and executed deep scrubs. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dvanders(a)gmail.com> Sent: 05 January 2023 15:36 To: Frank Schilder Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] increasing number of (deep) scrubs Hi Frank, What is your current osd_max_scrubs, and why don't you want to increase it? With 8+2, 8+3 pools each scrub is occupying the scrub slot on 10 or 11 OSDs, so at a minimum it could take 3-4x the amount of time to scrub the data than if those were replicated pools. If you want the scrub to complete in time, you need to increase the amount of scrub slots accordingly. On the other hand, IMHO the 1-week deadline for deep scrubs is often much too ambitious for large clusters -- increasing the scrub intervals is one solution, or I find it simpler to increase mon_warn_pg_not_scrubbed_ratio and mon_warn_pg_not_deep_scrubbed_ratio until you find a ratio that works for your cluster. Of course, all of this can impact detection of bit-rot, which anyway can be covered by client reads if most data is accessed periodically. But if the cluster is mostly idle or objects are generally not read, then it would be preferable to increase slots osd_max_scrubs. Cheers, Dan On Tue, Jan 3, 2023 at 2:30 AM Frank Schilder <frans(a)dtu.dk> wrote: > > Hi all, > > we are using 16T and 18T spinning drives as OSDs and I'm observing that they are not scrubbed as often as I would like. It looks like too few scrubs are scheduled for these large OSDs. My estimate is as follows: we have 852 spinning OSDs backing a 8+2 pool with 2024 and an 8+3 pool with 8192 PGs. On average I see something like 10PGs of pool 1 and 12 PGs of pool 2 (deep) scrubbing. This amounts to only 232 out of 852 OSDs scrubbing and seems to be due to a conservative rate of (deep) scrubs being scheduled. The PGs (dep) scrub fairly quickly. > > I would like to increase gently the number of scrubs scheduled for these drives and *not* the number of scrubs per OSD. I'm looking at parameters like: > > osd_scrub_backoff_ratio > osd_deep_scrub_randomize_ratio > > I'm wondering if lowering osd_scrub_backoff_ratio to 0.5 and, maybe, increasing osd_deep_scrub_randomize_ratio to 0.2 would have the desired effect? Are there other parameters to look at that allow gradual changes in the number of scrubs going on? > > Thanks a lot for your help! > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 3

Re: RGW access logs with bucket name
by Boris Behrens 02 Nov '23

02 Nov '23

Bringing up that topic again: is it possible to log the bucket name in the rgw client logs? currently I am only to know the bucket name when someone access the bucket via https://TLD/bucket/object instead of https://bucket.TLD/object. Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens <bb(a)kervyn.de>: > Hi, > I am looking forward to move our logs from > /var/log/ceph/ceph-client...log to our logaggregator. > > Is there a way to have the bucket name in the log file? > > Or can I write the rgw_enable_ops_log into a file? Maybe I could work with > this. > > Cheers and happy new year > Boris > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

4 6

Re: Status of Quincy 17.2.5 ?
by Christian Rohmann 30 Oct '23

30 Oct '23

Hey everyone, On 20/10/2022 10:12, Christian Rohmann wrote: > 1) May I bring up again my remarks about the timing: > > On 19/10/2022 11:46, Christian Rohmann wrote: > >> I believe the upload of a new release to the repo prior to the >> announcement happens quite regularly - it might just be due to the >> technical process of releasing. >> But I agree it would be nice to have a more "bit flip" approach to >> new releases in the repo and not have the packages appear as updates >> prior to the announcement and final release and update notes. > By my observations sometimes there are packages available on the > download servers via the "last stable" folders such as > https://download.ceph.com/debian-quincy/ quite some time before the > announcement of a release is out. > I know it's hard to time this right with mirrors requiring some time > to sync files, but would be nice to not see the packages or have > people install them before there are the release notes and potential > pointers to changes out. Todays 16.2.11 release shows the exact issue I described above .... 1) 16.2.11 packages are already available via e.g. https://download.ceph.com/debian-pacific 2) release notes not yet merged: (https://github.com/ceph/ceph/pull/49839) thus https://ceph.io/en/news/blog/2022/v16-2-11-pacific-released/ show a 404 :-) 3) No announcement like https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/QOCU563UD3… to the ML yet. Regards Christian

2 2

[Pacific] ceph orch device ls do not returns any HDD
by Patrick Begou 27 Oct '23

27 Oct '23

Hi everyone I'm new to CEPH, just a french 4 days training session with Octopus on VMs that convince me to build my first cluster. At this time I have 4 old identical nodes for testing with 3 HDDs each, 2 network interfaces and running Alma Linux8 (el8). I try to replay the training session but it fails, breaking the web interface because of some problems with podman 4.2 not compatible with Octopus. So I try to deploy Pacific with cephadm tool on my first node (mostha1) (to enable testing also an upgrade later). dnf -y install https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noar… monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }') cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \ --initial-dashboard-user admceph \ --allow-fqdn-hostname --cluster-network 10.1.0.0/16 This was sucessfull. But running "*c**eph orch device ls*" do not show any HDD even if I have /dev/sda (used by the OS), /dev/sdb and /dev/sdc The web interface shows a row capacity which is an aggregate of the sizes of the 3 HDDs for the node. I've also tried to reset /dev/sdb but cephadm do not see it: [ceph: root@mostha1 /]# ceph orch device zap mostha1.legi.grenoble-inp.fr /dev/sdb --force Error EINVAL: Device path '/dev/sdb' not found on host 'mostha1.legi.grenoble-inp.fr' On my first attempt with octopus, I was able to list the available HDD with this command line. Before moving to Pacific, the OS on this node has been reinstalled from scratch. Any advices for a CEPH beginner ? Thanks Patrick

9 51

libceph: mds1 IP+PORT wrong peer at address
by Frank Schilder 20 Sep '23

20 Sep '23

Hi all, we seem to have hit a bug in the ceph fs kernel client and I just want to confirm what action to take. We get the error "wrong peer at address" in dmesg and some jobs on that server seem to get stuck in fs access; log extract below. I found these 2 tracker items https://tracker.ceph.com/issues/23883 and https://tracker.ceph.com/issues/41519, which don't seem to have fixes. My questions: - Is this harmless or does it indicate invalid/corrupted client cache entries? - How to resolve, ignore, umount+mount or reboot? Here an extract from the dmesg log, the error has survived a couple of MDS restarts already: [Mon Mar 6 12:56:46 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:05:18 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-1572619386 [Mon Mar 6 13:05:18 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:13:50 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-1572619386 [Mon Mar 6 13:13:50 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state OPEN) [Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state OPEN) [Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start [Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start [Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success [Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success [Mon Mar 6 13:18:13 2023] ceph: update_snap_trace error -22 [Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state OPEN) [Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state OPEN) [Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed [Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect success [Mon Mar 6 13:18:29 2023] ceph: mds7 reconnect success [Mon Mar 6 13:18:35 2023] ceph: update_snap_trace error -22 [Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed [Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed [Mon Mar 6 13:22:22 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Mon Mar 6 13:22:22 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:30:54 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [...] [Thu Mar 9 09:37:24 2023] slurm.epilog.cl (31457): drop_caches: 3 [Thu Mar 9 09:38:26 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:38:26 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 09:46:58 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:46:58 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 09:55:30 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:55:30 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 10:04:02 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 10:04:02 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 6

quincy 17.2.6 - write performance continuously slowing down until OSD restart needed
by Nikola Ciprich 22 Jul '23

22 Jul '23

Hello dear CEPH users and developers, we're dealing with strange problems.. we're having 12 node alma linux 9 cluster, initially installed CEPH 15.2.16, then upgraded to 17.2.5. It's running bunch of KVM virtual machines accessing volumes using RBD. everything is working well, but there is strange and for us quite serious issue - speed of write operations (both sequential and random) is constantly degrading drastically to almost unusable numbers (in ~1week it drops from ~70k 4k writes/s from 1 VM to ~7k writes/s) When I restart all OSD daemons, numbers immediately return to normal.. volumes are stored on replicated pool of 4 replicas, on top of 7*12 = 84 INTEL SSDPE2KX080T8 NVMEs. I've updated cluster to 17.2.6 some time ago, but the problem persists. This is especially annoying in connection with https://tracker.ceph.com/issues/56896 as restarting OSDs is quite painfull when half of them crash.. I don't see anything suspicious, nodes load is quite low, no logs errors, network latency and throughput is OK too Anyone having simimar issue? I'd like to ask for hints on what should I check further.. we're running lots of 14.2.x and 15.2.x clusters, none showing similar issue, so I'm suspecting this is something related to quincy thanks a lot in advance with best regards nikola ciprich -- ------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis(a)linuxbox.cz -------------------------------------

7 16

rgw multisite sync not syncing data, error: RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data log shards
by Christian Rohmann 20 Jul '23

20 Jul '23

Hey ceph-users, I setup a multisite sync between two freshly setup Octopus clusters. In the first cluster I created a bucket with some data just to test the replication of actual data later. I then followed the instructions on https://docs.ceph.com/en/octopus/radosgw/multisite/#migrating-a-single-site… to add a second zone. Things went well and both zones are now happily reaching each other and the API endpoints are talking. Also the metadata is in sync already - both sides are happy and I can see bucket listings and users are "in sync": > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > metadata sync no sync (zone is master) > data sync source: c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > and on the other side ... > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > metadata sync syncing > full sync: 0/64 shards > incremental sync: 64/64 shards > metadata is caught up with master > data sync source: 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > also the newly created buckets (read: their metadata) is synced. What is apparently not working in the sync of actual data. Upon startup the radosgw on the second site shows: > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: start > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: realm > epoch=2 period id=f4553d7c-5cc5-4759-9253-9a22b051e736 > 2021-06-25T16:15:11.525+0000 7fe71dff3700 0 > RGW-SYNC:data:sync:init_data_sync_status: ERROR: failed to read remote > data log shards > also when issuing # radosgw-admin data sync init --source-zone obst-rgn it throws > 2021-06-25T16:20:29.167+0000 7f87c2aec080 0 > RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data > log shards Does anybody have any hints on where to look for what could be broken here? Thanks a bunch, Regards Christian

3 4

Re: Ceph Mgr/Dashboard Python depedencies: a new approach
by Casey Bodley 14 Jul '23

14 Jul '23

hi Ernesto and lists, > [1] https://github.com/ceph/ceph/pull/47501 are we planning to backport this to quincy so we can support centos 9 there? enabling that upgrade path on centos 9 was one of the conditions for dropping centos 8 support in reef, which i'm still keen to do if not, can we find another resolution to https://tracker.ceph.com/issues/58832? as i understand it, all of those python packages exist in centos 8. do we know why they were dropped for centos 9? have we looked into making those available in epel? (cc Ken and Kaleb) On Fri, Sep 2, 2022 at 12:01 PM Ernesto Puerta <epuertat(a)redhat.com> wrote: > > Hi Kevin, > >> >> Isn't this one of the reasons containers were pushed, so that the packaging isn't as big a deal? > > > Yes, but the Ceph community has a strong commitment to provide distro packages for those users who are not interested in moving to containers. > >> Is it the continued push to support lots of distros without using containers that is the problem? > > > If not a problem, it definitely makes it more challenging. Compiled components often sort this out by statically linking deps whose packages are not widely available in distros. The approach we're proposing here would be the closest equivalent to static linking for interpreted code (bundling). > > Thanks for sharing your questions! > > Kind regards, > Ernesto > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io

4 10

cephadm upgrade 16.2.10 to 16.2.11: osds crash and get stuck restarting
by Zakhar Kirpichenko 14 Jul '23

14 Jul '23

Hi, Attempted to upgrade 16.2.10 to 16.2.11, 2 OSDs out of many started crashing in a loop on the very 1st host: Jan 25 23:07:53 ceph01 bash[2553123]: Uptime(secs): 0.0 total, 0.0 interval Jan 25 23:07:53 ceph01 bash[2553123]: Flush(GB): cumulative 0.000, interval 0.000 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(GB): cumulative 0.000, interval 0.000 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(Total Files): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(L0 Files): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(Keys): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Jan 25 23:07:53 ceph01 bash[2553123]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Jan 25 23:07:53 ceph01 bash[2553123]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count Jan 25 23:07:53 ceph01 bash[2553123]: ** File Read Latency Histogram By Level [P] ** Jan 25 23:07:53 ceph01 bash[2553123]: debug -10> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986439) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -9> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986493) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -8> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986500) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -7> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986505) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -6> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: [compaction/compaction_job.cc:1676] [O-2] [JOB 9] Compacting 4@0 + 2@1 files to L1, score 1.0 0 Jan 25 23:07:53 ceph01 bash[2553123]: debug -5> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: [compaction/compaction_job.cc:1680] [O-2] Compaction start summary: Base version 33 Base level 0, inputs: [649058(959KB) 649046(1510KB) 649024(1323KB) 649002(1396KB)], [648981(66MB) 648982(52MB)] Jan 25 23:07:53 ceph01 bash[2553123]: debug -4> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1674688072986547, "job": 9, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [649058, 649046, 649024, 649002], "files_L1": [648981, 648982], "score": 1, "input_data_size": 129161327} Jan 25 23:07:53 ceph01 bash[2553123]: debug -3> 2023-01-25T23:07:52.990+0000 7f7c8b66e700 -1 bdev(0x5619bf0ce400 /var/lib/ceph/osd/ceph-3/block) _aio_thread got r=-1 ((1) Operation not permitted) Jan 25 23:07:53 ceph01 bash[2553123]: debug -2> 2023-01-25T23:07:52.990+0000 7f7c8b66e700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.11/rpm/el8/BUILD/ceph-16.2.11/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f7c8b66e700 time 2023-01-25T23:07:52.993976+0000 Jan 25 23:07:53 ceph01 bash[2553123]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.11/rpm/el8/BUILD/ceph-16.2.11/src/blk/kernel/KernelDevice.cc: 604: ceph_abort_msg("Unexpected IO error. This may suggest HW issue. Please check your dmesg!") Jan 25 23:07:53 ceph01 bash[2553123]: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) Jan 25 23:07:53 ceph01 bash[2553123]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x5619b2fc1adc] Jan 25 23:07:53 ceph01 bash[2553123]: 2: (KernelDevice::_aio_thread()+0x1285) [0x5619b3b2a4e5] Jan 25 23:07:53 ceph01 bash[2553123]: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5619b3b357b1] Jan 25 23:07:53 ceph01 bash[2553123]: 4: /lib64/libpthread.so.0(+0x81ca) [0x7f7c978851ca] Jan 25 23:07:53 ceph01 bash[2553123]: 5: clone() Jan 25 23:07:53 ceph01 bash[2553123]: debug -1> 2023-01-25T23:07:52.994+0000 7f7c8b66e700 -1 *** Caught signal (Aborted) ** Jan 25 23:07:53 ceph01 bash[2553123]: in thread 7f7c8b66e700 thread_name:bstore_aio Jan 25 23:07:53 ceph01 bash[2553123]: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) Jan 25 23:07:53 ceph01 bash[2553123]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7c9788fcf0] Jan 25 23:07:53 ceph01 bash[2553123]: 2: gsignal() Jan 25 23:07:53 ceph01 bash[2553123]: 3: abort() Jan 25 23:07:53 ceph01 bash[2553123]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x5619b2fc1bad] Jan 25 23:07:53 ceph01 bash[2553123]: 5: (KernelDevice::_aio_thread()+0x1285) [0x5619b3b2a4e5] Jan 25 23:07:53 ceph01 bash[2553123]: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5619b3b357b1] Jan 25 23:07:53 ceph01 bash[2553123]: 7: /lib64/libpthread.so.0(+0x81ca) [0x7f7c978851ca] OSD kept crashing until the host reboot, an OSD restart wouldn't help. This hasn't happened during any previous upgrades, so was a rather unexpected development. Unclear what caused this, but a host reboot seems to have fixed it. It happened to 1 other OSD on another host, exactly the same symptoms, also solved by a reboot. Best regards, Zakhar

4 4

upgrading from 15.2.17 to 16.2.11 - Health ERROR
by xadhoom76＠gmail.com 04 Jul '23

04 Jul '23

hi , starting upgrade from 15.2.17 i got this error Module 'cephadm' has failed: Expecting value: line 1 column 1 (char 0) Cluster was in health ok before starting.

5 9

CephFS metadata pool grows by two orders of magnitude while trimming (?) snapshots
by Janek Bevendorff 19 Jun '23

19 Jun '23

Hi, Perhaps this is a known issue and I was simply too dumb to find it, but we are having problems with our CephFS metadata pool filling up over night. Our cluster has a small SSD pool of around 15TB which hosts our CephFS metadata pool. Usually, that's more than enough. The normal size of the pool ranges between 200 and 800GiB (which is quite a lot of fluctuation already). Yesterday, we had suddenly had the pool fill up entirely and they only way to fix it was to add more capacity. I increased the pool size to 18TB by adding more SSDs and could resolve the problem. After a couple of hours of reshuffling, the pool size finally went back to 230GiB. But then we had another fill-up tonight to 7.6TiB. Luckily, I had adjusted the weights so that not all disks could fill up entirely like last time, so it ended there. I wasn't really able to identify the problem yesterday, but under the more controllable scenario today, I could check the MDS logs at debug_mds=10 and to me it seems like the problem is caused by snapshot trimming. The logs contain a lot of snapshot-related messages for paths that haven't been touched in a long time. The messages all look something like this: May 31 09:16:48 XXX ceph-mds[2947525]: 2023-05-31T09:16:48.292+0200 7f7ce1bd9700 10 mds.1.cache.ino(0x1000b3c3670) add_client_cap first cap, joining realm snaprealm(0x10000000000 seq 1b1c lc 1b1b cr 1 b1b cps 2 snaps={185f=snap(185f 0x10000000000 'monthly_20221201' 2022-12-01T00:00:01.530830+0100),18de=snap(18de 0x10000000000 'monthly_20230101' 2023-01-01T00:00:04.657252+0100),1941=snap(1941 0x10000000000 ... May 31 09:25:03 XXX ceph-mds[3268481]: 2023-05-31T09:25:03.396+0200 7f0e6a6ca700 10 mds.0.cache | |______ 3 rep [dir 0x100000218fe.101111101* /storage/REDACTED/| ptrwaiter=0 request=0 child=0 frozen=0 subtree=1 replicated=0 dirty=0 waiter=0 authpin=0 tempexporting=0 0x5607759d9600] May 31 09:25:03 XXX ceph-mds[3268481]: 2023-05-31T09:25:03.452+0200 7f0e6a6ca700 10 mds.0.cache | | |____ 4 rep [dir 0x100000ff904.100111101010* /storage/REDACTED/| ptrwaiter=0 request=0 child=0 frozen=0 subtree=1 importing=0 replicated=0 waiter=0 authpin=0 tempexporting=0 0x56034ed25200] May 31 09:25:03 XXX ceph-mds[3268481]: 2023-05-31T09:25:03.716+0200 7f0e6becd700 10 mds.0.server set_trace_dist snaprealm snaprealm(0x10000000000 seq 1b1c lc 1b1b cr 1b1b cps 2 snaps={185f=snap(185f 0x10000000000 'monthly_20221201' 2022-12-01T00:00:01.530830+0100),18de=snap(18de 0x10000000000 'monthly_20230101' 2023-01-01T00:00:04.657252+0100),1941=snap(1941 0x10000000000 'monthly_20230201' 2023-02-01T00:00:01.854059+0100),19a6=snap(19a6 0x10000000000 'monthly_20230301' 2023-03-01T00:00:01.215197+0100),1a24=snap(1a24 0x10000000000 'monthly_20230401' ...) len=384 May 31 09:25:36 deltaweb055 ceph-mds[3268481]: 2023-05-31T09:25:36.076+0200 7f0e6becd700 10 mds.0.cache.ino(0x10004d74911) remove_client_cap last cap, leaving realm snaprealm(0x10000000000 seq 1b1c lc 1b1b cr 1b1b cps 2 snaps={185f=snap(185f 0x10000000000 'monthly_20221201' 2022-12-01T00:00:01.530830+0100),18de=snap(18de 0x10000000000 'monthly_20230101' ...) The daily_*, montly_* etc. names are the names of our regular snapshots. I posted a larger log file snippet using ceph-post-file with the ID: da0eb93d-f340-4457-8a3f-434e8ef37d36 Is it possible that the MDS are trimming old snapshots without taking care not to fill up the entire metadata pool? Cheers Janek

3 16

RGW versioned bucket index issues
by Cory Snyder 16 Jun '23

16 Jun '23

Hi all, I wanted to call attention to some RGW issues that we've observed on a Pacific cluster over the past several weeks. The problems relate to versioned buckets and index entries that can be left behind after transactions complete abnormally. The scenario is multi-faceted and we're still investigating some of the details, but I wanted to provide a big-picture summary of what we've found so far. It looks like most of these issues should be reproducible on versions before and after Pacific as well. I'll enumerate the individual issues below: 1. PUT requests during reshard of versioned bucket fail with 404 and leave behind dark data Tracker: https://tracker.ceph.com/issues/61359 2. When bucket index ops are cancelled it can leave behind zombie index entries This one was merged a few months ago and did make the v16.2.13 release, but in our case we had billions of extra index entries by the time that we had upgraded to the patched version. Tracker: https://tracker.ceph.com/issues/58673 3. Issuing a delete for a key that already has a delete marker as the current version leaves behind index entries and OLH objects Note that the tracker's original description describes the problem a bit differently, but I've clarified the nature of the issue in a comment. Tracker: https://tracker.ceph.com/issues/59663 The extra index entries and OLH objects that are left behind due to these sorts of issues are obviously annoying in regards to the fact that they unnecessarily consume space, but we've found that they can also cause severe performance degradation for bucket listings, lifecycle processing, and other ops indirectly due to higher osd latencies. The reason for the performance impact is that bucket listing calls must repeatedly perform additional OSD ops until they find the requisite number of entries to return. The OSD cls method for bucket listing also does its own internal iteration for the same purpose. Since these entries are invalid, they are skipped. In the case that we observed, where some of our bucket indexes were filled with a sea of contiguous leftover entries, the process of continually iterating over and skipping invalid entries caused enormous read amplification. I believe that the following tracker is describing symptoms that are related to the same issue: https://tracker.ceph.com/issues/59164. Note that this can also cause LC processing to repeatedly fail in cases where there are enough contiguous invalid entries, since the OSD cls code eventually gives up and returns an error that isn't handled. The severity of these issues likely varies greatly based upon client behavior. If anyone has experienced similar problems, we'd love to hear about the nature of how they've manifested for you so that we can be more confident that we've plugged all of the holes. Thanks, Cory Snyder 11:11 Systems

3 5

bucket notification retries
by Yuval Lifshitz 12 Jun '23

12 Jun '23

Dear Community, I would like to collect your feedback on this issue. This is a followup from a discussion that started in the RGW refactoring meeting on 31-May-23 (thanks @Krunal Chheda <kchheda3(a)bloomberg.net> for bringing up this topic!). Currently persistent notifications are retried indefinitely. The only limiting mechanism that exists is that all notifications to a specific topic are stored in one RADOS object (of size 128MB). Assuming notifications are ~1KB at most, this would give us at least 128K notifications that can wait in the queue. When the queue fills up (e.g. kafka broker is down for 20 minutes, we are sending ~100 notifications per second) we start sending "slow down" replies to the client, and in this case the S3 operation will not be performed. This means that, for example, an outage of the kafka system would eventually cause an outage of our service. Note that this may also be a result of a misconfiguration of the kafka broker, or decommissioning of a broker. To avoid that, we propose several options: * use a fifo instead of a queue. This would allow us to hold more than 128K messages - survive longer broker outages, and at a higher message rate. there should still probably be a limit set on the size of the fifo * define maximum number of retries allowed for a notification * define maximum time the notification may stay in the queue before it is removed We should probably start with these definitions done as topic attributes, reflecting our delivery guarantees for this specific destination. Will try to capture the results of the discussion in this tracker: https://tracker.ceph.com/issues/61532 Thanks, Yuval

2 3

ceph Pacific - MDS activity freezes when one the MDSs is restarted
by Emmanuel Jaep 09 Jun '23

09 Jun '23

Hi, we are currently running a ceph fs cluster at the following version: MDS version: ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) The cluster is composed of 7 active MDSs and 1 standby MDS: RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active icadmin012 Reqs: 73 /s 1938k 1880k 85.3k 92.8k 1 active icadmin008 Reqs: 206 /s 2375k 2375k 7081 171k 2 active icadmin007 Reqs: 91 /s 5709k 5256k 149k 299k 3 active icadmin014 Reqs: 93 /s 679k 664k 40.1k 216k 4 active icadmin013 Reqs: 86 /s 3585k 3569k 12.7k 197k 5 active icadmin011 Reqs: 72 /s 225k 221k 8611 164k 6 active icadmin015 Reqs: 87 /s 1682k 1610k 27.9k 274k POOL TYPE USED AVAIL cephfs_metadata metadata 8552G 22.3T cephfs_data data 226T 22.3T STANDBY MDS icadmin006 When I restart one of the active MDSs, the standby MDS becomes active and its state becomes "replay". So far, so good! However, only one of the other "active" MDSs seems to remain active. All activities drop from the other ones: RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active icadmin012 Reqs: 0 /s 1938k 1881k 85.3k 9720 1 active icadmin008 Reqs: 0 /s 2375k 2375k 7080 2505 2 active icadmin007 Reqs: 2 /s 5709k 5256k 149k 26.5k 3 active icadmin014 Reqs: 0 /s 679k 664k 40.1k 3259 4 replay icadmin006 801k 801k 1279 0 5 active icadmin011 Reqs: 0 /s 225k 221k 8611 9241 6 active icadmin015 Reqs: 0 /s 1682k 1610k 27.9k 34.8k POOL TYPE USED AVAIL cephfs_metadata metadata 8539G 22.8T cephfs_data data 225T 22.8T STANDBY MDS icadmin013 In effect, the cluster becomes almost unavailable until the newly promoted MDS finishes rejoining the cluster. Obviously, this defeats the purpose of having 7MDSs. Is this behavior? If not, what configuration items should I check to go back to "normal" operations? Best, Emmanuel

5 12

RadosGW S3 API Multi-Tenancy
by Brad House 08 Jun '23

08 Jun '23

I'm in the process of exploring if it is worthwhile to add RadosGW to our existing ceph cluster. We've had a few internal requests for exposing the S3 API for some of our business units, right now we just use the ceph cluster for VM disk image storage via RBD. Everything looks pretty straight forward until we hit multitenancy. The page on multi-tenancy doesn't dive into permission delegation: https://docs.ceph.com/en/quincy/radosgw/multitenancy/ The end goal I want is to be able to create a single user per tenant (Business Unit) which will act as their 'administrator', where they can then do basically whatever they want under their tenant sandbox (though I don't think we need more advanced cases like creations of roles or policies, just create/delete their own users, buckets, objects). I was hopeful this would just work, and I asked on the ceph IRC channel on OFTC and was told once I grant a user caps="users=*", they would then be allowed to create users *outside* of their own tenant using the Rados Admin API and that I should explore IAM roles. I think it would make sense to add a feature, such as a flag that can be set on a user, to ensure they stay in their "sandbox". I'd assume this is probably a common use-case. Anyhow, if its possible to do today using iam roles/policies, then great, unfortunately this is my first time looking at this stuff and there are some things not immediately obvious. I saw this online about AWS itself and creating a permissions boundary, but that's for allowing creation of roles within a boundary: https://www.qloudx.com/delegate-aws-iam-user-and-role-creation-without-givi… I'm not sure what "Action" is associated with the Rados Admin API create user for applying a boundary that the user can only create users with the same tenant name. https://docs.ceph.com/en/quincy/radosgw/adminops/#create-user Any guidance on this would be extremely helpful. Thanks! -Brad

1 1

slow mds requests with random read test
by Ben 07 Jun '23

07 Jun '23

Hi, We are performing couple performance tests on CephFS using fio. fio is run in k8s pod and 3 pods will be up running mounting the same pvc to CephFS volume. Here is command line for random read: fio -direct=1 -iodepth=128 -rw=randread -ioengine=libaio -bs=4k -size=1G -numjobs=5 -runtime=500 -group_reporting -directory=/tmp/cache -name=Rand_Read_Testing_$BUILD_TIMESTAMP The random read is performed very slow. Here is the cluster log from dashboard: 5/30/23 8:13:16 PM [INF] Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 5/30/23 8:13:16 PM [INF] Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 5/30/23 8:13:16 PM [INF] MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 30 secs, oldest blocked for 33 secs 5/30/23 8:13:16 PM [INF] MDS health message cleared (mds.?): 1 slow requests are blocked > 30 secs 5/30/23 8:13:14 PM [WRN] Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 5/30/23 8:13:13 PM [INF] MDS health message cleared (mds.?): 1 slow requests are blocked > 30 secs 5/30/23 8:13:08 PM [WRN] Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 5/30/23 8:13:08 PM [WRN] Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 5/30/23 8:13:08 PM [WRN] slow request 34.213327 seconds old, received at 2023-05-30T12:12:33.951399+0000: client_request(client.270564:1406144 getattr pAsLsXsFs #0x700000103d0 2023-05-30T12:12:33.947323+0000 caller_uid=0, caller_gid=0{}) currently failed to rdlock, waiting 5/30/23 8:13:08 PM [WRN] 1 slow requests, 1 included below; oldest blocked for > 34.213328 secs 5/30/23 8:13:07 PM [WRN] slow request 33.169703 seconds old, received at 2023-05-30T12:12:33.952078+0000: peer_request:client.270564:1406144 currently dispatched 5/30/23 8:13:07 PM [WRN] 1 slow requests, 1 included below; oldest blocked for > 33.169704 secs 5/30/23 8:13:04 PM [INF] Cluster is now healthy 5/30/23 8:13:04 PM [INF] Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 5/30/23 8:13:04 PM [INF] Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 5/30/23 8:13:04 PM [INF] MDS health message cleared (mds.?): 9 slow metadata IOs are blocked > 30 secs, oldest blocked for 45 secs 5/30/23 8:13:04 PM [INF] MDS health message cleared (mds.?): 2 slow requests are blocked > 30 secs 5/30/23 8:12:57 PM [WRN] 2 slow requests, 0 included below; oldest blocked for > 44.954377 secs 5/30/23 8:12:52 PM [WRN] 2 slow requests, 0 included below; oldest blocked for > 39.954313 secs 5/30/23 8:12:48 PM [WRN] Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 5/30/23 8:12:47 PM [WRN] slow request 34.935921 seconds old, received at 2023-05-30T12:12:12.185614+0000: client_request(client.270564:1406139 create #0x7000001045b/atomic7966567911433736706tmp 2023-05-30T12:12:12.182999+0000 caller_uid=0, caller_gid=0{}) currently submit entry: journal_and_reply 5/30/23 8:12:47 PM [WRN] slow request 34.954254 seconds old, received at 2023-05-30T12:12:12.167281+0000: client_request(client.270564:1406138 rename #0x70000010457/build.xml #0x70000010457/atomic6590865221269854506tmp 2023-05-30T12:12:12.162999+0000 caller_uid=0, caller_gid=0{}) currently submit entry: journal_and_reply 5/30/23 8:12:47 PM [WRN] 2 slow requests, 2 included below; oldest blocked for > 34.954254 secs 5/30/23 8:12:44 PM [WRN] Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 5/30/23 8:12:41 PM [INF] Cluster is now healthy 5/30/23 8:12:41 PM [INF] Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 5/30/23 8:12:41 PM [INF] MDS health message cleared (mds.?): 1 slow requests are blocked > 30 secs 5/30/23 8:12:40 PM [INF] Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 5/30/23 8:12:40 PM [INF] MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 30 secs, oldest blocked for 38 secs However, random write test is performing very good. Any suggestions on the problem? Thanks, Ben

2 3

Orchestration seems not to work
by Thomas Widhalm 07 Jun '23

07 Jun '23

Hi, I'm in the process of upgrading my cluster from 17.2.5 to 17.2.6 but the following problem existed when I was still everywhere on 17.2.5 . I had a major issue in my cluster which could be solved with a lot of your help and even more trial and error. Right now it seems that most is already fixed but I can't rule out that there's still some problem hidden. The very issue I'm asking about started during the repair. When I want to orchestrate the cluster, it logs the command but it doesn't do anything. No matter if I use ceph dashboard or "ceph orch" in "cephadm shell". I don't get any error message when I try to deploy new services, redeploy them etc. The log only says "scheduled" and that's it. Same when I change placement rules. Usually I use tags. But since they don't work anymore, too, I tried host and umanaged. No success. The only way I can actually start and stop containers is via systemctl from the host itself. When I run "ceph orch ls" or "ceph orch ps" I see services I deployed for testing being deleted (for weeks now). Ans especially a lot of old MDS are listed as "error" or "starting". The list doesn't match reality at all because I had to start them by hand. I tried "ceph mgr fail" and even a complete shutdown of the whole cluster with all nodes including all mgs, mds even osd - everything during a maintenance window. Didn't change anything. Could you help me? To be honest I'm still rather new to Ceph and since I didn't find anything in the logs that caught my eye I would be thankful for hints how to debug. Cheers, Thomas -- http://www.widhalm.or.at GnuPG : 6265BAE6 , A84CB603 Threema: H7AV7D33 Telegram, Signal: widhalmt(a)widhalm.or.at

3 19

Encryption per user Howto
by huxiaoyu＠horebdata.cn 07 Jun '23

07 Jun '23

Dear Ceph folks, Recently one of our clients approached us with a request on encrpytion per user, i.e. using individual encrytion key for each user and encryption files and object store. Does anyone know (or have experience) how to do with CephFS and Ceph RGW? Any suggestionns or comments are highly appreciated, best regards, Samuel huxiaoyu(a)horebdata.cn

8 27

reef v18.1.0 QE Validation status
by Yuri Weinstein 07 Jun '23

07 Jun '23

Details of this release are summarized here: https://tracker.ceph.com/issues/61515#note-1 Release Notes - TBD Seeking approvals/reviews for: rados - Neha, Radek, Travis, Ernesto, Adam King (we still have to merge https://github.com/ceph/ceph/pull/51788 for the core) rgw - Casey fs - Venky orch - Adam King rbd - Ilya krbd - Ilya upgrade/octopus-x - deprecated upgrade/pacific-x - known issues, Ilya, Laura? upgrade/reef-p2p - N/A clients upgrades - not run yet powercycle - Brad ceph-volume - in progress Please reply to this email with approval and/or trackers of known issues/PRs to address them. gibba upgrade was done and will need to be done again this week. LRC upgrade TBD TIA

7 9

Unexpected behavior of directory mtime after being set explicitly
by Sandip Divekar 06 Jun '23

06 Jun '23

Hi Team, I'm writing to bring to your attention an issue we have encountered with the "mtime" (modification time) behavior for directories in the Ceph filesystem. Upon observation, we have noticed that when the mtime of a directory (let's say: dir1) is explicitly changed in CephFS, subsequent additions of files or directories within 'dir1' fail to update the directory's mtime as expected. This behavior appears to be specific to CephFS - we have reproduced this issue on both Quincy and Pacific. Similar steps work as expected in the ext4 filesystem amongst others. Reproduction steps: 1. Create a directory - mkdir dir1 2. Modify mtime using the touch command - touch dir1 3. Create a file or directory inside of 'dir1' - mkdir dir1/dir2 Expected result: mtime for dir1 should change to the time the file or directory was created in step 3 Actual result: there was no change to the mtime for 'dir1' Note : For more detail, kindly find the attached logs. Our queries are : 1. Is this expected behavior for CephFS? 2. If so, can you explain why the directory behavior is inconsistent depending on whether the mtime for the directory has previously been manually updated. Best Regards, Sandip Divekar Component QA Lead SDET.

7 14

Re: CEPH Version choice
by Frank Schilder 02 Jun '23

02 Jun '23

Hi Marc, I uploaded all scripts and a rudimentary readme to https://github.com/frans42/cephfs-bench . I hope it is sufficient to get started. I'm afraid its very much tailored to our deployment and I can't make it fully configurable anytime soon. I hope it serves a purpose though - at least I discovered a few bugs with it. We actually kept the benchmark running through an upgrade from mimic to octopus. Was quite interesting to see how certain performance properties change with that. This benchmark makes it possible to compare versions with live timings coming in. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Marc <Marc(a)f1-outsourcing.eu> Sent: Monday, May 15, 2023 11:28 PM To: Frank Schilder Subject: RE: [ceph-users] Re: CEPH Version choice > I planned to put it on-line. The hold-back is that the main test is un- > taring a nasty archive and this archive might contain personal > information, so I can't just upload it as is. I can try to put together > a similar archive from public sources. Please give me a bit of time. I'm > also a bit under stress right now with our users being hit by an FS meta > data corruption. That's also why I'm a bit trigger happy. > Ok thanks, very nice, no hurry!!!

2 2

NFS export of 2 disjoint sub-dir mounts
by Frank Schilder 02 Jun '23

02 Jun '23

Hi all, I have a problem with exporting 2 different sub-folder ceph-fs kernel mounts via nfsd to the same IP address. The top-level structure on the ceph fs is something like /A/S1 and /A/S2. On a file server I mount /A/S1 and /A/S2 as two different file systems under /mnt/S1 and /mnt/S2 using the ceph fs kernel client. Then, these 2 mounts are exported with lines like these in /etc/exports: /mnt/S1 -options NET /mnt/S2 -options IP IP is an element of NET, meaning that the host at IP should be the only host being able to access /mnt/S1 and /mnt/S2. What we observe is that any attempt to mount the export /mnt/S1 on the host at IP results in /mnt/S2 being mounted instead. My first guess was that here we have a clash of fsids and the ceph fs is simply reporting the same fsid to nfsd and, hence, nfsd thinks both mountpoints contain the same. So I modified the second export line to /mnt/S2 -options,fsid=100 IP to no avail. The two folders are completely disjoint, neither symlinks nor hard-links between them. So it should be safe to export these as 2 different file systems. Exporting such constructs to non-overlapping networks/IPs works as expected - even when exporting subdirs of a dir (like exporting /A/B and /A/B/C from the same file server to strictly different IPs). It seems the same-IP config that breaks expectations. Am I missing here a magic -yes-i-really-know-what-i-am-doing hack? The file server is on AlmaLinux release 8.7 (Stone Smilodon) and all ceph packages match the ceph version octopus latest of our cluster. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 1

PGs incomplete - Data loss
by Benno Wulf 01 Jun '23

01 Jun '23

Hi guys, I'm awake since 36h and try to restore a broken ceph Pool (2 PGs incomplete) My vm are all broken. Some Boot, some Dont Boot... Also I have 5 removed disk with Data of that Pool "in my Hands" - Dont ask... So my question is it possible to restore Data of these other disks and "add" them thee others for healing? Best regards Ben

2 1

cephadm does not honor container_image default value
by Daniel Krambrock 01 Jun '23

01 Jun '23

Hello. I think i found a bug in cephadm/ceph orch: Redeploying a container image (tested with alertmanager) after removing a custom `mgr/cephadm/container_image_alertmanager` value, deploys the previous container image and not the default container image. I'm running `cephadm` from ubuntu 22.04 pkg 17.2.5-0ubuntu0.22.04.3 and `ceph` version 17.2.6. Here is an example. Node clrz20-08 is the node altermanager is running on, clrz20-01 the node I'm controlling ceph from: * Get alertmanager version ``` root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name == "alertmanager")| .container_image_name' "quay.io/prometheus/alertmanager:v0.23.0" ``` * Set alertmanager image ``` root@clrz20-01:~# ceph config set mgr mgr/cephadm/container_image_alertmanager quay.io/prometheus/alertmanager root@clrz20-01:~# ceph config get mgr mgr/cephadm/container_image_alertmanager quay.io/prometheus/alertmanager ``` * redeploy altermanager ``` root@clrz20-01:~# ceph orch redeploy alertmanager Scheduled to redeploy alertmanager.clrz20-08 on host 'clrz20-08' ``` * Get alertmanager version ``` root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name == "alertmanager")| .container_image_name' "quay.io/prometheus/alertmanager:latest" ``` * Remove alertmanager image setting, revert to default: ``` root@clrz20-01:~# ceph config rm mgr mgr/cephadm/container_image_alertmanager root@clrz20-01:~# ceph config get mgr mgr/cephadm/container_image_alertmanager quay.io/prometheus/alertmanager:v0.23.0 ``` * redeploy altermanager ``` root@clrz20-01:~# ceph orch redeploy alertmanager Scheduled to redeploy alertmanager.clrz20-08 on host 'clrz20-08' ``` * Get alertmanager version ``` root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name == "alertmanager")| .container_image_name' "quay.io/prometheus/alertmanager:latest" ``` -> `mgr/cephadm/container_image_alertmanager` is set to `quay.io/prometheus/alertmanager:v0.23.0`, but redeploy uses `quay.io/prometheus/alertmanager:latest`. This looks like a bug. * Set alertmanager image explicitly to the default value ``` root@clrz20-01:~# ceph config set mgr mgr/cephadm/container_image_alertmanager quay.io/prometheus/alertmanager:v0.23.0 root@clrz20-01:~# ceph config get mgr mgr/cephadm/container_image_alertmanager quay.io/prometheus/alertmanager:v0.23.0 ``` * redeploy altermanager ``` root@clrz20-01:~# ceph orch redeploy alertmanager Scheduled to redeploy alertmanager.clrz20-08 on host 'clrz20-08' ``` * Get alertmanager version ``` root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name == "alertmanager")| .container_image_name' "quay.io/prometheus/alertmanager:v0.23.0" ``` -> Setting `mgr/cephadm/container_image_alertmanager` to the default setting fixes the issue. Bests, Daniel

3 3

Small RGW objects and RADOS 64KB minimun size
by Loïc Dachary 31 May '23

31 May '23

Bonjour, Reading Karan's blog post about benchmarking the insertion of billions objects to Ceph via S3 / RGW[0] from last year, it reads: > we decided to lower bluestore_min_alloc_size_hdd to 18KB and re-test. As represented in chart-5, the object creation rate found to be notably reduced after lowering the bluestore_min_alloc_size_hdd parameter from 64KB (default) to 18KB. As such, for objects larger than the bluestore_min_alloc_size_hdd , the default values seems to be optimal, smaller objects further require more investigation if you intended to reduce bluestore_min_alloc_size_hdd parameter. There also is a mail thread dated 2018 on this topic as well, with the same conclusion although using RADOS directly and not RGW[3]. I read the RGW data layout page in the documentation[1] and concluded that by default every object inserted with S3 / RGW will indeed use at least 64kb. A pull request from last year[2] seems to confirm it and also suggests modifying bluestore_min_alloc_size_hdd has adverse side effects. That being said, I'm curious to know if people developed strategies to cope with this overhead. Someone mentioned packing objects together client side to make them larger. But maybe there are simpler ways to do the same? Cheers [0] https://www.redhat.com/en/blog/scaling-ceph-billion-objects-and-beyond [1] https://docs.ceph.com/en/latest/radosgw/layout/ [2] https://github.com/ceph/ceph/pull/32809 [3] https://www.spinics.net/lists/ceph-users/msg45755.html -- Loïc Dachary, Artisan Logiciel Libre

4 6

BlueStore fragmentation woes
by Hector Martin 31 May '23

31 May '23

Hi, I've been seeing relatively large fragmentation numbers on all my OSDs: ceph daemon osd.13 bluestore allocator score block { "fragmentation_rating": 0.77251526920454427 } These aren't that old, as I recreated them all around July last year. They mostly hold CephFS data with erasure coding, with a mix of large and small files. The OSDs are at around 80%-85% utilization right now. Most of the data was written sequentially when the OSDs were created (I rsynced everything from a remote backup). Since then more data has been added, but not particularly quickly. At some point I noticed pathologically slow writes, and I couldn't figure out what was wrong. Eventually I did some block tracing and noticed the I/Os were very small, even though CephFS-side I was just writing one large file sequentially, and that's when I stumbled upon the free space fragmentation problem. Indeed, deleting some large files opened up some larger free extents and resolved the problem, but only until those get filled up and I'm back to fragmented tiny extents. So effectively I'm stuck at the current utilization, as trying to fill them up any more just slows down to an absolute crawl. I'm adding a few more OSDs and plan on doing the dance of removing one OSD at a time and replacing it with another one to hopefully improve the situation, but obviously this is going to take forever. Is there any plan for offering a defrag tool of some sort for bluestore? - Hector

7 27

[Pacific] Admin keys no longer works I get access denied URGENT!!!
by wodel youchi 31 May '23

31 May '23

Hi, After a wrong manipulation, the admin key no longer works, it seems it has been modified. My cluster is built using containers. When I execute ceph -s I get [root@controllera ceph]# ceph -s 2023-05-31T11:33:20.940+0100 7ff7b2d13700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] 2023-05-31T11:33:20.940+0100 7ff7b1d11700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] 2023-05-31T11:33:20.940+0100 7ff7b2512700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] [errno 13] RADOS permission denied (error connecting to the cluster) From the log file I am getting : May 31 11:03:02 controllera docker[214909]: debug 2023-05-31T11:03:02.714+0100 7fcfc0c91700 0 cephx server client.admin: unexpected key: req.key=5fea877f2a68548b expected_key=8c2074e03ffa449a How can I recover the correct key? Regards.

2 1

all buckets mtime = "0.000000" after upgrade to 17.2.6
by alyarb＠gmail.com 31 May '23

31 May '23

I was running on 17.2.5 since October, and just upgraded to 17.2.6, and now the "mtime" property on all my buckets is 0.000000. On all previous versions going back to Nautilus this wasn't an issue, and we do like to have that value present. radosgw-admin has no quick way to get the last object in the bucket. Here's my tracker submission: https://tracker.ceph.com/issues/61264#change-239348

2 1

MDS corrupt (also RADOS-level copy?)
by Jake Grimmett 31 May '23

31 May '23

Dear All, we are trying to recover from what we suspect is a corrupt MDS :( and have been following the guide here: <https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/> Symptoms: MDS SSD pool (2TB) filled completely over the weekend, normally uses less than 400GB, resulting in MDS crash. We added 4 x extra SSD to increase pool capacity to 3.5TB, however MDS did not recover # ceph fs status cephfs2 - 0 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed 1 resolve wilma-s3 8065 8063 8047 0 2 resolve wilma-s2 901k 802k 34.4k 0 POOL TYPE USED AVAIL mds_ssd metadata 2296G 3566G primary_fs_data data 0 3566G ec82pool data 2168T 3557T STANDBY MDS wilma-s1 wilma-s4 setting "ceph mds repaired 0" causes rank 0 to restart, and then immediately fail. Following the disaster-recovery-experts guide, the first step we did was to export the MDS journals, e.g: # cephfs-journal-tool --rank=cephfs2:0 journal export /root/backup.bin.0 journal is 9744716714163~658103700 wrote 658103700 bytes at offset 9744716714163 to /root/backup.bin.0 so far so good, however when we try to backup the final MDS the process consumes all available RAM (470GB) and needs to be killed after 14 minutes. # cephfs-journal-tool --rank=cephfs2:2 journal export /root/backup.bin.2 similarly, "recover_dentries summary" consumes all RAM when applied to MDS 2 # cephfs-journal-tool --rank=cephfs2:2 event recover_dentries summary We successfully ran "cephfs-journal-tool --rank=cephfs2:0 event recover_dentries summary" and "cephfs-journal-tool --rank=cephfs2:1 event recover_dentries summary" at this point, we tried to follow the instructions and make a RADOS level copy of the journal data, however the link in the docs doesn't explain how to do this and just points to <http://tracker.ceph.com/issues/9902> At this point we are tempted to reset the journal on MDS 2, but wanted to get a feeling from others about how dangerous this could be? We have a backup, but as there is 1.8PB of data, it's going to take a few weeks to restore.... any ideas gratefully received. Jake -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK.

2 3

how to use ctdb_mutex_ceph_rados_helper
by Angelo Höngens 31 May '23

31 May '23

Hey, I have a test setup with a 3-node samba cluster. This cluster consists of 3 vm's storing its locks on a replicated gluster volume. I want to switch to 2 physical smb-gateways for performance reasons (not enough money for 3), and since the 2-node cluster can't get quorum, I hope to switch to storing the ctdb lock in ceph and hope that will work reliably. (experiences with 2 node SMB clusters?) I am looking into the ctdb rados helper: [cluster] recovery lock = !/usr/lib/x86_64-linux-gnu/ctdb/ctdb_mutex_ceph_rados_helper ceph client.tenant1 cephfs_metadata ctdb_lock Now I do have a bit of experience with cephfs, rbd and rgw, but not rados. How do I give the user client.tenant1 permissions? We have a single cephfs, with 4 different tenants (departments). Each department has their own samba cluster. We're using cephfs permissions to limit the tenants to their own path (I hope). example of ceph auth: client.tenant1 key: ***** caps: [mds] allow rws fsname=cephfs path=/tenant1 caps: [mon] allow r fsname=cephfs caps: [osd] allow rw tag cephfs data=cephfs If I try some stuff manually (without really knowing how to specify objects or what that means), I get this permission denied error: root@tenant1-1:~# /usr/lib/x86_64-linux-gnu/ctdb/ctdb_mutex_ceph_rados_helper ceph client.tenant1 cephfs_metadata tenant1/ctdb_lock 1 /usr/lib/x86_64-linux-gnu/ctdb/ctdb_mutex_ceph_rados_helper: Failed to get lock on RADOS object 'tenant1/ctdb_lock' - (Operation not permitted) Angelo.

1 0

Important: RGW multisite bug may silently corrupt encrypted objects on replication
by Casey Bodley 31 May '23

31 May '23

Our downstream QE team recently observed an md5 mismatch of replicated objects when testing rgw's server-side encryption in multisite. This corruption is specific to s3 multipart uploads, and only affects the replicated copy - the original object remains intact. The bug likely affects Ceph releases all the way back to Luminous where server-side encryption was first introduced. To expand on the cause of this corruption: Encryption of multipart uploads requires special handling around the part boundaries, because each part is uploaded and encrypted separately. In multisite, objects are replicated in their encrypted form, and multipart uploads are replicated as a single part. As a result, the replicated copy loses its knowledge about the original part boundaries required to decrypt the data correctly. We don't have a fix yet, but we're tracking it in https://tracker.ceph.com/issues/46062. The fix will only modify the replication logic, so won't repair any objects that have already replicated incorrectly. We'll need to develop a radosgw-admin command to search for affected objects and reschedule their replication. In the meantime, I can only advise multisite users to avoid using encryption for multipart uploads. If you'd like to scan your cluster for existing encrypted multipart uploads, you can identify them with a s3 HeadObject request. The response would include a x-amz-server-side-encryption header, and the ETag header value (with "s removed) would be longer than 32 characters (multipart ETags are in the special form "<md5sum>-<num parts>"). Take care not to delete the corrupted replicas, because an active-active multisite configuration would go on to delete the original copy.

2 4

Custom CRUSH maps HOWTO?
by Thorne Lawler 31 May '23

31 May '23

Hi folks! I have a Ceph production 17.2.6 cluster with 6 machines in it - four newer, faster machines with 4x3.84TB NVME drives each, and two with 24x1.68TB SAS disks each. I know I should have done something smart with the CRUSH maps for this up front, but until now I have shied away from CRUSH maps as they sound really complex. Right now my cluster's performance, especially write performance, is not what it needs to be, and I am looking for advice: 1. How should I be structuring my crush map, and why? 2. How does one actually edit and manage a CRUSH map? What /commands/ does one use? This isn't clear at all in the documentation. Are there any GUI tools out there for managing CRUSH? 3. Is this going to impact production performance or availability while I'm configuring it? I have tens of thousands of users relying on this thing, so I can't take any risks. Thanks in advance! -- Regards, Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170 _DDNS /_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

3 3

Ceph client version vs server version inter-operability
by Mark Kirkwood 31 May '23

31 May '23

Hi, We are running a ceph cluster that is currently on Luminous. At this point most of our clients are also Luminous, but as we provision new client hosts we are using client versions that are more recent (e.g Octopus, Pacific and more recently Quincy). Is this safe? Is there a known list of what client versions are compatible with what server version? We are only using RBD and are specifying rbd_default_features (the same) on all server and client hosts. regards Mark

1 0

Seeking feedback on Improving cephadm bootstrap process
by Redouane Kachach 30 May '23

30 May '23

Dear ceph community, As you are aware, cephadm has become the default tool for installing Ceph on bare-metal systems. Currently, during the bootstrap process of a new cluster, if the user interrupts the process manually or if there are any issues causing the bootstrap process to fail, cephadm leaves behind the failed cluster files and processes on the current host. While this can be beneficial for debugging and resolving issues related to the cephadm bootstrap process, it can create difficulties for inexperienced users who need to delete the faulty cluster and proceed with the Ceph installation. The problem described in the tracker https://tracker.ceph.com/issues/57016 is a good example of this issue.In the cephadm development team, we are considering ways to enhance the user experience during the bootstrap of a new cluster. We have discussed the following options:1) Retain the cluster files without deleting them, but provide the user with a clear command to remove the broken/faulty cluster. 2) Automatically delete the broken/failed ceph installation and offer an option for the user to disable this behavior if desired.Both options have their advantages and disadvantages, which is why we are seeking your feedback. We would like to know which option you prefer and the reasoning behind your choice. Please provide reasonable arguments to justify your preference.Your feedback will be taken into careful consideration when we work on improving the ceph bootstrap process.Thank you, Redouane, On behalf of cephadm dev team.

6 8

RBD imagem mirroring doubt
by Work Ceph 30 May '23

30 May '23

Hello guys, What would happen if we set up an RBD mirroring configuration, and in the target system (the system where the RBD image is mirrored) we create snapshots of this image? Would that cause some problems? Also, what happens if we delete the source RBD image? Would that trigger a deletion in the target system RBD image as well? Thanks in advance!

1 0

Re: Newer linux kernel cephfs clients is more trouble?
by Dan van der Ster 30 May '23

30 May '23

Hi, Sorry for poking this old thread, but does this issue still persist in the 6.3 kernels? Cheers, Dan ______________________________ Clyso GmbH | https://www.clyso.com On Wed, Dec 7, 2022 at 3:42 AM William Edwards <wedwards(a)cyberfusion.nl> wrote: > > > > Op 7 dec. 2022 om 11:59 heeft Stefan Kooman <stefan(a)bit.nl> het volgende geschreven: > > > > On 5/13/22 09:38, Xiubo Li wrote: > >>> On 5/12/22 12:06 AM, Stefan Kooman wrote: > >>> Hi List, > >>> > >>> We have quite a few linux kernel clients for CephFS. One of our customers has been running mainline kernels (CentOS 7 elrepo) for the past two years. They started out with 3.x kernels (default CentOS 7), but upgraded to mainline when those kernels would frequently generate MDS warnings like "failing to respond to capability release". That worked fine until 5.14 kernel. 5.14 and up would use a lot of CPU and *way* more bandwidth on CephFS than older kernels (order of magnitude). After the MDS was upgraded from Nautilus to Octopus that behavior is gone (comparable CPU / bandwidth usage as older kernels). However, the newer kernels are now the ones that give "failing to respond to capability release", and worse, clients get evicted (unresponsive as far as the MDS is concerned). Even the latest 5.17 kernels have that. No difference is observed between using messenger v1 or v2. MDS version is 15.2.16. > >>> Surprisingly the latest stable kernels from CentOS 7 work flawlessly now. Although that is good news, newer operating systems come with newer kernels. > >>> > >>> Does anyone else observe the same behavior with newish kernel clients? > >> There have some known bugs, which have been fixed or under fixing recently, even in the mainline and, not sure whether are they related. Such as [1][2][3][4]. More detail please see ceph-client repo testing branch [5]. > > > > None of the issues you mentioned were related. We gained some more experience with newer kernel clients, specifically on Ubuntu Focal / Jammy (5.15). Performance issues seem to arise in certain workloads, specifically load-balanced Apache shared web hosting clusters with CephFS. We have tested linux kernel clients from 5.8 up to and including 6.0 with a production workload and the short summary is: > > > > < 5.13, everything works fine > > 5.13 and up is giving issues > > I see this issue on 6.0.0 as well. > > > > > We tested the 5.13.-rc1 as well, and already that kernel is giving issues. So something has changed in 5.13 that results in performance regression in certain workloads. And I wonder if it has something to do with the changes related to fscache that have, and are, happening in the kernel. These web servers might access the same directories / files concurrently. > > > > Note: we have quite a few 5.15 kernel clients not doing any (load-balanced) web based workload (container clusters on CephFS) that don't have any performance issue running these kernels. > > > > Issue: poor CephFS performance > > Symptom / result: excessive CephFS network usage (order of magnitude higher than for older kernels not having this issue), within a minute there are a bunch of slow web service processes, claiming loads of virtual memory, that result in heavy swap usage and basically rendering the node unusable slow. > > > > Other users that replied to this thread experienced similar symptoms. It is reproducible on both CentOS (EPEL mainline kernels) as well as on Ubuntu (hwe as well as default relase kernel). > > > > MDS version used: 15.2.16 (with a backported patch from 15.2.17) (single active / standby-replay) > > > > Does this ring a bell? > > > > Gr. Stefan > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 2

Pacific - mds How to know how many sequences still to be replayed
by Emmanuel Jaep 29 May '23

29 May '23

Hi, I just restarted one of our mds servers. I can find some "progress" in logs as below: mds.beacon.icadmin006 Sending beacon up:replay seq 461 mds.beacon.icadmin006 received beacon reply up:replay seq 461 rtt 0 How I know how long is the sequence (ie. when the node will be finished replaying)? Best, Emmanuel

1 0

Troubleshooting "N slow requests are blocked > 30 secs" on Pacific
by Emmanuel Jaep 29 May '23

29 May '23

Hi, we are running a cephfs cluster with the following version: ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) Several MDSs are reporting slow requests: HEALTH_WARN 4 MDSs report slow requests [WRN] MDS_SLOW_REQUEST: 4 MDSs report slow requests mds.icadmin011(mds.5): 1 slow requests are blocked > 30 secs mds.icadmin015(mds.6): 2 slow requests are blocked > 30 secs mds.icadmin006(mds.4): 8 slow requests are blocked > 30 secs mds.icadmin007(mds.2): 2 slow requests are blocked > 30 secs According to Quincy's documentation ( https://docs.ceph.com/en/quincy/cephfs/troubleshooting/) this can be investigated by issuing: ceph mds.icadmin011 dump cache /tmp/dump.txt Unfortunately, this command fails: no valid command found; 10 closest matches: pg stat pg getmap pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] pg dump_json [all|summary|sum|pools|osds|pgs...] pg dump_pools_json pg ls-by-pool <poolstr> [<states>...] pg ls-by-primary <id|osd.id> [<pool:int>] [<states>...] pg ls-by-osd <id|osd.id> [<pool:int>] [<states>...] pg ls [<pool:int>] [<states>...] pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [<threshold:int>] Error EINVAL: invalid command I imagine that it is related to the fact that we are running the Pacific version and not the Quincy version. When looking at the Pacific's documentation ( https://docs.ceph.com/en/pacific/cephfs/health-messages/) I should: > Use the ops admin socket command to list outstanding metadata operations. Unfortunately, I fail to really understand what I'm supposed to do. Can someone give a pointer? Best, Emmanuel

2 7

Recoveries without any misplaced objects?
by Hector Martin 29 May '23

29 May '23

Hi, I'm watching a cluster finish a bunch of backfilling, and I noticed that quite often PGs end up with zero misplaced objects, even though they are still backfilling. Right now the cluster is down to 6 backfilling PGs: data: volumes: 1/1 healthy pools: 6 pools, 268 pgs objects: 18.79M objects, 29 TiB usage: 49 TiB used, 25 TiB / 75 TiB avail pgs: 262 active+clean 6 active+remapped+backfilling But there are no misplaced objects, and the misplaced column in `ceph pg dump` is zero for all PGs. If I do a `ceph pg dump_json`, I can see `num_objects_recovered` increasing for these PGs... but the misplaced count is still 0. Is there something else that would cause recoveries/backfills other than misplaced objects? Or perhaps there is a bug somewhere causing the misplaced object count to be misreported as 0 sometimes? # ceph -v ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) - Hector

1 0

[Ceph | Quency ]The scheduled snapshots are not getting created till we create a manual backup.
by Lokendra Rathour 29 May '23

29 May '23

Hi Team, *Problem:* Create scheduled snapshots of the ceph subvolume. *Expected Result:* The scheduled snapshots should be created at the given scheduled time. *Actual Result:* The scheduled snapshots are not getting created till we create a manual backup. *Description:* *Ceph version: 17(quincy)* OS: Centos/Almalinux The scheduled snapshot creation is not working and we were only able to see the following logs in the file "ceph-mgr.storagenode3.log": *2023-05-29T04:59:35.101+0000 7f4cd3ad8700 0 [snap_schedule INFO mgr_util] scanning for idle connections..* *2023-05-29T04:59:35.101+0000 7f4cd3ad8700 0 [snap_schedule DEBUG mgr_util] fs_name (cephfs) connections ([])* *2023-05-29T04:59:35.101+0000 7f4cd3ad8700 0 [snap_schedule INFO mgr_util] cleaning up connections: [* The command which we were executing to add the snapshot schedule: *ceph fs snap-schedule add /volumes/<subvolume_group>/<subvolume> <frequency_of_snapshot> <Timeinformat_yyyymmddTHH:MM:ss>* *eg.* *ceph fs snap-schedule add /volumes/xyz/test_restore_53 1h 2023-05-26T11:05:00* We can make sure that the schedule has been created using the following commands: *#ceph fs snap-schedule list / --recursive=true* *#ceph fs snap-schedule status /volumes/xyz/test_restore_53* Even though we created the snapshot schedule, snapshots were not getting created. We then tried creating a manual snapshot for one of the sub-volumes using the following command: *#ceph fs subvolume snapshot create cephfs <subvolume> <snapshot-name> --group_name <subvolume_group>* *eg. ceph fs subvolume snapshot create cephfs test_restore_53 snapshot-1 --group_name xyz* To check the snapshots created we can use the following command: *ceph fs subvolume snapshot ls cephfs <subvolume> <snapshot-name> <subvolume_group>* *eg. ceph fs subvolume snapshot ls cephfs test_restore_53 snapshot-1 xyz* To delete the manually created snapshot: *ceph fs subvolume snapshot rm cephfs <subvolume> <snapshot-name> <subvolume_group>* *eg. ceph fs subvolume snapshot rm cephfs test_restore_53 snapshot-1 xyz* To our surprise, the scheduled snapshots started working. We also applied the retention policy and seems to be working fine. We re-tested this understanding for another subvolume. And the scheduled snapshots only started once we triggered a manual snapshot. Could you please help us out with this? -- ~ Lokendra skype: lokendrarathour

2 1

Creating a bucket with bucket constructor in Ceph v16.2.7
by Ramin Najjarbashi 29 May '23

29 May '23

Hi I'm currently using Ceph version 16.2.7 and facing an issue with bucket creation in a multi-zone configuration. My setup includes two zone groups: ZG1 (Master) and ZG2, with one zone in each zone group (zone-1 in ZG1 and zone-2 in ZG2). The objective is to create buckets in a specific zone group (ZG2) using the bucket constructor. However, despite setting the desired zone group (abrak) in the request, the bucket is still being created in the master zone group (ZG1). I have defined the following endpoint pattern for each zone group: s3.{zg}.mydomain.com I am using the s3cmd client to interact with the Ceph cluster. I have ensured that I provide the necessary endpoint and region information while executing the bucket creation command. Despite my efforts, the bucket consistently gets created in ZG1 instead of ZG2. - Ceph Version: 16.2.7 - Zone Group 1 (ZG1) Endpoint: http://s3.zonegroup1.mydomain.com - Zone Group 2 (ZG2) Endpoint: http://s3.zonegroup2.mydomain.com - Desired Bucket Creation Region: zg2-api-name have reviewed the Ceph documentation and made necessary configuration changes, but I have not been able to achieve the desired result.I kindly request your assistance in understanding why the bucket constructor is not honoring the specified region and always defaults to ZG1. I would greatly appreciate any insights, recommendations, or potential solutions to resolve this issue. Thank you for your time and support. --------------------------------- Here are the details of my setup: --------------------------------- ```sh s3cmd --region zg2-api-name mb s3://test-zg2s3cmd info s3://test-zg2s3://test-zg2/ (bucket): Location: zg2-api-name Payer: BucketOwner Expiration Rule: none Policy: none CORS: none ACL: development: FULL_CONTROL ``` this is my config file: ```ini [default] access_key = <KEY> secret_key = <SECRET> host_base = s3.zonegroup1.mydomain.com host_bucket = s3.%(location)s.mydomain.com #host_bucket = %(bucket)s.s3.zonegroup1.mydomain.com #host_bucket = s3.%(location)s.mydomain.com #host_bucket = s3.%(region)s.mydomain.com bucket_location = zg1-api-name use_https = False ``` Zonegroup configuration for the `zonegroup1` region: ```json { "id": "fb3f818a-ca9b-4b12-b431-7cdcd80006d", "name": "zg1-api-name", "api_name": "zg1-api-name", "is_master": "false", "endpoints": [ "http://s3.zonegroup1.mydomain.com", ], "hostnames": [ "s3.zonegroup1.mydomain.com", ], "hostnames_s3website": [ "s3-website.zonegroup1.mydomain.com", ], "master_zone": "at2-stg-zone", "zones": [ { "id": "at2-stg-zone", "name": "at2-stg-zone", "endpoints": [ "http://s3.zonegroup1.mydomain.com" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "fa2f8194-4a9d-4b98-b411-9cdcd1e5506a", "sync_policy": { "groups": [] } } ``` Zonegroup configuration for the `zonegroup2` region: ```json { "id": "a513d60c-44a2-4289-a23d-b7a511be6ee4", "name": "zg2-api-name", "api_name": "zg2-api-name", "is_master": "false", "endpoints": [ "http://s3.zonegroup2.mydomain.com" ], "hostnames": [ "s3.zonegroup2.mydomain.com" ], "hostnames_s3website": [], "master_zone": "zonegroup2-sh-1", "zones": [ { "id": "zonegroup2-sh-1", "name": "zonegroup2-sh-1", "endpoints": [ "http://s3.zonegroup2.mydomain.com" ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "fa2f8194-4a9d-4b98-b411-9cdcd1e5506a", "sync_policy": { "groups": [] } } ```

3 4

Can lifecycle transition cause a high load if many buckets do transition at the same time?
by huy nguyen 29 May '23

29 May '23

Hi, I have a plan to use the second storage class for cold data in the near future. But I have a question: if many old buckets with many objects are set to transition at the same time, can Ceph handle it by default, or do I have to configure QoS to avoid the high load? Thanks!

1 0

Ceph iscsi gateway semi deprecation warning?
by Mark Kirkwood 29 May '23

29 May '23

I am looking at using an iscsi gateway in front of a ceph setup. However the warning in the docs is concerning: The iSCSI gateway is in maintenance as of November 2022. This means that it is no longer in active development and will not be updated to add new features. Does this mean I should be wary of using it, or is it simply that it does all the stuff it needs to and no further development is needed? regards Mark

3 2

fail delete "daemon(s) not managed by cephadm"
by farhad kh 28 May '23

28 May '23

hi everyone i have a warning ` 1 stray daemon(s) not managed by cephadm` # ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon mon.apcepfpspsp0111 on host apcepfpspsp0111 not managed by cephadm and in detail : # ceph crash info 2023-01-17T18:25:19.601062Z_4affd3b9-f486-4ae5-801b-ede231e2b624 { "archived": "2023-01-18 19:31:58.200096", "assert_condition": "abort", "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.5/rpm/el8/BUILD/ceph-17.2.5/src/mon/MonitorDBStore.h", "assert_func": "int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)", "assert_line": 355, "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.5/rpm/el8/BUILD/ceph-17.2.5/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7fabeb737700 time 2023-01-17T18:25:19.568046+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.5/rpm/el8/BUILD/ceph-17.2.5/src/mon/MonitorDBStore.h: 355: ceph_abort_msg(\"failed to write to db\")\n", "assert_thread_name": "ms_dispatch", "backtrace": [ "/lib64/libpthread.so.0(+0x12cf0) [0x7fabf5949cf0]", "gsignal()", "abort()", "(ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x197) [0x7fabf799eb5f]", "(MonitorDBStore::apply_transaction(std::shared_ptr<MonitorDBStore::Transaction>)+0x88f) [0x55fe9cc3a68f]", "(Elector::persist_epoch(unsigned int)+0x184) [0x55fe9ccc64c4]", "(ElectionLogic::bump_epoch(unsigned int)+0x5d) [0x55fe9ccccd5d]", "(ElectionLogic::propose_classic_prefix(int, unsigned int)+0x3c7) [0x55fe9ccce8e7]", "(ElectionLogic::propose_classic_handler(int, unsigned int)+0x2b) [0x55fe9cccef9b]", "(Elector::handle_propose(boost::intrusive_ptr<MonOpRequest>)+0x6f5) [0x55fe9ccc6175]", "(Elector::dispatch(boost::intrusive_ptr<MonOpRequest>)+0xcdb) [0x55fe9ccc756b]", "(Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x11c2) [0x55fe9cc0aeb2]", "(Monitor::_ms_dispatch(Message*)+0x406) [0x55fe9cc0b5e6]", "(Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5d) [0x55fe9cc3bdad]", "(Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x478) [0x7fabf7c18c88]", "(DispatchQueue::entry()+0x50f) [0x7fabf7c160cf]", "(DispatchQueue::DispatchThread::entry()+0x11) [0x7fabf7cdd8f1]", "/lib64/libpthread.so.0(+0x81ca) [0x7fabf593f1ca]", "clone()" ], "ceph_version": "17.2.5", "crash_id": "2023-01-17T18:25:19.601062Z_4affd3b9-f486-4ae5-801b-ede231e2b624", "entity_name": "mon.apcepfpspsp0101", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "8", "os_version_id": "8", "process_name": "ceph-mon", "stack_sig": "74f29b024a3ec0f1395579505c5728e8ad5fd5a77c4a92437d696fa6fd1a8e20", "timestamp": "2023-01-17T18:25:19.601062Z", "utsname_hostname": "apcepfpspsp0101", "utsname_machine": "x86_64", "utsname_release": "5.4.17-2136.310.7.1.el8uek.x86_64", "utsname_sysname": "Linux", "utsname_version": "#2 SMP Wed Aug 17 15:14:08 PDT 2022" } but i have 3 instance from mon modul NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 3/3 2m ago 9M label:mon crash 6/6 9m ago 9M * grafana ?:3000 3/3 2m ago 4M count-per-host:1;label:mon mds.rook 3/3 2m ago 8M apcepfpspsp0101;apcepfpspsp0103;apcepfpspsp0105;count:3 mgr 2/2 2m ago 9M count:2 mon 3/3 2m ago 2h label:mon node-exporter ?:9100 6/6 9m ago 9M * osd 3 2m ago - <unmanaged> osd.cost_capacity 15 9m ago 4M * prometheus ?:9095 3/3 2m ago 9M label:mon whene i want remove this get # ceph orch daemon rm mon.apcepfpspsp0111 --force Error EINVAL: Unable to find daemon(s) ['mon.apcepfpspsp0111'] I don't have such a daemon mon.apcepfpspsp0101 apcepfpspsp0101 running (88m) mon.apcepfpspsp0103 apcepfpspsp0103 running (87m) mon.apcepfpspsp0105 apcepfpspsp0105 running (87m) What should I do to remove this warning?

2 1

cephfs-data-scan with multiple data pools
by Justin Li 26 May '23

26 May '23

Dear All, I'm trying to recover failed MDS metadata by following the link below but having troubles. Thanks in advance. Question1: how to scan 2 data pools with scan_extents (cmd 1). The cmd didn't work with two pools specified. Should I scan one then the other? Question2: As to scan_inodes (cmd 2), should I only specify the first data pool per the document. I'm concerned if the 2nd pool is not scanned then that'll cause metadata loss. *my fs name: cephfs, data pools: cephfs_hdd, cephfs_ssd* cmd 1: cephfs-data-scan scan_extents --filesystem cephfs cephfs_hdd cephfs_ssd cmd 2: cephfs-data-scan scan_inodes --filesystem cephfs cephfs_hdd cephfs-data-scan scan_extents [<data pool> [<extra data pool> ...]]cephfs-data-scan scan_inodes [<data pool>]cephfs-data-scan scan_links Note, the data pool parameters for ‘scan_extents’, ‘scan_inodes’ and ‘cleanup’ commands are optional, and usually the tool will be able to detect the pools automatically. Still you may override this. The ‘scan_extents’ command needs all data pools to be specified,* while ‘scan_inodes’ and ‘cleanup’ commands need only the main data pool.* *https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/ <https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/>* -- Best Regards, *Justin Li* IT Support/Systems Administrator *Justin.Li2030(a)Gmail.com <Justin.Li2030(a)Gmail.com>* <http://www.linkedin.com/in/justinli7>

1 0

Unable to online CephFS, MDS segfaults during mds log replay
by Alfred Heisner 26 May '23

26 May '23

Hello, I have a Ceph deployment using CephFS. Recently MDS failed and cannot start. Attempting to start MDS for this filesystem results in nearly immediate segfault in MDS. Logs below. cephfs-journal-tool shows Overall journal integrity state OK root@proxmox-2:/var/log/ceph# cephfs-journal-tool --rank=galaxy:all journal inspect Overall journal integrity: OK Stack dump / log from MDS: -14> 2023-05-26T15:01:09.204-0500 7f27c24b2700 1 mds.0.journaler.mdlog(ro) probing for end of the log -13> 2023-05-26T15:01:09.208-0500 7f27c34b4700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 4194304, expire 4194607, write 4194607, stream_format 1). probing for end of log (from 4194607)... -12> 2023-05-26T15:01:09.208-0500 7f27c34b4700 1 mds.0.journaler.pq(ro) probing for end of the log -11> 2023-05-26T15:01:09.412-0500 7f27c24b2700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 2388235687 (header had 2388213543). recovered. -10> 2023-05-26T15:01:09.412-0500 7f27c34b4700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 4194607 (header had 4194607). recovered. -9> 2023-05-26T15:01:09.412-0500 7f27c34b4700 4 mds.0.purge_queue operator(): open complete -8> 2023-05-26T15:01:09.412-0500 7f27c34b4700 1 mds.0.journaler.pq(ro) set_writeable -7> 2023-05-26T15:01:09.412-0500 7f27c1cb1700 4 mds.0.log Journal 0x200 recovered. -6> 2023-05-26T15:01:09.412-0500 7f27c1cb1700 4 mds.0.log Recovered journal 0x200 in format 1 -5> 2023-05-26T15:01:09.412-0500 7f27c1cb1700 2 mds.0.6403 Booting: 1: loading/discovering base inodes -4> 2023-05-26T15:01:09.412-0500 7f27c1cb1700 0 mds.0.cache creating system inode with ino:0x100 -3> 2023-05-26T15:01:09.412-0500 7f27c1cb1700 0 mds.0.cache creating system inode with ino:0x1 -2> 2023-05-26T15:01:09.416-0500 7f27c24b2700 2 mds.0.6403 Booting: 2: replaying mds log -1> 2023-05-26T15:01:09.416-0500 7f27c24b2700 2 mds.0.6403 Booting: 2: waiting for purge queue recovered 0> 2023-05-26T15:01:09.428-0500 7f27c0caf700 -1 *** Caught signal (Segmentation fault) ** in thread 7f27c0caf700 thread_name:md_log_replay ceph version 17.2.6 (995dec2cdae920da21db2d455e55efbc339bde24) quincy (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140) [0x7f27cd70c140] 2: (EMetaBlob::replay(MDSRank*, LogSegment*, MDPeerUpdate*)+0x66c2) [0x563540fc7372] 3: (EUpdate::replay(MDSRank*)+0x3c) [0x563540fc8abc] 4: (MDLog::_replay_thread()+0x7cb) [0x563540f4d0fb] 5: (MDLog::ReplayThread::entry()+0xd) [0x563540c1fbfd] 6: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f27cd700ea7] 7: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. What are the safest steps to recovery at this point? Thanks, Al

1 0

Pacific - MDS behind on trimming
by Emmanuel Jaep 26 May '23

26 May '23

Hi, lately, we have had some issues with our MDSs (Ceph version 16.2.10 Pacific). Part of them are related to MDS being behind on trimming. I checked the documentation and found the following information ( https://docs.ceph.com/en/pacific/cephfs/health-messages/) > CephFS maintains a metadata journal that is divided into *log segments*. The length of journal (in number of segments) is controlled by the setting mds_log_max_segments, and when the number of segments exceeds that setting the MDS starts writing back metadata so that it can remove (trim) the oldest segments. If this writeback is happening too slowly, or a software bug is preventing trimming, then this health message may appear. The threshold for this message to appear is controlled by the config option mds_log_warn_factor, the default is 2.0. Some resources on the web (https://www.suse.com/support/kb/doc/?id=000019740) indicated that a solution would be to change the `mds_log_max_segments`. Which I did: ``` ceph --cluster floki tell mds.* injectargs '--mds_log_max_segments=400000' ``` Of course, the warning disappeared, but I have a feeling that I just hid the problem. Pushing a value to 400'000 when the default value is 512 is a lot. Why is the trimming not taking place? How can I troubleshoot this further? Best, Emmanuel

2 1

mds dump inode crashes file system
by Frank Schilder 26 May '23

26 May '23

Hi all, I have an annoying problem with a specific ceph fs client. We have a file server on which we re-export kernel mounts via samba (all mounts with noshare option). On one of these re-exports we have recurring problems. Today I caught it with 2023-05-10T13:39:50.963685+0200 mds.ceph-23 (mds.1) 1761 : cluster [WRN] client.205899841 isn't responding to mclientcaps(revoke), ino 0x20011d3e5cb pending pAsLsXsFscr issued pAsLsXsFscr, sent 61.705410 seconds ago and I wanted to look up what path the inode 0x20011d3e5cb points to. Unfortunately, the command ceph tell "mds.*" dump inode 0x20011d3e5cb crashes an MDS in a way that it restarts itself, but doesn't seem to come back clean (it does not fail over to a stand-by). If I repeat the command above, it crashes the MDS again. Execution on other MDS daemons succeeds, for example: # ceph tell "mds.ceph-24" dump inode 0x20011d3e5cb 2023-05-10T14:14:37.091+0200 7fa47ffff700 0 client.210149523 ms_handle_reset on v2:192.168.32.88:6800/3216233914 2023-05-10T14:14:37.124+0200 7fa4857fa700 0 client.210374440 ms_handle_reset on v2:192.168.32.88:6800/3216233914 dump inode failed, wrong inode number or the inode is not cached The caps recall gets the client evicted at some point but it doesn't manage to come back clean. On a single ceph fs mount point I see this # ls /shares/samba/rit-oil ls: cannot access '/shares/samba/rit-oil': Stale file handle All other mount points are fine, just this one acts up. A "mount -o remount /shares/samba/rit-oil" crashed the entire server and I had to do a cold reboot. On reboot I see this message: https://imgur.com/a/bOSLxBb , which only occurs on this one file server (we are running a few of those). Does this point to a more serious problem, like a file system corruption? Should I try an fs scrub on the corresponding path? Some info about the system: The file server's kernel version is quite recent, updated two weeks ago: $ uname -r 4.18.0-486.el8.x86_64 # cat /etc/redhat-release CentOS Stream release 8 Our ceph cluster is octopus latest and we use the packages from the octopus el8 repo on this server. We have several such shares and they all work fine. It is only on one share where we have persistent problems with the mount point hanging or the server freezing and crashing. After working hours I will try a proper fail of the "broken" MDS to see if I can execute the dump inode command without it crashing everything. In the mean time, any hints would be appreciated. I see that we have an exceptionally large MDS log for the problematic one. Any hint what to look for would be appreciated, it contains a lot from the recovery operations: # pdsh -w ceph-[08-17,23-24] ls -lh "/var/log/ceph/ceph-mds.ceph-??.log" ceph-23: -rw-r--r--. 1 ceph ceph 15M May 10 14:28 /var/log/ceph/ceph-mds.ceph-23.log *** huge *** ceph-24: -rw-r--r--. 1 ceph ceph 14K May 10 14:28 /var/log/ceph/ceph-mds.ceph-24.log ceph-10: -rw-r--r--. 1 ceph ceph 394 May 10 14:02 /var/log/ceph/ceph-mds.ceph-10.log ceph-13: -rw-r--r--. 1 ceph ceph 394 May 10 14:02 /var/log/ceph/ceph-mds.ceph-13.log ceph-08: -rw-r--r--. 1 ceph ceph 394 May 10 14:02 /var/log/ceph/ceph-mds.ceph-08.log ceph-15: -rw-r--r--. 1 ceph ceph 14K May 10 14:28 /var/log/ceph/ceph-mds.ceph-15.log ceph-17: -rw-r--r--. 1 ceph ceph 14K May 10 14:28 /var/log/ceph/ceph-mds.ceph-17.log ceph-14: -rw-r--r--. 1 ceph ceph 16K May 10 14:28 /var/log/ceph/ceph-mds.ceph-14.log ceph-09: -rw-r--r--. 1 ceph ceph 16K May 10 14:28 /var/log/ceph/ceph-mds.ceph-09.log ceph-16: -rw-r--r--. 1 ceph ceph 15K May 10 14:28 /var/log/ceph/ceph-mds.ceph-16.log ceph-11: -rw-r--r--. 1 ceph ceph 14K May 10 14:28 /var/log/ceph/ceph-mds.ceph-11.log ceph-12: -rw-r--r--. 1 ceph ceph 394 May 10 14:02 /var/log/ceph/ceph-mds.ceph-12.log Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 24

Multi region RGW Config Questions - Quincy
by Deep Dish 26 May '23

26 May '23

Hello, I have a Qunicy (17.2.6) cluster, looking to create a multi-zone / multi-region RGW service and have a few questions with respect to published docs - https://docs.ceph.com/en/quincy/radosgw/multisite/. In general, I understand the process as: 1. Create a new REALM, ZONEGROUP, ZONE: radosgw-admin realm create --rgw-realm=my_new_realm [--default] radosgw-admin zonegroup create --rgw-zonegroup=my_country --endpoints=http://rgw1:80 --rgw-realm=my_new_realm --master –default radosgw-admin zone create --rgw-zonegroup=my_country --rgw-zone=my-region *\* --master --default *\* --endpoints={http://fqdn}[,{http://fqdn}] ## Question: If I have multiple RGWs deployed on my cluster, do I specify all of them as individual endpoints? OR specifying one rgw automatically propagates config throughout all? 2. Create SYSTEM user radosgw-admin user create --uid="synchronization-user" --display-name="Synchronization User" --system radosgw-admin zone modify --rgw-zone={zone-name} --access-key={access-key} --secret={secret} radosgw-admin period update --commit ## Question: The SYSTEM user is used only for replication? Will creating new REALM, ZONGROUP, ZONE reset any administrative access to management of RGWs through ceph-dashboard? 3. Remove DEFAULT REALM, ZONEGROUP, ZONE and supporting pools radosgw-admin zonegroup delete --rgw-zonegroup=default --rgw-zone=default radosgw-admin period update --commit radosgw-admin zone delete --rgw-zone=default radosgw-admin period update --commit radosgw-admin zonegroup delete --rgw-zonegroup=default radosgw-admin period update --commit ceph osd pool rm default.rgw.control default.rgw.control --yes-i-really-really-mean-it ceph osd pool rm default.rgw.data.root default.rgw.data.root --yes-i-really-really-mean-it ceph osd pool rm default.rgw.gc default.rgw.gc --yes-i-really-really-mean-it ceph osd pool rm default.rgw.log default.rgw.log --yes-i-really-really-mean-it ceph osd pool rm default.rgw.users.uid default.rgw.users.uid --yes-i-really-really-mean-it 4. UPDATING CEPH CONFIG FILE / RGW CONFIG VIA CEPH ORCH # QUESTION: Since I’m using ceph orch would I simply set rgw_zone property via CLUSTER -> CONFIGURATION on ceph-dashboard? Thank you.

1 0

ln: failed to create hard link 'file name': Read-only file system
by Frank Schilder 26 May '23

26 May '23

Hi all, on an NFS re-export of a ceph-fs (kernel client) I observe a very strange error. I'm un-taring a larger package (1.2G) and after some time I get these errors: ln: failed to create hard link 'file name': Read-only file system The strange thing is that this seems only temporary. When I used "ln src dst" for manual testing, the command failed as above. However, after that I tried "ln -v src dst" and this command created the hard link with exactly the same path arguments. During the period when the error occurs, I can't see any FS in read-only mode, neither on the NFS client nor the NFS server. Funny thing is that file creation and write still works, its only the hard-link creation that fails. For details, the set-up is: file-server: mount ceph-fs at /shares/path, export /shares/path as nfs4 to other server other server: mount /shares/path as NFS More precisely, on the file-server: fstab: MON-IPs:/shares/folder /shares/nfs/folder ceph defaults,noshare,name=NAME,secretfile=sec.file,mds_namespace=FS-NAME,_netdev 0 0 exports: /shares/nfs/folder -no_root_squash,rw,async,mountpoint,no_subtree_check DEST-IP On the host at DEST-IP: fstab: FILE-SERVER-IP:/shares/nfs/folder /mnt/folder nfs defaults,_netdev 0 0 Both, the file server and the client server are virtual machines. The file server is on Centos 8 stream (4.18.0-338.el8.x86_64) and the client machine is on AlmaLinux 8 (4.18.0-425.13.1.el8_7.x86_64). When I change the NFS export from "async" to "sync" everything works. However, that's a rather bad workaround and not a solution. Although this looks like an NFS issue, I'm afraid it is a problem with hard links and ceph-fs. It looks like a race with scheduling and executing operations on the ceph-fs kernel mount. Has anyone seen something like that? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 13

`ceph features` on Nautilus still reports "luminous"
by Oliver Schmidt 26 May '23

26 May '23

Dear Ceph community, on our way towards getting our cluster to a current Ceph release, we updated all hosts and clients to Nautilus 14.2.22. But despite setting `ceph osd set-require-min-compat-client nautilus`, the release reported by `ceph features` is still "luminous". Is this supposed to be like this? If not, does anyone have an idea what might be missing to make the features being reported as "nautilus" as well? ``` ~ # ceph mon dump epoch 66 fsid b67bad36-3273-11e3-a2ed-0200000311bf last_changed 2022-12-12 12:20:39.244333 created 2013-10-11 14:57:32.291514 min_mon_release 14 (nautilus) 0: [v2:172.20.4.10:3300/0,v1:172.20.4.10:6789/0] mon.host1 1: [v2:172.20.4.100:3300/0,v1:172.20.4.100:6789/0] mon.host2 2: [v2:172.20.4.101:3300/0,v1:172.20.4.101:6789/0] mon.host3 dumped monmap epoch 66 ~ # ceph features { "mon": [ { "features": "0x3ffddff8ffecffff", "release": "luminous", "num": 3 } ], "osd": [ { "features": "0x3ffddff8ffecffff", "release": "luminous", "num": 14 } ], "client": [ { "features": "0x3ffddff8ffecffff", "release": "luminous", "num": 137 } ], "mgr": [ { "features": "0x3ffddff8ffecffff", "release": "luminous", "num": 3 } ] } ``` All the best -- Oliver Schmidt · os(a)flyingcircus.io · Systems Engineer Flying Circus Internet Operations GmbH · http://flyingcircus.io Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick

5 6

Re: [Help appreciated] ceph mds damaged
by Justin Li 26 May '23

26 May '23

Sorry Patrick, last email was restricted as attachment size. I attached a link for you to download the log. Thanks. https://drive.google.com/drive/folders/1bV_X7vyma_-gTfLrPnEV27QzsdmgyK4g?us… Justin Li Senior Technical Officer School of Information Technology Faculty of Science, Engineering and Built Environment For ICT Support please see https://www.deakin.edu.au/sebeicthelp Deakin University Melbourne Burwood Campus, 221 Burwood Highway, Burwood, VIC 3125 +61 3 9246 8932 Justin.li(a)deakin.edu.au http://www.deakin.edu.au/ Deakin University CRICOS Provider Code 00113B Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone. Deakin University does not warrant that this email and any attachments are error or virus free. -----Original Message----- From: Justin Li Sent: Wednesday, May 24, 2023 8:21 AM To: Patrick Donnelly <pdonnell(a)redhat.com> Cc: ceph-users(a)ceph.io Subject: RE: [ceph-users] [Help appreciated] ceph mds damaged Hi Patrick, I attached two logs here. Those two servers are one of the monitors and MDSs. Let me know if you need more logs. Thanks. Justin Li Senior Technical Officer School of Information Technology Faculty of Science, Engineering and Built Environment For ICT Support please see https://www.deakin.edu.au/sebeicthelp Deakin University Melbourne Burwood Campus, 221 Burwood Highway, Burwood, VIC 3125 +61 3 9246 8932 Justin.li(a)deakin.edu.au http://www.deakin.edu.au/ Deakin University CRICOS Provider Code 00113B Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone. Deakin University does not warrant that this email and any attachments are error or virus free. -----Original Message----- From: Patrick Donnelly <pdonnell(a)redhat.com> Sent: Wednesday, May 24, 2023 7:35 AM To: Justin Li <justin.li(a)deakin.edu.au> Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] [Help appreciated] ceph mds damaged Hello Justin, On Tue, May 23, 2023 at 4:55 PM Justin Li <justin.li(a)deakin.edu.au> wrote: > > Dear All, > > After a unsuccessful upgrade to pacific, MDS were offline and could not get back on. Checked the MDS log and found below. See cluster info from below as well. Appreciate it if anyone can point me to the right direction. Thanks. > > > MDS log: > > 2023-05-24T06:21:36.831+1000 7efe56e7d700 1 mds.0.cache.den(0x600 > 1005480d3b2) loaded already corrupt dentry: [dentry > #0x100/stray0/1005480d3b2 [19ce,head] rep(a)0,-2.0<mailto:rep@0,-2.0> > NULL (dversion lock) pv=0 v=2154265030 ino=(nil) state=0 > 0x556433addb80] > > -5> 2023-05-24T06:21:36.831+1000 7efe56e7d700 -1 mds.0.damage > notify_dentry Damage to dentries in fragment * of ino 0x600is fatal > because it is a system directory for this rank > > -4> 2023-05-24T06:21:36.831+1000 7efe56e7d700 5 mds.beacon.posco > set_want_state: up:active -> down:damaged > > -3> 2023-05-24T06:21:36.831+1000 7efe56e7d700 5 mds.beacon.posco > Sending beacon down:damaged seq 5339 > > -2> 2023-05-24T06:21:36.831+1000 7efe56e7d700 10 monclient: > _send_mon_message to mon.ceph-3 at v2:10.120.0.146:3300/0 > > -1> 2023-05-24T06:21:37.659+1000 7efe60690700 5 mds.beacon.posco > received beacon reply down:damaged seq 5339 rtt 0.827966 > > 0> 2023-05-24T06:21:37.659+1000 7efe56e7d700 1 mds.posco respawn! > > > Cluster info: > root@ceph-1:~# ceph -s > cluster: > id: e2b93a76-2f97-4b34-8670-727d6ac72a64 > health: HEALTH_ERR > 1 filesystem is degraded > 1 filesystem is offline > 1 mds daemon damaged > > services: > mon: 3 daemons, quorum ceph-1,ceph-2,ceph-3 (age 26h) > mgr: ceph-3(active, since 15h), standbys: ceph-1, ceph-2 > mds: 0/1 daemons up, 3 standby > osd: 135 osds: 133 up (since 10h), 133 in (since 2w) > > data: > volumes: 0/1 healthy, 1 recovering; 1 damaged > pools: 4 pools, 4161 pgs > objects: 230.30M objects, 276 TiB > usage: 836 TiB used, 460 TiB / 1.3 PiB avail > pgs: 4138 active+clean > 13 active+clean+scrubbing > 10 active+clean+scrubbing+deep > > > > root@ceph-1:~# ceph health detail > HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds > daemon damaged [WRN] FS_DEGRADED: 1 filesystem is degraded > fs cephfs is degraded > [ERR] MDS_ALL_DOWN: 1 filesystem is offline > fs cephfs is offline because no MDS is active for it. > [ERR] MDS_DAMAGE: 1 mds daemon damaged > fs cephfs mds.0 is damaged Do you have a complete log you can share? Try: https://docs.ceph.com/en/quincy/man/8/ceph-post-file/ To get your upgrade to complete, you may set: ceph config set mds mds_go_bad_corrupt_dentry false -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone. Deakin University does not warrant that this email and any attachments are error or virus free.

2 4

Feature/Change Request: Don't send alert emails for --sticky muted WARN conditions
by Edward R Huyer 25 May '23

25 May '23

I recently upgraded to Quincy, and toggled on the BULK flag of a few pools. As a result, my cluster has been spending the last several days shuffling data while growing the pool pg counts. That in turn has resulted in a steadily increasing number of pgs being flagged PG_NOT_DEEP_SCRUBBED. And that has resulted in my getting hundreds of alert emails about pgs not being deep scrubbed, because I get a new email whenever the count changes. I tried using "ceph health mute PG_NOT_DEEP_SCRUBBED --sticky", but all that did (in terms of the email alerts) was make the emails say "HEALTH_OK" instead of "HEALTH WARN", which is less than helpful. I haven't found a way to stop the cluster from sending me these alert emails other than turning off email notifications entirely. If there is one, I'd love to know what it is. If not, I feel like there ought to be one, either as part of muting the health warning, or as a separate toggle. Hundreds of emails over what is expected behavior is rather silly. ----- Edward Huyer Golisano College of Computing and Information Sciences Rochester Institute of Technology Golisano 70-2373 152 Lomb Memorial Drive Rochester, NY 14623 585-475-6651 erhvks(a)rit.edu<mailto:erhvks@rit.edu> Obligatory Legalese: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this information.

1 0

MDS Upgrade from 17.2.5 to 17.2.6 not possible
by Henning Achterrath 25 May '23

25 May '23

Hi all, we did a major update from Pacific to Quincy (17.2.5) a month ago without any problems. Now we have tried a minor update from 17.2.5 to 17.2.6 (ceph orch upgrade). It stucks at mds upgrade phase. At this point the cluster tries to scale down mds (ceph fs set max_mds 1). We waited a few hours. We are running two active mds with 1 standby. No subdir pinning configured. CephFS data pool: 575 TB While Upgrading, Rank 1 MDS remains in state stopping. During this state clients are not able to reconnect. So we paused this upgrade and set max_mds to 2 back again and fail rank 1. After that, standby becomes active. In the mds (rank 1 in stopping state) logs we can see: waiting for strays to migrate In our second try, we have evicted all clients first without success. We make daily snapshots on / and rotate them via snapshot scheduler after one week. Is there a way to get rid of stray entries without scale down mds or do we have to wait longer? We had about the same amount of strays before we did the major upgrade. So, it is a bit curious. Current output from ceph perf dump Rank0: "num_strays": 417304, "num_strays_delayed": 3, "num_strays_enqueuing": 0, "strays_created": 567879, "strays_enqueued": 561803, "strays_reintegrated": 13751, "strays_migrated": 4, Rank1: ceph daemon mds.fdi-cephfs.ceph-service-13.rwdkqs perf dump | grep stray "num_strays": 172528, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "strays_created": 418365, "strays_enqueued": 396142, "strays_reintegrated": 67406, "strays_migrated": 4, Any help would be appreciated. best regards Henning

6 7

sub-read of librados
by johnnyjohnnypd＠gmail.com 25 May '23

25 May '23

Hi! I am trying to read only part of an object by specifying the non-trivial offset and length of the read function: `librados::IoCtxImpl::read(const object_t& oid, bufferlist& bl, size_t len, uint64_t off)` from `IoCtxImpl.cc`. However, after connecting to an erasure code pool (e.g., 12+4), I try to read data from a randomly chosen OSD (that is, 1/12 of the object), but the results of command `vmstat -d` and `iostat` show that the entire object was read, since read operations appeared on all 12 OSDs. So, I wonder if librados doesn't support the real sub-read of an object, and what should I do if I want to implement this function. Thanks!

1 0

mgr memory usage constantly increasing
by Tobias Hachmer 25 May '23

25 May '23

Hello list, we have found that the active mgr process in our 3-node CEPH cluster takes a lot of memory. After start the memory usage is constantly increasing. After 6 days the process takes ~67GB: ~# ps -p 7371 -o rss,%mem,cmd RSS %MEM CMD 71053880 26.9 /usr/bin/ceph-mgr -n mgr.hostname.nvwzhc -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false Cluster Specs: - 3-node 3-way mirror - each node has 256GB memory - each node has 10x 7.68TB NVMe, each NVMe is split into 2 OSDs - monitoring node is a separate VM (separate Hypervisor) - iSCSI and NFS Gateway are located on 2 separate VMs (separate Hypervisor) - main purpose is CephFS with currently ~15.42 million objects Is this normal behaviour or might we have a misconfiguration somewhere? What can we do to dig into this further? ~# ceph status cluster: id: f5129939-964b-11ed-bb6a-f7caa5af2f56 health: HEALTH_OK services: mon: 3 daemons, quorum host1,host2,host3 (age 6d) mgr: host2.nvwzhc(active, since 6d), standbys: host1.wwczzn mds: 1/1 daemons up, 1 standby, 1 hot standby osd: 60 osds: 60 up (since 6d), 60 in (since 8w) tcmu-runner: 2 portals active (2 hosts) data: volumes: 1/1 healthy pools: 6 pools, 2161 pgs objects: 15.42M objects, 45 TiB usage: 135 TiB used, 75 TiB / 210 TiB avail pgs: 2161 active+clean Thanks and kind regards Tobias Hachmer

3 4

Ceph Tech Talk For May 2023: RGW Lua Scripting Code Walkthrough
by Mike Perez 24 May '23

24 May '23

Hello everyone, Join us on May 24th at 17:00 UTC for a long overdue Ceph Tech Talk! This month, Yuval Lifshitz will give an RGW Lua Scripting Code Walkthrough. https://ceph.io/en/community/tech-talks/ You can also see Yuval's previous presentation at Ceph Month 2021, From Open Source to Open Ended in Ceph with Lua. https://www.youtube.com/watch?v=anQJugs27hE If you want to give a technical presentation for Ceph Tech Talks, please contact me directly with a title and description. Thank you! -- Mike Perez Community Manager Ceph Foundation

2 1

Training on ceph fs
by Emmanuel Jaep 24 May '23

24 May '23

Hi, I inherited a ceph fs cluster. Even if I have years of experience in systems management, I fail to grasp the complete logic of it fully. From what I found on the web, the documentation is either too "high level" or too detailed. Do you know any good resources to get fully acquainted with ceph (specifically, ceph fs)? Or any live training I could attend? Thanks in advance for your help, Emmanuel

2 1

Re: MDS crashes to damaged metadata
by Stefan Kooman 24 May '23

24 May '23

On 12/15/22 15:31, Stolte, Felix wrote: > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): Are you willing to share this script? I would like to use it to scan our CephFS before upgrading to 16.2.13. Do you run this script when the filesystem is online / active? Thanks, Gr. Stefan

2 3

cephfs max_file_size
by Dietmar Rieder 24 May '23

24 May '23

Hi, can the cephfs "max_file_size" setting be changed at any point in the lifetime of a cephfs? Or is it critical for existing data if it is changed after some time? Is there anything to consider when changing, let's say, from 1TB (default) to 4TB ? We are running the latest Nautilus release, BTW. Thanks in advance Dietmar

3 5

Slow recovery on Quincy
by Sake Paulusma 24 May '23

24 May '23

We noticed extremely slow performance when remapping is necessary. We didn't do anything special other than assigning the correct device_class (to ssd). When checking ceph status, we notice the number of objects recovering is around 17-25 (with watch -n 1 -c ceph status). How can we increase the recovery process? There isn't any client load, because we're going to migrate to this cluster in the future, so only a rsync once a while is being executed. [ceph: root@pwsoel12998 /]# ceph status cluster: id: da3ca2e4-ee5b-11ed-8096-0050569e8c3b health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set services: mon: 5 daemons, quorum pqsoel12997,pqsoel12996,pwsoel12994,pwsoel12998,prghygpl03 (age 3h) mgr: pwsoel12998.ylvjcb(active, since 3h), standbys: pqsoel12997.gagpbt mds: 4/4 daemons up, 2 standby osd: 32 osds: 32 up (since 73m), 32 in (since 6d); 10 remapped pgs flags noscrub,nodeep-scrub data: volumes: 2/2 healthy pools: 5 pools, 193 pgs objects: 13.97M objects, 853 GiB usage: 3.5 TiB used, 12 TiB / 16 TiB avail pgs: 755092/55882956 objects misplaced (1.351%) 183 active+clean 10 active+remapped+backfilling io: recovery: 2.3 MiB/s, 20 objects/s

8 17

[cephfs-data-scan] Estimate time for scanning extents and inodes
by Justin Li 24 May '23

24 May '23

Dear All, I'm using metadata repair tools to repair a damaged MDS following below document. My storage has about 276TB data. Cephfs-data-scan is using 32 workers. How long will it take to finish scanning extents? What about scanning inodes? It has run 6 hours and metadata pool dropped 1G. Is this normal? Thanks. ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 1.3 PiB 454 TiB 835 TiB 835 TiB 64.79 ssd 7.1 TiB 6.2 TiB 878 GiB 878 GiB 12.15 TOTAL 1.3 PiB 460 TiB 836 TiB 836 TiB 64.50 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL cephfs_data 1 4096 276 TiB 222.96M 828 TiB 71.28 111 TiB cephfs_metadata 2 32 101 GiB 5.83M 302 GiB 0.09 111 TiB device_health_metrics 10 1 239 MiB 251 716 MiB 0 112 TiB cephfs_ssd 11 32 117 GiB 1.49M 361 GiB 5.83 1.9 TiB https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/ Justin Li Senior Technical Officer School of Information Technology Faculty of Science, Engineering and Built Environment Request for assistance can be lodged to the SIT Technical Team using this form<https://deakinesmprod.service-now.com/esc?id=sc_cat_item&sys_id=7afa8fa5db6…> Deakin University Melbourne Burwood Campus, 221 Burwood Highway, Burwood, VIC 3125 +61 3 9246 8932 justin.li(a)deakin.edu.au<mailto:justin.li@deakin.edu.au> http://www.deakin.edu.au<http://www.deakin.edu.au/> Deakin University CRICOS Provider Code 00113B Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone. Deakin University does not warrant that this email and any attachments are error or virus free. Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone. Deakin University does not warrant that this email and any attachments are error or virus free.

1 0

RGW: How to restore the index and metadata of a bucket
by huy nguyen 24 May '23

24 May '23

There is a test bucket, I have removed its index and metadata: radosgw-admin bi purge --bucket abccc --yes-i-really-mean-it radosgw-admin metadata rm bucket.instance:abccc:17a4ce99-009e-40f2-a2d2-2afc218ebd9b.425824299.4 Now the index and metadata is gone, but how to clean its data? Or is there any way to restore the index and metadata? I have tried to fix the index, but it doesn't work: radosgw-admin bucket check --check-objects --fix --bucket abccc 2023-05-24 16:46:41.896 7f32d5890a80 -1 ERROR: get_bucket_instance_from_oid failed: -2 2023-05-24 16:46:41.896 7f32d5890a80 0 could not get bucket info for bucket=abccc Same result with bucket rm: radosgw-admin bucket rm --bucket abccc --purge-objects 2023-05-24 16:47:49.439 7fafb7939a80 -1 ERROR: get_bucket_instance_from_oid failed: -2 2023-05-24 16:47:49.439 7fafb7939a80 0 could not get bucket info for bucket=abccc

1 0

Timeout in Dashboard
by mailing-lists 24 May '23

24 May '23

Hey all, im facing a "minor" problem. I do not always get results when going to the dashboard, under Block->Images in the tab Images or Namespaces. The little refresh button will keep spinning and sometimes after several minutes it will finally show something. That is odd, because from the shell I am not seeing any problems like that. rbd namespace ls as well as: rbd ls --namespace someNS ...will give immediately their results. Does anyone know whats causing this? I did not yet find anything in the logs. Best Ken

1 0

[Help appreciated] ceph mds damaged
by Justin Li 23 May '23

23 May '23

Dear All, After a unsuccessful upgrade to pacific, MDS were offline and could not get back on. Checked the MDS log and found below. See cluster info from below as well. Appreciate it if anyone can point me to the right direction. Thanks. MDS log: 2023-05-24T06:21:36.831+1000 7efe56e7d700 1 mds.0.cache.den(0x600 1005480d3b2) loaded already corrupt dentry: [dentry #0x100/stray0/1005480d3b2 [19ce,head] rep(a)0,-2.0<mailto:rep@0,-2.0> NULL (dversion lock) pv=0 v=2154265030 ino=(nil) state=0 0x556433addb80] -5> 2023-05-24T06:21:36.831+1000 7efe56e7d700 -1 mds.0.damage notify_dentry Damage to dentries in fragment * of ino 0x600is fatal because it is a system directory for this rank -4> 2023-05-24T06:21:36.831+1000 7efe56e7d700 5 mds.beacon.posco set_want_state: up:active -> down:damaged -3> 2023-05-24T06:21:36.831+1000 7efe56e7d700 5 mds.beacon.posco Sending beacon down:damaged seq 5339 -2> 2023-05-24T06:21:36.831+1000 7efe56e7d700 10 monclient: _send_mon_message to mon.ceph-3 at v2:10.120.0.146:3300/0 -1> 2023-05-24T06:21:37.659+1000 7efe60690700 5 mds.beacon.posco received beacon reply down:damaged seq 5339 rtt 0.827966 0> 2023-05-24T06:21:37.659+1000 7efe56e7d700 1 mds.posco respawn! Cluster info: root@ceph-1:~# ceph -s cluster: id: e2b93a76-2f97-4b34-8670-727d6ac72a64 health: HEALTH_ERR 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged services: mon: 3 daemons, quorum ceph-1,ceph-2,ceph-3 (age 26h) mgr: ceph-3(active, since 15h), standbys: ceph-1, ceph-2 mds: 0/1 daemons up, 3 standby osd: 135 osds: 133 up (since 10h), 133 in (since 2w) data: volumes: 0/1 healthy, 1 recovering; 1 damaged pools: 4 pools, 4161 pgs objects: 230.30M objects, 276 TiB usage: 836 TiB used, 460 TiB / 1.3 PiB avail pgs: 4138 active+clean 13 active+clean+scrubbing 10 active+clean+scrubbing+deep root@ceph-1:~# ceph health detail HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged [WRN] FS_DEGRADED: 1 filesystem is degraded fs cephfs is degraded [ERR] MDS_ALL_DOWN: 1 filesystem is offline fs cephfs is offline because no MDS is active for it. [ERR] MDS_DAMAGE: 1 mds daemon damaged fs cephfs mds.0 is damaged Justin Li Senior Technical Officer School of Information Technology Faculty of Science, Engineering and Built Environment Request for assistance can be lodged to the SIT Technical Team using this form<https://deakinesmprod.service-now.com/esc?id=sc_cat_item&sys_id=7afa8fa5db6…> Deakin University Melbourne Burwood Campus, 221 Burwood Highway, Burwood, VIC 3125 +61 3 9246 8932 justin.li(a)deakin.edu.au<mailto:justin.li@deakin.edu.au> http://www.deakin.edu.au<http://www.deakin.edu.au/> Deakin University CRICOS Provider Code 00113B Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone. Deakin University does not warrant that this email and any attachments are error or virus free. Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone. Deakin University does not warrant that this email and any attachments are error or virus free.

3 4

Unable to change port range for OSDs in Pacific
by Renata Callado Borges 23 May '23

23 May '23

Dear all, How are you? I have a Pacific 3 nodes cluster, and the machines do double-duty as Ceph nodes and as Slurm clients. (I am well aware that this is not desirable, but my client wants it like this anyway). Our Slurm install uses the port 6818 for slurmd everywhere. In one of our Ceph/Slurm nodes, Ceph decided that port 6818 is great for an OSD. This prevents slurmd from running properly. Changing the slurmd port causes the Slurm master, slurmctld, to misread the OSD communication as Slurm "Insane length messages". I have tried unsuccessfully to change this port in Slurm and in Ceph. I wonder if someone here can help me limit the ports Ceph uses for its OSDs. I have tried this with no success: [root@darkside1 ~]# cat /etc/ceph/ceph.conf~ # minimal ceph.conf for 1902a026-496d-11ed-b43e-08c0eb320ec2 [global] fsid = 1902a026-496d-11ed-b43e-08c0eb320ec2 mon_host = [v2:172.22.132.188:3300/0,v1:172.22.132.188:6789/0] [osd] ms_bind_port_min = 6830 ms_bind_port_max = 7300 Then I restarted with "systemctl restart ceph.target" but the OSD keeps being re-bound to 6818. I also tried the same config but with the options under the [global] section. No luck there either. Tried reboot the Ceph/Slurm machine, the OSD is re-bound in 6818 also. Could someone help? Thanks in advance! Cordially, Renata.

2 2

Grafana service fails to start due to bad directory name after Quincy upgrade
by Adiga, Anantha 23 May '23

23 May '23

Hi Upgraded from Pacific 16.2.5 to 17.2.6 on May 8th However, Grafana fails to start due to bad folder path :/tmp# journalctl -u ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201 -n 25 -- Logs begin at Sun 2023-05-14 20:05:52 UTC, end at Tue 2023-05-16 19:07:51 UTC. -- May 16 19:05:00 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:00 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:00 fl31ca104ja0201 bash[2575021]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:00 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:00 fl31ca104ja0201 bash[2575030]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:00 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:10 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:10 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:10 fl31ca104ja0201 bash[2575273]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:10 fl31ca104ja0201 bash[2575282]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:20 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:20 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:20 fl31ca104ja0201 bash[2575369]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:20 fl31ca104ja0201 bash[2575370]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:30 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Start request r> May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:30 fl31ca104ja0201 systemd[1]: Failed to start Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ESCOC 19:07:51 UTC. -- 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory -be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Main process exited, code=exited, status=127/n/a ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory -be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Failed with result 'exit-code'. -be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Scheduled restart job, restart counter is at 3. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory -be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Main process exited, code=exited, status=127/n/a ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory -be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Failed with result 'exit-code'. -be05-a7a3a1d7a87e(a)grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Scheduled restart job, restart counter is at 4. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. Check if path exists: #<mailto:root@fl31ca104ja0201:/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201#> ls /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run ls: cannot access '/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run': No such file or directory Check if grafana.fl31ca104ja0201 directory exists: #<mailto:root@fl31ca104ja0201:/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201#> ls -l rt/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/unit.run total 36 drwxr-xr-x 4 167 167 4096 Apr 20 08:05 data drwxr-xr-x 3 167 167 4096 Apr 20 08:13 etc -rw------- 1 167 167 48 Apr 20 08:13 unit.created -rw------- 1 167 167 390 May 8 16:12 unit.stop -rw------- 1 167 167 390 May 8 16:12 unit.poststop -rw------- 1 167 167 365 May 8 16:12 unit.meta -rw------- 1 167 167 32 May 8 16:12 unit.image -rw------- 1 167 167 38 May 8 16:12 unit.configured -rw------- 1 167 167 1063 May 8 16:12 unit.run cat unit.run set -e # grafana.fl31ca104ja0201 ! /usr/bin/docker rm -f ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana.fl31ca104ja0201 2> /dev/null ! /usr/bin/docker rm -f ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 2> /dev/null /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --init --name ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 --user 472 -e CONTAINER_IMAGE=docker.io/grafana/grafana:6.7.4 -e NODE_NAME=fl31ca104ja0201 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/grafana.ini:/etc/grafana/grafana.ini:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/certs:/etc/grafana/certs:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/data/grafana.db:/var/lib/grafana/grafana.db:Z docker.io/grafana/grafana:6.7.4 Thank you, Anantha

3 5

Deleting a CephFS volume
by Conrad Hoffmann 23 May '23

23 May '23

Hi all, I'm having difficulties removing a CephFS volume that I set up for testing. I've been through this with RBDs, so I do know about `mon_allow_pool_delete`. However, it doesn't help in this case. It is a cluster with 3 monitors. You can find a console log of me verifying that `mon_allow_pool_delete` is indeed true on all monitors but still fail to remove the volume here: https://paste.sr.ht/~bitfehler/4b317e5b8c5823583c1b26b42e8495f54cbd5e2a I am happy to post config or logs as needed, I wasn't sure which ones might be relevant. I scanned monitor and MDS logs, but nothing stuck out. Using the volume worked fine in tests, so I don't think it is broken or anything. Any ideas would be greatly appreciated! Thanks a bunch, Conrad

3 4

Advices : RGW and Citrix ADC
by BEAUDICHON Hubert (Acoss) 22 May '23

22 May '23

Hi everyone, Is there any advices, best practices for a Citrix ADC VIP configuration in front of a dual RadosGW (same zone) ? I find many documentations for HAProxy but nothing for this hardware. For information, the CEPH cluster version is Pacific 16.2.10. Thanks for your help. Regards, Hubert [cid:image001.png@01D98CA0.D075D720] Hubert Beaudichon Secteur Infrastructures / Gamme CET-Infrastructures SDAT UT AUTOMATION ET CLOUD Hubert.beaudichon(a)acoss.fr<mailto:Hubert.beaudichon@acoss.fr> 01 82 30 77 03 * 06 00 00 00 00 36 rue de Valmy * 93108 Montreuil CEDEX www.urssaf.fr<http://www.urssaf.fr> [cid:image002.png@01D98CA1.F1ADE8F0]<https://www.youtube.com/channel/UCSjS8hOXCs5FlAnTi74rPrA>[cid:image003.png@01D98CA1.F1ADE8F0]<https://www.linkedin.com/company/acoss/>[cid:image002.png@01D98CA1.F1ADE8F0]<https://twitter.com/urssaf>[cid:image005.png@01D98CA0.D075D720]

1 0

RGW: are there any ways to limit access from user to a storage class?
by huy nguyen 22 May '23

22 May '23

There are tags for placement, so any user that has the tag is allowed to access the placement. But I can't find how to do it for storage class? Anyone have any ideal? Thanks.

1 0

Colocated RGW using same port, but different IPs
by Ondřej Kukla 22 May '23

22 May '23

Hello, As a part of performance/HW utilization testing I would like to compare how a VM with single RGW instance compares to a VM with 2 or more colocated RGWs. I’m deploying Quincy using Ansible but I’m struggling with setting the config to use the same port across all RGW instances, but to have a different IP as a beast endpoint. I would like to ask if anyone has any suggestion on this topic. Is this even supported scenario or it is better to have the instances using different ports? Any help is much appreciated. Kind regards, Ondrej

1 0

MDS migrations
by Odair M. 21 May '23

21 May '23

Hi, I'm doing a master's degree in distributed file systems, specifically CephFS in HPC, and I have some questions about the metadata migration process of MDSes that balances the MDS cluster. * Sage Weil Ph.D. thesis' and the documentation in the Ceph website say that before migrating to balance the MDS, the first step is to freeze the subtree and when the migration is complete, to unfreeze it. Doesn't it affect the performance since the subtree stays unavailable until migration is over? * Could you explain why the freezes are required at all? * Is It possible to change the migration process to migrate a copy of the inodes and then migrate inodes that have modifications after the migration? Like taking a snapshot of the subtree and migrate that, and then migrate any changes that happened after the snapshot migration is done, reducing the time spent with the subtree frozen. -- Odair M. Ditkun Jr Master's networks and distributed systems — University of State Parana — Brazil

1 0

RGW: request fields in Lua script return wrong data?
by viplanghe6＠gmail.com 21 May '23

21 May '23

Thanks for your answer. I was able to get the Lua debug log. But I think some request fields don't work: I have this lua script, for example: if Request.HTTP.StorageClass == 'COLD' then RGWDebugLog(Request.RGWOp .. " request with StorageClass: " .. Request.HTTP.StorageClass .. " Obj name: " .. Request.Object.Name .. " Obj ID: " .. Request.Object.Id .. " Size: " .. Request.Object.Size .. " MTime: " .. Request.Object.MTime ) end I put an 1kb object using boto3: s3.Object(bucket, 'test-object6').put(Body="0"*1000, ACL='public-read', StorageClass='COLD') But the size and mtime field provided wrong data? Lua INFO: put_obj request with StorageClass: COLD Obj name: test-object6 Obj ID: test-object6 Size: 0 MTime: 1970-01-01 08:00:00

3 3

Dedicated radosgw gateways
by Michal Strnad 19 May '23

19 May '23

Hi all, at Cephalocon 2023, it was mentioned several times that for service tasks such as data deletion via garbage collection or data replication in S3 via zoning, it is good to do them on dedicated radosgw gateways and not mix them with gateways used by users. How can this be achieved? How can we isolate these tasks? Will using dedicated keyrings instead of admin keys be sufficient? How do you operate this in your environment? Thx Michal

6 10

Ceph OSDs suddenly use public network for heardbeat_check
by Lee, H. (Hurng-Chun) 19 May '23

19 May '23

Hi all, We are running the Nautilus cluster. Today due to UPS work, we shut down the whole cluster. After we start the cluster, many OSDs go down and they seem to start doing the heardbeat_check using the public network. For example, we see the following logs: --- 2023-05-16 19:35:29.254 7efcd4ce7700 -1 osd.101 42916 heartbeat_check: no reply from 131.174.45.223:6825 osd.185 ever on either front or back, first ping sent 2023-05-16 19:34:48.593701 (oldest deadline 2023-05-16 19:35:08.593701) --- While I was expect the heardbeat to go through the cluster network, e.g. instead of 131.174.45.223, it should use 172.20.128.223. In fact, when we start up the cluster, we don't have DNS available to resolve the IP addresses, and for a short while, all OSDs are located in a new host called "localhost.localdomain". At that point, I fixed it by setting the static hostname using `hostnamectl set -hostname xxx`. Now we cannot bring the cluster back to healthy state. We are stuck at: --- cluster: id: 86c9bc85-b7f3-49a1-9e1f-8c9f2b31fca8 health: HEALTH_ERR 1 filesystem is degraded 1 filesystem has a failed mds daemon 1 filesystem is offline insufficient standby MDS daemons available pauserd,pausewr,noout,nobackfill,norebalance,norecover flag(s) set 88 osds down Reduced data availability: 2544 pgs inactive, 2369 pgs down, 159 pgs peering, 294 pgs stale Degraded data redundancy: 870424/2714593746 objects degraded (0.032%), 30 pgs degraded, 9 pgs undersized 8631 slow ops, oldest one blocked for 803 sec, mon.ceph- mon01 has slow ops services: mon: 3 daemons, quorum ceph-mon01,ceph-mon02,ceph-mon03 (age 2h) mgr: ceph-mon02(active, since 2h), standbys: ceph-mon01, ceph-mon03 mds: cephfs:0/1, 1 failed osd: 191 osds: 103 up (since 1.46827s), 191 in (since 4w) flags pauserd,pausewr,noout,nobackfill,norebalance,norecover data: pools: 2 pools, 2560 pgs objects: 456.13M objects, 657 TiB usage: 1.1 PiB used, 638 TiB / 1.7 PiB avail pgs: 100.000% pgs not active 870424/2714593746 objects degraded (0.032%) 2087 down 282 stale+down 129 peering 32 stale+peering 30 undersized+degraded+peered --- Any idea we could fix it and get the OSDs to use cluster network to do heartbeat checks. Any help would be highly appreciated. Thank you very much. Cheers, Hong -- Hurng-Chun (Hong) Lee, PhD ICT manager Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive Neuroimaging Radboud University Nijmegen e-mail: h.lee(a)donders.ru.nl tel: +31(0)631132518 web: http://www.ru.nl/donders/ pgp: 3AC505B2B787A8ABE2C551B1362976D838ABF09E * Mon, Tue and Thu at Trigon; Wed and Fri working from home

3 3

ls: cannot access '/cephfs': Stale file handle
by Harry G Coin 18 May '23

18 May '23

I have two autofs entries that mount the same cephfs file system to two different mountpoints. Accessing the first of the two fails with 'stale file handle'. The second works normally. Other than the name of the mount point, the lines in autofs are identical. No amount of 'umount -f' or restarting autofs resolves it. Any ideas?

2 1

User + Dev Monthly Meeting Minutes 2023-05-18
by Laura Flores 18 May '23

18 May '23

Hey all, Here are the minutes from today's meeting: Reef RC0 status? - ETA is the last week of May - Several users have volunteered to help with Reef scale testing; efforts are documented in this etherpad: https://pad.ceph.com/p/reef_scale_testing [Ben Gao] feature request: for non root user with password to install cluster with cephadm - There was work done on this previously, but it was closed in this PR: https://github.com/ceph/ceph/pull/44085 - Before, Sebastian questioned the use case of this feature and overhead - Ben's use case: Customer network security policy; currently ceph admin only supports root install - Adam King: We would be open to reviewing any patches submitted, but ultimately ssh keys are a better option. [Andras Pataki] Crush upmap fix: https://tracker.ceph.com/issues/51729 - There is a fix, but it needs to be well tested, as this is a high-traffic area of code - Andras is interested in testing the fix in a test environment; would need to upgrade the OSDs to include the crushwrapper code fix - Laura recommends against testing the fix on production clusters - Update the Tracker if you need a test build See you all next month! - Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

CLT Meeting minutes 17/05/23
by Nizamudeen A 18 May '23

18 May '23

Hey all, Ceph Quarterly announcement [Josh and Zac] One page digest that may published quarterly Planning for 1st of June, September and December Reef RC https://pad.ceph.com/p/reef_scale_testing https://pad.ceph.com/p/ceph-user-dev-monthly-minutes#L17 ETA last week of May Missing CentOS 9 Python deps Ken Dreyer has volunteered to help get the packages in epel Lab update There have been reimaging issues and kernel timeouts seen in the lab; infra team is working through fixing it Please raise infrastructure trackers for any bugs you see, as we have started to do monthly infrastructure bug scrubs Regards, Nizam

1 0

Deleting millions of objects
by Rok Jaklič 18 May '23

18 May '23

Hi, I would like to delete millions of objects in RGW instance with: mc rm --recursive --force ceph/archive/veeam but it seems it allows only 1000 (or 1002 exactly) removals per command. How can I delete/remove all objects with some prefix? Kind regards, Rok

7 9

Discussion thread for Known Pacific Performance Regressions
by Mark Nelson 17 May '23

17 May '23

Hi Everyone, This email was originally posted to dev(a)ceph.io, but Marc mentioned that he thought this would be useful to post on the user list so I'm re-posting here as well. David Orman mentioned in the CLT meeting this morning that there are a number of people on the mailing list asking about performance regressions in Pacific+ vs older releases. I want to document a couple of the bigger ones that we know about for the community's benefit. I want to be clear that Pacific does have a number of performance improvements over previous releases, and we do have tests showing improvement relative to nautilus (especially RBD on NVMe drives). Some of these regressions are going to have a bigger effect for some users than others. Having said that, let's get into them. ********** Regression #1: RocksDB Log File Recycling ********** Effects: More metadata updates to the underlying FS, higher write-amplification (Observed by Digital Ocean), Slower performance especially when the WAL device is saturated. When bluestore was created back in 2015 Sage implemented an optimization in RocksDB that allowed WAL log files to be recycled. The idea is that instead of deleting logs when they are flushed, rocksdb can simply reuse them. The benefit here is that it allows records to be written and fadatasync can be called without touching the inode for every IO. Sage did a pretty good job of explaining the benefit in the PR available here: https://github.com/facebook/rocksdb/pull/746 After much discussion, that PR was merged and received a couple of bug fixes over the years: Locking bug fix from Somnath back in 2016: https://github.com/facebook/rocksdb/pull/1313 Another bug fix from ajkr in 2020: https://github.com/facebook/rocksdb/pull/5900 In 2020, the RocksDB folks discovered there is a fundamental flaw in the way that the original PR works. It turns out that the feature to recycle log files is incompatible with RocksDB's kPointInTimeRecovery, kAbsoluteConsistency, and kTolerateCorruptedTailRecords recovery modes. One of the later PR's included a very good and concise description of the problem: "The two features are naturally incompatible. WAL recycling expects the recovery to succeed upon encountering a corrupt record at the point where new data ends and recycled data remains at the tail. However, WALRecoveryMode::kTolerateCorruptedTailRecords must fail upon encountering any such corrupt record, as it cannot differentiate between this and a real corruption, which would cause committed updates to be truncated." More background discussion on the RocksDB side available in these PRs and comments: https://github.com/facebook/rocksdb/pull/6351 https://github.com/facebook/rocksdb/pull/6351#issuecomment-672838284 https://github.com/facebook/rocksdb/pull/7252 https://github.com/facebook/rocksdb/pull/7271 On the Ceph side, there was a PR to try to re-enable the old behavior which we rejected as unsafe based on the analysis by the RocksDB folks (which we agree with): https://github.com/ceph/ceph/pull/36579 Sage also commented about a potential way forward: https://github.com/ceph/ceph/pull/36579#issuecomment-870884583 "tbh I think the best approach would be to create a new WAL file format that (1) is 4k block aligned and (2) has a header for each block that indicates the generation # for that log file (so we can see whether what we read is from a previous pass or corruption). That would be a fair bit of effort, though." On a side note, Igor tried to also disable WAL file recycling as a backport to Octopus but was thwarted by a BlueFS bug. That PR was eventually reverted leaving the old (dangerous!) behavior being left in place: https://github.com/ceph/ceph/pull/45040 https://github.com/ceph/ceph/pull/47053 The gist of it is that releases of Ceph older than Pacific are benefiting from the speed improvement of log file recycling but may be vulnerable to the issue as described above. This is likely one of the more impactful regressions that people upgrading to Pacific or later releases are seeing. Josh Baergen from Digital Ocean followed up that there is a slew of additional information on this issue in the following tracker as well: https://tracker.ceph.com/issues/58530 ********** Regression #1 Potential Fixes *********** Josh Baergen also mentioned that the write-amplification effect that was observed due to this issue is mitigated in by https://github.com/ceph/ceph/pull/48915 which was merged into 16.2.11 back in December. That however does not improve write IOPS amplification. Beyond that, we could follow Sage's idea and try to implement a new WAL file format. The risks here are that it could be a lot of work and we don't know if there is really any appetite on the RocksDB side to merge something like this upstream. My personal take is that we're already kind of abusing the RocksDB WAL for short lived PG log updates and I'm not thrilled about trying to add further code into RocksDB to try and support our use cases (though there is benefit here that goes beyond Ceph). We already maintain a custom version of RocksDB's LRU cache in our code to tie into our memory autotuning system but it would be really nice to avoid custom code like that in the future. One alternative: Igor Fedetov implemented a prototype WAL inside bluestore itself and we saw very good initial results from it with the RocksDB WAL disabled. These can be seen on slide 24 of my performance deck from Cephalocon 2023: https://www.linkedin.com/in/markhpc/overlay/experience/2113859303/multiple-… If Igor (or others) want to continue this work, I personally would be in favor of trying to move the WAL into Bluestore itself. I suspect we can make better decisions about PG log life cycles and have better BlueFS integration than what RocksDB provides us. Igor probably has a better idea of the pitfalls here though so I think we should hear out his thoughts on whether this is the right path forward. Igor also mentioned that he is continuing to work on his Bluestore WAL prototype with promising results, but that PG Log will (as expected) likely require a different solution that looks more like a specialized ring buffer. I think moving the WAL out of RocksDB is a good step toward that eventual goal. ********** Regression #2: (re-)Enabling BlueFS Buffered IO ********** Effects: Works around unexpected readahead behavior in RocksDB by utilizing underlying kernel page cache. Hurts write performance on fast devices. We're stuck between a bit of a rock and a hard place here. Over the years we have sea-sawed back and forth regarding when we should or should not use buffered IO: https://github.com/ceph/ceph/pull/11012 https://github.com/ceph/ceph/pull/11059 https://github.com/ceph/ceph/pull/18172 https://github.com/ceph/ceph/pull/20542 https://github.com/ceph/ceph/pull/34224 https://github.com/ceph/ceph/pull/38044 <-- lots of discussion here The gist of it is that there are upsides and downside to having bluefs_buffered_io=true. Direct IO is faster in some scenarios, especially more recent write tests on NVMe drives. The trade off is that RocksDB really seems to benefit from kernel buffer cache and there are other scenarios where bluefs_buffered_io is a big win. 2 years ago Adam and I did a walkthrough of the RocksDB code to try to understand the behavior regarding RocksDB readahead and we couldn't understand why it was re-reading data from the file system so often (or in the case of buffered IO the page cache!). I wrote up our walkthrough of the code here: https://github.com/ceph/ceph/pull/38044#issuecomment-790157415 ********** Regression #2 Potential Fixes ********** In a recent discussion with Mark Callaghan (of MyRocks/RocksDB performance tuning fame), he pointed out that RocksDB has an option to pre-populate the block cache with the data from SSTs created by memtable flush and that might help when O_DIRECT is used: https://github.com/facebook/rocksdb/blob/main/include/rocksdb/table.h#L600 We may want to experiment to see if this helps keep the block cache pre-populated after compaction and avoid (re)reads from the disk during iteration. We also might want to revisit this topic in general with the compact on iteration feature that was recently added and backported to pacific in 16.2.13. I'm still a little concerned however that we were seeing repeated overlapping reads for the same ranges during iteration that I would have expected to be cached by RocksDB on a previous read. Ultimately I think many of us would prefer to move entirely to directIO but there's more work to do to figure this one out. Josh Baergen provided further advice here: They have had good luck enabling buffered IO for rgw bucket index OSDs and disabling it everywhere else. This assumes that bucket indexes are on their own dedicated OSDs though, and personally I am a bit wary of hitting slow cases in RocksDB even on "regular" OSDs, but this might be something to consider as they've had good luck with this configuration for over a year. ********** Regression #3: RadosGW Coroutine and Request Timeout Changes ********** Effects: Higher RadosGW CPU usage, lower performance, especially for small object workloads Back when Pacific was released it was observed that RadosGW was showing much higher CPU usage and lower performance vs Nautilus for small (4KB) objects. It's likely that larger objects may be affected, though to a lesser degree A git bisection was performed and the results are summarized in the introduction section of the folliowing RGW performance analysis blog post here: https://ceph.io/en/news/blog/2023/reef-freeze-rgw-performance/ The bisection uncovered two primary PRs that were causing performance regression: https://github.com/ceph/ceph/pull/31580 https://github.com/ceph/ceph/pull/35355 The good news is that once those PRs were identified, the RGW team started working to improve things, especially for #35355: https://github.com/ceph/ceph/pull/43761 <-- Fixes issues introduced in #35355, backported to Pacific in 2022 *********** Regression #3 Potential Fixes ********** Quincy (and due to the backport likely Pacific) is showing significantly better behavior in recent tests due to PR #43761. The effects of #31580 are still present, but are considered a necessary trade-off. Other improvements since then may be helping, but we'll need to continue to make up the difference in other areas and start really investigating where we are spending cycles/time, especially in Reef. ********** Regression #4: Gradually slowing down OSDs ********** Effects: Significant slowdown after 1-2 weeks of OSD runtime Igor Fedetov pointed this one out in discussion earlier today: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/OXAUOK7CQXW… This one is pretty new and there is not much there yet other than perhaps low memory (and cache?) usage despite regular IO workload. Onode misses can absolutely cause performance degradation, but it's not clear yet whether this is memory related issue or something else. More investigation needed. Hopefully we'll get perf data from the users who encountered it to help diagnose what's going on here. ********** Conclusion ********** There may be other performance issues that I'm not remembering, but these are the big ones I can think of off the top of my head at the moment. Hopefully this helps clarify what's going on if people are seeing a regression, what to look for, and if they are hitting it, the why behind it. Thanks, Mark -- Best Regards, Mark Nelson Head of R&D (USA) Clyso GmbH p: +49 89 21552391 12 a: Loristraße 8 | 80335 München | Germany w: https://clyso.com | e: mark.nelson(a)clyso.com We are hiring: https://www.clyso.com/jobs/

2 2

Re: Enable Centralized Logging in Dashboard.
by Vladimir Brik 17 May '23

17 May '23

How do I create a user name and password that I could use to log in to grafana? Vlad On 11/16/22 08:42, E Taka wrote: > Thank you, Nizam. I wasn't aware that the Dashboard login is not the same > as the grafana login. Now I have accass to the logfiles. > > Am Mi., 16. Nov. 2022 um 15:06 Uhr schrieb Nizamudeen A <nia(a)redhat.com>: > >> Hi, >> >> Did you login to the grafana dashboard? For centralized logging you'll >> need to login to the >> grafana using your grafana username and password. If you do that and >> refresh the dashboard, >> I think the Loki page should be visible from the Daemon Logs page. >> >> Regards, >> Nizam >> >> On Wed, Nov 16, 2022 at 7:31 PM E Taka <0etaka0(a)gmail.com> wrote: >> >>> Ceph: 17.2.5, dockerized with Ubuntu 20.04 >>> >>> Hi all, >>> >>> I try to enable the Centralized Logging in Dashboard as described in >>> >>> >>> https://docs.ceph.com/en/quincy/cephadm/services/monitoring/#cephadm-monito… >>> >>> Logging inti files is enabled: >>> ceph config set global log_to_file true >>> ceph config set global mon_cluster_log_to_file true >>> >>> Loki is deployed at one host, promtail on every host: >>> >>> >>> service_type: loki >>> service_name: loki >>> placement: >>> hosts: >>> - ceph00 >>> --- >>> service_type: promtail >>> service_name: promtail >>> placement: >>> host_pattern: '*' >>> >>> >>> After applying the YAML above the log messages in »ceph -W cephadm« look >>> good (deploying loki+promtail and reconfiguring grafana). But the >>> Dashboard >>> "Cluster → Logs → Daemon Logs" just shows a standard grafana page without >>> any buttons for the Ceph Cluster. Its URL is https://ceph00. >>> >>> [mydomain]:3000/explore?orgId=1&left=["now-1h","now","Loki",{"refId":"A"}]&kiosk. >>> >>> Did I miss something for the centralized Logging in the Dashboard? >>> >>> Thank! >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

new install or change default registry to private registry
by farhad kh 17 May '23

17 May '23

i try deploy cluster from private registry and used this command {cephadm bootstrap ---mon-ip 10.10.128.68 --registry-url my.registry.xo --registry-username myuser1 --registry-password mypassword1 --dashboard-password-noupdate --initial-dashboard-password P@ssw0rd } even i changed section Default container images in cephadm script # Default container images ----------------------------------------------------- DEFAULT_IMAGE = ' my.registry.xo/ceph/ceph:v17' DEFAULT_IMAGE_IS_MASTER = False DEFAULT_IMAGE_RELEASE = 'quincy' DEFAULT_PROMETHEUS_IMAGE = ' my.registry.xo/ceph/prometheus:v2.33.4' DEFAULT_LOKI_IMAGE = ' my.registry.xo/ceph/loki:2.4.0' DEFAULT_PROMTAIL_IMAGE = my.registry.xo/ceph/promtail:2.4.0' DEFAULT_NODE_EXPORTER_IMAGE = ' my.registry.xo/ceph/node-exporter:v1.3.1' DEFAULT_ALERT_MANAGER_IMAGE = ' my.registry.xo/ceph/alertmanager:v0.23.0' DEFAULT_GRAFANA_IMAGE = ' my.registry.xo/ceph/ceph-grafana:8.3.5' DEFAULT_HAPROXY_IMAGE = ' my.registry.xo/ceph/haproxy:2.3' DEFAULT_KEEPALIVED_IMAGE = ' my.registry.xo/ceph//keepalived:2.1.5' DEFAULT_SNMP_GATEWAY_IMAGE = ' my.registry.xo/ceph/ snmp-notifier:v1.2.1' DEFAULT_REGISTRY = ' my.registry.xo' # normalize unqualified digests to this # ------------------------------------------------------------------------------ to my private registy but when i deploy new cluster cephadm try pull prometheus , node-exporter and ceph-grafana from quay.io . Why does it happen and what should I do to change it? Even the default registry is not changed #ceph config get mgr mgr/cephadm/default_registry docker.io Can someone guide me, what should I do?

2 1

how to enable multisite resharding feature?
by Alexander Mamonov 17 May '23

17 May '23

https://docs.ceph.com/en/latest/radosgw/multisite/#feature-resharding When I try this I get: root@ceph-m-02:~# radosgw-admin zone modify --rgw-zone=sel --enable-feature=resharding ERROR: invalid flag --enable-feature=resharding root@ceph-m-02:~# ceph version ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)

2 1

CEPH Version choice
by Tino Todino 17 May '23

17 May '23

Hi all, I've been reading through this email list for a while now, but one thing that I'm curious about is why a lot of installations out there aren't upgraded to the latest version of CEPH (Quincy). What are the main reasons for not upgrading to the latest and greatest? Thanks. Tino This E-mail is intended solely for the person or organisation to which it is addressed. It may contain privileged or confidential information and, if you are not the intended recipient, you must not copy, distribute or take any action in reliance upon it. Any views or opinions presented are solely those of the author and do not necessarily represent those of Marlan Maritime Technologies Ltd. If you have received this E-mail in error, please notify us as soon as possible and delete it from your computer. Marlan Maritime Technologies Ltd Registered in England & Wales 323 Mariners House, Norfolk Street, Liverpool. L1 0BG Company No. 08492427.

10 14

Best practice for expanding Ceph cluster
by huxiaoyu＠horebdata.cn 17 May '23

17 May '23

Dear Ceph folks, I am writing to ask for advice on best practice of expanding ceph cluster. We are running an 8-node Ceph cluster and RGW, and would like to add another 10 node, each of which have 10x 12TB HDD. The current 8-node has ca. 400TB user data. I am wondering whether to add 10 nodes at one shot and let the cluster to rebalance, or divide into 5 steps, each of which add 2 nodes and rebalance step by step? I do not know what would be the advantages or disadvantages with the one shot scheme vs 5 bataches of adding 2 nodes step-by-step. Any suggestions, experience sharing or advice are highly appreciated. thanks a lot in advance, Samuel huxiaoyu(a)horebdata.cn

4 6

CFP NOW OPEN: Ceph Days Vancouver Co-located with OpenInfra Summit
by Mike Perez 17 May '23

17 May '23

Hi everyone, I invite you to Ceph Days Vancouver on June 15, co-located with the OpenInfra Summit. Ceph Days are one-day events dedicated to multiple breakout and BoF sessions with a wide range of topics around Ceph. You can receive a nice discount to attend Ceph Days and the OpenInfra Summit if you register before May 12! https://ceph.io/en/community/events/2023/ceph-days-vancouver/ Our Call for proposals is open until May 17, and we have the following suggested topics: - Ceph operations, management, and development - New and proposed Ceph features, development status - Ceph development roadmap - Best practices - Ceph use-cases, solution architectures, and user experiences - Ceph performance and optimization - Platform Integrations - Kubernetes, OpenShift - OpenStack (Cinder, Manila, etc.) - Spark - Multi-site and multi-cluster data services - Persistent memory, ZNS SSDs, SMR HDDs, DPUs, and other new hardware technologies - Storage management, monitoring, and deployment automation - Experiences deploying and operating Ceph in production and/or at scale - Small-scale or edge deployments - Long-term, archival storage - Data compression, deduplication, and storage optimization - Developer processes, tools, challenges - Ceph testing infrastructure, tools - Ceph community issues, outreach, and project governance - Ceph documentation, training, and learner experience CFP: https://survey.zohopublic.com/zs/TVCCCQ See you all there soon! -- Mike Perez Community Manager Ceph Foundation

1 1

FW: Public Access URL returns "NoSuchBucket" when rgw_swift_account_in_url is True
by Beaman, Joshua 17 May '23

17 May '23

Thought I might forward this to the users list in case anyone else is experiencing or knows how to resolve. Thank you, Josh Beaman From: Beaman, Joshua <Joshua_Beaman(a)comcast.com> Date: Tuesday, May 16, 2023 at 3:23 PM To: dev(a)ceph.io <dev(a)ceph.io> Subject: Public Access URL returns "NoSuchBucket" when rgw_swift_account_in_url is True We have a continuing problem with Openstack Swift Public URLs returning “NoSuchBucket” even though the S3 equivalent URL works fine. This was not a problem in our Nautilus 14.2.22 clusters, but after upgrading to 16.2.9 or 16.2.10, this regression bug has returned. All 13 of our prod clusters were upgraded to 16.2.11 and still experience the problem, and now I’ve upgraded 1 prod cluster to Quincy/17.2.6 and it is still present. I’ve submitted the following bug report: https://tracker.ceph.com/issues/58019 Thanks in advance, Josh Beaman

1 0

Bucket-level redirect_zone
by Yixin Jin 16 May '23

16 May '23

Hi folks, I created a feature request ticket to call for bucket-level redirect_zone (https://tracker.ceph.com/issues/61199) which is basically an extension from zone-level redirect_zone. I found it helpful in realizing CopyObject with (x-copy-source) in multisite environments where bucket content don't exist in all zones. This feature is similar to what Matt Benjamin suggested about the concept of "bucket residence". In my own prototyping affect, redirecting feature at bucket level is fairly straightforward. Making use of it with CopyObject is trickier because I haven't found a good way to get the source object policy when RGWCopyObj::verify_permission() is called. Anyway, I will continue to explore this idea and hope to get more support of it. Thanks, Yixin

2 1

User + Dev Monthly Meeting happening this Thursday
by Laura Flores 16 May '23

16 May '23

Hi Ceph Users, The User + Dev Monthly Meeting is coming up next week on *Thursday, May 17th* *@* *14:00 UTC* (time conversions below). See meeting details at the bottom of this email. Please add any topics you'd like to discuss to the agenda: https://pad.ceph.com/p/ceph-user-dev-monthly-minutes See you there, Laura Flores Meeting link: https://meet.jit.si/ceph-user-dev-monthly Time Conversions: UTC: Wednesday, May 17, 14:00 UTC Mountain View, CA, US: Wednesday, May 17, 7:00 PDT Phoenix, AZ, US: Wednesday, May 17, 7:00 MST Denver, CO, US: Wednesday, May 17, 8:00 MDT Huntsville, AL, US: Wednesday, May 17, 9:00 CDT Raleigh, NC, US: Wednesday, May 17, 10:00 EDT London, England: Wednesday, May 17, 15:00 BST Paris, France: Wednesday, May 17, 16:00 CEST Helsinki, Finland: Wednesday, May 17, 17:00 EEST Tel Aviv, Israel: Wednesday, May 17, 17:00 IDT Pune, India: Wednesday, May 17, 19:30 IST Brisbane, Australia: Thursday, May 18, 0:00 AEST Singapore, Asia: Wednesday, May 17, 22:00 +08 Auckland, New Zealand: Thursday, May 18, 2:00 NZST -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 1

rbd mirror snapshot trash
by Eugen Block 16 May '23

16 May '23

Good morning, I would be grateful if anybody could shed some light on this, I can't reproduce it in my lab clusters so I was hoping for the community. A customer has 2 clusters with rbd mirroring (snapshots) enabled, it seems to work fine, they have regular checks and the images on the remote site are synced correctly. While investigating a different issue (could be related though) we noticed that there are quite a lot of snapshots in the trash namespace but for unkown reasons they are not purged. This is some example output from the remote site (the primary site looks similar): ---snip--- # rbd snap ls --all <pool>/<image> SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 23962608 42c2e4c3-e1ab-480f-9d29-aab5555c751b 20 GiB Tue Sep 20 20:33:18 2022 trash (.mirror.non_primary.61ef3a1e-6e5b-4147-ac62-552e4776dd70.ef345a74-2585-48ac-8e25-09b15c20c877) 96796437 24cc96d4-51c6-447e-870d-1545d8ec9308 20 GiB Tue Apr 18 12:15:28 2023 trash (.mirror.non_primary.61ef3a1e-6e5b-4147-ac62-552e4776dd70.a6ae46e4-cbb5-4264-b2e4-6b565820aa16) 110025019 .mirror.non_primary.61ef3a1e-6e5b-4147-ac62-552e4776dd70.0fc73e88-6671-4ba8-a118-8fcbcd422c22 20 GiB Mon May 15 16:27:39 2023 mirror (non-primary peer_uuids:[] 18ac37b9-a6c6-4f64-ba43-a1f0d02b3f96:115986586 copied) ---snip--- They're on latest Octopus and use all config defaults. There are more than 500 images in that pool with a snapshot schedule of 3 minutes for each images. A couple of questions arised and hopefully at least some of them can be answered: - Why are there snapshots in the trash namespace from September 2022, how can they be cleaned up and why aren't they cleaned up automatically? Not all images have trash entries though. - Could disabling mirroring for those images help getting rid of the trash, then enable mirroring again? - Is the default osd_max_trimming_pgs = 2 a bottleneck here? They have a week old sst files in the store.db, leading to long startup duration for MONs after reboot/restart. Would increasing that value help getting rid of the trash entries and maybe also trim the mon store? - Regarding the general snapshot mirroring procedure, the default of rbd_mirroring_max_mirroring_snapshots is 5, but I assume that the number of active snapshots would only grow in case of a disruption between those sites, correct? If the sync works there's no need to keep 5 snapshots, right? I'm still looking into these things myself but I'd appreciate anyone chiming in here. Thanks! Eugen

2 2